diff --git a/ATTRIBUTIONS-Python.md b/ATTRIBUTIONS-Python.md index 93df1c6ef1af..58c4d8440364 100644 --- a/ATTRIBUTIONS-Python.md +++ b/ATTRIBUTIONS-Python.md @@ -15404,7 +15404,7 @@ CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. - `Homepage`: https://github.com/lark-parser/lark -## llguidance (0.7.29) +## llguidance (1.7.6) ### Licenses License: `MIT` diff --git a/docs/source/deployment-guide/deployment-guide-for-mistral-large-4-on-trtllm.md b/docs/source/deployment-guide/deployment-guide-for-mistral-large-4-on-trtllm.md new file mode 100644 index 000000000000..a6b46ee8f468 --- /dev/null +++ b/docs/source/deployment-guide/deployment-guide-for-mistral-large-4-on-trtllm.md @@ -0,0 +1,592 @@ + + +# Deployment Guide for Mistral Large 4 on TensorRT LLM - Blackwell (GB300) + +## Introduction + +This guide describes how to serve Mistral Large 4 with TensorRT LLM on NVIDIA GB300 GPUs. It covers obtaining the checkpoint and its format, the supported deployments, installing the software, launching `trtllm-serve` on one or two nodes, the model-specific API behavior, example requests, accuracy verification and the known limitations. The configurations below are validated on GB300, most of them by the accuracy tests in `tests/integration/defs/accuracy/modeling_v2/mistral_large4/`; configurations outside these tables are not validated. + +This source checkout also supports NVIDIA VR200 (`sm_107`) for all six parallel layouts, with DSpark enabled or disabled. See [VR200 Accuracy Validation](#vr200-accuracy-validation) for the verified scope. The installation and launch recipes below describe GB300; VR200 requires a source checkout containing the SM107 targets. + +Mistral Large 4 is a Mixture-of-Experts model with about 1T parameters, 49B of them active per token (52B including the embedding and output layers). It takes text and image input, supports a context of up to 1,048,576 tokens, and uses latent attention with a sparse top-k indexer. Like Mistral Large 3, it is published on Hugging Face in the Mistral format: `params.json` for the configuration, `tekken.json` for the tokenizer and `consolidated-*.safetensors` for the weights. TensorRT LLM loads its text model as the architecture `MistralLarge4ForCausalLM`; the checkpoint itself names no architecture. The repository also ships Hugging Face tokenizer, chat-template, processor and generation-config files, but no Transformers `config.json`. TensorRT LLM reads the Mistral-format files only; see [Checkpoint Format](#checkpoint-format). + +TensorRT LLM serves Mistral Large 4 through **modeling_v2** targets (`tensorrt_llm/_torch/modeling_v2/models/mistral_large4/`). A modeling_v2 target is one implementation for one combination of checkpoint, GPU architecture and parallel layout. When the model is loaded, TensorRT LLM picks the target from the checkpoint's quantization, the GPU's compute capability, the parallel mapping and the speculative-decoding configuration. There is no other implementation of this architecture, so a deployment that no target claims fails to load. modeling_v2 is a prototype feature. + +## Prerequisites + +* GPU for the deployment recipes below: NVIDIA GB300 (`sm_103`). VR200 (`sm_107`) is also supported by this source checkout, with the validation scope described below. + * The 4-GPU deployments (TEP4, DEP4 and TP4) run on one node with 4 GPUs. + * The 8-GPU deployments (TEP8, DEP8, TP8) run on two nodes with 4 GPUs each. Both nodes must be in the same multi-node NVLink domain. +* Multi-node launcher: Slurm with the pyxis/enroot container plugin and PMIx (`srun --mpi=pmix`), or an equivalent MPI launcher, to start one rank per GPU. +* A shared filesystem visible to all ranks for the checkpoint, the configuration file and, for a source build, the repository. The NVFP4 checkpoint, including its speculative-decoding draft, is about 610 GB. +* Container runtime with NVIDIA GPU support, and TensorRT LLM with Mistral Large 4 support: the release container, the wheel or a source build. See [Install TensorRT LLM](#install-tensorrt-llm). +* Python packages from `requirements.txt`, including `mistral-common[opencv]` 1.13.0 or later, `xgrammar` and `llguidance`. The release container includes them, and installing the wheel installs them. Install mistral-common from PyPI: the checkpoint's tokenizer declares reasoning efforts that earlier releases do not know. Image input needs OpenCV, which the `opencv` extra installs. + +## Models + +* NVFP4 model: [mistralai/Mistral-Large-4-1T-A52B-NVFP4](https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B-NVFP4). This is the checkpoint TensorRT LLM serves. +* FP8 model: [mistralai/Mistral-Large-4-1T-A52B](https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B). Not supported by TensorRT LLM; it is refused at load time. + +| Checkpoint | Weights | Features | Deployments | +| :-- | :-- | :-- | :-- | +| NVFP4 | ModelOpt `MIXED_PRECISION`: NVFP4 routed experts (group size 16), FP8 128x128 block-scaled linear layers, BF16 vision encoder; the DSpark draft layers are part of the checkpoint | Text, images, DSpark speculative decoding, contexts up to 1,048,576 tokens | TEP4, DEP4, TP4, TEP8, DEP8, TP8 | + +### Download the Checkpoint + +Download the NVFP4 repository once to the shared filesystem. It is about 610 GB. + +```bash +hf auth login # if the repository asks for a Hugging Face access token +hf download mistralai/Mistral-Large-4-1T-A52B-NVFP4 --local-dir /path/to/mistral-large-4-nvfp4 +``` + +Pass the downloaded directory to TensorRT LLM as it is; no conversion is needed. The examples below use `/path/to/mistral-large-4-nvfp4`; replace it with your local path. The configurations in this guide are validated with a local directory, not with a Hugging Face model ID. + +### Checkpoint Format + +The repository contains two sets of files. TensorRT LLM reads the Mistral-format set and, from the Hugging Face set, only the sampling defaults: + +| Files | Contents | Read by TensorRT LLM | +| :-- | :-- | :-- | +| `params.json`, `consolidated.safetensors.index.json`, `consolidated-*.safetensors` | Mistral format: model configuration and weights, including the DSpark draft | Yes | +| `tekken.json` | Mistral format: tokenizer, chat format and reasoning efforts, read through mistral-common | Yes | +| `tokenizer.json`, `tokenizer_config.json`, `chat_template.jinja` | Hugging Face tokenizer and chat template | No | +| `processor_config.json` | Hugging Face image processor (`PixtralProcessor`) | No | +| `generation_config.json` | Hugging Face sampling defaults (temperature 1.0, top_p 0.95) | With `generation_config: auto` | + +Every Mistral Large 4 configuration must set `checkpoint_format: mistral`. With it, the model configuration, the weights, the tokenizer, chat rendering and image preprocessing all come from the Mistral-format files and mistral-common, and the Hugging Face tokenizer, chat template and image processor are ignored. Every component of the server that tokenizes, including the post-processing workers, the multimodal input loader and the disaggregated router, uses `tekken.json`. + +Without `checkpoint_format`, TensorRT LLM expects the Hugging Face format, which needs a `config.json`, and fails with an error that names `checkpoint_format='mistral'`. Mistral Large 4 is not served from the Hugging Face format. + +## Supported Configurations + +The validation results in this section describe GB300. VR200 supports the same layouts; its accuracy coverage is listed in [VR200 Accuracy Validation](#vr200-accuracy-validation). + +| Deployment | GPUs | Checkpoint | Parallel settings | Validated `max_seq_len` | +| :-- | --: | :-- | :-- | :-- | +| TEP4 | 4 | NVFP4 | `tensor_parallel_size: 4`, `moe_expert_parallel_size: 4`, `enable_attention_dp: false` | 32768, 1048576 | +| DEP4 | 4 | NVFP4 | `tensor_parallel_size: 4`, `moe_expert_parallel_size: 4`, `enable_attention_dp: true` | 32768, 1048576 | +| TP4 | 4 | NVFP4 | `tensor_parallel_size: 4`, `moe_tensor_parallel_size: 4`, `moe_expert_parallel_size: 1`, `enable_attention_dp: false` | 32768, 1048576 | +| TEP8 | 8 (2 x 4) | NVFP4 | `tensor_parallel_size: 8`, `moe_expert_parallel_size: 8`, `enable_attention_dp: false` | 32768, 1048576 | +| DEP8 | 8 (2 x 4) | NVFP4 | `tensor_parallel_size: 8`, `moe_expert_parallel_size: 8`, `enable_attention_dp: true` | 32768, 1048576 (prompts up to 128K tokens) | +| TP8 | 8 (2 x 4) | NVFP4 | `tensor_parallel_size: 8`, `moe_tensor_parallel_size: 8`, `moe_expert_parallel_size: 1`, `enable_attention_dp: false` | 32768, 1048576 | + +The 8-GPU deployments use the same targets as their 4-GPU counterparts. The 1M-token configurations of TEP8 and TP8 are validated with prompts of up to 1M tokens; DEP8 is validated with prompts of up to 128K tokens, and DEP4 covers 1M-token prompts under attention data parallelism. + +Features of the NVFP4 deployments: + +| Feature | Status | +| :-- | :-- | +| DSpark speculative decoding (3 draft tokens) | Supported; the draft is part of the checkpoint. It is the only speculative-decoding method. | +| Image input | Supported. Images are encoded on every rank. | +| KV cache block reuse (prefix caching) | Supported. | +| Chunked prefill | Supported; required when prompts can be longer than `max_num_tokens`. | +| CUDA graphs with padding | Supported. | +| Structured outputs (`response_format`) | Supported with the `xgrammar` and `llguidance` backends. See [`guided_decoding_backend`](#guided_decoding_backend). | +| Tool calling and reasoning output | Supported through the `mistral_large_4` tool and reasoning parsers. | +| Disaggregated serving | Validated for one TEP4 context server and one TEP4 generation server at `max_seq_len` 32768, without speculative decoding. See [Disaggregated Serving](#disaggregated-serving). | +| Pipeline parallelism, speculative-decoding methods other than DSpark | Not supported. | +| LoRA, beam search | Not validated. | + +## Deployment Steps + +### Install TensorRT LLM + +Mistral Large 4 support is released from the `feat/mistral-large-4` branch as TensorRT LLM `1.4.0a0.dev202610310501`, both as a release container and as a wheel. Use one of them, or build the branch from source. + +#### Release Container + +The release container has TensorRT LLM and its dependencies installed: + +```text +nvcr.io/nvidia/tensorrt-llm/release:feat-mistral-large-4-1.4.0a0.dev202610310501 +``` + +The tag covers x86_64 and aarch64; GB300 nodes use the aarch64 image. With Slurm and pyxis/enroot, import it once into a squashfs file on the shared filesystem: + +```bash +enroot import -o /path/to/tensorrt-llm-mistral-large-4.sqsh \ + "docker://nvcr.io#nvidia/tensorrt-llm/release:feat-mistral-large-4-1.4.0a0.dev202610310501" +``` + +The launch commands below use this file as the container image. Other container runtimes can use the tag directly. + +#### Wheel + +The wheel is built for the NGC PyTorch container `nvcr.io/nvidia/pytorch:26.09-py3` and published to a preview package index. Inside that container: + +```bash +pip3 install "tensorrt_llm==1.4.0a0.dev202610310501" \ + --extra-index-url https://pypi.nvidia.com/trtllm_ngcpytorch_preview/ +``` + +The launch commands start every rank in a fresh container, so save the container after the installation and use the saved image instead of the release container. With pyxis, on a GB300 node: + +```bash +srun -N 1 --ntasks 1 \ + --container-image=nvcr.io#nvidia/pytorch:26.09-py3 \ + --container-save=/path/to/tensorrt-llm-mistral-large-4.sqsh \ + pip3 install "tensorrt_llm==1.4.0a0.dev202610310501" \ + --extra-index-url https://pypi.nvidia.com/trtllm_ngcpytorch_preview/ +``` + +See [Install in an NGC PyTorch container](../installation/installation-guide.md#install-in-an-ngc-pytorch-container) for the general procedure. + +#### Build from Source + +Build in the TensorRT LLM development container built from `docker/Dockerfile.multi` (base `nvcr.io/nvidia/pytorch:26.09-py3`), using the SBSA (aarch64) variant on GB300. From the repository root: + +```bash +python3 scripts/build_wheel.py --use_ccache -a "103-real" --skip_building_wheel --install --yes +``` + +`-a "103-real"` builds for GB300. `--install` installs TensorRT LLM in editable mode into the virtual environment that `build_wheel.py` creates at the repository root, named after the container's Python version: `.venv-3.12` for the current containers. The source-build launch commands below put its `bin` directory first on `PATH`, so build with the repository at the same path the servers use. With `--build_root`, the virtual environment is created under the build root instead. See [Building from Source](../installation/build-from-source.md) for the other options. + +### Configuration File + +Create a YAML configuration file on the shared filesystem. The configuration below is the tested TEP4 configuration with DSpark speculative decoding at a 32K-token context: + +```shell +EXTRA_LLM_API_FILE=/path/to/mistral-large-4-config.yml + +cat << EOF > ${EXTRA_LLM_API_FILE} +checkpoint_format: mistral +modeling_v2: require +tensor_parallel_size: 4 +moe_expert_parallel_size: 4 +enable_attention_dp: false +max_batch_size: 16 +max_num_tokens: 32768 +max_seq_len: 32768 +enable_chunked_prefill: false +cuda_graph_config: + enable_padding: true + max_batch_size: 16 +kv_cache_config: + enable_block_reuse: true + free_gpu_memory_fraction: 0.85 +speculative_config: + decoding_type: DSpark + max_draft_len: 3 + speculative_model: /path/to/mistral-large-4-nvfp4 +guided_decoding_backend: xgrammar +EOF +``` + +Change these keys for the other deployments: + +| Deployment | Changes to the configuration above | +| :-- | :-- | +| DEP4 | `enable_attention_dp: true`. `max_batch_size` and `cuda_graph_config.max_batch_size` then apply to each rank. | +| TP4 | `moe_expert_parallel_size: 1` and `moe_tensor_parallel_size: 4` | +| TEP8, DEP8, TP8 | The TEP4, DEP4 or TP4 settings with `tensor_parallel_size: 8`, and `moe_expert_parallel_size: 8` (TEP8, DEP8) or `moe_tensor_parallel_size: 8` (TP8) | +| 1M-token context | `max_seq_len: 1048576` and `enable_chunked_prefill: true` | +| Without speculative decoding | Remove `speculative_config`. | + +Always set `max_seq_len`. Without it the engine sizes the KV cache for the checkpoint's 1,048,576-token limit. + +### Launch the TensorRT LLM Server + +Mistral Large 4 is launched through the `trtllm-llmapi-launch` wrapper, one rank per GPU. Rank 0 serves HTTP and the other ranks become workers. The example below starts the 4-GPU server with Slurm in the image from [Install TensorRT LLM](#install-tensorrt-llm): + +```bash +MODEL=/path/to/mistral-large-4-nvfp4 +IMAGE=/path/to/tensorrt-llm-mistral-large-4.sqsh + +srun -N 1 --ntasks 4 --ntasks-per-node 4 \ + --mpi=pmix --gpus-per-node=4 \ + --container-image=${IMAGE} \ + --container-mounts=${MODEL}:${MODEL},${EXTRA_LLM_API_FILE}:${EXTRA_LLM_API_FILE}:ro \ + trtllm-llmapi-launch trtllm-serve ${MODEL} \ + --config ${EXTRA_LLM_API_FILE} \ + --host 0.0.0.0 --port 8000 \ + --tool_parser mistral_large_4 \ + --reasoning_parser mistral_large_4 +``` + +For a source build, use the development container, mount the repository and put its virtual environment first on `PATH`: + +```bash +REPO=/path/to/TensorRT-LLM # the repository, built as in Build from Source + +srun -N 1 --ntasks 4 --ntasks-per-node 4 \ + --mpi=pmix --gpus-per-node=4 \ + --container-image=/path/to/tensorrt-llm-devel-container.sqsh \ + --container-mounts=${REPO}:${REPO},${MODEL}:${MODEL},${EXTRA_LLM_API_FILE}:${EXTRA_LLM_API_FILE}:ro \ + bash -c " + export PATH=\"${REPO}/.venv-3.12/bin:\$PATH\" + exec trtllm-llmapi-launch trtllm-serve ${MODEL} \ + --config ${EXTRA_LLM_API_FILE} \ + --host 0.0.0.0 --port 8000 \ + --tool_parser mistral_large_4 \ + --reasoning_parser mistral_large_4 + " +``` + +Outside Slurm, start the same command under MPI on one node, inside the container. For a source build, first run `export PATH="${REPO}/.venv-3.12/bin:$PATH"`. + +```bash +mpirun -n 4 trtllm-llmapi-launch trtllm-serve ${MODEL} \ + --config ${EXTRA_LLM_API_FILE} \ + --host 0.0.0.0 --port 8000 \ + --tool_parser mistral_large_4 \ + --reasoning_parser mistral_large_4 +``` + +For the 8-GPU deployments, use an 8-GPU configuration file and start 8 ranks on two nodes, for example `srun -N 2 --ntasks 8 --ntasks-per-node 4 --mpi=pmix --gpus-per-node=4 ...` with the same command. Both nodes must be in one multi-node NVLink domain (on Slurm clusters with topology segments, request both nodes in one segment, e.g. `--segment=2`). + +> [!NOTE] +> Pass `--tool_parser mistral_large_4` and `--reasoning_parser mistral_large_4` by name. `auto` cannot detect them, because it reads a Hugging Face `config.json`, which the checkpoint does not have. A command-line option that is also set in the YAML file overrides the YAML value. + +Startup is dominated by weight loading and depends mostly on the filesystem. On a networked filesystem, the validated servers became ready in 6 to 13 minutes, and the TP8 deployment in about 23 minutes. The server is ready when the following log is shown: + +```text +INFO: Application startup complete. +INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit) +``` + +### Quick Start Without a Server (LLM API) + +The script below runs the TEP4 configuration through the LLM API. Save it as `quickstart.py` in a directory on the shared filesystem: + +```python +from tensorrt_llm import LLM, SamplingParams +from tensorrt_llm.llmapi import (CudaGraphConfig, DSparkDecodingConfig, + KvCacheConfig) + + +def main(): + model = "/path/to/mistral-large-4-nvfp4" + with LLM( + model, + checkpoint_format="mistral", + modeling_v2="require", + tensor_parallel_size=4, + moe_expert_parallel_size=4, + enable_attention_dp=False, + max_batch_size=16, + max_num_tokens=32768, + max_seq_len=32768, + enable_chunked_prefill=False, + cuda_graph_config=CudaGraphConfig(max_batch_size=16, + enable_padding=True), + kv_cache_config=KvCacheConfig(enable_block_reuse=True, + free_gpu_memory_fraction=0.85), + speculative_config=DSparkDecodingConfig(max_draft_len=3, + speculative_model=model), + ) as llm: + outputs = llm.generate(["The capital of France is"], + SamplingParams(max_tokens=32, temperature=0)) + print(outputs[0].outputs[0].text) + + +if __name__ == "__main__": + main() +``` + +Run it with one rank per GPU under `trtllm-llmapi-launch`, as for the server. With Slurm: + +```bash +WORKDIR=/path/to/workdir # the directory that holds quickstart.py + +srun -N 1 --ntasks 4 --ntasks-per-node 4 \ + --mpi=pmix --gpus-per-node=4 \ + --container-image=${IMAGE} \ + --container-mounts=${WORKDIR}:${WORKDIR},${MODEL}:${MODEL} \ + trtllm-llmapi-launch python3 ${WORKDIR}/quickstart.py +``` + +For a source build, make the same changes as for the server: the development container, the repository mount and the virtual environment on `PATH`. + +Outside Slurm, `python3 quickstart.py` on a node with 4 GPUs also works: the LLM API then starts its worker ranks itself through MPI. Inside a Slurm job step, MPI cannot start them, so use `trtllm-llmapi-launch`. For the 8-GPU deployments, start 8 ranks on two nodes, as for the server. + +### Key YAML Options + +These options are set in the YAML file passed to `trtllm-serve --config`. See the [`TorchLlmArgs` class](https://nvidia.github.io/TensorRT-LLM/llm-api/reference.html#tensorrt_llm.llmapi.TorchLlmArgs) for the full reference. + +#### `checkpoint_format` + +* **Description:** Must be `mistral`, which reads the checkpoint in the Mistral format (see [Checkpoint Format](#checkpoint-format)). The default, the Hugging Face format, fails with an error that names `checkpoint_format='mistral'`. + +#### `modeling_v2` + +* **Description:** Set it to `require`. With the default, `off`, loading fails with `Unknown architecture for AutoModelForCausalLM: MistralLarge4ForCausalLM`. With `require`, a deployment that no target claims fails at load time with a message listing what was checked (compute capability, parallel mapping, speculative decoding and quantization) and which deployments the targets serve. `auto` serves the supported deployments as well, but for a deployment that no target claims it falls back to the default path and fails with the `Unknown architecture` error, which does not say why; use `require`. + +#### `tensor_parallel_size`, `moe_expert_parallel_size`, `moe_tensor_parallel_size`, `enable_attention_dp` + +* **Description:** Select the deployment, as listed in [Supported Configurations](#supported-configurations). `tensor_parallel_size` must be 4 or 8, and the world size must equal it: pipeline parallelism is not supported. + +#### `max_seq_len`, `max_num_tokens`, `enable_chunked_prefill` + +* **Description:** The validated values are `max_num_tokens: 32768` with either `max_seq_len: 32768` and chunked prefill off, or `max_seq_len: 1048576` and chunked prefill on. + +#### `max_batch_size` and `cuda_graph_config` + +* **Description:** The accuracy tests use `max_batch_size: 16` (per rank with attention DP) with CUDA graphs captured up to the same batch size and `enable_padding: true`. Larger batch sizes run, but are not covered by the accuracy tests. + +#### `kv_cache_config` + +* **Options:** + * `dtype`: Leave unset. Mistral Large 4 keeps its KV cache in the `fp8_ds_mla` format: an FP8 latent cache with its scales and a BF16 rotary part, plus FP8 indexer keys. `fp8` and `fp8_ds_mla` are also accepted; other values are rejected. + * `free_gpu_memory_fraction`: `0.85` is the validated value. + * `enable_block_reuse`: Prefix caching across requests. Validated on and off. + * `tokens_per_block`: Leave at the default (32). +* The KV cache manager V2 (`use_kv_cache_manager_v2`, on by default) is required. + +#### `speculative_config` + +* **Description:** DSpark speculative decoding with the draft carried in the NVFP4 checkpoint. `decoding_type` must be `DSpark`, `max_draft_len` must be `3` (the depth the draft was trained with), and `speculative_model`, if set, must be the checkpoint directory itself. Do not set `max_concurrency` or a draft-length schedule. + +#### `guided_decoding_backend` + +* **Description:** The guided-decoding backend: `xgrammar` or `llguidance`. It constrains structured outputs (`response_format`), and with `llguidance` also `tool_choice: "required"` and named tool calls. Without speculative decoding, Mistral Large 4 defaults to `xgrammar`. **With speculative decoding, guided decoding stays off unless this key is set**, and requests with `response_format` are then not constrained. With `xgrammar`, a JSON response starts at the first generated token, so no reasoning is returned with it; with `llguidance`, the JSON may follow a reasoning section. + +### Mistral Large 4 API Behavior + +* **Prompt rendering:** Chat messages are rendered by mistral-common from the checkpoint's `tekken.json`, not by a Jinja chat template; the checkpoint's `chat_template.jinja` is not used. A server-level `--chat_template` is ignored with a warning, and a request-level `chat_template` returns HTTP 400. Of `chat_template_kwargs`, only `reasoning_effort` and `continue_final_message` are read; other keys are ignored with a warning. Use the top-level `response_format` for structured outputs. +* **Reasoning:** `reasoning_effort` accepts `low`, `medium` and `high`. Unset, the checkpoint's default, `high`, applies, so the model reasons before it answers. `none` and other values return HTTP 400. The effort can be given at the top level of the request or in `chat_template_kwargs`; when both are given they must agree. With `--reasoning_parser mistral_large_4`, the reasoning text is returned in `reasoning_content` and the answer in `content`. `thinking_token_budget` is not supported. +* **Tool calling:** With `--tool_parser mistral_large_4`, tool calls are returned in `tool_calls`. `tool_choice: "auto"` works with every configuration. `tool_choice: "required"` and a named function need `guided_decoding_backend: llguidance`; the call and its arguments are then constrained to the declared tools. Otherwise these requests return HTTP 400, as they do when combined with `response_format`. Under `auto`, tools with `strict: true` are accepted, but their arguments are not constrained. For `tool_choice: "none"`, see [Known Limitations](#known-limitations). +* **Images:** The deployments accept images as `image_url` content parts, as data URLs (PNG or JPEG; the tested form) or as `http(s)` URLs that the server downloads. URLs that resolve to private addresses are refused unless `TRTLLM_MEDIA_ALLOW_PRIVATE_URLS=1` is set on the server; keep it unset in production. The server sets no limit on the number of images per prompt (prompts with ten 1540x1540 images have been tested), so enforce one in front of the server if you need it. All image tokens count against `max_seq_len`. Video, audio and precomputed image embeddings are not supported. +* **Sampling:** The model card recommends `temperature` 1.0 and `top_p` 0.95 at every reasoning effort, the values in the checkpoint's `generation_config.json`. Set `generation_config: auto` in the configuration to apply them to every request that sets neither, or set both in each request. Otherwise a chat request that sets neither samples with the server defaults, temperature 1.0 and top_p 1.0; an LLM API `SamplingParams` that sets no `temperature`, `top_p` or `top_k` decodes greedily. The examples below use `temperature: 0` for reproducible output. +* **Output length:** On `/v1/chat/completions`, a `max_tokens` or `max_completion_tokens` larger than `max_seq_len` returns HTTP 400. Prompts longer than `max_num_tokens` return HTTP 400 when chunked prefill is off. +* **Log probabilities:** Requests for `logprobs` or `top_logprobs` return HTTP 400 when speculative decoding is enabled. + +## Testing API Endpoint + +The server runs on the rank-0 node and listens on port `8000`. Send requests to that node's hostname; `localhost` only works on the rank-0 node itself. + +### Health Check + +```shell +curl -s -o /dev/null -w "Status: %{http_code}\n" "http://localhost:8000/health" +``` + +### Basic Chat + +```shell +curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ + "model": "/path/to/mistral-large-4-nvfp4", + "messages": [{"role": "user", "content": "What is the capital of France?"}], + "max_tokens": 256, + "temperature": 0 +}' -w "\n" +``` + +The response contains the answer in `choices[0].message.content` and, because the default reasoning effort is `high`, the model's reasoning in `choices[0].message.reasoning_content`. Add `"stream": true` to receive the answer as server-sent events. + +### Reasoning + +```shell +curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ + "model": "/path/to/mistral-large-4-nvfp4", + "messages": [{"role": "user", "content": "What is 17 * 23? Give the final number."}], + "reasoning_effort": "low", + "max_tokens": 2048, + "temperature": 0 +}' -w "\n" +``` + +`reasoning_effort` takes `low`, `medium` or `high` (the default). The reasoning is returned in `choices[0].message.reasoning_content` and the answer in `choices[0].message.content`. Leave room in `max_tokens` for the reasoning. + +### Tool Calling + +```shell +curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ + "model": "/path/to/mistral-large-4-nvfp4", + "messages": [{"role": "user", "content": "What is the weather in Paris?"}], + "tools": [{ + "type": "function", + "function": { + "name": "get_weather", + "description": "Get the current weather for a city.", + "parameters": { + "type": "object", + "properties": { + "city": {"type": "string", "description": "City name, e.g. Paris"}, + "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]} + }, + "required": ["city"] + } + } + }], + "tool_choice": "auto", + "max_tokens": 256, + "temperature": 0 +}' -w "\n" +``` + +The call is returned in `choices[0].message.tool_calls`, with `finish_reason` set to `tool_calls`. To continue the conversation, append the assistant message with its `tool_calls` and a `tool` message with the matching `tool_call_id` and the result. + +### Structured Output + +```shell +curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ + "model": "/path/to/mistral-large-4-nvfp4", + "messages": [{"role": "user", "content": "Describe a fictional software engineer."}], + "response_format": { + "type": "json_schema", + "json_schema": { + "name": "person", + "strict": true, + "schema": { + "type": "object", + "properties": { + "name": {"type": "string"}, + "age": {"type": "integer"}, + "languages": {"type": "array", "items": {"type": "string"}} + }, + "required": ["name", "age", "languages"], + "additionalProperties": false + } + } + }, + "max_tokens": 256, + "temperature": 0 +}' -w "\n" +``` + +`choices[0].message.content` holds a JSON object that matches the schema. `{"type": "json_object"}` requests any JSON object. With the `xgrammar` backend of the configuration above, the JSON starts at the first generated token and no reasoning is returned. With `llguidance`, the JSON may follow a reasoning section, which is returned in `reasoning_content` and counts against `max_tokens`; raise `max_tokens` or lower `reasoning_effort` so that the JSON is not cut off. + +### Image Input + +```shell +IMAGE_B64=$(base64 -w0 /path/to/image.png) + +curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ + "model": "/path/to/mistral-large-4-nvfp4", + "messages": [{ + "role": "user", + "content": [ + {"type": "image_url", "image_url": {"url": "data:image/png;base64,'"${IMAGE_B64}"'"}}, + {"type": "text", "text": "Describe this image in one sentence."} + ] + }], + "max_tokens": 256, + "temperature": 0 +}' -w "\n" +``` + +Several images can be passed in one message; they are seen in the order given. + +## Disaggregated Serving + +Disaggregated serving is validated with one context server and one generation server, each a TEP4 deployment on its own GB300 node, with the NVFP4 checkpoint without speculative decoding, at `max_seq_len` 32768. The KV cache is transferred with NIXL. Both servers use this configuration: + +```yaml +checkpoint_format: mistral +modeling_v2: require +tensor_parallel_size: 4 +moe_expert_parallel_size: 4 +enable_attention_dp: false +max_batch_size: 16 +max_num_tokens: 32768 +max_seq_len: 32768 +enable_chunked_prefill: false +cuda_graph_config: + enable_padding: true + max_batch_size: 16 +kv_cache_config: + enable_block_reuse: true + free_gpu_memory_fraction: 0.85 +cache_transceiver_config: + backend: NIXL + transceiver_runtime: PYTHON +``` + +Start each server as in [Launch the TensorRT LLM Server](#launch-the-tensorrt-llm-server), one per node, with `--port 8001`, and then the disaggregated router with the two server addresses: + +```yaml +# disagg_config.yaml +hostname: 0.0.0.0 +port: 8000 +context_servers: + num_instances: 1 + urls: + - ":8001" +generation_servers: + num_instances: 1 + urls: + - ":8001" +``` + +```bash +trtllm-serve disaggregated -c disagg_config.yaml --server_start_timeout 1800 +``` + +See [Disaggregated Serving](../features/disagg-serving.md) for the general setup. Other layouts, the 1M-token configuration, speculative decoding and image input with disaggregated serving are not validated. + +## Running Evaluations to Verify Accuracy (Optional) + +The accuracy tests in `tests/integration/defs/accuracy/modeling_v2/mistral_large4/` (one directory per target, the shared harness in `_common.py`) build the configurations above through the LLM API and compare the scores with the references in `tests/integration/defs/accuracy/references/` (`gsm8k.yaml`, `gpqa_diamond.yaml`, `mmmu.yaml`, `ruler.yaml` and `acceptance_length.yaml`). They read the checkpoint from `${LLM_MODELS_ROOT}/Mistral-Large-4-1T-A52B-NVFP4`, the [downloaded](#download-the-checkpoint) NVFP4 repository, and the datasets from `LLM_MODELS_ROOT`. The tests are not part of the release container or the wheel: run them from a checkout of this branch with its development requirements (`requirements-dev.txt`), as in the [source build](#build-from-source) environment. On a node with 4 GB300 GPUs: + +```bash +cd tests/integration/defs +export LLM_MODELS_ROOT=/path/to/models + +# GSM8K on the NVFP4 TEP4 deployment, without speculative decoding. +pytest -v "accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_accuracy.py::TestModelingV2MistralLarge4Nvfp4Sm103Tep::test_gsm8k[tep4]" + +# The quick profile on TEP4 with DSpark: GSM8K, RULER, image and acceptance-length checks. +pytest -v "accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_accuracy.py::TestModelingV2MistralLarge4Nvfp4Sm103Tep::test_accuracy[tep4-quick]" +``` + +`test_accuracy` is parameterized by profile and deployment, as `test_accuracy[-]`: + +| Profile | Deployments | Contents | +| :-- | :-- | :-- | +| `quick` | `tep4`, `dep4`, `tp4`, `tep8`, `dep8`, `tp8` and their `_1m` variants | GSM8K, a reduced RULER set, a reduced MMMU set, image, structured-output and cache checks, DSpark acceptance length | +| `full` | the same | The quick checks with the full RULER and MMMU sets, more image cases, GPQA Diamond, and GSM8K with speculative decoding off | +| `long_lite`, `long_full` | the `_1m` deployments | RULER at 64K, 128K and 1M tokens, cache and retrieval checks over prompts of up to 1M tokens, then the image checks and GSM8K | + +GPQA Diamond and MMMU run with `reasoning_effort: low`. + +The 8-GPU cases run on two nodes with 4 GPUs each, under `trtllm-llmapi-launch`. The test lists `tests/integration/test_lists/test-db/l0_gb300_multi_gpus.yml` and `l0_gb300_multi_nodes_node2_gpu8.yml` show the cases that CI runs. + +### VR200 Accuracy Validation + +The gate classes `TestModelingV2MistralLarge4Nvfp4Sm107Tep`, `TestModelingV2MistralLarge4Nvfp4Sm107Dep` and `TestModelingV2MistralLarge4Nvfp4Sm107Tp`, in the `mistral_large_4_nvfp4__sm_107__*` directories of `tests/integration/defs/accuracy/modeling_v2/mistral_large4/`, define the VR200 accuracy configurations: `test_accuracy` runs with DSpark and `test_accuracy_without_dspark` without it. Validation covered 24 base configurations: TEP4, TP4, DEP4, TEP8, TP8 and DEP8, each at `max_seq_len` 32768 and 1048576, with DSpark enabled and disabled: + +* The six TEP4, TP4 and DEP4 configurations at `max_seq_len` 1048576 each passed `full` and `long_full`, for 12 panels covering GSM8K, MMMU, GPQA Diamond, RULER, image, cache and execution-coverage checks. +* The other 18 configurations each passed the complete 1,319-question GSM8K evaluation. + +All 30 workloads passed with the existing accuracy references and tolerances unchanged. Each core `long_full` panel includes three fixed 1M-token NIAH/VT retrieval cases, in addition to the 64K and 128K sets. The 8-GPU 1M configurations were validated with GSM8K only, not with long prompts. The separate llguidance and disaggregated-serving recipes have not been validated on VR200. + +## Known Limitations + +* **GPU architectures.** GB300 (`sm_103`) and VR200 (`sm_107`) are supported. Other GPUs, including GB200 and B200, are not supported. +* **NVFP4 checkpoint only.** Other exports of Mistral Large 4, including the FP8 checkpoint, are refused at load time. +* **Mistral format only.** The checkpoint is read through `checkpoint_format: mistral`; the Hugging Face format is not supported. The Hugging Face tokenizer, chat template and image processor in the repository are not read. +* **Responses API.** Use `/v1/chat/completions`. On this revision `/v1/responses` drops image inputs, and with speculative decoding every `/v1/responses` request returns HTTP 400 because the request asks for log probabilities. +* **Streamed requests that fail validation.** A streamed request that the engine rejects after the response has started (for example, with chunked prefill on, a prompt longer than `max_seq_len`; or, with speculative decoding, a request for log probabilities) receives HTTP 200 and then a closed connection instead of an error status. The same request without streaming returns HTTP 400. +* **`tool_choice: "none"`.** The tools are still shown to the model. If it calls one anyway, the call is dropped and the response has empty content. To get a text answer, send the request without `tools`. +* **`tool_choice: "required"` and named tool calls** need `guided_decoding_backend: llguidance`. Under `auto`, tools with `strict: true` are not grammar-constrained. +* **`reasoning_effort`.** `low`, `medium` and `high` only; reasoning cannot be turned off. +* **OpenAI `store` field.** The server keeps no completions: chat requests with `store: true` return HTTP 400. `store: false` is accepted. +* **Log probabilities** are not available with speculative decoding. +* **Speculative decoding.** Only DSpark with the draft in the NVFP4 checkpoint, with 3 draft tokens. Guided decoding must be enabled explicitly with it (see `guided_decoding_backend` in [Key YAML Options](#key-yaml-options)). +* **Image count.** The server does not limit the number of images per prompt. +* **Long contexts.** A 1M-token request holds about 46 GiB of KV cache on each rank, so the 4-GPU deployments hold only one or two such requests at a time. TEP and TP keep a full copy of the KV cache on each rank; for many concurrent requests, use DEP. +* **Disaggregated serving.** Validated on GB300 only for TEP4 to TEP4 at a 32K-token context without speculative decoding. +* **Determinism.** Greedy decoding is not bitwise reproducible across runs and batch compositions; scores vary by a few tenths of a point between runs. + +## Troubleshooting Tips + +| Symptom | Cause and fix | +| :-- | :-- | +| `... is a Mistral-Large-4 checkpoint in the Mistral-native format; load it with checkpoint_format='mistral'` | Set `checkpoint_format: mistral`. | +| `Mistral-Large-4 is served from the ModelOpt MIXED_PRECISION export, whose params.json states quantization_config; found quantization=...` | The checkpoint is not the NVFP4 checkpoint. Only the NVFP4 checkpoint is supported. | +| `The tekken.json of ... declares reasoning efforts the installed mistral-common does not know` | Install mistral-common 1.13.0 or later from PyPI. | +| `Unknown architecture for AutoModelForCausalLM: MistralLarge4ForCausalLM` | Set `modeling_v2: require`. | +| `modeling_v2='require', but no target fits 'MistralLarge4ForCausalLM': ...` | The deployment is not supported. The message lists what was checked; compare the GPU, the parallel settings, the speculative-decoding configuration and the checkpoint with [Supported Configurations](#supported-configurations). | +| `Cannot auto-detect tool parser ...` or `Cannot auto-detect reasoning parser ...` | Pass `--tool_parser mistral_large_4` and `--reasoning_parser mistral_large_4` by name. | +| `the Mistral-Large-4 latent cache is fp8_ds_mla ...` | Leave `kv_cache_config.dtype` unset, or set it to `fp8` or `fp8_ds_mla`. | +| `DSA for Mistral-Large-4 requires KV cache manager V2` | Leave `use_kv_cache_manager_v2` unset, or set it to `true`. | +| `Mistral DSpark max_draft_len must equal the trained block depth 3` or `Native Mistral DSpark must use the target checkpoint` | Set `max_draft_len: 3`, and set `speculative_model` to the checkpoint directory or leave it unset. | +| Warning `max_seq_len ... is greater than max_num_tokens ... that can be allocated in kv cache manager, setting max_seq_len to ...` | Set `max_seq_len` explicitly; unset, it defaults to 1,048,576. | +| Image requests fail with an error about OpenCV (`cv2`) | Install `mistral-common[opencv]`, or `opencv-python-headless`. | +| Sampled outputs differ from another serving stack for the same request | The other stack may apply the checkpoint's `generation_config.json` (top_p 0.95); TensorRT LLM applies it only with `generation_config: auto`. Set that, or set `temperature` and `top_p` in the request. | +| CUDA out-of-memory errors at startup | Keep `free_gpu_memory_fraction` at 0.85 or lower, and check `max_batch_size` and `max_seq_len`. | +| The 8-GPU server does not start, or ranks cannot reach each other | Check that both nodes are in one multi-node NVLink domain and that the container image, the checkpoint, the configuration file and, for a source build, the checkout resolve to the same paths on both nodes. | diff --git a/docs/source/deployment-guide/index.rst b/docs/source/deployment-guide/index.rst index de135286934c..bdfffffa86e4 100644 --- a/docs/source/deployment-guide/index.rst +++ b/docs/source/deployment-guide/index.rst @@ -41,3 +41,4 @@ The deployment guides below provide more detailed instructions for serving speci deployment-guide-for-glm-5-on-trtllm.md deployment-guide-for-glm-5.3-flash-on-trtllm.md deployment-guide-for-minimax-m3-on-trtllm.md + deployment-guide-for-mistral-large-4-on-trtllm.md diff --git a/docs/source/models/supported-models.md b/docs/source/models/supported-models.md index 468e6021d3a7..f64c58f426bb 100644 --- a/docs/source/models/supported-models.md +++ b/docs/source/models/supported-models.md @@ -38,6 +38,7 @@ The following is a table of supported models for the PyTorch backend: | `MiniCPMV4_6ForConditionalGeneration` [^14]| MiniCPM-V 4.6 | `openbmb/MiniCPM-V-4.6` | | `MiniMaxM3SparseForConditionalGeneration` [^12]| MiniMax-M3 | `MiniMaxAI/MiniMax-M3` | | `MistralForCausalLM` | Mistral | `mistralai/Mistral-7B-v0.1` | +| `MistralLarge4ForCausalLM` [^26] | Mistral Large 4 | `mistralai/Mistral-Large-4-1T-A52B-NVFP4` | | `MixtralForCausalLM` | Mixtral | `mistralai/Mixtral-8x7B-v0.1` | | `MllamaForConditionalGeneration` | Llama 3.2 | `meta-llama/Llama-3.2-11B-Vision` | | `MuseSparkForConditionalGeneration` [^25] | Muse Spark Open | `meta-models/Muse-Spark-Open-816B-A42B-NVFP4` | @@ -109,6 +110,7 @@ statuses for the same architecture in the two matrices. | `Gemma4UnifiedForConditionalGeneration` | Yes | Yes | Yes | No | Yes | No | Yes | No | Yes | Yes | Yes | | `Step3p7ForConditionalGeneration`| Yes | Yes | Yes | Untested | Untested | MTP | Yes | No | Yes | Yes | Yes | | `MiniMaxM3SparseForConditionalGeneration` [^12] | Yes | Yes | Yes | Untested | Untested | EAGLE-3 (Linear), DSpark [^21] | Yes | No | N/A | Yes | Yes | +| `MistralLarge4ForCausalLM` [^26] | Yes | Yes | Yes | Yes [^27] | Yes | DSpark | Yes | Yes | N/A | Untested | Yes | [^1]: Chunked Prefill for MLA can only be enabled on SM90/SM100/SM103/SM107/SM120. [^2]: KV cache reuse for MLA can only be enabled on SM90/SM100/SM103/SM107/SM120/SM121 and in BF16/FP8 KV cache dtype. @@ -131,6 +133,8 @@ statuses for the same architecture in the two matrices. [^23]: Text-only decision model. `trtllm-serve embeddings` detects the native checkpoint and selects the FlashInfer attention backend automatically. Python `LLM.encode` callers must set `model_kwargs.architectures` to `[PplxDeciderForDecision]`. The embedding vector contains the 255 raw decision logits; apply the checkpoint's calibration and candidate normalization in the caller. [^24]: DeepSeek-V4.1 uses CSA2 sparse attention; see [DeepSeek-V4.1 CSA2](../features/sparse-attention.md#deepseek-v41-csa2). Checkpoints with candidate-consumer layers, including the released one, run only on SM100-family GPUs with a DeepGEMM build that includes the sparse MQA-logits kernels; otherwise model construction fails. Checkpoints without such layers can also use the SM90 and SM120/SM121 attention routes. The Engram tables live in pinned host memory, placed on a GPU's host NUMA node when the GPUs report one. When that node lacks free memory, model loading evicts the node's page cache, which every process on the node loses, and retries. Set `TRTLLM_ENGRAM_EVICT_PAGE_CACHE=0` to skip the eviction; such a node's tables then fall back to host memory that can span NUMA nodes, which can slow lookups. [^25]: Muse Spark Open is supported on NVIDIA Blackwell GPUs (`SM100` family); the validated deployment is TP4/EP4 on one 4-GPU GB300 NVL node — see the [Muse Spark Open deployment guide](../deployment-guide/deployment-guide-for-musespark-open-on-trtllm.md). Supports text, image, video, and audio inputs. On `SM103`, chunked prefill and KV cache reuse require the FP8 KV cache dtype (head-dimension-64 paged-context FMHA constraint); disaggregated serving runs only on the NIXL cache-transceiver backend with the Python transceiver runtime; some disaggregated configurations additionally require `TLLM_FMHA_LIBS=-fallback` in the server environment and, on GB300 (aarch64), `UCX_TLS=cuda_copy,cuda_ipc,sm,self,tcp`. Speculative decoding uses a DFlash2 drafter ([`nvidia/Muse-Spark-Open-816B-A42B-DFlash2`](https://huggingface.co/nvidia/Muse-Spark-Open-816B-A42B-DFlash2)). +[^26]: Mistral Large 4 runs only on NVIDIA GB300 (`SM103`) from the NVFP4 checkpoint in the Mistral format (`checkpoint_format: mistral`); the FP8 checkpoint and the Hugging Face format are not supported. The architecture name is the one TensorRT LLM assigns when it reads the checkpoint's `params.json`. It is served by three modeling_v2 targets, tensor-parallel attention with expert-parallel experts, attention data parallelism with expert-parallel experts, and tensor parallelism, each on 4 GPUs or on 8 GPUs over two nodes, with contexts of up to 1,048,576 tokens. Speculative decoding uses the DSpark draft in the checkpoint (3 draft tokens). See the [Mistral Large 4 deployment guide](../deployment-guide/deployment-guide-for-mistral-large-4-on-trtllm.md). +[^27]: Validated for one TEP4 context server and one TEP4 generation server at a 32K-token context, without speculative decoding. # Encoder-Decoder Feature Support Matrix (PyTorch Backend) @@ -181,6 +185,7 @@ not used to mark a feature as supported here. | `Qwen3_5ForConditionalGeneration` | Yes | Yes | Untested | Yes | No | Untested | Yes | L + I + V | | `Qwen3_5MoeForConditionalGeneration` | Yes | Yes | Untested | Yes | No | Untested | Yes | L + I + V | | `Qwen4ExpForConditionalGeneration` | Untested | Untested | Untested | Yes | Untested | Yes | No | L + I | +| `MistralLarge4ForCausalLM` [^26] | Yes | Yes | Yes | Yes | Yes | Untested | Untested | L + I | Note: - L: Language diff --git a/examples/serve/compatibility/README.md b/examples/serve/compatibility/README.md index 5351f269e82f..17104efe558d 100644 --- a/examples/serve/compatibility/README.md +++ b/examples/serve/compatibility/README.md @@ -15,6 +15,21 @@ This directory contains individual, self-contained examples demonstrating Tensor trtllm-serve Qwen/Qwen3-8B --reasoning_parser "qwen3" --tool_parser "qwen3" ``` + Native `mistral_large_4` required and named tool choices need the `llguidance` + guided-decoding backend. Override an xgrammar serving configuration with: + + ```bash + trtllm-serve /path/to/checkpoint --config serve.yaml \ + --tool_parser mistral_large_4 --reasoning_parser mistral_large_4 \ + --set guided_decoding_backend=llguidance + ``` + + Required tool choices allow parallel calls unless the request explicitly sets + `parallel_tool_calls` to `false`. Named choices constrain generation to one + call, including when `parallel_tool_calls` is `true`, matching the single-call + forced-choice post-processing. Native required and named choices return HTTP + 400 when the server uses xgrammar. + ## Running Examples diff --git a/jenkins/L0_Test.groovy b/jenkins/L0_Test.groovy index fc024094bb53..156332b74486 100644 --- a/jenkins/L0_Test.groovy +++ b/jenkins/L0_Test.groovy @@ -7884,7 +7884,7 @@ def launchTestJobs(pipeline, testFilter, globalVars) "GB300-4_GPUs-PyTorch-PerfSanity-KeyModels-Post-Merge-{split}_of_2", "auto:gb300-x4", "l0_gb300_multi_gpus_perf_sanity_key_models", - 2, + 4, 4, 1, true ) @@ -7986,8 +7986,10 @@ def launchTestJobs(pipeline, testFilter, globalVars) "GB300-16_GPUs-4_Nodes-PyTorch-PerfSanity-Node4-GPU16-Post-Merge-1_of_2": ["auto:gb300-flex", "l0_gb300_multi_nodes_perf_sanity_node4_gpu16", 1, 1, 16, 4], // gen_only_no_context: gen1 (8 nodes, 32 GPUs), no ctx fleet = 32 GPUs "GB300-32_GPUs-8_Nodes-PyTorch-PerfSanity-GEN1-NODE8-GPU32-Post-Merge-3_of_4": ["auto:gb300-flex", "l0_gb300_multi_nodes_perf_sanity_gen1_node8_gpu32", 1, 1, 32, 8], - // GLM-5.3-Flash FP8 8k1k con16: ctx1 (1 node, 4 GPUs) + gen1 tep4 (1 node, 4 GPUs) = 8 GPUs - "GB300-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-KeyModels-Post-Merge-1_of_2": ["auto:gb300-flex", "l0_gb300_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen1_node1_gpu4_key_models", 1, 1, 8, 2], + // GLM-5.3-Flash FP8 8k1k con16 and Mistral Large 4 NVFP4 8k1k con64/256/1024: + // ctx1 (1 node, 4 GPUs) + gen1 (1 node, 4 GPUs) = 8 GPUs, one split per build slot + "GB300-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-KeyModels-Post-Merge-1_of_2": ["auto:gb300-flex", "l0_gb300_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen1_node1_gpu4_key_models", 1, 2, 8, 2], + "GB300-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-KeyModels-Post-Merge-2_of_2": ["auto:gb300-flex", "l0_gb300_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen1_node1_gpu4_key_models", 2, 2, 8, 2], // Nemotron-Ultra-V3 8k64k con1: ctx1 (1 node, 4 GPUs) + gen1 tep4 (1 node, 4 GPUs) = 8 GPUs "GB300-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-3_of_4": ["auto:gb300-flex", "l0_gb300_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen1_node1_gpu4", 1, 1, 8, 2], ] diff --git a/requirements.txt b/requirements.txt index cc265f4669d7..0f8a96650577 100644 --- a/requirements.txt +++ b/requirements.txt @@ -76,7 +76,7 @@ einops fla-core==0.5.2 # the Kimi Linear / Kimi K3 KDA layers import fla.modules and fla.ops when they load flashinfer-python==0.7.3 xgrammar==0.2.8 -llguidance==0.7.29 +llguidance==1.7.6 jsonschema backoff nvtx @@ -106,7 +106,8 @@ mcp>=1.30.0,<2.0.0 # 2.0.0 removed mcp.server.fastmcp; scaffolding still uses th apache-tvm-ffi==0.1.13.post2 # required by the CUTLASS DSL TVM-FFI provider torch-c-dlpack-ext==0.1.3 # used for reduce nvidia-cutlass-dsl host overhead, optional package for improved torch tensor calling perf flash-attn-4==4.0.0b19 -mistral-common>=1.10.0 +mistral-common[opencv]>=1.13.0; python_version >= "3.11" +mistral-common[opencv]>=1.12.0; python_version < "3.11" # 1.13.0 requires Python 3.11+ torchao>=0.14.1,<0.16.0 cuda-core llist diff --git a/security_scanning/pyproject.toml b/security_scanning/pyproject.toml index f52eb6e17c4c..3954939a0f46 100644 --- a/security_scanning/pyproject.toml +++ b/security_scanning/pyproject.toml @@ -63,7 +63,7 @@ dependencies = [ "fla-core (==0.5.2)", "flashinfer-python (==0.7.3)", "xgrammar (==0.2.8)", - "llguidance (==0.7.29)", + "llguidance (==1.7.6)", "jsonschema (>=4.26.0,<5.0.0)", "backoff (>=2.2.1,<3.0.0)", "nvtx (>=0.2.17,<0.3.0)", @@ -86,7 +86,7 @@ dependencies = [ "mcp (>=1.30.0,<2.0.0)", "torch-c-dlpack-ext (==0.1.3)", "flash-attn-4 (==4.0.0b19)", - "mistral-common (>=1.10.0)", + "mistral-common[opencv] (>=1.13.0) ; python_version >= \"3.11\"", "torchao (>=0.14.1,<0.16.0)", "cuda-core (>=1.3.1,<2.0.0)", "llist (>=0.7.1,<0.8.0)", diff --git a/tensorrt_llm/_torch/attention/backends/sparse/dsa/cache_manager.py b/tensorrt_llm/_torch/attention/backends/sparse/dsa/cache_manager.py index c318aeb58ba4..7ee9036e2cd4 100644 --- a/tensorrt_llm/_torch/attention/backends/sparse/dsa/cache_manager.py +++ b/tensorrt_llm/_torch/attention/backends/sparse/dsa/cache_manager.py @@ -674,10 +674,13 @@ def __init__( ) primary_converters = [ self.impl.get_page_index_converter(local_layer_idx, Role.KEY) + if self.has_buffer(local_layer_idx, Role.KEY) + else None for local_layer_idx in range(self.num_local_layers) ] self._primary_pool_page_index_params = [ - (int(converter.scale), int(converter.layer_offset)) for converter in primary_converters + (int(converter.scale), int(converter.layer_offset)) if converter is not None else None + for converter in primary_converters ] local_indexer_mask = self.indexer_k_cache_local_layer_mask full_indexer_local_layers = [ @@ -719,8 +722,11 @@ def indexer_k_cache_local_layer_mask(self) -> list[bool]: # The base wrapper may remove empty layers while building the cache. return [self._indexer_k_cache_layer_mask[layer_idx] for layer_idx in self.pp_layers] - def get_primary_pool_page_index_params(self, local_layer_idx: int) -> Tuple[int, int]: - """Return the formal V2 page scale and layer offset for sparse MLA.""" + def get_primary_pool_page_index_params(self, local_layer_idx: int) -> Optional[Tuple[int, int]]: + """Return the formal V2 page scale and layer offset for sparse MLA. + + None for a layer that owns no latent KEY buffer. + """ return self._primary_pool_page_index_params[local_layer_idx] def _get_buffer_roles_for_layer(self, local_layer_idx: int) -> List[DataRole]: diff --git a/tensorrt_llm/_torch/attention/backends/sparse/mistral_large4/__init__.py b/tensorrt_llm/_torch/attention/backends/sparse/mistral_large4/__init__.py new file mode 100644 index 000000000000..86922aa08b68 --- /dev/null +++ b/tensorrt_llm/_torch/attention/backends/sparse/mistral_large4/__init__.py @@ -0,0 +1,6 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from .cache_manager import MistralLarge4CacheManager + +__all__ = ["MistralLarge4CacheManager"] diff --git a/tensorrt_llm/_torch/attention/backends/sparse/mistral_large4/cache_manager.py b/tensorrt_llm/_torch/attention/backends/sparse/mistral_large4/cache_manager.py new file mode 100644 index 000000000000..07e1967fd416 --- /dev/null +++ b/tensorrt_llm/_torch/attention/backends/sparse/mistral_large4/cache_manager.py @@ -0,0 +1,274 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""KV cache manager for Mistral-Large-4 checkpoints: an fp8_ds_mla latent pool with +cross-layer KV sharing. + +The checkpoint names, per layer, the layer whose latent it attends over +(``kv_source_layers``); a layer is a KV *owner* when that is itself. Only owners get +a latent page, so a sharing layer allocates no latent at all and its readers address +its source's page. Every latent row is the fp8_ds_mla token the reference +implementation writes:: + + [ 512 x e4m3 NoPE | 4 x fp32 power-of-two tile scale | 64 x bf16 RoPE ] = 656 B + +The indexer K cache is the DSA one (``index_head_dim`` e4m3 + one fp32 scale per key) +on every ``index_sharing == 'full'`` layer, owners and sharers alike. Indexer +pages belong to their logical layers independently of latent ownership. + +Bytes per token = owners x 656 + full-indexer layers x 132. +The manager is selected by ``sparse.registry`` for a ``MistralLarge4Config`` +checkpoint and requires KV cache manager V2. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, List, Optional, Union + +import torch + +from tensorrt_llm._torch.pyexecutor.kv_cache.kv_cache_manager_v2 import Role +from tensorrt_llm._utils import TensorWrapper, convert_to_torch_tensor +from tensorrt_llm.bindings import DataType +from tensorrt_llm.bindings.executor import KvCacheConfig +from tensorrt_llm.bindings.internal.batch_manager import CacheType as CacheTypeCpp +from tensorrt_llm.logger import logger +from tensorrt_llm.mapping import Mapping +from tensorrt_llm.runtime.kv_cache_manager_v2 import LayerId + +from ..dsa.cache_manager import ( + DSACacheManagerV2, + _get_indexer_k_cache_bytes_per_token, + derive_indexer_k_cache_layer_mask, +) +from ..dsa.params import DSAParams + +if TYPE_CHECKING: + import tensorrt_llm.bindings + from tensorrt_llm.llmapi.llm_args import DecodingBaseConfig, SparseAttentionConfig + + ModelConfig = tensorrt_llm.bindings.ModelConfig + +#: The latent row each owner layer stores per token, in bytes. +DIM_NOPE = 512 +DIM_ROPE = 64 +QUANT_TILE = 128 +SCALE_BYTES = DIM_NOPE // QUANT_TILE * 4 +ROPE_OFFSET = DIM_NOPE + SCALE_BYTES +TOKEN_BYTES = ROPE_OFFSET + DIM_ROPE * 2 # 656 +#: The layout name targets check before they address the pool. +LATENT_LAYOUT = "fp8_ds_mla" +_INDEXER_QUANT_BLOCK = 128 + + +def kv_source_layers(pretrained_config) -> Optional[List[int]]: + """The checkpoint's per-layer KV source, or None when it shares nothing.""" + sources = getattr(pretrained_config, "kv_source_layers", None) + if sources is None: + return None + return [int(s) for s in sources] + + +def latent_cache_bytes_per_token( + sources: List[int], indexer_mask: List[bool], indexer_bytes: int +) -> int: + """Owners x 656 + full-indexer layers x the indexer bytes.""" + owners = sum(1 for i, s in enumerate(sources) if s == i) + return owners * TOKEN_BYTES + sum(indexer_mask) * indexer_bytes + + +def _indexer_bytes(sparse_params: DSAParams) -> int: + return _get_indexer_k_cache_bytes_per_token( + sparse_params.index_head_dim, + _INDEXER_QUANT_BLOCK, + sparse_params.indexer_k_dtype == "fp4", + ) + + +class MistralLarge4CacheManager(DSACacheManagerV2): + """DSA cache manager with an owner-only fp8_ds_mla latent pool (see module doc).""" + + latent_layout = LATENT_LAYOUT + latent_token_bytes = TOKEN_BYTES + + def __init__( + self, + kv_cache_config: KvCacheConfig, + kv_cache_type: CacheTypeCpp, + *, + num_layers: int, + num_kv_heads: Union[int, List[Optional[int]]], + head_dim: int, + tokens_per_block: int, + max_seq_len: int, + max_batch_size: int, + mapping: Mapping, + dtype: DataType = DataType.HALF, + spec_config: Optional["DecodingBaseConfig"] = None, + layer_mask: Optional[List[bool]] = None, + max_num_tokens: int = 8192, + model_config: Optional["ModelConfig"] = None, + max_beam_width: int = 1, + sparse_attention_config: Optional["SparseAttentionConfig"] = None, + pretrained_config=None, + **kwargs, + ) -> None: + sources = kv_source_layers(pretrained_config) + if sparse_attention_config is None and model_config is not None: + sparse_attention_config = model_config.sparse_attention_config + indexer_mask = derive_indexer_k_cache_layer_mask( + sparse_attention_config, pretrained_config, num_layers + ) + # The routed targets fix this structure (mistral_large4/routing.py): one SELFKONLY + # 512+64 MLA latent per layer at pp_size 1, sources that own their latent, and an + # indexer K-cache on every latent-sharing layer. + assert ( + sources is not None + and len(sources) == num_layers + and all(sources[source] == source <= layer for layer, source in enumerate(sources)) + and all(indexer_mask[layer] for layer, source in enumerate(sources) if source != layer) + and mapping.pp_size == 1 + and kv_cache_type == CacheTypeCpp.SELFKONLY + and num_kv_heads + in (1, [1] * num_layers, [1 if s == i else None for i, s in enumerate(sources)]) + and head_dim == DIM_NOPE + DIM_ROPE + ), "the Mistral-Large-4 latent cache needs the routed targets' layer structure" + # Separate-draft allocation passes an all-target-layer mask. Physical + # latent ownership still comes exclusively from kv_source_layers. + if layer_mask is not None and layer_mask != [True] * len(sources): + raise NotImplementedError( + f"{type(self).__name__} derives its layer mask from kv_source_layers" + ) + if spec_config is not None and ( + not spec_config.spec_dec_mode.is_dspark() + or spec_config.draft_is_embedded_in_target + or spec_config._use_shared_kv_cache + ): + raise NotImplementedError( + f"{type(self).__name__} supports DSpark with a separate draft KV pool only" + ) + owner_heads = [1 if source == layer else None for layer, source in enumerate(sources)] + requested = getattr(kv_cache_config, "dtype", "auto") + if dtype != DataType.FP8 or requested not in ("auto", "fp8", "fp8_ds_mla"): + raise ValueError( + "the Mistral-Large-4 latent cache is fp8_ds_mla (656 B/token): set " + "kv_cache_config.dtype to auto (the checkpoint's fp8_kv_cache), fp8 or " + f"fp8_ds_mla; got dtype={requested!r} ({dtype})" + ) + if getattr(kv_cache_config, "enable_kv_pool_rebalance", False): + raise ValueError( + "the fp8_ds_mla latent cache does not support enable_kv_pool_rebalance" + ) + if requested == "fp8_ds_mla": + # The generic DSA check for this dtype string describes the SM120 FlashInfer + # inline-scale pages (576-element head, 64-token pages). This manager owns the + # 656-byte layout itself and takes any page size, so hand the base the plain + # FP8 element type. + kv_cache_config = kv_cache_config.model_copy(update={"dtype": "fp8"}) + + self.kv_source_layers = sources + self.kv_owner_layers = [i for i, s in enumerate(sources) if s == i] + self.kv_sharer_layers = [i for i, s in enumerate(sources) if s != i] + + super().__init__( + kv_cache_config, + kv_cache_type, + num_layers=num_layers, + num_kv_heads=owner_heads, + head_dim=TOKEN_BYTES, + tokens_per_block=tokens_per_block, + max_seq_len=max_seq_len, + max_batch_size=max_batch_size, + mapping=mapping, + dtype=DataType.FP8, + spec_config=spec_config, + layer_mask=None, + max_num_tokens=max_num_tokens, + model_config=model_config, + max_beam_width=max_beam_width, + sparse_attention_config=sparse_attention_config, + pretrained_config=pretrained_config, + **kwargs, + ) + actual_owners = [ + layer for local, layer in enumerate(self.pp_layers) if self.has_buffer(local, Role.KEY) + ] + if actual_owners != self.kv_owner_layers: + raise RuntimeError(f"Physical KEY owners {actual_owners} do not match KV sources") + + logger.info( + f"[{type(self).__name__}] {len(self.kv_owner_layers)} latent owners x " + f"{TOKEN_BYTES} B + {sum(indexer_mask)} indexer layers x " + f"{self.get_layer_bytes_per_token(0, Role.INDEX_KEY)} B = " + f"{self.get_cache_bytes_per_token()} B/token; " + f"sharers {self.kv_sharer_layers} read their source's latent" + ) + + def _kv_pool_mapping_offset( + self, layer_id: LayerId, layer_group_id: int, key_base_addr: int + ) -> int: + source = self.kv_source_layers[self.pp_layers[int(layer_id)]] + return super()._kv_pool_mapping_offset( + self.layer_offsets[source], layer_group_id, key_base_addr + ) + + def get_unique_primary_pool(self) -> torch.Tensor: + """Return the contiguous latent pool in physical owner order.""" + if self._unique_primary_pool is not None: + return self._unique_primary_pool + owners = self.kv_owner_layers + key_buffers = [buffer for buffer in self.impl.all_buffer_ids if buffer.role == Role.KEY] + pages = self.impl.get_aggregated_pages(key_buffers) + if len(pages) != 1: + raise ValueError("ML4 latent kernels require a single contiguous KEY pool") + page = pages[0] + bytes_per_slot = len(owners) * self.tokens_per_block * TOKEN_BYTES + if ( + int(page.size) != bytes_per_slot + or int(page.stride) != bytes_per_slot + or any(int(buffer.expansion) != 1 for buffer in page.buffers) + ): + raise ValueError("ML4 latent kernels require packed, unexpanded KEY pages") + # The descriptor supplies the pool's physical origin. Logical layers + # address their source's row through the page-index converter. + self._unique_primary_pool = convert_to_torch_tensor( + TensorWrapper( + page.base, + self.dtype, + [self.blocks_in_primary_pool, len(owners), 1, self.tokens_per_block * TOKEN_BYTES], + ) + ) + return self._unique_primary_pool + + @staticmethod + def get_cache_size_per_token( + model_config: "ModelConfig", + mapping: Mapping, + num_layers: Optional[int] = None, + **kwargs, + ) -> int: + """Owners x 656 + full-indexer layers x the indexer bytes.""" + config = model_config.pretrained_config + sources = kv_source_layers(config) + if sources is None or num_layers is not None: + raise ValueError("the Mistral-Large-4 cache is sized from its own kv_source_layers") + sparse = model_config.sparse_attention_config + indexer_mask = derive_indexer_k_cache_layer_mask(sparse, config, len(sources)) + return latent_cache_bytes_per_token( + sources, indexer_mask, _indexer_bytes(sparse.to_sparse_params(pretrained_config=config)) + ) + + # ---- accessors the targets read ---- + + def latent_pool_row(self, layer_idx: int) -> int: + """The latent page row (0..owners-1) layer `layer_idx` reads: its source's.""" + source = self.layer_offsets[self.kv_source_layers[layer_idx]] + return self.get_primary_pool_page_index_params(source)[1] + + def get_latent_pool(self) -> torch.Tensor: + """The whole latent pool as uint8 `[blocks, owners, tokens_per_block, 656]`; + the token at position p of a request whose block-table entry is b lives, for + layer L, at `[b, latent_pool_row(L), p % tokens_per_block]`.""" + pool = self.get_unique_primary_pool() + return pool.view(torch.uint8).view( + pool.shape[0], pool.shape[1], self.tokens_per_block, TOKEN_BYTES + ) diff --git a/tensorrt_llm/_torch/attention/backends/sparse/registry.py b/tensorrt_llm/_torch/attention/backends/sparse/registry.py index d993ec8955dd..a2ab15d4df3c 100644 --- a/tensorrt_llm/_torch/attention/backends/sparse/registry.py +++ b/tensorrt_llm/_torch/attention/backends/sparse/registry.py @@ -24,7 +24,10 @@ def get_sparse_attn_kv_cache_manager( sparse_attention_config: "SparseAttentionConfig", use_kv_cache_manager_v2: bool = False, + pretrained_config=None, ) -> type: + """``pretrained_config``, when given, lets a model-specific manager be chosen by the + checkpoint's config class, which the sparse config does not identify.""" from tensorrt_llm._torch.pyexecutor.kv_cache.kv_cache_manager_v2 import KVCacheManagerV2 from tensorrt_llm._torch.pyexecutor.resource_manager import KVCacheManager @@ -45,6 +48,15 @@ def get_sparse_attn_kv_cache_manager( if sparse_attention_config.algorithm == "rocket": return RocketKVCacheManager elif sparse_attention_config.algorithm == "dsa": + from tensorrt_llm._torch.configs.mistral_large4 import MistralLarge4Config + + # Mistral-Large-4 (cross-layer KV sharing): owner-only fp8_ds_mla latent pages. + if isinstance(pretrained_config, MistralLarge4Config): + if not use_kv_cache_manager_v2: + raise ValueError("DSA for Mistral-Large-4 requires KV cache manager V2") + from .mistral_large4 import MistralLarge4CacheManager + + return MistralLarge4CacheManager return DSACacheManagerV2 if use_kv_cache_manager_v2 else DSACacheManager elif sparse_attention_config.algorithm == "deepseek_v4": return DeepseekV4CacheManager diff --git a/tensorrt_llm/_torch/configs/__init__.py b/tensorrt_llm/_torch/configs/__init__.py index a1645d9a78e0..5d0543d41ee9 100644 --- a/tensorrt_llm/_torch/configs/__init__.py +++ b/tensorrt_llm/_torch/configs/__init__.py @@ -44,6 +44,7 @@ from tensorrt_llm._torch.configs.kimi_linear import KimiLinearConfig from tensorrt_llm._torch.configs.laguna import LagunaConfig from tensorrt_llm._torch.configs.minicpmv4_6 import MiniCPMV4_6Config, MiniCPMV4_6VisionConfig +from tensorrt_llm._torch.configs.mistral_large4 import MistralLarge4Config from tensorrt_llm._torch.configs.musespark_open import MuseSparkOpenConfig from tensorrt_llm._torch.configs.nemotron_h import NemotronHConfig from tensorrt_llm._torch.configs.qwen4_exp import ( @@ -168,6 +169,7 @@ def _register_custom_configs_with_transformers() -> None: "LagunaConfig", "MiniCPMV4_6Config", "MiniCPMV4_6VisionConfig", + "MistralLarge4Config", "MuseSparkOpenConfig", "NemotronHConfig", "Qwen4ExpConfig", diff --git a/tensorrt_llm/_torch/configs/mistral_large4.py b/tensorrt_llm/_torch/configs/mistral_large4.py new file mode 100644 index 000000000000..21dcb4d9a70d --- /dev/null +++ b/tensorrt_llm/_torch/configs/mistral_large4.py @@ -0,0 +1,62 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from typing import Optional + +from transformers.configuration_utils import PretrainedConfig + +# index_sharing names each layer's indexer role; the DSA top-k pattern spells +# the same roles with one letter per layer. +_INDEX_SHARING_LETTERS = {"full": "F", "shared": "S"} + + +# A Mistral-Large-4 language model's config, adapted from the checkpoint's +# Mistral-native params.json (models/checkpoints/mistral/config_loader.py) under +# model_type "mistral_large_4", which transformers does not know. Every key is +# kept as a plain attribute, exactly as adapted. DeepseekV3Config is not used +# because it fills in defaults the checkpoint does not state (an MTP layer +# count, special-token ids). +class MistralLarge4Config(PretrainedConfig): + model_type = "mistral_large_4" + keys_to_ignore_at_inference = ["past_key_values"] + + # The indexer rotates GPT-J interleaved pairs, as MLA does. The + # architecture has no other layout, so the checkpoint does not state it. + indexer_rope_interleave = True + + def __init__(self, vision_config=None, **kwargs): + super().__init__(**kwargs) + if vision_config is not None: + self.vision_config = ( + PretrainedConfig.from_dict(vision_config) + if isinstance(vision_config, dict) + else vision_config + ) + + @property + def index_topk_pattern(self) -> Optional[str]: + """``index_sharing`` as the DSA top-k pattern, one letter per layer. + + "F" marks a layer that runs its own indexer and owns an indexer-K + cache; "S" marks one that reuses the top-k of the most recent "F" + layer. ``DeepSeekSparseAttentionConfig`` reads this pattern. + """ + index_sharing = getattr(self, "index_sharing", None) + if index_sharing is None: + return None + unknown = sorted(set(index_sharing) - _INDEX_SHARING_LETTERS.keys()) + if unknown: + raise ValueError(f"index_sharing entries must be 'full' or 'shared', got {unknown}") + return "".join(_INDEX_SHARING_LETTERS[role] for role in index_sharing) diff --git a/tensorrt_llm/_torch/model_config.py b/tensorrt_llm/_torch/model_config.py index a89fa5451bb7..b4f38c31620d 100644 --- a/tensorrt_llm/_torch/model_config.py +++ b/tensorrt_llm/_torch/model_config.py @@ -1751,6 +1751,7 @@ def override_quant_algo(): def from_pretrained(cls, checkpoint_dir: str, trust_remote_code=False, + checkpoint_format: Optional[str] = None, **kwargs): def update_sparse_attention_indexer_config(pretrained_config, kwargs): @@ -1829,6 +1830,7 @@ def update_sparse_attention_indexer_config(pretrained_config, kwargs): pretrained_config = load_pretrained_config( checkpoint_dir, trust_remote_code=trust_remote_code, + checkpoint_format=checkpoint_format, **kwargs, ) # transformers 5 leaves `_name_or_path` empty when the config @@ -1840,7 +1842,8 @@ def update_sparse_attention_indexer_config(pretrained_config, kwargs): if not getattr(pretrained_config, '_name_or_path', None): pretrained_config._name_or_path = str(checkpoint_dir) if pretrained_config.architectures[0] in [ - "DeepseekV32ForCausalLM", "GlmMoeDsaForCausalLM" + "DeepseekV32ForCausalLM", "GlmMoeDsaForCausalLM", + "MistralLarge4ForCausalLM" ]: sparse_attention_config = kwargs.get( 'sparse_attention_config') diff --git a/tensorrt_llm/_torch/modeling_v2/README.md b/tensorrt_llm/_torch/modeling_v2/README.md index 3a00db3e0066..4e6d66806c7a 100644 --- a/tensorrt_llm/_torch/modeling_v2/README.md +++ b/tensorrt_llm/_torch/modeling_v2/README.md @@ -178,6 +178,10 @@ a Modeling V1 model class, and can override any of these steps: * **Constructor:** builds the model structure and parameters. * **`load_weights`:** loads checkpoint weights into the model. Put weight mapping, transformation and sharding in `weights.py`. +* **`load_checkpoint`** (optional): reads the checkpoint directory itself. When + a target defines it, the engine calls it in place of `load_weights` and opens + no checkpoint file of its own. Use it for a checkpoint the target streams + from its files, in its on-disk tensor names. * **`post_load_weights`:** prepares the loaded weights, for example by transforming them or precomputing derived state. The default implementation in `DecoderModelForCausalLM` runs alias setup, weight transforms and derived diff --git a/tensorrt_llm/_torch/modeling_v2/catalog/_op.py b/tensorrt_llm/_torch/modeling_v2/catalog/_op.py index a0494697de67..82c681dac114 100644 --- a/tensorrt_llm/_torch/modeling_v2/catalog/_op.py +++ b/tensorrt_llm/_torch/modeling_v2/catalog/_op.py @@ -45,6 +45,7 @@ import contextlib import enum +import zlib from abc import ABC, abstractmethod from dataclasses import dataclass, field from typing import Any, Mapping @@ -110,6 +111,14 @@ class Cell: why: str spec: Mapping[str, Any] = field(default_factory=dict) + #: The base of the cell's random inputs, for an entry whose test draws them. + #: None derives it from `why` (`seed_base`); set it when the inputs must not + #: follow the wording. + seed: int | None = None + + def seed_base(self) -> int: + """The base an entry's test derives this cell's random inputs from.""" + return zlib.crc32(self.why.encode()) if self.seed is None else self.seed class OpWrapper(ABC): diff --git a/tensorrt_llm/_torch/modeling_v2/catalog/activation/flashinfer_silu_and_mul.py b/tensorrt_llm/_torch/modeling_v2/catalog/activation/flashinfer_silu_and_mul.py index a7ac57303355..5bbc1be9046e 100644 --- a/tensorrt_llm/_torch/modeling_v2/catalog/activation/flashinfer_silu_and_mul.py +++ b/tensorrt_llm/_torch/modeling_v2/catalog/activation/flashinfer_silu_and_mul.py @@ -17,6 +17,11 @@ class FlashinferSiluAndMul(OpWrapper): The gate and up projections arrive concatenated along the last dim, which is why the op takes one tensor and halves it rather than taking two. + + `torch.ops.trtllm.flashinfer_silu_and_mul` (`custom_ops/flashinfer_custom_ops.py`) + calls `flashinfer.activation.silu_and_mul`, whose JIT module launches + `act_and_mul_kernel` (flashinfer's `activation.cuh`): one CTA per row, fp32 + math, one rounding to the input dtype on the store. """ ARCHS = frozenset({Arch.SM_100, Arch.SM_103}) @@ -40,9 +45,127 @@ class FlashinferSiluAndMul(OpWrapper): note = """ Registered only when flashinfer is importable (`IS_FLASHINFER_AVAILABLE`). - - fp16 and fp32 are accepted by the op. No shipped target uses them here, so - they are not cells. + Metadata consumed: none. + + Preconditions, each an `is_valid` guard: `x` is contiguous, its base is + 16-byte aligned, and each half of its last dim is a whole number of + 16-byte vectors. The kernel addresses row r at `r * x.shape[-1]` elements + from the base pointer and ignores strides: a column slice, or a view of + every other row, is accepted and every row but the first is computed from + the wrong elements (measured: 93.7% of the outputs wrong at widths 12288 and + 3072, 16 rows; at least 99.9% at 6144 and 1536). A + contiguous view that starts 2 or 8 bytes off a 16-byte boundary faults with + `torch.AcceleratorError: CUDA error: misaligned address`, which poisons the + context. A single row sliced out of a wider buffer is contiguous, admitted, + and computed right. + + Raised by the op itself, so not guarded (the test drives both): fp32 input, + `RuntimeError: Check failed: (false) is false: silu_and_mul(tvm::ffi::TensorView, + tvm::ffi::TensorView, bool):: failed to dispatch data type`; zero + rows, `RuntimeError: Check failed: (err == cudaSuccess) is false: Failed to + launch kernel: invalid argument` -- a caller with a possibly empty batch + skips the call. fp16 is accepted (one probe at 16 x 3072, inside torch's + fp16 band); no target uses it here, so it is not a cell. + + PDL. The trtllm op passes `enable_pdl` from `get_env_enable_pdl()` + (`TRTLLM_ENABLE_PDL`, on unless set to something other than "1"), read on + every call. The certifying run had it unset, so PDL on, which is how this + target's engine runs. With it at "0" the output was measured bitwise equal + at 9216 rows for widths 12288 and 3072, and at 16 and 32768 rows for widths + 49152, 6144 and 1536; that path is not a cell. + + Rows. The block size is set by the width alone (768 threads at 12288, 192 at + 3072, one 16-byte vector per thread per half) and the row count only sets + the grid. Batch-invariant at both target widths: every call on the leading + 1..9215 rows is bitwise equal to those rows of the 9216-row call, and every + call on the leading 1..32767 rows to those of the 32768-row call. CUDA + graphs: captured at 1..16 rows and replayed on new data at widths 12288 and + 3072, the output is bitwise equal to an eager call on that data; one eager + call on the capture stream precedes each capture. Widths 4096 and 36864 were + not captured. + + Widths 6144 and 1536 (dense layer 0 and the shared expert at tep8 and + tp8). Profiled on sm_103: grid = rows, 384 and 96 threads, 32 registers. + Each thread makes one vector pass over each half, as at 12288 and 3072, + and the scalar loop over the `d % (threads * 8)` remainder is empty. + Batch-invariant at both: every call on the leading 1..32767 rows is + bitwise equal to those rows of the 32768-row call. CUDA graphs captured at + 1..16 rows replay on new data bitwise equal to eager. + + Width 49152 (dep4's replicated dense layer 0). Profiled on sm_103: grid = + rows, 1024 threads (the launcher's cap; a half is 3072 vectors), 32 + registers, so two CTAs are resident per SM and a wave is 304 rows on 152 + SMs. Read from flashinfer's `activation.cuh`: each thread then makes + exactly three vector passes over each half, and the kernel's scalar loop + over the `d % (threads * 8)` remainder is empty (at 36864 it would cover + 2048 columns). Batch-invariant here too: every call on the leading + 1..32767 rows is bitwise equal to those rows of the 32768-row call (the + `large` cell; 20.9 s, 11.0 s of it the op calls), and pre-merge's sample + (939 counts, 0.27 s) holds the same. CUDA graphs captured at 1..16 rows + replay bitwise equal to eager. + + Numerics, on each eager cell's own randn draw. At 9216 rows the output is + bitwise equal to `reference` at both target widths; at 32768 rows one + output of each draw is one bf16 step off (1 of 201326592 at 12288, 1 of + 50331648 at 3072), at worst 0.11 of torch's bf16 band. At 16x that scale, + one step on 1730 of 56623104 outputs (9216 x 12288), 405 of 14155776 + (9216 x 3072), 5886 of 201326592 (32768 x 12288) and 1424 of 50331648 + (32768 x 3072), plus 3, 0, 1 and 1 outputs of +-0 from the `__expf` tail + below; none reads more than 0.073 of the band. At 32768 x 49152 the two + eager cells' draws put 2 and 4 of 805306368 outputs one step off, 0.071 of + the band; at 16x, 23458 and 24063 one step off plus 6 and 10 of +-0, 0.072 + of the band. At 32768 rows the 6144 cell's draw puts 1 of 100663296 outputs + one step off (0.011 of the band) and the 1536 cell's none of 25165824; at + 16x, 3033 and 712 one step off plus 3 and 0 of +-0, 0.072 of the band. + The kernel's `silu` uses `__expf`, which shows only in the tails, measured + with up = 1: a gate in + [-88.5, -87.5] returns +-0 where the fp32 reference is a normal float of + magnitude at most 8.8e-37, and a subnormal gate or up reads as zero. Both + are inside torch's bf16 band (atol 1e-5); an infinite up times such a gate + gives nan where the reference gives +-inf. inf and nan otherwise propagate + as in the reference (gate -inf gives nan in both). + + Uncertified: widths other than 4096, 12288, 3072, 36864, 49152, 6144 and + 1536; width 4096 at rows other than 1 and 2048, and width 36864 at rows + other than 16; rows above 32768 at 12288, 3072, 49152, 6144 and 1536 (the + probes below are not cells); fp16; PDL off. At 49152 pre-merge drives only + the sampled row counts; the other counts in 1..32767 are the `large` cell's. + + Hard limits (sm_103), at the target widths. Offsets are int64 in + `act_and_mul_kernel`. The serving envelopes reach none of these: 9216 rows + on the FP8 checkpoint's target, 32768 on the NVFP4 ones (at most 1610612736 + input elements, 75% of 2^31, at dep4's 49152, and 805306368 output + elements). The input's byte offset does pass 2^31 inside row 21845 at + 49152, so every 32768-row call there crosses it; offsets are typed-pointer + arithmetic, and those calls are gated against `reference`. The op reads no + KV pool, slot or page table, so no pool size bounds it. + in offset row * width, int64 2^31 elements, inside row 174762 (12288), + row 699050 (3072) and row 43690 (49152); + 2^32 inside row 87381 (49152) + 2^31 elements inside row 349525 (6144) + and row 1398101 (1536): computed, not + probed (32768 rows reach 201326592 + elements at 6144, 9.4% of 2^31) + out offset row * width / 2, int64 2^31 elements, inside row 349525 (12288), + row 1398101 (3072) and row 87381 (49152) + 2^31 elements inside row 699050 (6144) + and row 2796202 (1536): computed, not + probed + probed at 349526 rows (12288) and 1398102 rows (3072), crossing + both: passed -- all but 9 and 15 rows bitwise equal to the + reference, those scattered on both sides of the crossings and + 1 bf16 ulp off, inside torch's bf16 band + probed at 87382 rows (49152), crossing all three: passed -- + bitwise equal to the same rows called in 8192-row slices, the + rows straddling the crossings (43689..43692, 87379..87381) + bitwise equal to one-row calls, 9 outputs 1 bf16 step off the + reference, 0.11 of the band + rows one CTA per row, grid x 2^31 - 1 rows unprobed (13.2 TB of + input at width 3072) + flashinfer's generated `silu_and_mul` launcher narrows the int64 + row count to the grid's unsigned 32-bit x dimension (computed, + not probed) + No argument is an index carried through fp32. """ def __call__(self, x: torch.Tensor) -> torch.Tensor: @@ -71,6 +194,19 @@ def is_valid(self, x: torch.Tensor) -> None: f"x.shape[-1] // 2 must hold at least one 16-byte vector; {width} " f"gives {half_bytes} bytes, for which the computed block size is 0" ) + # The kernel reads row r at `r * width` from the base pointer and never + # looks at a stride: a view whose rows are not packed is accepted and + # every row past the first is computed from the wrong elements. + assert x.is_contiguous(), ( + f"x must be contiguous; got strides {tuple(x.stride())} for shape " + f"{tuple(x.shape)} -- the kernel ignores strides and would read the wrong rows" + ) + # Packed rows and aligned halves still fault if the base is off a + # 16-byte boundary, the same unrecoverable `misaligned address`. + assert x.data_ptr() % 16 == 0, ( + f"x must start 16-byte aligned; its base is {x.data_ptr() % 16} bytes " + f"off -- the kernel's vector loads would fault" + ) flashinfer_silu_and_mul = FlashinferSiluAndMul() diff --git a/tensorrt_llm/_torch/modeling_v2/catalog/comm/allgather.py b/tensorrt_llm/_torch/modeling_v2/catalog/comm/allgather.py index d8a4c8a81eeb..f82863e22dd9 100644 --- a/tensorrt_llm/_torch/modeling_v2/catalog/comm/allgather.py +++ b/tensorrt_llm/_torch/modeling_v2/catalog/comm/allgather.py @@ -9,9 +9,91 @@ import torch import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._utils import mpi_rank from .._op import Arch, Cell, OpWrapper +# Mistral Large 4 NVFP4 at dep4: the token gather before the router of every MoE +# layer (1..71). Each of the four ranks passes its post-attention-norm rows +# zero-padded to T_max = max(attn_metadata.all_rank_num_tokens), sizes=None, on +# the MPI-session communicator a dep4 world launched under mpirun reaches. +_DEP4_GATHER = dict( + trailing=(6144,), + dtype=torch.bfloat16, + sizes=None, + world_size=4, + process_group=None, +) +# The decode batches the dep4 engine captures per rank (padded), and the +# gathers one captured decode step issues. +_DEP4_GRAPH_ROWS = (1, 2, 4, 8, 16) +_DEP4_MOE_LAYERS = 71 +# Mistral Large 4 NVFP4 at tep4 with its 1M envelope (max_seq_len 1048576, chunked +# prefill, max_num_tokens 32768): the q-split's selection gather. Once a step's +# tc context rows reach the engine's q_split_threshold (8192), rank r of the TP +# group selects for rows [tc * r // 4, tc * (r + 1) // 4) only, and after the +# chunk loop every selecting indexer layer gathers the four shares' int32 top-k +# rows (index_topk 2048 wide) into rows [0, tc) of the step's selection buffer: +# sizes = q_sizes from the engine's `_split_prefill_queries`, uneven by at most +# one row, on the MPI-session communicator the mpirun-launched tep4 world +# reaches. Context steps are never captured, so every call is eager. +_TEP4_SPLIT = dict( + trailing=(2048,), + dtype=torch.int32, + world_size=4, + process_group=None, + q_split=True, +) +_TEP4_SELECTING_LAYERS = 44 +# Mistral Large 4 NVFP4 at tp4 with the same envelope makes the same call: its +# attention and indexer are sharded as at tep4 (the indexer replicated), so its +# PrefillTarget._indexer gathers the same q_sizes shares over the same TP group +# [0, 1, 2, 3] once per selecting layer, and the cells stating `q_split` serve +# both targets. The tp4 cells add the remaining split remainders, top-k +# selection payloads (key positions to 1048575 with -1 tails) and the target's +# copy-back into its selection buffer at every swept tc. +_TP4_TOP_POSITION = 1048575 + +# Mistral Large 4 NVFP4 at dep8: the same gather over the eight attention-DP ranks +# of two GB300 nodes of four (one NVL72 segment, so NCCL forms an MNNVL clique +# of 8), group [0..7] in all_rank_num_tokens order. Each rank passes its rows +# zero-padded to T_max, sizes=None, on the MPI-session communicator an +# eight-rank world launched under mpirun across both nodes reaches. `nodes` is +# the number of hosts the ranks span. +_DEP8_GATHER = dict( + trailing=(6144,), + dtype=torch.bfloat16, + sizes=None, + world_size=8, + process_group=None, + nodes=2, +) +# The decode batches a dep8 engine captures per rank -- the gates' padded +# 1/2/4/8/16 and the LLM API's unpadded default at max_batch_size 16, every +# count 1..16 -- and the gathers one captured decode step issues. +_DEP8_GRAPH_ROWS = tuple(range(1, 17)) +_DEP8_MOE_LAYERS = 71 +# Mistral Large 4 NVFP4 at tep8 with its 1M envelope (max_seq_len 1048576, chunked +# prefill, max_num_tokens 32768): the q-split's selection gather. Once a step's +# tc context rows reach the engine's q_split_threshold (8192), rank r of the +# eight-rank TP group selects for rows [tc * r // 8, tc * (r + 1) // 8) only, +# and after the chunk loop every selecting indexer layer gathers the eight +# shares' int32 top-k rows (index_topk 2048 wide) into rows [0, tc) of the +# step's selection buffer: sizes = q_sizes from the engine's +# `_split_prefill_queries`, uneven by at most one row, on the MPI-session +# communicator the mn_mpirun-launched tep8 world reaches, over two nodes of +# four (one MNNVL clique of 8). Context steps are never captured, so every +# call is eager. +_TEP8_SPLIT = dict( + trailing=(2048,), + dtype=torch.int32, + world_size=8, + nodes=2, + process_group=None, + q_split=True, +) +_TEP8_SELECTING_LAYERS = 44 + class Allgather(OpWrapper): """Gather `input` from every rank in `group`, concatenated along dim 0. @@ -23,12 +105,16 @@ class Allgather(OpWrapper): ARCHS = frozenset({Arch.SM_100, Arch.SM_103}) - # deepseek-r1 at dep4 and the gpt-oss default target under attention DP - # are the callers, and both always pass `sizes=None`: each pads every rank - # to a common height first, so the ragged form the op also supports is - # not something a shipped target reaches. Row counts span what an - # attention-DP rank feeds its expert-parallel MoE call, from one decode - # token to a full prefill chunk. + # deepseek-r1 at dep4, the gpt-oss default target under attention DP and + # Mistral Large 4 NVFP4 at dep4 and dep8 pass `sizes=None`: each pads every + # rank to a common height first. Their row counts span what an attention-DP + # rank feeds its expert-parallel MoE call, from one decode token to a whole + # prefill step. Mistral Large 4 NVFP4 at tep4, at tp4 and at tep8 passes the + # ragged form: its q-split gathers each rank's share of a context step's + # top-k rows with `sizes` = the engine's q_sizes (the cells stating + # `q_split`). The first three cells drove the `_pg` symbol (a boxed c10d + # group); every cell stating `process_group=None` drove the MPI-session + # symbol, at the world size it states. CELLS: tuple[Cell, ...] = ( Cell( why="decode: one row per rank, hidden 7168", @@ -42,18 +128,486 @@ class Allgather(OpWrapper): why="prefill chunk height", spec=dict(rows=2048, trailing=(7168,), dtype=torch.bfloat16, sizes=None), ), + *( + Cell( + why=f"dep4 MoE gather, decode graph at {rows} row(s) per rank: captured with " + f"{_DEP4_MOE_LAYERS} gathers (MoE layers 1..71) beside the other captured " + "batches in one pool, after an eager warmup of the group; every replay on new " + "data bitwise the concatenation", + spec=dict( + rows=rows, + graph_rows=_DEP4_GRAPH_ROWS, + sites=_DEP4_MOE_LAYERS, + **_DEP4_GATHER, + ), + ) + for rows in _DEP4_GRAPH_ROWS + ), + Cell( + why="dep4 MoE gather, eager rows every value 1..2048: every protocol and " + "channel-count switch NCCL's ring makes at this width (LL through 154 rows, LL128 " + "155..990, SIMPLE from 991), each bitwise the concatenation", + spec=dict(rows=2048, sweep_rows=(1, 2048), sweep_step=1, **_DEP4_GATHER), + ), + Cell( + why="dep4 MoE gather, eager rows 2049..32768, one NCCL tuning bucket (ring, " + "SIMPLE, 32 channels): a stride of 61 rows plus every 2^k - 1, 2^k and 2^k + 1 up " + "to max_num_tokens 32768 (a 131072 x 6144 result), each bitwise the concatenation", + spec=dict( + rows=32768, + sweep_rows=(2049, 32768), + sweep_step=61, + sweep_also=(4095, 4096, 4097, 8191, 8192, 8193, 16383, 16384, 16385, 32767), + **_DEP4_GATHER, + ), + ), + Cell( + why="dep4 MoE gather at max_num_tokens with the padding skew: one rank's 32768 " + "rows of arbitrary bf16 bit patterns (NaN, inf, -0.0, subnormals) and three " + "ranks' single rows zero-padded to 32768 move bitwise and repeat bitwise; the " + "input is left untouched and the output is fresh", + spec=dict(rows=32768, payload="bits", valid_rows=(32768, 1, 1, 1), **_DEP4_GATHER), + ), + *( + Cell( + why=f"dep8 MoE gather, decode graph at {rows} row(s) per rank, 8 ranks over two " + f"nodes: captured with {_DEP8_MOE_LAYERS} gathers (MoE layers 1..71) beside the " + "other captured batches 1..16 in one pool, after an eager warmup of the group; " + "every replay on new data bitwise the concatenation", + spec=dict( + rows=rows, + graph_rows=_DEP8_GRAPH_ROWS, + sites=_DEP8_MOE_LAYERS, + **_DEP8_GATHER, + ), + ) + for rows in _DEP8_GRAPH_ROWS + ), + Cell( + why="dep8 MoE gather, eager rows every value 1..2048, 8 ranks over two nodes: " + "every protocol and channel-count switch NCCL's ring makes at this width (LL " + "through 96 rows, LL128 97..1190, SIMPLE from 1191), each bitwise the " + "concatenation", + spec=dict(rows=2048, sweep_rows=(1, 2048), sweep_step=1, **_DEP8_GATHER), + ), + Cell( + why="dep8 MoE gather, eager rows 2049..32768, 8 ranks over two nodes, one NCCL " + "tuning bucket (ring, SIMPLE, 32 channels): a stride of 61 rows plus every " + "2^k - 1, 2^k and 2^k + 1 and rows 21845/21846/21847, where the result passes 2^31 " + "bytes, up to max_num_tokens 32768 (a 262144 x 6144 result, 3,221,225,472 bytes), " + "each bitwise the concatenation", + spec=dict( + rows=32768, + sweep_rows=(2049, 32768), + sweep_step=61, + sweep_also=( + 4095, + 4096, + 4097, + 8191, + 8192, + 8193, + 16383, + 16384, + 16385, + 21845, + 21846, + 21847, + 32767, + ), + **_DEP8_GATHER, + ), + ), + Cell( + why="dep8 MoE gather at max_num_tokens with the padding skew over two nodes: one " + "rank's 32768 rows of arbitrary bf16 bit patterns (NaN, inf, -0.0, subnormals) and " + "seven ranks' single rows zero-padded to 32768 move bitwise and repeat bitwise; the " + "input is left untouched and the output is fresh", + spec=dict( + rows=32768, payload="bits", valid_rows=(32768, 1, 1, 1, 1, 1, 1, 1), **_DEP8_GATHER + ), + ), + Cell( + why="tep4 q-split gather at tc 8192, the engine's q_split_threshold: four 2048-row " + "int32 shares of [8192, 2048] selections (uniform q_sizes, so one ncclAllGather), " + "each a view at its offset in the rank's selection buffer; three calls on new " + "payloads, each bitwise the concatenation, the input untouched, the output fresh", + spec=dict(rows=2048, sizes=[2048, 2048, 2048, 2048], tokens=8192, **_TEP4_SPLIT), + ), + Cell( + why="tep4 q-split gather at tc 8193, the first ragged split: q_sizes [2048, 2048, " + "2048, 2049], four ncclBroadcasts in one NCCL group, each rank's share a view at " + "its offset in the selection buffer; three calls on new payloads, each bitwise " + "the concatenation, the input untouched, the output fresh", + spec=dict(rows=2049, sizes=[2048, 2048, 2048, 2049], tokens=8193, **_TEP4_SPLIT), + ), + Cell( + why="tep4 q-split gather at tc 32767, the largest ragged split: q_sizes [8191, 8192, " + "8192, 8192], a [32767, 2048] int32 result; three calls on new payloads, each " + "bitwise the concatenation, the input untouched, the output fresh", + spec=dict(rows=8192, sizes=[8191, 8192, 8192, 8192], tokens=32767, **_TEP4_SPLIT), + ), + Cell( + why="tep4 q-split gather at max_num_tokens, tc 32768: four 8192-row shares of " + "arbitrary int32 bit patterns move bitwise and repeat bitwise over four calls into " + "four distinct fresh tensors; the input is left untouched", + spec=dict( + rows=8192, + sizes=[8192, 8192, 8192, 8192], + tokens=32768, + payload="bits", + **_TEP4_SPLIT, + ), + ), + Cell( + why="tep4 q-split gather, eager tc 8192..32768: a stride of 61 plus 8193..8195 (each " + "uneven pattern), 16383..16385 and 32767, each rank's input its q_sizes share, the " + "sizes the engine's _split_prefill_queries gives at every tc; one NCCL tuning " + "bucket (ring, SIMPLE, 32 channels) on both paths; each bitwise the concatenation", + spec=dict( + rows=8192, + sweep_tokens=(8192, 32768), + sweep_step=61, + sweep_also=(8193, 8194, 8195, 16383, 16384, 16385, 32767), + **_TEP4_SPLIT, + ), + ), + Cell( + why="tep4 q-split step at max_num_tokens: 44 gathers at tc 32768 (one per selecting " + "indexer layer, in layer order, no synchronization between), each copied into rows " + "[0, tc) of the sentinel-filled selection buffer its input is a view of, as the " + "target does; every layer's rows bitwise the concatenation, none left unwritten", + spec=dict( + rows=8192, + sizes=[8192, 8192, 8192, 8192], + tokens=32768, + sites=_TEP4_SELECTING_LAYERS, + **_TEP4_SPLIT, + ), + ), + Cell( + why="tep4 q-split step, ragged: the last 32751-row chunk of the envelope's " + "1048559-token prompt (31 x 32768 + 32751), q_sizes [8187, 8188, 8188, 8188], 44 " + "gathers in layer order into the sentinel-filled selection buffer; every layer's " + "rows bitwise the concatenation, none left unwritten", + spec=dict( + rows=8188, + sizes=[8187, 8188, 8188, 8188], + tokens=32751, + sites=_TEP4_SELECTING_LAYERS, + **_TEP4_SPLIT, + ), + ), + Cell( + why="tp4 q-split gather at tc 8194, the second remainder: q_sizes [2048, 2049, 2048, " + "2049], four ncclBroadcasts in one NCCL group, each rank's share a view at its " + "offset in the selection buffer; three calls on new payloads, each bitwise the " + "concatenation, the input untouched, the output fresh", + spec=dict(rows=2049, sizes=[2048, 2049, 2048, 2049], tokens=8194, **_TEP4_SPLIT), + ), + Cell( + why="tp4 q-split gather at tc 8195, the third remainder: q_sizes [2048, 2049, 2049, " + "2049], four ncclBroadcasts in one NCCL group, each rank's share a view at its " + "offset in the selection buffer; three calls on new payloads, each bitwise the " + "concatenation, the input untouched, the output fresh", + spec=dict(rows=2049, sizes=[2048, 2049, 2049, 2049], tokens=8195, **_TEP4_SPLIT), + ), + Cell( + why="tp4 q-split step at max_num_tokens on top-k selections: 44 gathers at tc 32768 " + "in layer order, each layer's rows key positions 0..1048575 with -1 tails (rows " + "that see fewer than 2048 keys, rows that see none), copied into rows [0, tc) of " + "the sentinel-filled selection buffer; every layer's rows bitwise the " + "concatenation, none left unwritten", + spec=dict( + rows=8192, + sizes=[8192, 8192, 8192, 8192], + tokens=32768, + sites=_TEP4_SELECTING_LAYERS, + payload="indices", + top_position=_TP4_TOP_POSITION, + **_TEP4_SPLIT, + ), + ), + Cell( + why="tp4 q-split copy-back, eager totals 8192..32768 at a stride of 61 and 32768, " + "sizes the floor split at each: every call's top-k selections (key positions to " + "1048575, -1 tails) staged as each rank's share in its sentinel-filled selection " + "buffer, gathered and copied into rows [0, tc) as PrefillTarget._indexer does; " + "each tc's rows bitwise the concatenation, none left unwritten", + spec=dict( + rows=8192, + sizes="floor_split", + sweep_totals=(8192, 32768), + sweep_step=61, + payload="indices", + top_position=_TP4_TOP_POSITION, + **_TEP4_SPLIT, + ), + ), + Cell( + why="tep8 q-split gather at tc 8192, the engine's q_split_threshold: eight 1024-row " + "int32 shares of [8192, 2048] selections over two nodes (uniform q_sizes, so one " + "ncclAllGather), each a view at its offset in the rank's selection buffer; three " + "calls on new payloads, each bitwise the concatenation, the input untouched, the " + "output fresh", + spec=dict(rows=1024, sizes=[1024] * 8, tokens=8192, **_TEP8_SPLIT), + ), + Cell( + why="tep8 q-split gather at tc 8193, the first ragged split: q_sizes [1024 x 7, " + "1025], eight ncclBroadcasts in one NCCL group over two nodes, each rank's share a " + "view at its offset in the selection buffer; three calls on new payloads, each " + "bitwise the concatenation, the input untouched, the output fresh", + spec=dict(rows=1025, sizes=[1024] * 7 + [1025], tokens=8193, **_TEP8_SPLIT), + ), + Cell( + why="tep8 q-split gather at tc 32736, the V2 scheduler's chunk beside 15 decode " + "rows: q_sizes [4092] * 8 (uniform), a [32736, 2048] int32 result over two nodes; " + "three calls on new payloads, each bitwise the concatenation, the input untouched, " + "the output fresh", + spec=dict(rows=4092, sizes=[4092] * 8, tokens=32736, **_TEP8_SPLIT), + ), + Cell( + why="tep8 q-split gather at tc 32767, the largest ragged split: q_sizes [4095, " + "4096 x 7], a [32767, 2048] int32 result over two nodes; three calls on new " + "payloads, each bitwise the concatenation, the input untouched, the output fresh", + spec=dict(rows=4096, sizes=[4095] + [4096] * 7, tokens=32767, **_TEP8_SPLIT), + ), + Cell( + why="tep8 q-split gather at max_num_tokens, tc 32768: eight 4096-row shares of " + "arbitrary int32 bit patterns over two nodes move bitwise and repeat bitwise over " + "four calls into four distinct fresh tensors; the input is left untouched", + spec=dict( + rows=4096, + sizes=[4096] * 8, + tokens=32768, + payload="bits", + **_TEP8_SPLIT, + ), + ), + Cell( + why="tep8 q-split gather, eager tc 8192..32768 over two nodes: a stride of 61 plus " + "8193..8199 (every uneven pattern at eight ranks), 16383..16385 and 32767, each " + "rank's input its q_sizes share, the sizes the engine's _split_prefill_queries " + "gives at every tc; each bitwise the concatenation", + spec=dict( + rows=4096, + sweep_tokens=(8192, 32768), + sweep_step=61, + sweep_also=(8193, 8194, 8195, 8196, 8197, 8198, 8199, 16383, 16384, 16385, 32767), + **_TEP8_SPLIT, + ), + ), + Cell( + why="tep8 q-split gather at NCCL's protocol switch: every tc 14273..14288 over two " + "nodes, whose largest share is 1785 rows (RING, LL128 through tc 14280) or 1786 " + "(RING, SIMPLE from tc 14281), NCCL tuning each call on 8 x its largest share's " + "bytes; each bitwise the concatenation", + spec=dict(rows=1786, sweep_tokens=(14273, 14288), sweep_step=1, **_TEP8_SPLIT), + ), + Cell( + why="tep8 q-split step at max_num_tokens: 44 gathers at tc 32768 over two nodes " + "(one per selecting indexer layer, in layer order, no synchronization between), each " + "copied into rows [0, tc) of the sentinel-filled selection buffer its input is a " + "view of, as the target does; every layer's rows bitwise the concatenation, none " + "left unwritten", + spec=dict( + rows=4096, + sizes=[4096] * 8, + tokens=32768, + sites=_TEP8_SELECTING_LAYERS, + **_TEP8_SPLIT, + ), + ), + Cell( + why="tep8 q-split step, ragged: the last 32751-row chunk of the envelope's " + "1048559-token prompt (31 x 32768 + 32751), q_sizes [4093, 4094 x 7], 44 gathers " + "in layer order over two nodes into the sentinel-filled selection buffer; every " + "layer's rows bitwise the concatenation, none left unwritten", + spec=dict( + rows=4094, + sizes=[4093] + [4094] * 7, + tokens=32751, + sites=_TEP8_SELECTING_LAYERS, + **_TEP8_SPLIT, + ), + ), ) note = """ - The ragged form (`sizes` as an explicit per-rank vector) and dtypes other - than bf16 work and are exercised by the op matrix, but no shipped target - reaches them, so they are not cells. - - CUDA graph behaviour -- capture at every engine batch size, replays - surviving eager calls of other shapes, the captured sizes vector being held - -- is not a cell axis either: it is about the op across capture and replay - rather than about an input configuration, and the matrix carries it as its - own named checks. + Two communicator flavours, one per symbol (see `__call__`). The cells that + state `process_group=None` drove `torch.ops.trtllm.allgather` on the + MPI-session communicator: at `world_size=4` four ranks of one node started + by mpirun, as a dep4, tep4 or tp4 world is; at `world_size=8` eight ranks + over two GB300 nodes of four (`nodes=2`, one NVL72 segment, one MNNVL + clique of 8) started by the session's MPI launcher across both nodes, as a + dep8 or tep8 world is. `getComm` (`cpp/tensorrt_llm/common/opUtils.cpp`) + builds one NCCL communicator per `group` set at that set's first call, + through the MPI session, and caches it for every op that names the same set + (the matrix captures this entry and reducescatter on it in one graph); it + sets NCCL_RUNTIME_CONNECT=0 and NCCL_GRAPH_REGISTER=0 where they are unset. + With `sizes=None` the call is one ncclAllGather of `input.numel()` elements + per rank. With a `sizes` list the result has `sum(sizes)` rows: when every + entry is equal it is that same one ncclAllGather, and otherwise one + ncclBroadcast per rank of the group inside one NCCL group, the i-th rank + (ascending) sending `sizes[i]` rows from its input to row `sum(sizes[:i])` + of every rank's result. The cells without the key drove `allgather_pg` on a + boxed c10d group. + + Preconditions on the MPI-session symbol: every rank of `group` makes the + same calls in the same order at the same shape. Calls pair by position on + the communicator; the op matrix's call-order check measures, on the `_pg` + symbol, that a mispairing at equal byte counts returns wrong data silently. + The group's first call builds the communicator, so it is made eagerly: the + graph cells warm the group before capture, as the engine's warmup does. + Capturing a group's first call is not certified on this symbol. With a + `sizes` list, `sizes[i]` must be the i-th rank's `input.shape[0]`: the op + takes the row count from `sizes`, never from the input. Measured at tc + 8193 on four ranks: a rank whose input was one row short of its entry sent + the row after its input in memory; one row long, its last row was dropped; + with every entry 2049 and every input 2048 rows, the blocks landed at + 2048-row offsets and the result's last four rows were never written. + Measured again at tc 8193 on eight ranks over two nodes, q_sizes [1024 x 7, + 1025]: the last rank's input one row short of its entry sent the row after + its input in memory; rank 0's input one row long, its last row was dropped; + with every entry 1025 and every input 1024 rows, the blocks landed at + 1024-row offsets and the result's last eight rows were never written. Both + times every rank got the same wrong result, and nothing raised. All silent, + so `is_valid` refuses a `sizes` entry that is not this rank's input height + on this symbol (the rank from the MPI session). + + Measured on 4 sm_103 ranks at width 6144 bf16 (NCCL's TUNING log, every + rows to 2048 and every 8th to 32768): the RING algorithm at every row + count, LL through 154 rows, LL128 from 155, SIMPLE from 991, with the + channel count moving between 3 and 32 below 621 rows and 32 from there on; + every rank made the same choice on every call, and the captured batches + 1/2/4/8/16 get eager's algorithm, protocol and channels. The op moves + bytes: every rows 1..32768 was driven once on the MPI-session symbol and + each result was the concatenation bit for bit (47.9 s, past one cell's + budget, so the cells keep every value to 2048 and a stride above it). + + Measured on 8 sm_103 ranks over two GB300 nodes at width 6144 bf16: the + eight ranks form one MNNVL clique (NCCL reports nNodes 1 and localRanks 8, + and connects every ring link P2P/MNNVL). Rank 0's TUNING log (NCCL logs + the choice on rank 0 only), every rows to 2048 and every 8th to 32768: the + RING algorithm at every row count -- NVLS is available on the devices and + never chosen for this op -- LL through 96 rows, LL128 from 97, SIMPLE from + 1191, with the channel count moving between 3 and 32 below 311 rows and 32 + from there on; the captured batches 1..16 get eager's algorithm, protocol + and channels. Every rows 1..32768 was driven once on the MPI-session symbol + at world 8 and each result was the concatenation bit for bit on all eight + ranks, a sweep past one cell's budget, so the dep8 cells too keep every + value to 2048 and a stride above it. + + The tep4 q-split gather (the cells stating `q_split` at `world_size=4`), + measured on 4 sm_103 ranks at width 2048 int32 (8 KiB rows), NCCL 2.31. Its + sizes are what the engine's `_split_prefill_queries` (`dsa/indexer.py`) + gives tp 4: checked against it at every tc 1..32768, unsplit below 8192 and + [(r + 1) * tc // 4 - r * tc // 4] from there on, so q_sizes are equal when + 4 divides tc and uneven by one row otherwise. NCCL's TUNING log over 420 + split sizes (tc 8192..8199, a stride of 61 to 32768, the edges): RING, + SIMPLE, 32 channels at every one, on every rank. The uniform ones run the + ncclAllGather, and NCCL tunes the ragged ones' grouped broadcasts as one + AllGather of the total bytes. The op moves bytes: every tc 8192..32768 was + driven once on the MPI-session symbol, each rank's input its share of a + fresh payload, and each result was the concatenation bit for bit (24577 + calls, 10.3 s, past one cell's budget, so the sweep cell keeps a stride of + 61 and the edges). A split step's 44 gathers, issued in layer order with no + synchronization between them and each copied into rows [0, tc) of the + buffer its input is a view of, left every layer's rows bitwise right and + no row unwritten, at tc 32768 and 32751. Warm, a ragged split's gather + costs more than the uniform one beside it. + + The tp4 deployment makes the same call (its indexer replicated, its TP group + the same four ranks, its split the same: `_split_prefill_queries` agrees on + its Mapping too, moe_tp_size 4). Measured again on a tp4 session (4 sm_103 + ranks, NCCL 2.31.2) in the target's own form -- each rank's share staged in + a sentinel-filled selection buffer, gathered, and the result copied into + rows [0, tc) of that buffer -- on top-k shaped payloads (key positions + 0..1048575 in a row's first slots, -1 in the rest: rows with a -1 tail, + rows of -1 only, rows of 2048 positions): every tc 8192..32768 once, a + fresh payload per call, each bitwise the shares' concatenation with no row + unwritten (24577 calls, 19.2 s, past one cell's budget, so the copy-back + cell keeps a stride of 61); a step's 44 such gathers in layer order left + every layer right at tc 32768, 32751 and 8195; NCCL's TUNING log over the + same 420 tc chose RING, SIMPLE, 32 channels on every rank. Warm, a gather + at a ragged remainder costs more than the uniform one beside it. + + The tep8 q-split gather (the cells stating `q_split` at `world_size=8`), + measured on 8 sm_103 ranks over two GB300 nodes of four (one MNNVL clique + of 8) at width 2048 int32 (8 KiB rows). Its sizes are what the engine's + `_split_prefill_queries` (`dsa/indexer.py`) gives the tep8 Mapping (tp 8, + moe_ep 8, attention DP off): checked against it at every tc 1..32768, + unsplit below 8192 and [(r + 1) * tc // 8 - r * tc // 8] from there on, so + q_sizes are equal when 8 divides tc and uneven by one row otherwise. NCCL's + TUNING log over 502 calls (tc 8192..8199, a stride of 61 to 32768, the + edges, every tc 14224..14304), read call by call: NCCL tunes a call when + its group ends, on 8 x the largest share's bytes, and chose RING with 32 + channels at every one, on every rank, with the LL128 protocol through tc + 14280 (largest share 1785 rows) and SIMPLE from tc 14281 (1786 rows), for + the uniform ncclAllGather and the ragged grouped ncclBroadcasts alike. The + op moves bytes: every tc 8192..32768 was driven once on the MPI-session + symbol, each rank's input its share of a fresh payload, and each result was + the concatenation bit for bit on all eight ranks (24577 calls, 11.0 s, past + one cell's budget, so one cell keeps a stride of 61 and the edges and + another every tc across the protocol switch). A split step's 44 gathers, + issued in layer order with no synchronization between them and each copied + into rows [0, tc) of the buffer its input is a view of, left every layer's + rows bitwise right and no row unwritten, at tc 32768, 32751 and 8193. Warm, + a ragged split's gather costs more than a uniform one beside it. + + The ragged form and dtypes other than bf16 and int32 also work on the + `_pg` symbol, which the op matrix exercises; only the q-split's ragged form + is a cell. Not certified on the MPI-session symbol: a ragged `sizes` other + than the q-split's (int32 at width 2048, summing to 8192..32768 over four + ranks of one node or eight ranks on two nodes, uneven by at most one row), + other dtypes and widths, a group other than all four ranks of one node or + all eight ranks of two nodes, other world sizes and node counts, rows above + 32768 (per rank in the uniform form, in all in the q-split's), and a ragged + gather captured in a CUDA graph. + + CUDA graph behaviour on the `_pg` symbol -- capture at every engine batch + size, replays surviving eager calls of other shapes, the captured sizes + vector being held -- is not a cell axis: it is about the op across capture + and replay rather than about an input configuration, and the matrix carries + it as its own named checks. On the MPI-session symbol the graph cells above + are the dep4 and dep8 decode graphs themselves. + + Hard limits (sm_103), the uniform form at width 6144 bf16, world 4 + output bytes 4 * rows * 12288 vs 2^31 reached at rows 43691 + not reached by dep4 (32768: 1,610,612,736 bytes) not probed + output numel 4 * rows * 6144 vs 2^31 reached at rows 87382 + not reached not probed + NCCL count input.numel() as size_t; no index passes through fp32 + + Hard limits (sm_103), the uniform form at width 6144 bf16, world 8 over two nodes + output bytes 8 * rows * 12288 vs 2^31 reached at rows 21846, inside dep8 + (32768: 3,221,225,472 bytes) probed at every rows 21845..32768 + once, and kept as cells at 21845, 21846, 21847, a stride of 61 + and 32767/32768 passed, bitwise on all eight ranks + output numel 8 * rows * 6144 vs 2^31 reached at rows 43691 + not reached by dep8 (32768: 1,610,612,736) not probed + NCCL count input.numel() as size_t; no index passes through fp32 + + Hard limits (sm_103), the q-split form at width 2048 int32, world 4 + output bytes tc * 8192 vs 2^31 reached at tc 262144 + not reached by tep4 or tp4 (32768: 268,435,456 bytes) + not probed + output numel tc * 2048 vs 2^31 reached at tc 1048576 + not reached (32768: 2^26 elements) not probed + sum(sizes) summed into an int wraps at 2^31 rows not reached + NCCL count per root numel_base * sizes[i] as size_t, numel_base (the + trailing elements, 2048) multiplied into an int; the row + offset is torch's int64; no index passes through fp32 + + Hard limits (sm_103), the q-split form at width 2048 int32, world 8 + output bytes tc * 8192 vs 2^31 reached at tc 262144 + not reached by tep8 (32768: 268,435,456 bytes) not probed + output numel tc * 2048 vs 2^31 reached at tc 1048576 + not reached not probed + sum(sizes) summed into an int wraps at 2^31 rows not reached + NCCL count per root numel_base * sizes[i] as size_t, numel_base (the + trailing elements, 2048) multiplied into an int; the row + offset is torch's int64; no index passes through fp32 """ def __call__( @@ -65,8 +619,9 @@ def __call__( ) -> torch.Tensor: """`process_group` selects the communicator flavor, mirroring the product router in `_torch/distributed/ops.py`: None runs the - MPI-session symbol -- the one a dep4 serving world calls -- and a boxed - c10d ProcessGroup runs the `_pg` symbol the non-MPI backends use. + MPI-session symbol -- the one a dep4, tep4, tp4, dep8 or tep8 serving + world calls -- and a boxed c10d ProcessGroup runs the `_pg` symbol the + non-MPI backends use. """ if process_group is None: return torch.ops.trtllm.allgather(input, sizes, group) @@ -112,6 +667,14 @@ def is_valid( "so a short list silently drops the trailing ranks and a long one " "appends uninitialized rows" ) + if process_group is None and sizes is not None and mpi_rank() in group: + mine = sorted(set(group)).index(mpi_rank()) + assert sizes[mine] == input.shape[0], ( + f"sizes[{mine}]={sizes[mine]} must equal this rank's input.shape[0]=" + f"{input.shape[0]}; the MPI-session op sends sizes[{mine}] rows from the " + "input whatever its height, so a shorter input is read past its end, a " + "longer one is cut, and equal sizes pack the blocks at the inputs' offsets" + ) allgather = Allgather() diff --git a/tensorrt_llm/_torch/modeling_v2/catalog/comm/reducescatter.py b/tensorrt_llm/_torch/modeling_v2/catalog/comm/reducescatter.py index 272efd7a64f2..2f807b50e94e 100644 --- a/tensorrt_llm/_torch/modeling_v2/catalog/comm/reducescatter.py +++ b/tensorrt_llm/_torch/modeling_v2/catalog/comm/reducescatter.py @@ -30,6 +30,43 @@ } ) +# Mistral Large 4 NVFP4 at dep4: the combine after the routed experts of every MoE +# layer (1..71). Each of the four ranks passes the runner's finalized +# [4 x T_max, 6144] bf16 partial and keeps its T_max rows, sizes=None, on the +# MPI-session communicator a dep4 world launched under mpirun reaches. +_DEP4_COMBINE = dict( + trailing=(6144,), + dtype=torch.bfloat16, + sizes=None, + world_size=4, + process_group=None, +) +# The decode batches the dep4 engine captures per rank (padded), and the +# combines one captured decode step issues. +_DEP4_GRAPH_ROWS = (1, 2, 4, 8, 16) +_DEP4_MOE_LAYERS = 71 + +# Mistral Large 4 NVFP4 at dep8: the same combine over the eight attention-DP ranks +# of two GB300 nodes of four (one NVL72 segment, so NCCL forms an MNNVL clique +# of 8), group [0..7]. Each rank passes the runner's finalized [8 x T_max, 6144] +# bf16 partial its 16-expert window wrote for every gathered token and keeps +# its T_max rows, sizes=None, on the MPI-session communicator an eight-rank +# world launched under mpirun across both nodes reaches. `nodes` is the number +# of hosts the ranks span. +_DEP8_COMBINE = dict( + trailing=(6144,), + dtype=torch.bfloat16, + sizes=None, + world_size=8, + process_group=None, + nodes=2, +) +# The decode batches a dep8 engine captures per rank -- the gates' padded +# 1/2/4/8/16 and the LLM API's unpadded default at max_batch_size 16, every +# count 1..16 -- and the combines one captured decode step issues. +_DEP8_GRAPH_ROWS = tuple(range(1, 17)) +_DEP8_MOE_LAYERS = 71 + class Reducescatter(OpWrapper): """Sum `input` across every rank in `group`, then keep this rank's slice. @@ -43,10 +80,13 @@ class Reducescatter(OpWrapper): ARCHS = frozenset({Arch.SM_100, Arch.SM_103}) - # deepseek-r1 at dep4 and the gpt-oss default target under attention DP - # are the callers, and both always pass `sizes=None`: each pads to a - # common height first. The row counts are the full heights an - # attention-DP step reduces, which is the rank count times a slice. + # deepseek-r1 at dep4, the gpt-oss default target under attention DP and + # Mistral Large 4 NVFP4 at dep4 and dep8 are the callers, and all pass + # `sizes=None`: each pads to a common height first. The row counts are the + # full heights an attention-DP step reduces, which is the rank count times a + # slice. The first three cells drove the `_pg` symbol (a boxed c10d group); + # every cell stating `process_group=None` drove the MPI-session symbol, at + # the world size it states. CELLS: tuple[Cell, ...] = ( Cell( why="decode: one row per rank after the split, hidden 7168", @@ -60,13 +100,185 @@ class Reducescatter(OpWrapper): why="prefill chunk height", spec=dict(rows_per_rank=2048, trailing=(7168,), dtype=torch.bfloat16, sizes=None), ), + *( + Cell( + why=f"dep4 MoE combine, decode graph at {rows} row(s) per rank: captured with " + f"{_DEP4_MOE_LAYERS} combines (MoE layers 1..71) beside the other captured " + "batches in one pool; every replay on new data bitwise equal to the reference " + "and to eager", + spec=dict( + rows_per_rank=rows, + graph_rows=_DEP4_GRAPH_ROWS, + sites=_DEP4_MOE_LAYERS, + **_DEP4_COMBINE, + ), + ) + for rows in _DEP4_GRAPH_ROWS + ), + Cell( + why="dep4 MoE combine, eager rows_per_rank every value 1..2048: every protocol and " + "channel-count switch NCCL's ring makes at this width (LL through 154 rows, LL128 " + "155..990, SIMPLE from 991), each bitwise equal to the reference", + spec=dict(rows_per_rank=2048, sweep_rows=(1, 2048), sweep_step=1, **_DEP4_COMBINE), + ), + Cell( + why="dep4 MoE combine, eager rows_per_rank 2049..32768: a stride of 61 rows through " + "the SIMPLE range plus every 2^k - 1, 2^k and 2^k + 1 up to max_num_tokens 32768 " + "(input 131072 x 6144), each bitwise equal to the reference", + spec=dict( + rows_per_rank=32768, + sweep_rows=(2049, 32768), + sweep_step=61, + sweep_also=(4095, 4096, 4097, 8191, 8192, 8193, 16383, 16384, 16385, 32767), + **_DEP4_COMBINE, + ), + ), + Cell( + why="dep4 MoE combine at max_num_tokens on every rank (rows_per_rank 32768): a " + "random payload stays inside the bf16 sequential-sum band of the reference and " + "repeats bitwise; the input is left untouched and the output is fresh", + spec=dict(rows_per_rank=32768, payload="random", **_DEP4_COMBINE), + ), + *( + Cell( + why=f"dep8 MoE combine, decode graph at {rows} row(s) per rank, 8 ranks over two " + f"nodes: captured with {_DEP8_MOE_LAYERS} combines (MoE layers 1..71) beside the " + "other captured batches 1..16 in one pool, after an eager warmup of the group; " + "every replay on new data bitwise equal to the reference and to eager", + spec=dict( + rows_per_rank=rows, + graph_rows=_DEP8_GRAPH_ROWS, + sites=_DEP8_MOE_LAYERS, + **_DEP8_COMBINE, + ), + ) + for rows in _DEP8_GRAPH_ROWS + ), + Cell( + why="dep8 MoE combine, eager rows_per_rank every value 1..2048, 8 ranks over two " + "nodes: every protocol and channel-count switch NCCL's ring makes at this width (LL " + "through 96 rows, LL128 97..1190, SIMPLE from 1191), each bitwise equal to the " + "reference", + spec=dict(rows_per_rank=2048, sweep_rows=(1, 2048), sweep_step=1, **_DEP8_COMBINE), + ), + Cell( + why="dep8 MoE combine, eager rows_per_rank 2049..32768, 8 ranks over two nodes, one " + "NCCL tuning bucket (ring, SIMPLE, 32 channels): a stride of 61 rows plus every " + "2^k - 1, 2^k and 2^k + 1 and rows_per_rank 21845/21846/21847, where the input passes " + "2^31 bytes, up to max_num_tokens 32768 (input 262144 x 6144, 3,221,225,472 bytes), " + "each bitwise equal to the reference", + spec=dict( + rows_per_rank=32768, + sweep_rows=(2049, 32768), + sweep_step=61, + sweep_also=( + 4095, + 4096, + 4097, + 8191, + 8192, + 8193, + 16383, + 16384, + 16385, + 21845, + 21846, + 21847, + 32767, + ), + **_DEP8_COMBINE, + ), + ), + Cell( + why="dep8 MoE combine at max_num_tokens on every rank (rows_per_rank 32768), 8 ranks " + "over two nodes: a random payload stays inside the bf16 sequential-sum band of the " + "reference at W = 8 and repeats bitwise; the input is left untouched and the output " + "is fresh", + spec=dict(rows_per_rank=32768, payload="random", **_DEP8_COMBINE), + ), ) note = """ + Two communicator flavours, one per symbol (see `__call__`). The cells that + state `process_group=None` drove `torch.ops.trtllm.reducescatter` on the + MPI-session communicator: at `world_size=4` four ranks of one node started + by mpirun, as a dep4 world is; at `world_size=8` eight ranks over two GB300 + nodes of four (`nodes=2`, one NVL72 segment) started by the session's MPI + launcher across both nodes, as a dep8 world is. + `getComm` (`cpp/tensorrt_llm/common/opUtils.cpp`) builds and caches one + NCCL communicator per `group` set at that group's first call, through the + MPI session, and sets NCCL_RUNTIME_CONNECT=0 and NCCL_GRAPH_REGISTER=0 where + they are unset. The cells without the key drove `reducescatter_pg` on a + boxed c10d group. + + Preconditions on the MPI-session symbol: every rank of `group` makes the + same calls in the same order at the same shape. Calls pair by position on + the communicator; the op matrix's call-order checks measure, on the `_pg` + symbol, that a mispairing at equal byte counts returns wrong data silently. + The group's first call builds the communicator, so it is made eagerly: the + graph cells warm the group before capture, as the engine's warmup does. + Capturing a group's first call is not certified. + + Numerics, measured on 4 sm_103 ranks at width 6144 bf16. NCCL's tuner (its + TUNING log) picks the RING algorithm at every rows_per_rank 1..1100 and at + every power of two up to 32768: LL through 154 rows, LL128 from 155, SIMPLE + from 991, with the channel count changing many times below 622 rows; the + captured batches 1/2/4/8/16 get eager's algorithm, protocol and channels. + On random payloads at 1, 16, 256, 2048 and 32768 rows the result is the + bf16 chain in ring order from the destination's successor, deterministic, + and bitwise the same on both symbols. It stays inside + (W - 1) * 2^-8 * sum_r |x_r| of the exact sum (largest ratio 0.956, at 32768 + rows) and differs from an fp32 sum rounded once in about a third of the + elements. + + Every rows_per_rank 1..32768 at width 6144 was driven once on the + MPI-session symbol, bitwise equal to the reference: 39.1 s, past one cell's + budget, so the cells keep every value to 2048 and a stride above it. + + Measured on 8 sm_103 ranks over two GB300 nodes at width 6144 bf16: the + eight ranks form one MNNVL clique (NCCL reports nNodes 1 and localRanks 8), + and every one of the communicator's 32 rings runs 0 1 2 ... 7, each link + P2P/MNNVL. Rank 0's TUNING log (NCCL logs the choice on rank 0 only), every + rows_per_rank to 2048 and every 8th to 32768: the RING algorithm at every + size -- NVLS is available on the devices and never chosen for this op -- LL + through 96 rows, LL128 from 97, SIMPLE from 1191, with the channel count + moving between 3 and 32 below 311 rows and 32 from there on; the captured + batches 1..16 get eager's algorithm, protocol and channels. On random + payloads at 1, 16, 256, 2048, 21846 and 32768 rows the result is again the + bf16 chain in ring order from the destination's successor, deterministic, + on all eight ranks. It stays inside (W - 1) * 2^-8 * sum_r |x_r| of the + exact sum, W = 8 (largest ratio 0.655, at 2048 rows; 0.649 at 32768 rows + over three payloads), and differs from an fp32 sum rounded once in about + half of the elements. + + Every rows_per_rank 1..32768 was driven once on the MPI-session symbol at + world 8 and each slice was the reference bit for bit on all eight ranks + (77.2 s), so the world-8 cells too keep every value to 2048 and a stride + above it. + The ragged form and dtypes other than bf16 work and are exercised by the op - matrix, but no shipped target reaches them, so they are not cells. CUDA - graph behaviour is carried by the matrix's own named checks rather than as - a cell axis. + matrix on the `_pg` symbol, but no shipped target reaches them, so they are + not cells. Not certified on the MPI-session symbol: a ragged `sizes`, other + dtypes and widths, a group other than all four ranks of one node or all + eight ranks of two nodes, other world sizes and node counts, and + rows_per_rank above 32768. + + Hard limits (sm_103), the uniform split at width 6144 bf16 + input bytes 4 * rows_per_rank * 12288 vs 2^31 reached at rows_per_rank 43691 + not reached by dep4 (32768: 1,610,612,736 bytes) not probed + input numel 4 * rows_per_rank * 6144 vs 2^31 reached at rows_per_rank 87382 + not reached not probed + NCCL count output.numel() as size_t; no index passes through fp32 + + Hard limits (sm_103), the uniform split at width 6144 bf16, world 8 over two nodes + input bytes 8 * rows_per_rank * 12288 vs 2^31 reached at rows_per_rank 21846, + inside dep8 (32768: 3,221,225,472 bytes) probed at every + rows_per_rank 21845..32768 once, and kept as cells at 21845, 21846, + 21847, a stride of 61 and 32767/32768 passed, bitwise on all + eight ranks + input numel 8 * rows_per_rank * 6144 vs 2^31 reached at rows_per_rank 43691 + not reached by dep8 (32768: 1,610,612,736) not probed + NCCL count output.numel() as size_t; no index passes through fp32 """ def __call__( diff --git a/tensorrt_llm/_torch/modeling_v2/catalog/gemm/bmm_out.py b/tensorrt_llm/_torch/modeling_v2/catalog/gemm/bmm_out.py index 65fa2f184d98..5e24996eeac6 100644 --- a/tensorrt_llm/_torch/modeling_v2/catalog/gemm/bmm_out.py +++ b/tensorrt_llm/_torch/modeling_v2/catalog/gemm/bmm_out.py @@ -11,20 +11,24 @@ class BmmOut(OpWrapper): - """`out[i] = a[i] @ b[i]` for every batch index, in place, returns None. + """`out[i] = a[i] @ b[i]` for every batch index, written into `out`, which + is returned. - The point of the op over `torch.bmm` is the caller-owned `out`: both - shipped call sites pass a transposed view of a buffer the rest of the + The point of the op over `torch.bmm` is the caller-owned `out`: every + shipped call site passes a transposed view of a buffer the rest of the forward already holds, so the product lands where it is needed with no copy. That is also the hazard -- see `is_valid`. + + Fusion boundary: the product only. The weight regroup and transposition, + and the buffers `a` and `out` are views of, are the caller's. """ ARCHS = frozenset({Arch.SM_100, Arch.SM_103}) - # deepseek-r1's MLA is the only caller, and it uses the op twice per layer - # in opposite orientations: absorbing the nope half of q into the latent - # space on the way in, and projecting the latent attention output back out - # to v. Batch is the head count (128, replicated under attention DP). + # deepseek-r1's MLA uses the op twice per layer in opposite orientations: + # absorbing the nope half of q into the latent space on the way in, and + # projecting the latent attention output back out to v. Batch is the head + # count (128, replicated under attention DP). CELLS: tuple[Cell, ...] = ( Cell( why="decode, q_nope -> latent: [H, 1, nope] @ [H, nope, kv_lora]", @@ -46,7 +50,142 @@ class BmmOut(OpWrapper): ) note = """ + What runs: `trtllm::bmm_out` is a `torch.library.custom_op` registered in + `tensorrt_llm/_torch/custom_ops/torch_custom_ops.py` whose body is + `torch.bmm(a, b, out=out)`, so one call is one cuBLAS strided-batched GEMM. + At Mistral Large 4's views the profiler shows a single `nvjet_sm103_tst_*` + kernel per call and no copy, at each of 34 values of m from 1 to 32768 and + on the trailing 15 rows of 32768-row buffers: the product is written + straight into the caller's view. cuBLAS picks the kernel by size -- over + those 34 values it named 14, 13 and 15 distinct kernels for K-B RoPE, K-B + NoPE and V-B, 5, 4 and 5 of them at the 13 values above 9216 -- and every + m's rows are still bitwise the leading rows of the full-size run, which + the test's 16-head sweep cells hold at 9216 and at 32768 rows. A transposed-view + `b` and the same values made contiguous give bitwise the same product. At + those three shapes and 32768 rows the product is within one bf16 rounding + of an fp64 product. The cells run under torch's matmul defaults + (`torch.backends.cuda.matmul.allow_bf16_reduced_precision_reduction` True). + + At DEP4's 64 heads the profiler shows one `nvjet_sm103_tst_*` launch per + call and no copy, at every m from 1 to 32768: 36, 42 and 67 + distinct kernels for K-B RoPE, K-B NoPE and V-B, in 770, 634 and 1936 runs + of m with one kernel and grid, every change between m = 8 and 9 or + between 16k and 16k + 1. Every m's rows are bitwise the leading + rows of the 32768-row run: the `large` cells drive each m, and pre-merge + drives every m to 16 and both edges of every 16-row window, which holds + both edges of every run. A transposed-view `b` and the same values made + contiguous give bitwise the same product here too. At these shapes and + 32768 rows the product is within one bf16 rounding of an fp64 product. + + At TEP8's and TP8's 8 heads the profiler shows one `nvjet_sm103_tst_*` + launch per call and no copy, at every m from 1 to 32768: 58, 59 and 86 + distinct kernels for K-B RoPE, K-B NoPE and V-B, in 615, 658 and 1563 runs + of m with one kernel and grid, every change between m = 8 and 9 or between + 16k and 16k + 1. Every m's rows are bitwise the leading rows of the + 32768-row run: the test's 8-head sweep cells drive each m in pre-merge. A + transposed-view `b` and the same values made contiguous give bitwise the + same product here too. At these shapes and 32768 rows the product is + within one bf16 rounding of an fp64 product. + + On the fp8_ds_mla latent KV a tep8 or tp8 rank's K-B output is heads 0..7 + of the `[T, 64, 576]` fused query FlashMLA takes (the test's `_ML4_8_FQ64`): the same + batch of 8, an output token stride of 64 x 576 = 36864 instead of 8 x 576, + and heads 8..63 left as the step zeroed them. Those cells hold the product + there at every m 1..32768, each m's rows bitwise the 32768-row run's (and + at m 1..16 and both edges of every 16-row window on their own), in CUDA + graphs at every batch 1..16 and on a mixed step's 15-row tail, with heads + 8..63 never written. The profiler shows one `nvjet_sm103_tst_*` launch + per call and no copy at each of 40 values of m from 1 to 32768 and on the + 15-row tail, at every one of them the kernel cuBLAS picks for the compact + 8-head layout: 18 distinct for K-B RoPE, 14 for K-B NoPE (tep8's + profile). At 32768 rows the product is within one bf16 rounding of an + fp64 product here too. + + On the fp8_ds_mla latent KV a tep4 (or tp4) rank's K-B output is heads + 0..15 of the `[T, 64, 576]` fused query FlashMLA takes (the test's `_ML4_16_FQ64`): the + same batch of 16, an output token stride of 64 x 576 = 36864 instead of + 16 x 576, and heads 16..63 left as the step zeroed them. Those cells hold + the product there at every m 1..32768, each m's rows bitwise the + 32768-row run's, in CUDA graphs at every batch 1..16 and on a mixed + step's 15-row tail, with heads 16..63 never written. The profiler shows + one `nvjet_sm103_tst_*` launch per call and no copy at each of 40 values + of m from 1 to 32768 and on the 15-row tail, at every one of them the + kernel cuBLAS picks for the compact 16-head layout: 16 distinct for K-B + RoPE, 14 for K-B NoPE (tep4's profile; tp4's, at 32 values of m and the + tail, found the same at every one, 13 distinct and 11). At 32768 rows the + product is within one bf16 rounding of an fp64 product here too. + `out` is written, not accumulated into: its prior contents are ignored. + m = 0 returns at once and writes nothing. Capturable: a captured call + replays bitwise equal to an eager one. Metadata consumed: none. + + Refused loudly by the op itself, each kind held by the test: `a` or `b` + not 3-D (`batch1 must be a 3D tensor`), a batch or K mismatch, and a mix of + dtypes -- every pairing of bf16, fp16 and fp32 with `a` and `b` differing, + under each `out` dtype, and the two `out` dtypes other than the operands' + that were driven, bf16 into fp32 and fp16 into bf16 (`expected scalar type + BFloat16 but found Half`, or `Expected out tensor to have dtype ...`). + Refused by `is_valid`, beside the 3-D and single-dtype checks it restates: + an `out` of the wrong shape, which the op resizes in place with only a + deprecation warning. A standalone `out` is + re-allocated, so a buffer the caller aliased with it never sees the + product. A Mistral Large 4 K-B view one token short is re-strided contiguous + over the fused query's storage: the product lands there scrambled for the + buffer's layout, and over its rope halves. + + Not in any cell: fp16 or fp32 operands (accepted), a contiguous `a` or + `out` (accepted), an `out` whose last dimension is not unit-stride, `a` + overlapping `out`, batch counts other than 8, 16 and 64, a batch written + into a wider head buffer other than 16 of 64 and 8 of 64, and m above + 32768 at 64 heads, at 16 of 64 or at 8 of 64 (four probes, below). + + Hard limits (sm_103). The NVFP4 target's m <= 32768 reaches at most + 1,207,959,552 elements into the fused query at 64 heads (56% of 2^31) -- + the 64-head fused query a tep4 or tp4 rank writes 16 heads of, and a tep8 + rank 8 heads of, is that size too, the last element written 1,207,931,839 + and 1,207,927,231 -- 301,989,888 at 16 and 150,994,944 at 8; the FP8 + target's m <= 9216 at most 84,934,656. + row offset m x token stride first m with an offset at 2^31: + 64 heads K-B out (36864) 58,255 -- probed at m 58260 + on K-B RoPE and NoPE: passed, every row + within the band, nothing written past n, + calls on the rows from 58247, 58254, 58255 + and 58257 bitwise the full call's rows and + nothing written ahead of them; + V-B a (32768) 65,537 -- probed at m 65540: + passed, every row within the band, calls + on the rows from 65529, 65536 and 65537 + the same; K-B a and V-B out (8192) + 262,145, computed + 16 heads K-B out (9216) 233,017; V-B a (8192) + 262,145; K-B a and V-B out (2048) + 1,048,577, computed + 8 heads K-B out (4608) 466,034; V-B a (4096) + 524,289; K-B a and V-B out (1024) + 2,097,153, computed + 16 of 64 heads K-B out (36864) 58,255 -- probed at m 58260 + on K-B RoPE and NoPE, heads 0..15 of a + 64-head buffer (last element written + 2,147,668,927): passed, every row within + the band, nothing written past n or into + heads 16..63, calls on the rows from + 58247, 58254, 58255 and 58257 (storage + offsets to 2,147,586,048) bitwise the + full call's rows and nothing written + ahead of them + 8 of 64 heads K-B out (36864) 58,255 -- probed at m 58260 + on K-B RoPE and NoPE, heads 0..7 of a + 64-head buffer (last element written + 2,147,664,319): passed, every row within + the band, nothing written past n or into + heads 8..63, calls on the rows from + 58247, 58254, 58255 and 58257 (storage + offsets to 2,147,586,048) bitwise the + full call's rows and nothing written + ahead of them + m the row count 2^31, computed + No index passes through fp32. The op touches no KV pool, only the step's + activations, so the pool's size sets no limit on it. """ def __call__(self, a: torch.Tensor, b: torch.Tensor, out: torch.Tensor) -> torch.Tensor: diff --git a/tensorrt_llm/_torch/modeling_v2/catalog/gemm/cublas_mm.py b/tensorrt_llm/_torch/modeling_v2/catalog/gemm/cublas_mm.py index a04be7d6f938..40058aaffe40 100644 --- a/tensorrt_llm/_torch/modeling_v2/catalog/gemm/cublas_mm.py +++ b/tensorrt_llm/_torch/modeling_v2/catalog/gemm/cublas_mm.py @@ -35,7 +35,119 @@ class CublasMm(OpWrapper): note = """ `output_buffer_kind` and `group` are the op's multi-stream and collective arguments. The covered targets leave them at their defaults, so neither is - a cell axis and neither is certified here. + a cell axis and neither is certified here. Metadata consumed: none. + + Kernel selection (sm_103). The thop caller first looks the call up in a + fixed bf16 algorithm table (`bf16_algo_list` in `thop/cublasScaledMMLut.h`, + consulted by `find_special_algo` in `thop/cublasScaledMM.cpp`, keyed on the + next power of two of m, at least 8, with k and n). No cell's key is in it, + so every cell runs on cuBLASLt's own heuristic choice. That choice moves + with m at a fine grain: over m 1..9216 at k 6144 the sweep cells changed + kernel 239 times (n 576), 274 (n 512), 75 (n 160) and 372 (n 128, fp32 + out) -- nvjet sm103 tiles with and without split-K (split-K adds a second + launch, `cublasLt::splitKreduce_kernel`), and at m 2 for n 160 and n 128 an + sm80 cutlass kernel. Every m in that range is driven, so whatever the + heuristic picks there is covered; a change of cuBLASLt can move the + buckets and the cells still cover them. + + Above 9216 rows (the 1..32768 cells; measured on a 152-SM GB300 from the + profiler's launches at every m, each launch's kernel, grid and block): + the launch configuration changes 955 times between m 9216 and 32768 + (n 576), 781 (n 512), 491 (n 160) and 861 (n 128). Every m above 9216 is + one nvjet launch without split-K (the last split-K m is 544, 432, 1984 and + 1680). None of the configurations chosen above 9216 is chosen at any m up + to 9216: 143, 196, 81 and 152 of them, over 29, 35, 14 and 41 tile + kernels, of which the 1..9216 sweep never launches 16, 22, 14 and 41. So + the 9216 cells say nothing about a longer step, and the 32768 cells drive + every m again rather than sample. Measured but not relied on: above 9216 + the choice changes only on 16-row boundaries. + + Above 32768 rows (the dep4 router's gathered cells, n 128 fp32 out, every + multiple of 4 in 32772..131072; the same profiler measurement): the launch + configuration changes 4532 times, over 457 configurations of 43 tile + kernels. 455 of those configurations are chosen at no m up to 32768 (which + chooses 358 over every m 1..32768), and the 1..32768 sweep never launches + 25 of the 43 kernels. Every one is a single nvjet launch without split-K. + So the 32768 cells say nothing about a gathered step past them, and the + gathered cell drives every multiple of 4 again. Measured but not relied + on: there too the choice changes only on 16-row boundaries. + + Not batch-invariant. The rows of an m-row call are not always bitwise + equal to the same rows of a larger call, because the kernel and its + split-K reduction order change with m. Measured at k 6144 against the + 9216-row call: equal for 8864 of the 9216 values of m (n 576), 8832 + (n 512), 7927 (n 160) and 7792 (n 128, fp32 out). Against the 32768-row + call, every m in 9217..32768 was equal at all four shapes (no split-K + there), and the counts up to 9216 were the same within one. Against the + 131072-row call at n 128 (fp32 out, zero padding rows in the operand), + every multiple of 4 in 32772..131072 was equal, and 7836 of the 8192 + multiples of 4 up to 32768. Every m is + within the test's band; a caller that needs a decode row to equal the + prefill row for the same token bit for bit does not get that here. + + fp32 out_dtype is an fp32 result, not a bf16 one widened: bf16 products + are exact in fp32 and the sum is accumulated in fp32. Measured worst case + over m 1..9216 at k 6144 n 128: 122 fp32 ulp of the row's largest logit, + where the same logits rounded through bf16 sit at 2.2e4 to 3.3e4; over + m 1..32768, 121 (124 at a 32768-row call on other operands), against + 3.3e4 rounded through bf16; over the gathered multiples of 4 in + 4..131072, 133 (at a 131072-row call), against 3.25e4 rounded through + bf16. + + Zero padding rows. An all-zero row of mat_a gives a row of +0.0 logits, + bit for bit, never -0.0: measured at every multiple of 4 in 4..131072 with + every 3rd row zero, in the replays at the gathered 4/8/16/32/64 and at + every multiple of 8 in 8..128, at the dep4 skew of T_max 32768 with + ranks at 32768 / 1 / 1 / 1 tokens (98301 zero rows of 131072), and at + the 174764- and 349527-row probes below. The band cannot see a -0.0 (the + reference row is +0.0, at zero distance), so the cells read the padding + rows' bits. + + CUDA graphs. Captured at m 1, 2, 4, 8, 16 and replayed on new data, the + output is bitwise equal to an eager call on that data, for each of the + four hidden-6144 shapes; for the router also at the gathered 4, 8, 16, + 32, 64 (four ranks) and at every multiple of 8 in 8..128 (eight ranks at + 1..16 rows each), with zero padding rows in the replayed data. Every + router call at m 8..128 is split-K (measured on sm_103: one nvjet sm103 + tile kernel, then `cublasLt::splitKreduce_kernel`), over four tile + kernels, one each for m 8..32, 40..64, 72..96 and 104..128; the last two + are run by none of m 8/16/32/64. The replays there were within 20.7 fp32 + ulp element-wise and 13.3 RMS of the reference. The thop caller keeps + one cuBLASLt workspace per (device, stream), created on the first + call on that stream (`getWorkspaceTensor` in `thop/cublasScaledMM.cpp`); + the cells call once on the capture stream before capturing, as an + engine's warm-up does. A capture with no call before it on that stream + was not measured. + + Uncertified: a fused bias at any k but 2880 or 7168; m above 32768 at k 6144, + except the router's (n 128, fp32 out) multiples of 4 up to 131072 -- an m + there that is not a multiple of 4, and any m above 131072, is in no cell; + any k, n or m outside the cells; a device with another SM count than the + 152-SM GB300 the cells ran on (cuBLASLt's choice was not measured there). + + Hard limits (sm_103), for the hidden-6144 cells. The serving envelope's + largest call is the dep4 router's 131072 gathered rows (n 128): mat_a is + 805306368 elements, 1610612736 bytes, below 2^31 bytes. The dep8 router + runs on chunks of at most 32768 of its up to 262144 gathered rows, so its + largest call is 32768 rows (one call over a gathered set above 131072 + rows would be in no cell). The two nearest boundaries were probed at + n 128, fp32 out, with every 3rd row zero, in a + call past each: the 2048 rows up to the boundary, every row past it and + the last 2048 rows were within the test's band against the reference, the + padding rows +0.0 and no output NaN. + m, n, k int32 in `cublas_gemm_caller` boundary m = 2^31 unprobed + mat_a bytes m * 12288, if counted in 32 bits 2^31 / 12288 = 174763 rows probed at 174764: passed + mat_a elems m * 6144, if counted in 32 bits 2^31 / 6144 = 349526 rows probed at 349527: passed + (and m * 12288 past 2^32 bytes from the same row) + out bytes m * n * 2 (bf16), * 4 (fp32 out) 2^31 / 1152 = 1864135 rows (n 576), + / 1024 = 2097152 (n 512), / 320 = 6710887 + (n 160), / 512 = 4194304 (n 128 fp32) unprobed + out elems m * n, if counted in 32 bits 2^31 / 576 = 3728271 rows, + / 512 = 4194304, / 160 = 13421773, + / 128 = 16777216 unprobed + The unprobed boundaries are 32x the router's envelope (131072 rows) and + 57x the other shapes' (32768 rows) or more. No argument is an index + carried through fp32. """ def __call__( diff --git a/tensorrt_llm/_torch/modeling_v2/catalog/moe/fp4_block_scale_moe_runner.py b/tensorrt_llm/_torch/modeling_v2/catalog/moe/fp4_block_scale_moe_runner.py index f0edb97d44d0..20f0851c3f18 100644 --- a/tensorrt_llm/_torch/modeling_v2/catalog/moe/fp4_block_scale_moe_runner.py +++ b/tensorrt_llm/_torch/modeling_v2/catalog/moe/fp4_block_scale_moe_runner.py @@ -24,8 +24,14 @@ #: `act_type` for SwiGlu, the only activation this entry certifies. ACT_TYPE_SWIGLU = 0 -#: The widest unchunked call a stock engine issues: trtllm's default -#: `max_num_tokens`. A caller whose own token count exceeds it must chunk. +#: The top of the certified token column at the R1 and qwen4_exp geometries: +#: trtllm's default `max_num_tokens`. A caller at those geometries whose own +#: token count exceeds it must chunk. The Mistral Large 4 cells certify 1..131072 +#: at theirs (H 6144 / I 6144 / 128 experts / top-4 / 32-wide windows): the +#: tep4 cells 1..32768, the dep4 cells the gathered 32769..131072; and the +#: tp4 and tp8 cells 1..32768 at theirs (H 6144 / I 1536 and 768 / all 128 +#: experts local / top-4); and the tep8 cells 1..32768 and the dep8 cells 1..32768 gathered +#: tokens per call at 16-wide windows. MAX_CERTIFIED_TOKENS = 8192 #: The 16 e2m1 code points in code order: sign bit 3, exponent bits 2:1, @@ -102,7 +108,17 @@ class Fp4BlockScaleMoeRunner(OpWrapper): and contributes zero for every other id, so an expert-parallel split is N calls whose outputs sum -- which is what the deepseek default target does under moe_ep, one call per rank over its window; qwen4_exp makes one call - over its whole expert stack. + over its whole expert stack, Mistral Large 4's tep4 + deployment one call per rank over its 32-wide window, Mistral Large 4's dep4 + deployment the same per-rank call over the four ranks' gathered tokens, and + Mistral Large 4's dep8 deployment one call per rank over its 16-wide window for + each chunk of at most 32768 of the eight ranks' gathered tokens, and its + tep8 deployment one call per rank over its 16-wide window and all of the + step's tokens. + Mistral Large 4's tp4 and tp8 deployments split the experts' width instead: one + call per rank over all 128 experts, each cut to that rank's 1536-wide + (tp4) or 768-wide (tp8) slice of its intermediate, and the ranks' outputs + sum to the layer. With `do_finalize=True` the result is a one-element list holding the combined `[T, H]` bf16 rows, or an empty tensor when `output` was given. @@ -147,8 +163,10 @@ class Fp4BlockScaleMoeRunner(OpWrapper): not a raise. The pre-routed path substitutes its own preprocess/postprocess pair before dispatch, and `routing_method_type` 0, 2, 4, 5, 6 were measured bitwise identical to 1 there. It picks its permutation kernel by row count - instead: a static-block kernel up to 4 rows, a dynamic-block one up to 16 - rows only at <= 256 experts, a single-cluster one up to 8192 rows. + instead, read back from the launches at 128 experts / top-4 and at 512 / + top-10: a static-block kernel up to 4 rows, a dynamic-block one for 5..12, + a single-cluster one for 13..8192, and from 8193 an expert-count init plus + a cooperative kernel (driven to 131072 rows at 128 experts). `act_type` 0 (SwiGlu) only. `gemm1_bias`, `gemm1_alpha`, `gemm1_beta`, `gemm1_clamp_limit` and `gemm2_bias` are `None`. `hidden_states_scale` is @@ -160,7 +178,8 @@ class Fp4BlockScaleMoeRunner(OpWrapper): `tune_max_num_tokens` is an autotuner bucket hint and was measured inert; it is not a token count. `T` is `hidden_states.shape[0]`, and 8192 is the - top of the certified column. A warm autotuner cache was measured bitwise + top of the certified column at the R1 and qwen4_exp geometries (131072 at + Mistral Large 4's, below). A warm autotuner cache was measured bitwise inert against a cold one at the R1 stack, its windows and the 512-expert stack; `local_num_experts` separates the tuner's cache entries, so each layout warms its own. @@ -170,6 +189,481 @@ class Fp4BlockScaleMoeRunner(OpWrapper): checks that, and getting it wrong is 300-3400 bf16 ulp with no error -- the entry's test builds the layout twice, once in torch and once through the trtllm preprocessing ops, and compares them before either reaches the op. + + Mistral Large 4 tep4 (H 6144, I 6144, top-4 of 128, windows of 32), on the + cold fallback tactic. Kernels by token count, read back from the + launches: permutation as above, cooperative from 8193 to 32768; both + GEMMs take the token tile `nextPowerOfTwo(int(T * top_k / + local_num_experts))` clamped to 8..256 -- 8 up to 71 tokens, 16 for + 72..135, 32 for 136..263, 64 for 264..519, 128 for 520..1031, 256 from + 1032; finalize -- scalar up to 49 tokens, vector loads from 50. Rows are + batch invariant on that path: every count of the sweep cell, across all + of those switches, is bitwise the leading rows of one 32768-token call. A + CUDA graph captured at 1..16 tokens replays bitwise equal to eager on new + tokens and new routing, the permutation kernels inside the graph. + + The tuned path. Every valid tactic of the op's tunable runner was run at + 1..16 tokens and at each autotuner bucket 32..8192 and 32768. From 256 + tokens up every one is bitwise the fallback. Below that, a few tile-8 + configs give other bits on some draws (8 of 216 at 11 tokens; 24 of 216 + at 32 and 64; 24 of 224 at 128), each inside the test's gate. So the + cells certify whichever tactic warm-up keeps, while bitwise batch + invariance holds on the cold path only. Pass `tune_max_num_tokens` = + max_num_tokens (32768), as the tep4 deployment does: the tuner then adds a + 32768 bucket and maps 8193..32768 onto it. Warming at 32768 tokens wrote + 15 cache entries in 14.7 s and left the result bitwise unchanged at every + count checked. Certified with TLLM_USE_FINE_GRAINED_SYNC unset (the + engine's default, `use_fine_grained_sync` False), so the runner takes + the non-fine-grained cubins. + + Mistral Large 4 dep4: the same windows over the four ranks' gathered tokens, + 4 x T_max up to 131072 (524288 expanded slots), on the cold fallback + tactic. From 8193 to 131072 the launches read back do not change: the + expert-count init plus the cooperative routing kernel, both GEMMs at + token tile 256, the vector-load finalize -- under uniform routing at + 8193, 32768, 32769, 65536 and 131072, and at 131072 with every slot in + one window and under the padding skew. Rows stay batch invariant: each + sweep count is bitwise the leading rows of the 131072-token call. Decode + graphs at 4..64 gathered tokens replay bitwise equal to eager. A padding + row is `fp4_quantize` of an all-zero row -- every code and every block + scale 0, read back from that op -- routed to experts 0..3 at 0.25, and + comes back +0.0; a step carries up to 98301 of them on window (0, 32). + The four windows over one gathered token set answer every slot exactly + once: their partials, summed, pass the test's gate against the summed + window references, while leaving one window out or running one twice + lands 118-175 rms ulp out. + The dep4 deployment passes `tune_max_num_tokens` = 4 x max_num_tokens = + 131072 and `use_dp` False, so the tuner's buckets are 1..8192 and 131072, + and 8193..131072 map onto 131072. The op offers 8 valid tactics there + (tiles 128 and 256), 8 at 8193 and 65537 too, and every one is + bitwise the fallback -- uniform, one-window and padded alike. Warming at + 131072 tokens wrote 15 cache entries in 12.3 s and left the result + bitwise unchanged at 8193, 32768, 65537 and 131072 in all three. + + Mistral Large 4 tep8 (H 6144, I 6144, top-4 of 128, windows of 16, every rank + over all T tokens), on the cold fallback tactic. Kernels by token count, + read back from the launches at every count of the tep8 sweep cell: the + permutation and finalize kernels switch at the counts they do at tep4 + (their choice reads the token count, not the window: + `runPostTopKPipeline` in `kernels/moe/trtllmGen/routing/ + RoutingFromTopKIds.cu`), and both GEMMs take the token tile at 16 local + experts -- 8 up to 35 tokens, 16 for 36..67, 32 for 68..131, 64 for + 132..259, 128 for 260..515, 256 from 516. With every slot in one window + at 32768 tokens the call launches what it does under uniform routing: + the expert-count init plus the cooperative kernel, tile 256, the + vector-load finalize. Rows stay batch invariant: every count of the + sweep cell is bitwise the leading rows of one 32768-token call, and + decode graphs at 1..16 tokens replay bitwise equal to eager at their own + window -- captured on rank 0's (0, 16) and, after those in the same + process, on rank 7's (112, 16). The offset reaches the kernels as a + per-call int (`run_moe`'s `local_expert_offset`; the C++ runner object + `FP4BlockScaleMoERunner.get_runner` caches is keyed on `act_type` + alone, so every offset shares it), and a graph keeps the one it was + captured at: the window-112 replays' data run at offset 0 land + 237-24805 rms ulp from `reference`. At decode counts a 16-wide window + has few live rows, so one FC1 value the kernel requantizes to the + neighbouring e2m1 code or e4m3 block scale -- read out of the kernel, + each such value within 5.24e-5 relative of the rounding midpoint -- + moves a step's relative RMS against `reference` by several bf16 ulp + (4.98 at one live row), so the test gates the decode graph replays on + the element figure alone (its `_compare`). The eight windows over one + 32768-token step answer every slot exactly once: their + partials, summed in fp32 and as a bf16 running sum in rank order, pass + the test's gate against `reference` over all 128 experts at once, while + leaving one window out or running one twice lands 84-119 rms ulp out. + How the tep8 deployment sums them is its closing all-reduce's + (`comm/cute_dsl_allreduce_add_rmsnorm`). + The tuned path at tep8: the op offers 216 valid tactics at 1..35 tokens + (tiles 8, 16 and 32), 224 at 36 and 64, 142 at 128, 80 at 256, 16 at 512 + and 8 (tiles 128 and 256) from 1024 to 32768. Every one is bitwise the + fallback except 24 tile-8 configs at 9 and 11 tokens on the test's + draws, each inside the test's gate; at 32768 tokens with every slot in + one window all 8 are bitwise the fallback. Warming at 32768 tokens with + `tune_max_num_tokens` 32768 wrote 15 cache entries in 8.5 s and left the + result bitwise unchanged at 40 counts from 1 to 32768 and on the + one-window draw. + + Mistral Large 4 dep8 (H 6144, I 6144, top-4 of 128, windows of 16): rank r + calls its window (16r, 16) over the eight ranks' gathered tokens in + calls of at most 32768 of them, on the cold fallback tactic. Kernels by + token count, read back from the launches at each side of every switch: + the permutation and finalize kernels switch at the counts they do at + tep4's 32-wide windows, and both GEMMs take the token tile at 16 local + experts -- 8 up to 35 tokens, 16 + for 36..67, 32 for 68..131, 64 for 132..259, 128 for 260..515, 256 from + 516. A 32768-token call with every slot in one window, and the padding + chunk below, launch what a uniform one does: the expert-count init plus + the cooperative kernel, tile 256, the vector-load finalize. Rows stay + batch invariant: every count 1..256 and every edge above it to 32768 is + bitwise the leading rows of one 32768-token call, so each chunk the + target issues (a multiple of 8, a step's last chunk the remainder) is + the leading rows of a certified call. Decode graphs at 8..128 gathered + tokens (8 ranks x 1..16) replay bitwise equal to eager, captured on rank + 0's window (0, 16) and, after those in the same process, on rank 7's + (112, 16). The offset reaches the kernels as a per-call int, and a graph + keeps the one it was captured at: the window-112 replays' data run at + offset 0 land 269-493 rms ulp from `reference`. With one rank at 32768 + tokens and seven at 1, chunks 1..7 of the 262144 gathered rows each hold + one live row and 32767 padding rows, whose 131068 slots ride experts + 0..3 of window (0, 16) at 0.25 and come back +0.0. The eight windows + over one 32768-token chunk answer every slot exactly once: their + partials, summed in fp32 and as a bf16 running sum in rank order, pass + the test's gate against `reference` over all 128 experts at once, while + leaving one window out or running one twice lands 83-118 rms ulp out. + How the dep8 deployment sums them is `comm/reducescatter`'s. + The dep8 deployment passes `tune_max_num_tokens` = its chunk, 32768, and + `use_dp` False, so the tuner's buckets are 1..8192 and 32768, and a call + runs the tactic of the power of two at or below its count (8193..32768 + that of 32768). Every tactic valid at that bucket was run at each decode + count 8..128 (216 at buckets 8..32, 224 at 64, 142 at 128), every valid + one at 256..8192 and 32768 (80, 16, then 8), and at 32768 on the + one-window and padding chunks: all bitwise the fallback except 24 tile-8 + configs of bucket 64 at 88 tokens on the test's draw, each inside the + test's gate. Warming at 32768 tokens wrote 15 cache entries in 8.5 s + (tile 8 kept at buckets 1..64, 16 at 128, 32 at 256, 64 at 512, 128 at + 1024..8192, 256 at 32768) and left the result bitwise unchanged at 46 + calls: 1..16, 24..128 step 8, 255 to 32768, and the one-window and + padding chunks. CUDA graphs captured on that warm cache at 8..128 + gathered tokens replayed bitwise equal to warm eager, each inside the + test's gate (a probe, not a cell). + + Operands, as a Mistral Large 4 loader builds them per local expert e (global + id = local_expert_offset + e) from layer N's `experts..{w1,w2,w3}`: + w1 is the gate, w3 the up and w2 the down projection; `.weight` is U8 + `[out, in/2]`, two e2m1 per byte (the op reads the even element from the + low nibble), and `.weight_scale` F8_E4M3 `[out, in/16]`, linear. + gemm1_weights[e] rows of cat([w3.weight, w1.weight]) gathered by + fc1_perm: [12288, 3072] uint8 + gemm1_weights_scale[e] the same gather of cat([w3.weight_scale, + w1.weight_scale]), then the 128x4 swizzle: + [12288, 384] float8_e4m3fn + gemm2_weights[e] w2.weight rows gathered by blk32_perm: [6144, 3072] + gemm2_weights_scale[e] the same gather of w2.weight_scale, then the + swizzle: [6144, 384] + `blk32_perm` is the 32-row shuffle: within each block of 32 rows, source + row 4u + v lands at row 8v + u. `fc1_perm` first interleaves the halves + (row 2i is up row i, row 2i + 1 gate row i), then applies `blk32_perm`. + `torch.ops.trtllm.shuffle_matrix(rows, perm)` is the row gather and + `torch.ops.trtllm.block_scale_interleave` the swizzle; the test proves + both equal to the torch recipe at these shapes, with no padding. The + scalars are fp32, per local expert, from the layer's single input_scale + per projection: output1_scale_gate_scalar = input_scale(w1) * + weight_scale_2(w1); output1_scale_scalar = that / input_scale(w2); + output2_scale_scalar = input_scale(w2) * weight_scale_2(w2). w1 and w3 + share both values on every expert. `hidden_states` and + `hidden_states_scale` are `fp4_quantize(x, 1 / input_scale(w1), 16, + False, False)`. The cells draw the operands at the checkpoint's own + magnitudes. Weight block scales are 8..448 and activation block scales + span the whole e4m3 range. weight_scale_2 covers its extremes over all + 9088 experts. The input scales are those of layers 1, 3, 45, 65 and 71: + the extremes of both global scales, and the last layer. At layer 65's + scalars the FC2 input runs far past its calibrated range -- on three of + four experts sampled, 91-100 % of its block scales saturate at 448; on + the 16-wide window's layer-65 cell's draws over its 16 experts, 84-100 % + on 11 and 1.4-17.8 % on the five with the smallest w1 weight_scale_2 (tep8 + and dep8 runs) -- and kernel and reference still agree. + + Hard limits (sm_103, Mistral Large 4 tep4, 32768 tokens) + gemm1_weights local expert e starts at e2m1 element e * 2I * H + (75,497,472 per expert); experts 29..31 start past 2^31 + elements, 28 straddles it. The tensor is 1,207,959,552 + bytes. probed: each of 29, 30, 31 gives the same bits + moved to slot 1 -- passed + permuted rows allocated (32 + (4T - 32) // tile) * tile = 139,008 at + T 32768, tile 256; gemm2 output 139,008 x 6144 bf16 = + 854,065,152 elements, 1,708,130,304 bytes; rows x 12,288 + bytes reaches 2^31 at 174,763 rows -- not reached. + probed at the envelope's top: all 131,072 slots in one + window, 32,768 of them on one expert -- passed + gemm1 output the NVFP4 FC2 input, 3072 bytes a row: 427,032,576 + bytes at 139,008 rows -- below 2^31 + the rest T * top_k = 131,072 expanded slots; hidden_states + 100,663,296 bytes; output 201,326,592 elements -- all + below 2^31. No index passes through fp32: the tile choice + truncates T * top_k / local_num_experts, 4096 at the top + T certified 1..32768 at this geometry by the tep4 cells, + to 131072 by the dep4 ones (below) + + Hard limits (sm_103, Mistral Large 4 dep4, 131072 gathered tokens) + permuted rows allocated (32 + (4T - 32) // tile) * tile = 532,224 at + T 131072, tile 256; the bf16 FC2 output is 3,269,984,256 + elements, 6,539,968,512 bytes. A row's byte offset + passes 2^31 at row 174,763 and its element offset at + row 349,526. probed: the one-window cell's draw (all + 524,288 slots in window (0, 32)) uses rows to 528,639 -- + 351,398 slots at or past row 174,763, 178,702 at or past + 349,526 -- and a padding-skew draw rows to 429,280; both + pass the gate, and the per-slot rows (`do_finalize=False`) + gathered and weighted in torch equal the finalized output + bitwise -- passed. A control that reads the rows past + either mark from that many rows earlier picks up + unwritten tile-padding rows: NaN, which the gate + refuses. Both finalize kernels (`finalizeKernel` + and `finalizeKernelVecLoad` in + `kernels/moe/trtllmGen/DevKernel.cu`) form the row offset + in 64 bits + gemm1 output 532,224 rows x 3072 bytes = 1,635,012,608 -- below 2^31 + the rest T * top_k = 524,288 expanded slots (int32 indices); + hidden_states 402,653,184 bytes; output 805,306,368 + elements -- all below 2^31. No index passes through + fp32: the tile choice truncates T * top_k / + local_num_experts, 16384 at the top + T certified 1..131072 at this geometry. The cooperative + routing kernel takes T <= (SMs - 8) * 128 * 64 / top_k = + 294,912 on sm_103; above it the routing switches path -- + computed, not probed + + Mistral Large 4 tp4 (H 6144, I 1536 -- each expert's slice of its 6144-wide + intermediate -- top-4 of 128, all 128 experts local at offset 0), on the + cold fallback tactic. Kernels by token count, read back from the launches + (the GEMM tile at every count of the tp4 sweep cell, the permutation and + finalize kernels on both sides of each switch): permutation as above, the + static block kernel to 4 tokens, the dynamic one for 5..12, the + single-cluster one for 13..8192, the expert-count init plus the + cooperative kernel from 8193 to 32768; both GEMMs take the token tile + `nextPowerOfTwo(int(T * top_k / local_num_experts))`, T / 32 here, clamped + to 8..256 -- 8 up to 287 tokens, 16 for 288..543, 32 for 544..1055, 64 for + 1056..2079, 128 for 2080..4127, 256 from 4128; finalize -- scalar up to 49 + tokens, vector loads from 50, as at tep4 (that switch reads the hidden + width and T, not the expert count). Rows are batch invariant on that path: + every count of the sweep cell is bitwise the leading rows of one + 32768-token call. A CUDA graph captured at 1..16 tokens replays bitwise + equal to eager on new tokens and new routing. + + The tuned path at tp4. The op's tunable runner offers 160 valid tactics + (tiles 8, 16 and 32) at every count read up to 287 tokens, 164 at 288 and + 512, 106 at 1024, 60 at 2048, 12 at 4096 and 8 (tiles 128 and 256) at + 8192, 8193, 16384 and 32768. Every one was run at 1..16 tokens and at each + autotuner bucket 32..8192 and 32768, and every one is bitwise the fallback + there. The target passes `tune_max_num_tokens` = max_num_tokens = 32768 + and `use_dp` False, so the buckets are 1..8192 and 32768, and 8193..32768 + map onto 32768. Warming at 32768 tokens wrote 15 cache entries in 13.9 s + and left the result bitwise unchanged at 33 counts from 1 to 32768. + + The four width slices. Rank r's call -- the same tokens, routing and + scalars, its own slice of every expert -- returns the partial sum over + intermediate columns [1536r, 1536r + 1536). The FC2-input requantization + runs in 16-element blocks under the layer's one global scale and 1536 = + 96 x 16, so no block straddles a slice boundary: the four partials, + summed in fp32 and as a bf16 running sum in rank order, pass the test's + gate against the whole-width (I 6144) reference, while one slice left out + lands 128 rms ulp out and slice 0 run twice 181. The element figure at the + slice is a max over a narrower contraction, so the test gates elements + at 32 ulp there (16 at the whole width) and RMS at 2 as everywhere but the + decode graph replays; its + `_compare` has the derivation and the measurements. How the tp4 deployment + sums the partials is its closing all-reduce's. + + Operands at tp4, as a loader builds them for rank r from layer N's + `experts..{w1,w2,w3}`, e = 0..127 (the slice is cut first, then laid + out: the layout is a function of the slice's row count): + gemm1_weights[e] rows [1536r, 1536r + 1536) of w3.weight, then + the same rows of w1.weight, gathered by + fc1_perm: [3072, 3072] uint8 + gemm1_weights_scale[e] the same rows of the two weight_scale tensors, + the same gather, then the 128x4 swizzle: + [3072, 384] float8_e4m3fn + gemm2_weights[e] w2.weight byte columns [768r, 768r + 768) + (inputs [1536r, 1536r + 1536)), rows gathered + by blk32_perm: [6144, 768] + gemm2_weights_scale[e] w2.weight_scale columns [96r, 96r + 96), the + same gather, then the swizzle: [6144, 96] + 3072 and 6144 rows are multiples of 128 and 384 and 96 scale columns + multiples of 4, so the swizzle pads nothing; the test proves + `block_scale_interleave` equal to the torch recipe at both scale shapes. + weight_scale_2 and input_scale are per expert and per layer, unsplit, so + the three scalars are tep4's, for all 128 experts. The cells draw the + slices at tep4's magnitudes and the same five layers' input scales; at + layer 65's the FC2 input saturates as at tep4 (99-100 % of the block + scales at 448 on three of four experts sampled). + + Hard limits (sm_103, Mistral Large 4 tp4, 32768 tokens) + gemm1_weights local expert e starts at e2m1 element e * 2I * H + (18,874,368 per expert at I 1536): expert 113 straddles + 2^31 elements, 114..127 start past it. The tensor is + 1,207,959,552 bytes. probed: each of 113..127 gives the + same bits moved to slot 1, and a cell puts all 32768 + tokens on each of 124..127 -- passed + permuted rows allocated (128 + (4T - 128) // tile) * tile = 163,584 at + T 32768, tile 256 -- read back as the per-slot rows of + `do_finalize=False` at 1, 16, 287, 288, 4128, 8193 and + 32768 tokens, each the formula's; the bf16 gemm2 output is + 1,005,060,096 elements, 2,010,120,192 bytes. A row's byte + offset passes 2^31 at row 174,763 -- not reached, 11,179 + rows short (a uniform draw used rows to 147,645, the + top_experts skew to 131,071) + gemm1 output the NVFP4 FC2 input, 768 bytes a row: 125,632,512 bytes + at 163,584 rows -- below 2^31 + the rest T * top_k = 131,072 expanded slots; gemm2_weights + 1,207,959,552 e2m1 elements; hidden_states 100,663,296 + bytes; output 201,326,592 elements -- all below 2^31. No + index passes through fp32: the tile choice truncates + T * top_k / local_num_experts, 1024 at the top + T certified 1..32768 at this geometry by the tp4 cells + + Mistral Large 4 tp8 (H 6144, I 768 -- each expert's slice of its 6144-wide + intermediate -- top-4 of 128, all 128 experts local at offset 0), on the + cold fallback tactic. Kernels by token count, read back from the launches + (the GEMM tile at every count of the tp8 sweep cell, the permutation and + finalize kernels on both sides of each switch): permutation as above, the + static block kernel to 4 tokens, the dynamic one for 5..12, the + single-cluster one for 13..8192, the expert-count init plus the + cooperative kernel from 8193 to 32768; both GEMMs take the token tile + `nextPowerOfTwo(int(T * top_k / local_num_experts))`, T / 32 here, clamped + to 8..256 -- 8 up to 287 tokens, 16 for 288..543, 32 for 544..1055, 64 for + 1056..2079, 128 for 2080..4127, 256 from 4128 (the expert count sets + them, not the width); finalize -- scalar up to 49 tokens, vector loads + from 50. Rows are batch invariant on that path: every count of the sweep + cell is bitwise the leading rows of one 32768-token call. A CUDA graph + captured at 1..16 tokens replays bitwise equal to eager on new tokens and + new routing. + + The tuned path at tp8. The op's tunable runner offers 152 valid tactics + (tiles 8, 16 and 32) at every count read up to 287 tokens, 160 for + 288..543 (tile 64 joins), 107 for 544..1055, 61 for 1056..2079, 13 for + 2080..4127 and 5 (tiles 128 and 256) from 4128 to 32768. Every one was + run at 1..16 tokens and at each autotuner bucket 32..8192 and 32768, and + every one is bitwise the fallback there. The target passes + `tune_max_num_tokens` = max_num_tokens = 32768 and `use_dp` False, so the + buckets are 1..8192 and 32768, and 8193..32768 map onto 32768. Warming at + 32768 tokens wrote 15 cache entries in 10.6 s and left the result bitwise + unchanged at 33 counts from 1 to 32768. + + The eight width slices. Rank r's call -- the same tokens, routing and + scalars, its own slice of every expert -- returns the partial sum over + intermediate columns [768r, 768r + 768). The FC2-input requantization + runs in 16-element blocks under the layer's one global scale and 768 = + 48 x 16, so no block straddles a slice boundary: the eight partials, + summed in fp32 and as a bf16 running sum in rank order, pass the test's + gate against the whole-width (I 6144) reference, while one slice left out + lands 90 rms ulp out and slice 0 run twice 128. The element figure at the + slice is a max over a narrower contraction, so the test gates elements + at 16 * sqrt(8) = 45.25 ulp there (16 at the whole width) and RMS at 2 as + everywhere but the decode graph replays; its `_compare` has the derivation + and the measurements. How + the tp8 deployment sums the partials is its closing all-reduce's. + + Decode at tp8 gates few rows: a step of T tokens is T live rows of 4 x 768 + FC1 outputs, so one FC1 output that kernel and reference round to + neighbouring e2m1 codes (or e4m3 block scales) at a rounding midpoint + moves the relative RMS figure of the whole call. Over 5120 fresh decode + draws a correct kernel exceeded the test's 2 ulp RMS gate on 21 (max + 7.05) with the element figure at or below 22.61; each of those differs + from an fp64 model only at FC1 rounding midpoints (`_compare` has the + protocol). The test gates the decode graph replays on the element figure + alone. + + Operands at tp8, as a loader builds them for rank r from layer N's + `experts..{w1,w2,w3}`, e = 0..127 (the slice is cut first, then laid + out: the layout is a function of the slice's row count): + gemm1_weights[e] rows [768r, 768r + 768) of w3.weight, then + the same rows of w1.weight, gathered by + fc1_perm: [1536, 3072] uint8 + gemm1_weights_scale[e] the same rows of the two weight_scale tensors, + the same gather, then the 128x4 swizzle: + [1536, 384] float8_e4m3fn + gemm2_weights[e] w2.weight byte columns [384r, 384r + 384) + (inputs [768r, 768r + 768)), rows gathered by + blk32_perm: [6144, 384] + gemm2_weights_scale[e] w2.weight_scale columns [48r, 48r + 48), the + same gather, then the swizzle: [6144, 48] + 1536 and 6144 rows are multiples of 128 and 48 scale columns a multiple + of 4, so the swizzle pads nothing; the test proves + `block_scale_interleave` equal to the torch recipe at both scale shapes. + weight_scale_2 and input_scale are per expert and per layer, unsplit, so + the three scalars are tep4's, for all 128 experts. The cells draw the + slices at tep4's magnitudes and the same five layers' input scales; at + layer 65's the FC2 input saturates as at tep4 (99.8-100 % of the block + scales at 448 on two of four experts sampled, 1.4 and 16.6 % on the + other two). + + Hard limits (sm_103, Mistral Large 4 tp8, 32768 tokens) + gemm1_weights local expert e starts at e2m1 element e * 2I * H + (9,437,184 per expert at I 768): expert 127 starts at + 1,198,522,368 and the tensor holds 1,207,959,552 + elements (603,979,776 bytes) -- below 2^31, computed, + not probed; a cell puts all 32768 tokens on each of + experts 124..127 + permuted rows allocated (128 + (4T - 128) // tile) * tile = 163,584 at + T 32768, tile 256 -- read back as the per-slot rows of + `do_finalize=False` at 1, 16, 287, 288, 4128, 8193 and + 32768 tokens, each the formula's; the bf16 gemm2 output is + 1,005,060,096 elements, 2,010,120,192 bytes. A row's byte + offset passes 2^31 at row 174,763 -- not reached, 11,179 + rows short (a uniform draw used rows to 147,645, the + top_experts skew to 131,071) + gemm1 output the NVFP4 FC2 input, 384 bytes a row: 62,816,256 bytes + at 163,584 rows -- below 2^31 + the rest T * top_k = 131,072 expanded slots; gemm2_weights + 603,979,776 e2m1 elements; hidden_states 100,663,296 + bytes; output 201,326,592 elements -- all below 2^31. No + index passes through fp32: the tile choice truncates + T * top_k / local_num_experts, 1024 at the top + T certified 1..32768 at this geometry by the tp8 cells + + Hard limits (sm_103, Mistral Large 4 tep8, 16-wide windows, 32768 tokens) + gemm1_weights local expert e starts at e2m1 element e * 2I * H + (75,497,472 per expert); the last, 15, at 1,132,462,080 + -- below 2^31, not reached. The tensor is 603,979,776 + bytes + permuted rows allocated (16 + (4T - 16) // tile) * tile = 134,912 at + T 32768, tile 256 -- read back (`do_finalize=False`) at + 20 counts from 1 to 32768 and with every slot in one + window; gemm2 output 134,912 x 6144 bf16 = 828,899,328 + elements, 1,657,798,656 bytes; rows x 12,288 bytes + reaches 2^31 at 174,763 rows -- not reached. The + one-window draw's slots reach row 133,375 + gemm1 output 134,912 rows x 3072 bytes = 414,449,664 -- below 2^31 + the rest as at tep4: 131,072 expanded slots, hidden_states + 100,663,296 bytes, output 201,326,592 elements. No index + passes through fp32: the tile choice truncates T * top_k + / local_num_experts, 8192 at the top + T certified 1..32768 at this geometry by the tep8 cells + + Hard limits (sm_103, Mistral Large 4 dep8, 16-wide windows, 32768-token calls) + gemm1_weights local expert e starts at e2m1 element e * 2I * H + (75,497,472 per expert); the last, 15, at 1,132,462,080 + -- below 2^31, not reached. The tensor is 603,979,776 + bytes + permuted rows allocated (16 + (4T - 16) // tile) * tile = 134,912 at + T 32768, tile 256 -- read back (`do_finalize=False`) at + the 20 kernel-edge counts and on the one-window and + padding chunks; the bf16 FC2 output is 828,899,328 + elements, 1,657,798,656 bytes. A row's byte offset + passes 2^31 at row 174,763 -- not reached: the slots' + rows, read back, reach 132,607 on the one-window chunk + and 131,072 on the padding chunk. A call of 65536 tokens + with every slot in one window would allocate 265,984 + rows, past it (computed); the 32768-token chunk keeps + every call below it + gemm1 output 134,912 rows x 3072 bytes = 414,449,664 -- below 2^31 + the rest T * top_k = 131,072 expanded slots; hidden_states + 100,663,296 bytes; output 201,326,592 elements -- all + below 2^31. No index passes through fp32: the tile choice + truncates T * top_k / local_num_experts, 8192 at the top + T certified 1..32768 per call at this geometry by the dep8 + cells; a gathered set past 32768 is the caller's to cut + + Not certified at Mistral Large 4: `do_finalize=False` and `output=` as a + caller's mode (the dep4, tep8 and dep8 tests read per-slot rows only to + locate and count them), the in-kernel routing path, + TLLM_USE_FINE_GRAINED_SYNC=1, more than 131072 tokens (more than 32768 + at 16-wide windows), gathered graph batches other than 4/8/16/32/64 at + 32-wide windows and 8..128 step 8 at 16-wide ones, tep8 and dep8 decode + graphs on windows (16, 16) to (96, 16) (those on 0 and 112 are cells), + graphs captured on a tuned tactic as cells (the graph cells capture the + cold fallback; every tactic was run eager, and probed at dep8's 8..128 on + window (0, 16): each replay bitwise warm eager, inside the gate), and the + windows in separate processes -- dep4's four, tep8's and dep8's eight: + the op holds no cross-rank state, each rank's call is this one at its own + offset, and the windows' partials were summed in one process, in fp32 and + as a bf16 running sum in rank order. How the dep4 and dep8 combines sum + them is `comm/reducescatter`'s; how the tep8 deployment's does, + `comm/cute_dsl_allreduce_add_rmsnorm`'s. At tp4 and tp8: more than 32768 + tokens, graph batches past 16, a width slice other than 1536 and 768 + (moe_tensor_parallel_size other than 4 and 8), and the four or eight width + slices in as many processes -- they were summed in one process, as the + windows were. """ def __call__( @@ -263,8 +757,14 @@ def reference( layout was built from, which is where a target's checkpoint starts too, plus the routing this call was entered with. - `up`, `gate` and `down` are each `(codes[E, M, K/2], scales[E, M, K/16])` - in their unshuffled, unswizzled form. `topk_ids` carries *global* ids; + `up`, `gate` and `down` are each `(codes[E, M, K], scales[E, M, K/16])` + in their unshuffled, unswizzled form, one e2m1 code (0..15) per weight + element. `scales` may be fp32 with the expert's global weight scale + (`weight_scale_2`) already multiplied in: the weight is dequantized as + `code * scale`, so a checkpoint whose experts carry their own global + scale is referenced exactly that way, and `hidden_states` is then the + activation with its own global scale undone (`code * scale * + input_scale`). `topk_ids` carries *global* ids; this rank contributes only for the ids its window owns, and every other slot contributes zero -- which is what makes an expert-parallel split summable. diff --git a/tensorrt_llm/_torch/modeling_v2/catalog/norm/flashinfer_rmsnorm.py b/tensorrt_llm/_torch/modeling_v2/catalog/norm/flashinfer_rmsnorm.py index e0555561922b..452fdc8f264c 100644 --- a/tensorrt_llm/_torch/modeling_v2/catalog/norm/flashinfer_rmsnorm.py +++ b/tensorrt_llm/_torch/modeling_v2/catalog/norm/flashinfer_rmsnorm.py @@ -13,9 +13,13 @@ import torch import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch.flashinfer_utils import IS_FLASHINFER_AVAILABLE, get_env_enable_pdl from .._op import Arch, Cell, OpWrapper +if IS_FLASHINFER_AVAILABLE: + from flashinfer.norm import rmsnorm as _rmsnorm_into + class FlashinferRmsnorm(OpWrapper): """`x / sqrt(mean(x^2, -1) + eps) * weight`, one op call, new tensor out. @@ -23,6 +27,17 @@ class FlashinferRmsnorm(OpWrapper): Fusion boundary: normalization and weight scaling only. No residual add (`flashinfer_fused_add_rmsnorm`), no gemma `(1 + w)` form (`flashinfer_gemma_rmsnorm`), no quantization. `x` is not mutated. + + `x` may be a row-strided view -- the leading columns of a wider row, such + as the latent `kv[:, :512]` of a 576-wide kv_a row -- and the result is + still a new contiguous tensor; the `note` says which views are accepted. + + With `out`, the result is written there instead and `out` is returned: + `x` itself (in place, the latent normalized inside its kv_a row, so the + row is the `[RMSNorm(c_kv) | k_pe]` latent with no concatenation) or the + leading columns of another buffer (a NoPE owner's 576-wide latent row, + whose rope tail the caller keeps zero). The columns of the row outside + `out` are not touched. """ ARCHS = frozenset({Arch.SM_100, Arch.SM_103}) @@ -43,8 +58,68 @@ class FlashinferRmsnorm(OpWrapper): note = """ Registered only when flashinfer is importable (`IS_FLASHINFER_AVAILABLE`). + The path the cells certify is FlashInfer's CuTe DSL kernel: + `flashinfer.norm.rmsnorm` runs `rmsnorm_cute` + (`flashinfer/norm/kernels/rmsnorm.py`) unless `FLASHINFER_USE_CUDA_NORM=1` + is set or the installed CuTe DSL cannot target the device; then it runs + the CUDA-JIT kernel, which this entry does not certify. `rmsnorm_cute` + compiles on the first call in a process for each (dtype, width, row + layout): a compact kernel when input and output are contiguous as torch + counts it, a strided one otherwise. A one-row view of a wider row is + contiguous to torch, so it takes the compact kernel; measured, and again + at 32768 rows on the 576-strided latent: compact at 1 row, strided from 2. + The cells drive both at width 512 and the compact one at every other + width. + PDL is controlled by `TRTLLM_ENABLE_PDL` (default on) inside the trtllm custom op; it changes scheduling, not results, and is not a cell axis. + The cells run with it on. + + Row-strided input. `x` may be a view whose last dim is contiguous, whose + row stride is a multiple of 8 elements and whose data is 16-byte aligned: + the leading columns of a wider row. The result is a new contiguous tensor + and the columns of the row outside `x` are not touched. Every other layout + was measured to raise before the launch rather than answer, so `is_valid` + does not restate them (the test pins each). Measured at width 512; every + message carries the same signature, which names the width: + last dim not contiguous (`buf[..., 0]` of a [T, 512, 2] buffer): + ValueError: Mismatched mX.strides[1] on argument #0 when calling: + `__call__(mX: Tensor([n0, 512], bfloat16), mW: Tensor([512], + bfloat16), mY: Tensor([n0, 512], bfloat16), M: int64, eps: float32)`, + expected to be 1 + row stride not a multiple of 8 (`buf[:, :512]` of a [T, 516] buffer): + ValueError: Invalid mX.strides[0] on argument #0 when calling: + `__call__(mX: Tensor([n0, 512], bfloat16), mW: Tensor([512], + bfloat16), mY: Tensor([n0, 512], bfloat16), M: int64, eps: float32)`, + expected to be divisible by 8 + data not 16-byte aligned (`buf[:, 4:516]` of a [T, 576] buffer): + ValueError: Misaligned Tensor data on argument #0 when calling: + `__call__(mX: Tensor([n0, 512], bfloat16), mW: Tensor([512], + bfloat16), mY: Tensor([n0, 512], bfloat16), M: int64, eps: float32)`, + expected data alignment=16 bytes + The first refusal in a process takes about 2 s inside the cute DSL's + TVM-FFI call wrapper; later ones are immediate. + + `out` runs the same kernel through `flashinfer.norm.rmsnorm(..., out=out)` + with the custom op's PDL setting (the op itself allocates its output), so + the result is bitwise the call without `out` (the `out` cells hold it so); + a 576-strided `out` takes the strided kernel from two rows, as `x` does. + The compact and strided kernels are compiled from one `RMSNormKernel` + program (`rmsnorm_cute`) and differ only in the layouts they are compiled + for, the row stride a constant or a runtime int64: the same operations at + the same precision, so a NoPE owner's latent written at row stride 576 + holds the bits the contiguous call returns. + In place is safe: each row is read whole before it is written, by the + same block (measured bitwise at every cell). An `out` that overlaps `x` + without being `x` is refused by `is_valid`. + + Rows are computed independently: at 6144 and at 512 in both layouts, + every row count in 1..9216 is bitwise equal to the leading rows of the + 9216-row call, every row count in 1..32768 to the leading rows of the + 32768-row call, and a CUDA-graph replay is bitwise equal to an eager call + on the same data. An all-zero row was measured to return zeros. The + 32768-row call is at most one bf16 ulp from the reference, on 6.44e-6 of + the elements at 6144 and 1.55e-6 to 1.85e-6 at 512. fp16 and fp32 inputs are accepted by the op and were measured to work, but no shipped target uses them, so they are not cells. TRT-LLM's own @@ -56,28 +131,59 @@ class FlashinferRmsnorm(OpWrapper): rejects it below. The CUDA-JIT path (`FLASHINFER_USE_CUDA_NORM=1`) does raise, but this entry does not certify that path. - """ - def __call__(self, x: torch.Tensor, weight: torch.Tensor, eps: float) -> torch.Tensor: - return torch.ops.trtllm.flashinfer_rmsnorm(x, weight, eps) + Hard limits (sm_103). Computed, not probed: the envelope's largest call, + 32768 x 6144 (the NVFP4 target's step), is 201,326,592 elements, 9.4% of + 2^31; its last row starts at element 201,320,448. + rows x width the compact kernel computes row * width in int32; + `rmsnorm_cute` switches to the strided kernel (int64 row + stride) itself once rows * width > 2^31 - 1, at 349526 + rows for 6144 and 4194304 for 512 (read in the source), + 10.7x and 128x the 32768-row step + row offset the strided kernel's row stride is int64: no 32-bit + product at row stride 576 (the kv_a buffer of a + 32768-row step is 18,874,368 elements) + rows int64 in the kernel; the grid is rows / rows-per-block, + 16384 blocks at 32768 x 6144 and 8192 at 32768 x 512, + against the 2^31 - 1 grid limit + fp32 no index passes through fp32 + KV pool none read: the op takes this step's rows only, so the + pool size sets no limit + """ - def reference(self, x: torch.Tensor, weight: torch.Tensor, eps: float) -> torch.Tensor: - """fp32-accumulated; the kernel reduces in fp32 and rounds once at the end.""" + def __call__( + self, x: torch.Tensor, weight: torch.Tensor, eps: float, out: torch.Tensor | None = None + ) -> torch.Tensor: + if out is None: + return torch.ops.trtllm.flashinfer_rmsnorm(x, weight, eps) + return _rmsnorm_into(x, weight, eps, out=out, enable_pdl=get_env_enable_pdl()) + + def reference( + self, x: torch.Tensor, weight: torch.Tensor, eps: float, out: torch.Tensor | None = None + ) -> torch.Tensor: + """fp32-accumulated; the kernel reduces in fp32 and rounds once at the end. A new + tensor either way; `out` is not read.""" xf = x.float() normed = xf * torch.rsqrt(xf.pow(2).mean(dim=-1, keepdim=True) + eps) return (normed * weight.float()).to(x.dtype) - def is_valid(self, x: torch.Tensor, weight: torch.Tensor, eps: float) -> None: - # Only the case the op takes and answers wrongly. Shape and dtype - # mismatches it rejects itself, loudly, and duplicating that here - # would only swap its error for a worse one. + def is_valid( + self, x: torch.Tensor, weight: torch.Tensor, eps: float, out: torch.Tensor | None = None + ) -> None: + # Only the case the op takes and answers wrongly. Shape, dtype and + # layout mismatches it rejects itself, loudly (the `note` quotes the + # layout errors), and duplicating that here would only swap its error + # for a worse one. assert x.dtype is not torch.float8_e4m3fn, ( "float8_e4m3fn is accepted by the kernel's dtype table and returns " "a plausible fp8 rmsnorm; dequantize before normalizing" ) - assert x.stride(-1) == 1, ( - f"the normalized dim must be contiguous; got stride {x.stride(-1)}" - ) + # A partial overlap reads rows the call has already written. + if out is not None and not (out.data_ptr() == x.data_ptr() and out.stride() == x.stride()): + x0, o0 = x.data_ptr(), out.data_ptr() + x1 = x0 + ((x.shape[0] - 1) * x.stride(0) + x.shape[1]) * x.element_size() + o1 = o0 + ((out.shape[0] - 1) * out.stride(0) + out.shape[1]) * out.element_size() + assert not (x0 < o1 and o0 < x1), "out must be x itself or not overlap it" flashinfer_rmsnorm = FlashinferRmsnorm() diff --git a/tensorrt_llm/_torch/modeling_v2/catalog/quantization/fp4_quantize.py b/tensorrt_llm/_torch/modeling_v2/catalog/quantization/fp4_quantize.py index 234d0f5e8503..bab2b44a367d 100644 --- a/tensorrt_llm/_torch/modeling_v2/catalog/quantization/fp4_quantize.py +++ b/tensorrt_llm/_torch/modeling_v2/catalog/quantization/fp4_quantize.py @@ -24,7 +24,8 @@ #: reciprocals, each ~2^-23 relative. A value within a few fp32 ulps of a #: midpoint can therefore round to either neighbour, and no reference can say #: which. Everything further out than this must match bit for bit. Largest -#: deviation observed on this machine: 7.9e-8 relative, ~12x inside the window. +#: deviation observed on this machine: 7.9e-8 relative, ~12x inside the window +#: (5.96e-8 across Mistral Large 4's 71 static global scales x every bf16 block max). TIE_WINDOW = 2.0**-20 @@ -80,8 +81,91 @@ class Fp4Quantize(OpWrapper): ) note = """ + Metadata consumed: none. Preconditions: `input` contiguous, its last dim a + whole number of `sf_vec_size` blocks; with `sf_use_ue8m0=False`, + `global_scale` a float32 device tensor holding one element (extra elements + are what `is_valid` refuses). + + The path the cells drive is the bf16 one of `quantize_with_block_size` + (`cpp/tensorrt_llm/kernels/quantization.cuh`, per block + `cvt_warp_fp16_to_fp4`): scale byte = e4m3 (round to nearest, saturating) + of `global_scale * max|x| / 6`; codes = e2m1 (round to nearest, saturating) + of `x * global_scale / scale`. At hidden 6144 the launcher + (`invokeFP4Quantization`) runs min(rows, 4 x SMs) CTAs of 512 threads, 608 + on this GB300, each walking rows at a stride of the grid, so past + 608 rows the launch is the same at every row count. Every row's output + depends on that row and `global_scale` alone: at every row count 1..32768; + at every dep4 gathered count 4 x T_max past it, up to 131072 (the `large` + cell; pre-merge drives either side of every grid wrap and rank boundary, + and all of the last grid period, 130465..131072); and in padded CUDA-graph + replays, dep4's (4 x 1..16 rows) and dep8's (8 x 1..16 rows, one live row + with seven ranks idle among them) with each rank's segment padded on its + own. A probe, not a + cell, drove every row count 32769..131072 (98304 calls, 51 s) against the + 131072-row call: each was bitwise its leading rows. A zero row, dep4's + padding of a rank up to T_max, comes out as zero code bytes under scale + byte 0. + + At dep8 each eager call is one chunk `o_all[lo:hi]` of the 8 x T_max + gathered rows, `lo` a multiple of 32768: a contiguous slice at a nonzero + storage offset. A probe, not a cell, quantized every chunk of a + 262144-row gathered set (byte offsets up to 2818572288) and the 8200-row + last chunk of a 73736-row one: each was bitwise the op on a contiguous + copy of the same rows, its scale bytes equal to `reference`. It also ran + dep8's padding skew (rank 0 at 32768 rows, ranks 1..7 at one, T_max + 32768), whose chunks 1..7 each hold one live row and 32767 zero rows: zero + codes under scale byte 0 there, the live row bitwise the op on it alone. + + `global_scale` states the range of the input. The canonical 448*6/amax of + the input itself puts its largest block at scale 448. A static, calibrated + scale (Mistral Large 4: 1 / input_scale(w1), one per MoE layer) states a range + a live input can leave, in both directions, measured at every one of the + checkpoint's 71 scales: + - above it, a block whose `global_scale * max|x| / 6` reaches 448 gets + scale byte 448 (0x7E) and its codes saturate at +-6; no scale byte is + e4m3 NaN, even at bf16's largest finite block max; + - below it, a block whose product rounds under the smallest e4m3 subnormal + gets scale byte 0, so it dequantizes to 0 whatever its codes: at + Mistral Large 4's scales every block whose max|x| is at most the bf16 value + 2.537e-4 (layer 65, 23.0112) down to 6.482e-7 (layer 3, 9039.45), the + largest such block maxima found. Its nonzero inputs come out +-6, + as in `reference`; an exact zero there makes a NaN product (0 x inf), + which the kernel writes as code 7 (+6) and `reference` as 0. The test + does not compare those codes: the block dequantizes to 0 either way. + + Cost on GB300 at hidden 6144: at 1..16 rows, and at dep8's gathered 8..128 + rows, a replay of a one-call CUDA graph is cheaper than an eager call. + `sf_use_ue8m0=True` selects power-of-two scales instead of e4m3. No shipped target uses it and this entry does not certify it. + + Hard limits (sm_103) + rows the leading dims' product, passed to the launcher + (`invokeFP4Quantization`) as `int m` boundary 2^31 rows + not probed: at least 2^35 elements, far past any envelope + (Mistral Large 4's largest call is dep4's 131072 gathered rows; + dep8's is one 32768-row chunk of its up to 262144) + data per 8-element vector, (row) * (K / 8) + col formed in int64 + no int32 boundary below 2^31 rows probed at 349526 x 6144 = + 2147487744 elements (past 2^31), linear: passed -- the rows + straddling element 2^31, the middle rows and the last rows + bitwise equal to the op on those rows alone, their scale + bytes equal to `reference` + scales linear row * (K / 16) + col in int64; swizzled 128x4 tile + offsets in int64 no int32 boundary below 2^31 rows + linear: same probe, passed; swizzled: not probed + the envelope's largest call, dep4's gathered 131072 x 6144 = 805306368 + elements (1610612736 input bytes, 402653184 code bytes, 50331648 scale + bytes), is 37.5% of 2^31 elements; its input bytes would reach 2^31 at + 174763 rows, which the data row's int64 vector offsets never form. The + probe above is 2.67x past it in rows, so it was not probed again. + + Not certified: `sf_use_ue8m0=True`; fp16 or e4m3 input; an input holding + inf or NaN; a `global_scale` that is not finite and positive; Mistral Large 4's + linear 6144 cells above 131072 rows (save the probe above), or with the + swizzled layout; row counts 32769..130464 that are not multiples of 4 (no + dep4 gather passes one; only the probe above drove them); CUDA-graph + capture of any other cell; `torch.compile`. """ def __call__( @@ -133,11 +217,12 @@ def near_tie(scaled: torch.Tensor) -> torch.Tensor: Inside `TIE_WINDOW` of an e2m1 midpoint the kernel's approximate reciprocals decide the direction, so a code that differs there is not a - defect. Outside it, a difference is. + defect. Outside it, a difference is. An infinite product is no tie: it + saturates at +-6 on both sides. """ mid = torch.tensor(_E2M1_MIDPOINTS, device=scaled.device, dtype=torch.float32) a = scaled.abs().unsqueeze(-1) - return ((a - mid).abs() <= TIE_WINDOW * a).any(-1) + return ((a - mid).abs() <= TIE_WINDOW * a).any(-1) & torch.isfinite(scaled) def is_valid( self, diff --git a/tensorrt_llm/_torch/modeling_v2/models/__init__.py b/tensorrt_llm/_torch/modeling_v2/models/__init__.py index 8e4227f5daed..537d81679e05 100644 --- a/tensorrt_llm/_torch/modeling_v2/models/__init__.py +++ b/tensorrt_llm/_torch/modeling_v2/models/__init__.py @@ -17,4 +17,5 @@ from .gpt_oss import routing as gpt_oss_routing # noqa: F401 from .kimi_k3_vl import routing as kimi_k3_vl_routing # noqa: F401 from .minimax_m3 import routing as minimax_m3_routing # noqa: F401 +from .mistral_large4 import routing as mistral_large4_routing # noqa: F401 from .qwen4_exp import routing as qwen4_exp_routing # noqa: F401 diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/__init__.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/__init__.py new file mode 100644 index 000000000000..2b7b7050659b --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MistralLarge4ForCausalLM targets.""" diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/kernels/__init__.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/kernels/__init__.py new file mode 100644 index 000000000000..52a7a9daf028 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/kernels/__init__.py @@ -0,0 +1,2 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/kernels/_ds_mla_sparse_decode_kernel.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/kernels/_ds_mla_sparse_decode_kernel.py new file mode 100644 index 000000000000..f85acf8d0b38 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/kernels/_ds_mla_sparse_decode_kernel.py @@ -0,0 +1,1300 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025 DeepSeek +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 AND MIT +# +# This kernel reproduces the arithmetic of FlashMLA's sm100 sparse fp8 decode and its split-K +# combine (https://github.com/deepseek-ai/FlashMLA, commit +# 15f13e5030374295491c5ce31b02d7e63a7772c6), which is distributed under the MIT License: +# +# MIT License +# +# Copyright (c) 2025 DeepSeek +# +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in all +# copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +# SOFTWARE. +"""The CuTe DSL kernel behind `cute_dsl_ds_mla_sparse_decode`: sparse MLA decode of 8 or 16 +query heads over the fp8_ds_mla latent pool (sm_100f). + +One query row per sequence, top-k 2048 selected pool rows. A 656-byte pool row holds 512 e4m3 +codes, four fp32 power-of-two scales (one per 128 codes) and 64 bf16 rope values. The +arithmetic is FlashMLA's sm100 ``head64`` sparse fp8 decode (V32 layout) and its split-K +combine, operation for operation and in the same order; the entry's ``note`` maps every stage +to FlashMLA's source. What differs is the work layout: the heads are not padded to 64, a +sequence's 2048 selections split over a cluster of 1, 2, 4 or 8 CTAs (a fixed partition, +where FlashMLA's scheduler partitions by load), and the split-K partials merge through the +cluster's shared memory instead of a second kernel. + +Per CTA: a contiguous 2048/splits slice of one row's selections, in tiles of 64 tokens. The +QK MMA is swapped, S^T = K Q^T with M = 64 tokens and N = 16 heads (8 heads run with 8 zero +query rows), as two fp32 accumulators -- FlashMLA's dual-GEMM half-sums. PV is +O^T = V^T P^T, M = 128 latent columns per stripe, N = 16. Warp roles: + +* W9: the query TMA, the index ring, the odd raw slots' TMA gathers. +* W10: the even raw slots' TMA gathers (whole 656-byte rows, ``tile::gather4``). +* W12-15: dequantization into the bf16 K/V tile (SW128, K-major). +* W8: the QK MMAs. W11: the PV MMAs. +* W0-3: the online softmax, one token per thread. +* W4-7: the O rescale, the softmax denominator (FlashMLA's summation order), the epilogue. +""" + +from __future__ import annotations + +import struct +from collections.abc import Callable + +import cuda.bindings.driver as cuda_driver +import cutlass +import cutlass.cute as cute +import cutlass.experimental.cuda as cuda +import cutlass.experimental.primitives.descriptors as _d +import torch +from cutlass.cutlass_dsl import dsl_user_op +from cutlass.experimental import primitives as prims + +D_NOPE = 512 +D_ROPE = 64 +D_QK = D_NOPE + D_ROPE +ROW_BYTES = 656 +ROW_WORDS = ROW_BYTES // 4 +SCALE_WORD = D_NOPE // 4 +TOPK = 2048 +BT = 64 # tokens per tile: the QK MMA is M = 64 (rows on TMEM lanes 32*(r//16) + r%16) +N_MMA = 16 +NUM_STRIPES = 4 +CHUNK_BYTES = BT * 128 # one 64-column bf16 chunk (or one 128-code e4m3 stripe) of a tile +CHUNK_WORDS = CHUNK_BYTES // 4 +ROPE_CHUNK = 8 +BUF_WORDS = 9 * CHUNK_WORDS # one tile: 8 NoPE chunks + the rope chunk +NUM_BUFS = 2 +KV_WORDS = NUM_BUFS * BUF_WORDS +Q_CHUNK_BYTES = N_MMA * 128 +RAW_TOK = 16 # tokens per raw slot (one whole 656-byte row each, TMA box of 82 int64) +RAW_SLOT_BYTES = RAW_TOK * ROW_BYTES # bytes one slot's TMA transactions deliver +RAW_GROUP_BYTES = 2688 # one gather4 (4 rows of 656 B) padded to a 128-byte boundary +RAW_GROUP_WORDS = RAW_GROUP_BYTES // 4 +RAW_SLOT_WORDS = (RAW_TOK // 4) * RAW_GROUP_WORDS +RAW_PER_TILE = BT // RAW_TOK +NUM_RAW = 5 +ST_S = 2 +ST_P = 2 +ST_I = 4 +ST_C = 1 +S_COLS = 2 * N_MMA # one S stage: FlashMLA's two QK half-sums, 16 columns each +TMEM_O = ST_S * S_COLS +TMEM_COLS = 128 +PF_STRIDE = 66 # fp32 P row stride (words) of the li hand-off: conflict-free both ways +THREADS = 512 +W_CORR0 = 4 +W_QK = 8 +W_IDX = 9 +W_TMA = 10 +W_PV = 11 +W_DQ0 = 12 +BAR_SOFTMAX = 1 +BAR_CORR = 2 +MAX_INIT = -1.0e30 +LOG2_E_F32 = 1.44269504 # FlashMLA's `LOG_2_E`, a float constant +SUPPORTED_HEADS = (8, 16) +SUPPORTED_SPLITS = (1, 2, 4, 8) +SW128 = prims.Tcgen05SmemSwizzle.SWIZZLE_128B + + +def f32(x: float) -> float: + """Round a Python float to fp32.""" + return struct.unpack("f", struct.pack("f", x))[0] + + +def scale_log2_of(softmax_scale: float) -> float: + """FlashMLA's `sm_scale_div_log2`: the fp32 product `float(scale) * 1.44269504f`.""" + return f32(f32(softmax_scale) * f32(LOG2_E_F32)) + + +# --------------------------------------------------------------------------- small helpers +@dsl_user_op +def _smem_desc(start_address, leading_byte_offset=0, stride_byte_offset=0, *, loc=None, ip=None): + """A tcgen05 SMEM matrix descriptor (128B swizzle), byte offsets.""" + if isinstance(start_address, (_d.Array, cutlass.Pointer)): + addr = cutlass.Int32(_d.llvm.ptrtoint(cutlass.Int32.mlir_type, start_address.ir_value())) + else: + addr = cutlass.Int32(start_address) + lbo = cutlass.Int32(_d._drop_low_bits(leading_byte_offset, 4, "leading_byte_offset")) + sbo = cutlass.Int32(_d._drop_low_bits(stride_byte_offset, 4, "stride_byte_offset")) + return _d.Tcgen05SmemDesc( + cutlass.Int64( + _d._nvvm.tcgen05_mma_smem_desc_v2( + addr >> 4, + lbo, + sbo, + cutlass.Int8(1), + cutlass.Int8(0), + cutlass.Boolean(0), + cutlass.Boolean(0), + cutlass.Int8(int(SW128)), + loc=loc, + ip=ip, + ) + ) + ) + + +def _ptx1(code, *args): + return cutlass.Float32( + prims.inline_ptx_hl(code, write_only_types=[cutlass.Float32], read_only_args=list(args)) + ) + + +def _ex2(x): # exp2f under --use_fast_math + return _ptx1("ex2.approx.ftz.f32 {$w0}, {$r0};", x) + + +def _lg2(x): # log2f under --use_fast_math + return _ptx1("lg2.approx.ftz.f32 {$w0}, {$r0};", x) + + +def _rcp(x): # __fdividef(1.0f, x) + return _ptx1("div.approx.ftz.f32 {$w0}, 0f3F800000, {$r0};", x) + + +def _fma_rn(a, b, c): # ku::float2_fma: fma.rn.f32x2 (inline asm, no ftz) + return _ptx1("fma.rn.f32 {$w0}, {$r0}, {$r1}, {$r2};", a, b, c) + + +def _fma_ftz(a, b, c): # fmaf under --use_fast_math + return _ptx1("fma.rn.ftz.f32 {$w0}, {$r0}, {$r1}, {$r2};", a, b, c) + + +def _mul_rn(a, b): # ku::float2_mul: mul.f32x2 (inline asm, no ftz) + return _ptx1("mul.rn.f32 {$w0}, {$r0}, {$r1};", a, b) + + +def _mul_ftz(a, b): # float * under --use_fast_math + return _ptx1("mul.rn.ftz.f32 {$w0}, {$r0}, {$r1};", a, b) + + +def _sub_ftz(a, b): + return _ptx1("sub.rn.ftz.f32 {$w0}, {$r0}, {$r1};", a, b) + + +def _add_ftz(a, b): # float + under --use_fast_math + return _ptx1("add.rn.ftz.f32 {$w0}, {$r0}, {$r1};", a, b) + + +def _add_rn(a, b): # ku::float2_add: add.f32x2 (inline asm, no ftz) + return _ptx1("add.rn.f32 {$w0}, {$r0}, {$r1};", a, b) + + +def _mul_into_zero(a, b): # `0.0f += a * b`, contracted: fma.rn.ftz(a, b, +0) + return _ptx1("fma.rn.ftz.f32 {$w0}, {$r0}, {$r1}, 0f00000000;", a, b) + + +def _fmax(a, b): + return _ptx1("max.ftz.f32 {$w0}, {$r0}, {$r1};", a, b) + + +def _bf16x2(lo, hi) -> cutlass.Int32: + """cvt.rn.bf16x2.f32 (`__float22bfloat162_rn`): lo in bits 0-15.""" + return cutlass.Int32( + prims.inline_ptx_hl( + "cvt.rn.bf16x2.f32 {$w0}, {$r0}, {$r1};", + write_only_types=[cutlass.Int32], + read_only_args=[hi, lo], + ) + ) + + +def _scales_to_bf16x2(lo, hi) -> cutlass.Int32: + """Two fp32 scales (as their bit patterns) to bf16x2, lo in bits 0-15: `cvt.rz.ue8m0x2.f32` + (`__nv_cvt_float2_to_e8m0x2(.., __NV_NOSAT, cudaRoundZero)`), then `cvt.rn.bf16x2.ue8m0x2` + (`__nv_cvt_e8m0x2_to_bf162raw`).""" + return cutlass.Int32( + prims.inline_ptx_hl( + "{\n\t.reg .b16 e;\n\tcvt.rz.ue8m0x2.f32 e, {$r1}, {$r0};\n\t" + "cvt.rn.bf16x2.ue8m0x2 {$w0}, e;\n\t}", + write_only_types=[cutlass.Int32], + read_only_args=[lo, hi], + ) + ) + + +def _sel(flag, a, b): + """flag (Int32, nonzero = true) ? a : b, fp32 registers (selp).""" + return _ptx1( + "{\n\t.reg .pred p;\n\tsetp.ne.s32 p, {$r0}, 0;\n\tselp.f32 {$w0}, {$r1}, {$r2}, p;\n\t}", + flag, + a, + b, + ) + + +def _sel_i32(flag, a, b): + """flag ? a : b (Int32 registers).""" + return cutlass.Int32( + prims.inline_ptx_hl( + "{\n\t.reg .pred p;\n\tsetp.ne.s32 p, {$r0}, 0;\n\tselp.b32 {$w0}, {$r1}, {$r2}, p;\n\t}", + write_only_types=[cutlass.Int32], + read_only_args=[flag, a, b], + ) + ) + + +# Constants live in the PTX text: inline-asm float operands must be registers. +F32_NEG_INF = "0fFF800000" +F32_POS_INF = "0f7F800000" +F32_ZERO = "0f00000000" +F32_SIX = "0f40C00000" + + +def _cmp_i32(op, a, lit) -> cutlass.Int32: + """1 if (a lit) for an fp32 PTX literal, else 0.""" + return cutlass.Int32( + prims.inline_ptx_hl( + "{\n\t.reg .pred p;\n\tsetp." + + op + + ".f32 p, {$r0}, " + + lit + + ";\n\tselp.s32 {$w0}, 1, 0, p;\n\t}", + write_only_types=[cutlass.Int32], + read_only_args=[a], + ) + ) + + +def _sel_lit(flag, a, lit): + """flag ? a : lit (fp32 PTX literal).""" + return _ptx1( + "{\n\t.reg .pred p;\n\tsetp.ne.s32 p, {$r0}, 0;\n\t" + "selp.f32 {$w0}, {$r1}, " + lit + ", p;\n\t}", + flag, + a, + ) + + +def _shfl_bfly(v, offset): + return cutlass.Float32( + prims.shfl_sync( + thread_mask=0xFFFFFFFF, val=v, offset=offset, mask_and_clamp=0x1F, kind=prims.Shfl.BFLY + ) + ) + + +def _pick_word(raw4, half, w: int): + """raw4[half * 2 + w] for a runtime half in {0, 1}.""" + return _sel_i32(half, raw4[2 + w], raw4[w]) + + +def _max_scatter16(vals, lane): + """Warp max of 16 per-lane values; returns, on lane l, the max of value (l >> 1) & 15 over + all 32 lanes (16 shuffles instead of 80).""" + v = list(vals) + n = 16 + for off in (16, 8, 4, 2): + up = (lane >> cutlass.Int32(off.bit_length() - 1)) & cutlass.Int32(1) + half = n // 2 + nv = [] + for i in range(half): + send = _sel(up, v[i], v[half + i]) + keep = _sel(up, v[half + i], v[i]) + other = _shfl_bfly(send, off) + nv.append(_fmax(keep, other)) + v = nv + n = half + return _fmax(v[0], _shfl_bfly(v[0], 1)) + + +def _butterfly_sum(terms): + """The sum a xor butterfly over lanes 0..n-1 (n a power of two) produces: pairs i, i + n/2 + first, then halving.""" + while len(terms) > 1: + hn = len(terms) // 2 + terms = [_add_ftz(terms[i], terms[i + hn]) for i in range(hn)] + return terms[0] + + +@cute.jit +def _issue_tile_raw( + kt, st, parity: cutlass.Constexpr[int], lane, sIdx, raw_full, raw_empty, sRaw, tma_raw +) -> None: + """Gather the raw slots q = parity, parity + 2 of tile kt (16 whole 656-byte rows each) + into the shared ring; lanes 0..3 issue one gather4 each.""" + rl = lane % cutlass.Int32(RAW_TOK // 4) + for qq in cutlass.range_constexpr(RAW_PER_TILE // 2): + q: cutlass.Constexpr[int] = 2 * qq + parity + rows4 = ( + sIdx.data_ptr() + + st * cutlass.Int32(BT) + + cutlass.Int32(q * RAW_TOK) + + rl * cutlass.Int32(4) + ).load(count=4, alignment=16) + g = kt * cutlass.Int32(RAW_PER_TILE) + cutlass.Int32(q) + slot = g % cutlass.Int32(NUM_RAW) + gph = (g // cutlass.Int32(NUM_RAW)) % cutlass.Int32(2) + _wait(raw_empty.data_ptr() + slot, gph ^ cutlass.Int32(1)) + if prims.elect_sync(): + prims.mbarrier_arrive_expect_tx(raw_full.data_ptr() + slot, RAW_SLOT_BYTES) + prims.bar_warp_sync(0xFFFFFFFF) + if lane < cutlass.Int32(RAW_TOK // 4): + _tma_gather4_lanes( + sRaw.data_ptr() + + slot * cutlass.Int32(RAW_SLOT_WORDS) + + rl * cutlass.Int32(RAW_GROUP_WORDS), + tma_raw.get_ptr(), + raw_full.data_ptr() + slot, + cutlass.Int32(0), + rows4[0], + rows4[1], + rows4[2], + rows4[3], + ) + + +@cute.jit +def _wait(bar, phase): + while not prims.mbarrier_try_wait_parity(bar, phase): + pass + + +@cute.jit +def _tma_gather4_lanes(smem_dst, tma_desc, mbarrier, col, row0, row1, row2, row3) -> None: + """Every lane issues its own `tile::gather4` TMA transaction (4 rows of one box): one + warp instruction moves 128 rows.""" + local_barrier = prims.cvta_to(mbarrier, prims.CvtaSpace.SHARED) + prims.inline_ptx_hl( + "cp.async.bulk.tensor.2d.shared::cta.global.tile::gather4" + ".mbarrier::complete_tx::bytes.L2::cache_hint" + " [{$r0}], [{$r1}, {{$r2}, {$r3}, {$r4}, {$r5}, {$r6}}], [{$r7}], {$r8};", + read_only_args=[ + smem_dst, + tma_desc, + col, + row0, + row1, + row2, + row3, + local_barrier, + cutlass.Int64(0), + ], + ) + + +# ----------------------------------------------------------------------------------- kernel +@cute.kernel +def _dsmla_decode_kernel( + tma_q: cutlass.GridConstant[cuda.TensorMap], + tma_raw: cutlass.GridConstant[cuda.TensorMap], + indices: cute.Tensor, + out: cute.Tensor, + scale_log2: cutlass.Float32, + num_heads: cutlass.Constexpr[int], + num_splits: cutlass.Constexpr[int], + global_partials: cutlass.Constexpr[bool] = False, +) -> None: + tidx, _, _ = cute.arch.thread_idx() + warp = cute.arch.warp_idx() + lane = tidx % cutlass.Int32(32) + split_x, _, row = cute.arch.block_idx() + split = split_x + if cutlass.const_expr(num_splits > 1 and not global_partials): + split = cute.arch.block_idx_in_cluster() + tokens_per_cta: cutlass.Constexpr[int] = TOPK // num_splits + num_kt: cutlass.Constexpr[int] = tokens_per_cta // BT + tok_base = row * cutlass.Int32(TOPK) + split * cutlass.Int32(tokens_per_cta) + + # ---- shared memory ------------------------------------------------------------------ + # The KV tile is typed fp32 so that it doubles as the fp32 split-K partial afterwards; + # dequantized bf16 pairs are stored as bit-cast 32-bit words. + sKV = cutlass.Array(cutlass.Float32, KV_WORDS, space=cutlass.AddressSpace.smem, alignment=1024) + sRaw = cutlass.Array( + cutlass.Int32, NUM_RAW * RAW_SLOT_WORDS, space=cutlass.AddressSpace.smem, alignment=1024 + ) + sQ = cutlass.Array( + cutlass.Int32, 9 * Q_CHUNK_BYTES // 4, space=cutlass.AddressSpace.smem, alignment=1024 + ) + sP = cutlass.Array( + cutlass.Int16, ST_P * N_MMA * 64, space=cutlass.AddressSpace.smem, alignment=1024 + ) + sIdx = cutlass.Array(cutlass.Int32, ST_I * BT, space=cutlass.AddressSpace.smem, alignment=16) + sMax = cutlass.Array( + cutlass.Float32, 2 * 4 * N_MMA, space=cutlass.AddressSpace.smem, alignment=16 + ) + sCorr = cutlass.Array( + cutlass.Float32, ST_C * N_MMA, space=cutlass.AddressSpace.smem, alignment=16 + ) + sFlag = cutlass.Array(cutlass.Int32, ST_C, space=cutlass.AddressSpace.smem, alignment=16) + sFin = cutlass.Array(cutlass.Float32, N_MMA, space=cutlass.AddressSpace.smem, alignment=16) + sLse = cutlass.Array(cutlass.Float32, N_MMA, space=cutlass.AddressSpace.smem, alignment=16) + # fp32 P of the current tile [head][token] for the denominator chains, and their partials. + sPf = cutlass.Array( + cutlass.Float32, N_MMA * PF_STRIDE, space=cutlass.AddressSpace.smem, alignment=16 + ) + sLi = cutlass.Array(cutlass.Float32, 2 * N_MMA, space=cutlass.AddressSpace.smem, alignment=16) + q_full = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem) + idx_full = cutlass.Array(cutlass.Int64, ST_I, space=cutlass.AddressSpace.smem) + idx_empty = cutlass.Array(cutlass.Int64, ST_I, space=cutlass.AddressSpace.smem) + raw_full = cutlass.Array(cutlass.Int64, NUM_RAW, space=cutlass.AddressSpace.smem) + raw_empty = cutlass.Array(cutlass.Int64, NUM_RAW, space=cutlass.AddressSpace.smem) + kv_full = cutlass.Array(cutlass.Int64, NUM_BUFS, space=cutlass.AddressSpace.smem) + kv_empty = cutlass.Array(cutlass.Int64, NUM_BUFS, space=cutlass.AddressSpace.smem) + s_full = cutlass.Array(cutlass.Int64, ST_S, space=cutlass.AddressSpace.smem) + s_empty = cutlass.Array(cutlass.Int64, ST_S, space=cutlass.AddressSpace.smem) + p_full = cutlass.Array(cutlass.Int64, ST_P, space=cutlass.AddressSpace.smem) + p_empty = cutlass.Array(cutlass.Int64, ST_P, space=cutlass.AddressSpace.smem) + c_full = cutlass.Array(cutlass.Int64, ST_C, space=cutlass.AddressSpace.smem) + c_empty = cutlass.Array(cutlass.Int64, ST_C, space=cutlass.AddressSpace.smem) + o_full = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem) + o_empty = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem) + fin_full = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem) + tmem_i32 = cutlass.Array(cutlass.Int32, 1, space=cutlass.AddressSpace.smem) + + if warp == 0: + if prims.elect_sync(): + prims.mbarrier_init(q_full, 1) + for i in cutlass.range_constexpr(ST_I): + prims.mbarrier_init(idx_full.data_ptr() + i, 1) + prims.mbarrier_init(idx_empty.data_ptr() + i, 6) + for i in cutlass.range_constexpr(NUM_RAW): + prims.mbarrier_init(raw_full.data_ptr() + i, 1) + prims.mbarrier_init(raw_empty.data_ptr() + i, 4) + for b in cutlass.range_constexpr(NUM_BUFS): + prims.mbarrier_init(kv_full.data_ptr() + b, 4 * RAW_PER_TILE) + prims.mbarrier_init(kv_empty.data_ptr() + b, 1) + for i in cutlass.range_constexpr(ST_S): + prims.mbarrier_init(s_full.data_ptr() + i, 1) + prims.mbarrier_init(s_empty.data_ptr() + i, 4) + for i in cutlass.range_constexpr(ST_P): + prims.mbarrier_init(p_full.data_ptr() + i, 4) + prims.mbarrier_init(p_empty.data_ptr() + i, 1) + for i in cutlass.range_constexpr(ST_C): + prims.mbarrier_init(c_full.data_ptr() + i, 4) + prims.mbarrier_init(c_empty.data_ptr() + i, 4) + prims.mbarrier_init(o_full, 1) + prims.mbarrier_init(o_empty, 4) + prims.mbarrier_init(fin_full, 1) + prims.fence_mbarrier_init() + if cutlass.const_expr(num_splits > 1 and not global_partials): + prims.barrier_cluster_arrive_relaxed() + if warp == W_PV: + prims.tcgen05_alloc(tmem_i32, TMEM_COLS, group="cta_1") + if cutlass.const_expr(num_splits > 1 and not global_partials): + prims.barrier_cluster_wait() + prims.barrier_cta_sync(0) + tmem_raw = tmem_i32.load() + if warp == W_PV: + prims.tcgen05_relinquish_alloc_permit(group="cta_1") + # Everything below reads what the preceding kernels wrote (query, indices, pool rows). + prims.griddepcontrol(prims.GridDepAction.WAIT) + + # ===================================================================================== + # W9: query TMA (9 chunks of 64 bf16 x heads, SW128) and the per-tile index ring. + # ===================================================================================== + if warp == W_IDX: + if cutlass.const_expr(num_heads < N_MMA): + # Rows num_heads..15 of every query chunk are zero (8-head mode runs N = 16). + zero = cutlass.Vector.from_elements( + tuple(cutlass.Int32(0) for _ in range(4)), dtype=cutlass.Int32 + ) + for c in cutlass.range_constexpr(9): + for rep_ in cutlass.range_constexpr((N_MMA - num_heads) * 128 // 16 // 32): + off = c * (Q_CHUNK_BYTES // 4) + num_heads * 32 + (lane + rep_ * 32) * 4 + (sQ.data_ptr() + off).store(zero, alignment=16) + prims.fence_proxy(prims.Proxy.ASYNC_SHARED, space=prims.SharedSpace.shared_cta) + prims.bar_warp_sync(0xFFFFFFFF) + if prims.elect_sync(): + prims.mbarrier_arrive_expect_tx(q_full, 9 * num_heads * 128) + for c in cutlass.range_constexpr(9): + prims.cp_async_bulk_tensor_shared_cta_global( + sQ.data_ptr() + c * (Q_CHUNK_BYTES // 4), + tma_q.get_ptr(), + [c * 64, 0, row], + q_full, + ) + idx_ptr = indices.iterator.raw_ptr() + for kt in cutlass.range(0, num_kt, 1, unroll=1): + st = kt % cutlass.Int32(ST_I) + ph = (kt // cutlass.Int32(ST_I)) % cutlass.Int32(2) + _wait(idx_empty.data_ptr() + st, ph ^ cutlass.Int32(1)) + if lane < cutlass.Int32(BT // 4): + vals = (idx_ptr + tok_base + kt * cutlass.Int32(BT) + lane * cutlass.Int32(4)).load( + count=4, alignment=16 + ) + (sIdx.data_ptr() + st * cutlass.Int32(BT) + lane * cutlass.Int32(4)).store( + vals, alignment=16 + ) + prims.bar_warp_sync(0xFFFFFFFF) + if prims.elect_sync(): + prims.mbarrier_arrive(idx_full.data_ptr() + st) + # W9 also issues the odd raw slots of this tile (W10 the even ones): the TMA unit + # serves one warp's requests one after another, two warps' in parallel. + _issue_tile_raw(kt, st, 1, lane, sIdx, raw_full, raw_empty, sRaw, tma_raw) + prims.bar_warp_sync(0xFFFFFFFF) + if prims.elect_sync(): + prims.mbarrier_arrive(idx_empty.data_ptr() + st) + + # ===================================================================================== + # W10: TMA gathers of the even raw slots (16 whole 656-byte rows each) into the 5-slot raw + # ring. A -1 index is an out-of-bounds row: TMA zero-fills it, codes and scales (FlashMLA + # sets `tma_coords = -1` and the scale bytes 0; the token is masked to -inf either way). + # ===================================================================================== + if warp == W_TMA: + for kt in cutlass.range(0, num_kt, 1, unroll=1): + st = kt % cutlass.Int32(ST_I) + ph = (kt // cutlass.Int32(ST_I)) % cutlass.Int32(2) + _wait(idx_full.data_ptr() + st, ph) + _issue_tile_raw(kt, st, 0, lane, sIdx, raw_full, raw_empty, sRaw, tma_raw) + prims.bar_warp_sync(0xFFFFFFFF) + if prims.elect_sync(): + prims.mbarrier_arrive(idx_empty.data_ptr() + st) + + # ===================================================================================== + # W12-15: dequantization, eight threads per token row. bf16(e4m3) * bf16(scale) with the + # scale rounded toward zero to a power of two (FlashMLA: e8m0 RZ, then e8m0 -> bf16), in + # the SW128 K-major layout the QK MMA reads and the PV MMA reads MN-major; the rope is + # copied as it is. + # ===================================================================================== + if warp >= W_DQ0: + dq = tidx - cutlass.Int32(W_DQ0 * 32) + t = dq >> cutlass.Int32(3) # token within a 16-token raw slot + part = dq & cutlass.Int32(7) # 16-byte pieces part, part+8, part+16, part+24 of the codes + for kt in cutlass.range(0, num_kt, 1, unroll=1): + b = kt % cutlass.Int32(NUM_BUFS) + u = kt // cutlass.Int32(NUM_BUFS) + kvb = b * cutlass.Int32(BUF_WORDS) + for q in cutlass.range_constexpr(RAW_PER_TILE): + g = kt * cutlass.Int32(RAW_PER_TILE) + cutlass.Int32(q) + slot = g % cutlass.Int32(NUM_RAW) + gph = (g // cutlass.Int32(NUM_RAW)) % cutlass.Int32(2) + _wait(raw_full.data_ptr() + slot, gph) + if cutlass.const_expr(q == 0): + _wait(kv_empty.data_ptr() + b, (u % cutlass.Int32(2)) ^ cutlass.Int32(1)) + rrow = ( + slot * cutlass.Int32(RAW_SLOT_WORDS) + + (t >> cutlass.Int32(2)) * cutlass.Int32(RAW_GROUP_WORDS) + + (t & cutlass.Int32(3)) * cutlass.Int32(ROW_WORDS) + ) + trow = cutlass.Int32(q * RAW_TOK) + t # row of the 64-token tile + sw = trow % cutlass.Int32(8) + sc = (sRaw.data_ptr() + rrow + cutlass.Int32(SCALE_WORD)).load( + count=4, alignment=16 + ) + # FlashMLA's two conversions: the fp32 scales to e8m0 toward zero, then to bf16. + pairs = [_scales_to_bf16x2(sc[0], sc[1]), _scales_to_bf16x2(sc[2], sc[3])] + for m in cutlass.range_constexpr(NUM_STRIPES): + sbits = (pairs[m // 2] >> cutlass.Int32(16 * (m % 2))) & cutlass.Int32(0xFFFF) + scale2 = sbits | (sbits << cutlass.Int32(16)) + k = part + cutlass.Int32(8 * m) # 16-code piece of the row + raw4 = (sRaw.data_ptr() + rrow + k * cutlass.Int32(4)).load( + count=4, alignment=16 + ) + chunk = k >> cutlass.Int32(2) + # Threads part and part+4 of a token write the same piece slots of two + # chunks 8 KB apart (same banks): the upper four take the halves in the + # opposite order so that each store instruction hits 8 distinct bank groups. + flip = part >> cutlass.Int32(2) + for hh in cutlass.range_constexpr(2): + half = cutlass.Int32(hh) ^ flip + packed = [] + for w in cutlass.range_constexpr(2): + word = _pick_word(raw4, half, w) + lo = prims.cvt_e4m3x2_word_to_bf16x2(word, 0) + hi = prims.cvt_e4m3x2_word_to_bf16x2(word, 16) + packed.append(prims.mul_bf16x2(lo, scale2)) + packed.append(prims.mul_bf16x2(hi, scale2)) + vec = cutlass.Vector.from_elements(tuple(packed), dtype=cutlass.Int32) + piece = (k & cutlass.Int32(3)) * cutlass.Int32(2) + half + dst = ( + kvb + + chunk * cutlass.Int32(CHUNK_WORDS) + + trow * cutlass.Int32(32) + + ((piece ^ sw) * cutlass.Int32(4)) + ) + (sKV.data_ptr() + dst).store(vec.bitcast(cutlass.Float32), alignment=16) + # Rope: 8 pieces of 8 bf16, copied as they are. + rope4 = ( + sRaw.data_ptr() + rrow + cutlass.Int32(SCALE_WORD + 4) + part * cutlass.Int32(4) + ).load(count=4, alignment=16) + rdst = ( + kvb + + cutlass.Int32(ROPE_CHUNK * CHUNK_WORDS) + + trow * cutlass.Int32(32) + + ((part ^ sw) * cutlass.Int32(4)) + ) + (sKV.data_ptr() + rdst).store(rope4.bitcast(cutlass.Float32), alignment=16) + prims.fence_proxy(prims.Proxy.ASYNC_SHARED, space=prims.SharedSpace.shared_cta) + prims.bar_warp_sync(0xFFFFFFFF) + if prims.elect_sync(): + prims.mbarrier_arrive(raw_empty.data_ptr() + slot) + prims.mbarrier_arrive(kv_full.data_ptr() + b) + + # ===================================================================================== + # W8: QK. S^T[tok, h] = K[tok, :] . Q[h, :] as FlashMLA's two half-sums (below), each + # its own fp32 accumulator in TMEM. + # ===================================================================================== + if warp == W_QK: + idesc_qk = prims.Tcgen05InstrDesc.build( + c_dtype=cutlass.Float32, + a_dtype=cutlass.BFloat16, + b_dtype=cutlass.BFloat16, + n_dim=N_MMA, + m_dim=BT, + ) + _wait(q_full, 0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + for kt in cutlass.range(0, num_kt, 1, unroll=1): + sst = kt % cutlass.Int32(ST_S) + sph = (kt // cutlass.Int32(ST_S)) % cutlass.Int32(2) + _wait(s_empty.data_ptr() + sst, sph ^ cutlass.Int32(1)) + # FlashMLA's dual GEMM (V32 layout): half 0 = rope dims 0-31, then NoPE chunks + # 0, 2, 4, 6; half 1 = rope dims 32-63, then chunks 1, 3, 5, 7. Each half is its own + # fp32 accumulator, K = 16 per MMA in that order; the softmax adds the two. + s_tm0 = tmem_raw + sst * cutlass.Int32(S_COLS) + b = kt % cutlass.Int32(NUM_BUFS) + u = kt // cutlass.Int32(NUM_BUFS) + kvb = sKV.data_ptr() + b * cutlass.Int32(BUF_WORDS) + _wait(kv_full.data_ptr() + b, u % cutlass.Int32(2)) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + dkr = _smem_desc(kvb + ROPE_CHUNK * CHUNK_WORDS, 16, 1024) + dqr = _smem_desc(sQ.data_ptr() + ROPE_CHUNK * (Q_CHUNK_BYTES // 4), 16, 1024) + for step in cutlass.range_constexpr(2): + for half in cutlass.range_constexpr(2): + s_tmem = cutlass.inttoptr(s_tm0 + half * N_MMA, 6, cutlass.Float32) + kb: cutlass.Constexpr[int] = 2 * half + step + if prims.elect_sync(): + prims.tcgen05_mma( + prims.Tcgen05MMAKind.F16, + prims.CTAGroup.CTA_1, + s_tmem, + dkr.advance_start_address(kb * 32), + dqr.advance_start_address(kb * 32), + idesc_qk, + step != 0, + ) + for j2 in cutlass.range_constexpr(4): + for kb in cutlass.range_constexpr(4): + for half in cutlass.range_constexpr(2): + c: cutlass.Constexpr[int] = 2 * j2 + half + s_tmem = cutlass.inttoptr(s_tm0 + half * N_MMA, 6, cutlass.Float32) + dk = _smem_desc(kvb + c * CHUNK_WORDS, 16, 1024) + dq = _smem_desc(sQ.data_ptr() + c * (Q_CHUNK_BYTES // 4), 16, 1024) + if prims.elect_sync(): + prims.tcgen05_mma( + prims.Tcgen05MMAKind.F16, + prims.CTAGroup.CTA_1, + s_tmem, + dk.advance_start_address(kb * 32), + dq.advance_start_address(kb * 32), + idesc_qk, + True, + ) + if prims.elect_sync(): + prims.tcgen05_commit(s_full.data_ptr() + sst, group=prims.CTAGroup.CTA_1) + + # ===================================================================================== + # W11: PV. O^T[d, h] += V^T[d, tok] . P^T[tok, h] per 128-column stripe; V^T is the + # dequantized NoPE tile read MN-major (two 64-column chunks 16 KB apart). + # ===================================================================================== + if warp == W_PV: + idesc_pv = prims.Tcgen05InstrDesc.build( + c_dtype=cutlass.Float32, + a_dtype=cutlass.BFloat16, + b_dtype=cutlass.BFloat16, + n_dim=N_MMA, + m_dim=128, + a_major=1, + b_major=0, + ) + for kt in cutlass.range(0, num_kt, 1, unroll=1): + pst = kt % cutlass.Int32(ST_P) + pph = (kt // cutlass.Int32(ST_P)) % cutlass.Int32(2) + _wait(p_full.data_ptr() + pst, pph) + _wait(o_empty, kt % cutlass.Int32(2)) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + b = kt % cutlass.Int32(NUM_BUFS) + kvb = sKV.data_ptr() + b * cutlass.Int32(BUF_WORDS) + dp = _smem_desc(sP.data_ptr() + pst * cutlass.Int32(N_MMA * 64), 16, 1024) + for s in cutlass.range_constexpr(NUM_STRIPES): + o_tmem = cutlass.inttoptr(tmem_raw + TMEM_O + s * N_MMA, 6, cutlass.Float32) + dv = _smem_desc(kvb + 2 * s * CHUNK_WORDS, CHUNK_BYTES, 1024) + for kk in cutlass.range_constexpr(BT // 16): + if prims.elect_sync(): + prims.tcgen05_mma( + prims.Tcgen05MMAKind.F16, + prims.CTAGroup.CTA_1, + o_tmem, + dv.advance_start_address(kk * 2048), + dp.advance_start_address(kk * 32), + idesc_pv, + (kt > cutlass.Int32(0)) or (kk != 0), + ) + if prims.elect_sync(): + prims.tcgen05_commit(kv_empty.data_ptr() + b, group=prims.CTAGroup.CTA_1) + prims.tcgen05_commit(o_full, group=prims.CTAGroup.CTA_1) + prims.tcgen05_commit(p_empty.data_ptr() + pst, group=prims.CTAGroup.CTA_1) + + # ===================================================================================== + # W0-3: online softmax, one token per thread (TMEM lane), 16 heads per thread. + # ===================================================================================== + if warp < 4: + # M = 64 accumulator: token 16*warp + lane on lanes 0..15 of the warp's quadrant. + t = warp * cutlass.Int32(16) + lane % cutlass.Int32(16) + live = cutlass.Int32(1) + if lane >= cutlass.Int32(16): + live = cutlass.Int32(0) + base = prims.TmemAddr(tmem_raw) + my_lane_addr = ((base.row_id + warp * cutlass.Int32(32)) << cutlass.Int32(16)) | base.col_id + neg_inf = cutlass.Float32(-float("inf")) + mi = cutlass.Vector.from_elements( + tuple(cutlass.Float32(MAX_INIT) for _ in range(N_MMA)), dtype=cutlass.Float32 + ) + real_mi = cutlass.Vector.from_elements( + tuple(neg_inf for _ in range(N_MMA)), dtype=cutlass.Float32 + ) + p_byte = t * cutlass.Int32(2) + for kt in cutlass.range(0, num_kt, 1, unroll=1): + sst = kt % cutlass.Int32(ST_S) + sph = (kt // cutlass.Int32(ST_S)) % cutlass.Int32(2) + idx_st = kt % cutlass.Int32(ST_I) + idx_ph = (kt // cutlass.Int32(ST_I)) % cutlass.Int32(2) + _wait(idx_full.data_ptr() + idx_st, idx_ph) + valid = live + if sIdx[idx_st * cutlass.Int32(BT) + t] == cutlass.Int32(-1): + valid = cutlass.Int32(0) + prims.bar_warp_sync(0xFFFFFFFF) + if prims.elect_sync(): + prims.mbarrier_arrive(idx_empty.data_ptr() + idx_st) + _wait(s_full.data_ptr() + sst, sph) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + raw = prims.tcgen05_ld( + "32x32b", + cutlass.inttoptr(my_lane_addr + sst * cutlass.Int32(S_COLS), 6, cutlass.Float32), + num=S_COLS, + ) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + prims.bar_warp_sync(0xFFFFFFFF) + if prims.elect_sync(): + prims.mbarrier_arrive(s_empty.data_ptr() + sst) + # The two QK half-sums added (ku::float2_add); invalid tokens masked to -inf. + sv = [ + _sel_lit(valid, _add_rn(raw[h], raw[N_MMA + h]), F32_NEG_INF) for h in range(N_MMA) + ] + # Per-head max over this tile's 128 tokens. + hmax = _max_scatter16(sv, lane) + mbuf = (kt % cutlass.Int32(2)) * cutlass.Int32(4 * N_MMA) + warp * cutlass.Int32(N_MMA) + if lane % cutlass.Int32(2) == cutlass.Int32(0): + sMax[mbuf + (lane >> cutlass.Int32(1))] = hmax + prims.barrier_cta_sync(BAR_SOFTMAX, thread_count=128) + mb0 = (kt % cutlass.Int32(2)) * cutlass.Int32(4 * N_MMA) + cur = [] + for h in cutlass.range_constexpr(N_MMA): + m = sMax[mb0 + h] + for w in cutlass.range_constexpr(1, 4): + m = _fmax(m, sMax[mb0 + w * N_MMA + h]) + cur.append(_mul_ftz(m, scale_log2)) + # FlashMLA's lazy rescale: move the running max only when some head's tile max + # exceeds it by more than 6 (log2 units); all live heads vote together. + should = cutlass.Int32(0) + for h in cutlass.range_constexpr(num_heads): + should = should | _cmp_i32("gt", _sub_ftz(cur[h], mi[h]), F32_SIX) + new_max = [_sel(should, _fmax(cur[h], mi[h]), mi[h]) for h in range(N_MMA)] + # exp2(0) is exactly 1: heads (or tiles) that keep their max get factor 1. + sfo = [_ex2(_sub_ftz(mi[h], new_max[h])) for h in range(N_MMA)] + real_mi = cutlass.Vector.from_elements( + tuple(_fmax(real_mi[h], cur[h]) for h in range(N_MMA)), dtype=cutlass.Float32 + ) + mi = cutlass.Vector.from_elements(tuple(new_max), dtype=cutlass.Float32) + pv = [_ex2(_fma_rn(sv[h], scale_log2, -new_max[h])) for h in range(N_MMA)] + # P^T [h][tok] bf16, K-major SW128 (one 64-token atom per head row). + pst = kt % cutlass.Int32(ST_P) + pph = (kt // cutlass.Int32(ST_P)) % cutlass.Int32(2) + _wait(p_empty.data_ptr() + pst, pph ^ cutlass.Int32(1)) + p_katom = pst * cutlass.Int32(N_MMA * 64) + for h2 in cutlass.range_constexpr(N_MMA // 2): + pk = _bf16x2(pv[2 * h2], pv[2 * h2 + 1]) + for e in cutlass.range_constexpr(2): + h: cutlass.Constexpr[int] = 2 * h2 + e + piece = (p_byte // cutlass.Int32(16)) ^ cutlass.Int32(h % 8) + elt = ( + p_katom + + cutlass.Int32(h * 64) + + piece * cutlass.Int32(8) + + (p_byte % cutlass.Int32(16)) // cutlass.Int32(2) + ) + if live != cutlass.Int32(0): + sP[elt] = cutlass.Int16( + (pk >> cutlass.Int32(16 * e)) & cutlass.Int32(0xFFFF) + ) + prims.fence_proxy(prims.Proxy.ASYNC_SHARED, space=prims.SharedSpace.shared_cta) + prims.bar_warp_sync(0xFFFFFFFF) + if prims.elect_sync(): + prims.mbarrier_arrive(p_full.data_ptr() + pst) + # Hand-off to the correction warps: the fp32 P of this tile (they sum it in + # FlashMLA's order), the factor on the old O and l, and whether O is rescaled. + cst = kt % cutlass.Int32(ST_C) + cph = (kt // cutlass.Int32(ST_C)) % cutlass.Int32(2) + _wait(c_empty.data_ptr() + cst, cph ^ cutlass.Int32(1)) + if live != cutlass.Int32(0): + for h in cutlass.range_constexpr(N_MMA): + sPf[cutlass.Int32(h * PF_STRIDE) + t] = pv[h] + if warp == 0: + if lane == cutlass.Int32(0): + for h in cutlass.range_constexpr(N_MMA): + sCorr[cst * cutlass.Int32(N_MMA) + h] = sfo[h] + sFlag[cst] = should + prims.bar_warp_sync(0xFFFFFFFF) + if prims.elect_sync(): + prims.mbarrier_arrive(c_full.data_ptr() + cst) + # Final max. No valid index at all: max = -inf, and l = 0 (FlashMLA). + if tidx == cutlass.Int32(0): + for h in cutlass.range_constexpr(N_MMA): + has = _cmp_i32("ne", real_mi[h], F32_NEG_INF) + sFin[h] = _sel_lit(has, mi[h], F32_NEG_INF) + prims.mbarrier_arrive(fin_full) + + # ===================================================================================== + # W4-7: the O rescale per tile, the softmax denominator, then the epilogue. Thread = TMEM + # lane = latent column of every 128-column O stripe. + # ===================================================================================== + if warp >= W_CORR0 and warp < W_QK: + cw = warp - cutlass.Int32(W_CORR0) + col = cw * cutlass.Int32(32) + lane + base = prims.TmemAddr(tmem_raw) + my_addr = ((base.row_id + cw * cutlass.Int32(32)) << cutlass.Int32(16)) | base.col_id + # Warps 4 and 5 keep FlashMLA's denominator: warp 4 the tokens 0-31 of every tile, warp + # 5 the tokens 32-63; lane = 2 * head + parity (even or odd tokens). + lh = lane >> cutlass.Int32(1) + lpar = lane & cutlass.Int32(1) + pf0 = lh * cutlass.Int32(PF_STRIDE) + cw * cutlass.Int32(32) + lpar + li_part = cutlass.Float32(0.0) + for kt in cutlass.range(0, num_kt, 1, unroll=1): + cst = kt % cutlass.Int32(ST_C) + cph = (kt // cutlass.Int32(ST_C)) % cutlass.Int32(2) + _wait(c_full.data_ptr() + cst, cph) + flag = sFlag[cst] + corr = (sCorr.data_ptr() + cst * cutlass.Int32(N_MMA)).load(count=N_MMA, alignment=16) + if cw < cutlass.Int32(2): + # ku::float2_add chain from 0: d0 + d2 + ... (exact first add), in token order. + tsum = sPf[pf0] + for i2 in cutlass.range_constexpr(1, 16): + tsum = _add_rn(tsum, sPf[pf0 + cutlass.Int32(2 * i2)]) + sfo_h = sCorr[cst * cutlass.Int32(N_MMA) + lh] + both = _add_ftz(tsum, _shfl_bfly(tsum, 1)) + li_part = _fma_ftz(li_part, sfo_h, both) + prims.bar_warp_sync(0xFFFFFFFF) + if prims.elect_sync(): + prims.mbarrier_arrive(c_empty.data_ptr() + cst) + if kt > cutlass.Int32(0): + _wait(o_full, (kt - cutlass.Int32(1)) % cutlass.Int32(2)) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + if flag != cutlass.Int32(0): + for s in cutlass.range_constexpr(NUM_STRIPES): + tp = cutlass.inttoptr(my_addr + TMEM_O + s * N_MMA, 6, cutlass.Float32) + ov = prims.tcgen05_ld("32x32b", tp, num=N_MMA) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + nv = cutlass.Vector.from_elements( + tuple(_mul_rn(ov[h], corr[h]) for h in range(N_MMA)), + dtype=cutlass.Float32, + ) + prims.tcgen05_st("32x32b", tp, nv) + prims.tcgen05_wait(prims.Tcgen05Wait.STORE) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + prims.bar_warp_sync(0xFFFFFFFF) + if prims.elect_sync(): + prims.mbarrier_arrive(o_empty) + _wait(o_full, (cutlass.Int32(num_kt) - cutlass.Int32(1)) % cutlass.Int32(2)) + _wait(fin_full, 0) + if cw < cutlass.Int32(2): + if lpar == cutlass.Int32(0): + sLi[cw * cutlass.Int32(N_MMA) + lh] = li_part + prims.barrier_cta_sync(BAR_CORR, thread_count=128) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + fin_m = sFin.data_ptr().load(count=N_MMA, alignment=16) + # l = l(tokens 0-31) + l(tokens 32-63); 0 when no index was valid. + fin_l = [ + _sel_lit( + _cmp_i32("ne", fin_m[h], F32_NEG_INF), _add_ftz(sLi[h], sLi[N_MMA + h]), F32_ZERO + ) + for h in range(N_MMA) + ] + osc = [ + _sel_lit(_cmp_i32("ne", fin_l[h], F32_ZERO), _rcp(fin_l[h]), F32_ZERO) + for h in range(N_MMA) + ] + if cutlass.const_expr(num_splits == 1): + optr = out.iterator.raw_ptr() + obase = row * cutlass.Int32(num_heads * D_NOPE) + for s in cutlass.range_constexpr(NUM_STRIPES): + tp = cutlass.inttoptr(my_addr + TMEM_O + s * N_MMA, 6, cutlass.Float32) + ov = prims.tcgen05_ld("32x32b", tp, num=N_MMA) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + for h2 in cutlass.range_constexpr(num_heads // 2): + pk = _bf16x2( + _mul_rn(ov[2 * h2], osc[2 * h2]), _mul_rn(ov[2 * h2 + 1], osc[2 * h2 + 1]) + ) + for e in cutlass.range_constexpr(2): + h: cutlass.Constexpr[int] = 2 * h2 + e + optr[obase + h * D_NOPE + s * 128 + col] = cutlass.Int16( + (pk >> cutlass.Int32(16 * e)) & cutlass.Int32(0xFFFF) + ) + else: + # fp32 partial O / l into the (now idle) KV tile, lse = lg2(l) + max (FlashMLA). + part = sKV + for s in cutlass.range_constexpr(NUM_STRIPES): + tp = cutlass.inttoptr(my_addr + TMEM_O + s * N_MMA, 6, cutlass.Float32) + ov = prims.tcgen05_ld("32x32b", tp, num=N_MMA) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + for h in cutlass.range_constexpr(num_heads): + value = _mul_rn(ov[h], osc[h]) + if cutlass.const_expr(global_partials): + offset = ( + (cutlass.Int64(row) * cutlass.Int64(num_splits) + cutlass.Int64(split)) + * cutlass.Int64(num_heads) + + cutlass.Int64(h) + ) * cutlass.Int64(516) + out.iterator.raw_ptr()[offset + cutlass.Int32(s * 128) + col] = value + else: + part[h * D_NOPE + s * 128 + col] = value + if cw == cutlass.Int32(0) and lane < cutlass.Int32(1): + for h in cutlass.range_constexpr(N_MMA): + lse = _add_ftz(_lg2(fin_l[h]), fin_m[h]) + if cutlass.const_expr(global_partials): + if cutlass.const_expr(h < num_heads): + offset = ( + ( + cutlass.Int64(row) * cutlass.Int64(num_splits) + + cutlass.Int64(split) + ) + * cutlass.Int64(num_heads) + + cutlass.Int64(h) + ) * cutlass.Int64(516) + out.iterator.raw_ptr()[offset + cutlass.Int32(512)] = lse + else: + sLse[h] = lse + + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + prims.barrier_cta_sync(0) + if warp == W_PV: + prims.tcgen05_dealloc( + cutlass.inttoptr(tmem_raw, 6, cutlass.Float32), TMEM_COLS, group="cta_1" + ) + if cutlass.const_expr(global_partials): + prims.griddepcontrol(prims.GridDepAction.LAUNCH_DEPENDENTS) + if cutlass.const_expr(num_splits > 1 and not global_partials): + # FlashMLA's combine (smxx/decode/combine/combine.cu) over the cluster's partials: + # each CTA merges 1/num_splits of the [heads x 512] output. + prims.barrier_cluster_arrive() + prims.barrier_cluster_wait() + total4: cutlass.Constexpr[int] = num_heads * D_NOPE // 4 + per_cta4: cutlass.Constexpr[int] = total4 // num_splits + iters: cutlass.Constexpr[int] = (per_cta4 + THREADS - 1) // THREADS + part_local = sKV.data_ptr() + for it in cutlass.range_constexpr(iters): + v4 = split * cutlass.Int32(per_cta4) + tidx + cutlass.Int32(it * THREADS) + if tidx + cutlass.Int32(it * THREADS) < cutlass.Int32(per_cta4): + h = v4 // cutlass.Int32(D_NOPE // 4) + d0 = (v4 % cutlass.Int32(D_NOPE // 4)) * cutlass.Int32(4) + lses = [] + for p in cutlass.range_constexpr(num_splits): + peer_lse = prims.mapa(sLse.data_ptr(), cutlass.Int32(p)) + lses.append(cutlass.Float32((peer_lse + h).load())) + max_lse = lses[0] + for p in cutlass.range_constexpr(1, num_splits): + max_lse = _fmax(max_lse, lses[p]) + max_lse = _sel_lit(_cmp_i32("ne", max_lse, F32_NEG_INF), max_lse, F32_ZERO) + # combine.cu sums over lanes with a xor butterfly (offsets 16 .. 1; lanes past + # the splits hold 0): the same adds as halving the list pairwise. + sum_lse = _butterfly_sum( + [_ex2(_sub_ftz(lses[p], max_lse)) for p in range(num_splits)] + ) + global_lse = _sel_lit( + _cmp_i32("ne", sum_lse, F32_ZERO), _add_ftz(_lg2(sum_lse), max_lse), F32_POS_INF + ) + acc = [] + for p in cutlass.range_constexpr(num_splits): + w = _ex2(_sub_ftz(lses[p], global_lse)) + peer_o = prims.mapa(part_local, cutlass.Int32(p)) + o4 = (peer_o + h * cutlass.Int32(D_NOPE) + d0).load(count=4, alignment=16) + if cutlass.const_expr(p == 0): + # FlashMLA: result = 0; result += w * o (an fma onto 0 is the product). + acc = [_mul_into_zero(w, o4[e]) for e in range(4)] + else: + acc = [_fma_ftz(w, o4[e], acc[e]) for e in range(4)] + lo = _bf16x2(acc[0], acc[1]) + hi = _bf16x2(acc[2], acc[3]) + ob = (row * cutlass.Int32(num_heads) + h) * cutlass.Int32(D_NOPE) + d0 + m16 = cutlass.Int32(0xFFFF) + halves = ( + lo & m16, + (lo >> cutlass.Int32(16)) & m16, + hi & m16, + (hi >> cutlass.Int32(16)) & m16, + ) + (out.iterator.raw_ptr() + ob).store( + cutlass.Vector.from_elements( + tuple(cutlass.Int16(x) for x in halves), dtype=cutlass.Int16 + ), + alignment=8, + ) + prims.barrier_cluster_arrive() + prims.barrier_cluster_wait() + prims.griddepcontrol(prims.GridDepAction.LAUNCH_DEPENDENTS) + + +@cute.jit +def _launch( + q: cute.Tensor, + pool: cute.Tensor, + indices: cute.Tensor, + out: cute.Tensor, + scale_log2: cutlass.Float32, + stream: cuda_driver.CUstream, + num_heads: cutlass.Constexpr[int], + num_splits: cutlass.Constexpr[int], + global_partials: cutlass.Constexpr[bool] = False, +) -> None: + rows = indices.shape[0] + h_q = q.shape[1] + # The query: the first `num_heads` of `h_q` heads, 64 dims per box (SW128). + tma_q = cuda.create_tensor_map_tiled( + global_address=q.iterator.toint(), + dtype=cutlass.BFloat16, + global_dims=[D_QK, h_q, rows], + global_strides=[D_QK * 2 // 16, h_q * (D_QK * 2 // 16)], + box_dims=[64, num_heads, 1], + swizzle=cuda.TensorMapSwizzle.s128b, + ) + # Whole pool rows, 82 x int64 a box; rows past the pool (the -1 slots) read as zeros. + tma_raw = cuda.create_tensor_map_tiled( + global_address=pool.iterator.toint(), + dtype=cutlass.Int64, + global_dims=[ROW_BYTES // 8, pool.shape[0]], + global_strides=[ROW_BYTES // 16], + box_dims=[ROW_BYTES // 8, 1], + swizzle=cuda.TensorMapSwizzle.none, + ) + cluster_splits: cutlass.Constexpr[int] = 1 if global_partials else num_splits + _dsmla_decode_kernel( + tma_q, tma_raw, indices, out, scale_log2, num_heads, num_splits, global_partials + ).launch( + grid=(num_splits, 1, rows), + block=(THREADS, 1, 1), + cluster=(cluster_splits, 1, 1), + stream=stream, + min_blocks_per_mp=1, + use_pdl=True, + ) + + +_COMPILED: dict[tuple[int, int, int], Callable] = {} + + +def compile_variant(num_heads: int, num_splits: int, device_index: int) -> Callable: + """The launcher for `num_heads` real heads and `num_splits` CTAs per row, compiled once per + process and device. Rows, the query's head count and the pool's row count are runtime.""" + key = (device_index, num_heads, num_splits) + if key not in _COMPILED: + if num_heads not in SUPPORTED_HEADS or num_splits not in SUPPORTED_SPLITS: + raise ValueError(f"no variant for {num_heads} heads and {num_splits} splits") + make = cute.runtime.make_fake_compact_tensor + rows, h_q, n_rows = cute.sym_int64(), cute.sym_int64(), cute.sym_int64() + cutlass.cuda.initialize_cuda_context(device_index) + _COMPILED[key] = cute.compile( + _launch, + make(cutlass.BFloat16, (rows, h_q, D_QK), stride_order=(2, 1, 0), assumed_align=16), + make(cutlass.Uint8, (n_rows, ROW_BYTES), stride_order=(1, 0), assumed_align=16), + make(cutlass.Int32, (rows, TOPK), stride_order=(1, 0), assumed_align=16), + make( + cutlass.Int16, (rows, num_heads, D_NOPE), stride_order=(2, 1, 0), assumed_align=16 + ), + cutlass.Float32(0.0), + cute.runtime.make_fake_stream(use_tvm_ffi_env_stream=False), + num_heads, + num_splits, + options="--enable-tvm-ffi --opt-level 2", + ) + return _COMPILED[key] + + +def pick_splits(batch: int, num_sms: int) -> int: + """CTAs per row (the cluster size) for a batch of `batch` rows. Measured on GB300 + at 8.7K and 32K contexts: an 8-CTA cluster stops paying past ~15 rows, a 4-CTA one past ~38, + two past ~76; the bounds scale with the SM count.""" + per = num_sms / 152.0 + if batch <= 15 * per: + return 8 + if batch <= 38 * per: + return 4 + if batch <= 76 * per: + return 2 + return 1 + + +def launch( + fn: Callable, + q: torch.Tensor, + kv_cache: torch.Tensor, + indices: torch.Tensor, + out: torch.Tensor, + softmax_scale: float, +) -> torch.Tensor: + """Run a compiled variant on the current stream; `out` written and returned.""" + pool = kv_cache.view(torch.uint8).view(-1, ROW_BYTES) + stream = cuda_driver.CUstream(torch.cuda.current_stream(q.device).cuda_stream) + fn(q, pool, indices, out.view(torch.int16), scale_log2_of(softmax_scale), stream) + return out + + +@cute.kernel +def _global_combine_kernel( + partials: cute.Tensor, + out: cute.Tensor, + num_heads: cutlass.Constexpr[int], + num_splits: cutlass.Constexpr[int], +) -> None: + """Merge independent CTA partials with the cluster combine's FP32 arithmetic.""" + tidx, _, _ = cute.arch.thread_idx() + row, head, _ = cute.arch.block_idx() + prims.griddepcontrol(prims.GridDepAction.WAIT) + ptr = partials.iterator.raw_ptr() + base = ( + cutlass.Int64(row) * cutlass.Int64(num_splits * num_heads) + cutlass.Int64(head) + ) * cutlass.Int64(516) + lses = [ + cutlass.Float32(ptr[base + cutlass.Int32(p * num_heads * 516 + 512)]) + for p in range(num_splits) + ] + max_lse = lses[0] + for p in cutlass.range_constexpr(1, num_splits): + max_lse = _fmax(max_lse, lses[p]) + max_lse = _sel_lit(_cmp_i32("ne", max_lse, F32_NEG_INF), max_lse, F32_ZERO) + sum_lse = _butterfly_sum([_ex2(_sub_ftz(lses[p], max_lse)) for p in range(num_splits)]) + global_lse = _sel_lit( + _cmp_i32("ne", sum_lse, F32_ZERO), _add_ftz(_lg2(sum_lse), max_lse), F32_POS_INF + ) + acc = [] + for p in cutlass.range_constexpr(num_splits): + weight = _ex2(_sub_ftz(lses[p], global_lse)) + offset = base + cutlass.Int32(p * num_heads * 516) + tidx * cutlass.Int32(4) + o4 = (ptr + offset).load(count=4, alignment=16) + if cutlass.const_expr(p == 0): + acc = [_mul_into_zero(weight, o4[e]) for e in range(4)] + else: + acc = [_fma_ftz(weight, o4[e], acc[e]) for e in range(4)] + lo, hi = _bf16x2(acc[0], acc[1]), _bf16x2(acc[2], acc[3]) + mask = cutlass.Int32(0xFFFF) + halves = ( + lo & mask, + (lo >> cutlass.Int32(16)) & mask, + hi & mask, + (hi >> cutlass.Int32(16)) & mask, + ) + offset = (cutlass.Int64(row) * cutlass.Int64(num_heads) + cutlass.Int64(head)) * cutlass.Int64( + D_NOPE + ) + cutlass.Int64(tidx * 4) + (out.iterator.raw_ptr() + offset).store( + cutlass.Vector.from_elements(tuple(cutlass.Int16(x) for x in halves), dtype=cutlass.Int16), + alignment=8, + ) + prims.barrier_cta_sync(0) + prims.griddepcontrol(prims.GridDepAction.LAUNCH_DEPENDENTS) + + +@cute.jit +def _launch_global_combine( + partials: cute.Tensor, + out: cute.Tensor, + stream: cuda_driver.CUstream, + num_heads: cutlass.Constexpr[int], + num_splits: cutlass.Constexpr[int], +) -> None: + _global_combine_kernel(partials, out, num_heads, num_splits).launch( + grid=(out.shape[0], num_heads, 1), + block=(128, 1, 1), + stream=stream, + use_pdl=True, + ) + + +_GLOBAL_COMPILED: dict[tuple[int, int, int], tuple[Callable, Callable]] = {} + + +def _compile_global_variant( + num_heads: int, num_splits: int, device_index: int +) -> tuple[Callable, Callable]: + """Compile independent split CTAs and a separate FP32 combine launch.""" + if num_heads not in SUPPORTED_HEADS or num_splits not in (8, 16): + raise ValueError(f"no global variant for {num_heads} heads and {num_splits} splits") + key = (device_index, num_heads, num_splits) + if key not in _GLOBAL_COMPILED: + make = cute.runtime.make_fake_compact_tensor + rows, h_q, n_rows = cute.sym_int64(), cute.sym_int64(), cute.sym_int64() + cutlass.cuda.initialize_cuda_context(device_index) + workspace = make( + cutlass.Float32, + (rows, num_splits, num_heads, 516), + stride_order=(3, 2, 1, 0), + assumed_align=16, + ) + output = make( + cutlass.Int16, (rows, num_heads, D_NOPE), stride_order=(2, 1, 0), assumed_align=16 + ) + stream = cute.runtime.make_fake_stream(use_tvm_ffi_env_stream=False) + partial = cute.compile( + _launch, + make(cutlass.BFloat16, (rows, h_q, D_QK), stride_order=(2, 1, 0), assumed_align=16), + make(cutlass.Uint8, (n_rows, ROW_BYTES), stride_order=(1, 0), assumed_align=16), + make(cutlass.Int32, (rows, TOPK), stride_order=(1, 0), assumed_align=16), + workspace, + cutlass.Float32(0.0), + stream, + num_heads, + num_splits, + True, + options="--enable-tvm-ffi --opt-level 2", + ) + combine = cute.compile( + _launch_global_combine, + workspace, + output, + stream, + num_heads, + num_splits, + options="--enable-tvm-ffi --opt-level 2", + ) + _GLOBAL_COMPILED[key] = (partial, combine) + return _GLOBAL_COMPILED[key] + + +def _launch_global( + fns: tuple[Callable, Callable], + q: torch.Tensor, + kv_cache: torch.Tensor, + indices: torch.Tensor, + out: torch.Tensor, + softmax_scale: float, + workspace: torch.Tensor, +) -> torch.Tensor: + """Write caller-owned workspace and output on the current stream in two launches. + + Workspace is FP32 [rows, splits, heads, 516]. The caller owns its lifetime and + must supply distinct workspaces to concurrently executing streams or graphs. + """ + stream = cuda_driver.CUstream(torch.cuda.current_stream(q.device).cuda_stream) + pool = kv_cache.view(torch.uint8).view(-1, ROW_BYTES) + partial, combine = fns + partial(q, pool, indices, workspace, scale_log2_of(softmax_scale), stream) + combine(workspace, out.view(torch.int16), stream) + return out diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/kernels/_indexer_k_cache_update_kernel.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/kernels/_indexer_k_cache_update_kernel.py new file mode 100644 index 000000000000..b81543b6c122 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/kernels/_indexer_k_cache_update_kernel.py @@ -0,0 +1,1044 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Fused DSA indexer K-cache update, a CuTe DSL kernel. + +The backend sequence this kernel replaces is: + + indexer_k BF16 [T, 128] + -> LayerNorm over the indexer-K row + -> RoPE on the first 64 K lanes + -> TRT-LLM fused FP8 cat/scale + -> TRT-LLM indexer K-cache scatter + -> final uint8 K cache [num_blocks, block_size, 1, 132] + +The fused kernel writes the final K-cache bytes directly. It intentionally +matches the backend's materialized BF16 precision boundaries before RoPE and +before FP8 quantization, so correctness is byte-for-byte equality of the final +cache image. +""" + +from collections.abc import Callable +from typing import TYPE_CHECKING + +import cuda.bindings.driver as cuda +import cutlass +import cutlass.cute as cute +from cutlass.cute.runtime import make_fake_compact_tensor, make_fake_stream, make_fake_tensor + +if TYPE_CHECKING: + import torch + + +_FUSED_CAT_WARP_SIZE = 32 +_FUSED_CAT_PE_DIM = 64 +_FUSED_CAT_NOPE_DIM = 64 +_FUSED_CAT_HEAD_DIM = _FUSED_CAT_PE_DIM + _FUSED_CAT_NOPE_DIM +_FUSED_CAT_ELEMS_PER_THREAD = 4 +_FUSED_CAT_FP8_MAX = 448.0 +_FUSED_CAT_INV_FP8_MAX = 1.0 / _FUSED_CAT_FP8_MAX +_FUSED_CAT_MIN_AMAX = 1.0e-12 +_FUSED_CAT_SCATTER_ROWS_PER_BLOCK = 8 +_FUSED_CAT_SCATTER_ALLOWED_ROWS_PER_BLOCK = (1, 2, 4, 8, 16) +_FUSED_CAT_THREADS_PER_ROW = 16 +_SMALL_TOKEN_THRESHOLD = 1024 +_SMALL_TOKEN_ROWS_PER_BLOCK = 2 +_INDEXER_K_NORM_EPS = 1.0e-6 +_INDEXER_K_CACHE_UPDATE_COMPILE_CACHE: dict[ + tuple[int, bool, bool, int, bool, bool, bool], Callable +] = {} + + +@cute.jit +def _indexer_k_norm_value_from_registers( + x: cutlass.Float32, + weight: cutlass.Float32, + bias: cutlass.Float32, + mean: cutlass.Float32, + inv_std: cutlass.Float32, +) -> cutlass.Float32: + """Apply the materialized-BF16 LayerNorm boundary to register values.""" + out = (x - mean) * inv_std * weight + bias + return out.to(cutlass.BFloat16).to(cutlass.Float32) + + +@cute.jit +def _subwarp16_reduction_sum(value: cutlass.Float32) -> cutlass.Float32: + """Reduce independently within both 16-lane halves of a physical warp.""" + active_mask = cute.arch.activemask() + value = value + cute.arch.shuffle_sync_bfly(value, 8, mask=active_mask) + value = value + cute.arch.shuffle_sync_bfly(value, 4, mask=active_mask) + value = value + cute.arch.shuffle_sync_bfly(value, 2, mask=active_mask) + return value + cute.arch.shuffle_sync_bfly(value, 1, mask=active_mask) + + +@cute.jit +def _subwarp16_reduction_max(value: cutlass.Float32) -> cutlass.Float32: + """Find the maximum independently within both physical subwarps.""" + active_mask = cute.arch.activemask() + value = cutlass.max( + value, + cute.arch.shuffle_sync_bfly(value, 8, mask=active_mask), + ) + value = cutlass.max( + value, + cute.arch.shuffle_sync_bfly(value, 4, mask=active_mask), + ) + value = cutlass.max( + value, + cute.arch.shuffle_sync_bfly(value, 2, mask=active_mask), + ) + return cutlass.max( + value, + cute.arch.shuffle_sync_bfly(value, 1, mask=active_mask), + ) + + +@cute.jit +def _quantize_pack_fp8x4( + x0: cutlass.Float32, + x1: cutlass.Float32, + x2: cutlass.Float32, + x3: cutlass.Float32, + inv_scale: cutlass.Float32, +) -> cutlass.Int32: + """Quantize four register values and pack the FP8 bytes into one word.""" + q0 = (x0 * inv_scale).to(cutlass.Float8E4M3FN).bitcast(cutlass.Int8) + q1 = (x1 * inv_scale).to(cutlass.Float8E4M3FN).bitcast(cutlass.Int8) + q2 = (x2 * inv_scale).to(cutlass.Float8E4M3FN).bitcast(cutlass.Int8) + q3 = (x3 * inv_scale).to(cutlass.Float8E4M3FN).bitcast(cutlass.Int8) + return ( + (cutlass.Int32(q0) & cutlass.Int32(0xFF)) + | ((cutlass.Int32(q1) & cutlass.Int32(0xFF)) << 8) + | ((cutlass.Int32(q2) & cutlass.Int32(0xFF)) << 16) + | ((cutlass.Int32(q3) & cutlass.Int32(0xFF)) << 24) + ) + + +@cute.jit +def _unpack_bf16x4( + packed: cutlass.Uint64, +) -> tuple[ + cutlass.Float32, + cutlass.Float32, + cutlass.Float32, + cutlass.Float32, +]: + """Unpack one naturally aligned 64-bit input word into four FP32 values.""" + x0 = cutlass.Uint16(packed & cutlass.Uint64(0xFFFF)) + x1 = cutlass.Uint16((packed >> 16) & cutlass.Uint64(0xFFFF)) + x2 = cutlass.Uint16((packed >> 32) & cutlass.Uint64(0xFFFF)) + x3 = cutlass.Uint16(packed >> 48) + return ( + x0.bitcast(cutlass.BFloat16).to(cutlass.Float32), + x1.bitcast(cutlass.BFloat16).to(cutlass.Float32), + x2.bitcast(cutlass.BFloat16).to(cutlass.Float32), + x3.bitcast(cutlass.BFloat16).to(cutlass.Float32), + ) + + +@cute.kernel +def self_attn_indexer_k_norm_rope_fp8_cache_update( + indexer_k: cute.Tensor, + norm_weight: cute.Tensor, + norm_bias: cute.Tensor, + position_ids: cute.Tensor, + rotary_cos_sin: cute.Tensor, + k_cache: cute.Tensor, + slot_mapping_fp8: cute.Tensor, + slot_mapping_scale: cute.Tensor, + num_tokens: cutlass.Int32, + cache_dim_0: cutlass.Int32, + cache_dim_1: cutlass.Int32, + cache_dim_2: cutlass.Int32, + cache_dim_3: cutlass.Int32, + cache_stride_0: cutlass.Int64, + cache_stride_1: cutlass.Int64, + cache_stride_2: cutlass.Int64, + cache_stride_3: cutlass.Int64, + USE_UE8M0: cutlass.Constexpr[bool], + IS_NEOX: cutlass.Constexpr[bool], + ROWS_PER_BLOCK: cutlass.Constexpr[int], + VECTORIZED_LOADS: cutlass.Constexpr[bool], + CACHE_CONTIGUOUS: cutlass.Constexpr[bool], + USE_PDL: cutlass.Constexpr[bool], +) -> None: + block_row, _, _ = cute.arch.block_idx() + tidx, _, _ = cute.arch.thread_idx() + warp_in_block = tidx // _FUSED_CAT_WARP_SIZE + lane_in_warp = tidx % _FUSED_CAT_WARP_SIZE + lane_in_row = lane_in_warp % cutlass.Int32(_FUSED_CAT_THREADS_PER_ROW) + row_in_warp = lane_in_warp // cutlass.Int32(_FUSED_CAT_THREADS_PER_ROW) + row_in_block = warp_in_block * cutlass.Int32(2) + row_in_warp + token_idx = block_row * ROWS_PER_BLOCK + row_in_block + + if cutlass.const_expr(USE_PDL): + # Every thread waits before it can issue an indexer-K load. This avoids + # paying an additional CTA barrier on the latency-sensitive path. + cute.arch.griddepcontrol_wait() + + if row_in_block < ROWS_PER_BLOCK and token_idx < num_tokens: + input_ptr = indexer_k.iterator + weight_ptr = norm_weight.iterator + bias_ptr = norm_bias.iterator + row_base = token_idx * _FUSED_CAT_HEAD_DIM + + # Physical lane i owns dimensions 4*i..4*i+3 and + # 64+4*i..64+4*i+3. + base_col = lane_in_row * _FUSED_CAT_ELEMS_PER_THREAD + second_base_col = base_col + cutlass.Int32(_FUSED_CAT_HEAD_DIM // 2) + if cutlass.const_expr(VECTORIZED_LOADS): + input_word_ptr = cute.recast_ptr( + (input_ptr + row_base + base_col).align(8), + dtype=cutlass.Uint64, + ) + x0, x1, x2, x3 = _unpack_bf16x4(input_word_ptr.load()) + else: + x0 = (input_ptr + row_base + base_col).load().to(cutlass.Float32) + x1 = (input_ptr + row_base + base_col + cutlass.Int32(1)).load().to(cutlass.Float32) + x2 = (input_ptr + row_base + base_col + cutlass.Int32(2)).load().to(cutlass.Float32) + x3 = (input_ptr + row_base + base_col + cutlass.Int32(3)).load().to(cutlass.Float32) + + if cutlass.const_expr(VECTORIZED_LOADS): + input_word_ptr = cute.recast_ptr( + (input_ptr + row_base + second_base_col).align(8), + dtype=cutlass.Uint64, + ) + y0, y1, y2, y3 = _unpack_bf16x4(input_word_ptr.load()) + else: + y0 = (input_ptr + row_base + second_base_col).load().to(cutlass.Float32) + y1 = ( + (input_ptr + row_base + second_base_col + cutlass.Int32(1)) + .load() + .to(cutlass.Float32) + ) + y2 = ( + (input_ptr + row_base + second_base_col + cutlass.Int32(2)) + .load() + .to(cutlass.Float32) + ) + y3 = ( + (input_ptr + row_base + second_base_col + cutlass.Int32(3)) + .load() + .to(cutlass.Float32) + ) + + local_sum = cutlass.Float32(0.0) + local_sum = local_sum + x0 + local_sum = local_sum + x1 + local_sum = local_sum + x2 + local_sum = local_sum + x3 + second_sum = cutlass.Float32(0.0) + second_sum = second_sum + y0 + second_sum = second_sum + y1 + second_sum = second_sum + y2 + second_sum = second_sum + y3 + local_sum = local_sum + second_sum + row_sum = _subwarp16_reduction_sum(local_sum) + mean = row_sum * cutlass.Float32(1.0 / _FUSED_CAT_HEAD_DIM) + + local_sq = cutlass.Float32(0.0) + centered = x0 - mean + local_sq = local_sq + centered * centered + centered = x1 - mean + local_sq = local_sq + centered * centered + centered = x2 - mean + local_sq = local_sq + centered * centered + centered = x3 - mean + local_sq = local_sq + centered * centered + second_sq = cutlass.Float32(0.0) + centered = y0 - mean + second_sq = second_sq + centered * centered + centered = y1 - mean + second_sq = second_sq + centered * centered + centered = y2 - mean + second_sq = second_sq + centered * centered + centered = y3 - mean + second_sq = second_sq + centered * centered + local_sq = local_sq + second_sq + row_sq = _subwarp16_reduction_sum(local_sq) + variance = row_sq * cutlass.Float32(1.0 / _FUSED_CAT_HEAD_DIM) + inv_std = cute.math.rsqrt( + variance + cutlass.Float32(_INDEXER_K_NORM_EPS), + fastmath=True, + ) + + weight0 = (weight_ptr + base_col).load().to(cutlass.Float32) + weight1 = (weight_ptr + base_col + cutlass.Int32(1)).load().to(cutlass.Float32) + weight2 = (weight_ptr + base_col + cutlass.Int32(2)).load().to(cutlass.Float32) + weight3 = (weight_ptr + base_col + cutlass.Int32(3)).load().to(cutlass.Float32) + bias0 = (bias_ptr + base_col).load().to(cutlass.Float32) + bias1 = (bias_ptr + base_col + cutlass.Int32(1)).load().to(cutlass.Float32) + bias2 = (bias_ptr + base_col + cutlass.Int32(2)).load().to(cutlass.Float32) + bias3 = (bias_ptr + base_col + cutlass.Int32(3)).load().to(cutlass.Float32) + x0 = _indexer_k_norm_value_from_registers( + x0, + weight0, + bias0, + mean, + inv_std, + ) + x1 = _indexer_k_norm_value_from_registers( + x1, + weight1, + bias1, + mean, + inv_std, + ) + x2 = _indexer_k_norm_value_from_registers( + x2, + weight2, + bias2, + mean, + inv_std, + ) + x3 = _indexer_k_norm_value_from_registers( + x3, + weight3, + bias3, + mean, + inv_std, + ) + weight0 = (weight_ptr + second_base_col).load().to(cutlass.Float32) + weight1 = (weight_ptr + second_base_col + cutlass.Int32(1)).load().to(cutlass.Float32) + weight2 = (weight_ptr + second_base_col + cutlass.Int32(2)).load().to(cutlass.Float32) + weight3 = (weight_ptr + second_base_col + cutlass.Int32(3)).load().to(cutlass.Float32) + bias0 = (bias_ptr + second_base_col).load().to(cutlass.Float32) + bias1 = (bias_ptr + second_base_col + cutlass.Int32(1)).load().to(cutlass.Float32) + bias2 = (bias_ptr + second_base_col + cutlass.Int32(2)).load().to(cutlass.Float32) + bias3 = (bias_ptr + second_base_col + cutlass.Int32(3)).load().to(cutlass.Float32) + y0 = _indexer_k_norm_value_from_registers( + y0, + weight0, + bias0, + mean, + inv_std, + ) + y1 = _indexer_k_norm_value_from_registers( + y1, + weight1, + bias1, + mean, + inv_std, + ) + y2 = _indexer_k_norm_value_from_registers( + y2, + weight2, + bias2, + mean, + inv_std, + ) + y3 = _indexer_k_norm_value_from_registers( + y3, + weight3, + bias3, + mean, + inv_std, + ) + + position = (position_ids.iterator + token_idx).load() + table_base = cutlass.Int64(position) * cutlass.Int64(_FUSED_CAT_PE_DIM) + cos_sin_ptr = rotary_cos_sin.iterator + if cutlass.const_expr(IS_NEOX): + # The NeoX partner is 32 dimensions away, or eight lanes away for + # this four-elements-per-lane ownership. Execute the shuffle in + # the full warp before the PE-lane predicate. + source_lane = row_in_warp * cutlass.Int32(16) + (lane_in_row ^ cutlass.Int32(8)) + active_mask = cute.arch.activemask() + partner0 = cute.arch.shuffle_sync(x0, source_lane, mask=active_mask) + partner1 = cute.arch.shuffle_sync(x1, source_lane, mask=active_mask) + partner2 = cute.arch.shuffle_sync(x2, source_lane, mask=active_mask) + partner3 = cute.arch.shuffle_sync(x3, source_lane, mask=active_mask) + if base_col < _FUSED_CAT_PE_DIM: + table_dim = base_col + if table_dim >= cutlass.Int32(_FUSED_CAT_PE_DIM // 2): + table_dim = table_dim - cutlass.Int32(_FUSED_CAT_PE_DIM // 2) + cos0 = ( + (cos_sin_ptr + table_base + cutlass.Int64(table_dim)).load().to(cutlass.Float32) + ) + cos1 = ( + (cos_sin_ptr + table_base + cutlass.Int64(table_dim + 1)) + .load() + .to(cutlass.Float32) + ) + cos2 = ( + (cos_sin_ptr + table_base + cutlass.Int64(table_dim + 2)) + .load() + .to(cutlass.Float32) + ) + cos3 = ( + (cos_sin_ptr + table_base + cutlass.Int64(table_dim + 3)) + .load() + .to(cutlass.Float32) + ) + sin_base = table_base + cutlass.Int64(_FUSED_CAT_PE_DIM // 2) + sin0 = ( + (cos_sin_ptr + sin_base + cutlass.Int64(table_dim)).load().to(cutlass.Float32) + ) + sin1 = ( + (cos_sin_ptr + sin_base + cutlass.Int64(table_dim + 1)) + .load() + .to(cutlass.Float32) + ) + sin2 = ( + (cos_sin_ptr + sin_base + cutlass.Int64(table_dim + 2)) + .load() + .to(cutlass.Float32) + ) + sin3 = ( + (cos_sin_ptr + sin_base + cutlass.Int64(table_dim + 3)) + .load() + .to(cutlass.Float32) + ) + out0 = x0 * cos0 + partner0 * sin0 + out1 = x1 * cos1 + partner1 * sin1 + out2 = x2 * cos2 + partner2 * sin2 + out3 = x3 * cos3 + partner3 * sin3 + if base_col < cutlass.Int32(_FUSED_CAT_PE_DIM // 2): + out0 = x0 * cos0 - partner0 * sin0 + out1 = x1 * cos1 - partner1 * sin1 + out2 = x2 * cos2 - partner2 * sin2 + out3 = x3 * cos3 - partner3 * sin3 + x0 = out0.to(cutlass.BFloat16).to(cutlass.Float32) + x1 = out1.to(cutlass.BFloat16).to(cutlass.Float32) + x2 = out2.to(cutlass.BFloat16).to(cutlass.Float32) + x3 = out3.to(cutlass.BFloat16).to(cutlass.Float32) + else: + if base_col < _FUSED_CAT_PE_DIM: + table_dim0 = base_col // cutlass.Int32(2) + table_dim1 = table_dim0 + cutlass.Int32(1) + cos0 = ( + (cos_sin_ptr + table_base + cutlass.Int64(table_dim0)) + .load() + .to(cutlass.Float32) + ) + cos1 = ( + (cos_sin_ptr + table_base + cutlass.Int64(table_dim1)) + .load() + .to(cutlass.Float32) + ) + sin_base = table_base + cutlass.Int64(_FUSED_CAT_PE_DIM // 2) + sin0 = ( + (cos_sin_ptr + sin_base + cutlass.Int64(table_dim0)).load().to(cutlass.Float32) + ) + sin1 = ( + (cos_sin_ptr + sin_base + cutlass.Int64(table_dim1)).load().to(cutlass.Float32) + ) + out0 = x0 * cos0 - x1 * sin0 + out1 = x1 * cos0 + x0 * sin0 + out2 = x2 * cos1 - x3 * sin1 + out3 = x3 * cos1 + x2 * sin1 + x0 = out0.to(cutlass.BFloat16).to(cutlass.Float32) + x1 = out1.to(cutlass.BFloat16).to(cutlass.Float32) + x2 = out2.to(cutlass.BFloat16).to(cutlass.Float32) + x3 = out3.to(cutlass.BFloat16).to(cutlass.Float32) + + ax0 = cute.math.absf(x0) + ax1 = cute.math.absf(x1) + ax2 = cute.math.absf(x2) + ax3 = cute.math.absf(x3) + local_max = ax0 + if ax1 > local_max: + local_max = ax1 + if ax2 > local_max: + local_max = ax2 + if ax3 > local_max: + local_max = ax3 + + ay0 = cute.math.absf(y0) + ay1 = cute.math.absf(y1) + ay2 = cute.math.absf(y2) + ay3 = cute.math.absf(y3) + if ay0 > local_max: + local_max = ay0 + if ay1 > local_max: + local_max = ay1 + if ay2 > local_max: + local_max = ay2 + if ay3 > local_max: + local_max = ay3 + amax = _subwarp16_reduction_max(local_max) + safe_amax = amax + if safe_amax < cutlass.Float32(_FUSED_CAT_MIN_AMAX): + safe_amax = cutlass.Float32(_FUSED_CAT_MIN_AMAX) + + scale = safe_amax * cutlass.Float32(_FUSED_CAT_INV_FP8_MAX) + if cutlass.const_expr(USE_UE8M0): + scale_bits = scale.bitcast(cutlass.Int32) + mantissa = scale_bits & cutlass.Int32(0x007FFFFF) + exp_bits = scale_bits & cutlass.Int32(0x7F800000) + if mantissa != cutlass.Int32(0): + exp_bits += cutlass.Int32(0x00800000) + scale = exp_bits.bitcast(cutlass.Float32) + inv_scale_bits = cutlass.Int32(0x7F000000) - exp_bits + inv_scale = inv_scale_bits.bitcast(cutlass.Float32) + else: + inv_scale = cutlass.Float32(1.0) / scale + + packed = _quantize_pack_fp8x4(x0, x1, x2, x3, inv_scale) + second_packed = _quantize_pack_fp8x4(y0, y1, y2, y3, inv_scale) + + flat_idx_fp8_base = (slot_mapping_fp8.iterator + token_idx).load() + flat_idx_scale_base = (slot_mapping_scale.iterator + token_idx).load() + if flat_idx_fp8_base >= 0 and flat_idx_scale_base >= 0: + if cutlass.const_expr(CACHE_CONTIGUOUS): + dst_offset = flat_idx_fp8_base + cutlass.Int64(base_col) + else: + dst_offset = _indexer_scatter_flat_to_strided_offset( + flat_idx_fp8_base + cutlass.Int64(base_col), + cache_dim_0, + cache_dim_1, + cache_dim_2, + cache_dim_3, + cache_stride_0, + cache_stride_1, + cache_stride_2, + cache_stride_3, + ) + dst_word_ptr = cute.recast_ptr( + (k_cache.iterator + dst_offset).align(4), + dtype=cutlass.Int32, + ) + dst_word_ptr.store(packed) + + if cutlass.const_expr(CACHE_CONTIGUOUS): + second_dst_offset = flat_idx_fp8_base + cutlass.Int64(second_base_col) + else: + second_dst_offset = _indexer_scatter_flat_to_strided_offset( + flat_idx_fp8_base + cutlass.Int64(second_base_col), + cache_dim_0, + cache_dim_1, + cache_dim_2, + cache_dim_3, + cache_stride_0, + cache_stride_1, + cache_stride_2, + cache_stride_3, + ) + second_dst_word_ptr = cute.recast_ptr( + (k_cache.iterator + second_dst_offset).align(4), + dtype=cutlass.Int32, + ) + second_dst_word_ptr.store(second_packed) + + if lane_in_row == 0: + if cutlass.const_expr(CACHE_CONTIGUOUS): + dst_offset_scale = flat_idx_scale_base + else: + dst_offset_scale = _indexer_scatter_flat_to_strided_offset( + flat_idx_scale_base, + cache_dim_0, + cache_dim_1, + cache_dim_2, + cache_dim_3, + cache_stride_0, + cache_stride_1, + cache_stride_2, + cache_stride_3, + ) + dst_scale_word_ptr = cute.recast_ptr( + (k_cache.iterator + dst_offset_scale).align(4), + dtype=cutlass.Int32, + ) + dst_scale_word_ptr.store(scale.bitcast(cutlass.Int32)) + + # Do not explicitly launch dependents here. The final cache store is at the + # end of the kernel, so there is no useful producer tail to overlap. Kernel + # completion provides the required implicit PDL release without paying for + # an explicit griddepcontrol signal on this latency-sensitive path. + + +@cute.jit +def _indexer_scatter_flat_to_strided_offset( + flat_idx: cutlass.Int64, + cache_dim_0: cutlass.Int32, + cache_dim_1: cutlass.Int32, + cache_dim_2: cutlass.Int32, + cache_dim_3: cutlass.Int32, + cache_stride_0: cutlass.Int64, + cache_stride_1: cutlass.Int64, + cache_stride_2: cutlass.Int64, + cache_stride_3: cutlass.Int64, +) -> cutlass.Int64: + lower_dim_size = ( + cutlass.Int64(cache_dim_1) * cutlass.Int64(cache_dim_2) * cutlass.Int64(cache_dim_3) + ) + lower_dims_contiguous = ( + cache_stride_3 == cutlass.Int64(1) + and cache_stride_2 == cutlass.Int64(cache_dim_3) + and cache_stride_1 == cutlass.Int64(cache_dim_2) * cutlass.Int64(cache_dim_3) + ) + offset = cutlass.Int64(0) + if lower_dims_contiguous: + # TRT-LLM's per-layer view has a gap between blocks, but each block is + # contiguous. Collapse the lower three dimensions to avoid the full + # four-dimensional Int64 div/mod chain for every cache store. + i0 = flat_idx // lower_dim_size + intra_block = flat_idx - i0 * lower_dim_size + offset = i0 * cache_stride_0 + intra_block + else: + i3 = flat_idx % cutlass.Int64(cache_dim_3) + flat_idx = flat_idx // cutlass.Int64(cache_dim_3) + i2 = flat_idx % cutlass.Int64(cache_dim_2) + flat_idx = flat_idx // cutlass.Int64(cache_dim_2) + i1 = flat_idx % cutlass.Int64(cache_dim_1) + flat_idx = flat_idx // cutlass.Int64(cache_dim_1) + i0 = flat_idx % cutlass.Int64(cache_dim_0) + offset = ( + i0 * cache_stride_0 + i1 * cache_stride_1 + i2 * cache_stride_2 + i3 * cache_stride_3 + ) + return offset + + +@cute.jit +def _launch_indexer_k_norm_rope_fp8_cache_update( + indexer_k: cute.Tensor, + norm_weight: cute.Tensor, + norm_bias: cute.Tensor, + position_ids: cute.Tensor, + rotary_cos_sin: cute.Tensor, + k_cache: cute.Tensor, + slot_mapping_fp8: cute.Tensor, + slot_mapping_scale: cute.Tensor, + num_tokens: cutlass.Int32, + cache_stride_0: cutlass.Int64, + cache_stride_1: cutlass.Int64, + cache_stride_2: cutlass.Int64, + cache_stride_3: cutlass.Int64, + stream: cuda.CUstream, + USE_UE8M0: cutlass.Constexpr[bool], + IS_NEOX: cutlass.Constexpr[bool], + ROWS_PER_BLOCK: cutlass.Constexpr[int], + VECTORIZED_LOADS: cutlass.Constexpr[bool], + CACHE_CONTIGUOUS: cutlass.Constexpr[bool], + USE_PDL: cutlass.Constexpr[bool], +) -> None: + cache_dim_0 = cute.size(k_cache, mode=[0]) + cache_dim_1 = cute.size(k_cache, mode=[1]) + cache_dim_2 = cute.size(k_cache, mode=[2]) + cache_dim_3 = cute.size(k_cache, mode=[3]) + + self_attn_indexer_k_norm_rope_fp8_cache_update( + indexer_k, + norm_weight, + norm_bias, + position_ids, + rotary_cos_sin, + k_cache, + slot_mapping_fp8, + slot_mapping_scale, + num_tokens, + cache_dim_0, + cache_dim_1, + cache_dim_2, + cache_dim_3, + cache_stride_0, + cache_stride_1, + cache_stride_2, + cache_stride_3, + USE_UE8M0, + IS_NEOX, + ROWS_PER_BLOCK, + VECTORIZED_LOADS, + CACHE_CONTIGUOUS, + USE_PDL, + ).launch( + grid=(cute.ceil_div(num_tokens, ROWS_PER_BLOCK), 1, 1), + block=( + _FUSED_CAT_WARP_SIZE + * cute.ceil_div( + ROWS_PER_BLOCK * _FUSED_CAT_THREADS_PER_ROW, + _FUSED_CAT_WARP_SIZE, + ), + 1, + 1, + ), + stream=stream, + use_pdl=USE_PDL, + ) + + +def compile_indexer_k_norm_rope_fp8_cache_update( + device_id: int = 0, + use_ue8m0: bool = True, + is_neox: bool = True, + rows_per_block: int = _FUSED_CAT_SCATTER_ROWS_PER_BLOCK, + *, + vectorized_loads: bool = False, + cache_contiguous: bool = False, + use_pdl: bool = True, +) -> Callable: + """Compile the fused kernel for one launch specialization. + + ``cache_contiguous`` defaults to the generic-stride path so separately + compiled callables remain correct for every cache layout accepted by the + public wrapper. ``use_pdl`` defaults on for the production launch chain: + the kernel waits on its incoming dependency and relies on kernel completion + to release a following PDL consumer. Callers can disable it for standalone + use. The wrapper specializes these options automatically when it owns + compilation. + """ + if rows_per_block not in _FUSED_CAT_SCATTER_ALLOWED_ROWS_PER_BLOCK: + raise ValueError( + "rows_per_block must be one of " + f"{_FUSED_CAT_SCATTER_ALLOWED_ROWS_PER_BLOCK}, got {rows_per_block}" + ) + key = ( + device_id, + bool(use_ue8m0), + bool(is_neox), + rows_per_block, + bool(vectorized_loads), + bool(cache_contiguous), + bool(use_pdl), + ) + if key in _INDEXER_K_CACHE_UPDATE_COMPILE_CACHE: + return _INDEXER_K_CACHE_UPDATE_COMPILE_CACHE[key] + + cutlass.cuda.initialize_cuda_context(device_id) + + sym_num_tokens = cute.sym_int32() + sym_max_positions = cute.sym_int32() + sym_num_blocks = cute.sym_int32() + sym_block_size = cute.sym_int32() + sym_cache_token_bytes = cute.sym_int32() + sym_cache_stride_0 = cute.sym_int64(symbol="indexer_k_cache_update_stride_0") + sym_cache_stride_1 = cute.sym_int64(symbol="indexer_k_cache_update_stride_1") + sym_cache_stride_2 = cute.sym_int64(symbol="indexer_k_cache_update_stride_2") + + fake_indexer_k = make_fake_compact_tensor( + cutlass.BFloat16, + (sym_num_tokens, _FUSED_CAT_HEAD_DIM), + stride_order=(1, 0), + assumed_align=8 if vectorized_loads else 2, + ) + fake_norm_weight = make_fake_compact_tensor( + cutlass.Float32, + (_FUSED_CAT_HEAD_DIM,), + assumed_align=4, + ) + fake_norm_bias = make_fake_compact_tensor( + cutlass.Float32, + (_FUSED_CAT_HEAD_DIM,), + assumed_align=4, + ) + fake_position_ids = make_fake_compact_tensor( + cutlass.Int32, + (sym_num_tokens,), + assumed_align=4, + ) + fake_cos_sin = make_fake_compact_tensor( + cutlass.Float32, + (sym_max_positions, 2, _FUSED_CAT_PE_DIM // 2), + stride_order=(2, 1, 0), + assumed_align=4, + ) + fake_k_cache = make_fake_tensor( + cutlass.Uint8, + (sym_num_blocks, sym_block_size, 1, sym_cache_token_bytes), + stride=(sym_cache_stride_0, sym_cache_stride_1, sym_cache_stride_2, 1), + assumed_align=4, + ) + fake_slot_mapping_fp8 = make_fake_compact_tensor( + cutlass.Int64, + (sym_num_tokens,), + assumed_align=8, + ) + fake_slot_mapping_scale = make_fake_compact_tensor( + cutlass.Int64, + (sym_num_tokens,), + assumed_align=8, + ) + + compiled = cute.compile( + _launch_indexer_k_norm_rope_fp8_cache_update, + fake_indexer_k, + fake_norm_weight, + fake_norm_bias, + fake_position_ids, + fake_cos_sin, + fake_k_cache, + fake_slot_mapping_fp8, + fake_slot_mapping_scale, + cutlass.Int32(0), + cutlass.Int64(0), + cutlass.Int64(0), + cutlass.Int64(0), + cutlass.Int64(1), + make_fake_stream(), + bool(use_ue8m0), + bool(is_neox), + rows_per_block, + bool(vectorized_loads), + bool(cache_contiguous), + bool(use_pdl), + options="--enable-tvm-ffi", + ) + _INDEXER_K_CACHE_UPDATE_COMPILE_CACHE[key] = compiled + return compiled + + +def _current_cuda_stream(torch, device_id: int | None = None): + stream = torch.cuda.current_stream(device_id) + return cuda.CUstream(stream.cuda_stream) + + +def _require_cuda_contiguous(torch, tensor, name: str, dtype) -> None: + if tensor.dtype != dtype: + raise TypeError(f"{name} must be {dtype}, got {tensor.dtype}") + if not tensor.is_cuda: + raise TypeError(f"{name} must be a CUDA tensor") + if not tensor.is_contiguous(): + raise ValueError(f"{name} must be contiguous") + + +def _require_current_cuda_device( + torch, + tensors: tuple[tuple["torch.Tensor", str], ...], + current_device: int, +) -> None: + for tensor, name in tensors: + device_index = tensor.device.index + if device_index is None: + device_index = current_device + if int(device_index) != int(current_device): + raise ValueError(f"{name} must be on cuda:{current_device}, got {tensor.device}") + + +def _can_vectorize_loads(indexer_k) -> bool: + """Return whether the packed indexer-K input load is naturally aligned.""" + input_address = indexer_k.data_ptr() + return input_address % 8 == 0 + + +def _select_rows_per_block(num_tokens: int) -> int: + """Choose enough blocks to avoid underfilling small-token launches.""" + if num_tokens <= _SMALL_TOKEN_THRESHOLD: + return _SMALL_TOKEN_ROWS_PER_BLOCK + return _FUSED_CAT_SCATTER_ROWS_PER_BLOCK + + +def _has_any(torch, predicate) -> bool: + return bool(torch.any(predicate).item()) + + +def _require_indexer_k_cache_update_ranges( + torch, + position_ids, + rotary_cos_sin, + k_cache, + slot_mapping_fp8, + slot_mapping_scale, + num_tokens: int, +) -> None: + active_positions = position_ids[:num_tokens] + if _has_any( + torch, + (active_positions < 0) | (active_positions >= rotary_cos_sin.shape[0]), + ): + raise ValueError("position_ids must be in [0, rotary_cos_sin.shape[0])") + + active_fp8 = slot_mapping_fp8[:num_tokens] + active_scale = slot_mapping_scale[:num_tokens] + if _has_any(torch, active_fp8 < -1) or _has_any(torch, active_scale < -1): + raise ValueError("slot mapping entries must be -1 or non-negative offsets") + + valid_fp8 = active_fp8 >= 0 + valid_scale = active_scale >= 0 + if _has_any(torch, valid_fp8 != valid_scale): + raise ValueError("slot_mapping_fp8 and slot_mapping_scale must have matching validity") + + logical_cache_elems = int( + k_cache.shape[0] * k_cache.shape[1] * k_cache.shape[2] * k_cache.shape[3] + ) + max_fp8_offset = logical_cache_elems - _FUSED_CAT_HEAD_DIM + max_scale_offset = logical_cache_elems - 4 + if _has_any( + torch, + valid_fp8 & (((active_fp8 % 4) != 0) | (active_fp8 > max_fp8_offset)), + ): + raise ValueError( + "slot_mapping_fp8 entries must be 4-byte aligned and leave room " + f"for {_FUSED_CAT_HEAD_DIM} bytes" + ) + if _has_any( + torch, + valid_scale & (((active_scale % 4) != 0) | (active_scale > max_scale_offset)), + ): + raise ValueError( + "slot_mapping_scale entries must be 4-byte aligned and leave room for 4 bytes" + ) + + +def _require_indexer_k_cache_update_inputs( + torch, + indexer_k, + norm_weight, + norm_bias, + position_ids, + rotary_cos_sin, + k_cache, + slot_mapping_fp8, + slot_mapping_scale, +) -> None: + _require_cuda_contiguous(torch, indexer_k, "indexer_k", torch.bfloat16) + _require_cuda_contiguous(torch, norm_weight, "norm_weight", torch.float32) + _require_cuda_contiguous(torch, norm_bias, "norm_bias", torch.float32) + _require_cuda_contiguous(torch, position_ids, "position_ids", torch.int32) + _require_cuda_contiguous(torch, rotary_cos_sin, "rotary_cos_sin", torch.float32) + if indexer_k.ndim != 2 or indexer_k.shape[1] != _FUSED_CAT_HEAD_DIM: + raise ValueError( + f"indexer_k must have shape [T,{_FUSED_CAT_HEAD_DIM}], got {tuple(indexer_k.shape)}" + ) + if tuple(norm_weight.shape) != (_FUSED_CAT_HEAD_DIM,): + raise ValueError( + f"norm_weight must have shape [{_FUSED_CAT_HEAD_DIM}], got {tuple(norm_weight.shape)}" + ) + if tuple(norm_bias.shape) != (_FUSED_CAT_HEAD_DIM,): + raise ValueError( + f"norm_bias must have shape [{_FUSED_CAT_HEAD_DIM}], got {tuple(norm_bias.shape)}" + ) + if position_ids.shape != (indexer_k.shape[0],): + raise ValueError( + f"position_ids must have shape [{indexer_k.shape[0]}], got {tuple(position_ids.shape)}" + ) + if rotary_cos_sin.ndim != 3 or tuple(rotary_cos_sin.shape[1:]) != ( + 2, + _FUSED_CAT_PE_DIM // 2, + ): + raise ValueError( + "rotary_cos_sin must have shape [max_positions,2,32], " + f"got {tuple(rotary_cos_sin.shape)}" + ) + if k_cache.dtype != torch.uint8: + raise TypeError(f"k_cache must be torch.uint8, got {k_cache.dtype}") + if not k_cache.is_cuda: + raise TypeError("k_cache must be a CUDA tensor") + if k_cache.ndim != 4 or k_cache.shape[2] != 1: + raise ValueError( + "k_cache must be 4D [num_blocks, block_size, 1, per_token_size], " + f"got {tuple(k_cache.shape)}" + ) + if k_cache.shape[3] < _FUSED_CAT_HEAD_DIM + 4: + raise ValueError( + f"k_cache.shape[3] must be at least {_FUSED_CAT_HEAD_DIM + 4}, got {k_cache.shape[3]}" + ) + if k_cache.stride(-1) != 1: + raise ValueError( + "k_cache must have innermost stride 1 for vectorized byte stores, " + f"got stride={tuple(k_cache.stride())}" + ) + + for tensor, name in ( + (slot_mapping_fp8, "slot_mapping_fp8"), + (slot_mapping_scale, "slot_mapping_scale"), + ): + if tensor.dtype != torch.int64: + raise TypeError(f"{name} must be torch.int64, got {tensor.dtype}") + if not tensor.is_cuda: + raise TypeError(f"{name} must be a CUDA tensor") + if tensor.ndim != 1: + raise ValueError(f"{name} must be 1D, got {tuple(tensor.shape)}") + if not tensor.is_contiguous(): + raise ValueError(f"{name} must be contiguous") + + +def indexer_k_norm_rope_fp8_cache_update( + indexer_k, + norm_weight, + norm_bias, + position_ids, + rotary_cos_sin, + k_cache, + slot_mapping_fp8, + slot_mapping_scale, + num_tokens: int, + use_ue8m0: bool = True, + is_neox: bool = True, + compiled_fn: Callable | None = None, + rows_per_block: int | None = None, + validate: bool = True, + *, + use_pdl: bool = True, +) -> None: + """Run the fused indexer-K norm + RoPE + FP8 cache update. + + When this wrapper owns compilation, ``rows_per_block=None`` selects two + rows for up to 1024 tokens and eight rows for larger workloads. Callers + that supply ``compiled_fn`` retain control of that specialization. + """ + import torch + + current_device = torch.cuda.current_device() + if validate: + _require_indexer_k_cache_update_inputs( + torch, + indexer_k, + norm_weight, + norm_bias, + position_ids, + rotary_cos_sin, + k_cache, + slot_mapping_fp8, + slot_mapping_scale, + ) + if num_tokens < 0 or num_tokens > indexer_k.shape[0]: + raise ValueError(f"num_tokens must be in [0, {indexer_k.shape[0]}], got {num_tokens}") + if slot_mapping_fp8.shape[0] < num_tokens or slot_mapping_scale.shape[0] < num_tokens: + raise ValueError("slot mapping tensors must contain at least num_tokens entries") + _require_current_cuda_device( + torch, + ( + (indexer_k, "indexer_k"), + (norm_weight, "norm_weight"), + (norm_bias, "norm_bias"), + (position_ids, "position_ids"), + (rotary_cos_sin, "rotary_cos_sin"), + (k_cache, "k_cache"), + (slot_mapping_fp8, "slot_mapping_fp8"), + (slot_mapping_scale, "slot_mapping_scale"), + ), + current_device, + ) + _require_indexer_k_cache_update_ranges( + torch, + position_ids, + rotary_cos_sin, + k_cache, + slot_mapping_fp8, + slot_mapping_scale, + num_tokens, + ) + + if compiled_fn is None: + if rows_per_block is None: + rows_per_block = _select_rows_per_block(num_tokens) + compiled_fn = compile_indexer_k_norm_rope_fp8_cache_update( + device_id=current_device, + use_ue8m0=bool(use_ue8m0), + is_neox=bool(is_neox), + rows_per_block=rows_per_block, + vectorized_loads=_can_vectorize_loads(indexer_k), + cache_contiguous=bool(k_cache.is_contiguous()), + use_pdl=bool(use_pdl), + ) + + active_rows = int(indexer_k.shape[0]) + active_slot_mapping_fp8 = slot_mapping_fp8[:active_rows] + active_slot_mapping_scale = slot_mapping_scale[:active_rows] + + compiled_fn( + indexer_k, + norm_weight, + norm_bias, + position_ids, + rotary_cos_sin, + k_cache, + active_slot_mapping_fp8, + active_slot_mapping_scale, + int(num_tokens), + int(k_cache.stride(0)), + int(k_cache.stride(1)), + int(k_cache.stride(2)), + int(k_cache.stride(3)), + _current_cuda_stream(torch, current_device), + ) diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/kernels/_indexer_q_side_fused_kernel.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/kernels/_indexer_q_side_fused_kernel.py new file mode 100644 index 000000000000..961d1c492800 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/kernels/_indexer_q_side_fused_kernel.py @@ -0,0 +1,633 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""DSA indexer, q-side post-projection fusion (decode and prefill rows). + +One launch replaces the four launches that follow the indexer ``q_b`` GEMM on a +full-indexer layer once the split-output pre-indexer projection has produced the +packed head weights:: + + q_pe, q_nope = split(q, [64, 64], -1) # q bf16 [T, 32, 128] + q_pe = rope_inplace(q_pe, pos, cos_sin, is_neox) # (1) flashinfer rope + q_fp8, q_scale = fused_cat_fp8(q_pe, q_nope, use_ue8m0) # (2) [pe|nope] e4m3 ue8m0 + weights = (wq * q_scale.view(T, 32)) * c # (3)(4) two fp32 muls + +ABI (all tensors CUDA, on the same device; ``T`` and the table length are runtime): + + inputs + q bf16 [T, 32, 128] contiguous (strides (4096, 128, 1)), 16 B aligned; + read only -- the rotated PE half is NOT written + back (nothing downstream reads it) + pos int32 [T] contiguous; caller contract + 0 <= pos < cos_sin.shape[0], not validated on + the device path + cos_sin fp32 [max_pos, 64] contiguous, 16 B aligned: ``[cos(32) | sin(32)]`` + per position (the indexer rope table, theta + already applied) + wq fp32 [T, 32] contiguous (the packed raw head weights) + c float ``idx_weight_scale = Di**-0.5 * Hi**-0.5`` + outputs (caller-owned, ``out=`` style, must not alias any input) + q_fp8 e4m3 [T, 32, 128] contiguous, 16 B aligned: ``[pe | nope]`` codes + weights fp32 [T, 32] contiguous -> stride (32, 1), + ``(wq * q_scale) * c`` + + ``q_scale`` (one ue8m0 power-of-two fp32 scale per (t, h)) is consumed inside + ``weights`` and not emitted, exactly as the forward does today. + +Mapping: one (token, head) row of 128 bf16 = 256 B per eight lanes, four rows per +warp, ``rows_per_block`` rows per CTA (default 16 -> 128 threads). Lane ``i`` of a row +owns PE dims ``8i..8i+7`` and NOPE dims ``64+8i..64+8i+7`` as two 16 B loads; the NeoX +partner 32 dims away is an xor-4 lane shuffle inside the row, the interleaved partner +is the neighbouring register. Per-row amax is a three-step butterfly over the eight +lanes; lane 0 of the row writes ``weights``. + +Precision boundaries reproduced from the reference chain, in order: bf16 inputs are +flushed to signed zero when subnormal (flashinfer's fast-math cast), rope is computed as +``fma(x, cos, (+/- partner) * sin)`` in fp32 and **rounded to bf16** (the materialised +in-place write of the reference), amax / scale / codes are then computed from those +bf16 values exactly as the TRT-LLM ``fused_cat_fp8`` op does (``scale = max(amax, 1e-12) +/ 448`` rounded up to a power of two, ``code = e4m3(x * (1/scale))``), and ``weights`` +is two separately rounded fp32 products, left-associated. ``q_fp8`` and ``weights`` +are bit-identical to the torch/flashinfer/TRT-LLM chain. + +``is_neox``: this checkpoint's forward passes ``indexer_is_neox = not +_INDEXER_ROPE_INTERLEAVE = True`` (NeoX halves rotate together); both modes are compiled +and verified because a wrong mode is a silent accuracy defect (max abs ~7.8 vs 2.4e-4). + +PDL: ``compile_indexer_q_side_fused(use_pdl=True)`` issues ``griddepcontrol.wait`` +before the first global read and launches with the programmatic-stream-serialisation +attribute; like the fused k entry it relies on kernel completion to release the next +dependent (no explicit ``launch_dependents``). The default builds the plain kernel. + +Kernel name: +``self_attn_indexer_q_side_fused__rpb_pdl<0|1>``. +""" + +from __future__ import annotations + +from collections.abc import Callable +from dataclasses import dataclass + +import cuda.bindings.driver as cuda +import cutlass +import cutlass.cute as cute +import torch +from cutlass.cute.runtime import make_fake_compact_tensor, make_fake_stream + +_WARP_SIZE = 32 +_NUM_HEADS = 32 +_HEAD_DIM = 128 +_PE_DIM = 64 +_ROPE_HALF = _PE_DIM // 2 +_LANES_PER_ROW = 8 +_ROWS_PER_WARP = _WARP_SIZE // _LANES_PER_ROW +_VALUES_PER_LANE = _PE_DIM // _LANES_PER_ROW # 8 PE + 8 NOPE values per lane +_BF16_VEC_BYTES = _VALUES_PER_LANE * 2 +_FP32_VEC_VALUES = 4 +_FP32_VEC_BYTES = _FP32_VEC_VALUES * 4 +_FP8_MAX = 448.0 +_INV_FP8_MAX = 1.0 / _FP8_MAX +_MIN_AMAX = 1.0e-12 +_DEFAULT_ROWS_PER_BLOCK = 16 +_ALLOWED_ROWS_PER_BLOCK = (4, 8, 16, 32, 64) +# The rope layout a caller gets when it names none; the entry always +# passes its own. +CHECKPOINT_IS_NEOX = True +KERNEL_NAME_PREFIX = "self_attn_indexer_q_side_fused" + + +@dataclass(frozen=True) +class QSideFusedConfig: + """Compile-time specialisation of the fused q-side kernel.""" + + is_neox: bool = CHECKPOINT_IS_NEOX + use_ue8m0: bool = True + rows_per_block: int = _DEFAULT_ROWS_PER_BLOCK + use_pdl: bool = False + + def __post_init__(self) -> None: + if self.rows_per_block not in _ALLOWED_ROWS_PER_BLOCK: + raise ValueError( + f"rows_per_block must be one of {_ALLOWED_ROWS_PER_BLOCK}, " + f"got {self.rows_per_block}" + ) + + @property + def threads(self) -> int: + return self.rows_per_block // _ROWS_PER_WARP * _WARP_SIZE + + @property + def kernel_name(self) -> str: + mode = "neox" if self.is_neox else "interleave" + return f"{KERNEL_NAME_PREFIX}_{mode}_rpb{self.rows_per_block}_pdl{int(self.use_pdl)}" + + +_COMPILE_CACHE: dict[tuple[int, QSideFusedConfig], Callable] = {} + + +@cute.jit +def _bf16_to_fp32_flashinfer_ftz(value: cutlass.BFloat16) -> cutlass.Float32: + """Match flashinfer's fast-math BF16 -> FP32 boundary: subnormals become +/-0.""" + bits = value.bitcast(cutlass.Uint16) + converted = value.to(cutlass.Float32) + if (bits & cutlass.Uint16(0x7F80)) == cutlass.Uint16(0): + signed_zero_bits = bits & cutlass.Uint16(0x8000) + converted = signed_zero_bits.bitcast(cutlass.BFloat16).to(cutlass.Float32) + return converted + + +@cute.jit +def _round_bf16(value: cutlass.Float32) -> cutlass.Float32: + """The materialised BF16 boundary of the in-place rope write.""" + return value.to(cutlass.BFloat16).to(cutlass.Float32) + + +@cute.jit +def _row8_reduction_max(value: cutlass.Float32) -> cutlass.Float32: + """Max over the eight lanes of one row (butterfly stays inside the row).""" + active_mask = cute.arch.activemask() + value = cutlass.max(value, cute.arch.shuffle_sync_bfly(value, 4, mask=active_mask)) + value = cutlass.max(value, cute.arch.shuffle_sync_bfly(value, 2, mask=active_mask)) + return cutlass.max(value, cute.arch.shuffle_sync_bfly(value, 1, mask=active_mask)) + + +@cute.jit +def _quantize_pack_fp8x4( + x0: cutlass.Float32, + x1: cutlass.Float32, + x2: cutlass.Float32, + x3: cutlass.Float32, + inv_scale: cutlass.Float32, +) -> cutlass.Int32: + """Quantise four values to E4M3 and pack the bytes little-endian into one word.""" + q0 = (x0 * inv_scale).to(cutlass.Float8E4M3FN).bitcast(cutlass.Int8) + q1 = (x1 * inv_scale).to(cutlass.Float8E4M3FN).bitcast(cutlass.Int8) + q2 = (x2 * inv_scale).to(cutlass.Float8E4M3FN).bitcast(cutlass.Int8) + q3 = (x3 * inv_scale).to(cutlass.Float8E4M3FN).bitcast(cutlass.Int8) + return ( + (cutlass.Int32(q0) & cutlass.Int32(0xFF)) + | ((cutlass.Int32(q1) & cutlass.Int32(0xFF)) << 8) + | ((cutlass.Int32(q2) & cutlass.Int32(0xFF)) << 16) + | ((cutlass.Int32(q3) & cutlass.Int32(0xFF)) << 24) + ) + + +@cute.jit +def _pack_words_i64(lo: cutlass.Int32, hi: cutlass.Int32) -> cutlass.Int64: + """Two little-endian 32-bit words -> one 64-bit word (``lo`` at the low address).""" + low = cutlass.Int64(lo) & cutlass.Int64(0xFFFFFFFF) + return (cutlass.Int64(hi) << 32) | low + + +@cute.kernel +def indexer_q_side_fused_kernel( + q: cute.Tensor, + position_ids: cute.Tensor, + cos_sin: cute.Tensor, + wq: cute.Tensor, + q_fp8: cute.Tensor, + weights: cute.Tensor, + rows: cutlass.Int32, + weight_scale_factor: cutlass.Float32, + IS_NEOX: cutlass.Constexpr[bool], + USE_UE8M0: cutlass.Constexpr[bool], + ROWS_PER_BLOCK: cutlass.Constexpr[int], + USE_PDL: cutlass.Constexpr[bool], +) -> None: + """Rope the PE half, quantise ``[pe | nope]`` to E4M3, finalise the head weight. + + ``rows = T * 32``; row ``r`` is token ``r // 32``, head ``r % 32``. All tensors are + addressed through their raw pointers as flat row-major storage. + """ + block, _, _ = cute.arch.block_idx() + tidx, _, _ = cute.arch.thread_idx() + warp_in_block = tidx // cutlass.Int32(_WARP_SIZE) + lane = tidx % cutlass.Int32(_WARP_SIZE) + row_in_warp = lane // cutlass.Int32(_LANES_PER_ROW) + lane_in_row = lane % cutlass.Int32(_LANES_PER_ROW) + row = ( + block * cutlass.Int32(ROWS_PER_BLOCK) + + warp_in_block * cutlass.Int32(_ROWS_PER_WARP) + + row_in_warp + ) + + if cutlass.const_expr(USE_PDL): + # Nothing above touches global memory; every thread waits before its first + # load of the predecessor GEMM's output (no CTA barrier on this path). + cute.arch.griddepcontrol_wait() + + if row < rows: + pe_col = lane_in_row * cutlass.Int32(_VALUES_PER_LANE) + nope_col = pe_col + cutlass.Int32(_PE_DIM) + q_base = row * cutlass.Int32(_HEAD_DIM) + q_ptr = q.iterator.raw_ptr() + pe_vals = (q_ptr + q_base + pe_col).load(count=_VALUES_PER_LANE, alignment=_BF16_VEC_BYTES) + nope_vals = (q_ptr + q_base + nope_col).load( + count=_VALUES_PER_LANE, alignment=_BF16_VEC_BYTES + ) + x0 = _bf16_to_fp32_flashinfer_ftz(pe_vals[0]) + x1 = _bf16_to_fp32_flashinfer_ftz(pe_vals[1]) + x2 = _bf16_to_fp32_flashinfer_ftz(pe_vals[2]) + x3 = _bf16_to_fp32_flashinfer_ftz(pe_vals[3]) + x4 = _bf16_to_fp32_flashinfer_ftz(pe_vals[4]) + x5 = _bf16_to_fp32_flashinfer_ftz(pe_vals[5]) + x6 = _bf16_to_fp32_flashinfer_ftz(pe_vals[6]) + x7 = _bf16_to_fp32_flashinfer_ftz(pe_vals[7]) + n0 = nope_vals[0].to(cutlass.Float32) + n1 = nope_vals[1].to(cutlass.Float32) + n2 = nope_vals[2].to(cutlass.Float32) + n3 = nope_vals[3].to(cutlass.Float32) + n4 = nope_vals[4].to(cutlass.Float32) + n5 = nope_vals[5].to(cutlass.Float32) + n6 = nope_vals[6].to(cutlass.Float32) + n7 = nope_vals[7].to(cutlass.Float32) + + token = row // cutlass.Int32(_NUM_HEADS) + position = (position_ids.iterator + token).load() + table_base = cutlass.Int64(position) * cutlass.Int64(_PE_DIM) + table_ptr = cos_sin.iterator.raw_ptr() + + if cutlass.const_expr(IS_NEOX): + # Partner is 32 dims = 4 lanes away; every lane of the row shuffles before + # any lane branches on its half. + active_mask = cute.arch.activemask() + source_lane = row_in_warp * cutlass.Int32(_LANES_PER_ROW) + ( + lane_in_row ^ cutlass.Int32(4) + ) + p0 = cute.arch.shuffle_sync(x0, source_lane, mask=active_mask) + p1 = cute.arch.shuffle_sync(x1, source_lane, mask=active_mask) + p2 = cute.arch.shuffle_sync(x2, source_lane, mask=active_mask) + p3 = cute.arch.shuffle_sync(x3, source_lane, mask=active_mask) + p4 = cute.arch.shuffle_sync(x4, source_lane, mask=active_mask) + p5 = cute.arch.shuffle_sync(x5, source_lane, mask=active_mask) + p6 = cute.arch.shuffle_sync(x6, source_lane, mask=active_mask) + p7 = cute.arch.shuffle_sync(x7, source_lane, mask=active_mask) + + table_dim = pe_col + if table_dim >= cutlass.Int32(_ROPE_HALF): + table_dim = table_dim - cutlass.Int32(_ROPE_HALF) + cos_base = table_base + cutlass.Int64(table_dim) + sin_base = cos_base + cutlass.Int64(_ROPE_HALF) + cos_lo = (table_ptr + cos_base).load(count=_FP32_VEC_VALUES, alignment=_FP32_VEC_BYTES) + cos_hi = (table_ptr + cos_base + cutlass.Int64(_FP32_VEC_VALUES)).load( + count=_FP32_VEC_VALUES, alignment=_FP32_VEC_BYTES + ) + sin_lo = (table_ptr + sin_base).load(count=_FP32_VEC_VALUES, alignment=_FP32_VEC_BYTES) + sin_hi = (table_ptr + sin_base + cutlass.Int64(_FP32_VEC_VALUES)).load( + count=_FP32_VEC_VALUES, alignment=_FP32_VEC_BYTES + ) + # flashinfer: vec[i] * cos[i] + (+/- partner[i]) * sin[i], contracted to + # one FMA with the (signed) partner product rounded first. + s0 = p0 * sin_lo[0] + s1 = p1 * sin_lo[1] + s2 = p2 * sin_lo[2] + s3 = p3 * sin_lo[3] + s4 = p4 * sin_hi[0] + s5 = p5 * sin_hi[1] + s6 = p6 * sin_hi[2] + s7 = p7 * sin_hi[3] + y0 = cute.math.fma(x0, cos_lo[0], s0) + y1 = cute.math.fma(x1, cos_lo[1], s1) + y2 = cute.math.fma(x2, cos_lo[2], s2) + y3 = cute.math.fma(x3, cos_lo[3], s3) + y4 = cute.math.fma(x4, cos_hi[0], s4) + y5 = cute.math.fma(x5, cos_hi[1], s5) + y6 = cute.math.fma(x6, cos_hi[2], s6) + y7 = cute.math.fma(x7, cos_hi[3], s7) + if pe_col < cutlass.Int32(_ROPE_HALF): + y0 = cute.math.fma(x0, cos_lo[0], -s0) + y1 = cute.math.fma(x1, cos_lo[1], -s1) + y2 = cute.math.fma(x2, cos_lo[2], -s2) + y3 = cute.math.fma(x3, cos_lo[3], -s3) + y4 = cute.math.fma(x4, cos_hi[0], -s4) + y5 = cute.math.fma(x5, cos_hi[1], -s5) + y6 = cute.math.fma(x6, cos_hi[2], -s6) + y7 = cute.math.fma(x7, cos_hi[3], -s7) + else: + # Interleaved: element e pairs with e ^ 1 and uses cos[e // 2], sin[e // 2]; + # this lane's four table values start at pe_col // 2 = 4 * lane_in_row. + cos_base = table_base + cutlass.Int64(pe_col // cutlass.Int32(2)) + sin_base = cos_base + cutlass.Int64(_ROPE_HALF) + cos_v = (table_ptr + cos_base).load(count=_FP32_VEC_VALUES, alignment=_FP32_VEC_BYTES) + sin_v = (table_ptr + sin_base).load(count=_FP32_VEC_VALUES, alignment=_FP32_VEC_BYTES) + y0 = cute.math.fma(x0, cos_v[0], -(x1 * sin_v[0])) + y1 = cute.math.fma(x1, cos_v[0], x0 * sin_v[0]) + y2 = cute.math.fma(x2, cos_v[1], -(x3 * sin_v[1])) + y3 = cute.math.fma(x3, cos_v[1], x2 * sin_v[1]) + y4 = cute.math.fma(x4, cos_v[2], -(x5 * sin_v[2])) + y5 = cute.math.fma(x5, cos_v[2], x4 * sin_v[2]) + y6 = cute.math.fma(x6, cos_v[3], -(x7 * sin_v[3])) + y7 = cute.math.fma(x7, cos_v[3], x6 * sin_v[3]) + + # The reference writes the rotated PE half back as bf16 and re-reads it. + r0 = _round_bf16(y0) + r1 = _round_bf16(y1) + r2 = _round_bf16(y2) + r3 = _round_bf16(y3) + r4 = _round_bf16(y4) + r5 = _round_bf16(y5) + r6 = _round_bf16(y6) + r7 = _round_bf16(y7) + + local_max = cute.math.absf(r0) + local_max = cutlass.max(local_max, cute.math.absf(r1)) + local_max = cutlass.max(local_max, cute.math.absf(r2)) + local_max = cutlass.max(local_max, cute.math.absf(r3)) + local_max = cutlass.max(local_max, cute.math.absf(r4)) + local_max = cutlass.max(local_max, cute.math.absf(r5)) + local_max = cutlass.max(local_max, cute.math.absf(r6)) + local_max = cutlass.max(local_max, cute.math.absf(r7)) + local_max = cutlass.max(local_max, cute.math.absf(n0)) + local_max = cutlass.max(local_max, cute.math.absf(n1)) + local_max = cutlass.max(local_max, cute.math.absf(n2)) + local_max = cutlass.max(local_max, cute.math.absf(n3)) + local_max = cutlass.max(local_max, cute.math.absf(n4)) + local_max = cutlass.max(local_max, cute.math.absf(n5)) + local_max = cutlass.max(local_max, cute.math.absf(n6)) + local_max = cutlass.max(local_max, cute.math.absf(n7)) + amax = _row8_reduction_max(local_max) + safe_amax = amax + if safe_amax < cutlass.Float32(_MIN_AMAX): + safe_amax = cutlass.Float32(_MIN_AMAX) + + scale = safe_amax * cutlass.Float32(_INV_FP8_MAX) + if cutlass.const_expr(USE_UE8M0): + scale_bits = scale.bitcast(cutlass.Int32) + mantissa = scale_bits & cutlass.Int32(0x007FFFFF) + exp_bits = scale_bits & cutlass.Int32(0x7F800000) + if mantissa != cutlass.Int32(0): + exp_bits += cutlass.Int32(0x00800000) + scale = exp_bits.bitcast(cutlass.Float32) + # 1 / 2^k is exact: negate the biased exponent. + inv_scale_bits = cutlass.Int32(0x7F000000) - exp_bits + inv_scale = inv_scale_bits.bitcast(cutlass.Float32) + else: + inv_scale = cutlass.Float32(1.0) / scale + + pe_word = _pack_words_i64( + _quantize_pack_fp8x4(r0, r1, r2, r3, inv_scale), + _quantize_pack_fp8x4(r4, r5, r6, r7, inv_scale), + ) + nope_word = _pack_words_i64( + _quantize_pack_fp8x4(n0, n1, n2, n3, inv_scale), + _quantize_pack_fp8x4(n4, n5, n6, n7, inv_scale), + ) + out_base = row * cutlass.Int32(_HEAD_DIM) + pe_out_ptr = cute.recast_ptr( + (q_fp8.iterator + out_base + pe_col).align(8), dtype=cutlass.Int64 + ) + pe_out_ptr.store(pe_word) + nope_out_ptr = cute.recast_ptr( + (q_fp8.iterator + out_base + nope_col).align(8), dtype=cutlass.Int64 + ) + nope_out_ptr.store(nope_word) + + if lane_in_row == cutlass.Int32(0): + # Two separately rounded fp32 products, left-associated, as the forward's + # ``mul(mul(wq, q_scale), c)``. + raw_weight = (wq.iterator + row).load() + product = raw_weight * scale + (weights.iterator + row).store(product * weight_scale_factor) + + +@cute.jit +def _launch_indexer_q_side_fused( + q: cute.Tensor, + position_ids: cute.Tensor, + cos_sin: cute.Tensor, + wq: cute.Tensor, + q_fp8: cute.Tensor, + weights: cute.Tensor, + num_tokens: cutlass.Int32, + weight_scale_factor: cutlass.Float32, + stream: cuda.CUstream, + IS_NEOX: cutlass.Constexpr[bool], + USE_UE8M0: cutlass.Constexpr[bool], + ROWS_PER_BLOCK: cutlass.Constexpr[int], + USE_PDL: cutlass.Constexpr[bool], +) -> None: + rows = num_tokens * cutlass.Int32(_NUM_HEADS) + threads = ROWS_PER_BLOCK // _ROWS_PER_WARP * _WARP_SIZE + indexer_q_side_fused_kernel( + q, + position_ids, + cos_sin, + wq, + q_fp8, + weights, + rows, + weight_scale_factor, + IS_NEOX, + USE_UE8M0, + ROWS_PER_BLOCK, + USE_PDL, + ).launch( + grid=(cute.ceil_div(rows, ROWS_PER_BLOCK), 1, 1), + block=(threads, 1, 1), + stream=stream, + use_pdl=USE_PDL, + ) + + +def compile_indexer_q_side_fused( + device_id: int = 0, + is_neox: bool = CHECKPOINT_IS_NEOX, + use_ue8m0: bool = True, + rows_per_block: int = _DEFAULT_ROWS_PER_BLOCK, + *, + use_pdl: bool = False, +) -> Callable: + """Compile (or fetch) one specialisation; allocates no GPU tensors. + + The compiled callable takes ``(q, position_ids, cos_sin, wq, q_fp8_out, + weights_out, num_tokens: int, weight_scale_factor: float, stream: cuda.CUstream)`` + and is what :func:`indexer_q_side_fused` launches. + """ + config = QSideFusedConfig( + is_neox=bool(is_neox), + use_ue8m0=bool(use_ue8m0), + rows_per_block=int(rows_per_block), + use_pdl=bool(use_pdl), + ) + key = (int(device_id), config) + cached = _COMPILE_CACHE.get(key) + if cached is not None: + return cached + + cutlass.cuda.initialize_cuda_context(device_id) + sym_tokens = cute.sym_int32() + sym_positions = cute.sym_int32() + fake_q = make_fake_compact_tensor( + cutlass.BFloat16, + (sym_tokens, _NUM_HEADS, _HEAD_DIM), + stride_order=(2, 1, 0), + assumed_align=_BF16_VEC_BYTES, + ) + fake_pos = make_fake_compact_tensor(cutlass.Int32, (sym_tokens,), assumed_align=4) + fake_cos_sin = make_fake_compact_tensor( + cutlass.Float32, + (sym_positions, _PE_DIM), + stride_order=(1, 0), + assumed_align=_FP32_VEC_BYTES, + ) + fake_wq = make_fake_compact_tensor( + cutlass.Float32, (sym_tokens, _NUM_HEADS), stride_order=(1, 0), assumed_align=4 + ) + fake_q_fp8 = make_fake_compact_tensor( + cutlass.Float8E4M3FN, + (sym_tokens, _NUM_HEADS, _HEAD_DIM), + stride_order=(2, 1, 0), + assumed_align=16, + ) + fake_weights = make_fake_compact_tensor( + cutlass.Float32, (sym_tokens, _NUM_HEADS), stride_order=(1, 0), assumed_align=4 + ) + indexer_q_side_fused_kernel.set_name_prefix(config.kernel_name, remove_cutlass_symbol=True) + compiled = cute.compile( + _launch_indexer_q_side_fused, + fake_q, + fake_pos, + fake_cos_sin, + fake_wq, + fake_q_fp8, + fake_weights, + cutlass.Int32(0), + cutlass.Float32(1.0), + make_fake_stream(), + config.is_neox, + config.use_ue8m0, + config.rows_per_block, + config.use_pdl, + options="--enable-tvm-ffi", + ) + _COMPILE_CACHE[key] = compiled + return compiled + + +def _byte_span(tensor: torch.Tensor) -> tuple[int, int]: + begin = tensor.data_ptr() + return begin, begin + tensor.numel() * tensor.element_size() + + +def _overlaps(left: torch.Tensor, right: torch.Tensor) -> bool: + lb, le = _byte_span(left) + rb, re = _byte_span(right) + return left.device == right.device and max(lb, rb) < min(le, re) + + +def _require(tensor: torch.Tensor, name: str, dtype, shape: tuple[int, ...], align: int): + if not tensor.is_cuda: + raise ValueError(f"{name} must be a CUDA tensor, got {tensor.device}") + if tensor.dtype != dtype: + raise TypeError(f"{name} must be {dtype}, got {tensor.dtype}") + if tuple(tensor.shape) != shape: + raise ValueError(f"{name} must have shape {shape}, got {tuple(tensor.shape)}") + if not tensor.is_contiguous(): + raise ValueError(f"{name} must be contiguous, got stride {tuple(tensor.stride())}") + if tensor.data_ptr() % align: + raise ValueError(f"{name} must be {align}-byte aligned") + + +def validate_abi( + q: torch.Tensor, + position_ids: torch.Tensor, + cos_sin: torch.Tensor, + wq: torch.Tensor, + q_fp8_out: torch.Tensor, + weights_out: torch.Tensor, +) -> int: + """Fail closed on every host-checkable property of the ABI; returns ``T``. + + Metadata only -- no device read, no synchronisation -- so it is safe inside a + CUDA-graph capture. The ``position_ids`` range is a caller contract. + """ + if q.ndim != 3: + raise ValueError(f"q must be [T, 32, 128], got {tuple(q.shape)}") + num_tokens = int(q.shape[0]) + if num_tokens < 1: + raise ValueError("q must hold at least one token") + if cos_sin.ndim != 2 or cos_sin.shape[1] != _PE_DIM: + raise ValueError(f"cos_sin must be [max_pos, {_PE_DIM}], got {tuple(cos_sin.shape)}") + _require(q, "q", torch.bfloat16, (num_tokens, _NUM_HEADS, _HEAD_DIM), _BF16_VEC_BYTES) + _require(position_ids, "position_ids", torch.int32, (num_tokens,), 4) + _require(cos_sin, "cos_sin", torch.float32, tuple(cos_sin.shape), _FP32_VEC_BYTES) + _require(wq, "wq", torch.float32, (num_tokens, _NUM_HEADS), 4) + _require( + q_fp8_out, + "q_fp8_out", + torch.float8_e4m3fn, + (num_tokens, _NUM_HEADS, _HEAD_DIM), + 16, + ) + _require(weights_out, "weights_out", torch.float32, (num_tokens, _NUM_HEADS), 4) + device = q.device + for name, tensor in ( + ("position_ids", position_ids), + ("cos_sin", cos_sin), + ("wq", wq), + ("q_fp8_out", q_fp8_out), + ("weights_out", weights_out), + ): + if tensor.device != device: + raise ValueError(f"{name} must be on {device}, got {tensor.device}") + for out_name, out in (("q_fp8_out", q_fp8_out), ("weights_out", weights_out)): + for in_name, inp in ( + ("q", q), + ("position_ids", position_ids), + ("cos_sin", cos_sin), + ("wq", wq), + ): + if _overlaps(out, inp): + raise ValueError(f"{out_name} must not alias {in_name}") + if _overlaps(q_fp8_out, weights_out): + raise ValueError("q_fp8_out and weights_out must not alias") + return num_tokens + + +def indexer_q_side_fused( + q: torch.Tensor, + position_ids: torch.Tensor, + cos_sin: torch.Tensor, + wq: torch.Tensor, + weight_scale_factor: float, + q_fp8_out: torch.Tensor, + weights_out: torch.Tensor, + *, + is_neox: bool = CHECKPOINT_IS_NEOX, + use_ue8m0: bool = True, + compiled_fn: Callable | None = None, + rows_per_block: int = _DEFAULT_ROWS_PER_BLOCK, + use_pdl: bool = False, + validate: bool = True, +) -> tuple[torch.Tensor, torch.Tensor]: + """Enqueue the fused q-side kernel on the current stream of ``q``'s device. + + Allocation-free and sync-free; writes ``q_fp8_out`` and ``weights_out`` and + returns them. ``compiled_fn`` (from :func:`compile_indexer_q_side_fused`) skips + the cache lookup; when given, ``is_neox`` / ``use_ue8m0`` / ``rows_per_block`` / + ``use_pdl`` must describe it. + """ + if validate: + num_tokens = validate_abi(q, position_ids, cos_sin, wq, q_fp8_out, weights_out) + else: + num_tokens = int(q.shape[0]) + if compiled_fn is None: + compiled_fn = compile_indexer_q_side_fused( + device_id=q.device.index if q.device.index is not None else 0, + is_neox=is_neox, + use_ue8m0=use_ue8m0, + rows_per_block=rows_per_block, + use_pdl=use_pdl, + ) + stream = cuda.CUstream(torch.cuda.current_stream(q.device).cuda_stream) + compiled_fn( + q, + position_ids, + cos_sin, + wq, + q_fp8_out, + weights_out, + num_tokens, + float(weight_scale_factor), + stream, + ) + return q_fp8_out, weights_out diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/kernels/_lamport.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/kernels/_lamport.py new file mode 100644 index 000000000000..3edda1e94de7 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/kernels/_lamport.py @@ -0,0 +1,264 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The workspace the TP4 Lamport all-reduce entry is handed. + +The kernel module is `_tp4_allreduce_fused_rms_kernel`, beside this one and +private to the family. `cute_dsl_allreduce_add_rmsnorm` is the entry that +launches from the workspace. + +`build_workspace` compiles four kernels: the pure all-reduce, the all-reduce +fused with the residual add and RMSNorm (one-shot and two-shot), and the +one-shot fused form that also emits an NVFP4-quantized norm output. They read +and write the same symmetric-memory peer buffers and rotate the same device +slot counter, so every collective of a forward goes through one +`LamportWorkspace`, built once at load by `build_workspace`. +`cute_dsl_allreduce_add_rmsnorm` launches the fused one-shot and two-shot +kernels; no entry launches the pure or the NVFP4-output one. + +The kernel reads its slot from the counter in device memory and advances it +itself, which is what lets one CUDA graph capture every collective of a +forward and still rotate slots on replay. The peer buffers are three slots x +world x the token budget, so every row count up to the budget fits. + +Not an entry: it holds what the entry is handed, and no entry owns it. +""" + +from __future__ import annotations + +import os +from dataclasses import dataclass + +import cutlass +import torch + +from . import _tp4_allreduce_fused_rms_kernel as _ar + +_PING_PONG = 3 +_SIGNAL_STATE = 6 +_BF16_VEC = 8 +# In-band readiness: bitwise bf16 -0.0 marks a slot not yet arrived, as TRT-LLM's own +# Lamport backend does. Separate ready flags double the peer traffic. +_NEG_ZERO_BF16 = -32768 +# PDL release points. The module default, `strict`, hangs here: it waits at entry for a +# producer release the flashinfer and torch kernels ahead of it never issue. `entry` on the +# fused ops releases a PDL-launched dependent at the kernel's entry, so it can overlap the +# allreduce's NVLink wait. +_PDL_POLICY = "output_safe" +_FUSED_PDL_POLICY = "entry" +_FUSED_QUANT_PDL_POLICY = "entry" +_FUSED_OPTIONS = dict( + all_to_all_cluster_reduction=True, + last_cta_signal_advance=True, + elide_self_rank=True, + rotate_peer_order=True, + rotated_poll_order=True, +) +#: Above this many rows the add+RMSNorm entry runs two-shot: one-shot has every rank pull +#: the whole message from every peer. Below it two-shot measured slower. The choice is a +#: function of the row count alone, and every rank derives the row count from the same +#: batch, so all ranks take the same side. +TWOSHOT_ABOVE = 3072 + + +def ensure_process_group(rank: int, world: int) -> None: + """Bring up `torch.distributed` for this rank if nothing has yet. + + `symm_mem.rendezvous` needs a process group. The engine's ranks are + MPI-spawned and TRT-LLM's collectives use their own transport, so there may + be none. Three cases, in order: + + * A group is already up (the engine brought one up, or a test launcher + did): it has to be this rank in a group of `world`, and is used as is. + * The rendezvous address is in the environment (`MASTER_ADDR`, + `MASTER_PORT`, `RANK`, `WORLD_SIZE`): the group comes up from it. + MPI is not touched. + * Otherwise (the engine under MPI): rank 0 draws a free port and the + address is broadcast over `MPI.COMM_WORLD`, as TRT-LLM's MegaMoE path + does. Only this case needs MPI, so only it checks that `COMM_WORLD` is + the TP group. + """ + import torch.distributed as dist + + if dist.is_initialized(): + assert dist.get_world_size() == world and dist.get_rank() == rank, ( + f"the process group is rank {dist.get_rank()} of {dist.get_world_size()}; " + f"the collective was built for rank {rank} of {world}" + ) + return + if not all(os.environ.get(k) for k in ("MASTER_ADDR", "MASTER_PORT", "RANK", "WORLD_SIZE")): + from mpi4py import MPI + + comm = MPI.COMM_WORLD + assert comm.Get_size() == world and comm.Get_rank() == rank, (comm.Get_size(), world) + addr = None + if rank == 0: + import socket + + from tensorrt_llm._torch.disaggregation.native.utils import get_local_ip + + with socket.socket() as sock: + sock.bind(("", 0)) + port = sock.getsockname()[1] + addr = (get_local_ip(), port) + host, port = comm.bcast(addr, root=0) + os.environ["MASTER_ADDR"] = host + os.environ["MASTER_PORT"] = str(port) + dist.init_process_group( + backend="nccl", + rank=rank, + world_size=world, + device_id=torch.device("cuda", torch.cuda.current_device()), + ) + + +@dataclass(frozen=True) +class LamportWorkspace: + """One set of peer buffers and compiled kernels serving every collective in the forward. + + Every compiled kernel is symbolic in the row count, so one compile each covers every + row count up to `max_rows`; the kernel reads the active rows off its input's shape. + `state` is the device slot counter: every launch of any of the four kernels reads and + advances it on the device, so the workspace's tensors are shared state even though the + dataclass is frozen. One workspace per rank per process. + """ + + hidden: int + eps: float + max_rows: int + rank: int + world: int + #: This rank's `[3, world, max_rows * hidden]` bf16 symmetric buffer and its + #: `[3, world, max_rows * hidden / 8]` int32 flags. + buffer: torch.Tensor + flags: torch.Tensor + #: Every rank's buffer and flags, mapped into this rank, in rank order. + peer_buffers: list + peer_flags: list + #: `[6]` int32: entry counter, current slot, per-slot vector counts, release epoch. + state: torch.Tensor + #: The multicast address of `buffer`, or 0 when any rank has none. + multicast_base: int + pure: object + fused: object + fused_quant: object + fused_twoshot: object + + +def build_workspace( + hidden: int, eps: float, max_rows: int, rank: int, world: int +) -> LamportWorkspace: + """Bring up the group, pin the peer buffers, agree multicast, compile the kernels. + + Collective: every rank of the group calls it, with the same `hidden`, `eps` and + `max_rows`. Allocates `3 * world * max_rows * hidden` bf16 plus a quarter of that + in int32 flags of symmetric memory per rank, and compiles four CuTe DSL kernels + (pure, fused, fused+quant, fused two-shot) for this rank on the current device. + Launches nothing beyond the buffer fills and the one-element multicast agreement. + + The multicast decision is agreed across the group: a rank compiling the multicast + variant while a peer did not would wait on a store that never lands. + """ + import torch.distributed as dist + import torch.distributed._symmetric_memory as symm_mem + + ensure_process_group(rank, world) + dev = torch.device("cuda", torch.cuda.current_device()) + numel = max_rows * hidden + buffer = symm_mem.empty((_PING_PONG, world, numel), dtype=torch.bfloat16, device=dev) + # Every slot starts not-ready, or the first exchange reads uninitialized memory as data. + buffer.view(torch.int16).fill_(_NEG_ZERO_BF16) + flags = symm_mem.empty( + (_PING_PONG, world, numel // _BF16_VEC), dtype=torch.int32, device=dev + ).zero_() + group = dist.group.WORLD + name = getattr(group, "group_name", group) + bh = symm_mem.rendezvous(buffer, group=name) + fh = symm_mem.rendezvous(flags, group=name) + peer_buffers = [bh.get_buffer(peer, buffer.shape, buffer.dtype) for peer in range(world)] + peer_flags = [fh.get_buffer(peer, flags.shape, flags.dtype) for peer in range(world)] + state = torch.zeros(_SIGNAL_STATE, dtype=torch.int32, device=dev) + try: + mc = int(getattr(bh, "multicast_ptr", 0) or 0) + except Exception: # noqa: BLE001 -- no multicast on this fabric is an answer + mc = 0 + # Agreed over the process group that `rendezvous` just used, so it needs no MPI: + # the minimum of every rank's "has multicast" flag. + agreed = torch.tensor([1 if mc != 0 else 0], dtype=torch.int32, device=dev) + dist.all_reduce(agreed, op=dist.ReduceOp.MIN, group=group) + use_mc = bool(agreed.item()) + + common = dict(use_ready_flags=False, use_pdl=True) + pure = _ar.compile_pure_for_rank( + rank, world, dev.index, hidden, pdl_release_policy=_PDL_POLICY, **common + ) + fused = _ar.compile_fused_for_rank( + rank, + world, + dev.index, + hidden, + eps=eps, + pdl_release_policy=_FUSED_PDL_POLICY, + prefetch_fused_inputs=True, + multicast_publish=use_mc, + **_FUSED_OPTIONS, + **common, + ) + # Same transport, slots and signal state as the fused op; only the epilogue differs, + # so the two interleave. + fused_quant = _ar.compile_fused_out_quant_for_rank( + rank, + world, + dev.index, + hidden, + eps=eps, + pdl_release_policy=_FUSED_QUANT_PDL_POLICY, + prefetch_fused_inputs=True, + multicast_publish=use_mc, + **_FUSED_OPTIONS, + **common, + ) + fused_twoshot = _ar.compile_fused_twoshot_for_rank( + rank, + world, + dev.index, + hidden, + eps=eps, + use_pdl=True, + pdl_release_policy=_PDL_POLICY, + ) + return LamportWorkspace( + hidden=hidden, + eps=eps, + max_rows=max_rows, + rank=rank, + world=world, + buffer=buffer, + flags=flags, + peer_buffers=peer_buffers, + peer_flags=peer_flags, + state=state, + multicast_base=mc if use_mc else 0, + pure=pure, + fused=fused, + fused_quant=fused_quant, + fused_twoshot=fused_twoshot, + ) + + +def tail(workspace: LamportWorkspace, fused: bool) -> list: + """The workspace operands every launch ends with. `signal` is dead with device signal + state on, so it is 0; only the fused signatures carry `multicast_base`.""" + out = [workspace.peer_buffers, workspace.peer_flags, workspace.state] + if fused: + out.append(cutlass.Int64(workspace.multicast_base)) + out.append(cutlass.Int64(0)) + out.append(torch.cuda.current_stream().cuda_stream) + return out + + +def check_rows(rows: int, workspace: LamportWorkspace) -> None: + """The kernel takes the row count from its input, so a call above the token budget the + peer buffers were sized for writes past every peer's buffer.""" + assert rows <= workspace.max_rows, ( + f"{rows} rows exceed the collective's budget of {workspace.max_rows}" + ) diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/kernels/_radix_topk_decode.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/kernels/_radix_topk_decode.py new file mode 100644 index 000000000000..564e3c0987dd --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/kernels/_radix_topk_decode.py @@ -0,0 +1,932 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The CuTe DSL kernel behind `cute_dsl_radix_topk_decode`: one CTA selects one decode row. + +The selection is the trtllm `indexer_topk_decode` op's on its split path, stated over that +op's own order and padding: + +* Scores are ranked by their fp32 bit pattern through the order-preserving map + `key(x) = bits if the sign bit is set else ~bits & 0x7FFFFFFF`, ascending key first: a strict + total order on bit patterns, +NaN above +inf, +0.0 above -0.0, -inf at the bottom but for + -NaN. Two scores tie only when their bits are identical. +* The op cuts a row whose causal end is `end` into `blocks_per_row` blocks: the first + `blocks_per_row - 1` hold `end // blocks_per_row` columns and the last the rest. A block of + at most `top_k` columns passes all of them on, padded to `top_k` with entries scored + -FLT_MAX that stand for column -1; a longer one passes its own top `top_k`. A merge takes + the `top_k` best of what the blocks pass, which is the `top_k` best of the row's columns and + `pad = sum(max(0, top_k - len))` pads. + +Here the pads are virtual: `pad` entries at `key(-FLT_MAX)`, ranked below any column with +that key. A row with `end <= top_k` keeps, in column order, each column whose key is at most +`key(-FLT_MAX)`. A longer row is resolved by an exact radix select over the full 32-bit key, +digit by digit: bits 31..20 are counted over the row (read in 256-bit vectors) into a +4096-bin shared-memory histogram; the columns of the better bins are emitted and the +threshold bin's columns go to a shared-memory list, over which bits 19..10 and 9..0 are +resolved. A threshold bin too large for the list is narrowed digit by digit over the row +instead. Of the columns whose key equals the final threshold, the number still needed is +taken in no particular order, columns before pads. The selected columns are written first +and every selected pad as a -1 after them. + +The op orders one value apart from its key: its first pass bins fp16(x), and the conversion +turns a negative NaN into the canonical positive NaN, the best score, while its later passes +rank it last by key. So a row holding a negative NaN takes `_emulate`, which runs the op's +stages as the op does: each block of more than `top_k` columns selects through the fp16 pass +when that pass settles it (its threshold bin holds at most 2048 entries) and by key +otherwise, and the merge does the same over the blocks' winners and the pads. + +One launch, nothing outside shared memory, no host read: the row count is symbolic and +`blocks_per_row` is a launch argument the caller computes from the call's shape. +""" + +from __future__ import annotations + +import cuda.bindings.driver as cuda +import cutlass +import cutlass.cute as cute +from cutlass._mlir.dialects import llvm +from cutlass.utils import SmemAllocator + +TOP_K = 2048 +NUM_THREADS = 1024 +NUM_WARPS = NUM_THREADS // 32 +#: fp32 scores per 256-bit load in the first pass over the row. +VEC = 8 +#: The row's select: the key's digits 31..20 (4096 bins), 19..10 and 9..0 (1024 bins each). +D1_SHIFT, D1_MASK, D1_BINS = 20, 0xFFF, 4096 +D2_SHIFT, D2_MASK, D2_BINS = 10, 0x3FF, 1024 +D3_MASK, D3_BINS = 0x3FF, 1024 +#: Bins of the emulation path's digits (the op's): fp16, then 31..21, 20..10 and 9..0. +NUM_BINS = 2048 +#: (key, column) slots for the columns that share the threshold prefix. +LIST_CAPACITY = 4096 +#: key(-FLT_MAX): -FLT_MAX is 0xFF7FFFFF and negative, so its key is its bits. +PAD_KEY = 0xFF7FFFFF +#: The widest row the emulation path's column mask covers. +MAX_COLUMNS = 65536 +#: The op's first-pass threshold bin is resolved by sorting it when it holds at most this many. +OP_FINAL_ITEMS = 2048 +_FLT_MAX = 3.4028234663852886e38 +#: key(-inf); a larger key is a negative NaN. +NEG_INF_KEY = 0xFF800000 + +# Scalar slots in shared memory. +_N_OUT = 0 # columns in the row's output buffer +_N_LIST = 1 # entries in the list +_N_TIE = 2 # tie-group columns claimed +_THR_BIN = 3 # threshold bin of the last scan +_THR_BEFORE = 4 # elements in the bins ranked above it +_THR_COUNT = 5 # elements in it +_N_NEG_NAN = 6 # columns holding a negative NaN +_NUM_SCALARS = 8 + + +@cute.jit +def _key(x): + """The order-preserving key of an fp32 score: ascending key is descending score.""" + bits = cutlass.Uint32(llvm.bitcast(cutlass.Uint32.mlir_type, x.ir_value())) + key = bits + if (bits & cutlass.Uint32(0x80000000)) == cutlass.Uint32(0): + key = (bits ^ cutlass.Uint32(0xFFFFFFFF)) & cutlass.Uint32(0x7FFFFFFF) + return key + + +@cute.jit +def _digit(key, shift: cutlass.Constexpr[int], mask: cutlass.Constexpr[int]): + return cutlass.Int32((key >> shift) & cutlass.Uint32(mask)) + + +@cute.jit +def _fp16_bin(x): + """The op's first-pass bin of a score: its fp16 image's 11-bit ordered key. + + Only the emulation path reads it, to reproduce when the op's fp16 pass decides a + selection. The conversion is the hardware's round-to-nearest one, which turns any NaN + into the canonical positive NaN, so a -NaN lands in the top bin here.""" + bits = cutlass.Uint16(llvm.bitcast(cutlass.Uint16.mlir_type, x.to(cutlass.Float16).ir_value())) + key = bits + if (bits & cutlass.Uint16(0x8000)) == cutlass.Uint16(0): + key = (bits ^ cutlass.Uint16(0xFFFF)) & cutlass.Uint16(0x7FFF) + return cutlass.Int32((key >> 5) & cutlass.Uint16(0x7FF)) + + +class RadixTopkDecodeKernel: + """One CTA of NUM_THREADS threads per row; `next_n` is a compile-time constant.""" + + def __init__(self, next_n: int): + self.next_n = next_n + self.T = NUM_THREADS + self.W = NUM_WARPS + + # ------------------------------------------------------------------ helpers + @cute.jit + def _clear(self, s_hist, nbins: cutlass.Constexpr[int], tid): + i = tid + while i < cutlass.Int32(nbins): + s_hist[i] = cutlass.Int32(0) + i = i + cutlass.Int32(self.T) + cute.arch.barrier() + + @cute.jit + def _row_tiles(self, score, size, tid): + """The row's first `size` scores (a multiple of VEC, 32-byte aligned) as per-thread + 256-bit vectors: the copy atom, the thread's source and coordinate partitions, and + the tile count (`self.T * VEC` scores per tile).""" + copy_atom = cute.make_copy_atom( + cute.nvgpu.CopyUniversalOp(), cutlass.Float32, num_bits_per_copy=32 * VEC + ) + tiled_copy = cute.make_tiled_copy_tv( + copy_atom, + cute.make_ordered_layout((1, self.T), order=(1, 0)), + cute.make_layout((1, VEC)), + ) + row_ptr = cute.make_ptr( + cutlass.Float32, score.iterator.toint(), cute.AddressSpace.gmem, assumed_align=32 + ) + row = cute.make_tensor(row_ptr, cute.make_layout((1, size), stride=(size, 1))) + tiler = (1, self.T * VEC) + g_row = cute.local_tile(row, tiler, (0, None)) + c_row = cute.local_tile(cute.make_identity_tensor((1, size)), tiler, (0, None)) + thr = tiled_copy.get_slice(tid) + return copy_atom, thr.partition_S(g_row), thr.partition_S(c_row), g_row.shape[2] + + @cute.jit + def _predicate(self, coords, limit): + """Which of the thread's vectors in a tile start below `limit`.""" + pred = cute.make_rmem_tensor( + cute.make_layout( + ( + cute.size(coords, mode=[0, 1]), + cute.size(coords, mode=[1]), + cute.size(coords, mode=[2]), + ), + stride=(cute.size(coords, mode=[2]), 0, 1), + ), + cutlass.Boolean, + ) + for rest_v in range(pred.shape[0]): + for rest_k in range(pred.shape[2]): + pred[rest_v, 0, rest_k] = cute.elem_less(coords[(0, rest_v), 0, rest_k][1], limit) + return pred + + @cute.jit + def _count(self, s_hist, b, n): + cute.arch.atomic_add(ptr=(s_hist.iterator + b).llvm_ptr, val=n, sem="relaxed", scope="cta") + + @cute.jit + def _warp_scan(self, x): + """Inclusive prefix sum of `x` over the warp's lanes.""" + lane = cute.arch.lane_idx() + for i in cutlass.range_constexpr(5): + up = cute.arch.shuffle_sync_up(x, 1 << i, mask=0xFFFFFFFF) + if lane >= cutlass.Int32(1 << i): + x = x + up + return x + + @cute.jit + def _warp_offsets(self, s_warp, warp): + """(sum of the warp totals in `s_warp` before `warp`, sum of all of them).""" + lane = cute.arch.lane_idx() + total = cutlass.Int32(0) + if lane < cutlass.Int32(self.W): + total = s_warp[lane] + inclusive = self._warp_scan(total) + before = cute.arch.shuffle_sync(inclusive, (warp + cutlass.Int32(31)) & cutlass.Int32(31)) + if warp == cutlass.Int32(0): + before = cutlass.Int32(0) + return before, cute.arch.shuffle_sync(inclusive, 31) + + @cute.jit + def _claim(self, s_scal, slot: cutlass.Constexpr[int], contrib): + """One position of the counter `s_scal[slot]` per lane with `contrib`, one atomic per + warp. Every lane of the warp calls it; the result is meaningful where `contrib`.""" + ballot = cute.arch.vote_ballot_sync(contrib) + pos = cutlass.Int32(0) + if ballot != 0: + lane = cute.arch.lane_idx() + base = cutlass.Int32(0) + if lane == cutlass.Int32(0): + base = cute.arch.atomic_add( + ptr=(s_scal.iterator + slot).llvm_ptr, + val=cute.arch.popc(ballot), + sem="relaxed", + scope="cta", + ) + below = (cutlass.Int32(1) << lane) - cutlass.Int32(1) + pos = cute.arch.shuffle_sync(base, 0) + cute.arch.popc(ballot & below) + return pos + + @cute.jit + def _threshold(self, s_hist, s_warp, s_scal, nbins: cutlass.Constexpr[int], need, tid): + """`(t, before, count)` of the bin with `before < need <= before + count`. + + The histogram's total is at least `need`, so exactly one bin qualifies. Each thread + owns `nbins // self.T` consecutive bins; a warp shuffle scan and the warp totals + give it its exclusive prefix. Two barriers.""" + per = cutlass.const_expr(nbins // self.T) + lane = cute.arch.lane_idx() + warp = tid // cutlass.Int32(32) + base = tid * cutlass.Int32(per) + local = cutlass.Int32(0) + for j in cutlass.range_constexpr(per): + local = local + s_hist[base + cutlass.Int32(j)] + inclusive = self._warp_scan(local) + if lane == cutlass.Int32(31): + s_warp[warp] = inclusive + cute.arch.barrier() + prefix, _ = self._warp_offsets(s_warp, warp) + prefix = prefix + inclusive - local + for j in cutlass.range_constexpr(per): + count = s_hist[base + cutlass.Int32(j)] + if (prefix < need) & (prefix + count >= need): + s_scal[_THR_BIN] = base + cutlass.Int32(j) + s_scal[_THR_BEFORE] = prefix + s_scal[_THR_COUNT] = count + prefix = prefix + count + cute.arch.barrier() + return s_scal[_THR_BIN], s_scal[_THR_BEFORE], s_scal[_THR_COUNT] + + @cute.jit + def _emit_tie(self, s_out, s_scal, col, share, take): + """Write `col` for the first `take` lanes with `share` (counted over the CTA).""" + slot = self._claim(s_scal, _N_TIE, share) + win = share & (slot < take) + pos = self._claim(s_scal, _N_OUT, win) + if win: + s_out[pos] = col + + # ------------------------------------------------------------------ row of end > top_k + @cute.jit + def _select(self, score, end, pad, s_hist, s_out, s_lkey, s_lcol, s_warp, s_scal, tid): + """The top_k of the row's columns and `pad` pads, given the histogram of digit 31..20 + (pads included) in `s_hist`. Writes the selected columns to `s_out`; returns their + count.""" + lane = cute.arch.lane_idx() + warp = tid // cutlass.Int32(32) + top_k = cutlass.Int32(TOP_K) + pad_key = cutlass.Uint32(PAD_KEY) + t1, before, count = self._threshold(s_hist, s_warp, s_scal, D1_BINS, top_k, tid) + need = top_k - before # still to take among the keys sharing `prefix` + prefix = cutlass.Uint32(t1) << D1_SHIFT + shift = cutlass.Int32(D1_SHIFT) # `prefix` holds the threshold key's bits from `shift` up + pad_in = cutlass.Int32(0) # pads sharing `prefix` + if _digit(pad_key, D1_SHIFT, D1_MASK) == t1: + pad_in = pad + + if count - pad_in > cutlass.Int32(LIST_CAPACITY): + # ---- digit 19..10 over the row, inside the prefix ------------------ + self._clear(s_hist, D2_BINS, tid) + i = tid + while i < end: + key = _key(score[i]) + if (key >> D1_SHIFT) == (prefix >> D1_SHIFT): + self._count(s_hist, _digit(key, D2_SHIFT, D2_MASK), cutlass.Int32(1)) + i = i + cutlass.Int32(self.T) + if (tid == cutlass.Int32(0)) & (pad_in > cutlass.Int32(0)): + self._count(s_hist, _digit(pad_key, D2_SHIFT, D2_MASK), pad_in) + cute.arch.barrier() + t2, before, count = self._threshold(s_hist, s_warp, s_scal, D2_BINS, need, tid) + need = need - before + prefix = prefix | (cutlass.Uint32(t2) << D2_SHIFT) + shift = cutlass.Int32(D2_SHIFT) + if _digit(pad_key, D2_SHIFT, D2_MASK) != t2: + pad_in = cutlass.Int32(0) + if count - pad_in > cutlass.Int32(LIST_CAPACITY): + # ---- digit 9..0 over the row, inside the prefix ---------------- + self._clear(s_hist, D3_BINS, tid) + i = tid + while i < end: + key = _key(score[i]) + if (key >> D2_SHIFT) == (prefix >> D2_SHIFT): + self._count(s_hist, _digit(key, 0, D3_MASK), cutlass.Int32(1)) + i = i + cutlass.Int32(self.T) + if (tid == cutlass.Int32(0)) & (pad_in > cutlass.Int32(0)): + self._count(s_hist, _digit(pad_key, 0, D3_MASK), pad_in) + cute.arch.barrier() + t3, before, count = self._threshold(s_hist, s_warp, s_scal, D3_BINS, need, tid) + need = need - before + prefix = prefix | cutlass.Uint32(t3) + shift = cutlass.Int32(0) + if _digit(pad_key, 0, D3_MASK) != t3: + pad_in = cutlass.Int32(0) + + # ---- one pass over the row: emit what ranks above the prefix ---------------- + # With the threshold key known (`shift == 0`) its tie group is taken here, columns + # before pads; otherwise the columns sharing the prefix go to the list. + take = need + if count - pad_in < take: + take = count - pad_in + wbase = warp * cutlass.Int32(32) + while wbase < end: + c = wbase + lane + live = c < end + key = pad_key + if live: + key = _key(score[c]) + above = live & ((key >> shift) < (prefix >> shift)) + share = live & ((key >> shift) == (prefix >> shift)) + pos = self._claim(s_scal, _N_OUT, above) + if above: + s_out[pos] = c + if shift == cutlass.Int32(0): + self._emit_tie(s_out, s_scal, c, share, take) + else: + slot = self._claim(s_scal, _N_LIST, share) + if share: + s_lkey[slot] = key + s_lcol[slot] = c + wbase = wbase + cutlass.Int32(self.T) + cute.arch.barrier() + + if shift != cutlass.Int32(0): + n_list = s_scal[_N_LIST] + if shift == cutlass.Int32(D1_SHIFT): + # ---- digit 19..10 over the list -------------------------------- + self._clear(s_hist, D2_BINS, tid) + j = tid + while j < n_list: + self._count(s_hist, _digit(s_lkey[j], D2_SHIFT, D2_MASK), cutlass.Int32(1)) + j = j + cutlass.Int32(self.T) + if (tid == cutlass.Int32(0)) & (pad_in > cutlass.Int32(0)): + self._count(s_hist, _digit(pad_key, D2_SHIFT, D2_MASK), pad_in) + cute.arch.barrier() + t2, before, count = self._threshold(s_hist, s_warp, s_scal, D2_BINS, need, tid) + need = need - before + prefix = prefix | (cutlass.Uint32(t2) << D2_SHIFT) + if _digit(pad_key, D2_SHIFT, D2_MASK) != t2: + pad_in = cutlass.Int32(0) + # ---- digit 9..0 over the list ------------------------------------------ + self._clear(s_hist, D3_BINS, tid) + j = tid + while j < n_list: + key = s_lkey[j] + if (key >> D2_SHIFT) == (prefix >> D2_SHIFT): + self._count(s_hist, _digit(key, 0, D3_MASK), cutlass.Int32(1)) + j = j + cutlass.Int32(self.T) + if (tid == cutlass.Int32(0)) & (pad_in > cutlass.Int32(0)): + self._count(s_hist, _digit(pad_key, 0, D3_MASK), pad_in) + cute.arch.barrier() + t3, before, count = self._threshold(s_hist, s_warp, s_scal, D3_BINS, need, tid) + need = need - before + thr = prefix | cutlass.Uint32(t3) + if _digit(pad_key, 0, D3_MASK) != t3: + pad_in = cutlass.Int32(0) + take = need + if count - pad_in < take: + take = count - pad_in + # Above the threshold key: selected; equal to it: `take` of them. + lbase = warp * cutlass.Int32(32) + while lbase < n_list: + j = lbase + lane + live = j < n_list + key = thr + col = cutlass.Int32(-1) + if live: + key = s_lkey[j] + col = s_lcol[j] + above = live & (key < thr) + share = live & (key == thr) + pos = self._claim(s_scal, _N_OUT, above) + if above: + s_out[pos] = col + self._emit_tie(s_out, s_scal, col, share, take) + lbase = lbase + cutlass.Int32(self.T) + cute.arch.barrier() + return s_scal[_N_OUT] + + # ------------------------------------------------------------------ emulation + @cute.jit + def _in_mask(self, s_mask, c): + word = s_mask[c >> cutlass.Int32(5)] + return ((word >> cutlass.Uint32(c & cutlass.Int32(31))) & cutlass.Uint32(1)) != 0 + + @cute.jit + def _mark_range(self, s_mask, lo, hi, tid): + """Set the mask bits of columns [lo, hi), one word per thread at a time.""" + w = (lo >> cutlass.Int32(5)) + tid + while w * cutlass.Int32(32) < hi: + first = w * cutlass.Int32(32) + a = lo - first + if a < cutlass.Int32(0): + a = cutlass.Int32(0) + b = hi - first + if b > cutlass.Int32(32): + b = cutlass.Int32(32) + upper = cutlass.Uint32(0xFFFFFFFF) + if b < cutlass.Int32(32): + upper = (cutlass.Uint32(1) << cutlass.Uint32(b)) - cutlass.Uint32(1) + lower = (cutlass.Uint32(1) << cutlass.Uint32(a)) - cutlass.Uint32(1) + s_mask[w] = s_mask[w] | (upper & (lower ^ cutlass.Uint32(0xFFFFFFFF))) + w = w + cutlass.Int32(self.T) + cute.arch.barrier() + + @cute.jit + def _stage( + self, + score, + lo, + hi, + use_mask: cutlass.Constexpr[bool], + pad, + emit: cutlass.Constexpr[bool], + s_hist, + s_out, + s_mask, + s_warp, + s_scal, + tid, + ): + """One selection of the op, as its kernel makes it: the top_k of the columns in + [lo, hi) (with `use_mask`, only those whose mask bit is set) and `pad` pads. + + Its first pass bins every candidate by `_fp16_bin` and finds the threshold bin. When + that bin holds at most OP_FINAL_ITEMS candidates the op takes every better bin whole + and sorts the threshold bin by key; otherwise it selects by key over all candidates. + Either way the key select runs here as three digit passes over the row. The winners + are set in `s_mask` (`emit` False) or written to `s_out` (`emit` True).""" + lane = cute.arch.lane_idx() + warp = tid // cutlass.Int32(32) + top_k = cutlass.Int32(TOP_K) + pad_key = cutlass.Uint32(PAD_KEY) + pad_bin = _fp16_bin(cutlass.Float32(-_FLT_MAX)) + if tid == cutlass.Int32(0): + s_scal[_N_TIE] = cutlass.Int32(0) + self._clear(s_hist, NUM_BINS, tid) + c = lo + tid + while c < hi: + ok = cutlass.Boolean(True) + if cutlass.const_expr(use_mask): + ok = self._in_mask(s_mask, c) + if ok: + self._count(s_hist, _fp16_bin(score[c]), cutlass.Int32(1)) + c = c + cutlass.Int32(self.T) + if (tid == cutlass.Int32(0)) & (pad > cutlass.Int32(0)): + self._count(s_hist, pad_bin, pad) + cute.arch.barrier() + t0, before, count = self._threshold(s_hist, s_warp, s_scal, NUM_BINS, top_k, tid) + resolved = count <= cutlass.Int32(OP_FINAL_ITEMS) + need = top_k + pad_in = pad + if resolved: + need = top_k - before + if pad_bin != t0: + pad_in = cutlass.Int32(0) + + prefix = cutlass.Uint32(0) + for d in cutlass.range_constexpr(3): + shift = cutlass.const_expr((21, 10, 0)[d]) + mask = cutlass.const_expr((0x7FF, 0x7FF, 0x3FF)[d]) + nbins = cutlass.const_expr((NUM_BINS, NUM_BINS, NUM_BINS // 2)[d]) + self._clear(s_hist, nbins, tid) + c = lo + tid + while c < hi: + ok = cutlass.Boolean(True) + if cutlass.const_expr(use_mask): + ok = self._in_mask(s_mask, c) + if ok: + x = score[c] + key = _key(x) + if resolved: + ok = _fp16_bin(x) == t0 + if cutlass.const_expr(d > 0): + above_shift = cutlass.const_expr((21, 10)[d - 1]) + ok = ok & ((key >> above_shift) == (prefix >> above_shift)) + if ok: + self._count(s_hist, _digit(key, shift, mask), cutlass.Int32(1)) + c = c + cutlass.Int32(self.T) + if (tid == cutlass.Int32(0)) & (pad_in > cutlass.Int32(0)): + self._count(s_hist, _digit(pad_key, shift, mask), pad_in) + cute.arch.barrier() + t, before, count = self._threshold(s_hist, s_warp, s_scal, nbins, need, tid) + need = need - before + prefix = prefix | (cutlass.Uint32(t) << shift) + if _digit(pad_key, shift, mask) != t: + pad_in = cutlass.Int32(0) + take = need + if count - pad_in < take: + take = count - pad_in + + wbase = (lo & cutlass.Int32(-32)) + warp * cutlass.Int32(32) + while wbase < hi: + c = wbase + lane + live = (c >= lo) & (c < hi) + if cutlass.const_expr(use_mask): + live = live & self._in_mask(s_mask, c) + key = pad_key + h = cutlass.Int32(0) + if live: + x = score[c] + key = _key(x) + h = _fp16_bin(x) + inbin = live + above = live & (key < prefix) + if resolved: + inbin = live & (h == t0) + above = (live & (h < t0)) | (inbin & (key < prefix)) + tie = inbin & (key == prefix) + slot = self._claim(s_scal, _N_TIE, tie) + chosen = above | (tie & (slot < take)) + if cutlass.const_expr(emit): + pos = self._claim(s_scal, _N_OUT, chosen) + if chosen: + s_out[pos] = c + else: + bits = cute.arch.vote_ballot_sync(chosen) + if lane == cutlass.Int32(0): + s_mask[wbase >> cutlass.Int32(5)] = s_mask[wbase >> cutlass.Int32(5)] | bits + wbase = wbase + cutlass.Int32(self.T) + cute.arch.barrier() + + @cute.jit + def _emulate(self, score, end, blocks_per_row, pad, s_hist, s_out, s_mask, s_warp, s_scal, tid): + """The op's two launches, stage by stage, for a row holding a negative NaN. + + That is the one value whose rank the op's fp16 first pass changes (the conversion + drops its sign), so only such a row needs this path. Each block of more than top_k + columns selects its top_k as the op's first launch does; a shorter one passes all its + columns. The merge then selects over those and the pads. Returns the columns written.""" + top_k = cutlass.Int32(TOP_K) + w = tid + while w < cutlass.Int32(MAX_COLUMNS // 32): + s_mask[w] = cutlass.Uint32(0) + w = w + cutlass.Int32(self.T) + cute.arch.barrier() + block = end // blocks_per_row + j = cutlass.Int32(0) + while j < blocks_per_row: + lo = j * block + hi = lo + block + if j == blocks_per_row - cutlass.Int32(1): + hi = end + if hi - lo <= top_k: + self._mark_range(s_mask, lo, hi, tid) + else: + self._stage( + score, lo, hi, False, cutlass.Int32(0), False, + s_hist, s_out, s_mask, s_warp, s_scal, tid, + ) # fmt: skip + j = j + cutlass.Int32(1) + self._stage(score, cutlass.Int32(0), end, True, pad, True, s_hist, s_out, s_mask, s_warp, + s_scal, tid) # fmt: skip + return s_scal[_N_OUT] + + # ------------------------------------------------------------------ kernel + @cute.kernel + def kernel( + self, + logits: cute.Tensor, + seq_lens: cute.Tensor, + out: cute.Tensor, + blocks_per_row: cutlass.Int32, + ): + row, _, _ = cute.arch.block_idx() + tid, _, _ = cute.arch.thread_idx() + lane = cute.arch.lane_idx() + warp = tid // cutlass.Int32(32) + next_n = cutlass.const_expr(self.next_n) + top_k = cutlass.Int32(TOP_K) + pad_key = cutlass.Uint32(PAD_KEY) + neg_inf_key = cutlass.Uint32(NEG_INF_KEY) + + smem = SmemAllocator() + s_hist = smem.allocate_tensor(cutlass.Int32, cute.make_layout((D1_BINS,)), 128) + s_out = smem.allocate_tensor(cutlass.Int32, cute.make_layout((TOP_K,)), 128) + s_lkey = smem.allocate_tensor(cutlass.Uint32, cute.make_layout((LIST_CAPACITY,)), 128) + s_lcol = smem.allocate_tensor(cutlass.Int32, cute.make_layout((LIST_CAPACITY,)), 128) + s_mask = smem.allocate_tensor(cutlass.Uint32, cute.make_layout((MAX_COLUMNS // 32,)), 128) + s_warp = smem.allocate_tensor(cutlass.Int32, cute.make_layout((self.W,)), 128) + s_scal = smem.allocate_tensor(cutlass.Int32, cute.make_layout((_NUM_SCALARS,)), 128) + + # The causal end as the op computes it (compress ratio 1), held inside the row. + cols = cute.size(logits, mode=[1]) + end = seq_lens[row // cutlass.Int32(next_n)] - cutlass.Int32(next_n) + end = end + row % cutlass.Int32(next_n) + cutlass.Int32(1) + if end > cols: + end = cols + if end < cutlass.Int32(0): + end = cutlass.Int32(0) + score = logits[row, None] + dst = out[row, None] + + # The op's pad entries: each of its blocks of at most top_k columns pads to top_k. + block = end // blocks_per_row + last = end - block * (blocks_per_row - cutlass.Int32(1)) + pad = cutlass.Int32(0) + if block < top_k: + pad = (blocks_per_row - cutlass.Int32(1)) * (top_k - block) + if last < top_k: + pad = pad + top_k - last + + if tid < cutlass.Int32(_NUM_SCALARS): + s_scal[tid] = cutlass.Int32(0) + cute.arch.barrier() + + # Each path's first pass also counts the negative NaNs; a row with one goes through + # the op's stages instead (`_emulate`). + n_sel = cutlass.Int32(0) + emulate = cutlass.Boolean(False) + if end <= top_k: + # At least top_k pads: every column whose key is at most key(-FLT_MAX) is kept, + # in column order (`per` consecutive columns per thread); a column below it + # loses to a pad. + per = cutlass.const_expr(TOP_K // self.T) + c0 = tid * cutlass.Int32(per) + keep = cute.make_rmem_tensor(cute.make_layout((per,)), cutlass.Int32) + neg = cutlass.Int32(0) + local = cutlass.Int32(0) + for e in cutlass.range_constexpr(per): + keep[e] = cutlass.Int32(0) + if c0 + cutlass.Int32(e) < end: + key = _key(score[c0 + cutlass.Int32(e)]) + if key <= pad_key: + keep[e] = cutlass.Int32(1) + if key > neg_inf_key: + neg = neg + cutlass.Int32(1) + local = local + keep[e] + if neg > cutlass.Int32(0): + self._count(s_scal, cutlass.Int32(_N_NEG_NAN), neg) + inclusive = self._warp_scan(local) + if lane == cutlass.Int32(31): + s_warp[warp] = inclusive + cute.arch.barrier() + if s_scal[_N_NEG_NAN] == cutlass.Int32(0): + pos0, total = self._warp_offsets(s_warp, warp) + pos0 = pos0 + inclusive - local + for e in cutlass.range_constexpr(per): + if keep[e] == cutlass.Int32(1): + s_out[pos0] = c0 + cutlass.Int32(e) + pos0 = pos0 + keep[e] + n_sel = total + else: + emulate = cutlass.Boolean(True) + else: + # ---- digit 31..20 over the row ----------------------------------- + self._clear(s_hist, D1_BINS, tid) + neg = cutlass.Int32(0) + size = (end + cutlass.Int32(VEC - 1)) // cutlass.Int32(VEC) * cutlass.Int32(VEC) + atom, t_src, t_crd, tiles = self._row_tiles(score, size, tid) + t_val = cute.make_fragment_like(t_src[None, None, None, 0]) + for tile in range(tiles): + pred = self._predicate(t_crd[None, None, None, tile], size) + cute.copy(atom, t_src[None, None, None, tile], t_val, pred=pred[None, None, None]) + crd = t_crd[(0, None), None, None, tile] + for v in cutlass.range(cute.size(t_val), unroll_full=True): + c = crd[v // VEC][1] + v % VEC + if pred[0, v // VEC, 0] & (c < end): + key = _key(t_val[v]) + self._count(s_hist, _digit(key, D1_SHIFT, D1_MASK), cutlass.Int32(1)) + if key > neg_inf_key: + neg = neg + cutlass.Int32(1) + if neg > cutlass.Int32(0): + self._count(s_scal, cutlass.Int32(_N_NEG_NAN), neg) + if (tid == cutlass.Int32(0)) & (pad > cutlass.Int32(0)): + self._count(s_hist, _digit(pad_key, D1_SHIFT, D1_MASK), pad) + cute.arch.barrier() + if s_scal[_N_NEG_NAN] != cutlass.Int32(0): + emulate = cutlass.Boolean(True) + else: + n_sel = self._select( + score, end, pad, s_hist, s_out, s_lkey, s_lcol, s_warp, s_scal, tid + ) + if emulate: + n_sel = self._emulate( + score, end, blocks_per_row, pad, s_hist, s_out, s_mask, s_warp, s_scal, tid + ) + cute.arch.barrier() + + # The selected columns, then -1 for every selected pad. + k = tid + while k < top_k: + v = cutlass.Int32(-1) + if k < n_sel: + v = s_out[k] + dst[k] = v + k = k + cutlass.Int32(self.T) + + @cute.jit + def __call__( + self, + logits: cute.Tensor, + seq_lens: cute.Tensor, + out: cute.Tensor, + blocks_per_row: cutlass.Int32, + stream: cuda.CUstream, + ): + rows = cute.size(logits, mode=[0]) + self.kernel(logits, seq_lens, out, blocks_per_row).launch( + grid=(rows, 1, 1), block=(self.T, 1, 1), stream=stream + ) + + +class SplitRadixTopkDecodeKernel(RadixTopkDecodeKernel): + """Partition selection and exact radix merge, with caller-owned candidate storage. + + A partition's first pass reads its scores as 256-bit vectors from the 8-column boundary + at or below its start, so the logits rows start 32-byte aligned and every row is readable + through its width rounded up to a multiple of VEC. The merge reads its candidates one + score per thread.""" + + def __init__(self, next_n: int, partitions: int, use_pdl: bool): + super().__init__(next_n) + self.partitions = partitions + self.use_pdl = use_pdl + + @cute.jit + def _begin_select(self, s_hist, s_scal, tid): + if tid < cutlass.Int32(_NUM_SCALARS): + s_scal[tid] = cutlass.Int32(0) + self._clear(s_hist, D1_BINS, tid) + + @cute.jit + def _count_aligned(self, src, start, length, s_hist, s_scal, tid): + """Digit 31..20 of `src[start:start + length]` into `s_hist` and its negative NaNs + into `s_scal`, read as 256-bit vectors from column `start` rounded down to VEC.""" + lo = start - start % cutlass.Int32(VEC) + head = start - lo + stop = head + length + size = (stop + cutlass.Int32(VEC - 1)) // cutlass.Int32(VEC) * cutlass.Int32(VEC) + window = cute.make_tensor(src.iterator + lo, cute.make_layout((size,))) + neg = cutlass.Int32(0) + atom, t_src, t_crd, tiles = self._row_tiles(window, size, tid) + t_val = cute.make_fragment_like(t_src[None, None, None, 0]) + for tile in range(tiles): + pred = self._predicate(t_crd[None, None, None, tile], size) + cute.copy(atom, t_src[None, None, None, tile], t_val, pred=pred[None, None, None]) + crd = t_crd[(0, None), None, None, tile] + for v in cutlass.range(cute.size(t_val), unroll_full=True): + c = crd[v // VEC][1] + v % VEC + if pred[0, v // VEC, 0] & (c >= head) & (c < stop): + key = _key(t_val[v]) + self._count(s_hist, _digit(key, D1_SHIFT, D1_MASK), cutlass.Int32(1)) + if key > cutlass.Uint32(NEG_INF_KEY): + neg = neg + cutlass.Int32(1) + if neg > cutlass.Int32(0): + self._count(s_scal, cutlass.Int32(_N_NEG_NAN), neg) + + @cute.jit + def _select_partition(self, score, end, s_hist, s_out, s_lkey, s_lcol, s_warp, s_scal, tid): + """Select a native stage's candidates, including its negative-NaN ordering.""" + self._begin_select(s_hist, s_scal, tid) + neg = cutlass.Int32(0) + col = tid + while col < end: + key = _key(score[col]) + self._count(s_hist, _digit(key, D1_SHIFT, D1_MASK), cutlass.Int32(1)) + if key > cutlass.Uint32(NEG_INF_KEY): + neg = neg + cutlass.Int32(1) + col = col + cutlass.Int32(self.T) + if neg > cutlass.Int32(0): + self._count(s_scal, cutlass.Int32(_N_NEG_NAN), neg) + self._select_counted(score, end, s_hist, s_out, s_lkey, s_lcol, s_warp, s_scal, tid) + + @cute.jit + def _select_counted(self, score, end, s_hist, s_out, s_lkey, s_lcol, s_warp, s_scal, tid): + """Finish a stage whose digit 31..20 and negative NaNs have been counted.""" + cute.arch.barrier() + if s_scal[_N_NEG_NAN] == cutlass.Int32(0): + self._select( + score, + end, + cutlass.Int32(0), + s_hist, + s_out, + s_lkey, + s_lcol, + s_warp, + s_scal, + tid, + ) + else: + # No mask is used by a standalone native selection stage. + self._stage( + score, + cutlass.Int32(0), + end, + False, + cutlass.Int32(0), + True, + s_hist, + s_out, + s_hist, + s_warp, + s_scal, + tid, + ) + cute.arch.barrier() + + @cute.kernel + def split_kernel( + self, + logits: cute.Tensor, + seq_lens: cute.Tensor, + aux_indices: cute.Tensor, + aux_logits: cute.Tensor, + ): + if cutlass.const_expr(self.use_pdl): + cute.arch.griddepcontrol_wait() + row, part, _ = cute.arch.block_idx() + tid, _, _ = cute.arch.thread_idx() + smem = SmemAllocator() + s_hist = smem.allocate_tensor(cutlass.Int32, cute.make_layout((D1_BINS,)), 128) + s_out = smem.allocate_tensor(cutlass.Int32, cute.make_layout((TOP_K,)), 128) + s_lkey = smem.allocate_tensor(cutlass.Uint32, cute.make_layout((LIST_CAPACITY,)), 128) + s_lcol = smem.allocate_tensor(cutlass.Int32, cute.make_layout((LIST_CAPACITY,)), 128) + s_warp = smem.allocate_tensor(cutlass.Int32, cute.make_layout((self.W,)), 128) + s_scal = smem.allocate_tensor(cutlass.Int32, cute.make_layout((_NUM_SCALARS,)), 128) + cols = cute.size(logits, mode=[1]) + end = seq_lens[row // cutlass.Int32(self.next_n)] - cutlass.Int32(self.next_n) + end = end + row % cutlass.Int32(self.next_n) + cutlass.Int32(1) + if (end < cutlass.Int32(0)) | (end > cols): + end = cutlass.Int32(0) + span = end // cutlass.Int32(self.partitions) + start = part * span + stop = start + span + if part == cutlass.Int32(self.partitions - 1): + stop = end + length = stop - start + src = logits[row, None] + score = cute.make_tensor(src.iterator + start, cute.make_layout((length,))) + dst = (row * cutlass.Int32(self.partitions) + part) * cutlass.Int32(TOP_K) + if length > cutlass.Int32(TOP_K): + self._begin_select(s_hist, s_scal, tid) + self._count_aligned(src, start, length, s_hist, s_scal, tid) + self._select_counted(score, length, s_hist, s_out, s_lkey, s_lcol, s_warp, s_scal, tid) + k = tid + while k < cutlass.Int32(TOP_K): + col = cutlass.Int32(-1) + value = cutlass.Float32(-_FLT_MAX) + if length > cutlass.Int32(TOP_K): + col = s_out[k] + value = score[col] + col = col + start + elif k < length: + col = start + k + value = score[k] + aux_indices[dst + k] = col + aux_logits[dst + k] = value + k = k + cutlass.Int32(self.T) + + @cute.kernel + def merge_kernel(self, aux_indices: cute.Tensor, aux_logits: cute.Tensor, out: cute.Tensor): + if cutlass.const_expr(self.use_pdl): + cute.arch.griddepcontrol_wait() + row, _, _ = cute.arch.block_idx() + tid, _, _ = cute.arch.thread_idx() + smem = SmemAllocator() + s_hist = smem.allocate_tensor(cutlass.Int32, cute.make_layout((D1_BINS,)), 128) + s_out = smem.allocate_tensor(cutlass.Int32, cute.make_layout((TOP_K,)), 128) + s_lkey = smem.allocate_tensor(cutlass.Uint32, cute.make_layout((LIST_CAPACITY,)), 128) + s_lcol = smem.allocate_tensor(cutlass.Int32, cute.make_layout((LIST_CAPACITY,)), 128) + s_warp = smem.allocate_tensor(cutlass.Int32, cute.make_layout((self.W,)), 128) + s_scal = smem.allocate_tensor(cutlass.Int32, cute.make_layout((_NUM_SCALARS,)), 128) + length = cutlass.Int32(self.partitions * TOP_K) + base = row * length + score = cute.make_tensor(aux_logits.iterator + base, cute.make_layout((length,))) + self._select_partition(score, length, s_hist, s_out, s_lkey, s_lcol, s_warp, s_scal, tid) + # Candidate padding participates in selection, but the output contract keeps + # every valid column before the -1 suffix. Compact within this merge launch. + if tid == cutlass.Int32(0): + s_scal[_N_OUT] = cutlass.Int32(0) + k = tid + while k < cutlass.Int32(TOP_K): + out[row, k] = cutlass.Int32(-1) + k = k + cutlass.Int32(self.T) + cute.arch.barrier() + k = tid + while k < cutlass.Int32(TOP_K): + col = aux_indices[base + s_out[k]] + valid = col >= cutlass.Int32(0) + pos = self._claim(s_scal, _N_OUT, valid) + if valid: + out[row, pos] = col + k = k + cutlass.Int32(self.T) + + @cute.jit + def split( + self, + logits: cute.Tensor, + seq_lens: cute.Tensor, + aux_indices: cute.Tensor, + aux_logits: cute.Tensor, + stream: cuda.CUstream, + ): + self.split_kernel(logits, seq_lens, aux_indices, aux_logits).launch( + grid=(cute.size(logits, mode=[0]), self.partitions, 1), + block=(self.T, 1, 1), + stream=stream, + use_pdl=self.use_pdl, + ) + + @cute.jit + def merge( + self, + aux_indices: cute.Tensor, + aux_logits: cute.Tensor, + out: cute.Tensor, + stream: cuda.CUstream, + ): + self.merge_kernel(aux_indices, aux_logits, out).launch( + grid=(cute.size(out, mode=[0]), 1, 1), + block=(self.T, 1, 1), + stream=stream, + use_pdl=self.use_pdl, + ) diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/kernels/_tp4_allreduce_fused_rms_kernel.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/kernels/_tp4_allreduce_fused_rms_kernel.py new file mode 100644 index 000000000000..68833aa7c56b --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/kernels/_tp4_allreduce_fused_rms_kernel.py @@ -0,0 +1,4742 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""TP allreduce and fused residual/RMSNorm, CuTe DSL kernels. + +These kernels implement the tensor-parallel output paths that follow local +projection GEMMs. Each TP rank owns a BF16 partial hidden vector for the same +token rows. The pure kernel only performs the TP sum. The fused kernel first +sums those partial vectors across TP ranks with a one-shot Lamport allreduce, +then immediately applies the residual add and the row-wise RMSNorm that consume +the reduced hidden state. + +The pure kernel computes, per rank: + + input_local[M, 6144] + -> one-shot Lamport allreduce across four TP ranks + -> output[M, 6144] + +The fused kernel computes, per rank: + + input_local[M, 6144] + -> one-shot Lamport allreduce across four TP ranks + -> + residual[M, 6144] + -> residual_output[M, 6144] + -> per-row RMSNorm with norm_weight[6144] + -> norm_output[M, 6144] + +The fused-out-quant variant keeps the same BF16 outputs and also emits the +linear-layout NVFP4 activation payload an NVFP4 MoE consumes: + + norm_output[M, 6144] + -> fp4_output[M, 3072] + scale_output[M, 384] + +The kernels run on four GPUs with symmetric peer memory. They compile through +TVM FFI, so the environment needs ``apache-tvm-ffi`` installed. + +The default path keeps payload data and readiness state separate: BF16 vectors +move through the Lamport payload buffers, while int32 ready flags keyed by +``(slot, source_rank, vector_index)`` publish vector availability. The split +row fused path can also compile an experimental in-band sentinel variant that +uses bitwise BF16 ``-0.0`` in the payload buffer as the not-ready marker, closer +to TensorRT-LLM's Lamport backend. + +Runs on SM100+ (Blackwell) GPUs. Rows are BF16 with a runtime M; the served +configuration is N=6144 with a TP world size of 4. +""" + +import cuda.bindings.driver as cuda +import cutlass +import cutlass.cute as cute +from cutlass.cute.runtime import make_fake_compact_tensor, make_fake_stream +from cutlass.experimental import primitives as prims + +try: + from cutlass._mlir import ir as _ir + from cutlass._mlir.dialects import llvm as _llvm + from cutlass.cutlass_dsl import T as _T + from cutlass.cutlass_dsl import dsl_user_op as _dsl_user_op +except ImportError: # pragma: no cover - older DSL packages + _ir = None + _llvm = None + _T = None + _dsl_user_op = None + + +_DEFAULT_N = 6144 +_THREADS_PER_BLOCK = 128 +_WARPS_PER_BLOCK = _THREADS_PER_BLOCK // 32 +_FUSED_CTAS_PER_ROW = 4 +_SPLIT_ROW_THREADS_PER_BLOCK = 192 +_PURE_THREADS_PER_BLOCK_CHOICES = (128, 192) +_FUSED_THREADS_PER_BLOCK_CHOICES = (128, 192, 256, 384) +_OUT_QUANT_THREADS_PER_BLOCK_CHOICES = (128, 192, 256, 384) +_PING_PONG_SIZE = 3 +_BF16_VEC_SIZE = 8 +_SCALE_BLOCK = 16 +_ALIGNMENT_BYTES = 16 +_READY_FLAG_ALIGNMENT_BYTES = 4 +_NEGATIVE_ZERO_BF16_BITS = -32768 +_RMS_EPS = 1e-6 +_USES_READY_FLAGS = True +_USES_DEVICE_SIGNAL_STATE = True +_PDL_RELEASE_POLICIES = ("strict", "output_safe", "pre_norm", "entry") +_FUSED_PDL_RELEASE_POLICIES = ( + "strict", + "output_safe", + "pre_norm", + "entry", +) +_SIGNAL_COUNTER_INDEX = 0 +_SIGNAL_VALUE_INDEX = 1 +_SIGNAL_SLOT_COUNTS_INDEX = 2 +_SIGNAL_RELEASE_EPOCH_INDEX = _SIGNAL_SLOT_COUNTS_INDEX + _PING_PONG_SIZE +_SIGNAL_STATE_SIZE = _SIGNAL_RELEASE_EPOCH_INDEX + 1 + + +def _validate_multicast_publish( + multicast_publish: bool, use_ready_flags: bool, split_rows: bool +) -> None: + if not multicast_publish: + return + if use_ready_flags: + raise ValueError( + "multicast publish requires sentinel readiness (the multicast store " + "carries the payload and the not-ready marker in one 16 B write)" + ) + if not split_rows: + raise ValueError("multicast publish is implemented only for split-row fused") + if _dsl_user_op is None or _llvm is None: + raise RuntimeError("multicast publish requires cutlass._mlir.dialects.llvm") + + +def _validate_ctas_per_row(ctas_per_row: int) -> None: + if ctas_per_row not in (2, 4, 8): + raise ValueError(f"ctas_per_row must be one of 2, 4, or 8, got {ctas_per_row}") + + +def _validate_signal_advance_flags( + last_cta_signal_advance: bool, + aggregate_signal_at_cluster_root: bool, + use_device_signal_state: bool, + use_pdl: bool, + pdl_release_policy: str, +) -> None: + if not last_cta_signal_advance: + return + if not use_device_signal_state: + raise ValueError("last-CTA signal advance requires device signal state") + if aggregate_signal_at_cluster_root: + raise ValueError( + "last-CTA signal advance and cluster-root signal aggregation are mutually exclusive" + ) + if use_pdl and pdl_release_policy == "strict": + raise ValueError( + "last-CTA signal advance does not implement the strict PDL " + "release epoch; use output_safe, pre_norm, or entry" + ) + + +def _validate_pdl_release_policy(policy: str) -> None: + if policy not in _PDL_RELEASE_POLICIES: + raise ValueError(f"pdl_release_policy must be 'strict' or 'output_safe', got {policy!r}") + + +def _validate_fused_pdl_release_policy(policy: str) -> None: + if policy not in _FUSED_PDL_RELEASE_POLICIES: + raise ValueError( + "fused RMSNorm pdl_release_policy must be 'strict', 'output_safe', " + f"'pre_norm', or 'entry', got {policy!r}" + ) + + +@cute.jit +def _warp_sum_f32(value): + value = value + cute.arch.shuffle_sync_bfly(value, 16) + value = value + cute.arch.shuffle_sync_bfly(value, 8) + value = value + cute.arch.shuffle_sync_bfly(value, 4) + value = value + cute.arch.shuffle_sync_bfly(value, 2) + return value + cute.arch.shuffle_sync_bfly(value, 1) + + +@cute.jit +def _advance_signal_if_last_cta( + state_counter_ptr, + state_signal_ptr, + signal_state, + signal_i32, + ping_slot, + active_vectors, + grid_x, +): + """Last-arrival Lamport state advance (replaces the bidx0 tail spin). + + Every CTA reads ``signal`` before calling this, then one thread per CTA + performs a returning ``atom.add`` on the entry counter. The CTA that + observes ``grid_x - 1`` is the last one to enter, so every CTA of this + launch has already snapshotted the current signal and the slot state can + be advanced immediately. No CTA ever spins on the counter and the scheme + is independent of the launch grid size (M may change between launches). + """ + entered_before = prims.atomicrmw( + prims.AtomicOp.ADD, + state_counter_ptr, + cutlass.Int32(1), + mem_order=prims.MemOrder.ACQ_REL, + syncscope=prims.MemScope.GPU, + ) + if entered_before == grid_x - cutlass.Int32(1): + ping_count_ptr = ( + signal_state.iterator + + cutlass.Int64(_SIGNAL_SLOT_COUNTS_INDEX) + + cutlass.Int64(ping_slot) + ).llvm_ptr + next_signal = (signal_i32 + cutlass.Int32(1)) % cutlass.Int32(_PING_PONG_SIZE) + cute.arch.store( + ping_count_ptr, + active_vectors, + sem="release", + scope="gpu", + ) + cute.arch.store( + state_counter_ptr, + cutlass.Int32(0), + sem="release", + scope="gpu", + ) + cute.arch.store( + state_signal_ptr, + next_signal, + sem="release", + scope="gpu", + ) + + +@cute.jit +def _publish_cta_partial_to_cluster(cluster_partials, cta_sum, cta_rank, CTAS_PER_ROW): + """Store this CTA's sum-of-squares partial into slot ``cta_rank`` of every + CTA in the cluster (including itself) through distributed shared memory. + + Each slot has exactly one writer, so no zero-initialisation or atomics are + needed and a single release/acquire cluster barrier afterwards makes all + partials visible everywhere. + """ + for peer_cta_rank in cutlass.range_constexpr(CTAS_PER_ROW): + for slot in cutlass.range_constexpr(CTAS_PER_ROW): + if cta_rank == slot: + peer_slot = prims.mapa( + cluster_partials.data_ptr() + slot, + cutlass.Int32(peer_cta_rank), + ) + peer_slot.store(cta_sum) + + +@cute.jit +def _cluster_inv_rms_from_partials(cluster_partials, CTAS_PER_ROW, N, eps): + """Every CTA sums the CTAS_PER_ROW partials in the same fixed order, so all + CTAs of a row compute a bit-identical inverse RMS without a broadcast.""" + cluster_sum = cutlass.Float32(0.0) + for slot in cutlass.range_constexpr(CTAS_PER_ROW): + cluster_sum = cluster_sum + cluster_partials[slot] + return cute.math.rsqrt( + cluster_sum * (1.0 / cutlass.Float32(N)) + eps, + fastmath=True, + ) + + +if _dsl_user_op is not None: + + @_dsl_user_op + def _multimem_st_v4_b32(mc_addr, word0, word1, word2, word3, *, loc=None, ip=None): + """``multimem.st.weak.global.v4`` of four b32 words to a multicast address. + + ``mc_addr`` is the 64-bit multicast virtual address (an ``Int64``); the + NVSwitch fabric replicates the 16 B store into every rank's copy of the + symmetric buffer, so one instruction replaces WORLD_SIZE-1 unicast + remote stores. Register bits are moved as-is (``.f32`` is a bit carrier). + """ + mc_ptr_int = mc_addr.ir_value(loc=loc, ip=ip) + _llvm.inline_asm( + None, + [ + mc_ptr_int, + word0.ir_value(loc=loc, ip=ip), + word1.ir_value(loc=loc, ip=ip), + word2.ir_value(loc=loc, ip=ip), + word3.ir_value(loc=loc, ip=ip), + ], + "multimem.st.weak.global.v4.f32 [$0], {$1, $2, $3, $4};", + "l,r,r,r,r,~{memory}", + has_side_effects=True, + asm_dialect=0, + loc=loc, + ip=ip, + ) + +else: # pragma: no cover - older DSL packages + + def _multimem_st_v4_b32(*args, **kwargs): + raise RuntimeError("multimem store requires cutlass._mlir.dialects.llvm") + + +@cute.jit +def _multicast_publish_bf16x8(mc_addr, values): + """Pack eight BF16 lanes into four b32 words and multicast-store them.""" + mask = cutlass.Int32(0xFFFF) + words = [cutlass.Int32(0), cutlass.Int32(0), cutlass.Int32(0), cutlass.Int32(0)] + for pair in cutlass.range_constexpr(_BF16_VEC_SIZE // 2): + low = values[2 * pair].bitcast(cutlass.Int16).to(cutlass.Int32) & mask + high = values[2 * pair + 1].bitcast(cutlass.Int16).to(cutlass.Int32) & mask + words[pair] = low | (high << cutlass.Int32(16)) + _multimem_st_v4_b32(mc_addr, words[0], words[1], words[2], words[3]) + + +@cute.jit +def _all_bf16x8_lanes_ready(values): + ready = cutlass.Boolean(True) + neg_zero_bits = cutlass.Int16(_NEGATIVE_ZERO_BF16_BITS) + for lane in cutlass.range_constexpr(_BF16_VEC_SIZE): + ready = ready & (values[lane].bitcast(cutlass.Int16) != neg_zero_bits) + return ready + + +@cute.jit +def _decode_positive_e4m3(code: cutlass.Int32) -> cutlass.Float32: + """Decode a finite non-negative E4M3 byte to FP32.""" + exponent = (code >> cutlass.Int32(3)) & cutlass.Int32(0xF) + mantissa = code & cutlass.Int32(0x7) + value = cutlass.Float32(mantissa) * cutlass.Float32(1.0 / 512.0) + if exponent != cutlass.Int32(0): + power = cutlass.Int32(exponent + cutlass.Int32(120)) << cutlass.Int32(23) + value = power.bitcast(cutlass.Float32) * ( + cutlass.Float32(1.0) + cutlass.Float32(mantissa) * cutlass.Float32(0.125) + ) + return value + + +@cute.kernel +def _tp_allreduce_bf16_oneshot_lamport_serving_kernel( + input_tensor: cute.Tensor, + output_tensor: cute.Tensor, + buffers: list[cute.Tensor], + ready_flags: list[cute.Tensor], + signal_state: cute.Tensor, + signal: cutlass.Int64, + N: cutlass.Constexpr[int], + LOCAL_RANK: cutlass.Constexpr[int], + WORLD_SIZE: cutlass.Constexpr[int], + THREADS_PER_BLOCK: cutlass.Constexpr[int], + USE_READY_FLAGS: cutlass.Constexpr[bool], + USE_DEVICE_SIGNAL_STATE: cutlass.Constexpr[bool], + ELIDE_SELF_RANK: cutlass.Constexpr[bool], + PEER_ORDER: cutlass.Constexpr[tuple[int, ...]], + FP32_ACCUMULATION: cutlass.Constexpr[bool], + BATCH_POLL_PEERS: cutlass.Constexpr[bool], + USE_PDL: cutlass.Constexpr[bool], + PDL_RELEASE_STRICT: cutlass.Constexpr[bool], + LAST_CTA_SIGNAL_ADVANCE: cutlass.Constexpr[bool], +): + tidx, _, _ = cute.arch.thread_idx() + bidx, _, _ = cute.arch.block_idx() + grid_x, _, _ = cute.arch.grid_dim() + bdim, _, _ = cute.arch.block_dim() + + input_array = cutlass.make_array_view(input_tensor) + output_array = cutlass.make_array_view(output_tensor) + buffer_arrays = [cutlass.make_array_view(buffer) for buffer in buffers] + local_buffer = buffer_arrays[LOCAL_RANK] + ready_flag_arrays = [cutlass.make_array_view(ready_flag) for ready_flag in ready_flags] + local_ready_flags = ready_flag_arrays[LOCAL_RANK] + + active_numel = cute.size(input_tensor, mode=[0]) * N + workspace_numel = cute.size(buffers[0], mode=[2]) + workspace_numel_i64 = cutlass.Int64(workspace_numel) + num_vectors = cute.size(ready_flags[0], mode=[2]) + num_vectors_i64 = cutlass.Int64(num_vectors) + state_counter_ptr = (signal_state.iterator + cutlass.Int64(_SIGNAL_COUNTER_INDEX)).llvm_ptr + state_signal_ptr = (signal_state.iterator + cutlass.Int64(_SIGNAL_VALUE_INDEX)).llvm_ptr + release_epoch_ptr = ( + signal_state.iterator + cutlass.Int64(_SIGNAL_RELEASE_EPOCH_INDEX) + ).llvm_ptr + + signal_i32 = cutlass.Int32(0) + signal_i64 = signal + entry_release_epoch = cutlass.Int32(0) + if cutlass.const_expr(USE_PDL): + cute.arch.griddepcontrol_wait() + if USE_DEVICE_SIGNAL_STATE: + if cutlass.const_expr(USE_PDL): + if cutlass.const_expr(PDL_RELEASE_STRICT): + entry_release_epoch = cute.arch.load( + release_epoch_ptr, + cutlass.Int32, + sem="acquire", + scope="gpu", + ) + signal_i32 = cute.arch.load( + state_signal_ptr, + cutlass.Int32, + sem="acquire", + scope="gpu", + ) + signal_i64 = cutlass.Int64(signal_i32) + + ping_slot = signal_i64 % _PING_PONG_SIZE + clear_slot = (signal_i64 + cutlass.Int64(1)) % _PING_PONG_SIZE + ping_base = cutlass.Int64(ping_slot) * cutlass.Int64(WORLD_SIZE) * workspace_numel_i64 + clear_base = cutlass.Int64(clear_slot) * cutlass.Int64(WORLD_SIZE) * workspace_numel_i64 + ping_ready_base = cutlass.Int64(ping_slot) * cutlass.Int64(WORLD_SIZE) * num_vectors_i64 + clear_ready_base = cutlass.Int64(clear_slot) * cutlass.Int64(WORLD_SIZE) * num_vectors_i64 + + neg_zero_vec = cutlass.Array( + cutlass.BFloat16, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + neg_zero_bits = cutlass.Int16(_NEGATIVE_ZERO_BF16_BITS) + for lane in cutlass.range_constexpr(_BF16_VEC_SIZE): + neg_zero_vec[lane] = neg_zero_bits.bitcast(cutlass.BFloat16) + ready_zero = cutlass.full((1,), 0, dtype=cutlass.Int32) + + if USE_DEVICE_SIGNAL_STATE: + clear_count_ptr = ( + signal_state.iterator + + cutlass.Int64(_SIGNAL_SLOT_COUNTS_INDEX) + + cutlass.Int64(clear_slot) + ).llvm_ptr + clear_vectors = cute.arch.load( + clear_count_ptr, + cutlass.Int32, + sem="acquire", + scope="gpu", + ) + clear_vector_idx = bidx * bdim + tidx + clear_stride = grid_x * bdim + while clear_vector_idx < clear_vectors: + clear_vector_idx_i64 = cutlass.Int64(clear_vector_idx) + if USE_READY_FLAGS: + for source_rank in cutlass.range_constexpr(WORLD_SIZE): + if (not ELIDE_SELF_RANK) or source_rank != LOCAL_RANK: + clear_ready_offset = ( + clear_ready_base + + cutlass.Int64(source_rank) * num_vectors_i64 + + clear_vector_idx_i64 + ) + local_ready_flags.store( + ready_zero, + clear_ready_offset, + 1, + alignment=_READY_FLAG_ALIGNMENT_BYTES, + is_volatile=True, + ) + else: + for source_rank in cutlass.range_constexpr(WORLD_SIZE): + if (not ELIDE_SELF_RANK) or source_rank != LOCAL_RANK: + clear_offset = ( + clear_base + + cutlass.Int64(source_rank) * workspace_numel_i64 + + clear_vector_idx_i64 * cutlass.Int64(_BF16_VEC_SIZE) + ) + local_buffer.store( + neg_zero_vec[ + 0:_BF16_VEC_SIZE, + cutlass.align(_ALIGNMENT_BYTES), + ], + clear_offset, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + is_volatile=True, + ) + clear_vector_idx = clear_vector_idx + clear_stride + + base_idx = (bidx * bdim + tidx) * _BF16_VEC_SIZE + if base_idx < active_numel: + base_idx_i64 = cutlass.Int64(base_idx) + vector_idx_i64 = base_idx_i64 // cutlass.Int64(_BF16_VEC_SIZE) + + if not USE_DEVICE_SIGNAL_STATE: + if USE_READY_FLAGS: + for source_rank in cutlass.range_constexpr(WORLD_SIZE): + if (not ELIDE_SELF_RANK) or source_rank != LOCAL_RANK: + clear_ready_offset = ( + clear_ready_base + + cutlass.Int64(source_rank) * num_vectors_i64 + + vector_idx_i64 + ) + local_ready_flags.store( + ready_zero, + clear_ready_offset, + 1, + alignment=_READY_FLAG_ALIGNMENT_BYTES, + is_volatile=True, + ) + else: + for source_rank in cutlass.range_constexpr(WORLD_SIZE): + if (not ELIDE_SELF_RANK) or source_rank != LOCAL_RANK: + clear_offset = ( + clear_base + + cutlass.Int64(source_rank) * workspace_numel_i64 + + base_idx_i64 + ) + local_buffer.store( + neg_zero_vec[ + 0:_BF16_VEC_SIZE, + cutlass.align(_ALIGNMENT_BYTES), + ], + clear_offset, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + is_volatile=True, + ) + + input_values = input_array.load( + base_idx, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + sanitized_input_values = cutlass.Array( + cutlass.BFloat16, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + if not USE_READY_FLAGS: + for lane in cutlass.range_constexpr(_BF16_VEC_SIZE): + lane_value = input_values[lane] + if lane_value.bitcast(cutlass.Int16) == neg_zero_bits: + lane_value = cutlass.BFloat16(0.0) + sanitized_input_values[lane] = lane_value + + for peer_ordinal in cutlass.range_constexpr(WORLD_SIZE): + target_rank = PEER_ORDER[peer_ordinal] + if (not ELIDE_SELF_RANK) or target_rank != LOCAL_RANK: + write_offset = ( + ping_base + cutlass.Int64(LOCAL_RANK) * workspace_numel_i64 + base_idx_i64 + ) + if USE_READY_FLAGS: + ready_offset = ( + ping_ready_base + + cutlass.Int64(LOCAL_RANK) * num_vectors_i64 + + vector_idx_i64 + ) + buffer_arrays[target_rank].store( + input_values, + write_offset, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + is_volatile=True, + ) + prims.red( + prims.ReductionOp.ADD, + prims.ReductionType.S32, + (ready_flags[target_rank].iterator + ready_offset).llvm_ptr, + cutlass.Int32(1), + mem_order=prims.MemOrder.RELEASE, + mem_scope=prims.MemScope.SYS, + ) + else: + buffer_arrays[target_rank].store( + sanitized_input_values[ + 0:_BF16_VEC_SIZE, + cutlass.align(_ALIGNMENT_BYTES), + ], + write_offset, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + is_volatile=True, + ) + + # Counter atomic is issued after publication so its round trip + # overlaps NVLink propagation instead of delaying the publish. + if USE_DEVICE_SIGNAL_STATE and LAST_CTA_SIGNAL_ADVANCE: + if tidx == 0: + active_vectors_i32 = ( + cutlass.Int32(cute.size(input_tensor, mode=[0])) * cutlass.Int32(N) + ) // cutlass.Int32(_BF16_VEC_SIZE) + _advance_signal_if_last_cta( + state_counter_ptr, + state_signal_ptr, + signal_state, + signal_i32, + ping_slot, + active_vectors_i32, + grid_x, + ) + + acc_f32 = cutlass.full((_BF16_VEC_SIZE,), 0.0, dtype=cutlass.Float32) + acc_bf16 = cutlass.full((_BF16_VEC_SIZE,), 0.0, dtype=cutlass.BFloat16) + + if BATCH_POLL_PEERS: + peer_values_all = cutlass.Array( + cutlass.BFloat16, + WORLD_SIZE * _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + all_peers_ready = cutlass.Boolean(False) + while not all_peers_ready: + all_peers_ready = cutlass.Boolean(True) + for source_rank in cutlass.range_constexpr(WORLD_SIZE): + if (not ELIDE_SELF_RANK) or source_rank != LOCAL_RANK: + read_offset = ( + ping_base + + cutlass.Int64(source_rank) * workspace_numel_i64 + + base_idx_i64 + ) + polled_values = local_buffer.load( + read_offset, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + is_volatile=True, + ) + peer_base = source_rank * _BF16_VEC_SIZE + peer_values_all[ + peer_base:_BF16_VEC_SIZE, + cutlass.align(_ALIGNMENT_BYTES), + ] = polled_values + all_peers_ready = all_peers_ready & _all_bf16x8_lanes_ready(polled_values) + + for source_rank in cutlass.range_constexpr(WORLD_SIZE): + if (not ELIDE_SELF_RANK) or source_rank != LOCAL_RANK: + peer_base = source_rank * _BF16_VEC_SIZE + peer_values = peer_values_all[ + peer_base:_BF16_VEC_SIZE, + cutlass.align(_ALIGNMENT_BYTES), + ] + if FP32_ACCUMULATION: + acc_f32 = acc_f32 + peer_values.to(cutlass.Float32) + else: + acc_bf16 = acc_bf16 + peer_values + if ELIDE_SELF_RANK and source_rank == LOCAL_RANK: + # Own partial enters the sum at its rank position so every + # rank accumulates 0 -> WORLD_SIZE-1 (bit-identical, as TRT-LLM). + if FP32_ACCUMULATION: + acc_f32 = acc_f32 + input_values.to(cutlass.Float32) + else: + acc_bf16 = acc_bf16 + input_values + else: + for source_rank in cutlass.range_constexpr(WORLD_SIZE): + if (not ELIDE_SELF_RANK) or source_rank != LOCAL_RANK: + read_offset = ( + ping_base + cutlass.Int64(source_rank) * workspace_numel_i64 + base_idx_i64 + ) + peer_values = cutlass.full( + (_BF16_VEC_SIZE,), + 0.0, + dtype=cutlass.BFloat16, + ) + if USE_READY_FLAGS: + read_ready_offset = ( + ping_ready_base + + cutlass.Int64(source_rank) * num_vectors_i64 + + vector_idx_i64 + ) + ready_value = cute.arch.load( + (ready_flags[LOCAL_RANK].iterator + read_ready_offset).llvm_ptr, + cutlass.Int32, + sem="acquire", + scope="sys", + ) + while ready_value < cutlass.Int32(1): + ready_value = cute.arch.load( + (ready_flags[LOCAL_RANK].iterator + read_ready_offset).llvm_ptr, + cutlass.Int32, + sem="acquire", + scope="sys", + ) + peer_values = local_buffer.load( + read_offset, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + is_volatile=True, + ) + else: + peer_values = local_buffer.load( + read_offset, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + is_volatile=True, + ) + while not _all_bf16x8_lanes_ready(peer_values): + peer_values = local_buffer.load( + read_offset, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + is_volatile=True, + ) + + if FP32_ACCUMULATION: + acc_f32 = acc_f32 + peer_values.to(cutlass.Float32) + else: + acc_bf16 = acc_bf16 + peer_values + if ELIDE_SELF_RANK and source_rank == LOCAL_RANK: + # Own partial enters the sum at its rank position so every + # rank accumulates 0 -> WORLD_SIZE-1 (bit-identical, as TRT-LLM). + if FP32_ACCUMULATION: + acc_f32 = acc_f32 + input_values.to(cutlass.Float32) + else: + acc_bf16 = acc_bf16 + input_values + output_values = cutlass.full((_BF16_VEC_SIZE,), 0.0, dtype=cutlass.BFloat16) + if FP32_ACCUMULATION: + output_values = acc_f32.to(cutlass.BFloat16) + else: + output_values = acc_bf16 + output_array.store( + output_values, + base_idx, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + + # A successor can launch here, but its entry wait precedes its signal-state + # load, so it cannot select a Lamport slot until this grid has published it. + if cutlass.const_expr(USE_PDL): + if cutlass.const_expr(not PDL_RELEASE_STRICT): + if tidx == 0: + cute.arch.griddepcontrol_launch_dependents() + + if USE_DEVICE_SIGNAL_STATE and not LAST_CTA_SIGNAL_ADVANCE: + if tidx == 0: + prims.red( + prims.ReductionOp.ADD, + prims.ReductionType.S32, + state_counter_ptr, + cutlass.Int32(1), + mem_order=prims.MemOrder.RELEASE, + mem_scope=prims.MemScope.GPU, + ) + if (bidx == 0) & (tidx == 0): + completed_ctas = cute.arch.load( + state_counter_ptr, + cutlass.Int32, + sem="acquire", + scope="gpu", + ) + while completed_ctas < grid_x: + completed_ctas = cute.arch.load( + state_counter_ptr, + cutlass.Int32, + sem="acquire", + scope="gpu", + ) + active_vectors = ( + cutlass.Int32(cute.size(input_tensor, mode=[0])) * cutlass.Int32(N) + ) // cutlass.Int32(_BF16_VEC_SIZE) + ping_count_ptr = ( + signal_state.iterator + + cutlass.Int64(_SIGNAL_SLOT_COUNTS_INDEX) + + cutlass.Int64(ping_slot) + ).llvm_ptr + next_signal = (signal_i32 + cutlass.Int32(1)) % cutlass.Int32(_PING_PONG_SIZE) + cute.arch.store( + ping_count_ptr, + active_vectors, + sem="release", + scope="gpu", + ) + cute.arch.store( + state_signal_ptr, + next_signal, + sem="release", + scope="gpu", + ) + cute.arch.store( + state_counter_ptr, + cutlass.Int32(0), + sem="release", + scope="gpu", + ) + if cutlass.const_expr(USE_PDL): + if cutlass.const_expr(PDL_RELEASE_STRICT): + prims.red( + prims.ReductionOp.ADD, + prims.ReductionType.S32, + release_epoch_ptr, + cutlass.Int32(1), + mem_order=prims.MemOrder.RELEASE, + mem_scope=prims.MemScope.GPU, + ) + + if cutlass.const_expr(USE_PDL): + if cutlass.const_expr(PDL_RELEASE_STRICT): + if tidx == 0: + if USE_DEVICE_SIGNAL_STATE: + published_epoch = cute.arch.load( + release_epoch_ptr, + cutlass.Int32, + sem="acquire", + scope="gpu", + ) + while published_epoch == entry_release_epoch: + published_epoch = cute.arch.load( + release_epoch_ptr, + cutlass.Int32, + sem="acquire", + scope="gpu", + ) + cute.arch.griddepcontrol_launch_dependents() + + +@cute.jit +def _tp_allreduce_bf16_oneshot_lamport_serving( + input_tensor: cute.Tensor, + output_tensor: cute.Tensor, + buffers: list[cute.Tensor], + ready_flags: list[cute.Tensor], + signal_state: cute.Tensor, + signal: cutlass.Int64, + stream: cuda.CUstream, + N: cutlass.Constexpr[int], + LOCAL_RANK: cutlass.Constexpr[int], + WORLD_SIZE: cutlass.Constexpr[int], + THREADS_PER_BLOCK: cutlass.Constexpr[int], + USE_READY_FLAGS: cutlass.Constexpr[bool], + USE_DEVICE_SIGNAL_STATE: cutlass.Constexpr[bool], + ELIDE_SELF_RANK: cutlass.Constexpr[bool], + PEER_ORDER: cutlass.Constexpr[tuple[int, ...]], + FP32_ACCUMULATION: cutlass.Constexpr[bool], + BATCH_POLL_PEERS: cutlass.Constexpr[bool], + USE_PDL: cutlass.Constexpr[bool], + PDL_RELEASE_STRICT: cutlass.Constexpr[bool], + LAST_CTA_SIGNAL_ADVANCE: cutlass.Constexpr[bool], +) -> None: + M = input_tensor.shape[0] + active_numel = M * N + elems_per_cta = THREADS_PER_BLOCK * _BF16_VEC_SIZE + num_blocks = cute.ceil_div(active_numel, elems_per_cta) + _tp_allreduce_bf16_oneshot_lamport_serving_kernel( + input_tensor, + output_tensor, + buffers, + ready_flags, + signal_state, + signal, + N, + LOCAL_RANK, + WORLD_SIZE, + THREADS_PER_BLOCK, + USE_READY_FLAGS, + USE_DEVICE_SIGNAL_STATE, + ELIDE_SELF_RANK, + PEER_ORDER, + FP32_ACCUMULATION, + BATCH_POLL_PEERS, + USE_PDL, + PDL_RELEASE_STRICT, + LAST_CTA_SIGNAL_ADVANCE, + ).launch( + grid=(num_blocks, 1, 1), + block=(THREADS_PER_BLOCK, 1, 1), + stream=stream, + use_pdl=USE_PDL, + ) + + +@cute.kernel +def _tp_allreduce_residual_rms_norm_bf16_oneshot_lamport_kernel( + input_tensor: cute.Tensor, + residual_tensor: cute.Tensor, + norm_weight_tensor: cute.Tensor, + norm_output_tensor: cute.Tensor, + residual_output_tensor: cute.Tensor, + buffers: list[cute.Tensor], + ready_flags: list[cute.Tensor], + signal: cutlass.Int64, + eps: cutlass.Constexpr[cutlass.Float32], + N: cutlass.Constexpr[int], + LOCAL_RANK: cutlass.Constexpr[int], + WORLD_SIZE: cutlass.Constexpr[int], +): + tidx, _, _ = cute.arch.thread_idx() + row, _, _ = cute.arch.block_idx() + + input_array = cutlass.make_array_view(input_tensor) + residual_array = cutlass.make_array_view(residual_tensor) + norm_weight_array = cutlass.make_array_view(norm_weight_tensor) + norm_output_array = cutlass.make_array_view(norm_output_tensor) + residual_output_array = cutlass.make_array_view(residual_output_tensor) + buffer_arrays = [cutlass.make_array_view(buffer) for buffer in buffers] + local_buffer = buffer_arrays[LOCAL_RANK] + ready_flag_arrays = [cutlass.make_array_view(ready_flag) for ready_flag in ready_flags] + local_ready_flags = ready_flag_arrays[LOCAL_RANK] + + row_values_per_thread = (N + _THREADS_PER_BLOCK - 1) // _THREADS_PER_BLOCK + row_values_padded = ( + (row_values_per_thread + _BF16_VEC_SIZE - 1) // _BF16_VEC_SIZE + ) * _BF16_VEC_SIZE + residual_values = cutlass.Array( + cutlass.BFloat16, + row_values_padded, + alignment=_ALIGNMENT_BYTES, + ) + weight_values = cutlass.Array( + cutlass.BFloat16, + row_values_padded, + alignment=_ALIGNMENT_BYTES, + ) + norm_values = cutlass.Array( + cutlass.BFloat16, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + warp_sums = cutlass.Array( + cutlass.Float32, + _WARPS_PER_BLOCK, + space=cutlass.AddressSpace.smem, + alignment=_ALIGNMENT_BYTES, + ) + + lane_id = tidx % 32 + warp_id = tidx // 32 + numel = cute.size(buffers[0], mode=[2]) + row_base = row * N + ping_slot = signal % _PING_PONG_SIZE + pong_slot = (signal + cutlass.Int64(1)) % _PING_PONG_SIZE + numel_i64 = cutlass.Int64(numel) + num_vectors = cute.size(ready_flags[0], mode=[2]) + num_vectors_i64 = cutlass.Int64(num_vectors) + ping_base = cutlass.Int64(ping_slot) * cutlass.Int64(WORLD_SIZE) * numel_i64 + ping_ready_base = cutlass.Int64(ping_slot) * cutlass.Int64(WORLD_SIZE) * num_vectors_i64 + pong_ready_base = cutlass.Int64(pong_slot) * cutlass.Int64(WORLD_SIZE) * num_vectors_i64 + + ready_zero = cutlass.full((1,), 0, dtype=cutlass.Int32) + local_sum_sq = cutlass.Float32(0.0) + + for j in cutlass.range_constexpr(0, row_values_per_thread, _BF16_VEC_SIZE): + col = tidx * _BF16_VEC_SIZE + j * _THREADS_PER_BLOCK + if col + _BF16_VEC_SIZE <= N: + base_idx = row_base + col + base_idx_i64 = cutlass.Int64(base_idx) + vector_idx_i64 = base_idx_i64 // cutlass.Int64(_BF16_VEC_SIZE) + + for source_rank in cutlass.range_constexpr(WORLD_SIZE): + clear_ready_offset = ( + pong_ready_base + cutlass.Int64(source_rank) * num_vectors_i64 + vector_idx_i64 + ) + local_ready_flags.store( + ready_zero, + clear_ready_offset, + 1, + alignment=_READY_FLAG_ALIGNMENT_BYTES, + is_volatile=True, + ) + + input_values = input_array[row, :].load( + col, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + + for target_rank in cutlass.range_constexpr(WORLD_SIZE): + write_offset = ping_base + cutlass.Int64(LOCAL_RANK) * numel_i64 + base_idx_i64 + ready_offset = ( + ping_ready_base + cutlass.Int64(LOCAL_RANK) * num_vectors_i64 + vector_idx_i64 + ) + buffer_arrays[target_rank].store( + input_values, + write_offset, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + is_volatile=True, + ) + prims.red( + prims.ReductionOp.ADD, + prims.ReductionType.S32, + (ready_flags[target_rank].iterator + ready_offset).llvm_ptr, + cutlass.Int32(1), + mem_order=prims.MemOrder.RELEASE, + mem_scope=prims.MemScope.SYS, + ) + + acc = cutlass.full((_BF16_VEC_SIZE,), 0.0, dtype=cutlass.Float32) + + for source_rank in cutlass.range_constexpr(WORLD_SIZE): + read_offset = ping_base + cutlass.Int64(source_rank) * numel_i64 + base_idx_i64 + read_ready_offset = ( + ping_ready_base + cutlass.Int64(source_rank) * num_vectors_i64 + vector_idx_i64 + ) + ready_value = cute.arch.load( + (ready_flags[LOCAL_RANK].iterator + read_ready_offset).llvm_ptr, + cutlass.Int32, + sem="acquire", + scope="sys", + ) + while ready_value < cutlass.Int32(1): + ready_value = cute.arch.load( + (ready_flags[LOCAL_RANK].iterator + read_ready_offset).llvm_ptr, + cutlass.Int32, + sem="acquire", + scope="sys", + ) + peer_values = local_buffer.load( + read_offset, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + is_volatile=True, + ) + + acc = acc + peer_values.to(cutlass.Float32) + + residual_vec = residual_array[row, :].load( + col, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + residual_out_vec = (acc + residual_vec.to(cutlass.Float32)).to(cutlass.BFloat16) + weight_vec = norm_weight_array.load( + col, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + residual_output_array[row, :].store( + residual_out_vec, + col, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + residual_values[j:_BF16_VEC_SIZE, cutlass.align(_ALIGNMENT_BYTES)] = residual_out_vec + weight_values[j:_BF16_VEC_SIZE, cutlass.align(_ALIGNMENT_BYTES)] = weight_vec + + residual_f32 = residual_out_vec.to(cutlass.Float32) + local_sum_sq = local_sum_sq + (residual_f32 * residual_f32).reduce("add") + + warp_sum = _warp_sum_f32(local_sum_sq) + if lane_id == 0: + warp_sums[warp_id] = warp_sum + + prims.barrier_cta_sync(0) + + if tidx == 0: + total_sum = cutlass.Float32(0.0) + for warp_idx in cutlass.range_constexpr(_WARPS_PER_BLOCK): + total_sum = total_sum + warp_sums[warp_idx] + warp_sums[0] = cute.math.rsqrt( + total_sum * (1.0 / cutlass.Float32(N)) + eps, + fastmath=True, + ) + + prims.barrier_cta_sync(0) + inv_rms = warp_sums[0] + + for j in cutlass.range_constexpr(0, row_values_per_thread, _BF16_VEC_SIZE): + col = tidx * _BF16_VEC_SIZE + j * _THREADS_PER_BLOCK + if col + _BF16_VEC_SIZE <= N: + residual_vec = residual_values[ + j:_BF16_VEC_SIZE, + cutlass.align(_ALIGNMENT_BYTES), + ] + weight_vec = weight_values[ + j:_BF16_VEC_SIZE, + cutlass.align(_ALIGNMENT_BYTES), + ] + norm_f32 = residual_vec.to(cutlass.Float32) * inv_rms + norm_f32 = norm_f32 * weight_vec.to(cutlass.Float32) + norm_values[0:_BF16_VEC_SIZE, cutlass.align(_ALIGNMENT_BYTES)] = norm_f32.to( + cutlass.BFloat16 + ) + norm_output_array[row, :].store( + norm_values[0:_BF16_VEC_SIZE, cutlass.align(_ALIGNMENT_BYTES)], + col, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + + +@cute.kernel +def _tp_allreduce_residual_rms_norm_bf16_oneshot_lamport_split_row_kernel( + input_tensor: cute.Tensor, + residual_tensor: cute.Tensor, + norm_weight_tensor: cute.Tensor, + norm_output_tensor: cute.Tensor, + residual_output_tensor: cute.Tensor, + buffers: list[cute.Tensor], + ready_flags: list[cute.Tensor], + signal_state: cute.Tensor, + multicast_base: cutlass.Int64, + signal: cutlass.Int64, + eps: cutlass.Constexpr[cutlass.Float32], + N: cutlass.Constexpr[int], + LOCAL_RANK: cutlass.Constexpr[int], + WORLD_SIZE: cutlass.Constexpr[int], + CTAS_PER_ROW: cutlass.Constexpr[int], + THREADS_PER_BLOCK: cutlass.Constexpr[int], + USE_READY_FLAGS: cutlass.Constexpr[bool], + USE_DEVICE_SIGNAL_STATE: cutlass.Constexpr[bool], + ELIDE_SELF_RANK: cutlass.Constexpr[bool], + PEER_ORDER: cutlass.Constexpr[tuple[int, ...]], + PREFETCH_FUSED_INPUTS: cutlass.Constexpr[bool], + DIRECT_CLUSTER_REDUCTION: cutlass.Constexpr[bool], + POST_NORM_CLUSTER_BARRIER: cutlass.Constexpr[bool], + FP32_ACCUMULATION: cutlass.Constexpr[bool], + BATCH_POLL_PEERS: cutlass.Constexpr[bool], + READY_MASK_POLL_PEERS: cutlass.Constexpr[bool], + AGGREGATE_SIGNAL_AT_CLUSTER_ROOT: cutlass.Constexpr[bool], + USE_PDL: cutlass.Constexpr[bool], + PDL_RELEASE_STRICT: cutlass.Constexpr[bool], + PDL_RELEASE_PRE_NORM: cutlass.Constexpr[bool], + PDL_RELEASE_AT_ENTRY: cutlass.Constexpr[bool], + LAST_CTA_SIGNAL_ADVANCE: cutlass.Constexpr[bool], + ALL_TO_ALL_CLUSTER_REDUCTION: cutlass.Constexpr[bool], + EARLY_INPUT_LOAD: cutlass.Constexpr[bool], + MULTICAST_PUBLISH: cutlass.Constexpr[bool], + IKET_PHASES: cutlass.Constexpr[bool], + ROTATED_POLL_ORDER: cutlass.Constexpr[bool], +): + tidx, _, _ = cute.arch.thread_idx() + bidx, _, _ = cute.arch.block_idx() + grid_x, _, _ = cute.arch.grid_dim() + cta_rank = cute.arch.block_idx_in_cluster() + row = bidx // CTAS_PER_ROW + + input_array = cutlass.make_array_view(input_tensor) + residual_array = cutlass.make_array_view(residual_tensor) + norm_weight_array = cutlass.make_array_view(norm_weight_tensor) + norm_output_array = cutlass.make_array_view(norm_output_tensor) + residual_output_array = cutlass.make_array_view(residual_output_tensor) + buffer_arrays = [cutlass.make_array_view(buffer) for buffer in buffers] + local_buffer = buffer_arrays[LOCAL_RANK] + ready_flag_arrays = [cutlass.make_array_view(ready_flag) for ready_flag in ready_flags] + local_ready_flags = ready_flag_arrays[LOCAL_RANK] + + cols_per_cta = N // CTAS_PER_ROW + chunk_values_per_thread = (cols_per_cta + THREADS_PER_BLOCK - 1) // THREADS_PER_BLOCK + chunk_values_padded = ( + (chunk_values_per_thread + _BF16_VEC_SIZE - 1) // _BF16_VEC_SIZE + ) * _BF16_VEC_SIZE + residual_values = cutlass.Array( + cutlass.BFloat16, + chunk_values_padded, + alignment=_ALIGNMENT_BYTES, + ) + weight_values = cutlass.Array( + cutlass.BFloat16, + chunk_values_padded, + alignment=_ALIGNMENT_BYTES, + ) + norm_values = cutlass.Array( + cutlass.BFloat16, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + warp_sums = cutlass.Array( + cutlass.Float32, + THREADS_PER_BLOCK // 32, + space=cutlass.AddressSpace.smem, + alignment=_ALIGNMENT_BYTES, + ) + row_sum = cutlass.Array( + cutlass.Float32, + 1, + space=cutlass.AddressSpace.smem, + alignment=_ALIGNMENT_BYTES, + ) + # One slot per CTA of the cluster; written remotely via DSMEM by the + # all-to-all reduction (each slot has exactly one writer). + cluster_partials = cutlass.Array( + cutlass.Float32, + CTAS_PER_ROW, + space=cutlass.AddressSpace.smem, + alignment=_ALIGNMENT_BYTES, + ) + + lane_id = tidx % 32 + warp_id = tidx // 32 + numel = cute.size(buffers[0], mode=[2]) + row_base = row * N + cta_col_base = cta_rank * cutlass.Int32(cols_per_cta) + state_counter_ptr = (signal_state.iterator + cutlass.Int64(_SIGNAL_COUNTER_INDEX)).llvm_ptr + state_signal_ptr = (signal_state.iterator + cutlass.Int64(_SIGNAL_VALUE_INDEX)).llvm_ptr + release_epoch_ptr = ( + signal_state.iterator + cutlass.Int64(_SIGNAL_RELEASE_EPOCH_INDEX) + ).llvm_ptr + signal_i32 = cutlass.Int32(0) + signal_i64 = signal + entry_release_epoch = cutlass.Int32(0) + if IKET_PHASES: + cute.experimental.iket.range_push("ar_entry") + if cutlass.const_expr(USE_PDL): + cute.arch.griddepcontrol_wait() + # Issue the local input load before the signal-state acquire load so the two + # global round trips overlap; publication then starts as soon as both + # arrive instead of after their sum. + early_input_values = cutlass.Array( + cutlass.BFloat16, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + if EARLY_INPUT_LOAD: + early_col_in_cta = tidx * _BF16_VEC_SIZE + if early_col_in_cta + _BF16_VEC_SIZE <= cols_per_cta: + early_input_values[0:_BF16_VEC_SIZE, cutlass.align(_ALIGNMENT_BYTES)] = input_array[ + row, : + ].load( + cta_col_base + early_col_in_cta, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + if USE_DEVICE_SIGNAL_STATE: + if cutlass.const_expr(USE_PDL): + if cutlass.const_expr(PDL_RELEASE_STRICT): + entry_release_epoch = cute.arch.load( + release_epoch_ptr, + cutlass.Int32, + sem="acquire", + scope="gpu", + ) + signal_i32 = cute.arch.load( + state_signal_ptr, + cutlass.Int32, + sem="acquire", + scope="gpu", + ) + signal_i64 = cutlass.Int64(signal_i32) + if (not AGGREGATE_SIGNAL_AT_CLUSTER_ROOT) and (not LAST_CTA_SIGNAL_ADVANCE) and tidx == 0: + prims.red( + prims.ReductionOp.ADD, + prims.ReductionType.S32, + state_counter_ptr, + cutlass.Int32(1), + mem_order=prims.MemOrder.RELEASE, + mem_scope=prims.MemScope.GPU, + ) + + # This is deliberately after the device signal snapshot: a PDL successor + # may start immediately, but its entry wait cannot observe stale slot state. + if cutlass.const_expr(USE_PDL): + if cutlass.const_expr(PDL_RELEASE_AT_ENTRY): + if tidx == 0: + cute.arch.griddepcontrol_launch_dependents() + ping_slot = signal_i64 % _PING_PONG_SIZE + clear_slot = (signal_i64 + cutlass.Int64(1)) % _PING_PONG_SIZE + numel_i64 = cutlass.Int64(numel) + ping_base = cutlass.Int64(ping_slot) * cutlass.Int64(WORLD_SIZE) * numel_i64 + clear_base = cutlass.Int64(clear_slot) * cutlass.Int64(WORLD_SIZE) * numel_i64 + num_vectors = cute.size(ready_flags[0], mode=[2]) + num_vectors_i64 = cutlass.Int64(num_vectors) + ping_ready_base = cutlass.Int64(ping_slot) * cutlass.Int64(WORLD_SIZE) * num_vectors_i64 + clear_ready_base = cutlass.Int64(clear_slot) * cutlass.Int64(WORLD_SIZE) * num_vectors_i64 + neg_zero_vec = cutlass.Array( + cutlass.BFloat16, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + neg_zero_bits = cutlass.Int16(_NEGATIVE_ZERO_BF16_BITS) + for lane in cutlass.range_constexpr(_BF16_VEC_SIZE): + neg_zero_vec[lane] = neg_zero_bits.bitcast(cutlass.BFloat16) + + ready_zero = cutlass.full((1,), 0, dtype=cutlass.Int32) + local_sum_sq = cutlass.Float32(0.0) + + if IKET_PHASES: + cute.experimental.iket.range_pop() + cute.experimental.iket.range_push("ar_clear") + if USE_DEVICE_SIGNAL_STATE: + clear_count_ptr = ( + signal_state.iterator + + cutlass.Int64(_SIGNAL_SLOT_COUNTS_INDEX) + + cutlass.Int64(clear_slot) + ).llvm_ptr + clear_vectors = cute.arch.load( + clear_count_ptr, + cutlass.Int32, + sem="acquire", + scope="gpu", + ) + clear_vector_idx = bidx * THREADS_PER_BLOCK + tidx + clear_stride = grid_x * THREADS_PER_BLOCK + while clear_vector_idx < clear_vectors: + clear_vector_idx_i64 = cutlass.Int64(clear_vector_idx) + if USE_READY_FLAGS: + for source_rank in cutlass.range_constexpr(WORLD_SIZE): + if (not ELIDE_SELF_RANK) or source_rank != LOCAL_RANK: + clear_ready_offset = ( + clear_ready_base + + cutlass.Int64(source_rank) * num_vectors_i64 + + clear_vector_idx_i64 + ) + local_ready_flags.store( + ready_zero, + clear_ready_offset, + 1, + alignment=_READY_FLAG_ALIGNMENT_BYTES, + is_volatile=True, + ) + else: + for source_rank in cutlass.range_constexpr(WORLD_SIZE): + if (not ELIDE_SELF_RANK) or source_rank != LOCAL_RANK: + clear_offset = ( + clear_base + + cutlass.Int64(source_rank) * numel_i64 + + clear_vector_idx_i64 * cutlass.Int64(_BF16_VEC_SIZE) + ) + local_buffer.store( + neg_zero_vec[ + 0:_BF16_VEC_SIZE, + cutlass.align(_ALIGNMENT_BYTES), + ], + clear_offset, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + is_volatile=True, + ) + clear_vector_idx = clear_vector_idx + clear_stride + + if IKET_PHASES: + cute.experimental.iket.range_pop() + cute.experimental.iket.range_push("ar_load_publish") + for j in cutlass.range_constexpr(0, chunk_values_per_thread, _BF16_VEC_SIZE): + col_in_cta = tidx * _BF16_VEC_SIZE + j * THREADS_PER_BLOCK + if col_in_cta + _BF16_VEC_SIZE <= cols_per_cta: + col = cta_col_base + col_in_cta + base_idx = row_base + col + base_idx_i64 = cutlass.Int64(base_idx) + vector_idx_i64 = base_idx_i64 // cutlass.Int64(_BF16_VEC_SIZE) + + if not USE_DEVICE_SIGNAL_STATE: + if USE_READY_FLAGS: + for source_rank in cutlass.range_constexpr(WORLD_SIZE): + if (not ELIDE_SELF_RANK) or source_rank != LOCAL_RANK: + clear_ready_offset = ( + clear_ready_base + + cutlass.Int64(source_rank) * num_vectors_i64 + + vector_idx_i64 + ) + local_ready_flags.store( + ready_zero, + clear_ready_offset, + 1, + alignment=_READY_FLAG_ALIGNMENT_BYTES, + is_volatile=True, + ) + else: + for source_rank in cutlass.range_constexpr(WORLD_SIZE): + if (not ELIDE_SELF_RANK) or source_rank != LOCAL_RANK: + clear_offset = ( + clear_base + cutlass.Int64(source_rank) * numel_i64 + base_idx_i64 + ) + local_buffer.store( + neg_zero_vec[ + 0:_BF16_VEC_SIZE, + cutlass.align(_ALIGNMENT_BYTES), + ], + clear_offset, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + is_volatile=True, + ) + + # Both arms assign into the same array so the frontend sees one + # container structure before and after the branch. + input_values_array = cutlass.Array( + cutlass.BFloat16, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + if EARLY_INPUT_LOAD and j == 0: + input_values_array[0:_BF16_VEC_SIZE, cutlass.align(_ALIGNMENT_BYTES)] = ( + early_input_values[0:_BF16_VEC_SIZE, cutlass.align(_ALIGNMENT_BYTES)] + ) + else: + input_values_array[0:_BF16_VEC_SIZE, cutlass.align(_ALIGNMENT_BYTES)] = input_array[ + row, : + ].load( + col, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + input_values = input_values_array[0:_BF16_VEC_SIZE, cutlass.align(_ALIGNMENT_BYTES)] + sanitized_input_values = cutlass.Array( + cutlass.BFloat16, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + if not USE_READY_FLAGS: + neg_zero_bits = cutlass.Int16(_NEGATIVE_ZERO_BF16_BITS) + for lane in cutlass.range_constexpr(_BF16_VEC_SIZE): + lane_value = input_values[lane] + if lane_value.bitcast(cutlass.Int16) == neg_zero_bits: + lane_value = cutlass.BFloat16(0.0) + sanitized_input_values[lane] = lane_value + + if MULTICAST_PUBLISH: + # One NVLS multicast store lands in slot[LOCAL_RANK] of every + # rank (including this one); the fabric does the replication, + # so egress bytes drop from (WORLD_SIZE-1)x to 1x. Sentinel + # polling on the receiving side is unchanged. + multicast_offset = ping_base + cutlass.Int64(LOCAL_RANK) * numel_i64 + base_idx_i64 + _multicast_publish_bf16x8( + multicast_base + multicast_offset * cutlass.Int64(2), + sanitized_input_values[ + 0:_BF16_VEC_SIZE, + cutlass.align(_ALIGNMENT_BYTES), + ], + ) + for peer_ordinal in cutlass.range_constexpr(WORLD_SIZE): + target_rank = PEER_ORDER[peer_ordinal] + if (not MULTICAST_PUBLISH) and ((not ELIDE_SELF_RANK) or target_rank != LOCAL_RANK): + write_offset = ping_base + cutlass.Int64(LOCAL_RANK) * numel_i64 + base_idx_i64 + if USE_READY_FLAGS: + ready_offset = ( + ping_ready_base + + cutlass.Int64(LOCAL_RANK) * num_vectors_i64 + + vector_idx_i64 + ) + buffer_arrays[target_rank].store( + input_values, + write_offset, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + is_volatile=True, + ) + prims.red( + prims.ReductionOp.ADD, + prims.ReductionType.S32, + (ready_flags[target_rank].iterator + ready_offset).llvm_ptr, + cutlass.Int32(1), + mem_order=prims.MemOrder.RELEASE, + mem_scope=prims.MemScope.SYS, + ) + else: + buffer_arrays[target_rank].store( + sanitized_input_values[ + 0:_BF16_VEC_SIZE, + cutlass.align(_ALIGNMENT_BYTES), + ], + write_offset, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + is_volatile=True, + ) + + if IKET_PHASES: + cute.experimental.iket.range_pop() + cute.experimental.iket.range_push("ar_poll") + # Counter atomic is issued after publication so its round trip + # overlaps NVLink propagation instead of delaying the publish. + if USE_DEVICE_SIGNAL_STATE and LAST_CTA_SIGNAL_ADVANCE and j == 0: + if tidx == 0: + active_vectors_i32 = ( + cutlass.Int32(cute.size(input_tensor, mode=[0])) * cutlass.Int32(N) + ) // cutlass.Int32(_BF16_VEC_SIZE) + _advance_signal_if_last_cta( + state_counter_ptr, + state_signal_ptr, + signal_state, + signal_i32, + ping_slot, + active_vectors_i32, + grid_x, + ) + + residual_vec = cutlass.full((_BF16_VEC_SIZE,), 0.0, dtype=cutlass.BFloat16) + weight_vec = cutlass.full((_BF16_VEC_SIZE,), 0.0, dtype=cutlass.BFloat16) + if PREFETCH_FUSED_INPUTS: + # Issue independent local loads after publication and before + # polling peers so their latency overlaps NVLink propagation. + residual_vec = residual_array[row, :].load( + col, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + weight_vec = norm_weight_array.load( + col, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + + acc_f32 = cutlass.full((_BF16_VEC_SIZE,), 0.0, dtype=cutlass.Float32) + acc_bf16 = cutlass.full((_BF16_VEC_SIZE,), 0.0, dtype=cutlass.BFloat16) + + if READY_MASK_POLL_PEERS: + # Cache each peer exactly once when its sentinel becomes ready. + # Accumulation is deferred until every peer is present so the + # arithmetic order remains PEER_ORDER, independent of arrival. + peer_values_all = cutlass.Array( + cutlass.BFloat16, + WORLD_SIZE * _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + peer_ready_mask = cutlass.Int32(0) + expected_peer_mask = cutlass.Int32(0) + for source_rank in cutlass.range_constexpr(WORLD_SIZE): + if (not ELIDE_SELF_RANK) or source_rank != LOCAL_RANK: + expected_peer_mask = expected_peer_mask | cutlass.Int32(1 << source_rank) + + while peer_ready_mask != expected_peer_mask: + for source_rank in cutlass.range_constexpr(WORLD_SIZE): + if (not ELIDE_SELF_RANK) or source_rank != LOCAL_RANK: + source_rank_mask = cutlass.Int32(1 << source_rank) + if (peer_ready_mask & source_rank_mask) == cutlass.Int32(0): + read_offset = ( + ping_base + + cutlass.Int64(source_rank) * numel_i64 + + base_idx_i64 + ) + polled_values = local_buffer.load( + read_offset, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + is_volatile=True, + ) + if _all_bf16x8_lanes_ready(polled_values): + peer_base = source_rank * _BF16_VEC_SIZE + peer_values_all[ + peer_base:_BF16_VEC_SIZE, + cutlass.align(_ALIGNMENT_BYTES), + ] = polled_values + peer_ready_mask = peer_ready_mask | source_rank_mask + + for source_rank in cutlass.range_constexpr(WORLD_SIZE): + if (not ELIDE_SELF_RANK) or source_rank != LOCAL_RANK: + peer_base = source_rank * _BF16_VEC_SIZE + peer_values = peer_values_all[ + peer_base:_BF16_VEC_SIZE, + cutlass.align(_ALIGNMENT_BYTES), + ] + if FP32_ACCUMULATION: + acc_f32 = acc_f32 + peer_values.to(cutlass.Float32) + else: + acc_bf16 = acc_bf16 + peer_values + if ELIDE_SELF_RANK and source_rank == LOCAL_RANK: + # Own partial enters the sum at its rank position so every + # rank accumulates 0 -> WORLD_SIZE-1 (bit-identical, as TRT-LLM). + if FP32_ACCUMULATION: + acc_f32 = acc_f32 + input_values.to(cutlass.Float32) + else: + acc_bf16 = acc_bf16 + input_values + elif BATCH_POLL_PEERS: + peer_values_all = cutlass.Array( + cutlass.BFloat16, + WORLD_SIZE * _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + all_peers_ready = cutlass.Boolean(False) + while not all_peers_ready: + all_peers_ready = cutlass.Boolean(True) + for source_rank in cutlass.range_constexpr(WORLD_SIZE): + if (not ELIDE_SELF_RANK) or source_rank != LOCAL_RANK: + read_offset = ( + ping_base + cutlass.Int64(source_rank) * numel_i64 + base_idx_i64 + ) + polled_values = local_buffer.load( + read_offset, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + is_volatile=True, + ) + peer_base = source_rank * _BF16_VEC_SIZE + peer_values_all[ + peer_base:_BF16_VEC_SIZE, + cutlass.align(_ALIGNMENT_BYTES), + ] = polled_values + all_peers_ready = all_peers_ready & _all_bf16x8_lanes_ready( + polled_values + ) + + for source_rank in cutlass.range_constexpr(WORLD_SIZE): + if (not ELIDE_SELF_RANK) or source_rank != LOCAL_RANK: + peer_base = source_rank * _BF16_VEC_SIZE + peer_values = peer_values_all[ + peer_base:_BF16_VEC_SIZE, + cutlass.align(_ALIGNMENT_BYTES), + ] + if FP32_ACCUMULATION: + acc_f32 = acc_f32 + peer_values.to(cutlass.Float32) + else: + acc_bf16 = acc_bf16 + peer_values + if ELIDE_SELF_RANK and source_rank == LOCAL_RANK: + # Own partial enters the sum at its rank position so every + # rank accumulates 0 -> WORLD_SIZE-1 (bit-identical, as TRT-LLM). + if FP32_ACCUMULATION: + acc_f32 = acc_f32 + input_values.to(cutlass.Float32) + else: + acc_bf16 = acc_bf16 + input_values + else: + peer_values_all = cutlass.Array( + cutlass.BFloat16, + WORLD_SIZE * _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + for peer_ordinal in cutlass.range_constexpr(WORLD_SIZE): + source_rank = PEER_ORDER[peer_ordinal] if ROTATED_POLL_ORDER else peer_ordinal + if (not ELIDE_SELF_RANK) or source_rank != LOCAL_RANK: + read_offset = ( + ping_base + cutlass.Int64(source_rank) * numel_i64 + base_idx_i64 + ) + peer_values = cutlass.full( + (_BF16_VEC_SIZE,), + 0.0, + dtype=cutlass.BFloat16, + ) + if USE_READY_FLAGS: + read_ready_offset = ( + ping_ready_base + + cutlass.Int64(source_rank) * num_vectors_i64 + + vector_idx_i64 + ) + ready_value = cute.arch.load( + (ready_flags[LOCAL_RANK].iterator + read_ready_offset).llvm_ptr, + cutlass.Int32, + sem="acquire", + scope="sys", + ) + while ready_value < cutlass.Int32(1): + ready_value = cute.arch.load( + (ready_flags[LOCAL_RANK].iterator + read_ready_offset).llvm_ptr, + cutlass.Int32, + sem="acquire", + scope="sys", + ) + peer_values = local_buffer.load( + read_offset, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + is_volatile=True, + ) + else: + peer_values = local_buffer.load( + read_offset, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + is_volatile=True, + ) + while not _all_bf16x8_lanes_ready(peer_values): + peer_values = local_buffer.load( + read_offset, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + is_volatile=True, + ) + + if ROTATED_POLL_ORDER: + peer_values_all[ + source_rank * _BF16_VEC_SIZE : _BF16_VEC_SIZE, + cutlass.align(_ALIGNMENT_BYTES), + ] = peer_values + elif FP32_ACCUMULATION: + acc_f32 = acc_f32 + peer_values.to(cutlass.Float32) + else: + acc_bf16 = acc_bf16 + peer_values + if (not ROTATED_POLL_ORDER) and ELIDE_SELF_RANK and source_rank == LOCAL_RANK: + # Own partial enters the sum at its rank position so every + # rank accumulates 0 -> WORLD_SIZE-1 (bit-identical, as TRT-LLM). + if FP32_ACCUMULATION: + acc_f32 = acc_f32 + input_values.to(cutlass.Float32) + else: + acc_bf16 = acc_bf16 + input_values + if ROTATED_POLL_ORDER: + # Polled in PEER_ORDER (latency), summed in absolute rank order (numerics). + for source_rank in cutlass.range_constexpr(WORLD_SIZE): + if (not ELIDE_SELF_RANK) or source_rank != LOCAL_RANK: + peer_values = peer_values_all[ + source_rank * _BF16_VEC_SIZE : _BF16_VEC_SIZE, + cutlass.align(_ALIGNMENT_BYTES), + ] + if FP32_ACCUMULATION: + acc_f32 = acc_f32 + peer_values.to(cutlass.Float32) + else: + acc_bf16 = acc_bf16 + peer_values + if ELIDE_SELF_RANK and source_rank == LOCAL_RANK: + if FP32_ACCUMULATION: + acc_f32 = acc_f32 + input_values.to(cutlass.Float32) + else: + acc_bf16 = acc_bf16 + input_values + if IKET_PHASES: + cute.experimental.iket.range_pop() + cute.experimental.iket.range_push("ar_residual") + if not PREFETCH_FUSED_INPUTS: + residual_vec = residual_array[row, :].load( + col, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + weight_vec = norm_weight_array.load( + col, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + # The DSL requires a value used after staged control flow to be + # defined before the branch, even when both constexpr arms assign it. + residual_out_vec = cutlass.full((_BF16_VEC_SIZE,), 0.0, dtype=cutlass.BFloat16) + if FP32_ACCUMULATION: + residual_out_vec = (acc_f32 + residual_vec.to(cutlass.Float32)).to(cutlass.BFloat16) + else: + residual_out_vec = acc_bf16 + residual_vec + residual_output_array[row, :].store( + residual_out_vec, + col, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + residual_values[j:_BF16_VEC_SIZE, cutlass.align(_ALIGNMENT_BYTES)] = residual_out_vec + weight_values[j:_BF16_VEC_SIZE, cutlass.align(_ALIGNMENT_BYTES)] = weight_vec + + residual_f32 = residual_out_vec.to(cutlass.Float32) + local_sum_sq = local_sum_sq + (residual_f32 * residual_f32).reduce("add") + + if IKET_PHASES: + cute.experimental.iket.range_pop() + cute.experimental.iket.range_push("ar_rms_reduce") + warp_sum = _warp_sum_f32(local_sum_sq) + if lane_id == 0: + warp_sums[warp_id] = warp_sum + + prims.barrier_cta_sync(0) + + cta_sum = cutlass.Float32(0.0) + if tidx == 0: + for warp_idx in cutlass.range_constexpr(THREADS_PER_BLOCK // 32): + cta_sum = cta_sum + warp_sums[warp_idx] + if ALL_TO_ALL_CLUSTER_REDUCTION: + _publish_cta_partial_to_cluster(cluster_partials, cta_sum, cta_rank, CTAS_PER_ROW) + elif DIRECT_CLUSTER_REDUCTION: + row_sum[0] = cta_sum + elif cta_rank == 0: + row_sum[0] = cutlass.Float32(0.0) + + inv_rms = cutlass.Float32(0.0) + if cutlass.const_expr(ALL_TO_ALL_CLUSTER_REDUCTION): + # Single cluster rendezvous: arrive.release publishes the DSMEM partial + # stores, wait.acquire makes every slot visible; afterwards no CTA + # touches another CTA's shared memory, so no lifetime barrier is needed. + prims.barrier_cluster_arrive() + prims.barrier_cluster_wait() + if cutlass.const_expr(USE_PDL): + if cutlass.const_expr(PDL_RELEASE_PRE_NORM): + if tidx == 0: + cute.arch.griddepcontrol_launch_dependents() + inv_rms = _cluster_inv_rms_from_partials(cluster_partials, CTAS_PER_ROW, N, eps) + elif cutlass.const_expr(not DIRECT_CLUSTER_REDUCTION): + prims.barrier_cta_sync(0) + prims.barrier_cluster_arrive() + prims.barrier_cluster_wait() + + if tidx == 0: + root_row_sum = prims.mapa(row_sum.data_ptr(), cutlass.Int32(0)) + prims.red( + prims.ReductionOp.ADD, + prims.ReductionType.F32, + root_row_sum, + cta_sum, + mem_order=prims.MemOrder.RELAXED, + mem_scope=prims.MemScope.CLUSTER, + ) + + prims.barrier_cluster_arrive() + prims.barrier_cluster_wait() + + if (cta_rank == 0) & (tidx == 0): + row_sum[0] = cute.math.rsqrt( + row_sum[0] * (1.0 / cutlass.Float32(N)) + eps, + fastmath=True, + ) + + prims.barrier_cta_sync(0) + else: + prims.barrier_cluster_arrive() + prims.barrier_cluster_wait() + + if (cta_rank == 0) & (tidx == 0): + cluster_sum = cutlass.Float32(0.0) + for peer_cta_rank in cutlass.range_constexpr(CTAS_PER_ROW): + peer_row_sum = prims.mapa(row_sum.data_ptr(), cutlass.Int32(peer_cta_rank)) + cluster_sum = cluster_sum + peer_row_sum.load() + row_sum[0] = cute.math.rsqrt( + cluster_sum * (1.0 / cutlass.Float32(N)) + eps, + fastmath=True, + ) + + if cutlass.const_expr(not ALL_TO_ALL_CLUSTER_REDUCTION): + prims.barrier_cluster_arrive() + prims.barrier_cluster_wait() + + # This is the earliest existing cluster-safe release point: every CTA + # has completed peer polling and residual/sumsq work, and no extra + # barrier is added. The successor may overlap this kernel's RMS + # normalization tail. + if cutlass.const_expr(USE_PDL): + if cutlass.const_expr(PDL_RELEASE_PRE_NORM): + if tidx == 0: + cute.arch.griddepcontrol_launch_dependents() + + root_row_sum = prims.mapa(row_sum.data_ptr(), cutlass.Int32(0)) + inv_rms = root_row_sum.load() + + if IKET_PHASES: + cute.experimental.iket.range_pop() + cute.experimental.iket.range_push("ar_tail_store") + for j in cutlass.range_constexpr(0, chunk_values_per_thread, _BF16_VEC_SIZE): + col_in_cta = tidx * _BF16_VEC_SIZE + j * THREADS_PER_BLOCK + if col_in_cta + _BF16_VEC_SIZE <= cols_per_cta: + col = cta_col_base + col_in_cta + residual_vec = residual_values[ + j:_BF16_VEC_SIZE, + cutlass.align(_ALIGNMENT_BYTES), + ] + weight_vec = weight_values[ + j:_BF16_VEC_SIZE, + cutlass.align(_ALIGNMENT_BYTES), + ] + norm_f32 = residual_vec.to(cutlass.Float32) * inv_rms + norm_f32 = norm_f32 * weight_vec.to(cutlass.Float32) + norm_values[0:_BF16_VEC_SIZE, cutlass.align(_ALIGNMENT_BYTES)] = norm_f32.to( + cutlass.BFloat16 + ) + norm_output_array[row, :].store( + norm_values[0:_BF16_VEC_SIZE, cutlass.align(_ALIGNMENT_BYTES)], + col, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + + if IKET_PHASES: + cute.experimental.iket.range_pop() + if POST_NORM_CLUSTER_BARRIER and not ALL_TO_ALL_CLUSTER_REDUCTION: + prims.barrier_cluster_arrive_relaxed() + prims.barrier_cluster_wait() + + # A successor can launch here, but its entry wait precedes its signal-state + # load, so it cannot select a Lamport slot until this grid has published it. + if cutlass.const_expr(USE_PDL): + if cutlass.const_expr( + not PDL_RELEASE_STRICT and not PDL_RELEASE_PRE_NORM and not PDL_RELEASE_AT_ENTRY + ): + if tidx == 0: + cute.arch.griddepcontrol_launch_dependents() + + if USE_DEVICE_SIGNAL_STATE and not LAST_CTA_SIGNAL_ADVANCE: + if AGGREGATE_SIGNAL_AT_CLUSTER_ROOT: + if (cta_rank == 0) & (tidx == 0): + prims.red( + prims.ReductionOp.ADD, + prims.ReductionType.S32, + state_counter_ptr, + cutlass.Int32(1), + mem_order=prims.MemOrder.RELEASE, + mem_scope=prims.MemScope.GPU, + ) + if (bidx == 0) & (tidx == 0): + if AGGREGATE_SIGNAL_AT_CLUSTER_ROOT: + completed_clusters = cute.arch.load( + state_counter_ptr, + cutlass.Int32, + sem="acquire", + scope="gpu", + ) + while completed_clusters < grid_x // CTAS_PER_ROW: + completed_clusters = cute.arch.load( + state_counter_ptr, + cutlass.Int32, + sem="acquire", + scope="gpu", + ) + else: + entered_ctas = cute.arch.load( + state_counter_ptr, + cutlass.Int32, + sem="acquire", + scope="gpu", + ) + while entered_ctas < grid_x: + entered_ctas = cute.arch.load( + state_counter_ptr, + cutlass.Int32, + sem="acquire", + scope="gpu", + ) + active_vectors = ( + cutlass.Int32(cute.size(input_tensor, mode=[0])) * cutlass.Int32(N) + ) // cutlass.Int32(_BF16_VEC_SIZE) + ping_count_ptr = ( + signal_state.iterator + + cutlass.Int64(_SIGNAL_SLOT_COUNTS_INDEX) + + cutlass.Int64(ping_slot) + ).llvm_ptr + next_signal = (signal_i32 + cutlass.Int32(1)) % cutlass.Int32(_PING_PONG_SIZE) + cute.arch.store( + ping_count_ptr, + active_vectors, + sem="release", + scope="gpu", + ) + cute.arch.store( + state_signal_ptr, + next_signal, + sem="release", + scope="gpu", + ) + cute.arch.store( + state_counter_ptr, + cutlass.Int32(0), + sem="release", + scope="gpu", + ) + if cutlass.const_expr(USE_PDL): + if cutlass.const_expr(PDL_RELEASE_STRICT): + prims.red( + prims.ReductionOp.ADD, + prims.ReductionType.S32, + release_epoch_ptr, + cutlass.Int32(1), + mem_order=prims.MemOrder.RELEASE, + mem_scope=prims.MemScope.GPU, + ) + + if cutlass.const_expr(USE_PDL): + if cutlass.const_expr(PDL_RELEASE_STRICT): + if tidx == 0: + if USE_DEVICE_SIGNAL_STATE: + published_epoch = cute.arch.load( + release_epoch_ptr, + cutlass.Int32, + sem="acquire", + scope="gpu", + ) + while published_epoch == entry_release_epoch: + published_epoch = cute.arch.load( + release_epoch_ptr, + cutlass.Int32, + sem="acquire", + scope="gpu", + ) + cute.arch.griddepcontrol_launch_dependents() + + +@cute.kernel +def _tp_allreduce_residual_rms_norm_out_quant_nvfp4_oneshot_lamport_split_row_kernel( + input_tensor: cute.Tensor, + residual_tensor: cute.Tensor, + norm_weight_tensor: cute.Tensor, + global_scale_tensor: cute.Tensor, + norm_output_tensor: cute.Tensor, + fp4_output_tensor: cute.Tensor, + scale_output_tensor: cute.Tensor, + residual_output_tensor: cute.Tensor, + buffers: list[cute.Tensor], + ready_flags: list[cute.Tensor], + signal_state: cute.Tensor, + multicast_base: cutlass.Int64, + signal: cutlass.Int64, + eps: cutlass.Constexpr[cutlass.Float32], + N: cutlass.Constexpr[int], + LOCAL_RANK: cutlass.Constexpr[int], + WORLD_SIZE: cutlass.Constexpr[int], + CTAS_PER_ROW: cutlass.Constexpr[int], + THREADS_PER_BLOCK: cutlass.Constexpr[int], + USE_READY_FLAGS: cutlass.Constexpr[bool], + USE_DEVICE_SIGNAL_STATE: cutlass.Constexpr[bool], + PEER_ORDER: cutlass.Constexpr[tuple[int, ...]], + ELIDE_SELF_RANK: cutlass.Constexpr[bool], + AGGREGATE_SIGNAL_AT_CLUSTER_ROOT: cutlass.Constexpr[bool], + USE_PDL: cutlass.Constexpr[bool], + PDL_RELEASE_STRICT: cutlass.Constexpr[bool], + PDL_RELEASE_PRE_NORM: cutlass.Constexpr[bool], + PDL_RELEASE_AT_ENTRY: cutlass.Constexpr[bool], + LAST_CTA_SIGNAL_ADVANCE: cutlass.Constexpr[bool], + ALL_TO_ALL_CLUSTER_REDUCTION: cutlass.Constexpr[bool], + EARLY_INPUT_LOAD: cutlass.Constexpr[bool], + MULTICAST_PUBLISH: cutlass.Constexpr[bool], + IKET_PHASES: cutlass.Constexpr[bool], + ROTATED_POLL_ORDER: cutlass.Constexpr[bool], + PREFETCH_FUSED_INPUTS: cutlass.Constexpr[bool], +): + tidx, _, _ = cute.arch.thread_idx() + bidx, _, _ = cute.arch.block_idx() + grid_x, _, _ = cute.arch.grid_dim() + cta_rank = cute.arch.block_idx_in_cluster() + row = bidx // CTAS_PER_ROW + + input_array = cutlass.make_array_view(input_tensor) + residual_array = cutlass.make_array_view(residual_tensor) + norm_weight_array = cutlass.make_array_view(norm_weight_tensor) + norm_output_array = cutlass.make_array_view(norm_output_tensor) + residual_output_array = cutlass.make_array_view(residual_output_tensor) + buffer_arrays = [cutlass.make_array_view(buffer) for buffer in buffers] + local_buffer = buffer_arrays[LOCAL_RANK] + ready_flag_arrays = [cutlass.make_array_view(ready_flag) for ready_flag in ready_flags] + local_ready_flags = ready_flag_arrays[LOCAL_RANK] + + cols_per_cta = N // CTAS_PER_ROW + chunk_values_per_thread = (cols_per_cta + THREADS_PER_BLOCK - 1) // THREADS_PER_BLOCK + chunk_values_padded = ( + (chunk_values_per_thread + _BF16_VEC_SIZE - 1) // _BF16_VEC_SIZE + ) * _BF16_VEC_SIZE + residual_values = cutlass.Array( + cutlass.BFloat16, + chunk_values_padded, + alignment=_ALIGNMENT_BYTES, + ) + weight_values = cutlass.Array( + cutlass.BFloat16, + chunk_values_padded, + alignment=_ALIGNMENT_BYTES, + ) + norm_values = cutlass.Array( + cutlass.BFloat16, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + warp_sums = cutlass.Array( + cutlass.Float32, + (THREADS_PER_BLOCK // 32), + space=cutlass.AddressSpace.smem, + alignment=_ALIGNMENT_BYTES, + ) + row_sum = cutlass.Array( + cutlass.Float32, + 1, + space=cutlass.AddressSpace.smem, + alignment=_ALIGNMENT_BYTES, + ) + # One slot per CTA of the cluster; written remotely via DSMEM by the + # all-to-all reduction (each slot has exactly one writer). + cluster_partials = cutlass.Array( + cutlass.Float32, + CTAS_PER_ROW, + space=cutlass.AddressSpace.smem, + alignment=_ALIGNMENT_BYTES, + ) + + lane_id = tidx % 32 + warp_id = tidx // 32 + numel = cute.size(buffers[0], mode=[2]) + row_base = row * N + cta_col_base = cta_rank * cutlass.Int32(cols_per_cta) + state_counter_ptr = (signal_state.iterator + cutlass.Int64(_SIGNAL_COUNTER_INDEX)).llvm_ptr + state_signal_ptr = (signal_state.iterator + cutlass.Int64(_SIGNAL_VALUE_INDEX)).llvm_ptr + release_epoch_ptr = ( + signal_state.iterator + cutlass.Int64(_SIGNAL_RELEASE_EPOCH_INDEX) + ).llvm_ptr + signal_i32 = cutlass.Int32(0) + signal_i64 = signal + entry_release_epoch = cutlass.Int32(0) + if IKET_PHASES: + cute.experimental.iket.range_push("ar_entry") + if cutlass.const_expr(USE_PDL): + cute.arch.griddepcontrol_wait() + # The NVFP4 global scale is a per-layer constant; issuing its load here lets + # the round trip overlap the signal-state load and the Lamport protocol + # instead of sitting on the critical path of the quant tail. + global_scale_ptr = global_scale_tensor.iterator.raw_ptr() + global_scale = cutlass.Float32(0.0) + if PREFETCH_FUSED_INPUTS: + global_scale = prims.load_ext(global_scale_ptr, dtype=cutlass.Float32) + # Issue the local input load before the signal-state acquire load so the two + # global round trips overlap; publication then starts as soon as both + # arrive instead of after their sum. + early_input_values = cutlass.Array( + cutlass.BFloat16, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + if EARLY_INPUT_LOAD: + early_col_in_cta = tidx * _BF16_VEC_SIZE + if early_col_in_cta + _BF16_VEC_SIZE <= cols_per_cta: + early_input_values[0:_BF16_VEC_SIZE, cutlass.align(_ALIGNMENT_BYTES)] = input_array[ + row, : + ].load( + cta_col_base + early_col_in_cta, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + if USE_DEVICE_SIGNAL_STATE: + if cutlass.const_expr(USE_PDL): + if cutlass.const_expr(PDL_RELEASE_STRICT): + entry_release_epoch = cute.arch.load( + release_epoch_ptr, + cutlass.Int32, + sem="acquire", + scope="gpu", + ) + signal_i32 = cute.arch.load( + state_signal_ptr, + cutlass.Int32, + sem="acquire", + scope="gpu", + ) + signal_i64 = cutlass.Int64(signal_i32) + if (not AGGREGATE_SIGNAL_AT_CLUSTER_ROOT) and (not LAST_CTA_SIGNAL_ADVANCE) and tidx == 0: + prims.red( + prims.ReductionOp.ADD, + prims.ReductionType.S32, + state_counter_ptr, + cutlass.Int32(1), + mem_order=prims.MemOrder.RELEASE, + mem_scope=prims.MemScope.GPU, + ) + + # Deliberately after the device signal snapshot: a PDL successor may start + # immediately, but its entry wait cannot observe stale slot state. + if cutlass.const_expr(USE_PDL): + if cutlass.const_expr(PDL_RELEASE_AT_ENTRY): + if tidx == 0: + cute.arch.griddepcontrol_launch_dependents() + ping_slot = signal_i64 % _PING_PONG_SIZE + clear_slot = (signal_i64 + cutlass.Int64(1)) % _PING_PONG_SIZE + numel_i64 = cutlass.Int64(numel) + ping_base = cutlass.Int64(ping_slot) * cutlass.Int64(WORLD_SIZE) * numel_i64 + clear_base = cutlass.Int64(clear_slot) * cutlass.Int64(WORLD_SIZE) * numel_i64 + num_vectors = cute.size(ready_flags[0], mode=[2]) + num_vectors_i64 = cutlass.Int64(num_vectors) + ping_ready_base = cutlass.Int64(ping_slot) * cutlass.Int64(WORLD_SIZE) * num_vectors_i64 + clear_ready_base = cutlass.Int64(clear_slot) * cutlass.Int64(WORLD_SIZE) * num_vectors_i64 + neg_zero_vec = cutlass.Array( + cutlass.BFloat16, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + neg_zero_bits = cutlass.Int16(_NEGATIVE_ZERO_BF16_BITS) + for lane in cutlass.range_constexpr(_BF16_VEC_SIZE): + neg_zero_vec[lane] = neg_zero_bits.bitcast(cutlass.BFloat16) + + ready_zero = cutlass.full((1,), 0, dtype=cutlass.Int32) + local_sum_sq = cutlass.Float32(0.0) + + if IKET_PHASES: + cute.experimental.iket.range_pop() + cute.experimental.iket.range_push("ar_clear") + if USE_DEVICE_SIGNAL_STATE: + clear_count_ptr = ( + signal_state.iterator + + cutlass.Int64(_SIGNAL_SLOT_COUNTS_INDEX) + + cutlass.Int64(clear_slot) + ).llvm_ptr + clear_vectors = cute.arch.load( + clear_count_ptr, + cutlass.Int32, + sem="acquire", + scope="gpu", + ) + clear_vector_idx = bidx * THREADS_PER_BLOCK + tidx + clear_stride = grid_x * THREADS_PER_BLOCK + while clear_vector_idx < clear_vectors: + clear_vector_idx_i64 = cutlass.Int64(clear_vector_idx) + if USE_READY_FLAGS: + for source_rank in cutlass.range_constexpr(WORLD_SIZE): + if (not ELIDE_SELF_RANK) or source_rank != LOCAL_RANK: + clear_ready_offset = ( + clear_ready_base + + cutlass.Int64(source_rank) * num_vectors_i64 + + clear_vector_idx_i64 + ) + local_ready_flags.store( + ready_zero, + clear_ready_offset, + 1, + alignment=_READY_FLAG_ALIGNMENT_BYTES, + is_volatile=True, + ) + else: + for source_rank in cutlass.range_constexpr(WORLD_SIZE): + if (not ELIDE_SELF_RANK) or source_rank != LOCAL_RANK: + clear_offset = ( + clear_base + + cutlass.Int64(source_rank) * numel_i64 + + clear_vector_idx_i64 * cutlass.Int64(_BF16_VEC_SIZE) + ) + local_buffer.store( + neg_zero_vec[ + 0:_BF16_VEC_SIZE, + cutlass.align(_ALIGNMENT_BYTES), + ], + clear_offset, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + is_volatile=True, + ) + clear_vector_idx = clear_vector_idx + clear_stride + + if IKET_PHASES: + cute.experimental.iket.range_pop() + cute.experimental.iket.range_push("ar_load_publish") + for j in cutlass.range_constexpr(0, chunk_values_per_thread, _BF16_VEC_SIZE): + col_in_cta = tidx * _BF16_VEC_SIZE + j * THREADS_PER_BLOCK + if col_in_cta + _BF16_VEC_SIZE <= cols_per_cta: + col = cta_col_base + col_in_cta + base_idx = row_base + col + base_idx_i64 = cutlass.Int64(base_idx) + vector_idx_i64 = base_idx_i64 // cutlass.Int64(_BF16_VEC_SIZE) + + if not USE_DEVICE_SIGNAL_STATE: + if USE_READY_FLAGS: + for source_rank in cutlass.range_constexpr(WORLD_SIZE): + if (not ELIDE_SELF_RANK) or source_rank != LOCAL_RANK: + clear_ready_offset = ( + clear_ready_base + + cutlass.Int64(source_rank) * num_vectors_i64 + + vector_idx_i64 + ) + local_ready_flags.store( + ready_zero, + clear_ready_offset, + 1, + alignment=_READY_FLAG_ALIGNMENT_BYTES, + is_volatile=True, + ) + else: + for source_rank in cutlass.range_constexpr(WORLD_SIZE): + if (not ELIDE_SELF_RANK) or source_rank != LOCAL_RANK: + clear_offset = ( + clear_base + cutlass.Int64(source_rank) * numel_i64 + base_idx_i64 + ) + local_buffer.store( + neg_zero_vec[ + 0:_BF16_VEC_SIZE, + cutlass.align(_ALIGNMENT_BYTES), + ], + clear_offset, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + is_volatile=True, + ) + + # Both arms assign into the same array so the frontend sees one + # container structure before and after the branch. + input_values_array = cutlass.Array( + cutlass.BFloat16, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + if EARLY_INPUT_LOAD and j == 0: + input_values_array[0:_BF16_VEC_SIZE, cutlass.align(_ALIGNMENT_BYTES)] = ( + early_input_values[0:_BF16_VEC_SIZE, cutlass.align(_ALIGNMENT_BYTES)] + ) + else: + input_values_array[0:_BF16_VEC_SIZE, cutlass.align(_ALIGNMENT_BYTES)] = input_array[ + row, : + ].load( + col, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + input_values = input_values_array[0:_BF16_VEC_SIZE, cutlass.align(_ALIGNMENT_BYTES)] + sanitized_input_values = cutlass.Array( + cutlass.BFloat16, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + if not USE_READY_FLAGS: + neg_zero_bits = cutlass.Int16(_NEGATIVE_ZERO_BF16_BITS) + for lane in cutlass.range_constexpr(_BF16_VEC_SIZE): + lane_value = input_values[lane] + if lane_value.bitcast(cutlass.Int16) == neg_zero_bits: + lane_value = cutlass.BFloat16(0.0) + sanitized_input_values[lane] = lane_value + + if MULTICAST_PUBLISH: + # One NVLS multicast store lands in slot[LOCAL_RANK] of every + # rank (including this one); the fabric does the replication, + # so egress bytes drop from (WORLD_SIZE-1)x to 1x. Sentinel + # polling on the receiving side is unchanged. + multicast_offset = ping_base + cutlass.Int64(LOCAL_RANK) * numel_i64 + base_idx_i64 + _multicast_publish_bf16x8( + multicast_base + multicast_offset * cutlass.Int64(2), + sanitized_input_values[ + 0:_BF16_VEC_SIZE, + cutlass.align(_ALIGNMENT_BYTES), + ], + ) + for peer_ordinal in cutlass.range_constexpr(WORLD_SIZE): + target_rank = PEER_ORDER[peer_ordinal] + if (not MULTICAST_PUBLISH) and ((not ELIDE_SELF_RANK) or target_rank != LOCAL_RANK): + write_offset = ping_base + cutlass.Int64(LOCAL_RANK) * numel_i64 + base_idx_i64 + if USE_READY_FLAGS: + ready_offset = ( + ping_ready_base + + cutlass.Int64(LOCAL_RANK) * num_vectors_i64 + + vector_idx_i64 + ) + buffer_arrays[target_rank].store( + input_values, + write_offset, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + is_volatile=True, + ) + prims.red( + prims.ReductionOp.ADD, + prims.ReductionType.S32, + (ready_flags[target_rank].iterator + ready_offset).llvm_ptr, + cutlass.Int32(1), + mem_order=prims.MemOrder.RELEASE, + mem_scope=prims.MemScope.SYS, + ) + else: + buffer_arrays[target_rank].store( + sanitized_input_values[ + 0:_BF16_VEC_SIZE, + cutlass.align(_ALIGNMENT_BYTES), + ], + write_offset, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + is_volatile=True, + ) + + # Defined before the constexpr branches so the value survives staged + # control flow (DSL requirement, as in the fused RMSNorm kernel). + residual_vec = cutlass.full((_BF16_VEC_SIZE,), 0.0, dtype=cutlass.BFloat16) + weight_vec = cutlass.full((_BF16_VEC_SIZE,), 0.0, dtype=cutlass.BFloat16) + if PREFETCH_FUSED_INPUTS: + # Issue the residual and RMS-weight loads after publication and + # before polling so their latency hides under NVLink propagation + # (the fused RMSNorm kernel does the same). + residual_vec = residual_array[row, :].load( + col, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + weight_vec = norm_weight_array.load( + col, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + if IKET_PHASES: + cute.experimental.iket.range_pop() + cute.experimental.iket.range_push("ar_poll") + # Counter atomic is issued after publication so its round trip + # overlaps NVLink propagation instead of delaying the publish. + if USE_DEVICE_SIGNAL_STATE and LAST_CTA_SIGNAL_ADVANCE and j == 0: + if tidx == 0: + active_vectors_i32 = ( + cutlass.Int32(cute.size(input_tensor, mode=[0])) * cutlass.Int32(N) + ) // cutlass.Int32(_BF16_VEC_SIZE) + _advance_signal_if_last_cta( + state_counter_ptr, + state_signal_ptr, + signal_state, + signal_i32, + ping_slot, + active_vectors_i32, + grid_x, + ) + + acc = cutlass.full((_BF16_VEC_SIZE,), 0.0, dtype=cutlass.Float32) + + peer_values_all = cutlass.Array( + cutlass.BFloat16, + WORLD_SIZE * _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + for peer_ordinal in cutlass.range_constexpr(WORLD_SIZE): + source_rank = PEER_ORDER[peer_ordinal] if ROTATED_POLL_ORDER else peer_ordinal + if (not ELIDE_SELF_RANK) or source_rank != LOCAL_RANK: + read_offset = ping_base + cutlass.Int64(source_rank) * numel_i64 + base_idx_i64 + peer_values = cutlass.full( + (_BF16_VEC_SIZE,), + 0.0, + dtype=cutlass.BFloat16, + ) + if USE_READY_FLAGS: + read_ready_offset = ( + ping_ready_base + + cutlass.Int64(source_rank) * num_vectors_i64 + + vector_idx_i64 + ) + ready_value = cute.arch.load( + (ready_flags[LOCAL_RANK].iterator + read_ready_offset).llvm_ptr, + cutlass.Int32, + sem="acquire", + scope="sys", + ) + while ready_value < cutlass.Int32(1): + ready_value = cute.arch.load( + (ready_flags[LOCAL_RANK].iterator + read_ready_offset).llvm_ptr, + cutlass.Int32, + sem="acquire", + scope="sys", + ) + peer_values = local_buffer.load( + read_offset, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + is_volatile=True, + ) + else: + peer_values = local_buffer.load( + read_offset, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + is_volatile=True, + ) + while not _all_bf16x8_lanes_ready(peer_values): + peer_values = local_buffer.load( + read_offset, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + is_volatile=True, + ) + + if ROTATED_POLL_ORDER: + peer_values_all[ + source_rank * _BF16_VEC_SIZE : _BF16_VEC_SIZE, + cutlass.align(_ALIGNMENT_BYTES), + ] = peer_values + else: + acc = acc + peer_values.to(cutlass.Float32) + if (not ROTATED_POLL_ORDER) and ELIDE_SELF_RANK and source_rank == LOCAL_RANK: + # Own partial enters the sum at its rank position so every + # rank accumulates 0 -> WORLD_SIZE-1 (bit-identical, as TRT-LLM). + acc = acc + input_values.to(cutlass.Float32) + if ROTATED_POLL_ORDER: + # Polled in PEER_ORDER (latency), summed in absolute rank order (numerics). + for source_rank in cutlass.range_constexpr(WORLD_SIZE): + if (not ELIDE_SELF_RANK) or source_rank != LOCAL_RANK: + peer_values = peer_values_all[ + source_rank * _BF16_VEC_SIZE : _BF16_VEC_SIZE, + cutlass.align(_ALIGNMENT_BYTES), + ] + acc = acc + peer_values.to(cutlass.Float32) + if ELIDE_SELF_RANK and source_rank == LOCAL_RANK: + acc = acc + input_values.to(cutlass.Float32) + if IKET_PHASES: + cute.experimental.iket.range_pop() + cute.experimental.iket.range_push("ar_residual") + if not PREFETCH_FUSED_INPUTS: + residual_vec = residual_array[row, :].load( + col, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + weight_vec = norm_weight_array.load( + col, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + residual_out_vec = (acc + residual_vec.to(cutlass.Float32)).to(cutlass.BFloat16) + residual_output_array[row, :].store( + residual_out_vec, + col, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + residual_values[j:_BF16_VEC_SIZE, cutlass.align(_ALIGNMENT_BYTES)] = residual_out_vec + weight_values[j:_BF16_VEC_SIZE, cutlass.align(_ALIGNMENT_BYTES)] = weight_vec + + residual_f32 = residual_out_vec.to(cutlass.Float32) + local_sum_sq = local_sum_sq + (residual_f32 * residual_f32).reduce("add") + + if IKET_PHASES: + cute.experimental.iket.range_pop() + cute.experimental.iket.range_push("ar_rms_reduce") + warp_sum = _warp_sum_f32(local_sum_sq) + if lane_id == 0: + warp_sums[warp_id] = warp_sum + + prims.barrier_cta_sync(0) + + cta_sum = cutlass.Float32(0.0) + if tidx == 0: + for warp_idx in cutlass.range_constexpr((THREADS_PER_BLOCK // 32)): + cta_sum = cta_sum + warp_sums[warp_idx] + if ALL_TO_ALL_CLUSTER_REDUCTION: + _publish_cta_partial_to_cluster(cluster_partials, cta_sum, cta_rank, CTAS_PER_ROW) + elif cta_rank == 0: + row_sum[0] = cutlass.Float32(0.0) + + inv_rms = cutlass.Float32(0.0) + if cutlass.const_expr(ALL_TO_ALL_CLUSTER_REDUCTION): + # Single cluster rendezvous; afterwards no CTA touches another CTA's + # shared memory, so no lifetime barrier is needed before exit. + prims.barrier_cluster_arrive() + prims.barrier_cluster_wait() + if cutlass.const_expr(USE_PDL): + if cutlass.const_expr(PDL_RELEASE_PRE_NORM): + if tidx == 0: + cute.arch.griddepcontrol_launch_dependents() + inv_rms = _cluster_inv_rms_from_partials(cluster_partials, CTAS_PER_ROW, N, eps) + else: + prims.barrier_cta_sync(0) + prims.barrier_cluster_arrive() + prims.barrier_cluster_wait() + + if tidx == 0: + root_row_sum = prims.mapa(row_sum.data_ptr(), cutlass.Int32(0)) + prims.red( + prims.ReductionOp.ADD, + prims.ReductionType.F32, + root_row_sum, + cta_sum, + mem_order=prims.MemOrder.RELAXED, + mem_scope=prims.MemScope.CLUSTER, + ) + + prims.barrier_cluster_arrive() + prims.barrier_cluster_wait() + + if (cta_rank == 0) & (tidx == 0): + row_sum[0] = cute.math.rsqrt( + row_sum[0] * (1.0 / cutlass.Float32(N)) + eps, + fastmath=True, + ) + + prims.barrier_cta_sync(0) + prims.barrier_cluster_arrive() + prims.barrier_cluster_wait() + + if cutlass.const_expr(USE_PDL): + if cutlass.const_expr(PDL_RELEASE_PRE_NORM): + if tidx == 0: + cute.arch.griddepcontrol_launch_dependents() + + root_row_sum = prims.mapa(row_sum.data_ptr(), cutlass.Int32(0)) + inv_rms = root_row_sum.load() + if IKET_PHASES: + cute.experimental.iket.range_pop() + cute.experimental.iket.range_push("ar_tail_store") + fp4_ptr = fp4_output_tensor.iterator.raw_ptr() + sf_ptr = scale_output_tensor.iterator.raw_ptr() + if not PREFETCH_FUSED_INPUTS: + global_scale = prims.load_ext(global_scale_ptr, dtype=cutlass.Float32) + scale_blocks_per_row: cutlass.Constexpr = N // _SCALE_BLOCK + + for j in cutlass.range_constexpr(0, chunk_values_per_thread, _BF16_VEC_SIZE): + col_in_cta = tidx * _BF16_VEC_SIZE + j * THREADS_PER_BLOCK + if col_in_cta + _BF16_VEC_SIZE <= cols_per_cta: + col = cta_col_base + col_in_cta + residual_vec = residual_values[ + j:_BF16_VEC_SIZE, + cutlass.align(_ALIGNMENT_BYTES), + ] + weight_vec = weight_values[ + j:_BF16_VEC_SIZE, + cutlass.align(_ALIGNMENT_BYTES), + ] + norm_f32 = residual_vec.to(cutlass.Float32) * inv_rms + norm_f32 = norm_f32 * weight_vec.to(cutlass.Float32) + norm_values[0:_BF16_VEC_SIZE, cutlass.align(_ALIGNMENT_BYTES)] = norm_f32.to( + cutlass.BFloat16 + ) + quant_f32 = norm_f32.to(cutlass.BFloat16).to(cutlass.Float32) + norm_output_array[row, :].store( + norm_values[0:_BF16_VEC_SIZE, cutlass.align(_ALIGNMENT_BYTES)], + col, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + + local_max = cutlass.Float32(0.0) + for lane in cutlass.range_constexpr(_BF16_VEC_SIZE): + magnitude = cute.math.abs(quant_f32[lane]) + if magnitude > local_max: + local_max = magnitude + + block_max = local_max + peer_max = cute.arch.shuffle_sync_bfly(block_max, 1) + if peer_max > block_max: + block_max = peer_max + + sf_unquantized = global_scale * block_max * cutlass.Float32(1.0 / 6.0) + sf_packed = prims.cvt_f32x2_to_f8x2( + sf_unquantized, + sf_unquantized, + cutlass.Float8E4M3FN, + rnd=prims.FPRoundingMode.RN, + sat=prims.SaturationMode.SATFINITE, + ) + sf_code = cutlass.Int32(sf_packed) & cutlass.Int32(0xFF) + half_block = (col // cutlass.Int32(_BF16_VEC_SIZE)) & cutlass.Int32(1) + scale_block = col // cutlass.Int32(_SCALE_BLOCK) + if half_block == cutlass.Int32(0): + sf_index = row * cutlass.Int32(scale_blocks_per_row) + scale_block + prims.store_ext(cutlass.Int8(sf_code), sf_ptr + sf_index) + + narrowed_sf = _decode_positive_e4m3(sf_code) + output_scale = cutlass.Float32(0.0) + if block_max != cutlass.Float32(0.0): + output_scale = cute.arch.rcp_approx( + narrowed_sf * cute.arch.rcp_approx(global_scale) + ) + + packed_word = cutlass.Int32(0) + for pair in cutlass.range_constexpr(_BF16_VEC_SIZE // 2): + low = quant_f32[2 * pair] * output_scale + high = quant_f32[2 * pair + 1] * output_scale + packed_byte = prims.cvt_f32x2_to_f4x2( + high, + low, + cutlass.Float4E2M1FN, + rnd=prims.FPRoundingMode.RN, + ) & cutlass.Int32(0xFF) + packed_word = packed_word | (packed_byte << cutlass.Int32(pair * 8)) + fp4_index = ( + row * cutlass.Int32(N // 2) + + scale_block * cutlass.Int32(_SCALE_BLOCK // 2) + + half_block * cutlass.Int32(_BF16_VEC_SIZE // 2) + ) + prims.store_ext(packed_word, fp4_ptr + fp4_index) + + if IKET_PHASES: + cute.experimental.iket.range_pop() + if cutlass.const_expr(not ALL_TO_ALL_CLUSTER_REDUCTION): + # Keeps the root CTA's shared memory alive until every peer has read + # inv_rms through DSMEM; the all-to-all path has no such read. + prims.barrier_cluster_arrive_relaxed() + prims.barrier_cluster_wait() + + # A successor can launch here, but its entry wait precedes its signal-state + # load, so it cannot select a Lamport slot until this grid has published it. + if cutlass.const_expr(USE_PDL): + if cutlass.const_expr( + not PDL_RELEASE_STRICT and not PDL_RELEASE_PRE_NORM and not PDL_RELEASE_AT_ENTRY + ): + if tidx == 0: + cute.arch.griddepcontrol_launch_dependents() + + if USE_DEVICE_SIGNAL_STATE and not LAST_CTA_SIGNAL_ADVANCE: + if AGGREGATE_SIGNAL_AT_CLUSTER_ROOT and (cta_rank == 0) and (tidx == 0): + prims.red( + prims.ReductionOp.ADD, + prims.ReductionType.S32, + state_counter_ptr, + cutlass.Int32(CTAS_PER_ROW), + mem_order=prims.MemOrder.RELEASE, + mem_scope=prims.MemScope.GPU, + ) + if (bidx == 0) & (tidx == 0): + entered_ctas = cute.arch.load( + state_counter_ptr, + cutlass.Int32, + sem="acquire", + scope="gpu", + ) + while entered_ctas < grid_x: + entered_ctas = cute.arch.load( + state_counter_ptr, + cutlass.Int32, + sem="acquire", + scope="gpu", + ) + active_vectors = ( + cutlass.Int32(cute.size(input_tensor, mode=[0])) * cutlass.Int32(N) + ) // cutlass.Int32(_BF16_VEC_SIZE) + ping_count_ptr = ( + signal_state.iterator + + cutlass.Int64(_SIGNAL_SLOT_COUNTS_INDEX) + + cutlass.Int64(ping_slot) + ).llvm_ptr + next_signal = (signal_i32 + cutlass.Int32(1)) % cutlass.Int32(_PING_PONG_SIZE) + cute.arch.store( + ping_count_ptr, + active_vectors, + sem="release", + scope="gpu", + ) + cute.arch.store( + state_signal_ptr, + next_signal, + sem="release", + scope="gpu", + ) + cute.arch.store( + state_counter_ptr, + cutlass.Int32(0), + sem="release", + scope="gpu", + ) + if cutlass.const_expr(USE_PDL): + if cutlass.const_expr(PDL_RELEASE_STRICT): + prims.red( + prims.ReductionOp.ADD, + prims.ReductionType.S32, + release_epoch_ptr, + cutlass.Int32(1), + mem_order=prims.MemOrder.RELEASE, + mem_scope=prims.MemScope.GPU, + ) + + if cutlass.const_expr(USE_PDL): + if cutlass.const_expr(PDL_RELEASE_STRICT): + if tidx == 0: + if USE_DEVICE_SIGNAL_STATE: + published_epoch = cute.arch.load( + release_epoch_ptr, + cutlass.Int32, + sem="acquire", + scope="gpu", + ) + while published_epoch == entry_release_epoch: + published_epoch = cute.arch.load( + release_epoch_ptr, + cutlass.Int32, + sem="acquire", + scope="gpu", + ) + cute.arch.griddepcontrol_launch_dependents() + + +@cute.jit +def _tp_allreduce_residual_rms_norm_bf16_oneshot_lamport( + input_tensor: cute.Tensor, + residual_tensor: cute.Tensor, + norm_weight_tensor: cute.Tensor, + norm_output_tensor: cute.Tensor, + residual_output_tensor: cute.Tensor, + buffers: list[cute.Tensor], + ready_flags: list[cute.Tensor], + signal: cutlass.Int64, + stream: cuda.CUstream, + eps: cutlass.Constexpr[cutlass.Float32], + N: cutlass.Constexpr[int], + LOCAL_RANK: cutlass.Constexpr[int], + WORLD_SIZE: cutlass.Constexpr[int], +) -> None: + M = input_tensor.shape[0] + _tp_allreduce_residual_rms_norm_bf16_oneshot_lamport_kernel( + input_tensor, + residual_tensor, + norm_weight_tensor, + norm_output_tensor, + residual_output_tensor, + buffers, + ready_flags, + signal, + eps, + N, + LOCAL_RANK, + WORLD_SIZE, + ).launch( + grid=(M, 1, 1), + block=(_THREADS_PER_BLOCK, 1, 1), + stream=stream, + ) + + +@cute.jit +def _tp_allreduce_residual_rms_norm_bf16_oneshot_lamport_split_row( + input_tensor: cute.Tensor, + residual_tensor: cute.Tensor, + norm_weight_tensor: cute.Tensor, + norm_output_tensor: cute.Tensor, + residual_output_tensor: cute.Tensor, + buffers: list[cute.Tensor], + ready_flags: list[cute.Tensor], + signal_state: cute.Tensor, + multicast_base: cutlass.Int64, + signal: cutlass.Int64, + stream: cuda.CUstream, + eps: cutlass.Constexpr[cutlass.Float32], + N: cutlass.Constexpr[int], + LOCAL_RANK: cutlass.Constexpr[int], + WORLD_SIZE: cutlass.Constexpr[int], + CTAS_PER_ROW: cutlass.Constexpr[int], + THREADS_PER_BLOCK: cutlass.Constexpr[int], + USE_READY_FLAGS: cutlass.Constexpr[bool], + USE_DEVICE_SIGNAL_STATE: cutlass.Constexpr[bool], + ELIDE_SELF_RANK: cutlass.Constexpr[bool], + PEER_ORDER: cutlass.Constexpr[tuple[int, ...]], + PREFETCH_FUSED_INPUTS: cutlass.Constexpr[bool], + DIRECT_CLUSTER_REDUCTION: cutlass.Constexpr[bool], + POST_NORM_CLUSTER_BARRIER: cutlass.Constexpr[bool], + FP32_ACCUMULATION: cutlass.Constexpr[bool], + BATCH_POLL_PEERS: cutlass.Constexpr[bool], + READY_MASK_POLL_PEERS: cutlass.Constexpr[bool], + AGGREGATE_SIGNAL_AT_CLUSTER_ROOT: cutlass.Constexpr[bool], + USE_PDL: cutlass.Constexpr[bool], + PDL_RELEASE_STRICT: cutlass.Constexpr[bool], + PDL_RELEASE_PRE_NORM: cutlass.Constexpr[bool], + PDL_RELEASE_AT_ENTRY: cutlass.Constexpr[bool], + LAST_CTA_SIGNAL_ADVANCE: cutlass.Constexpr[bool], + ALL_TO_ALL_CLUSTER_REDUCTION: cutlass.Constexpr[bool], + EARLY_INPUT_LOAD: cutlass.Constexpr[bool], + MULTICAST_PUBLISH: cutlass.Constexpr[bool], + IKET_PHASES: cutlass.Constexpr[bool], + ROTATED_POLL_ORDER: cutlass.Constexpr[bool], +) -> None: + M = input_tensor.shape[0] + _tp_allreduce_residual_rms_norm_bf16_oneshot_lamport_split_row_kernel( + input_tensor, + residual_tensor, + norm_weight_tensor, + norm_output_tensor, + residual_output_tensor, + buffers, + ready_flags, + signal_state, + multicast_base, + signal, + eps, + N, + LOCAL_RANK, + WORLD_SIZE, + CTAS_PER_ROW, + THREADS_PER_BLOCK, + USE_READY_FLAGS, + USE_DEVICE_SIGNAL_STATE, + ELIDE_SELF_RANK, + PEER_ORDER, + PREFETCH_FUSED_INPUTS, + DIRECT_CLUSTER_REDUCTION, + POST_NORM_CLUSTER_BARRIER, + FP32_ACCUMULATION, + BATCH_POLL_PEERS, + READY_MASK_POLL_PEERS, + AGGREGATE_SIGNAL_AT_CLUSTER_ROOT, + USE_PDL, + PDL_RELEASE_STRICT, + PDL_RELEASE_PRE_NORM, + PDL_RELEASE_AT_ENTRY, + LAST_CTA_SIGNAL_ADVANCE, + ALL_TO_ALL_CLUSTER_REDUCTION, + EARLY_INPUT_LOAD, + MULTICAST_PUBLISH, + IKET_PHASES, + ROTATED_POLL_ORDER, + ).launch( + grid=(M * CTAS_PER_ROW, 1, 1), + block=(THREADS_PER_BLOCK, 1, 1), + cluster=(CTAS_PER_ROW, 1, 1), + stream=stream, + use_pdl=USE_PDL, + ) + + +@cute.jit +def _tp_allreduce_residual_rms_norm_out_quant_nvfp4_oneshot_lamport_split_row( + input_tensor: cute.Tensor, + residual_tensor: cute.Tensor, + norm_weight_tensor: cute.Tensor, + global_scale_tensor: cute.Tensor, + norm_output_tensor: cute.Tensor, + fp4_output_tensor: cute.Tensor, + scale_output_tensor: cute.Tensor, + residual_output_tensor: cute.Tensor, + buffers: list[cute.Tensor], + ready_flags: list[cute.Tensor], + signal_state: cute.Tensor, + multicast_base: cutlass.Int64, + signal: cutlass.Int64, + stream: cuda.CUstream, + eps: cutlass.Constexpr[cutlass.Float32], + N: cutlass.Constexpr[int], + LOCAL_RANK: cutlass.Constexpr[int], + WORLD_SIZE: cutlass.Constexpr[int], + CTAS_PER_ROW: cutlass.Constexpr[int], + THREADS_PER_BLOCK: cutlass.Constexpr[int], + USE_READY_FLAGS: cutlass.Constexpr[bool], + USE_DEVICE_SIGNAL_STATE: cutlass.Constexpr[bool], + PEER_ORDER: cutlass.Constexpr[tuple[int, ...]], + ELIDE_SELF_RANK: cutlass.Constexpr[bool], + AGGREGATE_SIGNAL_AT_CLUSTER_ROOT: cutlass.Constexpr[bool], + USE_PDL: cutlass.Constexpr[bool], + PDL_RELEASE_STRICT: cutlass.Constexpr[bool], + PDL_RELEASE_PRE_NORM: cutlass.Constexpr[bool], + PDL_RELEASE_AT_ENTRY: cutlass.Constexpr[bool], + LAST_CTA_SIGNAL_ADVANCE: cutlass.Constexpr[bool], + ALL_TO_ALL_CLUSTER_REDUCTION: cutlass.Constexpr[bool], + EARLY_INPUT_LOAD: cutlass.Constexpr[bool], + MULTICAST_PUBLISH: cutlass.Constexpr[bool], + IKET_PHASES: cutlass.Constexpr[bool], + ROTATED_POLL_ORDER: cutlass.Constexpr[bool], + PREFETCH_FUSED_INPUTS: cutlass.Constexpr[bool], +) -> None: + M = input_tensor.shape[0] + _tp_allreduce_residual_rms_norm_out_quant_nvfp4_oneshot_lamport_split_row_kernel( + input_tensor, + residual_tensor, + norm_weight_tensor, + global_scale_tensor, + norm_output_tensor, + fp4_output_tensor, + scale_output_tensor, + residual_output_tensor, + buffers, + ready_flags, + signal_state, + multicast_base, + signal, + eps, + N, + LOCAL_RANK, + WORLD_SIZE, + CTAS_PER_ROW, + THREADS_PER_BLOCK, + USE_READY_FLAGS, + USE_DEVICE_SIGNAL_STATE, + PEER_ORDER, + ELIDE_SELF_RANK, + AGGREGATE_SIGNAL_AT_CLUSTER_ROOT, + USE_PDL, + PDL_RELEASE_STRICT, + PDL_RELEASE_PRE_NORM, + PDL_RELEASE_AT_ENTRY, + LAST_CTA_SIGNAL_ADVANCE, + ALL_TO_ALL_CLUSTER_REDUCTION, + EARLY_INPUT_LOAD, + MULTICAST_PUBLISH, + IKET_PHASES, + PREFETCH_FUSED_INPUTS, + ROTATED_POLL_ORDER, + ).launch( + grid=(M * CTAS_PER_ROW, 1, 1), + block=(THREADS_PER_BLOCK, 1, 1), + cluster=(CTAS_PER_ROW, 1, 1), + stream=stream, + use_pdl=USE_PDL, + ) + + +# --------------------------------------------------------------------------- +# Prefill-class two-shot NVLS variant of the fused RMSNorm and out-quant kernels. +# +# The one-shot kernels above need every rank to receive the three other +# partials in full (ingress 3P), which is the algorithmic floor at prefill M +# (P = 100.7 MB at M=8192). This variant keeps the fused epilogue local and +# changes only how the sum is formed: +# +# step 1 every CTA copies its rows of the local partial into region +# ``_TWOSHOT_INPUT_REGION`` of the current Lamport slot (local HBM +# write) and release-adds the block's input-ready flag on the owner +# rank (owner = block % WORLD_SIZE). +# step 2 the owner CTA waits for WORLD_SIZE input-ready arrivals, then for +# each of its rows issues ``multimem.ld_reduce`` on the multicast +# address of the input region: the NVSwitch fetches the four +# partials and returns their f32-accumulated BF16 sum. The same +# 16 B words are pushed back with one ``multimem.st`` into region +# ``_TWOSHOT_SUM_REGION`` of every rank, and the owner runs the +# residual/RMSNorm(/NVFP4) epilogue on its rows from registers. +# It then release-adds the block's sum-ready flag on each peer. +# step 3 the three non-owner CTAs wait for the sum-ready flag and run the +# same epilogue on the rows that arrived via multicast. +# +# Per rank and call this moves ~P out and ~P in instead of 3P in; the flags +# are reset by their single reader, so the 3-slot rotation, the clear phase +# and the device signal state are shared unchanged with the one-shot kernels +# (a decode launch may follow a prefill launch on the same workspace). +# --------------------------------------------------------------------------- +_TWOSHOT_INPUT_REGION = 0 +_TWOSHOT_SUM_REGION = 1 +_TWOSHOT_FLAG_INPUT_READY = 0 +_TWOSHOT_FLAG_SUM_READY = 1 +_TWOSHOT_THREADS_PER_BLOCK = 256 +_TWOSHOT_THREADS_PER_BLOCK_CHOICES = (128, 192, 256, 384, 768) +_TWOSHOT_ROWS_PER_CTA = 4 +# Diagnostic option, 0 in normal use: 3 runs the full two-shot protocol but skips the +# sentinel clear of the next slot. +_TWOSHOT_DEBUG_PHASE = 0 +# Cap on the number of persistent two-shot CTAs (0 = one CTA per row-block group, no pipelining). +_TWOSHOT_MAX_CTAS = 256 +# Number of dedicated CONSUMER CTAs (peer-block epilogues). 0 = every CTA does both roles. +# With C > 0 the grid is P producer CTAs (staging, ld_reduce, multicast, own epilogue) followed by +# C consumer CTAs, so reduce-scatter egress, all-gather ingress and the local staging copy proceed +# concurrently instead of phase by phase. +_TWOSHOT_CONSUMER_CTAS = 1024 +# 1 (default): reduce-scatter / all-gather over direct peer loads (unicast NVLink reads, fp32 accumulation +# in fixed rank order by the block owner). 0 selects the multimem.ld_reduce / multimem.st (NVLS) transport, +# which needs a multicast_ptr and is slower than direct loads at prefill sizes. +_TWOSHOT_DIRECT = 1 + + +if _dsl_user_op is not None: + + @_dsl_user_op + def _multimem_ld_reduce_bf16x8_words(mc_addr, *, loc=None, ip=None): + """``multimem.ld_reduce.acc::f32.v4.bf16x2`` of one 16 B vector. + + The NVSwitch fetches the eight BF16 lanes at ``mc_addr`` from every + rank of the multicast group, accumulates them in f32 and returns the + BF16 sum as four b32 words (raw MLIR values, like the store operand). + """ + mc_ptr_int = mc_addr.ir_value(loc=loc, ip=ip) + return_struct = _llvm.inline_asm( + _ir.Type.parse("!llvm.struct<(i32,i32,i32,i32)>"), + [mc_ptr_int], + "multimem.ld_reduce.weak.global.add.acc::f32.v4.bf16x2 {$0, $1, $2, $3}, [$4];", + "=r,=r,=r,=r,l,~{memory}", + has_side_effects=True, + asm_dialect=0, + loc=loc, + ip=ip, + ) + return tuple( + _llvm.extractvalue(_T.i32(), return_struct, [index], loc=loc, ip=ip) + for index in range(4) + ) + +else: # pragma: no cover - older DSL packages + + def _multimem_ld_reduce_bf16x8_words(*args, **kwargs): + raise RuntimeError("multimem ld_reduce requires cutlass._mlir.dialects.llvm") + + +@cute.jit +def _twoshot_row_epilogue( + row, + sum_values, + residual_array, + norm_weight_array, + norm_output_array, + residual_output_array, + warp_sums, + tidx, + lane_id, + warp_id, + global_scale, + fp4_ptr, + sf_ptr, + eps: cutlass.Constexpr[cutlass.Float32], + N: cutlass.Constexpr[int], + THREADS_PER_BLOCK: cutlass.Constexpr[int], + VECS_PER_THREAD: cutlass.Constexpr[int], + EMIT_NVFP4: cutlass.Constexpr[bool], +): + """Residual add, row RMSNorm and optional NVFP4 quant of one full row. + + ``sum_values`` holds this thread's ``VECS_PER_THREAD`` BF16x8 vectors of + the reduced row (same lane layout as the input copy: vector ``v`` covers + columns ``(tidx + v * THREADS_PER_BLOCK) * 8``). Every thread of the CTA + must call this for the same row; the whole CTA owns the row so the + sum-of-squares reduction is a plain CTA reduction, no cluster needed. + """ + residual_out_values = cutlass.Array( + cutlass.BFloat16, + VECS_PER_THREAD * _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + weight_values = cutlass.Array( + cutlass.BFloat16, + VECS_PER_THREAD * _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + local_sum_sq = cutlass.Float32(0.0) + for v in cutlass.range_constexpr(VECS_PER_THREAD): + col = (tidx + cutlass.Int32(v * THREADS_PER_BLOCK)) * cutlass.Int32(_BF16_VEC_SIZE) + lane_base = v * _BF16_VEC_SIZE + sum_vec = sum_values[lane_base:_BF16_VEC_SIZE, cutlass.align(_ALIGNMENT_BYTES)] + residual_vec = residual_array[row, :].load(col, _BF16_VEC_SIZE, alignment=_ALIGNMENT_BYTES) + weight_vec = norm_weight_array.load(col, _BF16_VEC_SIZE, alignment=_ALIGNMENT_BYTES) + residual_out_vec = (sum_vec.to(cutlass.Float32) + residual_vec.to(cutlass.Float32)).to( + cutlass.BFloat16 + ) + residual_output_array[row, :].store( + residual_out_vec, col, _BF16_VEC_SIZE, alignment=_ALIGNMENT_BYTES + ) + residual_out_values[lane_base:_BF16_VEC_SIZE, cutlass.align(_ALIGNMENT_BYTES)] = ( + residual_out_vec + ) + weight_values[lane_base:_BF16_VEC_SIZE, cutlass.align(_ALIGNMENT_BYTES)] = weight_vec + residual_f32 = residual_out_vec.to(cutlass.Float32) + local_sum_sq = local_sum_sq + (residual_f32 * residual_f32).reduce("add") + + warp_sum = _warp_sum_f32(local_sum_sq) + if lane_id == 0: + warp_sums[warp_id] = warp_sum + prims.barrier_cta_sync(0) + # Every thread sums the warp partials in the same fixed order, so the row + # gets one bit-identical inverse RMS without a broadcast. + cta_sum = cutlass.Float32(0.0) + for warp_idx in cutlass.range_constexpr(THREADS_PER_BLOCK // 32): + cta_sum = cta_sum + warp_sums[warp_idx] + inv_rms = cute.math.rsqrt(cta_sum * (1.0 / cutlass.Float32(N)) + eps, fastmath=True) + # The next row reuses warp_sums; keep the slow readers from seeing the + # next row's partials. + prims.barrier_cta_sync(0) + + for v in cutlass.range_constexpr(VECS_PER_THREAD): + col = (tidx + cutlass.Int32(v * THREADS_PER_BLOCK)) * cutlass.Int32(_BF16_VEC_SIZE) + lane_base = v * _BF16_VEC_SIZE + residual_vec = residual_out_values[ + lane_base:_BF16_VEC_SIZE, cutlass.align(_ALIGNMENT_BYTES) + ] + weight_vec = weight_values[lane_base:_BF16_VEC_SIZE, cutlass.align(_ALIGNMENT_BYTES)] + norm_f32 = residual_vec.to(cutlass.Float32) * inv_rms + norm_f32 = norm_f32 * weight_vec.to(cutlass.Float32) + norm_bf16 = norm_f32.to(cutlass.BFloat16) + norm_output_array[row, :].store(norm_bf16, col, _BF16_VEC_SIZE, alignment=_ALIGNMENT_BYTES) + if cutlass.const_expr(EMIT_NVFP4): + # Same linear-layout NVFP4 contract as the one-shot out-quant kernel: + # lanes (tidx, tidx^1) hold the two 8-column halves of one + # 16-column scale block, so a butterfly shuffle of distance 1 + # gives the block max. + quant_f32 = norm_bf16.to(cutlass.Float32) + local_max = cutlass.Float32(0.0) + for lane in cutlass.range_constexpr(_BF16_VEC_SIZE): + magnitude = cute.math.abs(quant_f32[lane]) + if magnitude > local_max: + local_max = magnitude + block_max = local_max + peer_max = cute.arch.shuffle_sync_bfly(block_max, 1) + if peer_max > block_max: + block_max = peer_max + sf_unquantized = global_scale * block_max * cutlass.Float32(1.0 / 6.0) + sf_packed = prims.cvt_f32x2_to_f8x2( + sf_unquantized, + sf_unquantized, + cutlass.Float8E4M3FN, + rnd=prims.FPRoundingMode.RN, + sat=prims.SaturationMode.SATFINITE, + ) + sf_code = cutlass.Int32(sf_packed) & cutlass.Int32(0xFF) + half_block = (col // cutlass.Int32(_BF16_VEC_SIZE)) & cutlass.Int32(1) + scale_block = col // cutlass.Int32(_SCALE_BLOCK) + if half_block == cutlass.Int32(0): + sf_index = row * cutlass.Int32(N // _SCALE_BLOCK) + scale_block + prims.store_ext(cutlass.Int8(sf_code), sf_ptr + sf_index) + narrowed_sf = _decode_positive_e4m3(sf_code) + output_scale = cutlass.Float32(0.0) + if block_max != cutlass.Float32(0.0): + output_scale = cute.arch.rcp_approx( + narrowed_sf * cute.arch.rcp_approx(global_scale) + ) + packed_word = cutlass.Int32(0) + for pair in cutlass.range_constexpr(_BF16_VEC_SIZE // 2): + low = quant_f32[2 * pair] * output_scale + high = quant_f32[2 * pair + 1] * output_scale + packed_byte = prims.cvt_f32x2_to_f4x2( + high, + low, + cutlass.Float4E2M1FN, + rnd=prims.FPRoundingMode.RN, + ) & cutlass.Int32(0xFF) + packed_word = packed_word | (packed_byte << cutlass.Int32(pair * 8)) + fp4_index = ( + row * cutlass.Int32(N // 2) + + scale_block * cutlass.Int32(_SCALE_BLOCK // 2) + + half_block * cutlass.Int32(_BF16_VEC_SIZE // 2) + ) + prims.store_ext(packed_word, fp4_ptr + fp4_index) + + +@cute.jit +def _twoshot_stage_group( + group, + tidx, + m_rows, + input_array, + local_buffer, + ready_flags, + stage_values, + input_region_base, + flag_in_base, + N: cutlass.Constexpr[int], + WORLD_SIZE: cutlass.Constexpr[int], + THREADS_PER_BLOCK: cutlass.Constexpr[int], + ROWS_PER_CTA: cutlass.Constexpr[int], + VECS_PER_THREAD: cutlass.Constexpr[int], +): + """Copy this rank's rows of the WORLD_SIZE row blocks of ``group`` into the input region + and release one input-ready flag per block on that block's owner rank. + + The loads of a whole block (ROWS_PER_CTA x VECS_PER_THREAD vectors per thread) are issued + before its stores so the copy is not a chain of dependent 16 B load->store pairs. + """ + row_block_rows = cutlass.Int32(ROWS_PER_CTA) + for r in cutlass.range_constexpr(WORLD_SIZE): + block = group * cutlass.Int32(WORLD_SIZE) + cutlass.Int32(r) + row0 = block * row_block_rows + if row0 < m_rows: + for rr in cutlass.range_constexpr(ROWS_PER_CTA): + row = row0 + cutlass.Int32(rr) + if row < m_rows: + for v in cutlass.range_constexpr(VECS_PER_THREAD): + col = (tidx + cutlass.Int32(v * THREADS_PER_BLOCK)) * cutlass.Int32( + _BF16_VEC_SIZE + ) + base = (rr * VECS_PER_THREAD + v) * _BF16_VEC_SIZE + stage_values[base:_BF16_VEC_SIZE, cutlass.align(_ALIGNMENT_BYTES)] = ( + input_array[row, :].load( + col, _BF16_VEC_SIZE, alignment=_ALIGNMENT_BYTES + ) + ) + for rr in cutlass.range_constexpr(ROWS_PER_CTA): + row = row0 + cutlass.Int32(rr) + if row < m_rows: + row_offset = input_region_base + cutlass.Int64(row) * cutlass.Int64(N) + for v in cutlass.range_constexpr(VECS_PER_THREAD): + col = (tidx + cutlass.Int32(v * THREADS_PER_BLOCK)) * cutlass.Int32( + _BF16_VEC_SIZE + ) + base = (rr * VECS_PER_THREAD + v) * _BF16_VEC_SIZE + local_buffer.store( + stage_values[base:_BF16_VEC_SIZE, cutlass.align(_ALIGNMENT_BYTES)], + row_offset + cutlass.Int64(col), + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + prims.barrier_cta_sync(0) + if tidx == 0: + for r in cutlass.range_constexpr(WORLD_SIZE): + block = group * cutlass.Int32(WORLD_SIZE) + cutlass.Int32(r) + if block * row_block_rows < m_rows: + prims.red( + prims.ReductionOp.ADD, + prims.ReductionType.S32, + (ready_flags[r].iterator + flag_in_base + cutlass.Int64(block)).llvm_ptr, + cutlass.Int32(1), + mem_order=prims.MemOrder.RELEASE, + mem_scope=prims.MemScope.SYS, + ) + + +@cute.kernel +def _tp_allreduce_residual_rms_norm_twoshot_nvls_kernel( + input_tensor: cute.Tensor, + residual_tensor: cute.Tensor, + norm_weight_tensor: cute.Tensor, + global_scale_tensor: cute.Tensor, + norm_output_tensor: cute.Tensor, + fp4_output_tensor: cute.Tensor, + scale_output_tensor: cute.Tensor, + residual_output_tensor: cute.Tensor, + buffers: list[cute.Tensor], + ready_flags: list[cute.Tensor], + signal_state: cute.Tensor, + multicast_base: cutlass.Int64, + signal: cutlass.Int64, + eps: cutlass.Constexpr[cutlass.Float32], + N: cutlass.Constexpr[int], + LOCAL_RANK: cutlass.Constexpr[int], + WORLD_SIZE: cutlass.Constexpr[int], + THREADS_PER_BLOCK: cutlass.Constexpr[int], + ROWS_PER_CTA: cutlass.Constexpr[int], + USE_DEVICE_SIGNAL_STATE: cutlass.Constexpr[bool], + USE_PDL: cutlass.Constexpr[bool], + PDL_RELEASE_AT_ENTRY: cutlass.Constexpr[bool], + EMIT_NVFP4: cutlass.Constexpr[bool], +): + tidx, _, _ = cute.arch.thread_idx() + bidx, _, _ = cute.arch.block_idx() + grid_x, _, _ = cute.arch.grid_dim() + lane_id = tidx % 32 + warp_id = tidx // 32 + VECS_PER_THREAD = N // (_BF16_VEC_SIZE * THREADS_PER_BLOCK) + + m_rows = cutlass.Int32(cute.size(input_tensor, mode=[0])) + input_array = cutlass.make_array_view(input_tensor) + residual_array = cutlass.make_array_view(residual_tensor) + norm_weight_array = cutlass.make_array_view(norm_weight_tensor) + norm_output_array = cutlass.make_array_view(norm_output_tensor) + residual_output_array = cutlass.make_array_view(residual_output_tensor) + buffer_arrays = [cutlass.make_array_view(buffer) for buffer in buffers] + local_buffer = buffer_arrays[LOCAL_RANK] + fp4_ptr = fp4_output_tensor.iterator.raw_ptr() + sf_ptr = scale_output_tensor.iterator.raw_ptr() + + warp_sums = cutlass.Array( + cutlass.Float32, + (THREADS_PER_BLOCK // 32), + space=cutlass.AddressSpace.smem, + alignment=_ALIGNMENT_BYTES, + ) + + numel = cute.size(buffers[0], mode=[2]) + numel_i64 = cutlass.Int64(numel) + num_vectors = cute.size(ready_flags[0], mode=[2]) + num_vectors_i64 = cutlass.Int64(num_vectors) + state_counter_ptr = (signal_state.iterator + cutlass.Int64(_SIGNAL_COUNTER_INDEX)).llvm_ptr + state_signal_ptr = (signal_state.iterator + cutlass.Int64(_SIGNAL_VALUE_INDEX)).llvm_ptr + + if cutlass.const_expr(USE_PDL): + cute.arch.griddepcontrol_wait() + global_scale = cutlass.Float32(0.0) + if cutlass.const_expr(EMIT_NVFP4): + global_scale = prims.load_ext(global_scale_tensor.iterator.raw_ptr(), dtype=cutlass.Float32) + + signal_i32 = cutlass.Int32(0) + signal_i64 = signal + if USE_DEVICE_SIGNAL_STATE: + signal_i32 = cute.arch.load( + state_signal_ptr, + cutlass.Int32, + sem="acquire", + scope="gpu", + ) + signal_i64 = cutlass.Int64(signal_i32) + ping_slot = signal_i64 % _PING_PONG_SIZE + clear_slot = (signal_i64 + cutlass.Int64(1)) % _PING_PONG_SIZE + ping_base = cutlass.Int64(ping_slot) * cutlass.Int64(WORLD_SIZE) * numel_i64 + clear_base = cutlass.Int64(clear_slot) * cutlass.Int64(WORLD_SIZE) * numel_i64 + input_region_base = ping_base + cutlass.Int64(_TWOSHOT_INPUT_REGION) * numel_i64 + sum_region_base = ping_base + cutlass.Int64(_TWOSHOT_SUM_REGION) * numel_i64 + flag_base = cutlass.Int64(ping_slot) * cutlass.Int64(WORLD_SIZE) * num_vectors_i64 + active_vectors_i32 = (m_rows * cutlass.Int32(N)) // cutlass.Int32(_BF16_VEC_SIZE) + if USE_DEVICE_SIGNAL_STATE: + if tidx == 0: + _advance_signal_if_last_cta( + state_counter_ptr, + state_signal_ptr, + signal_state, + signal_i32, + ping_slot, + active_vectors_i32, + grid_x, + ) + # Deliberately after the device signal snapshot (see the one-shot kernels). + if cutlass.const_expr(USE_PDL): + if cutlass.const_expr(PDL_RELEASE_AT_ENTRY): + if tidx == 0: + cute.arch.griddepcontrol_launch_dependents() + + # Clear the slot the next launch will use back to the sentinel, exactly as + # the one-shot kernels do, so a decode one-shot launch can follow. + neg_zero_vec = cutlass.Array( + cutlass.BFloat16, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + neg_zero_bits = cutlass.Int16(_NEGATIVE_ZERO_BF16_BITS) + for lane in cutlass.range_constexpr(_BF16_VEC_SIZE): + neg_zero_vec[lane] = neg_zero_bits.bitcast(cutlass.BFloat16) + if cutlass.const_expr(_TWOSHOT_DEBUG_PHASE != 3): + if USE_DEVICE_SIGNAL_STATE: + clear_count_ptr = ( + signal_state.iterator + + cutlass.Int64(_SIGNAL_SLOT_COUNTS_INDEX) + + cutlass.Int64(clear_slot) + ).llvm_ptr + clear_vectors = cute.arch.load( + clear_count_ptr, + cutlass.Int32, + sem="acquire", + scope="gpu", + ) + clear_vector_idx = bidx * THREADS_PER_BLOCK + tidx + clear_stride = grid_x * THREADS_PER_BLOCK + while clear_vector_idx < clear_vectors: + clear_vector_idx_i64 = cutlass.Int64(clear_vector_idx) + for source_rank in cutlass.range_constexpr(WORLD_SIZE): + clear_offset = ( + clear_base + + cutlass.Int64(source_rank) * numel_i64 + + clear_vector_idx_i64 * cutlass.Int64(_BF16_VEC_SIZE) + ) + local_buffer.store( + neg_zero_vec[0:_BF16_VEC_SIZE, cutlass.align(_ALIGNMENT_BYTES)], + clear_offset, + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + is_volatile=True, + ) + clear_vector_idx = clear_vector_idx + clear_stride + + # Persistent CTAs over row-block GROUPS (group g = blocks g*WORLD_SIZE+r, + # this rank owns block g*WORLD_SIZE+LOCAL_RANK). Each CTA walks groups bidx, bidx+grid, ... + # and stages group g+1 BEFORE waiting on group g, so the local HBM copy of the next group + # overlaps the switch round trips (ld_reduce / multicast / peer flags) of the current one. + # With grid == number of groups (default) each CTA handles one group (no pipelining). + group_rows = cutlass.Int32(ROWS_PER_CTA * WORLD_SIZE) + num_groups = (m_rows + group_rows - cutlass.Int32(1)) // group_rows + row_block_rows = cutlass.Int32(ROWS_PER_CTA) + flag_in_base = flag_base + cutlass.Int64(_TWOSHOT_FLAG_INPUT_READY) * num_vectors_i64 + flag_sum_base = flag_base + cutlass.Int64(_TWOSHOT_FLAG_SUM_READY) * num_vectors_i64 + + stage_values = cutlass.Array( + cutlass.BFloat16, + ROWS_PER_CTA * VECS_PER_THREAD * _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + sum_values = cutlass.Array( + cutlass.BFloat16, + VECS_PER_THREAD * _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + # All ld_reduce results of the owned block are issued before any is consumed so the + # switch round trips overlap (ROWS_PER_CTA x VECS_PER_THREAD x 4 words in flight). + owner_words = cutlass.Array( + cutlass.Int32, + ROWS_PER_CTA * VECS_PER_THREAD * 4, + alignment=_ALIGNMENT_BYTES, + ) + + if cutlass.const_expr(_TWOSHOT_CONSUMER_CTAS == 0): + # Single-role schedule: every CTA stages, reduces its own block and then consumes the peers. + group = bidx + if group < num_groups: + _twoshot_stage_group( + group, + tidx, + m_rows, + input_array, + local_buffer, + ready_flags, + stage_values, + input_region_base, + flag_in_base, + N, + WORLD_SIZE, + THREADS_PER_BLOCK, + ROWS_PER_CTA, + VECS_PER_THREAD, + ) + while group < num_groups: + next_group = group + grid_x + if next_group < num_groups: + _twoshot_stage_group( + next_group, + tidx, + m_rows, + input_array, + local_buffer, + ready_flags, + stage_values, + input_region_base, + flag_in_base, + N, + WORLD_SIZE, + THREADS_PER_BLOCK, + ROWS_PER_CTA, + VECS_PER_THREAD, + ) + + # ---- own block: wait for the WORLD_SIZE input-ready arrivals, switch-reduce, + # multicast the sum to every rank, local fused epilogue, release sum-ready on peers. + own_block = group * cutlass.Int32(WORLD_SIZE) + cutlass.Int32(LOCAL_RANK) + own_row0 = own_block * row_block_rows + if own_row0 < m_rows: + own_in_flag_ptr = ( + ready_flags[LOCAL_RANK].iterator + flag_in_base + cutlass.Int64(own_block) + ).llvm_ptr + if tidx == 0: + arrivals = cute.arch.load( + own_in_flag_ptr, cutlass.Int32, sem="acquire", scope="sys" + ) + while arrivals < cutlass.Int32(WORLD_SIZE): + arrivals = cute.arch.load( + own_in_flag_ptr, cutlass.Int32, sem="acquire", scope="sys" + ) + # Single reader: consume the arrivals so the slot is clean for its next use. + prims.red( + prims.ReductionOp.ADD, + prims.ReductionType.S32, + own_in_flag_ptr, + cutlass.Int32(-WORLD_SIZE), + mem_order=prims.MemOrder.RELAXED, + mem_scope=prims.MemScope.SYS, + ) + prims.barrier_cta_sync(0) + if cutlass.const_expr(_TWOSHOT_DIRECT): + # Direct reduce-scatter: read the block rows of every rank (fixed order 0..W-1, fp32), + # keep the BF16 sums in registers, then store them once into the local sum region. + for rr in cutlass.range_constexpr(ROWS_PER_CTA): + row = own_row0 + cutlass.Int32(rr) + if row < m_rows: + in_row_offset = input_region_base + cutlass.Int64(row) * cutlass.Int64( + N + ) + for v in cutlass.range_constexpr(VECS_PER_THREAD): + col = (tidx + cutlass.Int32(v * THREADS_PER_BLOCK)) * cutlass.Int32( + _BF16_VEC_SIZE + ) + acc_f32 = ( + buffer_arrays[0] + .load( + in_row_offset + cutlass.Int64(col), + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + is_volatile=True, + ) + .to(cutlass.Float32) + ) + for src in cutlass.range_constexpr(1, WORLD_SIZE): + acc_f32 = acc_f32 + buffer_arrays[src].load( + in_row_offset + cutlass.Int64(col), + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + is_volatile=True, + ).to(cutlass.Float32) + pbase = (rr * VECS_PER_THREAD + v) * _BF16_VEC_SIZE + stage_values[ + pbase:_BF16_VEC_SIZE, + cutlass.align(_ALIGNMENT_BYTES), + ] = acc_f32.to(cutlass.BFloat16) + for rr in cutlass.range_constexpr(ROWS_PER_CTA): + row = own_row0 + cutlass.Int32(rr) + if row < m_rows: + sum_row_offset = sum_region_base + cutlass.Int64(row) * cutlass.Int64(N) + for v in cutlass.range_constexpr(VECS_PER_THREAD): + col = (tidx + cutlass.Int32(v * THREADS_PER_BLOCK)) * cutlass.Int32( + _BF16_VEC_SIZE + ) + pbase = (rr * VECS_PER_THREAD + v) * _BF16_VEC_SIZE + local_buffer.store( + stage_values[ + pbase:_BF16_VEC_SIZE, + cutlass.align(_ALIGNMENT_BYTES), + ], + sum_row_offset + cutlass.Int64(col), + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + for e in cutlass.range_constexpr(_BF16_VEC_SIZE): + sum_values[v * _BF16_VEC_SIZE + e] = stage_values[pbase + e] + _twoshot_row_epilogue( + row, + sum_values, + residual_array, + norm_weight_array, + norm_output_array, + residual_output_array, + warp_sums, + tidx, + lane_id, + warp_id, + global_scale, + fp4_ptr, + sf_ptr, + eps, + N, + THREADS_PER_BLOCK, + VECS_PER_THREAD, + EMIT_NVFP4, + ) + else: + for rr in cutlass.range_constexpr(ROWS_PER_CTA): + row = own_row0 + cutlass.Int32(rr) + if row < m_rows: + in_row_offset = input_region_base + cutlass.Int64(row) * cutlass.Int64( + N + ) + for v in cutlass.range_constexpr(VECS_PER_THREAD): + col = (tidx + cutlass.Int32(v * THREADS_PER_BLOCK)) * cutlass.Int32( + _BF16_VEC_SIZE + ) + word0, word1, word2, word3 = _multimem_ld_reduce_bf16x8_words( + multicast_base + + (in_row_offset + cutlass.Int64(col)) * cutlass.Int64(2) + ) + wbase = (rr * VECS_PER_THREAD + v) * 4 + owner_words[wbase] = cutlass.Int32(word0) + owner_words[wbase + 1] = cutlass.Int32(word1) + owner_words[wbase + 2] = cutlass.Int32(word2) + owner_words[wbase + 3] = cutlass.Int32(word3) + for rr in cutlass.range_constexpr(ROWS_PER_CTA): + row = own_row0 + cutlass.Int32(rr) + if row < m_rows: + sum_row_offset = sum_region_base + cutlass.Int64(row) * cutlass.Int64(N) + for v in cutlass.range_constexpr(VECS_PER_THREAD): + col = (tidx + cutlass.Int32(v * THREADS_PER_BLOCK)) * cutlass.Int32( + _BF16_VEC_SIZE + ) + wbase = (rr * VECS_PER_THREAD + v) * 4 + w0 = owner_words[wbase] + w1 = owner_words[wbase + 1] + w2 = owner_words[wbase + 2] + w3 = owner_words[wbase + 3] + _multimem_st_v4_b32( + multicast_base + + (sum_row_offset + cutlass.Int64(col)) * cutlass.Int64(2), + w0, + w1, + w2, + w3, + ) + words = [w0, w1, w2, w3] + lane_base = v * _BF16_VEC_SIZE + for pair in cutlass.range_constexpr(_BF16_VEC_SIZE // 2): + word = words[pair] + low_bits = (word << cutlass.Int32(16)) >> cutlass.Int32(16) + high_bits = word >> cutlass.Int32(16) + # Constructor narrowing (as cutlass.Int8(sf_code) in the out-quant kernel); + # both halves were sign-extended by the arithmetic shifts. + sum_values[lane_base + 2 * pair] = cutlass.Int16( + low_bits + ).bitcast(cutlass.BFloat16) + sum_values[lane_base + 2 * pair + 1] = cutlass.Int16( + high_bits + ).bitcast(cutlass.BFloat16) + _twoshot_row_epilogue( + row, + sum_values, + residual_array, + norm_weight_array, + norm_output_array, + residual_output_array, + warp_sums, + tidx, + lane_id, + warp_id, + global_scale, + fp4_ptr, + sf_ptr, + eps, + N, + THREADS_PER_BLOCK, + VECS_PER_THREAD, + EMIT_NVFP4, + ) + prims.barrier_cta_sync(0) + if tidx == 0: + for target_rank in cutlass.range_constexpr(WORLD_SIZE): + if cutlass.const_expr(target_rank != LOCAL_RANK): + prims.red( + prims.ReductionOp.ADD, + prims.ReductionType.S32, + ( + ready_flags[target_rank].iterator + + flag_sum_base + + cutlass.Int64(own_block) + ).llvm_ptr, + cutlass.Int32(1), + mem_order=prims.MemOrder.RELEASE, + mem_scope=prims.MemScope.SYS, + ) + + # ---- the three peer blocks of the group: wait for each owner's multicast sum + # (rotated start so the four ranks do not all wait on the same owner first). + for peer_ordinal in cutlass.range_constexpr(1, WORLD_SIZE): + peer_rank = (LOCAL_RANK + peer_ordinal) % WORLD_SIZE + peer_block = group * cutlass.Int32(WORLD_SIZE) + cutlass.Int32(peer_rank) + peer_row0 = peer_block * row_block_rows + if peer_row0 < m_rows: + peer_sum_flag_ptr = ( + ready_flags[LOCAL_RANK].iterator + flag_sum_base + cutlass.Int64(peer_block) + ).llvm_ptr + if tidx == 0: + ready = cute.arch.load( + peer_sum_flag_ptr, cutlass.Int32, sem="acquire", scope="sys" + ) + while ready < cutlass.Int32(1): + ready = cute.arch.load( + peer_sum_flag_ptr, + cutlass.Int32, + sem="acquire", + scope="sys", + ) + prims.red( + prims.ReductionOp.ADD, + prims.ReductionType.S32, + peer_sum_flag_ptr, + cutlass.Int32(-1), + mem_order=prims.MemOrder.RELAXED, + mem_scope=prims.MemScope.SYS, + ) + prims.barrier_cta_sync(0) + # Batched: issue every row's loads of the block before the first epilogue so the + # CTA is not one latency-bound row at a time. + for rr in cutlass.range_constexpr(ROWS_PER_CTA): + row = peer_row0 + cutlass.Int32(rr) + if row < m_rows: + sum_row_offset = sum_region_base + cutlass.Int64(row) * cutlass.Int64(N) + for v in cutlass.range_constexpr(VECS_PER_THREAD): + col = (tidx + cutlass.Int32(v * THREADS_PER_BLOCK)) * cutlass.Int32( + _BF16_VEC_SIZE + ) + pbase = (rr * VECS_PER_THREAD + v) * _BF16_VEC_SIZE + if cutlass.const_expr(_TWOSHOT_DIRECT): + stage_values[ + pbase:_BF16_VEC_SIZE, + cutlass.align(_ALIGNMENT_BYTES), + ] = buffer_arrays[peer_rank].load( + sum_row_offset + cutlass.Int64(col), + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + is_volatile=True, + ) + else: + stage_values[ + pbase:_BF16_VEC_SIZE, + cutlass.align(_ALIGNMENT_BYTES), + ] = local_buffer.load( + sum_row_offset + cutlass.Int64(col), + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + is_volatile=True, + ) + for rr in cutlass.range_constexpr(ROWS_PER_CTA): + row = peer_row0 + cutlass.Int32(rr) + if row < m_rows: + for e in cutlass.range_constexpr(VECS_PER_THREAD * _BF16_VEC_SIZE): + sum_values[e] = stage_values[ + rr * VECS_PER_THREAD * _BF16_VEC_SIZE + e + ] + _twoshot_row_epilogue( + row, + sum_values, + residual_array, + norm_weight_array, + norm_output_array, + residual_output_array, + warp_sums, + tidx, + lane_id, + warp_id, + global_scale, + fp4_ptr, + sf_ptr, + eps, + N, + THREADS_PER_BLOCK, + VECS_PER_THREAD, + EMIT_NVFP4, + ) + + group = next_group + else: + # Split-role schedule: CTAs [0, P) are producers, CTAs [P, P + C) consumers (C constexpr). + num_producers = grid_x - cutlass.Int32(_TWOSHOT_CONSUMER_CTAS) + if bidx < num_producers: + group = bidx + if group < num_groups: + _twoshot_stage_group( + group, + tidx, + m_rows, + input_array, + local_buffer, + ready_flags, + stage_values, + input_region_base, + flag_in_base, + N, + WORLD_SIZE, + THREADS_PER_BLOCK, + ROWS_PER_CTA, + VECS_PER_THREAD, + ) + while group < num_groups: + next_group = group + num_producers + if next_group < num_groups: + _twoshot_stage_group( + next_group, + tidx, + m_rows, + input_array, + local_buffer, + ready_flags, + stage_values, + input_region_base, + flag_in_base, + N, + WORLD_SIZE, + THREADS_PER_BLOCK, + ROWS_PER_CTA, + VECS_PER_THREAD, + ) + # ---- own block: wait for the WORLD_SIZE input-ready arrivals, switch-reduce, + # multicast the sum to every rank, local fused epilogue, release sum-ready on peers. + own_block = group * cutlass.Int32(WORLD_SIZE) + cutlass.Int32(LOCAL_RANK) + own_row0 = own_block * row_block_rows + if own_row0 < m_rows: + own_in_flag_ptr = ( + ready_flags[LOCAL_RANK].iterator + flag_in_base + cutlass.Int64(own_block) + ).llvm_ptr + if tidx == 0: + arrivals = cute.arch.load( + own_in_flag_ptr, cutlass.Int32, sem="acquire", scope="sys" + ) + while arrivals < cutlass.Int32(WORLD_SIZE): + arrivals = cute.arch.load( + own_in_flag_ptr, + cutlass.Int32, + sem="acquire", + scope="sys", + ) + # Single reader: consume the arrivals so the slot is clean for its next use. + prims.red( + prims.ReductionOp.ADD, + prims.ReductionType.S32, + own_in_flag_ptr, + cutlass.Int32(-WORLD_SIZE), + mem_order=prims.MemOrder.RELAXED, + mem_scope=prims.MemScope.SYS, + ) + prims.barrier_cta_sync(0) + if cutlass.const_expr(_TWOSHOT_DIRECT): + # Direct reduce-scatter: read the block rows of every rank (fixed order 0..W-1, fp32), + # keep the BF16 sums in registers, then store them once into the local sum region. + for rr in cutlass.range_constexpr(ROWS_PER_CTA): + row = own_row0 + cutlass.Int32(rr) + if row < m_rows: + in_row_offset = input_region_base + cutlass.Int64( + row + ) * cutlass.Int64(N) + for v in cutlass.range_constexpr(VECS_PER_THREAD): + col = ( + tidx + cutlass.Int32(v * THREADS_PER_BLOCK) + ) * cutlass.Int32(_BF16_VEC_SIZE) + acc_f32 = ( + buffer_arrays[0] + .load( + in_row_offset + cutlass.Int64(col), + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + is_volatile=True, + ) + .to(cutlass.Float32) + ) + for src in cutlass.range_constexpr(1, WORLD_SIZE): + acc_f32 = acc_f32 + buffer_arrays[src].load( + in_row_offset + cutlass.Int64(col), + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + is_volatile=True, + ).to(cutlass.Float32) + pbase = (rr * VECS_PER_THREAD + v) * _BF16_VEC_SIZE + stage_values[ + pbase:_BF16_VEC_SIZE, + cutlass.align(_ALIGNMENT_BYTES), + ] = acc_f32.to(cutlass.BFloat16) + for rr in cutlass.range_constexpr(ROWS_PER_CTA): + row = own_row0 + cutlass.Int32(rr) + if row < m_rows: + sum_row_offset = sum_region_base + cutlass.Int64( + row + ) * cutlass.Int64(N) + for v in cutlass.range_constexpr(VECS_PER_THREAD): + col = ( + tidx + cutlass.Int32(v * THREADS_PER_BLOCK) + ) * cutlass.Int32(_BF16_VEC_SIZE) + pbase = (rr * VECS_PER_THREAD + v) * _BF16_VEC_SIZE + local_buffer.store( + stage_values[ + pbase:_BF16_VEC_SIZE, + cutlass.align(_ALIGNMENT_BYTES), + ], + sum_row_offset + cutlass.Int64(col), + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + ) + for e in cutlass.range_constexpr(_BF16_VEC_SIZE): + sum_values[v * _BF16_VEC_SIZE + e] = stage_values[pbase + e] + _twoshot_row_epilogue( + row, + sum_values, + residual_array, + norm_weight_array, + norm_output_array, + residual_output_array, + warp_sums, + tidx, + lane_id, + warp_id, + global_scale, + fp4_ptr, + sf_ptr, + eps, + N, + THREADS_PER_BLOCK, + VECS_PER_THREAD, + EMIT_NVFP4, + ) + else: + for rr in cutlass.range_constexpr(ROWS_PER_CTA): + row = own_row0 + cutlass.Int32(rr) + if row < m_rows: + in_row_offset = input_region_base + cutlass.Int64( + row + ) * cutlass.Int64(N) + for v in cutlass.range_constexpr(VECS_PER_THREAD): + col = ( + tidx + cutlass.Int32(v * THREADS_PER_BLOCK) + ) * cutlass.Int32(_BF16_VEC_SIZE) + word0, word1, word2, word3 = _multimem_ld_reduce_bf16x8_words( + multicast_base + + (in_row_offset + cutlass.Int64(col)) * cutlass.Int64(2) + ) + wbase = (rr * VECS_PER_THREAD + v) * 4 + owner_words[wbase] = cutlass.Int32(word0) + owner_words[wbase + 1] = cutlass.Int32(word1) + owner_words[wbase + 2] = cutlass.Int32(word2) + owner_words[wbase + 3] = cutlass.Int32(word3) + for rr in cutlass.range_constexpr(ROWS_PER_CTA): + row = own_row0 + cutlass.Int32(rr) + if row < m_rows: + sum_row_offset = sum_region_base + cutlass.Int64( + row + ) * cutlass.Int64(N) + for v in cutlass.range_constexpr(VECS_PER_THREAD): + col = ( + tidx + cutlass.Int32(v * THREADS_PER_BLOCK) + ) * cutlass.Int32(_BF16_VEC_SIZE) + wbase = (rr * VECS_PER_THREAD + v) * 4 + w0 = owner_words[wbase] + w1 = owner_words[wbase + 1] + w2 = owner_words[wbase + 2] + w3 = owner_words[wbase + 3] + _multimem_st_v4_b32( + multicast_base + + (sum_row_offset + cutlass.Int64(col)) * cutlass.Int64(2), + w0, + w1, + w2, + w3, + ) + words = [w0, w1, w2, w3] + lane_base = v * _BF16_VEC_SIZE + for pair in cutlass.range_constexpr(_BF16_VEC_SIZE // 2): + word = words[pair] + low_bits = (word << cutlass.Int32(16)) >> cutlass.Int32(16) + high_bits = word >> cutlass.Int32(16) + # Constructor narrowing (as cutlass.Int8(sf_code) in the out-quant kernel); + # both halves were sign-extended by the arithmetic shifts. + sum_values[lane_base + 2 * pair] = cutlass.Int16( + low_bits + ).bitcast(cutlass.BFloat16) + sum_values[lane_base + 2 * pair + 1] = cutlass.Int16( + high_bits + ).bitcast(cutlass.BFloat16) + _twoshot_row_epilogue( + row, + sum_values, + residual_array, + norm_weight_array, + norm_output_array, + residual_output_array, + warp_sums, + tidx, + lane_id, + warp_id, + global_scale, + fp4_ptr, + sf_ptr, + eps, + N, + THREADS_PER_BLOCK, + VECS_PER_THREAD, + EMIT_NVFP4, + ) + prims.barrier_cta_sync(0) + if tidx == 0: + for target_rank in cutlass.range_constexpr(WORLD_SIZE): + if cutlass.const_expr(target_rank != LOCAL_RANK): + prims.red( + prims.ReductionOp.ADD, + prims.ReductionType.S32, + ( + ready_flags[target_rank].iterator + + flag_sum_base + + cutlass.Int64(own_block) + ).llvm_ptr, + cutlass.Int32(1), + mem_order=prims.MemOrder.RELEASE, + mem_scope=prims.MemScope.SYS, + ) + + group = next_group + else: + group = bidx - num_producers + while group < num_groups: + # ---- the three peer blocks of the group: wait for each owner's multicast sum + # (rotated start so the four ranks do not all wait on the same owner first). + for peer_ordinal in cutlass.range_constexpr(1, WORLD_SIZE): + peer_rank = (LOCAL_RANK + peer_ordinal) % WORLD_SIZE + peer_block = group * cutlass.Int32(WORLD_SIZE) + cutlass.Int32(peer_rank) + peer_row0 = peer_block * row_block_rows + if peer_row0 < m_rows: + peer_sum_flag_ptr = ( + ready_flags[LOCAL_RANK].iterator + + flag_sum_base + + cutlass.Int64(peer_block) + ).llvm_ptr + if tidx == 0: + ready = cute.arch.load( + peer_sum_flag_ptr, + cutlass.Int32, + sem="acquire", + scope="sys", + ) + while ready < cutlass.Int32(1): + ready = cute.arch.load( + peer_sum_flag_ptr, + cutlass.Int32, + sem="acquire", + scope="sys", + ) + prims.red( + prims.ReductionOp.ADD, + prims.ReductionType.S32, + peer_sum_flag_ptr, + cutlass.Int32(-1), + mem_order=prims.MemOrder.RELAXED, + mem_scope=prims.MemScope.SYS, + ) + prims.barrier_cta_sync(0) + # Batched: issue every row's loads of the block before the first epilogue so the + # CTA is not one latency-bound row at a time. + for rr in cutlass.range_constexpr(ROWS_PER_CTA): + row = peer_row0 + cutlass.Int32(rr) + if row < m_rows: + sum_row_offset = sum_region_base + cutlass.Int64( + row + ) * cutlass.Int64(N) + for v in cutlass.range_constexpr(VECS_PER_THREAD): + col = ( + tidx + cutlass.Int32(v * THREADS_PER_BLOCK) + ) * cutlass.Int32(_BF16_VEC_SIZE) + pbase = (rr * VECS_PER_THREAD + v) * _BF16_VEC_SIZE + if cutlass.const_expr(_TWOSHOT_DIRECT): + stage_values[ + pbase:_BF16_VEC_SIZE, + cutlass.align(_ALIGNMENT_BYTES), + ] = buffer_arrays[peer_rank].load( + sum_row_offset + cutlass.Int64(col), + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + is_volatile=True, + ) + else: + stage_values[ + pbase:_BF16_VEC_SIZE, + cutlass.align(_ALIGNMENT_BYTES), + ] = local_buffer.load( + sum_row_offset + cutlass.Int64(col), + _BF16_VEC_SIZE, + alignment=_ALIGNMENT_BYTES, + is_volatile=True, + ) + for rr in cutlass.range_constexpr(ROWS_PER_CTA): + row = peer_row0 + cutlass.Int32(rr) + if row < m_rows: + for e in cutlass.range_constexpr(VECS_PER_THREAD * _BF16_VEC_SIZE): + sum_values[e] = stage_values[ + rr * VECS_PER_THREAD * _BF16_VEC_SIZE + e + ] + _twoshot_row_epilogue( + row, + sum_values, + residual_array, + norm_weight_array, + norm_output_array, + residual_output_array, + warp_sums, + tidx, + lane_id, + warp_id, + global_scale, + fp4_ptr, + sf_ptr, + eps, + N, + THREADS_PER_BLOCK, + VECS_PER_THREAD, + EMIT_NVFP4, + ) + + group = group + cutlass.Int32(_TWOSHOT_CONSUMER_CTAS) + + # Output-safe release: every output row of this CTA is written. + if cutlass.const_expr(USE_PDL): + if cutlass.const_expr(not PDL_RELEASE_AT_ENTRY): + if tidx == 0: + cute.arch.griddepcontrol_launch_dependents() + + +@cute.jit +def _tp_allreduce_residual_rms_norm_bf16_twoshot_nvls( + input_tensor: cute.Tensor, + residual_tensor: cute.Tensor, + norm_weight_tensor: cute.Tensor, + norm_output_tensor: cute.Tensor, + residual_output_tensor: cute.Tensor, + buffers: list[cute.Tensor], + ready_flags: list[cute.Tensor], + signal_state: cute.Tensor, + multicast_base: cutlass.Int64, + signal: cutlass.Int64, + stream: cuda.CUstream, + eps: cutlass.Constexpr[cutlass.Float32], + N: cutlass.Constexpr[int], + LOCAL_RANK: cutlass.Constexpr[int], + WORLD_SIZE: cutlass.Constexpr[int], + THREADS_PER_BLOCK: cutlass.Constexpr[int], + ROWS_PER_CTA: cutlass.Constexpr[int], + USE_DEVICE_SIGNAL_STATE: cutlass.Constexpr[bool], + USE_PDL: cutlass.Constexpr[bool], + PDL_RELEASE_AT_ENTRY: cutlass.Constexpr[bool], +) -> None: + M = input_tensor.shape[0] + grid_x = (M + ROWS_PER_CTA * WORLD_SIZE - 1) // (ROWS_PER_CTA * WORLD_SIZE) + if cutlass.const_expr(_TWOSHOT_MAX_CTAS > 0): + grid_x = cutlass.min(grid_x, cutlass.Int32(_TWOSHOT_MAX_CTAS)) + if cutlass.const_expr(_TWOSHOT_CONSUMER_CTAS > 0): + grid_x = grid_x + cutlass.Int32(_TWOSHOT_CONSUMER_CTAS) + # The fused RMSNorm kernel has no NVFP4 outputs; the placeholder tensors are never touched + # because EMIT_NVFP4 is False (all uses sit under const_expr guards). + _tp_allreduce_residual_rms_norm_twoshot_nvls_kernel( + input_tensor, + residual_tensor, + norm_weight_tensor, + norm_weight_tensor, + norm_output_tensor, + norm_output_tensor, + norm_output_tensor, + residual_output_tensor, + buffers, + ready_flags, + signal_state, + multicast_base, + signal, + eps, + N, + LOCAL_RANK, + WORLD_SIZE, + THREADS_PER_BLOCK, + ROWS_PER_CTA, + USE_DEVICE_SIGNAL_STATE, + USE_PDL, + PDL_RELEASE_AT_ENTRY, + False, + ).launch( + grid=(grid_x, 1, 1), + block=(THREADS_PER_BLOCK, 1, 1), + stream=stream, + use_pdl=USE_PDL, + ) + + +def compile_pure_for_rank( + local_rank: int = 0, + world_size: int = 1, + device_id: int = 0, + n: int = _DEFAULT_N, + pure_threads_per_block: int = _THREADS_PER_BLOCK, + pure_elide_self_rank: bool = False, + pure_rotate_peer_order: bool = False, + pure_fp32_accumulation: bool = True, + pure_batch_poll_peers: bool = False, + use_ready_flags: bool = _USES_READY_FLAGS, + use_device_signal_state: bool = _USES_DEVICE_SIGNAL_STATE, + use_pdl: bool = False, + pdl_release_policy: str = "strict", + last_cta_signal_advance: bool = False, +): + cutlass.cuda.initialize_cuda_context(device_id) + _validate_signal_advance_flags( + last_cta_signal_advance, + False, + use_device_signal_state, + use_pdl, + pdl_release_policy, + ) + if pure_threads_per_block not in _PURE_THREADS_PER_BLOCK_CHOICES: + raise ValueError( + "pure_threads_per_block must be one of " + f"{_PURE_THREADS_PER_BLOCK_CHOICES}, got {pure_threads_per_block}" + ) + _validate_pdl_release_policy(pdl_release_policy) + if pure_batch_poll_peers and (world_size != 4 or use_ready_flags): + raise ValueError( + "allreduce batch peer polling requires world_size=4 and sentinel readiness" + ) + if n % _BF16_VEC_SIZE != 0: + raise ValueError(f"n must be divisible by {_BF16_VEC_SIZE}, got {n}") + + sym_m = cute.sym_int32() + input_tensor = make_fake_compact_tensor( + cutlass.BFloat16, + (sym_m, n), + stride_order=(1, 0), + assumed_align=_ALIGNMENT_BYTES, + ) + output_tensor = make_fake_compact_tensor( + cutlass.BFloat16, + (sym_m, n), + stride_order=(1, 0), + assumed_align=_ALIGNMENT_BYTES, + ) + + workspace_numel = cute.sym_int32(divisibility=_BF16_VEC_SIZE) + workspace_num_vectors = workspace_numel // _BF16_VEC_SIZE + fake_buffers = [] + fake_ready_flags = [] + for _ in range(world_size): + buffer_tensor = make_fake_compact_tensor( + cutlass.BFloat16, + (_PING_PONG_SIZE, world_size, workspace_numel), + stride_order=(2, 1, 0), + assumed_align=_ALIGNMENT_BYTES, + ) + fake_buffers.append(buffer_tensor) + ready_flag_tensor = make_fake_compact_tensor( + cutlass.Int32, + (_PING_PONG_SIZE, world_size, workspace_num_vectors), + stride_order=(2, 1, 0), + assumed_align=_READY_FLAG_ALIGNMENT_BYTES, + ) + fake_ready_flags.append(ready_flag_tensor) + + fake_signal_state = make_fake_compact_tensor( + cutlass.Int32, + (_SIGNAL_STATE_SIZE,), + assumed_align=_READY_FLAG_ALIGNMENT_BYTES, + ) + peer_order = tuple(range(world_size)) + if pure_rotate_peer_order: + peer_order = tuple( + (local_rank + peer_ordinal) % world_size for peer_ordinal in range(world_size) + ) + + return cute.compile( + _tp_allreduce_bf16_oneshot_lamport_serving, + input_tensor, + output_tensor, + fake_buffers, + fake_ready_flags, + fake_signal_state, + cutlass.Int64(0), + make_fake_stream(), + n, + local_rank, + world_size, + pure_threads_per_block, + use_ready_flags, + use_device_signal_state, + pure_elide_self_rank, + peer_order, + pure_fp32_accumulation, + pure_batch_poll_peers, + use_pdl, + pdl_release_policy == "strict", + last_cta_signal_advance, + options="--enable-tvm-ffi", + ) + + +def compile_fused_for_rank( + local_rank: int = 0, + world_size: int = 1, + device_id: int = 0, + n: int = _DEFAULT_N, + eps: float = _RMS_EPS, + split_rows: bool = True, + use_ready_flags: bool = _USES_READY_FLAGS, + use_device_signal_state: bool = _USES_DEVICE_SIGNAL_STATE, + elide_self_rank: bool = False, + rotate_peer_order: bool = False, + aggregate_signal_at_cluster_root: bool = False, + ctas_per_row: int = _FUSED_CTAS_PER_ROW, + fused_threads_per_block: int = _SPLIT_ROW_THREADS_PER_BLOCK, + prefetch_fused_inputs: bool = False, + fused_direct_cluster_reduction: bool = False, + fused_post_norm_cluster_barrier: bool = True, + fused_fp32_accumulation: bool = True, + fused_batch_poll_peers: bool = False, + fused_ready_mask_poll_peers: bool = False, + use_pdl: bool = False, + pdl_release_policy: str = "strict", + last_cta_signal_advance: bool = False, + all_to_all_cluster_reduction: bool = False, + early_input_load: bool = False, + multicast_publish: bool = False, + iket_phases: bool = False, + rotated_poll_order: bool = False, +): + cutlass.cuda.initialize_cuda_context(device_id) + _validate_ctas_per_row(ctas_per_row) + _validate_signal_advance_flags( + last_cta_signal_advance, + aggregate_signal_at_cluster_root, + use_device_signal_state, + use_pdl, + pdl_release_policy, + ) + if fused_threads_per_block not in _FUSED_THREADS_PER_BLOCK_CHOICES: + raise ValueError( + "fused_threads_per_block must be one of " + f"{_FUSED_THREADS_PER_BLOCK_CHOICES}, got {fused_threads_per_block}" + ) + _validate_fused_pdl_release_policy(pdl_release_policy) + if fused_batch_poll_peers and fused_ready_mask_poll_peers: + raise ValueError("fused RMSNorm batch and ready-mask peer polling are mutually exclusive") + if rotated_poll_order and (fused_batch_poll_peers or fused_ready_mask_poll_peers): + raise ValueError( + "rotated_poll_order applies to the sequential peer poll only; the batch " + "and ready-mask variants already poll every peer each iteration" + ) + if (fused_batch_poll_peers or fused_ready_mask_poll_peers) and ( + world_size != 4 or use_ready_flags + ): + raise ValueError( + "fused RMSNorm batch and ready-mask peer polling require " + "world_size=4 and sentinel readiness" + ) + if n % _BF16_VEC_SIZE != 0: + raise ValueError(f"n must be divisible by {_BF16_VEC_SIZE}, got {n}") + split_row_alignment = ctas_per_row * _BF16_VEC_SIZE + if split_rows and n % split_row_alignment != 0: + raise ValueError( + f"n must be divisible by {split_row_alignment} for split-row fused launch, got {n}" + ) + if not use_ready_flags and not split_rows: + raise ValueError("sentinel readiness is implemented only for split-row fused") + if use_device_signal_state and not split_rows: + raise ValueError("device signal state is implemented only for split-row fused") + if use_pdl and not split_rows: + raise ValueError("PDL is implemented only for split-row fused") + _validate_multicast_publish(multicast_publish, use_ready_flags, split_rows) + + sym_m = cute.sym_int32() + input_tensor = make_fake_compact_tensor( + cutlass.BFloat16, + (sym_m, n), + stride_order=(1, 0), + assumed_align=_ALIGNMENT_BYTES, + ) + residual_tensor = make_fake_compact_tensor( + cutlass.BFloat16, + (sym_m, n), + stride_order=(1, 0), + assumed_align=_ALIGNMENT_BYTES, + ) + norm_weight_tensor = make_fake_compact_tensor( + cutlass.BFloat16, + (n,), + assumed_align=_ALIGNMENT_BYTES, + ) + norm_output_tensor = make_fake_compact_tensor( + cutlass.BFloat16, + (sym_m, n), + stride_order=(1, 0), + assumed_align=_ALIGNMENT_BYTES, + ) + residual_output_tensor = make_fake_compact_tensor( + cutlass.BFloat16, + (sym_m, n), + stride_order=(1, 0), + assumed_align=_ALIGNMENT_BYTES, + ) + + workspace_numel = cute.sym_int32(divisibility=_BF16_VEC_SIZE) + workspace_num_vectors = workspace_numel // _BF16_VEC_SIZE + fake_buffers = [] + fake_ready_flags = [] + for _ in range(world_size): + buffer_tensor = make_fake_compact_tensor( + cutlass.BFloat16, + (_PING_PONG_SIZE, world_size, workspace_numel), + stride_order=(2, 1, 0), + assumed_align=_ALIGNMENT_BYTES, + ) + fake_buffers.append(buffer_tensor) + ready_flag_tensor = make_fake_compact_tensor( + cutlass.Int32, + (_PING_PONG_SIZE, world_size, workspace_num_vectors), + stride_order=(2, 1, 0), + assumed_align=_READY_FLAG_ALIGNMENT_BYTES, + ) + fake_ready_flags.append(ready_flag_tensor) + + fake_signal_state = make_fake_compact_tensor( + cutlass.Int32, + (_SIGNAL_STATE_SIZE,), + assumed_align=_READY_FLAG_ALIGNMENT_BYTES, + ) + + fused_host = ( + _tp_allreduce_residual_rms_norm_bf16_oneshot_lamport_split_row + if split_rows + else _tp_allreduce_residual_rms_norm_bf16_oneshot_lamport + ) + compile_args = [ + input_tensor, + residual_tensor, + norm_weight_tensor, + norm_output_tensor, + residual_output_tensor, + fake_buffers, + fake_ready_flags, + ] + if split_rows: + compile_args.append(fake_signal_state) + compile_args.append(cutlass.Int64(0)) + compile_args.extend( + [ + cutlass.Int64(0), + make_fake_stream(), + cutlass.Float32(eps), + n, + local_rank, + world_size, + ] + ) + if split_rows: + peer_order = tuple(range(world_size)) + if rotate_peer_order: + peer_order = tuple( + (local_rank + peer_ordinal) % world_size for peer_ordinal in range(world_size) + ) + compile_args.append(ctas_per_row) + compile_args.append(fused_threads_per_block) + compile_args.append(use_ready_flags) + compile_args.append(use_device_signal_state) + compile_args.append(elide_self_rank) + compile_args.append(peer_order) + compile_args.append(prefetch_fused_inputs) + compile_args.append(fused_direct_cluster_reduction) + compile_args.append(fused_post_norm_cluster_barrier) + compile_args.append(fused_fp32_accumulation) + compile_args.append(fused_batch_poll_peers) + compile_args.append(fused_ready_mask_poll_peers) + compile_args.append(aggregate_signal_at_cluster_root) + compile_args.append(use_pdl) + compile_args.append(pdl_release_policy == "strict") + compile_args.append(pdl_release_policy == "pre_norm") + compile_args.append(pdl_release_policy == "entry") + compile_args.append(last_cta_signal_advance) + compile_args.append(all_to_all_cluster_reduction) + compile_args.append(early_input_load) + compile_args.append(multicast_publish) + compile_args.append(iket_phases) + compile_args.append(rotated_poll_order) + + return cute.compile( + fused_host, + *compile_args, + options="--enable-tvm-ffi", + ) + + +def compile_fused_out_quant_for_rank( + local_rank: int = 0, + world_size: int = 1, + device_id: int = 0, + n: int = _DEFAULT_N, + eps: float = _RMS_EPS, + split_rows: bool = True, + use_ready_flags: bool = _USES_READY_FLAGS, + use_device_signal_state: bool = _USES_DEVICE_SIGNAL_STATE, + rotate_peer_order: bool = False, + elide_self_rank: bool = False, + aggregate_signal_at_cluster_root: bool = False, + ctas_per_row: int = _FUSED_CTAS_PER_ROW, + use_pdl: bool | None = None, + pdl_release_policy: str = "strict", + last_cta_signal_advance: bool = False, + all_to_all_cluster_reduction: bool = False, + threads_per_block: int = _SPLIT_ROW_THREADS_PER_BLOCK, + early_input_load: bool = False, + multicast_publish: bool = False, + iket_phases: bool = False, + prefetch_fused_inputs: bool = False, + rotated_poll_order: bool = False, +): + cutlass.cuda.initialize_cuda_context(device_id) + if n % _SCALE_BLOCK != 0: + raise ValueError(f"n must be divisible by {_SCALE_BLOCK}, got {n}") + _validate_ctas_per_row(ctas_per_row) + if threads_per_block not in _OUT_QUANT_THREADS_PER_BLOCK_CHOICES: + raise ValueError( + "threads_per_block must be one of " + f"{_OUT_QUANT_THREADS_PER_BLOCK_CHOICES}, got {threads_per_block}" + ) + _validate_pdl_release_policy(pdl_release_policy) + _validate_multicast_publish(multicast_publish, use_ready_flags, split_rows) + if use_pdl is None: + use_pdl = True + _validate_signal_advance_flags( + last_cta_signal_advance, + aggregate_signal_at_cluster_root, + use_device_signal_state, + use_pdl, + pdl_release_policy, + ) + split_row_alignment = ctas_per_row * _SCALE_BLOCK + if not split_rows: + raise ValueError( + "NVFP4 out-quant fused allreduce is implemented only for the split-row serving launch." + ) + if n % split_row_alignment != 0: + raise ValueError( + f"n must be divisible by {split_row_alignment} for split-row " + f"NVFP4 out-quant launch, got {n}" + ) + + sym_m = cute.sym_int32() + input_tensor = make_fake_compact_tensor( + cutlass.BFloat16, + (sym_m, n), + stride_order=(1, 0), + assumed_align=_ALIGNMENT_BYTES, + ) + residual_tensor = make_fake_compact_tensor( + cutlass.BFloat16, + (sym_m, n), + stride_order=(1, 0), + assumed_align=_ALIGNMENT_BYTES, + ) + norm_weight_tensor = make_fake_compact_tensor( + cutlass.BFloat16, + (n,), + assumed_align=_ALIGNMENT_BYTES, + ) + global_scale_tensor = make_fake_compact_tensor( + cutlass.Float32, + (1,), + assumed_align=4, + ) + norm_output_tensor = make_fake_compact_tensor( + cutlass.BFloat16, + (sym_m, n), + stride_order=(1, 0), + assumed_align=_ALIGNMENT_BYTES, + ) + fp4_output_tensor = make_fake_compact_tensor( + cutlass.Int8, + (sym_m, n // 2), + stride_order=(1, 0), + assumed_align=_ALIGNMENT_BYTES, + ) + scale_output_tensor = make_fake_compact_tensor( + cutlass.Int8, + (sym_m, n // _SCALE_BLOCK), + stride_order=(1, 0), + assumed_align=_ALIGNMENT_BYTES, + ) + residual_output_tensor = make_fake_compact_tensor( + cutlass.BFloat16, + (sym_m, n), + stride_order=(1, 0), + assumed_align=_ALIGNMENT_BYTES, + ) + + workspace_numel = cute.sym_int32(divisibility=_BF16_VEC_SIZE) + workspace_num_vectors = workspace_numel // _BF16_VEC_SIZE + fake_buffers = [] + fake_ready_flags = [] + for _ in range(world_size): + buffer_tensor = make_fake_compact_tensor( + cutlass.BFloat16, + (_PING_PONG_SIZE, world_size, workspace_numel), + stride_order=(2, 1, 0), + assumed_align=_ALIGNMENT_BYTES, + ) + fake_buffers.append(buffer_tensor) + ready_flag_tensor = make_fake_compact_tensor( + cutlass.Int32, + (_PING_PONG_SIZE, world_size, workspace_num_vectors), + stride_order=(2, 1, 0), + assumed_align=_READY_FLAG_ALIGNMENT_BYTES, + ) + fake_ready_flags.append(ready_flag_tensor) + + fake_signal_state = make_fake_compact_tensor( + cutlass.Int32, + (_SIGNAL_STATE_SIZE,), + assumed_align=_READY_FLAG_ALIGNMENT_BYTES, + ) + + peer_order = tuple(range(world_size)) + if rotate_peer_order: + peer_order = tuple( + (local_rank + peer_ordinal) % world_size for peer_ordinal in range(world_size) + ) + + return cute.compile( + _tp_allreduce_residual_rms_norm_out_quant_nvfp4_oneshot_lamport_split_row, + input_tensor, + residual_tensor, + norm_weight_tensor, + global_scale_tensor, + norm_output_tensor, + fp4_output_tensor, + scale_output_tensor, + residual_output_tensor, + fake_buffers, + fake_ready_flags, + fake_signal_state, + cutlass.Int64(0), + cutlass.Int64(0), + make_fake_stream(), + cutlass.Float32(eps), + n, + local_rank, + world_size, + ctas_per_row, + threads_per_block, + use_ready_flags, + use_device_signal_state, + peer_order, + elide_self_rank, + aggregate_signal_at_cluster_root, + use_pdl, + pdl_release_policy == "strict", + pdl_release_policy == "pre_norm", + pdl_release_policy == "entry", + last_cta_signal_advance, + all_to_all_cluster_reduction, + early_input_load, + multicast_publish, + iket_phases, + prefetch_fused_inputs, + rotated_poll_order, + options="--enable-tvm-ffi", + ) + + +def _compile_twoshot_common( + jit_entry, + *, + local_rank: int, + world_size: int, + device_id: int, + n: int, + eps: float, + use_device_signal_state: bool, + use_pdl: bool | None, + pdl_release_policy: str, + threads_per_block: int, + rows_per_cta: int, + emit_nvfp4: bool, +): + cutlass.cuda.initialize_cuda_context(device_id) + if threads_per_block not in _TWOSHOT_THREADS_PER_BLOCK_CHOICES: + raise ValueError( + "threads_per_block must be one of " + f"{_TWOSHOT_THREADS_PER_BLOCK_CHOICES}, got {threads_per_block}" + ) + if n % (_BF16_VEC_SIZE * threads_per_block) != 0: + raise ValueError( + "two-shot kernel needs n divisible by 8 x threads_per_block " + f"(each thread owns whole 16 B vectors of a row), got n={n}, " + f"threads_per_block={threads_per_block}" + ) + if emit_nvfp4 and n % _SCALE_BLOCK != 0: + raise ValueError(f"n must be divisible by {_SCALE_BLOCK}, got {n}") + if rows_per_cta < 1: + raise ValueError(f"rows_per_cta must be positive, got {rows_per_cta}") + if pdl_release_policy not in ("entry", "output_safe"): + raise ValueError( + "two-shot pdl_release_policy must be 'entry' or 'output_safe', " + f"got {pdl_release_policy!r}" + ) + if use_pdl is None: + use_pdl = True + if not use_device_signal_state: + raise ValueError("two-shot kernel requires device signal state") + # The CTA-role options are module constants. + # A negative consumer count would + # make the producer stride grid_x - C exceed the producer count and skip + # row-block groups, so reject it here for every caller. + if _TWOSHOT_MAX_CTAS < 0 or _TWOSHOT_CONSUMER_CTAS < 0: + raise ValueError( + "two-shot CTA counts must be non-negative, got max_ctas=" + f"{_TWOSHOT_MAX_CTAS}, consumer_ctas={_TWOSHOT_CONSUMER_CTAS}" + ) + if _TWOSHOT_DIRECT not in (0, 1): + raise ValueError(f"_TWOSHOT_DIRECT must be 0 or 1, got {_TWOSHOT_DIRECT}") + + sym_m = cute.sym_int32() + input_tensor = make_fake_compact_tensor( + cutlass.BFloat16, + (sym_m, n), + stride_order=(1, 0), + assumed_align=_ALIGNMENT_BYTES, + ) + residual_tensor = make_fake_compact_tensor( + cutlass.BFloat16, + (sym_m, n), + stride_order=(1, 0), + assumed_align=_ALIGNMENT_BYTES, + ) + norm_weight_tensor = make_fake_compact_tensor( + cutlass.BFloat16, (n,), assumed_align=_ALIGNMENT_BYTES + ) + norm_output_tensor = make_fake_compact_tensor( + cutlass.BFloat16, + (sym_m, n), + stride_order=(1, 0), + assumed_align=_ALIGNMENT_BYTES, + ) + residual_output_tensor = make_fake_compact_tensor( + cutlass.BFloat16, + (sym_m, n), + stride_order=(1, 0), + assumed_align=_ALIGNMENT_BYTES, + ) + workspace_numel = cute.sym_int32(divisibility=_BF16_VEC_SIZE) + workspace_num_vectors = workspace_numel // _BF16_VEC_SIZE + fake_buffers = [] + fake_ready_flags = [] + for _ in range(world_size): + fake_buffers.append( + make_fake_compact_tensor( + cutlass.BFloat16, + (_PING_PONG_SIZE, world_size, workspace_numel), + stride_order=(2, 1, 0), + assumed_align=_ALIGNMENT_BYTES, + ) + ) + fake_ready_flags.append( + make_fake_compact_tensor( + cutlass.Int32, + (_PING_PONG_SIZE, world_size, workspace_num_vectors), + stride_order=(2, 1, 0), + assumed_align=_READY_FLAG_ALIGNMENT_BYTES, + ) + ) + fake_signal_state = make_fake_compact_tensor( + cutlass.Int32, (_SIGNAL_STATE_SIZE,), assumed_align=_READY_FLAG_ALIGNMENT_BYTES + ) + tail = ( + fake_buffers, + fake_ready_flags, + fake_signal_state, + cutlass.Int64(0), + cutlass.Int64(0), + make_fake_stream(), + cutlass.Float32(eps), + n, + local_rank, + world_size, + threads_per_block, + rows_per_cta, + use_device_signal_state, + use_pdl, + pdl_release_policy == "entry", + ) + if emit_nvfp4: + global_scale_tensor = make_fake_compact_tensor(cutlass.Float32, (1,), assumed_align=4) + fp4_output_tensor = make_fake_compact_tensor( + cutlass.Int8, + (sym_m, n // 2), + stride_order=(1, 0), + assumed_align=_ALIGNMENT_BYTES, + ) + scale_output_tensor = make_fake_compact_tensor( + cutlass.Int8, + (sym_m, n // _SCALE_BLOCK), + stride_order=(1, 0), + assumed_align=_ALIGNMENT_BYTES, + ) + return cute.compile( + jit_entry, + input_tensor, + residual_tensor, + norm_weight_tensor, + global_scale_tensor, + norm_output_tensor, + fp4_output_tensor, + scale_output_tensor, + residual_output_tensor, + *tail, + options="--enable-tvm-ffi", + ) + return cute.compile( + jit_entry, + input_tensor, + residual_tensor, + norm_weight_tensor, + norm_output_tensor, + residual_output_tensor, + *tail, + options="--enable-tvm-ffi", + ) + + +def compile_fused_twoshot_for_rank( + local_rank: int = 0, + world_size: int = 1, + device_id: int = 0, + n: int = _DEFAULT_N, + eps: float = _RMS_EPS, + use_device_signal_state: bool = _USES_DEVICE_SIGNAL_STATE, + use_pdl: bool | None = None, + pdl_release_policy: str = "output_safe", + threads_per_block: int = _TWOSHOT_THREADS_PER_BLOCK, + rows_per_cta: int = _TWOSHOT_ROWS_PER_CTA, + **_ignored_oneshot_options, +): + """Fused residual-add + RMSNorm (RESIDUAL_RMS_NORM) two-shot NVLS kernel; same + launch signature as ``compile_fused_for_rank`` so adapters can hold both and pick by M. + One-shot-only options passed by shared adapter code are accepted and ignored.""" + return _compile_twoshot_common( + _tp_allreduce_residual_rms_norm_bf16_twoshot_nvls, + local_rank=local_rank, + world_size=world_size, + device_id=device_id, + n=n, + eps=eps, + use_device_signal_state=use_device_signal_state, + use_pdl=use_pdl, + pdl_release_policy=pdl_release_policy, + threads_per_block=threads_per_block, + rows_per_cta=rows_per_cta, + emit_nvfp4=False, + ) diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/__init__.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/__init__.py new file mode 100644 index 000000000000..d7dbf40094ca --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Mistral-Large-4-1T-A52B-NVFP4 / sm_103 / dep, the language model only.""" diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/modeling.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/modeling.py new file mode 100644 index 000000000000..587eb8b45749 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/modeling.py @@ -0,0 +1,2597 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""ModelingV2 target: mistral-large-4-nvfp4 / sm_103 / dep, native text and image inference. + +72 layers, hidden 6144, 64 query heads. Attention is MLA without a q-LoRA (`wq` straight to +64 x 128), a 512-wide latent plus a 64-wide rope key, and DeepSeek Sparse Attention: an +indexer on the 44 `index_sharing == 'full'` layers selects 2048 positions per query row and +each 'shared' layer reuses the selection of the most recent full layer. Every fourth layer +(`i % 4 == 3`) takes no rope, for MLA and for its indexer; the NoPE layers 39, 43, ..., 71 own +no latent and attend over layer 35's. Queries are scaled by SSMax, `s * log(pos + 1) + b` per +head, on MLA and on the indexer. Layer 0 is a dense SwiGLU MLP, layers 1-71 MoE (128 experts, +softmax top-4 renormalised, one shared expert). The final logits are soft-capped at +`softmax_tanh_gating`. + +The checkpoint is ModelOpt `MIXED_PRECISION`. The routed experts are NVFP4 W4A4: e2m1 weights +with one e4m3 scale per 16 inputs and a global scale per expert, and activations quantized to +NVFP4 under one static global scale per layer and projection. `wq`, `wo`, the indexer's `wq_b` +and the dense and shared MLPs are FP8 e4m3 with 128x128 power-of-two block scales and dynamic +1x128 activation scales. The rest is bf16. The latent KV pool is fp8_ds_mla (656 B per token +and owner layer: 512 e4m3 codes with one power-of-two fp32 scale per 128, and the 64-wide rope +key in bf16), owned by `MistralLarge4CacheManager` and per rank under attention DP; the indexer +key pool is fp8 with one scale per key. + +`dep` is attention data parallel over the ranks of the mapping, four on one node or eight on +two nodes of four (dep8): each rank serves its own requests with the whole model outside the +routed experts -- all 64 heads, the latent, the indexer, the dense layer-0 MLP, the shared +expert and the full-vocabulary `lm_head` -- and holds the KV of its own requests only. Nothing +before the routed experts communicates. The routed experts are expert parallel, +`n_routed_experts / moe_ep_size` of the 128 per rank (32 at dep4, 16 at dep8): each MoE layer +zero-pads the rank's rows to the largest rank's count, gathers every rank's rows before the +router (so every rank routes the identical token set and the windows tile the expert ids +exactly once), runs the router, the routing, the NVFP4 quantization and this rank's window over +the gathered rows in calls of at most `expert_call_tokens` rows, and reduce-scatters the partial +sums back to each rank's own rows. Every rank issues the same collective sequence whatever its +batch -- one gather and one reduce-scatter per MoE layer, of the same size -- in the prefill and +the decode target alike. + +The absorbed form throughout: `fused_q = [q_nope @ W_UK | rope(q_pe)]` (576 wide) against the +pooled `[c_kv | k_pe]` latent rows, then `@ W_UV`. Context rows attend over the selection (the +engine's causal index on a step it skipped the indexer) with FlashMLA's bf16 sparse prefill +over the requests' latent rows dequantized out of the pool, so they read the FP8-rounded latent +as the reference does; generation rows run FlashMLA's sparse fp8 decode over the pool itself. +Both kernels take the rank's 64 heads in one call. + +No path is selected by the serving configuration: each step chooses from its own sizes, and +every buffer is sized from the engine's quantities (the attention metadata's max_num_tokens, +max_num_requests, max_seq_len and runtime_features). The context rows take one of two paths, +chosen per step by the rank's own context KV (every context request's cached and new rows): +while it fits the held workspace, the sparse prefill over those rows dequantized into it; past +it, each context row as a one-row sequence of the sparse fp8 decode over the pool itself, as +generation rows always run. The held workspace, allocated on the first context step, has room +for the most context KV one step can attend, twice over, up to 2**19 rows +(`_ctx_workspace_rows`): a step whose context KV fits twice keeps layer 35's rows beside its +own for the NoPE layers attending over that latent, a longer one gathers them again for each +of those layers. An indexer chunk the engine packs past `max_seq_len` keys is scored request +by request, the indexer's logits are tiled under a budget from the engine's max_num_tokens and +max_seq_len (`_prefill_logits_budget`), each tile scored with a split-KV schedule built once per +step (every SM busy whatever the tile's row count) and selected with the self-sampling GVR +top-k (`_select_ctx`), and the expert path's calls take the world's `_EXPERT_CALL_TOKENS`, or +`_EXPERT_CALL_TOKENS_CHUNKED` under chunked prefill.""" + +from __future__ import annotations + +import itertools +import math +import os +import time +from dataclasses import replace +from types import SimpleNamespace +from typing import NamedTuple, Optional, cast + +import torch +from torch import nn +from transformers import PretrainedConfig + +from tensorrt_llm._torch.attention.backends.interface import AttentionMetadata +from tensorrt_llm._torch.attention.backends.sparse.params import ( + get_indexer_mqa_logits_workspace_bytes, +) +from tensorrt_llm._torch.attention.backends.trtllm import TrtllmAttentionMetadata +from tensorrt_llm._torch.model_config import ModelConfig +from tensorrt_llm._torch.modeling_v2._router_index import step_contract_enabled +from tensorrt_llm._torch.modeling_v2.catalog.activation.flashinfer_silu_and_mul import ( + flashinfer_silu_and_mul, +) +from tensorrt_llm._torch.modeling_v2.catalog.attention.thop_attention import thop_attention +from tensorrt_llm._torch.modeling_v2.catalog.comm.allgather import allgather +from tensorrt_llm._torch.modeling_v2.catalog.comm.reducescatter import reducescatter +from tensorrt_llm._torch.modeling_v2.catalog.gemm.bmm_out import bmm_out +from tensorrt_llm._torch.modeling_v2.catalog.gemm.cublas_mm import cublas_mm +from tensorrt_llm._torch.modeling_v2.catalog.moe.fp4_block_scale_moe_runner import ( + fp4_block_scale_moe_runner, +) +from tensorrt_llm._torch.modeling_v2.catalog.norm.flashinfer_fused_add_rmsnorm import ( + flashinfer_fused_add_rmsnorm, +) +from tensorrt_llm._torch.modeling_v2.catalog.norm.flashinfer_rmsnorm import flashinfer_rmsnorm +from tensorrt_llm._torch.modeling_v2.catalog.quantization.fp4_quantize import fp4_quantize +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.activation.gelu_erf import gelu_erf +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.activation.tanh_softcap import ( + tanh_softcap, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.convert_req_index_to_global import ( # noqa: E501 + convert_req_index_to_global, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_fp8_paged_mqa_logits import ( # noqa: E501 + cute_dsl_fp8_paged_mqa_logits, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_gvr_topk_prefill import ( # noqa: E501 + cute_dsl_gvr_topk_prefill, + gvr_warmup, + gvr_workspace, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_indexer_k_cache_update import ( # noqa: E501 + build as build_ix_k_cache, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_indexer_k_cache_update import ( # noqa: E501 + cute_dsl_indexer_k_cache_update, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_indexer_q_side_fused import ( # noqa: E501 + build as build_ix_q_side, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_indexer_q_side_fused import ( # noqa: E501 + cute_dsl_indexer_q_side_fused, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_radix_topk_decode import ( # noqa: E501 + build as build_ix_topk_decode, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_radix_topk_decode import ( # noqa: E501 + cute_dsl_radix_topk_decode, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ds_mla_dequant_gather import ( + ds_mla_dequant_gather, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ds_mla_rope_append import ( + ds_mla_rope_append, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.flash_mla_sparse_decode import ( + flash_mla_sparse_decode, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.flash_mla_sparse_prefill import ( + flash_mla_sparse_prefill, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.fp8_mqa_logits import ( + fp8_mqa_logits, + mqa_logits_schedule, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.indexer_k_cache_gather_op import ( # noqa: E501 + indexer_k_cache_gather_op, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.indexer_topk_decode import ( + indexer_topk_decode, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ssmax_indexer_q_side_fused import ( # noqa: E501 + ssmax_indexer_q_side_fused, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ssmax_scaling import ( + ssmax_scaling, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ssmax_scaling_fused_q import ( + ssmax_scaling_fused_q, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.gemm.fp8_prequantized_swap_ab_gemm import ( + fp8_prequantized_swap_ab_gemm, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.moe.renorm_moe_routing_op import ( + renorm_moe_routing_op, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.norm.layer_norm import layer_norm +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.quantization.fp8_quantize_1x128_packed_ue8m0 import ( # noqa: E501 + fp8_quantize_1x128_packed_ue8m0, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.vision.pixtral_rope_2d import ( + pixtral_rope_2d, +) +from tensorrt_llm._torch.models.modeling_multimodal_utils import ( + find_input_mm_embeds, + get_multimodal_embeddings, +) +from tensorrt_llm._torch.models.modeling_utils import DecoderModel, DecoderModelForCausalLM +from tensorrt_llm._torch.pyexecutor.guided_decoder import CapturableGuidedDecoder +from tensorrt_llm._torch.speculative import get_spec_worker +from tensorrt_llm._utils import mpi_disabled +from tensorrt_llm.flash_mla import get_mla_metadata +from tensorrt_llm.inputs.multimodal import MultimodalParams +from tensorrt_llm.logger import logger +from tensorrt_llm.models.modeling_utils import QuantConfig + +from . import weights as _weights + +#: eps 1e-5, the reference's value; params.json's `vision_encoder` states none. +_VISION_EPS = 1e-5 + +#: The thop attention arguments the vision tower's call holds fixed: the no-KV-cache +#: mode (an encoder: the three cache-addressing arguments None), one context sequence +#: per image, the fused `[q | k | v]` rows already rotated (no rope in the op), +#: bidirectional within each sequence (mask_type 0), softmax scale head_dim^-0.5. +_VISION_ATTN = dict( + k=None, + v=None, + output_sf=None, + max_context_q_len_override=None, + kv_cache_block_offsets=None, + host_kv_cache_pool_pointers=None, + host_kv_cache_pool_mapping=None, + cache_indirection=None, + kv_scale_orig_quant=None, + kv_scale_quant_orig=None, + out_scale=None, + rotary_inv_freq=None, + rotary_cos_sin=None, + latent_cache=None, + q_pe=None, + block_ids_per_seq=None, + attention_sinks=None, + is_fused_qkv=True, + update_kv_cache=True, + predicted_tokens_per_seq=1, + local_layer_idx=0, + tokens_per_block=None, + beam_width=1, + mask_type=0, + quant_mode=0, + q_scaling=1.0, + position_embedding_type=0, + rope_dim=0, + rope_base=10000.0, + rope_scale_type=0, + rope_scale=1.0, + rope_short_m_scale=1.0, + rope_long_m_scale=1.0, + rope_max_positions=1024, + rope_original_max_positions=1024, + use_paged_context_fmha=False, + attention_input_type=0, + is_mla_enable=False, + chunked_prefill_buffer_batch_size=1, + q_lora_rank=None, + kv_lora_rank=None, + qk_nope_head_dim=None, + qk_rope_head_dim=None, + v_head_dim=None, + rope_append=None, + mrope_rotary_cos_sin=None, + mrope_position_deltas=None, + helix_position_offsets=None, + helix_is_inactive_rank=None, + attention_chunk_size=None, + softmax_stats_tensor=None, + is_spec_decoding_enabled=False, + use_spec_decoding=False, + is_spec_dec_tree=False, + spec_decoding_generation_lengths=None, + spec_decoding_position_offsets=None, + spec_decoding_packed_mask=None, + spec_decoding_bl_tree_mask_offset=None, + spec_decoding_bl_tree_mask=None, + spec_bl_tree_first_sparse_mask_offset_kv=None, + sparse_kv_indices=None, + sparse_kv_offsets=None, + sparse_attn_indices=None, + sparse_attn_offsets=None, + sparse_attn_indices_block_size=0, +) + +# The latent pool layout this target addresses (`MistralLarge4CacheManager.latent_layout`). +_LATENT_LAYOUT = "fp8_ds_mla" +# The query heads FlashMLA's sm100 sparse kernels run: under attention DP a rank's own 64. +_FLASH_MLA_HEADS = 64 +# The pre-routed MoE call: the runner's routing method is inert once ids and weights are +# given; 1 (renormalize) is the value its cells drive. +_MOE_ROUTING_METHOD = 1 +# The routed experts' SwiGLU, the runner's `act_type` 0. +_MOE_ACT_SWIGLU = 0 +# NVFP4: one e4m3 scale per 16 activations, laid out row-major (linear), the order the MoE +# runner reads; the 128x4-swizzled order is another buffer of the same length at 128-row +# multiples, and silently wrong there. +_NVFP4_SF_VEC = 16 +_NVFP4_SF_LINEAR = False +# fp8_mqa_logits allocates a `[q_rows, kv_rows]` fp32 score matrix per call; the context +# queries are tiled to keep it under the engine's budget (`_prefill_logits_budget`). Context +# steps are not captured. +# capacity: fixed: the budget's ceiling (256 MiB): 64 rows at 1048576 keys. The engine's first +# context step leaves one tile's block cached, device memory held once up beside the KV pool: a +# 1 GiB tile (1 << 28) left a dep4 rank under the 2%-of-device free-memory margin at 0.9 of free +# memory to KV at max_seq_len 1048576. +_PREFILL_LOGITS_CEILING = 1 << 26 +# The decode indexer's top-k: True runs cute_dsl_radix_topk_decode, one launch that selects the +# trtllm indexer_topk_decode op's set (up to exact ties), False that op itself. The CuTe DSL +# selector serves scorer rows from 4096 columns (where the op splits each row) to 65536 (one +# CTA scans a whole row), the widths its cells cover; the scorer's column count is the engine's +# max_seq_len, so an engine outside that range keeps the op (`_derive_first_step`). +_TOPK_DECODE_CUTE_DSL = True +_TOPK_DECODE_COLUMNS = (4096, 65536) +# The attention block's glue as separate ops instead of the fused entries, for old-versus-new +# verification (both compute the same bits; the fused entries' tests hold each pair equal): +# ssmax_scaling plus the rope tail's copy or zero fill for ssmax_scaling_fused_q; +# ssmax_scaling, the head weights' fp32 copy and cute_dsl_indexer_q_side_fused for +# ssmax_indexer_q_side_fused; a fresh RMSNorm and torch.cat for the in-place latent; the copy +# out of FlashMLA's 64-head output for the in-place read. Set +# TRTLLM_MODELING_V2_UNFUSED_GLUE=1 in every rank's environment at launch; off by default. +# It sets the core's `unfused_glue`, which every site reads. +_UNFUSED_GLUE = os.environ.get("TRTLLM_MODELING_V2_UNFUSED_GLUE", "0") == "1" +# capacity: fixed: by attention-DP world size, the most gathered tokens one router / routing / +# NVFP4-quantize / expert call takes, the token column those entries are certified to at that +# world's expert window. At 4 ranks (32 local experts) the gathered set is up to +# 4 x max_num_tokens = 131072 rows, one call per MoE layer. At 8 ranks (16 local experts) it is +# up to 8 x max_num_tokens = 262144 rows, in calls of 32768: the call bounds the runner's +# workspace, whose FC2 rows at 32768 tokens and top-4 stay below 2^31 bytes. A larger gathered +# set runs in chunks of this many rows, whose boundaries depend on the gathered row count alone, +# the same on every rank. A world size not listed here is refused. +_EXPERT_CALL_TOKENS = {4: 131072, 8: 32768} +# capacity: fixed: by attention-DP world size, the calls of an engine under chunked prefill, +# whose steps attend a cached prefix up to max_seq_len and hold the large context workspace: +# 32768 at both, a token column the expert path's entries are certified to at that world's +# expert window. A call's workspace grows with its rows: at 4 ranks one call over all 131072 +# gathered rows leaves the engine about 5.6 GiB less free memory once up, more than max_seq_len +# 1048576's margin spares; so the 131072 rows run in four calls. The chunks give one call's bits +# (test_modeling_v2_routed_experts.py). +_EXPERT_CALL_TOKENS_CHUNKED = {4: 32768, 8: 32768} +# The context path's crossover, chosen once per step on this rank's host ints, and the held +# context workspace's ceiling (`_ctx_workspace_rows`). A step whose context KV (every context +# request's cached and new rows) fits the held workspace has it gathered, dequantized, into it +# (bf16 [rows, 576]) for FlashMLA's sparse prefill; a longer one attends its context rows +# straight off the pool with the sparse decode kernel, whose cost does not grow with the prefix +# as the gather's does. +# capacity: fixed: 2**19 rows, 576 MiB, the gather-vs-decode crossover. +_CTX_WORKSPACE_ROWS = 1 << 19 +# The most context rows one sparse-decode call takes on that path: FlashMLA's decode scheduler +# exits the process, raising nothing, past 11622 one-row sequences of 64 heads on sm_103. +# capacity: fixed: the kernel's scheduler, not an engine limit. +_CTX_DECODE_ROWS = 8192 + + +def _prefill_logits_budget(max_num_tokens: int, max_seq_len: int) -> int: + """The most fp32 logits one indexer prefill tile holds, from the engine's max_num_tokens and + max_seq_len: mainline's reachable MQA-logits tile (`get_indexer_mqa_logits_workspace_bytes`, + 4 bytes a logit; TLLM_INDEXER_MQA_LOGITS_ELEM_BUDGET lowers it), within + `_PREFILL_LOGITS_CEILING`, and within max_seq_len / 64 query rows at max_seq_len keys. + + The last bound grows the tile with the engine's longest request, which it scores in at + least 64 tiles: 2**24 logits (512 rows of 32768 keys) at max_seq_len 32768, the ceiling + (64 rows of 1048576 keys) from max_seq_len 65536 on. Those are the budgets the 32768 and + 1048576 serving configs were certified at, so their tiles, and the fp8_mqa_logits and + prefill top-k cells that cover them, stay as they were.""" + reachable = get_indexer_mqa_logits_workspace_bytes(max_num_tokens, max_seq_len) // 4 + return min(reachable, _PREFILL_LOGITS_CEILING, max_seq_len * (max_seq_len >> 6)) + + +def _most_ctx_kv(md) -> int: + """The most context KV (cached and new rows over a step's context requests) one step can + attend, from the attention metadata alone. `runtime_features` is built once by the executor + for the service (chunked prefill from enable_chunked_prefill, cache reuse from the + KvCacheConfig's enable_block_reuse). With neither, every context row is new: + max_num_tokens. Otherwise mainline's never-stall worst case for a workspace chunked prefill + does not bound (`get_mla_context_workspace_kv_len_cap` without an + fp8_context_mla_kv_len_cap override; the gather reads every attended row, cached ones + included): min(max_num_requests, max_num_tokens) * max_seq_len. The target does not read + fp8_context_mla_kv_len_cap: the KvCacheConfig is not reachable from the model, and for these + targets the engine installs no admission cap that would bound a step to it.""" + features = md.runtime_features + if not (features.chunked_prefill or features.cache_reuse): + return md.max_num_tokens + return min(md.max_num_requests, md.max_num_tokens) * md.max_seq_len + + +def _ctx_workspace_rows(md) -> int: + """The held context workspace's rows: the most context KV one step can attend + (`_most_ctx_kv`) twice over, a step's rows and a shared owner's (layer 35's) kept beside + them, up to `_CTX_WORKSPACE_ROWS`. At max_num_tokens 32768 with neither chunked prefill nor + block reuse that is 65536 rows, the two `[32768, 576]` gathers a step there makes; with either, + at 16 requests of max_seq_len 32768 or more, 2**19.""" + return min(2 * _most_ctx_kv(md), _CTX_WORKSPACE_ROWS) + + +def _step_common(md: TrtllmAttentionMetadata, indexer_layers: list[int]) -> dict: + """The batch state both targets read: KV lengths and pool addressing for the MLA ops, the + indexer's key pools and slot mappings, the top-k conversion tables, the decode scorer's + schedule, and every rank's token count this step (attention DP: identical on every rank, + in rank order, unpadded). The indexer pool views are taken per step: the engine replaces its + KV cache manager during boot, and a view of the old one writes into a dead allocation.""" + mgr = md.kv_cache_manager + latent_pool = mgr.get_latent_pool() + return dict( + num_seqs=md.num_seqs, + num_tokens=md.num_tokens, + all_rank_num_tokens=[int(n) for n in md.all_rank_num_tokens], + kv_lens_cuda=md.kv_lens_cuda_runtime, + kv_lens_host=md.kv_lens_runtime, + prompt_lens_host=md.prompt_lens_cpu_runtime, + kv_cache_block_offsets=md.kv_cache_block_offsets, + host_kv_cache_pool_pointers=md.host_kv_cache_pool_pointers, + host_kv_cache_pool_mapping=md.host_kv_cache_pool_mapping, + tokens_per_block=md.tokens_per_block, + max_seq_len=md.max_seq_len, + # [blocks, owners, tokens_per_block, 656] bytes, and the same bytes as FlashMLA's + # pages: page `block * owners + row`, slot `page * tokens_per_block + offset`. + latent_pool=latent_pool, + latent_kv=latent_pool.view(-1, latent_pool.shape[2], 1, latent_pool.shape[3]), + ix_pools={i: mgr.get_indexer_k_cache_buffers(i) for i in indexer_layers}, + block_table=md.block_table, + req_idx_per_token=md.req_idx_per_token, + slot_mapping_fp8=md.slot_mapping_fp8, + slot_mapping_scale=md.slot_mapping_scale, + skip_gen=bool(md.skip_indexer_for_gen_reqs), + # Allocated only while the engine's short-sequence indexer skip is on; the two skip + # flags are False whenever it is off. + topk_indices_buffer=md.topk_indices_buffer if md.enable_indexer_skip else None, + indexer_k_cache_block_offsets=md.indexer_k_cache_block_offsets, + scheduler_metadata_buffer=md.scheduler_metadata_buffer, + radix_aux_indices=md.radix_aux_indices, + radix_aux_logits=md.radix_aux_logits, + ) + + +def _padded_rows(core: "MistralLarge4DepModelingV2", step: dict, rows: int) -> int: + """The row count every rank pads its block to before the expert path's collectives: the + largest rank's count this step. Every rank reads the same list, so every rank's collectives + move the same bytes; this rank's own entry must be the rows it holds, or its block would + be cut or overrun.""" + counts = step["all_rank_num_tokens"] + assert len(counts) == len(core.dp_group) and counts[core.dp_rank] == rows, ( + f"rank {core.dp_rank} holds {rows} rows; all_rank_num_tokens is {counts}" + ) + return max(counts) + + +def _padding_rows(counts: list[int], padded: int, device) -> torch.Tensor | None: + """Which of the gathered rows are padding: bool `[len(counts) * padded]` on `device`, or None + when no rank pads (every count is `padded`). Rank r's segment is rows `[r * padded, + (r + 1) * padded)`, its first `counts[r]` its own and the rest the zero rows `_moe` pads it + with, at `[r * padded + counts[r], (r + 1) * padded)`. Built from the step's host counts.""" + if all(n == padded for n in counts): + return None + own = torch.tensor(counts, dtype=torch.int64, device=device) + return (torch.arange(padded, device=device)[None, :] >= own[:, None]).reshape(-1) + + +def _ctx_kv_offsets(md: TrtllmAttentionMetadata, num_contexts: int) -> dict: + """The context rope/append op's per-request offsets: int64 prefix sums of the cached and + of the total (cached + new) lengths over the context requests, and the longest new run. + + The engine keeps them only under `enable_context_mla_with_cached_kv` (KV reuse or chunked + prefill). Without either, no context request has a cached prefix, so the cached sums are + zero and the totals are the new lengths: the int32 prefix sum of `seq_lens_cuda` the + metadata always builds, widened to the op's int64. + """ + if md.enable_context_mla_with_cached_kv: + return dict( + cu_ctx_cached=md.ctx_cached_token_indptr[: num_contexts + 1], + cu_ctx_kv=md.ctx_kv_indptr[: num_contexts + 1], + max_ctx_new=int(md.max_ctx_seq_len), + ctx_cached_tokens=int(md.num_ctx_cached_tokens), + ) + cu = md.mla_prepare_ctx_cu_seqlens() + return dict( + cu_ctx_cached=torch.zeros(num_contexts + 1, dtype=torch.int64, device=cu.device), + cu_ctx_kv=cu.to(torch.int64), + max_ctx_new=int(md.seq_lens[:num_contexts].max()), + ctx_cached_tokens=0, + ) + + +def _ctx_host_offsets(md: TrtllmAttentionMetadata, num_contexts: int) -> dict: + """The host lists over the context requests: the prefix sums of the new and of the total + (cached + new) lengths, which place each context request's rows in the step and its keys in + the indexer's full-KV gather mappings (`_prefill_blocks`). + + The engine's host copies under `enable_context_mla_with_cached_kv`; without it no request + has a cached prefix, so both are the prefix sum of the host `seq_lens`. Every source is a + host tensor the metadata fills on the host, so taking them each context step syncs nothing. + """ + if md.enable_context_mla_with_cached_kv: + return dict( + host_cu_ctx_new=md.host_ctx_uncached_token_indptr[: num_contexts + 1].tolist(), + host_cu_ctx_kv=md.host_ctx_kv_indptr[: num_contexts + 1].tolist(), + ) + host_cu = [0, *itertools.accumulate(md.seq_lens[:num_contexts].tolist())] + return dict(host_cu_ctx_new=host_cu, host_cu_ctx_kv=host_cu) + + +def _prefill_blocks( + chunk, cu_new: list[int], cu_kv: list[int], max_keys: int +) -> list[tuple[int, int, int, int]]: + """The indexer's scoring blocks over one engine prefill chunk, as host ints + `(token_start, token_end, k_token_start, n_k)`: rows `[token_start, token_end)` of the + step's context rows scored against keys `[k_token_start, k_token_start + n_k)` of its + full-KV gather mappings. + + One block, the chunk itself, while it holds at most `max_keys` keys (max_seq_len, the most + one request holds). The engine packs requests into one chunk while (summed new tokens) x + (summed KV) stays within its packed-logits budget, so a few short last chunks over long + prefixes pack past max_seq_len (two 5-token chunks over 983040 cached keys each: 1966090 + keys). Such a chunk is cut at its request boundaries, each request's rows against its own + keys: a row's window lies inside its own request's keys. `cu_new` / `cu_kv` are the + step's prefix sums of the new and of the cached + new tokens over its context requests, + in the order of the context rows and of the full-KV mappings.""" + n_k = chunk.k_token_end - chunk.k_token_start + if n_k <= max_keys: + return [(chunk.token_start, chunk.token_end, chunk.k_token_start, n_k)] + blocks = [] + for r in range(len(cu_new) - 1): + lo, hi = max(chunk.token_start, cu_new[r]), min(chunk.token_end, cu_new[r + 1]) + if lo >= hi: + continue + k_lo, k_hi = max(chunk.k_token_start, cu_kv[r]), min(chunk.k_token_end, cu_kv[r + 1]) + assert 0 < k_hi - k_lo <= max_keys, ( + f"context request {r} of an indexer chunk holds keys [{k_lo}, {k_hi}), past " + f"max_seq_len {max_keys}" + ) + blocks.append((lo, hi, k_lo, k_hi - k_lo)) + assert blocks and blocks[0][0] == chunk.token_start and blocks[-1][1] == chunk.token_end, ( + f"the requests {cu_new} do not tile the chunk's rows " + f"[{chunk.token_start}, {chunk.token_end})" + ) + return blocks + + +def _yarn_lerope_inv_freq(cfg, alpha: torch.Tensor) -> torch.Tensor: + """The rotary inverse frequencies, fp32, as the reference implementation builds + them: YaRN's blend of the extrapolated and the interpolated frequency, both scaled by + LeRoPE's `exp(alpha)`, over a ramp taken on the frequencies without LeRoPE.""" + rope = cfg.rope_parameters + dim = cfg.qk_rope_head_dim + theta = float(rope["rope_theta"]) + factor = float(rope["factor"]) + original = float(rope["original_max_position_embeddings"]) + pos_freqs = theta ** (torch.arange(0, dim, 2, dtype=torch.float32) / dim) + lerope = torch.exp(alpha.detach().to(device="cpu", dtype=torch.float32)) + extrapolated = 1.0 / pos_freqs * lerope + interpolated = 1.0 / (factor * pos_freqs) * lerope + + def correction_dim(rotations: float) -> float: + return dim * math.log(original / (rotations * 2 * math.pi)) / (2 * math.log(theta)) + + low = max(math.floor(correction_dim(float(rope.get("beta_fast", 32)))), 0) + high = min(math.ceil(correction_dim(float(rope.get("beta_slow", 1)))), dim - 1) + if low == high: + high += 0.001 + ramp = ((torch.arange(dim // 2, dtype=torch.float32) - low) / (high - low)).clamp(0, 1) + extrapolation = 1 - ramp + return interpolated * (1 - extrapolation) + extrapolated * extrapolation + + +def _rope_tables(cfg, alpha: torch.Tensor, positions: int, dtype, device) -> SimpleNamespace: + """The tables the rope consumers read, built from one fp32 angle grid and rounded to the + activation dtype first, as the reference's cos/sin cache is held. + + `indexer` is `[rows, 64]`, `[cos(32) | sin(32)]`: the layout the indexer q side and + `ds_mla_rope_append` (the MLA query tail and the latent's rope key) read; the indexer key + side reads the same storage as `[rows, 2, 32]`. `identity` (cos 1, sin 0) is the table of a + NoPE indexer layer, whose kernels have no no-rope mode. Every row depends only on its own + index, so a longer table is the same table with more rows. + """ + inv = _yarn_lerope_inv_freq(cfg, alpha) + t = torch.arange(positions, dtype=torch.float32) + angle = torch.einsum("i,j -> ij", t, inv) + cos = angle.cos().to(dtype).float() + sin = angle.sin().to(dtype).float() + identity = torch.cat([torch.ones_like(cos), torch.zeros_like(sin)], dim=1) + return SimpleNamespace( + rows=positions, + indexer=torch.cat([cos, sin], dim=1).to(device).contiguous(), + identity=identity.to(device).contiguous(), + ) + + +def _moe_scalars(isc1, isc2, ws2_13, ws2_2) -> SimpleNamespace: + """The NVFP4 MoE's call scalars from the checkpoint's per-tensor ones, fp32. + + ModelOpt stores `input_scale` and `weight_scale_2` as the reciprocals of the global scales a + quantizer divides by (`amax / (448 * 6)`), one `isc1` / `isc2` per layer and one + `ws2_13[e]` / `ws2_2[e]` per local expert. So the activation quantizer's global scale is + `1 / isc1`; FC1's accumulator dequantizes by `isc1 * ws2_13[e]` (the gate half, + `output1_scale_gate_scalar`) and, because the FC1 epilogue requantizes the SwiGLU output to + NVFP4 under `1 / isc2`, the linear half by `isc1 * ws2_13[e] / isc2` + (`output1_scale_scalar`); FC2's accumulator dequantizes by `isc2 * ws2_2[e]` + (`output2_scale_scalar`). The reference's TRT-LLM-Gen path folds the same three + (`g1_alphas`, `g1_scale_c`, `g2_alphas`).""" + gate1 = (isc1 * ws2_13).contiguous() + return SimpleNamespace( + gscale=(1.0 / isc1).contiguous(), + out1=(gate1 / isc2).contiguous(), + out1_gate=gate1, + out2=(isc2 * ws2_2).contiguous(), + ) + + +def _layer_views(core: "MistralLarge4DepModelingV2") -> list[SimpleNamespace]: + """Every layer's operands, as the views the entries read: GEMM operands transposed (`.t()`, + the column-major `[K, N]` cuBLAS and the FP8 GEMM read), the FP8 scale storage transposed to + the MN-major operand, `wkv_b` split into the absorption pair over all 64 heads, and the + NVFP4 MoE's call scalars (`_moe_scalars`). `next_norm` is the next layer's input norm, the + final norm after the last layer. A sharer's absorption pair is its own `wkv_b`; its latent is + its source's.""" + w = core.w + heads, v_dim, kv_lora = core.heads, core.v_dim, core.kv_lora + layers = [] + for i in range(core.num_layers): + p = f"l{i}_" + nope = core.qk_dim if core.nope_layer[i] else core.nope + kvb = w[p + "wkv_b"] + hn = heads * nope + owner = core.kv_source[i] == i + ix = None + if core.full_indexer[i]: + ix = SimpleNamespace( + q_norm_w=w[p + "ix_q_norm_w"], + q_norm_b=w[p + "ix_q_norm_b"], + wq_b=w[p + "ix_wq_b"], + wq_b_s=w[p + "ix_wq_b_s"].t(), + wk_wp=w[p + "ix_wk_wp"].t(), + k_norm_w=w[p + "ix_k_norm_w"], + k_norm_b=w[p + "ix_k_norm_b"], + ssmax_s=w[p + "ix_ssmax_s"], + ssmax_b=w[p + "ix_ssmax_b"], + rope=not core.indexer_nope[i], + ) + moe = None + if i >= core.dense_layers: + moe = SimpleNamespace( + router=w[p + "router"].t(), + fc1_w=w[p + "fc1_w"], + fc1_s=w[p + "fc1_s"], + fc2_w=w[p + "fc2_w"], + fc2_s=w[p + "fc2_s"], + **vars(_moe_scalars(w[p + "isc1"], w[p + "isc2"], w[p + "ws2_13"], w[p + "ws2_2"])), + ) + layers.append( + SimpleNamespace( + index=i, + rope=not core.nope_layer[i], + nope=nope, + owner=owner, + kv_source=core.kv_source[i], + attn_norm=w[p + "attn_norm"], + wq=w[p + "wq"], + wq_s=w[p + "wq_s"].t(), + ssmax_s=w[p + "ssmax_s"], + ssmax_b=w[p + "ssmax_b"], + wkv_a=w[p + "wkv_a"].t() if owner else None, + kv_norm=w[p + "kv_a_norm"] if owner else None, + k_b=kvb[:hn].reshape(heads, nope, kv_lora), + v_b_t=torch.transpose(kvb[hn:].reshape(heads, v_dim, kv_lora), 1, 2), + wo=w[p + "wo"], + wo_s=w[p + "wo_s"].t(), + ffn_norm=w[p + "ffn_norm"], + mlp_gu=w[p + "mlp_gu"], + mlp_gu_s=w[p + "mlp_gu_s"].t(), + mlp_dn=w[p + "mlp_dn"], + mlp_dn_s=w[p + "mlp_dn_s"].t(), + indexer=ix, + moe=moe, + next_norm=( + w[f"l{i + 1}_attn_norm"] if i + 1 < core.num_layers else w["final_norm"] + ), + ) + ) + return layers + + +class MistralLarge4DepModelingV2(DecoderModel): + def __init__(self, model_config: ModelConfig): + super().__init__(model_config) + cfg = model_config.pretrained_config + mapping = model_config.mapping + # The expert path's communicator: the attention-DP ranks, four on one node or eight over + # both nodes, in the order `all_rank_num_tokens` lists them and the gather concatenates + # them. + self.dp_group = list(mapping.tp_group) + self.dp_rank = mapping.tp_rank + # The attention block's glue: the fused entries (ssmax_scaling_fused_q, + # ssmax_indexer_q_side_fused, the latent RMSNorm in place, FlashMLA's decode output + # read in place) at every max_seq_len, whose cells cover the 32K and 1M configs' + # positions (to 1048575), rope tables and step shapes; the separate ops they replace + # under `_UNFUSED_GLUE`, at every site. + self.unfused_glue = _UNFUSED_GLUE + world = mapping.world_size + if world not in _EXPERT_CALL_TOKENS or len(self.dp_group) != world: + raise ValueError( + f"modeling_v2 Mistral Large 4 NVFP4 attention-DP target: world_size {world} with an " + f"attention-DP group of {len(self.dp_group)} ranks; it serves 4 ranks on one " + "node or 8 over two nodes of four, every rank in the one group" + ) + + self.num_layers = cfg.num_hidden_layers + self.hidden = cfg.hidden_size + self.eps = cfg.rms_norm_eps + # Attention DP: every rank runs every head on its own requests. + self.heads = cfg.num_attention_heads + self.nope = cfg.qk_nope_head_dim + self.rope_dim = cfg.qk_rope_head_dim + # A NoPE layer spends the rope dims on nope: the query head stays this wide. + self.qk_dim = self.nope + self.rope_dim + self.v_dim = cfg.v_head_dim + self.kv_lora = cfg.kv_lora_rank + self.lat_dim = self.kv_lora + self.rope_dim + # Every layer's softmax scale, NoPE layers included: qk_head_dim**-0.5, no YaRN + # temperature (its magnitude is 1). + self.softmax_scale = self.qk_dim**-0.5 + + self.nope_layer = [bool(f) for f in cfg.nope_layer_flags] + self.indexer_nope = [bool(f) for f in cfg.indexer_nope_layer_flags] + self.kv_source = [int(s) for s in cfg.kv_source_layers] + # An owner some later layer attends over (layer 35): a step whose context KV fits the + # held workspace twice keeps its gather there for the sharers; a longer one has each + # sharer gather its source's rows again (`_ctx_workspace`). + self.shared_source = [ + any(s == i and j != i for j, s in enumerate(self.kv_source)) + for i in range(self.num_layers) + ] + self.full_indexer = [role == "full" for role in cfg.index_sharing] + # A 'shared' layer attends with the most recent full layer's selection: each target + # keeps one selection buffer per step, which every full layer overwrites in order. + self.indexer_layers = [i for i in range(self.num_layers) if self.full_indexer[i]] + + self.idx_heads = cfg.index_n_heads + self.idx_dim = cfg.index_head_dim + self.idx_topk = cfg.index_topk + self.idx_eps = cfg.indexer_norm_eps + # The indexer's softmax scale and head average: its scorers take none, so the q side + # folds both into the head weights together with the q quantization scale. + self.idx_weight_scale = self.idx_dim**-0.5 * self.idx_heads**-0.5 + + self.dense_layers = cfg.first_k_dense_replace + self.num_experts = cfg.n_routed_experts + self.topk = cfg.num_experts_per_tok + self.moe_inter = cfg.moe_intermediate_size + self.local_experts = self.num_experts // mapping.moe_ep_size + self.expert_offset = self.local_experts * mapping.moe_ep_rank + # The rope tables' first size; grown on the first step to the engine's max_seq_len. + self.rope_rows = model_config.max_seq_len or cfg.max_position_embeddings + + self.w = _weights.MODEL_WEIGHTS.declare(_weights.MODEL_WEIGHTS.dims(self)) + + self._rope: SimpleNamespace | None = None + self._derived_for = None + self._ix_k = None + self._ix_k_checked = False + self._pool_row: list[int] = [] + self._stride_factor = 0 + self._max_ctx_len = 0 + # The engine's sizes, fixed on the first step (`_derive_first_step`): the indexer's + # prefill logits budget, the held context workspace's rows, the most gathered rows one + # expert-path call takes and the MoE autotuner's largest bucket. + self.prefill_logits_budget = 0 + self.ctx_workspace_rows = 0 + self.expert_call_tokens = 0 + self.expert_tune_tokens = 0 + # The decode top-k's handle, built on the first step at a scorer width it serves. + self._ix_topk = None + self._side_stream: torch.cuda.Stream | None = None + self.dp_pg = None + self._prefill = self._decode = None + + def post_load_weights(self) -> None: + """Derive what the weights fix, build the kernels, the shared expert's stream and the + communicator handle, then the targets. + + Named for the shell hook that drives it. Everything compiles here or on the first + (eager) step: a first compile inside a CUDA-graph capture breaks the graph. The side + stream is made here too: a stream created inside a capture is not capturable. + """ + dev = torch.cuda.current_device() + self.build_kernels(dev) + self._side_stream = torch.cuda.Stream(device=dev) + # The collectives' communicator flavor, as the engine's own collectives pick it: the + # MPI session's, or the tensor-parallel process group's when MPI is disabled. + mapping = self.model_config.mapping + self.dp_pg = mapping.tp_group_pg.boxed() if mpi_disabled() else None + self._prefill = PrefillTarget(self) + self._decode = DecodeTarget(self) + + def build_kernels(self, dev: int) -> None: + """The rank-local derivations: the rope tables (and, for the unfused glue, the indexer + q side).""" + self._rope = _rope_tables( + self.model_config.pretrained_config, + self.w["lerope_alpha"], + self.rope_rows, + self.w["final_norm"].dtype, + self.w["final_norm"].device, + ) + if self.unfused_glue: + # The checkpoint's rope is interleaved (GPT-J) for the indexer too. + self._ix_q = build_ix_q_side(dev, is_neox=False) + + def _derive_first_step(self, md) -> None: + """What depends on values the engine resolves, fixed on the first step it hands over + (eager, ahead of any capture) and again for a replaced KV cache manager.""" + if md.kv_cache_manager is self._derived_for: + return + dev = torch.cuda.current_device() + mgr = md.kv_cache_manager + # The pool layout is the KV cache manager's, which routing cannot see; every latent + # op here reads and writes fp8_ds_mla rows addressed by owner row. + if getattr(mgr, "latent_layout", None) != _LATENT_LAYOUT: + raise RuntimeError( + f"this target attends over an {_LATENT_LAYOUT} latent pool with owner-only " + f"pages (MistralLarge4CacheManager); the engine built {type(mgr).__name__} " + f"with layout {getattr(mgr, 'latent_layout', None)!r}. Use KV cache manager V2 " + "and kv_cache_config.dtype auto, fp8 or fp8_ds_mla." + ) + if list(mgr.kv_source_layers) != self.kv_source: + raise RuntimeError("the KV cache manager's kv_source_layers differ from the config's") + # A table shorter than max_seq_len is read past its end with no check. + if md.max_seq_len > self._rope.rows: + self.rope_rows = md.max_seq_len + self._rope = _rope_tables( + self.model_config.pretrained_config, + self.w["lerope_alpha"], + self.rope_rows, + self.w["final_norm"].dtype, + self.w["final_norm"].device, + ) + # Layer i reads owner row `_pool_row[i]` (its source's); a block holds every owner's + # page, so a global row is `block * owners * tpb + row * tpb + offset`. + self._pool_row = [mgr.latent_pool_row(i) for i in range(self.num_layers)] + self._stride_factor = len(mgr.kv_owner_layers) * md.tokens_per_block + # The decode scorer's column count: frozen into a capture, so an engine constant. + self._max_ctx_len = md.max_seq_len + # The indexer's prefill logits budget and the held context workspace's rows, from the + # engine's max_num_tokens, max_num_requests, max_seq_len and runtime features. + self.prefill_logits_budget = _prefill_logits_budget(md.max_num_tokens, md.max_seq_len) + self.ctx_workspace_rows = _ctx_workspace_rows(md) + # The expert path's calls at this world size: the smaller ones under chunked prefill, + # the engines whose steps attend a cached prefix up to max_seq_len and hold the large + # context workspace. + calls = ( + _EXPERT_CALL_TOKENS_CHUNKED + if md.runtime_features.chunked_prefill + else _EXPERT_CALL_TOKENS + ) + self.expert_call_tokens = calls[len(self.dp_group)] + # capacity: max_num_tokens -- the MoE autotuner's largest bucket, the most rows one + # expert call holds: every rank's max_num_tokens gathered, capped at one call. + self.expert_tune_tokens = min( + self.expert_call_tokens, len(self.dp_group) * md.max_num_tokens + ) + logger.info( + f"modeling_v2 Mistral Large 4 NVFP4: sized from the engine: prefill logits budget " + f"{self.prefill_logits_budget}, context workspace {self.ctx_workspace_rows} rows, " + f"expert calls of {self.expert_call_tokens} rows" + ) + # The decode selector, by the scorer's column count: cute_dsl_radix_topk_decode's + # handle (built once) at a width it serves, else none and the trtllm op selects. + lo, hi = _TOPK_DECODE_COLUMNS + if not (_TOPK_DECODE_CUTE_DSL and lo <= self._max_ctx_len <= hi): + self._ix_topk = None + elif self._ix_topk is None: + self._ix_topk = build_ix_topk_decode(dev, next_n=1) + # Both key-pool layouts: a launcher compiled for a contiguous view addresses a strided + # one as if it were dense. + if self._ix_k is None: + self._ix_k = { + contiguous: build_ix_k_cache(dev, is_neox=False, cache_contiguous=contiguous) + for contiguous in (False, True) + } + self._derived_for = md.kv_cache_manager + + def _select_target(self, attn_metadata) -> "PrefillTarget | DecodeTarget": + """Which target runs this step: one host-side integer read. A mixed batch goes to + prefill, so prefill is the general case and decode the specialization that may assume + it has no context rows. The ranks' choices differ in one step and need not agree: both + targets issue the same collectives.""" + return self._decode if attn_metadata.num_contexts == 0 else self._prefill + + def forward( + self, + attn_metadata: AttentionMetadata, + input_ids: torch.IntTensor | None = None, + position_ids: torch.IntTensor | None = None, + inputs_embeds: torch.FloatTensor | None = None, + lora_params: dict | None = None, + **kwargs, + ) -> torch.Tensor: + """Derive what the first step fixes, then route.""" + self._derive_first_step(attn_metadata) + logger.debug( + f"modeling_v2 Mistral Large 4 NVFP4 attention-DP rank {self.dp_rank}: all_rank_num_tokens " + f"{attn_metadata.all_rank_num_tokens}, num_contexts {attn_metadata.num_contexts}" + ) + return self._select_target(attn_metadata).forward( + attn_metadata, + input_ids=input_ids, + position_ids=position_ids, + inputs_embeds=inputs_embeds, + lora_params=lora_params, + **kwargs, + ) + + +class PrefillTarget: + """The general case: context rows, and possibly generation rows behind them.""" + + # The held context workspace, bf16 [core.ctx_workspace_rows, 576]: allocated on the first + # context step and held from then on (`_ctx_workspace`). + _ws_buf: torch.Tensor | None = None + # The indexer's prefill top-k scratch (`gvr_workspace`, 20 MiB): allocated on the first + # context step that selects and held, used on the main stream only (`_select_ctx`). + _gvr_ws: torch.Tensor | None = None + + def __init__(self, core: MistralLarge4DepModelingV2) -> None: + """Take the per-layer views once, off the real weights. Everything an engine value + fixes is read off `self.core` at the call.""" + self.core = core + # Off unless TRTLLM_MODELING_V2_VALIDATE asks for it. Each target owns its own flag: + # the two `step_args` projections differ, so probing one does not prove the other. + self._contract_pending = step_contract_enabled() + self._layers = _layer_views(core) + self._embed = core.w["embed"] + self._norm0 = core.w["l0_attn_norm"] + + def _check_step_contract(self, md) -> None: + """Opt-in first-forward fail-fast: calling `step_args` reads every metadata field this + target consumes (private trtllm surface). Fixed at engine construction, so once per + target instance, and off in a served engine.""" + if not self._contract_pending: + return + try: + self.step_args(md) + except AttributeError as exc: + raise AssertionError(f"attention metadata surface drifted: {exc}") from exc + self._contract_pending = False + + def step_args(self, md: TrtllmAttentionMetadata) -> dict: + """The common batch state, plus the context rows': their counts and lengths, the + indexer's prefill windows and key gather mappings, and the context requests' KV + offsets the context workspace is laid out by and their host prefix sums.""" + step = _step_common(md, self.core.indexer_layers) + ctx = md.num_contexts + # The engine splits the indexer's prefill queries across ranks only without attention + # DP; under it each rank's chunks cover every context row of its own requests, and a + # split would leave the rows outside this rank's share unselected. + assert md.indexer_prefill_q_split is None, ( + f"the indexer's prefill queries are split across ranks " + f"({md.indexer_prefill_q_split}); this target selects every context row itself" + ) + step.update( + num_contexts=ctx, + num_ctx_tokens=md.num_ctx_tokens, + seq_lens=md.seq_lens, + skip_ctx=bool(md.skip_indexer_for_ctx_reqs), + cu_seqlen_ks=md.cu_seqlen_ks, + cu_seqlen_ke=md.cu_seqlen_ke, + indexer_prefill_chunks=md.indexer_prefill_chunks, + num_ctx_kv_tokens=md.num_ctx_kv_tokens, + slot_mapping_fp8_fullkv=md.slot_mapping_fp8_fullkv, + slot_mapping_scale_fullkv=md.slot_mapping_scale_fullkv, + **_ctx_kv_offsets(md, ctx), + **_ctx_host_offsets(md, ctx), + ) + return step + + # ---- one layer's pieces, context and generation rows ---- + + def _linear_fp8(self, x, w, w_s): + """`x @ w.T`: the bf16 activation quantized 1x128 with power-of-two scales, then the + DeepGEMM block-scale GEMM against the load-time UE8M0 weight scales.""" + codes, scale = fp8_quantize_1x128_packed_ue8m0(x) + return fp8_prequantized_swap_ab_gemm(codes, scale, w, w_s, torch.bfloat16, True) + + def _latent(self, x, L, nope_lat): + """`[RMSNorm(c_kv) | k_pe]`, 576 wide: a RoPE owner's kv_a row, normalized in place; a + NoPE owner's 512-wide latent normalized into `nope_lat`, whose rope key stays zero.""" + core = self.core + kv = cublas_mm(x, L.wkv_a) + if core.unfused_glue: + c_kv = flashinfer_rmsnorm(kv[:, : core.kv_lora], L.kv_norm, core.eps) + if L.rope: + k_pe = kv[:, core.kv_lora :] + else: + k_pe = torch.zeros([kv.shape[0], core.rope_dim], dtype=kv.dtype, device=kv.device) + return torch.cat([c_kv, k_pe], dim=-1) + if L.rope: + c_kv = kv[:, : core.kv_lora] + flashinfer_rmsnorm(c_kv, L.kv_norm, core.eps, out=c_kv) + return kv + flashinfer_rmsnorm(kv, L.kv_norm, core.eps, out=nope_lat[:, : core.kv_lora]) + return nope_lat + + def _indexer(self, step, x, pos, L, topk_buf, tc: int, skip_ctx: bool, skip_gen: bool): + """Append this step's keys to the layer's key pool; select `index_topk` positions per + row into `topk_buf` for the rows whose phase the engine did not skip.""" + core = self.core + ix = L.indexer + rows = x.shape[0] + table = core._rope.indexer if ix.rope else core._rope.identity + kw = cublas_mm(x, ix.wk_wp) + k = kw[:, : core.idx_dim].contiguous() + k_pool = step["ix_pools"][L.index] + k_pool = cute_dsl_indexer_k_cache_update( + k, ix.k_norm_w, ix.k_norm_b, pos, table.view(-1, 2, core.rope_dim // 2), k_pool, + step["slot_mapping_fp8"], step["slot_mapping_scale"], rows, + is_neox=False, compiled=core._ix_k[k_pool.is_contiguous()], + # Its range check is a host sync: once, on the first (eager) call. + validate=not core._ix_k_checked, + ) # fmt: skip + core._ix_k_checked = True + if skip_ctx and skip_gen: + return + qn = layer_norm(x, ix.q_norm_w, ix.q_norm_b, core.idx_eps) + q = torch.reshape(self._linear_fp8(qn, ix.wq_b, ix.wq_b_s), [rows, core.idx_heads, -1]) + q_fp8 = torch.empty_like(q, dtype=torch.float8_e4m3fn) + weights = torch.empty([rows, core.idx_heads], dtype=torch.float32, device=x.device) + if core.unfused_glue: + q = ssmax_scaling(q, pos, ix.ssmax_s, ix.ssmax_b) + q_fp8, weights = cute_dsl_indexer_q_side_fused( + q, pos, table, kw[:, core.idx_dim :].float().contiguous(), core.idx_weight_scale, + q_fp8, weights, is_neox=False, compiled=core._ix_q, + ) # fmt: skip + else: + # SSMax, rope, e4m3 and the head weights (read in place out of `kw`) in one launch. + q_fp8, weights = ssmax_indexer_q_side_fused( + q, pos, ix.ssmax_s, ix.ssmax_b, table, kw[:, core.idx_dim :], + core.idx_weight_scale, q_fp8, weights, + ) # fmt: skip + if tc and not skip_ctx: + self._select_ctx(step, k_pool, q_fp8, weights, topk_buf, tc) + gen = rows - tc + if gen and not skip_gen: + ctx = step["num_contexts"] + num_gens = step["num_seqs"] - ctx + assert gen % num_gens == 0, (gen, num_gens) + next_n = gen // num_gens + kv_lens = step["kv_lens_cuda"][ctx : ctx + num_gens] + logits = cute_dsl_fp8_paged_mqa_logits( + torch.reshape(q_fp8[tc:], [num_gens, next_n, core.idx_heads, core.idx_dim]), + k_pool, + weights[tc:], + kv_lens, + step["indexer_k_cache_block_offsets"][ctx : ctx + num_gens], + step["scheduler_metadata_buffer"], + core._max_ctx_len, + ) + if next_n != 1 or core._ix_topk is None: + indexer_topk_decode( + logits, kv_lens, topk_buf[tc:], next_n, core.idx_topk, 1, + step["radix_aux_indices"], step["radix_aux_logits"], + ) # fmt: skip + else: + cute_dsl_radix_topk_decode( + logits, kv_lens, topk_buf[tc:], next_n=1, compiled=core._ix_topk + ) + + def _ix_plan(self, step, tc: int, heads: int) -> list[SimpleNamespace]: + """The step's indexer scoring plan, the same for each of its indexer layers: per + scoring block of each engine prefill chunk, its rows, its keys in the step's full-KV + mappings and its query tiles of at most the engine's `prefill_logits_budget` logits, + each tile with its rows' windows and the split-KV schedule of its logits call + (`mqa_logits_schedule`). A chunk past max_seq_len keys is cut at its requests + (`_prefill_blocks`): a block that starts past its chunk's first key takes its rows' + windows rebased to its own first key. A block with no key has no tile. + + Built on the step's first indexer layer and kept in `step`: the windows, the key + counts and so the schedules depend on the step alone, and the layers differ only in + the keys and queries they score.""" + core = self.core + chunks = step["indexer_prefill_chunks"] + if chunks is None: + # One chunk over every context row: the engine plans chunks only past its chunk + # size, or under KV reuse or chunked prefill. + chunks = [ + SimpleNamespace( + token_start=0, + token_end=tc, + k_token_start=0, + k_token_end=step["num_ctx_kv_tokens"], + cu_seqlen_ks=step["cu_seqlen_ks"][:tc], + cu_seqlen_ke=step["cu_seqlen_ke"][:tc], + ) + ] + plan = [] + for chunk in chunks: + blocks = _prefill_blocks( + chunk, step["host_cu_ctx_new"], step["host_cu_ctx_kv"], core._max_ctx_len + ) + for token_start, token_end, k_token_start, n_k in blocks: + if token_end <= token_start: + continue + tiles = [] + if n_k > 0: + shift = k_token_start - chunk.k_token_start + tile = max(1, core.prefill_logits_budget // n_k) + for lo in range(token_start, token_end, tile): + hi = min(lo + tile, token_end) + # The windows are the chunk's own, indexed from its first row and key. + ks = chunk.cu_seqlen_ks[lo - chunk.token_start : hi - chunk.token_start] + ke = chunk.cu_seqlen_ke[lo - chunk.token_start : hi - chunk.token_start] + if shift: + ks, ke = torch.sub(ks, shift), torch.sub(ke, shift) + tiles.append((lo, hi, ks, ke, mqa_logits_schedule(ks, ke, n_k, heads))) + plan.append( + SimpleNamespace( + token_start=token_start, + token_end=token_end, + k_token_start=k_token_start, + n_k=n_k, + tiles=tiles, + ) + ) + return plan + + def _select_ctx(self, step, k_pool, q_fp8, weights, topk_buf, tc: int) -> None: + """Select `index_topk` positions for each of the `tc` context rows into `topk_buf`, + window-local, along the step's scoring plan (`_ix_plan`): per block, gather its keys + out of the layer's key pool through the step's full-KV mappings, then per query tile + score its rows against them with the tile's split-KV schedule, which spreads each + query block's keys over the SMs (a 64-row tile at 1048576 keys keeps every SM busy, + the grid-stride call 16), and select over each row's window with the self-sampling + GVR top-k, one streaming pass per row. Both answer as the grid-stride scorer and the + radix top-k: the logits bitwise, the selection the same set up to the keys tied at a + row's 2048th score. A block with no key leaves its rows' slots empty (-1), and each + tile's logits are freed before the next tile's are allocated.""" + core = self.core + plan = step.get("ix_plan") + if plan is None: + plan = step["ix_plan"] = self._ix_plan(step, tc, q_fp8.shape[1]) + if self._gvr_ws is None: + # On the first context step, which the KV-capacity estimate's profiling forward + # is: the pool is sized with the top-k's scratch held, and every top-k launch this + # engine's tiles reach compiles here rather than on a served step (each tile holds + # at most the logits budget, one block at most max_seq_len keys). + self._gvr_ws = gvr_workspace(q_fp8.device) + t0 = time.monotonic() + calls = gvr_warmup( + core.idx_topk, core._max_ctx_len, core.prefill_logits_budget, q_fp8.device, + self._gvr_ws, + ) # fmt: skip + logger.info( + f"modeling_v2 Mistral Large 4 NVFP4: indexer prefill top-k compiled for this " + f"engine's tiles ({calls} warm calls, {time.monotonic() - t0:.1f} s)" + ) + for block in plan: + if block.n_k <= 0: + # No key to select from: every slot of the block's rows is empty. + topk_buf[block.token_start : block.token_end].fill_(-1) + continue + ck, cs = indexer_k_cache_gather_op( + k_pool, step["slot_mapping_fp8_fullkv"], step["slot_mapping_scale_fullkv"], + block.k_token_start, block.n_k, core.idx_dim, + ) # fmt: skip + cs = torch.reshape(cs, [-1]) + for lo, hi, ks, ke, schedule in block.tiles: + logits = fp8_mqa_logits( + q_fp8[lo:hi], (ck, cs), weights[lo:hi], ks, ke, schedule_meta=schedule + ) + cute_dsl_gvr_topk_prefill( + logits, ks, ke, topk_buf[lo:hi], core.idx_topk, self._gvr_ws + ) + # Freed before the next tile's is allocated: one tile is live at a time. + del logits + + def _attn_ctx(self, step, L, fused_q, lat_out, req_idx, topk_buf, ctx): + """Sparse MLA over the context rows into `lat_out`, on the step's path (`ctx`). + + Workspace path: FlashMLA's bf16 sparse prefill over the context requests' latent rows + of the source layer, dequantized out of the pool (this step's rows included, so every + row is read FP8-rounded) into the held workspace's first rows. A shared source (layer + 35) is gathered once into the rows the step keeps beside them, when it has them, and + read there by every layer that shares it; otherwise each of those layers gathers it + again into the first rows. Decode path, a step past the held workspace: the selections + as pool slots, then FlashMLA's sparse fp8 decode over the pool with every context row a + one-row sequence, in the step's sub-batches.""" + core = self.core + tc = fused_q.shape[0] + src = L.kv_source + if ctx.ws_buf is None: + slots = convert_req_index_to_global( + req_idx[:tc], step["block_table"], topk_buf[:tc], step["tokens_per_block"], + core.idx_topk, core._stride_factor, core._pool_row[src], + ) # fmt: skip + for lo, hi in ctx.sub_batches: + # Past 11622 rows FlashMLA's scheduler exits the process instead of raising. + assert hi - lo <= _CTX_DECODE_ROWS, (lo, hi) + flash_mla_sparse_decode( + fused_q[lo:hi], step["latent_kv"], slots[lo:hi], ctx.sched[hi - lo], + core.softmax_scale, lat_out[lo:hi], + ) # fmt: skip + return + ws = ctx.ws.get(src) + if ws is None: + keep = core.shared_source[src] and ctx.kept is not None + ws = ds_mla_dequant_gather( + step["latent_pool"], core._pool_row[src], ctx.kv_req, ctx.kv_pos, + step["block_table"], ctx.kept if keep else ctx.ws_buf, + ) # fmt: skip + if keep: + ctx.ws[src] = ws + # Request-local selections to workspace rows: each request's run starts at its + # cumulative KV offset. + sel = topk_buf[:tc] + idx = torch.where(sel >= 0, sel + ctx.ws_off[req_idx[:tc]][:, None], sel) + flash_mla_sparse_prefill(fused_q, ws, idx.to(torch.int32), core.softmax_scale, lat_out) + + def _attn_gen(self, step, L, fused_q, lat_out, slots, sched): + """Sparse MLA over the generation rows behind the context rows into `lat_out`: FlashMLA's + sparse fp8 decode over the pool, one scheduler shared by the step's layers.""" + flash_mla_sparse_decode( + fused_q, step["latent_kv"], slots, sched, self.core.softmax_scale, lat_out + ) + + def _dense(self, o, L): + """The SwiGLU MLP on this rank's own rows, whole: layer 0's, or a MoE layer's shared + expert.""" + act = flashinfer_silu_and_mul(self._linear_fp8(o, L.mlp_gu, L.mlp_gu_s)) + return self._linear_fp8(act, L.mlp_dn, L.mlp_dn_s) + + def _moe(self, o, L, padded: int, pad_rows: torch.Tensor | None = None): + """A MoE layer's MLP on this rank's own rows: the shared expert on its own rows, on the + side stream, beside the routed experts over every rank's gathered rows. + + The rows are zero-padded to `padded` (the largest rank's count) and gathered before the + router, so every rank routes the identical `world_size * padded` rows. Router, routing, + NVFP4 quantization and this rank's expert window (32 experts at 4 ranks, 16 at 8) run + over them in chunks of at most `expert_call_tokens`, each writing its rows of one + output, which the reduce-scatter sums over the windows and splits back to `padded` rows + per rank. The padding rows (`pad_rows`, `_padding_rows`) take expert id -1 after the + routing, so no rank's window holds them and no expert computes them; their output rows + come back +0.0 and are dropped. Without `pad_rows` they route to experts 0..3 (an exact + tie at +0.0) and contribute zero.""" + core = self.core + m = L.moe + rows = o.shape[0] + side = core._side_stream + main = torch.cuda.current_stream() + side.wait_stream(main) + with torch.cuda.stream(side): + shared = self._dense(o, L) + o_pad = o if rows == padded else nn.functional.pad(o, [0, 0, 0, padded - rows]) + o_all = allgather(o_pad, None, core.dp_group, core.dp_pg) + total = o_all.shape[0] + call = core.expert_call_tokens + routed_all = None if total <= call else torch.empty_like(o_all) + for lo in range(0, total, call): + hi = min(lo + call, total) + x = o_all[lo:hi] + logits = cublas_mm(x, m.router, None, torch.float32) + ids, weights = renorm_moe_routing_op(logits, core.topk, torch.bfloat16) + if pad_rows is not None: + # Every routing kernel of the pre-routed runner drops a slot whose id lies + # outside the rank's window (the small-batch kernels test `expertIdx > -1`, the + # others a signed `id - first_expert >= 0`), so -1 is local to no rank: the + # padding rows reach no expert's GEMMs, and the finalize writes a row with no + # local slot as +0.0. The router, the routing and the quantization still run + # over every row, so the call's shapes, tile and tactic are the same. + ids.masked_fill_(pad_rows[lo:hi, None], -1) + # One NVFP4 quantization under the layer's static scale serves every expert on + # every rank: the windows' partials sum. + xq, xs = fp4_quantize(x, m.gscale, _NVFP4_SF_VEC, False, _NVFP4_SF_LINEAR) + part = fp4_block_scale_moe_runner( + None, None, xq, xs.view(torch.float8_e4m3fn), m.fc1_w, m.fc1_s, None, None, + None, None, m.fc2_w, m.fc2_s, None, m.out1, m.out1_gate, m.out2, + core.num_experts, core.topk, None, None, core.moe_inter, core.expert_offset, + core.local_experts, None, _MOE_ROUTING_METHOD, True, _MOE_ACT_SWIGLU, + topk_weights=weights, topk_ids=ids, tune_max_num_tokens=core.expert_tune_tokens, + )[0] # fmt: skip + if routed_all is None: + routed_all = part + else: + routed_all[lo:hi].copy_(part) + routed = reducescatter(routed_all, None, core.dp_group, core.dp_pg)[:rows] + main.wait_stream(side) + return torch.add(routed, shared) + + def _topk_buf(self, step, rows: int, tc: int, skip_ctx: bool, skip_gen: bool): + """Where this step's selections land, request-local positions. A phase the engine + skipped (every sequence within index_topk) takes the dense causal index it wrote.""" + if skip_ctx and skip_gen: + return step["topk_indices_buffer"][:rows] + buf = torch.empty( + [rows, self.core.idx_topk], dtype=torch.int32, device=step["block_table"].device + ) + if skip_ctx and tc: + buf[:tc].copy_(step["topk_indices_buffer"][:tc]) + if skip_gen and rows > tc: + buf[tc:].copy_(step["topk_indices_buffer"][tc:rows]) + return buf + + def forward( + self, + attn_metadata: AttentionMetadata, + input_ids: torch.IntTensor | None = None, + position_ids: torch.IntTensor | None = None, + inputs_embeds: torch.FloatTensor | None = None, + lora_params: dict | None = None, + **kwargs, + ) -> torch.Tensor: + core = self.core + self._check_step_contract(attn_metadata) + step = self.step_args(attn_metadata) + h = ( + nn.functional.embedding(input_ids, self._embed) + if inputs_embeds is None + else inputs_embeds + ) + rows = h.shape[0] + dev = h.device + # The slot mappings and windows follow the prepared token order; a padded input + # would scatter keys into stale slots. + assert rows == step["num_tokens"], (rows, step["num_tokens"]) + padded = _padded_rows(core, step, rows) + # The gathered rows that are another rank's padding (or this one's), once per step. + pad_rows = _padding_rows(step["all_rank_num_tokens"], padded, dev) + tc = step["num_ctx_tokens"] + gen = rows - tc + skip_ctx = step["skip_ctx"] + skip_gen = step["skip_gen"] if gen else True + pos = torch.reshape(position_ids, [-1]) + req_idx = step["req_idx_per_token"][:rows] + topk_buf = self._topk_buf(step, rows, tc, skip_ctx, skip_gen) + scratch = self._scratch(rows, h.dtype, dev) + ctx = self._ctx_workspace(step, tc, dev) if tc else None + + x = flashinfer_rmsnorm(h, self._norm0, core.eps) + residual = h + spec_metadata = kwargs.get("spec_metadata") + for layer_idx, L in enumerate(self._layers): + attn_out = self._attention( + step, L, x, pos, req_idx, topk_buf, scratch, ctx, tc, skip_ctx, skip_gen + ) + residual = torch.add(residual, self._linear_fp8(attn_out, L.wo, L.wo_s)) + o = flashinfer_rmsnorm(residual, L.ffn_norm, core.eps) + mlp = self._dense(o, L) if L.moe is None else self._moe(o, L, padded, pad_rows) + residual = torch.add(residual, mlp) + x = flashinfer_rmsnorm(residual, L.next_norm, core.eps) + if spec_metadata is not None: + spec_metadata.maybe_capture_hidden_states(layer_idx, residual) + return x + + def _scratch(self, rows: int, dtype, dev) -> SimpleNamespace: + """Per-step scratch every layer rewrites in turn: the attention output, the absorbed + query at the kernels' 64 heads (this rank's first, the rest zero for the step), the + latent attention output, a NoPE owner's 576-wide latent rows (the rope key zero for + the step), and the generation rows' FlashMLA scheduler, filled by the step's first + decode call and shared by the rest.""" + core = self.core + bf16 = dict(dtype=dtype, device=dev) + return SimpleNamespace( + attn_out=torch.empty([rows, core.heads * core.v_dim], **bf16), + fused_q=torch.zeros([rows, _FLASH_MLA_HEADS, core.lat_dim], **bf16), + lat_out=torch.empty([rows, core.heads, core.kv_lora], **bf16), + nope_lat=None if core.unfused_glue else torch.zeros([rows, core.lat_dim], **bf16), + sched=get_mla_metadata()[0], + ) + + def _ctx_workspace(self, step, tc: int, dev) -> SimpleNamespace: + """The context rows' path for the step, chosen once on this rank's host ints from its + context KV: every context request's cached prefix and new tokens (the engine's + `num_ctx_kv_tokens` counts the new ones only). Under attention DP the step holds this + rank's own requests only and the attention has no collective, so each rank takes its + own path; ranks on different paths meet again only at the expert path's collectives. + + The held workspace, bf16 `[core.ctx_workspace_rows, 576]` (`_ctx_workspace_rows`), is + allocated on the first context step, whichever path that step takes, and held. The + engine's KV-capacity estimate runs context steps as its profiling forward and sizes the + pool after their peak memory (`KvCacheCreator.configure_kv_cache_capacity`), so the pool + is sized with the workspace held. A generation-only engine runs no context step and + holds none. + + The workspace path, a step whose context KV fits the held rows: request and position + per workspace row as the dequantizing gather and the prefill kernel address them, each + request's offset, the held workspace's first `n_kv` rows, which every layer's gather + rewrites, and, when the context KV fits twice, the next `n_kv` rows (`kept`), where a + shared owner's gather stays for the step (`ws`). + + The decode path, a step past the held rows, with no workspace: the context rows' + sub-batches of at most `_CTX_DECODE_ROWS`, and one FlashMLA scheduler per sub-batch + size, filled by its first call and shared by the step's layers.""" + # Under KV reuse a context request attends over its cached prefix and its new tokens. + n_kv = step["ctx_cached_tokens"] + step["num_ctx_tokens"] + if self._ws_buf is None: + # On the first context step: the KV-capacity estimate's profiling forward is one, + # so the peak it measures, which sizes the pool, holds this buffer. + self._ws_buf = torch.empty( + [self.core.ctx_workspace_rows, self.core.lat_dim], dtype=torch.bfloat16, device=dev + ) + logger.info( + f"modeling_v2 Mistral Large 4 NVFP4: context workspace held, {self._ws_buf.shape[0]} " + f"rows x {self.core.lat_dim} bf16 ({self._ws_buf.nbytes / 2**20:.0f} MiB)" + ) + held = self._ws_buf.shape[0] + if n_kv > held: + bounds = [(lo, min(lo + _CTX_DECODE_ROWS, tc)) for lo in range(0, tc, _CTX_DECODE_ROWS)] + logger.debug( + f"modeling_v2 Mistral Large 4 NVFP4: context KV {n_kv} past the held {held} rows: " + f"decode path, {len(bounds)} sub-batches" + ) + return SimpleNamespace( + ws_buf=None, + sub_batches=bounds, + sched={rows: get_mla_metadata()[0] for rows in {hi - lo for lo, hi in bounds}}, + ) + ctx = step["num_contexts"] + cu = step["cu_ctx_kv"] + lens = cu[1:] - cu[:-1] + kv_req = torch.repeat_interleave( + torch.arange(ctx, dtype=torch.int32, device=dev), lens, output_size=n_kv + ) + kv_pos = (torch.arange(n_kv, device=dev) - cu[:-1][kv_req]).to(torch.int32) + keep = 2 * n_kv <= held + logger.debug( + f"modeling_v2 Mistral Large 4 NVFP4: context KV {n_kv} of the held {held} rows: " + f"workspace path, the shared source {'kept' if keep else 'gathered again'}" + ) + return SimpleNamespace( + kv_req=kv_req, + kv_pos=kv_pos, + ws_off=cu[:-1], + ws_buf=self._ws_buf[:n_kv], + kept=self._ws_buf[n_kv : 2 * n_kv] if keep else None, + ws={}, + ) + + def _attention(self, step, L, x, pos, req_idx, topk_buf, s, ctx, tc, skip_ctx, skip_gen): + """One layer's attention block, both phases, into `s.attn_out` (returned): the query + with SSMax, the indexer's selection, the absorbed query, the owner's rope and + fp8_ds_mla append, sparse MLA over the context and the generation rows, and the V + up-projection.""" + core = self.core + rows = x.shape[0] + q = self._linear_fp8(x, L.wq, L.wq_s) + q3 = torch.reshape(q, [rows, core.heads, core.qk_dim]) + fq = s.fused_q[:, : core.heads] + if core.unfused_glue: + q3 = ssmax_scaling(q3, pos, L.ssmax_s, L.ssmax_b) + if L.rope: + fq[..., core.kv_lora :].copy_(q3[..., L.nope :]) + else: + fq[..., core.kv_lora :].zero_() + else: + # SSMax in place; its rope half (a NoPE layer's zero) into the absorbed query's tail. + ssmax_scaling_fused_q( + q3, pos, L.ssmax_s, L.ssmax_b, q3, fq[..., core.kv_lora :], L.rope + ) + if L.indexer is not None: + self._indexer(step, x, pos, L, topk_buf, tc, skip_ctx, skip_gen) + bmm_out( + torch.transpose(q3[..., : L.nope], 0, 1), L.k_b, + torch.transpose(fq[..., : core.kv_lora], 0, 1), + ) # fmt: skip + if L.owner: + # Every row's latent, context and generation, before either phase reads the pool; + # the query tail is roped in the same call. + ds_mla_rope_append( + self._latent(x, L, s.nope_lat), fq, pos, core._rope.indexer, req_idx, + step["block_table"], step["latent_pool"], core._pool_row[L.index], L.rope, + ) # fmt: skip + if tc: + self._attn_ctx(step, L, s.fused_q[:tc], s.lat_out[:tc], req_idx, topk_buf, ctx) + if rows > tc: + slots = convert_req_index_to_global( + req_idx[tc:], step["block_table"], topk_buf[tc:], step["tokens_per_block"], + core.idx_topk, core._stride_factor, core._pool_row[L.kv_source], + ) # fmt: skip + self._attn_gen(step, L, s.fused_q[tc:], s.lat_out[tc:], slots, s.sched) + bmm_out( + torch.transpose(s.lat_out, 0, 1), + L.v_b_t, + torch.transpose(torch.reshape(s.attn_out, [rows, core.heads, core.v_dim]), 0, 1), + ) + return s.attn_out + + def dspark_forward( + self, + draft, + noise_embedding, + query_positions, + num_ctx_per_req, + ctx_k_cache, + ctx_v_cache, + ctx_cache_batch_idx, + ctx_kv_cache=None, + ctx_page_table=None, + block_placement=None, + ): + # Only the worker's fused paged-latent preparation places the block + # (block_placement), and a drafter with embed_draft_block does not take + # it; this decode reads its context window through ctx_page_table. + del block_placement + draft._build_fused_kv_buffers() + draft._ensure_rope() + core = self.core + batch, block, hidden = noise_embedding.shape + rows = batch * block + # Gather a bounded window, not every page in the configured context. + # Invalid padding slots are masked even if their pages are allocated. + lengths = num_ctx_per_req[:batch].long() + offsets = torch.arange(draft.window, device=noise_embedding.device) + positions = (lengths - draft.window).clamp_min(0)[:, None] + offsets[None, :] + valid = positions < lengths[:, None] + slots = ctx_cache_batch_idx.long() + if ctx_page_table is not None: + page_size = ctx_kv_cache[0].shape[-2] + table = ctx_page_table.index_select(0, slots).long() + safe_positions = positions.clamp(max=table.shape[1] * page_size - 1) + pages = table.gather(1, safe_positions // page_size).clamp_min(0) + page_offsets = safe_positions % page_size + else: + safe_positions = positions.clamp(max=ctx_k_cache.shape[2] - 1) + key_valid = torch.cat( + (valid, torch.ones(batch, block, dtype=torch.bool, device=valid.device)), dim=1 + ) + query = torch.empty( + batch, + block, + core.heads, + core.lat_dim, + dtype=noise_embedding.dtype, + device=noise_embedding.device, + ) + values = torch.empty( + rows, core.heads, core.v_dim, dtype=noise_embedding.dtype, device=noise_embedding.device + ) + residual = noise_embedding.reshape(rows, hidden).clone() + x = flashinfer_rmsnorm(residual, draft._layers[0].attn_norm, core.eps) + for i, layer in enumerate(draft._layers): + q = cublas_mm(x, layer.q, None, None).view(batch, block, core.heads, core.qk_dim) + bmm_out( + q[..., : core.nope].reshape(rows, core.heads, core.nope).transpose(0, 1), + layer.k_b, + query[..., : core.kv_lora].view(rows, core.heads, core.kv_lora).transpose(0, 1), + ) + query[..., core.kv_lora :].copy_(draft._rotate(q[..., core.nope :], query_positions)) + kv = cublas_mm(x, layer.kv_a, None, None) + content = flashinfer_rmsnorm( + kv[:, : core.kv_lora].contiguous(), layer.kv_norm, core.eps + ) + block_kv = torch.cat( + ( + content.view(batch, block, core.kv_lora), + draft._rotate( + kv[:, core.kv_lora :].view(batch, block, core.rope_dim), query_positions + ), + ), + dim=-1, + ) + if ctx_page_table is not None: + context = ctx_kv_cache[i][:, 0, 0][pages, page_offsets] + else: + context = ctx_k_cache[slots[:, None], i, safe_positions, 0] + # Zero masked values as well as scores: 0 * an uninitialized + # NaN in the latent pool must not contaminate the weighted sum. + context = context.masked_fill(~valid.unsqueeze(-1), 0) + keys = torch.cat((context, block_kv), dim=1) + scores = torch.einsum("bshd,btd->bsht", query, keys).float() * core.softmax_scale + scores = scores.masked_fill(~key_valid[:, None, None, :], float("-inf")) + probabilities = torch.softmax(scores, dim=-1).to(keys.dtype) + latent = torch.einsum("bsht,btc->bshc", probabilities, keys[..., : core.kv_lora]) + bmm_out( + latent.reshape(rows, core.heads, core.kv_lora).transpose(0, 1), + layer.v_b.transpose(1, 2), + values.transpose(0, 1), + ) + partial = cublas_mm(values.reshape(rows, -1), layer.out, None, None) + residual = residual + partial + o = flashinfer_rmsnorm(residual, layer.ffn_norm, core.eps) + gate = cublas_mm(o, layer.gate, None, None) + up = cublas_mm(o, layer.up, None, None) + activated = flashinfer_silu_and_mul(torch.cat((gate, up), dim=-1)) + mlp = cublas_mm(activated, layer.down, None, None) + residual = residual + mlp + x = flashinfer_rmsnorm(residual, layer.next_norm, core.eps) + return x + + +class DecodeTarget: + """The specialization: routed to only when there are no context rows. One query row per + sequence (no speculative decoding); CUDA-graph captured. Its MoE layers issue the same + collectives as the prefill target's, at the same sizes: a peer rank may be prefilling.""" + + def __init__(self, core: MistralLarge4DepModelingV2) -> None: + """Take the per-layer views once, off the real weights. Everything an engine value + fixes is read off `self.core` at the call.""" + self.core = core + self._contract_pending = step_contract_enabled() + self._layers = _layer_views(core) + self._embed = core.w["embed"] + self._norm0 = core.w["l0_attn_norm"] + + def _check_step_contract(self, md) -> None: + """Opt-in first-forward fail-fast: calling `step_args` reads every metadata field this + target consumes (private trtllm surface). Fixed at engine construction, so once per + target instance, and off in a served engine.""" + if not self._contract_pending: + return + try: + self.step_args(md) + except AttributeError as exc: + raise AssertionError(f"attention metadata surface drifted: {exc}") from exc + self._contract_pending = False + + def step_args(self, md: TrtllmAttentionMetadata) -> dict: + """The common batch state. States the two context counts instead of reading them + back: this target runs only when there are no context rows.""" + step = _step_common(md, self.core.indexer_layers) + step.update(num_contexts=0, num_ctx_tokens=0) + return step + + def _linear_fp8(self, x, w, w_s): + """`x @ w.T`: the bf16 activation quantized 1x128 with power-of-two scales, then the + DeepGEMM block-scale GEMM against the load-time UE8M0 weight scales.""" + codes, scale = fp8_quantize_1x128_packed_ue8m0(x) + return fp8_prequantized_swap_ab_gemm(codes, scale, w, w_s, torch.bfloat16, True) + + def _latent(self, x, L, nope_lat): + """`[RMSNorm(c_kv) | k_pe]`, 576 wide: a RoPE owner's kv_a row, normalized in place; a + NoPE owner's 512-wide latent normalized into `nope_lat`, whose rope key stays zero.""" + core = self.core + kv = cublas_mm(x, L.wkv_a) + if core.unfused_glue: + c_kv = flashinfer_rmsnorm(kv[:, : core.kv_lora], L.kv_norm, core.eps) + if L.rope: + k_pe = kv[:, core.kv_lora :] + else: + k_pe = torch.zeros([kv.shape[0], core.rope_dim], dtype=kv.dtype, device=kv.device) + return torch.cat([c_kv, k_pe], dim=-1) + if L.rope: + c_kv = kv[:, : core.kv_lora] + flashinfer_rmsnorm(c_kv, L.kv_norm, core.eps, out=c_kv) + return kv + flashinfer_rmsnorm(kv, L.kv_norm, core.eps, out=nope_lat[:, : core.kv_lora]) + return nope_lat + + def _indexer(self, step, x, pos, L, topk_buf, skip_gen: bool): + """Append this step's keys to the layer's key pool; unless the engine skipped the + indexer (every sequence within index_topk), score every sequence's paged keys and + select `index_topk` positions per row into `topk_buf`.""" + core = self.core + ix = L.indexer + rows = x.shape[0] + table = core._rope.indexer if ix.rope else core._rope.identity + kw = cublas_mm(x, ix.wk_wp) + k = kw[:, : core.idx_dim].contiguous() + k_pool = step["ix_pools"][L.index] + k_pool = cute_dsl_indexer_k_cache_update( + k, ix.k_norm_w, ix.k_norm_b, pos, table.view(-1, 2, core.rope_dim // 2), k_pool, + step["slot_mapping_fp8"], step["slot_mapping_scale"], rows, + is_neox=False, compiled=core._ix_k[k_pool.is_contiguous()], validate=False, + ) # fmt: skip + if skip_gen: + return + qn = layer_norm(x, ix.q_norm_w, ix.q_norm_b, core.idx_eps) + q = torch.reshape(self._linear_fp8(qn, ix.wq_b, ix.wq_b_s), [rows, core.idx_heads, -1]) + q_fp8 = torch.empty_like(q, dtype=torch.float8_e4m3fn) + weights = torch.empty([rows, core.idx_heads], dtype=torch.float32, device=x.device) + if core.unfused_glue: + q = ssmax_scaling(q, pos, ix.ssmax_s, ix.ssmax_b) + q_fp8, weights = cute_dsl_indexer_q_side_fused( + q, pos, table, kw[:, core.idx_dim :].float().contiguous(), core.idx_weight_scale, + q_fp8, weights, is_neox=False, compiled=core._ix_q, + ) # fmt: skip + else: + # SSMax, rope, e4m3 and the head weights (read in place out of `kw`) in one launch. + q_fp8, weights = ssmax_indexer_q_side_fused( + q, pos, ix.ssmax_s, ix.ssmax_b, table, kw[:, core.idx_dim :], + core.idx_weight_scale, q_fp8, weights, + ) # fmt: skip + num_gens = step["num_seqs"] + next_n = rows // num_gens + kv_lens = step["kv_lens_cuda"][:num_gens] + logits = cute_dsl_fp8_paged_mqa_logits( + torch.reshape(q_fp8, [num_gens, next_n, core.idx_heads, core.idx_dim]), + k_pool, + weights, + kv_lens, + step["indexer_k_cache_block_offsets"][:num_gens], + step["scheduler_metadata_buffer"], + core._max_ctx_len, + ) + if next_n != 1 or core._ix_topk is None: + indexer_topk_decode( + logits, kv_lens, topk_buf, next_n, core.idx_topk, 1, step["radix_aux_indices"], + step["radix_aux_logits"], + ) # fmt: skip + else: + cute_dsl_radix_topk_decode(logits, kv_lens, topk_buf, next_n=1, compiled=core._ix_topk) + + def _dense(self, o, L): + """The SwiGLU MLP on this rank's own rows, whole: layer 0's, or a MoE layer's shared + expert.""" + act = flashinfer_silu_and_mul(self._linear_fp8(o, L.mlp_gu, L.mlp_gu_s)) + return self._linear_fp8(act, L.mlp_dn, L.mlp_dn_s) + + def _moe(self, o, L, padded: int, pad_rows: torch.Tensor | None = None): + """A MoE layer's MLP on this rank's own rows: the shared expert on its own rows, on the + side stream, beside the routed experts over every rank's gathered rows. + + The rows are zero-padded to `padded` (the largest rank's count) and gathered before the + router, so every rank routes the identical `world_size * padded` rows. Router, routing, + NVFP4 quantization and this rank's expert window (32 experts at 4 ranks, 16 at 8) run + over them in chunks of at most `expert_call_tokens`, each writing its rows of one + output, which the reduce-scatter sums over the windows and splits back to `padded` rows + per rank. Beside a + prefilling peer this rank's few rows gather up to `world_size * max_num_tokens`, so the + chunking is this target's too. The padding rows (`pad_rows`, `_padding_rows`) take + expert id -1 after the routing, so no rank's window holds them and no expert computes + them; their output rows come back +0.0 and are dropped. Without `pad_rows` they route + to experts 0..3 (an exact tie at +0.0) and contribute zero.""" + core = self.core + m = L.moe + rows = o.shape[0] + side = core._side_stream + main = torch.cuda.current_stream() + side.wait_stream(main) + with torch.cuda.stream(side): + shared = self._dense(o, L) + o_pad = o if rows == padded else nn.functional.pad(o, [0, 0, 0, padded - rows]) + o_all = allgather(o_pad, None, core.dp_group, core.dp_pg) + total = o_all.shape[0] + call = core.expert_call_tokens + routed_all = None if total <= call else torch.empty_like(o_all) + for lo in range(0, total, call): + hi = min(lo + call, total) + x = o_all[lo:hi] + logits = cublas_mm(x, m.router, None, torch.float32) + ids, weights = renorm_moe_routing_op(logits, core.topk, torch.bfloat16) + if pad_rows is not None: + # Every routing kernel of the pre-routed runner drops a slot whose id lies + # outside the rank's window (the small-batch kernels test `expertIdx > -1`, the + # others a signed `id - first_expert >= 0`), so -1 is local to no rank: the + # padding rows reach no expert's GEMMs, and the finalize writes a row with no + # local slot as +0.0. The router, the routing and the quantization still run + # over every row, so the call's shapes, tile and tactic are the same. + ids.masked_fill_(pad_rows[lo:hi, None], -1) + # One NVFP4 quantization under the layer's static scale serves every expert on + # every rank: the windows' partials sum. + xq, xs = fp4_quantize(x, m.gscale, _NVFP4_SF_VEC, False, _NVFP4_SF_LINEAR) + part = fp4_block_scale_moe_runner( + None, None, xq, xs.view(torch.float8_e4m3fn), m.fc1_w, m.fc1_s, None, None, + None, None, m.fc2_w, m.fc2_s, None, m.out1, m.out1_gate, m.out2, + core.num_experts, core.topk, None, None, core.moe_inter, core.expert_offset, + core.local_experts, None, _MOE_ROUTING_METHOD, True, _MOE_ACT_SWIGLU, + topk_weights=weights, topk_ids=ids, tune_max_num_tokens=core.expert_tune_tokens, + )[0] # fmt: skip + if routed_all is None: + routed_all = part + else: + routed_all[lo:hi].copy_(part) + routed = reducescatter(routed_all, None, core.dp_group, core.dp_pg)[:rows] + main.wait_stream(side) + return torch.add(routed, shared) + + def forward( + self, + attn_metadata: AttentionMetadata, + input_ids: torch.IntTensor | None = None, + position_ids: torch.IntTensor | None = None, + inputs_embeds: torch.FloatTensor | None = None, + lora_params: dict | None = None, + **kwargs, + ) -> torch.Tensor: + core = self.core + self._check_step_contract(attn_metadata) + step = self.step_args(attn_metadata) + h = ( + nn.functional.embedding(input_ids, self._embed) + if inputs_embeds is None + else inputs_embeds + ) + rows = h.shape[0] + dev = h.device + assert rows == step["num_tokens"] and rows % step["num_seqs"] == 0, ( + rows, + step["num_tokens"], + step["num_seqs"], + ) + # Every rank's count, frozen into a capture: the engine replays a decode graph only + # when every rank is generation-only at the same padded batch, the counts it captured. + padded = _padded_rows(core, step, rows) + # The gathered rows that are padding, in an eager step beside a prefilling peer. A + # captured step takes none: the engine replays a decode graph only when every rank + # decodes at the captured batch, every count equal, so a graph pads nothing. + pad_rows = ( + None + if torch.cuda.is_current_stream_capturing() + else _padding_rows(step["all_rank_num_tokens"], padded, dev) + ) + # A separate graph is captured for steps whose sequences all fit in index_topk, so + # this host branch is a per-capture constant. + skip_gen = step["skip_gen"] + pos = torch.reshape(position_ids, [-1]) + req_idx = step["req_idx_per_token"][:rows] + if skip_gen: + topk_buf = step["topk_indices_buffer"][:rows] + else: + topk_buf = torch.empty([rows, core.idx_topk], dtype=torch.int32, device=dev) + scratch = self._scratch(rows, h.dtype, dev) + + x = flashinfer_rmsnorm(h, self._norm0, core.eps) + residual = h + spec_metadata = kwargs.get("spec_metadata") + for layer_idx, L in enumerate(self._layers): + attn_out = self._attention(step, L, x, pos, req_idx, topk_buf, scratch, skip_gen) + residual = torch.add(residual, self._linear_fp8(attn_out, L.wo, L.wo_s)) + o = flashinfer_rmsnorm(residual, L.ffn_norm, core.eps) + mlp = self._dense(o, L) if L.moe is None else self._moe(o, L, padded, pad_rows) + residual = torch.add(residual, mlp) + x = flashinfer_rmsnorm(residual, L.next_norm, core.eps) + if spec_metadata is not None: + spec_metadata.maybe_capture_hidden_states(layer_idx, residual) + return x + + def _scratch(self, rows: int, dtype, dev) -> SimpleNamespace: + """Per-step scratch every layer rewrites in turn: the attention output, the absorbed + query at the kernel's 64 heads (this rank's first, the rest zero for the step), a NoPE + owner's 576-wide latent rows (the rope key zero for the step), the latent attention + output the unfused glue copies FlashMLA's live heads into, and the FlashMLA scheduler + the step's first layer fills and the rest share (captured with the step).""" + core = self.core + bf16 = dict(dtype=dtype, device=dev) + return SimpleNamespace( + attn_out=torch.empty([rows, core.heads * core.v_dim], **bf16), + fused_q=torch.zeros([rows, _FLASH_MLA_HEADS, core.lat_dim], **bf16), + nope_lat=None if core.unfused_glue else torch.zeros([rows, core.lat_dim], **bf16), + lat_out=( + torch.empty([rows, core.heads, core.kv_lora], **bf16) if core.unfused_glue else None + ), + sched=get_mla_metadata()[0], + ) + + def _attention(self, step, L, x, pos, req_idx, topk_buf, s, skip_gen: bool): + """One layer's attention block into `s.attn_out` (returned): the query with SSMax, the + indexer's selection, the absorbed query, the owner's rope and fp8_ds_mla append, + FlashMLA sparse decode over the pool, and the V up-projection of the kernel's live + heads, read in place.""" + core = self.core + rows = x.shape[0] + q = self._linear_fp8(x, L.wq, L.wq_s) + q3 = torch.reshape(q, [rows, core.heads, core.qk_dim]) + fq = s.fused_q[:, : core.heads] + if core.unfused_glue: + q3 = ssmax_scaling(q3, pos, L.ssmax_s, L.ssmax_b) + if L.rope: + fq[..., core.kv_lora :].copy_(q3[..., L.nope :]) + else: + fq[..., core.kv_lora :].zero_() + else: + # SSMax in place; its rope half (a NoPE layer's zero) into the absorbed query's tail. + ssmax_scaling_fused_q( + q3, pos, L.ssmax_s, L.ssmax_b, q3, fq[..., core.kv_lora :], L.rope + ) + if L.indexer is not None: + self._indexer(step, x, pos, L, topk_buf, skip_gen) + slots = convert_req_index_to_global( + req_idx, step["block_table"], topk_buf, step["tokens_per_block"], core.idx_topk, + core._stride_factor, core._pool_row[L.kv_source], + ) # fmt: skip + bmm_out( + torch.transpose(q3[..., : L.nope], 0, 1), L.k_b, + torch.transpose(fq[..., : core.kv_lora], 0, 1), + ) # fmt: skip + if L.owner: + ds_mla_rope_append( + self._latent(x, L, s.nope_lat), fq, pos, core._rope.indexer, req_idx, + step["block_table"], step["latent_pool"], core._pool_row[L.index], L.rope, + ) # fmt: skip + if core.unfused_glue: + lat = flash_mla_sparse_decode( + s.fused_q, step["latent_kv"], slots, s.sched, core.softmax_scale, s.lat_out + ) + else: + lat = flash_mla_sparse_decode( + s.fused_q, step["latent_kv"], slots, s.sched, core.softmax_scale, None + )[:, : core.heads] + bmm_out( + torch.transpose(lat, 0, 1), + L.v_b_t, + torch.transpose(torch.reshape(s.attn_out, [rows, core.heads, core.v_dim]), 0, 1), + ) + return s.attn_out + + def dspark_forward( + self, + draft, + noise_embedding, + query_positions, + num_ctx_per_req, + ctx_k_cache, + ctx_v_cache, + ctx_cache_batch_idx, + ctx_kv_cache=None, + ctx_page_table=None, + block_placement=None, + ): + # Only the worker's fused paged-latent preparation places the block + # (block_placement), and a drafter with embed_draft_block does not take + # it; this decode reads its context window through ctx_page_table. + del block_placement + draft._build_fused_kv_buffers() + draft._ensure_rope() + core = self.core + batch, block, hidden = noise_embedding.shape + rows = batch * block + # Gather a bounded window, not every page in the configured context. + # Invalid padding slots are masked even if their pages are allocated. + lengths = num_ctx_per_req[:batch].long() + offsets = torch.arange(draft.window, device=noise_embedding.device) + positions = (lengths - draft.window).clamp_min(0)[:, None] + offsets[None, :] + valid = positions < lengths[:, None] + slots = ctx_cache_batch_idx.long() + if ctx_page_table is not None: + page_size = ctx_kv_cache[0].shape[-2] + table = ctx_page_table.index_select(0, slots).long() + safe_positions = positions.clamp(max=table.shape[1] * page_size - 1) + pages = table.gather(1, safe_positions // page_size).clamp_min(0) + page_offsets = safe_positions % page_size + else: + safe_positions = positions.clamp(max=ctx_k_cache.shape[2] - 1) + key_valid = torch.cat( + (valid, torch.ones(batch, block, dtype=torch.bool, device=valid.device)), dim=1 + ) + query = torch.empty( + batch, + block, + core.heads, + core.lat_dim, + dtype=noise_embedding.dtype, + device=noise_embedding.device, + ) + values = torch.empty( + rows, core.heads, core.v_dim, dtype=noise_embedding.dtype, device=noise_embedding.device + ) + residual = noise_embedding.reshape(rows, hidden).clone() + x = flashinfer_rmsnorm(residual, draft._layers[0].attn_norm, core.eps) + for i, layer in enumerate(draft._layers): + q = cublas_mm(x, layer.q, None, None).view(batch, block, core.heads, core.qk_dim) + bmm_out( + q[..., : core.nope].reshape(rows, core.heads, core.nope).transpose(0, 1), + layer.k_b, + query[..., : core.kv_lora].view(rows, core.heads, core.kv_lora).transpose(0, 1), + ) + query[..., core.kv_lora :].copy_(draft._rotate(q[..., core.nope :], query_positions)) + kv = cublas_mm(x, layer.kv_a, None, None) + content = flashinfer_rmsnorm( + kv[:, : core.kv_lora].contiguous(), layer.kv_norm, core.eps + ) + block_kv = torch.cat( + ( + content.view(batch, block, core.kv_lora), + draft._rotate( + kv[:, core.kv_lora :].view(batch, block, core.rope_dim), query_positions + ), + ), + dim=-1, + ) + if ctx_page_table is not None: + context = ctx_kv_cache[i][:, 0, 0][pages, page_offsets] + else: + context = ctx_k_cache[slots[:, None], i, safe_positions, 0] + # Zero masked values as well as scores: 0 * an uninitialized + # NaN in the latent pool must not contaminate the weighted sum. + context = context.masked_fill(~valid.unsqueeze(-1), 0) + keys = torch.cat((context, block_kv), dim=1) + scores = torch.einsum("bshd,btd->bsht", query, keys).float() * core.softmax_scale + scores = scores.masked_fill(~key_valid[:, None, None, :], float("-inf")) + probabilities = torch.softmax(scores, dim=-1).to(keys.dtype) + latent = torch.einsum("bsht,btc->bshc", probabilities, keys[..., : core.kv_lora]) + bmm_out( + latent.reshape(rows, core.heads, core.kv_lora).transpose(0, 1), + layer.v_b.transpose(1, 2), + values.transpose(0, 1), + ) + partial = cublas_mm(values.reshape(rows, -1), layer.out, None, None) + residual = residual + partial + o = flashinfer_rmsnorm(residual, layer.ffn_norm, core.eps) + gate = cublas_mm(o, layer.gate, None, None) + up = cublas_mm(o, layer.up, None, None) + activated = flashinfer_silu_and_mul(torch.cat((gate, up), dim=-1)) + mlp = cublas_mm(activated, layer.down, None, None) + residual = residual + mlp + x = flashinfer_rmsnorm(residual, layer.next_norm, core.eps) + return x + + +class DraftModel(nn.Module): + """This specialization's BF16 DSpark backbone and recurrent head. + + Runtime owns accepted-token bookkeeping and the paged context pool. The + draft attends the last trained window of captured target features plus + its whole non-causal query block. All parameters live in the core table. + """ + + _uses_worker_attention_backend = False + _paged_ctx_cache = True + _kv_factor = 1 + _dspark_shift_label = True + has_markov_head = True + has_recurrent_markov_head = True + + def __init__(self, shell, draft_config): + super().__init__() + # Keep references without registering the parent as a child module. + object.__setattr__(self, "_shell", shell) + self.model_config = draft_config + self.config = draft_config.pretrained_config + core = shell.model + spec = shell.spec_config + native = spec._draft_model_config + self.block_size = spec.block_size + self.mask_token_id = spec.mask_token_id + self.target_layer_ids = tuple(spec.target_layer_ids) + self.window = int(native["ragged_attention"].split(",")[0]) + self._num_attn_layers = int(native["n_layers"]) + self._num_heads = core.heads + self._num_kv_heads = 1 + self._head_dim = core.lat_dim + self.dflash_attention_backend = "VANILLA" + self._runtime_position_ceiling = None + self._rope = None + self._layers = None + + @property + def core(self): + return self._shell.model + + @property + def lm_head(self): + return self._shell.lm_head + + @property + def markov_w2(self): + return self.core.w["draft_markov_output"] + + def logits_processor(self, hidden, lm_head, attn_metadata, gather): + logits = self._shell.logits_processor(hidden, lm_head, attn_metadata, gather) + # The shared-head logits are soft-capped before the RNN bias is added. + return tanh_softcap(logits, self._shell.softcap) + + def apply_draft_logit_transform(self, logits): + """Return ``logits`` unchanged: :meth:`logits_processor` already soft-caps them. + + The worker calls this on the block logits after ``logits_processor``. + This drafter declares no ``logit_multiplier`` or ``logit_softcap``, so the + cap is applied once, in ``logits_processor``. + """ + return logits + + def _build_fused_kv_buffers(self): + # The worker calls this before sizing its cache. Take views once after + # materialization; context projection is explicit per layer here. + if self._layers is not None: + return + w, core = self.core.w, self.core + self._layers = [] + for i in range(self._num_attn_layers): + prefix = f"l{i}_draft_" + kv_b = w[prefix + "wkv_b"].view(core.heads, core.nope + core.v_dim, core.kv_lora) + self._layers.append( + SimpleNamespace( + attn_norm=w[prefix + "attn_norm"], + q=w[prefix + "wq"].t(), + kv_a=w[prefix + "wkv_a"].t(), + kv_norm=w[prefix + "kv_a_norm"], + k_b=kv_b[:, : core.nope].contiguous(), + v_b=kv_b[:, core.nope :].contiguous(), + out=w[prefix + "wo"].t(), + ffn_norm=w[prefix + "ffn_norm"], + gate=w[prefix + "w1"].t(), + up=w[prefix + "w3"].t(), + down=w[prefix + "w2"].t(), + next_norm=( + w[f"l{i + 1}_draft_attn_norm"] + if i + 1 < self._num_attn_layers + else w["draft_final_norm"] + ), + ) + ) + + def _ensure_rope(self): + cap = self._runtime_position_ceiling or self.core.rope_rows + if self._rope is None or self._rope.rows != cap: + w = self.core.w["draft_final_norm"] + # alpha=0 removes LeRoPE; all draft layers use plain YaRN. + self._rope = _rope_tables( + self.core.model_config.pretrained_config, + torch.zeros(self.core.rope_dim // 2), + cap, + w.dtype, + w.device, + ) + + def _rotate(self, x, positions): + table = self._rope.indexer[positions.long()] + cos, sin = table.chunk(2, dim=-1) + # One head axis for Q, none for the shared latent K. + if x.ndim == positions.ndim + 2: + cos, sin = cos.unsqueeze(-2), sin.unsqueeze(-2) + even, odd = x.float()[..., 0::2], x.float()[..., 1::2] + return ( + torch.stack((even * cos - odd * sin, even * sin + odd * cos), -1) + .flatten(-2) + .to(x.dtype) + ) + + def embed_draft_block(self, anchors, block_size): + # The route pins the trained block width; no mask token is looked up + # in the vocabulary (its sentinel is deliberately out of range). + weight = self.core.w["embed"] + valid = (anchors >= 0) & (anchors < weight.shape[0]) + embedded = nn.functional.embedding(anchors.clamp(0, weight.shape[0] - 1), weight) + embedded = embedded.masked_fill(~valid.unsqueeze(-1), 0) + masks = self.core.w["draft_mask"].unsqueeze(0).expand(anchors.shape[0], -1, -1) + return torch.cat((embedded.unsqueeze(1), masks), dim=1) + + def project_target_hidden(self, hidden): + w = self.core.w + projected = cublas_mm(hidden, w["draft_target_projection"].t(), None, None) + return flashinfer_rmsnorm(projected, w["draft_target_norm"], self.core.eps) + + def precompute_context_kv(self, projected_hidden, positions): + self._build_fused_kv_buffers() + self._ensure_rope() + core = self.core + latent = [] + for layer in self._layers: + kv = cublas_mm(projected_hidden, layer.kv_a, None, None) + content = flashinfer_rmsnorm( + kv[:, : core.kv_lora].contiguous(), layer.kv_norm, core.eps + ) + rope = self._rotate(kv[:, core.kv_lora :], positions.reshape(-1)) + latent.append(torch.cat((content, rope), dim=-1)) + return torch.stack(latent, dim=1).unsqueeze(2), None + + def dflash_forward_with_metadata(self, *, attn_metadata, **kwargs): + # The phase belongs to this engine step, including mixed batches. + return self.core._select_target(attn_metadata).dspark_forward(self, **kwargs) + + def apply_markov_chain_logits(self, base_logits, first_prev_tokens, *, hidden_states): + """The recurrent Markov bias over full-vocab ``[B, K, vocab]`` block logits. + + Each position's greedy token (the argmax of its biased logits) is the + next position's previous token; ``hidden_states`` is the ``[B, K, + hidden]`` block output the logits were projected from. + """ + w = self.core.w + output = w["draft_markov_output"] + previous = first_prev_tokens.long() + state = None + refined = [] + for i in range(base_logits.shape[1]): + weight = w["draft_markov_embedding"] + valid = (previous >= 0) & (previous < weight.shape[0]) + embedding = nn.functional.embedding(previous.clamp(0, weight.shape[0] - 1), weight) + embedding = embedding.masked_fill(~valid.unsqueeze(-1), 0) + if state is None: + state = torch.zeros_like(embedding) + joint = cublas_mm( + torch.cat((state, embedding, hidden_states[:, i]), dim=-1), + w["draft_joint"].t(), + w["draft_joint_bias"], + None, + ) + gate, candidate, emitted = joint.chunk(3, dim=-1) + gate = torch.sigmoid(gate) + state = gate * state + (1 - gate) * torch.tanh(candidate) + bias = cublas_mm(torch.tanh(emitted), output.t(), None, None) + logits = base_logits[:, i] + bias + refined.append(logits) + previous = logits.argmax(dim=-1) + return torch.stack(refined, dim=1) + + +class _VisionBlock(NamedTuple): + """One vision transformer block's weights, as the views the ops read: the GEMM + operands column-major (`.t()` of the stored rows).""" + + attn_norm: torch.Tensor + qkv: torch.Tensor + wo: torch.Tensor + ffn_norm: torch.Tensor + w13: torch.Tensor + w2: torch.Tensor + + +def _vision_step(seg_lens: list[int], device, workspace: torch.Tensor) -> dict: + """The batch state of one vision attention call: one context sequence per image, + no KV cache. Every value is host-known from the images' patch counts.""" + total = sum(seg_lens) + longest = max(seg_lens) + lens_dev = torch.tensor(seg_lens, dtype=torch.int32, device=device) + lens_host = torch.tensor(seg_lens, dtype=torch.int32) + return dict( + workspace=workspace, + sequence_length=lens_dev, + host_past_key_value_lengths=lens_host, + host_total_kv_lens=torch.tensor([total, 0], dtype=torch.int32), + context_lengths=lens_dev, + host_context_lengths=lens_host, + host_request_types=torch.zeros(len(seg_lens), dtype=torch.int32), + max_num_requests=len(seg_lens), + max_context_length=longest, + max_seq_len=longest, + attention_window_size=longest, + num_contexts=len(seg_lens), + num_ctx_tokens=total, + ) + + +def _vision_attention( + x: torch.Tensor, + B: _VisionBlock, + positions: torch.Tensor, + step: dict, + heads: int, + head_dim: int, + theta: float, +) -> torch.Tensor: + """One vision block's attention over the patch rows of every image of the call: + `[wq ; wk ; wv]` as one GEMM, q and k rotated by each patch's (row, col) in its + own image, attention bidirectional within each image, then `wo`.""" + rows = x.shape[0] + width = heads * head_dim + qkv = cublas_mm(x, B.qkv) + q, k = pixtral_rope_2d( + torch.reshape(qkv[:, :width], [rows, heads, head_dim]), + torch.reshape(qkv[:, width : 2 * width], [rows, heads, head_dim]), + positions, + theta, + ) + qkv = torch.cat( + [torch.reshape(q, [rows, width]), torch.reshape(k, [rows, width]), qkv[:, 2 * width :]], -1 + ) + out = torch.empty([rows, width], dtype=x.dtype, device=x.device) + out = thop_attention( + q=qkv, + output=out, + num_heads=heads, + num_kv_heads=heads, + head_size=head_dim, + **step, + **_VISION_ATTN, + ) + return cublas_mm(out, B.wo) + + +def _merge_tiles(x: torch.Tensor, grid_h: int, grid_w: int, merge: int) -> torch.Tensor: + """One image's `[grid_h * grid_w, d]` patch rows -> `[grid_h * grid_w / merge^2, + d * merge^2]`: each merge x merge tile of the grid as one row, tiles row-major (the + order of the image's [IMG] tokens), each row laid out (channel, tile row, tile + column) as the merging layer's input columns are.""" + d = x.shape[-1] + grid = torch.permute(torch.reshape(x, [grid_h, grid_w, d]), [2, 0, 1]) + tiles = nn.functional.unfold(torch.unsqueeze(grid, 0), kernel_size=merge, stride=merge) + return torch.t(tiles[0]) + + +class VisionTower: + """The image input path: a request's pixels -> one row per [IMG] token, in the + language model's hidden width. Run by the shell, in context steps only, never + captured; every rank runs the whole tower (all heads) over its own requests. + + Per image: the 14x14 patch embedding as an unfold and one GEMM, `ln_pre`, the + transformer blocks (RMSNorm, attention with 2-D RoPE, RMSNorm, SwiGLU, each with + its residual), the pre-projector RMSNorm, the 2x2 patch merger and the adapter + (`w_in`, exact GELU, `w_out`). The images of one call are concatenated along the + rows; only the attention needs their boundaries.""" + + def __init__(self, core) -> None: + """Take the tower's weight views and geometry, once, off the real weights.""" + d = _weights.MODEL_WEIGHTS.dims(core) + w = core.w + self.hidden = d.v_hidden + self.heads = d.v_heads + self.head_dim = d.v_head_dim + self.patch = d.v_patch + self.merge = d.v_merge + self.theta = d.v_theta + self.dtype = d.dtype + self.out_hidden = d.hidden + self._validate = step_contract_enabled() + self._rank = core.model_config.mapping.rank + self.patch_w = w["vis_patch"].t() + self.ln_pre = w["vis_ln_pre"] + self.blocks = [ + _VisionBlock( + attn_norm=w[f"v{j}_attn_norm"], + qkv=w[f"v{j}_qkv"].t(), + wo=w[f"v{j}_wo"].t(), + ffn_norm=w[f"v{j}_ffn_norm"], + w13=w[f"v{j}_w13"].t(), + w2=w[f"v{j}_w2"].t(), + ) + for j in range(d.v_layers) + ] + self.proj_norm = w["vis_proj_norm"] + self.merge_w = w["vis_merge"].t() + self.w_in = w["vis_w_in"].t() + self.w_out = w["vis_w_out"].t() + # The prompt token the tower's rows replace, on the host: the executor finds a + # step's image positions with it before anything reaches the device. + self.image_token_ids = torch.tensor([d.image_token_id], dtype=torch.int32) + # The attention op sizes its workspace in place per call and never shrinks it. + self._workspace = torch.empty(0, dtype=torch.int8, device=self.ln_pre.device) + + def _images(self, multimodal_params: list[MultimodalParams]) -> list[tuple]: + """`(pixels [C, H, W], grid rows, grid cols)` of every image, in order. The + mistral-native processor pads the batch and preserves each image's true size.""" + images = [] + unit = self.patch * self.merge + for param in multimodal_params: + image = (param.multimodal_data or {}).get("image") + if image is None: + continue + pixels = image["pixel_values"] + sizes = image.get("image_sizes") + if pixels.ndim != 4 or pixels.shape[1] != 3: + raise ValueError("Image pixels must have shape [N, 3, H, W]") + if sizes is None or len(sizes) != pixels.shape[0]: + raise ValueError("Image sizes must describe every image in the padded batch") + for b in range(pixels.shape[0]): + if len(sizes[b]) != 2: + raise ValueError("Each image size must be [height, width]") + height, width = int(sizes[b][0]), int(sizes[b][1]) + if not (0 < height <= pixels.shape[-2] and 0 < width <= pixels.shape[-1]): + raise ValueError("Image sizes must fit within their padded pixels") + if height % unit or width % unit: + raise ValueError( + f"image {height}x{width} is not a whole number of {unit}-pixel " + "merge tiles; the processor resizes to them" + ) + images.append( + (pixels[b, :, :height, :width], height // self.patch, width // self.patch) + ) + return images + + def forward(self, multimodal_params: list[MultimodalParams]) -> torch.Tensor: + """The rows of every image of `multimodal_params`, request by request, image by + image: `[sum of merged tiles, text hidden]`.""" + images = self._images(multimodal_params) + dev = self.ln_pre.device + if not images: + return torch.empty([0, self.out_hidden], dtype=self.dtype, device=dev) + seg = [gh * gw for _, gh, gw in images] + if self._validate: + logger.info( + "modeling_v2 image forward target=%s rank=%d grids=%s merged_rows=%d", + __name__.rsplit(".", 1)[0], + self._rank, + [(gh, gw) for _, gh, gw in images], + sum(seg) // self.merge**2, + ) + # Each patch as one row, (channel, row, col) within it, patches row-major. + patches = torch.cat( + [ + torch.t( + nn.functional.unfold( + torch.unsqueeze(px.to(device=dev, dtype=self.dtype), 0), + kernel_size=self.patch, + stride=self.patch, + )[0] + ) + for px, _, _ in images + ], + 0, + ) + positions = torch.cat( + [ + torch.reshape( + torch.stack( + torch.meshgrid( + torch.arange(gh, device=dev), + torch.arange(gw, device=dev), + indexing="ij", + ), + -1, + ), + [-1, 2], + ) + for _, gh, gw in images + ], + 0, + ) + step = _vision_step(seg, dev, self._workspace) + + h = flashinfer_rmsnorm(cublas_mm(patches, self.patch_w), self.ln_pre, _VISION_EPS) + x = flashinfer_rmsnorm(h, self.blocks[0].attn_norm, _VISION_EPS) + residual = h + for j, B in enumerate(self.blocks): + attn = _vision_attention(x, B, positions, step, self.heads, self.head_dim, self.theta) + x, residual = flashinfer_fused_add_rmsnorm(attn, residual, B.ffn_norm, _VISION_EPS) + mlp = cublas_mm(flashinfer_silu_and_mul(cublas_mm(x, B.w13)), B.w2) + norm_next = self.blocks[j + 1].attn_norm if j + 1 < len(self.blocks) else self.proj_norm + x, residual = flashinfer_fused_add_rmsnorm(mlp, residual, norm_next, _VISION_EPS) + + # x is the pre-projector-normed tower output. + tiles = torch.cat( + [ + _merge_tiles(xi, gh, gw, self.merge) + for xi, (_, gh, gw) in zip(torch.split(x, seg, 0), images) + ], + 0, + ) + merged = cublas_mm(tiles, self.merge_w) + return cublas_mm(gelu_erf(cublas_mm(merged, self.w_in)), self.w_out) + + +def _image_rows( + vision: Optional[VisionTower], multimodal_params: list[MultimodalParams] +) -> Optional[torch.Tensor]: + """This step's image rows, one per [IMG] position its context chunks cover, in + the order those positions appear; None when the step covers none. + + The tower runs once per request, on the first step that needs it; its rows stay + on the request's multimodal data for the later chunks. A chunk -- under chunked + prefill, or resuming after a reused prefix -- takes rows `[cached, cached + in + chunk)` of its request's, counted by the request's [IMG] cumsum.""" + if vision is None: + if any((p.multimodal_data or {}).get("image") is not None for p in multimodal_params): + raise ValueError( + "an image reached a Mistral Large 4 target whose checkpoint has no vision encoder" + ) + return None + # Filter before encoding and gathering so inactive requests neither run the + # tower nor shift the embedding offsets of active requests in a mixed batch. + multimodal_params = [ + param + for param in multimodal_params + if param.multimodal_runtime is None or param.multimodal_runtime.num_mm_tokens_in_chunk > 0 + ] + embeds = find_input_mm_embeds( + get_multimodal_embeddings(vision.forward, multimodal_params), multimodal_params + ) + if not embeds: + return None + rows = embeds[0] if len(embeds) == 1 else torch.cat(embeds, 0) + return rows if rows.shape[0] else None + + +def _splice_image_rows( + embed: torch.Tensor, + input_ids: torch.Tensor, + rows: torch.Tensor, + mm_token_indices: Optional[torch.Tensor], +) -> torch.Tensor: + """The step's input embeddings with `rows` written over its [IMG] positions, + `mm_token_indices` (which the executor computes on the host from the shell's + `multimodal_token_ids`); every other position keeps its token embedding.""" + if mm_token_indices is None: + raise ValueError("an image step arrived without the executor's mm_token_indices") + if mm_token_indices.shape[0] != rows.shape[0]: + raise ValueError( + f"{mm_token_indices.shape[0]} [IMG] positions in this step but " + f"{rows.shape[0]} image rows for them" + ) + h = nn.functional.embedding(input_ids, embed) + return h.index_copy_(0, mm_token_indices, rows.to(h.dtype)) + + +class ModelingV2MistralLarge4Nvfp4Sm103Dep( + DecoderModelForCausalLM[MistralLarge4DepModelingV2, PretrainedConfig] +): + def __init__(self, model_config: ModelConfig): + cfg = model_config.pretrained_config + super().__init__( + MistralLarge4DepModelingV2(model_config), + config=model_config, + hidden_size=cfg.hidden_size, + vocab_size=cfg.vocab_size, + ) + self.softcap = float(cfg.softmax_tanh_gating) + self._vision: Optional[VisionTower] = None + vis = getattr(cfg, "vision_config", None) + self._image_token_ids = ( + torch.tensor([vis.image_token_id], dtype=torch.int32) if vis is not None else None + ) + + self.spec_config = model_config.spec_config + self.draft_model = None + self.draft_config = None + self.spec_worker = None + if self.spec_config is not None: + # This is allocation geometry for the independently paged BF16 + # draft, not a second registry-resolved model or target cache. + draft_cfg = PretrainedConfig( + num_hidden_layers=int(self.spec_config._draft_model_config["n_layers"]), + hidden_size=cfg.hidden_size, + num_attention_heads=cfg.num_attention_heads, + num_key_value_heads=1, + kv_lora_rank=cfg.kv_lora_rank, + qk_rope_head_dim=cfg.qk_rope_head_dim, + qk_nope_head_dim=cfg.qk_nope_head_dim, + v_head_dim=cfg.v_head_dim, + intermediate_size=cfg.intermediate_size, + vocab_size=cfg.vocab_size, + max_position_embeddings=cfg.max_position_embeddings, + torch_dtype=torch.bfloat16, + ) + self.draft_config = replace( + model_config, + pretrained_config=draft_cfg, + quant_config=QuantConfig(), + quant_config_dict=None, + sparse_attention_config=None, + spec_config=None, + modeling_v2_target=None, + ) + self.draft_model = DraftModel(self, self.draft_config) + self.spec_worker = get_spec_worker( + self.spec_config, + model_config, + model_config.mapping, + use_separate_draft_kv_cache=True, + ) + self.spec_worker.set_draft_model(self.draft_model) + self.epilogue.append(self.spec_worker) + + def set_guided_decoder(self, guided_decoder: CapturableGuidedDecoder) -> bool: + if self.spec_worker is None: + return False + return self.spec_worker.set_guided_decoder(guided_decoder) + + @property + def multimodal_token_ids(self) -> Optional[torch.Tensor]: + return self._image_token_ids + + def forward( + self, + attn_metadata: AttentionMetadata, + input_ids: torch.IntTensor | None = None, + position_ids: torch.IntTensor | None = None, + inputs_embeds: torch.FloatTensor | None = None, + return_context_logits: bool = False, + spec_metadata=None, + lora_params: dict | None = None, + **kwargs, + ) -> torch.Tensor: + """Soft-cap target logits before verification and return the worker result.""" + worker_input_ids = input_ids + multimodal_params = kwargs.get("multimodal_params") or [] + if multimodal_params: + rows = _image_rows(self._vision, multimodal_params) + if rows is not None: + inputs_embeds = _splice_image_rows( + self.model.w["embed"], input_ids, rows, kwargs.get("mm_token_indices") + ) + input_ids = None + if self.spec_worker is None: + logits = super().forward( + attn_metadata, + cast(torch.IntTensor, input_ids), + position_ids, + inputs_embeds, + return_context_logits, + spec_metadata, + lora_params, + **kwargs, + ) + return tanh_softcap(logits, self.softcap) + hidden = self.model( + attn_metadata=attn_metadata, + input_ids=input_ids, + position_ids=position_ids, + inputs_embeds=inputs_embeds, + spec_metadata=spec_metadata, + lora_params=lora_params, + **kwargs, + ) + logits = self.logits_processor( + nn.functional.embedding(spec_metadata.gather_ids, hidden), + self.lm_head, + attn_metadata, + True, + ) + return self.spec_worker( + input_ids=worker_input_ids, + position_ids=position_ids, + hidden_states=hidden, + logits=tanh_softcap(logits, self.softcap), + attn_metadata=attn_metadata, + spec_metadata=spec_metadata, + draft_model=self.draft_model, + resource_manager=kwargs.get("resource_manager"), + ) + + def load_weights(self, weights, *args, **kwargs): + _weights.MODEL_WEIGHTS.load(self, weights) + + def load_checkpoint(self, checkpoint_dir: str) -> None: + _weights.MODEL_WEIGHTS.load_checkpoint(self, checkpoint_dir) + + def post_load_weights(self): + super().post_load_weights() + self.model.post_load_weights() + if step_contract_enabled(): + logger.info( + "modeling_v2 loaded target=%s class=%s rank=%d world=%d vision=%s", + __name__.rsplit(".", 1)[0], + type(self).__name__, + self.model_config.mapping.rank, + self.model_config.mapping.world_size, + getattr(self.config, "vision_config", None) is not None, + ) + if getattr(self.config, "vision_config", None) is not None: + self._vision = VisionTower(self.model) + if self.draft_model is not None: + self.draft_model._build_fused_kv_buffers() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/weights.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/weights.py new file mode 100644 index 000000000000..6a1c392eb5e7 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/weights.py @@ -0,0 +1,1250 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Weight table and loader: mistral-large-4-nvfp4 / sm_103 / dep, the language model. + +The checkpoint is Mistral-native (`consolidated-*.safetensors`, `layers.N.attention.wq`, ...) +and ModelOpt `MIXED_PRECISION`: the routed experts NVFP4, the other big linears FP8 128x128 +block (`FP8_PB_WO`), everything else bf16. Every rank is handed the whole of it, and under +attention data parallelism the topology divides the requests, not the weights, so the table +makes one split only: + +* **expert windows** -- rank `r` (its `moe_ep_rank`) loads routed experts `[Er, Er + E)`, + `E = n_routed_experts / moe_ep_size` whole experts (32 at 4 ranks, 16 at 8 ranks over two + nodes of four), the windows tiling the 128 once; +* **replicated, whole** -- everything else: `wq`, `wkv_b`, `wo` and the MLA SSMax `s`/`b` over + all 64 heads; `wkv_a_with_mqa` and `kv_a_norm`; the whole indexer; the dense layer's and the + shared expert's `w1`/`w3`/`w2` at full width, which the forward runs on each rank's own + tokens; the router, the embedding, every norm and `lerope.alpha`; and the shell's `lm_head`, + which under attention DP is the full `[vocab, hidden]` on every rank. + +So no row slices one of its checkpoint tensors: each source is read whole, and only the expert +rows choose, by rank, which tensors they read. + +Layer sets come from the config: `wkv_a_with_mqa` and `kv_a_norm` exist on the KV-owning layers +only (`kv_source_layers[i] == i`; the sharers read their source's latent), the indexer on the +`index_sharing == 'full'` layers only, the dense MLP on the first `first_k_dense_replace` +layers and the router, shared expert and routed experts on the rest. + +Load-time layouts, each the one its consumer reads: + +* `wkv_b` regrouped per head `[k_nope | v]` -> `[all k_nope ; all v]` over all heads (k_nope is + 64 wide on RoPE layers and 128 on NoPE layers); +* the dense MLP and the shared expert `[w1 ; w3]`, gate rows first, the half order + `flashinfer_silu_and_mul` reads; +* FP8 linears (`wq`, `wo`, the indexer's `wq_b`, the dense and shared `w1`/`w3`/`w2`): e4m3 + `[N, K]` as stored, and the fp32 block scales -- stored as `.weight_scale` + `[N/128, 1, K/128, 1]`, squeezed here to the `[N/128, K/128]` grid -- in DeepGEMM's packed + UE8M0 operand, `transform_sf_into_required_layout(recipe=(1, 128, 128))`: what + `FP8BlockScalesLinearMethod.transform_weights` installs as `weight_scale` on sm_100f. That + operand is an MN-major `[N, ceil(K/512)]` int32 view; the parameter holds the dense + `[ceil(K/512), N]` behind it and the GEMM takes `.t()`. The method's + `resmooth_to_fp8_e8m0` is not run: every scale is asserted an exact power of two, which makes + it a power-of-two re-exponent of the same values; +* the indexer's `wk` and `weights_proj` stacked into one `[128 + 32, hidden]` bf16 GEMM operand; + its `k_norm` and `lerope.alpha` widened to fp32. + +The routed experts are this rank's window stacked on an expert axis, in the NVFP4 MoE runner's +layout: e2m1 codes, two per byte, `[E, 2I, H/2]` / `[E, H, I/2]` uint8 and their e4m3 block +scales, one per 16 inputs, `[E, 2I, H/16]` / `[E, H, I/16]`. Each source lands as stored -- +FC1's rows `[w3 ; w1]`, up rows first, the order the runner's FC1 concatenates them in -- and +once every source of an expert stack has landed, its row's `layout` rearranges each expert in +place (the runner reads nothing else, and a wrong layout costs 300-3400 bf16 ulp and raises +nothing): + +* FC1, codes and block scales alike: the halves interleaved (row `2i` up row `i`, row `2i + 1` + gate row `i`), then the 32-row block shuffle (within each 32 rows, row `4u + v` moves to + `8v + u`); +* FC2: the 32-row block shuffle; +* both block scales: then the 128x4 swizzle (`torch.ops.trtllm.block_scale_interleave`'s order, + written here in torch so the table also runs on meta tensors). + +The runner's output scalars are not derived here: the four per-tensor scalars are handed over +as stored -- `isc1` and `isc2`, the one FC1 and FC2 activation `input_scale` of the layer, and +per local expert `ws2_13` (`w1`'s `weight_scale_2`, which `w3` shares) and `ws2_2`. ModelOpt +stores these as reciprocals of the global scales a quantizer divides by. + +Before any of that is read, `load` reads the per-tensor scalars of all the layer's experts, +the other ranks' included, and asserts what the forward relies on: `w1` and `w3` store the same +`input_scale` and `weight_scale_2` on every expert, so one FC1 alpha serves the stacked +`[w3 ; w1]`; and each projection stores one `input_scale` for every expert, so one NVFP4 +quantization of the gathered activations is right for every expert on every rank and the +windows' partial outputs sum. + +The BF16 vision encoder, pre-projector norm, patch merger and language adapter are +replicated on every rank when the configuration declares a vision encoder. + +`expected_unconsumed` accounts for off-rank routed experts and disabled DSpark draft weights. +Off-rank expert scalars are still read to check shared scales; their codes and block scales +are never read. +At `moe_ep_size` 1 every expert belongs to the rank and is read, sliced. +""" + +from __future__ import annotations + +import collections +import functools +import json +import os +from dataclasses import dataclass +from pathlib import Path +from types import SimpleNamespace +from typing import Any, Callable + +import torch +import torch.nn as nn +from safetensors import safe_open + +from tensorrt_llm.logger import logger +from tensorrt_llm.quantization.utils.fp8_utils import transform_sf_into_required_layout + +F8, F32, I32, U8 = torch.float8_e4m3fn, torch.float32, torch.int32, torch.uint8 + +#: The FP8 block edge on both weight axes (the checkpoint's `FP8_PB_WO` 128x128 blocks), and +#: the K granularity of the DeepGEMM scale recipe `(1, 128, 128)`. +BLOCK = 128 +#: UE8M0 bytes per int32 word of the DeepGEMM scale operand: four K blocks. +SF_PACK = 4 +#: NVFP4: one e4m3 block scale per 16 inputs (the checkpoint's `group_size`), two e2m1 codes +#: per byte. +NVFP4_BLOCK = 16 +E2M1_PER_BYTE = 2 + +#: A routed expert's three projections: `w1` gate, `w3` up, `w2` down. +PROJECTIONS = ("w1", "w2", "w3") +#: The per-tensor NVFP4 scalars each of them carries beside its codes and block scales. +EXPERT_SCALARS = ("weight_scale_2", "input_scale") + +#: The vision tower's checkpoint prefixes. +VISION_PREFIXES = ( + "vision_encoder.", + "patch_merger.", + "pre_mm_projector_norm.", + "vision_language_adapter.", +) + + +#: `layers=ALL_LAYERS` is the common case and the default. A sentinel rather +#: than a lambda so the table reads as data, and rather than `True` so the +#: field has one type of answer: which layers. +ALL_LAYERS = object() + + +@dataclass(frozen=True) +class W: + """One weight role: what it is, where its bytes come from, what happens on the way. + + `shape` takes the dimension bundle and returns the declared shape. + + `src` says where the bytes come from: a checkpoint key template with `{p}` + for the layer prefix (`layers.N`), or a callable `(d, i) -> [(key, index), ...]` + where `index` is where in the parameter that tensor lands (`None`: all of it). + A `key` may be a tuple of per-tensor scalars, which `transform` is handed together. + Every source is read whole. + + `layers` says which layers carry this weight: `ALL_LAYERS`, `None` for a + weight that is not per-layer, or a callable `d -> iterable[int]`. Two rows + may share a name over disjoint layer sets, with different shapes. + + `transform` runs on the source tensor, on the destination's device. + + `layout` runs once every source of the parameter has landed, on each slot of its leading + (expert) axis, and the slot is overwritten with the result: a rearrangement that mixes rows + several sources fill, which no single source's `transform` can make. + """ + + name: str + shape: Callable[[Any], tuple[int, ...]] + src: str | Callable + dtype: torch.dtype | None = None + transform: Callable | None = None + layers: object | None = ALL_LAYERS + layout: Callable | None = None + when: Callable | None = None + tag: str = "l" + + def layer_indices(self, d: SimpleNamespace) -> list[int] | None: + """The layers this weight exists on, or None if it is not per-layer.""" + if self.layers is None: + return None + if self.layers is ALL_LAYERS: + return list(range(d.num_layers)) + return list(self.layers(d)) + + def sources(self, d: SimpleNamespace, i: int | None, prefix: str) -> list[tuple[Any, Any]]: + """`[(checkpoint key | key tuple, index into the parameter | None), ...]`.""" + if callable(self.src): + return list(self.src(d, i)) + return [(self.src.format(p=prefix), None)] + + +def _keys(ckpt) -> tuple[str, ...]: + """The checkpoint keys of one source: itself, or the members of a key tuple.""" + return ckpt if isinstance(ckpt, tuple) else (ckpt,) + + +def _sf_words(k: int) -> int: + """int32 words per row of the DeepGEMM scale operand of a K-wide weight.""" + return -(-k // (BLOCK * SF_PACK)) + + +def _assert_ue8m0(s: torch.Tensor) -> None: + """Every block scale is a positive normal power of two, so its UE8M0 byte (the fp32 + exponent field) holds it exactly. A meta tensor carries no values to check.""" + if s.is_meta: + return + bits = s.view(torch.int32) + exp = (bits >> 23) & 0xFF + exact = ((bits & 0x807FFFFF) == 0) & (exp > 0) & (exp < 255) + assert bool(exact.all()), ( + f"weight_scale: {int((~exact).sum())} of {s.numel()} block scales are not a positive " + "normal power of two (UE8M0); the scale layouts below would change their values" + ) + + +def _deepgemm_sf(s: torch.Tensor, core) -> torch.Tensor: + """fp32 `[N/128, K/128]` block scales -> the dense `[ceil(K/512), N]` int32 storage of + DeepGEMM's MN-major packed UE8M0 operand, whose `.t()` is that operand. N is a multiple of + 128 for every role here, so the operand's row stride is N and its storage has no padding.""" + _assert_ue8m0(s) + n_blk, k_blk = s.shape + sf = transform_sf_into_required_layout( + s, mn=n_blk * BLOCK, k=k_blk * BLOCK, recipe=(1, BLOCK, BLOCK), is_sfa=False + ) + return sf.t() + + +def _block_sf(s: torch.Tensor, core) -> torch.Tensor: + """An FP8 linear's block scales as stored, fp32 `[N/128, 1, K/128, 1]`: squeezed to the + `[N/128, K/128]` grid, then `_deepgemm_sf`.""" + assert s.dim() == 4 and s.shape[1] == 1 and s.shape[3] == 1, ( + f"FP8 block scale stored as {tuple(s.shape)}, not [N/128, 1, K/128, 1]" + ) + return _deepgemm_sf(s.reshape(s.shape[0], s.shape[2]), core) + + +def _shared_scalar(t: tuple[torch.Tensor, ...], core) -> torch.Tensor: + """One per-tensor scalar the checkpoint stores several times (`w1`'s and `w3`'s, or every + expert's of the window): asserted the same value everywhere, then that value.""" + first = t[0] + if not first.is_meta: + values = torch.stack(t) + assert bool((values == first).all()), ( + f"{int((values != first).sum())} of {len(t)} copies of one shared scale differ from " + f"the first, {first.item()!r}" + ) + return first + + +def _regroup_kv_b(t: torch.Tensor, core) -> torch.Tensor: + """All heads' `[heads * (k_nope + v), kv_lora]` rows, per-head `[k_nope | v]` -> + `[heads * k_nope ; heads * v]`: the K-B operand, then the V-B one. k_nope is read off the + tensor: 64 on RoPE layers, 128 on NoPE layers.""" + cfg = core.model_config.pretrained_config + heads = cfg.num_attention_heads + v = cfg.v_head_dim + t3 = t.reshape(heads, t.shape[0] // heads, t.shape[1]) + nope = t3.shape[1] - v + return torch.cat( + [t3[:, :nope, :].reshape(heads * nope, -1), t3[:, nope:, :].reshape(heads * v, -1)], dim=0 + ).contiguous() + + +def _to_fp32(t: torch.Tensor, core) -> torch.Tensor: + return t.to(torch.float32) + + +# --- the NVFP4 MoE runner's expert layout, one expert at a time --- + +#: The runner's row shuffle works on blocks of this many rows. +SHUFFLE_ROWS = 32 +#: The block-scale swizzle's tile: 128 rows by 4 scale columns. +SWIZZLE_ROWS, SWIZZLE_COLS = 128, 4 + + +@functools.lru_cache(maxsize=None) +def _block32_perm(rows: int, device) -> torch.Tensor: + """Gather index of the 32-row block shuffle: within each block of 32 rows, source row + `4u + v` lands at row `8v + u`.""" + assert rows % SHUFFLE_ROWS == 0, rows + j = torch.arange(SHUFFLE_ROWS) + one = torch.empty(SHUFFLE_ROWS, dtype=torch.long) + one[(j % 4) * 8 + j // 4] = j + blocks = torch.arange(rows // SHUFFLE_ROWS).repeat_interleave(SHUFFLE_ROWS) * SHUFFLE_ROWS + return (one.repeat(rows // SHUFFLE_ROWS) + blocks).to(device) + + +@functools.lru_cache(maxsize=None) +def _fc1_perm(rows: int, device) -> torch.Tensor: + """Gather index of FC1's `[up ; gate]` rows: interleaved (row `2i` up row `i`, row `2i + 1` + gate row `i`), then block-shuffled.""" + interleave = torch.empty(rows, dtype=torch.long) + interleave[0::2] = torch.arange(rows // 2) + interleave[1::2] = torch.arange(rows // 2, rows) + return interleave.to(device)[_block32_perm(rows, device)] + + +def _swizzle(s: torch.Tensor) -> torch.Tensor: + """One expert's `[M, C]` e4m3 block scales in the 128x4-swizzled byte order, kept `[M, C]`: + byte `(m, c)` lands at flat offset `(m // 128) * 128 * C + (c // 4) * 512 + (m % 32) * 16 + + ((m % 128) // 32) * 4 + c % 4`. Both of this target's scale stacks tile it exactly (M a + multiple of 128, C of 4), so there is no padding.""" + m, c = s.shape + assert m % SWIZZLE_ROWS == 0 and c % SWIZZLE_COLS == 0, (m, c) + tiles = s.view(torch.uint8).reshape(m // SWIZZLE_ROWS, 4, 32, c // SWIZZLE_COLS, SWIZZLE_COLS) + return tiles.permute(0, 3, 2, 1, 4).reshape(m, c).view(s.dtype) + + +def _fc1_codes(w: torch.Tensor) -> torch.Tensor: + return torch.index_select(w, 0, _fc1_perm(w.shape[0], w.device)) + + +def _fc1_scales(s: torch.Tensor) -> torch.Tensor: + rows = torch.index_select(s.view(torch.uint8), 0, _fc1_perm(s.shape[0], s.device)) + return _swizzle(rows.view(s.dtype)) + + +def _fc2_codes(w: torch.Tensor) -> torch.Tensor: + return torch.index_select(w, 0, _block32_perm(w.shape[0], w.device)) + + +def _fc2_scales(s: torch.Tensor) -> torch.Tensor: + rows = torch.index_select(s.view(torch.uint8), 0, _block32_perm(s.shape[0], s.device)) + return _swizzle(rows.view(s.dtype)) + + +def _materialize(entry) -> torch.Tensor: + """One checkpoint entry's bytes, whole: a tensor, or a lazy safetensors slice, which reads + them only now. `[...]`, not `[:]`: the per-tensor scalars are 0-dim.""" + return entry[...] + + +#: The checkpoint's shard index, beside its `consolidated-*.safetensors`. +SHARD_INDEX = "consolidated.safetensors.index.json" + + +def _checkpoint_shards(checkpoint_dir) -> tuple[Path, dict[str, str]]: + """`(directory, key -> shard file)` of a checkpoint directory: its index when it ships + one, else the header of each `consolidated*.safetensors`.""" + directory = Path(checkpoint_dir) + index = directory / SHARD_INDEX + if index.is_file(): + return directory, json.loads(index.read_text())["weight_map"] + weight_map: dict[str, str] = {} + for path in sorted(directory.glob("consolidated*.safetensors")): + with safe_open(str(path), framework="pt", device="cpu") as handle: + weight_map.update(dict.fromkeys(handle.keys(), path.name)) + if not weight_map: + raise FileNotFoundError(f"no consolidated*.safetensors in {directory}") + return directory, weight_map + + +def _shard_files(core, weights) -> tuple[Path, dict[str, str]] | None: + """`(directory, key -> shard file)` of the checkpoint a dict of sources was read from, or + None when the sources are not the shard files of one (the directory the config records + lacks the index, or the index does not name every key `weights` holds).""" + cfg = getattr(core.model_config, "pretrained_config", None) + root = getattr(cfg, "_name_or_path", None) + if not root: + return None + index = Path(root) / SHARD_INDEX + if not index.is_file(): + return None + weight_map = json.loads(index.read_text())["weight_map"] + if not set(weights.keys()) <= set(weight_map): + return None + return Path(root), weight_map + + +def _drop_page_cache(path: Path) -> None: + """Advise the kernel to drop a consumed shard file's cached pages. Advice only: a + filesystem that ignores it costs memory pressure, not correctness.""" + try: + fd = os.open(path, os.O_RDONLY) + except OSError: + return + try: + os.posix_fadvise(fd, 0, 0, os.POSIX_FADV_DONTNEED) + except OSError: + pass + finally: + os.close(fd) + + +# --- which layers carry a weight --- + + +def _owners(d): + """Layers that own a latent KV. The others read `kv_source_layers[i]`'s and project none.""" + return [i for i in range(d.num_layers) if d.kv_source[i] == i] + + +def _rope_owners(d): + return [i for i in _owners(d) if not d.nope_layer[i]] + + +def _nope_owners(d): + return [i for i in _owners(d) if d.nope_layer[i]] + + +def _rope_layers(d): + return [i for i in range(d.num_layers) if not d.nope_layer[i]] + + +def _nope_layers(d): + return [i for i in range(d.num_layers) if d.nope_layer[i]] + + +def _indexer_layers(d): + """Layers that own an indexer. The 'shared' ones reuse the most recent one's selection.""" + return [i for i in range(d.num_layers) if d.full_indexer[i]] + + +def _dense_layers(d): + return range(d.dense_layers) + + +def _moe_layers(d): + return range(d.dense_layers, d.num_layers) + + +# --- where the bytes come from --- + + +def _stack(*parts, prefix="layers"): + """Several checkpoint tensors into one parameter: each `(key template, index(d))`, the + index being where in the parameter that tensor lands.""" + return lambda d, i: [(t.format(p=f"{prefix}.{i}"), index(d)) for t, index in parts] + + +def _expert_key(i: int, e: int, suffix: str) -> str: + return f"layers.{i}.experts.{e}.{suffix}" + + +def _expert_window(*parts): + """This rank's routed experts, local slot `s` <- global expert `expert_offset + s`: each + `(checkpoint suffix | suffix tuple, index(d) after the expert axis)`.""" + + def key(i, e, suffix): + if isinstance(suffix, tuple): + return tuple(_expert_key(i, e, s) for s in suffix) + return _expert_key(i, e, suffix) + + def build(d, i): + return [ + (key(i, d.expert_offset + slot, suffix), (slot, *index(d))) + for slot in range(d.local_experts) + for suffix, index in parts + ] + + return build + + +def _window_scalar(*suffixes): + """One per-tensor scalar every expert of the window stores: all of the window's copies, one + key tuple, into element 0.""" + return lambda d, i: [ + ( + tuple( + _expert_key(i, d.expert_offset + slot, s) + for slot in range(d.local_experts) + for s in suffixes + ), + (0,), + ) + ] + + +def _expert_scalar_keys(d) -> list[str]: + """Every routed expert's per-tensor scalars on every MoE layer, the other ranks' included: + what `load` reads, and checks, before anything else.""" + return [ + _expert_key(i, e, f"{proj}.{s}") + for i in _moe_layers(d) + for e in range(d.num_experts) + for proj in PROJECTIONS + for s in EXPERT_SCALARS + ] + + +def _check_expert_scalars(scalars: dict[str, torch.Tensor], d) -> None: + """Per MoE layer, over all of its experts: `w1` and `w3` store the same `input_scale` and + `weight_scale_2` on every expert; `w1` and `w2` each store one `input_scale` for every expert; + every scalar is a positive finite fp32. Meta tensors carry no values to check.""" + if scalars and next(iter(scalars.values())).is_meta: + return + experts = range(d.num_experts) + for i in _moe_layers(d): + v = { + (proj, s): torch.stack([scalars[_expert_key(i, e, f"{proj}.{s}")] for e in experts]) + for proj in PROJECTIONS + for s in EXPERT_SCALARS + } + for (proj, s), t in v.items(): + bad = (~(torch.isfinite(t) & (t > 0))).nonzero().flatten() + assert bad.numel() == 0, ( + f"layers.{i}: {proj}.{s} of expert {int(bad[0])} is {t[bad[0]].item()!r}, " + "not a positive finite scale" + ) + for s in EXPERT_SCALARS: + bad = (v["w1", s] != v["w3", s]).nonzero().flatten() + assert bad.numel() == 0, ( + f"layers.{i}: w1 and w3 {s} differ on {bad.numel()} of {len(experts)} experts, " + f"first expert {int(bad[0])}: {v['w1', s][bad[0]].item()!r} vs " + f"{v['w3', s][bad[0]].item()!r}" + ) + for proj in ("w1", "w2"): + isc = v[proj, "input_scale"] + bad = (isc != isc[0]).nonzero().flatten() + assert bad.numel() == 0, ( + f"layers.{i}: {proj}.input_scale is not one value over the experts: expert " + f"{int(bad[0])} stores {isc[bad[0]].item()!r}, expert 0 {isc[0].item()!r}" + ) + + +def _read_scalars(keys: list[str], weights, shards) -> dict[str, torch.Tensor]: + """The given per-tensor scalars, off the shard files one file at a time when they are + found, otherwise off the given dict.""" + if shards is None: + return {key: _materialize(weights[key]) for key in keys} + directory, weight_map = shards + by_file: dict[str, list] = collections.defaultdict(list) + for key in keys: + by_file[weight_map[key]].append(key) + out = {} + for name in sorted(by_file): + with safe_open(str(directory / name), framework="pt", device="cpu") as handle: + for key in by_file[name]: + out[key] = handle.get_slice(key)[...].clone() + return out + + +def _fp8_linear(name, key, n, k, layers): + """An FP8 128x128-block linear: e4m3 `[N, K]` and its DeepGEMM scale operand's storage.""" + return ( + W(name, shape=lambda d: (n(d), k(d)), dtype=F8, src=key, layers=layers), + W( + f"{name}_s", + shape=lambda d: (_sf_words(k(d)), n(d)), + dtype=I32, + src=key.replace(".weight", ".weight_scale"), + transform=_block_sf, + layers=layers, + ), + ) + + +def _mlp(module, inter, layers): + """A SwiGLU MLP at full width: `[w1 ; w3]` stacked by output rows, gate first (the half + order `flashinfer_silu_and_mul` reads), and `w2`. The scales follow; in the scale storage + the stacked axis is the second one.""" + p = "{p}." + module + return ( + W( + "mlp_gu", + shape=lambda d: (2 * inter(d), d.hidden), + dtype=F8, + src=_stack( + (f"{p}.w1.weight", lambda d: (slice(0, inter(d)),)), + (f"{p}.w3.weight", lambda d: (slice(inter(d), 2 * inter(d)),)), + ), + layers=layers, + ), + W( + "mlp_gu_s", + shape=lambda d: (_sf_words(d.hidden), 2 * inter(d)), + dtype=I32, + src=_stack( + (f"{p}.w1.weight_scale", lambda d: (slice(None), slice(0, inter(d)))), + (f"{p}.w3.weight_scale", lambda d: (slice(None), slice(inter(d), 2 * inter(d)))), + ), + transform=_block_sf, + layers=layers, + ), + *_fp8_linear("mlp_dn", f"{p}.w2.weight", lambda d: d.hidden, inter, layers), + ) + + +def _kv_b(nope, layers): + """`wkv_b`, all heads, regrouped. Per-head rows are `nope(d) + v` wide.""" + return W( + "wkv_b", + shape=lambda d: (d.heads * (nope(d) + d.v_dim), d.kv_lora), + src="{p}.attention.wkv_b.weight", + transform=_regroup_kv_b, + layers=layers, + ) + + +def _vision_blocks(d): + return list(range(d.v_layers)) + + +def _has_vision(d): + return d.has_vision + + +def _flatten_patch_conv(tensor, core): + return tensor.flatten(1) + + +def _draft_layers(d): + return range(d.draft_layers) + + +def _draft_source(suffix): + return lambda d, i: [(f"mtp_dspark.backbone.layers.{i}.{suffix}", None)] + + +class MistralLarge4DepWeights: + """mistral-large-4-nvfp4 / sm_103 / dep.""" + + WEIGHTS: tuple[W, ...] = ( + # --- MLA: every layer, KV owners only where marked --- + W("attn_norm", shape=lambda d: (d.hidden,), src="{p}.attention_norm.weight"), + # Per head `[nope 64 | rope 64]` on RoPE layers, 128 nope on NoPE layers: 128 either way. + *_fp8_linear( + "wq", + "{p}.attention.wq.weight", + lambda d: d.heads * d.qk_dim, + lambda d: d.hidden, + ALL_LAYERS, + ), + W("ssmax_s", shape=lambda d: (d.heads,), src="{p}.attention.ssmax_scaling.s"), + W("ssmax_b", shape=lambda d: (d.heads,), src="{p}.attention.ssmax_scaling.b"), + # `[c_kv 512 | k_pe 64]` on RoPE owners; NoPE owners project the 512 latent alone. + W( + "wkv_a", + shape=lambda d: (d.lat_dim, d.hidden), + src="{p}.attention.wkv_a_with_mqa.weight", + layers=_rope_owners, + ), + W( + "wkv_a", + shape=lambda d: (d.kv_lora, d.hidden), + src="{p}.attention.wkv_a_with_mqa.weight", + layers=_nope_owners, + ), + W( + "kv_a_norm", + shape=lambda d: (d.kv_lora,), + src="{p}.attention.kv_a_norm.weight", + layers=_owners, + ), + # Every layer absorbs through its own `wkv_b`, KV sharers included. + _kv_b(lambda d: d.nope, _rope_layers), + _kv_b(lambda d: d.qk_dim, _nope_layers), + *_fp8_linear( + "wo", + "{p}.attention.wo.weight", + lambda d: d.hidden, + lambda d: d.heads * d.v_dim, + ALL_LAYERS, + ), + W("ffn_norm", shape=lambda d: (d.hidden,), src="{p}.ffn_norm.weight"), + # --- the DSA indexer, on the 'full' layers --- + W( + "ix_q_norm_w", + shape=lambda d: (d.hidden,), + src="{p}.attention.indexer.q_input_norm.weight", + layers=_indexer_layers, + ), + W( + "ix_q_norm_b", + shape=lambda d: (d.hidden,), + src="{p}.attention.indexer.q_input_norm.bias", + layers=_indexer_layers, + ), + # Per head `[rope 64 | nope 64]`, the reverse of MLA's. + *_fp8_linear( + "ix_wq_b", + "{p}.attention.indexer.wq_b.weight", + lambda d: d.idx_heads * d.idx_dim, + lambda d: d.hidden, + _indexer_layers, + ), + # `[wk ; weights_proj]`: the key and the head weights, one GEMM. + W( + "ix_wk_wp", + shape=lambda d: (d.idx_dim + d.idx_heads, d.hidden), + src=_stack( + ("{p}.attention.indexer.wk.weight", lambda d: (slice(0, d.idx_dim),)), + ( + "{p}.attention.indexer.weights_proj.weight", + lambda d: (slice(d.idx_dim, d.idx_dim + d.idx_heads),), + ), + ), + layers=_indexer_layers, + ), + # fp32, as the fused key entry (LayerNorm in fp32) takes them. + W( + "ix_k_norm_w", + shape=lambda d: (d.idx_dim,), + dtype=F32, + src="{p}.attention.indexer.k_norm.weight", + transform=_to_fp32, + layers=_indexer_layers, + ), + W( + "ix_k_norm_b", + shape=lambda d: (d.idx_dim,), + dtype=F32, + src="{p}.attention.indexer.k_norm.bias", + transform=_to_fp32, + layers=_indexer_layers, + ), + W( + "ix_ssmax_s", + shape=lambda d: (d.idx_heads,), + src="{p}.attention.indexer.ssmax_scaling.s", + layers=_indexer_layers, + ), + W( + "ix_ssmax_b", + shape=lambda d: (d.idx_heads,), + src="{p}.attention.indexer.ssmax_scaling.b", + layers=_indexer_layers, + ), + # --- the SwiGLU MLP at full width: the dense layer's own, then each MoE layer's shared + # expert, one key --- + *_mlp("feed_forward", lambda d: d.dense_inter, _dense_layers), + *_mlp("shared_experts", lambda d: d.shared_inter, _moe_layers), + # --- routing --- + W( + "router", + shape=lambda d: (d.num_experts, d.hidden), + src="{p}.gate.weight", + layers=_moe_layers, + ), + # --- this rank's routed experts, NVFP4: `[w3 ; w1]` as stored, then each expert in the + # MoE runner's layout --- + W( + "fc1_w", + shape=lambda d: (d.local_experts, 2 * d.moe_inter, d.hidden // E2M1_PER_BYTE), + dtype=U8, + src=_expert_window( + ("w3.weight", lambda d: (slice(0, d.moe_inter),)), + ("w1.weight", lambda d: (slice(d.moe_inter, 2 * d.moe_inter),)), + ), + layers=_moe_layers, + layout=_fc1_codes, + ), + W( + "fc1_s", + shape=lambda d: (d.local_experts, 2 * d.moe_inter, d.hidden // NVFP4_BLOCK), + dtype=F8, + src=_expert_window( + ("w3.weight_scale", lambda d: (slice(0, d.moe_inter),)), + ("w1.weight_scale", lambda d: (slice(d.moe_inter, 2 * d.moe_inter),)), + ), + layers=_moe_layers, + layout=_fc1_scales, + ), + W( + "fc2_w", + shape=lambda d: (d.local_experts, d.hidden, d.moe_inter // E2M1_PER_BYTE), + dtype=U8, + src=_expert_window(("w2.weight", lambda d: ())), + layers=_moe_layers, + layout=_fc2_codes, + ), + W( + "fc2_s", + shape=lambda d: (d.local_experts, d.hidden, d.moe_inter // NVFP4_BLOCK), + dtype=F8, + src=_expert_window(("w2.weight_scale", lambda d: ())), + layers=_moe_layers, + layout=_fc2_scales, + ), + # The per-tensor scalars: the layer's one FC1 and one FC2 activation scale, and each + # local expert's global weight scales (`w3`'s are `w1`'s). + W( + "isc1", + shape=lambda d: (1,), + dtype=F32, + src=_window_scalar("w1.input_scale", "w3.input_scale"), + transform=_shared_scalar, + layers=_moe_layers, + ), + W( + "isc2", + shape=lambda d: (1,), + dtype=F32, + src=_window_scalar("w2.input_scale"), + transform=_shared_scalar, + layers=_moe_layers, + ), + W( + "ws2_13", + shape=lambda d: (d.local_experts,), + dtype=F32, + src=_expert_window((("w1.weight_scale_2", "w3.weight_scale_2"), lambda d: ())), + transform=_shared_scalar, + layers=_moe_layers, + ), + W( + "ws2_2", + shape=lambda d: (d.local_experts,), + dtype=F32, + src=_expert_window(("w2.weight_scale_2", lambda d: ())), + layers=_moe_layers, + ), + # --- not per-layer --- + W("final_norm", shape=lambda d: (d.hidden,), src="norm.weight", layers=None), + W("embed", shape=lambda d: (d.vocab, d.hidden), src="tok_embeddings.weight", layers=None), + # The LeRoPE per-frequency log-scales the rope tables are built from, fp32 as the + # reference widens them. + W( + "lerope_alpha", + shape=lambda d: (d.rope_dim // 2,), + dtype=F32, + src="lerope.alpha", + transform=_to_fp32, + layers=None, + ), + # BF16 vision encoder and projector, replicated on every rank. + W( + "vis_patch", + shape=lambda d: (d.v_hidden, d.v_channels * d.v_patch**2), + src="vision_encoder.patch_conv.weight", + transform=_flatten_patch_conv, + layers=None, + when=_has_vision, + ), + W( + "vis_ln_pre", + shape=lambda d: (d.v_hidden,), + src="vision_encoder.ln_pre.weight", + layers=None, + when=_has_vision, + ), + W( + "attn_norm", + shape=lambda d: (d.v_hidden,), + src="{p}.attention_norm.weight", + layers=_vision_blocks, + tag="v", + ), + W( + "qkv", + shape=lambda d: (3 * d.v_hidden, d.v_hidden), + src=_stack( + ("{p}.attention.wq.weight", lambda d: (slice(0, d.v_hidden),)), + ("{p}.attention.wk.weight", lambda d: (slice(d.v_hidden, 2 * d.v_hidden),)), + ("{p}.attention.wv.weight", lambda d: (slice(2 * d.v_hidden, 3 * d.v_hidden),)), + prefix="vision_encoder.transformer.layers", + ), + layers=_vision_blocks, + tag="v", + ), + W( + "wo", + shape=lambda d: (d.v_hidden, d.v_hidden), + src="{p}.attention.wo.weight", + layers=_vision_blocks, + tag="v", + ), + W( + "ffn_norm", + shape=lambda d: (d.v_hidden,), + src="{p}.ffn_norm.weight", + layers=_vision_blocks, + tag="v", + ), + W( + "w13", + shape=lambda d: (2 * d.v_inter, d.v_hidden), + src=_stack( + ("{p}.feed_forward.w1.weight", lambda d: (slice(0, d.v_inter),)), + ("{p}.feed_forward.w3.weight", lambda d: (slice(d.v_inter, 2 * d.v_inter),)), + prefix="vision_encoder.transformer.layers", + ), + layers=_vision_blocks, + tag="v", + ), + W( + "w2", + shape=lambda d: (d.v_hidden, d.v_inter), + src="{p}.feed_forward.w2.weight", + layers=_vision_blocks, + tag="v", + ), + W( + "vis_proj_norm", + shape=lambda d: (d.v_hidden,), + src="pre_mm_projector_norm.weight", + layers=None, + when=_has_vision, + ), + W( + "vis_merge", + shape=lambda d: (d.v_hidden, d.v_hidden * d.v_merge**2), + src="patch_merger.merging_layer.weight", + layers=None, + when=_has_vision, + ), + W( + "vis_w_in", + shape=lambda d: (d.hidden, d.v_hidden), + src="vision_language_adapter.w_in.weight", + layers=None, + when=_has_vision, + ), + W( + "vis_w_out", + shape=lambda d: (d.hidden, d.hidden), + src="vision_language_adapter.w_out.weight", + layers=None, + when=_has_vision, + ), + ) + + # BF16 draft weights use the same attention and dense-MLP partition as + # this target. The RNN and target-feature projection are replicated. + DRAFT_WEIGHTS: tuple[W, ...] = ( + W( + "draft_attn_norm", + lambda d: (d.hidden,), + _draft_source("attention_norm.weight"), + layers=_draft_layers, + ), + W( + "draft_wq", + lambda d: (d.heads * d.qk_dim, d.hidden), + _draft_source("attention.wq.weight"), + layers=_draft_layers, + ), + W( + "draft_wkv_a", + lambda d: (d.lat_dim, d.hidden), + _draft_source("attention.wkv_a_with_mqa.weight"), + layers=_draft_layers, + ), + W( + "draft_kv_a_norm", + lambda d: (d.kv_lora,), + _draft_source("attention.kv_a_norm.weight"), + layers=_draft_layers, + ), + W( + "draft_wkv_b", + lambda d: (d.heads * (d.nope + d.v_dim), d.kv_lora), + _draft_source("attention.wkv_b.weight"), + layers=_draft_layers, + ), + W( + "draft_wo", + lambda d: (d.hidden, d.heads * d.v_dim), + _draft_source("attention.wo.weight"), + layers=_draft_layers, + ), + W( + "draft_ffn_norm", + lambda d: (d.hidden,), + _draft_source("ffn_norm.weight"), + layers=_draft_layers, + ), + W( + "draft_w1", + lambda d: (d.dense_inter, d.hidden), + _draft_source("feed_forward.w1.weight"), + layers=_draft_layers, + ), + W( + "draft_w3", + lambda d: (d.dense_inter, d.hidden), + _draft_source("feed_forward.w3.weight"), + layers=_draft_layers, + ), + W( + "draft_w2", + lambda d: (d.hidden, d.dense_inter), + _draft_source("feed_forward.w2.weight"), + layers=_draft_layers, + ), + W( + "draft_final_norm", + lambda d: (d.hidden,), + "mtp_dspark.backbone.final_norm.weight", + layers=None, + ), + W( + "draft_target_norm", + lambda d: (d.hidden,), + "mtp_dspark.backbone.target_norm.weight", + layers=None, + ), + W( + "draft_target_projection", + lambda d: (d.hidden, d.hidden * d.draft_capture_layers), + "mtp_dspark.backbone.target_projection.weight", + layers=None, + ), + W( + "draft_mask", + lambda d: (d.draft_depth - 1, d.hidden), + "mtp_dspark.backbone.mask_embedding", + layers=None, + ), + W( + "draft_markov_embedding", + lambda d: (d.vocab, d.draft_rank), + "mtp_dspark.autoregressive_head.embedding.weight", + layers=None, + ), + W( + "draft_markov_output", + lambda d: (d.vocab, d.draft_rank), + "mtp_dspark.autoregressive_head.output.weight", + layers=None, + ), + W( + "draft_joint", + lambda d: (3 * d.draft_rank, d.hidden + 2 * d.draft_rank), + "mtp_dspark.autoregressive_head.joint_proj.weight", + layers=None, + ), + W( + "draft_joint_bias", + lambda d: (3 * d.draft_rank,), + "mtp_dspark.autoregressive_head.joint_proj.bias", + layers=None, + ), + ) + + def _entries(self, d): + return self.WEIGHTS + (self.DRAFT_WEIGHTS if d.draft_layers else ()) + + def declare(self, d: SimpleNamespace) -> nn.ParameterDict: + """Allocate every weight the table declares. + + Meta-init intercepts `torch.empty` here -- real CUDA storage arrives + when the engine materializes the registry. + """ + w = nn.ParameterDict() + for entry in self._entries(d): + dtype = entry.dtype or d.dtype + for key, _, _ in self._targets(entry, d): + assert key not in w, f"two rows declare {key}" + w[key] = nn.Parameter( + torch.empty(*entry.shape(d), dtype=dtype), requires_grad=False + ) + return w + + def _targets(self, entry: W, d: SimpleNamespace) -> list[tuple[str, int | None, str]]: + """`[(param key, layer index | None, checkpoint layer prefix), ...]`. + + One place decides which parameters an entry declares, so `declare` and + `manifest` cannot disagree about whether a weight exists. + """ + if entry.when is not None and not entry.when(d): + return [] + layers = entry.layer_indices(d) + if layers is None: + return [(entry.name, None, "")] + prefix = "vision_encoder.transformer.layers" if entry.tag == "v" else "layers" + return [(f"{entry.tag}{i}_{entry.name}", i, f"{prefix}.{i}") for i in layers] + + def manifest(self, core) -> dict: + """param key -> [(checkpoint key | key tuple, index into the param | None, transform)].""" + d = self.dims(core) + rows: dict = {} + for entry in self._entries(d): + for key, i, prefix in self._targets(entry, d): + assert key not in rows, f"two rows declare {key}" + rows[key] = [ + (ckpt, index, entry.transform) for ckpt, index in entry.sources(d, i, prefix) + ] + return rows + + def expected_unconsumed(self, core, weights) -> dict[str, str]: + """Declared unused keys: off-rank experts and disabled draft weights. Off-rank expert + scalars are still read to validate the layer's shared scales.""" + d = self.dims(core) + lo, hi = d.expert_offset, d.expert_offset + d.local_experts + keys = { + k: "DSpark draft disabled" + for k in weights.keys() + if not d.draft_layers and k.startswith("mtp_dspark.") + } + window = f"routed expert outside this rank's window [{lo}, {hi})" + never_read = f"{window}: never read" + checked = f"{window}: read only to check the layer's shared scales" + for i in _moe_layers(d): + for e in range(d.num_experts): + if lo <= e < hi: + continue + for proj in PROJECTIONS: + for s in ("weight", "weight_scale"): + key = _expert_key(i, e, f"{proj}.{s}") + if key in weights: + keys[key] = never_read + for s in EXPERT_SCALARS: + keys[_expert_key(i, e, f"{proj}.{s}")] = checked + return keys + + def load_checkpoint(self, model, checkpoint_dir) -> None: + """Fill every declared weight from a checkpoint directory, which is what the engine + loads: `load` over the directory's keys, every source read from its shard file.""" + directory, weight_map = _checkpoint_shards(checkpoint_dir) + self.load(model, weight_map, shards=(directory, weight_map)) + + def load(self, model, weights, shards=None) -> None: + """Fill every declared weight from the checkpoint: multi-source rows, the shell's whole + `lm_head`, and coverage asserted in both directions. + + First every routed expert's per-tensor scalars are read and checked, the other ranks' + included, so a checkpoint that breaks what the forward relies on fails before any + weight moves. When the checkpoint's own shard files can be found, every source is read + from them one file at a time, this rank's expert window and the replicated rest only, + and each file's cached pages are dropped once its sources are copied. Mapping every + shard file on every rank and reading through the mappings instead would fault + the expert windows of a node's four ranks -- 539 GiB, the whole expert stack, at 4 + ranks; half of it on each of the two nodes at 8 -- and the replicated rest into that + node's host memory. + Last, each row's `layout` puts the expert stacks into the MoE runner's layout, so a + second `load` starts again from the stored bytes. + """ + core = model.model + manifest = self.manifest(core) + consumed: set = set() + + def fill(param, ckpt, entry, index, transform) -> None: + dst = param.data if index is None else param.data[index] + if isinstance(ckpt, tuple): + src = transform( + tuple(_materialize(e).to(dst.device, non_blocking=True) for e in entry), core + ) + else: + src = _materialize(entry) + if transform is not None: + src = transform(src.to(dst.device, non_blocking=True), core) + assert tuple(dst.shape) == tuple(src.shape), (ckpt, tuple(dst.shape), tuple(src.shape)) + assert src.dtype == dst.dtype, (ckpt, src.dtype, dst.dtype) + dst.copy_(src, non_blocking=True) + consumed.update(_keys(ckpt)) + + assert set(manifest.keys()) == set(core.w.keys()), ( + "manifest/parameter drift", + sorted(set(manifest.keys()) ^ set(core.w.keys()))[:12], + ) + d = self.dims(core) + plan = [ + (core.w[param_key], ckpt, index, transform) + for param_key, sources in manifest.items() + for ckpt, index, transform in sources + ] + # Under attention DP the shell builds lm_head whole, `[vocab, hidden]` on every rank: + # the checkpoint's whole `output.weight`. + plan.append((model.lm_head.weight, "output.weight", None, None)) + scalar_keys = _expert_scalar_keys(d) + for key in (*(k for row in plan for k in _keys(row[1])), *scalar_keys): + assert key in weights, f"checkpoint key missing: {key}" + if shards is None: + shards = _shard_files(core, weights) + scalars = _read_scalars(scalar_keys, weights, shards) + _check_expert_scalars(scalars, d) + + streamed = [] + for param, ckpt, index, transform in plan: + if all(k in scalars for k in _keys(ckpt)): + entry = tuple(scalars[k] for k in _keys(ckpt)) + entry = entry if isinstance(ckpt, tuple) else entry[0] + fill(param, ckpt, entry, index, transform) + else: + assert isinstance(ckpt, str), f"a key tuple of other than expert scalars: {ckpt}" + streamed.append((param, ckpt, index, transform)) + if shards is None: + for param, ckpt, index, transform in streamed: + fill(param, ckpt, weights[ckpt], index, transform) + else: + directory, weight_map = shards + by_file: dict[str, list] = collections.defaultdict(list) + for row in streamed: + by_file[weight_map[row[1]]].append(row) + on_device = model.lm_head.weight.device.type == "cuda" + for name in sorted(by_file): + with safe_open(str(directory / name), framework="pt", device="cpu") as handle: + for param, ckpt, index, transform in by_file[name]: + fill(param, ckpt, handle.get_slice(ckpt), index, transform) + if on_device: + torch.cuda.synchronize() + _drop_page_cache(directory / name) + # Every source has landed: rearrange each expert in place, one at a time. + for entry in self._entries(d): + if entry.layout is None: + continue + for key, _, _ in self._targets(entry, d): + param = core.w[key].data + for slot in range(param.shape[0]): + param[slot].copy_(entry.layout(param[slot])) + shell = {n for n, _ in model.named_parameters() if not n.startswith("model.w.")} + assert shell == {"lm_head.weight"}, f"unfed shell parameters: {sorted(shell)}" + + if model.lm_head.weight.device.type == "cuda": + torch.cuda.synchronize() + leftover = set(weights.keys()) - consumed + expected = self.expected_unconsumed(core, weights) + assert leftover == set(expected), ( + "checkpoint coverage: unexpected " + f"{sorted(leftover - set(expected))[:8]}, missed {sorted(set(expected) - leftover)[:8]}" + ) + not_loaded = collections.Counter(expected.values()) + logger.info( + f"modeling_v2 weights: {len(consumed)} checkpoint tensors loaded " + f"({'streamed per shard file' if shards else 'from the given dict'}), " + f"{len(scalars)} expert scalars checked; not loaded: " + + "; ".join(f"{n} x {reason}" for reason, n in sorted(not_loaded.items())) + ) + + def dims(self, core) -> SimpleNamespace: + cfg = core.model_config.pretrained_config + vis = getattr(cfg, "vision_config", None) + mapping = core.model_config.mapping + spec = getattr(core.model_config, "spec_config", None) + draft = getattr(spec, "_draft_model_config", None) or {} + local_experts = cfg.n_routed_experts // mapping.moe_ep_size + return SimpleNamespace( + draft_layers=int(draft.get("n_layers", 0)), + draft_rank=int((draft.get("dspark") or {}).get("markov_rank", 0)), + draft_depth=int(draft.get("depth", 0)), + draft_capture_layers=len(spec.target_layer_ids) if draft else 0, + draft_tp_rank=0, + num_layers=cfg.num_hidden_layers, + hidden=cfg.hidden_size, + heads=cfg.num_attention_heads, + nope=cfg.qk_nope_head_dim, + rope_dim=cfg.qk_rope_head_dim, + qk_dim=cfg.qk_nope_head_dim + cfg.qk_rope_head_dim, + v_dim=cfg.v_head_dim, + kv_lora=cfg.kv_lora_rank, + lat_dim=cfg.kv_lora_rank + cfg.qk_rope_head_dim, + nope_layer=tuple(bool(f) for f in cfg.nope_layer_flags), + kv_source=tuple(int(s) for s in cfg.kv_source_layers), + full_indexer=tuple(t == "full" for t in cfg.index_sharing), + idx_heads=cfg.index_n_heads, + idx_dim=cfg.index_head_dim, + dense_layers=cfg.first_k_dense_replace, + num_experts=cfg.n_routed_experts, + moe_inter=cfg.moe_intermediate_size, + local_experts=local_experts, + expert_offset=local_experts * mapping.moe_ep_rank, + shared_inter=cfg.moe_intermediate_size * cfg.n_shared_experts, + dense_inter=cfg.intermediate_size, + vocab=cfg.vocab_size, + dtype=core.model_config.torch_dtype, + # The vision tower (replicated; absent without a vision encoder). + has_vision=vis is not None, + v_layers=vis.num_hidden_layers if vis is not None else 0, + v_hidden=vis.hidden_size if vis is not None else 0, + v_heads=vis.num_attention_heads if vis is not None else 0, + v_head_dim=vis.hidden_size // vis.num_attention_heads if vis is not None else 0, + v_inter=vis.intermediate_size if vis is not None else 0, + v_channels=vis.num_channels if vis is not None else 0, + v_patch=vis.patch_size if vis is not None else 0, + v_merge=vis.spatial_merge_size if vis is not None else 0, + v_theta=float(vis.rope_theta) if vis is not None else 0.0, + image_token_id=vis.image_token_id if vis is not None else None, + ) + + +MODEL_WEIGHTS = MistralLarge4DepWeights() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/__init__.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/__init__.py new file mode 100644 index 000000000000..95c1f4b15eee --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Mistral-Large-4-1T-A52B-NVFP4 / sm_103 / tep, the language model only.""" diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/modeling.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/modeling.py new file mode 100644 index 000000000000..fc8fca41f258 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/modeling.py @@ -0,0 +1,2511 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""ModelingV2 target: mistral-large-4-nvfp4 / sm_103 / tep, native text and image inference. + +72 layers, hidden 6144, 64 query heads (16 per rank at tep4, 8 at tep8). Attention is MLA +without a q-LoRA (`wq` straight to 64 x 128), a 512-wide latent plus a 64-wide rope key, and +DeepSeek Sparse Attention: an indexer on the 44 `index_sharing == 'full'` layers selects 2048 +positions per query row and each 'shared' layer reuses the selection of the most recent full +layer. Every fourth layer (`i % 4 == 3`) takes no rope, for MLA and for its indexer; the NoPE +layers 39, 43, ..., 71 own no latent and attend over layer 35's. Queries are scaled by SSMax, +`s * log(pos + 1) + b` per head, on MLA and on the indexer. Layer 0 is a dense SwiGLU MLP, +layers 1-71 MoE (128 experts, softmax top-4 renormalised, one shared expert). The final logits +are soft-capped at `softmax_tanh_gating`. + +The checkpoint is ModelOpt `MIXED_PRECISION`. The routed experts are NVFP4 W4A4: e2m1 weights +with one e4m3 scale per 16 inputs and a global scale per expert, and activations quantized to +NVFP4 under one static global scale per layer and projection. `wq`, `wo`, the indexer's `wq_b` +and the dense and shared MLPs are FP8 e4m3 with 128x128 power-of-two block scales and dynamic +1x128 activation scales. The rest is bf16. The latent KV pool is fp8_ds_mla (656 B per token +and owner layer: 512 e4m3 codes with one power-of-two fp32 scale per 128, and the 64-wide rope +key in bf16), owned by `MistralLarge4CacheManager`; the indexer key pool is fp8 with one scale +per key. + +`tep` shards the attention heads, the dense and shared MLPs and `lm_head` over every rank and +gives each rank `n_routed_experts / moe_ep_size` of the 128 routed experts: four ways and 32 +experts on one node of four, or eight ways and 16 experts over two nodes of four (tep8), every +width read off the mapping. The latent projection, the indexer and the router are replicated. +Every layer closes twice on the all-reduce over every rank fused with the residual add and the +next RMSNorm. + +The absorbed form throughout: `fused_q = [q_nope @ W_UK | rope(q_pe)]` (576 wide) against the +pooled `[c_kv | k_pe]` latent rows, then `@ W_UV`. Context rows attend over the selection (the +engine's causal index on a step it skipped the indexer) with FlashMLA's bf16 sparse prefill +over the requests' latent rows dequantized out of the pool, so they read the FP8-rounded latent +as the reference does; generation rows run sparse fp8 decode over the pool itself, at this +rank's 16 (tep4) or 8 (tep8) heads at every max_seq_len: cute_dsl_ds_mla_sparse_decode, +FlashMLA's sparse decode arithmetic without the head padding. The prefill kernel, and the +sparse decode a context row past the held workspace runs (FlashMLA's), take 64 heads: this +rank's go first in a 64-head query whose other heads are zero; heads do not interact, so the +zero heads change nothing the live ones compute. `_GEN_ATTN_FLASH_MLA` (for A/B verification) +runs the generation rows on FlashMLA's 64-head decode too, the same padding. + +No path is selected by the serving configuration: each step chooses from its own sizes, and +every buffer is sized from the engine's quantities (the attention metadata's max_num_tokens, +max_num_requests, max_seq_len and runtime_features). The context rows take one of two paths, +chosen per step by the step's context KV (every context request's cached and new rows): while +it fits the held workspace, the sparse prefill over those rows dequantized into it; past it, +each context row as a one-row sequence of the sparse fp8 decode over the pool itself, as +generation rows always run. The held workspace, allocated on the first context step, has room +for the most context KV one step can attend, twice over, up to 2**19 rows +(`_ctx_workspace_rows`): a step whose context KV fits twice keeps layer 35's rows beside its +own for the NoPE layers attending over that latent, a longer one gathers them again for each +of those layers. An indexer chunk the engine packs past `max_seq_len` keys is scored request +by request, and the indexer's logits are tiled under a budget from the engine's max_num_tokens +and max_seq_len (`_prefill_logits_budget`), each tile scored with a split-KV schedule built once +per step (every SM busy whatever the tile's row count) and selected with the self-sampling GVR +top-k (`_select_ctx`). + +Under chunked prefill the engine splits a step's context queries across every rank (four, or +eight over two nodes) from `q_split_threshold` rows on: each rank selects for its share, and the +shares are all-gathered into every rank's selection buffer before a layer reads it. +""" + +from __future__ import annotations + +import itertools +import math +import os +import time +from dataclasses import replace +from types import SimpleNamespace +from typing import NamedTuple, Optional, cast + +import torch +from torch import nn +from transformers import PretrainedConfig + +from tensorrt_llm._torch.attention.backends.interface import AttentionMetadata +from tensorrt_llm._torch.attention.backends.sparse.params import ( + get_indexer_mqa_logits_workspace_bytes, +) +from tensorrt_llm._torch.attention.backends.trtllm import TrtllmAttentionMetadata +from tensorrt_llm._torch.model_config import ModelConfig +from tensorrt_llm._torch.modeling_v2._router_index import step_contract_enabled +from tensorrt_llm._torch.modeling_v2.catalog.activation.flashinfer_silu_and_mul import ( + flashinfer_silu_and_mul, +) +from tensorrt_llm._torch.modeling_v2.catalog.attention.thop_attention import thop_attention +from tensorrt_llm._torch.modeling_v2.catalog.comm.allgather import allgather +from tensorrt_llm._torch.modeling_v2.catalog.gemm.bmm_out import bmm_out +from tensorrt_llm._torch.modeling_v2.catalog.gemm.cublas_mm import cublas_mm +from tensorrt_llm._torch.modeling_v2.catalog.moe.fp4_block_scale_moe_runner import ( + fp4_block_scale_moe_runner, +) +from tensorrt_llm._torch.modeling_v2.catalog.norm.flashinfer_fused_add_rmsnorm import ( + flashinfer_fused_add_rmsnorm, +) +from tensorrt_llm._torch.modeling_v2.catalog.norm.flashinfer_rmsnorm import flashinfer_rmsnorm +from tensorrt_llm._torch.modeling_v2.catalog.quantization.fp4_quantize import fp4_quantize +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.activation.gelu_erf import gelu_erf +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.activation.tanh_softcap import ( + tanh_softcap, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.convert_req_index_to_global import ( # noqa: E501 + convert_req_index_to_global, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_ds_mla_sparse_decode import ( # noqa: E501 + build as build_ds_mla_decode, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_ds_mla_sparse_decode import ( # noqa: E501 + cute_dsl_ds_mla_sparse_decode, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_fp8_paged_mqa_logits import ( # noqa: E501 + cute_dsl_fp8_paged_mqa_logits, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_gvr_topk_prefill import ( # noqa: E501 + cute_dsl_gvr_topk_prefill, + gvr_warmup, + gvr_workspace, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_indexer_k_cache_update import ( # noqa: E501 + build as build_ix_k_cache, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_indexer_k_cache_update import ( # noqa: E501 + cute_dsl_indexer_k_cache_update, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_indexer_q_side_fused import ( # noqa: E501 + build as build_ix_q_side, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_indexer_q_side_fused import ( # noqa: E501 + cute_dsl_indexer_q_side_fused, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_radix_topk_decode import ( # noqa: E501 + build as build_ix_topk_decode, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_radix_topk_decode import ( # noqa: E501 + cute_dsl_radix_topk_decode, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ds_mla_dequant_gather import ( + ds_mla_dequant_gather, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ds_mla_rope_append import ( + ds_mla_rope_append, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.flash_mla_sparse_decode import ( + flash_mla_sparse_decode, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.flash_mla_sparse_prefill import ( + flash_mla_sparse_prefill, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.fp8_mqa_logits import ( + fp8_mqa_logits, + mqa_logits_schedule, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.indexer_k_cache_gather_op import ( # noqa: E501 + indexer_k_cache_gather_op, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.indexer_topk_decode import ( + indexer_topk_decode, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ssmax_indexer_q_side_fused import ( # noqa: E501 + ssmax_indexer_q_side_fused, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ssmax_scaling import ( + ssmax_scaling, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ssmax_scaling_fused_q import ( + ssmax_scaling_fused_q, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.comm.cute_dsl_allreduce_add_rmsnorm import ( # noqa: E501 + build_workspace as build_lamport_workspace, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.comm.cute_dsl_allreduce_add_rmsnorm import ( # noqa: E501 + cute_dsl_allreduce_add_rmsnorm, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.gemm.fp8_prequantized_swap_ab_gemm import ( + fp8_prequantized_swap_ab_gemm, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.moe.renorm_moe_routing_op import ( + renorm_moe_routing_op, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.norm.layer_norm import layer_norm +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.quantization.fp8_quantize_1x128_packed_ue8m0 import ( # noqa: E501 + fp8_quantize_1x128_packed_ue8m0, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.vision.pixtral_rope_2d import ( + pixtral_rope_2d, +) +from tensorrt_llm._torch.models.modeling_multimodal_utils import ( + find_input_mm_embeds, + get_multimodal_embeddings, +) +from tensorrt_llm._torch.models.modeling_utils import DecoderModel, DecoderModelForCausalLM +from tensorrt_llm._torch.pyexecutor.guided_decoder import CapturableGuidedDecoder +from tensorrt_llm._torch.speculative import get_spec_worker +from tensorrt_llm._utils import mpi_disabled +from tensorrt_llm.flash_mla import get_mla_metadata +from tensorrt_llm.inputs.multimodal import MultimodalParams +from tensorrt_llm.logger import logger +from tensorrt_llm.models.modeling_utils import QuantConfig + +from . import weights as _weights + +#: eps 1e-5, the reference's value; params.json's `vision_encoder` states none. +_VISION_EPS = 1e-5 + +#: The thop attention arguments the vision tower's call holds fixed: the no-KV-cache +#: mode (an encoder: the three cache-addressing arguments None), one context sequence +#: per image, the fused `[q | k | v]` rows already rotated (no rope in the op), +#: bidirectional within each sequence (mask_type 0), softmax scale head_dim^-0.5. +_VISION_ATTN = dict( + k=None, + v=None, + output_sf=None, + max_context_q_len_override=None, + kv_cache_block_offsets=None, + host_kv_cache_pool_pointers=None, + host_kv_cache_pool_mapping=None, + cache_indirection=None, + kv_scale_orig_quant=None, + kv_scale_quant_orig=None, + out_scale=None, + rotary_inv_freq=None, + rotary_cos_sin=None, + latent_cache=None, + q_pe=None, + block_ids_per_seq=None, + attention_sinks=None, + is_fused_qkv=True, + update_kv_cache=True, + predicted_tokens_per_seq=1, + local_layer_idx=0, + tokens_per_block=None, + beam_width=1, + mask_type=0, + quant_mode=0, + q_scaling=1.0, + position_embedding_type=0, + rope_dim=0, + rope_base=10000.0, + rope_scale_type=0, + rope_scale=1.0, + rope_short_m_scale=1.0, + rope_long_m_scale=1.0, + rope_max_positions=1024, + rope_original_max_positions=1024, + use_paged_context_fmha=False, + attention_input_type=0, + is_mla_enable=False, + chunked_prefill_buffer_batch_size=1, + q_lora_rank=None, + kv_lora_rank=None, + qk_nope_head_dim=None, + qk_rope_head_dim=None, + v_head_dim=None, + rope_append=None, + mrope_rotary_cos_sin=None, + mrope_position_deltas=None, + helix_position_offsets=None, + helix_is_inactive_rank=None, + attention_chunk_size=None, + softmax_stats_tensor=None, + is_spec_decoding_enabled=False, + use_spec_decoding=False, + is_spec_dec_tree=False, + spec_decoding_generation_lengths=None, + spec_decoding_position_offsets=None, + spec_decoding_packed_mask=None, + spec_decoding_bl_tree_mask_offset=None, + spec_decoding_bl_tree_mask=None, + spec_bl_tree_first_sparse_mask_offset_kv=None, + sparse_kv_indices=None, + sparse_kv_offsets=None, + sparse_attn_indices=None, + sparse_attn_offsets=None, + sparse_attn_indices_block_size=0, +) + +# The latent pool layout this target addresses (`MistralLarge4CacheManager.latent_layout`). +_LATENT_LAYOUT = "fp8_ds_mla" +# The query heads FlashMLA's sm100 sparse kernels run; this rank's heads go first. +_FLASH_MLA_HEADS = 64 +# Generation rows' sparse MLA. False: cute_dsl_ds_mla_sparse_decode at this rank's heads, the +# shipped path. True: flash_mla_sparse_decode over the 64-head padded query, the kernel this +# target ran before (bit-identical where the two split a row alike; see the entry's note) -- +# a switch for A/B verification, not a configuration. It sets the core's +# `gen_attn_flash_mla`, which every generation-row site reads; the native entry's cells cover +# the 32K and the 1M configs (key lengths to 1048576, the 1M pools' slots past 2**31 bytes). +_GEN_ATTN_FLASH_MLA = False +# The pre-routed MoE call: the runner's routing method is inert once ids and weights are +# given; 1 (renormalize) is the value its cells drive. +_MOE_ROUTING_METHOD = 1 +# The routed experts' SwiGLU, the runner's `act_type` 0. +_MOE_ACT_SWIGLU = 0 +# NVFP4: one e4m3 scale per 16 activations, laid out row-major (linear), the order the MoE +# runner reads; the 128x4-swizzled order is another buffer of the same length at 128-row +# multiples, and silently wrong there. +_NVFP4_SF_VEC = 16 +_NVFP4_SF_LINEAR = False +# fp8_mqa_logits allocates a `[q_rows, kv_rows]` fp32 score matrix per call; the context +# queries are tiled to keep it under the engine's budget (`_prefill_logits_budget`). Context +# steps are not captured. +# capacity: fixed: the budget's ceiling, one 1 GiB fp32 tile (256 rows at 1048576 keys). +# Mainline's default reachable tile (2**31 logits) would take 8 GiB from the KV pool. +_PREFILL_LOGITS_CEILING = 1 << 28 +# The context path's crossover, chosen once per step on host ints, and the held context +# workspace's ceiling (`_ctx_workspace_rows`). A step whose context KV (every context request's +# cached and new rows) fits the held workspace has it gathered, dequantized, into it (bf16 +# [rows, 576]) for FlashMLA's sparse prefill; a longer one attends its context rows straight off +# the pool with the sparse decode kernel, whose cost does not grow with the prefix as the +# gather's does. +# capacity: fixed: 2**19 rows, 576 MiB, the gather-vs-decode crossover. +_CTX_WORKSPACE_ROWS = 1 << 19 +# The most context rows one sparse-decode call takes on that path. +# capacity: fixed: FlashMLA's decode scheduler, which exits the process, raising nothing, past +# 11622 one-row sequences on sm_103; not an engine limit. +_CTX_DECODE_ROWS = 8192 +# The decode indexer's top-k: True runs cute_dsl_radix_topk_decode, one launch that selects the +# trtllm indexer_topk_decode op's set (up to exact ties), False that op itself. The CuTe DSL +# selector serves scorer rows from 4096 columns (where the op splits each row) to 65536 (one +# CTA scans a whole row), the widths its cells cover; the scorer's column count is the engine's +# max_seq_len, so an engine outside that range keeps the op (`_derive_first_step`). +_TOPK_DECODE_CUTE_DSL = True +_TOPK_DECODE_COLUMNS = (4096, 65536) +# The attention block's glue as separate ops instead of the fused entries, for old-versus-new +# verification (both compute the same bits; the fused entries' tests hold each pair equal): +# ssmax_scaling plus the rope tail's copy or zero fill for ssmax_scaling_fused_q; +# ssmax_scaling, the head weights' fp32 copy and cute_dsl_indexer_q_side_fused for +# ssmax_indexer_q_side_fused; a fresh RMSNorm and torch.cat for the in-place latent; the copy +# out of FlashMLA's 64-head output for the in-place read. Set +# TRTLLM_MODELING_V2_UNFUSED_GLUE=1 in every rank's environment at launch; off by default. +# It sets the core's `unfused_glue`, which every site reads. +_UNFUSED_GLUE = os.environ.get("TRTLLM_MODELING_V2_UNFUSED_GLUE", "0") == "1" + + +def _prefill_tiles( + token_start: int, token_end: int, n_k: int, budget: int +) -> list[tuple[int, int]]: + """The query-row tiles `[lo, hi)` of one indexer prefill chunk over `n_k` keys under a + logits `budget`: `floor(budget / n_k)` rows each (at least one), the last one short.""" + tile = max(1, budget // n_k) + return [(lo, min(lo + tile, token_end)) for lo in range(token_start, token_end, tile)] + + +def _prefill_logits_budget(max_num_tokens: int, max_seq_len: int) -> int: + """The most fp32 logits one indexer prefill tile holds, from the engine's max_num_tokens and + max_seq_len: mainline's reachable MQA-logits tile (`get_indexer_mqa_logits_workspace_bytes`, + 4 bytes a logit; TLLM_INDEXER_MQA_LOGITS_ELEM_BUDGET lowers it), within + `_PREFILL_LOGITS_CEILING`, and within max_seq_len / 64 query rows at max_seq_len keys. + + The last bound grows the tile with the engine's longest request, which it scores in at + least 64 tiles: 2**24 logits (512 rows of 32768 keys) at max_seq_len 32768, the ceiling + (256 rows of 1048576 keys) from max_seq_len 131072 on. Those are the budgets the 32768 and + 1048576 serving configs were certified at, so their tiles, and the fp8_mqa_logits and + prefill top-k cells that cover them, stay as they were.""" + reachable = get_indexer_mqa_logits_workspace_bytes(max_num_tokens, max_seq_len) // 4 + return min(reachable, _PREFILL_LOGITS_CEILING, max_seq_len * (max_seq_len >> 6)) + + +def _most_ctx_kv(md) -> int: + """The most context KV (cached and new rows over a step's context requests) one step can + attend, from the attention metadata alone. `runtime_features` is built once by the executor + for the service (chunked prefill from enable_chunked_prefill, cache reuse from the + KvCacheConfig's enable_block_reuse). With neither, every context row is new: + max_num_tokens. Otherwise mainline's never-stall worst case for a workspace chunked prefill + does not bound (`get_mla_context_workspace_kv_len_cap` without an + fp8_context_mla_kv_len_cap override; the gather reads every attended row, cached ones + included): min(max_num_requests, max_num_tokens) * max_seq_len. The target does not read + fp8_context_mla_kv_len_cap: the KvCacheConfig is not reachable from the model, and for these + targets the engine installs no admission cap that would bound a step to it.""" + features = md.runtime_features + if not (features.chunked_prefill or features.cache_reuse): + return md.max_num_tokens + return min(md.max_num_requests, md.max_num_tokens) * md.max_seq_len + + +def _ctx_workspace_rows(md) -> int: + """The held context workspace's rows: the most context KV one step can attend + (`_most_ctx_kv`) twice over, a step's rows and a shared owner's (layer 35's) kept beside + them, up to `_CTX_WORKSPACE_ROWS`. At max_num_tokens 32768 with neither chunked prefill nor + block reuse that is 65536 rows, the two `[32768, 576]` gathers a step there makes; with either, + at 16 requests of max_seq_len 32768 or more, 2**19.""" + return min(2 * _most_ctx_kv(md), _CTX_WORKSPACE_ROWS) + + +def _step_common(md: TrtllmAttentionMetadata, indexer_layers: list[int]) -> dict: + """The batch state both targets read: KV lengths and pool addressing for the MLA ops, the + indexer's key pools and slot mappings, the top-k conversion tables and the decode scorer's + schedule. The indexer pool views are taken per step: the engine replaces its KV cache + manager during boot, and a view of the old one writes into a dead allocation.""" + mgr = md.kv_cache_manager + latent_pool = mgr.get_latent_pool() + return dict( + num_seqs=md.num_seqs, + num_tokens=md.num_tokens, + kv_lens_cuda=md.kv_lens_cuda_runtime, + kv_lens_host=md.kv_lens_runtime, + prompt_lens_host=md.prompt_lens_cpu_runtime, + kv_cache_block_offsets=md.kv_cache_block_offsets, + host_kv_cache_pool_pointers=md.host_kv_cache_pool_pointers, + host_kv_cache_pool_mapping=md.host_kv_cache_pool_mapping, + tokens_per_block=md.tokens_per_block, + max_seq_len=md.max_seq_len, + # [blocks, owners, tokens_per_block, 656] bytes, and the same bytes as FlashMLA's + # pages: page `block * owners + row`, slot `page * tokens_per_block + offset`. + latent_pool=latent_pool, + latent_kv=latent_pool.view(-1, latent_pool.shape[2], 1, latent_pool.shape[3]), + ix_pools={i: mgr.get_indexer_k_cache_buffers(i) for i in indexer_layers}, + block_table=md.block_table, + req_idx_per_token=md.req_idx_per_token, + slot_mapping_fp8=md.slot_mapping_fp8, + slot_mapping_scale=md.slot_mapping_scale, + skip_gen=bool(md.skip_indexer_for_gen_reqs), + # Allocated only while the engine's short-sequence indexer skip is on; the two skip + # flags are False whenever it is off. + topk_indices_buffer=md.topk_indices_buffer if md.enable_indexer_skip else None, + indexer_k_cache_block_offsets=md.indexer_k_cache_block_offsets, + scheduler_metadata_buffer=md.scheduler_metadata_buffer, + radix_aux_indices=md.radix_aux_indices, + radix_aux_logits=md.radix_aux_logits, + ) + + +def _ctx_kv_offsets(md: TrtllmAttentionMetadata, num_contexts: int) -> dict: + """The context rope/append op's per-request offsets: int64 prefix sums of the cached and + of the total (cached + new) lengths over the context requests, and the longest new run. + + The engine keeps them only under `enable_context_mla_with_cached_kv` (KV reuse or chunked + prefill). Without either, no context request has a cached prefix, so the cached sums are + zero and the totals are the new lengths: the int32 prefix sum of `seq_lens_cuda` the + metadata always builds, widened to the op's int64. + """ + if md.enable_context_mla_with_cached_kv: + return dict( + cu_ctx_cached=md.ctx_cached_token_indptr[: num_contexts + 1], + cu_ctx_kv=md.ctx_kv_indptr[: num_contexts + 1], + max_ctx_new=int(md.max_ctx_seq_len), + ctx_cached_tokens=int(md.num_ctx_cached_tokens), + ) + cu = md.mla_prepare_ctx_cu_seqlens() + return dict( + cu_ctx_cached=torch.zeros(num_contexts + 1, dtype=torch.int64, device=cu.device), + cu_ctx_kv=cu.to(torch.int64), + max_ctx_new=int(md.seq_lens[:num_contexts].max()), + ctx_cached_tokens=0, + ) + + +def _ctx_host_offsets(md: TrtllmAttentionMetadata, num_contexts: int) -> dict: + """The host lists over the context requests: the prefix sums of the new and of the total + (cached + new) lengths, which place each context request's rows in the step and its keys in + the indexer's full-KV gather mappings (`_indexer_blocks`). + + The engine's host copies under `enable_context_mla_with_cached_kv`; without it no request + has a cached prefix, so both are the prefix sum of the host `seq_lens`. Every source is a + host tensor the metadata fills on the host, so taking them each context step syncs nothing. + """ + if md.enable_context_mla_with_cached_kv: + return dict( + host_cu_ctx_new=md.host_ctx_uncached_token_indptr[: num_contexts + 1].tolist(), + host_cu_ctx_kv=md.host_ctx_kv_indptr[: num_contexts + 1].tolist(), + ) + host_cu = [0, *itertools.accumulate(md.seq_lens[:num_contexts].tolist())] + return dict(host_cu_ctx_new=host_cu, host_cu_ctx_kv=host_cu) + + +def _indexer_blocks(chunks, step: dict) -> list: + """The context indexer's scoring blocks for one step, in the engine's chunk order: rows + `[token_start, token_end)` of the context rows scored against keys + `[k_token_start, k_token_end)` of the full-KV gather mappings, each row's window + (`cu_seqlen_ks` / `cu_seqlen_ke`) counted from the block's first key. + + An engine prefill chunk is one block, as planned, while it holds at most `max_seq_len` keys, + the most one request holds. The engine packs consecutive context requests into one chunk + while (summed new tokens) x (summed KV) stays within its packed-logits budget (2^28), so + short last chunks of long requests pack past max_seq_len: a 100-token tail of a + 1048559-token request beside a 150-token prompt is one chunk of 1048709 keys, wider than + the key scorer's and the prefill top-k's cells. Such a chunk is cut at its request + boundaries (`_ctx_host_offsets`), each request's rows against its own keys with their + windows re-based to its first key: a row's window lies inside its own request's keys, so + each request selects as it would alone and no block holds more than max_seq_len keys. + """ + tc = step["num_ctx_tokens"] + if chunks is None: + # One chunk over every context row: the engine plans chunks only past its chunk + # size, or under KV reuse or chunked prefill. + chunks = [ + SimpleNamespace( + token_start=0, + token_end=tc, + k_token_start=0, + k_token_end=step["num_ctx_kv_tokens"], + cu_seqlen_ks=step["cu_seqlen_ks"][:tc], + cu_seqlen_ke=step["cu_seqlen_ke"][:tc], + ) + ] + max_keys = step["max_seq_len"] + cu_new, cu_kv = step["host_cu_ctx_new"], step["host_cu_ctx_kv"] + blocks = [] + for chunk in chunks: + if chunk.k_token_end - chunk.k_token_start <= max_keys: + blocks.append(chunk) + continue + for r in range(len(cu_new) - 1): + lo, hi = max(chunk.token_start, cu_new[r]), min(chunk.token_end, cu_new[r + 1]) + if lo >= hi: + continue + k_lo, k_hi = max(chunk.k_token_start, cu_kv[r]), min(chunk.k_token_end, cu_kv[r + 1]) + assert k_hi - k_lo <= max_keys, ( + f"context request {r} holds keys [{k_lo}, {k_hi}) of an indexer chunk, past " + f"max_seq_len {max_keys}" + ) + shift = k_lo - chunk.k_token_start + rows = slice(lo - chunk.token_start, hi - chunk.token_start) + blocks.append( + SimpleNamespace( + token_start=lo, + token_end=hi, + k_token_start=k_lo, + k_token_end=k_hi, + cu_seqlen_ks=torch.sub(chunk.cu_seqlen_ks[rows], shift), + cu_seqlen_ke=torch.sub(chunk.cu_seqlen_ke[rows], shift), + ) + ) + return blocks + + +def _yarn_lerope_inv_freq(cfg, alpha: torch.Tensor) -> torch.Tensor: + """The rotary inverse frequencies, fp32, as the reference implementation builds + them: YaRN's blend of the extrapolated and the interpolated frequency, both scaled by + LeRoPE's `exp(alpha)`, over a ramp taken on the frequencies without LeRoPE.""" + rope = cfg.rope_parameters + dim = cfg.qk_rope_head_dim + theta = float(rope["rope_theta"]) + factor = float(rope["factor"]) + original = float(rope["original_max_position_embeddings"]) + pos_freqs = theta ** (torch.arange(0, dim, 2, dtype=torch.float32) / dim) + lerope = torch.exp(alpha.detach().to(device="cpu", dtype=torch.float32)) + extrapolated = 1.0 / pos_freqs * lerope + interpolated = 1.0 / (factor * pos_freqs) * lerope + + def correction_dim(rotations: float) -> float: + return dim * math.log(original / (rotations * 2 * math.pi)) / (2 * math.log(theta)) + + low = max(math.floor(correction_dim(float(rope.get("beta_fast", 32)))), 0) + high = min(math.ceil(correction_dim(float(rope.get("beta_slow", 1)))), dim - 1) + if low == high: + high += 0.001 + ramp = ((torch.arange(dim // 2, dtype=torch.float32) - low) / (high - low)).clamp(0, 1) + extrapolation = 1 - ramp + return interpolated * (1 - extrapolation) + extrapolated * extrapolation + + +def _rope_tables(cfg, alpha: torch.Tensor, positions: int, dtype, device) -> SimpleNamespace: + """The tables the rope consumers read, built from one fp32 angle grid and rounded to the + activation dtype first, as the reference's cos/sin cache is held. + + `indexer` is `[rows, 64]`, `[cos(32) | sin(32)]`: the layout the indexer q side and + `ds_mla_rope_append` (the MLA query tail and the latent's rope key) read; the indexer key + side reads the same storage as `[rows, 2, 32]`. `identity` (cos 1, sin 0) is the table of a + NoPE indexer layer, whose kernels have no no-rope mode. Every row depends only on its own + index, so a longer table is the same table with more rows. + """ + inv = _yarn_lerope_inv_freq(cfg, alpha) + t = torch.arange(positions, dtype=torch.float32) + angle = torch.einsum("i,j -> ij", t, inv) + cos = angle.cos().to(dtype).float() + sin = angle.sin().to(dtype).float() + identity = torch.cat([torch.ones_like(cos), torch.zeros_like(sin)], dim=1) + return SimpleNamespace( + rows=positions, + indexer=torch.cat([cos, sin], dim=1).to(device).contiguous(), + identity=identity.to(device).contiguous(), + ) + + +def _moe_scalars(isc1, isc2, ws2_13, ws2_2) -> SimpleNamespace: + """The NVFP4 MoE's call scalars from the checkpoint's per-tensor ones, fp32. + + ModelOpt stores `input_scale` and `weight_scale_2` as the reciprocals of the global scales a + quantizer divides by (`amax / (448 * 6)`), one `isc1` / `isc2` per layer and one + `ws2_13[e]` / `ws2_2[e]` per local expert. So the activation quantizer's global scale is + `1 / isc1`; FC1's accumulator dequantizes by `isc1 * ws2_13[e]` (the gate half, + `output1_scale_gate_scalar`) and, because the FC1 epilogue requantizes the SwiGLU output to + NVFP4 under `1 / isc2`, the linear half by `isc1 * ws2_13[e] / isc2` + (`output1_scale_scalar`); FC2's accumulator dequantizes by `isc2 * ws2_2[e]` + (`output2_scale_scalar`). The reference's TRT-LLM-Gen path folds the same three + (`g1_alphas`, `g1_scale_c`, `g2_alphas`).""" + gate1 = (isc1 * ws2_13).contiguous() + return SimpleNamespace( + gscale=(1.0 / isc1).contiguous(), + out1=(gate1 / isc2).contiguous(), + out1_gate=gate1, + out2=(isc2 * ws2_2).contiguous(), + ) + + +def _layer_views(core: "MistralLarge4TepModelingV2") -> list[SimpleNamespace]: + """Every layer's operands, as the views the entries read: GEMM operands transposed (`.t()`, + the column-major `[K, N]` cuBLAS and the FP8 GEMM read), the FP8 scale storage transposed to + the MN-major operand, `wkv_b` split into the absorption pair, and the NVFP4 MoE's call + scalars (`_moe_scalars`). `next_norm` is the next layer's input norm, the final norm after + the last layer. A sharer's absorption pair is its own `wkv_b`; its latent is its source's.""" + w = core.w + heads, v_dim, kv_lora = core.heads, core.v_dim, core.kv_lora + layers = [] + for i in range(core.num_layers): + p = f"l{i}_" + nope = core.qk_dim if core.nope_layer[i] else core.nope + kvb = w[p + "wkv_b"] + hn = heads * nope + owner = core.kv_source[i] == i + ix = None + if core.full_indexer[i]: + ix = SimpleNamespace( + q_norm_w=w[p + "ix_q_norm_w"], + q_norm_b=w[p + "ix_q_norm_b"], + wq_b=w[p + "ix_wq_b"], + wq_b_s=w[p + "ix_wq_b_s"].t(), + wk_wp=w[p + "ix_wk_wp"].t(), + k_norm_w=w[p + "ix_k_norm_w"], + k_norm_b=w[p + "ix_k_norm_b"], + ssmax_s=w[p + "ix_ssmax_s"], + ssmax_b=w[p + "ix_ssmax_b"], + rope=not core.indexer_nope[i], + ) + moe = None + if i >= core.dense_layers: + moe = SimpleNamespace( + router=w[p + "router"].t(), + fc1_w=w[p + "fc1_w"], + fc1_s=w[p + "fc1_s"], + fc2_w=w[p + "fc2_w"], + fc2_s=w[p + "fc2_s"], + **vars(_moe_scalars(w[p + "isc1"], w[p + "isc2"], w[p + "ws2_13"], w[p + "ws2_2"])), + ) + layers.append( + SimpleNamespace( + index=i, + rope=not core.nope_layer[i], + nope=nope, + owner=owner, + kv_source=core.kv_source[i], + attn_norm=w[p + "attn_norm"], + wq=w[p + "wq"], + wq_s=w[p + "wq_s"].t(), + ssmax_s=w[p + "ssmax_s"], + ssmax_b=w[p + "ssmax_b"], + wkv_a=w[p + "wkv_a"].t() if owner else None, + kv_norm=w[p + "kv_a_norm"] if owner else None, + k_b=kvb[:hn].reshape(heads, nope, kv_lora), + v_b_t=torch.transpose(kvb[hn:].reshape(heads, v_dim, kv_lora), 1, 2), + wo=w[p + "wo"], + wo_s=w[p + "wo_s"].t(), + ffn_norm=w[p + "ffn_norm"], + mlp_gu=w[p + "mlp_gu"], + mlp_gu_s=w[p + "mlp_gu_s"].t(), + mlp_dn=w[p + "mlp_dn"], + mlp_dn_s=w[p + "mlp_dn_s"].t(), + indexer=ix, + moe=moe, + next_norm=( + w[f"l{i + 1}_attn_norm"] if i + 1 < core.num_layers else w["final_norm"] + ), + ) + ) + return layers + + +class MistralLarge4TepModelingV2(DecoderModel): + def __init__(self, model_config: ModelConfig): + super().__init__(model_config) + cfg = model_config.pretrained_config + mapping = model_config.mapping + self.tp_rank = mapping.tp_rank + self.tp_size = mapping.tp_size + # The context selection's gather: the TP ranks in the order the engine sizes their + # query shares, on the MPI session's communicator (`post_load_weights` swaps in the TP + # process group when MPI is disabled). + self.tp_group = list(mapping.tp_group) + self.tp_pg = None + # The attention block's glue: the fused entries (ssmax_scaling_fused_q, + # ssmax_indexer_q_side_fused, the latent RMSNorm in place, FlashMLA's decode output + # read in place) at every max_seq_len, whose cells cover the 32K and 1M configs' + # positions (to 1048575), rope tables and step shapes; the separate ops they replace + # under `_UNFUSED_GLUE`, at every site. + self.unfused_glue = _UNFUSED_GLUE + # The generation rows' sparse MLA: cute_dsl_ds_mla_sparse_decode at the rank's heads at + # every max_seq_len, whose cells cover the 32K and 1M configs' pools and key lengths; + # flash_mla_sparse_decode over the 64-head padded query under `_GEN_ATTN_FLASH_MLA`. + self.gen_attn_flash_mla = _GEN_ATTN_FLASH_MLA + + self.num_layers = cfg.num_hidden_layers + self.hidden = cfg.hidden_size + self.eps = cfg.rms_norm_eps + self.heads = cfg.num_attention_heads // mapping.tp_size + self.nope = cfg.qk_nope_head_dim + self.rope_dim = cfg.qk_rope_head_dim + # A NoPE layer spends the rope dims on nope: the query head stays this wide. + self.qk_dim = self.nope + self.rope_dim + self.v_dim = cfg.v_head_dim + self.kv_lora = cfg.kv_lora_rank + self.lat_dim = self.kv_lora + self.rope_dim + # Every layer's softmax scale, NoPE layers included: qk_head_dim**-0.5, no YaRN + # temperature (its magnitude is 1). + self.softmax_scale = self.qk_dim**-0.5 + + self.nope_layer = [bool(f) for f in cfg.nope_layer_flags] + self.indexer_nope = [bool(f) for f in cfg.indexer_nope_layer_flags] + self.kv_source = [int(s) for s in cfg.kv_source_layers] + # An owner some later layer attends over (layer 35): a step whose context KV fits the + # held workspace twice keeps its gather there for the sharers; a longer one has each + # sharer gather its source's rows again (`_ctx_workspace`). + self.shared_source = [ + any(s == i and j != i for j, s in enumerate(self.kv_source)) + for i in range(self.num_layers) + ] + self.full_indexer = [role == "full" for role in cfg.index_sharing] + # A 'shared' layer attends with the most recent full layer's selection: each target + # keeps one selection buffer per step, which every full layer overwrites in order. + self.indexer_layers = [i for i in range(self.num_layers) if self.full_indexer[i]] + + self.idx_heads = cfg.index_n_heads + self.idx_dim = cfg.index_head_dim + self.idx_topk = cfg.index_topk + self.idx_eps = cfg.indexer_norm_eps + # The indexer's softmax scale and head average: its scorers take none, so the q side + # folds both into the head weights together with the q quantization scale. + self.idx_weight_scale = self.idx_dim**-0.5 * self.idx_heads**-0.5 + + self.dense_layers = cfg.first_k_dense_replace + self.num_experts = cfg.n_routed_experts + self.topk = cfg.num_experts_per_tok + self.moe_inter = cfg.moe_intermediate_size + self.local_experts = self.num_experts // mapping.moe_ep_size + self.expert_offset = self.local_experts * mapping.moe_ep_rank + # capacity: max_num_tokens -- the collective's peer buffers and the MoE autotuner's + # largest bucket. + self.max_num_tokens = model_config.max_num_tokens + # The rope tables' first size; grown on the first step to the engine's max_seq_len. + self.rope_rows = model_config.max_seq_len or cfg.max_position_embeddings + + self.w = _weights.MODEL_WEIGHTS.declare(_weights.MODEL_WEIGHTS.dims(self)) + + self._rope: SimpleNamespace | None = None + self._derived_for = None + self._ix_k = None + self._ix_k_checked = False + self._pool_row: list[int] = [] + self._stride_factor = 0 + self._max_ctx_len = 0 + # The engine's sizes, fixed on the first step (`_derive_first_step`). + self.prefill_logits_budget = 0 + self.ctx_workspace_rows = 0 + # The decode top-k's handle, built on the first step at a scorer width it serves. + self._ix_topk = None + self._prefill = self._decode = None + + def post_load_weights(self) -> None: + """Derive what the weights fix, build the kernels and the collective, then the targets. + + Named for the shell hook that drives it. Everything compiles here or on the first + (eager) step: a first compile inside a CUDA-graph capture breaks the graph. + """ + dev = torch.cuda.current_device() + self.build_kernels(dev) + # Collective: every rank builds it with the same arguments. The peer buffers hold + # the whole token budget, so one workspace serves every closing collective. + self._lamport = build_lamport_workspace( + self.hidden, self.eps, self.max_num_tokens, self.tp_rank, self.tp_size + ) + # The gather's communicator flavor, as the engine's own collectives pick it. + if mpi_disabled(): + self.tp_pg = self.model_config.mapping.tp_group_pg.boxed() + self._prefill = PrefillTarget(self) + self._decode = DecodeTarget(self) + + def build_kernels(self, dev: int) -> None: + """The rank-local derivations: the rope tables, the indexer q side for the unfused glue, + and the generation rows' sparse decode at this rank's heads.""" + self._rope = _rope_tables( + self.model_config.pretrained_config, + self.w["lerope_alpha"], + self.rope_rows, + self.w["final_norm"].dtype, + self.w["final_norm"].device, + ) + if self.unfused_glue: + # The checkpoint's rope is interleaved (GPT-J) for the indexer too. + self._ix_q = build_ix_q_side(dev, is_neox=False) + self._mla_decode = ( + None if self.gen_attn_flash_mla else build_ds_mla_decode(dev, num_heads=self.heads) + ) + + def _derive_first_step(self, md) -> None: + """What depends on values the engine resolves, fixed on the first step it hands over + (eager, ahead of any capture) and again for a replaced KV cache manager.""" + if md.kv_cache_manager is self._derived_for: + return + dev = torch.cuda.current_device() + mgr = md.kv_cache_manager + # The pool layout is the KV cache manager's, which routing cannot see; every latent + # op here reads and writes fp8_ds_mla rows addressed by owner row. + if getattr(mgr, "latent_layout", None) != _LATENT_LAYOUT: + raise RuntimeError( + f"this target attends over an {_LATENT_LAYOUT} latent pool with owner-only " + f"pages (MistralLarge4CacheManager); the engine built {type(mgr).__name__} " + f"with layout {getattr(mgr, 'latent_layout', None)!r}. Use KV cache manager V2 " + "and kv_cache_config.dtype auto, fp8 or fp8_ds_mla." + ) + if list(mgr.kv_source_layers) != self.kv_source: + raise RuntimeError("the KV cache manager's kv_source_layers differ from the config's") + # A table shorter than max_seq_len is read past its end with no check. + if md.max_seq_len > self._rope.rows: + self.rope_rows = md.max_seq_len + self._rope = _rope_tables( + self.model_config.pretrained_config, + self.w["lerope_alpha"], + self.rope_rows, + self.w["final_norm"].dtype, + self.w["final_norm"].device, + ) + # Layer i reads owner row `_pool_row[i]` (its source's); a block holds every owner's + # page, so a global row is `block * owners * tpb + row * tpb + offset`. + self._pool_row = [mgr.latent_pool_row(i) for i in range(self.num_layers)] + self._stride_factor = len(mgr.kv_owner_layers) * md.tokens_per_block + # The decode scorer's column count: frozen into a capture, so an engine constant. + self._max_ctx_len = md.max_seq_len + # The indexer's prefill logits budget and the held context workspace's rows, from the + # engine's max_num_tokens, max_num_requests, max_seq_len and runtime features. + self.prefill_logits_budget = _prefill_logits_budget(md.max_num_tokens, md.max_seq_len) + self.ctx_workspace_rows = _ctx_workspace_rows(md) + logger.info( + f"modeling_v2 Mistral Large 4 NVFP4: sized from the engine: prefill logits budget " + f"{self.prefill_logits_budget}, context workspace {self.ctx_workspace_rows} rows" + ) + # The decode selector, by the scorer's column count: cute_dsl_radix_topk_decode's + # handle (built once) at a width it serves, else none and the trtllm op selects. + lo, hi = _TOPK_DECODE_COLUMNS + if not (_TOPK_DECODE_CUTE_DSL and lo <= self._max_ctx_len <= hi): + self._ix_topk = None + elif self._ix_topk is None: + self._ix_topk = build_ix_topk_decode(dev, next_n=1) + # Both key-pool layouts: a launcher compiled for a contiguous view addresses a strided + # one as if it were dense. + if self._ix_k is None: + self._ix_k = { + contiguous: build_ix_k_cache(dev, is_neox=False, cache_contiguous=contiguous) + for contiguous in (False, True) + } + self._derived_for = md.kv_cache_manager + + def _select_target(self, attn_metadata) -> "PrefillTarget | DecodeTarget": + """Which target runs this step: one host-side integer read. A mixed batch goes to + prefill, so prefill is the general case and decode the specialization that may assume + it has no context rows.""" + return self._decode if attn_metadata.num_contexts == 0 else self._prefill + + def forward( + self, + attn_metadata: AttentionMetadata, + input_ids: torch.IntTensor | None = None, + position_ids: torch.IntTensor | None = None, + inputs_embeds: torch.FloatTensor | None = None, + lora_params: dict | None = None, + **kwargs, + ) -> torch.Tensor: + """Derive what the first step fixes, then route.""" + self._derive_first_step(attn_metadata) + return self._select_target(attn_metadata).forward( + attn_metadata, + input_ids=input_ids, + position_ids=position_ids, + inputs_embeds=inputs_embeds, + lora_params=lora_params, + **kwargs, + ) + + +class PrefillTarget: + """The general case: context rows, and possibly generation rows behind them.""" + + # The held context workspace, bf16 [core.ctx_workspace_rows, 576]: allocated on the first + # context step and held from then on (`_ctx_workspace`). + _ws_buf: torch.Tensor | None = None + # The indexer's prefill top-k scratch (`gvr_workspace`, 20 MiB): allocated on the first + # context step that selects and held, used on the main stream only (`_select_ctx`). + _gvr_ws: torch.Tensor | None = None + + def __init__(self, core: MistralLarge4TepModelingV2) -> None: + """Take the per-layer views once, off the real weights. Everything an engine value + fixes is read off `self.core` at the call.""" + self.core = core + # Off unless TRTLLM_MODELING_V2_VALIDATE asks for it. Each target owns its own flag: + # the two `step_args` projections differ, so probing one does not prove the other. + self._contract_pending = step_contract_enabled() + self._layers = _layer_views(core) + self._embed = core.w["embed"] + self._norm0 = core.w["l0_attn_norm"] + + def _check_step_contract(self, md) -> None: + """Opt-in first-forward fail-fast: calling `step_args` reads every metadata field this + target consumes (private trtllm surface). Fixed at engine construction, so once per + target instance, and off in a served engine.""" + if not self._contract_pending: + return + try: + self.step_args(md) + except AttributeError as exc: + raise AssertionError(f"attention metadata surface drifted: {exc}") from exc + self._contract_pending = False + + def step_args(self, md: TrtllmAttentionMetadata) -> dict: + """The common batch state, plus the context rows': their counts and lengths, the + indexer's prefill windows and scoring blocks, its query split across the TP ranks and + its key gather mappings, and the context rope/append op's offsets and their host prefix + sums.""" + step = _step_common(md, self.core.indexer_layers) + ctx = md.num_contexts + step.update( + num_contexts=ctx, + num_ctx_tokens=md.num_ctx_tokens, + seq_lens=md.seq_lens, + skip_ctx=bool(md.skip_indexer_for_ctx_reqs), + cu_seqlen_ks=md.cu_seqlen_ks, + cu_seqlen_ke=md.cu_seqlen_ke, + # (q_begin, q_end, per-rank sizes) once the step's context rows reach the engine's + # q_split_threshold under chunked prefill: the chunks then cover this rank's + # [q_begin, q_end) only. + indexer_prefill_q_split=md.indexer_prefill_q_split, + num_ctx_kv_tokens=md.num_ctx_kv_tokens, + slot_mapping_fp8_fullkv=md.slot_mapping_fp8_fullkv, + slot_mapping_scale_fullkv=md.slot_mapping_scale_fullkv, + **_ctx_kv_offsets(md, ctx), + **_ctx_host_offsets(md, ctx), + ) + step["indexer_blocks"] = _indexer_blocks(md.indexer_prefill_chunks, step) + return step + + # ---- one layer's pieces, context and generation rows ---- + + def _linear_fp8(self, x, w, w_s): + """`x @ w.T`: the bf16 activation quantized 1x128 with power-of-two scales, then the + DeepGEMM block-scale GEMM against the load-time UE8M0 weight scales.""" + codes, scale = fp8_quantize_1x128_packed_ue8m0(x) + return fp8_prequantized_swap_ab_gemm(codes, scale, w, w_s, torch.bfloat16, True) + + def _latent(self, x, L, nope_lat): + """`[RMSNorm(c_kv) | k_pe]`, 576 wide: a RoPE owner's kv_a row, normalized in place; a + NoPE owner's 512-wide latent normalized into `nope_lat`, whose rope key stays zero.""" + core = self.core + kv = cublas_mm(x, L.wkv_a) + if core.unfused_glue: + c_kv = flashinfer_rmsnorm(kv[:, : core.kv_lora], L.kv_norm, core.eps) + if L.rope: + k_pe = kv[:, core.kv_lora :] + else: + k_pe = torch.zeros([kv.shape[0], core.rope_dim], dtype=kv.dtype, device=kv.device) + return torch.cat([c_kv, k_pe], dim=-1) + if L.rope: + c_kv = kv[:, : core.kv_lora] + flashinfer_rmsnorm(c_kv, L.kv_norm, core.eps, out=c_kv) + return kv + flashinfer_rmsnorm(kv, L.kv_norm, core.eps, out=nope_lat[:, : core.kv_lora]) + return nope_lat + + def _indexer(self, step, x, pos, L, topk_buf, tc: int, skip_ctx: bool, skip_gen: bool): + """Append this step's keys to the layer's key pool; select `index_topk` positions per + row into `topk_buf` for the rows whose phase the engine did not skip. Under the + engine's query split each rank selects its share of the context rows, and the shares + are gathered into rows [0, tc) before any layer reads them.""" + core = self.core + ix = L.indexer + rows = x.shape[0] + table = core._rope.indexer if ix.rope else core._rope.identity + kw = cublas_mm(x, ix.wk_wp) + k = kw[:, : core.idx_dim].contiguous() + k_pool = step["ix_pools"][L.index] + k_pool = cute_dsl_indexer_k_cache_update( + k, ix.k_norm_w, ix.k_norm_b, pos, table.view(-1, 2, core.rope_dim // 2), k_pool, + step["slot_mapping_fp8"], step["slot_mapping_scale"], rows, + is_neox=False, compiled=core._ix_k[k_pool.is_contiguous()], + # Its range check is a host sync: once, on the first (eager) call. + validate=not core._ix_k_checked, + ) # fmt: skip + core._ix_k_checked = True + if skip_ctx and skip_gen: + return + qn = layer_norm(x, ix.q_norm_w, ix.q_norm_b, core.idx_eps) + q = torch.reshape(self._linear_fp8(qn, ix.wq_b, ix.wq_b_s), [rows, core.idx_heads, -1]) + q_fp8 = torch.empty_like(q, dtype=torch.float8_e4m3fn) + weights = torch.empty([rows, core.idx_heads], dtype=torch.float32, device=x.device) + if core.unfused_glue: + q = ssmax_scaling(q, pos, ix.ssmax_s, ix.ssmax_b) + q_fp8, weights = cute_dsl_indexer_q_side_fused( + q, pos, table, kw[:, core.idx_dim :].float().contiguous(), core.idx_weight_scale, + q_fp8, weights, is_neox=False, compiled=core._ix_q, + ) # fmt: skip + else: + # SSMax, rope, e4m3 and the head weights (read in place out of `kw`) in one launch. + q_fp8, weights = ssmax_indexer_q_side_fused( + q, pos, ix.ssmax_s, ix.ssmax_b, table, kw[:, core.idx_dim :], + core.idx_weight_scale, q_fp8, weights, + ) # fmt: skip + if tc and not skip_ctx: + self._select_ctx(step, k_pool, q_fp8, weights, topk_buf) + q_split = step["indexer_prefill_q_split"] + if q_split is not None: + # The chunks covered this rank's share only. Every rank of the group makes + # this call once per selecting layer, in layer order: the communicator pairs + # calls by position. + q_begin, q_end, q_sizes = q_split + topk_buf[:tc].copy_( + allgather(topk_buf[q_begin:q_end], q_sizes, core.tp_group, core.tp_pg) + ) + gen = rows - tc + if gen and not skip_gen: + ctx = step["num_contexts"] + num_gens = step["num_seqs"] - ctx + assert gen % num_gens == 0, (gen, num_gens) + next_n = gen // num_gens + kv_lens = step["kv_lens_cuda"][ctx : ctx + num_gens] + logits = cute_dsl_fp8_paged_mqa_logits( + torch.reshape(q_fp8[tc:], [num_gens, next_n, core.idx_heads, core.idx_dim]), + k_pool, + weights[tc:], + kv_lens, + step["indexer_k_cache_block_offsets"][ctx : ctx + num_gens], + step["scheduler_metadata_buffer"], + core._max_ctx_len, + ) + if next_n != 1 or core._ix_topk is None: + indexer_topk_decode( + logits, kv_lens, topk_buf[tc:], next_n, core.idx_topk, 1, + step["radix_aux_indices"], step["radix_aux_logits"], + ) # fmt: skip + else: + cute_dsl_radix_topk_decode( + logits, kv_lens, topk_buf[tc:], next_n=1, compiled=core._ix_topk + ) + + def _ix_plan(self, step, heads: int) -> list[SimpleNamespace]: + """The step's indexer scoring plan, the same for each of its indexer layers: per scoring + block (`_indexer_blocks`, this rank's share of the context rows under the engine's + query split), its rows, its keys in the step's full-KV mappings and its query tiles of + at most the engine's `prefill_logits_budget` logits (`_prefill_tiles`), each tile with + its rows' windows and the split-KV schedule of its logits call (`mqa_logits_schedule`). + A block with no key has no tile. + + Built on the step's first indexer layer and kept in `step`: the windows, the key counts + and so the schedules depend on the step alone, and the layers differ only in the keys + and queries they score.""" + core = self.core + plan = [] + for block in step["indexer_blocks"]: + n_k = block.k_token_end - block.k_token_start + if block.token_end <= block.token_start: + continue + tiles = [] + if n_k > 0: + for lo, hi in _prefill_tiles( + block.token_start, block.token_end, n_k, core.prefill_logits_budget + ): + # The windows are the block's own, indexed from its first row. + ks = block.cu_seqlen_ks[lo - block.token_start : hi - block.token_start] + ke = block.cu_seqlen_ke[lo - block.token_start : hi - block.token_start] + tiles.append((lo, hi, ks, ke, mqa_logits_schedule(ks, ke, n_k, heads))) + plan.append( + SimpleNamespace( + token_start=block.token_start, + token_end=block.token_end, + k_token_start=block.k_token_start, + n_k=n_k, + tiles=tiles, + ) + ) + return plan + + def _select_ctx(self, step, k_pool, q_fp8, weights, topk_buf) -> None: + """Select `index_topk` positions, window-local, for each context row this rank scores, + along the step's scoring plan (`_ix_plan`): per block, gather its keys out of the + layer's key pool through the step's full-KV mappings, then per query tile score its + rows against them with the tile's split-KV schedule, which spreads each query block's + keys over the SMs (a 256-row tile at 1048576 keys keeps every SM busy, the grid-stride + call 64), and select over each row's window into `topk_buf` with the self-sampling GVR + top-k, one streaming pass per row. Both answer as the grid-stride scorer and the radix + top-k: the logits bitwise, the selection the same set up to the keys tied at a row's + 2048th score. A block with no key leaves its rows' slots empty (-1), and each tile's + logits are freed before the next tile's are allocated.""" + core = self.core + plan = step.get("ix_plan") + if plan is None: + plan = step["ix_plan"] = self._ix_plan(step, q_fp8.shape[1]) + if self._gvr_ws is None: + # On the first context step, which the KV-capacity estimate's profiling forward + # is: the pool is sized with the top-k's scratch held, and every top-k launch this + # engine's tiles reach compiles here rather than on a served step (each tile holds + # at most the logits budget, one block at most max_seq_len keys). + self._gvr_ws = gvr_workspace(q_fp8.device) + t0 = time.monotonic() + calls = gvr_warmup( + core.idx_topk, core._max_ctx_len, core.prefill_logits_budget, q_fp8.device, + self._gvr_ws, + ) # fmt: skip + logger.info( + f"modeling_v2 Mistral Large 4 NVFP4: indexer prefill top-k compiled for this " + f"engine's tiles ({calls} warm calls, {time.monotonic() - t0:.1f} s)" + ) + for block in plan: + if block.n_k <= 0: + # No key to select from: every slot of the block's rows is empty. + topk_buf[block.token_start : block.token_end].fill_(-1) + continue + ck, cs = indexer_k_cache_gather_op( + k_pool, step["slot_mapping_fp8_fullkv"], step["slot_mapping_scale_fullkv"], + block.k_token_start, block.n_k, core.idx_dim, + ) # fmt: skip + cs = torch.reshape(cs, [-1]) + for lo, hi, ks, ke, schedule in block.tiles: + logits = fp8_mqa_logits( + q_fp8[lo:hi], (ck, cs), weights[lo:hi], ks, ke, schedule_meta=schedule + ) + cute_dsl_gvr_topk_prefill( + logits, ks, ke, topk_buf[lo:hi], core.idx_topk, self._gvr_ws + ) + # Freed before the next tile's is allocated: one tile is live at a time. + del logits + + def _attn_ctx(self, step, L, fused_q, lat_out, req_idx, topk_buf, ctx): + """Sparse MLA over the context rows into `lat_out`, on the step's path (`ctx`). + + Workspace path: FlashMLA's bf16 sparse prefill over the context requests' latent rows + of the source layer, dequantized out of the pool (this step's rows included, so every + row is read FP8-rounded) into the held workspace's first rows. A shared source (layer + 35) is gathered once into the rows the step keeps beside them, when it has them, and + read there by every layer that shares it; otherwise each of those layers gathers it + again into the first rows. Decode path, a step past the held workspace: the selections + as pool slots, then FlashMLA's sparse fp8 decode over the pool with every context row a + one-row sequence, in the step's sub-batches.""" + core = self.core + tc = fused_q.shape[0] + src = L.kv_source + if ctx.ws_buf is None: + slots = convert_req_index_to_global( + req_idx[:tc], step["block_table"], topk_buf[:tc], step["tokens_per_block"], + core.idx_topk, core._stride_factor, core._pool_row[src], + ) # fmt: skip + for lo, hi in ctx.sub_batches: + # Past 11622 rows FlashMLA's scheduler exits the process instead of raising. + assert hi - lo <= _CTX_DECODE_ROWS, (lo, hi) + flash_mla_sparse_decode( + fused_q[lo:hi], step["latent_kv"], slots[lo:hi], ctx.sched[hi - lo], + core.softmax_scale, lat_out[lo:hi], + ) # fmt: skip + return + ws = ctx.ws.get(src) + if ws is None: + keep = core.shared_source[src] and ctx.kept is not None + ws = ds_mla_dequant_gather( + step["latent_pool"], core._pool_row[src], ctx.kv_req, ctx.kv_pos, + step["block_table"], ctx.kept if keep else ctx.ws_buf, + ) # fmt: skip + if keep: + ctx.ws[src] = ws + # Request-local selections to workspace rows: each request's run starts at its + # cumulative KV offset. + sel = topk_buf[:tc] + idx = torch.where(sel >= 0, sel + ctx.ws_off[req_idx[:tc]][:, None], sel) + flash_mla_sparse_prefill(fused_q, ws, idx.to(torch.int32), core.softmax_scale, lat_out) + + def _attn_gen(self, step, L, fused_q, lat_out, slots, sched): + """Sparse MLA over the generation rows behind the context rows into `lat_out`, over the + pool: this rank's heads read off the 64-head scratch the prefill kernel shares (or, under + `gen_attn_flash_mla`, FlashMLA's decode over all 64, one scheduler for the step).""" + core = self.core + if core.gen_attn_flash_mla: + flash_mla_sparse_decode( + fused_q, step["latent_kv"], slots, sched, core.softmax_scale, lat_out + ) + else: + cute_dsl_ds_mla_sparse_decode( + fused_q, step["latent_kv"], slots, core.softmax_scale, lat_out, + compiled=core._mla_decode, + ) # fmt: skip + + def _mlp(self, o, L): + """This rank's share of the layer's MLP, a partial sum: the dense SwiGLU slice on the + dense layer; on a MoE layer the expert window (32 experts at tep4, 16 at tep8) plus the + shared expert's slice.""" + core = self.core + act = flashinfer_silu_and_mul(self._linear_fp8(o, L.mlp_gu, L.mlp_gu_s)) + shared = self._linear_fp8(act, L.mlp_dn, L.mlp_dn_s) + m = L.moe + if m is None: + return shared + logits = cublas_mm(o, m.router, None, torch.float32) + ids, weights = renorm_moe_routing_op(logits, core.topk, torch.bfloat16) + # One NVFP4 quantization under the layer's static scale serves every expert on every + # rank: the windows' partials sum. + xq, xs = fp4_quantize(o, m.gscale, _NVFP4_SF_VEC, False, _NVFP4_SF_LINEAR) + routed = fp4_block_scale_moe_runner( + None, None, xq, xs.view(torch.float8_e4m3fn), m.fc1_w, m.fc1_s, None, None, None, + None, m.fc2_w, m.fc2_s, None, m.out1, m.out1_gate, m.out2, core.num_experts, + core.topk, None, None, core.moe_inter, core.expert_offset, core.local_experts, None, + _MOE_ROUTING_METHOD, True, _MOE_ACT_SWIGLU, topk_weights=weights, topk_ids=ids, + tune_max_num_tokens=core.max_num_tokens, + )[0] # fmt: skip + return torch.add(routed, shared) + + def _topk_buf(self, step, rows: int, tc: int, skip_ctx: bool, skip_gen: bool): + """Where this step's selections land, request-local positions. A phase the engine + skipped (every sequence within index_topk) takes the dense causal index it wrote.""" + if skip_ctx and skip_gen: + return step["topk_indices_buffer"][:rows] + buf = torch.empty( + [rows, self.core.idx_topk], dtype=torch.int32, device=step["block_table"].device + ) + if skip_ctx and tc: + buf[:tc].copy_(step["topk_indices_buffer"][:tc]) + if skip_gen and rows > tc: + buf[tc:].copy_(step["topk_indices_buffer"][tc:rows]) + return buf + + def forward( + self, + attn_metadata: AttentionMetadata, + input_ids: torch.IntTensor | None = None, + position_ids: torch.IntTensor | None = None, + inputs_embeds: torch.FloatTensor | None = None, + lora_params: dict | None = None, + **kwargs, + ) -> torch.Tensor: + core = self.core + self._check_step_contract(attn_metadata) + step = self.step_args(attn_metadata) + h = ( + nn.functional.embedding(input_ids, self._embed) + if inputs_embeds is None + else inputs_embeds + ) + rows = h.shape[0] + dev = h.device + # The slot mappings and windows follow the prepared token order; a padded input + # would scatter keys into stale slots. + assert rows == step["num_tokens"], (rows, step["num_tokens"]) + tc = step["num_ctx_tokens"] + gen = rows - tc + skip_ctx = step["skip_ctx"] + skip_gen = step["skip_gen"] if gen else True + pos = torch.reshape(position_ids, [-1]) + req_idx = step["req_idx_per_token"][:rows] + topk_buf = self._topk_buf(step, rows, tc, skip_ctx, skip_gen) + scratch = self._scratch(rows, h.dtype, dev) + ctx = self._ctx_workspace(step, tc, dev) if tc else None + + x = flashinfer_rmsnorm(h, self._norm0, core.eps) + residual = h + spec_metadata = kwargs.get("spec_metadata") + for layer_idx, L in enumerate(self._layers): + attn_out = self._attention( + step, L, x, pos, req_idx, topk_buf, scratch, ctx, tc, skip_ctx, skip_gen + ) + partial = self._linear_fp8(attn_out, L.wo, L.wo_s) + o = cute_dsl_allreduce_add_rmsnorm( + partial, residual, L.ffn_norm, torch.empty_like(partial), core._lamport + ) + mlp = self._mlp(o, L) + x = cute_dsl_allreduce_add_rmsnorm( + mlp, residual, L.next_norm, torch.empty_like(mlp), core._lamport + ) + if spec_metadata is not None: + spec_metadata.maybe_capture_hidden_states(layer_idx, residual) + return x + + def _scratch(self, rows: int, dtype, dev) -> SimpleNamespace: + """Per-step scratch every layer rewrites in turn: the attention output, the absorbed + query at the prefill kernel's 64 heads (this rank's first, the rest zero for the step), + the latent attention output, a NoPE owner's 576-wide latent rows (the rope key zero for + the step), and, under `gen_attn_flash_mla`, the generation rows' FlashMLA scheduler, + filled by the step's first decode call and shared by the rest.""" + core = self.core + bf16 = dict(dtype=dtype, device=dev) + return SimpleNamespace( + attn_out=torch.empty([rows, core.heads * core.v_dim], **bf16), + fused_q=torch.zeros([rows, _FLASH_MLA_HEADS, core.lat_dim], **bf16), + lat_out=torch.empty([rows, core.heads, core.kv_lora], **bf16), + nope_lat=None if core.unfused_glue else torch.zeros([rows, core.lat_dim], **bf16), + sched=get_mla_metadata()[0] if core.gen_attn_flash_mla else None, + ) + + def _ctx_workspace(self, step, tc: int, dev) -> SimpleNamespace: + """The context rows' path for the step, chosen once on host ints by its context KV: + every context request's cached prefix and new tokens (the host + `ctx_kv_indptr[num_contexts]`; the engine's `num_ctx_kv_tokens` counts the new ones + only, and under chunked prefill every chunk after a prompt's first has a prefix). + + The held workspace, bf16 `[core.ctx_workspace_rows, 576]` (`_ctx_workspace_rows`), is + allocated on the first context step, whichever path that step takes, and held. The + engine's KV-capacity estimate runs context steps as its profiling forward and sizes the + pool after their peak memory (`KvCacheCreator.configure_kv_cache_capacity`), so the pool + is sized with the workspace held. A generation-only engine runs no context step and + holds none. + + The workspace path, a step whose context KV fits the held rows: request and position + per workspace row as the dequantizing gather and the prefill kernel address them, each + request's offset, the held workspace's first `n_kv` rows, which every layer's gather + rewrites, and, when the context KV fits twice, the next `n_kv` rows (`kept`), where a + shared owner's gather stays for the step (`ws`). + + The decode path, a step past the held rows, with no workspace: the context rows' + sub-batches of at most `_CTX_DECODE_ROWS`, and one FlashMLA scheduler per sub-batch + size, filled by its first call and shared by the step's layers.""" + # Under KV reuse a context request attends over its cached prefix and its new tokens. + n_kv = step["ctx_cached_tokens"] + step["num_ctx_tokens"] + if self._ws_buf is None: + # On the first context step: the KV-capacity estimate's profiling forward is one, + # so the peak it measures, which sizes the pool, holds this buffer. + self._ws_buf = torch.empty( + [self.core.ctx_workspace_rows, self.core.lat_dim], dtype=torch.bfloat16, device=dev + ) + logger.info( + f"modeling_v2 Mistral Large 4 NVFP4: context workspace held, {self._ws_buf.shape[0]} " + f"rows x {self.core.lat_dim} bf16 ({self._ws_buf.nbytes / 2**20:.0f} MiB)" + ) + held = self._ws_buf.shape[0] + if n_kv > held: + bounds = [(lo, min(lo + _CTX_DECODE_ROWS, tc)) for lo in range(0, tc, _CTX_DECODE_ROWS)] + logger.debug( + f"modeling_v2 Mistral Large 4 NVFP4: context KV {n_kv} past the held {held} rows: " + f"decode path, {len(bounds)} sub-batches" + ) + return SimpleNamespace( + ws_buf=None, + sub_batches=bounds, + sched={rows: get_mla_metadata()[0] for rows in {hi - lo for lo, hi in bounds}}, + ) + ctx = step["num_contexts"] + cu = step["cu_ctx_kv"] + lens = cu[1:] - cu[:-1] + kv_req = torch.repeat_interleave( + torch.arange(ctx, dtype=torch.int32, device=dev), lens, output_size=n_kv + ) + kv_pos = (torch.arange(n_kv, device=dev) - cu[:-1][kv_req]).to(torch.int32) + keep = 2 * n_kv <= held + logger.debug( + f"modeling_v2 Mistral Large 4 NVFP4: context KV {n_kv} of the held {held} rows: " + f"workspace path, the shared source {'kept' if keep else 'gathered again'}" + ) + return SimpleNamespace( + kv_req=kv_req, + kv_pos=kv_pos, + ws_off=cu[:-1], + ws_buf=self._ws_buf[:n_kv], + kept=self._ws_buf[n_kv : 2 * n_kv] if keep else None, + ws={}, + ) + + def _attention(self, step, L, x, pos, req_idx, topk_buf, s, ctx, tc, skip_ctx, skip_gen): + """One layer's attention block, both phases, into `s.attn_out` (returned): the query + with SSMax, the indexer's selection, the absorbed query, the owner's rope and + fp8_ds_mla append, sparse MLA over the context and the generation rows, and the V + up-projection.""" + core = self.core + rows = x.shape[0] + q = self._linear_fp8(x, L.wq, L.wq_s) + q3 = torch.reshape(q, [rows, core.heads, core.qk_dim]) + fq = s.fused_q[:, : core.heads] + if core.unfused_glue: + q3 = ssmax_scaling(q3, pos, L.ssmax_s, L.ssmax_b) + if L.rope: + fq[..., core.kv_lora :].copy_(q3[..., L.nope :]) + else: + fq[..., core.kv_lora :].zero_() + else: + # SSMax in place; its rope half (a NoPE layer's zero) into the absorbed query's tail. + ssmax_scaling_fused_q( + q3, pos, L.ssmax_s, L.ssmax_b, q3, fq[..., core.kv_lora :], L.rope + ) + if L.indexer is not None: + self._indexer(step, x, pos, L, topk_buf, tc, skip_ctx, skip_gen) + bmm_out( + torch.transpose(q3[..., : L.nope], 0, 1), L.k_b, + torch.transpose(fq[..., : core.kv_lora], 0, 1), + ) # fmt: skip + if L.owner: + # Every row's latent, context and generation, before either phase reads the pool; + # the query tail is roped in the same call. + ds_mla_rope_append( + self._latent(x, L, s.nope_lat), fq, pos, core._rope.indexer, req_idx, + step["block_table"], step["latent_pool"], core._pool_row[L.index], L.rope, + ) # fmt: skip + if tc: + self._attn_ctx(step, L, s.fused_q[:tc], s.lat_out[:tc], req_idx, topk_buf, ctx) + if rows > tc: + slots = convert_req_index_to_global( + req_idx[tc:], step["block_table"], topk_buf[tc:], step["tokens_per_block"], + core.idx_topk, core._stride_factor, core._pool_row[L.kv_source], + ) # fmt: skip + self._attn_gen(step, L, s.fused_q[tc:], s.lat_out[tc:], slots, s.sched) + bmm_out( + torch.transpose(s.lat_out, 0, 1), + L.v_b_t, + torch.transpose(torch.reshape(s.attn_out, [rows, core.heads, core.v_dim]), 0, 1), + ) + return s.attn_out + + def dspark_forward( + self, + draft, + noise_embedding, + query_positions, + num_ctx_per_req, + ctx_k_cache, + ctx_v_cache, + ctx_cache_batch_idx, + ctx_kv_cache=None, + ctx_page_table=None, + block_placement=None, + ): + # Only the worker's fused paged-latent preparation places the block + # (block_placement), and a drafter with embed_draft_block does not take + # it; this decode reads its context window through ctx_page_table. + del block_placement + draft._build_fused_kv_buffers() + draft._ensure_rope() + core = self.core + batch, block, hidden = noise_embedding.shape + rows = batch * block + # Gather a bounded window, not every page in the configured context. + # Invalid padding slots are masked even if their pages are allocated. + lengths = num_ctx_per_req[:batch].long() + offsets = torch.arange(draft.window, device=noise_embedding.device) + positions = (lengths - draft.window).clamp_min(0)[:, None] + offsets[None, :] + valid = positions < lengths[:, None] + slots = ctx_cache_batch_idx.long() + if ctx_page_table is not None: + page_size = ctx_kv_cache[0].shape[-2] + table = ctx_page_table.index_select(0, slots).long() + safe_positions = positions.clamp(max=table.shape[1] * page_size - 1) + pages = table.gather(1, safe_positions // page_size).clamp_min(0) + page_offsets = safe_positions % page_size + else: + safe_positions = positions.clamp(max=ctx_k_cache.shape[2] - 1) + key_valid = torch.cat( + (valid, torch.ones(batch, block, dtype=torch.bool, device=valid.device)), dim=1 + ) + query = torch.empty( + batch, + block, + core.heads, + core.lat_dim, + dtype=noise_embedding.dtype, + device=noise_embedding.device, + ) + values = torch.empty( + rows, core.heads, core.v_dim, dtype=noise_embedding.dtype, device=noise_embedding.device + ) + residual = noise_embedding.reshape(rows, hidden).clone() + normalized = torch.empty_like(residual) + x = flashinfer_rmsnorm(residual, draft._layers[0].attn_norm, core.eps) + for i, layer in enumerate(draft._layers): + q = cublas_mm(x, layer.q, None, None).view(batch, block, core.heads, core.qk_dim) + bmm_out( + q[..., : core.nope].reshape(rows, core.heads, core.nope).transpose(0, 1), + layer.k_b, + query[..., : core.kv_lora].view(rows, core.heads, core.kv_lora).transpose(0, 1), + ) + query[..., core.kv_lora :].copy_(draft._rotate(q[..., core.nope :], query_positions)) + kv = cublas_mm(x, layer.kv_a, None, None) + content = flashinfer_rmsnorm( + kv[:, : core.kv_lora].contiguous(), layer.kv_norm, core.eps + ) + block_kv = torch.cat( + ( + content.view(batch, block, core.kv_lora), + draft._rotate( + kv[:, core.kv_lora :].view(batch, block, core.rope_dim), query_positions + ), + ), + dim=-1, + ) + if ctx_page_table is not None: + context = ctx_kv_cache[i][:, 0, 0][pages, page_offsets] + else: + context = ctx_k_cache[slots[:, None], i, safe_positions, 0] + # Zero masked values as well as scores: 0 * an uninitialized + # NaN in the latent pool must not contaminate the weighted sum. + context = context.masked_fill(~valid.unsqueeze(-1), 0) + keys = torch.cat((context, block_kv), dim=1) + scores = torch.einsum("bshd,btd->bsht", query, keys).float() * core.softmax_scale + scores = scores.masked_fill(~key_valid[:, None, None, :], float("-inf")) + probabilities = torch.softmax(scores, dim=-1).to(keys.dtype) + latent = torch.einsum("bsht,btc->bshc", probabilities, keys[..., : core.kv_lora]) + bmm_out( + latent.reshape(rows, core.heads, core.kv_lora).transpose(0, 1), + layer.v_b.transpose(1, 2), + values.transpose(0, 1), + ) + partial = cublas_mm(values.reshape(rows, -1), layer.out, None, None) + o = cute_dsl_allreduce_add_rmsnorm( + partial, residual, layer.ffn_norm, normalized, core._lamport + ) + gate = cublas_mm(o, layer.gate, None, None) + up = cublas_mm(o, layer.up, None, None) + activated = flashinfer_silu_and_mul(torch.cat((gate, up), dim=-1)) + mlp = cublas_mm(activated, layer.down, None, None) + x = cute_dsl_allreduce_add_rmsnorm( + mlp, residual, layer.next_norm, normalized, core._lamport + ) + return x + + +class DecodeTarget: + """Generation-only steps, including uniform DSpark verification blocks; graph captured.""" + + def __init__(self, core: MistralLarge4TepModelingV2) -> None: + """Take the per-layer views once, off the real weights. Everything an engine value + fixes is read off `self.core` at the call.""" + self.core = core + self._contract_pending = step_contract_enabled() + self._layers = _layer_views(core) + self._embed = core.w["embed"] + self._norm0 = core.w["l0_attn_norm"] + + def _check_step_contract(self, md) -> None: + """Opt-in first-forward fail-fast: calling `step_args` reads every metadata field this + target consumes (private trtllm surface). Fixed at engine construction, so once per + target instance, and off in a served engine.""" + if not self._contract_pending: + return + try: + self.step_args(md) + except AttributeError as exc: + raise AssertionError(f"attention metadata surface drifted: {exc}") from exc + self._contract_pending = False + + def step_args(self, md: TrtllmAttentionMetadata) -> dict: + """The common batch state. States the two context counts instead of reading them + back: this target runs only when there are no context rows.""" + step = _step_common(md, self.core.indexer_layers) + step.update(num_contexts=0, num_ctx_tokens=0) + return step + + def _linear_fp8(self, x, w, w_s): + """`x @ w.T`: the bf16 activation quantized 1x128 with power-of-two scales, then the + DeepGEMM block-scale GEMM against the load-time UE8M0 weight scales.""" + codes, scale = fp8_quantize_1x128_packed_ue8m0(x) + return fp8_prequantized_swap_ab_gemm(codes, scale, w, w_s, torch.bfloat16, True) + + def _latent(self, x, L, nope_lat): + """`[RMSNorm(c_kv) | k_pe]`, 576 wide: a RoPE owner's kv_a row, normalized in place; a + NoPE owner's 512-wide latent normalized into `nope_lat`, whose rope key stays zero.""" + core = self.core + kv = cublas_mm(x, L.wkv_a) + if core.unfused_glue: + c_kv = flashinfer_rmsnorm(kv[:, : core.kv_lora], L.kv_norm, core.eps) + if L.rope: + k_pe = kv[:, core.kv_lora :] + else: + k_pe = torch.zeros([kv.shape[0], core.rope_dim], dtype=kv.dtype, device=kv.device) + return torch.cat([c_kv, k_pe], dim=-1) + if L.rope: + c_kv = kv[:, : core.kv_lora] + flashinfer_rmsnorm(c_kv, L.kv_norm, core.eps, out=c_kv) + return kv + flashinfer_rmsnorm(kv, L.kv_norm, core.eps, out=nope_lat[:, : core.kv_lora]) + return nope_lat + + def _indexer(self, step, x, pos, L, topk_buf, skip_gen: bool): + """Append this step's keys to the layer's key pool; unless the engine skipped the + indexer (every sequence within index_topk), score every sequence's paged keys and + select `index_topk` positions per row into `topk_buf`.""" + core = self.core + ix = L.indexer + rows = x.shape[0] + table = core._rope.indexer if ix.rope else core._rope.identity + kw = cublas_mm(x, ix.wk_wp) + k = kw[:, : core.idx_dim].contiguous() + k_pool = step["ix_pools"][L.index] + k_pool = cute_dsl_indexer_k_cache_update( + k, ix.k_norm_w, ix.k_norm_b, pos, table.view(-1, 2, core.rope_dim // 2), k_pool, + step["slot_mapping_fp8"], step["slot_mapping_scale"], rows, + is_neox=False, compiled=core._ix_k[k_pool.is_contiguous()], validate=False, + ) # fmt: skip + if skip_gen: + return + qn = layer_norm(x, ix.q_norm_w, ix.q_norm_b, core.idx_eps) + q = torch.reshape(self._linear_fp8(qn, ix.wq_b, ix.wq_b_s), [rows, core.idx_heads, -1]) + q_fp8 = torch.empty_like(q, dtype=torch.float8_e4m3fn) + weights = torch.empty([rows, core.idx_heads], dtype=torch.float32, device=x.device) + if core.unfused_glue: + q = ssmax_scaling(q, pos, ix.ssmax_s, ix.ssmax_b) + q_fp8, weights = cute_dsl_indexer_q_side_fused( + q, pos, table, kw[:, core.idx_dim :].float().contiguous(), core.idx_weight_scale, + q_fp8, weights, is_neox=False, compiled=core._ix_q, + ) # fmt: skip + else: + # SSMax, rope, e4m3 and the head weights (read in place out of `kw`) in one launch. + q_fp8, weights = ssmax_indexer_q_side_fused( + q, pos, ix.ssmax_s, ix.ssmax_b, table, kw[:, core.idx_dim :], + core.idx_weight_scale, q_fp8, weights, + ) # fmt: skip + num_gens = step["num_seqs"] + next_n = rows // num_gens + kv_lens = step["kv_lens_cuda"][:num_gens] + logits = cute_dsl_fp8_paged_mqa_logits( + torch.reshape(q_fp8, [num_gens, next_n, core.idx_heads, core.idx_dim]), + k_pool, + weights, + kv_lens, + step["indexer_k_cache_block_offsets"][:num_gens], + step["scheduler_metadata_buffer"], + core._max_ctx_len, + ) + if next_n != 1 or core._ix_topk is None: + indexer_topk_decode( + logits, kv_lens, topk_buf, next_n, core.idx_topk, 1, step["radix_aux_indices"], + step["radix_aux_logits"], + ) # fmt: skip + else: + cute_dsl_radix_topk_decode(logits, kv_lens, topk_buf, next_n=1, compiled=core._ix_topk) + + def _mlp(self, o, L): + """This rank's share of the layer's MLP, a partial sum: the dense SwiGLU slice on the + dense layer; on a MoE layer the expert window (32 experts at tep4, 16 at tep8) plus the + shared expert's slice.""" + core = self.core + act = flashinfer_silu_and_mul(self._linear_fp8(o, L.mlp_gu, L.mlp_gu_s)) + shared = self._linear_fp8(act, L.mlp_dn, L.mlp_dn_s) + m = L.moe + if m is None: + return shared + logits = cublas_mm(o, m.router, None, torch.float32) + ids, weights = renorm_moe_routing_op(logits, core.topk, torch.bfloat16) + # One NVFP4 quantization under the layer's static scale serves every expert on every + # rank: the windows' partials sum. + xq, xs = fp4_quantize(o, m.gscale, _NVFP4_SF_VEC, False, _NVFP4_SF_LINEAR) + routed = fp4_block_scale_moe_runner( + None, None, xq, xs.view(torch.float8_e4m3fn), m.fc1_w, m.fc1_s, None, None, None, + None, m.fc2_w, m.fc2_s, None, m.out1, m.out1_gate, m.out2, core.num_experts, + core.topk, None, None, core.moe_inter, core.expert_offset, core.local_experts, None, + _MOE_ROUTING_METHOD, True, _MOE_ACT_SWIGLU, topk_weights=weights, topk_ids=ids, + tune_max_num_tokens=core.max_num_tokens, + )[0] # fmt: skip + return torch.add(routed, shared) + + def forward( + self, + attn_metadata: AttentionMetadata, + input_ids: torch.IntTensor | None = None, + position_ids: torch.IntTensor | None = None, + inputs_embeds: torch.FloatTensor | None = None, + lora_params: dict | None = None, + **kwargs, + ) -> torch.Tensor: + core = self.core + self._check_step_contract(attn_metadata) + step = self.step_args(attn_metadata) + h = ( + nn.functional.embedding(input_ids, self._embed) + if inputs_embeds is None + else inputs_embeds + ) + rows = h.shape[0] + dev = h.device + assert rows == step["num_tokens"] and rows % step["num_seqs"] == 0, ( + rows, + step["num_tokens"], + step["num_seqs"], + ) + # A separate graph is captured for steps whose sequences all fit in index_topk, so + # this host branch is a per-capture constant. + skip_gen = step["skip_gen"] + pos = torch.reshape(position_ids, [-1]) + req_idx = step["req_idx_per_token"][:rows] + if skip_gen: + topk_buf = step["topk_indices_buffer"][:rows] + else: + topk_buf = torch.empty([rows, core.idx_topk], dtype=torch.int32, device=dev) + scratch = self._scratch(rows, h.dtype, dev) + + x = flashinfer_rmsnorm(h, self._norm0, core.eps) + residual = h + spec_metadata = kwargs.get("spec_metadata") + for layer_idx, L in enumerate(self._layers): + attn_out = self._attention(step, L, x, pos, req_idx, topk_buf, scratch, skip_gen) + partial = self._linear_fp8(attn_out, L.wo, L.wo_s) + o = cute_dsl_allreduce_add_rmsnorm( + partial, residual, L.ffn_norm, torch.empty_like(partial), core._lamport + ) + mlp = self._mlp(o, L) + x = cute_dsl_allreduce_add_rmsnorm( + mlp, residual, L.next_norm, torch.empty_like(mlp), core._lamport + ) + if spec_metadata is not None: + spec_metadata.maybe_capture_hidden_states(layer_idx, residual) + return x + + def _scratch(self, rows: int, dtype, dev) -> SimpleNamespace: + """Per-step scratch every layer rewrites in turn: the attention output, the absorbed + query at this rank's heads (every element written by each layer: the absorption BMM + and the SSMax op's rope tail or zeros), a NoPE owner's 576-wide latent rows (the rope + key zero for the step), and the latent attention output the native decode writes. + Under `gen_attn_flash_mla` the query is FlashMLA's 64 heads (this rank's first, the + rest zero for the step), the scratch carries the FlashMLA scheduler the step's first + layer fills and the rest share (captured with the step), and the latent output is kept + only for the unfused glue, which copies FlashMLA's live heads into it.""" + core = self.core + bf16 = dict(dtype=dtype, device=dev) + if core.gen_attn_flash_mla: + fused_q = torch.zeros([rows, _FLASH_MLA_HEADS, core.lat_dim], **bf16) + else: + fused_q = torch.empty([rows, core.heads, core.lat_dim], **bf16) + return SimpleNamespace( + attn_out=torch.empty([rows, core.heads * core.v_dim], **bf16), + fused_q=fused_q, + nope_lat=None if core.unfused_glue else torch.zeros([rows, core.lat_dim], **bf16), + lat_out=( + torch.empty([rows, core.heads, core.kv_lora], **bf16) + if core.unfused_glue or not core.gen_attn_flash_mla + else None + ), + sched=get_mla_metadata()[0] if core.gen_attn_flash_mla else None, + ) + + def _attention(self, step, L, x, pos, req_idx, topk_buf, s, skip_gen: bool): + """One layer's attention block into `s.attn_out` (returned): the query with SSMax, the + indexer's selection, the absorbed query, the owner's rope and fp8_ds_mla append, + sparse decode over the pool, and the V up-projection of the latent output (FlashMLA's + live heads read in place under `gen_attn_flash_mla` with the fused glue).""" + core = self.core + rows = x.shape[0] + q = self._linear_fp8(x, L.wq, L.wq_s) + q3 = torch.reshape(q, [rows, core.heads, core.qk_dim]) + fq = s.fused_q[:, : core.heads] + if core.unfused_glue: + q3 = ssmax_scaling(q3, pos, L.ssmax_s, L.ssmax_b) + if L.rope: + fq[..., core.kv_lora :].copy_(q3[..., L.nope :]) + else: + fq[..., core.kv_lora :].zero_() + else: + # SSMax in place; its rope half (a NoPE layer's zero) into the absorbed query's tail. + ssmax_scaling_fused_q( + q3, pos, L.ssmax_s, L.ssmax_b, q3, fq[..., core.kv_lora :], L.rope + ) + if L.indexer is not None: + self._indexer(step, x, pos, L, topk_buf, skip_gen) + slots = convert_req_index_to_global( + req_idx, step["block_table"], topk_buf, step["tokens_per_block"], core.idx_topk, + core._stride_factor, core._pool_row[L.kv_source], + ) # fmt: skip + bmm_out( + torch.transpose(q3[..., : L.nope], 0, 1), L.k_b, + torch.transpose(fq[..., : core.kv_lora], 0, 1), + ) # fmt: skip + if L.owner: + ds_mla_rope_append( + self._latent(x, L, s.nope_lat), fq, pos, core._rope.indexer, req_idx, + step["block_table"], step["latent_pool"], core._pool_row[L.index], L.rope, + ) # fmt: skip + if core.gen_attn_flash_mla and core.unfused_glue: + lat = flash_mla_sparse_decode( + s.fused_q, step["latent_kv"], slots, s.sched, core.softmax_scale, s.lat_out + ) + elif core.gen_attn_flash_mla: + lat = flash_mla_sparse_decode( + s.fused_q, step["latent_kv"], slots, s.sched, core.softmax_scale, None + )[:, : core.heads] + else: + # The rank's heads, unpadded, into the latent output. + cute_dsl_ds_mla_sparse_decode( + s.fused_q, step["latent_kv"], slots, core.softmax_scale, s.lat_out, + compiled=core._mla_decode, + ) # fmt: skip + lat = s.lat_out + bmm_out( + torch.transpose(lat, 0, 1), + L.v_b_t, + torch.transpose(torch.reshape(s.attn_out, [rows, core.heads, core.v_dim]), 0, 1), + ) + return s.attn_out + + def dspark_forward( + self, + draft, + noise_embedding, + query_positions, + num_ctx_per_req, + ctx_k_cache, + ctx_v_cache, + ctx_cache_batch_idx, + ctx_kv_cache=None, + ctx_page_table=None, + block_placement=None, + ): + # Only the worker's fused paged-latent preparation places the block + # (block_placement), and a drafter with embed_draft_block does not take + # it; this decode reads its context window through ctx_page_table. + del block_placement + draft._build_fused_kv_buffers() + draft._ensure_rope() + core = self.core + batch, block, hidden = noise_embedding.shape + rows = batch * block + # Gather a bounded window, not every page in the configured context. + # Invalid padding slots are masked even if their pages are allocated. + lengths = num_ctx_per_req[:batch].long() + offsets = torch.arange(draft.window, device=noise_embedding.device) + positions = (lengths - draft.window).clamp_min(0)[:, None] + offsets[None, :] + valid = positions < lengths[:, None] + slots = ctx_cache_batch_idx.long() + if ctx_page_table is not None: + page_size = ctx_kv_cache[0].shape[-2] + table = ctx_page_table.index_select(0, slots).long() + safe_positions = positions.clamp(max=table.shape[1] * page_size - 1) + pages = table.gather(1, safe_positions // page_size).clamp_min(0) + page_offsets = safe_positions % page_size + else: + safe_positions = positions.clamp(max=ctx_k_cache.shape[2] - 1) + key_valid = torch.cat( + (valid, torch.ones(batch, block, dtype=torch.bool, device=valid.device)), dim=1 + ) + query = torch.empty( + batch, + block, + core.heads, + core.lat_dim, + dtype=noise_embedding.dtype, + device=noise_embedding.device, + ) + values = torch.empty( + rows, core.heads, core.v_dim, dtype=noise_embedding.dtype, device=noise_embedding.device + ) + residual = noise_embedding.reshape(rows, hidden).clone() + normalized = torch.empty_like(residual) + x = flashinfer_rmsnorm(residual, draft._layers[0].attn_norm, core.eps) + for i, layer in enumerate(draft._layers): + q = cublas_mm(x, layer.q, None, None).view(batch, block, core.heads, core.qk_dim) + bmm_out( + q[..., : core.nope].reshape(rows, core.heads, core.nope).transpose(0, 1), + layer.k_b, + query[..., : core.kv_lora].view(rows, core.heads, core.kv_lora).transpose(0, 1), + ) + query[..., core.kv_lora :].copy_(draft._rotate(q[..., core.nope :], query_positions)) + kv = cublas_mm(x, layer.kv_a, None, None) + content = flashinfer_rmsnorm( + kv[:, : core.kv_lora].contiguous(), layer.kv_norm, core.eps + ) + block_kv = torch.cat( + ( + content.view(batch, block, core.kv_lora), + draft._rotate( + kv[:, core.kv_lora :].view(batch, block, core.rope_dim), query_positions + ), + ), + dim=-1, + ) + if ctx_page_table is not None: + context = ctx_kv_cache[i][:, 0, 0][pages, page_offsets] + else: + context = ctx_k_cache[slots[:, None], i, safe_positions, 0] + # Zero masked values as well as scores: 0 * an uninitialized + # NaN in the latent pool must not contaminate the weighted sum. + context = context.masked_fill(~valid.unsqueeze(-1), 0) + keys = torch.cat((context, block_kv), dim=1) + scores = torch.einsum("bshd,btd->bsht", query, keys).float() * core.softmax_scale + scores = scores.masked_fill(~key_valid[:, None, None, :], float("-inf")) + probabilities = torch.softmax(scores, dim=-1).to(keys.dtype) + latent = torch.einsum("bsht,btc->bshc", probabilities, keys[..., : core.kv_lora]) + bmm_out( + latent.reshape(rows, core.heads, core.kv_lora).transpose(0, 1), + layer.v_b.transpose(1, 2), + values.transpose(0, 1), + ) + partial = cublas_mm(values.reshape(rows, -1), layer.out, None, None) + o = cute_dsl_allreduce_add_rmsnorm( + partial, residual, layer.ffn_norm, normalized, core._lamport + ) + gate = cublas_mm(o, layer.gate, None, None) + up = cublas_mm(o, layer.up, None, None) + activated = flashinfer_silu_and_mul(torch.cat((gate, up), dim=-1)) + mlp = cublas_mm(activated, layer.down, None, None) + x = cute_dsl_allreduce_add_rmsnorm( + mlp, residual, layer.next_norm, normalized, core._lamport + ) + return x + + +class DraftModel(nn.Module): + """This specialization's BF16 DSpark backbone and recurrent head. + + Runtime owns accepted-token bookkeeping and the paged context pool. The + draft attends the last trained window of captured target features plus + its whole non-causal query block. All parameters live in the core table. + """ + + _uses_worker_attention_backend = False + _paged_ctx_cache = True + _kv_factor = 1 + _dspark_shift_label = True + has_markov_head = True + has_recurrent_markov_head = True + + def __init__(self, shell, draft_config): + super().__init__() + # Keep references without registering the parent as a child module. + object.__setattr__(self, "_shell", shell) + self.model_config = draft_config + self.config = draft_config.pretrained_config + core = shell.model + spec = shell.spec_config + native = spec._draft_model_config + self.block_size = spec.block_size + self.mask_token_id = spec.mask_token_id + self.target_layer_ids = tuple(spec.target_layer_ids) + self.window = int(native["ragged_attention"].split(",")[0]) + self._num_attn_layers = int(native["n_layers"]) + self._num_heads = core.heads + self._num_kv_heads = 1 + self._head_dim = core.lat_dim + self.dflash_attention_backend = "VANILLA" + self._runtime_position_ceiling = None + self._rope = None + self._layers = None + + @property + def core(self): + return self._shell.model + + @property + def lm_head(self): + return self._shell.lm_head + + @property + def markov_w2(self): + return self.core.w["draft_markov_output"] + + def logits_processor(self, hidden, lm_head, attn_metadata, gather): + logits = self._shell.logits_processor(hidden, lm_head, attn_metadata, gather) + # The shared-head logits are soft-capped before the RNN bias is added. + return tanh_softcap(logits, self._shell.softcap) + + def apply_draft_logit_transform(self, logits): + """Return ``logits`` unchanged: :meth:`logits_processor` already soft-caps them. + + The worker calls this on the block logits after ``logits_processor``. + This drafter declares no ``logit_multiplier`` or ``logit_softcap``, so the + cap is applied once, in ``logits_processor``. + """ + return logits + + def _build_fused_kv_buffers(self): + # The worker calls this before sizing its cache. Take views once after + # materialization; context projection is explicit per layer here. + if self._layers is not None: + return + w, core = self.core.w, self.core + self._layers = [] + for i in range(self._num_attn_layers): + prefix = f"l{i}_draft_" + kv_b = w[prefix + "wkv_b"].view(core.heads, core.nope + core.v_dim, core.kv_lora) + self._layers.append( + SimpleNamespace( + attn_norm=w[prefix + "attn_norm"], + q=w[prefix + "wq"].t(), + kv_a=w[prefix + "wkv_a"].t(), + kv_norm=w[prefix + "kv_a_norm"], + k_b=kv_b[:, : core.nope].contiguous(), + v_b=kv_b[:, core.nope :].contiguous(), + out=w[prefix + "wo"].t(), + ffn_norm=w[prefix + "ffn_norm"], + gate=w[prefix + "w1"].t(), + up=w[prefix + "w3"].t(), + down=w[prefix + "w2"].t(), + next_norm=( + w[f"l{i + 1}_draft_attn_norm"] + if i + 1 < self._num_attn_layers + else w["draft_final_norm"] + ), + ) + ) + + def _ensure_rope(self): + cap = self._runtime_position_ceiling or self.core.rope_rows + if self._rope is None or self._rope.rows != cap: + w = self.core.w["draft_final_norm"] + # alpha=0 removes LeRoPE; all draft layers use plain YaRN. + self._rope = _rope_tables( + self.core.model_config.pretrained_config, + torch.zeros(self.core.rope_dim // 2), + cap, + w.dtype, + w.device, + ) + + def _rotate(self, x, positions): + table = self._rope.indexer[positions.long()] + cos, sin = table.chunk(2, dim=-1) + # One head axis for Q, none for the shared latent K. + if x.ndim == positions.ndim + 2: + cos, sin = cos.unsqueeze(-2), sin.unsqueeze(-2) + even, odd = x.float()[..., 0::2], x.float()[..., 1::2] + return ( + torch.stack((even * cos - odd * sin, even * sin + odd * cos), -1) + .flatten(-2) + .to(x.dtype) + ) + + def embed_draft_block(self, anchors, block_size): + # The route pins the trained block width; no mask token is looked up + # in the vocabulary (its sentinel is deliberately out of range). + weight = self.core.w["embed"] + valid = (anchors >= 0) & (anchors < weight.shape[0]) + embedded = nn.functional.embedding(anchors.clamp(0, weight.shape[0] - 1), weight) + embedded = embedded.masked_fill(~valid.unsqueeze(-1), 0) + masks = self.core.w["draft_mask"].unsqueeze(0).expand(anchors.shape[0], -1, -1) + return torch.cat((embedded.unsqueeze(1), masks), dim=1) + + def project_target_hidden(self, hidden): + w = self.core.w + projected = cublas_mm(hidden, w["draft_target_projection"].t(), None, None) + return flashinfer_rmsnorm(projected, w["draft_target_norm"], self.core.eps) + + def precompute_context_kv(self, projected_hidden, positions): + self._build_fused_kv_buffers() + self._ensure_rope() + core = self.core + latent = [] + for layer in self._layers: + kv = cublas_mm(projected_hidden, layer.kv_a, None, None) + content = flashinfer_rmsnorm( + kv[:, : core.kv_lora].contiguous(), layer.kv_norm, core.eps + ) + rope = self._rotate(kv[:, core.kv_lora :], positions.reshape(-1)) + latent.append(torch.cat((content, rope), dim=-1)) + return torch.stack(latent, dim=1).unsqueeze(2), None + + def dflash_forward_with_metadata(self, *, attn_metadata, **kwargs): + # The phase belongs to this engine step, including mixed batches. + return self.core._select_target(attn_metadata).dspark_forward(self, **kwargs) + + def apply_markov_chain_logits(self, base_logits, first_prev_tokens, *, hidden_states): + """The recurrent Markov bias over full-vocab ``[B, K, vocab]`` block logits. + + Each position's greedy token (the argmax of its biased logits) is the + next position's previous token; ``hidden_states`` is the ``[B, K, + hidden]`` block output the logits were projected from. + """ + w = self.core.w + output = w["draft_markov_output"] + previous = first_prev_tokens.long() + state = None + refined = [] + for i in range(base_logits.shape[1]): + weight = w["draft_markov_embedding"] + valid = (previous >= 0) & (previous < weight.shape[0]) + embedding = nn.functional.embedding(previous.clamp(0, weight.shape[0] - 1), weight) + embedding = embedding.masked_fill(~valid.unsqueeze(-1), 0) + if state is None: + state = torch.zeros_like(embedding) + joint = cublas_mm( + torch.cat((state, embedding, hidden_states[:, i]), dim=-1), + w["draft_joint"].t(), + w["draft_joint_bias"], + None, + ) + gate, candidate, emitted = joint.chunk(3, dim=-1) + gate = torch.sigmoid(gate) + state = gate * state + (1 - gate) * torch.tanh(candidate) + bias = cublas_mm(torch.tanh(emitted), output.t(), None, None) + logits = base_logits[:, i] + bias + refined.append(logits) + previous = logits.argmax(dim=-1) + return torch.stack(refined, dim=1) + + +class _VisionBlock(NamedTuple): + """One vision transformer block's weights, as the views the ops read: the GEMM + operands column-major (`.t()` of the stored rows).""" + + attn_norm: torch.Tensor + qkv: torch.Tensor + wo: torch.Tensor + ffn_norm: torch.Tensor + w13: torch.Tensor + w2: torch.Tensor + + +def _vision_step(seg_lens: list[int], device, workspace: torch.Tensor) -> dict: + """The batch state of one vision attention call: one context sequence per image, + no KV cache. Every value is host-known from the images' patch counts.""" + total = sum(seg_lens) + longest = max(seg_lens) + lens_dev = torch.tensor(seg_lens, dtype=torch.int32, device=device) + lens_host = torch.tensor(seg_lens, dtype=torch.int32) + return dict( + workspace=workspace, + sequence_length=lens_dev, + host_past_key_value_lengths=lens_host, + host_total_kv_lens=torch.tensor([total, 0], dtype=torch.int32), + context_lengths=lens_dev, + host_context_lengths=lens_host, + host_request_types=torch.zeros(len(seg_lens), dtype=torch.int32), + max_num_requests=len(seg_lens), + max_context_length=longest, + max_seq_len=longest, + attention_window_size=longest, + num_contexts=len(seg_lens), + num_ctx_tokens=total, + ) + + +def _vision_attention( + x: torch.Tensor, + B: _VisionBlock, + positions: torch.Tensor, + step: dict, + heads: int, + head_dim: int, + theta: float, +) -> torch.Tensor: + """One vision block's attention over the patch rows of every image of the call: + `[wq ; wk ; wv]` as one GEMM, q and k rotated by each patch's (row, col) in its + own image, attention bidirectional within each image, then `wo`.""" + rows = x.shape[0] + width = heads * head_dim + qkv = cublas_mm(x, B.qkv) + q, k = pixtral_rope_2d( + torch.reshape(qkv[:, :width], [rows, heads, head_dim]), + torch.reshape(qkv[:, width : 2 * width], [rows, heads, head_dim]), + positions, + theta, + ) + qkv = torch.cat( + [torch.reshape(q, [rows, width]), torch.reshape(k, [rows, width]), qkv[:, 2 * width :]], -1 + ) + out = torch.empty([rows, width], dtype=x.dtype, device=x.device) + out = thop_attention( + q=qkv, + output=out, + num_heads=heads, + num_kv_heads=heads, + head_size=head_dim, + **step, + **_VISION_ATTN, + ) + return cublas_mm(out, B.wo) + + +def _merge_tiles(x: torch.Tensor, grid_h: int, grid_w: int, merge: int) -> torch.Tensor: + """One image's `[grid_h * grid_w, d]` patch rows -> `[grid_h * grid_w / merge^2, + d * merge^2]`: each merge x merge tile of the grid as one row, tiles row-major (the + order of the image's [IMG] tokens), each row laid out (channel, tile row, tile + column) as the merging layer's input columns are.""" + d = x.shape[-1] + grid = torch.permute(torch.reshape(x, [grid_h, grid_w, d]), [2, 0, 1]) + tiles = nn.functional.unfold(torch.unsqueeze(grid, 0), kernel_size=merge, stride=merge) + return torch.t(tiles[0]) + + +class VisionTower: + """The image input path: a request's pixels -> one row per [IMG] token, in the + language model's hidden width. Run by the shell, in context steps only, never + captured; every rank runs the whole tower (all heads) over its own requests. + + Per image: the 14x14 patch embedding as an unfold and one GEMM, `ln_pre`, the + transformer blocks (RMSNorm, attention with 2-D RoPE, RMSNorm, SwiGLU, each with + its residual), the pre-projector RMSNorm, the 2x2 patch merger and the adapter + (`w_in`, exact GELU, `w_out`). The images of one call are concatenated along the + rows; only the attention needs their boundaries.""" + + def __init__(self, core) -> None: + """Take the tower's weight views and geometry, once, off the real weights.""" + d = _weights.MODEL_WEIGHTS.dims(core) + w = core.w + self.hidden = d.v_hidden + self.heads = d.v_heads + self.head_dim = d.v_head_dim + self.patch = d.v_patch + self.merge = d.v_merge + self.theta = d.v_theta + self.dtype = d.dtype + self.out_hidden = d.hidden + self._validate = step_contract_enabled() + self._rank = core.model_config.mapping.rank + self.patch_w = w["vis_patch"].t() + self.ln_pre = w["vis_ln_pre"] + self.blocks = [ + _VisionBlock( + attn_norm=w[f"v{j}_attn_norm"], + qkv=w[f"v{j}_qkv"].t(), + wo=w[f"v{j}_wo"].t(), + ffn_norm=w[f"v{j}_ffn_norm"], + w13=w[f"v{j}_w13"].t(), + w2=w[f"v{j}_w2"].t(), + ) + for j in range(d.v_layers) + ] + self.proj_norm = w["vis_proj_norm"] + self.merge_w = w["vis_merge"].t() + self.w_in = w["vis_w_in"].t() + self.w_out = w["vis_w_out"].t() + # The prompt token the tower's rows replace, on the host: the executor finds a + # step's image positions with it before anything reaches the device. + self.image_token_ids = torch.tensor([d.image_token_id], dtype=torch.int32) + # The attention op sizes its workspace in place per call and never shrinks it. + self._workspace = torch.empty(0, dtype=torch.int8, device=self.ln_pre.device) + + def _images(self, multimodal_params: list[MultimodalParams]) -> list[tuple]: + """`(pixels [C, H, W], grid rows, grid cols)` of every image, in order. The + mistral-native processor pads the batch and preserves each image's true size.""" + images = [] + unit = self.patch * self.merge + for param in multimodal_params: + image = (param.multimodal_data or {}).get("image") + if image is None: + continue + pixels = image["pixel_values"] + sizes = image.get("image_sizes") + if pixels.ndim != 4 or pixels.shape[1] != 3: + raise ValueError("Image pixels must have shape [N, 3, H, W]") + if sizes is None or len(sizes) != pixels.shape[0]: + raise ValueError("Image sizes must describe every image in the padded batch") + for b in range(pixels.shape[0]): + if len(sizes[b]) != 2: + raise ValueError("Each image size must be [height, width]") + height, width = int(sizes[b][0]), int(sizes[b][1]) + if not (0 < height <= pixels.shape[-2] and 0 < width <= pixels.shape[-1]): + raise ValueError("Image sizes must fit within their padded pixels") + if height % unit or width % unit: + raise ValueError( + f"image {height}x{width} is not a whole number of {unit}-pixel " + "merge tiles; the processor resizes to them" + ) + images.append( + (pixels[b, :, :height, :width], height // self.patch, width // self.patch) + ) + return images + + def forward(self, multimodal_params: list[MultimodalParams]) -> torch.Tensor: + """The rows of every image of `multimodal_params`, request by request, image by + image: `[sum of merged tiles, text hidden]`.""" + images = self._images(multimodal_params) + dev = self.ln_pre.device + if not images: + return torch.empty([0, self.out_hidden], dtype=self.dtype, device=dev) + seg = [gh * gw for _, gh, gw in images] + if self._validate: + logger.info( + "modeling_v2 image forward target=%s rank=%d grids=%s merged_rows=%d", + __name__.rsplit(".", 1)[0], + self._rank, + [(gh, gw) for _, gh, gw in images], + sum(seg) // self.merge**2, + ) + # Each patch as one row, (channel, row, col) within it, patches row-major. + patches = torch.cat( + [ + torch.t( + nn.functional.unfold( + torch.unsqueeze(px.to(device=dev, dtype=self.dtype), 0), + kernel_size=self.patch, + stride=self.patch, + )[0] + ) + for px, _, _ in images + ], + 0, + ) + positions = torch.cat( + [ + torch.reshape( + torch.stack( + torch.meshgrid( + torch.arange(gh, device=dev), + torch.arange(gw, device=dev), + indexing="ij", + ), + -1, + ), + [-1, 2], + ) + for _, gh, gw in images + ], + 0, + ) + step = _vision_step(seg, dev, self._workspace) + + h = flashinfer_rmsnorm(cublas_mm(patches, self.patch_w), self.ln_pre, _VISION_EPS) + x = flashinfer_rmsnorm(h, self.blocks[0].attn_norm, _VISION_EPS) + residual = h + for j, B in enumerate(self.blocks): + attn = _vision_attention(x, B, positions, step, self.heads, self.head_dim, self.theta) + x, residual = flashinfer_fused_add_rmsnorm(attn, residual, B.ffn_norm, _VISION_EPS) + mlp = cublas_mm(flashinfer_silu_and_mul(cublas_mm(x, B.w13)), B.w2) + norm_next = self.blocks[j + 1].attn_norm if j + 1 < len(self.blocks) else self.proj_norm + x, residual = flashinfer_fused_add_rmsnorm(mlp, residual, norm_next, _VISION_EPS) + + # x is the pre-projector-normed tower output. + tiles = torch.cat( + [ + _merge_tiles(xi, gh, gw, self.merge) + for xi, (_, gh, gw) in zip(torch.split(x, seg, 0), images) + ], + 0, + ) + merged = cublas_mm(tiles, self.merge_w) + return cublas_mm(gelu_erf(cublas_mm(merged, self.w_in)), self.w_out) + + +def _image_rows( + vision: Optional[VisionTower], multimodal_params: list[MultimodalParams] +) -> Optional[torch.Tensor]: + """This step's image rows, one per [IMG] position its context chunks cover, in + the order those positions appear; None when the step covers none. + + The tower runs once per request, on the first step that needs it; its rows stay + on the request's multimodal data for the later chunks. A chunk -- under chunked + prefill, or resuming after a reused prefix -- takes rows `[cached, cached + in + chunk)` of its request's, counted by the request's [IMG] cumsum.""" + if vision is None: + if any((p.multimodal_data or {}).get("image") is not None for p in multimodal_params): + raise ValueError( + "an image reached a Mistral Large 4 target whose checkpoint has no vision encoder" + ) + return None + # Filter before encoding and gathering so inactive requests neither run the + # tower nor shift the embedding offsets of active requests in a mixed batch. + multimodal_params = [ + param + for param in multimodal_params + if param.multimodal_runtime is None or param.multimodal_runtime.num_mm_tokens_in_chunk > 0 + ] + embeds = find_input_mm_embeds( + get_multimodal_embeddings(vision.forward, multimodal_params), multimodal_params + ) + if not embeds: + return None + rows = embeds[0] if len(embeds) == 1 else torch.cat(embeds, 0) + return rows if rows.shape[0] else None + + +def _splice_image_rows( + embed: torch.Tensor, + input_ids: torch.Tensor, + rows: torch.Tensor, + mm_token_indices: Optional[torch.Tensor], +) -> torch.Tensor: + """The step's input embeddings with `rows` written over its [IMG] positions, + `mm_token_indices` (which the executor computes on the host from the shell's + `multimodal_token_ids`); every other position keeps its token embedding.""" + if mm_token_indices is None: + raise ValueError("an image step arrived without the executor's mm_token_indices") + if mm_token_indices.shape[0] != rows.shape[0]: + raise ValueError( + f"{mm_token_indices.shape[0]} [IMG] positions in this step but " + f"{rows.shape[0]} image rows for them" + ) + h = nn.functional.embedding(input_ids, embed) + return h.index_copy_(0, mm_token_indices, rows.to(h.dtype)) + + +class ModelingV2MistralLarge4Nvfp4Sm103Tep( + DecoderModelForCausalLM[MistralLarge4TepModelingV2, PretrainedConfig] +): + def __init__(self, model_config: ModelConfig): + cfg = model_config.pretrained_config + super().__init__( + MistralLarge4TepModelingV2(model_config), + config=model_config, + hidden_size=cfg.hidden_size, + vocab_size=cfg.vocab_size, + ) + self.softcap = float(cfg.softmax_tanh_gating) + self._vision: Optional[VisionTower] = None + vis = getattr(cfg, "vision_config", None) + self._image_token_ids = ( + torch.tensor([vis.image_token_id], dtype=torch.int32) if vis is not None else None + ) + + self.spec_config = model_config.spec_config + self.draft_model = None + self.draft_config = None + self.spec_worker = None + if self.spec_config is not None: + # This is allocation geometry for the independently paged BF16 + # draft, not a second registry-resolved model or target cache. + draft_cfg = PretrainedConfig( + num_hidden_layers=int(self.spec_config._draft_model_config["n_layers"]), + hidden_size=cfg.hidden_size, + num_attention_heads=cfg.num_attention_heads, + num_key_value_heads=1, + kv_lora_rank=cfg.kv_lora_rank, + qk_rope_head_dim=cfg.qk_rope_head_dim, + qk_nope_head_dim=cfg.qk_nope_head_dim, + v_head_dim=cfg.v_head_dim, + intermediate_size=cfg.intermediate_size, + vocab_size=cfg.vocab_size, + max_position_embeddings=cfg.max_position_embeddings, + torch_dtype=torch.bfloat16, + ) + self.draft_config = replace( + model_config, + pretrained_config=draft_cfg, + quant_config=QuantConfig(), + quant_config_dict=None, + sparse_attention_config=None, + spec_config=None, + modeling_v2_target=None, + ) + self.draft_model = DraftModel(self, self.draft_config) + self.spec_worker = get_spec_worker( + self.spec_config, + model_config, + model_config.mapping, + use_separate_draft_kv_cache=True, + ) + self.spec_worker.set_draft_model(self.draft_model) + self.epilogue.append(self.spec_worker) + + def set_guided_decoder(self, guided_decoder: CapturableGuidedDecoder) -> bool: + if self.spec_worker is None: + return False + return self.spec_worker.set_guided_decoder(guided_decoder) + + @property + def multimodal_token_ids(self) -> Optional[torch.Tensor]: + return self._image_token_ids + + def forward( + self, + attn_metadata: AttentionMetadata, + input_ids: torch.IntTensor | None = None, + position_ids: torch.IntTensor | None = None, + inputs_embeds: torch.FloatTensor | None = None, + return_context_logits: bool = False, + spec_metadata=None, + lora_params: dict | None = None, + **kwargs, + ) -> torch.Tensor: + """Soft-cap target logits before verification and return the worker result.""" + worker_input_ids = input_ids + multimodal_params = kwargs.get("multimodal_params") or [] + if multimodal_params: + rows = _image_rows(self._vision, multimodal_params) + if rows is not None: + inputs_embeds = _splice_image_rows( + self.model.w["embed"], input_ids, rows, kwargs.get("mm_token_indices") + ) + input_ids = None + if self.spec_worker is None: + logits = super().forward( + attn_metadata, + cast(torch.IntTensor, input_ids), + position_ids, + inputs_embeds, + return_context_logits, + spec_metadata, + lora_params, + **kwargs, + ) + return tanh_softcap(logits, self.softcap) + hidden = self.model( + attn_metadata=attn_metadata, + input_ids=input_ids, + position_ids=position_ids, + inputs_embeds=inputs_embeds, + spec_metadata=spec_metadata, + lora_params=lora_params, + **kwargs, + ) + logits = self.logits_processor( + nn.functional.embedding(spec_metadata.gather_ids, hidden), + self.lm_head, + attn_metadata, + True, + ) + return self.spec_worker( + input_ids=worker_input_ids, + position_ids=position_ids, + hidden_states=hidden, + logits=tanh_softcap(logits, self.softcap), + attn_metadata=attn_metadata, + spec_metadata=spec_metadata, + draft_model=self.draft_model, + resource_manager=kwargs.get("resource_manager"), + ) + + def load_weights(self, weights, *args, **kwargs): + _weights.MODEL_WEIGHTS.load(self, weights) + + def load_checkpoint(self, checkpoint_dir: str) -> None: + _weights.MODEL_WEIGHTS.load_checkpoint(self, checkpoint_dir) + + def post_load_weights(self): + super().post_load_weights() + self.model.post_load_weights() + if step_contract_enabled(): + logger.info( + "modeling_v2 loaded target=%s class=%s rank=%d world=%d vision=%s", + __name__.rsplit(".", 1)[0], + type(self).__name__, + self.model_config.mapping.rank, + self.model_config.mapping.world_size, + getattr(self.config, "vision_config", None) is not None, + ) + if getattr(self.config, "vision_config", None) is not None: + self._vision = VisionTower(self.model) + if self.draft_model is not None: + self.draft_model._build_fused_kv_buffers() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/weights.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/weights.py new file mode 100644 index 000000000000..6ebc369954ec --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/weights.py @@ -0,0 +1,1337 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Weight table and loader: mistral-large-4-nvfp4 / sm_103 / tep, the language model. + +The checkpoint is Mistral-native (`consolidated-*.safetensors`, `layers.N.attention.wq`, ...) +and ModelOpt `MIXED_PRECISION`: the routed experts NVFP4, the other big linears FP8 128x128 +block (`FP8_PB_WO`), everything else bf16. Every rank is handed the whole of it, so every split +is this table's, every width read off the mapping (`tp_size` and `moe_ep_size` 4 on one node, or +8 over two nodes of four): + +* **tensor-parallel slices** -- `wq` by output rows and `wo` by input columns (this rank's 16 + heads at 4 ranks, 8 heads -- 1024 rows or columns -- at 8); `wkv_b` by heads; the MLA SSMax + `s`/`b` by heads; the dense layer's and the shared expert's `w1`/`w3` by rows and `w2` by + columns (6144 and 1536 per rank at 4 ranks, 3072 and 768 at 8); the shell's `lm_head` by + vocabulary rows (51200 or 25600 per rank). Every slice is whole 128-blocks, and a block scale + is sliced on the same boundary. The slice is taken on the checkpoint entry before the copy to + the device, so a rank moves only its own bytes. That is the `shard` column of `W`. +* **expert windows** -- rank `r` loads routed experts `[Er, Er + E)`, `E = n_routed_experts / + moe_ep_size`: 32 at 4 ranks, 16 at 8. +* **replicated** -- `wkv_a_with_mqa` and `kv_a_norm`, the whole indexer (all 32 of its heads), + the router, the embedding, every norm and `lerope.alpha`. + +Layer sets come from the config: `wkv_a_with_mqa` and `kv_a_norm` exist on the KV-owning layers +only (`kv_source_layers[i] == i`; the sharers read their source's latent), the indexer on the +`index_sharing == 'full'` layers only, the dense MLP on the first `first_k_dense_replace` +layers and the router, shared expert and routed experts on the rest. + +Load-time layouts, each the one its consumer reads: + +* `wkv_b` regrouped per head `[k_nope | v]` -> `[all k_nope ; all v]` over this rank's heads + (k_nope is 64 wide on RoPE layers and 128 on NoPE layers); +* the dense MLP and the shared expert `[w1 ; w3]`, gate rows first, the half order + `flashinfer_silu_and_mul` reads; +* FP8 linears (`wq`, `wo`, the indexer's `wq_b`, the dense and shared `w1`/`w3`/`w2`): e4m3 + `[N, K]` as stored, and the fp32 block scales -- stored as `.weight_scale` + `[N/128, 1, K/128, 1]`, squeezed here to the `[N/128, K/128]` grid -- in DeepGEMM's packed + UE8M0 operand, `transform_sf_into_required_layout(recipe=(1, 128, 128))`: what + `FP8BlockScalesLinearMethod.transform_weights` installs as `weight_scale` on sm_100f. That + operand is an MN-major `[N, ceil(K/512)]` int32 view; the parameter holds the dense + `[ceil(K/512), N]` behind it and the GEMM takes `.t()`. Where K/128 is not a multiple of four + the last word of each row is partly padding: the shared expert's `w2` at K 768 (8 ranks) has 6 + blocks in 2 words, and bytes 2 and 3 of the second word are zero. The method's + `resmooth_to_fp8_e8m0` is not run: every scale is asserted an exact power of two, which makes + it a power-of-two re-exponent of the same values; +* the indexer's `wk` and `weights_proj` stacked into one `[128 + 32, hidden]` bf16 GEMM operand; + its `k_norm` and `lerope.alpha` widened to fp32. + +The routed experts are this rank's window stacked on an expert axis, in the NVFP4 MoE runner's +layout: e2m1 codes, two per byte, `[E, 2I, H/2]` / `[E, H, I/2]` uint8 and their e4m3 block +scales, one per 16 inputs, `[E, 2I, H/16]` / `[E, H, I/16]`. Each source lands as stored -- +FC1's rows `[w3 ; w1]`, up rows first, the order the runner's FC1 concatenates them in -- and +once every source of an expert stack has landed, its row's `layout` rearranges each expert in +place (the runner reads nothing else, and a wrong layout costs 300-3400 bf16 ulp and raises +nothing): + +* FC1, codes and block scales alike: the halves interleaved (row `2i` up row `i`, row `2i + 1` + gate row `i`), then the 32-row block shuffle (within each 32 rows, row `4u + v` moves to + `8v + u`); +* FC2: the 32-row block shuffle; +* both block scales: then the 128x4 swizzle (`torch.ops.trtllm.block_scale_interleave`'s order, + written here in torch so the table also runs on meta tensors). + +The runner's output scalars are not derived here: the four per-tensor scalars are handed over +as stored -- `isc1` and `isc2`, the one FC1 and FC2 activation `input_scale` of the layer, and +per local expert `ws2_13` (`w1`'s `weight_scale_2`, which `w3` shares) and `ws2_2`. ModelOpt +stores these as reciprocals of the global scales a quantizer divides by. + +Before any of that is read, `load` reads the per-tensor scalars of all the layer's experts, +the other ranks' included, and asserts what the forward relies on: `w1` and `w3` store the same +`input_scale` and `weight_scale_2` on every expert, so one FC1 alpha serves the stacked +`[w3 ; w1]`; and each projection stores one `input_scale` for every expert, so one NVFP4 +quantization of the replicated activations is right for every expert on every rank and the +windows' partial outputs sum. + +The BF16 vision encoder, pre-projector norm, patch merger and language adapter are +replicated on every rank when the configuration declares a vision encoder. + +`expected_unconsumed` accounts for off-rank routed experts and disabled DSpark draft weights. +Off-rank expert scalars are still read to check shared scales; their codes and block scales +are never read. +At `moe_ep_size` 1 every expert belongs to the rank and is read, sliced. +""" + +from __future__ import annotations + +import collections +import functools +import json +import os +from dataclasses import dataclass +from pathlib import Path +from types import SimpleNamespace +from typing import Any, Callable + +import torch +import torch.nn as nn +from safetensors import safe_open + +from tensorrt_llm.logger import logger +from tensorrt_llm.quantization.utils.fp8_utils import transform_sf_into_required_layout + +F8, F32, I32, U8 = torch.float8_e4m3fn, torch.float32, torch.int32, torch.uint8 + +#: The FP8 block edge on both weight axes (the checkpoint's `FP8_PB_WO` 128x128 blocks), and +#: the K granularity of the DeepGEMM scale recipe `(1, 128, 128)`. +BLOCK = 128 +#: UE8M0 bytes per int32 word of the DeepGEMM scale operand: four K blocks. +SF_PACK = 4 +#: NVFP4: one e4m3 block scale per 16 inputs (the checkpoint's `group_size`), two e2m1 codes +#: per byte. +NVFP4_BLOCK = 16 +E2M1_PER_BYTE = 2 + +#: A routed expert's three projections: `w1` gate, `w3` up, `w2` down. +PROJECTIONS = ("w1", "w2", "w3") +#: The per-tensor NVFP4 scalars each of them carries beside its codes and block scales. +EXPERT_SCALARS = ("weight_scale_2", "input_scale") + +#: The vision tower's checkpoint prefixes. +VISION_PREFIXES = ( + "vision_encoder.", + "patch_merger.", + "pre_mm_projector_norm.", + "vision_language_adapter.", +) + + +#: `layers=ALL_LAYERS` is the common case and the default. A sentinel rather +#: than a lambda so the table reads as data, and rather than `True` so the +#: field has one type of answer: which layers. +ALL_LAYERS = object() + + +@dataclass(frozen=True) +class W: + """One weight role: what it is, where its bytes come from, what happens on the way. + + `shape` takes the dimension bundle and returns the declared shape. + + `src` says where the bytes come from: a checkpoint key template with `{p}` + for the layer prefix (`layers.N`), or a callable `(d, i) -> [(key, index), ...]` + where `index` is where in the parameter that tensor lands (`None`: all of it). + A `key` may be a tuple of per-tensor scalars, which `transform` is handed together. + + `layers` says which layers carry this weight: `ALL_LAYERS`, `None` for a + weight that is not per-layer, or a callable `d -> iterable[int]`. Two rows + may share a name over disjoint layer sets, with different shapes. + + `shard` takes the dimension bundle and returns a function of the checkpoint + entry: this rank's tensor-parallel slice, applied to every source of the row. + It runs before the copy to the device and before `transform`, which is what + `wkv_b` relies on: the regroup reads this rank's head count off the slice. + + `transform` runs on the (sliced) source tensor, on the destination's device. + + `layout` runs once every source of the parameter has landed, on each slot of its leading + (expert) axis, and the slot is overwritten with the result: a rearrangement that mixes rows + several sources fill, which no single source's `transform` can make. + """ + + name: str + shape: Callable[[Any], tuple[int, ...]] + src: str | Callable + dtype: torch.dtype | None = None + transform: Callable | None = None + layers: object | None = ALL_LAYERS + shard: Callable | None = None + layout: Callable | None = None + when: Callable | None = None + tag: str = "l" + + def layer_indices(self, d: SimpleNamespace) -> list[int] | None: + """The layers this weight exists on, or None if it is not per-layer.""" + if self.layers is None: + return None + if self.layers is ALL_LAYERS: + return list(range(d.num_layers)) + return list(self.layers(d)) + + def sources(self, d: SimpleNamespace, i: int | None, prefix: str) -> list[tuple[Any, Any]]: + """`[(checkpoint key | key tuple, index into the parameter | None), ...]`.""" + if callable(self.src): + return list(self.src(d, i)) + return [(self.src.format(p=prefix), None)] + + +def _keys(ckpt) -> tuple[str, ...]: + """The checkpoint keys of one source: itself, or the members of a key tuple.""" + return ckpt if isinstance(ckpt, tuple) else (ckpt,) + + +def _rows(start, width): + return lambda d: (lambda e: e[start(d) : start(d) + width(d)]) + + +def _cols(start, width): + return lambda d: (lambda e: e[:, start(d) : start(d) + width(d)]) + + +def _sf_cols(start, width): + """`_cols` on an FP8 block scale as stored, `[N/128, 1, K/128, 1]`: its K blocks are axis 2.""" + return lambda d: (lambda e: e[:, :, start(d) : start(d) + width(d)]) + + +def _sf_words(k: int) -> int: + """int32 words per row of the DeepGEMM scale operand of a K-wide weight.""" + return -(-k // (BLOCK * SF_PACK)) + + +def _assert_ue8m0(s: torch.Tensor) -> None: + """Every block scale is a positive normal power of two, so its UE8M0 byte (the fp32 + exponent field) holds it exactly. A meta tensor carries no values to check.""" + if s.is_meta: + return + bits = s.view(torch.int32) + exp = (bits >> 23) & 0xFF + exact = ((bits & 0x807FFFFF) == 0) & (exp > 0) & (exp < 255) + assert bool(exact.all()), ( + f"weight_scale: {int((~exact).sum())} of {s.numel()} block scales are not a positive " + "normal power of two (UE8M0); the scale layouts below would change their values" + ) + + +def _deepgemm_sf(s: torch.Tensor, core) -> torch.Tensor: + """fp32 `[N/128, K/128]` block scales -> the dense `[ceil(K/512), N]` int32 storage of + DeepGEMM's MN-major packed UE8M0 operand, whose `.t()` is that operand. N is a multiple of + 128 for every role here, so the operand's row stride is N and its storage has no MN padding. + A K of a block count that is not a multiple of four (the shared expert's `w2` at K 768 on 8 + ranks: 6 blocks) leaves the high bytes of each row's last word zero.""" + _assert_ue8m0(s) + n_blk, k_blk = s.shape + sf = transform_sf_into_required_layout( + s, mn=n_blk * BLOCK, k=k_blk * BLOCK, recipe=(1, BLOCK, BLOCK), is_sfa=False + ) + return sf.t() + + +def _block_sf(s: torch.Tensor, core) -> torch.Tensor: + """An FP8 linear's block scales as stored, fp32 `[N/128, 1, K/128, 1]`: squeezed to the + `[N/128, K/128]` grid, then `_deepgemm_sf`.""" + assert s.dim() == 4 and s.shape[1] == 1 and s.shape[3] == 1, ( + f"FP8 block scale stored as {tuple(s.shape)}, not [N/128, 1, K/128, 1]" + ) + return _deepgemm_sf(s.reshape(s.shape[0], s.shape[2]), core) + + +def _shared_scalar(t: tuple[torch.Tensor, ...], core) -> torch.Tensor: + """One per-tensor scalar the checkpoint stores several times (`w1`'s and `w3`'s, or every + expert's of the window): asserted the same value everywhere, then that value.""" + first = t[0] + if not first.is_meta: + values = torch.stack(t) + assert bool((values == first).all()), ( + f"{int((values != first).sum())} of {len(t)} copies of one shared scale differ from " + f"the first, {first.item()!r}" + ) + return first + + +def _regroup_kv_b(t: torch.Tensor, core) -> torch.Tensor: + """This rank's `[heads * (k_nope + v), kv_lora]` rows, per-head `[k_nope | v]` -> + `[heads * k_nope ; heads * v]`: the K-B operand, then the V-B one. k_nope is read off the + slice: 64 on RoPE layers, 128 on NoPE layers.""" + cfg = core.model_config.pretrained_config + heads = cfg.num_attention_heads // core.model_config.mapping.tp_size + v = cfg.v_head_dim + t3 = t.reshape(heads, t.shape[0] // heads, t.shape[1]) + nope = t3.shape[1] - v + return torch.cat( + [t3[:, :nope, :].reshape(heads * nope, -1), t3[:, nope:, :].reshape(heads * v, -1)], dim=0 + ).contiguous() + + +def _to_fp32(t: torch.Tensor, core) -> torch.Tensor: + return t.to(torch.float32) + + +# --- the NVFP4 MoE runner's expert layout, one expert at a time --- + +#: The runner's row shuffle works on blocks of this many rows. +SHUFFLE_ROWS = 32 +#: The block-scale swizzle's tile: 128 rows by 4 scale columns. +SWIZZLE_ROWS, SWIZZLE_COLS = 128, 4 + + +@functools.lru_cache(maxsize=None) +def _block32_perm(rows: int, device) -> torch.Tensor: + """Gather index of the 32-row block shuffle: within each block of 32 rows, source row + `4u + v` lands at row `8v + u`.""" + assert rows % SHUFFLE_ROWS == 0, rows + j = torch.arange(SHUFFLE_ROWS) + one = torch.empty(SHUFFLE_ROWS, dtype=torch.long) + one[(j % 4) * 8 + j // 4] = j + blocks = torch.arange(rows // SHUFFLE_ROWS).repeat_interleave(SHUFFLE_ROWS) * SHUFFLE_ROWS + return (one.repeat(rows // SHUFFLE_ROWS) + blocks).to(device) + + +@functools.lru_cache(maxsize=None) +def _fc1_perm(rows: int, device) -> torch.Tensor: + """Gather index of FC1's `[up ; gate]` rows: interleaved (row `2i` up row `i`, row `2i + 1` + gate row `i`), then block-shuffled.""" + interleave = torch.empty(rows, dtype=torch.long) + interleave[0::2] = torch.arange(rows // 2) + interleave[1::2] = torch.arange(rows // 2, rows) + return interleave.to(device)[_block32_perm(rows, device)] + + +def _swizzle(s: torch.Tensor) -> torch.Tensor: + """One expert's `[M, C]` e4m3 block scales in the 128x4-swizzled byte order, kept `[M, C]`: + byte `(m, c)` lands at flat offset `(m // 128) * 128 * C + (c // 4) * 512 + (m % 32) * 16 + + ((m % 128) // 32) * 4 + c % 4`. Both of this target's scale stacks tile it exactly (M a + multiple of 128, C of 4), so there is no padding.""" + m, c = s.shape + assert m % SWIZZLE_ROWS == 0 and c % SWIZZLE_COLS == 0, (m, c) + tiles = s.view(torch.uint8).reshape(m // SWIZZLE_ROWS, 4, 32, c // SWIZZLE_COLS, SWIZZLE_COLS) + return tiles.permute(0, 3, 2, 1, 4).reshape(m, c).view(s.dtype) + + +def _fc1_codes(w: torch.Tensor) -> torch.Tensor: + return torch.index_select(w, 0, _fc1_perm(w.shape[0], w.device)) + + +def _fc1_scales(s: torch.Tensor) -> torch.Tensor: + rows = torch.index_select(s.view(torch.uint8), 0, _fc1_perm(s.shape[0], s.device)) + return _swizzle(rows.view(s.dtype)) + + +def _fc2_codes(w: torch.Tensor) -> torch.Tensor: + return torch.index_select(w, 0, _block32_perm(w.shape[0], w.device)) + + +def _fc2_scales(s: torch.Tensor) -> torch.Tensor: + rows = torch.index_select(s.view(torch.uint8), 0, _block32_perm(s.shape[0], s.device)) + return _swizzle(rows.view(s.dtype)) + + +def _materialize(entry, shard) -> torch.Tensor: + """This rank's bytes of one checkpoint entry: a tensor, or a lazy safetensors slice, which + reads only what the index selects. `[...]`, not `[:]`: the per-tensor scalars are 0-dim.""" + return entry[...] if shard is None else shard(entry) + + +#: The checkpoint's shard index, beside its `consolidated-*.safetensors`. +SHARD_INDEX = "consolidated.safetensors.index.json" + + +def _checkpoint_shards(checkpoint_dir) -> tuple[Path, dict[str, str]]: + """`(directory, key -> shard file)` of a checkpoint directory: its index when it ships + one, else the header of each `consolidated*.safetensors`.""" + directory = Path(checkpoint_dir) + index = directory / SHARD_INDEX + if index.is_file(): + return directory, json.loads(index.read_text())["weight_map"] + weight_map: dict[str, str] = {} + for path in sorted(directory.glob("consolidated*.safetensors")): + with safe_open(str(path), framework="pt", device="cpu") as handle: + weight_map.update(dict.fromkeys(handle.keys(), path.name)) + if not weight_map: + raise FileNotFoundError(f"no consolidated*.safetensors in {directory}") + return directory, weight_map + + +def _shard_files(core, weights) -> tuple[Path, dict[str, str]] | None: + """`(directory, key -> shard file)` of the checkpoint a dict of sources was read from, or + None when the sources are not the shard files of one (the directory the config records + lacks the index, or the index does not name every key `weights` holds).""" + cfg = getattr(core.model_config, "pretrained_config", None) + root = getattr(cfg, "_name_or_path", None) + if not root: + return None + index = Path(root) / SHARD_INDEX + if not index.is_file(): + return None + weight_map = json.loads(index.read_text())["weight_map"] + if not set(weights.keys()) <= set(weight_map): + return None + return Path(root), weight_map + + +def _drop_page_cache(path: Path) -> None: + """Advise the kernel to drop a consumed shard file's cached pages. Advice only: a + filesystem that ignores it costs memory pressure, not correctness.""" + try: + fd = os.open(path, os.O_RDONLY) + except OSError: + return + try: + os.posix_fadvise(fd, 0, 0, os.POSIX_FADV_DONTNEED) + except OSError: + pass + finally: + os.close(fd) + + +# --- which layers carry a weight --- + + +def _owners(d): + """Layers that own a latent KV. The others read `kv_source_layers[i]`'s and project none.""" + return [i for i in range(d.num_layers) if d.kv_source[i] == i] + + +def _rope_owners(d): + return [i for i in _owners(d) if not d.nope_layer[i]] + + +def _nope_owners(d): + return [i for i in _owners(d) if d.nope_layer[i]] + + +def _rope_layers(d): + return [i for i in range(d.num_layers) if not d.nope_layer[i]] + + +def _nope_layers(d): + return [i for i in range(d.num_layers) if d.nope_layer[i]] + + +def _indexer_layers(d): + """Layers that own an indexer. The 'shared' ones reuse the most recent one's selection.""" + return [i for i in range(d.num_layers) if d.full_indexer[i]] + + +def _dense_layers(d): + return range(d.dense_layers) + + +def _moe_layers(d): + return range(d.dense_layers, d.num_layers) + + +# --- where the bytes come from --- + + +def _stack(*parts, prefix="layers"): + """Several checkpoint tensors into one parameter: each `(key template, index(d))`, the + index being where in the parameter that tensor lands.""" + return lambda d, i: [(t.format(p=f"{prefix}.{i}"), index(d)) for t, index in parts] + + +def _expert_key(i: int, e: int, suffix: str) -> str: + return f"layers.{i}.experts.{e}.{suffix}" + + +def _expert_window(*parts): + """This rank's routed experts, local slot `s` <- global expert `expert_offset + s`: each + `(checkpoint suffix | suffix tuple, index(d) after the expert axis)`.""" + + def key(i, e, suffix): + if isinstance(suffix, tuple): + return tuple(_expert_key(i, e, s) for s in suffix) + return _expert_key(i, e, suffix) + + def build(d, i): + return [ + (key(i, d.expert_offset + slot, suffix), (slot, *index(d))) + for slot in range(d.local_experts) + for suffix, index in parts + ] + + return build + + +def _window_scalar(*suffixes): + """One per-tensor scalar every expert of the window stores: all of the window's copies, one + key tuple, into element 0.""" + return lambda d, i: [ + ( + tuple( + _expert_key(i, d.expert_offset + slot, s) + for slot in range(d.local_experts) + for s in suffixes + ), + (0,), + ) + ] + + +def _expert_scalar_keys(d) -> list[str]: + """Every routed expert's per-tensor scalars on every MoE layer, the other ranks' included: + what `load` reads, and checks, before anything else.""" + return [ + _expert_key(i, e, f"{proj}.{s}") + for i in _moe_layers(d) + for e in range(d.num_experts) + for proj in PROJECTIONS + for s in EXPERT_SCALARS + ] + + +def _check_expert_scalars(scalars: dict[str, torch.Tensor], d) -> None: + """Per MoE layer, over all of its experts: `w1` and `w3` store the same `input_scale` and + `weight_scale_2` on every expert; `w1` and `w2` each store one `input_scale` for every expert; + every scalar is a positive finite fp32. Meta tensors carry no values to check.""" + if scalars and next(iter(scalars.values())).is_meta: + return + experts = range(d.num_experts) + for i in _moe_layers(d): + v = { + (proj, s): torch.stack([scalars[_expert_key(i, e, f"{proj}.{s}")] for e in experts]) + for proj in PROJECTIONS + for s in EXPERT_SCALARS + } + for (proj, s), t in v.items(): + bad = (~(torch.isfinite(t) & (t > 0))).nonzero().flatten() + assert bad.numel() == 0, ( + f"layers.{i}: {proj}.{s} of expert {int(bad[0])} is {t[bad[0]].item()!r}, " + "not a positive finite scale" + ) + for s in EXPERT_SCALARS: + bad = (v["w1", s] != v["w3", s]).nonzero().flatten() + assert bad.numel() == 0, ( + f"layers.{i}: w1 and w3 {s} differ on {bad.numel()} of {len(experts)} experts, " + f"first expert {int(bad[0])}: {v['w1', s][bad[0]].item()!r} vs " + f"{v['w3', s][bad[0]].item()!r}" + ) + for proj in ("w1", "w2"): + isc = v[proj, "input_scale"] + bad = (isc != isc[0]).nonzero().flatten() + assert bad.numel() == 0, ( + f"layers.{i}: {proj}.input_scale is not one value over the experts: expert " + f"{int(bad[0])} stores {isc[bad[0]].item()!r}, expert 0 {isc[0].item()!r}" + ) + + +def _read_scalars(keys: list[str], weights, shards) -> dict[str, torch.Tensor]: + """The given per-tensor scalars, off the shard files one file at a time when they are + found, otherwise off the given dict.""" + if shards is None: + return {key: _materialize(weights[key], None) for key in keys} + directory, weight_map = shards + by_file: dict[str, list] = collections.defaultdict(list) + for key in keys: + by_file[weight_map[key]].append(key) + out = {} + for name in sorted(by_file): + with safe_open(str(directory / name), framework="pt", device="cpu") as handle: + for key in by_file[name]: + out[key] = handle.get_slice(key)[...].clone() + return out + + +def _fp8_linear(name, key, n, k, layers, w_shard=None, s_shard=None): + """An FP8 128x128-block linear: e4m3 `[N, K]` and its DeepGEMM scale operand's storage.""" + return ( + W(name, shape=lambda d: (n(d), k(d)), dtype=F8, src=key, layers=layers, shard=w_shard), + W( + f"{name}_s", + shape=lambda d: (_sf_words(k(d)), n(d)), + dtype=I32, + src=key.replace(".weight", ".weight_scale"), + transform=_block_sf, + layers=layers, + shard=s_shard, + ), + ) + + +def _tp_mlp(module, inter, layers): + """A SwiGLU MLP split over its intermediate dimension: `[w1 ; w3]` by output rows, gate + first (the half order `flashinfer_silu_and_mul` reads), `w2` by input columns. The same + row slice applies to `w1` and `w3` before they are stacked, and the scales follow in + 128-blocks; in the scale storage the stacked axis is the second one.""" + blocks = lambda d: inter(d) // BLOCK # noqa: E731 + start = lambda d: d.tp_rank * inter(d) # noqa: E731 + b_start = lambda d: d.tp_rank * blocks(d) # noqa: E731 + p = "{p}." + module + return ( + W( + "mlp_gu", + shape=lambda d: (2 * inter(d), d.hidden), + dtype=F8, + src=_stack( + (f"{p}.w1.weight", lambda d: (slice(0, inter(d)),)), + (f"{p}.w3.weight", lambda d: (slice(inter(d), 2 * inter(d)),)), + ), + layers=layers, + shard=_rows(start, inter), + ), + W( + "mlp_gu_s", + shape=lambda d: (_sf_words(d.hidden), 2 * inter(d)), + dtype=I32, + src=_stack( + (f"{p}.w1.weight_scale", lambda d: (slice(None), slice(0, inter(d)))), + (f"{p}.w3.weight_scale", lambda d: (slice(None), slice(inter(d), 2 * inter(d)))), + ), + transform=_block_sf, + layers=layers, + shard=_rows(b_start, blocks), + ), + *_fp8_linear( + "mlp_dn", + f"{p}.w2.weight", + lambda d: d.hidden, + inter, + layers, + w_shard=_cols(start, inter), + s_shard=_sf_cols(b_start, blocks), + ), + ) + + +def _kv_b(nope, layers): + """`wkv_b`, this rank's heads, regrouped. Per-head rows are `nope(d) + v` wide.""" + width = lambda d: d.heads * (nope(d) + d.v_dim) # noqa: E731 + return W( + "wkv_b", + shape=lambda d: (width(d), d.kv_lora), + src="{p}.attention.wkv_b.weight", + transform=_regroup_kv_b, + layers=layers, + shard=_rows(lambda d: d.tp_rank * width(d), width), + ) + + +def _heads(width): + """This rank's slice of a per-head axis of the given per-head width.""" + return _rows(lambda d: d.tp_rank * d.heads * width(d), lambda d: d.heads * width(d)) + + +def _vision_blocks(d): + return list(range(d.v_layers)) + + +def _has_vision(d): + return d.has_vision + + +def _flatten_patch_conv(tensor, core): + return tensor.flatten(1) + + +def _draft_layers(d): + return range(d.draft_layers) + + +def _draft_source(suffix): + return lambda d, i: [(f"mtp_dspark.backbone.layers.{i}.{suffix}", None)] + + +class MistralLarge4TepWeights: + """mistral-large-4-nvfp4 / sm_103 / tep.""" + + WEIGHTS: tuple[W, ...] = ( + # --- MLA: every layer, KV owners only where marked --- + W("attn_norm", shape=lambda d: (d.hidden,), src="{p}.attention_norm.weight"), + # Per head `[nope 64 | rope 64]` on RoPE layers, 128 nope on NoPE layers: 128 either way. + *_fp8_linear( + "wq", + "{p}.attention.wq.weight", + lambda d: d.heads * d.qk_dim, + lambda d: d.hidden, + ALL_LAYERS, + w_shard=_heads(lambda d: d.qk_dim), + s_shard=_heads(lambda d: d.qk_dim // BLOCK), + ), + W( + "ssmax_s", + shape=lambda d: (d.heads,), + src="{p}.attention.ssmax_scaling.s", + shard=_heads(lambda d: 1), + ), + W( + "ssmax_b", + shape=lambda d: (d.heads,), + src="{p}.attention.ssmax_scaling.b", + shard=_heads(lambda d: 1), + ), + # `[c_kv 512 | k_pe 64]` on RoPE owners; NoPE owners project the 512 latent alone. + W( + "wkv_a", + shape=lambda d: (d.lat_dim, d.hidden), + src="{p}.attention.wkv_a_with_mqa.weight", + layers=_rope_owners, + ), + W( + "wkv_a", + shape=lambda d: (d.kv_lora, d.hidden), + src="{p}.attention.wkv_a_with_mqa.weight", + layers=_nope_owners, + ), + W( + "kv_a_norm", + shape=lambda d: (d.kv_lora,), + src="{p}.attention.kv_a_norm.weight", + layers=_owners, + ), + # Every layer absorbs through its own `wkv_b`, KV sharers included. + _kv_b(lambda d: d.nope, _rope_layers), + _kv_b(lambda d: d.qk_dim, _nope_layers), + *_fp8_linear( + "wo", + "{p}.attention.wo.weight", + lambda d: d.hidden, + lambda d: d.heads * d.v_dim, + ALL_LAYERS, + w_shard=_cols(lambda d: d.tp_rank * d.heads * d.v_dim, lambda d: d.heads * d.v_dim), + s_shard=_sf_cols( + lambda d: d.tp_rank * d.heads * d.v_dim // BLOCK, + lambda d: d.heads * d.v_dim // BLOCK, + ), + ), + W("ffn_norm", shape=lambda d: (d.hidden,), src="{p}.ffn_norm.weight"), + # --- the DSA indexer: replicated, on the 'full' layers --- + W( + "ix_q_norm_w", + shape=lambda d: (d.hidden,), + src="{p}.attention.indexer.q_input_norm.weight", + layers=_indexer_layers, + ), + W( + "ix_q_norm_b", + shape=lambda d: (d.hidden,), + src="{p}.attention.indexer.q_input_norm.bias", + layers=_indexer_layers, + ), + # Per head `[rope 64 | nope 64]`, the reverse of MLA's. + *_fp8_linear( + "ix_wq_b", + "{p}.attention.indexer.wq_b.weight", + lambda d: d.idx_heads * d.idx_dim, + lambda d: d.hidden, + _indexer_layers, + ), + # `[wk ; weights_proj]`: the key and the head weights, one GEMM. + W( + "ix_wk_wp", + shape=lambda d: (d.idx_dim + d.idx_heads, d.hidden), + src=_stack( + ("{p}.attention.indexer.wk.weight", lambda d: (slice(0, d.idx_dim),)), + ( + "{p}.attention.indexer.weights_proj.weight", + lambda d: (slice(d.idx_dim, d.idx_dim + d.idx_heads),), + ), + ), + layers=_indexer_layers, + ), + # fp32, as the fused key entry (LayerNorm in fp32) takes them. + W( + "ix_k_norm_w", + shape=lambda d: (d.idx_dim,), + dtype=F32, + src="{p}.attention.indexer.k_norm.weight", + transform=_to_fp32, + layers=_indexer_layers, + ), + W( + "ix_k_norm_b", + shape=lambda d: (d.idx_dim,), + dtype=F32, + src="{p}.attention.indexer.k_norm.bias", + transform=_to_fp32, + layers=_indexer_layers, + ), + W( + "ix_ssmax_s", + shape=lambda d: (d.idx_heads,), + src="{p}.attention.indexer.ssmax_scaling.s", + layers=_indexer_layers, + ), + W( + "ix_ssmax_b", + shape=lambda d: (d.idx_heads,), + src="{p}.attention.indexer.ssmax_scaling.b", + layers=_indexer_layers, + ), + # --- the TP-sliced SwiGLU MLP: the dense layer's own, then each MoE layer's shared + # expert, one key --- + *_tp_mlp("feed_forward", lambda d: d.dense_inter, _dense_layers), + *_tp_mlp("shared_experts", lambda d: d.shared_inter, _moe_layers), + # --- routing --- + W( + "router", + shape=lambda d: (d.num_experts, d.hidden), + src="{p}.gate.weight", + layers=_moe_layers, + ), + # --- this rank's routed experts, NVFP4: `[w3 ; w1]` as stored, then each expert in the + # MoE runner's layout --- + W( + "fc1_w", + shape=lambda d: (d.local_experts, 2 * d.moe_inter, d.hidden // E2M1_PER_BYTE), + dtype=U8, + src=_expert_window( + ("w3.weight", lambda d: (slice(0, d.moe_inter),)), + ("w1.weight", lambda d: (slice(d.moe_inter, 2 * d.moe_inter),)), + ), + layers=_moe_layers, + layout=_fc1_codes, + ), + W( + "fc1_s", + shape=lambda d: (d.local_experts, 2 * d.moe_inter, d.hidden // NVFP4_BLOCK), + dtype=F8, + src=_expert_window( + ("w3.weight_scale", lambda d: (slice(0, d.moe_inter),)), + ("w1.weight_scale", lambda d: (slice(d.moe_inter, 2 * d.moe_inter),)), + ), + layers=_moe_layers, + layout=_fc1_scales, + ), + W( + "fc2_w", + shape=lambda d: (d.local_experts, d.hidden, d.moe_inter // E2M1_PER_BYTE), + dtype=U8, + src=_expert_window(("w2.weight", lambda d: ())), + layers=_moe_layers, + layout=_fc2_codes, + ), + W( + "fc2_s", + shape=lambda d: (d.local_experts, d.hidden, d.moe_inter // NVFP4_BLOCK), + dtype=F8, + src=_expert_window(("w2.weight_scale", lambda d: ())), + layers=_moe_layers, + layout=_fc2_scales, + ), + # The per-tensor scalars: the layer's one FC1 and one FC2 activation scale, and each + # local expert's global weight scales (`w3`'s are `w1`'s). + W( + "isc1", + shape=lambda d: (1,), + dtype=F32, + src=_window_scalar("w1.input_scale", "w3.input_scale"), + transform=_shared_scalar, + layers=_moe_layers, + ), + W( + "isc2", + shape=lambda d: (1,), + dtype=F32, + src=_window_scalar("w2.input_scale"), + transform=_shared_scalar, + layers=_moe_layers, + ), + W( + "ws2_13", + shape=lambda d: (d.local_experts,), + dtype=F32, + src=_expert_window((("w1.weight_scale_2", "w3.weight_scale_2"), lambda d: ())), + transform=_shared_scalar, + layers=_moe_layers, + ), + W( + "ws2_2", + shape=lambda d: (d.local_experts,), + dtype=F32, + src=_expert_window(("w2.weight_scale_2", lambda d: ())), + layers=_moe_layers, + ), + # --- not per-layer --- + W("final_norm", shape=lambda d: (d.hidden,), src="norm.weight", layers=None), + W("embed", shape=lambda d: (d.vocab, d.hidden), src="tok_embeddings.weight", layers=None), + # The LeRoPE per-frequency log-scales the rope tables are built from, fp32 as the + # reference widens them. + W( + "lerope_alpha", + shape=lambda d: (d.rope_dim // 2,), + dtype=F32, + src="lerope.alpha", + transform=_to_fp32, + layers=None, + ), + # BF16 vision encoder and projector, replicated on every rank. + W( + "vis_patch", + shape=lambda d: (d.v_hidden, d.v_channels * d.v_patch**2), + src="vision_encoder.patch_conv.weight", + transform=_flatten_patch_conv, + layers=None, + when=_has_vision, + ), + W( + "vis_ln_pre", + shape=lambda d: (d.v_hidden,), + src="vision_encoder.ln_pre.weight", + layers=None, + when=_has_vision, + ), + W( + "attn_norm", + shape=lambda d: (d.v_hidden,), + src="{p}.attention_norm.weight", + layers=_vision_blocks, + tag="v", + ), + W( + "qkv", + shape=lambda d: (3 * d.v_hidden, d.v_hidden), + src=_stack( + ("{p}.attention.wq.weight", lambda d: (slice(0, d.v_hidden),)), + ("{p}.attention.wk.weight", lambda d: (slice(d.v_hidden, 2 * d.v_hidden),)), + ("{p}.attention.wv.weight", lambda d: (slice(2 * d.v_hidden, 3 * d.v_hidden),)), + prefix="vision_encoder.transformer.layers", + ), + layers=_vision_blocks, + tag="v", + ), + W( + "wo", + shape=lambda d: (d.v_hidden, d.v_hidden), + src="{p}.attention.wo.weight", + layers=_vision_blocks, + tag="v", + ), + W( + "ffn_norm", + shape=lambda d: (d.v_hidden,), + src="{p}.ffn_norm.weight", + layers=_vision_blocks, + tag="v", + ), + W( + "w13", + shape=lambda d: (2 * d.v_inter, d.v_hidden), + src=_stack( + ("{p}.feed_forward.w1.weight", lambda d: (slice(0, d.v_inter),)), + ("{p}.feed_forward.w3.weight", lambda d: (slice(d.v_inter, 2 * d.v_inter),)), + prefix="vision_encoder.transformer.layers", + ), + layers=_vision_blocks, + tag="v", + ), + W( + "w2", + shape=lambda d: (d.v_hidden, d.v_inter), + src="{p}.feed_forward.w2.weight", + layers=_vision_blocks, + tag="v", + ), + W( + "vis_proj_norm", + shape=lambda d: (d.v_hidden,), + src="pre_mm_projector_norm.weight", + layers=None, + when=_has_vision, + ), + W( + "vis_merge", + shape=lambda d: (d.v_hidden, d.v_hidden * d.v_merge**2), + src="patch_merger.merging_layer.weight", + layers=None, + when=_has_vision, + ), + W( + "vis_w_in", + shape=lambda d: (d.hidden, d.v_hidden), + src="vision_language_adapter.w_in.weight", + layers=None, + when=_has_vision, + ), + W( + "vis_w_out", + shape=lambda d: (d.hidden, d.hidden), + src="vision_language_adapter.w_out.weight", + layers=None, + when=_has_vision, + ), + ) + + # BF16 draft weights use the same attention and dense-MLP partition as + # this target. The RNN and target-feature projection are replicated. + DRAFT_WEIGHTS: tuple[W, ...] = ( + W( + "draft_attn_norm", + lambda d: (d.hidden,), + _draft_source("attention_norm.weight"), + layers=_draft_layers, + ), + W( + "draft_wq", + lambda d: (d.heads * d.qk_dim, d.hidden), + _draft_source("attention.wq.weight"), + layers=_draft_layers, + shard=_rows( + lambda d: d.draft_tp_rank * d.heads * d.qk_dim, lambda d: d.heads * d.qk_dim + ), + ), + W( + "draft_wkv_a", + lambda d: (d.lat_dim, d.hidden), + _draft_source("attention.wkv_a_with_mqa.weight"), + layers=_draft_layers, + ), + W( + "draft_kv_a_norm", + lambda d: (d.kv_lora,), + _draft_source("attention.kv_a_norm.weight"), + layers=_draft_layers, + ), + W( + "draft_wkv_b", + lambda d: (d.heads * (d.nope + d.v_dim), d.kv_lora), + _draft_source("attention.wkv_b.weight"), + layers=_draft_layers, + shard=_rows( + lambda d: d.draft_tp_rank * d.heads * (d.nope + d.v_dim), + lambda d: d.heads * (d.nope + d.v_dim), + ), + ), + W( + "draft_wo", + lambda d: (d.hidden, d.heads * d.v_dim), + _draft_source("attention.wo.weight"), + layers=_draft_layers, + shard=_cols(lambda d: d.draft_tp_rank * d.heads * d.v_dim, lambda d: d.heads * d.v_dim), + ), + W( + "draft_ffn_norm", + lambda d: (d.hidden,), + _draft_source("ffn_norm.weight"), + layers=_draft_layers, + ), + W( + "draft_w1", + lambda d: (d.dense_inter, d.hidden), + _draft_source("feed_forward.w1.weight"), + layers=_draft_layers, + shard=_rows(lambda d: d.draft_tp_rank * d.dense_inter, lambda d: d.dense_inter), + ), + W( + "draft_w3", + lambda d: (d.dense_inter, d.hidden), + _draft_source("feed_forward.w3.weight"), + layers=_draft_layers, + shard=_rows(lambda d: d.draft_tp_rank * d.dense_inter, lambda d: d.dense_inter), + ), + W( + "draft_w2", + lambda d: (d.hidden, d.dense_inter), + _draft_source("feed_forward.w2.weight"), + layers=_draft_layers, + shard=_cols(lambda d: d.draft_tp_rank * d.dense_inter, lambda d: d.dense_inter), + ), + W( + "draft_final_norm", + lambda d: (d.hidden,), + "mtp_dspark.backbone.final_norm.weight", + layers=None, + ), + W( + "draft_target_norm", + lambda d: (d.hidden,), + "mtp_dspark.backbone.target_norm.weight", + layers=None, + ), + W( + "draft_target_projection", + lambda d: (d.hidden, d.hidden * d.draft_capture_layers), + "mtp_dspark.backbone.target_projection.weight", + layers=None, + ), + W( + "draft_mask", + lambda d: (d.draft_depth - 1, d.hidden), + "mtp_dspark.backbone.mask_embedding", + layers=None, + ), + W( + "draft_markov_embedding", + lambda d: (d.vocab, d.draft_rank), + "mtp_dspark.autoregressive_head.embedding.weight", + layers=None, + ), + W( + "draft_markov_output", + lambda d: (d.vocab, d.draft_rank), + "mtp_dspark.autoregressive_head.output.weight", + layers=None, + ), + W( + "draft_joint", + lambda d: (3 * d.draft_rank, d.hidden + 2 * d.draft_rank), + "mtp_dspark.autoregressive_head.joint_proj.weight", + layers=None, + ), + W( + "draft_joint_bias", + lambda d: (3 * d.draft_rank,), + "mtp_dspark.autoregressive_head.joint_proj.bias", + layers=None, + ), + ) + + def _entries(self, d): + return self.WEIGHTS + (self.DRAFT_WEIGHTS if d.draft_layers else ()) + + def declare(self, d: SimpleNamespace) -> nn.ParameterDict: + """Allocate every weight the table declares. + + Meta-init intercepts `torch.empty` here -- real CUDA storage arrives + when the engine materializes the registry. + """ + w = nn.ParameterDict() + for entry in self._entries(d): + dtype = entry.dtype or d.dtype + for key, _, _ in self._targets(entry, d): + assert key not in w, f"two rows declare {key}" + w[key] = nn.Parameter( + torch.empty(*entry.shape(d), dtype=dtype), requires_grad=False + ) + return w + + def _targets(self, entry: W, d: SimpleNamespace) -> list[tuple[str, int | None, str]]: + """`[(param key, layer index | None, checkpoint layer prefix), ...]`. + + One place decides which parameters an entry declares, so `declare` and + `manifest` cannot disagree about whether a weight exists. + """ + if entry.when is not None and not entry.when(d): + return [] + layers = entry.layer_indices(d) + if layers is None: + return [(entry.name, None, "")] + prefix = "vision_encoder.transformer.layers" if entry.tag == "v" else "layers" + return [(f"{entry.tag}{i}_{entry.name}", i, f"{prefix}.{i}") for i in layers] + + def manifest(self, core) -> dict: + """param key -> [(checkpoint key | key tuple, index into the param | None, transform, + shard)].""" + d = self.dims(core) + rows: dict = {} + for entry in self._entries(d): + shard = entry.shard(d) if entry.shard else None + for key, i, prefix in self._targets(entry, d): + assert key not in rows, f"two rows declare {key}" + rows[key] = [ + (ckpt, index, entry.transform, shard) + for ckpt, index in entry.sources(d, i, prefix) + ] + return rows + + def expected_unconsumed(self, core, weights) -> dict[str, str]: + """Declared unused keys: off-rank experts and disabled draft weights. Off-rank expert + scalars are still read to validate the layer's shared scales.""" + d = self.dims(core) + lo, hi = d.expert_offset, d.expert_offset + d.local_experts + keys = { + k: "DSpark draft disabled" + for k in weights.keys() + if not d.draft_layers and k.startswith("mtp_dspark.") + } + window = f"routed expert outside this rank's window [{lo}, {hi})" + never_read = f"{window}: never read" + checked = f"{window}: read only to check the layer's shared scales" + for i in _moe_layers(d): + for e in range(d.num_experts): + if lo <= e < hi: + continue + for proj in PROJECTIONS: + for s in ("weight", "weight_scale"): + key = _expert_key(i, e, f"{proj}.{s}") + if key in weights: + keys[key] = never_read + for s in EXPERT_SCALARS: + keys[_expert_key(i, e, f"{proj}.{s}")] = checked + return keys + + def load_checkpoint(self, model, checkpoint_dir) -> None: + """Fill every declared weight from a checkpoint directory, which is what the engine + loads: `load` over the directory's keys, every source read from its shard file.""" + directory, weight_map = _checkpoint_shards(checkpoint_dir) + self.load(model, weight_map, shards=(directory, weight_map)) + + def load(self, model, weights, shards=None) -> None: + """Fill every declared weight from the checkpoint: the shard column, multi-source rows, + the vocab-parallel `lm_head`, and coverage asserted in both directions. + + First every routed expert's per-tensor scalars are read and checked, the other ranks' + included, so a checkpoint that breaks what the forward relies on fails before any + weight moves. When the checkpoint's own shard files can be found, every source is read + from them one file at a time, this rank's slices only, and each file's cached pages are + dropped once its sources are copied. Mapping every shard file on every rank and reading + through the mappings instead would fault a node's four ranks' expert + windows -- 539 GiB, the whole expert stack, on one node of four; 270 GiB, half of it, + on each node of two -- into that node's host memory. Last, each row's `layout` puts the + expert stacks into the MoE runner's layout, so a second `load` starts again from the + stored bytes. + """ + core = model.model + manifest = self.manifest(core) + consumed: set = set() + + def fill(param, ckpt, entry, index, transform, shard) -> None: + dst = param.data if index is None else param.data[index] + if isinstance(ckpt, tuple): + src = transform( + tuple(_materialize(e, shard).to(dst.device, non_blocking=True) for e in entry), + core, + ) + else: + src = _materialize(entry, shard) + if transform is not None: + # A column shard is a strided view: make it dense before a transform reads it. + src = transform(src.to(dst.device, non_blocking=True).contiguous(), core) + assert tuple(dst.shape) == tuple(src.shape), (ckpt, tuple(dst.shape), tuple(src.shape)) + assert src.dtype == dst.dtype, (ckpt, src.dtype, dst.dtype) + dst.copy_(src, non_blocking=True) + consumed.update(_keys(ckpt)) + + assert set(manifest.keys()) == set(core.w.keys()), ( + "manifest/parameter drift", + sorted(set(manifest.keys()) ^ set(core.w.keys()))[:12], + ) + # The shell builds lm_head vocab-parallel under TP: this rank's block of rows. + d = self.dims(core) + rows = d.vocab // d.tp_size + plan = [ + (core.w[param_key], ckpt, index, transform, shard) + for param_key, sources in manifest.items() + for ckpt, index, transform, shard in sources + ] + plan.append( + ( + model.lm_head.weight, + "output.weight", + None, + None, + _rows(lambda _: d.tp_rank * rows, lambda _: rows)(d), + ) + ) + scalar_keys = _expert_scalar_keys(d) + for key in (*(k for row in plan for k in _keys(row[1])), *scalar_keys): + assert key in weights, f"checkpoint key missing: {key}" + if shards is None: + shards = _shard_files(core, weights) + scalars = _read_scalars(scalar_keys, weights, shards) + _check_expert_scalars(scalars, d) + + streamed = [] + for param, ckpt, index, transform, shard in plan: + if all(k in scalars for k in _keys(ckpt)): + entry = tuple(scalars[k] for k in _keys(ckpt)) + entry = entry if isinstance(ckpt, tuple) else entry[0] + fill(param, ckpt, entry, index, transform, shard) + else: + assert isinstance(ckpt, str), f"a key tuple of other than expert scalars: {ckpt}" + streamed.append((param, ckpt, index, transform, shard)) + if shards is None: + for param, ckpt, index, transform, shard in streamed: + fill(param, ckpt, weights[ckpt], index, transform, shard) + else: + directory, weight_map = shards + by_file: dict[str, list] = collections.defaultdict(list) + for row in streamed: + by_file[weight_map[row[1]]].append(row) + on_device = model.lm_head.weight.device.type == "cuda" + for name in sorted(by_file): + with safe_open(str(directory / name), framework="pt", device="cpu") as handle: + for param, ckpt, index, transform, shard in by_file[name]: + fill(param, ckpt, handle.get_slice(ckpt), index, transform, shard) + if on_device: + torch.cuda.synchronize() + _drop_page_cache(directory / name) + # Every source has landed: rearrange each expert in place, one at a time. + for entry in self._entries(d): + if entry.layout is None: + continue + for key, _, _ in self._targets(entry, d): + param = core.w[key].data + for slot in range(param.shape[0]): + param[slot].copy_(entry.layout(param[slot])) + shell = {n for n, _ in model.named_parameters() if not n.startswith("model.w.")} + assert shell == {"lm_head.weight"}, f"unfed shell parameters: {sorted(shell)}" + + if model.lm_head.weight.device.type == "cuda": + torch.cuda.synchronize() + leftover = set(weights.keys()) - consumed + expected = self.expected_unconsumed(core, weights) + assert leftover == set(expected), ( + "checkpoint coverage: unexpected " + f"{sorted(leftover - set(expected))[:8]}, missed {sorted(set(expected) - leftover)[:8]}" + ) + not_loaded = collections.Counter(expected.values()) + logger.info( + f"modeling_v2 weights: {len(consumed)} checkpoint tensors loaded " + f"({'streamed per shard file' if shards else 'from the given dict'}), " + f"{len(scalars)} expert scalars checked; not loaded: " + + "; ".join(f"{n} x {reason}" for reason, n in sorted(not_loaded.items())) + ) + + def dims(self, core) -> SimpleNamespace: + cfg = core.model_config.pretrained_config + vis = getattr(cfg, "vision_config", None) + mapping = core.model_config.mapping + spec = getattr(core.model_config, "spec_config", None) + draft = getattr(spec, "_draft_model_config", None) or {} + tp = mapping.tp_size + local_experts = cfg.n_routed_experts // mapping.moe_ep_size + return SimpleNamespace( + draft_layers=int(draft.get("n_layers", 0)), + draft_rank=int((draft.get("dspark") or {}).get("markov_rank", 0)), + draft_depth=int(draft.get("depth", 0)), + draft_capture_layers=len(spec.target_layer_ids) if draft else 0, + draft_tp_rank=mapping.tp_rank, + num_layers=cfg.num_hidden_layers, + hidden=cfg.hidden_size, + heads=cfg.num_attention_heads // tp, + nope=cfg.qk_nope_head_dim, + rope_dim=cfg.qk_rope_head_dim, + qk_dim=cfg.qk_nope_head_dim + cfg.qk_rope_head_dim, + v_dim=cfg.v_head_dim, + kv_lora=cfg.kv_lora_rank, + lat_dim=cfg.kv_lora_rank + cfg.qk_rope_head_dim, + nope_layer=tuple(bool(f) for f in cfg.nope_layer_flags), + kv_source=tuple(int(s) for s in cfg.kv_source_layers), + full_indexer=tuple(t == "full" for t in cfg.index_sharing), + idx_heads=cfg.index_n_heads, + idx_dim=cfg.index_head_dim, + dense_layers=cfg.first_k_dense_replace, + num_experts=cfg.n_routed_experts, + moe_inter=cfg.moe_intermediate_size, + local_experts=local_experts, + expert_offset=local_experts * mapping.moe_ep_rank, + shared_inter=cfg.moe_intermediate_size * cfg.n_shared_experts // tp, + dense_inter=cfg.intermediate_size // tp, + vocab=cfg.vocab_size, + tp_rank=mapping.tp_rank, + tp_size=tp, + dtype=core.model_config.torch_dtype, + # The vision tower (replicated; absent without a vision encoder). + has_vision=vis is not None, + v_layers=vis.num_hidden_layers if vis is not None else 0, + v_hidden=vis.hidden_size if vis is not None else 0, + v_heads=vis.num_attention_heads if vis is not None else 0, + v_head_dim=vis.hidden_size // vis.num_attention_heads if vis is not None else 0, + v_inter=vis.intermediate_size if vis is not None else 0, + v_channels=vis.num_channels if vis is not None else 0, + v_patch=vis.patch_size if vis is not None else 0, + v_merge=vis.spatial_merge_size if vis is not None else 0, + v_theta=float(vis.rope_theta) if vis is not None else 0.0, + image_token_id=vis.image_token_id if vis is not None else None, + ) + + +MODEL_WEIGHTS = MistralLarge4TepWeights() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/__init__.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/__init__.py new file mode 100644 index 000000000000..bdfa7c377dfb --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Mistral-Large-4-1T-A52B-NVFP4 / sm_103 / tp, the language model only.""" diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/modeling.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/modeling.py new file mode 100644 index 000000000000..ac4cbfe0b3bb --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/modeling.py @@ -0,0 +1,2508 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""ModelingV2 target: mistral-large-4-nvfp4 / sm_103 / tp, native text and image inference. + +72 layers, hidden 6144, 64 query heads (16 per rank at tp4, 8 at tp8). Attention is MLA without +a q-LoRA (`wq` straight to 64 x 128), a 512-wide latent plus a 64-wide rope key, and DeepSeek +Sparse Attention: an indexer on the 44 `index_sharing == 'full'` layers selects 2048 positions +per query row and each 'shared' layer reuses the selection of the most recent full layer. Every +fourth layer (`i % 4 == 3`) takes no rope, for MLA and for its indexer; the NoPE layers 39, +43, ..., 71 own no latent and attend over layer 35's. Queries are scaled by SSMax, +`s * log(pos + 1) + b` per head, on MLA and on the indexer. Layer 0 is a dense SwiGLU MLP, +layers 1-71 MoE (128 experts, softmax top-4 renormalised, one shared expert). The final logits +are soft-capped at `softmax_tanh_gating`. + +The checkpoint is ModelOpt `MIXED_PRECISION`. The routed experts are NVFP4 W4A4: e2m1 weights +with one e4m3 scale per 16 inputs and a global scale per expert, and activations quantized to +NVFP4 under one static global scale per layer and projection. `wq`, `wo`, the indexer's `wq_b` +and the dense and shared MLPs are FP8 e4m3 with 128x128 power-of-two block scales and dynamic +1x128 activation scales. The rest is bf16. The latent KV pool is fp8_ds_mla (656 B per token +and owner layer: 512 e4m3 codes with one power-of-two fp32 scale per 128, and the 64-wide rope +key in bf16), owned by `MistralLarge4CacheManager`; the indexer key pool is fp8 with one scale +per key. + +`tp` shards the attention heads, the dense and shared MLPs and `lm_head` over every rank -- +four ways on one node of four, or eight ways over two nodes of four (tp8), every width read off +the mapping -- and the routed experts by width: every rank holds all 128 experts, each cut to +its `moe_intermediate_size / moe_tp_size`-wide slice of the intermediate (1536 at tp4, 768 at +tp8), so one runner call over every token returns this rank's partial sum over its slice. The +latent projection, the indexer and the router are replicated. Every layer closes twice on the +all-reduce over every rank fused with the residual add and the next RMSNorm; the second one +sums the ranks' routed and shared partials together. Under chunked prefill, a step of +`q_split_threshold` context rows or more has the indexer's context queries split over every +rank by the engine: each rank selects for its share, and every selecting layer all-gathers the +shares before any layer attends. + +The absorbed form throughout: `fused_q = [q_nope @ W_UK | rope(q_pe)]` (576 wide) against the +pooled `[c_kv | k_pe]` latent rows, then `@ W_UV`. Context rows attend over the selection (the +engine's causal index on a step it skipped the indexer) with FlashMLA's bf16 sparse prefill +over the requests' latent rows dequantized out of the pool, so they read the FP8-rounded latent +as the reference does; generation rows run sparse fp8 decode over the pool itself, at this +rank's 16 (tp4) or 8 (tp8) heads at every max_seq_len: cute_dsl_ds_mla_sparse_decode, +FlashMLA's sparse decode arithmetic without the head padding. The prefill kernel, and the +sparse decode a context row past the held workspace runs (FlashMLA's), take 64 heads: this +rank's go first in a 64-head query whose other heads are zero; heads do not interact, so the +zero heads change nothing the live ones compute. `_GEN_ATTN_FLASH_MLA` (for A/B verification) +runs the generation rows on FlashMLA's 64-head decode too, the same padding. + +No path is selected by the serving configuration: each step chooses from its own sizes, and +every buffer is sized from the engine's quantities (the attention metadata's max_num_tokens, +max_num_requests, max_seq_len and runtime_features). The context rows take one of two paths, +chosen per step by the step's context KV (every context request's cached and new rows): while +it fits the held workspace, the sparse prefill over those rows dequantized into it; past it, +each context row as a one-row sequence of the sparse fp8 decode over the pool itself, as +generation rows always run. The held workspace, allocated on the first context step, has room +for the most context KV one step can attend, twice over, up to 2**19 rows +(`_ctx_workspace_rows`): a step whose context KV fits twice keeps layer 35's rows beside its +own for the NoPE layers attending over that latent, a longer one gathers them again for each +of those layers. An indexer chunk the engine packs past `max_seq_len` keys is scored request +by request, and the indexer's logits are tiled under a budget from the engine's max_num_tokens +and max_seq_len (`_prefill_logits_budget`), each tile scored with a split-KV schedule built once +per step (every SM busy whatever the tile's row count) and selected with the self-sampling GVR +top-k (`_select_ctx`).""" + +from __future__ import annotations + +import itertools +import math +import os +import time +from dataclasses import replace +from types import SimpleNamespace +from typing import NamedTuple, Optional, cast + +import torch +from torch import nn +from transformers import PretrainedConfig + +from tensorrt_llm._torch.attention.backends.interface import AttentionMetadata +from tensorrt_llm._torch.attention.backends.sparse.params import ( + get_indexer_mqa_logits_workspace_bytes, +) +from tensorrt_llm._torch.attention.backends.trtllm import TrtllmAttentionMetadata +from tensorrt_llm._torch.model_config import ModelConfig +from tensorrt_llm._torch.modeling_v2._router_index import step_contract_enabled +from tensorrt_llm._torch.modeling_v2.catalog.activation.flashinfer_silu_and_mul import ( + flashinfer_silu_and_mul, +) +from tensorrt_llm._torch.modeling_v2.catalog.attention.thop_attention import thop_attention +from tensorrt_llm._torch.modeling_v2.catalog.comm.allgather import allgather +from tensorrt_llm._torch.modeling_v2.catalog.gemm.bmm_out import bmm_out +from tensorrt_llm._torch.modeling_v2.catalog.gemm.cublas_mm import cublas_mm +from tensorrt_llm._torch.modeling_v2.catalog.moe.fp4_block_scale_moe_runner import ( + fp4_block_scale_moe_runner, +) +from tensorrt_llm._torch.modeling_v2.catalog.norm.flashinfer_fused_add_rmsnorm import ( + flashinfer_fused_add_rmsnorm, +) +from tensorrt_llm._torch.modeling_v2.catalog.norm.flashinfer_rmsnorm import flashinfer_rmsnorm +from tensorrt_llm._torch.modeling_v2.catalog.quantization.fp4_quantize import fp4_quantize +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.activation.gelu_erf import gelu_erf +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.activation.tanh_softcap import ( + tanh_softcap, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.convert_req_index_to_global import ( # noqa: E501 + convert_req_index_to_global, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_ds_mla_sparse_decode import ( # noqa: E501 + build as build_ds_mla_decode, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_ds_mla_sparse_decode import ( # noqa: E501 + cute_dsl_ds_mla_sparse_decode, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_fp8_paged_mqa_logits import ( # noqa: E501 + cute_dsl_fp8_paged_mqa_logits, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_gvr_topk_prefill import ( # noqa: E501 + cute_dsl_gvr_topk_prefill, + gvr_warmup, + gvr_workspace, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_indexer_k_cache_update import ( # noqa: E501 + build as build_ix_k_cache, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_indexer_k_cache_update import ( # noqa: E501 + cute_dsl_indexer_k_cache_update, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_indexer_q_side_fused import ( # noqa: E501 + build as build_ix_q_side, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_indexer_q_side_fused import ( # noqa: E501 + cute_dsl_indexer_q_side_fused, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_radix_topk_decode import ( # noqa: E501 + build as build_ix_topk_decode, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_radix_topk_decode import ( # noqa: E501 + cute_dsl_radix_topk_decode, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ds_mla_dequant_gather import ( + ds_mla_dequant_gather, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ds_mla_rope_append import ( + ds_mla_rope_append, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.flash_mla_sparse_decode import ( + flash_mla_sparse_decode, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.flash_mla_sparse_prefill import ( + flash_mla_sparse_prefill, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.fp8_mqa_logits import ( + fp8_mqa_logits, + mqa_logits_schedule, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.indexer_k_cache_gather_op import ( # noqa: E501 + indexer_k_cache_gather_op, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.indexer_topk_decode import ( + indexer_topk_decode, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ssmax_indexer_q_side_fused import ( # noqa: E501 + ssmax_indexer_q_side_fused, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ssmax_scaling import ( + ssmax_scaling, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ssmax_scaling_fused_q import ( + ssmax_scaling_fused_q, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.comm.cute_dsl_allreduce_add_rmsnorm import ( # noqa: E501 + build_workspace as build_lamport_workspace, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.comm.cute_dsl_allreduce_add_rmsnorm import ( # noqa: E501 + cute_dsl_allreduce_add_rmsnorm, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.gemm.fp8_prequantized_swap_ab_gemm import ( + fp8_prequantized_swap_ab_gemm, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.moe.renorm_moe_routing_op import ( + renorm_moe_routing_op, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.norm.layer_norm import layer_norm +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.quantization.fp8_quantize_1x128_packed_ue8m0 import ( # noqa: E501 + fp8_quantize_1x128_packed_ue8m0, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.vision.pixtral_rope_2d import ( + pixtral_rope_2d, +) +from tensorrt_llm._torch.models.modeling_multimodal_utils import ( + find_input_mm_embeds, + get_multimodal_embeddings, +) +from tensorrt_llm._torch.models.modeling_utils import DecoderModel, DecoderModelForCausalLM +from tensorrt_llm._torch.pyexecutor.guided_decoder import CapturableGuidedDecoder +from tensorrt_llm._torch.speculative import get_spec_worker +from tensorrt_llm._utils import mpi_disabled +from tensorrt_llm.flash_mla import get_mla_metadata +from tensorrt_llm.inputs.multimodal import MultimodalParams +from tensorrt_llm.logger import logger +from tensorrt_llm.models.modeling_utils import QuantConfig + +from . import weights as _weights + +#: eps 1e-5, the reference's value; params.json's `vision_encoder` states none. +_VISION_EPS = 1e-5 + +#: The thop attention arguments the vision tower's call holds fixed: the no-KV-cache +#: mode (an encoder: the three cache-addressing arguments None), one context sequence +#: per image, the fused `[q | k | v]` rows already rotated (no rope in the op), +#: bidirectional within each sequence (mask_type 0), softmax scale head_dim^-0.5. +_VISION_ATTN = dict( + k=None, + v=None, + output_sf=None, + max_context_q_len_override=None, + kv_cache_block_offsets=None, + host_kv_cache_pool_pointers=None, + host_kv_cache_pool_mapping=None, + cache_indirection=None, + kv_scale_orig_quant=None, + kv_scale_quant_orig=None, + out_scale=None, + rotary_inv_freq=None, + rotary_cos_sin=None, + latent_cache=None, + q_pe=None, + block_ids_per_seq=None, + attention_sinks=None, + is_fused_qkv=True, + update_kv_cache=True, + predicted_tokens_per_seq=1, + local_layer_idx=0, + tokens_per_block=None, + beam_width=1, + mask_type=0, + quant_mode=0, + q_scaling=1.0, + position_embedding_type=0, + rope_dim=0, + rope_base=10000.0, + rope_scale_type=0, + rope_scale=1.0, + rope_short_m_scale=1.0, + rope_long_m_scale=1.0, + rope_max_positions=1024, + rope_original_max_positions=1024, + use_paged_context_fmha=False, + attention_input_type=0, + is_mla_enable=False, + chunked_prefill_buffer_batch_size=1, + q_lora_rank=None, + kv_lora_rank=None, + qk_nope_head_dim=None, + qk_rope_head_dim=None, + v_head_dim=None, + rope_append=None, + mrope_rotary_cos_sin=None, + mrope_position_deltas=None, + helix_position_offsets=None, + helix_is_inactive_rank=None, + attention_chunk_size=None, + softmax_stats_tensor=None, + is_spec_decoding_enabled=False, + use_spec_decoding=False, + is_spec_dec_tree=False, + spec_decoding_generation_lengths=None, + spec_decoding_position_offsets=None, + spec_decoding_packed_mask=None, + spec_decoding_bl_tree_mask_offset=None, + spec_decoding_bl_tree_mask=None, + spec_bl_tree_first_sparse_mask_offset_kv=None, + sparse_kv_indices=None, + sparse_kv_offsets=None, + sparse_attn_indices=None, + sparse_attn_offsets=None, + sparse_attn_indices_block_size=0, +) + +# The latent pool layout this target addresses (`MistralLarge4CacheManager.latent_layout`). +_LATENT_LAYOUT = "fp8_ds_mla" +# The query heads FlashMLA's sm100 sparse kernels run; this rank's heads go first. +_FLASH_MLA_HEADS = 64 +# Generation rows' sparse MLA. False: cute_dsl_ds_mla_sparse_decode at this rank's heads, the +# shipped path. True: flash_mla_sparse_decode over the 64-head padded query, the kernel this +# target ran before (bit-identical where the two split a row alike; see the entry's note) -- +# a switch for A/B verification, not a configuration. It sets the core's +# `gen_attn_flash_mla`, which every generation-row site reads; the native entry's cells cover +# the 32K and the 1M configs (key lengths to 1048576, the 1M pools' slots past 2**31 bytes). +_GEN_ATTN_FLASH_MLA = False +# The pre-routed MoE call: the runner's routing method is inert once ids and weights are +# given; 1 (renormalize) is the value its cells drive. +_MOE_ROUTING_METHOD = 1 +# The routed experts' SwiGLU, the runner's `act_type` 0. +_MOE_ACT_SWIGLU = 0 +# NVFP4: one e4m3 scale per 16 activations, laid out row-major (linear), the order the MoE +# runner reads; the 128x4-swizzled order is another buffer of the same length at 128-row +# multiples, and silently wrong there. +_NVFP4_SF_VEC = 16 +_NVFP4_SF_LINEAR = False +# fp8_mqa_logits allocates a `[q_rows, kv_rows]` fp32 score matrix per call; the context +# queries are tiled to keep it under the engine's budget (`_prefill_logits_budget`). Context +# steps are not captured. +# capacity: fixed: the budget's ceiling, one 1 GiB fp32 tile (256 rows at 1048576 keys). +# Mainline's default reachable tile (2**31 logits) would take 8 GiB from the KV pool. +_PREFILL_LOGITS_CEILING = 1 << 28 +# The context path's crossover, chosen once per step on host ints, and the held context +# workspace's ceiling (`_ctx_workspace_rows`). A step whose context KV (every context request's +# cached and new rows) fits the held workspace has it gathered, dequantized, into it (bf16 +# [rows, 576]) for FlashMLA's sparse prefill; a longer one attends its context rows straight off +# the pool with the sparse decode kernel, whose cost does not grow with the prefix as the +# gather's does. +# capacity: fixed: 2**19 rows, 576 MiB, the gather-vs-decode crossover. +_CTX_WORKSPACE_ROWS = 1 << 19 +# The most context rows one sparse-decode call takes on that path: FlashMLA's decode scheduler +# exits the process, raising nothing, past 11622 one-row sequences on sm_103. +# capacity: fixed: the kernel's scheduler, not an engine limit. +_CTX_DECODE_ROWS = 8192 +# The decode indexer's top-k: True runs cute_dsl_radix_topk_decode, one launch that selects the +# trtllm indexer_topk_decode op's set (up to exact ties), False that op itself. The CuTe DSL +# selector serves scorer rows from 4096 columns (where the op splits each row) to 65536 (one +# CTA scans a whole row), the widths its cells cover; the scorer's column count is the engine's +# max_seq_len, so an engine outside that range keeps the op (`_derive_first_step`). +_TOPK_DECODE_CUTE_DSL = True +_TOPK_DECODE_COLUMNS = (4096, 65536) +# The attention block's glue as separate ops instead of the fused entries, for old-versus-new +# verification (both compute the same bits; the fused entries' tests hold each pair equal): +# ssmax_scaling plus the rope tail's copy or zero fill for ssmax_scaling_fused_q; +# ssmax_scaling, the head weights' fp32 copy and cute_dsl_indexer_q_side_fused for +# ssmax_indexer_q_side_fused; a fresh RMSNorm and torch.cat for the in-place latent; the copy +# out of FlashMLA's 64-head output for the in-place read. Set +# TRTLLM_MODELING_V2_UNFUSED_GLUE=1 in every rank's environment at launch; off by default. +# It sets the core's `unfused_glue`, which every site reads. +_UNFUSED_GLUE = os.environ.get("TRTLLM_MODELING_V2_UNFUSED_GLUE", "0") == "1" + + +def _prefill_logits_budget(max_num_tokens: int, max_seq_len: int) -> int: + """The most fp32 logits one indexer prefill tile holds, from the engine's max_num_tokens and + max_seq_len: mainline's reachable MQA-logits tile (`get_indexer_mqa_logits_workspace_bytes`, + 4 bytes a logit; TLLM_INDEXER_MQA_LOGITS_ELEM_BUDGET lowers it), within + `_PREFILL_LOGITS_CEILING`, and within max_seq_len / 64 query rows at max_seq_len keys. + + The last bound grows the tile with the engine's longest request, which it scores in at + least 64 tiles: 2**24 logits (512 rows of 32768 keys) at max_seq_len 32768, the ceiling + (256 rows of 1048576 keys) from max_seq_len 131072 on. Those are the budgets the 32768 and + 1048576 serving configs were certified at, so their tiles, and the fp8_mqa_logits and + prefill top-k cells that cover them, stay as they were.""" + reachable = get_indexer_mqa_logits_workspace_bytes(max_num_tokens, max_seq_len) // 4 + return min(reachable, _PREFILL_LOGITS_CEILING, max_seq_len * (max_seq_len >> 6)) + + +def _most_ctx_kv(md) -> int: + """The most context KV (cached and new rows over a step's context requests) one step can + attend, from the attention metadata alone. `runtime_features` is built once by the executor + for the service (chunked prefill from enable_chunked_prefill, cache reuse from the + KvCacheConfig's enable_block_reuse). With neither, every context row is new: + max_num_tokens. Otherwise mainline's never-stall worst case for a workspace chunked prefill + does not bound (`get_mla_context_workspace_kv_len_cap` without an + fp8_context_mla_kv_len_cap override; the gather reads every attended row, cached ones + included): min(max_num_requests, max_num_tokens) * max_seq_len. The target does not read + fp8_context_mla_kv_len_cap: the KvCacheConfig is not reachable from the model, and for these + targets the engine installs no admission cap that would bound a step to it.""" + features = md.runtime_features + if not (features.chunked_prefill or features.cache_reuse): + return md.max_num_tokens + return min(md.max_num_requests, md.max_num_tokens) * md.max_seq_len + + +def _ctx_workspace_rows(md) -> int: + """The held context workspace's rows: the most context KV one step can attend + (`_most_ctx_kv`) twice over, a step's rows and a shared owner's (layer 35's) kept beside + them, up to `_CTX_WORKSPACE_ROWS`. At max_num_tokens 32768 with neither chunked prefill nor + block reuse that is 65536 rows, the two `[32768, 576]` gathers a step there makes; with either, + at 16 requests of max_seq_len 32768 or more, 2**19.""" + return min(2 * _most_ctx_kv(md), _CTX_WORKSPACE_ROWS) + + +def _step_common(md: TrtllmAttentionMetadata, indexer_layers: list[int]) -> dict: + """The batch state both targets read: KV lengths and pool addressing for the MLA ops, the + indexer's key pools and slot mappings, the top-k conversion tables and the decode scorer's + schedule. The indexer pool views are taken per step: the engine replaces its KV cache + manager during boot, and a view of the old one writes into a dead allocation.""" + mgr = md.kv_cache_manager + latent_pool = mgr.get_latent_pool() + return dict( + num_seqs=md.num_seqs, + num_tokens=md.num_tokens, + kv_lens_cuda=md.kv_lens_cuda_runtime, + kv_lens_host=md.kv_lens_runtime, + prompt_lens_host=md.prompt_lens_cpu_runtime, + kv_cache_block_offsets=md.kv_cache_block_offsets, + host_kv_cache_pool_pointers=md.host_kv_cache_pool_pointers, + host_kv_cache_pool_mapping=md.host_kv_cache_pool_mapping, + tokens_per_block=md.tokens_per_block, + max_seq_len=md.max_seq_len, + # [blocks, owners, tokens_per_block, 656] bytes, and the same bytes as FlashMLA's + # pages: page `block * owners + row`, slot `page * tokens_per_block + offset`. + latent_pool=latent_pool, + latent_kv=latent_pool.view(-1, latent_pool.shape[2], 1, latent_pool.shape[3]), + ix_pools={i: mgr.get_indexer_k_cache_buffers(i) for i in indexer_layers}, + block_table=md.block_table, + req_idx_per_token=md.req_idx_per_token, + slot_mapping_fp8=md.slot_mapping_fp8, + slot_mapping_scale=md.slot_mapping_scale, + skip_gen=bool(md.skip_indexer_for_gen_reqs), + # Allocated only while the engine's short-sequence indexer skip is on; the two skip + # flags are False whenever it is off. + topk_indices_buffer=md.topk_indices_buffer if md.enable_indexer_skip else None, + indexer_k_cache_block_offsets=md.indexer_k_cache_block_offsets, + scheduler_metadata_buffer=md.scheduler_metadata_buffer, + radix_aux_indices=md.radix_aux_indices, + radix_aux_logits=md.radix_aux_logits, + ) + + +def _ctx_kv_offsets(md: TrtllmAttentionMetadata, num_contexts: int) -> dict: + """The context rope/append op's per-request offsets: int64 prefix sums of the cached and + of the total (cached + new) lengths over the context requests, and the longest new run. + + The engine keeps them only under `enable_context_mla_with_cached_kv` (KV reuse or chunked + prefill). Without either, no context request has a cached prefix, so the cached sums are + zero and the totals are the new lengths: the int32 prefix sum of `seq_lens_cuda` the + metadata always builds, widened to the op's int64. + """ + if md.enable_context_mla_with_cached_kv: + return dict( + cu_ctx_cached=md.ctx_cached_token_indptr[: num_contexts + 1], + cu_ctx_kv=md.ctx_kv_indptr[: num_contexts + 1], + max_ctx_new=int(md.max_ctx_seq_len), + ctx_cached_tokens=int(md.num_ctx_cached_tokens), + ) + cu = md.mla_prepare_ctx_cu_seqlens() + return dict( + cu_ctx_cached=torch.zeros(num_contexts + 1, dtype=torch.int64, device=cu.device), + cu_ctx_kv=cu.to(torch.int64), + max_ctx_new=int(md.seq_lens[:num_contexts].max()), + ctx_cached_tokens=0, + ) + + +def _ctx_host_offsets(md: TrtllmAttentionMetadata, num_contexts: int) -> dict: + """The host lists over the context requests: the prefix sums of the new and of the total + (cached + new) lengths, which place each context request's rows in the step and its keys in + the indexer's full-KV gather mappings (`_indexer_blocks`). + + The engine's host copies under `enable_context_mla_with_cached_kv`; without it no request + has a cached prefix, so both are the prefix sum of the host `seq_lens`. Every source is a + host tensor the metadata fills on the host, so taking them each context step syncs nothing. + """ + if md.enable_context_mla_with_cached_kv: + return dict( + host_cu_ctx_new=md.host_ctx_uncached_token_indptr[: num_contexts + 1].tolist(), + host_cu_ctx_kv=md.host_ctx_kv_indptr[: num_contexts + 1].tolist(), + ) + host_cu = [0, *itertools.accumulate(md.seq_lens[:num_contexts].tolist())] + return dict(host_cu_ctx_new=host_cu, host_cu_ctx_kv=host_cu) + + +def _indexer_blocks(chunks, step: dict) -> list: + """The context indexer's scoring blocks for one step, in the engine's chunk order: rows + `[token_start, token_end)` of the context rows scored against keys + `[k_token_start, k_token_end)` of the full-KV gather mappings, each row's window + (`cu_seqlen_ks` / `cu_seqlen_ke`) counted from the block's first key. + + An engine prefill chunk is one block, as planned, while it holds at most `max_seq_len` keys, + the most one request holds. The engine packs consecutive context requests into one chunk + while (summed new tokens) x (summed KV) stays within its packed-logits budget (2^28), so + short last chunks of long requests pack past max_seq_len: a 100-token tail of a + 1048559-token request beside a 150-token prompt is one chunk of 1048709 keys, wider than + the key scorer's and the prefill top-k's cells. Such a chunk is cut at its request + boundaries (`_ctx_host_offsets`), each request's rows against its own keys with their + windows re-based to its first key: a row's window lies inside its own request's keys, so + each request selects as it would alone and no block holds more than max_seq_len keys. + """ + tc = step["num_ctx_tokens"] + if chunks is None: + # One chunk over every context row: the engine plans chunks only past its chunk + # size, or under KV reuse or chunked prefill. + chunks = [ + SimpleNamespace( + token_start=0, + token_end=tc, + k_token_start=0, + k_token_end=step["num_ctx_kv_tokens"], + cu_seqlen_ks=step["cu_seqlen_ks"][:tc], + cu_seqlen_ke=step["cu_seqlen_ke"][:tc], + ) + ] + max_keys = step["max_seq_len"] + cu_new, cu_kv = step["host_cu_ctx_new"], step["host_cu_ctx_kv"] + blocks = [] + for chunk in chunks: + if chunk.k_token_end - chunk.k_token_start <= max_keys: + blocks.append(chunk) + continue + for r in range(len(cu_new) - 1): + lo, hi = max(chunk.token_start, cu_new[r]), min(chunk.token_end, cu_new[r + 1]) + if lo >= hi: + continue + k_lo, k_hi = max(chunk.k_token_start, cu_kv[r]), min(chunk.k_token_end, cu_kv[r + 1]) + assert k_hi - k_lo <= max_keys, ( + f"context request {r} holds keys [{k_lo}, {k_hi}) of an indexer chunk, past " + f"max_seq_len {max_keys}" + ) + shift = k_lo - chunk.k_token_start + rows = slice(lo - chunk.token_start, hi - chunk.token_start) + blocks.append( + SimpleNamespace( + token_start=lo, + token_end=hi, + k_token_start=k_lo, + k_token_end=k_hi, + cu_seqlen_ks=torch.sub(chunk.cu_seqlen_ks[rows], shift), + cu_seqlen_ke=torch.sub(chunk.cu_seqlen_ke[rows], shift), + ) + ) + return blocks + + +def _yarn_lerope_inv_freq(cfg, alpha: torch.Tensor) -> torch.Tensor: + """The rotary inverse frequencies, fp32, as the reference implementation builds + them: YaRN's blend of the extrapolated and the interpolated frequency, both scaled by + LeRoPE's `exp(alpha)`, over a ramp taken on the frequencies without LeRoPE.""" + rope = cfg.rope_parameters + dim = cfg.qk_rope_head_dim + theta = float(rope["rope_theta"]) + factor = float(rope["factor"]) + original = float(rope["original_max_position_embeddings"]) + pos_freqs = theta ** (torch.arange(0, dim, 2, dtype=torch.float32) / dim) + lerope = torch.exp(alpha.detach().to(device="cpu", dtype=torch.float32)) + extrapolated = 1.0 / pos_freqs * lerope + interpolated = 1.0 / (factor * pos_freqs) * lerope + + def correction_dim(rotations: float) -> float: + return dim * math.log(original / (rotations * 2 * math.pi)) / (2 * math.log(theta)) + + low = max(math.floor(correction_dim(float(rope.get("beta_fast", 32)))), 0) + high = min(math.ceil(correction_dim(float(rope.get("beta_slow", 1)))), dim - 1) + if low == high: + high += 0.001 + ramp = ((torch.arange(dim // 2, dtype=torch.float32) - low) / (high - low)).clamp(0, 1) + extrapolation = 1 - ramp + return interpolated * (1 - extrapolation) + extrapolated * extrapolation + + +def _rope_tables(cfg, alpha: torch.Tensor, positions: int, dtype, device) -> SimpleNamespace: + """The tables the rope consumers read, built from one fp32 angle grid and rounded to the + activation dtype first, as the reference's cos/sin cache is held. + + `indexer` is `[rows, 64]`, `[cos(32) | sin(32)]`: the layout the indexer q side and + `ds_mla_rope_append` (the MLA query tail and the latent's rope key) read; the indexer key + side reads the same storage as `[rows, 2, 32]`. `identity` (cos 1, sin 0) is the table of a + NoPE indexer layer, whose kernels have no no-rope mode. Every row depends only on its own + index, so a longer table is the same table with more rows. + """ + inv = _yarn_lerope_inv_freq(cfg, alpha) + t = torch.arange(positions, dtype=torch.float32) + angle = torch.einsum("i,j -> ij", t, inv) + cos = angle.cos().to(dtype).float() + sin = angle.sin().to(dtype).float() + identity = torch.cat([torch.ones_like(cos), torch.zeros_like(sin)], dim=1) + return SimpleNamespace( + rows=positions, + indexer=torch.cat([cos, sin], dim=1).to(device).contiguous(), + identity=identity.to(device).contiguous(), + ) + + +def _moe_scalars(isc1, isc2, ws2_13, ws2_2) -> SimpleNamespace: + """The NVFP4 MoE's call scalars from the checkpoint's per-tensor ones, fp32. + + ModelOpt stores `input_scale` and `weight_scale_2` as the reciprocals of the global scales a + quantizer divides by (`amax / (448 * 6)`), one `isc1` / `isc2` per layer and one + `ws2_13[e]` / `ws2_2[e]` per local expert. So the activation quantizer's global scale is + `1 / isc1`; FC1's accumulator dequantizes by `isc1 * ws2_13[e]` (the gate half, + `output1_scale_gate_scalar`) and, because the FC1 epilogue requantizes the SwiGLU output to + NVFP4 under `1 / isc2`, the linear half by `isc1 * ws2_13[e] / isc2` + (`output1_scale_scalar`); FC2's accumulator dequantizes by `isc2 * ws2_2[e]` + (`output2_scale_scalar`). The reference's TRT-LLM-Gen path folds the same three + (`g1_alphas`, `g1_scale_c`, `g2_alphas`).""" + gate1 = (isc1 * ws2_13).contiguous() + return SimpleNamespace( + gscale=(1.0 / isc1).contiguous(), + out1=(gate1 / isc2).contiguous(), + out1_gate=gate1, + out2=(isc2 * ws2_2).contiguous(), + ) + + +def _layer_views(core: "MistralLarge4TpModelingV2") -> list[SimpleNamespace]: + """Every layer's operands, as the views the entries read: GEMM operands transposed (`.t()`, + the column-major `[K, N]` cuBLAS and the FP8 GEMM read), the FP8 scale storage transposed to + the MN-major operand, `wkv_b` split into the absorption pair, and the NVFP4 MoE's call + scalars (`_moe_scalars`). `next_norm` is the next layer's input norm, the final norm after + the last layer. A sharer's absorption pair is its own `wkv_b`; its latent is its source's.""" + w = core.w + heads, v_dim, kv_lora = core.heads, core.v_dim, core.kv_lora + layers = [] + for i in range(core.num_layers): + p = f"l{i}_" + nope = core.qk_dim if core.nope_layer[i] else core.nope + kvb = w[p + "wkv_b"] + hn = heads * nope + owner = core.kv_source[i] == i + ix = None + if core.full_indexer[i]: + ix = SimpleNamespace( + q_norm_w=w[p + "ix_q_norm_w"], + q_norm_b=w[p + "ix_q_norm_b"], + wq_b=w[p + "ix_wq_b"], + wq_b_s=w[p + "ix_wq_b_s"].t(), + wk_wp=w[p + "ix_wk_wp"].t(), + k_norm_w=w[p + "ix_k_norm_w"], + k_norm_b=w[p + "ix_k_norm_b"], + ssmax_s=w[p + "ix_ssmax_s"], + ssmax_b=w[p + "ix_ssmax_b"], + rope=not core.indexer_nope[i], + ) + moe = None + if i >= core.dense_layers: + moe = SimpleNamespace( + router=w[p + "router"].t(), + fc1_w=w[p + "fc1_w"], + fc1_s=w[p + "fc1_s"], + fc2_w=w[p + "fc2_w"], + fc2_s=w[p + "fc2_s"], + **vars(_moe_scalars(w[p + "isc1"], w[p + "isc2"], w[p + "ws2_13"], w[p + "ws2_2"])), + ) + layers.append( + SimpleNamespace( + index=i, + rope=not core.nope_layer[i], + nope=nope, + owner=owner, + kv_source=core.kv_source[i], + attn_norm=w[p + "attn_norm"], + wq=w[p + "wq"], + wq_s=w[p + "wq_s"].t(), + ssmax_s=w[p + "ssmax_s"], + ssmax_b=w[p + "ssmax_b"], + wkv_a=w[p + "wkv_a"].t() if owner else None, + kv_norm=w[p + "kv_a_norm"] if owner else None, + k_b=kvb[:hn].reshape(heads, nope, kv_lora), + v_b_t=torch.transpose(kvb[hn:].reshape(heads, v_dim, kv_lora), 1, 2), + wo=w[p + "wo"], + wo_s=w[p + "wo_s"].t(), + ffn_norm=w[p + "ffn_norm"], + mlp_gu=w[p + "mlp_gu"], + mlp_gu_s=w[p + "mlp_gu_s"].t(), + mlp_dn=w[p + "mlp_dn"], + mlp_dn_s=w[p + "mlp_dn_s"].t(), + indexer=ix, + moe=moe, + next_norm=( + w[f"l{i + 1}_attn_norm"] if i + 1 < core.num_layers else w["final_norm"] + ), + ) + ) + return layers + + +class MistralLarge4TpModelingV2(DecoderModel): + def __init__(self, model_config: ModelConfig): + super().__init__(model_config) + cfg = model_config.pretrained_config + mapping = model_config.mapping + self.tp_rank = mapping.tp_rank + self.tp_size = mapping.tp_size + # The context selection's gather group: the TP ranks, in the order the engine sizes + # their query shares and the gather concatenates them. + self.tp_group = list(mapping.tp_group) + # The attention block's glue: the fused entries (ssmax_scaling_fused_q, + # ssmax_indexer_q_side_fused, the latent RMSNorm in place, FlashMLA's decode output + # read in place) at every max_seq_len, whose cells cover the 32K and 1M configs' + # positions (to 1048575), rope tables and step shapes; the separate ops they replace + # under `_UNFUSED_GLUE`, at every site. + self.unfused_glue = _UNFUSED_GLUE + # The generation rows' sparse MLA: cute_dsl_ds_mla_sparse_decode at the rank's heads at + # every max_seq_len, whose cells cover the 32K and 1M configs' pools and key lengths; + # flash_mla_sparse_decode over the 64-head padded query under `_GEN_ATTN_FLASH_MLA`. + self.gen_attn_flash_mla = _GEN_ATTN_FLASH_MLA + + self.num_layers = cfg.num_hidden_layers + self.hidden = cfg.hidden_size + self.eps = cfg.rms_norm_eps + self.heads = cfg.num_attention_heads // mapping.tp_size + self.nope = cfg.qk_nope_head_dim + self.rope_dim = cfg.qk_rope_head_dim + # A NoPE layer spends the rope dims on nope: the query head stays this wide. + self.qk_dim = self.nope + self.rope_dim + self.v_dim = cfg.v_head_dim + self.kv_lora = cfg.kv_lora_rank + self.lat_dim = self.kv_lora + self.rope_dim + # Every layer's softmax scale, NoPE layers included: qk_head_dim**-0.5, no YaRN + # temperature (its magnitude is 1). + self.softmax_scale = self.qk_dim**-0.5 + + self.nope_layer = [bool(f) for f in cfg.nope_layer_flags] + self.indexer_nope = [bool(f) for f in cfg.indexer_nope_layer_flags] + self.kv_source = [int(s) for s in cfg.kv_source_layers] + # An owner some later layer attends over (layer 35): a step whose context KV fits the + # held workspace twice keeps its gather there for the sharers; a longer one has each + # sharer gather its source's rows again (`_ctx_workspace`). + self.shared_source = [ + any(s == i and j != i for j, s in enumerate(self.kv_source)) + for i in range(self.num_layers) + ] + self.full_indexer = [role == "full" for role in cfg.index_sharing] + # A 'shared' layer attends with the most recent full layer's selection: each target + # keeps one selection buffer per step, which every full layer overwrites in order. + self.indexer_layers = [i for i in range(self.num_layers) if self.full_indexer[i]] + + self.idx_heads = cfg.index_n_heads + self.idx_dim = cfg.index_head_dim + self.idx_topk = cfg.index_topk + self.idx_eps = cfg.indexer_norm_eps + # The indexer's softmax scale and head average: its scorers take none, so the q side + # folds both into the head weights together with the q quantization scale. + self.idx_weight_scale = self.idx_dim**-0.5 * self.idx_heads**-0.5 + + self.dense_layers = cfg.first_k_dense_replace + self.num_experts = cfg.n_routed_experts + self.topk = cfg.num_experts_per_tok + # The routed experts' tensor-parallel split: every expert this rank answers for (all of + # them at moe_ep_size 1, at offset 0), each at its moe_intermediate_size / moe_tp_size + # slice of the intermediate -- the width the weight table declares and the runner runs. + self.moe_inter = cfg.moe_intermediate_size // mapping.moe_tp_size + self.local_experts = self.num_experts // mapping.moe_ep_size + self.expert_offset = self.local_experts * mapping.moe_ep_rank + # capacity: max_num_tokens -- the collective's peer buffers and the MoE autotuner's + # largest bucket. + self.max_num_tokens = model_config.max_num_tokens + # The rope tables' first size; grown on the first step to the engine's max_seq_len. + self.rope_rows = model_config.max_seq_len or cfg.max_position_embeddings + + self.w = _weights.MODEL_WEIGHTS.declare(_weights.MODEL_WEIGHTS.dims(self)) + + self._rope: SimpleNamespace | None = None + self._derived_for = None + self._ix_k = None + self._ix_k_checked = False + self._pool_row: list[int] = [] + self._stride_factor = 0 + self._max_ctx_len = 0 + self.tp_pg = None + # The engine's sizes, fixed on the first step (`_derive_first_step`). + self.prefill_logits_budget = 0 + self.ctx_workspace_rows = 0 + # The decode top-k's handle, built on the first step at a scorer width it serves. + self._ix_topk = None + self._prefill = self._decode = None + + def post_load_weights(self) -> None: + """Derive what the weights fix, build the kernels and the collectives, then the + targets. + + Named for the shell hook that drives it. Everything compiles here or on the first + (eager) step: a first compile inside a CUDA-graph capture breaks the graph. + """ + dev = torch.cuda.current_device() + self.build_kernels(dev) + # Collective: every rank builds it with the same arguments. The peer buffers hold + # the whole token budget, so one workspace serves every closing collective. + self._lamport = build_lamport_workspace( + self.hidden, self.eps, self.max_num_tokens, self.tp_rank, self.tp_size + ) + # The selection gather's communicator flavor, as the engine's own collectives pick it: + # the MPI session's, or the TP process group's when MPI is disabled. + mapping = self.model_config.mapping + self.tp_pg = mapping.tp_group_pg.boxed() if mpi_disabled() else None + self._prefill = PrefillTarget(self) + self._decode = DecodeTarget(self) + + def build_kernels(self, dev: int) -> None: + """The rank-local derivations: the rope tables, the indexer q side for the unfused glue, + and the generation rows' sparse decode at this rank's heads.""" + self._rope = _rope_tables( + self.model_config.pretrained_config, + self.w["lerope_alpha"], + self.rope_rows, + self.w["final_norm"].dtype, + self.w["final_norm"].device, + ) + if self.unfused_glue: + # The checkpoint's rope is interleaved (GPT-J) for the indexer too. + self._ix_q = build_ix_q_side(dev, is_neox=False) + self._mla_decode = ( + None if self.gen_attn_flash_mla else build_ds_mla_decode(dev, num_heads=self.heads) + ) + + def _derive_first_step(self, md) -> None: + """What depends on values the engine resolves, fixed on the first step it hands over + (eager, ahead of any capture) and again for a replaced KV cache manager.""" + if md.kv_cache_manager is self._derived_for: + return + dev = torch.cuda.current_device() + mgr = md.kv_cache_manager + # The pool layout is the KV cache manager's, which routing cannot see; every latent + # op here reads and writes fp8_ds_mla rows addressed by owner row. + if getattr(mgr, "latent_layout", None) != _LATENT_LAYOUT: + raise RuntimeError( + f"this target attends over an {_LATENT_LAYOUT} latent pool with owner-only " + f"pages (MistralLarge4CacheManager); the engine built {type(mgr).__name__} " + f"with layout {getattr(mgr, 'latent_layout', None)!r}. Use KV cache manager V2 " + "and kv_cache_config.dtype auto, fp8 or fp8_ds_mla." + ) + if list(mgr.kv_source_layers) != self.kv_source: + raise RuntimeError("the KV cache manager's kv_source_layers differ from the config's") + # A table shorter than max_seq_len is read past its end with no check. + if md.max_seq_len > self._rope.rows: + self.rope_rows = md.max_seq_len + self._rope = _rope_tables( + self.model_config.pretrained_config, + self.w["lerope_alpha"], + self.rope_rows, + self.w["final_norm"].dtype, + self.w["final_norm"].device, + ) + # Layer i reads owner row `_pool_row[i]` (its source's); a block holds every owner's + # page, so a global row is `block * owners * tpb + row * tpb + offset`. + self._pool_row = [mgr.latent_pool_row(i) for i in range(self.num_layers)] + self._stride_factor = len(mgr.kv_owner_layers) * md.tokens_per_block + # The decode scorer's column count: frozen into a capture, so an engine constant. + self._max_ctx_len = md.max_seq_len + # The indexer's prefill logits budget and the held context workspace's rows, from the + # engine's max_num_tokens, max_num_requests, max_seq_len and runtime features. + self.prefill_logits_budget = _prefill_logits_budget(md.max_num_tokens, md.max_seq_len) + self.ctx_workspace_rows = _ctx_workspace_rows(md) + logger.info( + f"modeling_v2 Mistral Large 4 NVFP4: sized from the engine: prefill logits budget " + f"{self.prefill_logits_budget}, context workspace {self.ctx_workspace_rows} rows" + ) + # The decode selector, by the scorer's column count: cute_dsl_radix_topk_decode's + # handle (built once) at a width it serves, else none and the trtllm op selects. + lo, hi = _TOPK_DECODE_COLUMNS + if not (_TOPK_DECODE_CUTE_DSL and lo <= self._max_ctx_len <= hi): + self._ix_topk = None + elif self._ix_topk is None: + self._ix_topk = build_ix_topk_decode(dev, next_n=1) + # Both key-pool layouts: a launcher compiled for a contiguous view addresses a strided + # one as if it were dense. + if self._ix_k is None: + self._ix_k = { + contiguous: build_ix_k_cache(dev, is_neox=False, cache_contiguous=contiguous) + for contiguous in (False, True) + } + self._derived_for = md.kv_cache_manager + + def _select_target(self, attn_metadata) -> "PrefillTarget | DecodeTarget": + """Which target runs this step: one host-side integer read. A mixed batch goes to + prefill, so prefill is the general case and decode the specialization that may assume + it has no context rows.""" + return self._decode if attn_metadata.num_contexts == 0 else self._prefill + + def forward( + self, + attn_metadata: AttentionMetadata, + input_ids: torch.IntTensor | None = None, + position_ids: torch.IntTensor | None = None, + inputs_embeds: torch.FloatTensor | None = None, + lora_params: dict | None = None, + **kwargs, + ) -> torch.Tensor: + """Derive what the first step fixes, then route.""" + self._derive_first_step(attn_metadata) + return self._select_target(attn_metadata).forward( + attn_metadata, + input_ids=input_ids, + position_ids=position_ids, + inputs_embeds=inputs_embeds, + lora_params=lora_params, + **kwargs, + ) + + +class PrefillTarget: + """The general case: context rows, and possibly generation rows behind them.""" + + # The held context workspace, bf16 [core.ctx_workspace_rows, 576]: allocated on the first + # context step and held from then on (`_ctx_workspace`). + _ws_buf: torch.Tensor | None = None + # The indexer's prefill top-k scratch (`gvr_workspace`, 20 MiB): allocated on the first + # context step that selects and held, used on the main stream only (`_select_ctx`). + _gvr_ws: torch.Tensor | None = None + + def __init__(self, core: MistralLarge4TpModelingV2) -> None: + """Take the per-layer views once, off the real weights. Everything an engine value + fixes is read off `self.core` at the call.""" + self.core = core + # Off unless TRTLLM_MODELING_V2_VALIDATE asks for it. Each target owns its own flag: + # the two `step_args` projections differ, so probing one does not prove the other. + self._contract_pending = step_contract_enabled() + self._layers = _layer_views(core) + self._embed = core.w["embed"] + self._norm0 = core.w["l0_attn_norm"] + + def _check_step_contract(self, md) -> None: + """Opt-in first-forward fail-fast: calling `step_args` reads every metadata field this + target consumes (private trtllm surface). Fixed at engine construction, so once per + target instance, and off in a served engine.""" + if not self._contract_pending: + return + try: + self.step_args(md) + except AttributeError as exc: + raise AssertionError(f"attention metadata surface drifted: {exc}") from exc + self._contract_pending = False + + def step_args(self, md: TrtllmAttentionMetadata) -> dict: + """The common batch state, plus the context rows': their counts and lengths, the + indexer's prefill windows and scoring blocks, its query split over the TP ranks and its + key gather mappings, and the context rope/append op's offsets and their host prefix + sums.""" + step = _step_common(md, self.core.indexer_layers) + ctx = md.num_contexts + step.update( + num_contexts=ctx, + num_ctx_tokens=md.num_ctx_tokens, + seq_lens=md.seq_lens, + skip_ctx=bool(md.skip_indexer_for_ctx_reqs), + cu_seqlen_ks=md.cu_seqlen_ks, + cu_seqlen_ke=md.cu_seqlen_ke, + indexer_prefill_chunks=md.indexer_prefill_chunks, + # (q_begin, q_end, per-rank row counts) once a chunked-prefill step's context rows + # reach the engine's q_split_threshold, None below it: the chunks then cover this + # rank's rows [q_begin, q_end) only. + indexer_prefill_q_split=md.indexer_prefill_q_split, + num_ctx_kv_tokens=md.num_ctx_kv_tokens, + slot_mapping_fp8_fullkv=md.slot_mapping_fp8_fullkv, + slot_mapping_scale_fullkv=md.slot_mapping_scale_fullkv, + **_ctx_kv_offsets(md, ctx), + **_ctx_host_offsets(md, ctx), + ) + step["indexer_blocks"] = _indexer_blocks(md.indexer_prefill_chunks, step) + return step + + # ---- one layer's pieces, context and generation rows ---- + + def _linear_fp8(self, x, w, w_s): + """`x @ w.T`: the bf16 activation quantized 1x128 with power-of-two scales, then the + DeepGEMM block-scale GEMM against the load-time UE8M0 weight scales.""" + codes, scale = fp8_quantize_1x128_packed_ue8m0(x) + return fp8_prequantized_swap_ab_gemm(codes, scale, w, w_s, torch.bfloat16, True) + + def _latent(self, x, L, nope_lat): + """`[RMSNorm(c_kv) | k_pe]`, 576 wide: a RoPE owner's kv_a row, normalized in place; a + NoPE owner's 512-wide latent normalized into `nope_lat`, whose rope key stays zero.""" + core = self.core + kv = cublas_mm(x, L.wkv_a) + if core.unfused_glue: + c_kv = flashinfer_rmsnorm(kv[:, : core.kv_lora], L.kv_norm, core.eps) + if L.rope: + k_pe = kv[:, core.kv_lora :] + else: + k_pe = torch.zeros([kv.shape[0], core.rope_dim], dtype=kv.dtype, device=kv.device) + return torch.cat([c_kv, k_pe], dim=-1) + if L.rope: + c_kv = kv[:, : core.kv_lora] + flashinfer_rmsnorm(c_kv, L.kv_norm, core.eps, out=c_kv) + return kv + flashinfer_rmsnorm(kv, L.kv_norm, core.eps, out=nope_lat[:, : core.kv_lora]) + return nope_lat + + def _indexer(self, step, x, pos, L, topk_buf, tc: int, skip_ctx: bool, skip_gen: bool): + """Append this step's keys to the layer's key pool; select `index_topk` positions per + row into `topk_buf` for the rows whose phase the engine did not skip. Under the + engine's query split this rank selects for its share of the context rows, and the + shares are gathered into rows [0, tc) before any layer reads them.""" + core = self.core + ix = L.indexer + rows = x.shape[0] + table = core._rope.indexer if ix.rope else core._rope.identity + kw = cublas_mm(x, ix.wk_wp) + k = kw[:, : core.idx_dim].contiguous() + k_pool = step["ix_pools"][L.index] + k_pool = cute_dsl_indexer_k_cache_update( + k, ix.k_norm_w, ix.k_norm_b, pos, table.view(-1, 2, core.rope_dim // 2), k_pool, + step["slot_mapping_fp8"], step["slot_mapping_scale"], rows, + is_neox=False, compiled=core._ix_k[k_pool.is_contiguous()], + # Its range check is a host sync: once, on the first (eager) call. + validate=not core._ix_k_checked, + ) # fmt: skip + core._ix_k_checked = True + if skip_ctx and skip_gen: + return + qn = layer_norm(x, ix.q_norm_w, ix.q_norm_b, core.idx_eps) + q = torch.reshape(self._linear_fp8(qn, ix.wq_b, ix.wq_b_s), [rows, core.idx_heads, -1]) + q_fp8 = torch.empty_like(q, dtype=torch.float8_e4m3fn) + weights = torch.empty([rows, core.idx_heads], dtype=torch.float32, device=x.device) + if core.unfused_glue: + q = ssmax_scaling(q, pos, ix.ssmax_s, ix.ssmax_b) + q_fp8, weights = cute_dsl_indexer_q_side_fused( + q, pos, table, kw[:, core.idx_dim :].float().contiguous(), core.idx_weight_scale, + q_fp8, weights, is_neox=False, compiled=core._ix_q, + ) # fmt: skip + else: + # SSMax, rope, e4m3 and the head weights (read in place out of `kw`) in one launch. + q_fp8, weights = ssmax_indexer_q_side_fused( + q, pos, ix.ssmax_s, ix.ssmax_b, table, kw[:, core.idx_dim :], + core.idx_weight_scale, q_fp8, weights, + ) # fmt: skip + if tc and not skip_ctx: + self._select_ctx(step, k_pool, q_fp8, weights, topk_buf) + q_split = step["indexer_prefill_q_split"] + if q_split is not None: + # The chunks covered this rank's share only. The split depends on the step's + # counts alone, so every rank of the group makes this gather once per selecting + # layer, in layer order, at the same sizes: the communicator pairs calls by + # position. + q_begin, q_end, q_sizes = q_split + topk_buf[:tc].copy_( + allgather(topk_buf[q_begin:q_end], q_sizes, core.tp_group, core.tp_pg) + ) + gen = rows - tc + if gen and not skip_gen: + ctx = step["num_contexts"] + num_gens = step["num_seqs"] - ctx + assert gen % num_gens == 0, (gen, num_gens) + next_n = gen // num_gens + kv_lens = step["kv_lens_cuda"][ctx : ctx + num_gens] + logits = cute_dsl_fp8_paged_mqa_logits( + torch.reshape(q_fp8[tc:], [num_gens, next_n, core.idx_heads, core.idx_dim]), + k_pool, + weights[tc:], + kv_lens, + step["indexer_k_cache_block_offsets"][ctx : ctx + num_gens], + step["scheduler_metadata_buffer"], + core._max_ctx_len, + ) + if next_n != 1 or core._ix_topk is None: + indexer_topk_decode( + logits, kv_lens, topk_buf[tc:], next_n, core.idx_topk, 1, + step["radix_aux_indices"], step["radix_aux_logits"], + ) # fmt: skip + else: + cute_dsl_radix_topk_decode( + logits, kv_lens, topk_buf[tc:], next_n=1, compiled=core._ix_topk + ) + + def _ix_plan(self, step, heads: int) -> list[SimpleNamespace]: + """The step's indexer scoring plan, the same for each of its indexer layers: per scoring + block (`_indexer_blocks`, this rank's share of the context rows under the engine's + query split), its rows, its keys in the step's full-KV mappings and its query tiles of + `floor(prefill_logits_budget / keys)` rows (at least one, the last short), each tile + with its rows' windows and the split-KV schedule of its logits call + (`mqa_logits_schedule`). A block with no key has no tile. + + Built on the step's first indexer layer and kept in `step`: the windows, the key counts + and so the schedules depend on the step alone, and the layers differ only in the keys + and queries they score.""" + core = self.core + plan = [] + for block in step["indexer_blocks"]: + n_k = block.k_token_end - block.k_token_start + if block.token_end <= block.token_start: + continue + tiles = [] + if n_k > 0: + tile = max(1, core.prefill_logits_budget // n_k) + for lo in range(block.token_start, block.token_end, tile): + hi = min(lo + tile, block.token_end) + # The windows are the block's own, indexed from its first row. + ks = block.cu_seqlen_ks[lo - block.token_start : hi - block.token_start] + ke = block.cu_seqlen_ke[lo - block.token_start : hi - block.token_start] + tiles.append((lo, hi, ks, ke, mqa_logits_schedule(ks, ke, n_k, heads))) + plan.append( + SimpleNamespace( + token_start=block.token_start, + token_end=block.token_end, + k_token_start=block.k_token_start, + n_k=n_k, + tiles=tiles, + ) + ) + return plan + + def _select_ctx(self, step, k_pool, q_fp8, weights, topk_buf) -> None: + """Select `index_topk` positions, window-local, for each context row this rank scores, + along the step's scoring plan (`_ix_plan`): per block, gather its keys out of the + layer's key pool through the step's full-KV mappings, then per query tile score its + rows against them with the tile's split-KV schedule, which spreads each query block's + keys over the SMs (a 256-row tile at 1048576 keys keeps every SM busy, the grid-stride + call 64), and select over each row's window into `topk_buf` with the self-sampling GVR + top-k, one streaming pass per row. Both answer as the grid-stride scorer and the radix + top-k: the logits bitwise, the selection the same set up to the keys tied at a row's + 2048th score. A block with no key leaves its rows' slots empty (-1), and each tile's + logits are freed before the next tile's are allocated.""" + core = self.core + plan = step.get("ix_plan") + if plan is None: + plan = step["ix_plan"] = self._ix_plan(step, q_fp8.shape[1]) + if self._gvr_ws is None: + # On the first context step, which the KV-capacity estimate's profiling forward + # is: the pool is sized with the top-k's scratch held, and every top-k launch this + # engine's tiles reach compiles here rather than on a served step (each tile holds + # at most the logits budget, one block at most max_seq_len keys). + self._gvr_ws = gvr_workspace(q_fp8.device) + t0 = time.monotonic() + calls = gvr_warmup( + core.idx_topk, core._max_ctx_len, core.prefill_logits_budget, q_fp8.device, + self._gvr_ws, + ) # fmt: skip + logger.info( + f"modeling_v2 Mistral Large 4 NVFP4: indexer prefill top-k compiled for this " + f"engine's tiles ({calls} warm calls, {time.monotonic() - t0:.1f} s)" + ) + for block in plan: + if block.n_k <= 0: + # No key to select from: every slot of the block's rows is empty. + topk_buf[block.token_start : block.token_end].fill_(-1) + continue + ck, cs = indexer_k_cache_gather_op( + k_pool, step["slot_mapping_fp8_fullkv"], step["slot_mapping_scale_fullkv"], + block.k_token_start, block.n_k, core.idx_dim, + ) # fmt: skip + cs = torch.reshape(cs, [-1]) + for lo, hi, ks, ke, schedule in block.tiles: + logits = fp8_mqa_logits( + q_fp8[lo:hi], (ck, cs), weights[lo:hi], ks, ke, schedule_meta=schedule + ) + cute_dsl_gvr_topk_prefill( + logits, ks, ke, topk_buf[lo:hi], core.idx_topk, self._gvr_ws + ) + # Freed before the next tile's is allocated: one tile is live at a time. + del logits + + def _attn_ctx(self, step, L, fused_q, lat_out, req_idx, topk_buf, ctx): + """Sparse MLA over the context rows into `lat_out`, on the step's path (`ctx`). + + Workspace path: FlashMLA's bf16 sparse prefill over the context requests' latent rows + of the source layer, dequantized out of the pool (this step's rows included, so every + row is read FP8-rounded) into the held workspace's first rows. A shared source (layer + 35) is gathered once into the rows the step keeps beside them, when it has them, and + read there by every layer that shares it; otherwise each of those layers gathers it + again into the first rows. Decode path, a step past the held workspace: the selections + as pool slots, then FlashMLA's sparse fp8 decode over the pool with every context row a + one-row sequence, in the step's sub-batches.""" + core = self.core + tc = fused_q.shape[0] + src = L.kv_source + if ctx.ws_buf is None: + slots = convert_req_index_to_global( + req_idx[:tc], step["block_table"], topk_buf[:tc], step["tokens_per_block"], + core.idx_topk, core._stride_factor, core._pool_row[src], + ) # fmt: skip + for lo, hi in ctx.sub_batches: + # Past 11622 rows FlashMLA's scheduler exits the process instead of raising. + assert hi - lo <= _CTX_DECODE_ROWS, (lo, hi) + flash_mla_sparse_decode( + fused_q[lo:hi], step["latent_kv"], slots[lo:hi], ctx.sched[hi - lo], + core.softmax_scale, lat_out[lo:hi], + ) # fmt: skip + return + ws = ctx.ws.get(src) + if ws is None: + keep = core.shared_source[src] and ctx.kept is not None + ws = ds_mla_dequant_gather( + step["latent_pool"], core._pool_row[src], ctx.kv_req, ctx.kv_pos, + step["block_table"], ctx.kept if keep else ctx.ws_buf, + ) # fmt: skip + if keep: + ctx.ws[src] = ws + # Request-local selections to workspace rows: each request's run starts at its + # cumulative KV offset. + sel = topk_buf[:tc] + idx = torch.where(sel >= 0, sel + ctx.ws_off[req_idx[:tc]][:, None], sel) + flash_mla_sparse_prefill(fused_q, ws, idx.to(torch.int32), core.softmax_scale, lat_out) + + def _attn_gen(self, step, L, fused_q, lat_out, slots, sched): + """Sparse MLA over the generation rows behind the context rows into `lat_out`, over the + pool: this rank's heads read off the 64-head scratch the prefill kernel shares (or, under + `gen_attn_flash_mla`, FlashMLA's decode over all 64, one scheduler for the step).""" + core = self.core + if core.gen_attn_flash_mla: + flash_mla_sparse_decode( + fused_q, step["latent_kv"], slots, sched, core.softmax_scale, lat_out + ) + else: + cute_dsl_ds_mla_sparse_decode( + fused_q, step["latent_kv"], slots, core.softmax_scale, lat_out, + compiled=core._mla_decode, + ) # fmt: skip + + def _mlp(self, o, L): + """This rank's share of the layer's MLP, a partial sum: the dense SwiGLU slice on the + dense layer; on a MoE layer every routed expert's width slice plus the shared expert's + slice.""" + core = self.core + act = flashinfer_silu_and_mul(self._linear_fp8(o, L.mlp_gu, L.mlp_gu_s)) + shared = self._linear_fp8(act, L.mlp_dn, L.mlp_dn_s) + m = L.moe + if m is None: + return shared + logits = cublas_mm(o, m.router, None, torch.float32) + ids, weights = renorm_moe_routing_op(logits, core.topk, torch.bfloat16) + # One NVFP4 quantization under the layer's static scale serves every expert's slice on + # every rank: the width slices' partials sum. + xq, xs = fp4_quantize(o, m.gscale, _NVFP4_SF_VEC, False, _NVFP4_SF_LINEAR) + routed = fp4_block_scale_moe_runner( + None, None, xq, xs.view(torch.float8_e4m3fn), m.fc1_w, m.fc1_s, None, None, None, + None, m.fc2_w, m.fc2_s, None, m.out1, m.out1_gate, m.out2, core.num_experts, + core.topk, None, None, core.moe_inter, core.expert_offset, core.local_experts, None, + _MOE_ROUTING_METHOD, True, _MOE_ACT_SWIGLU, topk_weights=weights, topk_ids=ids, + tune_max_num_tokens=core.max_num_tokens, + )[0] # fmt: skip + return torch.add(routed, shared) + + def _topk_buf(self, step, rows: int, tc: int, skip_ctx: bool, skip_gen: bool): + """Where this step's selections land, request-local positions. A phase the engine + skipped (every sequence within index_topk) takes the dense causal index it wrote.""" + if skip_ctx and skip_gen: + return step["topk_indices_buffer"][:rows] + buf = torch.empty( + [rows, self.core.idx_topk], dtype=torch.int32, device=step["block_table"].device + ) + if skip_ctx and tc: + buf[:tc].copy_(step["topk_indices_buffer"][:tc]) + if skip_gen and rows > tc: + buf[tc:].copy_(step["topk_indices_buffer"][tc:rows]) + return buf + + def forward( + self, + attn_metadata: AttentionMetadata, + input_ids: torch.IntTensor | None = None, + position_ids: torch.IntTensor | None = None, + inputs_embeds: torch.FloatTensor | None = None, + lora_params: dict | None = None, + **kwargs, + ) -> torch.Tensor: + core = self.core + self._check_step_contract(attn_metadata) + step = self.step_args(attn_metadata) + h = ( + nn.functional.embedding(input_ids, self._embed) + if inputs_embeds is None + else inputs_embeds + ) + rows = h.shape[0] + dev = h.device + # The slot mappings and windows follow the prepared token order; a padded input + # would scatter keys into stale slots. + assert rows == step["num_tokens"], (rows, step["num_tokens"]) + tc = step["num_ctx_tokens"] + gen = rows - tc + skip_ctx = step["skip_ctx"] + skip_gen = step["skip_gen"] if gen else True + pos = torch.reshape(position_ids, [-1]) + req_idx = step["req_idx_per_token"][:rows] + topk_buf = self._topk_buf(step, rows, tc, skip_ctx, skip_gen) + scratch = self._scratch(rows, h.dtype, dev) + ctx = self._ctx_workspace(step, tc, dev) if tc else None + + x = flashinfer_rmsnorm(h, self._norm0, core.eps) + residual = h + spec_metadata = kwargs.get("spec_metadata") + for layer_idx, L in enumerate(self._layers): + attn_out = self._attention( + step, L, x, pos, req_idx, topk_buf, scratch, ctx, tc, skip_ctx, skip_gen + ) + partial = self._linear_fp8(attn_out, L.wo, L.wo_s) + o = cute_dsl_allreduce_add_rmsnorm( + partial, residual, L.ffn_norm, torch.empty_like(partial), core._lamport + ) + mlp = self._mlp(o, L) + x = cute_dsl_allreduce_add_rmsnorm( + mlp, residual, L.next_norm, torch.empty_like(mlp), core._lamport + ) + if spec_metadata is not None: + spec_metadata.maybe_capture_hidden_states(layer_idx, residual) + return x + + def _scratch(self, rows: int, dtype, dev) -> SimpleNamespace: + """Per-step scratch every layer rewrites in turn: the attention output, the absorbed + query at the prefill kernel's 64 heads (this rank's first, the rest zero for the step), + the latent attention output, a NoPE owner's 576-wide latent rows (the rope key zero for + the step), and, under `gen_attn_flash_mla`, the generation rows' FlashMLA scheduler, + filled by the step's first decode call and shared by the rest.""" + core = self.core + bf16 = dict(dtype=dtype, device=dev) + return SimpleNamespace( + attn_out=torch.empty([rows, core.heads * core.v_dim], **bf16), + fused_q=torch.zeros([rows, _FLASH_MLA_HEADS, core.lat_dim], **bf16), + lat_out=torch.empty([rows, core.heads, core.kv_lora], **bf16), + nope_lat=None if core.unfused_glue else torch.zeros([rows, core.lat_dim], **bf16), + sched=get_mla_metadata()[0] if core.gen_attn_flash_mla else None, + ) + + def _ctx_workspace(self, step, tc: int, dev) -> SimpleNamespace: + """The context rows' path for the step, chosen once on host ints by its context KV: + every context request's cached prefix and new tokens (the host + `ctx_kv_indptr[num_contexts]`; the engine's `num_ctx_kv_tokens` counts the new ones + only, and under chunked prefill every chunk after a prompt's first has a prefix). + + The held workspace, bf16 `[core.ctx_workspace_rows, 576]` (`_ctx_workspace_rows`), is + allocated on the first context step, whichever path that step takes, and held. The + engine's KV-capacity estimate runs context steps as its profiling forward and sizes the + pool after their peak memory (`KvCacheCreator.configure_kv_cache_capacity`), so the pool + is sized with the workspace held. A generation-only engine runs no context step and + holds none. + + The workspace path, a step whose context KV fits the held rows: request and position + per workspace row as the dequantizing gather and the prefill kernel address them, each + request's offset, the held workspace's first `n_kv` rows, which every layer's gather + rewrites, and, when the context KV fits twice, the next `n_kv` rows (`kept`), where a + shared owner's gather stays for the step (`ws`). + + The decode path, a step past the held rows, with no workspace: the context rows' + sub-batches of at most `_CTX_DECODE_ROWS`, and one FlashMLA scheduler per sub-batch + size, filled by its first call and shared by the step's layers.""" + # Under KV reuse a context request attends over its cached prefix and its new tokens. + n_kv = step["ctx_cached_tokens"] + step["num_ctx_tokens"] + if self._ws_buf is None: + # On the first context step: the KV-capacity estimate's profiling forward is one, + # so the peak it measures, which sizes the pool, holds this buffer. + self._ws_buf = torch.empty( + [self.core.ctx_workspace_rows, self.core.lat_dim], dtype=torch.bfloat16, device=dev + ) + logger.info( + f"modeling_v2 Mistral Large 4 NVFP4: context workspace held, {self._ws_buf.shape[0]} " + f"rows x {self.core.lat_dim} bf16 ({self._ws_buf.nbytes / 2**20:.0f} MiB)" + ) + held = self._ws_buf.shape[0] + if n_kv > held: + bounds = [(lo, min(lo + _CTX_DECODE_ROWS, tc)) for lo in range(0, tc, _CTX_DECODE_ROWS)] + logger.debug( + f"modeling_v2 Mistral Large 4 NVFP4: context KV {n_kv} past the held {held} rows: " + f"decode path, {len(bounds)} sub-batches" + ) + return SimpleNamespace( + ws_buf=None, + sub_batches=bounds, + sched={rows: get_mla_metadata()[0] for rows in {hi - lo for lo, hi in bounds}}, + ) + ctx = step["num_contexts"] + cu = step["cu_ctx_kv"] + lens = cu[1:] - cu[:-1] + kv_req = torch.repeat_interleave( + torch.arange(ctx, dtype=torch.int32, device=dev), lens, output_size=n_kv + ) + kv_pos = (torch.arange(n_kv, device=dev) - cu[:-1][kv_req]).to(torch.int32) + keep = 2 * n_kv <= held + logger.debug( + f"modeling_v2 Mistral Large 4 NVFP4: context KV {n_kv} of the held {held} rows: " + f"workspace path, the shared source {'kept' if keep else 'gathered again'}" + ) + return SimpleNamespace( + kv_req=kv_req, + kv_pos=kv_pos, + ws_off=cu[:-1], + ws_buf=self._ws_buf[:n_kv], + kept=self._ws_buf[n_kv : 2 * n_kv] if keep else None, + ws={}, + ) + + def _attention(self, step, L, x, pos, req_idx, topk_buf, s, ctx, tc, skip_ctx, skip_gen): + """One layer's attention block, both phases, into `s.attn_out` (returned): the query + with SSMax, the indexer's selection, the absorbed query, the owner's rope and + fp8_ds_mla append, sparse MLA over the context and the generation rows, and the V + up-projection.""" + core = self.core + rows = x.shape[0] + q = self._linear_fp8(x, L.wq, L.wq_s) + q3 = torch.reshape(q, [rows, core.heads, core.qk_dim]) + fq = s.fused_q[:, : core.heads] + if core.unfused_glue: + q3 = ssmax_scaling(q3, pos, L.ssmax_s, L.ssmax_b) + if L.rope: + fq[..., core.kv_lora :].copy_(q3[..., L.nope :]) + else: + fq[..., core.kv_lora :].zero_() + else: + # SSMax in place; its rope half (a NoPE layer's zero) into the absorbed query's tail. + ssmax_scaling_fused_q( + q3, pos, L.ssmax_s, L.ssmax_b, q3, fq[..., core.kv_lora :], L.rope + ) + if L.indexer is not None: + self._indexer(step, x, pos, L, topk_buf, tc, skip_ctx, skip_gen) + bmm_out( + torch.transpose(q3[..., : L.nope], 0, 1), L.k_b, + torch.transpose(fq[..., : core.kv_lora], 0, 1), + ) # fmt: skip + if L.owner: + # Every row's latent, context and generation, before either phase reads the pool; + # the query tail is roped in the same call. + ds_mla_rope_append( + self._latent(x, L, s.nope_lat), fq, pos, core._rope.indexer, req_idx, + step["block_table"], step["latent_pool"], core._pool_row[L.index], L.rope, + ) # fmt: skip + if tc: + self._attn_ctx(step, L, s.fused_q[:tc], s.lat_out[:tc], req_idx, topk_buf, ctx) + if rows > tc: + slots = convert_req_index_to_global( + req_idx[tc:], step["block_table"], topk_buf[tc:], step["tokens_per_block"], + core.idx_topk, core._stride_factor, core._pool_row[L.kv_source], + ) # fmt: skip + self._attn_gen(step, L, s.fused_q[tc:], s.lat_out[tc:], slots, s.sched) + bmm_out( + torch.transpose(s.lat_out, 0, 1), + L.v_b_t, + torch.transpose(torch.reshape(s.attn_out, [rows, core.heads, core.v_dim]), 0, 1), + ) + return s.attn_out + + def dspark_forward( + self, + draft, + noise_embedding, + query_positions, + num_ctx_per_req, + ctx_k_cache, + ctx_v_cache, + ctx_cache_batch_idx, + ctx_kv_cache=None, + ctx_page_table=None, + block_placement=None, + ): + # Only the worker's fused paged-latent preparation places the block + # (block_placement), and a drafter with embed_draft_block does not take + # it; this decode reads its context window through ctx_page_table. + del block_placement + draft._build_fused_kv_buffers() + draft._ensure_rope() + core = self.core + batch, block, hidden = noise_embedding.shape + rows = batch * block + # Gather a bounded window, not every page in the configured context. + # Invalid padding slots are masked even if their pages are allocated. + lengths = num_ctx_per_req[:batch].long() + offsets = torch.arange(draft.window, device=noise_embedding.device) + positions = (lengths - draft.window).clamp_min(0)[:, None] + offsets[None, :] + valid = positions < lengths[:, None] + slots = ctx_cache_batch_idx.long() + if ctx_page_table is not None: + page_size = ctx_kv_cache[0].shape[-2] + table = ctx_page_table.index_select(0, slots).long() + safe_positions = positions.clamp(max=table.shape[1] * page_size - 1) + pages = table.gather(1, safe_positions // page_size).clamp_min(0) + page_offsets = safe_positions % page_size + else: + safe_positions = positions.clamp(max=ctx_k_cache.shape[2] - 1) + key_valid = torch.cat( + (valid, torch.ones(batch, block, dtype=torch.bool, device=valid.device)), dim=1 + ) + query = torch.empty( + batch, + block, + core.heads, + core.lat_dim, + dtype=noise_embedding.dtype, + device=noise_embedding.device, + ) + values = torch.empty( + rows, core.heads, core.v_dim, dtype=noise_embedding.dtype, device=noise_embedding.device + ) + residual = noise_embedding.reshape(rows, hidden).clone() + normalized = torch.empty_like(residual) + x = flashinfer_rmsnorm(residual, draft._layers[0].attn_norm, core.eps) + for i, layer in enumerate(draft._layers): + q = cublas_mm(x, layer.q, None, None).view(batch, block, core.heads, core.qk_dim) + bmm_out( + q[..., : core.nope].reshape(rows, core.heads, core.nope).transpose(0, 1), + layer.k_b, + query[..., : core.kv_lora].view(rows, core.heads, core.kv_lora).transpose(0, 1), + ) + query[..., core.kv_lora :].copy_(draft._rotate(q[..., core.nope :], query_positions)) + kv = cublas_mm(x, layer.kv_a, None, None) + content = flashinfer_rmsnorm( + kv[:, : core.kv_lora].contiguous(), layer.kv_norm, core.eps + ) + block_kv = torch.cat( + ( + content.view(batch, block, core.kv_lora), + draft._rotate( + kv[:, core.kv_lora :].view(batch, block, core.rope_dim), query_positions + ), + ), + dim=-1, + ) + if ctx_page_table is not None: + context = ctx_kv_cache[i][:, 0, 0][pages, page_offsets] + else: + context = ctx_k_cache[slots[:, None], i, safe_positions, 0] + # Zero masked values as well as scores: 0 * an uninitialized + # NaN in the latent pool must not contaminate the weighted sum. + context = context.masked_fill(~valid.unsqueeze(-1), 0) + keys = torch.cat((context, block_kv), dim=1) + scores = torch.einsum("bshd,btd->bsht", query, keys).float() * core.softmax_scale + scores = scores.masked_fill(~key_valid[:, None, None, :], float("-inf")) + probabilities = torch.softmax(scores, dim=-1).to(keys.dtype) + latent = torch.einsum("bsht,btc->bshc", probabilities, keys[..., : core.kv_lora]) + bmm_out( + latent.reshape(rows, core.heads, core.kv_lora).transpose(0, 1), + layer.v_b.transpose(1, 2), + values.transpose(0, 1), + ) + partial = cublas_mm(values.reshape(rows, -1), layer.out, None, None) + o = cute_dsl_allreduce_add_rmsnorm( + partial, residual, layer.ffn_norm, normalized, core._lamport + ) + gate = cublas_mm(o, layer.gate, None, None) + up = cublas_mm(o, layer.up, None, None) + activated = flashinfer_silu_and_mul(torch.cat((gate, up), dim=-1)) + mlp = cublas_mm(activated, layer.down, None, None) + x = cute_dsl_allreduce_add_rmsnorm( + mlp, residual, layer.next_norm, normalized, core._lamport + ) + return x + + +class DecodeTarget: + """Generation-only steps, including uniform DSpark verification blocks; graph captured.""" + + def __init__(self, core: MistralLarge4TpModelingV2) -> None: + """Take the per-layer views once, off the real weights. Everything an engine value + fixes is read off `self.core` at the call.""" + self.core = core + self._contract_pending = step_contract_enabled() + self._layers = _layer_views(core) + self._embed = core.w["embed"] + self._norm0 = core.w["l0_attn_norm"] + + def _check_step_contract(self, md) -> None: + """Opt-in first-forward fail-fast: calling `step_args` reads every metadata field this + target consumes (private trtllm surface). Fixed at engine construction, so once per + target instance, and off in a served engine.""" + if not self._contract_pending: + return + try: + self.step_args(md) + except AttributeError as exc: + raise AssertionError(f"attention metadata surface drifted: {exc}") from exc + self._contract_pending = False + + def step_args(self, md: TrtllmAttentionMetadata) -> dict: + """The common batch state. States the two context counts instead of reading them + back: this target runs only when there are no context rows.""" + step = _step_common(md, self.core.indexer_layers) + step.update(num_contexts=0, num_ctx_tokens=0) + return step + + def _linear_fp8(self, x, w, w_s): + """`x @ w.T`: the bf16 activation quantized 1x128 with power-of-two scales, then the + DeepGEMM block-scale GEMM against the load-time UE8M0 weight scales.""" + codes, scale = fp8_quantize_1x128_packed_ue8m0(x) + return fp8_prequantized_swap_ab_gemm(codes, scale, w, w_s, torch.bfloat16, True) + + def _latent(self, x, L, nope_lat): + """`[RMSNorm(c_kv) | k_pe]`, 576 wide: a RoPE owner's kv_a row, normalized in place; a + NoPE owner's 512-wide latent normalized into `nope_lat`, whose rope key stays zero.""" + core = self.core + kv = cublas_mm(x, L.wkv_a) + if core.unfused_glue: + c_kv = flashinfer_rmsnorm(kv[:, : core.kv_lora], L.kv_norm, core.eps) + if L.rope: + k_pe = kv[:, core.kv_lora :] + else: + k_pe = torch.zeros([kv.shape[0], core.rope_dim], dtype=kv.dtype, device=kv.device) + return torch.cat([c_kv, k_pe], dim=-1) + if L.rope: + c_kv = kv[:, : core.kv_lora] + flashinfer_rmsnorm(c_kv, L.kv_norm, core.eps, out=c_kv) + return kv + flashinfer_rmsnorm(kv, L.kv_norm, core.eps, out=nope_lat[:, : core.kv_lora]) + return nope_lat + + def _indexer(self, step, x, pos, L, topk_buf, skip_gen: bool): + """Append this step's keys to the layer's key pool; unless the engine skipped the + indexer (every sequence within index_topk), score every sequence's paged keys and + select `index_topk` positions per row into `topk_buf`.""" + core = self.core + ix = L.indexer + rows = x.shape[0] + table = core._rope.indexer if ix.rope else core._rope.identity + kw = cublas_mm(x, ix.wk_wp) + k = kw[:, : core.idx_dim].contiguous() + k_pool = step["ix_pools"][L.index] + k_pool = cute_dsl_indexer_k_cache_update( + k, ix.k_norm_w, ix.k_norm_b, pos, table.view(-1, 2, core.rope_dim // 2), k_pool, + step["slot_mapping_fp8"], step["slot_mapping_scale"], rows, + is_neox=False, compiled=core._ix_k[k_pool.is_contiguous()], validate=False, + ) # fmt: skip + if skip_gen: + return + qn = layer_norm(x, ix.q_norm_w, ix.q_norm_b, core.idx_eps) + q = torch.reshape(self._linear_fp8(qn, ix.wq_b, ix.wq_b_s), [rows, core.idx_heads, -1]) + q_fp8 = torch.empty_like(q, dtype=torch.float8_e4m3fn) + weights = torch.empty([rows, core.idx_heads], dtype=torch.float32, device=x.device) + if core.unfused_glue: + q = ssmax_scaling(q, pos, ix.ssmax_s, ix.ssmax_b) + q_fp8, weights = cute_dsl_indexer_q_side_fused( + q, pos, table, kw[:, core.idx_dim :].float().contiguous(), core.idx_weight_scale, + q_fp8, weights, is_neox=False, compiled=core._ix_q, + ) # fmt: skip + else: + # SSMax, rope, e4m3 and the head weights (read in place out of `kw`) in one launch. + q_fp8, weights = ssmax_indexer_q_side_fused( + q, pos, ix.ssmax_s, ix.ssmax_b, table, kw[:, core.idx_dim :], + core.idx_weight_scale, q_fp8, weights, + ) # fmt: skip + num_gens = step["num_seqs"] + next_n = rows // num_gens + kv_lens = step["kv_lens_cuda"][:num_gens] + logits = cute_dsl_fp8_paged_mqa_logits( + torch.reshape(q_fp8, [num_gens, next_n, core.idx_heads, core.idx_dim]), + k_pool, + weights, + kv_lens, + step["indexer_k_cache_block_offsets"][:num_gens], + step["scheduler_metadata_buffer"], + core._max_ctx_len, + ) + if next_n != 1 or core._ix_topk is None: + indexer_topk_decode( + logits, kv_lens, topk_buf, next_n, core.idx_topk, 1, step["radix_aux_indices"], + step["radix_aux_logits"], + ) # fmt: skip + else: + cute_dsl_radix_topk_decode(logits, kv_lens, topk_buf, next_n=1, compiled=core._ix_topk) + + def _mlp(self, o, L): + """This rank's share of the layer's MLP, a partial sum: the dense SwiGLU slice on the + dense layer; on a MoE layer every routed expert's width slice plus the shared expert's + slice.""" + core = self.core + act = flashinfer_silu_and_mul(self._linear_fp8(o, L.mlp_gu, L.mlp_gu_s)) + shared = self._linear_fp8(act, L.mlp_dn, L.mlp_dn_s) + m = L.moe + if m is None: + return shared + logits = cublas_mm(o, m.router, None, torch.float32) + ids, weights = renorm_moe_routing_op(logits, core.topk, torch.bfloat16) + # One NVFP4 quantization under the layer's static scale serves every expert's slice on + # every rank: the width slices' partials sum. + xq, xs = fp4_quantize(o, m.gscale, _NVFP4_SF_VEC, False, _NVFP4_SF_LINEAR) + routed = fp4_block_scale_moe_runner( + None, None, xq, xs.view(torch.float8_e4m3fn), m.fc1_w, m.fc1_s, None, None, None, + None, m.fc2_w, m.fc2_s, None, m.out1, m.out1_gate, m.out2, core.num_experts, + core.topk, None, None, core.moe_inter, core.expert_offset, core.local_experts, None, + _MOE_ROUTING_METHOD, True, _MOE_ACT_SWIGLU, topk_weights=weights, topk_ids=ids, + tune_max_num_tokens=core.max_num_tokens, + )[0] # fmt: skip + return torch.add(routed, shared) + + def forward( + self, + attn_metadata: AttentionMetadata, + input_ids: torch.IntTensor | None = None, + position_ids: torch.IntTensor | None = None, + inputs_embeds: torch.FloatTensor | None = None, + lora_params: dict | None = None, + **kwargs, + ) -> torch.Tensor: + core = self.core + self._check_step_contract(attn_metadata) + step = self.step_args(attn_metadata) + h = ( + nn.functional.embedding(input_ids, self._embed) + if inputs_embeds is None + else inputs_embeds + ) + rows = h.shape[0] + dev = h.device + assert rows == step["num_tokens"] and rows % step["num_seqs"] == 0, ( + rows, + step["num_tokens"], + step["num_seqs"], + ) + # A separate graph is captured for steps whose sequences all fit in index_topk, so + # this host branch is a per-capture constant. + skip_gen = step["skip_gen"] + pos = torch.reshape(position_ids, [-1]) + req_idx = step["req_idx_per_token"][:rows] + if skip_gen: + topk_buf = step["topk_indices_buffer"][:rows] + else: + topk_buf = torch.empty([rows, core.idx_topk], dtype=torch.int32, device=dev) + scratch = self._scratch(rows, h.dtype, dev) + + x = flashinfer_rmsnorm(h, self._norm0, core.eps) + residual = h + spec_metadata = kwargs.get("spec_metadata") + for layer_idx, L in enumerate(self._layers): + attn_out = self._attention(step, L, x, pos, req_idx, topk_buf, scratch, skip_gen) + partial = self._linear_fp8(attn_out, L.wo, L.wo_s) + o = cute_dsl_allreduce_add_rmsnorm( + partial, residual, L.ffn_norm, torch.empty_like(partial), core._lamport + ) + mlp = self._mlp(o, L) + x = cute_dsl_allreduce_add_rmsnorm( + mlp, residual, L.next_norm, torch.empty_like(mlp), core._lamport + ) + if spec_metadata is not None: + spec_metadata.maybe_capture_hidden_states(layer_idx, residual) + return x + + def _scratch(self, rows: int, dtype, dev) -> SimpleNamespace: + """Per-step scratch every layer rewrites in turn: the attention output, the absorbed + query at this rank's heads (every element written by each layer: the absorption BMM + and the SSMax op's rope tail or zeros), a NoPE owner's 576-wide latent rows (the rope + key zero for the step), and the latent attention output the native decode writes. + Under `gen_attn_flash_mla` the query is FlashMLA's 64 heads (this rank's first, the + rest zero for the step), the scratch carries the FlashMLA scheduler the step's first + layer fills and the rest share (captured with the step), and the latent output is kept + only for the unfused glue, which copies FlashMLA's live heads into it.""" + core = self.core + bf16 = dict(dtype=dtype, device=dev) + if core.gen_attn_flash_mla: + fused_q = torch.zeros([rows, _FLASH_MLA_HEADS, core.lat_dim], **bf16) + else: + fused_q = torch.empty([rows, core.heads, core.lat_dim], **bf16) + return SimpleNamespace( + attn_out=torch.empty([rows, core.heads * core.v_dim], **bf16), + fused_q=fused_q, + nope_lat=None if core.unfused_glue else torch.zeros([rows, core.lat_dim], **bf16), + lat_out=( + torch.empty([rows, core.heads, core.kv_lora], **bf16) + if core.unfused_glue or not core.gen_attn_flash_mla + else None + ), + sched=get_mla_metadata()[0] if core.gen_attn_flash_mla else None, + ) + + def _attention(self, step, L, x, pos, req_idx, topk_buf, s, skip_gen: bool): + """One layer's attention block into `s.attn_out` (returned): the query with SSMax, the + indexer's selection, the absorbed query, the owner's rope and fp8_ds_mla append, + sparse decode over the pool, and the V up-projection of the latent output (FlashMLA's + live heads read in place under `gen_attn_flash_mla` with the fused glue).""" + core = self.core + rows = x.shape[0] + q = self._linear_fp8(x, L.wq, L.wq_s) + q3 = torch.reshape(q, [rows, core.heads, core.qk_dim]) + fq = s.fused_q[:, : core.heads] + if core.unfused_glue: + q3 = ssmax_scaling(q3, pos, L.ssmax_s, L.ssmax_b) + if L.rope: + fq[..., core.kv_lora :].copy_(q3[..., L.nope :]) + else: + fq[..., core.kv_lora :].zero_() + else: + # SSMax in place; its rope half (a NoPE layer's zero) into the absorbed query's tail. + ssmax_scaling_fused_q( + q3, pos, L.ssmax_s, L.ssmax_b, q3, fq[..., core.kv_lora :], L.rope + ) + if L.indexer is not None: + self._indexer(step, x, pos, L, topk_buf, skip_gen) + slots = convert_req_index_to_global( + req_idx, step["block_table"], topk_buf, step["tokens_per_block"], core.idx_topk, + core._stride_factor, core._pool_row[L.kv_source], + ) # fmt: skip + bmm_out( + torch.transpose(q3[..., : L.nope], 0, 1), L.k_b, + torch.transpose(fq[..., : core.kv_lora], 0, 1), + ) # fmt: skip + if L.owner: + ds_mla_rope_append( + self._latent(x, L, s.nope_lat), fq, pos, core._rope.indexer, req_idx, + step["block_table"], step["latent_pool"], core._pool_row[L.index], L.rope, + ) # fmt: skip + if core.gen_attn_flash_mla and core.unfused_glue: + lat = flash_mla_sparse_decode( + s.fused_q, step["latent_kv"], slots, s.sched, core.softmax_scale, s.lat_out + ) + elif core.gen_attn_flash_mla: + lat = flash_mla_sparse_decode( + s.fused_q, step["latent_kv"], slots, s.sched, core.softmax_scale, None + )[:, : core.heads] + else: + # The rank's heads, unpadded, into the latent output. + cute_dsl_ds_mla_sparse_decode( + s.fused_q, step["latent_kv"], slots, core.softmax_scale, s.lat_out, + compiled=core._mla_decode, + ) # fmt: skip + lat = s.lat_out + bmm_out( + torch.transpose(lat, 0, 1), + L.v_b_t, + torch.transpose(torch.reshape(s.attn_out, [rows, core.heads, core.v_dim]), 0, 1), + ) + return s.attn_out + + def dspark_forward( + self, + draft, + noise_embedding, + query_positions, + num_ctx_per_req, + ctx_k_cache, + ctx_v_cache, + ctx_cache_batch_idx, + ctx_kv_cache=None, + ctx_page_table=None, + block_placement=None, + ): + # Only the worker's fused paged-latent preparation places the block + # (block_placement), and a drafter with embed_draft_block does not take + # it; this decode reads its context window through ctx_page_table. + del block_placement + draft._build_fused_kv_buffers() + draft._ensure_rope() + core = self.core + batch, block, hidden = noise_embedding.shape + rows = batch * block + # Gather a bounded window, not every page in the configured context. + # Invalid padding slots are masked even if their pages are allocated. + lengths = num_ctx_per_req[:batch].long() + offsets = torch.arange(draft.window, device=noise_embedding.device) + positions = (lengths - draft.window).clamp_min(0)[:, None] + offsets[None, :] + valid = positions < lengths[:, None] + slots = ctx_cache_batch_idx.long() + if ctx_page_table is not None: + page_size = ctx_kv_cache[0].shape[-2] + table = ctx_page_table.index_select(0, slots).long() + safe_positions = positions.clamp(max=table.shape[1] * page_size - 1) + pages = table.gather(1, safe_positions // page_size).clamp_min(0) + page_offsets = safe_positions % page_size + else: + safe_positions = positions.clamp(max=ctx_k_cache.shape[2] - 1) + key_valid = torch.cat( + (valid, torch.ones(batch, block, dtype=torch.bool, device=valid.device)), dim=1 + ) + query = torch.empty( + batch, + block, + core.heads, + core.lat_dim, + dtype=noise_embedding.dtype, + device=noise_embedding.device, + ) + values = torch.empty( + rows, core.heads, core.v_dim, dtype=noise_embedding.dtype, device=noise_embedding.device + ) + residual = noise_embedding.reshape(rows, hidden).clone() + normalized = torch.empty_like(residual) + x = flashinfer_rmsnorm(residual, draft._layers[0].attn_norm, core.eps) + for i, layer in enumerate(draft._layers): + q = cublas_mm(x, layer.q, None, None).view(batch, block, core.heads, core.qk_dim) + bmm_out( + q[..., : core.nope].reshape(rows, core.heads, core.nope).transpose(0, 1), + layer.k_b, + query[..., : core.kv_lora].view(rows, core.heads, core.kv_lora).transpose(0, 1), + ) + query[..., core.kv_lora :].copy_(draft._rotate(q[..., core.nope :], query_positions)) + kv = cublas_mm(x, layer.kv_a, None, None) + content = flashinfer_rmsnorm( + kv[:, : core.kv_lora].contiguous(), layer.kv_norm, core.eps + ) + block_kv = torch.cat( + ( + content.view(batch, block, core.kv_lora), + draft._rotate( + kv[:, core.kv_lora :].view(batch, block, core.rope_dim), query_positions + ), + ), + dim=-1, + ) + if ctx_page_table is not None: + context = ctx_kv_cache[i][:, 0, 0][pages, page_offsets] + else: + context = ctx_k_cache[slots[:, None], i, safe_positions, 0] + # Zero masked values as well as scores: 0 * an uninitialized + # NaN in the latent pool must not contaminate the weighted sum. + context = context.masked_fill(~valid.unsqueeze(-1), 0) + keys = torch.cat((context, block_kv), dim=1) + scores = torch.einsum("bshd,btd->bsht", query, keys).float() * core.softmax_scale + scores = scores.masked_fill(~key_valid[:, None, None, :], float("-inf")) + probabilities = torch.softmax(scores, dim=-1).to(keys.dtype) + latent = torch.einsum("bsht,btc->bshc", probabilities, keys[..., : core.kv_lora]) + bmm_out( + latent.reshape(rows, core.heads, core.kv_lora).transpose(0, 1), + layer.v_b.transpose(1, 2), + values.transpose(0, 1), + ) + partial = cublas_mm(values.reshape(rows, -1), layer.out, None, None) + o = cute_dsl_allreduce_add_rmsnorm( + partial, residual, layer.ffn_norm, normalized, core._lamport + ) + gate = cublas_mm(o, layer.gate, None, None) + up = cublas_mm(o, layer.up, None, None) + activated = flashinfer_silu_and_mul(torch.cat((gate, up), dim=-1)) + mlp = cublas_mm(activated, layer.down, None, None) + x = cute_dsl_allreduce_add_rmsnorm( + mlp, residual, layer.next_norm, normalized, core._lamport + ) + return x + + +class DraftModel(nn.Module): + """This specialization's BF16 DSpark backbone and recurrent head. + + Runtime owns accepted-token bookkeeping and the paged context pool. The + draft attends the last trained window of captured target features plus + its whole non-causal query block. All parameters live in the core table. + """ + + _uses_worker_attention_backend = False + _paged_ctx_cache = True + _kv_factor = 1 + _dspark_shift_label = True + has_markov_head = True + has_recurrent_markov_head = True + + def __init__(self, shell, draft_config): + super().__init__() + # Keep references without registering the parent as a child module. + object.__setattr__(self, "_shell", shell) + self.model_config = draft_config + self.config = draft_config.pretrained_config + core = shell.model + spec = shell.spec_config + native = spec._draft_model_config + self.block_size = spec.block_size + self.mask_token_id = spec.mask_token_id + self.target_layer_ids = tuple(spec.target_layer_ids) + self.window = int(native["ragged_attention"].split(",")[0]) + self._num_attn_layers = int(native["n_layers"]) + self._num_heads = core.heads + self._num_kv_heads = 1 + self._head_dim = core.lat_dim + self.dflash_attention_backend = "VANILLA" + self._runtime_position_ceiling = None + self._rope = None + self._layers = None + + @property + def core(self): + return self._shell.model + + @property + def lm_head(self): + return self._shell.lm_head + + @property + def markov_w2(self): + return self.core.w["draft_markov_output"] + + def logits_processor(self, hidden, lm_head, attn_metadata, gather): + logits = self._shell.logits_processor(hidden, lm_head, attn_metadata, gather) + # The shared-head logits are soft-capped before the RNN bias is added. + return tanh_softcap(logits, self._shell.softcap) + + def apply_draft_logit_transform(self, logits): + """Return ``logits`` unchanged: :meth:`logits_processor` already soft-caps them. + + The worker calls this on the block logits after ``logits_processor``. + This drafter declares no ``logit_multiplier`` or ``logit_softcap``, so the + cap is applied once, in ``logits_processor``. + """ + return logits + + def _build_fused_kv_buffers(self): + # The worker calls this before sizing its cache. Take views once after + # materialization; context projection is explicit per layer here. + if self._layers is not None: + return + w, core = self.core.w, self.core + self._layers = [] + for i in range(self._num_attn_layers): + prefix = f"l{i}_draft_" + kv_b = w[prefix + "wkv_b"].view(core.heads, core.nope + core.v_dim, core.kv_lora) + self._layers.append( + SimpleNamespace( + attn_norm=w[prefix + "attn_norm"], + q=w[prefix + "wq"].t(), + kv_a=w[prefix + "wkv_a"].t(), + kv_norm=w[prefix + "kv_a_norm"], + k_b=kv_b[:, : core.nope].contiguous(), + v_b=kv_b[:, core.nope :].contiguous(), + out=w[prefix + "wo"].t(), + ffn_norm=w[prefix + "ffn_norm"], + gate=w[prefix + "w1"].t(), + up=w[prefix + "w3"].t(), + down=w[prefix + "w2"].t(), + next_norm=( + w[f"l{i + 1}_draft_attn_norm"] + if i + 1 < self._num_attn_layers + else w["draft_final_norm"] + ), + ) + ) + + def _ensure_rope(self): + cap = self._runtime_position_ceiling or self.core.rope_rows + if self._rope is None or self._rope.rows != cap: + w = self.core.w["draft_final_norm"] + # alpha=0 removes LeRoPE; all draft layers use plain YaRN. + self._rope = _rope_tables( + self.core.model_config.pretrained_config, + torch.zeros(self.core.rope_dim // 2), + cap, + w.dtype, + w.device, + ) + + def _rotate(self, x, positions): + table = self._rope.indexer[positions.long()] + cos, sin = table.chunk(2, dim=-1) + # One head axis for Q, none for the shared latent K. + if x.ndim == positions.ndim + 2: + cos, sin = cos.unsqueeze(-2), sin.unsqueeze(-2) + even, odd = x.float()[..., 0::2], x.float()[..., 1::2] + return ( + torch.stack((even * cos - odd * sin, even * sin + odd * cos), -1) + .flatten(-2) + .to(x.dtype) + ) + + def embed_draft_block(self, anchors, block_size): + # The route pins the trained block width; no mask token is looked up + # in the vocabulary (its sentinel is deliberately out of range). + weight = self.core.w["embed"] + valid = (anchors >= 0) & (anchors < weight.shape[0]) + embedded = nn.functional.embedding(anchors.clamp(0, weight.shape[0] - 1), weight) + embedded = embedded.masked_fill(~valid.unsqueeze(-1), 0) + masks = self.core.w["draft_mask"].unsqueeze(0).expand(anchors.shape[0], -1, -1) + return torch.cat((embedded.unsqueeze(1), masks), dim=1) + + def project_target_hidden(self, hidden): + w = self.core.w + projected = cublas_mm(hidden, w["draft_target_projection"].t(), None, None) + return flashinfer_rmsnorm(projected, w["draft_target_norm"], self.core.eps) + + def precompute_context_kv(self, projected_hidden, positions): + self._build_fused_kv_buffers() + self._ensure_rope() + core = self.core + latent = [] + for layer in self._layers: + kv = cublas_mm(projected_hidden, layer.kv_a, None, None) + content = flashinfer_rmsnorm( + kv[:, : core.kv_lora].contiguous(), layer.kv_norm, core.eps + ) + rope = self._rotate(kv[:, core.kv_lora :], positions.reshape(-1)) + latent.append(torch.cat((content, rope), dim=-1)) + return torch.stack(latent, dim=1).unsqueeze(2), None + + def dflash_forward_with_metadata(self, *, attn_metadata, **kwargs): + # The phase belongs to this engine step, including mixed batches. + return self.core._select_target(attn_metadata).dspark_forward(self, **kwargs) + + def apply_markov_chain_logits(self, base_logits, first_prev_tokens, *, hidden_states): + """The recurrent Markov bias over full-vocab ``[B, K, vocab]`` block logits. + + Each position's greedy token (the argmax of its biased logits) is the + next position's previous token; ``hidden_states`` is the ``[B, K, + hidden]`` block output the logits were projected from. + """ + w = self.core.w + output = w["draft_markov_output"] + previous = first_prev_tokens.long() + state = None + refined = [] + for i in range(base_logits.shape[1]): + weight = w["draft_markov_embedding"] + valid = (previous >= 0) & (previous < weight.shape[0]) + embedding = nn.functional.embedding(previous.clamp(0, weight.shape[0] - 1), weight) + embedding = embedding.masked_fill(~valid.unsqueeze(-1), 0) + if state is None: + state = torch.zeros_like(embedding) + joint = cublas_mm( + torch.cat((state, embedding, hidden_states[:, i]), dim=-1), + w["draft_joint"].t(), + w["draft_joint_bias"], + None, + ) + gate, candidate, emitted = joint.chunk(3, dim=-1) + gate = torch.sigmoid(gate) + state = gate * state + (1 - gate) * torch.tanh(candidate) + bias = cublas_mm(torch.tanh(emitted), output.t(), None, None) + logits = base_logits[:, i] + bias + refined.append(logits) + previous = logits.argmax(dim=-1) + return torch.stack(refined, dim=1) + + +class _VisionBlock(NamedTuple): + """One vision transformer block's weights, as the views the ops read: the GEMM + operands column-major (`.t()` of the stored rows).""" + + attn_norm: torch.Tensor + qkv: torch.Tensor + wo: torch.Tensor + ffn_norm: torch.Tensor + w13: torch.Tensor + w2: torch.Tensor + + +def _vision_step(seg_lens: list[int], device, workspace: torch.Tensor) -> dict: + """The batch state of one vision attention call: one context sequence per image, + no KV cache. Every value is host-known from the images' patch counts.""" + total = sum(seg_lens) + longest = max(seg_lens) + lens_dev = torch.tensor(seg_lens, dtype=torch.int32, device=device) + lens_host = torch.tensor(seg_lens, dtype=torch.int32) + return dict( + workspace=workspace, + sequence_length=lens_dev, + host_past_key_value_lengths=lens_host, + host_total_kv_lens=torch.tensor([total, 0], dtype=torch.int32), + context_lengths=lens_dev, + host_context_lengths=lens_host, + host_request_types=torch.zeros(len(seg_lens), dtype=torch.int32), + max_num_requests=len(seg_lens), + max_context_length=longest, + max_seq_len=longest, + attention_window_size=longest, + num_contexts=len(seg_lens), + num_ctx_tokens=total, + ) + + +def _vision_attention( + x: torch.Tensor, + B: _VisionBlock, + positions: torch.Tensor, + step: dict, + heads: int, + head_dim: int, + theta: float, +) -> torch.Tensor: + """One vision block's attention over the patch rows of every image of the call: + `[wq ; wk ; wv]` as one GEMM, q and k rotated by each patch's (row, col) in its + own image, attention bidirectional within each image, then `wo`.""" + rows = x.shape[0] + width = heads * head_dim + qkv = cublas_mm(x, B.qkv) + q, k = pixtral_rope_2d( + torch.reshape(qkv[:, :width], [rows, heads, head_dim]), + torch.reshape(qkv[:, width : 2 * width], [rows, heads, head_dim]), + positions, + theta, + ) + qkv = torch.cat( + [torch.reshape(q, [rows, width]), torch.reshape(k, [rows, width]), qkv[:, 2 * width :]], -1 + ) + out = torch.empty([rows, width], dtype=x.dtype, device=x.device) + out = thop_attention( + q=qkv, + output=out, + num_heads=heads, + num_kv_heads=heads, + head_size=head_dim, + **step, + **_VISION_ATTN, + ) + return cublas_mm(out, B.wo) + + +def _merge_tiles(x: torch.Tensor, grid_h: int, grid_w: int, merge: int) -> torch.Tensor: + """One image's `[grid_h * grid_w, d]` patch rows -> `[grid_h * grid_w / merge^2, + d * merge^2]`: each merge x merge tile of the grid as one row, tiles row-major (the + order of the image's [IMG] tokens), each row laid out (channel, tile row, tile + column) as the merging layer's input columns are.""" + d = x.shape[-1] + grid = torch.permute(torch.reshape(x, [grid_h, grid_w, d]), [2, 0, 1]) + tiles = nn.functional.unfold(torch.unsqueeze(grid, 0), kernel_size=merge, stride=merge) + return torch.t(tiles[0]) + + +class VisionTower: + """The image input path: a request's pixels -> one row per [IMG] token, in the + language model's hidden width. Run by the shell, in context steps only, never + captured; every rank runs the whole tower (all heads) over its own requests. + + Per image: the 14x14 patch embedding as an unfold and one GEMM, `ln_pre`, the + transformer blocks (RMSNorm, attention with 2-D RoPE, RMSNorm, SwiGLU, each with + its residual), the pre-projector RMSNorm, the 2x2 patch merger and the adapter + (`w_in`, exact GELU, `w_out`). The images of one call are concatenated along the + rows; only the attention needs their boundaries.""" + + def __init__(self, core) -> None: + """Take the tower's weight views and geometry, once, off the real weights.""" + d = _weights.MODEL_WEIGHTS.dims(core) + w = core.w + self.hidden = d.v_hidden + self.heads = d.v_heads + self.head_dim = d.v_head_dim + self.patch = d.v_patch + self.merge = d.v_merge + self.theta = d.v_theta + self.dtype = d.dtype + self.out_hidden = d.hidden + self._validate = step_contract_enabled() + self._rank = core.model_config.mapping.rank + self.patch_w = w["vis_patch"].t() + self.ln_pre = w["vis_ln_pre"] + self.blocks = [ + _VisionBlock( + attn_norm=w[f"v{j}_attn_norm"], + qkv=w[f"v{j}_qkv"].t(), + wo=w[f"v{j}_wo"].t(), + ffn_norm=w[f"v{j}_ffn_norm"], + w13=w[f"v{j}_w13"].t(), + w2=w[f"v{j}_w2"].t(), + ) + for j in range(d.v_layers) + ] + self.proj_norm = w["vis_proj_norm"] + self.merge_w = w["vis_merge"].t() + self.w_in = w["vis_w_in"].t() + self.w_out = w["vis_w_out"].t() + # The prompt token the tower's rows replace, on the host: the executor finds a + # step's image positions with it before anything reaches the device. + self.image_token_ids = torch.tensor([d.image_token_id], dtype=torch.int32) + # The attention op sizes its workspace in place per call and never shrinks it. + self._workspace = torch.empty(0, dtype=torch.int8, device=self.ln_pre.device) + + def _images(self, multimodal_params: list[MultimodalParams]) -> list[tuple]: + """`(pixels [C, H, W], grid rows, grid cols)` of every image, in order. The + mistral-native processor pads the batch and preserves each image's true size.""" + images = [] + unit = self.patch * self.merge + for param in multimodal_params: + image = (param.multimodal_data or {}).get("image") + if image is None: + continue + pixels = image["pixel_values"] + sizes = image.get("image_sizes") + if pixels.ndim != 4 or pixels.shape[1] != 3: + raise ValueError("Image pixels must have shape [N, 3, H, W]") + if sizes is None or len(sizes) != pixels.shape[0]: + raise ValueError("Image sizes must describe every image in the padded batch") + for b in range(pixels.shape[0]): + if len(sizes[b]) != 2: + raise ValueError("Each image size must be [height, width]") + height, width = int(sizes[b][0]), int(sizes[b][1]) + if not (0 < height <= pixels.shape[-2] and 0 < width <= pixels.shape[-1]): + raise ValueError("Image sizes must fit within their padded pixels") + if height % unit or width % unit: + raise ValueError( + f"image {height}x{width} is not a whole number of {unit}-pixel " + "merge tiles; the processor resizes to them" + ) + images.append( + (pixels[b, :, :height, :width], height // self.patch, width // self.patch) + ) + return images + + def forward(self, multimodal_params: list[MultimodalParams]) -> torch.Tensor: + """The rows of every image of `multimodal_params`, request by request, image by + image: `[sum of merged tiles, text hidden]`.""" + images = self._images(multimodal_params) + dev = self.ln_pre.device + if not images: + return torch.empty([0, self.out_hidden], dtype=self.dtype, device=dev) + seg = [gh * gw for _, gh, gw in images] + if self._validate: + logger.info( + "modeling_v2 image forward target=%s rank=%d grids=%s merged_rows=%d", + __name__.rsplit(".", 1)[0], + self._rank, + [(gh, gw) for _, gh, gw in images], + sum(seg) // self.merge**2, + ) + # Each patch as one row, (channel, row, col) within it, patches row-major. + patches = torch.cat( + [ + torch.t( + nn.functional.unfold( + torch.unsqueeze(px.to(device=dev, dtype=self.dtype), 0), + kernel_size=self.patch, + stride=self.patch, + )[0] + ) + for px, _, _ in images + ], + 0, + ) + positions = torch.cat( + [ + torch.reshape( + torch.stack( + torch.meshgrid( + torch.arange(gh, device=dev), + torch.arange(gw, device=dev), + indexing="ij", + ), + -1, + ), + [-1, 2], + ) + for _, gh, gw in images + ], + 0, + ) + step = _vision_step(seg, dev, self._workspace) + + h = flashinfer_rmsnorm(cublas_mm(patches, self.patch_w), self.ln_pre, _VISION_EPS) + x = flashinfer_rmsnorm(h, self.blocks[0].attn_norm, _VISION_EPS) + residual = h + for j, B in enumerate(self.blocks): + attn = _vision_attention(x, B, positions, step, self.heads, self.head_dim, self.theta) + x, residual = flashinfer_fused_add_rmsnorm(attn, residual, B.ffn_norm, _VISION_EPS) + mlp = cublas_mm(flashinfer_silu_and_mul(cublas_mm(x, B.w13)), B.w2) + norm_next = self.blocks[j + 1].attn_norm if j + 1 < len(self.blocks) else self.proj_norm + x, residual = flashinfer_fused_add_rmsnorm(mlp, residual, norm_next, _VISION_EPS) + + # x is the pre-projector-normed tower output. + tiles = torch.cat( + [ + _merge_tiles(xi, gh, gw, self.merge) + for xi, (_, gh, gw) in zip(torch.split(x, seg, 0), images) + ], + 0, + ) + merged = cublas_mm(tiles, self.merge_w) + return cublas_mm(gelu_erf(cublas_mm(merged, self.w_in)), self.w_out) + + +def _image_rows( + vision: Optional[VisionTower], multimodal_params: list[MultimodalParams] +) -> Optional[torch.Tensor]: + """This step's image rows, one per [IMG] position its context chunks cover, in + the order those positions appear; None when the step covers none. + + The tower runs once per request, on the first step that needs it; its rows stay + on the request's multimodal data for the later chunks. A chunk -- under chunked + prefill, or resuming after a reused prefix -- takes rows `[cached, cached + in + chunk)` of its request's, counted by the request's [IMG] cumsum.""" + if vision is None: + if any((p.multimodal_data or {}).get("image") is not None for p in multimodal_params): + raise ValueError( + "an image reached a Mistral Large 4 target whose checkpoint has no vision encoder" + ) + return None + # Filter before encoding and gathering so inactive requests neither run the + # tower nor shift the embedding offsets of active requests in a mixed batch. + multimodal_params = [ + param + for param in multimodal_params + if param.multimodal_runtime is None or param.multimodal_runtime.num_mm_tokens_in_chunk > 0 + ] + embeds = find_input_mm_embeds( + get_multimodal_embeddings(vision.forward, multimodal_params), multimodal_params + ) + if not embeds: + return None + rows = embeds[0] if len(embeds) == 1 else torch.cat(embeds, 0) + return rows if rows.shape[0] else None + + +def _splice_image_rows( + embed: torch.Tensor, + input_ids: torch.Tensor, + rows: torch.Tensor, + mm_token_indices: Optional[torch.Tensor], +) -> torch.Tensor: + """The step's input embeddings with `rows` written over its [IMG] positions, + `mm_token_indices` (which the executor computes on the host from the shell's + `multimodal_token_ids`); every other position keeps its token embedding.""" + if mm_token_indices is None: + raise ValueError("an image step arrived without the executor's mm_token_indices") + if mm_token_indices.shape[0] != rows.shape[0]: + raise ValueError( + f"{mm_token_indices.shape[0]} [IMG] positions in this step but " + f"{rows.shape[0]} image rows for them" + ) + h = nn.functional.embedding(input_ids, embed) + return h.index_copy_(0, mm_token_indices, rows.to(h.dtype)) + + +class ModelingV2MistralLarge4Nvfp4Sm103Tp( + DecoderModelForCausalLM[MistralLarge4TpModelingV2, PretrainedConfig] +): + def __init__(self, model_config: ModelConfig): + cfg = model_config.pretrained_config + super().__init__( + MistralLarge4TpModelingV2(model_config), + config=model_config, + hidden_size=cfg.hidden_size, + vocab_size=cfg.vocab_size, + ) + self.softcap = float(cfg.softmax_tanh_gating) + self._vision: Optional[VisionTower] = None + vis = getattr(cfg, "vision_config", None) + self._image_token_ids = ( + torch.tensor([vis.image_token_id], dtype=torch.int32) if vis is not None else None + ) + + self.spec_config = model_config.spec_config + self.draft_model = None + self.draft_config = None + self.spec_worker = None + if self.spec_config is not None: + # This is allocation geometry for the independently paged BF16 + # draft, not a second registry-resolved model or target cache. + draft_cfg = PretrainedConfig( + num_hidden_layers=int(self.spec_config._draft_model_config["n_layers"]), + hidden_size=cfg.hidden_size, + num_attention_heads=cfg.num_attention_heads, + num_key_value_heads=1, + kv_lora_rank=cfg.kv_lora_rank, + qk_rope_head_dim=cfg.qk_rope_head_dim, + qk_nope_head_dim=cfg.qk_nope_head_dim, + v_head_dim=cfg.v_head_dim, + intermediate_size=cfg.intermediate_size, + vocab_size=cfg.vocab_size, + max_position_embeddings=cfg.max_position_embeddings, + torch_dtype=torch.bfloat16, + ) + self.draft_config = replace( + model_config, + pretrained_config=draft_cfg, + quant_config=QuantConfig(), + quant_config_dict=None, + sparse_attention_config=None, + spec_config=None, + modeling_v2_target=None, + ) + self.draft_model = DraftModel(self, self.draft_config) + self.spec_worker = get_spec_worker( + self.spec_config, + model_config, + model_config.mapping, + use_separate_draft_kv_cache=True, + ) + self.spec_worker.set_draft_model(self.draft_model) + self.epilogue.append(self.spec_worker) + + def set_guided_decoder(self, guided_decoder: CapturableGuidedDecoder) -> bool: + if self.spec_worker is None: + return False + return self.spec_worker.set_guided_decoder(guided_decoder) + + @property + def multimodal_token_ids(self) -> Optional[torch.Tensor]: + return self._image_token_ids + + def forward( + self, + attn_metadata: AttentionMetadata, + input_ids: torch.IntTensor | None = None, + position_ids: torch.IntTensor | None = None, + inputs_embeds: torch.FloatTensor | None = None, + return_context_logits: bool = False, + spec_metadata=None, + lora_params: dict | None = None, + **kwargs, + ) -> torch.Tensor: + """Soft-cap target logits before verification and return the worker result.""" + worker_input_ids = input_ids + multimodal_params = kwargs.get("multimodal_params") or [] + if multimodal_params: + rows = _image_rows(self._vision, multimodal_params) + if rows is not None: + inputs_embeds = _splice_image_rows( + self.model.w["embed"], input_ids, rows, kwargs.get("mm_token_indices") + ) + input_ids = None + if self.spec_worker is None: + logits = super().forward( + attn_metadata, + cast(torch.IntTensor, input_ids), + position_ids, + inputs_embeds, + return_context_logits, + spec_metadata, + lora_params, + **kwargs, + ) + return tanh_softcap(logits, self.softcap) + hidden = self.model( + attn_metadata=attn_metadata, + input_ids=input_ids, + position_ids=position_ids, + inputs_embeds=inputs_embeds, + spec_metadata=spec_metadata, + lora_params=lora_params, + **kwargs, + ) + logits = self.logits_processor( + nn.functional.embedding(spec_metadata.gather_ids, hidden), + self.lm_head, + attn_metadata, + True, + ) + return self.spec_worker( + input_ids=worker_input_ids, + position_ids=position_ids, + hidden_states=hidden, + logits=tanh_softcap(logits, self.softcap), + attn_metadata=attn_metadata, + spec_metadata=spec_metadata, + draft_model=self.draft_model, + resource_manager=kwargs.get("resource_manager"), + ) + + def load_weights(self, weights, *args, **kwargs): + _weights.MODEL_WEIGHTS.load(self, weights) + + def load_checkpoint(self, checkpoint_dir: str) -> None: + _weights.MODEL_WEIGHTS.load_checkpoint(self, checkpoint_dir) + + def post_load_weights(self): + super().post_load_weights() + self.model.post_load_weights() + if step_contract_enabled(): + logger.info( + "modeling_v2 loaded target=%s class=%s rank=%d world=%d vision=%s", + __name__.rsplit(".", 1)[0], + type(self).__name__, + self.model_config.mapping.rank, + self.model_config.mapping.world_size, + getattr(self.config, "vision_config", None) is not None, + ) + if getattr(self.config, "vision_config", None) is not None: + self._vision = VisionTower(self.model) + if self.draft_model is not None: + self.draft_model._build_fused_kv_buffers() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/weights.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/weights.py new file mode 100644 index 000000000000..8e4bb04df89c --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/weights.py @@ -0,0 +1,1377 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Weight table and loader: mistral-large-4-nvfp4 / sm_103 / tp, the language model. + +The checkpoint is Mistral-native (`consolidated-*.safetensors`, `layers.N.attention.wq`, ...) +and ModelOpt `MIXED_PRECISION`: the routed experts NVFP4, the other big linears FP8 128x128 +block (`FP8_PB_WO`), everything else bf16. Every rank is handed the whole of it, so every split +is this table's, every width read off the mapping (`tp_size` and `moe_tp_size` 4 on one node, or +8 over two nodes of four, and `moe_ep_size` 1): + +* **tensor-parallel slices** -- `wq` by output rows and `wo` by input columns (this rank's 16 + heads at 4 ranks, 8 heads -- 1024 rows or columns -- at 8); `wkv_b` by heads; the MLA SSMax + `s`/`b` by heads; the dense layer's and the shared expert's `w1`/`w3` by rows and `w2` by + columns (6144 and 1536 per rank at 4 ranks, 3072 and 768 at 8); the shell's `lm_head` by + vocabulary rows (51200 or 25600 per rank). Every slice is whole 128-blocks, and a block scale + is sliced on the same boundary. The slice is taken on the checkpoint entry before the copy to + the device, so a rank moves only its own bytes. That is the `shard` column of `W`. +* **routed-expert width slices** -- every rank loads all 128 routed experts (the window + `[expert_offset, expert_offset + local_experts)` that `moe_ep_size` 1 makes `[0, 128)`), each + cut along its intermediate width to `moe_inter = moe_intermediate_size / moe_tp_size` (1536 at + 4 ranks, 768 at 8) at `moe_tp_rank * moe_inter`: `w1` and `w3`, codes and block scales alike, + by output rows `[Ir, Ir + I)`; `w2`'s codes by input bytes `[Ir/2, Ir/2 + I/2)` (two e2m1 per + byte: 768 or 384 bytes) and its block scales by columns `[Ir/16, Ir/16 + I/16)` (one per 16 + inputs, 96 or 48 columns, so the cut splits no block). These are `shard`s too: a row read, and + for `w2` a strided column read, of the checkpoint entry. +* **replicated** -- `wkv_a_with_mqa` and `kv_a_norm`, the whole indexer (all 32 of its heads), + the router, the embedding, every norm, `lerope.alpha`, and the routed experts' per-tensor + scalars. + +Layer sets come from the config: `wkv_a_with_mqa` and `kv_a_norm` exist on the KV-owning layers +only (`kv_source_layers[i] == i`; the sharers read their source's latent), the indexer on the +`index_sharing == 'full'` layers only, the dense MLP on the first `first_k_dense_replace` +layers and the router, shared expert and routed experts on the rest. + +Load-time layouts, each the one its consumer reads: + +* `wkv_b` regrouped per head `[k_nope | v]` -> `[all k_nope ; all v]` over this rank's heads + (k_nope is 64 wide on RoPE layers and 128 on NoPE layers); +* the dense MLP and the shared expert `[w1 ; w3]`, gate rows first, the half order + `flashinfer_silu_and_mul` reads; +* FP8 linears (`wq`, `wo`, the indexer's `wq_b`, the dense and shared `w1`/`w3`/`w2`): e4m3 + `[N, K]` as stored, and the fp32 block scales -- stored as `.weight_scale` + `[N/128, 1, K/128, 1]`, squeezed here to the `[N/128, K/128]` grid -- in DeepGEMM's packed + UE8M0 operand, `transform_sf_into_required_layout(recipe=(1, 128, 128))`: what + `FP8BlockScalesLinearMethod.transform_weights` installs as `weight_scale` on sm_100f. That + operand is an MN-major `[N, ceil(K/512)]` int32 view; the parameter holds the dense + `[ceil(K/512), N]` behind it and the GEMM takes `.t()`. Where K/128 is not a multiple of four + the last word of each row is partly padding: the shared expert's `w2` at K 768 (8 ranks) has 6 + blocks in 2 words, and bytes 2 and 3 of the second word are zero. The method's + `resmooth_to_fp8_e8m0` is not run: every scale is asserted an exact power of two, which makes + it a power-of-two re-exponent of the same values; +* the indexer's `wk` and `weights_proj` stacked into one `[128 + 32, hidden]` bf16 GEMM operand; + its `k_norm` and `lerope.alpha` widened to fp32. + +The routed experts are this rank's width slice of every expert, stacked on an expert axis, in +the NVFP4 MoE runner's layout: e2m1 codes, two per byte, `[E, 2I, H/2]` / `[E, H, I/2]` uint8 +and their e4m3 block scales, one per 16 inputs, `[E, 2I, H/16]` / `[E, H, I/16]`, with `E` 128 +and `I` this rank's 1536 (4 ranks) or 768 (8 ranks). Each sliced source lands as stored -- FC1's +rows `[w3 ; w1]`, up rows first, the order the runner's FC1 concatenates them in -- and once +every source of an expert stack has landed, its row's `layout` rearranges each expert's slice in +place (the runner reads nothing else, and a wrong layout costs 300-3400 bf16 ulp and raises +nothing): + +* FC1, codes and block scales alike: the halves interleaved (row `2i` up row `i`, row `2i + 1` + gate row `i`), then the 32-row block shuffle (within each 32 rows, row `4u + v` moves to + `8v + u`); +* FC2: the 32-row block shuffle; +* both block scales: then the 128x4 swizzle (`torch.ops.trtllm.block_scale_interleave`'s order, + written here in torch so the table also runs on meta tensors). + +The layout is a function of the slice's shape, so it runs on the slice and never on a whole +expert: the swizzle's byte order depends on the scale column count (FC2's 96 or 48 here, not +the stored 384), and the interleave pairs this rank's up and gate rows. At either per-rank width +every rule is met without padding: FC1's 2 x 1536 = 3072 or 2 x 768 = 1536 rows and FC2's 6144 +are multiples of the 32-row shuffle block and of the swizzle's 128 rows, and FC1's 384 and FC2's +96 or 48 scale columns of its 4. + +The runner's output scalars are not derived here: the four per-tensor scalars are handed over +as stored, unsliced -- `isc1` and `isc2`, the one FC1 and FC2 activation `input_scale` of the +layer, and per local expert `ws2_13` (`w1`'s `weight_scale_2`, which `w3` shares) and `ws2_2`. +ModelOpt stores these as reciprocals of the global scales a quantizer divides by. A per-tensor +scale holds for every slice of its tensor, so each rank's slice dequantizes with the whole +expert's scalars. + +Before any of that is read, `load` reads the per-tensor scalars of all the layer's experts and +asserts what the forward relies on: `w1` and `w3` store the same `input_scale` and +`weight_scale_2` on every expert, so one FC1 alpha serves the stacked `[w3 ; w1]`; and each +projection stores one `input_scale` for every expert, so one NVFP4 quantization of the +replicated activations is right for every expert on every rank, and the ranks' partial +outputs, each over its slice of every expert, sum. + +The BF16 vision encoder, pre-projector norm, patch merger and language adapter are +replicated on every rank when the configuration declares a vision encoder. + +`expected_unconsumed` accounts for off-rank routed experts and disabled DSpark draft weights. +Off-rank expert scalars are still read to check shared scales; their codes and block scales +are never read. +At `moe_ep_size` 1 every expert belongs to the rank and is read, sliced. +""" + +from __future__ import annotations + +import collections +import functools +import json +import os +from dataclasses import dataclass +from pathlib import Path +from types import SimpleNamespace +from typing import Any, Callable + +import torch +import torch.nn as nn +from safetensors import safe_open + +from tensorrt_llm.logger import logger +from tensorrt_llm.quantization.utils.fp8_utils import transform_sf_into_required_layout + +F8, F32, I32, U8 = torch.float8_e4m3fn, torch.float32, torch.int32, torch.uint8 + +#: The FP8 block edge on both weight axes (the checkpoint's `FP8_PB_WO` 128x128 blocks), and +#: the K granularity of the DeepGEMM scale recipe `(1, 128, 128)`. +BLOCK = 128 +#: UE8M0 bytes per int32 word of the DeepGEMM scale operand: four K blocks. +SF_PACK = 4 +#: NVFP4: one e4m3 block scale per 16 inputs (the checkpoint's `group_size`), two e2m1 codes +#: per byte. +NVFP4_BLOCK = 16 +E2M1_PER_BYTE = 2 + +#: A routed expert's three projections: `w1` gate, `w3` up, `w2` down. +PROJECTIONS = ("w1", "w2", "w3") +#: The per-tensor NVFP4 scalars each of them carries beside its codes and block scales. +EXPERT_SCALARS = ("weight_scale_2", "input_scale") + +#: The vision tower's checkpoint prefixes. +VISION_PREFIXES = ( + "vision_encoder.", + "patch_merger.", + "pre_mm_projector_norm.", + "vision_language_adapter.", +) + + +#: `layers=ALL_LAYERS` is the common case and the default. A sentinel rather +#: than a lambda so the table reads as data, and rather than `True` so the +#: field has one type of answer: which layers. +ALL_LAYERS = object() + + +@dataclass(frozen=True) +class W: + """One weight role: what it is, where its bytes come from, what happens on the way. + + `shape` takes the dimension bundle and returns the declared shape. + + `src` says where the bytes come from: a checkpoint key template with `{p}` + for the layer prefix (`layers.N`), or a callable `(d, i) -> [(key, index), ...]` + where `index` is where in the parameter that tensor lands (`None`: all of it). + A `key` may be a tuple of per-tensor scalars, which `transform` is handed together. + + `layers` says which layers carry this weight: `ALL_LAYERS`, `None` for a + weight that is not per-layer, or a callable `d -> iterable[int]`. Two rows + may share a name over disjoint layer sets, with different shapes. + + `shard` takes the dimension bundle and returns a function of the checkpoint + entry: this rank's tensor-parallel slice, applied to every source of the row. + It runs before the copy to the device and before `transform`, which is what + `wkv_b` relies on: the regroup reads this rank's head count off the slice. + + `transform` runs on the (sliced) source tensor, on the destination's device. + + `layout` runs once every source of the parameter has landed, on each slot of its leading + (expert) axis, and the slot is overwritten with the result: a rearrangement that mixes rows + several sources fill, which no single source's `transform` can make. + """ + + name: str + shape: Callable[[Any], tuple[int, ...]] + src: str | Callable + dtype: torch.dtype | None = None + transform: Callable | None = None + layers: object | None = ALL_LAYERS + shard: Callable | None = None + layout: Callable | None = None + when: Callable | None = None + tag: str = "l" + + def layer_indices(self, d: SimpleNamespace) -> list[int] | None: + """The layers this weight exists on, or None if it is not per-layer.""" + if self.layers is None: + return None + if self.layers is ALL_LAYERS: + return list(range(d.num_layers)) + return list(self.layers(d)) + + def sources(self, d: SimpleNamespace, i: int | None, prefix: str) -> list[tuple[Any, Any]]: + """`[(checkpoint key | key tuple, index into the parameter | None), ...]`.""" + if callable(self.src): + return list(self.src(d, i)) + return [(self.src.format(p=prefix), None)] + + +def _keys(ckpt) -> tuple[str, ...]: + """The checkpoint keys of one source: itself, or the members of a key tuple.""" + return ckpt if isinstance(ckpt, tuple) else (ckpt,) + + +def _rows(start, width): + return lambda d: (lambda e: e[start(d) : start(d) + width(d)]) + + +def _cols(start, width): + return lambda d: (lambda e: e[:, start(d) : start(d) + width(d)]) + + +def _sf_cols(start, width): + """`_cols` on an FP8 block scale as stored, `[N/128, 1, K/128, 1]`: its K blocks are axis 2.""" + return lambda d: (lambda e: e[:, :, start(d) : start(d) + width(d)]) + + +def _sf_words(k: int) -> int: + """int32 words per row of the DeepGEMM scale operand of a K-wide weight.""" + return -(-k // (BLOCK * SF_PACK)) + + +def _assert_ue8m0(s: torch.Tensor) -> None: + """Every block scale is a positive normal power of two, so its UE8M0 byte (the fp32 + exponent field) holds it exactly. A meta tensor carries no values to check.""" + if s.is_meta: + return + bits = s.view(torch.int32) + exp = (bits >> 23) & 0xFF + exact = ((bits & 0x807FFFFF) == 0) & (exp > 0) & (exp < 255) + assert bool(exact.all()), ( + f"weight_scale: {int((~exact).sum())} of {s.numel()} block scales are not a positive " + "normal power of two (UE8M0); the scale layouts below would change their values" + ) + + +def _deepgemm_sf(s: torch.Tensor, core) -> torch.Tensor: + """fp32 `[N/128, K/128]` block scales -> the dense `[ceil(K/512), N]` int32 storage of + DeepGEMM's MN-major packed UE8M0 operand, whose `.t()` is that operand. N is a multiple of + 128 for every role here, so the operand's row stride is N and its storage has no MN padding. + A K of a block count that is not a multiple of four (the shared expert's `w2` at K 768 on 8 + ranks: 6 blocks) leaves the high bytes of each row's last word zero.""" + _assert_ue8m0(s) + n_blk, k_blk = s.shape + sf = transform_sf_into_required_layout( + s, mn=n_blk * BLOCK, k=k_blk * BLOCK, recipe=(1, BLOCK, BLOCK), is_sfa=False + ) + return sf.t() + + +def _block_sf(s: torch.Tensor, core) -> torch.Tensor: + """An FP8 linear's block scales as stored, fp32 `[N/128, 1, K/128, 1]`: squeezed to the + `[N/128, K/128]` grid, then `_deepgemm_sf`.""" + assert s.dim() == 4 and s.shape[1] == 1 and s.shape[3] == 1, ( + f"FP8 block scale stored as {tuple(s.shape)}, not [N/128, 1, K/128, 1]" + ) + return _deepgemm_sf(s.reshape(s.shape[0], s.shape[2]), core) + + +def _shared_scalar(t: tuple[torch.Tensor, ...], core) -> torch.Tensor: + """One per-tensor scalar the checkpoint stores several times (`w1`'s and `w3`'s, or every + expert's of the window): asserted the same value everywhere, then that value.""" + first = t[0] + if not first.is_meta: + values = torch.stack(t) + assert bool((values == first).all()), ( + f"{int((values != first).sum())} of {len(t)} copies of one shared scale differ from " + f"the first, {first.item()!r}" + ) + return first + + +def _regroup_kv_b(t: torch.Tensor, core) -> torch.Tensor: + """This rank's `[heads * (k_nope + v), kv_lora]` rows, per-head `[k_nope | v]` -> + `[heads * k_nope ; heads * v]`: the K-B operand, then the V-B one. k_nope is read off the + slice: 64 on RoPE layers, 128 on NoPE layers.""" + cfg = core.model_config.pretrained_config + heads = cfg.num_attention_heads // core.model_config.mapping.tp_size + v = cfg.v_head_dim + t3 = t.reshape(heads, t.shape[0] // heads, t.shape[1]) + nope = t3.shape[1] - v + return torch.cat( + [t3[:, :nope, :].reshape(heads * nope, -1), t3[:, nope:, :].reshape(heads * v, -1)], dim=0 + ).contiguous() + + +def _to_fp32(t: torch.Tensor, core) -> torch.Tensor: + return t.to(torch.float32) + + +# --- the NVFP4 MoE runner's expert layout, one expert at a time --- + +#: The runner's row shuffle works on blocks of this many rows. +SHUFFLE_ROWS = 32 +#: The block-scale swizzle's tile: 128 rows by 4 scale columns. +SWIZZLE_ROWS, SWIZZLE_COLS = 128, 4 + + +@functools.lru_cache(maxsize=None) +def _block32_perm(rows: int, device) -> torch.Tensor: + """Gather index of the 32-row block shuffle: within each block of 32 rows, source row + `4u + v` lands at row `8v + u`.""" + assert rows % SHUFFLE_ROWS == 0, rows + j = torch.arange(SHUFFLE_ROWS) + one = torch.empty(SHUFFLE_ROWS, dtype=torch.long) + one[(j % 4) * 8 + j // 4] = j + blocks = torch.arange(rows // SHUFFLE_ROWS).repeat_interleave(SHUFFLE_ROWS) * SHUFFLE_ROWS + return (one.repeat(rows // SHUFFLE_ROWS) + blocks).to(device) + + +@functools.lru_cache(maxsize=None) +def _fc1_perm(rows: int, device) -> torch.Tensor: + """Gather index of FC1's `[up ; gate]` rows: interleaved (row `2i` up row `i`, row `2i + 1` + gate row `i`), then block-shuffled.""" + interleave = torch.empty(rows, dtype=torch.long) + interleave[0::2] = torch.arange(rows // 2) + interleave[1::2] = torch.arange(rows // 2, rows) + return interleave.to(device)[_block32_perm(rows, device)] + + +def _swizzle(s: torch.Tensor) -> torch.Tensor: + """One expert's `[M, C]` e4m3 block scales in the 128x4-swizzled byte order, kept `[M, C]`: + byte `(m, c)` lands at flat offset `(m // 128) * 128 * C + (c // 4) * 512 + (m % 32) * 16 + + ((m % 128) // 32) * 4 + c % 4`. Both of this target's scale stacks tile it exactly (M a + multiple of 128, C of 4), so there is no padding.""" + m, c = s.shape + assert m % SWIZZLE_ROWS == 0 and c % SWIZZLE_COLS == 0, (m, c) + tiles = s.view(torch.uint8).reshape(m // SWIZZLE_ROWS, 4, 32, c // SWIZZLE_COLS, SWIZZLE_COLS) + return tiles.permute(0, 3, 2, 1, 4).reshape(m, c).view(s.dtype) + + +def _fc1_codes(w: torch.Tensor) -> torch.Tensor: + return torch.index_select(w, 0, _fc1_perm(w.shape[0], w.device)) + + +def _fc1_scales(s: torch.Tensor) -> torch.Tensor: + rows = torch.index_select(s.view(torch.uint8), 0, _fc1_perm(s.shape[0], s.device)) + return _swizzle(rows.view(s.dtype)) + + +def _fc2_codes(w: torch.Tensor) -> torch.Tensor: + return torch.index_select(w, 0, _block32_perm(w.shape[0], w.device)) + + +def _fc2_scales(s: torch.Tensor) -> torch.Tensor: + rows = torch.index_select(s.view(torch.uint8), 0, _block32_perm(s.shape[0], s.device)) + return _swizzle(rows.view(s.dtype)) + + +def _materialize(entry, shard) -> torch.Tensor: + """This rank's bytes of one checkpoint entry: a tensor, or a lazy safetensors slice, which + reads only what the index selects. `[...]`, not `[:]`: the per-tensor scalars are 0-dim.""" + return entry[...] if shard is None else shard(entry) + + +#: The checkpoint's shard index, beside its `consolidated-*.safetensors`. +SHARD_INDEX = "consolidated.safetensors.index.json" + + +def _checkpoint_shards(checkpoint_dir) -> tuple[Path, dict[str, str]]: + """`(directory, key -> shard file)` of a checkpoint directory: its index when it ships + one, else the header of each `consolidated*.safetensors`.""" + directory = Path(checkpoint_dir) + index = directory / SHARD_INDEX + if index.is_file(): + return directory, json.loads(index.read_text())["weight_map"] + weight_map: dict[str, str] = {} + for path in sorted(directory.glob("consolidated*.safetensors")): + with safe_open(str(path), framework="pt", device="cpu") as handle: + weight_map.update(dict.fromkeys(handle.keys(), path.name)) + if not weight_map: + raise FileNotFoundError(f"no consolidated*.safetensors in {directory}") + return directory, weight_map + + +def _shard_files(core, weights) -> tuple[Path, dict[str, str]] | None: + """`(directory, key -> shard file)` of the checkpoint a dict of sources was read from, or + None when the sources are not the shard files of one (the directory the config records + lacks the index, or the index does not name every key `weights` holds).""" + cfg = getattr(core.model_config, "pretrained_config", None) + root = getattr(cfg, "_name_or_path", None) + if not root: + return None + index = Path(root) / SHARD_INDEX + if not index.is_file(): + return None + weight_map = json.loads(index.read_text())["weight_map"] + if not set(weights.keys()) <= set(weight_map): + return None + return Path(root), weight_map + + +def _drop_page_cache(path: Path) -> None: + """Advise the kernel to drop a consumed shard file's cached pages. Advice only: a + filesystem that ignores it costs memory pressure, not correctness. Every rank reads every + shard file, a node's ranks through that node's page cache, so the advice can also drop pages + a sibling rank has yet to read, which that rank then reads again from disk; the kernel keeps + the pages a process has mapped.""" + try: + fd = os.open(path, os.O_RDONLY) + except OSError: + return + try: + os.posix_fadvise(fd, 0, 0, os.POSIX_FADV_DONTNEED) + except OSError: + pass + finally: + os.close(fd) + + +# --- which layers carry a weight --- + + +def _owners(d): + """Layers that own a latent KV. The others read `kv_source_layers[i]`'s and project none.""" + return [i for i in range(d.num_layers) if d.kv_source[i] == i] + + +def _rope_owners(d): + return [i for i in _owners(d) if not d.nope_layer[i]] + + +def _nope_owners(d): + return [i for i in _owners(d) if d.nope_layer[i]] + + +def _rope_layers(d): + return [i for i in range(d.num_layers) if not d.nope_layer[i]] + + +def _nope_layers(d): + return [i for i in range(d.num_layers) if d.nope_layer[i]] + + +def _indexer_layers(d): + """Layers that own an indexer. The 'shared' ones reuse the most recent one's selection.""" + return [i for i in range(d.num_layers) if d.full_indexer[i]] + + +def _dense_layers(d): + return range(d.dense_layers) + + +def _moe_layers(d): + return range(d.dense_layers, d.num_layers) + + +# --- where the bytes come from --- + + +def _stack(*parts, prefix="layers"): + """Several checkpoint tensors into one parameter: each `(key template, index(d))`, the + index being where in the parameter that tensor lands.""" + return lambda d, i: [(t.format(p=f"{prefix}.{i}"), index(d)) for t, index in parts] + + +def _expert_key(i: int, e: int, suffix: str) -> str: + return f"layers.{i}.experts.{e}.{suffix}" + + +def _expert_window(*parts): + """This rank's routed experts, local slot `s` <- global expert `expert_offset + s`: each + `(checkpoint suffix | suffix tuple, index(d) after the expert axis)`.""" + + def key(i, e, suffix): + if isinstance(suffix, tuple): + return tuple(_expert_key(i, e, s) for s in suffix) + return _expert_key(i, e, suffix) + + def build(d, i): + return [ + (key(i, d.expert_offset + slot, suffix), (slot, *index(d))) + for slot in range(d.local_experts) + for suffix, index in parts + ] + + return build + + +def _window_scalar(*suffixes): + """One per-tensor scalar every expert of the window stores: all of the window's copies, one + key tuple, into element 0.""" + return lambda d, i: [ + ( + tuple( + _expert_key(i, d.expert_offset + slot, s) + for slot in range(d.local_experts) + for s in suffixes + ), + (0,), + ) + ] + + +def _expert_rows(): + """This rank's slice of an expert's intermediate output rows: `w1` and `w3`, codes and block + scales alike, `[moe_tp_rank * moe_inter, + moe_inter)`.""" + return _rows(lambda d: d.moe_tp_rank * d.moe_inter, lambda d: d.moe_inter) + + +def _expert_cols(per_column: int): + """This rank's slice of an expert's intermediate input axis, `w2`'s, as stored with + `per_column` inputs to a column: 2 for the e2m1 codes, 16 for the block scales.""" + width = lambda d: d.moe_inter // per_column # noqa: E731 + return _cols(lambda d: d.moe_tp_rank * width(d), width) + + +def _expert_scalar_keys(d) -> list[str]: + """Every routed expert's per-tensor scalars on every MoE layer: what `load` reads, and + checks, before anything else.""" + return [ + _expert_key(i, e, f"{proj}.{s}") + for i in _moe_layers(d) + for e in range(d.num_experts) + for proj in PROJECTIONS + for s in EXPERT_SCALARS + ] + + +def _check_expert_scalars(scalars: dict[str, torch.Tensor], d) -> None: + """Per MoE layer, over all of its experts: `w1` and `w3` store the same `input_scale` and + `weight_scale_2` on every expert; `w1` and `w2` each store one `input_scale` for every expert; + every scalar is a positive finite fp32. Meta tensors carry no values to check.""" + if scalars and next(iter(scalars.values())).is_meta: + return + experts = range(d.num_experts) + for i in _moe_layers(d): + v = { + (proj, s): torch.stack([scalars[_expert_key(i, e, f"{proj}.{s}")] for e in experts]) + for proj in PROJECTIONS + for s in EXPERT_SCALARS + } + for (proj, s), t in v.items(): + bad = (~(torch.isfinite(t) & (t > 0))).nonzero().flatten() + assert bad.numel() == 0, ( + f"layers.{i}: {proj}.{s} of expert {int(bad[0])} is {t[bad[0]].item()!r}, " + "not a positive finite scale" + ) + for s in EXPERT_SCALARS: + bad = (v["w1", s] != v["w3", s]).nonzero().flatten() + assert bad.numel() == 0, ( + f"layers.{i}: w1 and w3 {s} differ on {bad.numel()} of {len(experts)} experts, " + f"first expert {int(bad[0])}: {v['w1', s][bad[0]].item()!r} vs " + f"{v['w3', s][bad[0]].item()!r}" + ) + for proj in ("w1", "w2"): + isc = v[proj, "input_scale"] + bad = (isc != isc[0]).nonzero().flatten() + assert bad.numel() == 0, ( + f"layers.{i}: {proj}.input_scale is not one value over the experts: expert " + f"{int(bad[0])} stores {isc[bad[0]].item()!r}, expert 0 {isc[0].item()!r}" + ) + + +def _read_scalars(keys: list[str], weights, shards) -> dict[str, torch.Tensor]: + """The given per-tensor scalars, off the shard files one file at a time when they are + found, otherwise off the given dict.""" + if shards is None: + return {key: _materialize(weights[key], None) for key in keys} + directory, weight_map = shards + by_file: dict[str, list] = collections.defaultdict(list) + for key in keys: + by_file[weight_map[key]].append(key) + out = {} + for name in sorted(by_file): + with safe_open(str(directory / name), framework="pt", device="cpu") as handle: + for key in by_file[name]: + out[key] = handle.get_slice(key)[...].clone() + return out + + +def _fp8_linear(name, key, n, k, layers, w_shard=None, s_shard=None): + """An FP8 128x128-block linear: e4m3 `[N, K]` and its DeepGEMM scale operand's storage.""" + return ( + W(name, shape=lambda d: (n(d), k(d)), dtype=F8, src=key, layers=layers, shard=w_shard), + W( + f"{name}_s", + shape=lambda d: (_sf_words(k(d)), n(d)), + dtype=I32, + src=key.replace(".weight", ".weight_scale"), + transform=_block_sf, + layers=layers, + shard=s_shard, + ), + ) + + +def _tp_mlp(module, inter, layers): + """A SwiGLU MLP split over its intermediate dimension: `[w1 ; w3]` by output rows, gate + first (the half order `flashinfer_silu_and_mul` reads), `w2` by input columns. The same + row slice applies to `w1` and `w3` before they are stacked, and the scales follow in + 128-blocks; in the scale storage the stacked axis is the second one.""" + blocks = lambda d: inter(d) // BLOCK # noqa: E731 + start = lambda d: d.tp_rank * inter(d) # noqa: E731 + b_start = lambda d: d.tp_rank * blocks(d) # noqa: E731 + p = "{p}." + module + return ( + W( + "mlp_gu", + shape=lambda d: (2 * inter(d), d.hidden), + dtype=F8, + src=_stack( + (f"{p}.w1.weight", lambda d: (slice(0, inter(d)),)), + (f"{p}.w3.weight", lambda d: (slice(inter(d), 2 * inter(d)),)), + ), + layers=layers, + shard=_rows(start, inter), + ), + W( + "mlp_gu_s", + shape=lambda d: (_sf_words(d.hidden), 2 * inter(d)), + dtype=I32, + src=_stack( + (f"{p}.w1.weight_scale", lambda d: (slice(None), slice(0, inter(d)))), + (f"{p}.w3.weight_scale", lambda d: (slice(None), slice(inter(d), 2 * inter(d)))), + ), + transform=_block_sf, + layers=layers, + shard=_rows(b_start, blocks), + ), + *_fp8_linear( + "mlp_dn", + f"{p}.w2.weight", + lambda d: d.hidden, + inter, + layers, + w_shard=_cols(start, inter), + s_shard=_sf_cols(b_start, blocks), + ), + ) + + +def _kv_b(nope, layers): + """`wkv_b`, this rank's heads, regrouped. Per-head rows are `nope(d) + v` wide.""" + width = lambda d: d.heads * (nope(d) + d.v_dim) # noqa: E731 + return W( + "wkv_b", + shape=lambda d: (width(d), d.kv_lora), + src="{p}.attention.wkv_b.weight", + transform=_regroup_kv_b, + layers=layers, + shard=_rows(lambda d: d.tp_rank * width(d), width), + ) + + +def _heads(width): + """This rank's slice of a per-head axis of the given per-head width.""" + return _rows(lambda d: d.tp_rank * d.heads * width(d), lambda d: d.heads * width(d)) + + +def _vision_blocks(d): + return list(range(d.v_layers)) + + +def _has_vision(d): + return d.has_vision + + +def _flatten_patch_conv(tensor, core): + return tensor.flatten(1) + + +def _draft_layers(d): + return range(d.draft_layers) + + +def _draft_source(suffix): + return lambda d, i: [(f"mtp_dspark.backbone.layers.{i}.{suffix}", None)] + + +class MistralLarge4TpWeights: + """mistral-large-4-nvfp4 / sm_103 / tp.""" + + WEIGHTS: tuple[W, ...] = ( + # --- MLA: every layer, KV owners only where marked --- + W("attn_norm", shape=lambda d: (d.hidden,), src="{p}.attention_norm.weight"), + # Per head `[nope 64 | rope 64]` on RoPE layers, 128 nope on NoPE layers: 128 either way. + *_fp8_linear( + "wq", + "{p}.attention.wq.weight", + lambda d: d.heads * d.qk_dim, + lambda d: d.hidden, + ALL_LAYERS, + w_shard=_heads(lambda d: d.qk_dim), + s_shard=_heads(lambda d: d.qk_dim // BLOCK), + ), + W( + "ssmax_s", + shape=lambda d: (d.heads,), + src="{p}.attention.ssmax_scaling.s", + shard=_heads(lambda d: 1), + ), + W( + "ssmax_b", + shape=lambda d: (d.heads,), + src="{p}.attention.ssmax_scaling.b", + shard=_heads(lambda d: 1), + ), + # `[c_kv 512 | k_pe 64]` on RoPE owners; NoPE owners project the 512 latent alone. + W( + "wkv_a", + shape=lambda d: (d.lat_dim, d.hidden), + src="{p}.attention.wkv_a_with_mqa.weight", + layers=_rope_owners, + ), + W( + "wkv_a", + shape=lambda d: (d.kv_lora, d.hidden), + src="{p}.attention.wkv_a_with_mqa.weight", + layers=_nope_owners, + ), + W( + "kv_a_norm", + shape=lambda d: (d.kv_lora,), + src="{p}.attention.kv_a_norm.weight", + layers=_owners, + ), + # Every layer absorbs through its own `wkv_b`, KV sharers included. + _kv_b(lambda d: d.nope, _rope_layers), + _kv_b(lambda d: d.qk_dim, _nope_layers), + *_fp8_linear( + "wo", + "{p}.attention.wo.weight", + lambda d: d.hidden, + lambda d: d.heads * d.v_dim, + ALL_LAYERS, + w_shard=_cols(lambda d: d.tp_rank * d.heads * d.v_dim, lambda d: d.heads * d.v_dim), + s_shard=_sf_cols( + lambda d: d.tp_rank * d.heads * d.v_dim // BLOCK, + lambda d: d.heads * d.v_dim // BLOCK, + ), + ), + W("ffn_norm", shape=lambda d: (d.hidden,), src="{p}.ffn_norm.weight"), + # --- the DSA indexer: replicated, on the 'full' layers --- + W( + "ix_q_norm_w", + shape=lambda d: (d.hidden,), + src="{p}.attention.indexer.q_input_norm.weight", + layers=_indexer_layers, + ), + W( + "ix_q_norm_b", + shape=lambda d: (d.hidden,), + src="{p}.attention.indexer.q_input_norm.bias", + layers=_indexer_layers, + ), + # Per head `[rope 64 | nope 64]`, the reverse of MLA's. + *_fp8_linear( + "ix_wq_b", + "{p}.attention.indexer.wq_b.weight", + lambda d: d.idx_heads * d.idx_dim, + lambda d: d.hidden, + _indexer_layers, + ), + # `[wk ; weights_proj]`: the key and the head weights, one GEMM. + W( + "ix_wk_wp", + shape=lambda d: (d.idx_dim + d.idx_heads, d.hidden), + src=_stack( + ("{p}.attention.indexer.wk.weight", lambda d: (slice(0, d.idx_dim),)), + ( + "{p}.attention.indexer.weights_proj.weight", + lambda d: (slice(d.idx_dim, d.idx_dim + d.idx_heads),), + ), + ), + layers=_indexer_layers, + ), + # fp32, as the fused key entry (LayerNorm in fp32) takes them. + W( + "ix_k_norm_w", + shape=lambda d: (d.idx_dim,), + dtype=F32, + src="{p}.attention.indexer.k_norm.weight", + transform=_to_fp32, + layers=_indexer_layers, + ), + W( + "ix_k_norm_b", + shape=lambda d: (d.idx_dim,), + dtype=F32, + src="{p}.attention.indexer.k_norm.bias", + transform=_to_fp32, + layers=_indexer_layers, + ), + W( + "ix_ssmax_s", + shape=lambda d: (d.idx_heads,), + src="{p}.attention.indexer.ssmax_scaling.s", + layers=_indexer_layers, + ), + W( + "ix_ssmax_b", + shape=lambda d: (d.idx_heads,), + src="{p}.attention.indexer.ssmax_scaling.b", + layers=_indexer_layers, + ), + # --- the TP-sliced SwiGLU MLP: the dense layer's own, then each MoE layer's shared + # expert, one key --- + *_tp_mlp("feed_forward", lambda d: d.dense_inter, _dense_layers), + *_tp_mlp("shared_experts", lambda d: d.shared_inter, _moe_layers), + # --- routing --- + W( + "router", + shape=lambda d: (d.num_experts, d.hidden), + src="{p}.gate.weight", + layers=_moe_layers, + ), + # --- the routed experts, NVFP4: this rank's width slice of every expert, `[w3 ; w1]` as + # stored, then each expert's slice in the MoE runner's layout --- + W( + "fc1_w", + shape=lambda d: (d.local_experts, 2 * d.moe_inter, d.hidden // E2M1_PER_BYTE), + dtype=U8, + src=_expert_window( + ("w3.weight", lambda d: (slice(0, d.moe_inter),)), + ("w1.weight", lambda d: (slice(d.moe_inter, 2 * d.moe_inter),)), + ), + layers=_moe_layers, + shard=_expert_rows(), + layout=_fc1_codes, + ), + W( + "fc1_s", + shape=lambda d: (d.local_experts, 2 * d.moe_inter, d.hidden // NVFP4_BLOCK), + dtype=F8, + src=_expert_window( + ("w3.weight_scale", lambda d: (slice(0, d.moe_inter),)), + ("w1.weight_scale", lambda d: (slice(d.moe_inter, 2 * d.moe_inter),)), + ), + layers=_moe_layers, + shard=_expert_rows(), + layout=_fc1_scales, + ), + W( + "fc2_w", + shape=lambda d: (d.local_experts, d.hidden, d.moe_inter // E2M1_PER_BYTE), + dtype=U8, + src=_expert_window(("w2.weight", lambda d: ())), + layers=_moe_layers, + shard=_expert_cols(E2M1_PER_BYTE), + layout=_fc2_codes, + ), + W( + "fc2_s", + shape=lambda d: (d.local_experts, d.hidden, d.moe_inter // NVFP4_BLOCK), + dtype=F8, + src=_expert_window(("w2.weight_scale", lambda d: ())), + layers=_moe_layers, + shard=_expert_cols(NVFP4_BLOCK), + layout=_fc2_scales, + ), + # The per-tensor scalars, whole: the layer's one FC1 and one FC2 activation scale, and + # each local expert's global weight scales (`w3`'s are `w1`'s). + W( + "isc1", + shape=lambda d: (1,), + dtype=F32, + src=_window_scalar("w1.input_scale", "w3.input_scale"), + transform=_shared_scalar, + layers=_moe_layers, + ), + W( + "isc2", + shape=lambda d: (1,), + dtype=F32, + src=_window_scalar("w2.input_scale"), + transform=_shared_scalar, + layers=_moe_layers, + ), + W( + "ws2_13", + shape=lambda d: (d.local_experts,), + dtype=F32, + src=_expert_window((("w1.weight_scale_2", "w3.weight_scale_2"), lambda d: ())), + transform=_shared_scalar, + layers=_moe_layers, + ), + W( + "ws2_2", + shape=lambda d: (d.local_experts,), + dtype=F32, + src=_expert_window(("w2.weight_scale_2", lambda d: ())), + layers=_moe_layers, + ), + # --- not per-layer --- + W("final_norm", shape=lambda d: (d.hidden,), src="norm.weight", layers=None), + W("embed", shape=lambda d: (d.vocab, d.hidden), src="tok_embeddings.weight", layers=None), + # The LeRoPE per-frequency log-scales the rope tables are built from, fp32 as the + # reference widens them. + W( + "lerope_alpha", + shape=lambda d: (d.rope_dim // 2,), + dtype=F32, + src="lerope.alpha", + transform=_to_fp32, + layers=None, + ), + # BF16 vision encoder and projector, replicated on every rank. + W( + "vis_patch", + shape=lambda d: (d.v_hidden, d.v_channels * d.v_patch**2), + src="vision_encoder.patch_conv.weight", + transform=_flatten_patch_conv, + layers=None, + when=_has_vision, + ), + W( + "vis_ln_pre", + shape=lambda d: (d.v_hidden,), + src="vision_encoder.ln_pre.weight", + layers=None, + when=_has_vision, + ), + W( + "attn_norm", + shape=lambda d: (d.v_hidden,), + src="{p}.attention_norm.weight", + layers=_vision_blocks, + tag="v", + ), + W( + "qkv", + shape=lambda d: (3 * d.v_hidden, d.v_hidden), + src=_stack( + ("{p}.attention.wq.weight", lambda d: (slice(0, d.v_hidden),)), + ("{p}.attention.wk.weight", lambda d: (slice(d.v_hidden, 2 * d.v_hidden),)), + ("{p}.attention.wv.weight", lambda d: (slice(2 * d.v_hidden, 3 * d.v_hidden),)), + prefix="vision_encoder.transformer.layers", + ), + layers=_vision_blocks, + tag="v", + ), + W( + "wo", + shape=lambda d: (d.v_hidden, d.v_hidden), + src="{p}.attention.wo.weight", + layers=_vision_blocks, + tag="v", + ), + W( + "ffn_norm", + shape=lambda d: (d.v_hidden,), + src="{p}.ffn_norm.weight", + layers=_vision_blocks, + tag="v", + ), + W( + "w13", + shape=lambda d: (2 * d.v_inter, d.v_hidden), + src=_stack( + ("{p}.feed_forward.w1.weight", lambda d: (slice(0, d.v_inter),)), + ("{p}.feed_forward.w3.weight", lambda d: (slice(d.v_inter, 2 * d.v_inter),)), + prefix="vision_encoder.transformer.layers", + ), + layers=_vision_blocks, + tag="v", + ), + W( + "w2", + shape=lambda d: (d.v_hidden, d.v_inter), + src="{p}.feed_forward.w2.weight", + layers=_vision_blocks, + tag="v", + ), + W( + "vis_proj_norm", + shape=lambda d: (d.v_hidden,), + src="pre_mm_projector_norm.weight", + layers=None, + when=_has_vision, + ), + W( + "vis_merge", + shape=lambda d: (d.v_hidden, d.v_hidden * d.v_merge**2), + src="patch_merger.merging_layer.weight", + layers=None, + when=_has_vision, + ), + W( + "vis_w_in", + shape=lambda d: (d.hidden, d.v_hidden), + src="vision_language_adapter.w_in.weight", + layers=None, + when=_has_vision, + ), + W( + "vis_w_out", + shape=lambda d: (d.hidden, d.hidden), + src="vision_language_adapter.w_out.weight", + layers=None, + when=_has_vision, + ), + ) + + # BF16 draft weights use the same attention and dense-MLP partition as + # this target. The RNN and target-feature projection are replicated. + DRAFT_WEIGHTS: tuple[W, ...] = ( + W( + "draft_attn_norm", + lambda d: (d.hidden,), + _draft_source("attention_norm.weight"), + layers=_draft_layers, + ), + W( + "draft_wq", + lambda d: (d.heads * d.qk_dim, d.hidden), + _draft_source("attention.wq.weight"), + layers=_draft_layers, + shard=_rows( + lambda d: d.draft_tp_rank * d.heads * d.qk_dim, lambda d: d.heads * d.qk_dim + ), + ), + W( + "draft_wkv_a", + lambda d: (d.lat_dim, d.hidden), + _draft_source("attention.wkv_a_with_mqa.weight"), + layers=_draft_layers, + ), + W( + "draft_kv_a_norm", + lambda d: (d.kv_lora,), + _draft_source("attention.kv_a_norm.weight"), + layers=_draft_layers, + ), + W( + "draft_wkv_b", + lambda d: (d.heads * (d.nope + d.v_dim), d.kv_lora), + _draft_source("attention.wkv_b.weight"), + layers=_draft_layers, + shard=_rows( + lambda d: d.draft_tp_rank * d.heads * (d.nope + d.v_dim), + lambda d: d.heads * (d.nope + d.v_dim), + ), + ), + W( + "draft_wo", + lambda d: (d.hidden, d.heads * d.v_dim), + _draft_source("attention.wo.weight"), + layers=_draft_layers, + shard=_cols(lambda d: d.draft_tp_rank * d.heads * d.v_dim, lambda d: d.heads * d.v_dim), + ), + W( + "draft_ffn_norm", + lambda d: (d.hidden,), + _draft_source("ffn_norm.weight"), + layers=_draft_layers, + ), + W( + "draft_w1", + lambda d: (d.dense_inter, d.hidden), + _draft_source("feed_forward.w1.weight"), + layers=_draft_layers, + shard=_rows(lambda d: d.draft_tp_rank * d.dense_inter, lambda d: d.dense_inter), + ), + W( + "draft_w3", + lambda d: (d.dense_inter, d.hidden), + _draft_source("feed_forward.w3.weight"), + layers=_draft_layers, + shard=_rows(lambda d: d.draft_tp_rank * d.dense_inter, lambda d: d.dense_inter), + ), + W( + "draft_w2", + lambda d: (d.hidden, d.dense_inter), + _draft_source("feed_forward.w2.weight"), + layers=_draft_layers, + shard=_cols(lambda d: d.draft_tp_rank * d.dense_inter, lambda d: d.dense_inter), + ), + W( + "draft_final_norm", + lambda d: (d.hidden,), + "mtp_dspark.backbone.final_norm.weight", + layers=None, + ), + W( + "draft_target_norm", + lambda d: (d.hidden,), + "mtp_dspark.backbone.target_norm.weight", + layers=None, + ), + W( + "draft_target_projection", + lambda d: (d.hidden, d.hidden * d.draft_capture_layers), + "mtp_dspark.backbone.target_projection.weight", + layers=None, + ), + W( + "draft_mask", + lambda d: (d.draft_depth - 1, d.hidden), + "mtp_dspark.backbone.mask_embedding", + layers=None, + ), + W( + "draft_markov_embedding", + lambda d: (d.vocab, d.draft_rank), + "mtp_dspark.autoregressive_head.embedding.weight", + layers=None, + ), + W( + "draft_markov_output", + lambda d: (d.vocab, d.draft_rank), + "mtp_dspark.autoregressive_head.output.weight", + layers=None, + ), + W( + "draft_joint", + lambda d: (3 * d.draft_rank, d.hidden + 2 * d.draft_rank), + "mtp_dspark.autoregressive_head.joint_proj.weight", + layers=None, + ), + W( + "draft_joint_bias", + lambda d: (3 * d.draft_rank,), + "mtp_dspark.autoregressive_head.joint_proj.bias", + layers=None, + ), + ) + + def _entries(self, d): + return self.WEIGHTS + (self.DRAFT_WEIGHTS if d.draft_layers else ()) + + def declare(self, d: SimpleNamespace) -> nn.ParameterDict: + """Allocate every weight the table declares. + + Meta-init intercepts `torch.empty` here -- real CUDA storage arrives + when the engine materializes the registry. + """ + w = nn.ParameterDict() + for entry in self._entries(d): + dtype = entry.dtype or d.dtype + for key, _, _ in self._targets(entry, d): + assert key not in w, f"two rows declare {key}" + w[key] = nn.Parameter( + torch.empty(*entry.shape(d), dtype=dtype), requires_grad=False + ) + return w + + def _targets(self, entry: W, d: SimpleNamespace) -> list[tuple[str, int | None, str]]: + """`[(param key, layer index | None, checkpoint layer prefix), ...]`. + + One place decides which parameters an entry declares, so `declare` and + `manifest` cannot disagree about whether a weight exists. + """ + if entry.when is not None and not entry.when(d): + return [] + layers = entry.layer_indices(d) + if layers is None: + return [(entry.name, None, "")] + prefix = "vision_encoder.transformer.layers" if entry.tag == "v" else "layers" + return [(f"{entry.tag}{i}_{entry.name}", i, f"{prefix}.{i}") for i in layers] + + def manifest(self, core) -> dict: + """param key -> [(checkpoint key | key tuple, index into the param | None, transform, + shard)].""" + d = self.dims(core) + rows: dict = {} + for entry in self._entries(d): + shard = entry.shard(d) if entry.shard else None + for key, i, prefix in self._targets(entry, d): + assert key not in rows, f"two rows declare {key}" + rows[key] = [ + (ckpt, index, entry.transform, shard) + for ckpt, index in entry.sources(d, i, prefix) + ] + return rows + + def expected_unconsumed(self, core, weights) -> dict[str, str]: + """Declared unused keys: off-rank experts and disabled draft weights. Off-rank expert + scalars are still read to validate the layer's shared scales.""" + d = self.dims(core) + lo, hi = d.expert_offset, d.expert_offset + d.local_experts + keys = { + k: "DSpark draft disabled" + for k in weights.keys() + if not d.draft_layers and k.startswith("mtp_dspark.") + } + window = f"routed expert outside this rank's window [{lo}, {hi})" + never_read = f"{window}: never read" + checked = f"{window}: read only to check the layer's shared scales" + for i in _moe_layers(d): + for e in range(d.num_experts): + if lo <= e < hi: + continue + for proj in PROJECTIONS: + for s in ("weight", "weight_scale"): + key = _expert_key(i, e, f"{proj}.{s}") + if key in weights: + keys[key] = never_read + for s in EXPERT_SCALARS: + keys[_expert_key(i, e, f"{proj}.{s}")] = checked + return keys + + def load_checkpoint(self, model, checkpoint_dir) -> None: + """Fill every declared weight from a checkpoint directory, which is what the engine + loads: `load` over the directory's keys, every source read from its shard file.""" + directory, weight_map = _checkpoint_shards(checkpoint_dir) + self.load(model, weight_map, shards=(directory, weight_map)) + + def load(self, model, weights, shards=None) -> None: + """Fill every declared weight from the checkpoint: the shard column, multi-source rows, + the vocab-parallel `lm_head`, and coverage asserted in both directions. + + First every routed expert's per-tensor scalars are read and checked, so a checkpoint + that breaks what the forward relies on fails before any weight moves. When the + checkpoint's own shard files can be found, every source is read from them one file at a + time, this rank's slices only -- of the routed experts its width slice of each, never a + whole expert -- and each file's cached pages are dropped once its sources are copied. + Mapping every shard file on every rank and reading through the mappings instead + would fault a node's four ranks' slices into that node's host memory: on one + node of four, between them every expert, 539 GiB, the whole expert stack; on each of + two nodes, half of every expert's `w1` and `w3` and every page of every `w2`, which a + strided column read touches whole, 359 GiB. Last, each row's `layout` puts each + expert's slice into the MoE runner's layout, so a second `load` starts again from the + stored bytes. + """ + core = model.model + manifest = self.manifest(core) + consumed: set = set() + + def fill(param, ckpt, entry, index, transform, shard) -> None: + dst = param.data if index is None else param.data[index] + if isinstance(ckpt, tuple): + src = transform( + tuple(_materialize(e, shard).to(dst.device, non_blocking=True) for e in entry), + core, + ) + else: + src = _materialize(entry, shard) + if transform is not None: + # A column shard is a strided view: make it dense before a transform reads it. + src = transform(src.to(dst.device, non_blocking=True).contiguous(), core) + assert tuple(dst.shape) == tuple(src.shape), (ckpt, tuple(dst.shape), tuple(src.shape)) + assert src.dtype == dst.dtype, (ckpt, src.dtype, dst.dtype) + dst.copy_(src, non_blocking=True) + consumed.update(_keys(ckpt)) + + assert set(manifest.keys()) == set(core.w.keys()), ( + "manifest/parameter drift", + sorted(set(manifest.keys()) ^ set(core.w.keys()))[:12], + ) + # The shell builds lm_head vocab-parallel under TP: this rank's block of rows. + d = self.dims(core) + rows = d.vocab // d.tp_size + plan = [ + (core.w[param_key], ckpt, index, transform, shard) + for param_key, sources in manifest.items() + for ckpt, index, transform, shard in sources + ] + plan.append( + ( + model.lm_head.weight, + "output.weight", + None, + None, + _rows(lambda _: d.tp_rank * rows, lambda _: rows)(d), + ) + ) + scalar_keys = _expert_scalar_keys(d) + for key in (*(k for row in plan for k in _keys(row[1])), *scalar_keys): + assert key in weights, f"checkpoint key missing: {key}" + if shards is None: + shards = _shard_files(core, weights) + scalars = _read_scalars(scalar_keys, weights, shards) + _check_expert_scalars(scalars, d) + + streamed = [] + for param, ckpt, index, transform, shard in plan: + if all(k in scalars for k in _keys(ckpt)): + entry = tuple(scalars[k] for k in _keys(ckpt)) + entry = entry if isinstance(ckpt, tuple) else entry[0] + fill(param, ckpt, entry, index, transform, shard) + else: + assert isinstance(ckpt, str), f"a key tuple of other than expert scalars: {ckpt}" + streamed.append((param, ckpt, index, transform, shard)) + if shards is None: + for param, ckpt, index, transform, shard in streamed: + fill(param, ckpt, weights[ckpt], index, transform, shard) + else: + directory, weight_map = shards + by_file: dict[str, list] = collections.defaultdict(list) + for row in streamed: + by_file[weight_map[row[1]]].append(row) + on_device = model.lm_head.weight.device.type == "cuda" + for name in sorted(by_file): + with safe_open(str(directory / name), framework="pt", device="cpu") as handle: + for param, ckpt, index, transform, shard in by_file[name]: + fill(param, ckpt, handle.get_slice(ckpt), index, transform, shard) + if on_device: + torch.cuda.synchronize() + _drop_page_cache(directory / name) + # Every source has landed: rearrange each expert in place, one at a time. + for entry in self._entries(d): + if entry.layout is None: + continue + for key, _, _ in self._targets(entry, d): + param = core.w[key].data + for slot in range(param.shape[0]): + param[slot].copy_(entry.layout(param[slot])) + shell = {n for n, _ in model.named_parameters() if not n.startswith("model.w.")} + assert shell == {"lm_head.weight"}, f"unfed shell parameters: {sorted(shell)}" + + if model.lm_head.weight.device.type == "cuda": + torch.cuda.synchronize() + leftover = set(weights.keys()) - consumed + expected = self.expected_unconsumed(core, weights) + assert leftover == set(expected), ( + "checkpoint coverage: unexpected " + f"{sorted(leftover - set(expected))[:8]}, missed {sorted(set(expected) - leftover)[:8]}" + ) + not_loaded = collections.Counter(expected.values()) + logger.info( + f"modeling_v2 weights: {len(consumed)} checkpoint tensors loaded " + f"({'streamed per shard file' if shards else 'from the given dict'}), " + f"{len(scalars)} expert scalars checked; not loaded: " + + "; ".join(f"{n} x {reason}" for reason, n in sorted(not_loaded.items())) + ) + + def dims(self, core) -> SimpleNamespace: + cfg = core.model_config.pretrained_config + vis = getattr(cfg, "vision_config", None) + mapping = core.model_config.mapping + spec = getattr(core.model_config, "spec_config", None) + draft = getattr(spec, "_draft_model_config", None) or {} + tp = mapping.tp_size + local_experts = cfg.n_routed_experts // mapping.moe_ep_size + return SimpleNamespace( + draft_layers=int(draft.get("n_layers", 0)), + draft_rank=int((draft.get("dspark") or {}).get("markov_rank", 0)), + draft_depth=int(draft.get("depth", 0)), + draft_capture_layers=len(spec.target_layer_ids) if draft else 0, + draft_tp_rank=mapping.tp_rank, + num_layers=cfg.num_hidden_layers, + hidden=cfg.hidden_size, + heads=cfg.num_attention_heads // tp, + nope=cfg.qk_nope_head_dim, + rope_dim=cfg.qk_rope_head_dim, + qk_dim=cfg.qk_nope_head_dim + cfg.qk_rope_head_dim, + v_dim=cfg.v_head_dim, + kv_lora=cfg.kv_lora_rank, + lat_dim=cfg.kv_lora_rank + cfg.qk_rope_head_dim, + nope_layer=tuple(bool(f) for f in cfg.nope_layer_flags), + kv_source=tuple(int(s) for s in cfg.kv_source_layers), + full_indexer=tuple(t == "full" for t in cfg.index_sharing), + idx_heads=cfg.index_n_heads, + idx_dim=cfg.index_head_dim, + dense_layers=cfg.first_k_dense_replace, + num_experts=cfg.n_routed_experts, + moe_inter=cfg.moe_intermediate_size // mapping.moe_tp_size, + moe_tp_rank=mapping.moe_tp_rank, + local_experts=local_experts, + expert_offset=local_experts * mapping.moe_ep_rank, + shared_inter=cfg.moe_intermediate_size * cfg.n_shared_experts // tp, + dense_inter=cfg.intermediate_size // tp, + vocab=cfg.vocab_size, + tp_rank=mapping.tp_rank, + tp_size=tp, + dtype=core.model_config.torch_dtype, + # The vision tower (replicated; absent without a vision encoder). + has_vision=vis is not None, + v_layers=vis.num_hidden_layers if vis is not None else 0, + v_hidden=vis.hidden_size if vis is not None else 0, + v_heads=vis.num_attention_heads if vis is not None else 0, + v_head_dim=vis.hidden_size // vis.num_attention_heads if vis is not None else 0, + v_inter=vis.intermediate_size if vis is not None else 0, + v_channels=vis.num_channels if vis is not None else 0, + v_patch=vis.patch_size if vis is not None else 0, + v_merge=vis.spatial_merge_size if vis is not None else 0, + v_theta=float(vis.rope_theta) if vis is not None else 0.0, + image_token_id=vis.image_token_id if vis is not None else None, + ) + + +MODEL_WEIGHTS = MistralLarge4TpWeights() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__dep/__init__.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__dep/__init__.py new file mode 100644 index 000000000000..b75576946e8d --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__dep/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Mistral-Large-4-1T-A52B-NVFP4 / sm_107 / dep, the language model only.""" diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__dep/modeling.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__dep/modeling.py new file mode 100644 index 000000000000..388eadf3a0df --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__dep/modeling.py @@ -0,0 +1,2577 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""ModelingV2 target: mistral-large-4-nvfp4 / sm_107 / dep, native text and image inference. + +72 layers, hidden 6144, 64 query heads. Attention is MLA without a q-LoRA (`wq` straight to +64 x 128), a 512-wide latent plus a 64-wide rope key, and DeepSeek Sparse Attention: an +indexer on the 44 `index_sharing == 'full'` layers selects 2048 positions per query row and +each 'shared' layer reuses the selection of the most recent full layer. Every fourth layer +(`i % 4 == 3`) takes no rope, for MLA and for its indexer; the NoPE layers 39, 43, ..., 71 own +no latent and attend over layer 35's. Queries are scaled by SSMax, `s * log(pos + 1) + b` per +head, on MLA and on the indexer. Layer 0 is a dense SwiGLU MLP, layers 1-71 MoE (128 experts, +softmax top-4 renormalised, one shared expert). The final logits are soft-capped at +`softmax_tanh_gating`. + +The checkpoint is ModelOpt `MIXED_PRECISION`. The routed experts are NVFP4 W4A4: e2m1 weights +with one e4m3 scale per 16 inputs and a global scale per expert, and activations quantized to +NVFP4 under one static global scale per layer and projection. `wq`, `wo`, the indexer's `wq_b` +and the dense and shared MLPs are FP8 e4m3 with 128x128 power-of-two block scales and dynamic +1x128 activation scales. The rest is bf16. The latent KV pool is fp8_ds_mla (656 B per token +and owner layer: 512 e4m3 codes with one power-of-two fp32 scale per 128, and the 64-wide rope +key in bf16), owned by `MistralLarge4CacheManager` and per rank under attention DP; the indexer +key pool is fp8 with one scale per key. + +`dep` is attention data parallel over the ranks of the mapping, four on one node or eight on +two nodes of four (dep8): each rank serves its own requests with the whole model outside the +routed experts -- all 64 heads, the latent, the indexer, the dense layer-0 MLP, the shared +expert and the full-vocabulary `lm_head` -- and holds the KV of its own requests only. Nothing +before the routed experts communicates. The routed experts are expert parallel, +`n_routed_experts / moe_ep_size` of the 128 per rank (32 at dep4, 16 at dep8): each MoE layer +zero-pads the rank's rows to the largest rank's count, gathers every rank's rows before the +router (so every rank routes the identical token set and the windows tile the expert ids +exactly once), runs the router, the routing, the NVFP4 quantization and this rank's window over +the gathered rows in calls of at most `expert_call_tokens` rows, and reduce-scatters the partial +sums back to each rank's own rows. Every rank issues the same collective sequence whatever its +batch -- one gather and one reduce-scatter per MoE layer, of the same size -- in the prefill and +the decode target alike. + +The absorbed form throughout: `fused_q = [q_nope @ W_UK | rope(q_pe)]` (576 wide) against the +pooled `[c_kv | k_pe]` latent rows, then `@ W_UV`. Context rows attend over the selection (the +engine's causal index on a step it skipped the indexer) with FlashMLA's bf16 sparse prefill +over the requests' latent rows dequantized out of the pool, so they read the FP8-rounded latent +as the reference does; generation rows run FlashMLA's sparse fp8 decode over the pool itself. +Both kernels take the rank's 64 heads in one call. + +No path is selected by the serving configuration: each step chooses from its own sizes, and +every buffer is sized from the engine's quantities (the attention metadata's max_num_tokens, +max_num_requests, max_seq_len and runtime_features). The context rows take one of two paths, +chosen per step by the rank's own context KV (every context request's cached and new rows): +while it fits the held workspace, the sparse prefill over those rows dequantized into it; past +it, each context row as a one-row sequence of the sparse fp8 decode over the pool itself, as +generation rows always run. The held workspace, allocated on the first context step, has room +for the most context KV one step can attend, twice over, up to 2**19 rows +(`_ctx_workspace_rows`): a step whose context KV fits twice keeps layer 35's rows beside its +own for the NoPE layers attending over that latent, a longer one gathers them again for each +of those layers. An indexer chunk the engine packs past `max_seq_len` keys is scored request +by request, the indexer's logits are tiled under a budget from the engine's max_num_tokens and +max_seq_len (`_prefill_logits_budget`), each tile scored with the Triton FP8 MQA scorer +and selected with the self-sampling GVR +top-k (`_select_ctx`), and the expert path's calls take the world's `_EXPERT_CALL_TOKENS`, or +`_EXPERT_CALL_TOKENS_CHUNKED` under chunked prefill.""" + +from __future__ import annotations + +import itertools +import math +import os +import time +from dataclasses import replace +from types import SimpleNamespace +from typing import NamedTuple, Optional, cast + +import torch +from torch import nn +from transformers import PretrainedConfig + +from tensorrt_llm._torch.attention.backends.interface import AttentionMetadata +from tensorrt_llm._torch.attention.backends.sparse.params import ( + get_indexer_mqa_logits_workspace_bytes, +) +from tensorrt_llm._torch.attention.backends.trtllm import TrtllmAttentionMetadata +from tensorrt_llm._torch.model_config import ModelConfig +from tensorrt_llm._torch.modeling_v2._router_index import step_contract_enabled +from tensorrt_llm._torch.modeling_v2.catalog.activation.flashinfer_silu_and_mul import ( + flashinfer_silu_and_mul, +) +from tensorrt_llm._torch.modeling_v2.catalog.attention.thop_attention import thop_attention +from tensorrt_llm._torch.modeling_v2.catalog.comm.allgather import allgather +from tensorrt_llm._torch.modeling_v2.catalog.comm.reducescatter import reducescatter +from tensorrt_llm._torch.modeling_v2.catalog.gemm.bmm_out import bmm_out +from tensorrt_llm._torch.modeling_v2.catalog.gemm.cublas_mm import cublas_mm +from tensorrt_llm._torch.modeling_v2.catalog.moe.fp4_block_scale_moe_runner import ( + fp4_block_scale_moe_runner, +) +from tensorrt_llm._torch.modeling_v2.catalog.norm.flashinfer_fused_add_rmsnorm import ( + flashinfer_fused_add_rmsnorm, +) +from tensorrt_llm._torch.modeling_v2.catalog.norm.flashinfer_rmsnorm import flashinfer_rmsnorm +from tensorrt_llm._torch.modeling_v2.catalog.quantization.fp4_quantize import fp4_quantize +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.activation.gelu_erf import gelu_erf +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.activation.tanh_softcap import ( + tanh_softcap, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.convert_req_index_to_global import ( # noqa: E501 + convert_req_index_to_global, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_fp8_paged_mqa_logits import ( # noqa: E501 + cute_dsl_fp8_paged_mqa_logits, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_gvr_topk_prefill import ( # noqa: E501 + cute_dsl_gvr_topk_prefill, + gvr_warmup, + gvr_workspace, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_indexer_k_cache_update import ( # noqa: E501 + build as build_ix_k_cache, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_indexer_k_cache_update import ( # noqa: E501 + cute_dsl_indexer_k_cache_update, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_indexer_q_side_fused import ( # noqa: E501 + build as build_ix_q_side, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_indexer_q_side_fused import ( # noqa: E501 + cute_dsl_indexer_q_side_fused, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_radix_topk_decode import ( # noqa: E501 + build as build_ix_topk_decode, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_radix_topk_decode import ( # noqa: E501 + cute_dsl_radix_topk_decode, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ds_mla_dequant_gather import ( + ds_mla_dequant_gather, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ds_mla_rope_append import ( + ds_mla_rope_append, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.flash_mla_sparse_decode import ( + flash_mla_sparse_decode, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.flash_mla_sparse_prefill import ( + flash_mla_sparse_prefill, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.indexer_k_cache_gather_op import ( # noqa: E501 + indexer_k_cache_gather_op, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.indexer_topk_decode import ( + indexer_topk_decode, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ssmax_indexer_q_side_fused import ( # noqa: E501 + ssmax_indexer_q_side_fused, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ssmax_scaling import ( + ssmax_scaling, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ssmax_scaling_fused_q import ( + ssmax_scaling_fused_q, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.triton_fp8_mqa_logits import ( + triton_fp8_mqa_logits, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.gemm.cute_dsl_fp8_block_scale_linear import ( # noqa: E501 + cute_dsl_fp8_block_scale_linear, + prepare_weight_scale, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.moe.renorm_moe_routing_op import ( + renorm_moe_routing_op, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.norm.layer_norm import layer_norm +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.vision.pixtral_rope_2d import ( + pixtral_rope_2d, +) +from tensorrt_llm._torch.models.modeling_multimodal_utils import ( + find_input_mm_embeds, + get_multimodal_embeddings, +) +from tensorrt_llm._torch.models.modeling_utils import DecoderModel, DecoderModelForCausalLM +from tensorrt_llm._torch.pyexecutor.guided_decoder import CapturableGuidedDecoder +from tensorrt_llm._torch.speculative import get_spec_worker +from tensorrt_llm._utils import mpi_disabled +from tensorrt_llm.flash_mla import get_mla_metadata +from tensorrt_llm.inputs.multimodal import MultimodalParams +from tensorrt_llm.logger import logger +from tensorrt_llm.models.modeling_utils import QuantConfig + +from . import weights as _weights + +#: eps 1e-5, the reference's value; params.json's `vision_encoder` states none. +_VISION_EPS = 1e-5 + +#: The thop attention arguments the vision tower's call holds fixed: the no-KV-cache +#: mode (an encoder: the three cache-addressing arguments None), one context sequence +#: per image, the fused `[q | k | v]` rows already rotated (no rope in the op), +#: bidirectional within each sequence (mask_type 0), softmax scale head_dim^-0.5. +_VISION_ATTN = dict( + k=None, + v=None, + output_sf=None, + max_context_q_len_override=None, + kv_cache_block_offsets=None, + host_kv_cache_pool_pointers=None, + host_kv_cache_pool_mapping=None, + cache_indirection=None, + kv_scale_orig_quant=None, + kv_scale_quant_orig=None, + out_scale=None, + rotary_inv_freq=None, + rotary_cos_sin=None, + latent_cache=None, + q_pe=None, + block_ids_per_seq=None, + attention_sinks=None, + is_fused_qkv=True, + update_kv_cache=True, + predicted_tokens_per_seq=1, + local_layer_idx=0, + tokens_per_block=None, + beam_width=1, + mask_type=0, + quant_mode=0, + q_scaling=1.0, + position_embedding_type=0, + rope_dim=0, + rope_base=10000.0, + rope_scale_type=0, + rope_scale=1.0, + rope_short_m_scale=1.0, + rope_long_m_scale=1.0, + rope_max_positions=1024, + rope_original_max_positions=1024, + use_paged_context_fmha=False, + attention_input_type=0, + is_mla_enable=False, + chunked_prefill_buffer_batch_size=1, + q_lora_rank=None, + kv_lora_rank=None, + qk_nope_head_dim=None, + qk_rope_head_dim=None, + v_head_dim=None, + rope_append=None, + mrope_rotary_cos_sin=None, + mrope_position_deltas=None, + helix_position_offsets=None, + helix_is_inactive_rank=None, + attention_chunk_size=None, + softmax_stats_tensor=None, + is_spec_decoding_enabled=False, + use_spec_decoding=False, + is_spec_dec_tree=False, + spec_decoding_generation_lengths=None, + spec_decoding_position_offsets=None, + spec_decoding_packed_mask=None, + spec_decoding_bl_tree_mask_offset=None, + spec_decoding_bl_tree_mask=None, + spec_bl_tree_first_sparse_mask_offset_kv=None, + sparse_kv_indices=None, + sparse_kv_offsets=None, + sparse_attn_indices=None, + sparse_attn_offsets=None, + sparse_attn_indices_block_size=0, +) + +# The latent pool layout this target addresses (`MistralLarge4CacheManager.latent_layout`). +_LATENT_LAYOUT = "fp8_ds_mla" +# The query heads FlashMLA's sm100 sparse kernels run: under attention DP a rank's own 64. +_FLASH_MLA_HEADS = 64 +# The pre-routed MoE call: the runner's routing method is inert once ids and weights are +# given; 1 (renormalize) is the value its cells drive. +_MOE_ROUTING_METHOD = 1 +# The routed experts' SwiGLU, the runner's `act_type` 0. +_MOE_ACT_SWIGLU = 0 +# NVFP4: one e4m3 scale per 16 activations, laid out row-major (linear), the order the MoE +# runner reads; the 128x4-swizzled order is another buffer of the same length at 128-row +# multiples, and silently wrong there. +_NVFP4_SF_VEC = 16 +_NVFP4_SF_LINEAR = False +# triton_fp8_mqa_logits allocates a `[q_rows, kv_rows]` fp32 score matrix per call; the context +# queries are tiled to keep it under the engine's budget (`_prefill_logits_budget`). Context +# steps are not captured. +# capacity: fixed: the budget's ceiling (256 MiB): 64 rows at 1048576 keys. The engine's first +# context step leaves one tile's block cached, device memory held once up beside the KV pool: a +# 1 GiB tile (1 << 28) left a dep4 rank under the 2%-of-device free-memory margin at 0.9 of free +# memory to KV at max_seq_len 1048576. +_PREFILL_LOGITS_CEILING = 1 << 26 +# The decode indexer's top-k: True runs cute_dsl_radix_topk_decode, one launch that selects the +# trtllm indexer_topk_decode op's set (up to exact ties), False that op itself. The CuTe DSL +# selector serves scorer rows from 4096 columns (where the op splits each row) to 65536 (one +# CTA scans a whole row), the widths its cells cover; the scorer's column count is the engine's +# max_seq_len, so an engine outside that range keeps the op (`_derive_first_step`). +_TOPK_DECODE_CUTE_DSL = True +_TOPK_DECODE_COLUMNS = (4096, 65536) +# The attention block's glue as separate ops instead of the fused entries, for old-versus-new +# verification (both compute the same bits; the fused entries' tests hold each pair equal): +# ssmax_scaling plus the rope tail's copy or zero fill for ssmax_scaling_fused_q; +# ssmax_scaling, the head weights' fp32 copy and cute_dsl_indexer_q_side_fused for +# ssmax_indexer_q_side_fused; a fresh RMSNorm and torch.cat for the in-place latent; the copy +# out of FlashMLA's 64-head output for the in-place read. Set +# TRTLLM_MODELING_V2_UNFUSED_GLUE=1 in every rank's environment at launch; off by default. +# It sets the core's `unfused_glue`, which every site reads. +_UNFUSED_GLUE = os.environ.get("TRTLLM_MODELING_V2_UNFUSED_GLUE", "0") == "1" +# capacity: fixed: by attention-DP world size, the most gathered tokens one router / routing / +# NVFP4-quantize / expert call takes, the token column those entries are certified to at that +# world's expert window. At 4 ranks (32 local experts) the gathered set is up to +# 4 x max_num_tokens = 131072 rows, one call per MoE layer. At 8 ranks (16 local experts) it is +# up to 8 x max_num_tokens = 262144 rows, in calls of 32768: the call bounds the runner's +# workspace, whose FC2 rows at 32768 tokens and top-4 stay below 2^31 bytes. A larger gathered +# set runs in chunks of this many rows, whose boundaries depend on the gathered row count alone, +# the same on every rank. A world size not listed here is refused. +_EXPERT_CALL_TOKENS = {4: 131072, 8: 32768} +# capacity: fixed: by attention-DP world size, the calls of an engine under chunked prefill, +# whose steps attend a cached prefix up to max_seq_len and hold the large context workspace: +# 32768 at both, a token column the expert path's entries are certified to at that world's +# expert window. A call's workspace grows with its rows: at 4 ranks one call over all 131072 +# gathered rows leaves the engine about 5.6 GiB less free memory once up, more than max_seq_len +# 1048576's margin spares; so the 131072 rows run in four calls. The chunks give one call's bits +# (test_modeling_v2_routed_experts.py). +_EXPERT_CALL_TOKENS_CHUNKED = {4: 32768, 8: 32768} +# The context path's crossover, chosen once per step on this rank's host ints, and the held +# context workspace's ceiling (`_ctx_workspace_rows`). A step whose context KV (every context +# request's cached and new rows) fits the held workspace has it gathered, dequantized, into it +# (bf16 [rows, 576]) for FlashMLA's sparse prefill; a longer one attends its context rows +# straight off the pool with the sparse decode kernel, whose cost does not grow with the prefix +# as the gather's does. +# capacity: fixed: 2**19 rows, 576 MiB, the gather-vs-decode crossover. +_CTX_WORKSPACE_ROWS = 1 << 19 +# The most context rows one sparse-decode call takes on that path: FlashMLA's decode scheduler +# exits the process, raising nothing, past 11622 one-row sequences of 64 heads on sm_103. +# capacity: fixed: the kernel's scheduler, not an engine limit. +_CTX_DECODE_ROWS = 8192 + + +def _prefill_logits_budget(max_num_tokens: int, max_seq_len: int) -> int: + """The most fp32 logits one indexer prefill tile holds, from the engine's max_num_tokens and + max_seq_len: mainline's reachable MQA-logits tile (`get_indexer_mqa_logits_workspace_bytes`, + 4 bytes a logit; TLLM_INDEXER_MQA_LOGITS_ELEM_BUDGET lowers it), within + `_PREFILL_LOGITS_CEILING`, and within max_seq_len / 64 query rows at max_seq_len keys. + + The last bound grows the tile with the engine's longest request, which it scores in at + least 64 tiles: 2**24 logits (512 rows of 32768 keys) at max_seq_len 32768, the ceiling + (64 rows of 1048576 keys) from max_seq_len 65536 on. Those are the budgets the 32768 and + 1048576 serving configs were certified at, so their tiles, and the fp8_mqa_logits and + prefill top-k cells that cover them, stay as they were.""" + reachable = get_indexer_mqa_logits_workspace_bytes(max_num_tokens, max_seq_len) // 4 + return min(reachable, _PREFILL_LOGITS_CEILING, max_seq_len * (max_seq_len >> 6)) + + +def _most_ctx_kv(md) -> int: + """The most context KV (cached and new rows over a step's context requests) one step can + attend, from the attention metadata alone. `runtime_features` is built once by the executor + for the service (chunked prefill from enable_chunked_prefill, cache reuse from the + KvCacheConfig's enable_block_reuse). With neither, every context row is new: + max_num_tokens. Otherwise mainline's never-stall worst case for a workspace chunked prefill + does not bound (`get_mla_context_workspace_kv_len_cap` without an + fp8_context_mla_kv_len_cap override; the gather reads every attended row, cached ones + included): min(max_num_requests, max_num_tokens) * max_seq_len. The target does not read + fp8_context_mla_kv_len_cap: the KvCacheConfig is not reachable from the model, and for these + targets the engine installs no admission cap that would bound a step to it.""" + features = md.runtime_features + if not (features.chunked_prefill or features.cache_reuse): + return md.max_num_tokens + return min(md.max_num_requests, md.max_num_tokens) * md.max_seq_len + + +def _ctx_workspace_rows(md) -> int: + """The held context workspace's rows: the most context KV one step can attend + (`_most_ctx_kv`) twice over, a step's rows and a shared owner's (layer 35's) kept beside + them, up to `_CTX_WORKSPACE_ROWS`. At max_num_tokens 32768 with neither chunked prefill nor + block reuse that is 65536 rows, the two `[32768, 576]` gathers a step there makes; with either, + at 16 requests of max_seq_len 32768 or more, 2**19.""" + return min(2 * _most_ctx_kv(md), _CTX_WORKSPACE_ROWS) + + +def _step_common(md: TrtllmAttentionMetadata, indexer_layers: list[int]) -> dict: + """The batch state both targets read: KV lengths and pool addressing for the MLA ops, the + indexer's key pools and slot mappings, the top-k conversion tables, the decode scorer's + schedule, and every rank's token count this step (attention DP: identical on every rank, + in rank order, unpadded). The indexer pool views are taken per step: the engine replaces its + KV cache manager during boot, and a view of the old one writes into a dead allocation.""" + mgr = md.kv_cache_manager + latent_pool = mgr.get_latent_pool() + return dict( + num_seqs=md.num_seqs, + num_tokens=md.num_tokens, + all_rank_num_tokens=[int(n) for n in md.all_rank_num_tokens], + kv_lens_cuda=md.kv_lens_cuda_runtime, + kv_lens_host=md.kv_lens_runtime, + prompt_lens_host=md.prompt_lens_cpu_runtime, + kv_cache_block_offsets=md.kv_cache_block_offsets, + host_kv_cache_pool_pointers=md.host_kv_cache_pool_pointers, + host_kv_cache_pool_mapping=md.host_kv_cache_pool_mapping, + tokens_per_block=md.tokens_per_block, + max_seq_len=md.max_seq_len, + # [blocks, owners, tokens_per_block, 656] bytes, and the same bytes as FlashMLA's + # pages: page `block * owners + row`, slot `page * tokens_per_block + offset`. + latent_pool=latent_pool, + latent_kv=latent_pool.view(-1, latent_pool.shape[2], 1, latent_pool.shape[3]), + ix_pools={i: mgr.get_indexer_k_cache_buffers(i) for i in indexer_layers}, + block_table=md.block_table, + req_idx_per_token=md.req_idx_per_token, + slot_mapping_fp8=md.slot_mapping_fp8, + slot_mapping_scale=md.slot_mapping_scale, + skip_gen=bool(md.skip_indexer_for_gen_reqs), + # Allocated only while the engine's short-sequence indexer skip is on; the two skip + # flags are False whenever it is off. + topk_indices_buffer=md.topk_indices_buffer if md.enable_indexer_skip else None, + indexer_k_cache_block_offsets=md.indexer_k_cache_block_offsets, + scheduler_metadata_buffer=md.scheduler_metadata_buffer, + radix_aux_indices=md.radix_aux_indices, + radix_aux_logits=md.radix_aux_logits, + ) + + +def _padded_rows(core: "MistralLarge4DepModelingV2", step: dict, rows: int) -> int: + """The row count every rank pads its block to before the expert path's collectives: the + largest rank's count this step. Every rank reads the same list, so every rank's collectives + move the same bytes; this rank's own entry must be the rows it holds, or its block would + be cut or overrun.""" + counts = step["all_rank_num_tokens"] + assert len(counts) == len(core.dp_group) and counts[core.dp_rank] == rows, ( + f"rank {core.dp_rank} holds {rows} rows; all_rank_num_tokens is {counts}" + ) + return max(counts) + + +def _padding_rows(counts: list[int], padded: int, device) -> torch.Tensor | None: + """Which of the gathered rows are padding: bool `[len(counts) * padded]` on `device`, or None + when no rank pads (every count is `padded`). Rank r's segment is rows `[r * padded, + (r + 1) * padded)`, its first `counts[r]` its own and the rest the zero rows `_moe` pads it + with, at `[r * padded + counts[r], (r + 1) * padded)`. Built from the step's host counts.""" + if all(n == padded for n in counts): + return None + own = torch.tensor(counts, dtype=torch.int64, device=device) + return (torch.arange(padded, device=device)[None, :] >= own[:, None]).reshape(-1) + + +def _ctx_kv_offsets(md: TrtllmAttentionMetadata, num_contexts: int) -> dict: + """The context rope/append op's per-request offsets: int64 prefix sums of the cached and + of the total (cached + new) lengths over the context requests, and the longest new run. + + The engine keeps them only under `enable_context_mla_with_cached_kv` (KV reuse or chunked + prefill). Without either, no context request has a cached prefix, so the cached sums are + zero and the totals are the new lengths: the int32 prefix sum of `seq_lens_cuda` the + metadata always builds, widened to the op's int64. + """ + if md.enable_context_mla_with_cached_kv: + return dict( + cu_ctx_cached=md.ctx_cached_token_indptr[: num_contexts + 1], + cu_ctx_kv=md.ctx_kv_indptr[: num_contexts + 1], + max_ctx_new=int(md.max_ctx_seq_len), + ctx_cached_tokens=int(md.num_ctx_cached_tokens), + ) + cu = md.mla_prepare_ctx_cu_seqlens() + return dict( + cu_ctx_cached=torch.zeros(num_contexts + 1, dtype=torch.int64, device=cu.device), + cu_ctx_kv=cu.to(torch.int64), + max_ctx_new=int(md.seq_lens[:num_contexts].max()), + ctx_cached_tokens=0, + ) + + +def _ctx_host_offsets(md: TrtllmAttentionMetadata, num_contexts: int) -> dict: + """The host lists over the context requests: the prefix sums of the new and of the total + (cached + new) lengths, which place each context request's rows in the step and its keys in + the indexer's full-KV gather mappings (`_prefill_blocks`). + + The engine's host copies under `enable_context_mla_with_cached_kv`; without it no request + has a cached prefix, so both are the prefix sum of the host `seq_lens`. Every source is a + host tensor the metadata fills on the host, so taking them each context step syncs nothing. + """ + if md.enable_context_mla_with_cached_kv: + return dict( + host_cu_ctx_new=md.host_ctx_uncached_token_indptr[: num_contexts + 1].tolist(), + host_cu_ctx_kv=md.host_ctx_kv_indptr[: num_contexts + 1].tolist(), + ) + host_cu = [0, *itertools.accumulate(md.seq_lens[:num_contexts].tolist())] + return dict(host_cu_ctx_new=host_cu, host_cu_ctx_kv=host_cu) + + +def _prefill_blocks( + chunk, cu_new: list[int], cu_kv: list[int], max_keys: int +) -> list[tuple[int, int, int, int]]: + """The indexer's scoring blocks over one engine prefill chunk, as host ints + `(token_start, token_end, k_token_start, n_k)`: rows `[token_start, token_end)` of the + step's context rows scored against keys `[k_token_start, k_token_start + n_k)` of its + full-KV gather mappings. + + One block, the chunk itself, while it holds at most `max_keys` keys (max_seq_len, the most + one request holds). The engine packs requests into one chunk while (summed new tokens) x + (summed KV) stays within its packed-logits budget, so a few short last chunks over long + prefixes pack past max_seq_len (two 5-token chunks over 983040 cached keys each: 1966090 + keys). Such a chunk is cut at its request boundaries, each request's rows against its own + keys: a row's window lies inside its own request's keys. `cu_new` / `cu_kv` are the + step's prefix sums of the new and of the cached + new tokens over its context requests, + in the order of the context rows and of the full-KV mappings.""" + n_k = chunk.k_token_end - chunk.k_token_start + if n_k <= max_keys: + return [(chunk.token_start, chunk.token_end, chunk.k_token_start, n_k)] + blocks = [] + for r in range(len(cu_new) - 1): + lo, hi = max(chunk.token_start, cu_new[r]), min(chunk.token_end, cu_new[r + 1]) + if lo >= hi: + continue + k_lo, k_hi = max(chunk.k_token_start, cu_kv[r]), min(chunk.k_token_end, cu_kv[r + 1]) + assert 0 < k_hi - k_lo <= max_keys, ( + f"context request {r} of an indexer chunk holds keys [{k_lo}, {k_hi}), past " + f"max_seq_len {max_keys}" + ) + blocks.append((lo, hi, k_lo, k_hi - k_lo)) + assert blocks and blocks[0][0] == chunk.token_start and blocks[-1][1] == chunk.token_end, ( + f"the requests {cu_new} do not tile the chunk's rows " + f"[{chunk.token_start}, {chunk.token_end})" + ) + return blocks + + +def _yarn_lerope_inv_freq(cfg, alpha: torch.Tensor) -> torch.Tensor: + """The rotary inverse frequencies, fp32, as the reference implementation builds + them: YaRN's blend of the extrapolated and the interpolated frequency, both scaled by + LeRoPE's `exp(alpha)`, over a ramp taken on the frequencies without LeRoPE.""" + rope = cfg.rope_parameters + dim = cfg.qk_rope_head_dim + theta = float(rope["rope_theta"]) + factor = float(rope["factor"]) + original = float(rope["original_max_position_embeddings"]) + pos_freqs = theta ** (torch.arange(0, dim, 2, dtype=torch.float32) / dim) + lerope = torch.exp(alpha.detach().to(device="cpu", dtype=torch.float32)) + extrapolated = 1.0 / pos_freqs * lerope + interpolated = 1.0 / (factor * pos_freqs) * lerope + + def correction_dim(rotations: float) -> float: + return dim * math.log(original / (rotations * 2 * math.pi)) / (2 * math.log(theta)) + + low = max(math.floor(correction_dim(float(rope.get("beta_fast", 32)))), 0) + high = min(math.ceil(correction_dim(float(rope.get("beta_slow", 1)))), dim - 1) + if low == high: + high += 0.001 + ramp = ((torch.arange(dim // 2, dtype=torch.float32) - low) / (high - low)).clamp(0, 1) + extrapolation = 1 - ramp + return interpolated * (1 - extrapolation) + extrapolated * extrapolation + + +def _rope_tables(cfg, alpha: torch.Tensor, positions: int, dtype, device) -> SimpleNamespace: + """The tables the rope consumers read, built from one fp32 angle grid and rounded to the + activation dtype first, as the reference's cos/sin cache is held. + + `indexer` is `[rows, 64]`, `[cos(32) | sin(32)]`: the layout the indexer q side and + `ds_mla_rope_append` (the MLA query tail and the latent's rope key) read; the indexer key + side reads the same storage as `[rows, 2, 32]`. `identity` (cos 1, sin 0) is the table of a + NoPE indexer layer, whose kernels have no no-rope mode. Every row depends only on its own + index, so a longer table is the same table with more rows. + """ + inv = _yarn_lerope_inv_freq(cfg, alpha) + t = torch.arange(positions, dtype=torch.float32) + angle = torch.einsum("i,j -> ij", t, inv) + cos = angle.cos().to(dtype).float() + sin = angle.sin().to(dtype).float() + identity = torch.cat([torch.ones_like(cos), torch.zeros_like(sin)], dim=1) + return SimpleNamespace( + rows=positions, + indexer=torch.cat([cos, sin], dim=1).to(device).contiguous(), + identity=identity.to(device).contiguous(), + ) + + +def _moe_scalars(isc1, isc2, ws2_13, ws2_2) -> SimpleNamespace: + """The NVFP4 MoE's call scalars from the checkpoint's per-tensor ones, fp32. + + ModelOpt stores `input_scale` and `weight_scale_2` as the reciprocals of the global scales a + quantizer divides by (`amax / (448 * 6)`), one `isc1` / `isc2` per layer and one + `ws2_13[e]` / `ws2_2[e]` per local expert. So the activation quantizer's global scale is + `1 / isc1`; FC1's accumulator dequantizes by `isc1 * ws2_13[e]` (the gate half, + `output1_scale_gate_scalar`) and, because the FC1 epilogue requantizes the SwiGLU output to + NVFP4 under `1 / isc2`, the linear half by `isc1 * ws2_13[e] / isc2` + (`output1_scale_scalar`); FC2's accumulator dequantizes by `isc2 * ws2_2[e]` + (`output2_scale_scalar`). The reference's TRT-LLM-Gen path folds the same three + (`g1_alphas`, `g1_scale_c`, `g2_alphas`).""" + gate1 = (isc1 * ws2_13).contiguous() + return SimpleNamespace( + gscale=(1.0 / isc1).contiguous(), + out1=(gate1 / isc2).contiguous(), + out1_gate=gate1, + out2=(isc2 * ws2_2).contiguous(), + ) + + +def _layer_views(core: "MistralLarge4DepModelingV2") -> list[SimpleNamespace]: + """Every layer's operands, as the views the entries read: GEMM operands transposed (`.t()`, + the column-major `[K, N]` cuBLAS operand), FP8 scales converted once to + the Rubin R128c4 K32 layout, `wkv_b` split into the absorption pair over all 64 heads, and the + NVFP4 MoE's call scalars (`_moe_scalars`). `next_norm` is the next layer's input norm, the + final norm after the last layer. A sharer's absorption pair is its own `wkv_b`; its latent is + its source's.""" + w = core.w + heads, v_dim, kv_lora = core.heads, core.v_dim, core.kv_lora + layers = [] + for i in range(core.num_layers): + p = f"l{i}_" + nope = core.qk_dim if core.nope_layer[i] else core.nope + kvb = w[p + "wkv_b"] + hn = heads * nope + owner = core.kv_source[i] == i + ix = None + if core.full_indexer[i]: + ix = SimpleNamespace( + q_norm_w=w[p + "ix_q_norm_w"], + q_norm_b=w[p + "ix_q_norm_b"], + wq_b=w[p + "ix_wq_b"], + wq_b_s=prepare_weight_scale(w[p + "ix_wq_b_s"].t(), w[p + "ix_wq_b"].shape[1]), + wk_wp=w[p + "ix_wk_wp"].t(), + k_norm_w=w[p + "ix_k_norm_w"], + k_norm_b=w[p + "ix_k_norm_b"], + ssmax_s=w[p + "ix_ssmax_s"], + ssmax_b=w[p + "ix_ssmax_b"], + rope=not core.indexer_nope[i], + ) + moe = None + if i >= core.dense_layers: + moe = SimpleNamespace( + router=w[p + "router"].t(), + fc1_w=w[p + "fc1_w"], + fc1_s=w[p + "fc1_s"], + fc2_w=w[p + "fc2_w"], + fc2_s=w[p + "fc2_s"], + **vars(_moe_scalars(w[p + "isc1"], w[p + "isc2"], w[p + "ws2_13"], w[p + "ws2_2"])), + ) + layers.append( + SimpleNamespace( + index=i, + rope=not core.nope_layer[i], + nope=nope, + owner=owner, + kv_source=core.kv_source[i], + attn_norm=w[p + "attn_norm"], + wq=w[p + "wq"], + wq_s=prepare_weight_scale(w[p + "wq_s"].t(), w[p + "wq"].shape[1]), + ssmax_s=w[p + "ssmax_s"], + ssmax_b=w[p + "ssmax_b"], + wkv_a=w[p + "wkv_a"].t() if owner else None, + kv_norm=w[p + "kv_a_norm"] if owner else None, + k_b=kvb[:hn].reshape(heads, nope, kv_lora), + v_b_t=torch.transpose(kvb[hn:].reshape(heads, v_dim, kv_lora), 1, 2), + wo=w[p + "wo"], + wo_s=prepare_weight_scale(w[p + "wo_s"].t(), w[p + "wo"].shape[1]), + ffn_norm=w[p + "ffn_norm"], + mlp_gu=w[p + "mlp_gu"], + mlp_gu_s=prepare_weight_scale(w[p + "mlp_gu_s"].t(), w[p + "mlp_gu"].shape[1]), + mlp_dn=w[p + "mlp_dn"], + mlp_dn_s=prepare_weight_scale(w[p + "mlp_dn_s"].t(), w[p + "mlp_dn"].shape[1]), + indexer=ix, + moe=moe, + next_norm=( + w[f"l{i + 1}_attn_norm"] if i + 1 < core.num_layers else w["final_norm"] + ), + ) + ) + return layers + + +class MistralLarge4DepModelingV2(DecoderModel): + def __init__(self, model_config: ModelConfig): + super().__init__(model_config) + cfg = model_config.pretrained_config + mapping = model_config.mapping + # The expert path's communicator: the attention-DP ranks, four on one node or eight over + # both nodes, in the order `all_rank_num_tokens` lists them and the gather concatenates + # them. + self.dp_group = list(mapping.tp_group) + self.dp_rank = mapping.tp_rank + # The attention block's glue: the fused entries (ssmax_scaling_fused_q, + # ssmax_indexer_q_side_fused, the latent RMSNorm in place, FlashMLA's decode output + # read in place) at every max_seq_len, whose cells cover the 32K and 1M configs' + # positions (to 1048575), rope tables and step shapes; the separate ops they replace + # under `_UNFUSED_GLUE`, at every site. + self.unfused_glue = _UNFUSED_GLUE + world = mapping.world_size + if world not in _EXPERT_CALL_TOKENS or len(self.dp_group) != world: + raise ValueError( + f"modeling_v2 Mistral Large 4 NVFP4 attention-DP target: world_size {world} with an " + f"attention-DP group of {len(self.dp_group)} ranks; it serves 4 ranks on one " + "node or 8 over two nodes of four, every rank in the one group" + ) + + self.num_layers = cfg.num_hidden_layers + self.hidden = cfg.hidden_size + self.eps = cfg.rms_norm_eps + # Attention DP: every rank runs every head on its own requests. + self.heads = cfg.num_attention_heads + self.nope = cfg.qk_nope_head_dim + self.rope_dim = cfg.qk_rope_head_dim + # A NoPE layer spends the rope dims on nope: the query head stays this wide. + self.qk_dim = self.nope + self.rope_dim + self.v_dim = cfg.v_head_dim + self.kv_lora = cfg.kv_lora_rank + self.lat_dim = self.kv_lora + self.rope_dim + # Every layer's softmax scale, NoPE layers included: qk_head_dim**-0.5, no YaRN + # temperature (its magnitude is 1). + self.softmax_scale = self.qk_dim**-0.5 + + self.nope_layer = [bool(f) for f in cfg.nope_layer_flags] + self.indexer_nope = [bool(f) for f in cfg.indexer_nope_layer_flags] + self.kv_source = [int(s) for s in cfg.kv_source_layers] + # An owner some later layer attends over (layer 35): a step whose context KV fits the + # held workspace twice keeps its gather there for the sharers; a longer one has each + # sharer gather its source's rows again (`_ctx_workspace`). + self.shared_source = [ + any(s == i and j != i for j, s in enumerate(self.kv_source)) + for i in range(self.num_layers) + ] + self.full_indexer = [role == "full" for role in cfg.index_sharing] + # A 'shared' layer attends with the most recent full layer's selection: each target + # keeps one selection buffer per step, which every full layer overwrites in order. + self.indexer_layers = [i for i in range(self.num_layers) if self.full_indexer[i]] + + self.idx_heads = cfg.index_n_heads + self.idx_dim = cfg.index_head_dim + self.idx_topk = cfg.index_topk + self.idx_eps = cfg.indexer_norm_eps + # The indexer's softmax scale and head average: its scorers take none, so the q side + # folds both into the head weights together with the q quantization scale. + self.idx_weight_scale = self.idx_dim**-0.5 * self.idx_heads**-0.5 + + self.dense_layers = cfg.first_k_dense_replace + self.num_experts = cfg.n_routed_experts + self.topk = cfg.num_experts_per_tok + self.moe_inter = cfg.moe_intermediate_size + self.local_experts = self.num_experts // mapping.moe_ep_size + self.expert_offset = self.local_experts * mapping.moe_ep_rank + # The rope tables' first size; grown on the first step to the engine's max_seq_len. + self.rope_rows = model_config.max_seq_len or cfg.max_position_embeddings + + self.w = _weights.MODEL_WEIGHTS.declare(_weights.MODEL_WEIGHTS.dims(self)) + + self._rope: SimpleNamespace | None = None + self._derived_for = None + self._ix_k = None + self._ix_k_checked = False + self._pool_row: list[int] = [] + self._stride_factor = 0 + self._max_ctx_len = 0 + # The engine's sizes, fixed on the first step (`_derive_first_step`): the indexer's + # prefill logits budget, the held context workspace's rows, the most gathered rows one + # expert-path call takes and the MoE autotuner's largest bucket. + self.prefill_logits_budget = 0 + self.ctx_workspace_rows = 0 + self.expert_call_tokens = 0 + self.expert_tune_tokens = 0 + # The decode top-k's handle, built on the first step at a scorer width it serves. + self._ix_topk = None + self._side_stream: torch.cuda.Stream | None = None + self.dp_pg = None + self._prefill = self._decode = None + + def post_load_weights(self) -> None: + """Derive what the weights fix, build the kernels, the shared expert's stream and the + communicator handle, then the targets. + + Named for the shell hook that drives it. Everything compiles here or on the first + (eager) step: a first compile inside a CUDA-graph capture breaks the graph. The side + stream is made here too: a stream created inside a capture is not capturable. + """ + dev = torch.cuda.current_device() + self.build_kernels(dev) + self._side_stream = torch.cuda.Stream(device=dev) + # The collectives' communicator flavor, as the engine's own collectives pick it: the + # MPI session's, or the tensor-parallel process group's when MPI is disabled. + mapping = self.model_config.mapping + self.dp_pg = mapping.tp_group_pg.boxed() if mpi_disabled() else None + self._prefill = PrefillTarget(self) + self._decode = DecodeTarget(self) + + def build_kernels(self, dev: int) -> None: + """The rank-local derivations: the rope tables (and, for the unfused glue, the indexer + q side).""" + self._rope = _rope_tables( + self.model_config.pretrained_config, + self.w["lerope_alpha"], + self.rope_rows, + self.w["final_norm"].dtype, + self.w["final_norm"].device, + ) + if self.unfused_glue: + # The checkpoint's rope is interleaved (GPT-J) for the indexer too. + self._ix_q = build_ix_q_side(dev, is_neox=False) + + def _derive_first_step(self, md) -> None: + """What depends on values the engine resolves, fixed on the first step it hands over + (eager, ahead of any capture) and again for a replaced KV cache manager.""" + if md.kv_cache_manager is self._derived_for: + return + dev = torch.cuda.current_device() + mgr = md.kv_cache_manager + # The pool layout is the KV cache manager's, which routing cannot see; every latent + # op here reads and writes fp8_ds_mla rows addressed by owner row. + if getattr(mgr, "latent_layout", None) != _LATENT_LAYOUT: + raise RuntimeError( + f"this target attends over an {_LATENT_LAYOUT} latent pool with owner-only " + f"pages (MistralLarge4CacheManager); the engine built {type(mgr).__name__} " + f"with layout {getattr(mgr, 'latent_layout', None)!r}. Use KV cache manager V2 " + "and kv_cache_config.dtype auto, fp8 or fp8_ds_mla." + ) + if list(mgr.kv_source_layers) != self.kv_source: + raise RuntimeError("the KV cache manager's kv_source_layers differ from the config's") + # A table shorter than max_seq_len is read past its end with no check. + if md.max_seq_len > self._rope.rows: + self.rope_rows = md.max_seq_len + self._rope = _rope_tables( + self.model_config.pretrained_config, + self.w["lerope_alpha"], + self.rope_rows, + self.w["final_norm"].dtype, + self.w["final_norm"].device, + ) + # Layer i reads owner row `_pool_row[i]` (its source's); a block holds every owner's + # page, so a global row is `block * owners * tpb + row * tpb + offset`. + self._pool_row = [mgr.latent_pool_row(i) for i in range(self.num_layers)] + self._stride_factor = len(mgr.kv_owner_layers) * md.tokens_per_block + # The decode scorer's column count: frozen into a capture, so an engine constant. + self._max_ctx_len = md.max_seq_len + # The indexer's prefill logits budget and the held context workspace's rows, from the + # engine's max_num_tokens, max_num_requests, max_seq_len and runtime features. + self.prefill_logits_budget = _prefill_logits_budget(md.max_num_tokens, md.max_seq_len) + self.ctx_workspace_rows = _ctx_workspace_rows(md) + # The expert path's calls at this world size: the smaller ones under chunked prefill, + # the engines whose steps attend a cached prefix up to max_seq_len and hold the large + # context workspace. + calls = ( + _EXPERT_CALL_TOKENS_CHUNKED + if md.runtime_features.chunked_prefill + else _EXPERT_CALL_TOKENS + ) + self.expert_call_tokens = calls[len(self.dp_group)] + # capacity: max_num_tokens -- the MoE autotuner's largest bucket, the most rows one + # expert call holds: every rank's max_num_tokens gathered, capped at one call. + self.expert_tune_tokens = min( + self.expert_call_tokens, len(self.dp_group) * md.max_num_tokens + ) + logger.info( + f"modeling_v2 Mistral Large 4 NVFP4: sized from the engine: prefill logits budget " + f"{self.prefill_logits_budget}, context workspace {self.ctx_workspace_rows} rows, " + f"expert calls of {self.expert_call_tokens} rows" + ) + # The decode selector, by the scorer's column count: cute_dsl_radix_topk_decode's + # handle (built once) at a width it serves, else none and the trtllm op selects. + lo, hi = _TOPK_DECODE_COLUMNS + if not (_TOPK_DECODE_CUTE_DSL and lo <= self._max_ctx_len <= hi): + self._ix_topk = None + elif self._ix_topk is None: + self._ix_topk = build_ix_topk_decode(dev, next_n=1) + # Both key-pool layouts: a launcher compiled for a contiguous view addresses a strided + # one as if it were dense. + if self._ix_k is None: + self._ix_k = { + contiguous: build_ix_k_cache(dev, is_neox=False, cache_contiguous=contiguous) + for contiguous in (False, True) + } + self._derived_for = md.kv_cache_manager + + def _select_target(self, attn_metadata) -> "PrefillTarget | DecodeTarget": + """Which target runs this step: one host-side integer read. A mixed batch goes to + prefill, so prefill is the general case and decode the specialization that may assume + it has no context rows. The ranks' choices differ in one step and need not agree: both + targets issue the same collectives.""" + return self._decode if attn_metadata.num_contexts == 0 else self._prefill + + def forward( + self, + attn_metadata: AttentionMetadata, + input_ids: torch.IntTensor | None = None, + position_ids: torch.IntTensor | None = None, + inputs_embeds: torch.FloatTensor | None = None, + lora_params: dict | None = None, + **kwargs, + ) -> torch.Tensor: + """Derive what the first step fixes, then route.""" + self._derive_first_step(attn_metadata) + logger.debug( + f"modeling_v2 Mistral Large 4 NVFP4 attention-DP rank {self.dp_rank}: all_rank_num_tokens " + f"{attn_metadata.all_rank_num_tokens}, num_contexts {attn_metadata.num_contexts}" + ) + return self._select_target(attn_metadata).forward( + attn_metadata, + input_ids=input_ids, + position_ids=position_ids, + inputs_embeds=inputs_embeds, + lora_params=lora_params, + **kwargs, + ) + + +class PrefillTarget: + """The general case: context rows, and possibly generation rows behind them.""" + + # The held context workspace, bf16 [core.ctx_workspace_rows, 576]: allocated on the first + # context step and held from then on (`_ctx_workspace`). + _ws_buf: torch.Tensor | None = None + # The indexer's prefill top-k scratch (`gvr_workspace`, 20 MiB): allocated on the first + # context step that selects and held, used on the main stream only (`_select_ctx`). + _gvr_ws: torch.Tensor | None = None + + def __init__(self, core: MistralLarge4DepModelingV2) -> None: + """Take the per-layer views once, off the real weights. Everything an engine value + fixes is read off `self.core` at the call.""" + self.core = core + # Off unless TRTLLM_MODELING_V2_VALIDATE asks for it. Each target owns its own flag: + # the two `step_args` projections differ, so probing one does not prove the other. + self._contract_pending = step_contract_enabled() + self._layers = _layer_views(core) + self._embed = core.w["embed"] + self._norm0 = core.w["l0_attn_norm"] + + def _check_step_contract(self, md) -> None: + """Opt-in first-forward fail-fast: calling `step_args` reads every metadata field this + target consumes (private trtllm surface). Fixed at engine construction, so once per + target instance, and off in a served engine.""" + if not self._contract_pending: + return + try: + self.step_args(md) + except AttributeError as exc: + raise AssertionError(f"attention metadata surface drifted: {exc}") from exc + self._contract_pending = False + + def step_args(self, md: TrtllmAttentionMetadata) -> dict: + """The common batch state, plus the context rows': their counts and lengths, the + indexer's prefill windows and key gather mappings, and the context requests' KV + offsets the context workspace is laid out by and their host prefix sums.""" + step = _step_common(md, self.core.indexer_layers) + ctx = md.num_contexts + # The engine splits the indexer's prefill queries across ranks only without attention + # DP; under it each rank's chunks cover every context row of its own requests, and a + # split would leave the rows outside this rank's share unselected. + assert md.indexer_prefill_q_split is None, ( + f"the indexer's prefill queries are split across ranks " + f"({md.indexer_prefill_q_split}); this target selects every context row itself" + ) + step.update( + num_contexts=ctx, + num_ctx_tokens=md.num_ctx_tokens, + seq_lens=md.seq_lens, + skip_ctx=bool(md.skip_indexer_for_ctx_reqs), + cu_seqlen_ks=md.cu_seqlen_ks, + cu_seqlen_ke=md.cu_seqlen_ke, + indexer_prefill_chunks=md.indexer_prefill_chunks, + num_ctx_kv_tokens=md.num_ctx_kv_tokens, + slot_mapping_fp8_fullkv=md.slot_mapping_fp8_fullkv, + slot_mapping_scale_fullkv=md.slot_mapping_scale_fullkv, + **_ctx_kv_offsets(md, ctx), + **_ctx_host_offsets(md, ctx), + ) + return step + + # ---- one layer's pieces, context and generation rows ---- + + def _linear_fp8(self, x, w, w_s): + """K128-quantized FP8 linear through the Rubin MXFP8 kernel.""" + return cute_dsl_fp8_block_scale_linear(x, w, w_s) + + def _latent(self, x, L, nope_lat): + """`[RMSNorm(c_kv) | k_pe]`, 576 wide: a RoPE owner's kv_a row, normalized in place; a + NoPE owner's 512-wide latent normalized into `nope_lat`, whose rope key stays zero.""" + core = self.core + kv = cublas_mm(x, L.wkv_a) + if core.unfused_glue: + c_kv = flashinfer_rmsnorm(kv[:, : core.kv_lora], L.kv_norm, core.eps) + if L.rope: + k_pe = kv[:, core.kv_lora :] + else: + k_pe = torch.zeros([kv.shape[0], core.rope_dim], dtype=kv.dtype, device=kv.device) + return torch.cat([c_kv, k_pe], dim=-1) + if L.rope: + c_kv = kv[:, : core.kv_lora] + flashinfer_rmsnorm(c_kv, L.kv_norm, core.eps, out=c_kv) + return kv + flashinfer_rmsnorm(kv, L.kv_norm, core.eps, out=nope_lat[:, : core.kv_lora]) + return nope_lat + + def _indexer(self, step, x, pos, L, topk_buf, tc: int, skip_ctx: bool, skip_gen: bool): + """Append this step's keys to the layer's key pool; select `index_topk` positions per + row into `topk_buf` for the rows whose phase the engine did not skip.""" + core = self.core + ix = L.indexer + rows = x.shape[0] + table = core._rope.indexer if ix.rope else core._rope.identity + kw = cublas_mm(x, ix.wk_wp) + k = kw[:, : core.idx_dim].contiguous() + k_pool = step["ix_pools"][L.index] + k_pool = cute_dsl_indexer_k_cache_update( + k, ix.k_norm_w, ix.k_norm_b, pos, table.view(-1, 2, core.rope_dim // 2), k_pool, + step["slot_mapping_fp8"], step["slot_mapping_scale"], rows, + is_neox=False, compiled=core._ix_k[k_pool.is_contiguous()], + # Its range check is a host sync: once, on the first (eager) call. + validate=not core._ix_k_checked, + ) # fmt: skip + core._ix_k_checked = True + if skip_ctx and skip_gen: + return + qn = layer_norm(x, ix.q_norm_w, ix.q_norm_b, core.idx_eps) + q = torch.reshape(self._linear_fp8(qn, ix.wq_b, ix.wq_b_s), [rows, core.idx_heads, -1]) + q_fp8 = torch.empty_like(q, dtype=torch.float8_e4m3fn) + weights = torch.empty([rows, core.idx_heads], dtype=torch.float32, device=x.device) + if core.unfused_glue: + q = ssmax_scaling(q, pos, ix.ssmax_s, ix.ssmax_b) + q_fp8, weights = cute_dsl_indexer_q_side_fused( + q, pos, table, kw[:, core.idx_dim :].float().contiguous(), core.idx_weight_scale, + q_fp8, weights, is_neox=False, compiled=core._ix_q, + ) # fmt: skip + else: + # SSMax, rope, e4m3 and the head weights (read in place out of `kw`) in one launch. + q_fp8, weights = ssmax_indexer_q_side_fused( + q, pos, ix.ssmax_s, ix.ssmax_b, table, kw[:, core.idx_dim :], + core.idx_weight_scale, q_fp8, weights, + ) # fmt: skip + if tc and not skip_ctx: + self._select_ctx(step, k_pool, q_fp8, weights, topk_buf, tc) + gen = rows - tc + if gen and not skip_gen: + ctx = step["num_contexts"] + num_gens = step["num_seqs"] - ctx + assert gen % num_gens == 0, (gen, num_gens) + next_n = gen // num_gens + kv_lens = step["kv_lens_cuda"][ctx : ctx + num_gens] + logits = cute_dsl_fp8_paged_mqa_logits( + torch.reshape(q_fp8[tc:], [num_gens, next_n, core.idx_heads, core.idx_dim]), + k_pool, + weights[tc:], + kv_lens, + step["indexer_k_cache_block_offsets"][ctx : ctx + num_gens], + step["scheduler_metadata_buffer"], + core._max_ctx_len, + ) + if next_n != 1 or core._ix_topk is None: + indexer_topk_decode( + logits, kv_lens, topk_buf[tc:], next_n, core.idx_topk, 1, + step["radix_aux_indices"], step["radix_aux_logits"], + ) # fmt: skip + else: + cute_dsl_radix_topk_decode( + logits, kv_lens, topk_buf[tc:], next_n=1, compiled=core._ix_topk + ) + + def _ix_plan(self, step, tc: int) -> list[SimpleNamespace]: + """Plan the context rows and key windows scored by each query tile. + + The engine's chunk and query-split mappings determine each block's rows, + keys and window offsets. Tiles stay within the prefill logits budget. + The plan is reused across indexer layers within this step.""" + core = self.core + chunks = step["indexer_prefill_chunks"] + if chunks is None: + # One chunk over every context row: the engine plans chunks only past its chunk + # size, or under KV reuse or chunked prefill. + chunks = [ + SimpleNamespace( + token_start=0, + token_end=tc, + k_token_start=0, + k_token_end=step["num_ctx_kv_tokens"], + cu_seqlen_ks=step["cu_seqlen_ks"][:tc], + cu_seqlen_ke=step["cu_seqlen_ke"][:tc], + ) + ] + plan = [] + for chunk in chunks: + blocks = _prefill_blocks( + chunk, step["host_cu_ctx_new"], step["host_cu_ctx_kv"], core._max_ctx_len + ) + for token_start, token_end, k_token_start, n_k in blocks: + if token_end <= token_start: + continue + tiles = [] + if n_k > 0: + shift = k_token_start - chunk.k_token_start + tile = max(1, core.prefill_logits_budget // n_k) + for lo in range(token_start, token_end, tile): + hi = min(lo + tile, token_end) + # The windows are the chunk's own, indexed from its first row and key. + ks = chunk.cu_seqlen_ks[lo - chunk.token_start : hi - chunk.token_start] + ke = chunk.cu_seqlen_ke[lo - chunk.token_start : hi - chunk.token_start] + if shift: + ks, ke = torch.sub(ks, shift), torch.sub(ke, shift) + tiles.append((lo, hi, ks, ke)) + plan.append( + SimpleNamespace( + token_start=token_start, + token_end=token_end, + k_token_start=k_token_start, + n_k=n_k, + tiles=tiles, + ) + ) + return plan + + def _select_ctx(self, step, k_pool, q_fp8, weights, topk_buf, tc: int) -> None: + """Select window-local top-k keys for the context rows assigned to this rank. + + Gather each block's FP8 keys, score each query tile with the Triton MQA + scorer, and pass its aligned logits to GVR top-k. Empty blocks retain + -1 indices. Only one tile's logits are live at a time.""" + core = self.core + plan = step.get("ix_plan") + if plan is None: + plan = step["ix_plan"] = self._ix_plan(step, tc) + if self._gvr_ws is None: + # On the first context step, which the KV-capacity estimate's profiling forward + # is: the pool is sized with the top-k's scratch held, and every top-k launch this + # engine's tiles reach compiles here rather than on a served step (each tile holds + # at most the logits budget, one block at most max_seq_len keys). + self._gvr_ws = gvr_workspace(q_fp8.device) + t0 = time.monotonic() + calls = gvr_warmup( + core.idx_topk, core._max_ctx_len, core.prefill_logits_budget, q_fp8.device, + self._gvr_ws, + ) # fmt: skip + logger.info( + f"modeling_v2 Mistral Large 4 NVFP4: indexer prefill top-k compiled for this " + f"engine's tiles ({calls} warm calls, {time.monotonic() - t0:.1f} s)" + ) + for block in plan: + if block.n_k <= 0: + # No key to select from: every slot of the block's rows is empty. + topk_buf[block.token_start : block.token_end].fill_(-1) + continue + ck, cs = indexer_k_cache_gather_op( + k_pool, step["slot_mapping_fp8_fullkv"], step["slot_mapping_scale_fullkv"], + block.k_token_start, block.n_k, core.idx_dim, + ) # fmt: skip + cs = torch.reshape(cs, [-1]) + for lo, hi, ks, ke in block.tiles: + logits = triton_fp8_mqa_logits(q_fp8[lo:hi], (ck, cs), weights[lo:hi], ks, ke) + cute_dsl_gvr_topk_prefill( + logits, ks, ke, topk_buf[lo:hi], core.idx_topk, self._gvr_ws + ) + # Freed before the next tile's is allocated: one tile is live at a time. + del logits + + def _attn_ctx(self, step, L, fused_q, lat_out, req_idx, topk_buf, ctx): + """Sparse MLA over the context rows into `lat_out`, on the step's path (`ctx`). + + Workspace path: FlashMLA's bf16 sparse prefill over the context requests' latent rows + of the source layer, dequantized out of the pool (this step's rows included, so every + row is read FP8-rounded) into the held workspace's first rows. A shared source (layer + 35) is gathered once into the rows the step keeps beside them, when it has them, and + read there by every layer that shares it; otherwise each of those layers gathers it + again into the first rows. Decode path, a step past the held workspace: the selections + as pool slots, then FlashMLA's sparse fp8 decode over the pool with every context row a + one-row sequence, in the step's sub-batches.""" + core = self.core + tc = fused_q.shape[0] + src = L.kv_source + if ctx.ws_buf is None: + slots = convert_req_index_to_global( + req_idx[:tc], step["block_table"], topk_buf[:tc], step["tokens_per_block"], + core.idx_topk, core._stride_factor, core._pool_row[src], + ) # fmt: skip + for lo, hi in ctx.sub_batches: + # Past 11622 rows FlashMLA's scheduler exits the process instead of raising. + assert hi - lo <= _CTX_DECODE_ROWS, (lo, hi) + flash_mla_sparse_decode( + fused_q[lo:hi], step["latent_kv"], slots[lo:hi], ctx.sched[hi - lo], + core.softmax_scale, lat_out[lo:hi], + ) # fmt: skip + return + ws = ctx.ws.get(src) + if ws is None: + keep = core.shared_source[src] and ctx.kept is not None + ws = ds_mla_dequant_gather( + step["latent_pool"], core._pool_row[src], ctx.kv_req, ctx.kv_pos, + step["block_table"], ctx.kept if keep else ctx.ws_buf, + ) # fmt: skip + if keep: + ctx.ws[src] = ws + # Request-local selections to workspace rows: each request's run starts at its + # cumulative KV offset. + sel = topk_buf[:tc] + idx = torch.where(sel >= 0, sel + ctx.ws_off[req_idx[:tc]][:, None], sel) + flash_mla_sparse_prefill(fused_q, ws, idx.to(torch.int32), core.softmax_scale, lat_out) + + def _attn_gen(self, step, L, fused_q, lat_out, slots, sched): + """Sparse MLA over the generation rows behind the context rows into `lat_out`: FlashMLA's + sparse fp8 decode over the pool, one scheduler shared by the step's layers.""" + flash_mla_sparse_decode( + fused_q, step["latent_kv"], slots, sched, self.core.softmax_scale, lat_out + ) + + def _dense(self, o, L): + """The SwiGLU MLP on this rank's own rows, whole: layer 0's, or a MoE layer's shared + expert.""" + act = flashinfer_silu_and_mul(self._linear_fp8(o, L.mlp_gu, L.mlp_gu_s)) + return self._linear_fp8(act, L.mlp_dn, L.mlp_dn_s) + + def _moe(self, o, L, padded: int, pad_rows: torch.Tensor | None = None): + """A MoE layer's MLP on this rank's own rows: the shared expert on its own rows, on the + side stream, beside the routed experts over every rank's gathered rows. + + The rows are zero-padded to `padded` (the largest rank's count) and gathered before the + router, so every rank routes the identical `world_size * padded` rows. Router, routing, + NVFP4 quantization and this rank's expert window (32 experts at 4 ranks, 16 at 8) run + over them in chunks of at most `expert_call_tokens`, each writing its rows of one + output, which the reduce-scatter sums over the windows and splits back to `padded` rows + per rank. The padding rows (`pad_rows`, `_padding_rows`) take expert id -1 after the + routing, so no rank's window holds them and no expert computes them; their output rows + come back +0.0 and are dropped. Without `pad_rows` they route to experts 0..3 (an exact + tie at +0.0) and contribute zero.""" + core = self.core + m = L.moe + rows = o.shape[0] + side = core._side_stream + main = torch.cuda.current_stream() + side.wait_stream(main) + with torch.cuda.stream(side): + shared = self._dense(o, L) + o_pad = o if rows == padded else nn.functional.pad(o, [0, 0, 0, padded - rows]) + o_all = allgather(o_pad, None, core.dp_group, core.dp_pg) + total = o_all.shape[0] + call = core.expert_call_tokens + routed_all = None if total <= call else torch.empty_like(o_all) + for lo in range(0, total, call): + hi = min(lo + call, total) + x = o_all[lo:hi] + logits = cublas_mm(x, m.router, None, torch.float32) + ids, weights = renorm_moe_routing_op(logits, core.topk, torch.bfloat16) + if pad_rows is not None: + # Every routing kernel of the pre-routed runner drops a slot whose id lies + # outside the rank's window (the small-batch kernels test `expertIdx > -1`, the + # others a signed `id - first_expert >= 0`), so -1 is local to no rank: the + # padding rows reach no expert's GEMMs, and the finalize writes a row with no + # local slot as +0.0. The router, the routing and the quantization still run + # over every row, so the call's shapes, tile and tactic are the same. + ids.masked_fill_(pad_rows[lo:hi, None], -1) + # One NVFP4 quantization under the layer's static scale serves every expert on + # every rank: the windows' partials sum. + xq, xs = fp4_quantize(x, m.gscale, _NVFP4_SF_VEC, False, _NVFP4_SF_LINEAR) + part = fp4_block_scale_moe_runner( + None, None, xq, xs.view(torch.float8_e4m3fn), m.fc1_w, m.fc1_s, None, None, + None, None, m.fc2_w, m.fc2_s, None, m.out1, m.out1_gate, m.out2, + core.num_experts, core.topk, None, None, core.moe_inter, core.expert_offset, + core.local_experts, None, _MOE_ROUTING_METHOD, True, _MOE_ACT_SWIGLU, + topk_weights=weights, topk_ids=ids, tune_max_num_tokens=core.expert_tune_tokens, + )[0] # fmt: skip + if routed_all is None: + routed_all = part + else: + routed_all[lo:hi].copy_(part) + routed = reducescatter(routed_all, None, core.dp_group, core.dp_pg)[:rows] + main.wait_stream(side) + return torch.add(routed, shared) + + def _topk_buf(self, step, rows: int, tc: int, skip_ctx: bool, skip_gen: bool): + """Where this step's selections land, request-local positions. A phase the engine + skipped (every sequence within index_topk) takes the dense causal index it wrote.""" + if skip_ctx and skip_gen: + return step["topk_indices_buffer"][:rows] + buf = torch.empty( + [rows, self.core.idx_topk], dtype=torch.int32, device=step["block_table"].device + ) + if skip_ctx and tc: + buf[:tc].copy_(step["topk_indices_buffer"][:tc]) + if skip_gen and rows > tc: + buf[tc:].copy_(step["topk_indices_buffer"][tc:rows]) + return buf + + def forward( + self, + attn_metadata: AttentionMetadata, + input_ids: torch.IntTensor | None = None, + position_ids: torch.IntTensor | None = None, + inputs_embeds: torch.FloatTensor | None = None, + lora_params: dict | None = None, + **kwargs, + ) -> torch.Tensor: + core = self.core + self._check_step_contract(attn_metadata) + step = self.step_args(attn_metadata) + h = ( + nn.functional.embedding(input_ids, self._embed) + if inputs_embeds is None + else inputs_embeds + ) + rows = h.shape[0] + dev = h.device + # The slot mappings and windows follow the prepared token order; a padded input + # would scatter keys into stale slots. + assert rows == step["num_tokens"], (rows, step["num_tokens"]) + padded = _padded_rows(core, step, rows) + # The gathered rows that are another rank's padding (or this one's), once per step. + pad_rows = _padding_rows(step["all_rank_num_tokens"], padded, dev) + tc = step["num_ctx_tokens"] + gen = rows - tc + skip_ctx = step["skip_ctx"] + skip_gen = step["skip_gen"] if gen else True + pos = torch.reshape(position_ids, [-1]) + req_idx = step["req_idx_per_token"][:rows] + topk_buf = self._topk_buf(step, rows, tc, skip_ctx, skip_gen) + scratch = self._scratch(rows, h.dtype, dev) + ctx = self._ctx_workspace(step, tc, dev) if tc else None + + x = flashinfer_rmsnorm(h, self._norm0, core.eps) + residual = h + spec_metadata = kwargs.get("spec_metadata") + for layer_idx, L in enumerate(self._layers): + attn_out = self._attention( + step, L, x, pos, req_idx, topk_buf, scratch, ctx, tc, skip_ctx, skip_gen + ) + residual = torch.add(residual, self._linear_fp8(attn_out, L.wo, L.wo_s)) + o = flashinfer_rmsnorm(residual, L.ffn_norm, core.eps) + mlp = self._dense(o, L) if L.moe is None else self._moe(o, L, padded, pad_rows) + residual = torch.add(residual, mlp) + x = flashinfer_rmsnorm(residual, L.next_norm, core.eps) + if spec_metadata is not None: + spec_metadata.maybe_capture_hidden_states(layer_idx, residual) + return x + + def _scratch(self, rows: int, dtype, dev) -> SimpleNamespace: + """Per-step scratch every layer rewrites in turn: the attention output, the absorbed + query at the kernels' 64 heads (this rank's first, the rest zero for the step), the + latent attention output, a NoPE owner's 576-wide latent rows (the rope key zero for + the step), and the generation rows' FlashMLA scheduler, filled by the step's first + decode call and shared by the rest.""" + core = self.core + bf16 = dict(dtype=dtype, device=dev) + return SimpleNamespace( + attn_out=torch.empty([rows, core.heads * core.v_dim], **bf16), + fused_q=torch.zeros([rows, _FLASH_MLA_HEADS, core.lat_dim], **bf16), + lat_out=torch.empty([rows, core.heads, core.kv_lora], **bf16), + nope_lat=None if core.unfused_glue else torch.zeros([rows, core.lat_dim], **bf16), + sched=get_mla_metadata()[0], + ) + + def _ctx_workspace(self, step, tc: int, dev) -> SimpleNamespace: + """The context rows' path for the step, chosen once on this rank's host ints from its + context KV: every context request's cached prefix and new tokens (the engine's + `num_ctx_kv_tokens` counts the new ones only). Under attention DP the step holds this + rank's own requests only and the attention has no collective, so each rank takes its + own path; ranks on different paths meet again only at the expert path's collectives. + + The held workspace, bf16 `[core.ctx_workspace_rows, 576]` (`_ctx_workspace_rows`), is + allocated on the first context step, whichever path that step takes, and held. The + engine's KV-capacity estimate runs context steps as its profiling forward and sizes the + pool after their peak memory (`KvCacheCreator.configure_kv_cache_capacity`), so the pool + is sized with the workspace held. A generation-only engine runs no context step and + holds none. + + The workspace path, a step whose context KV fits the held rows: request and position + per workspace row as the dequantizing gather and the prefill kernel address them, each + request's offset, the held workspace's first `n_kv` rows, which every layer's gather + rewrites, and, when the context KV fits twice, the next `n_kv` rows (`kept`), where a + shared owner's gather stays for the step (`ws`). + + The decode path, a step past the held rows, with no workspace: the context rows' + sub-batches of at most `_CTX_DECODE_ROWS`, and one FlashMLA scheduler per sub-batch + size, filled by its first call and shared by the step's layers.""" + # Under KV reuse a context request attends over its cached prefix and its new tokens. + n_kv = step["ctx_cached_tokens"] + step["num_ctx_tokens"] + if self._ws_buf is None: + # On the first context step: the KV-capacity estimate's profiling forward is one, + # so the peak it measures, which sizes the pool, holds this buffer. + self._ws_buf = torch.empty( + [self.core.ctx_workspace_rows, self.core.lat_dim], dtype=torch.bfloat16, device=dev + ) + logger.info( + f"modeling_v2 Mistral Large 4 NVFP4: context workspace held, {self._ws_buf.shape[0]} " + f"rows x {self.core.lat_dim} bf16 ({self._ws_buf.nbytes / 2**20:.0f} MiB)" + ) + held = self._ws_buf.shape[0] + if n_kv > held: + bounds = [(lo, min(lo + _CTX_DECODE_ROWS, tc)) for lo in range(0, tc, _CTX_DECODE_ROWS)] + logger.debug( + f"modeling_v2 Mistral Large 4 NVFP4: context KV {n_kv} past the held {held} rows: " + f"decode path, {len(bounds)} sub-batches" + ) + return SimpleNamespace( + ws_buf=None, + sub_batches=bounds, + sched={rows: get_mla_metadata()[0] for rows in {hi - lo for lo, hi in bounds}}, + ) + ctx = step["num_contexts"] + cu = step["cu_ctx_kv"] + lens = cu[1:] - cu[:-1] + kv_req = torch.repeat_interleave( + torch.arange(ctx, dtype=torch.int32, device=dev), lens, output_size=n_kv + ) + kv_pos = (torch.arange(n_kv, device=dev) - cu[:-1][kv_req]).to(torch.int32) + keep = 2 * n_kv <= held + logger.debug( + f"modeling_v2 Mistral Large 4 NVFP4: context KV {n_kv} of the held {held} rows: " + f"workspace path, the shared source {'kept' if keep else 'gathered again'}" + ) + return SimpleNamespace( + kv_req=kv_req, + kv_pos=kv_pos, + ws_off=cu[:-1], + ws_buf=self._ws_buf[:n_kv], + kept=self._ws_buf[n_kv : 2 * n_kv] if keep else None, + ws={}, + ) + + def _attention(self, step, L, x, pos, req_idx, topk_buf, s, ctx, tc, skip_ctx, skip_gen): + """One layer's attention block, both phases, into `s.attn_out` (returned): the query + with SSMax, the indexer's selection, the absorbed query, the owner's rope and + fp8_ds_mla append, sparse MLA over the context and the generation rows, and the V + up-projection.""" + core = self.core + rows = x.shape[0] + q = self._linear_fp8(x, L.wq, L.wq_s) + q3 = torch.reshape(q, [rows, core.heads, core.qk_dim]) + fq = s.fused_q[:, : core.heads] + if core.unfused_glue: + q3 = ssmax_scaling(q3, pos, L.ssmax_s, L.ssmax_b) + if L.rope: + fq[..., core.kv_lora :].copy_(q3[..., L.nope :]) + else: + fq[..., core.kv_lora :].zero_() + else: + # SSMax in place; its rope half (a NoPE layer's zero) into the absorbed query's tail. + ssmax_scaling_fused_q( + q3, pos, L.ssmax_s, L.ssmax_b, q3, fq[..., core.kv_lora :], L.rope + ) + if L.indexer is not None: + self._indexer(step, x, pos, L, topk_buf, tc, skip_ctx, skip_gen) + bmm_out( + torch.transpose(q3[..., : L.nope], 0, 1), L.k_b, + torch.transpose(fq[..., : core.kv_lora], 0, 1), + ) # fmt: skip + if L.owner: + # Every row's latent, context and generation, before either phase reads the pool; + # the query tail is roped in the same call. + ds_mla_rope_append( + self._latent(x, L, s.nope_lat), fq, pos, core._rope.indexer, req_idx, + step["block_table"], step["latent_pool"], core._pool_row[L.index], L.rope, + ) # fmt: skip + if tc: + self._attn_ctx(step, L, s.fused_q[:tc], s.lat_out[:tc], req_idx, topk_buf, ctx) + if rows > tc: + slots = convert_req_index_to_global( + req_idx[tc:], step["block_table"], topk_buf[tc:], step["tokens_per_block"], + core.idx_topk, core._stride_factor, core._pool_row[L.kv_source], + ) # fmt: skip + self._attn_gen(step, L, s.fused_q[tc:], s.lat_out[tc:], slots, s.sched) + bmm_out( + torch.transpose(s.lat_out, 0, 1), + L.v_b_t, + torch.transpose(torch.reshape(s.attn_out, [rows, core.heads, core.v_dim]), 0, 1), + ) + return s.attn_out + + def dspark_forward( + self, + draft, + noise_embedding, + query_positions, + num_ctx_per_req, + ctx_k_cache, + ctx_v_cache, + ctx_cache_batch_idx, + ctx_kv_cache=None, + ctx_page_table=None, + block_placement=None, + ): + # Only the worker's fused paged-latent preparation places the block + # (block_placement), and a drafter with embed_draft_block does not take + # it; this decode reads its context window through ctx_page_table. + del block_placement + draft._build_fused_kv_buffers() + draft._ensure_rope() + core = self.core + batch, block, hidden = noise_embedding.shape + rows = batch * block + # Gather a bounded window, not every page in the configured context. + # Invalid padding slots are masked even if their pages are allocated. + lengths = num_ctx_per_req[:batch].long() + offsets = torch.arange(draft.window, device=noise_embedding.device) + positions = (lengths - draft.window).clamp_min(0)[:, None] + offsets[None, :] + valid = positions < lengths[:, None] + slots = ctx_cache_batch_idx.long() + if ctx_page_table is not None: + page_size = ctx_kv_cache[0].shape[-2] + table = ctx_page_table.index_select(0, slots).long() + safe_positions = positions.clamp(max=table.shape[1] * page_size - 1) + pages = table.gather(1, safe_positions // page_size).clamp_min(0) + page_offsets = safe_positions % page_size + else: + safe_positions = positions.clamp(max=ctx_k_cache.shape[2] - 1) + key_valid = torch.cat( + (valid, torch.ones(batch, block, dtype=torch.bool, device=valid.device)), dim=1 + ) + query = torch.empty( + batch, + block, + core.heads, + core.lat_dim, + dtype=noise_embedding.dtype, + device=noise_embedding.device, + ) + values = torch.empty( + rows, core.heads, core.v_dim, dtype=noise_embedding.dtype, device=noise_embedding.device + ) + residual = noise_embedding.reshape(rows, hidden).clone() + x = flashinfer_rmsnorm(residual, draft._layers[0].attn_norm, core.eps) + for i, layer in enumerate(draft._layers): + q = cublas_mm(x, layer.q, None, None).view(batch, block, core.heads, core.qk_dim) + bmm_out( + q[..., : core.nope].reshape(rows, core.heads, core.nope).transpose(0, 1), + layer.k_b, + query[..., : core.kv_lora].view(rows, core.heads, core.kv_lora).transpose(0, 1), + ) + query[..., core.kv_lora :].copy_(draft._rotate(q[..., core.nope :], query_positions)) + kv = cublas_mm(x, layer.kv_a, None, None) + content = flashinfer_rmsnorm( + kv[:, : core.kv_lora].contiguous(), layer.kv_norm, core.eps + ) + block_kv = torch.cat( + ( + content.view(batch, block, core.kv_lora), + draft._rotate( + kv[:, core.kv_lora :].view(batch, block, core.rope_dim), query_positions + ), + ), + dim=-1, + ) + if ctx_page_table is not None: + context = ctx_kv_cache[i][:, 0, 0][pages, page_offsets] + else: + context = ctx_k_cache[slots[:, None], i, safe_positions, 0] + # Zero masked values as well as scores: 0 * an uninitialized + # NaN in the latent pool must not contaminate the weighted sum. + context = context.masked_fill(~valid.unsqueeze(-1), 0) + keys = torch.cat((context, block_kv), dim=1) + scores = torch.einsum("bshd,btd->bsht", query, keys).float() * core.softmax_scale + scores = scores.masked_fill(~key_valid[:, None, None, :], float("-inf")) + probabilities = torch.softmax(scores, dim=-1).to(keys.dtype) + latent = torch.einsum("bsht,btc->bshc", probabilities, keys[..., : core.kv_lora]) + bmm_out( + latent.reshape(rows, core.heads, core.kv_lora).transpose(0, 1), + layer.v_b.transpose(1, 2), + values.transpose(0, 1), + ) + partial = cublas_mm(values.reshape(rows, -1), layer.out, None, None) + residual = residual + partial + o = flashinfer_rmsnorm(residual, layer.ffn_norm, core.eps) + gate = cublas_mm(o, layer.gate, None, None) + up = cublas_mm(o, layer.up, None, None) + activated = flashinfer_silu_and_mul(torch.cat((gate, up), dim=-1)) + mlp = cublas_mm(activated, layer.down, None, None) + residual = residual + mlp + x = flashinfer_rmsnorm(residual, layer.next_norm, core.eps) + return x + + +class DecodeTarget: + """The specialization: routed to only when there are no context rows. One query row per + sequence (no speculative decoding); CUDA-graph captured. Its MoE layers issue the same + collectives as the prefill target's, at the same sizes: a peer rank may be prefilling.""" + + def __init__(self, core: MistralLarge4DepModelingV2) -> None: + """Take the per-layer views once, off the real weights. Everything an engine value + fixes is read off `self.core` at the call.""" + self.core = core + self._contract_pending = step_contract_enabled() + self._layers = _layer_views(core) + self._embed = core.w["embed"] + self._norm0 = core.w["l0_attn_norm"] + + def _check_step_contract(self, md) -> None: + """Opt-in first-forward fail-fast: calling `step_args` reads every metadata field this + target consumes (private trtllm surface). Fixed at engine construction, so once per + target instance, and off in a served engine.""" + if not self._contract_pending: + return + try: + self.step_args(md) + except AttributeError as exc: + raise AssertionError(f"attention metadata surface drifted: {exc}") from exc + self._contract_pending = False + + def step_args(self, md: TrtllmAttentionMetadata) -> dict: + """The common batch state. States the two context counts instead of reading them + back: this target runs only when there are no context rows.""" + step = _step_common(md, self.core.indexer_layers) + step.update(num_contexts=0, num_ctx_tokens=0) + return step + + def _linear_fp8(self, x, w, w_s): + """K128-quantized FP8 linear through the Rubin MXFP8 kernel.""" + return cute_dsl_fp8_block_scale_linear(x, w, w_s) + + def _latent(self, x, L, nope_lat): + """`[RMSNorm(c_kv) | k_pe]`, 576 wide: a RoPE owner's kv_a row, normalized in place; a + NoPE owner's 512-wide latent normalized into `nope_lat`, whose rope key stays zero.""" + core = self.core + kv = cublas_mm(x, L.wkv_a) + if core.unfused_glue: + c_kv = flashinfer_rmsnorm(kv[:, : core.kv_lora], L.kv_norm, core.eps) + if L.rope: + k_pe = kv[:, core.kv_lora :] + else: + k_pe = torch.zeros([kv.shape[0], core.rope_dim], dtype=kv.dtype, device=kv.device) + return torch.cat([c_kv, k_pe], dim=-1) + if L.rope: + c_kv = kv[:, : core.kv_lora] + flashinfer_rmsnorm(c_kv, L.kv_norm, core.eps, out=c_kv) + return kv + flashinfer_rmsnorm(kv, L.kv_norm, core.eps, out=nope_lat[:, : core.kv_lora]) + return nope_lat + + def _indexer(self, step, x, pos, L, topk_buf, skip_gen: bool): + """Append this step's keys to the layer's key pool; unless the engine skipped the + indexer (every sequence within index_topk), score every sequence's paged keys and + select `index_topk` positions per row into `topk_buf`.""" + core = self.core + ix = L.indexer + rows = x.shape[0] + table = core._rope.indexer if ix.rope else core._rope.identity + kw = cublas_mm(x, ix.wk_wp) + k = kw[:, : core.idx_dim].contiguous() + k_pool = step["ix_pools"][L.index] + k_pool = cute_dsl_indexer_k_cache_update( + k, ix.k_norm_w, ix.k_norm_b, pos, table.view(-1, 2, core.rope_dim // 2), k_pool, + step["slot_mapping_fp8"], step["slot_mapping_scale"], rows, + is_neox=False, compiled=core._ix_k[k_pool.is_contiguous()], validate=False, + ) # fmt: skip + if skip_gen: + return + qn = layer_norm(x, ix.q_norm_w, ix.q_norm_b, core.idx_eps) + q = torch.reshape(self._linear_fp8(qn, ix.wq_b, ix.wq_b_s), [rows, core.idx_heads, -1]) + q_fp8 = torch.empty_like(q, dtype=torch.float8_e4m3fn) + weights = torch.empty([rows, core.idx_heads], dtype=torch.float32, device=x.device) + if core.unfused_glue: + q = ssmax_scaling(q, pos, ix.ssmax_s, ix.ssmax_b) + q_fp8, weights = cute_dsl_indexer_q_side_fused( + q, pos, table, kw[:, core.idx_dim :].float().contiguous(), core.idx_weight_scale, + q_fp8, weights, is_neox=False, compiled=core._ix_q, + ) # fmt: skip + else: + # SSMax, rope, e4m3 and the head weights (read in place out of `kw`) in one launch. + q_fp8, weights = ssmax_indexer_q_side_fused( + q, pos, ix.ssmax_s, ix.ssmax_b, table, kw[:, core.idx_dim :], + core.idx_weight_scale, q_fp8, weights, + ) # fmt: skip + num_gens = step["num_seqs"] + next_n = rows // num_gens + kv_lens = step["kv_lens_cuda"][:num_gens] + logits = cute_dsl_fp8_paged_mqa_logits( + torch.reshape(q_fp8, [num_gens, next_n, core.idx_heads, core.idx_dim]), + k_pool, + weights, + kv_lens, + step["indexer_k_cache_block_offsets"][:num_gens], + step["scheduler_metadata_buffer"], + core._max_ctx_len, + ) + if next_n != 1 or core._ix_topk is None: + indexer_topk_decode( + logits, kv_lens, topk_buf, next_n, core.idx_topk, 1, step["radix_aux_indices"], + step["radix_aux_logits"], + ) # fmt: skip + else: + cute_dsl_radix_topk_decode(logits, kv_lens, topk_buf, next_n=1, compiled=core._ix_topk) + + def _dense(self, o, L): + """The SwiGLU MLP on this rank's own rows, whole: layer 0's, or a MoE layer's shared + expert.""" + act = flashinfer_silu_and_mul(self._linear_fp8(o, L.mlp_gu, L.mlp_gu_s)) + return self._linear_fp8(act, L.mlp_dn, L.mlp_dn_s) + + def _moe(self, o, L, padded: int, pad_rows: torch.Tensor | None = None): + """A MoE layer's MLP on this rank's own rows: the shared expert on its own rows, on the + side stream, beside the routed experts over every rank's gathered rows. + + The rows are zero-padded to `padded` (the largest rank's count) and gathered before the + router, so every rank routes the identical `world_size * padded` rows. Router, routing, + NVFP4 quantization and this rank's expert window (32 experts at 4 ranks, 16 at 8) run + over them in chunks of at most `expert_call_tokens`, each writing its rows of one + output, which the reduce-scatter sums over the windows and splits back to `padded` rows + per rank. Beside a + prefilling peer this rank's few rows gather up to `world_size * max_num_tokens`, so the + chunking is this target's too. The padding rows (`pad_rows`, `_padding_rows`) take + expert id -1 after the routing, so no rank's window holds them and no expert computes + them; their output rows come back +0.0 and are dropped. Without `pad_rows` they route + to experts 0..3 (an exact tie at +0.0) and contribute zero.""" + core = self.core + m = L.moe + rows = o.shape[0] + side = core._side_stream + main = torch.cuda.current_stream() + side.wait_stream(main) + with torch.cuda.stream(side): + shared = self._dense(o, L) + o_pad = o if rows == padded else nn.functional.pad(o, [0, 0, 0, padded - rows]) + o_all = allgather(o_pad, None, core.dp_group, core.dp_pg) + total = o_all.shape[0] + call = core.expert_call_tokens + routed_all = None if total <= call else torch.empty_like(o_all) + for lo in range(0, total, call): + hi = min(lo + call, total) + x = o_all[lo:hi] + logits = cublas_mm(x, m.router, None, torch.float32) + ids, weights = renorm_moe_routing_op(logits, core.topk, torch.bfloat16) + if pad_rows is not None: + # Every routing kernel of the pre-routed runner drops a slot whose id lies + # outside the rank's window (the small-batch kernels test `expertIdx > -1`, the + # others a signed `id - first_expert >= 0`), so -1 is local to no rank: the + # padding rows reach no expert's GEMMs, and the finalize writes a row with no + # local slot as +0.0. The router, the routing and the quantization still run + # over every row, so the call's shapes, tile and tactic are the same. + ids.masked_fill_(pad_rows[lo:hi, None], -1) + # One NVFP4 quantization under the layer's static scale serves every expert on + # every rank: the windows' partials sum. + xq, xs = fp4_quantize(x, m.gscale, _NVFP4_SF_VEC, False, _NVFP4_SF_LINEAR) + part = fp4_block_scale_moe_runner( + None, None, xq, xs.view(torch.float8_e4m3fn), m.fc1_w, m.fc1_s, None, None, + None, None, m.fc2_w, m.fc2_s, None, m.out1, m.out1_gate, m.out2, + core.num_experts, core.topk, None, None, core.moe_inter, core.expert_offset, + core.local_experts, None, _MOE_ROUTING_METHOD, True, _MOE_ACT_SWIGLU, + topk_weights=weights, topk_ids=ids, tune_max_num_tokens=core.expert_tune_tokens, + )[0] # fmt: skip + if routed_all is None: + routed_all = part + else: + routed_all[lo:hi].copy_(part) + routed = reducescatter(routed_all, None, core.dp_group, core.dp_pg)[:rows] + main.wait_stream(side) + return torch.add(routed, shared) + + def forward( + self, + attn_metadata: AttentionMetadata, + input_ids: torch.IntTensor | None = None, + position_ids: torch.IntTensor | None = None, + inputs_embeds: torch.FloatTensor | None = None, + lora_params: dict | None = None, + **kwargs, + ) -> torch.Tensor: + core = self.core + self._check_step_contract(attn_metadata) + step = self.step_args(attn_metadata) + h = ( + nn.functional.embedding(input_ids, self._embed) + if inputs_embeds is None + else inputs_embeds + ) + rows = h.shape[0] + dev = h.device + assert rows == step["num_tokens"] and rows % step["num_seqs"] == 0, ( + rows, + step["num_tokens"], + step["num_seqs"], + ) + # Every rank's count, frozen into a capture: the engine replays a decode graph only + # when every rank is generation-only at the same padded batch, the counts it captured. + padded = _padded_rows(core, step, rows) + # The gathered rows that are padding, in an eager step beside a prefilling peer. A + # captured step takes none: the engine replays a decode graph only when every rank + # decodes at the captured batch, every count equal, so a graph pads nothing. + pad_rows = ( + None + if torch.cuda.is_current_stream_capturing() + else _padding_rows(step["all_rank_num_tokens"], padded, dev) + ) + # A separate graph is captured for steps whose sequences all fit in index_topk, so + # this host branch is a per-capture constant. + skip_gen = step["skip_gen"] + pos = torch.reshape(position_ids, [-1]) + req_idx = step["req_idx_per_token"][:rows] + if skip_gen: + topk_buf = step["topk_indices_buffer"][:rows] + else: + topk_buf = torch.empty([rows, core.idx_topk], dtype=torch.int32, device=dev) + scratch = self._scratch(rows, h.dtype, dev) + + x = flashinfer_rmsnorm(h, self._norm0, core.eps) + residual = h + spec_metadata = kwargs.get("spec_metadata") + for layer_idx, L in enumerate(self._layers): + attn_out = self._attention(step, L, x, pos, req_idx, topk_buf, scratch, skip_gen) + residual = torch.add(residual, self._linear_fp8(attn_out, L.wo, L.wo_s)) + o = flashinfer_rmsnorm(residual, L.ffn_norm, core.eps) + mlp = self._dense(o, L) if L.moe is None else self._moe(o, L, padded, pad_rows) + residual = torch.add(residual, mlp) + x = flashinfer_rmsnorm(residual, L.next_norm, core.eps) + if spec_metadata is not None: + spec_metadata.maybe_capture_hidden_states(layer_idx, residual) + return x + + def _scratch(self, rows: int, dtype, dev) -> SimpleNamespace: + """Per-step scratch every layer rewrites in turn: the attention output, the absorbed + query at the kernel's 64 heads (this rank's first, the rest zero for the step), a NoPE + owner's 576-wide latent rows (the rope key zero for the step), the latent attention + output the unfused glue copies FlashMLA's live heads into, and the FlashMLA scheduler + the step's first layer fills and the rest share (captured with the step).""" + core = self.core + bf16 = dict(dtype=dtype, device=dev) + return SimpleNamespace( + attn_out=torch.empty([rows, core.heads * core.v_dim], **bf16), + fused_q=torch.zeros([rows, _FLASH_MLA_HEADS, core.lat_dim], **bf16), + nope_lat=None if core.unfused_glue else torch.zeros([rows, core.lat_dim], **bf16), + lat_out=( + torch.empty([rows, core.heads, core.kv_lora], **bf16) if core.unfused_glue else None + ), + sched=get_mla_metadata()[0], + ) + + def _attention(self, step, L, x, pos, req_idx, topk_buf, s, skip_gen: bool): + """One layer's attention block into `s.attn_out` (returned): the query with SSMax, the + indexer's selection, the absorbed query, the owner's rope and fp8_ds_mla append, + FlashMLA sparse decode over the pool, and the V up-projection of the kernel's live + heads, read in place.""" + core = self.core + rows = x.shape[0] + q = self._linear_fp8(x, L.wq, L.wq_s) + q3 = torch.reshape(q, [rows, core.heads, core.qk_dim]) + fq = s.fused_q[:, : core.heads] + if core.unfused_glue: + q3 = ssmax_scaling(q3, pos, L.ssmax_s, L.ssmax_b) + if L.rope: + fq[..., core.kv_lora :].copy_(q3[..., L.nope :]) + else: + fq[..., core.kv_lora :].zero_() + else: + # SSMax in place; its rope half (a NoPE layer's zero) into the absorbed query's tail. + ssmax_scaling_fused_q( + q3, pos, L.ssmax_s, L.ssmax_b, q3, fq[..., core.kv_lora :], L.rope + ) + if L.indexer is not None: + self._indexer(step, x, pos, L, topk_buf, skip_gen) + slots = convert_req_index_to_global( + req_idx, step["block_table"], topk_buf, step["tokens_per_block"], core.idx_topk, + core._stride_factor, core._pool_row[L.kv_source], + ) # fmt: skip + bmm_out( + torch.transpose(q3[..., : L.nope], 0, 1), L.k_b, + torch.transpose(fq[..., : core.kv_lora], 0, 1), + ) # fmt: skip + if L.owner: + ds_mla_rope_append( + self._latent(x, L, s.nope_lat), fq, pos, core._rope.indexer, req_idx, + step["block_table"], step["latent_pool"], core._pool_row[L.index], L.rope, + ) # fmt: skip + if core.unfused_glue: + lat = flash_mla_sparse_decode( + s.fused_q, step["latent_kv"], slots, s.sched, core.softmax_scale, s.lat_out + ) + else: + lat = flash_mla_sparse_decode( + s.fused_q, step["latent_kv"], slots, s.sched, core.softmax_scale, None + )[:, : core.heads] + bmm_out( + torch.transpose(lat, 0, 1), + L.v_b_t, + torch.transpose(torch.reshape(s.attn_out, [rows, core.heads, core.v_dim]), 0, 1), + ) + return s.attn_out + + def dspark_forward( + self, + draft, + noise_embedding, + query_positions, + num_ctx_per_req, + ctx_k_cache, + ctx_v_cache, + ctx_cache_batch_idx, + ctx_kv_cache=None, + ctx_page_table=None, + block_placement=None, + ): + # Only the worker's fused paged-latent preparation places the block + # (block_placement), and a drafter with embed_draft_block does not take + # it; this decode reads its context window through ctx_page_table. + del block_placement + draft._build_fused_kv_buffers() + draft._ensure_rope() + core = self.core + batch, block, hidden = noise_embedding.shape + rows = batch * block + # Gather a bounded window, not every page in the configured context. + # Invalid padding slots are masked even if their pages are allocated. + lengths = num_ctx_per_req[:batch].long() + offsets = torch.arange(draft.window, device=noise_embedding.device) + positions = (lengths - draft.window).clamp_min(0)[:, None] + offsets[None, :] + valid = positions < lengths[:, None] + slots = ctx_cache_batch_idx.long() + if ctx_page_table is not None: + page_size = ctx_kv_cache[0].shape[-2] + table = ctx_page_table.index_select(0, slots).long() + safe_positions = positions.clamp(max=table.shape[1] * page_size - 1) + pages = table.gather(1, safe_positions // page_size).clamp_min(0) + page_offsets = safe_positions % page_size + else: + safe_positions = positions.clamp(max=ctx_k_cache.shape[2] - 1) + key_valid = torch.cat( + (valid, torch.ones(batch, block, dtype=torch.bool, device=valid.device)), dim=1 + ) + query = torch.empty( + batch, + block, + core.heads, + core.lat_dim, + dtype=noise_embedding.dtype, + device=noise_embedding.device, + ) + values = torch.empty( + rows, core.heads, core.v_dim, dtype=noise_embedding.dtype, device=noise_embedding.device + ) + residual = noise_embedding.reshape(rows, hidden).clone() + x = flashinfer_rmsnorm(residual, draft._layers[0].attn_norm, core.eps) + for i, layer in enumerate(draft._layers): + q = cublas_mm(x, layer.q, None, None).view(batch, block, core.heads, core.qk_dim) + bmm_out( + q[..., : core.nope].reshape(rows, core.heads, core.nope).transpose(0, 1), + layer.k_b, + query[..., : core.kv_lora].view(rows, core.heads, core.kv_lora).transpose(0, 1), + ) + query[..., core.kv_lora :].copy_(draft._rotate(q[..., core.nope :], query_positions)) + kv = cublas_mm(x, layer.kv_a, None, None) + content = flashinfer_rmsnorm( + kv[:, : core.kv_lora].contiguous(), layer.kv_norm, core.eps + ) + block_kv = torch.cat( + ( + content.view(batch, block, core.kv_lora), + draft._rotate( + kv[:, core.kv_lora :].view(batch, block, core.rope_dim), query_positions + ), + ), + dim=-1, + ) + if ctx_page_table is not None: + context = ctx_kv_cache[i][:, 0, 0][pages, page_offsets] + else: + context = ctx_k_cache[slots[:, None], i, safe_positions, 0] + # Zero masked values as well as scores: 0 * an uninitialized + # NaN in the latent pool must not contaminate the weighted sum. + context = context.masked_fill(~valid.unsqueeze(-1), 0) + keys = torch.cat((context, block_kv), dim=1) + scores = torch.einsum("bshd,btd->bsht", query, keys).float() * core.softmax_scale + scores = scores.masked_fill(~key_valid[:, None, None, :], float("-inf")) + probabilities = torch.softmax(scores, dim=-1).to(keys.dtype) + latent = torch.einsum("bsht,btc->bshc", probabilities, keys[..., : core.kv_lora]) + bmm_out( + latent.reshape(rows, core.heads, core.kv_lora).transpose(0, 1), + layer.v_b.transpose(1, 2), + values.transpose(0, 1), + ) + partial = cublas_mm(values.reshape(rows, -1), layer.out, None, None) + residual = residual + partial + o = flashinfer_rmsnorm(residual, layer.ffn_norm, core.eps) + gate = cublas_mm(o, layer.gate, None, None) + up = cublas_mm(o, layer.up, None, None) + activated = flashinfer_silu_and_mul(torch.cat((gate, up), dim=-1)) + mlp = cublas_mm(activated, layer.down, None, None) + residual = residual + mlp + x = flashinfer_rmsnorm(residual, layer.next_norm, core.eps) + return x + + +class DraftModel(nn.Module): + """This specialization's BF16 DSpark backbone and recurrent head. + + Runtime owns accepted-token bookkeeping and the paged context pool. The + draft attends the last trained window of captured target features plus + its whole non-causal query block. All parameters live in the core table. + """ + + _uses_worker_attention_backend = False + _paged_ctx_cache = True + _kv_factor = 1 + _dspark_shift_label = True + has_markov_head = True + has_recurrent_markov_head = True + + def __init__(self, shell, draft_config): + super().__init__() + # Keep references without registering the parent as a child module. + object.__setattr__(self, "_shell", shell) + self.model_config = draft_config + self.config = draft_config.pretrained_config + core = shell.model + spec = shell.spec_config + native = spec._draft_model_config + self.block_size = spec.block_size + self.mask_token_id = spec.mask_token_id + self.target_layer_ids = tuple(spec.target_layer_ids) + self.window = int(native["ragged_attention"].split(",")[0]) + self._num_attn_layers = int(native["n_layers"]) + self._num_heads = core.heads + self._num_kv_heads = 1 + self._head_dim = core.lat_dim + self.dflash_attention_backend = "VANILLA" + self._runtime_position_ceiling = None + self._rope = None + self._layers = None + + @property + def core(self): + return self._shell.model + + @property + def lm_head(self): + return self._shell.lm_head + + @property + def markov_w2(self): + return self.core.w["draft_markov_output"] + + def logits_processor(self, hidden, lm_head, attn_metadata, gather): + logits = self._shell.logits_processor(hidden, lm_head, attn_metadata, gather) + # The shared-head logits are soft-capped before the RNN bias is added. + return tanh_softcap(logits, self._shell.softcap) + + def apply_draft_logit_transform(self, logits): + """Return ``logits`` unchanged: :meth:`logits_processor` already soft-caps them. + + The worker calls this on the block logits after ``logits_processor``. + This drafter declares no ``logit_multiplier`` or ``logit_softcap``, so the + cap is applied once, in ``logits_processor``. + """ + return logits + + def _build_fused_kv_buffers(self): + # The worker calls this before sizing its cache. Take views once after + # materialization; context projection is explicit per layer here. + if self._layers is not None: + return + w, core = self.core.w, self.core + self._layers = [] + for i in range(self._num_attn_layers): + prefix = f"l{i}_draft_" + kv_b = w[prefix + "wkv_b"].view(core.heads, core.nope + core.v_dim, core.kv_lora) + self._layers.append( + SimpleNamespace( + attn_norm=w[prefix + "attn_norm"], + q=w[prefix + "wq"].t(), + kv_a=w[prefix + "wkv_a"].t(), + kv_norm=w[prefix + "kv_a_norm"], + k_b=kv_b[:, : core.nope].contiguous(), + v_b=kv_b[:, core.nope :].contiguous(), + out=w[prefix + "wo"].t(), + ffn_norm=w[prefix + "ffn_norm"], + gate=w[prefix + "w1"].t(), + up=w[prefix + "w3"].t(), + down=w[prefix + "w2"].t(), + next_norm=( + w[f"l{i + 1}_draft_attn_norm"] + if i + 1 < self._num_attn_layers + else w["draft_final_norm"] + ), + ) + ) + + def _ensure_rope(self): + cap = self._runtime_position_ceiling or self.core.rope_rows + if self._rope is None or self._rope.rows != cap: + w = self.core.w["draft_final_norm"] + # alpha=0 removes LeRoPE; all draft layers use plain YaRN. + self._rope = _rope_tables( + self.core.model_config.pretrained_config, + torch.zeros(self.core.rope_dim // 2), + cap, + w.dtype, + w.device, + ) + + def _rotate(self, x, positions): + table = self._rope.indexer[positions.long()] + cos, sin = table.chunk(2, dim=-1) + # One head axis for Q, none for the shared latent K. + if x.ndim == positions.ndim + 2: + cos, sin = cos.unsqueeze(-2), sin.unsqueeze(-2) + even, odd = x.float()[..., 0::2], x.float()[..., 1::2] + return ( + torch.stack((even * cos - odd * sin, even * sin + odd * cos), -1) + .flatten(-2) + .to(x.dtype) + ) + + def embed_draft_block(self, anchors, block_size): + # The route pins the trained block width; no mask token is looked up + # in the vocabulary (its sentinel is deliberately out of range). + weight = self.core.w["embed"] + valid = (anchors >= 0) & (anchors < weight.shape[0]) + embedded = nn.functional.embedding(anchors.clamp(0, weight.shape[0] - 1), weight) + embedded = embedded.masked_fill(~valid.unsqueeze(-1), 0) + masks = self.core.w["draft_mask"].unsqueeze(0).expand(anchors.shape[0], -1, -1) + return torch.cat((embedded.unsqueeze(1), masks), dim=1) + + def project_target_hidden(self, hidden): + w = self.core.w + projected = cublas_mm(hidden, w["draft_target_projection"].t(), None, None) + return flashinfer_rmsnorm(projected, w["draft_target_norm"], self.core.eps) + + def precompute_context_kv(self, projected_hidden, positions): + self._build_fused_kv_buffers() + self._ensure_rope() + core = self.core + latent = [] + for layer in self._layers: + kv = cublas_mm(projected_hidden, layer.kv_a, None, None) + content = flashinfer_rmsnorm( + kv[:, : core.kv_lora].contiguous(), layer.kv_norm, core.eps + ) + rope = self._rotate(kv[:, core.kv_lora :], positions.reshape(-1)) + latent.append(torch.cat((content, rope), dim=-1)) + return torch.stack(latent, dim=1).unsqueeze(2), None + + def dflash_forward_with_metadata(self, *, attn_metadata, **kwargs): + # The phase belongs to this engine step, including mixed batches. + return self.core._select_target(attn_metadata).dspark_forward(self, **kwargs) + + def apply_markov_chain_logits(self, base_logits, first_prev_tokens, *, hidden_states): + """The recurrent Markov bias over full-vocab ``[B, K, vocab]`` block logits. + + Each position's greedy token (the argmax of its biased logits) is the + next position's previous token; ``hidden_states`` is the ``[B, K, + hidden]`` block output the logits were projected from. + """ + w = self.core.w + output = w["draft_markov_output"] + previous = first_prev_tokens.long() + state = None + refined = [] + for i in range(base_logits.shape[1]): + weight = w["draft_markov_embedding"] + valid = (previous >= 0) & (previous < weight.shape[0]) + embedding = nn.functional.embedding(previous.clamp(0, weight.shape[0] - 1), weight) + embedding = embedding.masked_fill(~valid.unsqueeze(-1), 0) + if state is None: + state = torch.zeros_like(embedding) + joint = cublas_mm( + torch.cat((state, embedding, hidden_states[:, i]), dim=-1), + w["draft_joint"].t(), + w["draft_joint_bias"], + None, + ) + gate, candidate, emitted = joint.chunk(3, dim=-1) + gate = torch.sigmoid(gate) + state = gate * state + (1 - gate) * torch.tanh(candidate) + bias = cublas_mm(torch.tanh(emitted), output.t(), None, None) + logits = base_logits[:, i] + bias + refined.append(logits) + previous = logits.argmax(dim=-1) + return torch.stack(refined, dim=1) + + +class _VisionBlock(NamedTuple): + """One vision transformer block's weights, as the views the ops read: the GEMM + operands column-major (`.t()` of the stored rows).""" + + attn_norm: torch.Tensor + qkv: torch.Tensor + wo: torch.Tensor + ffn_norm: torch.Tensor + w13: torch.Tensor + w2: torch.Tensor + + +def _vision_step(seg_lens: list[int], device, workspace: torch.Tensor) -> dict: + """The batch state of one vision attention call: one context sequence per image, + no KV cache. Every value is host-known from the images' patch counts.""" + total = sum(seg_lens) + longest = max(seg_lens) + lens_dev = torch.tensor(seg_lens, dtype=torch.int32, device=device) + lens_host = torch.tensor(seg_lens, dtype=torch.int32) + return dict( + workspace=workspace, + sequence_length=lens_dev, + host_past_key_value_lengths=lens_host, + host_total_kv_lens=torch.tensor([total, 0], dtype=torch.int32), + context_lengths=lens_dev, + host_context_lengths=lens_host, + host_request_types=torch.zeros(len(seg_lens), dtype=torch.int32), + max_num_requests=len(seg_lens), + max_context_length=longest, + max_seq_len=longest, + attention_window_size=longest, + num_contexts=len(seg_lens), + num_ctx_tokens=total, + ) + + +def _vision_attention( + x: torch.Tensor, + B: _VisionBlock, + positions: torch.Tensor, + step: dict, + heads: int, + head_dim: int, + theta: float, +) -> torch.Tensor: + """One vision block's attention over the patch rows of every image of the call: + `[wq ; wk ; wv]` as one GEMM, q and k rotated by each patch's (row, col) in its + own image, attention bidirectional within each image, then `wo`.""" + rows = x.shape[0] + width = heads * head_dim + qkv = cublas_mm(x, B.qkv) + q, k = pixtral_rope_2d( + torch.reshape(qkv[:, :width], [rows, heads, head_dim]), + torch.reshape(qkv[:, width : 2 * width], [rows, heads, head_dim]), + positions, + theta, + ) + qkv = torch.cat( + [torch.reshape(q, [rows, width]), torch.reshape(k, [rows, width]), qkv[:, 2 * width :]], -1 + ) + out = torch.empty([rows, width], dtype=x.dtype, device=x.device) + out = thop_attention( + q=qkv, + output=out, + num_heads=heads, + num_kv_heads=heads, + head_size=head_dim, + **step, + **_VISION_ATTN, + ) + return cublas_mm(out, B.wo) + + +def _merge_tiles(x: torch.Tensor, grid_h: int, grid_w: int, merge: int) -> torch.Tensor: + """One image's `[grid_h * grid_w, d]` patch rows -> `[grid_h * grid_w / merge^2, + d * merge^2]`: each merge x merge tile of the grid as one row, tiles row-major (the + order of the image's [IMG] tokens), each row laid out (channel, tile row, tile + column) as the merging layer's input columns are.""" + d = x.shape[-1] + grid = torch.permute(torch.reshape(x, [grid_h, grid_w, d]), [2, 0, 1]) + tiles = nn.functional.unfold(torch.unsqueeze(grid, 0), kernel_size=merge, stride=merge) + return torch.t(tiles[0]) + + +class VisionTower: + """The image input path: a request's pixels -> one row per [IMG] token, in the + language model's hidden width. Run by the shell, in context steps only, never + captured; every rank runs the whole tower (all heads) over its own requests. + + Per image: the 14x14 patch embedding as an unfold and one GEMM, `ln_pre`, the + transformer blocks (RMSNorm, attention with 2-D RoPE, RMSNorm, SwiGLU, each with + its residual), the pre-projector RMSNorm, the 2x2 patch merger and the adapter + (`w_in`, exact GELU, `w_out`). The images of one call are concatenated along the + rows; only the attention needs their boundaries.""" + + def __init__(self, core) -> None: + """Take the tower's weight views and geometry, once, off the real weights.""" + d = _weights.MODEL_WEIGHTS.dims(core) + w = core.w + self.hidden = d.v_hidden + self.heads = d.v_heads + self.head_dim = d.v_head_dim + self.patch = d.v_patch + self.merge = d.v_merge + self.theta = d.v_theta + self.dtype = d.dtype + self.out_hidden = d.hidden + self._validate = step_contract_enabled() + self._rank = core.model_config.mapping.rank + self.patch_w = w["vis_patch"].t() + self.ln_pre = w["vis_ln_pre"] + self.blocks = [ + _VisionBlock( + attn_norm=w[f"v{j}_attn_norm"], + qkv=w[f"v{j}_qkv"].t(), + wo=w[f"v{j}_wo"].t(), + ffn_norm=w[f"v{j}_ffn_norm"], + w13=w[f"v{j}_w13"].t(), + w2=w[f"v{j}_w2"].t(), + ) + for j in range(d.v_layers) + ] + self.proj_norm = w["vis_proj_norm"] + self.merge_w = w["vis_merge"].t() + self.w_in = w["vis_w_in"].t() + self.w_out = w["vis_w_out"].t() + # The prompt token the tower's rows replace, on the host: the executor finds a + # step's image positions with it before anything reaches the device. + self.image_token_ids = torch.tensor([d.image_token_id], dtype=torch.int32) + # The attention op sizes its workspace in place per call and never shrinks it. + self._workspace = torch.empty(0, dtype=torch.int8, device=self.ln_pre.device) + + def _images(self, multimodal_params: list[MultimodalParams]) -> list[tuple]: + """`(pixels [C, H, W], grid rows, grid cols)` of every image, in order. The + mistral-native processor pads the batch and preserves each image's true size.""" + images = [] + unit = self.patch * self.merge + for param in multimodal_params: + image = (param.multimodal_data or {}).get("image") + if image is None: + continue + pixels = image["pixel_values"] + sizes = image.get("image_sizes") + if pixels.ndim != 4 or pixels.shape[1] != 3: + raise ValueError("Image pixels must have shape [N, 3, H, W]") + if sizes is None or len(sizes) != pixels.shape[0]: + raise ValueError("Image sizes must describe every image in the padded batch") + for b in range(pixels.shape[0]): + if len(sizes[b]) != 2: + raise ValueError("Each image size must be [height, width]") + height, width = int(sizes[b][0]), int(sizes[b][1]) + if not (0 < height <= pixels.shape[-2] and 0 < width <= pixels.shape[-1]): + raise ValueError("Image sizes must fit within their padded pixels") + if height % unit or width % unit: + raise ValueError( + f"image {height}x{width} is not a whole number of {unit}-pixel " + "merge tiles; the processor resizes to them" + ) + images.append( + (pixels[b, :, :height, :width], height // self.patch, width // self.patch) + ) + return images + + def forward(self, multimodal_params: list[MultimodalParams]) -> torch.Tensor: + """The rows of every image of `multimodal_params`, request by request, image by + image: `[sum of merged tiles, text hidden]`.""" + images = self._images(multimodal_params) + dev = self.ln_pre.device + if not images: + return torch.empty([0, self.out_hidden], dtype=self.dtype, device=dev) + seg = [gh * gw for _, gh, gw in images] + if self._validate: + logger.info( + "modeling_v2 image forward target=%s rank=%d grids=%s merged_rows=%d", + __name__.rsplit(".", 1)[0], + self._rank, + [(gh, gw) for _, gh, gw in images], + sum(seg) // self.merge**2, + ) + # Each patch as one row, (channel, row, col) within it, patches row-major. + patches = torch.cat( + [ + torch.t( + nn.functional.unfold( + torch.unsqueeze(px.to(device=dev, dtype=self.dtype), 0), + kernel_size=self.patch, + stride=self.patch, + )[0] + ) + for px, _, _ in images + ], + 0, + ) + positions = torch.cat( + [ + torch.reshape( + torch.stack( + torch.meshgrid( + torch.arange(gh, device=dev), + torch.arange(gw, device=dev), + indexing="ij", + ), + -1, + ), + [-1, 2], + ) + for _, gh, gw in images + ], + 0, + ) + step = _vision_step(seg, dev, self._workspace) + + h = flashinfer_rmsnorm(cublas_mm(patches, self.patch_w), self.ln_pre, _VISION_EPS) + x = flashinfer_rmsnorm(h, self.blocks[0].attn_norm, _VISION_EPS) + residual = h + for j, B in enumerate(self.blocks): + attn = _vision_attention(x, B, positions, step, self.heads, self.head_dim, self.theta) + x, residual = flashinfer_fused_add_rmsnorm(attn, residual, B.ffn_norm, _VISION_EPS) + mlp = cublas_mm(flashinfer_silu_and_mul(cublas_mm(x, B.w13)), B.w2) + norm_next = self.blocks[j + 1].attn_norm if j + 1 < len(self.blocks) else self.proj_norm + x, residual = flashinfer_fused_add_rmsnorm(mlp, residual, norm_next, _VISION_EPS) + + # x is the pre-projector-normed tower output. + tiles = torch.cat( + [ + _merge_tiles(xi, gh, gw, self.merge) + for xi, (_, gh, gw) in zip(torch.split(x, seg, 0), images) + ], + 0, + ) + merged = cublas_mm(tiles, self.merge_w) + return cublas_mm(gelu_erf(cublas_mm(merged, self.w_in)), self.w_out) + + +def _image_rows( + vision: Optional[VisionTower], multimodal_params: list[MultimodalParams] +) -> Optional[torch.Tensor]: + """This step's image rows, one per [IMG] position its context chunks cover, in + the order those positions appear; None when the step covers none. + + The tower runs once per request, on the first step that needs it; its rows stay + on the request's multimodal data for the later chunks. A chunk -- under chunked + prefill, or resuming after a reused prefix -- takes rows `[cached, cached + in + chunk)` of its request's, counted by the request's [IMG] cumsum.""" + if vision is None: + if any((p.multimodal_data or {}).get("image") is not None for p in multimodal_params): + raise ValueError( + "an image reached a Mistral Large 4 target whose checkpoint has no vision encoder" + ) + return None + # Filter before encoding and gathering so inactive requests neither run the + # tower nor shift the embedding offsets of active requests in a mixed batch. + multimodal_params = [ + param + for param in multimodal_params + if param.multimodal_runtime is None or param.multimodal_runtime.num_mm_tokens_in_chunk > 0 + ] + embeds = find_input_mm_embeds( + get_multimodal_embeddings(vision.forward, multimodal_params), multimodal_params + ) + if not embeds: + return None + rows = embeds[0] if len(embeds) == 1 else torch.cat(embeds, 0) + return rows if rows.shape[0] else None + + +def _splice_image_rows( + embed: torch.Tensor, + input_ids: torch.Tensor, + rows: torch.Tensor, + mm_token_indices: Optional[torch.Tensor], +) -> torch.Tensor: + """The step's input embeddings with `rows` written over its [IMG] positions, + `mm_token_indices` (which the executor computes on the host from the shell's + `multimodal_token_ids`); every other position keeps its token embedding.""" + if mm_token_indices is None: + raise ValueError("an image step arrived without the executor's mm_token_indices") + if mm_token_indices.shape[0] != rows.shape[0]: + raise ValueError( + f"{mm_token_indices.shape[0]} [IMG] positions in this step but " + f"{rows.shape[0]} image rows for them" + ) + h = nn.functional.embedding(input_ids, embed) + return h.index_copy_(0, mm_token_indices, rows.to(h.dtype)) + + +class ModelingV2MistralLarge4Nvfp4Sm107Dep( + DecoderModelForCausalLM[MistralLarge4DepModelingV2, PretrainedConfig] +): + def __init__(self, model_config: ModelConfig): + cfg = model_config.pretrained_config + super().__init__( + MistralLarge4DepModelingV2(model_config), + config=model_config, + hidden_size=cfg.hidden_size, + vocab_size=cfg.vocab_size, + ) + self.softcap = float(cfg.softmax_tanh_gating) + self._vision: Optional[VisionTower] = None + vis = getattr(cfg, "vision_config", None) + self._image_token_ids = ( + torch.tensor([vis.image_token_id], dtype=torch.int32) if vis is not None else None + ) + + self.spec_config = model_config.spec_config + self.draft_model = None + self.draft_config = None + self.spec_worker = None + if self.spec_config is not None: + # This is allocation geometry for the independently paged BF16 + # draft, not a second registry-resolved model or target cache. + draft_cfg = PretrainedConfig( + num_hidden_layers=int(self.spec_config._draft_model_config["n_layers"]), + hidden_size=cfg.hidden_size, + num_attention_heads=cfg.num_attention_heads, + num_key_value_heads=1, + kv_lora_rank=cfg.kv_lora_rank, + qk_rope_head_dim=cfg.qk_rope_head_dim, + qk_nope_head_dim=cfg.qk_nope_head_dim, + v_head_dim=cfg.v_head_dim, + intermediate_size=cfg.intermediate_size, + vocab_size=cfg.vocab_size, + max_position_embeddings=cfg.max_position_embeddings, + torch_dtype=torch.bfloat16, + ) + self.draft_config = replace( + model_config, + pretrained_config=draft_cfg, + quant_config=QuantConfig(), + quant_config_dict=None, + sparse_attention_config=None, + spec_config=None, + modeling_v2_target=None, + ) + self.draft_model = DraftModel(self, self.draft_config) + self.spec_worker = get_spec_worker( + self.spec_config, + model_config, + model_config.mapping, + use_separate_draft_kv_cache=True, + ) + self.spec_worker.set_draft_model(self.draft_model) + self.epilogue.append(self.spec_worker) + + def set_guided_decoder(self, guided_decoder: CapturableGuidedDecoder) -> bool: + if self.spec_worker is None: + return False + return self.spec_worker.set_guided_decoder(guided_decoder) + + @property + def multimodal_token_ids(self) -> Optional[torch.Tensor]: + return self._image_token_ids + + def forward( + self, + attn_metadata: AttentionMetadata, + input_ids: torch.IntTensor | None = None, + position_ids: torch.IntTensor | None = None, + inputs_embeds: torch.FloatTensor | None = None, + return_context_logits: bool = False, + spec_metadata=None, + lora_params: dict | None = None, + **kwargs, + ) -> torch.Tensor: + """Soft-cap target logits before verification and return the worker result.""" + worker_input_ids = input_ids + multimodal_params = kwargs.get("multimodal_params") or [] + if multimodal_params: + rows = _image_rows(self._vision, multimodal_params) + if rows is not None: + inputs_embeds = _splice_image_rows( + self.model.w["embed"], input_ids, rows, kwargs.get("mm_token_indices") + ) + input_ids = None + if self.spec_worker is None: + logits = super().forward( + attn_metadata, + cast(torch.IntTensor, input_ids), + position_ids, + inputs_embeds, + return_context_logits, + spec_metadata, + lora_params, + **kwargs, + ) + return tanh_softcap(logits, self.softcap) + hidden = self.model( + attn_metadata=attn_metadata, + input_ids=input_ids, + position_ids=position_ids, + inputs_embeds=inputs_embeds, + spec_metadata=spec_metadata, + lora_params=lora_params, + **kwargs, + ) + logits = self.logits_processor( + nn.functional.embedding(spec_metadata.gather_ids, hidden), + self.lm_head, + attn_metadata, + True, + ) + return self.spec_worker( + input_ids=worker_input_ids, + position_ids=position_ids, + hidden_states=hidden, + logits=tanh_softcap(logits, self.softcap), + attn_metadata=attn_metadata, + spec_metadata=spec_metadata, + draft_model=self.draft_model, + resource_manager=kwargs.get("resource_manager"), + ) + + def load_weights(self, weights, *args, **kwargs): + _weights.MODEL_WEIGHTS.load(self, weights) + + def load_checkpoint(self, checkpoint_dir: str) -> None: + _weights.MODEL_WEIGHTS.load_checkpoint(self, checkpoint_dir) + + def post_load_weights(self): + super().post_load_weights() + self.model.post_load_weights() + if step_contract_enabled(): + logger.info( + "modeling_v2 loaded target=%s class=%s rank=%d world=%d vision=%s", + __name__.rsplit(".", 1)[0], + type(self).__name__, + self.model_config.mapping.rank, + self.model_config.mapping.world_size, + getattr(self.config, "vision_config", None) is not None, + ) + if getattr(self.config, "vision_config", None) is not None: + self._vision = VisionTower(self.model) + if self.draft_model is not None: + self.draft_model._build_fused_kv_buffers() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__dep/weights.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__dep/weights.py new file mode 100644 index 000000000000..13f91a34ab8a --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__dep/weights.py @@ -0,0 +1,1251 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Weight table and loader: mistral-large-4-nvfp4 / sm_107 / dep, the language model. + +The checkpoint is Mistral-native (`consolidated-*.safetensors`, `layers.N.attention.wq`, ...) +and ModelOpt `MIXED_PRECISION`: the routed experts NVFP4, the other big linears FP8 128x128 +block (`FP8_PB_WO`), everything else bf16. Every rank is handed the whole of it, and under +attention data parallelism the topology divides the requests, not the weights, so the table +makes one split only: + +* **expert windows** -- rank `r` (its `moe_ep_rank`) loads routed experts `[Er, Er + E)`, + `E = n_routed_experts / moe_ep_size` whole experts (32 at 4 ranks, 16 at 8 ranks over two + nodes of four), the windows tiling the 128 once; +* **replicated, whole** -- everything else: `wq`, `wkv_b`, `wo` and the MLA SSMax `s`/`b` over + all 64 heads; `wkv_a_with_mqa` and `kv_a_norm`; the whole indexer; the dense layer's and the + shared expert's `w1`/`w3`/`w2` at full width, which the forward runs on each rank's own + tokens; the router, the embedding, every norm and `lerope.alpha`; and the shell's `lm_head`, + which under attention DP is the full `[vocab, hidden]` on every rank. + +So no row slices one of its checkpoint tensors: each source is read whole, and only the expert +rows choose, by rank, which tensors they read. + +Layer sets come from the config: `wkv_a_with_mqa` and `kv_a_norm` exist on the KV-owning layers +only (`kv_source_layers[i] == i`; the sharers read their source's latent), the indexer on the +`index_sharing == 'full'` layers only, the dense MLP on the first `first_k_dense_replace` +layers and the router, shared expert and routed experts on the rest. + +Load-time layouts, each the one its consumer reads: + +* `wkv_b` regrouped per head `[k_nope | v]` -> `[all k_nope ; all v]` over all heads (k_nope is + 64 wide on RoPE layers and 128 on NoPE layers); +* the dense MLP and the shared expert `[w1 ; w3]`, gate rows first, the half order + `flashinfer_silu_and_mul` reads; +* FP8 linears (`wq`, `wo`, the indexer's `wq_b`, the dense and shared `w1`/`w3`/`w2`): e4m3 + `[N, K]` as stored, and the fp32 block scales -- stored as `.weight_scale` + `[N/128, 1, K/128, 1]`, squeezed here to the `[N/128, K/128]` grid -- in DeepGEMM's packed + UE8M0 operand, `transform_sf_into_required_layout(recipe=(1, 128, 128))`: what + `FP8BlockScalesLinearMethod.transform_weights` installs as `weight_scale` on sm_100f. That + operand is an MN-major `[N, ceil(K/512)]` int32 view; the parameter holds the dense + `[ceil(K/512), N]` behind it. The layer views convert `.t()` to Rubin R128c4 K32 scales once, + replicating each K128 exponent four times. The method's + `resmooth_to_fp8_e8m0` is not run: every scale is asserted an exact power of two, which makes + it a power-of-two re-exponent of the same values; +* the indexer's `wk` and `weights_proj` stacked into one `[128 + 32, hidden]` bf16 GEMM operand; + its `k_norm` and `lerope.alpha` widened to fp32. + +The routed experts are this rank's window stacked on an expert axis, in the NVFP4 MoE runner's +layout: e2m1 codes, two per byte, `[E, 2I, H/2]` / `[E, H, I/2]` uint8 and their e4m3 block +scales, one per 16 inputs, `[E, 2I, H/16]` / `[E, H, I/16]`. Each source lands as stored -- +FC1's rows `[w3 ; w1]`, up rows first, the order the runner's FC1 concatenates them in -- and +once every source of an expert stack has landed, its row's `layout` rearranges each expert in +place (the runner reads nothing else, and a wrong layout costs 300-3400 bf16 ulp and raises +nothing): + +* FC1, codes and block scales alike: the halves interleaved (row `2i` up row `i`, row `2i + 1` + gate row `i`), then the 32-row block shuffle (within each 32 rows, row `4u + v` moves to + `8v + u`); +* FC2: the 32-row block shuffle; +* both block scales: then the 128x4 swizzle (`torch.ops.trtllm.block_scale_interleave`'s order, + written here in torch so the table also runs on meta tensors). + +The runner's output scalars are not derived here: the four per-tensor scalars are handed over +as stored -- `isc1` and `isc2`, the one FC1 and FC2 activation `input_scale` of the layer, and +per local expert `ws2_13` (`w1`'s `weight_scale_2`, which `w3` shares) and `ws2_2`. ModelOpt +stores these as reciprocals of the global scales a quantizer divides by. + +Before any of that is read, `load` reads the per-tensor scalars of all the layer's experts, +the other ranks' included, and asserts what the forward relies on: `w1` and `w3` store the same +`input_scale` and `weight_scale_2` on every expert, so one FC1 alpha serves the stacked +`[w3 ; w1]`; and each projection stores one `input_scale` for every expert, so one NVFP4 +quantization of the gathered activations is right for every expert on every rank and the +windows' partial outputs sum. + +The BF16 vision encoder, pre-projector norm, patch merger and language adapter are +replicated on every rank when the configuration declares a vision encoder. + +`expected_unconsumed` accounts for off-rank routed experts and disabled DSpark draft weights. +Off-rank expert scalars are still read to check shared scales; their codes and block scales +are never read. +At `moe_ep_size` 1 every expert belongs to the rank and is read, sliced. +""" + +from __future__ import annotations + +import collections +import functools +import json +import os +from dataclasses import dataclass +from pathlib import Path +from types import SimpleNamespace +from typing import Any, Callable + +import torch +import torch.nn as nn +from safetensors import safe_open + +from tensorrt_llm.logger import logger +from tensorrt_llm.quantization.utils.fp8_utils import transform_sf_into_required_layout + +F8, F32, I32, U8 = torch.float8_e4m3fn, torch.float32, torch.int32, torch.uint8 + +#: The FP8 block edge on both weight axes (the checkpoint's `FP8_PB_WO` 128x128 blocks), and +#: the K granularity of the DeepGEMM scale recipe `(1, 128, 128)`. +BLOCK = 128 +#: UE8M0 bytes per int32 word of the DeepGEMM scale operand: four K blocks. +SF_PACK = 4 +#: NVFP4: one e4m3 block scale per 16 inputs (the checkpoint's `group_size`), two e2m1 codes +#: per byte. +NVFP4_BLOCK = 16 +E2M1_PER_BYTE = 2 + +#: A routed expert's three projections: `w1` gate, `w3` up, `w2` down. +PROJECTIONS = ("w1", "w2", "w3") +#: The per-tensor NVFP4 scalars each of them carries beside its codes and block scales. +EXPERT_SCALARS = ("weight_scale_2", "input_scale") + +#: The vision tower's checkpoint prefixes. +VISION_PREFIXES = ( + "vision_encoder.", + "patch_merger.", + "pre_mm_projector_norm.", + "vision_language_adapter.", +) + + +#: `layers=ALL_LAYERS` is the common case and the default. A sentinel rather +#: than a lambda so the table reads as data, and rather than `True` so the +#: field has one type of answer: which layers. +ALL_LAYERS = object() + + +@dataclass(frozen=True) +class W: + """One weight role: what it is, where its bytes come from, what happens on the way. + + `shape` takes the dimension bundle and returns the declared shape. + + `src` says where the bytes come from: a checkpoint key template with `{p}` + for the layer prefix (`layers.N`), or a callable `(d, i) -> [(key, index), ...]` + where `index` is where in the parameter that tensor lands (`None`: all of it). + A `key` may be a tuple of per-tensor scalars, which `transform` is handed together. + Every source is read whole. + + `layers` says which layers carry this weight: `ALL_LAYERS`, `None` for a + weight that is not per-layer, or a callable `d -> iterable[int]`. Two rows + may share a name over disjoint layer sets, with different shapes. + + `transform` runs on the source tensor, on the destination's device. + + `layout` runs once every source of the parameter has landed, on each slot of its leading + (expert) axis, and the slot is overwritten with the result: a rearrangement that mixes rows + several sources fill, which no single source's `transform` can make. + """ + + name: str + shape: Callable[[Any], tuple[int, ...]] + src: str | Callable + dtype: torch.dtype | None = None + transform: Callable | None = None + layers: object | None = ALL_LAYERS + layout: Callable | None = None + when: Callable | None = None + tag: str = "l" + + def layer_indices(self, d: SimpleNamespace) -> list[int] | None: + """The layers this weight exists on, or None if it is not per-layer.""" + if self.layers is None: + return None + if self.layers is ALL_LAYERS: + return list(range(d.num_layers)) + return list(self.layers(d)) + + def sources(self, d: SimpleNamespace, i: int | None, prefix: str) -> list[tuple[Any, Any]]: + """`[(checkpoint key | key tuple, index into the parameter | None), ...]`.""" + if callable(self.src): + return list(self.src(d, i)) + return [(self.src.format(p=prefix), None)] + + +def _keys(ckpt) -> tuple[str, ...]: + """The checkpoint keys of one source: itself, or the members of a key tuple.""" + return ckpt if isinstance(ckpt, tuple) else (ckpt,) + + +def _sf_words(k: int) -> int: + """int32 words per row of the DeepGEMM scale operand of a K-wide weight.""" + return -(-k // (BLOCK * SF_PACK)) + + +def _assert_ue8m0(s: torch.Tensor) -> None: + """Every block scale is a positive normal power of two, so its UE8M0 byte (the fp32 + exponent field) holds it exactly. A meta tensor carries no values to check.""" + if s.is_meta: + return + bits = s.view(torch.int32) + exp = (bits >> 23) & 0xFF + exact = ((bits & 0x807FFFFF) == 0) & (exp > 0) & (exp < 255) + assert bool(exact.all()), ( + f"weight_scale: {int((~exact).sum())} of {s.numel()} block scales are not a positive " + "normal power of two (UE8M0); the scale layouts below would change their values" + ) + + +def _deepgemm_sf(s: torch.Tensor, core) -> torch.Tensor: + """fp32 `[N/128, K/128]` block scales -> the dense `[ceil(K/512), N]` int32 storage of + DeepGEMM's MN-major packed UE8M0 operand, whose `.t()` is that operand. N is a multiple of + 128 for every role here, so the operand's row stride is N and its storage has no padding.""" + _assert_ue8m0(s) + n_blk, k_blk = s.shape + sf = transform_sf_into_required_layout( + s, mn=n_blk * BLOCK, k=k_blk * BLOCK, recipe=(1, BLOCK, BLOCK), is_sfa=False + ) + return sf.t() + + +def _block_sf(s: torch.Tensor, core) -> torch.Tensor: + """An FP8 linear's block scales as stored, fp32 `[N/128, 1, K/128, 1]`: squeezed to the + `[N/128, K/128]` grid, then `_deepgemm_sf`.""" + assert s.dim() == 4 and s.shape[1] == 1 and s.shape[3] == 1, ( + f"FP8 block scale stored as {tuple(s.shape)}, not [N/128, 1, K/128, 1]" + ) + return _deepgemm_sf(s.reshape(s.shape[0], s.shape[2]), core) + + +def _shared_scalar(t: tuple[torch.Tensor, ...], core) -> torch.Tensor: + """One per-tensor scalar the checkpoint stores several times (`w1`'s and `w3`'s, or every + expert's of the window): asserted the same value everywhere, then that value.""" + first = t[0] + if not first.is_meta: + values = torch.stack(t) + assert bool((values == first).all()), ( + f"{int((values != first).sum())} of {len(t)} copies of one shared scale differ from " + f"the first, {first.item()!r}" + ) + return first + + +def _regroup_kv_b(t: torch.Tensor, core) -> torch.Tensor: + """All heads' `[heads * (k_nope + v), kv_lora]` rows, per-head `[k_nope | v]` -> + `[heads * k_nope ; heads * v]`: the K-B operand, then the V-B one. k_nope is read off the + tensor: 64 on RoPE layers, 128 on NoPE layers.""" + cfg = core.model_config.pretrained_config + heads = cfg.num_attention_heads + v = cfg.v_head_dim + t3 = t.reshape(heads, t.shape[0] // heads, t.shape[1]) + nope = t3.shape[1] - v + return torch.cat( + [t3[:, :nope, :].reshape(heads * nope, -1), t3[:, nope:, :].reshape(heads * v, -1)], dim=0 + ).contiguous() + + +def _to_fp32(t: torch.Tensor, core) -> torch.Tensor: + return t.to(torch.float32) + + +# --- the NVFP4 MoE runner's expert layout, one expert at a time --- + +#: The runner's row shuffle works on blocks of this many rows. +SHUFFLE_ROWS = 32 +#: The block-scale swizzle's tile: 128 rows by 4 scale columns. +SWIZZLE_ROWS, SWIZZLE_COLS = 128, 4 + + +@functools.lru_cache(maxsize=None) +def _block32_perm(rows: int, device) -> torch.Tensor: + """Gather index of the 32-row block shuffle: within each block of 32 rows, source row + `4u + v` lands at row `8v + u`.""" + assert rows % SHUFFLE_ROWS == 0, rows + j = torch.arange(SHUFFLE_ROWS) + one = torch.empty(SHUFFLE_ROWS, dtype=torch.long) + one[(j % 4) * 8 + j // 4] = j + blocks = torch.arange(rows // SHUFFLE_ROWS).repeat_interleave(SHUFFLE_ROWS) * SHUFFLE_ROWS + return (one.repeat(rows // SHUFFLE_ROWS) + blocks).to(device) + + +@functools.lru_cache(maxsize=None) +def _fc1_perm(rows: int, device) -> torch.Tensor: + """Gather index of FC1's `[up ; gate]` rows: interleaved (row `2i` up row `i`, row `2i + 1` + gate row `i`), then block-shuffled.""" + interleave = torch.empty(rows, dtype=torch.long) + interleave[0::2] = torch.arange(rows // 2) + interleave[1::2] = torch.arange(rows // 2, rows) + return interleave.to(device)[_block32_perm(rows, device)] + + +def _swizzle(s: torch.Tensor) -> torch.Tensor: + """One expert's `[M, C]` e4m3 block scales in the 128x4-swizzled byte order, kept `[M, C]`: + byte `(m, c)` lands at flat offset `(m // 128) * 128 * C + (c // 4) * 512 + (m % 32) * 16 + + ((m % 128) // 32) * 4 + c % 4`. Both of this target's scale stacks tile it exactly (M a + multiple of 128, C of 4), so there is no padding.""" + m, c = s.shape + assert m % SWIZZLE_ROWS == 0 and c % SWIZZLE_COLS == 0, (m, c) + tiles = s.view(torch.uint8).reshape(m // SWIZZLE_ROWS, 4, 32, c // SWIZZLE_COLS, SWIZZLE_COLS) + return tiles.permute(0, 3, 2, 1, 4).reshape(m, c).view(s.dtype) + + +def _fc1_codes(w: torch.Tensor) -> torch.Tensor: + return torch.index_select(w, 0, _fc1_perm(w.shape[0], w.device)) + + +def _fc1_scales(s: torch.Tensor) -> torch.Tensor: + rows = torch.index_select(s.view(torch.uint8), 0, _fc1_perm(s.shape[0], s.device)) + return _swizzle(rows.view(s.dtype)) + + +def _fc2_codes(w: torch.Tensor) -> torch.Tensor: + return torch.index_select(w, 0, _block32_perm(w.shape[0], w.device)) + + +def _fc2_scales(s: torch.Tensor) -> torch.Tensor: + rows = torch.index_select(s.view(torch.uint8), 0, _block32_perm(s.shape[0], s.device)) + return _swizzle(rows.view(s.dtype)) + + +def _materialize(entry) -> torch.Tensor: + """One checkpoint entry's bytes, whole: a tensor, or a lazy safetensors slice, which reads + them only now. `[...]`, not `[:]`: the per-tensor scalars are 0-dim.""" + return entry[...] + + +#: The checkpoint's shard index, beside its `consolidated-*.safetensors`. +SHARD_INDEX = "consolidated.safetensors.index.json" + + +def _checkpoint_shards(checkpoint_dir) -> tuple[Path, dict[str, str]]: + """`(directory, key -> shard file)` of a checkpoint directory: its index when it ships + one, else the header of each `consolidated*.safetensors`.""" + directory = Path(checkpoint_dir) + index = directory / SHARD_INDEX + if index.is_file(): + return directory, json.loads(index.read_text())["weight_map"] + weight_map: dict[str, str] = {} + for path in sorted(directory.glob("consolidated*.safetensors")): + with safe_open(str(path), framework="pt", device="cpu") as handle: + weight_map.update(dict.fromkeys(handle.keys(), path.name)) + if not weight_map: + raise FileNotFoundError(f"no consolidated*.safetensors in {directory}") + return directory, weight_map + + +def _shard_files(core, weights) -> tuple[Path, dict[str, str]] | None: + """`(directory, key -> shard file)` of the checkpoint a dict of sources was read from, or + None when the sources are not the shard files of one (the directory the config records + lacks the index, or the index does not name every key `weights` holds).""" + cfg = getattr(core.model_config, "pretrained_config", None) + root = getattr(cfg, "_name_or_path", None) + if not root: + return None + index = Path(root) / SHARD_INDEX + if not index.is_file(): + return None + weight_map = json.loads(index.read_text())["weight_map"] + if not set(weights.keys()) <= set(weight_map): + return None + return Path(root), weight_map + + +def _drop_page_cache(path: Path) -> None: + """Advise the kernel to drop a consumed shard file's cached pages. Advice only: a + filesystem that ignores it costs memory pressure, not correctness.""" + try: + fd = os.open(path, os.O_RDONLY) + except OSError: + return + try: + os.posix_fadvise(fd, 0, 0, os.POSIX_FADV_DONTNEED) + except OSError: + pass + finally: + os.close(fd) + + +# --- which layers carry a weight --- + + +def _owners(d): + """Layers that own a latent KV. The others read `kv_source_layers[i]`'s and project none.""" + return [i for i in range(d.num_layers) if d.kv_source[i] == i] + + +def _rope_owners(d): + return [i for i in _owners(d) if not d.nope_layer[i]] + + +def _nope_owners(d): + return [i for i in _owners(d) if d.nope_layer[i]] + + +def _rope_layers(d): + return [i for i in range(d.num_layers) if not d.nope_layer[i]] + + +def _nope_layers(d): + return [i for i in range(d.num_layers) if d.nope_layer[i]] + + +def _indexer_layers(d): + """Layers that own an indexer. The 'shared' ones reuse the most recent one's selection.""" + return [i for i in range(d.num_layers) if d.full_indexer[i]] + + +def _dense_layers(d): + return range(d.dense_layers) + + +def _moe_layers(d): + return range(d.dense_layers, d.num_layers) + + +# --- where the bytes come from --- + + +def _stack(*parts, prefix="layers"): + """Several checkpoint tensors into one parameter: each `(key template, index(d))`, the + index being where in the parameter that tensor lands.""" + return lambda d, i: [(t.format(p=f"{prefix}.{i}"), index(d)) for t, index in parts] + + +def _expert_key(i: int, e: int, suffix: str) -> str: + return f"layers.{i}.experts.{e}.{suffix}" + + +def _expert_window(*parts): + """This rank's routed experts, local slot `s` <- global expert `expert_offset + s`: each + `(checkpoint suffix | suffix tuple, index(d) after the expert axis)`.""" + + def key(i, e, suffix): + if isinstance(suffix, tuple): + return tuple(_expert_key(i, e, s) for s in suffix) + return _expert_key(i, e, suffix) + + def build(d, i): + return [ + (key(i, d.expert_offset + slot, suffix), (slot, *index(d))) + for slot in range(d.local_experts) + for suffix, index in parts + ] + + return build + + +def _window_scalar(*suffixes): + """One per-tensor scalar every expert of the window stores: all of the window's copies, one + key tuple, into element 0.""" + return lambda d, i: [ + ( + tuple( + _expert_key(i, d.expert_offset + slot, s) + for slot in range(d.local_experts) + for s in suffixes + ), + (0,), + ) + ] + + +def _expert_scalar_keys(d) -> list[str]: + """Every routed expert's per-tensor scalars on every MoE layer, the other ranks' included: + what `load` reads, and checks, before anything else.""" + return [ + _expert_key(i, e, f"{proj}.{s}") + for i in _moe_layers(d) + for e in range(d.num_experts) + for proj in PROJECTIONS + for s in EXPERT_SCALARS + ] + + +def _check_expert_scalars(scalars: dict[str, torch.Tensor], d) -> None: + """Per MoE layer, over all of its experts: `w1` and `w3` store the same `input_scale` and + `weight_scale_2` on every expert; `w1` and `w2` each store one `input_scale` for every expert; + every scalar is a positive finite fp32. Meta tensors carry no values to check.""" + if scalars and next(iter(scalars.values())).is_meta: + return + experts = range(d.num_experts) + for i in _moe_layers(d): + v = { + (proj, s): torch.stack([scalars[_expert_key(i, e, f"{proj}.{s}")] for e in experts]) + for proj in PROJECTIONS + for s in EXPERT_SCALARS + } + for (proj, s), t in v.items(): + bad = (~(torch.isfinite(t) & (t > 0))).nonzero().flatten() + assert bad.numel() == 0, ( + f"layers.{i}: {proj}.{s} of expert {int(bad[0])} is {t[bad[0]].item()!r}, " + "not a positive finite scale" + ) + for s in EXPERT_SCALARS: + bad = (v["w1", s] != v["w3", s]).nonzero().flatten() + assert bad.numel() == 0, ( + f"layers.{i}: w1 and w3 {s} differ on {bad.numel()} of {len(experts)} experts, " + f"first expert {int(bad[0])}: {v['w1', s][bad[0]].item()!r} vs " + f"{v['w3', s][bad[0]].item()!r}" + ) + for proj in ("w1", "w2"): + isc = v[proj, "input_scale"] + bad = (isc != isc[0]).nonzero().flatten() + assert bad.numel() == 0, ( + f"layers.{i}: {proj}.input_scale is not one value over the experts: expert " + f"{int(bad[0])} stores {isc[bad[0]].item()!r}, expert 0 {isc[0].item()!r}" + ) + + +def _read_scalars(keys: list[str], weights, shards) -> dict[str, torch.Tensor]: + """The given per-tensor scalars, off the shard files one file at a time when they are + found, otherwise off the given dict.""" + if shards is None: + return {key: _materialize(weights[key]) for key in keys} + directory, weight_map = shards + by_file: dict[str, list] = collections.defaultdict(list) + for key in keys: + by_file[weight_map[key]].append(key) + out = {} + for name in sorted(by_file): + with safe_open(str(directory / name), framework="pt", device="cpu") as handle: + for key in by_file[name]: + out[key] = handle.get_slice(key)[...].clone() + return out + + +def _fp8_linear(name, key, n, k, layers): + """An FP8 128x128-block linear: e4m3 `[N, K]` and its DeepGEMM scale operand's storage.""" + return ( + W(name, shape=lambda d: (n(d), k(d)), dtype=F8, src=key, layers=layers), + W( + f"{name}_s", + shape=lambda d: (_sf_words(k(d)), n(d)), + dtype=I32, + src=key.replace(".weight", ".weight_scale"), + transform=_block_sf, + layers=layers, + ), + ) + + +def _mlp(module, inter, layers): + """A SwiGLU MLP at full width: `[w1 ; w3]` stacked by output rows, gate first (the half + order `flashinfer_silu_and_mul` reads), and `w2`. The scales follow; in the scale storage + the stacked axis is the second one.""" + p = "{p}." + module + return ( + W( + "mlp_gu", + shape=lambda d: (2 * inter(d), d.hidden), + dtype=F8, + src=_stack( + (f"{p}.w1.weight", lambda d: (slice(0, inter(d)),)), + (f"{p}.w3.weight", lambda d: (slice(inter(d), 2 * inter(d)),)), + ), + layers=layers, + ), + W( + "mlp_gu_s", + shape=lambda d: (_sf_words(d.hidden), 2 * inter(d)), + dtype=I32, + src=_stack( + (f"{p}.w1.weight_scale", lambda d: (slice(None), slice(0, inter(d)))), + (f"{p}.w3.weight_scale", lambda d: (slice(None), slice(inter(d), 2 * inter(d)))), + ), + transform=_block_sf, + layers=layers, + ), + *_fp8_linear("mlp_dn", f"{p}.w2.weight", lambda d: d.hidden, inter, layers), + ) + + +def _kv_b(nope, layers): + """`wkv_b`, all heads, regrouped. Per-head rows are `nope(d) + v` wide.""" + return W( + "wkv_b", + shape=lambda d: (d.heads * (nope(d) + d.v_dim), d.kv_lora), + src="{p}.attention.wkv_b.weight", + transform=_regroup_kv_b, + layers=layers, + ) + + +def _vision_blocks(d): + return list(range(d.v_layers)) + + +def _has_vision(d): + return d.has_vision + + +def _flatten_patch_conv(tensor, core): + return tensor.flatten(1) + + +def _draft_layers(d): + return range(d.draft_layers) + + +def _draft_source(suffix): + return lambda d, i: [(f"mtp_dspark.backbone.layers.{i}.{suffix}", None)] + + +class MistralLarge4DepWeights: + """mistral-large-4-nvfp4 / sm_107 / dep.""" + + WEIGHTS: tuple[W, ...] = ( + # --- MLA: every layer, KV owners only where marked --- + W("attn_norm", shape=lambda d: (d.hidden,), src="{p}.attention_norm.weight"), + # Per head `[nope 64 | rope 64]` on RoPE layers, 128 nope on NoPE layers: 128 either way. + *_fp8_linear( + "wq", + "{p}.attention.wq.weight", + lambda d: d.heads * d.qk_dim, + lambda d: d.hidden, + ALL_LAYERS, + ), + W("ssmax_s", shape=lambda d: (d.heads,), src="{p}.attention.ssmax_scaling.s"), + W("ssmax_b", shape=lambda d: (d.heads,), src="{p}.attention.ssmax_scaling.b"), + # `[c_kv 512 | k_pe 64]` on RoPE owners; NoPE owners project the 512 latent alone. + W( + "wkv_a", + shape=lambda d: (d.lat_dim, d.hidden), + src="{p}.attention.wkv_a_with_mqa.weight", + layers=_rope_owners, + ), + W( + "wkv_a", + shape=lambda d: (d.kv_lora, d.hidden), + src="{p}.attention.wkv_a_with_mqa.weight", + layers=_nope_owners, + ), + W( + "kv_a_norm", + shape=lambda d: (d.kv_lora,), + src="{p}.attention.kv_a_norm.weight", + layers=_owners, + ), + # Every layer absorbs through its own `wkv_b`, KV sharers included. + _kv_b(lambda d: d.nope, _rope_layers), + _kv_b(lambda d: d.qk_dim, _nope_layers), + *_fp8_linear( + "wo", + "{p}.attention.wo.weight", + lambda d: d.hidden, + lambda d: d.heads * d.v_dim, + ALL_LAYERS, + ), + W("ffn_norm", shape=lambda d: (d.hidden,), src="{p}.ffn_norm.weight"), + # --- the DSA indexer, on the 'full' layers --- + W( + "ix_q_norm_w", + shape=lambda d: (d.hidden,), + src="{p}.attention.indexer.q_input_norm.weight", + layers=_indexer_layers, + ), + W( + "ix_q_norm_b", + shape=lambda d: (d.hidden,), + src="{p}.attention.indexer.q_input_norm.bias", + layers=_indexer_layers, + ), + # Per head `[rope 64 | nope 64]`, the reverse of MLA's. + *_fp8_linear( + "ix_wq_b", + "{p}.attention.indexer.wq_b.weight", + lambda d: d.idx_heads * d.idx_dim, + lambda d: d.hidden, + _indexer_layers, + ), + # `[wk ; weights_proj]`: the key and the head weights, one GEMM. + W( + "ix_wk_wp", + shape=lambda d: (d.idx_dim + d.idx_heads, d.hidden), + src=_stack( + ("{p}.attention.indexer.wk.weight", lambda d: (slice(0, d.idx_dim),)), + ( + "{p}.attention.indexer.weights_proj.weight", + lambda d: (slice(d.idx_dim, d.idx_dim + d.idx_heads),), + ), + ), + layers=_indexer_layers, + ), + # fp32, as the fused key entry (LayerNorm in fp32) takes them. + W( + "ix_k_norm_w", + shape=lambda d: (d.idx_dim,), + dtype=F32, + src="{p}.attention.indexer.k_norm.weight", + transform=_to_fp32, + layers=_indexer_layers, + ), + W( + "ix_k_norm_b", + shape=lambda d: (d.idx_dim,), + dtype=F32, + src="{p}.attention.indexer.k_norm.bias", + transform=_to_fp32, + layers=_indexer_layers, + ), + W( + "ix_ssmax_s", + shape=lambda d: (d.idx_heads,), + src="{p}.attention.indexer.ssmax_scaling.s", + layers=_indexer_layers, + ), + W( + "ix_ssmax_b", + shape=lambda d: (d.idx_heads,), + src="{p}.attention.indexer.ssmax_scaling.b", + layers=_indexer_layers, + ), + # --- the SwiGLU MLP at full width: the dense layer's own, then each MoE layer's shared + # expert, one key --- + *_mlp("feed_forward", lambda d: d.dense_inter, _dense_layers), + *_mlp("shared_experts", lambda d: d.shared_inter, _moe_layers), + # --- routing --- + W( + "router", + shape=lambda d: (d.num_experts, d.hidden), + src="{p}.gate.weight", + layers=_moe_layers, + ), + # --- this rank's routed experts, NVFP4: `[w3 ; w1]` as stored, then each expert in the + # MoE runner's layout --- + W( + "fc1_w", + shape=lambda d: (d.local_experts, 2 * d.moe_inter, d.hidden // E2M1_PER_BYTE), + dtype=U8, + src=_expert_window( + ("w3.weight", lambda d: (slice(0, d.moe_inter),)), + ("w1.weight", lambda d: (slice(d.moe_inter, 2 * d.moe_inter),)), + ), + layers=_moe_layers, + layout=_fc1_codes, + ), + W( + "fc1_s", + shape=lambda d: (d.local_experts, 2 * d.moe_inter, d.hidden // NVFP4_BLOCK), + dtype=F8, + src=_expert_window( + ("w3.weight_scale", lambda d: (slice(0, d.moe_inter),)), + ("w1.weight_scale", lambda d: (slice(d.moe_inter, 2 * d.moe_inter),)), + ), + layers=_moe_layers, + layout=_fc1_scales, + ), + W( + "fc2_w", + shape=lambda d: (d.local_experts, d.hidden, d.moe_inter // E2M1_PER_BYTE), + dtype=U8, + src=_expert_window(("w2.weight", lambda d: ())), + layers=_moe_layers, + layout=_fc2_codes, + ), + W( + "fc2_s", + shape=lambda d: (d.local_experts, d.hidden, d.moe_inter // NVFP4_BLOCK), + dtype=F8, + src=_expert_window(("w2.weight_scale", lambda d: ())), + layers=_moe_layers, + layout=_fc2_scales, + ), + # The per-tensor scalars: the layer's one FC1 and one FC2 activation scale, and each + # local expert's global weight scales (`w3`'s are `w1`'s). + W( + "isc1", + shape=lambda d: (1,), + dtype=F32, + src=_window_scalar("w1.input_scale", "w3.input_scale"), + transform=_shared_scalar, + layers=_moe_layers, + ), + W( + "isc2", + shape=lambda d: (1,), + dtype=F32, + src=_window_scalar("w2.input_scale"), + transform=_shared_scalar, + layers=_moe_layers, + ), + W( + "ws2_13", + shape=lambda d: (d.local_experts,), + dtype=F32, + src=_expert_window((("w1.weight_scale_2", "w3.weight_scale_2"), lambda d: ())), + transform=_shared_scalar, + layers=_moe_layers, + ), + W( + "ws2_2", + shape=lambda d: (d.local_experts,), + dtype=F32, + src=_expert_window(("w2.weight_scale_2", lambda d: ())), + layers=_moe_layers, + ), + # --- not per-layer --- + W("final_norm", shape=lambda d: (d.hidden,), src="norm.weight", layers=None), + W("embed", shape=lambda d: (d.vocab, d.hidden), src="tok_embeddings.weight", layers=None), + # The LeRoPE per-frequency log-scales the rope tables are built from, fp32 as the + # reference widens them. + W( + "lerope_alpha", + shape=lambda d: (d.rope_dim // 2,), + dtype=F32, + src="lerope.alpha", + transform=_to_fp32, + layers=None, + ), + # BF16 vision encoder and projector, replicated on every rank. + W( + "vis_patch", + shape=lambda d: (d.v_hidden, d.v_channels * d.v_patch**2), + src="vision_encoder.patch_conv.weight", + transform=_flatten_patch_conv, + layers=None, + when=_has_vision, + ), + W( + "vis_ln_pre", + shape=lambda d: (d.v_hidden,), + src="vision_encoder.ln_pre.weight", + layers=None, + when=_has_vision, + ), + W( + "attn_norm", + shape=lambda d: (d.v_hidden,), + src="{p}.attention_norm.weight", + layers=_vision_blocks, + tag="v", + ), + W( + "qkv", + shape=lambda d: (3 * d.v_hidden, d.v_hidden), + src=_stack( + ("{p}.attention.wq.weight", lambda d: (slice(0, d.v_hidden),)), + ("{p}.attention.wk.weight", lambda d: (slice(d.v_hidden, 2 * d.v_hidden),)), + ("{p}.attention.wv.weight", lambda d: (slice(2 * d.v_hidden, 3 * d.v_hidden),)), + prefix="vision_encoder.transformer.layers", + ), + layers=_vision_blocks, + tag="v", + ), + W( + "wo", + shape=lambda d: (d.v_hidden, d.v_hidden), + src="{p}.attention.wo.weight", + layers=_vision_blocks, + tag="v", + ), + W( + "ffn_norm", + shape=lambda d: (d.v_hidden,), + src="{p}.ffn_norm.weight", + layers=_vision_blocks, + tag="v", + ), + W( + "w13", + shape=lambda d: (2 * d.v_inter, d.v_hidden), + src=_stack( + ("{p}.feed_forward.w1.weight", lambda d: (slice(0, d.v_inter),)), + ("{p}.feed_forward.w3.weight", lambda d: (slice(d.v_inter, 2 * d.v_inter),)), + prefix="vision_encoder.transformer.layers", + ), + layers=_vision_blocks, + tag="v", + ), + W( + "w2", + shape=lambda d: (d.v_hidden, d.v_inter), + src="{p}.feed_forward.w2.weight", + layers=_vision_blocks, + tag="v", + ), + W( + "vis_proj_norm", + shape=lambda d: (d.v_hidden,), + src="pre_mm_projector_norm.weight", + layers=None, + when=_has_vision, + ), + W( + "vis_merge", + shape=lambda d: (d.v_hidden, d.v_hidden * d.v_merge**2), + src="patch_merger.merging_layer.weight", + layers=None, + when=_has_vision, + ), + W( + "vis_w_in", + shape=lambda d: (d.hidden, d.v_hidden), + src="vision_language_adapter.w_in.weight", + layers=None, + when=_has_vision, + ), + W( + "vis_w_out", + shape=lambda d: (d.hidden, d.hidden), + src="vision_language_adapter.w_out.weight", + layers=None, + when=_has_vision, + ), + ) + + # BF16 draft weights use the same attention and dense-MLP partition as + # this target. The RNN and target-feature projection are replicated. + DRAFT_WEIGHTS: tuple[W, ...] = ( + W( + "draft_attn_norm", + lambda d: (d.hidden,), + _draft_source("attention_norm.weight"), + layers=_draft_layers, + ), + W( + "draft_wq", + lambda d: (d.heads * d.qk_dim, d.hidden), + _draft_source("attention.wq.weight"), + layers=_draft_layers, + ), + W( + "draft_wkv_a", + lambda d: (d.lat_dim, d.hidden), + _draft_source("attention.wkv_a_with_mqa.weight"), + layers=_draft_layers, + ), + W( + "draft_kv_a_norm", + lambda d: (d.kv_lora,), + _draft_source("attention.kv_a_norm.weight"), + layers=_draft_layers, + ), + W( + "draft_wkv_b", + lambda d: (d.heads * (d.nope + d.v_dim), d.kv_lora), + _draft_source("attention.wkv_b.weight"), + layers=_draft_layers, + ), + W( + "draft_wo", + lambda d: (d.hidden, d.heads * d.v_dim), + _draft_source("attention.wo.weight"), + layers=_draft_layers, + ), + W( + "draft_ffn_norm", + lambda d: (d.hidden,), + _draft_source("ffn_norm.weight"), + layers=_draft_layers, + ), + W( + "draft_w1", + lambda d: (d.dense_inter, d.hidden), + _draft_source("feed_forward.w1.weight"), + layers=_draft_layers, + ), + W( + "draft_w3", + lambda d: (d.dense_inter, d.hidden), + _draft_source("feed_forward.w3.weight"), + layers=_draft_layers, + ), + W( + "draft_w2", + lambda d: (d.hidden, d.dense_inter), + _draft_source("feed_forward.w2.weight"), + layers=_draft_layers, + ), + W( + "draft_final_norm", + lambda d: (d.hidden,), + "mtp_dspark.backbone.final_norm.weight", + layers=None, + ), + W( + "draft_target_norm", + lambda d: (d.hidden,), + "mtp_dspark.backbone.target_norm.weight", + layers=None, + ), + W( + "draft_target_projection", + lambda d: (d.hidden, d.hidden * d.draft_capture_layers), + "mtp_dspark.backbone.target_projection.weight", + layers=None, + ), + W( + "draft_mask", + lambda d: (d.draft_depth - 1, d.hidden), + "mtp_dspark.backbone.mask_embedding", + layers=None, + ), + W( + "draft_markov_embedding", + lambda d: (d.vocab, d.draft_rank), + "mtp_dspark.autoregressive_head.embedding.weight", + layers=None, + ), + W( + "draft_markov_output", + lambda d: (d.vocab, d.draft_rank), + "mtp_dspark.autoregressive_head.output.weight", + layers=None, + ), + W( + "draft_joint", + lambda d: (3 * d.draft_rank, d.hidden + 2 * d.draft_rank), + "mtp_dspark.autoregressive_head.joint_proj.weight", + layers=None, + ), + W( + "draft_joint_bias", + lambda d: (3 * d.draft_rank,), + "mtp_dspark.autoregressive_head.joint_proj.bias", + layers=None, + ), + ) + + def _entries(self, d): + return self.WEIGHTS + (self.DRAFT_WEIGHTS if d.draft_layers else ()) + + def declare(self, d: SimpleNamespace) -> nn.ParameterDict: + """Allocate every weight the table declares. + + Meta-init intercepts `torch.empty` here -- real CUDA storage arrives + when the engine materializes the registry. + """ + w = nn.ParameterDict() + for entry in self._entries(d): + dtype = entry.dtype or d.dtype + for key, _, _ in self._targets(entry, d): + assert key not in w, f"two rows declare {key}" + w[key] = nn.Parameter( + torch.empty(*entry.shape(d), dtype=dtype), requires_grad=False + ) + return w + + def _targets(self, entry: W, d: SimpleNamespace) -> list[tuple[str, int | None, str]]: + """`[(param key, layer index | None, checkpoint layer prefix), ...]`. + + One place decides which parameters an entry declares, so `declare` and + `manifest` cannot disagree about whether a weight exists. + """ + if entry.when is not None and not entry.when(d): + return [] + layers = entry.layer_indices(d) + if layers is None: + return [(entry.name, None, "")] + prefix = "vision_encoder.transformer.layers" if entry.tag == "v" else "layers" + return [(f"{entry.tag}{i}_{entry.name}", i, f"{prefix}.{i}") for i in layers] + + def manifest(self, core) -> dict: + """param key -> [(checkpoint key | key tuple, index into the param | None, transform)].""" + d = self.dims(core) + rows: dict = {} + for entry in self._entries(d): + for key, i, prefix in self._targets(entry, d): + assert key not in rows, f"two rows declare {key}" + rows[key] = [ + (ckpt, index, entry.transform) for ckpt, index in entry.sources(d, i, prefix) + ] + return rows + + def expected_unconsumed(self, core, weights) -> dict[str, str]: + """Declared unused keys: off-rank experts and disabled draft weights. Off-rank expert + scalars are still read to validate the layer's shared scales.""" + d = self.dims(core) + lo, hi = d.expert_offset, d.expert_offset + d.local_experts + keys = { + k: "DSpark draft disabled" + for k in weights.keys() + if not d.draft_layers and k.startswith("mtp_dspark.") + } + window = f"routed expert outside this rank's window [{lo}, {hi})" + never_read = f"{window}: never read" + checked = f"{window}: read only to check the layer's shared scales" + for i in _moe_layers(d): + for e in range(d.num_experts): + if lo <= e < hi: + continue + for proj in PROJECTIONS: + for s in ("weight", "weight_scale"): + key = _expert_key(i, e, f"{proj}.{s}") + if key in weights: + keys[key] = never_read + for s in EXPERT_SCALARS: + keys[_expert_key(i, e, f"{proj}.{s}")] = checked + return keys + + def load_checkpoint(self, model, checkpoint_dir) -> None: + """Fill every declared weight from a checkpoint directory, which is what the engine + loads: `load` over the directory's keys, every source read from its shard file.""" + directory, weight_map = _checkpoint_shards(checkpoint_dir) + self.load(model, weight_map, shards=(directory, weight_map)) + + def load(self, model, weights, shards=None) -> None: + """Fill every declared weight from the checkpoint: multi-source rows, the shell's whole + `lm_head`, and coverage asserted in both directions. + + First every routed expert's per-tensor scalars are read and checked, the other ranks' + included, so a checkpoint that breaks what the forward relies on fails before any + weight moves. When the checkpoint's own shard files can be found, every source is read + from them one file at a time, this rank's expert window and the replicated rest only, + and each file's cached pages are dropped once its sources are copied. Mapping every + shard file on every rank and reading through the mappings instead would fault + the expert windows of a node's four ranks -- 539 GiB, the whole expert stack, at 4 + ranks; half of it on each of the two nodes at 8 -- and the replicated rest into that + node's host memory. + Last, each row's `layout` puts the expert stacks into the MoE runner's layout, so a + second `load` starts again from the stored bytes. + """ + core = model.model + manifest = self.manifest(core) + consumed: set = set() + + def fill(param, ckpt, entry, index, transform) -> None: + dst = param.data if index is None else param.data[index] + if isinstance(ckpt, tuple): + src = transform( + tuple(_materialize(e).to(dst.device, non_blocking=True) for e in entry), core + ) + else: + src = _materialize(entry) + if transform is not None: + src = transform(src.to(dst.device, non_blocking=True), core) + assert tuple(dst.shape) == tuple(src.shape), (ckpt, tuple(dst.shape), tuple(src.shape)) + assert src.dtype == dst.dtype, (ckpt, src.dtype, dst.dtype) + dst.copy_(src, non_blocking=True) + consumed.update(_keys(ckpt)) + + assert set(manifest.keys()) == set(core.w.keys()), ( + "manifest/parameter drift", + sorted(set(manifest.keys()) ^ set(core.w.keys()))[:12], + ) + d = self.dims(core) + plan = [ + (core.w[param_key], ckpt, index, transform) + for param_key, sources in manifest.items() + for ckpt, index, transform in sources + ] + # Under attention DP the shell builds lm_head whole, `[vocab, hidden]` on every rank: + # the checkpoint's whole `output.weight`. + plan.append((model.lm_head.weight, "output.weight", None, None)) + scalar_keys = _expert_scalar_keys(d) + for key in (*(k for row in plan for k in _keys(row[1])), *scalar_keys): + assert key in weights, f"checkpoint key missing: {key}" + if shards is None: + shards = _shard_files(core, weights) + scalars = _read_scalars(scalar_keys, weights, shards) + _check_expert_scalars(scalars, d) + + streamed = [] + for param, ckpt, index, transform in plan: + if all(k in scalars for k in _keys(ckpt)): + entry = tuple(scalars[k] for k in _keys(ckpt)) + entry = entry if isinstance(ckpt, tuple) else entry[0] + fill(param, ckpt, entry, index, transform) + else: + assert isinstance(ckpt, str), f"a key tuple of other than expert scalars: {ckpt}" + streamed.append((param, ckpt, index, transform)) + if shards is None: + for param, ckpt, index, transform in streamed: + fill(param, ckpt, weights[ckpt], index, transform) + else: + directory, weight_map = shards + by_file: dict[str, list] = collections.defaultdict(list) + for row in streamed: + by_file[weight_map[row[1]]].append(row) + on_device = model.lm_head.weight.device.type == "cuda" + for name in sorted(by_file): + with safe_open(str(directory / name), framework="pt", device="cpu") as handle: + for param, ckpt, index, transform in by_file[name]: + fill(param, ckpt, handle.get_slice(ckpt), index, transform) + if on_device: + torch.cuda.synchronize() + _drop_page_cache(directory / name) + # Every source has landed: rearrange each expert in place, one at a time. + for entry in self._entries(d): + if entry.layout is None: + continue + for key, _, _ in self._targets(entry, d): + param = core.w[key].data + for slot in range(param.shape[0]): + param[slot].copy_(entry.layout(param[slot])) + shell = {n for n, _ in model.named_parameters() if not n.startswith("model.w.")} + assert shell == {"lm_head.weight"}, f"unfed shell parameters: {sorted(shell)}" + + if model.lm_head.weight.device.type == "cuda": + torch.cuda.synchronize() + leftover = set(weights.keys()) - consumed + expected = self.expected_unconsumed(core, weights) + assert leftover == set(expected), ( + "checkpoint coverage: unexpected " + f"{sorted(leftover - set(expected))[:8]}, missed {sorted(set(expected) - leftover)[:8]}" + ) + not_loaded = collections.Counter(expected.values()) + logger.info( + f"modeling_v2 weights: {len(consumed)} checkpoint tensors loaded " + f"({'streamed per shard file' if shards else 'from the given dict'}), " + f"{len(scalars)} expert scalars checked; not loaded: " + + "; ".join(f"{n} x {reason}" for reason, n in sorted(not_loaded.items())) + ) + + def dims(self, core) -> SimpleNamespace: + cfg = core.model_config.pretrained_config + vis = getattr(cfg, "vision_config", None) + mapping = core.model_config.mapping + spec = getattr(core.model_config, "spec_config", None) + draft = getattr(spec, "_draft_model_config", None) or {} + local_experts = cfg.n_routed_experts // mapping.moe_ep_size + return SimpleNamespace( + draft_layers=int(draft.get("n_layers", 0)), + draft_rank=int((draft.get("dspark") or {}).get("markov_rank", 0)), + draft_depth=int(draft.get("depth", 0)), + draft_capture_layers=len(spec.target_layer_ids) if draft else 0, + draft_tp_rank=0, + num_layers=cfg.num_hidden_layers, + hidden=cfg.hidden_size, + heads=cfg.num_attention_heads, + nope=cfg.qk_nope_head_dim, + rope_dim=cfg.qk_rope_head_dim, + qk_dim=cfg.qk_nope_head_dim + cfg.qk_rope_head_dim, + v_dim=cfg.v_head_dim, + kv_lora=cfg.kv_lora_rank, + lat_dim=cfg.kv_lora_rank + cfg.qk_rope_head_dim, + nope_layer=tuple(bool(f) for f in cfg.nope_layer_flags), + kv_source=tuple(int(s) for s in cfg.kv_source_layers), + full_indexer=tuple(t == "full" for t in cfg.index_sharing), + idx_heads=cfg.index_n_heads, + idx_dim=cfg.index_head_dim, + dense_layers=cfg.first_k_dense_replace, + num_experts=cfg.n_routed_experts, + moe_inter=cfg.moe_intermediate_size, + local_experts=local_experts, + expert_offset=local_experts * mapping.moe_ep_rank, + shared_inter=cfg.moe_intermediate_size * cfg.n_shared_experts, + dense_inter=cfg.intermediate_size, + vocab=cfg.vocab_size, + dtype=core.model_config.torch_dtype, + # The vision tower (replicated; absent without a vision encoder). + has_vision=vis is not None, + v_layers=vis.num_hidden_layers if vis is not None else 0, + v_hidden=vis.hidden_size if vis is not None else 0, + v_heads=vis.num_attention_heads if vis is not None else 0, + v_head_dim=vis.hidden_size // vis.num_attention_heads if vis is not None else 0, + v_inter=vis.intermediate_size if vis is not None else 0, + v_channels=vis.num_channels if vis is not None else 0, + v_patch=vis.patch_size if vis is not None else 0, + v_merge=vis.spatial_merge_size if vis is not None else 0, + v_theta=float(vis.rope_theta) if vis is not None else 0.0, + image_token_id=vis.image_token_id if vis is not None else None, + ) + + +MODEL_WEIGHTS = MistralLarge4DepWeights() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tep/__init__.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tep/__init__.py new file mode 100644 index 000000000000..a3338824a5b5 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tep/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Mistral-Large-4-1T-A52B-NVFP4 / sm_107 / tep, the language model only.""" diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tep/modeling.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tep/modeling.py new file mode 100644 index 000000000000..9c7c069217e8 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tep/modeling.py @@ -0,0 +1,2491 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""ModelingV2 target: mistral-large-4-nvfp4 / sm_107 / tep, native text and image inference. + +72 layers, hidden 6144, 64 query heads (16 per rank at tep4, 8 at tep8). Attention is MLA +without a q-LoRA (`wq` straight to 64 x 128), a 512-wide latent plus a 64-wide rope key, and +DeepSeek Sparse Attention: an indexer on the 44 `index_sharing == 'full'` layers selects 2048 +positions per query row and each 'shared' layer reuses the selection of the most recent full +layer. Every fourth layer (`i % 4 == 3`) takes no rope, for MLA and for its indexer; the NoPE +layers 39, 43, ..., 71 own no latent and attend over layer 35's. Queries are scaled by SSMax, +`s * log(pos + 1) + b` per head, on MLA and on the indexer. Layer 0 is a dense SwiGLU MLP, +layers 1-71 MoE (128 experts, softmax top-4 renormalised, one shared expert). The final logits +are soft-capped at `softmax_tanh_gating`. + +The checkpoint is ModelOpt `MIXED_PRECISION`. The routed experts are NVFP4 W4A4: e2m1 weights +with one e4m3 scale per 16 inputs and a global scale per expert, and activations quantized to +NVFP4 under one static global scale per layer and projection. `wq`, `wo`, the indexer's `wq_b` +and the dense and shared MLPs are FP8 e4m3 with 128x128 power-of-two block scales and dynamic +1x128 activation scales. The rest is bf16. The latent KV pool is fp8_ds_mla (656 B per token +and owner layer: 512 e4m3 codes with one power-of-two fp32 scale per 128, and the 64-wide rope +key in bf16), owned by `MistralLarge4CacheManager`; the indexer key pool is fp8 with one scale +per key. + +`tep` shards the attention heads, the dense and shared MLPs and `lm_head` over every rank and +gives each rank `n_routed_experts / moe_ep_size` of the 128 routed experts: four ways and 32 +experts on one node of four, or eight ways and 16 experts over two nodes of four (tep8), every +width read off the mapping. The latent projection, the indexer and the router are replicated. +Every layer closes twice on the all-reduce over every rank fused with the residual add and the +next RMSNorm. + +The absorbed form throughout: `fused_q = [q_nope @ W_UK | rope(q_pe)]` (576 wide) against the +pooled `[c_kv | k_pe]` latent rows, then `@ W_UV`. Context rows attend over the selection (the +engine's causal index on a step it skipped the indexer) with FlashMLA's bf16 sparse prefill +over the requests' latent rows dequantized out of the pool, so they read the FP8-rounded latent +as the reference does; generation rows run sparse fp8 decode over the pool itself, at this +rank's 16 (tep4) or 8 (tep8) heads at every max_seq_len: cute_dsl_ds_mla_sparse_decode, +FlashMLA's sparse decode arithmetic without the head padding. The prefill kernel, and the +sparse decode a context row past the held workspace runs (FlashMLA's), take 64 heads: this +rank's go first in a 64-head query whose other heads are zero; heads do not interact, so the +zero heads change nothing the live ones compute. `_GEN_ATTN_FLASH_MLA` (for A/B verification) +runs the generation rows on FlashMLA's 64-head decode too, the same padding. + +No path is selected by the serving configuration: each step chooses from its own sizes, and +every buffer is sized from the engine's quantities (the attention metadata's max_num_tokens, +max_num_requests, max_seq_len and runtime_features). The context rows take one of two paths, +chosen per step by the step's context KV (every context request's cached and new rows): while +it fits the held workspace, the sparse prefill over those rows dequantized into it; past it, +each context row as a one-row sequence of the sparse fp8 decode over the pool itself, as +generation rows always run. The held workspace, allocated on the first context step, has room +for the most context KV one step can attend, twice over, up to 2**19 rows +(`_ctx_workspace_rows`): a step whose context KV fits twice keeps layer 35's rows beside its +own for the NoPE layers attending over that latent, a longer one gathers them again for each +of those layers. An indexer chunk the engine packs past `max_seq_len` keys is scored request +by request, and the indexer's logits are tiled under a budget from the engine's max_num_tokens +and max_seq_len (`_prefill_logits_budget`), each tile scored with the Triton FP8 MQA scorer +and selected with the self-sampling GVR top-k (`_select_ctx`). + +Under chunked prefill the engine splits a step's context queries across all four or eight ranks +from `q_split_threshold` rows on: each rank selects for its share, and the +shares are all-gathered into every rank's selection buffer before a layer reads it. +""" + +from __future__ import annotations + +import itertools +import math +import os +import time +from dataclasses import replace +from types import SimpleNamespace +from typing import NamedTuple, Optional, cast + +import torch +from torch import nn +from transformers import PretrainedConfig + +from tensorrt_llm._torch.attention.backends.interface import AttentionMetadata +from tensorrt_llm._torch.attention.backends.sparse.params import ( + get_indexer_mqa_logits_workspace_bytes, +) +from tensorrt_llm._torch.attention.backends.trtllm import TrtllmAttentionMetadata +from tensorrt_llm._torch.model_config import ModelConfig +from tensorrt_llm._torch.modeling_v2._router_index import step_contract_enabled +from tensorrt_llm._torch.modeling_v2.catalog.activation.flashinfer_silu_and_mul import ( + flashinfer_silu_and_mul, +) +from tensorrt_llm._torch.modeling_v2.catalog.attention.thop_attention import thop_attention +from tensorrt_llm._torch.modeling_v2.catalog.comm.allgather import allgather +from tensorrt_llm._torch.modeling_v2.catalog.gemm.bmm_out import bmm_out +from tensorrt_llm._torch.modeling_v2.catalog.gemm.cublas_mm import cublas_mm +from tensorrt_llm._torch.modeling_v2.catalog.moe.fp4_block_scale_moe_runner import ( + fp4_block_scale_moe_runner, +) +from tensorrt_llm._torch.modeling_v2.catalog.norm.flashinfer_fused_add_rmsnorm import ( + flashinfer_fused_add_rmsnorm, +) +from tensorrt_llm._torch.modeling_v2.catalog.norm.flashinfer_rmsnorm import flashinfer_rmsnorm +from tensorrt_llm._torch.modeling_v2.catalog.quantization.fp4_quantize import fp4_quantize +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.activation.gelu_erf import gelu_erf +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.activation.tanh_softcap import ( + tanh_softcap, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.convert_req_index_to_global import ( # noqa: E501 + convert_req_index_to_global, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_ds_mla_sparse_decode import ( # noqa: E501 + build as build_ds_mla_decode, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_ds_mla_sparse_decode import ( # noqa: E501 + cute_dsl_ds_mla_sparse_decode, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_fp8_paged_mqa_logits import ( # noqa: E501 + cute_dsl_fp8_paged_mqa_logits, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_gvr_topk_prefill import ( # noqa: E501 + cute_dsl_gvr_topk_prefill, + gvr_warmup, + gvr_workspace, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_indexer_k_cache_update import ( # noqa: E501 + build as build_ix_k_cache, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_indexer_k_cache_update import ( # noqa: E501 + cute_dsl_indexer_k_cache_update, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_indexer_q_side_fused import ( # noqa: E501 + build as build_ix_q_side, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_indexer_q_side_fused import ( # noqa: E501 + cute_dsl_indexer_q_side_fused, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_radix_topk_decode import ( # noqa: E501 + build as build_ix_topk_decode, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_radix_topk_decode import ( # noqa: E501 + cute_dsl_radix_topk_decode, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ds_mla_dequant_gather import ( + ds_mla_dequant_gather, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ds_mla_rope_append import ( + ds_mla_rope_append, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.flash_mla_sparse_decode import ( + flash_mla_sparse_decode, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.flash_mla_sparse_prefill import ( + flash_mla_sparse_prefill, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.indexer_k_cache_gather_op import ( # noqa: E501 + indexer_k_cache_gather_op, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.indexer_topk_decode import ( + indexer_topk_decode, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ssmax_indexer_q_side_fused import ( # noqa: E501 + ssmax_indexer_q_side_fused, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ssmax_scaling import ( + ssmax_scaling, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ssmax_scaling_fused_q import ( + ssmax_scaling_fused_q, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.triton_fp8_mqa_logits import ( + triton_fp8_mqa_logits, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.comm.cute_dsl_allreduce_add_rmsnorm import ( # noqa: E501 + build_workspace as build_lamport_workspace, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.comm.cute_dsl_allreduce_add_rmsnorm import ( # noqa: E501 + cute_dsl_allreduce_add_rmsnorm, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.gemm.cute_dsl_fp8_block_scale_linear import ( # noqa: E501 + cute_dsl_fp8_block_scale_linear, + prepare_weight_scale, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.moe.renorm_moe_routing_op import ( + renorm_moe_routing_op, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.norm.layer_norm import layer_norm +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.vision.pixtral_rope_2d import ( + pixtral_rope_2d, +) +from tensorrt_llm._torch.models.modeling_multimodal_utils import ( + find_input_mm_embeds, + get_multimodal_embeddings, +) +from tensorrt_llm._torch.models.modeling_utils import DecoderModel, DecoderModelForCausalLM +from tensorrt_llm._torch.pyexecutor.guided_decoder import CapturableGuidedDecoder +from tensorrt_llm._torch.speculative import get_spec_worker +from tensorrt_llm._utils import mpi_disabled +from tensorrt_llm.flash_mla import get_mla_metadata +from tensorrt_llm.inputs.multimodal import MultimodalParams +from tensorrt_llm.logger import logger +from tensorrt_llm.models.modeling_utils import QuantConfig + +from . import weights as _weights + +#: eps 1e-5, the reference's value; params.json's `vision_encoder` states none. +_VISION_EPS = 1e-5 + +#: The thop attention arguments the vision tower's call holds fixed: the no-KV-cache +#: mode (an encoder: the three cache-addressing arguments None), one context sequence +#: per image, the fused `[q | k | v]` rows already rotated (no rope in the op), +#: bidirectional within each sequence (mask_type 0), softmax scale head_dim^-0.5. +_VISION_ATTN = dict( + k=None, + v=None, + output_sf=None, + max_context_q_len_override=None, + kv_cache_block_offsets=None, + host_kv_cache_pool_pointers=None, + host_kv_cache_pool_mapping=None, + cache_indirection=None, + kv_scale_orig_quant=None, + kv_scale_quant_orig=None, + out_scale=None, + rotary_inv_freq=None, + rotary_cos_sin=None, + latent_cache=None, + q_pe=None, + block_ids_per_seq=None, + attention_sinks=None, + is_fused_qkv=True, + update_kv_cache=True, + predicted_tokens_per_seq=1, + local_layer_idx=0, + tokens_per_block=None, + beam_width=1, + mask_type=0, + quant_mode=0, + q_scaling=1.0, + position_embedding_type=0, + rope_dim=0, + rope_base=10000.0, + rope_scale_type=0, + rope_scale=1.0, + rope_short_m_scale=1.0, + rope_long_m_scale=1.0, + rope_max_positions=1024, + rope_original_max_positions=1024, + use_paged_context_fmha=False, + attention_input_type=0, + is_mla_enable=False, + chunked_prefill_buffer_batch_size=1, + q_lora_rank=None, + kv_lora_rank=None, + qk_nope_head_dim=None, + qk_rope_head_dim=None, + v_head_dim=None, + rope_append=None, + mrope_rotary_cos_sin=None, + mrope_position_deltas=None, + helix_position_offsets=None, + helix_is_inactive_rank=None, + attention_chunk_size=None, + softmax_stats_tensor=None, + is_spec_decoding_enabled=False, + use_spec_decoding=False, + is_spec_dec_tree=False, + spec_decoding_generation_lengths=None, + spec_decoding_position_offsets=None, + spec_decoding_packed_mask=None, + spec_decoding_bl_tree_mask_offset=None, + spec_decoding_bl_tree_mask=None, + spec_bl_tree_first_sparse_mask_offset_kv=None, + sparse_kv_indices=None, + sparse_kv_offsets=None, + sparse_attn_indices=None, + sparse_attn_offsets=None, + sparse_attn_indices_block_size=0, +) + +# The latent pool layout this target addresses (`MistralLarge4CacheManager.latent_layout`). +_LATENT_LAYOUT = "fp8_ds_mla" +# The query heads FlashMLA's sm100 sparse kernels run; this rank's heads go first. +_FLASH_MLA_HEADS = 64 +# Generation rows' sparse MLA. False: cute_dsl_ds_mla_sparse_decode at this rank's heads, the +# shipped path. True: flash_mla_sparse_decode over the 64-head padded query, the kernel this +# target ran before (bit-identical where the two split a row alike; see the entry's note) -- +# a switch for A/B verification, not a configuration. It sets the core's +# `gen_attn_flash_mla`, which every generation-row site reads; the native entry's cells cover +# the 32K and the 1M configs (key lengths to 1048576, the 1M pools' slots past 2**31 bytes). +_GEN_ATTN_FLASH_MLA = False +# The pre-routed MoE call: the runner's routing method is inert once ids and weights are +# given; 1 (renormalize) is the value its cells drive. +_MOE_ROUTING_METHOD = 1 +# The routed experts' SwiGLU, the runner's `act_type` 0. +_MOE_ACT_SWIGLU = 0 +# NVFP4: one e4m3 scale per 16 activations, laid out row-major (linear), the order the MoE +# runner reads; the 128x4-swizzled order is another buffer of the same length at 128-row +# multiples, and silently wrong there. +_NVFP4_SF_VEC = 16 +_NVFP4_SF_LINEAR = False +# triton_fp8_mqa_logits allocates a `[q_rows, kv_rows]` fp32 score matrix per call; the context +# queries are tiled to keep it under the engine's budget (`_prefill_logits_budget`). Context +# steps are not captured. +# capacity: fixed: the budget's ceiling, one 1 GiB fp32 tile (256 rows at 1048576 keys). +# Mainline's default reachable tile (2**31 logits) would take 8 GiB from the KV pool. +_PREFILL_LOGITS_CEILING = 1 << 28 +# The context path's crossover, chosen once per step on host ints, and the held context +# workspace's ceiling (`_ctx_workspace_rows`). A step whose context KV (every context request's +# cached and new rows) fits the held workspace has it gathered, dequantized, into it (bf16 +# [rows, 576]) for FlashMLA's sparse prefill; a longer one attends its context rows straight off +# the pool with the sparse decode kernel, whose cost does not grow with the prefix as the +# gather's does. +# capacity: fixed: 2**19 rows, 576 MiB, the gather-vs-decode crossover. +_CTX_WORKSPACE_ROWS = 1 << 19 +# The most context rows one sparse-decode call takes on that path. +# capacity: fixed: FlashMLA's decode scheduler, which exits the process, raising nothing, past +# 11622 one-row sequences on sm_103; not an engine limit. +_CTX_DECODE_ROWS = 8192 +# The decode indexer's top-k: True runs cute_dsl_radix_topk_decode, one launch that selects the +# trtllm indexer_topk_decode op's set (up to exact ties), False that op itself. The CuTe DSL +# selector serves scorer rows from 4096 columns (where the op splits each row) to 65536 (one +# CTA scans a whole row), the widths its cells cover; the scorer's column count is the engine's +# max_seq_len, so an engine outside that range keeps the op (`_derive_first_step`). +_TOPK_DECODE_CUTE_DSL = True +_TOPK_DECODE_COLUMNS = (4096, 65536) +# The attention block's glue as separate ops instead of the fused entries, for old-versus-new +# verification (both compute the same bits; the fused entries' tests hold each pair equal): +# ssmax_scaling plus the rope tail's copy or zero fill for ssmax_scaling_fused_q; +# ssmax_scaling, the head weights' fp32 copy and cute_dsl_indexer_q_side_fused for +# ssmax_indexer_q_side_fused; a fresh RMSNorm and torch.cat for the in-place latent; the copy +# out of FlashMLA's 64-head output for the in-place read. Set +# TRTLLM_MODELING_V2_UNFUSED_GLUE=1 in every rank's environment at launch; off by default. +# It sets the core's `unfused_glue`, which every site reads. +_UNFUSED_GLUE = os.environ.get("TRTLLM_MODELING_V2_UNFUSED_GLUE", "0") == "1" + + +def _prefill_tiles( + token_start: int, token_end: int, n_k: int, budget: int +) -> list[tuple[int, int]]: + """The query-row tiles `[lo, hi)` of one indexer prefill chunk over `n_k` keys under a + logits `budget`: `floor(budget / n_k)` rows each (at least one), the last one short.""" + tile = max(1, budget // n_k) + return [(lo, min(lo + tile, token_end)) for lo in range(token_start, token_end, tile)] + + +def _prefill_logits_budget(max_num_tokens: int, max_seq_len: int) -> int: + """The most fp32 logits one indexer prefill tile holds, from the engine's max_num_tokens and + max_seq_len: mainline's reachable MQA-logits tile (`get_indexer_mqa_logits_workspace_bytes`, + 4 bytes a logit; TLLM_INDEXER_MQA_LOGITS_ELEM_BUDGET lowers it), within + `_PREFILL_LOGITS_CEILING`, and within max_seq_len / 64 query rows at max_seq_len keys. + + The last bound grows the tile with the engine's longest request, which it scores in at + least 64 tiles: 2**24 logits (512 rows of 32768 keys) at max_seq_len 32768, the ceiling + (256 rows of 1048576 keys) from max_seq_len 131072 on. Those are the budgets the 32768 and + 1048576 serving configs were certified at, so their tiles, and the fp8_mqa_logits and + prefill top-k cells that cover them, stay as they were.""" + reachable = get_indexer_mqa_logits_workspace_bytes(max_num_tokens, max_seq_len) // 4 + return min(reachable, _PREFILL_LOGITS_CEILING, max_seq_len * (max_seq_len >> 6)) + + +def _most_ctx_kv(md) -> int: + """The most context KV (cached and new rows over a step's context requests) one step can + attend, from the attention metadata alone. `runtime_features` is built once by the executor + for the service (chunked prefill from enable_chunked_prefill, cache reuse from the + KvCacheConfig's enable_block_reuse). With neither, every context row is new: + max_num_tokens. Otherwise mainline's never-stall worst case for a workspace chunked prefill + does not bound (`get_mla_context_workspace_kv_len_cap` without an + fp8_context_mla_kv_len_cap override; the gather reads every attended row, cached ones + included): min(max_num_requests, max_num_tokens) * max_seq_len. The target does not read + fp8_context_mla_kv_len_cap: the KvCacheConfig is not reachable from the model, and for these + targets the engine installs no admission cap that would bound a step to it.""" + features = md.runtime_features + if not (features.chunked_prefill or features.cache_reuse): + return md.max_num_tokens + return min(md.max_num_requests, md.max_num_tokens) * md.max_seq_len + + +def _ctx_workspace_rows(md) -> int: + """The held context workspace's rows: the most context KV one step can attend + (`_most_ctx_kv`) twice over, a step's rows and a shared owner's (layer 35's) kept beside + them, up to `_CTX_WORKSPACE_ROWS`. At max_num_tokens 32768 with neither chunked prefill nor + block reuse that is 65536 rows, the two `[32768, 576]` gathers a step there makes; with either, + at 16 requests of max_seq_len 32768 or more, 2**19.""" + return min(2 * _most_ctx_kv(md), _CTX_WORKSPACE_ROWS) + + +def _step_common(md: TrtllmAttentionMetadata, indexer_layers: list[int]) -> dict: + """The batch state both targets read: KV lengths and pool addressing for the MLA ops, the + indexer's key pools and slot mappings, the top-k conversion tables and the decode scorer's + schedule. The indexer pool views are taken per step: the engine replaces its KV cache + manager during boot, and a view of the old one writes into a dead allocation.""" + mgr = md.kv_cache_manager + latent_pool = mgr.get_latent_pool() + return dict( + num_seqs=md.num_seqs, + num_tokens=md.num_tokens, + kv_lens_cuda=md.kv_lens_cuda_runtime, + kv_lens_host=md.kv_lens_runtime, + prompt_lens_host=md.prompt_lens_cpu_runtime, + kv_cache_block_offsets=md.kv_cache_block_offsets, + host_kv_cache_pool_pointers=md.host_kv_cache_pool_pointers, + host_kv_cache_pool_mapping=md.host_kv_cache_pool_mapping, + tokens_per_block=md.tokens_per_block, + max_seq_len=md.max_seq_len, + # [blocks, owners, tokens_per_block, 656] bytes, and the same bytes as FlashMLA's + # pages: page `block * owners + row`, slot `page * tokens_per_block + offset`. + latent_pool=latent_pool, + latent_kv=latent_pool.view(-1, latent_pool.shape[2], 1, latent_pool.shape[3]), + ix_pools={i: mgr.get_indexer_k_cache_buffers(i) for i in indexer_layers}, + block_table=md.block_table, + req_idx_per_token=md.req_idx_per_token, + slot_mapping_fp8=md.slot_mapping_fp8, + slot_mapping_scale=md.slot_mapping_scale, + skip_gen=bool(md.skip_indexer_for_gen_reqs), + # Allocated only while the engine's short-sequence indexer skip is on; the two skip + # flags are False whenever it is off. + topk_indices_buffer=md.topk_indices_buffer if md.enable_indexer_skip else None, + indexer_k_cache_block_offsets=md.indexer_k_cache_block_offsets, + scheduler_metadata_buffer=md.scheduler_metadata_buffer, + radix_aux_indices=md.radix_aux_indices, + radix_aux_logits=md.radix_aux_logits, + ) + + +def _ctx_kv_offsets(md: TrtllmAttentionMetadata, num_contexts: int) -> dict: + """The context rope/append op's per-request offsets: int64 prefix sums of the cached and + of the total (cached + new) lengths over the context requests, and the longest new run. + + The engine keeps them only under `enable_context_mla_with_cached_kv` (KV reuse or chunked + prefill). Without either, no context request has a cached prefix, so the cached sums are + zero and the totals are the new lengths: the int32 prefix sum of `seq_lens_cuda` the + metadata always builds, widened to the op's int64. + """ + if md.enable_context_mla_with_cached_kv: + return dict( + cu_ctx_cached=md.ctx_cached_token_indptr[: num_contexts + 1], + cu_ctx_kv=md.ctx_kv_indptr[: num_contexts + 1], + max_ctx_new=int(md.max_ctx_seq_len), + ctx_cached_tokens=int(md.num_ctx_cached_tokens), + ) + cu = md.mla_prepare_ctx_cu_seqlens() + return dict( + cu_ctx_cached=torch.zeros(num_contexts + 1, dtype=torch.int64, device=cu.device), + cu_ctx_kv=cu.to(torch.int64), + max_ctx_new=int(md.seq_lens[:num_contexts].max()), + ctx_cached_tokens=0, + ) + + +def _ctx_host_offsets(md: TrtllmAttentionMetadata, num_contexts: int) -> dict: + """The host lists over the context requests: the prefix sums of the new and of the total + (cached + new) lengths, which place each context request's rows in the step and its keys in + the indexer's full-KV gather mappings (`_indexer_blocks`). + + The engine's host copies under `enable_context_mla_with_cached_kv`; without it no request + has a cached prefix, so both are the prefix sum of the host `seq_lens`. Every source is a + host tensor the metadata fills on the host, so taking them each context step syncs nothing. + """ + if md.enable_context_mla_with_cached_kv: + return dict( + host_cu_ctx_new=md.host_ctx_uncached_token_indptr[: num_contexts + 1].tolist(), + host_cu_ctx_kv=md.host_ctx_kv_indptr[: num_contexts + 1].tolist(), + ) + host_cu = [0, *itertools.accumulate(md.seq_lens[:num_contexts].tolist())] + return dict(host_cu_ctx_new=host_cu, host_cu_ctx_kv=host_cu) + + +def _indexer_blocks(chunks, step: dict) -> list: + """The context indexer's scoring blocks for one step, in the engine's chunk order: rows + `[token_start, token_end)` of the context rows scored against keys + `[k_token_start, k_token_end)` of the full-KV gather mappings, each row's window + (`cu_seqlen_ks` / `cu_seqlen_ke`) counted from the block's first key. + + An engine prefill chunk is one block, as planned, while it holds at most `max_seq_len` keys, + the most one request holds. The engine packs consecutive context requests into one chunk + while (summed new tokens) x (summed KV) stays within its packed-logits budget (2^28), so + short last chunks of long requests pack past max_seq_len: a 100-token tail of a + 1048559-token request beside a 150-token prompt is one chunk of 1048709 keys, wider than + the key scorer's and the prefill top-k's cells. Such a chunk is cut at its request + boundaries (`_ctx_host_offsets`), each request's rows against its own keys with their + windows re-based to its first key: a row's window lies inside its own request's keys, so + each request selects as it would alone and no block holds more than max_seq_len keys. + """ + tc = step["num_ctx_tokens"] + if chunks is None: + # One chunk over every context row: the engine plans chunks only past its chunk + # size, or under KV reuse or chunked prefill. + chunks = [ + SimpleNamespace( + token_start=0, + token_end=tc, + k_token_start=0, + k_token_end=step["num_ctx_kv_tokens"], + cu_seqlen_ks=step["cu_seqlen_ks"][:tc], + cu_seqlen_ke=step["cu_seqlen_ke"][:tc], + ) + ] + max_keys = step["max_seq_len"] + cu_new, cu_kv = step["host_cu_ctx_new"], step["host_cu_ctx_kv"] + blocks = [] + for chunk in chunks: + if chunk.k_token_end - chunk.k_token_start <= max_keys: + blocks.append(chunk) + continue + for r in range(len(cu_new) - 1): + lo, hi = max(chunk.token_start, cu_new[r]), min(chunk.token_end, cu_new[r + 1]) + if lo >= hi: + continue + k_lo, k_hi = max(chunk.k_token_start, cu_kv[r]), min(chunk.k_token_end, cu_kv[r + 1]) + assert k_hi - k_lo <= max_keys, ( + f"context request {r} holds keys [{k_lo}, {k_hi}) of an indexer chunk, past " + f"max_seq_len {max_keys}" + ) + shift = k_lo - chunk.k_token_start + rows = slice(lo - chunk.token_start, hi - chunk.token_start) + blocks.append( + SimpleNamespace( + token_start=lo, + token_end=hi, + k_token_start=k_lo, + k_token_end=k_hi, + cu_seqlen_ks=torch.sub(chunk.cu_seqlen_ks[rows], shift), + cu_seqlen_ke=torch.sub(chunk.cu_seqlen_ke[rows], shift), + ) + ) + return blocks + + +def _yarn_lerope_inv_freq(cfg, alpha: torch.Tensor) -> torch.Tensor: + """The rotary inverse frequencies, fp32, as the reference implementation builds + them: YaRN's blend of the extrapolated and the interpolated frequency, both scaled by + LeRoPE's `exp(alpha)`, over a ramp taken on the frequencies without LeRoPE.""" + rope = cfg.rope_parameters + dim = cfg.qk_rope_head_dim + theta = float(rope["rope_theta"]) + factor = float(rope["factor"]) + original = float(rope["original_max_position_embeddings"]) + pos_freqs = theta ** (torch.arange(0, dim, 2, dtype=torch.float32) / dim) + lerope = torch.exp(alpha.detach().to(device="cpu", dtype=torch.float32)) + extrapolated = 1.0 / pos_freqs * lerope + interpolated = 1.0 / (factor * pos_freqs) * lerope + + def correction_dim(rotations: float) -> float: + return dim * math.log(original / (rotations * 2 * math.pi)) / (2 * math.log(theta)) + + low = max(math.floor(correction_dim(float(rope.get("beta_fast", 32)))), 0) + high = min(math.ceil(correction_dim(float(rope.get("beta_slow", 1)))), dim - 1) + if low == high: + high += 0.001 + ramp = ((torch.arange(dim // 2, dtype=torch.float32) - low) / (high - low)).clamp(0, 1) + extrapolation = 1 - ramp + return interpolated * (1 - extrapolation) + extrapolated * extrapolation + + +def _rope_tables(cfg, alpha: torch.Tensor, positions: int, dtype, device) -> SimpleNamespace: + """The tables the rope consumers read, built from one fp32 angle grid and rounded to the + activation dtype first, as the reference's cos/sin cache is held. + + `indexer` is `[rows, 64]`, `[cos(32) | sin(32)]`: the layout the indexer q side and + `ds_mla_rope_append` (the MLA query tail and the latent's rope key) read; the indexer key + side reads the same storage as `[rows, 2, 32]`. `identity` (cos 1, sin 0) is the table of a + NoPE indexer layer, whose kernels have no no-rope mode. Every row depends only on its own + index, so a longer table is the same table with more rows. + """ + inv = _yarn_lerope_inv_freq(cfg, alpha) + t = torch.arange(positions, dtype=torch.float32) + angle = torch.einsum("i,j -> ij", t, inv) + cos = angle.cos().to(dtype).float() + sin = angle.sin().to(dtype).float() + identity = torch.cat([torch.ones_like(cos), torch.zeros_like(sin)], dim=1) + return SimpleNamespace( + rows=positions, + indexer=torch.cat([cos, sin], dim=1).to(device).contiguous(), + identity=identity.to(device).contiguous(), + ) + + +def _moe_scalars(isc1, isc2, ws2_13, ws2_2) -> SimpleNamespace: + """The NVFP4 MoE's call scalars from the checkpoint's per-tensor ones, fp32. + + ModelOpt stores `input_scale` and `weight_scale_2` as the reciprocals of the global scales a + quantizer divides by (`amax / (448 * 6)`), one `isc1` / `isc2` per layer and one + `ws2_13[e]` / `ws2_2[e]` per local expert. So the activation quantizer's global scale is + `1 / isc1`; FC1's accumulator dequantizes by `isc1 * ws2_13[e]` (the gate half, + `output1_scale_gate_scalar`) and, because the FC1 epilogue requantizes the SwiGLU output to + NVFP4 under `1 / isc2`, the linear half by `isc1 * ws2_13[e] / isc2` + (`output1_scale_scalar`); FC2's accumulator dequantizes by `isc2 * ws2_2[e]` + (`output2_scale_scalar`). The reference's TRT-LLM-Gen path folds the same three + (`g1_alphas`, `g1_scale_c`, `g2_alphas`).""" + gate1 = (isc1 * ws2_13).contiguous() + return SimpleNamespace( + gscale=(1.0 / isc1).contiguous(), + out1=(gate1 / isc2).contiguous(), + out1_gate=gate1, + out2=(isc2 * ws2_2).contiguous(), + ) + + +def _layer_views(core: "MistralLarge4TepModelingV2") -> list[SimpleNamespace]: + """Every layer's operands, as the views the entries read: GEMM operands transposed (`.t()`, + the column-major `[K, N]` cuBLAS operand), FP8 scales converted once to + the Rubin R128c4 K32 layout, `wkv_b` split into the absorption pair, and the NVFP4 MoE's call + scalars (`_moe_scalars`). `next_norm` is the next layer's input norm, the final norm after + the last layer. A sharer's absorption pair is its own `wkv_b`; its latent is its source's.""" + w = core.w + heads, v_dim, kv_lora = core.heads, core.v_dim, core.kv_lora + layers = [] + for i in range(core.num_layers): + p = f"l{i}_" + nope = core.qk_dim if core.nope_layer[i] else core.nope + kvb = w[p + "wkv_b"] + hn = heads * nope + owner = core.kv_source[i] == i + ix = None + if core.full_indexer[i]: + ix = SimpleNamespace( + q_norm_w=w[p + "ix_q_norm_w"], + q_norm_b=w[p + "ix_q_norm_b"], + wq_b=w[p + "ix_wq_b"], + wq_b_s=prepare_weight_scale(w[p + "ix_wq_b_s"].t(), w[p + "ix_wq_b"].shape[1]), + wk_wp=w[p + "ix_wk_wp"].t(), + k_norm_w=w[p + "ix_k_norm_w"], + k_norm_b=w[p + "ix_k_norm_b"], + ssmax_s=w[p + "ix_ssmax_s"], + ssmax_b=w[p + "ix_ssmax_b"], + rope=not core.indexer_nope[i], + ) + moe = None + if i >= core.dense_layers: + moe = SimpleNamespace( + router=w[p + "router"].t(), + fc1_w=w[p + "fc1_w"], + fc1_s=w[p + "fc1_s"], + fc2_w=w[p + "fc2_w"], + fc2_s=w[p + "fc2_s"], + **vars(_moe_scalars(w[p + "isc1"], w[p + "isc2"], w[p + "ws2_13"], w[p + "ws2_2"])), + ) + layers.append( + SimpleNamespace( + index=i, + rope=not core.nope_layer[i], + nope=nope, + owner=owner, + kv_source=core.kv_source[i], + attn_norm=w[p + "attn_norm"], + wq=w[p + "wq"], + wq_s=prepare_weight_scale(w[p + "wq_s"].t(), w[p + "wq"].shape[1]), + ssmax_s=w[p + "ssmax_s"], + ssmax_b=w[p + "ssmax_b"], + wkv_a=w[p + "wkv_a"].t() if owner else None, + kv_norm=w[p + "kv_a_norm"] if owner else None, + k_b=kvb[:hn].reshape(heads, nope, kv_lora), + v_b_t=torch.transpose(kvb[hn:].reshape(heads, v_dim, kv_lora), 1, 2), + wo=w[p + "wo"], + wo_s=prepare_weight_scale(w[p + "wo_s"].t(), w[p + "wo"].shape[1]), + ffn_norm=w[p + "ffn_norm"], + mlp_gu=w[p + "mlp_gu"], + mlp_gu_s=prepare_weight_scale(w[p + "mlp_gu_s"].t(), w[p + "mlp_gu"].shape[1]), + mlp_dn=w[p + "mlp_dn"], + mlp_dn_s=prepare_weight_scale(w[p + "mlp_dn_s"].t(), w[p + "mlp_dn"].shape[1]), + indexer=ix, + moe=moe, + next_norm=( + w[f"l{i + 1}_attn_norm"] if i + 1 < core.num_layers else w["final_norm"] + ), + ) + ) + return layers + + +class MistralLarge4TepModelingV2(DecoderModel): + def __init__(self, model_config: ModelConfig): + super().__init__(model_config) + cfg = model_config.pretrained_config + mapping = model_config.mapping + self.tp_rank = mapping.tp_rank + self.tp_size = mapping.tp_size + # The context selection's gather: the TP ranks in the order the engine sizes their + # query shares, on the MPI session's communicator (`post_load_weights` swaps in the TP + # process group when MPI is disabled). + self.tp_group = list(mapping.tp_group) + self.tp_pg = None + # The attention block's glue: the fused entries (ssmax_scaling_fused_q, + # ssmax_indexer_q_side_fused, the latent RMSNorm in place, FlashMLA's decode output + # read in place) at every max_seq_len, whose cells cover the 32K and 1M configs' + # positions (to 1048575), rope tables and step shapes; the separate ops they replace + # under `_UNFUSED_GLUE`, at every site. + self.unfused_glue = _UNFUSED_GLUE + # The generation rows' sparse MLA: cute_dsl_ds_mla_sparse_decode at the rank's heads at + # every max_seq_len, whose cells cover the 32K and 1M configs' pools and key lengths; + # flash_mla_sparse_decode over the 64-head padded query under `_GEN_ATTN_FLASH_MLA`. + self.gen_attn_flash_mla = _GEN_ATTN_FLASH_MLA + + self.num_layers = cfg.num_hidden_layers + self.hidden = cfg.hidden_size + self.eps = cfg.rms_norm_eps + self.heads = cfg.num_attention_heads // mapping.tp_size + self.nope = cfg.qk_nope_head_dim + self.rope_dim = cfg.qk_rope_head_dim + # A NoPE layer spends the rope dims on nope: the query head stays this wide. + self.qk_dim = self.nope + self.rope_dim + self.v_dim = cfg.v_head_dim + self.kv_lora = cfg.kv_lora_rank + self.lat_dim = self.kv_lora + self.rope_dim + # Every layer's softmax scale, NoPE layers included: qk_head_dim**-0.5, no YaRN + # temperature (its magnitude is 1). + self.softmax_scale = self.qk_dim**-0.5 + + self.nope_layer = [bool(f) for f in cfg.nope_layer_flags] + self.indexer_nope = [bool(f) for f in cfg.indexer_nope_layer_flags] + self.kv_source = [int(s) for s in cfg.kv_source_layers] + # An owner some later layer attends over (layer 35): a step whose context KV fits the + # held workspace twice keeps its gather there for the sharers; a longer one has each + # sharer gather its source's rows again (`_ctx_workspace`). + self.shared_source = [ + any(s == i and j != i for j, s in enumerate(self.kv_source)) + for i in range(self.num_layers) + ] + self.full_indexer = [role == "full" for role in cfg.index_sharing] + # A 'shared' layer attends with the most recent full layer's selection: each target + # keeps one selection buffer per step, which every full layer overwrites in order. + self.indexer_layers = [i for i in range(self.num_layers) if self.full_indexer[i]] + + self.idx_heads = cfg.index_n_heads + self.idx_dim = cfg.index_head_dim + self.idx_topk = cfg.index_topk + self.idx_eps = cfg.indexer_norm_eps + # The indexer's softmax scale and head average: its scorers take none, so the q side + # folds both into the head weights together with the q quantization scale. + self.idx_weight_scale = self.idx_dim**-0.5 * self.idx_heads**-0.5 + + self.dense_layers = cfg.first_k_dense_replace + self.num_experts = cfg.n_routed_experts + self.topk = cfg.num_experts_per_tok + self.moe_inter = cfg.moe_intermediate_size + self.local_experts = self.num_experts // mapping.moe_ep_size + self.expert_offset = self.local_experts * mapping.moe_ep_rank + # capacity: max_num_tokens -- the collective's peer buffers and the MoE autotuner's + # largest bucket. + self.max_num_tokens = model_config.max_num_tokens + # The rope tables' first size; grown on the first step to the engine's max_seq_len. + self.rope_rows = model_config.max_seq_len or cfg.max_position_embeddings + + self.w = _weights.MODEL_WEIGHTS.declare(_weights.MODEL_WEIGHTS.dims(self)) + + self._rope: SimpleNamespace | None = None + self._derived_for = None + self._ix_k = None + self._ix_k_checked = False + self._pool_row: list[int] = [] + self._stride_factor = 0 + self._max_ctx_len = 0 + # The engine's sizes, fixed on the first step (`_derive_first_step`). + self.prefill_logits_budget = 0 + self.ctx_workspace_rows = 0 + # The decode top-k's handle, built on the first step at a scorer width it serves. + self._ix_topk = None + self._prefill = self._decode = None + + def post_load_weights(self) -> None: + """Derive what the weights fix, build the kernels and the collective, then the targets. + + Named for the shell hook that drives it. Everything compiles here or on the first + (eager) step: a first compile inside a CUDA-graph capture breaks the graph. + """ + dev = torch.cuda.current_device() + self.build_kernels(dev) + # Collective: every rank builds it with the same arguments. The peer buffers hold + # the whole token budget, so one workspace serves every closing collective. + self._lamport = build_lamport_workspace( + self.hidden, self.eps, self.max_num_tokens, self.tp_rank, self.tp_size + ) + # The gather's communicator flavor, as the engine's own collectives pick it. + if mpi_disabled(): + self.tp_pg = self.model_config.mapping.tp_group_pg.boxed() + self._prefill = PrefillTarget(self) + self._decode = DecodeTarget(self) + + def build_kernels(self, dev: int) -> None: + """The rank-local derivations: the rope tables, the indexer q side for the unfused glue, + and the generation rows' sparse decode at this rank's heads.""" + self._rope = _rope_tables( + self.model_config.pretrained_config, + self.w["lerope_alpha"], + self.rope_rows, + self.w["final_norm"].dtype, + self.w["final_norm"].device, + ) + if self.unfused_glue: + # The checkpoint's rope is interleaved (GPT-J) for the indexer too. + self._ix_q = build_ix_q_side(dev, is_neox=False) + self._mla_decode = ( + None if self.gen_attn_flash_mla else build_ds_mla_decode(dev, num_heads=self.heads) + ) + + def _derive_first_step(self, md) -> None: + """What depends on values the engine resolves, fixed on the first step it hands over + (eager, ahead of any capture) and again for a replaced KV cache manager.""" + if md.kv_cache_manager is self._derived_for: + return + dev = torch.cuda.current_device() + mgr = md.kv_cache_manager + # The pool layout is the KV cache manager's, which routing cannot see; every latent + # op here reads and writes fp8_ds_mla rows addressed by owner row. + if getattr(mgr, "latent_layout", None) != _LATENT_LAYOUT: + raise RuntimeError( + f"this target attends over an {_LATENT_LAYOUT} latent pool with owner-only " + f"pages (MistralLarge4CacheManager); the engine built {type(mgr).__name__} " + f"with layout {getattr(mgr, 'latent_layout', None)!r}. Use KV cache manager V2 " + "and kv_cache_config.dtype auto, fp8 or fp8_ds_mla." + ) + if list(mgr.kv_source_layers) != self.kv_source: + raise RuntimeError("the KV cache manager's kv_source_layers differ from the config's") + # A table shorter than max_seq_len is read past its end with no check. + if md.max_seq_len > self._rope.rows: + self.rope_rows = md.max_seq_len + self._rope = _rope_tables( + self.model_config.pretrained_config, + self.w["lerope_alpha"], + self.rope_rows, + self.w["final_norm"].dtype, + self.w["final_norm"].device, + ) + # Layer i reads owner row `_pool_row[i]` (its source's); a block holds every owner's + # page, so a global row is `block * owners * tpb + row * tpb + offset`. + self._pool_row = [mgr.latent_pool_row(i) for i in range(self.num_layers)] + self._stride_factor = len(mgr.kv_owner_layers) * md.tokens_per_block + # The decode scorer's column count: frozen into a capture, so an engine constant. + self._max_ctx_len = md.max_seq_len + # The indexer's prefill logits budget and the held context workspace's rows, from the + # engine's max_num_tokens, max_num_requests, max_seq_len and runtime features. + self.prefill_logits_budget = _prefill_logits_budget(md.max_num_tokens, md.max_seq_len) + self.ctx_workspace_rows = _ctx_workspace_rows(md) + logger.info( + f"modeling_v2 Mistral Large 4 NVFP4: sized from the engine: prefill logits budget " + f"{self.prefill_logits_budget}, context workspace {self.ctx_workspace_rows} rows" + ) + # The decode selector, by the scorer's column count: cute_dsl_radix_topk_decode's + # handle (built once) at a width it serves, else none and the trtllm op selects. + lo, hi = _TOPK_DECODE_COLUMNS + if not (_TOPK_DECODE_CUTE_DSL and lo <= self._max_ctx_len <= hi): + self._ix_topk = None + elif self._ix_topk is None: + self._ix_topk = build_ix_topk_decode(dev, next_n=1) + # Both key-pool layouts: a launcher compiled for a contiguous view addresses a strided + # one as if it were dense. + if self._ix_k is None: + self._ix_k = { + contiguous: build_ix_k_cache(dev, is_neox=False, cache_contiguous=contiguous) + for contiguous in (False, True) + } + self._derived_for = md.kv_cache_manager + + def _select_target(self, attn_metadata) -> "PrefillTarget | DecodeTarget": + """Which target runs this step: one host-side integer read. A mixed batch goes to + prefill, so prefill is the general case and decode the specialization that may assume + it has no context rows.""" + return self._decode if attn_metadata.num_contexts == 0 else self._prefill + + def forward( + self, + attn_metadata: AttentionMetadata, + input_ids: torch.IntTensor | None = None, + position_ids: torch.IntTensor | None = None, + inputs_embeds: torch.FloatTensor | None = None, + lora_params: dict | None = None, + **kwargs, + ) -> torch.Tensor: + """Derive what the first step fixes, then route.""" + self._derive_first_step(attn_metadata) + return self._select_target(attn_metadata).forward( + attn_metadata, + input_ids=input_ids, + position_ids=position_ids, + inputs_embeds=inputs_embeds, + lora_params=lora_params, + **kwargs, + ) + + +class PrefillTarget: + """The general case: context rows, and possibly generation rows behind them.""" + + # The held context workspace, bf16 [core.ctx_workspace_rows, 576]: allocated on the first + # context step and held from then on (`_ctx_workspace`). + _ws_buf: torch.Tensor | None = None + # The indexer's prefill top-k scratch (`gvr_workspace`, 20 MiB): allocated on the first + # context step that selects and held, used on the main stream only (`_select_ctx`). + _gvr_ws: torch.Tensor | None = None + + def __init__(self, core: MistralLarge4TepModelingV2) -> None: + """Take the per-layer views once, off the real weights. Everything an engine value + fixes is read off `self.core` at the call.""" + self.core = core + # Off unless TRTLLM_MODELING_V2_VALIDATE asks for it. Each target owns its own flag: + # the two `step_args` projections differ, so probing one does not prove the other. + self._contract_pending = step_contract_enabled() + self._layers = _layer_views(core) + self._embed = core.w["embed"] + self._norm0 = core.w["l0_attn_norm"] + + def _check_step_contract(self, md) -> None: + """Opt-in first-forward fail-fast: calling `step_args` reads every metadata field this + target consumes (private trtllm surface). Fixed at engine construction, so once per + target instance, and off in a served engine.""" + if not self._contract_pending: + return + try: + self.step_args(md) + except AttributeError as exc: + raise AssertionError(f"attention metadata surface drifted: {exc}") from exc + self._contract_pending = False + + def step_args(self, md: TrtllmAttentionMetadata) -> dict: + """The common batch state, plus the context rows': their counts and lengths, the + indexer's prefill windows and scoring blocks, its query split across the TP ranks and + its key gather mappings, and the context rope/append op's offsets and their host prefix + sums.""" + step = _step_common(md, self.core.indexer_layers) + ctx = md.num_contexts + step.update( + num_contexts=ctx, + num_ctx_tokens=md.num_ctx_tokens, + seq_lens=md.seq_lens, + skip_ctx=bool(md.skip_indexer_for_ctx_reqs), + cu_seqlen_ks=md.cu_seqlen_ks, + cu_seqlen_ke=md.cu_seqlen_ke, + # (q_begin, q_end, per-rank sizes) once the step's context rows reach the engine's + # q_split_threshold under chunked prefill: the chunks then cover this rank's + # [q_begin, q_end) only. + indexer_prefill_q_split=md.indexer_prefill_q_split, + num_ctx_kv_tokens=md.num_ctx_kv_tokens, + slot_mapping_fp8_fullkv=md.slot_mapping_fp8_fullkv, + slot_mapping_scale_fullkv=md.slot_mapping_scale_fullkv, + **_ctx_kv_offsets(md, ctx), + **_ctx_host_offsets(md, ctx), + ) + step["indexer_blocks"] = _indexer_blocks(md.indexer_prefill_chunks, step) + return step + + # ---- one layer's pieces, context and generation rows ---- + + def _linear_fp8(self, x, w, w_s): + """K128-quantized FP8 linear through the Rubin MXFP8 kernel.""" + return cute_dsl_fp8_block_scale_linear(x, w, w_s) + + def _latent(self, x, L, nope_lat): + """`[RMSNorm(c_kv) | k_pe]`, 576 wide: a RoPE owner's kv_a row, normalized in place; a + NoPE owner's 512-wide latent normalized into `nope_lat`, whose rope key stays zero.""" + core = self.core + kv = cublas_mm(x, L.wkv_a) + if core.unfused_glue: + c_kv = flashinfer_rmsnorm(kv[:, : core.kv_lora], L.kv_norm, core.eps) + if L.rope: + k_pe = kv[:, core.kv_lora :] + else: + k_pe = torch.zeros([kv.shape[0], core.rope_dim], dtype=kv.dtype, device=kv.device) + return torch.cat([c_kv, k_pe], dim=-1) + if L.rope: + c_kv = kv[:, : core.kv_lora] + flashinfer_rmsnorm(c_kv, L.kv_norm, core.eps, out=c_kv) + return kv + flashinfer_rmsnorm(kv, L.kv_norm, core.eps, out=nope_lat[:, : core.kv_lora]) + return nope_lat + + def _indexer(self, step, x, pos, L, topk_buf, tc: int, skip_ctx: bool, skip_gen: bool): + """Append this step's keys to the layer's key pool; select `index_topk` positions per + row into `topk_buf` for the rows whose phase the engine did not skip. Under the + engine's query split each rank selects its share of the context rows, and the shares + are gathered into rows [0, tc) before any layer reads them.""" + core = self.core + ix = L.indexer + rows = x.shape[0] + table = core._rope.indexer if ix.rope else core._rope.identity + kw = cublas_mm(x, ix.wk_wp) + k = kw[:, : core.idx_dim].contiguous() + k_pool = step["ix_pools"][L.index] + k_pool = cute_dsl_indexer_k_cache_update( + k, ix.k_norm_w, ix.k_norm_b, pos, table.view(-1, 2, core.rope_dim // 2), k_pool, + step["slot_mapping_fp8"], step["slot_mapping_scale"], rows, + is_neox=False, compiled=core._ix_k[k_pool.is_contiguous()], + # Its range check is a host sync: once, on the first (eager) call. + validate=not core._ix_k_checked, + ) # fmt: skip + core._ix_k_checked = True + if skip_ctx and skip_gen: + return + qn = layer_norm(x, ix.q_norm_w, ix.q_norm_b, core.idx_eps) + q = torch.reshape(self._linear_fp8(qn, ix.wq_b, ix.wq_b_s), [rows, core.idx_heads, -1]) + q_fp8 = torch.empty_like(q, dtype=torch.float8_e4m3fn) + weights = torch.empty([rows, core.idx_heads], dtype=torch.float32, device=x.device) + if core.unfused_glue: + q = ssmax_scaling(q, pos, ix.ssmax_s, ix.ssmax_b) + q_fp8, weights = cute_dsl_indexer_q_side_fused( + q, pos, table, kw[:, core.idx_dim :].float().contiguous(), core.idx_weight_scale, + q_fp8, weights, is_neox=False, compiled=core._ix_q, + ) # fmt: skip + else: + # SSMax, rope, e4m3 and the head weights (read in place out of `kw`) in one launch. + q_fp8, weights = ssmax_indexer_q_side_fused( + q, pos, ix.ssmax_s, ix.ssmax_b, table, kw[:, core.idx_dim :], + core.idx_weight_scale, q_fp8, weights, + ) # fmt: skip + if tc and not skip_ctx: + self._select_ctx(step, k_pool, q_fp8, weights, topk_buf) + q_split = step["indexer_prefill_q_split"] + if q_split is not None: + # The chunks covered this rank's share only. Every rank of the group makes + # this call once per selecting layer, in layer order: the communicator pairs + # calls by position. + q_begin, q_end, q_sizes = q_split + topk_buf[:tc].copy_( + allgather(topk_buf[q_begin:q_end], q_sizes, core.tp_group, core.tp_pg) + ) + gen = rows - tc + if gen and not skip_gen: + ctx = step["num_contexts"] + num_gens = step["num_seqs"] - ctx + assert gen % num_gens == 0, (gen, num_gens) + next_n = gen // num_gens + kv_lens = step["kv_lens_cuda"][ctx : ctx + num_gens] + logits = cute_dsl_fp8_paged_mqa_logits( + torch.reshape(q_fp8[tc:], [num_gens, next_n, core.idx_heads, core.idx_dim]), + k_pool, + weights[tc:], + kv_lens, + step["indexer_k_cache_block_offsets"][ctx : ctx + num_gens], + step["scheduler_metadata_buffer"], + core._max_ctx_len, + ) + if next_n != 1 or core._ix_topk is None: + indexer_topk_decode( + logits, kv_lens, topk_buf[tc:], next_n, core.idx_topk, 1, + step["radix_aux_indices"], step["radix_aux_logits"], + ) # fmt: skip + else: + cute_dsl_radix_topk_decode( + logits, kv_lens, topk_buf[tc:], next_n=1, compiled=core._ix_topk + ) + + def _ix_plan(self, step) -> list[SimpleNamespace]: + """Plan the context rows and key windows scored by each query tile. + + The engine's chunk and query-split mappings determine each block's rows, + keys and window offsets. Tiles stay within the prefill logits budget. + The plan is reused across indexer layers within this step.""" + core = self.core + plan = [] + for block in step["indexer_blocks"]: + n_k = block.k_token_end - block.k_token_start + if block.token_end <= block.token_start: + continue + tiles = [] + if n_k > 0: + for lo, hi in _prefill_tiles( + block.token_start, block.token_end, n_k, core.prefill_logits_budget + ): + # The windows are the block's own, indexed from its first row. + ks = block.cu_seqlen_ks[lo - block.token_start : hi - block.token_start] + ke = block.cu_seqlen_ke[lo - block.token_start : hi - block.token_start] + tiles.append((lo, hi, ks, ke)) + plan.append( + SimpleNamespace( + token_start=block.token_start, + token_end=block.token_end, + k_token_start=block.k_token_start, + n_k=n_k, + tiles=tiles, + ) + ) + return plan + + def _select_ctx(self, step, k_pool, q_fp8, weights, topk_buf) -> None: + """Select window-local top-k keys for the context rows assigned to this rank. + + Gather each block's FP8 keys, score each query tile with the Triton MQA + scorer, and pass its aligned logits to GVR top-k. Empty blocks retain + -1 indices. Only one tile's logits are live at a time.""" + core = self.core + plan = step.get("ix_plan") + if plan is None: + plan = step["ix_plan"] = self._ix_plan(step) + if self._gvr_ws is None: + # On the first context step, which the KV-capacity estimate's profiling forward + # is: the pool is sized with the top-k's scratch held, and every top-k launch this + # engine's tiles reach compiles here rather than on a served step (each tile holds + # at most the logits budget, one block at most max_seq_len keys). + self._gvr_ws = gvr_workspace(q_fp8.device) + t0 = time.monotonic() + calls = gvr_warmup( + core.idx_topk, core._max_ctx_len, core.prefill_logits_budget, q_fp8.device, + self._gvr_ws, + ) # fmt: skip + logger.info( + f"modeling_v2 Mistral Large 4 NVFP4: indexer prefill top-k compiled for this " + f"engine's tiles ({calls} warm calls, {time.monotonic() - t0:.1f} s)" + ) + for block in plan: + if block.n_k <= 0: + # No key to select from: every slot of the block's rows is empty. + topk_buf[block.token_start : block.token_end].fill_(-1) + continue + ck, cs = indexer_k_cache_gather_op( + k_pool, step["slot_mapping_fp8_fullkv"], step["slot_mapping_scale_fullkv"], + block.k_token_start, block.n_k, core.idx_dim, + ) # fmt: skip + cs = torch.reshape(cs, [-1]) + for lo, hi, ks, ke in block.tiles: + logits = triton_fp8_mqa_logits(q_fp8[lo:hi], (ck, cs), weights[lo:hi], ks, ke) + cute_dsl_gvr_topk_prefill( + logits, ks, ke, topk_buf[lo:hi], core.idx_topk, self._gvr_ws + ) + # Freed before the next tile's is allocated: one tile is live at a time. + del logits + + def _attn_ctx(self, step, L, fused_q, lat_out, req_idx, topk_buf, ctx): + """Sparse MLA over the context rows into `lat_out`, on the step's path (`ctx`). + + Workspace path: FlashMLA's bf16 sparse prefill over the context requests' latent rows + of the source layer, dequantized out of the pool (this step's rows included, so every + row is read FP8-rounded) into the held workspace's first rows. A shared source (layer + 35) is gathered once into the rows the step keeps beside them, when it has them, and + read there by every layer that shares it; otherwise each of those layers gathers it + again into the first rows. Decode path, a step past the held workspace: the selections + as pool slots, then FlashMLA's sparse fp8 decode over the pool with every context row a + one-row sequence, in the step's sub-batches.""" + core = self.core + tc = fused_q.shape[0] + src = L.kv_source + if ctx.ws_buf is None: + slots = convert_req_index_to_global( + req_idx[:tc], step["block_table"], topk_buf[:tc], step["tokens_per_block"], + core.idx_topk, core._stride_factor, core._pool_row[src], + ) # fmt: skip + for lo, hi in ctx.sub_batches: + # Past 11622 rows FlashMLA's scheduler exits the process instead of raising. + assert hi - lo <= _CTX_DECODE_ROWS, (lo, hi) + flash_mla_sparse_decode( + fused_q[lo:hi], step["latent_kv"], slots[lo:hi], ctx.sched[hi - lo], + core.softmax_scale, lat_out[lo:hi], + ) # fmt: skip + return + ws = ctx.ws.get(src) + if ws is None: + keep = core.shared_source[src] and ctx.kept is not None + ws = ds_mla_dequant_gather( + step["latent_pool"], core._pool_row[src], ctx.kv_req, ctx.kv_pos, + step["block_table"], ctx.kept if keep else ctx.ws_buf, + ) # fmt: skip + if keep: + ctx.ws[src] = ws + # Request-local selections to workspace rows: each request's run starts at its + # cumulative KV offset. + sel = topk_buf[:tc] + idx = torch.where(sel >= 0, sel + ctx.ws_off[req_idx[:tc]][:, None], sel) + flash_mla_sparse_prefill(fused_q, ws, idx.to(torch.int32), core.softmax_scale, lat_out) + + def _attn_gen(self, step, L, fused_q, lat_out, slots, sched): + """Sparse MLA over the generation rows behind the context rows into `lat_out`, over the + pool: this rank's heads read off the 64-head scratch the prefill kernel shares (or, under + `gen_attn_flash_mla`, FlashMLA's decode over all 64, one scheduler for the step).""" + core = self.core + if core.gen_attn_flash_mla: + flash_mla_sparse_decode( + fused_q, step["latent_kv"], slots, sched, core.softmax_scale, lat_out + ) + else: + cute_dsl_ds_mla_sparse_decode( + fused_q, step["latent_kv"], slots, core.softmax_scale, lat_out, + compiled=core._mla_decode, + ) # fmt: skip + + def _mlp(self, o, L): + """This rank's share of the layer's MLP, a partial sum: the dense SwiGLU slice on the + dense layer; on a MoE layer the expert window (32 experts at tep4, 16 at tep8) plus the + shared expert's slice.""" + core = self.core + act = flashinfer_silu_and_mul(self._linear_fp8(o, L.mlp_gu, L.mlp_gu_s)) + shared = self._linear_fp8(act, L.mlp_dn, L.mlp_dn_s) + m = L.moe + if m is None: + return shared + logits = cublas_mm(o, m.router, None, torch.float32) + ids, weights = renorm_moe_routing_op(logits, core.topk, torch.bfloat16) + # One NVFP4 quantization under the layer's static scale serves every expert on every + # rank: the windows' partials sum. + xq, xs = fp4_quantize(o, m.gscale, _NVFP4_SF_VEC, False, _NVFP4_SF_LINEAR) + routed = fp4_block_scale_moe_runner( + None, None, xq, xs.view(torch.float8_e4m3fn), m.fc1_w, m.fc1_s, None, None, None, + None, m.fc2_w, m.fc2_s, None, m.out1, m.out1_gate, m.out2, core.num_experts, + core.topk, None, None, core.moe_inter, core.expert_offset, core.local_experts, None, + _MOE_ROUTING_METHOD, True, _MOE_ACT_SWIGLU, topk_weights=weights, topk_ids=ids, + tune_max_num_tokens=core.max_num_tokens, + )[0] # fmt: skip + return torch.add(routed, shared) + + def _topk_buf(self, step, rows: int, tc: int, skip_ctx: bool, skip_gen: bool): + """Where this step's selections land, request-local positions. A phase the engine + skipped (every sequence within index_topk) takes the dense causal index it wrote.""" + if skip_ctx and skip_gen: + return step["topk_indices_buffer"][:rows] + buf = torch.empty( + [rows, self.core.idx_topk], dtype=torch.int32, device=step["block_table"].device + ) + if skip_ctx and tc: + buf[:tc].copy_(step["topk_indices_buffer"][:tc]) + if skip_gen and rows > tc: + buf[tc:].copy_(step["topk_indices_buffer"][tc:rows]) + return buf + + def forward( + self, + attn_metadata: AttentionMetadata, + input_ids: torch.IntTensor | None = None, + position_ids: torch.IntTensor | None = None, + inputs_embeds: torch.FloatTensor | None = None, + lora_params: dict | None = None, + **kwargs, + ) -> torch.Tensor: + core = self.core + self._check_step_contract(attn_metadata) + step = self.step_args(attn_metadata) + h = ( + nn.functional.embedding(input_ids, self._embed) + if inputs_embeds is None + else inputs_embeds + ) + rows = h.shape[0] + dev = h.device + # The slot mappings and windows follow the prepared token order; a padded input + # would scatter keys into stale slots. + assert rows == step["num_tokens"], (rows, step["num_tokens"]) + tc = step["num_ctx_tokens"] + gen = rows - tc + skip_ctx = step["skip_ctx"] + skip_gen = step["skip_gen"] if gen else True + pos = torch.reshape(position_ids, [-1]) + req_idx = step["req_idx_per_token"][:rows] + topk_buf = self._topk_buf(step, rows, tc, skip_ctx, skip_gen) + scratch = self._scratch(rows, h.dtype, dev) + ctx = self._ctx_workspace(step, tc, dev) if tc else None + + x = flashinfer_rmsnorm(h, self._norm0, core.eps) + residual = h + spec_metadata = kwargs.get("spec_metadata") + for layer_idx, L in enumerate(self._layers): + attn_out = self._attention( + step, L, x, pos, req_idx, topk_buf, scratch, ctx, tc, skip_ctx, skip_gen + ) + partial = self._linear_fp8(attn_out, L.wo, L.wo_s) + o = cute_dsl_allreduce_add_rmsnorm( + partial, residual, L.ffn_norm, torch.empty_like(partial), core._lamport + ) + mlp = self._mlp(o, L) + x = cute_dsl_allreduce_add_rmsnorm( + mlp, residual, L.next_norm, torch.empty_like(mlp), core._lamport + ) + if spec_metadata is not None: + spec_metadata.maybe_capture_hidden_states(layer_idx, residual) + return x + + def _scratch(self, rows: int, dtype, dev) -> SimpleNamespace: + """Per-step scratch every layer rewrites in turn: the attention output, the absorbed + query at the prefill kernel's 64 heads (this rank's first, the rest zero for the step), + the latent attention output, a NoPE owner's 576-wide latent rows (the rope key zero for + the step), and, under `gen_attn_flash_mla`, the generation rows' FlashMLA scheduler, + filled by the step's first decode call and shared by the rest.""" + core = self.core + bf16 = dict(dtype=dtype, device=dev) + return SimpleNamespace( + attn_out=torch.empty([rows, core.heads * core.v_dim], **bf16), + fused_q=torch.zeros([rows, _FLASH_MLA_HEADS, core.lat_dim], **bf16), + lat_out=torch.empty([rows, core.heads, core.kv_lora], **bf16), + nope_lat=None if core.unfused_glue else torch.zeros([rows, core.lat_dim], **bf16), + sched=get_mla_metadata()[0] if core.gen_attn_flash_mla else None, + ) + + def _ctx_workspace(self, step, tc: int, dev) -> SimpleNamespace: + """The context rows' path for the step, chosen once on host ints by its context KV: + every context request's cached prefix and new tokens (the host + `ctx_kv_indptr[num_contexts]`; the engine's `num_ctx_kv_tokens` counts the new ones + only, and under chunked prefill every chunk after a prompt's first has a prefix). + + The held workspace, bf16 `[core.ctx_workspace_rows, 576]` (`_ctx_workspace_rows`), is + allocated on the first context step, whichever path that step takes, and held. The + engine's KV-capacity estimate runs context steps as its profiling forward and sizes the + pool after their peak memory (`KvCacheCreator.configure_kv_cache_capacity`), so the pool + is sized with the workspace held. A generation-only engine runs no context step and + holds none. + + The workspace path, a step whose context KV fits the held rows: request and position + per workspace row as the dequantizing gather and the prefill kernel address them, each + request's offset, the held workspace's first `n_kv` rows, which every layer's gather + rewrites, and, when the context KV fits twice, the next `n_kv` rows (`kept`), where a + shared owner's gather stays for the step (`ws`). + + The decode path, a step past the held rows, with no workspace: the context rows' + sub-batches of at most `_CTX_DECODE_ROWS`, and one FlashMLA scheduler per sub-batch + size, filled by its first call and shared by the step's layers.""" + # Under KV reuse a context request attends over its cached prefix and its new tokens. + n_kv = step["ctx_cached_tokens"] + step["num_ctx_tokens"] + if self._ws_buf is None: + # On the first context step: the KV-capacity estimate's profiling forward is one, + # so the peak it measures, which sizes the pool, holds this buffer. + self._ws_buf = torch.empty( + [self.core.ctx_workspace_rows, self.core.lat_dim], dtype=torch.bfloat16, device=dev + ) + logger.info( + f"modeling_v2 Mistral Large 4 NVFP4: context workspace held, {self._ws_buf.shape[0]} " + f"rows x {self.core.lat_dim} bf16 ({self._ws_buf.nbytes / 2**20:.0f} MiB)" + ) + held = self._ws_buf.shape[0] + if n_kv > held: + bounds = [(lo, min(lo + _CTX_DECODE_ROWS, tc)) for lo in range(0, tc, _CTX_DECODE_ROWS)] + logger.debug( + f"modeling_v2 Mistral Large 4 NVFP4: context KV {n_kv} past the held {held} rows: " + f"decode path, {len(bounds)} sub-batches" + ) + return SimpleNamespace( + ws_buf=None, + sub_batches=bounds, + sched={rows: get_mla_metadata()[0] for rows in {hi - lo for lo, hi in bounds}}, + ) + ctx = step["num_contexts"] + cu = step["cu_ctx_kv"] + lens = cu[1:] - cu[:-1] + kv_req = torch.repeat_interleave( + torch.arange(ctx, dtype=torch.int32, device=dev), lens, output_size=n_kv + ) + kv_pos = (torch.arange(n_kv, device=dev) - cu[:-1][kv_req]).to(torch.int32) + keep = 2 * n_kv <= held + logger.debug( + f"modeling_v2 Mistral Large 4 NVFP4: context KV {n_kv} of the held {held} rows: " + f"workspace path, the shared source {'kept' if keep else 'gathered again'}" + ) + return SimpleNamespace( + kv_req=kv_req, + kv_pos=kv_pos, + ws_off=cu[:-1], + ws_buf=self._ws_buf[:n_kv], + kept=self._ws_buf[n_kv : 2 * n_kv] if keep else None, + ws={}, + ) + + def _attention(self, step, L, x, pos, req_idx, topk_buf, s, ctx, tc, skip_ctx, skip_gen): + """One layer's attention block, both phases, into `s.attn_out` (returned): the query + with SSMax, the indexer's selection, the absorbed query, the owner's rope and + fp8_ds_mla append, sparse MLA over the context and the generation rows, and the V + up-projection.""" + core = self.core + rows = x.shape[0] + q = self._linear_fp8(x, L.wq, L.wq_s) + q3 = torch.reshape(q, [rows, core.heads, core.qk_dim]) + fq = s.fused_q[:, : core.heads] + if core.unfused_glue: + q3 = ssmax_scaling(q3, pos, L.ssmax_s, L.ssmax_b) + if L.rope: + fq[..., core.kv_lora :].copy_(q3[..., L.nope :]) + else: + fq[..., core.kv_lora :].zero_() + else: + # SSMax in place; its rope half (a NoPE layer's zero) into the absorbed query's tail. + ssmax_scaling_fused_q( + q3, pos, L.ssmax_s, L.ssmax_b, q3, fq[..., core.kv_lora :], L.rope + ) + if L.indexer is not None: + self._indexer(step, x, pos, L, topk_buf, tc, skip_ctx, skip_gen) + bmm_out( + torch.transpose(q3[..., : L.nope], 0, 1), L.k_b, + torch.transpose(fq[..., : core.kv_lora], 0, 1), + ) # fmt: skip + if L.owner: + # Every row's latent, context and generation, before either phase reads the pool; + # the query tail is roped in the same call. + ds_mla_rope_append( + self._latent(x, L, s.nope_lat), fq, pos, core._rope.indexer, req_idx, + step["block_table"], step["latent_pool"], core._pool_row[L.index], L.rope, + ) # fmt: skip + if tc: + self._attn_ctx(step, L, s.fused_q[:tc], s.lat_out[:tc], req_idx, topk_buf, ctx) + if rows > tc: + slots = convert_req_index_to_global( + req_idx[tc:], step["block_table"], topk_buf[tc:], step["tokens_per_block"], + core.idx_topk, core._stride_factor, core._pool_row[L.kv_source], + ) # fmt: skip + self._attn_gen(step, L, s.fused_q[tc:], s.lat_out[tc:], slots, s.sched) + bmm_out( + torch.transpose(s.lat_out, 0, 1), + L.v_b_t, + torch.transpose(torch.reshape(s.attn_out, [rows, core.heads, core.v_dim]), 0, 1), + ) + return s.attn_out + + def dspark_forward( + self, + draft, + noise_embedding, + query_positions, + num_ctx_per_req, + ctx_k_cache, + ctx_v_cache, + ctx_cache_batch_idx, + ctx_kv_cache=None, + ctx_page_table=None, + block_placement=None, + ): + # Only the worker's fused paged-latent preparation places the block + # (block_placement), and a drafter with embed_draft_block does not take + # it; this decode reads its context window through ctx_page_table. + del block_placement + draft._build_fused_kv_buffers() + draft._ensure_rope() + core = self.core + batch, block, hidden = noise_embedding.shape + rows = batch * block + # Gather a bounded window, not every page in the configured context. + # Invalid padding slots are masked even if their pages are allocated. + lengths = num_ctx_per_req[:batch].long() + offsets = torch.arange(draft.window, device=noise_embedding.device) + positions = (lengths - draft.window).clamp_min(0)[:, None] + offsets[None, :] + valid = positions < lengths[:, None] + slots = ctx_cache_batch_idx.long() + if ctx_page_table is not None: + page_size = ctx_kv_cache[0].shape[-2] + table = ctx_page_table.index_select(0, slots).long() + safe_positions = positions.clamp(max=table.shape[1] * page_size - 1) + pages = table.gather(1, safe_positions // page_size).clamp_min(0) + page_offsets = safe_positions % page_size + else: + safe_positions = positions.clamp(max=ctx_k_cache.shape[2] - 1) + key_valid = torch.cat( + (valid, torch.ones(batch, block, dtype=torch.bool, device=valid.device)), dim=1 + ) + query = torch.empty( + batch, + block, + core.heads, + core.lat_dim, + dtype=noise_embedding.dtype, + device=noise_embedding.device, + ) + values = torch.empty( + rows, core.heads, core.v_dim, dtype=noise_embedding.dtype, device=noise_embedding.device + ) + residual = noise_embedding.reshape(rows, hidden).clone() + normalized = torch.empty_like(residual) + x = flashinfer_rmsnorm(residual, draft._layers[0].attn_norm, core.eps) + for i, layer in enumerate(draft._layers): + q = cublas_mm(x, layer.q, None, None).view(batch, block, core.heads, core.qk_dim) + bmm_out( + q[..., : core.nope].reshape(rows, core.heads, core.nope).transpose(0, 1), + layer.k_b, + query[..., : core.kv_lora].view(rows, core.heads, core.kv_lora).transpose(0, 1), + ) + query[..., core.kv_lora :].copy_(draft._rotate(q[..., core.nope :], query_positions)) + kv = cublas_mm(x, layer.kv_a, None, None) + content = flashinfer_rmsnorm( + kv[:, : core.kv_lora].contiguous(), layer.kv_norm, core.eps + ) + block_kv = torch.cat( + ( + content.view(batch, block, core.kv_lora), + draft._rotate( + kv[:, core.kv_lora :].view(batch, block, core.rope_dim), query_positions + ), + ), + dim=-1, + ) + if ctx_page_table is not None: + context = ctx_kv_cache[i][:, 0, 0][pages, page_offsets] + else: + context = ctx_k_cache[slots[:, None], i, safe_positions, 0] + # Zero masked values as well as scores: 0 * an uninitialized + # NaN in the latent pool must not contaminate the weighted sum. + context = context.masked_fill(~valid.unsqueeze(-1), 0) + keys = torch.cat((context, block_kv), dim=1) + scores = torch.einsum("bshd,btd->bsht", query, keys).float() * core.softmax_scale + scores = scores.masked_fill(~key_valid[:, None, None, :], float("-inf")) + probabilities = torch.softmax(scores, dim=-1).to(keys.dtype) + latent = torch.einsum("bsht,btc->bshc", probabilities, keys[..., : core.kv_lora]) + bmm_out( + latent.reshape(rows, core.heads, core.kv_lora).transpose(0, 1), + layer.v_b.transpose(1, 2), + values.transpose(0, 1), + ) + partial = cublas_mm(values.reshape(rows, -1), layer.out, None, None) + o = cute_dsl_allreduce_add_rmsnorm( + partial, residual, layer.ffn_norm, normalized, core._lamport + ) + gate = cublas_mm(o, layer.gate, None, None) + up = cublas_mm(o, layer.up, None, None) + activated = flashinfer_silu_and_mul(torch.cat((gate, up), dim=-1)) + mlp = cublas_mm(activated, layer.down, None, None) + x = cute_dsl_allreduce_add_rmsnorm( + mlp, residual, layer.next_norm, normalized, core._lamport + ) + return x + + +class DecodeTarget: + """Generation-only steps, including uniform DSpark verification blocks; graph captured.""" + + def __init__(self, core: MistralLarge4TepModelingV2) -> None: + """Take the per-layer views once, off the real weights. Everything an engine value + fixes is read off `self.core` at the call.""" + self.core = core + self._contract_pending = step_contract_enabled() + self._layers = _layer_views(core) + self._embed = core.w["embed"] + self._norm0 = core.w["l0_attn_norm"] + + def _check_step_contract(self, md) -> None: + """Opt-in first-forward fail-fast: calling `step_args` reads every metadata field this + target consumes (private trtllm surface). Fixed at engine construction, so once per + target instance, and off in a served engine.""" + if not self._contract_pending: + return + try: + self.step_args(md) + except AttributeError as exc: + raise AssertionError(f"attention metadata surface drifted: {exc}") from exc + self._contract_pending = False + + def step_args(self, md: TrtllmAttentionMetadata) -> dict: + """The common batch state. States the two context counts instead of reading them + back: this target runs only when there are no context rows.""" + step = _step_common(md, self.core.indexer_layers) + step.update(num_contexts=0, num_ctx_tokens=0) + return step + + def _linear_fp8(self, x, w, w_s): + """K128-quantized FP8 linear through the Rubin MXFP8 kernel.""" + return cute_dsl_fp8_block_scale_linear(x, w, w_s) + + def _latent(self, x, L, nope_lat): + """`[RMSNorm(c_kv) | k_pe]`, 576 wide: a RoPE owner's kv_a row, normalized in place; a + NoPE owner's 512-wide latent normalized into `nope_lat`, whose rope key stays zero.""" + core = self.core + kv = cublas_mm(x, L.wkv_a) + if core.unfused_glue: + c_kv = flashinfer_rmsnorm(kv[:, : core.kv_lora], L.kv_norm, core.eps) + if L.rope: + k_pe = kv[:, core.kv_lora :] + else: + k_pe = torch.zeros([kv.shape[0], core.rope_dim], dtype=kv.dtype, device=kv.device) + return torch.cat([c_kv, k_pe], dim=-1) + if L.rope: + c_kv = kv[:, : core.kv_lora] + flashinfer_rmsnorm(c_kv, L.kv_norm, core.eps, out=c_kv) + return kv + flashinfer_rmsnorm(kv, L.kv_norm, core.eps, out=nope_lat[:, : core.kv_lora]) + return nope_lat + + def _indexer(self, step, x, pos, L, topk_buf, skip_gen: bool): + """Append this step's keys to the layer's key pool; unless the engine skipped the + indexer (every sequence within index_topk), score every sequence's paged keys and + select `index_topk` positions per row into `topk_buf`.""" + core = self.core + ix = L.indexer + rows = x.shape[0] + table = core._rope.indexer if ix.rope else core._rope.identity + kw = cublas_mm(x, ix.wk_wp) + k = kw[:, : core.idx_dim].contiguous() + k_pool = step["ix_pools"][L.index] + k_pool = cute_dsl_indexer_k_cache_update( + k, ix.k_norm_w, ix.k_norm_b, pos, table.view(-1, 2, core.rope_dim // 2), k_pool, + step["slot_mapping_fp8"], step["slot_mapping_scale"], rows, + is_neox=False, compiled=core._ix_k[k_pool.is_contiguous()], validate=False, + ) # fmt: skip + if skip_gen: + return + qn = layer_norm(x, ix.q_norm_w, ix.q_norm_b, core.idx_eps) + q = torch.reshape(self._linear_fp8(qn, ix.wq_b, ix.wq_b_s), [rows, core.idx_heads, -1]) + q_fp8 = torch.empty_like(q, dtype=torch.float8_e4m3fn) + weights = torch.empty([rows, core.idx_heads], dtype=torch.float32, device=x.device) + if core.unfused_glue: + q = ssmax_scaling(q, pos, ix.ssmax_s, ix.ssmax_b) + q_fp8, weights = cute_dsl_indexer_q_side_fused( + q, pos, table, kw[:, core.idx_dim :].float().contiguous(), core.idx_weight_scale, + q_fp8, weights, is_neox=False, compiled=core._ix_q, + ) # fmt: skip + else: + # SSMax, rope, e4m3 and the head weights (read in place out of `kw`) in one launch. + q_fp8, weights = ssmax_indexer_q_side_fused( + q, pos, ix.ssmax_s, ix.ssmax_b, table, kw[:, core.idx_dim :], + core.idx_weight_scale, q_fp8, weights, + ) # fmt: skip + num_gens = step["num_seqs"] + next_n = rows // num_gens + kv_lens = step["kv_lens_cuda"][:num_gens] + logits = cute_dsl_fp8_paged_mqa_logits( + torch.reshape(q_fp8, [num_gens, next_n, core.idx_heads, core.idx_dim]), + k_pool, + weights, + kv_lens, + step["indexer_k_cache_block_offsets"][:num_gens], + step["scheduler_metadata_buffer"], + core._max_ctx_len, + ) + if next_n != 1 or core._ix_topk is None: + indexer_topk_decode( + logits, kv_lens, topk_buf, next_n, core.idx_topk, 1, step["radix_aux_indices"], + step["radix_aux_logits"], + ) # fmt: skip + else: + cute_dsl_radix_topk_decode(logits, kv_lens, topk_buf, next_n=1, compiled=core._ix_topk) + + def _mlp(self, o, L): + """This rank's share of the layer's MLP, a partial sum: the dense SwiGLU slice on the + dense layer; on a MoE layer the expert window (32 experts at tep4, 16 at tep8) plus the + shared expert's slice.""" + core = self.core + act = flashinfer_silu_and_mul(self._linear_fp8(o, L.mlp_gu, L.mlp_gu_s)) + shared = self._linear_fp8(act, L.mlp_dn, L.mlp_dn_s) + m = L.moe + if m is None: + return shared + logits = cublas_mm(o, m.router, None, torch.float32) + ids, weights = renorm_moe_routing_op(logits, core.topk, torch.bfloat16) + # One NVFP4 quantization under the layer's static scale serves every expert on every + # rank: the windows' partials sum. + xq, xs = fp4_quantize(o, m.gscale, _NVFP4_SF_VEC, False, _NVFP4_SF_LINEAR) + routed = fp4_block_scale_moe_runner( + None, None, xq, xs.view(torch.float8_e4m3fn), m.fc1_w, m.fc1_s, None, None, None, + None, m.fc2_w, m.fc2_s, None, m.out1, m.out1_gate, m.out2, core.num_experts, + core.topk, None, None, core.moe_inter, core.expert_offset, core.local_experts, None, + _MOE_ROUTING_METHOD, True, _MOE_ACT_SWIGLU, topk_weights=weights, topk_ids=ids, + tune_max_num_tokens=core.max_num_tokens, + )[0] # fmt: skip + return torch.add(routed, shared) + + def forward( + self, + attn_metadata: AttentionMetadata, + input_ids: torch.IntTensor | None = None, + position_ids: torch.IntTensor | None = None, + inputs_embeds: torch.FloatTensor | None = None, + lora_params: dict | None = None, + **kwargs, + ) -> torch.Tensor: + core = self.core + self._check_step_contract(attn_metadata) + step = self.step_args(attn_metadata) + h = ( + nn.functional.embedding(input_ids, self._embed) + if inputs_embeds is None + else inputs_embeds + ) + rows = h.shape[0] + dev = h.device + assert rows == step["num_tokens"] and rows % step["num_seqs"] == 0, ( + rows, + step["num_tokens"], + step["num_seqs"], + ) + # A separate graph is captured for steps whose sequences all fit in index_topk, so + # this host branch is a per-capture constant. + skip_gen = step["skip_gen"] + pos = torch.reshape(position_ids, [-1]) + req_idx = step["req_idx_per_token"][:rows] + if skip_gen: + topk_buf = step["topk_indices_buffer"][:rows] + else: + topk_buf = torch.empty([rows, core.idx_topk], dtype=torch.int32, device=dev) + scratch = self._scratch(rows, h.dtype, dev) + + x = flashinfer_rmsnorm(h, self._norm0, core.eps) + residual = h + spec_metadata = kwargs.get("spec_metadata") + for layer_idx, L in enumerate(self._layers): + attn_out = self._attention(step, L, x, pos, req_idx, topk_buf, scratch, skip_gen) + partial = self._linear_fp8(attn_out, L.wo, L.wo_s) + o = cute_dsl_allreduce_add_rmsnorm( + partial, residual, L.ffn_norm, torch.empty_like(partial), core._lamport + ) + mlp = self._mlp(o, L) + x = cute_dsl_allreduce_add_rmsnorm( + mlp, residual, L.next_norm, torch.empty_like(mlp), core._lamport + ) + if spec_metadata is not None: + spec_metadata.maybe_capture_hidden_states(layer_idx, residual) + return x + + def _scratch(self, rows: int, dtype, dev) -> SimpleNamespace: + """Per-step scratch every layer rewrites in turn: the attention output, the absorbed + query at this rank's heads (every element written by each layer: the absorption BMM + and the SSMax op's rope tail or zeros), a NoPE owner's 576-wide latent rows (the rope + key zero for the step), and the latent attention output the native decode writes. + Under `gen_attn_flash_mla` the query is FlashMLA's 64 heads (this rank's first, the + rest zero for the step), the scratch carries the FlashMLA scheduler the step's first + layer fills and the rest share (captured with the step), and the latent output is kept + only for the unfused glue, which copies FlashMLA's live heads into it.""" + core = self.core + bf16 = dict(dtype=dtype, device=dev) + if core.gen_attn_flash_mla: + fused_q = torch.zeros([rows, _FLASH_MLA_HEADS, core.lat_dim], **bf16) + else: + fused_q = torch.empty([rows, core.heads, core.lat_dim], **bf16) + return SimpleNamespace( + attn_out=torch.empty([rows, core.heads * core.v_dim], **bf16), + fused_q=fused_q, + nope_lat=None if core.unfused_glue else torch.zeros([rows, core.lat_dim], **bf16), + lat_out=( + torch.empty([rows, core.heads, core.kv_lora], **bf16) + if core.unfused_glue or not core.gen_attn_flash_mla + else None + ), + sched=get_mla_metadata()[0] if core.gen_attn_flash_mla else None, + ) + + def _attention(self, step, L, x, pos, req_idx, topk_buf, s, skip_gen: bool): + """One layer's attention block into `s.attn_out` (returned): the query with SSMax, the + indexer's selection, the absorbed query, the owner's rope and fp8_ds_mla append, + sparse decode over the pool, and the V up-projection of the latent output (FlashMLA's + live heads read in place under `gen_attn_flash_mla` with the fused glue).""" + core = self.core + rows = x.shape[0] + q = self._linear_fp8(x, L.wq, L.wq_s) + q3 = torch.reshape(q, [rows, core.heads, core.qk_dim]) + fq = s.fused_q[:, : core.heads] + if core.unfused_glue: + q3 = ssmax_scaling(q3, pos, L.ssmax_s, L.ssmax_b) + if L.rope: + fq[..., core.kv_lora :].copy_(q3[..., L.nope :]) + else: + fq[..., core.kv_lora :].zero_() + else: + # SSMax in place; its rope half (a NoPE layer's zero) into the absorbed query's tail. + ssmax_scaling_fused_q( + q3, pos, L.ssmax_s, L.ssmax_b, q3, fq[..., core.kv_lora :], L.rope + ) + if L.indexer is not None: + self._indexer(step, x, pos, L, topk_buf, skip_gen) + slots = convert_req_index_to_global( + req_idx, step["block_table"], topk_buf, step["tokens_per_block"], core.idx_topk, + core._stride_factor, core._pool_row[L.kv_source], + ) # fmt: skip + bmm_out( + torch.transpose(q3[..., : L.nope], 0, 1), L.k_b, + torch.transpose(fq[..., : core.kv_lora], 0, 1), + ) # fmt: skip + if L.owner: + ds_mla_rope_append( + self._latent(x, L, s.nope_lat), fq, pos, core._rope.indexer, req_idx, + step["block_table"], step["latent_pool"], core._pool_row[L.index], L.rope, + ) # fmt: skip + if core.gen_attn_flash_mla and core.unfused_glue: + lat = flash_mla_sparse_decode( + s.fused_q, step["latent_kv"], slots, s.sched, core.softmax_scale, s.lat_out + ) + elif core.gen_attn_flash_mla: + lat = flash_mla_sparse_decode( + s.fused_q, step["latent_kv"], slots, s.sched, core.softmax_scale, None + )[:, : core.heads] + else: + # The rank's heads, unpadded, into the latent output. + cute_dsl_ds_mla_sparse_decode( + s.fused_q, step["latent_kv"], slots, core.softmax_scale, s.lat_out, + compiled=core._mla_decode, + ) # fmt: skip + lat = s.lat_out + bmm_out( + torch.transpose(lat, 0, 1), + L.v_b_t, + torch.transpose(torch.reshape(s.attn_out, [rows, core.heads, core.v_dim]), 0, 1), + ) + return s.attn_out + + def dspark_forward( + self, + draft, + noise_embedding, + query_positions, + num_ctx_per_req, + ctx_k_cache, + ctx_v_cache, + ctx_cache_batch_idx, + ctx_kv_cache=None, + ctx_page_table=None, + block_placement=None, + ): + # Only the worker's fused paged-latent preparation places the block + # (block_placement), and a drafter with embed_draft_block does not take + # it; this decode reads its context window through ctx_page_table. + del block_placement + draft._build_fused_kv_buffers() + draft._ensure_rope() + core = self.core + batch, block, hidden = noise_embedding.shape + rows = batch * block + # Gather a bounded window, not every page in the configured context. + # Invalid padding slots are masked even if their pages are allocated. + lengths = num_ctx_per_req[:batch].long() + offsets = torch.arange(draft.window, device=noise_embedding.device) + positions = (lengths - draft.window).clamp_min(0)[:, None] + offsets[None, :] + valid = positions < lengths[:, None] + slots = ctx_cache_batch_idx.long() + if ctx_page_table is not None: + page_size = ctx_kv_cache[0].shape[-2] + table = ctx_page_table.index_select(0, slots).long() + safe_positions = positions.clamp(max=table.shape[1] * page_size - 1) + pages = table.gather(1, safe_positions // page_size).clamp_min(0) + page_offsets = safe_positions % page_size + else: + safe_positions = positions.clamp(max=ctx_k_cache.shape[2] - 1) + key_valid = torch.cat( + (valid, torch.ones(batch, block, dtype=torch.bool, device=valid.device)), dim=1 + ) + query = torch.empty( + batch, + block, + core.heads, + core.lat_dim, + dtype=noise_embedding.dtype, + device=noise_embedding.device, + ) + values = torch.empty( + rows, core.heads, core.v_dim, dtype=noise_embedding.dtype, device=noise_embedding.device + ) + residual = noise_embedding.reshape(rows, hidden).clone() + normalized = torch.empty_like(residual) + x = flashinfer_rmsnorm(residual, draft._layers[0].attn_norm, core.eps) + for i, layer in enumerate(draft._layers): + q = cublas_mm(x, layer.q, None, None).view(batch, block, core.heads, core.qk_dim) + bmm_out( + q[..., : core.nope].reshape(rows, core.heads, core.nope).transpose(0, 1), + layer.k_b, + query[..., : core.kv_lora].view(rows, core.heads, core.kv_lora).transpose(0, 1), + ) + query[..., core.kv_lora :].copy_(draft._rotate(q[..., core.nope :], query_positions)) + kv = cublas_mm(x, layer.kv_a, None, None) + content = flashinfer_rmsnorm( + kv[:, : core.kv_lora].contiguous(), layer.kv_norm, core.eps + ) + block_kv = torch.cat( + ( + content.view(batch, block, core.kv_lora), + draft._rotate( + kv[:, core.kv_lora :].view(batch, block, core.rope_dim), query_positions + ), + ), + dim=-1, + ) + if ctx_page_table is not None: + context = ctx_kv_cache[i][:, 0, 0][pages, page_offsets] + else: + context = ctx_k_cache[slots[:, None], i, safe_positions, 0] + # Zero masked values as well as scores: 0 * an uninitialized + # NaN in the latent pool must not contaminate the weighted sum. + context = context.masked_fill(~valid.unsqueeze(-1), 0) + keys = torch.cat((context, block_kv), dim=1) + scores = torch.einsum("bshd,btd->bsht", query, keys).float() * core.softmax_scale + scores = scores.masked_fill(~key_valid[:, None, None, :], float("-inf")) + probabilities = torch.softmax(scores, dim=-1).to(keys.dtype) + latent = torch.einsum("bsht,btc->bshc", probabilities, keys[..., : core.kv_lora]) + bmm_out( + latent.reshape(rows, core.heads, core.kv_lora).transpose(0, 1), + layer.v_b.transpose(1, 2), + values.transpose(0, 1), + ) + partial = cublas_mm(values.reshape(rows, -1), layer.out, None, None) + o = cute_dsl_allreduce_add_rmsnorm( + partial, residual, layer.ffn_norm, normalized, core._lamport + ) + gate = cublas_mm(o, layer.gate, None, None) + up = cublas_mm(o, layer.up, None, None) + activated = flashinfer_silu_and_mul(torch.cat((gate, up), dim=-1)) + mlp = cublas_mm(activated, layer.down, None, None) + x = cute_dsl_allreduce_add_rmsnorm( + mlp, residual, layer.next_norm, normalized, core._lamport + ) + return x + + +class DraftModel(nn.Module): + """This specialization's BF16 DSpark backbone and recurrent head. + + Runtime owns accepted-token bookkeeping and the paged context pool. The + draft attends the last trained window of captured target features plus + its whole non-causal query block. All parameters live in the core table. + """ + + _uses_worker_attention_backend = False + _paged_ctx_cache = True + _kv_factor = 1 + _dspark_shift_label = True + has_markov_head = True + has_recurrent_markov_head = True + + def __init__(self, shell, draft_config): + super().__init__() + # Keep references without registering the parent as a child module. + object.__setattr__(self, "_shell", shell) + self.model_config = draft_config + self.config = draft_config.pretrained_config + core = shell.model + spec = shell.spec_config + native = spec._draft_model_config + self.block_size = spec.block_size + self.mask_token_id = spec.mask_token_id + self.target_layer_ids = tuple(spec.target_layer_ids) + self.window = int(native["ragged_attention"].split(",")[0]) + self._num_attn_layers = int(native["n_layers"]) + self._num_heads = core.heads + self._num_kv_heads = 1 + self._head_dim = core.lat_dim + self.dflash_attention_backend = "VANILLA" + self._runtime_position_ceiling = None + self._rope = None + self._layers = None + + @property + def core(self): + return self._shell.model + + @property + def lm_head(self): + return self._shell.lm_head + + @property + def markov_w2(self): + return self.core.w["draft_markov_output"] + + def logits_processor(self, hidden, lm_head, attn_metadata, gather): + logits = self._shell.logits_processor(hidden, lm_head, attn_metadata, gather) + # The shared-head logits are soft-capped before the RNN bias is added. + return tanh_softcap(logits, self._shell.softcap) + + def apply_draft_logit_transform(self, logits): + """Return ``logits`` unchanged: :meth:`logits_processor` already soft-caps them. + + The worker calls this on the block logits after ``logits_processor``. + This drafter declares no ``logit_multiplier`` or ``logit_softcap``, so the + cap is applied once, in ``logits_processor``. + """ + return logits + + def _build_fused_kv_buffers(self): + # The worker calls this before sizing its cache. Take views once after + # materialization; context projection is explicit per layer here. + if self._layers is not None: + return + w, core = self.core.w, self.core + self._layers = [] + for i in range(self._num_attn_layers): + prefix = f"l{i}_draft_" + kv_b = w[prefix + "wkv_b"].view(core.heads, core.nope + core.v_dim, core.kv_lora) + self._layers.append( + SimpleNamespace( + attn_norm=w[prefix + "attn_norm"], + q=w[prefix + "wq"].t(), + kv_a=w[prefix + "wkv_a"].t(), + kv_norm=w[prefix + "kv_a_norm"], + k_b=kv_b[:, : core.nope].contiguous(), + v_b=kv_b[:, core.nope :].contiguous(), + out=w[prefix + "wo"].t(), + ffn_norm=w[prefix + "ffn_norm"], + gate=w[prefix + "w1"].t(), + up=w[prefix + "w3"].t(), + down=w[prefix + "w2"].t(), + next_norm=( + w[f"l{i + 1}_draft_attn_norm"] + if i + 1 < self._num_attn_layers + else w["draft_final_norm"] + ), + ) + ) + + def _ensure_rope(self): + cap = self._runtime_position_ceiling or self.core.rope_rows + if self._rope is None or self._rope.rows != cap: + w = self.core.w["draft_final_norm"] + # alpha=0 removes LeRoPE; all draft layers use plain YaRN. + self._rope = _rope_tables( + self.core.model_config.pretrained_config, + torch.zeros(self.core.rope_dim // 2), + cap, + w.dtype, + w.device, + ) + + def _rotate(self, x, positions): + table = self._rope.indexer[positions.long()] + cos, sin = table.chunk(2, dim=-1) + # One head axis for Q, none for the shared latent K. + if x.ndim == positions.ndim + 2: + cos, sin = cos.unsqueeze(-2), sin.unsqueeze(-2) + even, odd = x.float()[..., 0::2], x.float()[..., 1::2] + return ( + torch.stack((even * cos - odd * sin, even * sin + odd * cos), -1) + .flatten(-2) + .to(x.dtype) + ) + + def embed_draft_block(self, anchors, block_size): + # The route pins the trained block width; no mask token is looked up + # in the vocabulary (its sentinel is deliberately out of range). + weight = self.core.w["embed"] + valid = (anchors >= 0) & (anchors < weight.shape[0]) + embedded = nn.functional.embedding(anchors.clamp(0, weight.shape[0] - 1), weight) + embedded = embedded.masked_fill(~valid.unsqueeze(-1), 0) + masks = self.core.w["draft_mask"].unsqueeze(0).expand(anchors.shape[0], -1, -1) + return torch.cat((embedded.unsqueeze(1), masks), dim=1) + + def project_target_hidden(self, hidden): + w = self.core.w + projected = cublas_mm(hidden, w["draft_target_projection"].t(), None, None) + return flashinfer_rmsnorm(projected, w["draft_target_norm"], self.core.eps) + + def precompute_context_kv(self, projected_hidden, positions): + self._build_fused_kv_buffers() + self._ensure_rope() + core = self.core + latent = [] + for layer in self._layers: + kv = cublas_mm(projected_hidden, layer.kv_a, None, None) + content = flashinfer_rmsnorm( + kv[:, : core.kv_lora].contiguous(), layer.kv_norm, core.eps + ) + rope = self._rotate(kv[:, core.kv_lora :], positions.reshape(-1)) + latent.append(torch.cat((content, rope), dim=-1)) + return torch.stack(latent, dim=1).unsqueeze(2), None + + def dflash_forward_with_metadata(self, *, attn_metadata, **kwargs): + # The phase belongs to this engine step, including mixed batches. + return self.core._select_target(attn_metadata).dspark_forward(self, **kwargs) + + def apply_markov_chain_logits(self, base_logits, first_prev_tokens, *, hidden_states): + """The recurrent Markov bias over full-vocab ``[B, K, vocab]`` block logits. + + Each position's greedy token (the argmax of its biased logits) is the + next position's previous token; ``hidden_states`` is the ``[B, K, + hidden]`` block output the logits were projected from. + """ + w = self.core.w + output = w["draft_markov_output"] + previous = first_prev_tokens.long() + state = None + refined = [] + for i in range(base_logits.shape[1]): + weight = w["draft_markov_embedding"] + valid = (previous >= 0) & (previous < weight.shape[0]) + embedding = nn.functional.embedding(previous.clamp(0, weight.shape[0] - 1), weight) + embedding = embedding.masked_fill(~valid.unsqueeze(-1), 0) + if state is None: + state = torch.zeros_like(embedding) + joint = cublas_mm( + torch.cat((state, embedding, hidden_states[:, i]), dim=-1), + w["draft_joint"].t(), + w["draft_joint_bias"], + None, + ) + gate, candidate, emitted = joint.chunk(3, dim=-1) + gate = torch.sigmoid(gate) + state = gate * state + (1 - gate) * torch.tanh(candidate) + bias = cublas_mm(torch.tanh(emitted), output.t(), None, None) + logits = base_logits[:, i] + bias + refined.append(logits) + previous = logits.argmax(dim=-1) + return torch.stack(refined, dim=1) + + +class _VisionBlock(NamedTuple): + """One vision transformer block's weights, as the views the ops read: the GEMM + operands column-major (`.t()` of the stored rows).""" + + attn_norm: torch.Tensor + qkv: torch.Tensor + wo: torch.Tensor + ffn_norm: torch.Tensor + w13: torch.Tensor + w2: torch.Tensor + + +def _vision_step(seg_lens: list[int], device, workspace: torch.Tensor) -> dict: + """The batch state of one vision attention call: one context sequence per image, + no KV cache. Every value is host-known from the images' patch counts.""" + total = sum(seg_lens) + longest = max(seg_lens) + lens_dev = torch.tensor(seg_lens, dtype=torch.int32, device=device) + lens_host = torch.tensor(seg_lens, dtype=torch.int32) + return dict( + workspace=workspace, + sequence_length=lens_dev, + host_past_key_value_lengths=lens_host, + host_total_kv_lens=torch.tensor([total, 0], dtype=torch.int32), + context_lengths=lens_dev, + host_context_lengths=lens_host, + host_request_types=torch.zeros(len(seg_lens), dtype=torch.int32), + max_num_requests=len(seg_lens), + max_context_length=longest, + max_seq_len=longest, + attention_window_size=longest, + num_contexts=len(seg_lens), + num_ctx_tokens=total, + ) + + +def _vision_attention( + x: torch.Tensor, + B: _VisionBlock, + positions: torch.Tensor, + step: dict, + heads: int, + head_dim: int, + theta: float, +) -> torch.Tensor: + """One vision block's attention over the patch rows of every image of the call: + `[wq ; wk ; wv]` as one GEMM, q and k rotated by each patch's (row, col) in its + own image, attention bidirectional within each image, then `wo`.""" + rows = x.shape[0] + width = heads * head_dim + qkv = cublas_mm(x, B.qkv) + q, k = pixtral_rope_2d( + torch.reshape(qkv[:, :width], [rows, heads, head_dim]), + torch.reshape(qkv[:, width : 2 * width], [rows, heads, head_dim]), + positions, + theta, + ) + qkv = torch.cat( + [torch.reshape(q, [rows, width]), torch.reshape(k, [rows, width]), qkv[:, 2 * width :]], -1 + ) + out = torch.empty([rows, width], dtype=x.dtype, device=x.device) + out = thop_attention( + q=qkv, + output=out, + num_heads=heads, + num_kv_heads=heads, + head_size=head_dim, + **step, + **_VISION_ATTN, + ) + return cublas_mm(out, B.wo) + + +def _merge_tiles(x: torch.Tensor, grid_h: int, grid_w: int, merge: int) -> torch.Tensor: + """One image's `[grid_h * grid_w, d]` patch rows -> `[grid_h * grid_w / merge^2, + d * merge^2]`: each merge x merge tile of the grid as one row, tiles row-major (the + order of the image's [IMG] tokens), each row laid out (channel, tile row, tile + column) as the merging layer's input columns are.""" + d = x.shape[-1] + grid = torch.permute(torch.reshape(x, [grid_h, grid_w, d]), [2, 0, 1]) + tiles = nn.functional.unfold(torch.unsqueeze(grid, 0), kernel_size=merge, stride=merge) + return torch.t(tiles[0]) + + +class VisionTower: + """The image input path: a request's pixels -> one row per [IMG] token, in the + language model's hidden width. Run by the shell, in context steps only, never + captured; every rank runs the whole tower (all heads) over its own requests. + + Per image: the 14x14 patch embedding as an unfold and one GEMM, `ln_pre`, the + transformer blocks (RMSNorm, attention with 2-D RoPE, RMSNorm, SwiGLU, each with + its residual), the pre-projector RMSNorm, the 2x2 patch merger and the adapter + (`w_in`, exact GELU, `w_out`). The images of one call are concatenated along the + rows; only the attention needs their boundaries.""" + + def __init__(self, core) -> None: + """Take the tower's weight views and geometry, once, off the real weights.""" + d = _weights.MODEL_WEIGHTS.dims(core) + w = core.w + self.hidden = d.v_hidden + self.heads = d.v_heads + self.head_dim = d.v_head_dim + self.patch = d.v_patch + self.merge = d.v_merge + self.theta = d.v_theta + self.dtype = d.dtype + self.out_hidden = d.hidden + self._validate = step_contract_enabled() + self._rank = core.model_config.mapping.rank + self.patch_w = w["vis_patch"].t() + self.ln_pre = w["vis_ln_pre"] + self.blocks = [ + _VisionBlock( + attn_norm=w[f"v{j}_attn_norm"], + qkv=w[f"v{j}_qkv"].t(), + wo=w[f"v{j}_wo"].t(), + ffn_norm=w[f"v{j}_ffn_norm"], + w13=w[f"v{j}_w13"].t(), + w2=w[f"v{j}_w2"].t(), + ) + for j in range(d.v_layers) + ] + self.proj_norm = w["vis_proj_norm"] + self.merge_w = w["vis_merge"].t() + self.w_in = w["vis_w_in"].t() + self.w_out = w["vis_w_out"].t() + # The prompt token the tower's rows replace, on the host: the executor finds a + # step's image positions with it before anything reaches the device. + self.image_token_ids = torch.tensor([d.image_token_id], dtype=torch.int32) + # The attention op sizes its workspace in place per call and never shrinks it. + self._workspace = torch.empty(0, dtype=torch.int8, device=self.ln_pre.device) + + def _images(self, multimodal_params: list[MultimodalParams]) -> list[tuple]: + """`(pixels [C, H, W], grid rows, grid cols)` of every image, in order. The + mistral-native processor pads the batch and preserves each image's true size.""" + images = [] + unit = self.patch * self.merge + for param in multimodal_params: + image = (param.multimodal_data or {}).get("image") + if image is None: + continue + pixels = image["pixel_values"] + sizes = image.get("image_sizes") + if pixels.ndim != 4 or pixels.shape[1] != 3: + raise ValueError("Image pixels must have shape [N, 3, H, W]") + if sizes is None or len(sizes) != pixels.shape[0]: + raise ValueError("Image sizes must describe every image in the padded batch") + for b in range(pixels.shape[0]): + if len(sizes[b]) != 2: + raise ValueError("Each image size must be [height, width]") + height, width = int(sizes[b][0]), int(sizes[b][1]) + if not (0 < height <= pixels.shape[-2] and 0 < width <= pixels.shape[-1]): + raise ValueError("Image sizes must fit within their padded pixels") + if height % unit or width % unit: + raise ValueError( + f"image {height}x{width} is not a whole number of {unit}-pixel " + "merge tiles; the processor resizes to them" + ) + images.append( + (pixels[b, :, :height, :width], height // self.patch, width // self.patch) + ) + return images + + def forward(self, multimodal_params: list[MultimodalParams]) -> torch.Tensor: + """The rows of every image of `multimodal_params`, request by request, image by + image: `[sum of merged tiles, text hidden]`.""" + images = self._images(multimodal_params) + dev = self.ln_pre.device + if not images: + return torch.empty([0, self.out_hidden], dtype=self.dtype, device=dev) + seg = [gh * gw for _, gh, gw in images] + if self._validate: + logger.info( + "modeling_v2 image forward target=%s rank=%d grids=%s merged_rows=%d", + __name__.rsplit(".", 1)[0], + self._rank, + [(gh, gw) for _, gh, gw in images], + sum(seg) // self.merge**2, + ) + # Each patch as one row, (channel, row, col) within it, patches row-major. + patches = torch.cat( + [ + torch.t( + nn.functional.unfold( + torch.unsqueeze(px.to(device=dev, dtype=self.dtype), 0), + kernel_size=self.patch, + stride=self.patch, + )[0] + ) + for px, _, _ in images + ], + 0, + ) + positions = torch.cat( + [ + torch.reshape( + torch.stack( + torch.meshgrid( + torch.arange(gh, device=dev), + torch.arange(gw, device=dev), + indexing="ij", + ), + -1, + ), + [-1, 2], + ) + for _, gh, gw in images + ], + 0, + ) + step = _vision_step(seg, dev, self._workspace) + + h = flashinfer_rmsnorm(cublas_mm(patches, self.patch_w), self.ln_pre, _VISION_EPS) + x = flashinfer_rmsnorm(h, self.blocks[0].attn_norm, _VISION_EPS) + residual = h + for j, B in enumerate(self.blocks): + attn = _vision_attention(x, B, positions, step, self.heads, self.head_dim, self.theta) + x, residual = flashinfer_fused_add_rmsnorm(attn, residual, B.ffn_norm, _VISION_EPS) + mlp = cublas_mm(flashinfer_silu_and_mul(cublas_mm(x, B.w13)), B.w2) + norm_next = self.blocks[j + 1].attn_norm if j + 1 < len(self.blocks) else self.proj_norm + x, residual = flashinfer_fused_add_rmsnorm(mlp, residual, norm_next, _VISION_EPS) + + # x is the pre-projector-normed tower output. + tiles = torch.cat( + [ + _merge_tiles(xi, gh, gw, self.merge) + for xi, (_, gh, gw) in zip(torch.split(x, seg, 0), images) + ], + 0, + ) + merged = cublas_mm(tiles, self.merge_w) + return cublas_mm(gelu_erf(cublas_mm(merged, self.w_in)), self.w_out) + + +def _image_rows( + vision: Optional[VisionTower], multimodal_params: list[MultimodalParams] +) -> Optional[torch.Tensor]: + """This step's image rows, one per [IMG] position its context chunks cover, in + the order those positions appear; None when the step covers none. + + The tower runs once per request, on the first step that needs it; its rows stay + on the request's multimodal data for the later chunks. A chunk -- under chunked + prefill, or resuming after a reused prefix -- takes rows `[cached, cached + in + chunk)` of its request's, counted by the request's [IMG] cumsum.""" + if vision is None: + if any((p.multimodal_data or {}).get("image") is not None for p in multimodal_params): + raise ValueError( + "an image reached a Mistral Large 4 target whose checkpoint has no vision encoder" + ) + return None + # Filter before encoding and gathering so inactive requests neither run the + # tower nor shift the embedding offsets of active requests in a mixed batch. + multimodal_params = [ + param + for param in multimodal_params + if param.multimodal_runtime is None or param.multimodal_runtime.num_mm_tokens_in_chunk > 0 + ] + embeds = find_input_mm_embeds( + get_multimodal_embeddings(vision.forward, multimodal_params), multimodal_params + ) + if not embeds: + return None + rows = embeds[0] if len(embeds) == 1 else torch.cat(embeds, 0) + return rows if rows.shape[0] else None + + +def _splice_image_rows( + embed: torch.Tensor, + input_ids: torch.Tensor, + rows: torch.Tensor, + mm_token_indices: Optional[torch.Tensor], +) -> torch.Tensor: + """The step's input embeddings with `rows` written over its [IMG] positions, + `mm_token_indices` (which the executor computes on the host from the shell's + `multimodal_token_ids`); every other position keeps its token embedding.""" + if mm_token_indices is None: + raise ValueError("an image step arrived without the executor's mm_token_indices") + if mm_token_indices.shape[0] != rows.shape[0]: + raise ValueError( + f"{mm_token_indices.shape[0]} [IMG] positions in this step but " + f"{rows.shape[0]} image rows for them" + ) + h = nn.functional.embedding(input_ids, embed) + return h.index_copy_(0, mm_token_indices, rows.to(h.dtype)) + + +class ModelingV2MistralLarge4Nvfp4Sm107Tep( + DecoderModelForCausalLM[MistralLarge4TepModelingV2, PretrainedConfig] +): + def __init__(self, model_config: ModelConfig): + cfg = model_config.pretrained_config + super().__init__( + MistralLarge4TepModelingV2(model_config), + config=model_config, + hidden_size=cfg.hidden_size, + vocab_size=cfg.vocab_size, + ) + self.softcap = float(cfg.softmax_tanh_gating) + self._vision: Optional[VisionTower] = None + vis = getattr(cfg, "vision_config", None) + self._image_token_ids = ( + torch.tensor([vis.image_token_id], dtype=torch.int32) if vis is not None else None + ) + + self.spec_config = model_config.spec_config + self.draft_model = None + self.draft_config = None + self.spec_worker = None + if self.spec_config is not None: + # This is allocation geometry for the independently paged BF16 + # draft, not a second registry-resolved model or target cache. + draft_cfg = PretrainedConfig( + num_hidden_layers=int(self.spec_config._draft_model_config["n_layers"]), + hidden_size=cfg.hidden_size, + num_attention_heads=cfg.num_attention_heads, + num_key_value_heads=1, + kv_lora_rank=cfg.kv_lora_rank, + qk_rope_head_dim=cfg.qk_rope_head_dim, + qk_nope_head_dim=cfg.qk_nope_head_dim, + v_head_dim=cfg.v_head_dim, + intermediate_size=cfg.intermediate_size, + vocab_size=cfg.vocab_size, + max_position_embeddings=cfg.max_position_embeddings, + torch_dtype=torch.bfloat16, + ) + self.draft_config = replace( + model_config, + pretrained_config=draft_cfg, + quant_config=QuantConfig(), + quant_config_dict=None, + sparse_attention_config=None, + spec_config=None, + modeling_v2_target=None, + ) + self.draft_model = DraftModel(self, self.draft_config) + self.spec_worker = get_spec_worker( + self.spec_config, + model_config, + model_config.mapping, + use_separate_draft_kv_cache=True, + ) + self.spec_worker.set_draft_model(self.draft_model) + self.epilogue.append(self.spec_worker) + + def set_guided_decoder(self, guided_decoder: CapturableGuidedDecoder) -> bool: + if self.spec_worker is None: + return False + return self.spec_worker.set_guided_decoder(guided_decoder) + + @property + def multimodal_token_ids(self) -> Optional[torch.Tensor]: + return self._image_token_ids + + def forward( + self, + attn_metadata: AttentionMetadata, + input_ids: torch.IntTensor | None = None, + position_ids: torch.IntTensor | None = None, + inputs_embeds: torch.FloatTensor | None = None, + return_context_logits: bool = False, + spec_metadata=None, + lora_params: dict | None = None, + **kwargs, + ) -> torch.Tensor: + """Soft-cap target logits before verification and return the worker result.""" + worker_input_ids = input_ids + multimodal_params = kwargs.get("multimodal_params") or [] + if multimodal_params: + rows = _image_rows(self._vision, multimodal_params) + if rows is not None: + inputs_embeds = _splice_image_rows( + self.model.w["embed"], input_ids, rows, kwargs.get("mm_token_indices") + ) + input_ids = None + if self.spec_worker is None: + logits = super().forward( + attn_metadata, + cast(torch.IntTensor, input_ids), + position_ids, + inputs_embeds, + return_context_logits, + spec_metadata, + lora_params, + **kwargs, + ) + return tanh_softcap(logits, self.softcap) + hidden = self.model( + attn_metadata=attn_metadata, + input_ids=input_ids, + position_ids=position_ids, + inputs_embeds=inputs_embeds, + spec_metadata=spec_metadata, + lora_params=lora_params, + **kwargs, + ) + logits = self.logits_processor( + nn.functional.embedding(spec_metadata.gather_ids, hidden), + self.lm_head, + attn_metadata, + True, + ) + return self.spec_worker( + input_ids=worker_input_ids, + position_ids=position_ids, + hidden_states=hidden, + logits=tanh_softcap(logits, self.softcap), + attn_metadata=attn_metadata, + spec_metadata=spec_metadata, + draft_model=self.draft_model, + resource_manager=kwargs.get("resource_manager"), + ) + + def load_weights(self, weights, *args, **kwargs): + _weights.MODEL_WEIGHTS.load(self, weights) + + def load_checkpoint(self, checkpoint_dir: str) -> None: + _weights.MODEL_WEIGHTS.load_checkpoint(self, checkpoint_dir) + + def post_load_weights(self): + super().post_load_weights() + self.model.post_load_weights() + if step_contract_enabled(): + logger.info( + "modeling_v2 loaded target=%s class=%s rank=%d world=%d vision=%s", + __name__.rsplit(".", 1)[0], + type(self).__name__, + self.model_config.mapping.rank, + self.model_config.mapping.world_size, + getattr(self.config, "vision_config", None) is not None, + ) + if getattr(self.config, "vision_config", None) is not None: + self._vision = VisionTower(self.model) + if self.draft_model is not None: + self.draft_model._build_fused_kv_buffers() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tep/weights.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tep/weights.py new file mode 100644 index 000000000000..36c2c5aa709f --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tep/weights.py @@ -0,0 +1,1338 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Weight table and loader: mistral-large-4-nvfp4 / sm_107 / tep, the language model. + +The checkpoint is Mistral-native (`consolidated-*.safetensors`, `layers.N.attention.wq`, ...) +and ModelOpt `MIXED_PRECISION`: the routed experts NVFP4, the other big linears FP8 128x128 +block (`FP8_PB_WO`), everything else bf16. Every rank is handed the whole of it, so every split +is this table's, every width read off the mapping (`tp_size` and `moe_ep_size` 4 on one node, or +8 over two nodes of four): + +* **tensor-parallel slices** -- `wq` by output rows and `wo` by input columns (this rank's 16 + heads at 4 ranks, 8 heads -- 1024 rows or columns -- at 8); `wkv_b` by heads; the MLA SSMax + `s`/`b` by heads; the dense layer's and the shared expert's `w1`/`w3` by rows and `w2` by + columns (6144 and 1536 per rank at 4 ranks, 3072 and 768 at 8); the shell's `lm_head` by + vocabulary rows (51200 or 25600 per rank). Every slice is whole 128-blocks, and a block scale + is sliced on the same boundary. The slice is taken on the checkpoint entry before the copy to + the device, so a rank moves only its own bytes. That is the `shard` column of `W`. +* **expert windows** -- rank `r` loads routed experts `[Er, Er + E)`, `E = n_routed_experts / + moe_ep_size`: 32 at 4 ranks, 16 at 8. +* **replicated** -- `wkv_a_with_mqa` and `kv_a_norm`, the whole indexer (all 32 of its heads), + the router, the embedding, every norm and `lerope.alpha`. + +Layer sets come from the config: `wkv_a_with_mqa` and `kv_a_norm` exist on the KV-owning layers +only (`kv_source_layers[i] == i`; the sharers read their source's latent), the indexer on the +`index_sharing == 'full'` layers only, the dense MLP on the first `first_k_dense_replace` +layers and the router, shared expert and routed experts on the rest. + +Load-time layouts, each the one its consumer reads: + +* `wkv_b` regrouped per head `[k_nope | v]` -> `[all k_nope ; all v]` over this rank's heads + (k_nope is 64 wide on RoPE layers and 128 on NoPE layers); +* the dense MLP and the shared expert `[w1 ; w3]`, gate rows first, the half order + `flashinfer_silu_and_mul` reads; +* FP8 linears (`wq`, `wo`, the indexer's `wq_b`, the dense and shared `w1`/`w3`/`w2`): e4m3 + `[N, K]` as stored, and the fp32 block scales -- stored as `.weight_scale` + `[N/128, 1, K/128, 1]`, squeezed here to the `[N/128, K/128]` grid -- in DeepGEMM's packed + UE8M0 operand, `transform_sf_into_required_layout(recipe=(1, 128, 128))`: what + `FP8BlockScalesLinearMethod.transform_weights` installs as `weight_scale` on sm_100f. That + operand is an MN-major `[N, ceil(K/512)]` int32 view; the parameter holds the dense + `[ceil(K/512), N]` behind it. The layer views convert `.t()` to Rubin R128c4 K32 scales once, + replicating each K128 exponent four times. Where K/128 is not a multiple of four + the last word of each row is partly padding: the shared expert's `w2` at K 768 (8 ranks) has 6 + blocks in 2 words, and bytes 2 and 3 of the second word are zero. The method's + `resmooth_to_fp8_e8m0` is not run: every scale is asserted an exact power of two, which makes + it a power-of-two re-exponent of the same values; +* the indexer's `wk` and `weights_proj` stacked into one `[128 + 32, hidden]` bf16 GEMM operand; + its `k_norm` and `lerope.alpha` widened to fp32. + +The routed experts are this rank's window stacked on an expert axis, in the NVFP4 MoE runner's +layout: e2m1 codes, two per byte, `[E, 2I, H/2]` / `[E, H, I/2]` uint8 and their e4m3 block +scales, one per 16 inputs, `[E, 2I, H/16]` / `[E, H, I/16]`. Each source lands as stored -- +FC1's rows `[w3 ; w1]`, up rows first, the order the runner's FC1 concatenates them in -- and +once every source of an expert stack has landed, its row's `layout` rearranges each expert in +place (the runner reads nothing else, and a wrong layout costs 300-3400 bf16 ulp and raises +nothing): + +* FC1, codes and block scales alike: the halves interleaved (row `2i` up row `i`, row `2i + 1` + gate row `i`), then the 32-row block shuffle (within each 32 rows, row `4u + v` moves to + `8v + u`); +* FC2: the 32-row block shuffle; +* both block scales: then the 128x4 swizzle (`torch.ops.trtllm.block_scale_interleave`'s order, + written here in torch so the table also runs on meta tensors). + +The runner's output scalars are not derived here: the four per-tensor scalars are handed over +as stored -- `isc1` and `isc2`, the one FC1 and FC2 activation `input_scale` of the layer, and +per local expert `ws2_13` (`w1`'s `weight_scale_2`, which `w3` shares) and `ws2_2`. ModelOpt +stores these as reciprocals of the global scales a quantizer divides by. + +Before any of that is read, `load` reads the per-tensor scalars of all the layer's experts, +the other ranks' included, and asserts what the forward relies on: `w1` and `w3` store the same +`input_scale` and `weight_scale_2` on every expert, so one FC1 alpha serves the stacked +`[w3 ; w1]`; and each projection stores one `input_scale` for every expert, so one NVFP4 +quantization of the replicated activations is right for every expert on every rank and the +windows' partial outputs sum. + +The BF16 vision encoder, pre-projector norm, patch merger and language adapter are +replicated on every rank when the configuration declares a vision encoder. + +`expected_unconsumed` accounts for off-rank routed experts and disabled DSpark draft weights. +Off-rank expert scalars are still read to check shared scales; their codes and block scales +are never read. +At `moe_ep_size` 1 every expert belongs to the rank and is read, sliced. +""" + +from __future__ import annotations + +import collections +import functools +import json +import os +from dataclasses import dataclass +from pathlib import Path +from types import SimpleNamespace +from typing import Any, Callable + +import torch +import torch.nn as nn +from safetensors import safe_open + +from tensorrt_llm.logger import logger +from tensorrt_llm.quantization.utils.fp8_utils import transform_sf_into_required_layout + +F8, F32, I32, U8 = torch.float8_e4m3fn, torch.float32, torch.int32, torch.uint8 + +#: The FP8 block edge on both weight axes (the checkpoint's `FP8_PB_WO` 128x128 blocks), and +#: the K granularity of the DeepGEMM scale recipe `(1, 128, 128)`. +BLOCK = 128 +#: UE8M0 bytes per int32 word of the DeepGEMM scale operand: four K blocks. +SF_PACK = 4 +#: NVFP4: one e4m3 block scale per 16 inputs (the checkpoint's `group_size`), two e2m1 codes +#: per byte. +NVFP4_BLOCK = 16 +E2M1_PER_BYTE = 2 + +#: A routed expert's three projections: `w1` gate, `w3` up, `w2` down. +PROJECTIONS = ("w1", "w2", "w3") +#: The per-tensor NVFP4 scalars each of them carries beside its codes and block scales. +EXPERT_SCALARS = ("weight_scale_2", "input_scale") + +#: The vision tower's checkpoint prefixes. +VISION_PREFIXES = ( + "vision_encoder.", + "patch_merger.", + "pre_mm_projector_norm.", + "vision_language_adapter.", +) + + +#: `layers=ALL_LAYERS` is the common case and the default. A sentinel rather +#: than a lambda so the table reads as data, and rather than `True` so the +#: field has one type of answer: which layers. +ALL_LAYERS = object() + + +@dataclass(frozen=True) +class W: + """One weight role: what it is, where its bytes come from, what happens on the way. + + `shape` takes the dimension bundle and returns the declared shape. + + `src` says where the bytes come from: a checkpoint key template with `{p}` + for the layer prefix (`layers.N`), or a callable `(d, i) -> [(key, index), ...]` + where `index` is where in the parameter that tensor lands (`None`: all of it). + A `key` may be a tuple of per-tensor scalars, which `transform` is handed together. + + `layers` says which layers carry this weight: `ALL_LAYERS`, `None` for a + weight that is not per-layer, or a callable `d -> iterable[int]`. Two rows + may share a name over disjoint layer sets, with different shapes. + + `shard` takes the dimension bundle and returns a function of the checkpoint + entry: this rank's tensor-parallel slice, applied to every source of the row. + It runs before the copy to the device and before `transform`, which is what + `wkv_b` relies on: the regroup reads this rank's head count off the slice. + + `transform` runs on the (sliced) source tensor, on the destination's device. + + `layout` runs once every source of the parameter has landed, on each slot of its leading + (expert) axis, and the slot is overwritten with the result: a rearrangement that mixes rows + several sources fill, which no single source's `transform` can make. + """ + + name: str + shape: Callable[[Any], tuple[int, ...]] + src: str | Callable + dtype: torch.dtype | None = None + transform: Callable | None = None + layers: object | None = ALL_LAYERS + shard: Callable | None = None + layout: Callable | None = None + when: Callable | None = None + tag: str = "l" + + def layer_indices(self, d: SimpleNamespace) -> list[int] | None: + """The layers this weight exists on, or None if it is not per-layer.""" + if self.layers is None: + return None + if self.layers is ALL_LAYERS: + return list(range(d.num_layers)) + return list(self.layers(d)) + + def sources(self, d: SimpleNamespace, i: int | None, prefix: str) -> list[tuple[Any, Any]]: + """`[(checkpoint key | key tuple, index into the parameter | None), ...]`.""" + if callable(self.src): + return list(self.src(d, i)) + return [(self.src.format(p=prefix), None)] + + +def _keys(ckpt) -> tuple[str, ...]: + """The checkpoint keys of one source: itself, or the members of a key tuple.""" + return ckpt if isinstance(ckpt, tuple) else (ckpt,) + + +def _rows(start, width): + return lambda d: (lambda e: e[start(d) : start(d) + width(d)]) + + +def _cols(start, width): + return lambda d: (lambda e: e[:, start(d) : start(d) + width(d)]) + + +def _sf_cols(start, width): + """`_cols` on an FP8 block scale as stored, `[N/128, 1, K/128, 1]`: its K blocks are axis 2.""" + return lambda d: (lambda e: e[:, :, start(d) : start(d) + width(d)]) + + +def _sf_words(k: int) -> int: + """int32 words per row of the DeepGEMM scale operand of a K-wide weight.""" + return -(-k // (BLOCK * SF_PACK)) + + +def _assert_ue8m0(s: torch.Tensor) -> None: + """Every block scale is a positive normal power of two, so its UE8M0 byte (the fp32 + exponent field) holds it exactly. A meta tensor carries no values to check.""" + if s.is_meta: + return + bits = s.view(torch.int32) + exp = (bits >> 23) & 0xFF + exact = ((bits & 0x807FFFFF) == 0) & (exp > 0) & (exp < 255) + assert bool(exact.all()), ( + f"weight_scale: {int((~exact).sum())} of {s.numel()} block scales are not a positive " + "normal power of two (UE8M0); the scale layouts below would change their values" + ) + + +def _deepgemm_sf(s: torch.Tensor, core) -> torch.Tensor: + """fp32 `[N/128, K/128]` block scales -> the dense `[ceil(K/512), N]` int32 storage of + DeepGEMM's MN-major packed UE8M0 operand, whose `.t()` is that operand. N is a multiple of + 128 for every role here, so the operand's row stride is N and its storage has no MN padding. + A K of a block count that is not a multiple of four (the shared expert's `w2` at K 768 on 8 + ranks: 6 blocks) leaves the high bytes of each row's last word zero.""" + _assert_ue8m0(s) + n_blk, k_blk = s.shape + sf = transform_sf_into_required_layout( + s, mn=n_blk * BLOCK, k=k_blk * BLOCK, recipe=(1, BLOCK, BLOCK), is_sfa=False + ) + return sf.t() + + +def _block_sf(s: torch.Tensor, core) -> torch.Tensor: + """An FP8 linear's block scales as stored, fp32 `[N/128, 1, K/128, 1]`: squeezed to the + `[N/128, K/128]` grid, then `_deepgemm_sf`.""" + assert s.dim() == 4 and s.shape[1] == 1 and s.shape[3] == 1, ( + f"FP8 block scale stored as {tuple(s.shape)}, not [N/128, 1, K/128, 1]" + ) + return _deepgemm_sf(s.reshape(s.shape[0], s.shape[2]), core) + + +def _shared_scalar(t: tuple[torch.Tensor, ...], core) -> torch.Tensor: + """One per-tensor scalar the checkpoint stores several times (`w1`'s and `w3`'s, or every + expert's of the window): asserted the same value everywhere, then that value.""" + first = t[0] + if not first.is_meta: + values = torch.stack(t) + assert bool((values == first).all()), ( + f"{int((values != first).sum())} of {len(t)} copies of one shared scale differ from " + f"the first, {first.item()!r}" + ) + return first + + +def _regroup_kv_b(t: torch.Tensor, core) -> torch.Tensor: + """This rank's `[heads * (k_nope + v), kv_lora]` rows, per-head `[k_nope | v]` -> + `[heads * k_nope ; heads * v]`: the K-B operand, then the V-B one. k_nope is read off the + slice: 64 on RoPE layers, 128 on NoPE layers.""" + cfg = core.model_config.pretrained_config + heads = cfg.num_attention_heads // core.model_config.mapping.tp_size + v = cfg.v_head_dim + t3 = t.reshape(heads, t.shape[0] // heads, t.shape[1]) + nope = t3.shape[1] - v + return torch.cat( + [t3[:, :nope, :].reshape(heads * nope, -1), t3[:, nope:, :].reshape(heads * v, -1)], dim=0 + ).contiguous() + + +def _to_fp32(t: torch.Tensor, core) -> torch.Tensor: + return t.to(torch.float32) + + +# --- the NVFP4 MoE runner's expert layout, one expert at a time --- + +#: The runner's row shuffle works on blocks of this many rows. +SHUFFLE_ROWS = 32 +#: The block-scale swizzle's tile: 128 rows by 4 scale columns. +SWIZZLE_ROWS, SWIZZLE_COLS = 128, 4 + + +@functools.lru_cache(maxsize=None) +def _block32_perm(rows: int, device) -> torch.Tensor: + """Gather index of the 32-row block shuffle: within each block of 32 rows, source row + `4u + v` lands at row `8v + u`.""" + assert rows % SHUFFLE_ROWS == 0, rows + j = torch.arange(SHUFFLE_ROWS) + one = torch.empty(SHUFFLE_ROWS, dtype=torch.long) + one[(j % 4) * 8 + j // 4] = j + blocks = torch.arange(rows // SHUFFLE_ROWS).repeat_interleave(SHUFFLE_ROWS) * SHUFFLE_ROWS + return (one.repeat(rows // SHUFFLE_ROWS) + blocks).to(device) + + +@functools.lru_cache(maxsize=None) +def _fc1_perm(rows: int, device) -> torch.Tensor: + """Gather index of FC1's `[up ; gate]` rows: interleaved (row `2i` up row `i`, row `2i + 1` + gate row `i`), then block-shuffled.""" + interleave = torch.empty(rows, dtype=torch.long) + interleave[0::2] = torch.arange(rows // 2) + interleave[1::2] = torch.arange(rows // 2, rows) + return interleave.to(device)[_block32_perm(rows, device)] + + +def _swizzle(s: torch.Tensor) -> torch.Tensor: + """One expert's `[M, C]` e4m3 block scales in the 128x4-swizzled byte order, kept `[M, C]`: + byte `(m, c)` lands at flat offset `(m // 128) * 128 * C + (c // 4) * 512 + (m % 32) * 16 + + ((m % 128) // 32) * 4 + c % 4`. Both of this target's scale stacks tile it exactly (M a + multiple of 128, C of 4), so there is no padding.""" + m, c = s.shape + assert m % SWIZZLE_ROWS == 0 and c % SWIZZLE_COLS == 0, (m, c) + tiles = s.view(torch.uint8).reshape(m // SWIZZLE_ROWS, 4, 32, c // SWIZZLE_COLS, SWIZZLE_COLS) + return tiles.permute(0, 3, 2, 1, 4).reshape(m, c).view(s.dtype) + + +def _fc1_codes(w: torch.Tensor) -> torch.Tensor: + return torch.index_select(w, 0, _fc1_perm(w.shape[0], w.device)) + + +def _fc1_scales(s: torch.Tensor) -> torch.Tensor: + rows = torch.index_select(s.view(torch.uint8), 0, _fc1_perm(s.shape[0], s.device)) + return _swizzle(rows.view(s.dtype)) + + +def _fc2_codes(w: torch.Tensor) -> torch.Tensor: + return torch.index_select(w, 0, _block32_perm(w.shape[0], w.device)) + + +def _fc2_scales(s: torch.Tensor) -> torch.Tensor: + rows = torch.index_select(s.view(torch.uint8), 0, _block32_perm(s.shape[0], s.device)) + return _swizzle(rows.view(s.dtype)) + + +def _materialize(entry, shard) -> torch.Tensor: + """This rank's bytes of one checkpoint entry: a tensor, or a lazy safetensors slice, which + reads only what the index selects. `[...]`, not `[:]`: the per-tensor scalars are 0-dim.""" + return entry[...] if shard is None else shard(entry) + + +#: The checkpoint's shard index, beside its `consolidated-*.safetensors`. +SHARD_INDEX = "consolidated.safetensors.index.json" + + +def _checkpoint_shards(checkpoint_dir) -> tuple[Path, dict[str, str]]: + """`(directory, key -> shard file)` of a checkpoint directory: its index when it ships + one, else the header of each `consolidated*.safetensors`.""" + directory = Path(checkpoint_dir) + index = directory / SHARD_INDEX + if index.is_file(): + return directory, json.loads(index.read_text())["weight_map"] + weight_map: dict[str, str] = {} + for path in sorted(directory.glob("consolidated*.safetensors")): + with safe_open(str(path), framework="pt", device="cpu") as handle: + weight_map.update(dict.fromkeys(handle.keys(), path.name)) + if not weight_map: + raise FileNotFoundError(f"no consolidated*.safetensors in {directory}") + return directory, weight_map + + +def _shard_files(core, weights) -> tuple[Path, dict[str, str]] | None: + """`(directory, key -> shard file)` of the checkpoint a dict of sources was read from, or + None when the sources are not the shard files of one (the directory the config records + lacks the index, or the index does not name every key `weights` holds).""" + cfg = getattr(core.model_config, "pretrained_config", None) + root = getattr(cfg, "_name_or_path", None) + if not root: + return None + index = Path(root) / SHARD_INDEX + if not index.is_file(): + return None + weight_map = json.loads(index.read_text())["weight_map"] + if not set(weights.keys()) <= set(weight_map): + return None + return Path(root), weight_map + + +def _drop_page_cache(path: Path) -> None: + """Advise the kernel to drop a consumed shard file's cached pages. Advice only: a + filesystem that ignores it costs memory pressure, not correctness.""" + try: + fd = os.open(path, os.O_RDONLY) + except OSError: + return + try: + os.posix_fadvise(fd, 0, 0, os.POSIX_FADV_DONTNEED) + except OSError: + pass + finally: + os.close(fd) + + +# --- which layers carry a weight --- + + +def _owners(d): + """Layers that own a latent KV. The others read `kv_source_layers[i]`'s and project none.""" + return [i for i in range(d.num_layers) if d.kv_source[i] == i] + + +def _rope_owners(d): + return [i for i in _owners(d) if not d.nope_layer[i]] + + +def _nope_owners(d): + return [i for i in _owners(d) if d.nope_layer[i]] + + +def _rope_layers(d): + return [i for i in range(d.num_layers) if not d.nope_layer[i]] + + +def _nope_layers(d): + return [i for i in range(d.num_layers) if d.nope_layer[i]] + + +def _indexer_layers(d): + """Layers that own an indexer. The 'shared' ones reuse the most recent one's selection.""" + return [i for i in range(d.num_layers) if d.full_indexer[i]] + + +def _dense_layers(d): + return range(d.dense_layers) + + +def _moe_layers(d): + return range(d.dense_layers, d.num_layers) + + +# --- where the bytes come from --- + + +def _stack(*parts, prefix="layers"): + """Several checkpoint tensors into one parameter: each `(key template, index(d))`, the + index being where in the parameter that tensor lands.""" + return lambda d, i: [(t.format(p=f"{prefix}.{i}"), index(d)) for t, index in parts] + + +def _expert_key(i: int, e: int, suffix: str) -> str: + return f"layers.{i}.experts.{e}.{suffix}" + + +def _expert_window(*parts): + """This rank's routed experts, local slot `s` <- global expert `expert_offset + s`: each + `(checkpoint suffix | suffix tuple, index(d) after the expert axis)`.""" + + def key(i, e, suffix): + if isinstance(suffix, tuple): + return tuple(_expert_key(i, e, s) for s in suffix) + return _expert_key(i, e, suffix) + + def build(d, i): + return [ + (key(i, d.expert_offset + slot, suffix), (slot, *index(d))) + for slot in range(d.local_experts) + for suffix, index in parts + ] + + return build + + +def _window_scalar(*suffixes): + """One per-tensor scalar every expert of the window stores: all of the window's copies, one + key tuple, into element 0.""" + return lambda d, i: [ + ( + tuple( + _expert_key(i, d.expert_offset + slot, s) + for slot in range(d.local_experts) + for s in suffixes + ), + (0,), + ) + ] + + +def _expert_scalar_keys(d) -> list[str]: + """Every routed expert's per-tensor scalars on every MoE layer, the other ranks' included: + what `load` reads, and checks, before anything else.""" + return [ + _expert_key(i, e, f"{proj}.{s}") + for i in _moe_layers(d) + for e in range(d.num_experts) + for proj in PROJECTIONS + for s in EXPERT_SCALARS + ] + + +def _check_expert_scalars(scalars: dict[str, torch.Tensor], d) -> None: + """Per MoE layer, over all of its experts: `w1` and `w3` store the same `input_scale` and + `weight_scale_2` on every expert; `w1` and `w2` each store one `input_scale` for every expert; + every scalar is a positive finite fp32. Meta tensors carry no values to check.""" + if scalars and next(iter(scalars.values())).is_meta: + return + experts = range(d.num_experts) + for i in _moe_layers(d): + v = { + (proj, s): torch.stack([scalars[_expert_key(i, e, f"{proj}.{s}")] for e in experts]) + for proj in PROJECTIONS + for s in EXPERT_SCALARS + } + for (proj, s), t in v.items(): + bad = (~(torch.isfinite(t) & (t > 0))).nonzero().flatten() + assert bad.numel() == 0, ( + f"layers.{i}: {proj}.{s} of expert {int(bad[0])} is {t[bad[0]].item()!r}, " + "not a positive finite scale" + ) + for s in EXPERT_SCALARS: + bad = (v["w1", s] != v["w3", s]).nonzero().flatten() + assert bad.numel() == 0, ( + f"layers.{i}: w1 and w3 {s} differ on {bad.numel()} of {len(experts)} experts, " + f"first expert {int(bad[0])}: {v['w1', s][bad[0]].item()!r} vs " + f"{v['w3', s][bad[0]].item()!r}" + ) + for proj in ("w1", "w2"): + isc = v[proj, "input_scale"] + bad = (isc != isc[0]).nonzero().flatten() + assert bad.numel() == 0, ( + f"layers.{i}: {proj}.input_scale is not one value over the experts: expert " + f"{int(bad[0])} stores {isc[bad[0]].item()!r}, expert 0 {isc[0].item()!r}" + ) + + +def _read_scalars(keys: list[str], weights, shards) -> dict[str, torch.Tensor]: + """The given per-tensor scalars, off the shard files one file at a time when they are + found, otherwise off the given dict.""" + if shards is None: + return {key: _materialize(weights[key], None) for key in keys} + directory, weight_map = shards + by_file: dict[str, list] = collections.defaultdict(list) + for key in keys: + by_file[weight_map[key]].append(key) + out = {} + for name in sorted(by_file): + with safe_open(str(directory / name), framework="pt", device="cpu") as handle: + for key in by_file[name]: + out[key] = handle.get_slice(key)[...].clone() + return out + + +def _fp8_linear(name, key, n, k, layers, w_shard=None, s_shard=None): + """An FP8 128x128-block linear: e4m3 `[N, K]` and its DeepGEMM scale operand's storage.""" + return ( + W(name, shape=lambda d: (n(d), k(d)), dtype=F8, src=key, layers=layers, shard=w_shard), + W( + f"{name}_s", + shape=lambda d: (_sf_words(k(d)), n(d)), + dtype=I32, + src=key.replace(".weight", ".weight_scale"), + transform=_block_sf, + layers=layers, + shard=s_shard, + ), + ) + + +def _tp_mlp(module, inter, layers): + """A SwiGLU MLP split over its intermediate dimension: `[w1 ; w3]` by output rows, gate + first (the half order `flashinfer_silu_and_mul` reads), `w2` by input columns. The same + row slice applies to `w1` and `w3` before they are stacked, and the scales follow in + 128-blocks; in the scale storage the stacked axis is the second one.""" + blocks = lambda d: inter(d) // BLOCK # noqa: E731 + start = lambda d: d.tp_rank * inter(d) # noqa: E731 + b_start = lambda d: d.tp_rank * blocks(d) # noqa: E731 + p = "{p}." + module + return ( + W( + "mlp_gu", + shape=lambda d: (2 * inter(d), d.hidden), + dtype=F8, + src=_stack( + (f"{p}.w1.weight", lambda d: (slice(0, inter(d)),)), + (f"{p}.w3.weight", lambda d: (slice(inter(d), 2 * inter(d)),)), + ), + layers=layers, + shard=_rows(start, inter), + ), + W( + "mlp_gu_s", + shape=lambda d: (_sf_words(d.hidden), 2 * inter(d)), + dtype=I32, + src=_stack( + (f"{p}.w1.weight_scale", lambda d: (slice(None), slice(0, inter(d)))), + (f"{p}.w3.weight_scale", lambda d: (slice(None), slice(inter(d), 2 * inter(d)))), + ), + transform=_block_sf, + layers=layers, + shard=_rows(b_start, blocks), + ), + *_fp8_linear( + "mlp_dn", + f"{p}.w2.weight", + lambda d: d.hidden, + inter, + layers, + w_shard=_cols(start, inter), + s_shard=_sf_cols(b_start, blocks), + ), + ) + + +def _kv_b(nope, layers): + """`wkv_b`, this rank's heads, regrouped. Per-head rows are `nope(d) + v` wide.""" + width = lambda d: d.heads * (nope(d) + d.v_dim) # noqa: E731 + return W( + "wkv_b", + shape=lambda d: (width(d), d.kv_lora), + src="{p}.attention.wkv_b.weight", + transform=_regroup_kv_b, + layers=layers, + shard=_rows(lambda d: d.tp_rank * width(d), width), + ) + + +def _heads(width): + """This rank's slice of a per-head axis of the given per-head width.""" + return _rows(lambda d: d.tp_rank * d.heads * width(d), lambda d: d.heads * width(d)) + + +def _vision_blocks(d): + return list(range(d.v_layers)) + + +def _has_vision(d): + return d.has_vision + + +def _flatten_patch_conv(tensor, core): + return tensor.flatten(1) + + +def _draft_layers(d): + return range(d.draft_layers) + + +def _draft_source(suffix): + return lambda d, i: [(f"mtp_dspark.backbone.layers.{i}.{suffix}", None)] + + +class MistralLarge4TepWeights: + """mistral-large-4-nvfp4 / sm_107 / tep.""" + + WEIGHTS: tuple[W, ...] = ( + # --- MLA: every layer, KV owners only where marked --- + W("attn_norm", shape=lambda d: (d.hidden,), src="{p}.attention_norm.weight"), + # Per head `[nope 64 | rope 64]` on RoPE layers, 128 nope on NoPE layers: 128 either way. + *_fp8_linear( + "wq", + "{p}.attention.wq.weight", + lambda d: d.heads * d.qk_dim, + lambda d: d.hidden, + ALL_LAYERS, + w_shard=_heads(lambda d: d.qk_dim), + s_shard=_heads(lambda d: d.qk_dim // BLOCK), + ), + W( + "ssmax_s", + shape=lambda d: (d.heads,), + src="{p}.attention.ssmax_scaling.s", + shard=_heads(lambda d: 1), + ), + W( + "ssmax_b", + shape=lambda d: (d.heads,), + src="{p}.attention.ssmax_scaling.b", + shard=_heads(lambda d: 1), + ), + # `[c_kv 512 | k_pe 64]` on RoPE owners; NoPE owners project the 512 latent alone. + W( + "wkv_a", + shape=lambda d: (d.lat_dim, d.hidden), + src="{p}.attention.wkv_a_with_mqa.weight", + layers=_rope_owners, + ), + W( + "wkv_a", + shape=lambda d: (d.kv_lora, d.hidden), + src="{p}.attention.wkv_a_with_mqa.weight", + layers=_nope_owners, + ), + W( + "kv_a_norm", + shape=lambda d: (d.kv_lora,), + src="{p}.attention.kv_a_norm.weight", + layers=_owners, + ), + # Every layer absorbs through its own `wkv_b`, KV sharers included. + _kv_b(lambda d: d.nope, _rope_layers), + _kv_b(lambda d: d.qk_dim, _nope_layers), + *_fp8_linear( + "wo", + "{p}.attention.wo.weight", + lambda d: d.hidden, + lambda d: d.heads * d.v_dim, + ALL_LAYERS, + w_shard=_cols(lambda d: d.tp_rank * d.heads * d.v_dim, lambda d: d.heads * d.v_dim), + s_shard=_sf_cols( + lambda d: d.tp_rank * d.heads * d.v_dim // BLOCK, + lambda d: d.heads * d.v_dim // BLOCK, + ), + ), + W("ffn_norm", shape=lambda d: (d.hidden,), src="{p}.ffn_norm.weight"), + # --- the DSA indexer: replicated, on the 'full' layers --- + W( + "ix_q_norm_w", + shape=lambda d: (d.hidden,), + src="{p}.attention.indexer.q_input_norm.weight", + layers=_indexer_layers, + ), + W( + "ix_q_norm_b", + shape=lambda d: (d.hidden,), + src="{p}.attention.indexer.q_input_norm.bias", + layers=_indexer_layers, + ), + # Per head `[rope 64 | nope 64]`, the reverse of MLA's. + *_fp8_linear( + "ix_wq_b", + "{p}.attention.indexer.wq_b.weight", + lambda d: d.idx_heads * d.idx_dim, + lambda d: d.hidden, + _indexer_layers, + ), + # `[wk ; weights_proj]`: the key and the head weights, one GEMM. + W( + "ix_wk_wp", + shape=lambda d: (d.idx_dim + d.idx_heads, d.hidden), + src=_stack( + ("{p}.attention.indexer.wk.weight", lambda d: (slice(0, d.idx_dim),)), + ( + "{p}.attention.indexer.weights_proj.weight", + lambda d: (slice(d.idx_dim, d.idx_dim + d.idx_heads),), + ), + ), + layers=_indexer_layers, + ), + # fp32, as the fused key entry (LayerNorm in fp32) takes them. + W( + "ix_k_norm_w", + shape=lambda d: (d.idx_dim,), + dtype=F32, + src="{p}.attention.indexer.k_norm.weight", + transform=_to_fp32, + layers=_indexer_layers, + ), + W( + "ix_k_norm_b", + shape=lambda d: (d.idx_dim,), + dtype=F32, + src="{p}.attention.indexer.k_norm.bias", + transform=_to_fp32, + layers=_indexer_layers, + ), + W( + "ix_ssmax_s", + shape=lambda d: (d.idx_heads,), + src="{p}.attention.indexer.ssmax_scaling.s", + layers=_indexer_layers, + ), + W( + "ix_ssmax_b", + shape=lambda d: (d.idx_heads,), + src="{p}.attention.indexer.ssmax_scaling.b", + layers=_indexer_layers, + ), + # --- the TP-sliced SwiGLU MLP: the dense layer's own, then each MoE layer's shared + # expert, one key --- + *_tp_mlp("feed_forward", lambda d: d.dense_inter, _dense_layers), + *_tp_mlp("shared_experts", lambda d: d.shared_inter, _moe_layers), + # --- routing --- + W( + "router", + shape=lambda d: (d.num_experts, d.hidden), + src="{p}.gate.weight", + layers=_moe_layers, + ), + # --- this rank's routed experts, NVFP4: `[w3 ; w1]` as stored, then each expert in the + # MoE runner's layout --- + W( + "fc1_w", + shape=lambda d: (d.local_experts, 2 * d.moe_inter, d.hidden // E2M1_PER_BYTE), + dtype=U8, + src=_expert_window( + ("w3.weight", lambda d: (slice(0, d.moe_inter),)), + ("w1.weight", lambda d: (slice(d.moe_inter, 2 * d.moe_inter),)), + ), + layers=_moe_layers, + layout=_fc1_codes, + ), + W( + "fc1_s", + shape=lambda d: (d.local_experts, 2 * d.moe_inter, d.hidden // NVFP4_BLOCK), + dtype=F8, + src=_expert_window( + ("w3.weight_scale", lambda d: (slice(0, d.moe_inter),)), + ("w1.weight_scale", lambda d: (slice(d.moe_inter, 2 * d.moe_inter),)), + ), + layers=_moe_layers, + layout=_fc1_scales, + ), + W( + "fc2_w", + shape=lambda d: (d.local_experts, d.hidden, d.moe_inter // E2M1_PER_BYTE), + dtype=U8, + src=_expert_window(("w2.weight", lambda d: ())), + layers=_moe_layers, + layout=_fc2_codes, + ), + W( + "fc2_s", + shape=lambda d: (d.local_experts, d.hidden, d.moe_inter // NVFP4_BLOCK), + dtype=F8, + src=_expert_window(("w2.weight_scale", lambda d: ())), + layers=_moe_layers, + layout=_fc2_scales, + ), + # The per-tensor scalars: the layer's one FC1 and one FC2 activation scale, and each + # local expert's global weight scales (`w3`'s are `w1`'s). + W( + "isc1", + shape=lambda d: (1,), + dtype=F32, + src=_window_scalar("w1.input_scale", "w3.input_scale"), + transform=_shared_scalar, + layers=_moe_layers, + ), + W( + "isc2", + shape=lambda d: (1,), + dtype=F32, + src=_window_scalar("w2.input_scale"), + transform=_shared_scalar, + layers=_moe_layers, + ), + W( + "ws2_13", + shape=lambda d: (d.local_experts,), + dtype=F32, + src=_expert_window((("w1.weight_scale_2", "w3.weight_scale_2"), lambda d: ())), + transform=_shared_scalar, + layers=_moe_layers, + ), + W( + "ws2_2", + shape=lambda d: (d.local_experts,), + dtype=F32, + src=_expert_window(("w2.weight_scale_2", lambda d: ())), + layers=_moe_layers, + ), + # --- not per-layer --- + W("final_norm", shape=lambda d: (d.hidden,), src="norm.weight", layers=None), + W("embed", shape=lambda d: (d.vocab, d.hidden), src="tok_embeddings.weight", layers=None), + # The LeRoPE per-frequency log-scales the rope tables are built from, fp32 as the + # reference widens them. + W( + "lerope_alpha", + shape=lambda d: (d.rope_dim // 2,), + dtype=F32, + src="lerope.alpha", + transform=_to_fp32, + layers=None, + ), + # BF16 vision encoder and projector, replicated on every rank. + W( + "vis_patch", + shape=lambda d: (d.v_hidden, d.v_channels * d.v_patch**2), + src="vision_encoder.patch_conv.weight", + transform=_flatten_patch_conv, + layers=None, + when=_has_vision, + ), + W( + "vis_ln_pre", + shape=lambda d: (d.v_hidden,), + src="vision_encoder.ln_pre.weight", + layers=None, + when=_has_vision, + ), + W( + "attn_norm", + shape=lambda d: (d.v_hidden,), + src="{p}.attention_norm.weight", + layers=_vision_blocks, + tag="v", + ), + W( + "qkv", + shape=lambda d: (3 * d.v_hidden, d.v_hidden), + src=_stack( + ("{p}.attention.wq.weight", lambda d: (slice(0, d.v_hidden),)), + ("{p}.attention.wk.weight", lambda d: (slice(d.v_hidden, 2 * d.v_hidden),)), + ("{p}.attention.wv.weight", lambda d: (slice(2 * d.v_hidden, 3 * d.v_hidden),)), + prefix="vision_encoder.transformer.layers", + ), + layers=_vision_blocks, + tag="v", + ), + W( + "wo", + shape=lambda d: (d.v_hidden, d.v_hidden), + src="{p}.attention.wo.weight", + layers=_vision_blocks, + tag="v", + ), + W( + "ffn_norm", + shape=lambda d: (d.v_hidden,), + src="{p}.ffn_norm.weight", + layers=_vision_blocks, + tag="v", + ), + W( + "w13", + shape=lambda d: (2 * d.v_inter, d.v_hidden), + src=_stack( + ("{p}.feed_forward.w1.weight", lambda d: (slice(0, d.v_inter),)), + ("{p}.feed_forward.w3.weight", lambda d: (slice(d.v_inter, 2 * d.v_inter),)), + prefix="vision_encoder.transformer.layers", + ), + layers=_vision_blocks, + tag="v", + ), + W( + "w2", + shape=lambda d: (d.v_hidden, d.v_inter), + src="{p}.feed_forward.w2.weight", + layers=_vision_blocks, + tag="v", + ), + W( + "vis_proj_norm", + shape=lambda d: (d.v_hidden,), + src="pre_mm_projector_norm.weight", + layers=None, + when=_has_vision, + ), + W( + "vis_merge", + shape=lambda d: (d.v_hidden, d.v_hidden * d.v_merge**2), + src="patch_merger.merging_layer.weight", + layers=None, + when=_has_vision, + ), + W( + "vis_w_in", + shape=lambda d: (d.hidden, d.v_hidden), + src="vision_language_adapter.w_in.weight", + layers=None, + when=_has_vision, + ), + W( + "vis_w_out", + shape=lambda d: (d.hidden, d.hidden), + src="vision_language_adapter.w_out.weight", + layers=None, + when=_has_vision, + ), + ) + + # BF16 draft weights use the same attention and dense-MLP partition as + # this target. The RNN and target-feature projection are replicated. + DRAFT_WEIGHTS: tuple[W, ...] = ( + W( + "draft_attn_norm", + lambda d: (d.hidden,), + _draft_source("attention_norm.weight"), + layers=_draft_layers, + ), + W( + "draft_wq", + lambda d: (d.heads * d.qk_dim, d.hidden), + _draft_source("attention.wq.weight"), + layers=_draft_layers, + shard=_rows( + lambda d: d.draft_tp_rank * d.heads * d.qk_dim, lambda d: d.heads * d.qk_dim + ), + ), + W( + "draft_wkv_a", + lambda d: (d.lat_dim, d.hidden), + _draft_source("attention.wkv_a_with_mqa.weight"), + layers=_draft_layers, + ), + W( + "draft_kv_a_norm", + lambda d: (d.kv_lora,), + _draft_source("attention.kv_a_norm.weight"), + layers=_draft_layers, + ), + W( + "draft_wkv_b", + lambda d: (d.heads * (d.nope + d.v_dim), d.kv_lora), + _draft_source("attention.wkv_b.weight"), + layers=_draft_layers, + shard=_rows( + lambda d: d.draft_tp_rank * d.heads * (d.nope + d.v_dim), + lambda d: d.heads * (d.nope + d.v_dim), + ), + ), + W( + "draft_wo", + lambda d: (d.hidden, d.heads * d.v_dim), + _draft_source("attention.wo.weight"), + layers=_draft_layers, + shard=_cols(lambda d: d.draft_tp_rank * d.heads * d.v_dim, lambda d: d.heads * d.v_dim), + ), + W( + "draft_ffn_norm", + lambda d: (d.hidden,), + _draft_source("ffn_norm.weight"), + layers=_draft_layers, + ), + W( + "draft_w1", + lambda d: (d.dense_inter, d.hidden), + _draft_source("feed_forward.w1.weight"), + layers=_draft_layers, + shard=_rows(lambda d: d.draft_tp_rank * d.dense_inter, lambda d: d.dense_inter), + ), + W( + "draft_w3", + lambda d: (d.dense_inter, d.hidden), + _draft_source("feed_forward.w3.weight"), + layers=_draft_layers, + shard=_rows(lambda d: d.draft_tp_rank * d.dense_inter, lambda d: d.dense_inter), + ), + W( + "draft_w2", + lambda d: (d.hidden, d.dense_inter), + _draft_source("feed_forward.w2.weight"), + layers=_draft_layers, + shard=_cols(lambda d: d.draft_tp_rank * d.dense_inter, lambda d: d.dense_inter), + ), + W( + "draft_final_norm", + lambda d: (d.hidden,), + "mtp_dspark.backbone.final_norm.weight", + layers=None, + ), + W( + "draft_target_norm", + lambda d: (d.hidden,), + "mtp_dspark.backbone.target_norm.weight", + layers=None, + ), + W( + "draft_target_projection", + lambda d: (d.hidden, d.hidden * d.draft_capture_layers), + "mtp_dspark.backbone.target_projection.weight", + layers=None, + ), + W( + "draft_mask", + lambda d: (d.draft_depth - 1, d.hidden), + "mtp_dspark.backbone.mask_embedding", + layers=None, + ), + W( + "draft_markov_embedding", + lambda d: (d.vocab, d.draft_rank), + "mtp_dspark.autoregressive_head.embedding.weight", + layers=None, + ), + W( + "draft_markov_output", + lambda d: (d.vocab, d.draft_rank), + "mtp_dspark.autoregressive_head.output.weight", + layers=None, + ), + W( + "draft_joint", + lambda d: (3 * d.draft_rank, d.hidden + 2 * d.draft_rank), + "mtp_dspark.autoregressive_head.joint_proj.weight", + layers=None, + ), + W( + "draft_joint_bias", + lambda d: (3 * d.draft_rank,), + "mtp_dspark.autoregressive_head.joint_proj.bias", + layers=None, + ), + ) + + def _entries(self, d): + return self.WEIGHTS + (self.DRAFT_WEIGHTS if d.draft_layers else ()) + + def declare(self, d: SimpleNamespace) -> nn.ParameterDict: + """Allocate every weight the table declares. + + Meta-init intercepts `torch.empty` here -- real CUDA storage arrives + when the engine materializes the registry. + """ + w = nn.ParameterDict() + for entry in self._entries(d): + dtype = entry.dtype or d.dtype + for key, _, _ in self._targets(entry, d): + assert key not in w, f"two rows declare {key}" + w[key] = nn.Parameter( + torch.empty(*entry.shape(d), dtype=dtype), requires_grad=False + ) + return w + + def _targets(self, entry: W, d: SimpleNamespace) -> list[tuple[str, int | None, str]]: + """`[(param key, layer index | None, checkpoint layer prefix), ...]`. + + One place decides which parameters an entry declares, so `declare` and + `manifest` cannot disagree about whether a weight exists. + """ + if entry.when is not None and not entry.when(d): + return [] + layers = entry.layer_indices(d) + if layers is None: + return [(entry.name, None, "")] + prefix = "vision_encoder.transformer.layers" if entry.tag == "v" else "layers" + return [(f"{entry.tag}{i}_{entry.name}", i, f"{prefix}.{i}") for i in layers] + + def manifest(self, core) -> dict: + """param key -> [(checkpoint key | key tuple, index into the param | None, transform, + shard)].""" + d = self.dims(core) + rows: dict = {} + for entry in self._entries(d): + shard = entry.shard(d) if entry.shard else None + for key, i, prefix in self._targets(entry, d): + assert key not in rows, f"two rows declare {key}" + rows[key] = [ + (ckpt, index, entry.transform, shard) + for ckpt, index in entry.sources(d, i, prefix) + ] + return rows + + def expected_unconsumed(self, core, weights) -> dict[str, str]: + """Declared unused keys: off-rank experts and disabled draft weights. Off-rank expert + scalars are still read to validate the layer's shared scales.""" + d = self.dims(core) + lo, hi = d.expert_offset, d.expert_offset + d.local_experts + keys = { + k: "DSpark draft disabled" + for k in weights.keys() + if not d.draft_layers and k.startswith("mtp_dspark.") + } + window = f"routed expert outside this rank's window [{lo}, {hi})" + never_read = f"{window}: never read" + checked = f"{window}: read only to check the layer's shared scales" + for i in _moe_layers(d): + for e in range(d.num_experts): + if lo <= e < hi: + continue + for proj in PROJECTIONS: + for s in ("weight", "weight_scale"): + key = _expert_key(i, e, f"{proj}.{s}") + if key in weights: + keys[key] = never_read + for s in EXPERT_SCALARS: + keys[_expert_key(i, e, f"{proj}.{s}")] = checked + return keys + + def load_checkpoint(self, model, checkpoint_dir) -> None: + """Fill every declared weight from a checkpoint directory, which is what the engine + loads: `load` over the directory's keys, every source read from its shard file.""" + directory, weight_map = _checkpoint_shards(checkpoint_dir) + self.load(model, weight_map, shards=(directory, weight_map)) + + def load(self, model, weights, shards=None) -> None: + """Fill every declared weight from the checkpoint: the shard column, multi-source rows, + the vocab-parallel `lm_head`, and coverage asserted in both directions. + + First every routed expert's per-tensor scalars are read and checked, the other ranks' + included, so a checkpoint that breaks what the forward relies on fails before any + weight moves. When the checkpoint's own shard files can be found, every source is read + from them one file at a time, this rank's slices only, and each file's cached pages are + dropped once its sources are copied. Mapping every shard file on every rank and reading + through the mappings instead would fault a node's four ranks' expert + windows -- 539 GiB, the whole expert stack, on one node of four; 270 GiB, half of it, + on each node of two -- into that node's host memory. Last, each row's `layout` puts the + expert stacks into the MoE runner's layout, so a second `load` starts again from the + stored bytes. + """ + core = model.model + manifest = self.manifest(core) + consumed: set = set() + + def fill(param, ckpt, entry, index, transform, shard) -> None: + dst = param.data if index is None else param.data[index] + if isinstance(ckpt, tuple): + src = transform( + tuple(_materialize(e, shard).to(dst.device, non_blocking=True) for e in entry), + core, + ) + else: + src = _materialize(entry, shard) + if transform is not None: + # A column shard is a strided view: make it dense before a transform reads it. + src = transform(src.to(dst.device, non_blocking=True).contiguous(), core) + assert tuple(dst.shape) == tuple(src.shape), (ckpt, tuple(dst.shape), tuple(src.shape)) + assert src.dtype == dst.dtype, (ckpt, src.dtype, dst.dtype) + dst.copy_(src, non_blocking=True) + consumed.update(_keys(ckpt)) + + assert set(manifest.keys()) == set(core.w.keys()), ( + "manifest/parameter drift", + sorted(set(manifest.keys()) ^ set(core.w.keys()))[:12], + ) + # The shell builds lm_head vocab-parallel under TP: this rank's block of rows. + d = self.dims(core) + rows = d.vocab // d.tp_size + plan = [ + (core.w[param_key], ckpt, index, transform, shard) + for param_key, sources in manifest.items() + for ckpt, index, transform, shard in sources + ] + plan.append( + ( + model.lm_head.weight, + "output.weight", + None, + None, + _rows(lambda _: d.tp_rank * rows, lambda _: rows)(d), + ) + ) + scalar_keys = _expert_scalar_keys(d) + for key in (*(k for row in plan for k in _keys(row[1])), *scalar_keys): + assert key in weights, f"checkpoint key missing: {key}" + if shards is None: + shards = _shard_files(core, weights) + scalars = _read_scalars(scalar_keys, weights, shards) + _check_expert_scalars(scalars, d) + + streamed = [] + for param, ckpt, index, transform, shard in plan: + if all(k in scalars for k in _keys(ckpt)): + entry = tuple(scalars[k] for k in _keys(ckpt)) + entry = entry if isinstance(ckpt, tuple) else entry[0] + fill(param, ckpt, entry, index, transform, shard) + else: + assert isinstance(ckpt, str), f"a key tuple of other than expert scalars: {ckpt}" + streamed.append((param, ckpt, index, transform, shard)) + if shards is None: + for param, ckpt, index, transform, shard in streamed: + fill(param, ckpt, weights[ckpt], index, transform, shard) + else: + directory, weight_map = shards + by_file: dict[str, list] = collections.defaultdict(list) + for row in streamed: + by_file[weight_map[row[1]]].append(row) + on_device = model.lm_head.weight.device.type == "cuda" + for name in sorted(by_file): + with safe_open(str(directory / name), framework="pt", device="cpu") as handle: + for param, ckpt, index, transform, shard in by_file[name]: + fill(param, ckpt, handle.get_slice(ckpt), index, transform, shard) + if on_device: + torch.cuda.synchronize() + _drop_page_cache(directory / name) + # Every source has landed: rearrange each expert in place, one at a time. + for entry in self._entries(d): + if entry.layout is None: + continue + for key, _, _ in self._targets(entry, d): + param = core.w[key].data + for slot in range(param.shape[0]): + param[slot].copy_(entry.layout(param[slot])) + shell = {n for n, _ in model.named_parameters() if not n.startswith("model.w.")} + assert shell == {"lm_head.weight"}, f"unfed shell parameters: {sorted(shell)}" + + if model.lm_head.weight.device.type == "cuda": + torch.cuda.synchronize() + leftover = set(weights.keys()) - consumed + expected = self.expected_unconsumed(core, weights) + assert leftover == set(expected), ( + "checkpoint coverage: unexpected " + f"{sorted(leftover - set(expected))[:8]}, missed {sorted(set(expected) - leftover)[:8]}" + ) + not_loaded = collections.Counter(expected.values()) + logger.info( + f"modeling_v2 weights: {len(consumed)} checkpoint tensors loaded " + f"({'streamed per shard file' if shards else 'from the given dict'}), " + f"{len(scalars)} expert scalars checked; not loaded: " + + "; ".join(f"{n} x {reason}" for reason, n in sorted(not_loaded.items())) + ) + + def dims(self, core) -> SimpleNamespace: + cfg = core.model_config.pretrained_config + vis = getattr(cfg, "vision_config", None) + mapping = core.model_config.mapping + spec = getattr(core.model_config, "spec_config", None) + draft = getattr(spec, "_draft_model_config", None) or {} + tp = mapping.tp_size + local_experts = cfg.n_routed_experts // mapping.moe_ep_size + return SimpleNamespace( + draft_layers=int(draft.get("n_layers", 0)), + draft_rank=int((draft.get("dspark") or {}).get("markov_rank", 0)), + draft_depth=int(draft.get("depth", 0)), + draft_capture_layers=len(spec.target_layer_ids) if draft else 0, + draft_tp_rank=mapping.tp_rank, + num_layers=cfg.num_hidden_layers, + hidden=cfg.hidden_size, + heads=cfg.num_attention_heads // tp, + nope=cfg.qk_nope_head_dim, + rope_dim=cfg.qk_rope_head_dim, + qk_dim=cfg.qk_nope_head_dim + cfg.qk_rope_head_dim, + v_dim=cfg.v_head_dim, + kv_lora=cfg.kv_lora_rank, + lat_dim=cfg.kv_lora_rank + cfg.qk_rope_head_dim, + nope_layer=tuple(bool(f) for f in cfg.nope_layer_flags), + kv_source=tuple(int(s) for s in cfg.kv_source_layers), + full_indexer=tuple(t == "full" for t in cfg.index_sharing), + idx_heads=cfg.index_n_heads, + idx_dim=cfg.index_head_dim, + dense_layers=cfg.first_k_dense_replace, + num_experts=cfg.n_routed_experts, + moe_inter=cfg.moe_intermediate_size, + local_experts=local_experts, + expert_offset=local_experts * mapping.moe_ep_rank, + shared_inter=cfg.moe_intermediate_size * cfg.n_shared_experts // tp, + dense_inter=cfg.intermediate_size // tp, + vocab=cfg.vocab_size, + tp_rank=mapping.tp_rank, + tp_size=tp, + dtype=core.model_config.torch_dtype, + # The vision tower (replicated; absent without a vision encoder). + has_vision=vis is not None, + v_layers=vis.num_hidden_layers if vis is not None else 0, + v_hidden=vis.hidden_size if vis is not None else 0, + v_heads=vis.num_attention_heads if vis is not None else 0, + v_head_dim=vis.hidden_size // vis.num_attention_heads if vis is not None else 0, + v_inter=vis.intermediate_size if vis is not None else 0, + v_channels=vis.num_channels if vis is not None else 0, + v_patch=vis.patch_size if vis is not None else 0, + v_merge=vis.spatial_merge_size if vis is not None else 0, + v_theta=float(vis.rope_theta) if vis is not None else 0.0, + image_token_id=vis.image_token_id if vis is not None else None, + ) + + +MODEL_WEIGHTS = MistralLarge4TepWeights() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/__init__.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/__init__.py new file mode 100644 index 000000000000..ae4088f9770c --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Mistral-Large-4-1T-A52B-NVFP4 / sm_107 / tp, the language model only.""" diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/modeling.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/modeling.py new file mode 100644 index 000000000000..ff688dbf9480 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/modeling.py @@ -0,0 +1,2549 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""ModelingV2 target: mistral-large-4-nvfp4 / sm_107 / tp, native text and image inference. + +72 layers, hidden 6144, 64 query heads (16 per rank at tp4, 8 at tp8). Attention is MLA without +a q-LoRA (`wq` straight to 64 x 128), a 512-wide latent plus a 64-wide rope key, and DeepSeek +Sparse Attention: an indexer on the 44 `index_sharing == 'full'` layers selects 2048 positions +per query row and each 'shared' layer reuses the selection of the most recent full layer. Every +fourth layer (`i % 4 == 3`) takes no rope, for MLA and for its indexer; the NoPE layers 39, +43, ..., 71 own no latent and attend over layer 35's. Queries are scaled by SSMax, +`s * log(pos + 1) + b` per head, on MLA and on the indexer. Layer 0 is a dense SwiGLU MLP, +layers 1-71 MoE (128 experts, softmax top-4 renormalised, one shared expert). The final logits +are soft-capped at `softmax_tanh_gating`. + +The checkpoint is ModelOpt `MIXED_PRECISION`. The routed experts are NVFP4 W4A4: e2m1 weights +with one e4m3 scale per 16 inputs and a global scale per expert, and activations quantized to +NVFP4 under one static global scale per layer and projection. `wq`, `wo`, the indexer's `wq_b` +and the dense and shared MLPs are FP8 e4m3 with 128x128 power-of-two block scales and dynamic +1x128 activation scales. The rest is bf16. The latent KV pool is fp8_ds_mla (656 B per token +and owner layer: 512 e4m3 codes with one power-of-two fp32 scale per 128, and the 64-wide rope +key in bf16), owned by `MistralLarge4CacheManager`; the indexer key pool is fp8 with one scale +per key. + +`tp` shards the attention heads, the dense and shared MLPs and `lm_head` over every rank -- +four ways on one node of four, or eight ways over two nodes of four (tp8), every width read off +the mapping -- and the routed experts by width: every rank holds all 128 experts, each cut to +its `moe_intermediate_size / moe_tp_size`-wide slice of the intermediate (1536 at tp4, 768 at +tp8), so one runner call over every token returns this rank's partial sum over its slice. The +latent projection, the indexer and the router are replicated. Every layer closes twice on the +all-reduce over every rank fused with the residual add and the next RMSNorm; the second one +sums the ranks' routed and shared partials together. Under chunked prefill, a step of +`q_split_threshold` context rows or more has the indexer's context queries split over every +rank by the engine: each rank selects for its share, and every selecting layer all-gathers the +shares before any layer attends. + +The absorbed form throughout: `fused_q = [q_nope @ W_UK | rope(q_pe)]` (576 wide) against the +pooled `[c_kv | k_pe]` latent rows, then `@ W_UV`. Context rows attend over the selection (the +engine's causal index on a step it skipped the indexer) with FlashMLA's bf16 sparse prefill +over the requests' latent rows dequantized out of the pool, so they read the FP8-rounded latent +as the reference does; generation rows run sparse fp8 decode over the pool itself, at this +rank's 16 (tp4) or 8 (tp8) heads at every max_seq_len: cute_dsl_ds_mla_sparse_decode, +FlashMLA's sparse decode arithmetic without the head padding. The prefill kernel, and the +sparse decode a context row past the held workspace runs (FlashMLA's), take 64 heads: this +rank's go first in a 64-head query whose other heads are zero; heads do not interact, so the +zero heads change nothing the live ones compute. `_GEN_ATTN_FLASH_MLA` (for A/B verification) +runs the generation rows on FlashMLA's 64-head decode too, the same padding. + +No path is selected by the serving configuration: each step chooses from its own sizes, and +every buffer is sized from the engine's quantities (the attention metadata's max_num_tokens, +max_num_requests, max_seq_len and runtime_features). The context rows take one of two paths, +chosen per step by the step's context KV (every context request's cached and new rows): while +it fits the held workspace, the sparse prefill over those rows dequantized into it; past it, +each context row as a one-row sequence of the sparse fp8 decode over the pool itself, as +generation rows always run. The held workspace, allocated on the first context step, has room +for the most context KV one step can attend, twice over, up to 2**19 rows +(`_ctx_workspace_rows`): a step whose context KV fits twice keeps layer 35's rows beside its +own for the NoPE layers attending over that latent, a longer one gathers them again for each +of those layers. An indexer chunk the engine packs past `max_seq_len` keys is scored request +by request, and the indexer's logits are tiled under a budget from the engine's max_num_tokens +and max_seq_len (`_prefill_logits_budget`), each tile scored with the Triton FP8 MQA scorer +and selected with the self-sampling GVR +top-k (`_select_ctx`).""" + +from __future__ import annotations + +import itertools +import math +import os +import time +from dataclasses import replace +from types import SimpleNamespace +from typing import NamedTuple, Optional, cast + +import torch +from torch import nn +from transformers import PretrainedConfig + +from tensorrt_llm._torch.attention.backends.interface import AttentionMetadata +from tensorrt_llm._torch.attention.backends.sparse.params import ( + get_indexer_mqa_logits_workspace_bytes, +) +from tensorrt_llm._torch.attention.backends.trtllm import TrtllmAttentionMetadata +from tensorrt_llm._torch.model_config import ModelConfig +from tensorrt_llm._torch.modeling_v2._router_index import step_contract_enabled +from tensorrt_llm._torch.modeling_v2.catalog.activation.flashinfer_silu_and_mul import ( + flashinfer_silu_and_mul, +) +from tensorrt_llm._torch.modeling_v2.catalog.attention.thop_attention import thop_attention +from tensorrt_llm._torch.modeling_v2.catalog.comm.allgather import allgather +from tensorrt_llm._torch.modeling_v2.catalog.gemm.bmm_out import bmm_out +from tensorrt_llm._torch.modeling_v2.catalog.moe.fp4_block_scale_moe_runner import ( + fp4_block_scale_moe_runner, +) +from tensorrt_llm._torch.modeling_v2.catalog.norm.flashinfer_fused_add_rmsnorm import ( + flashinfer_fused_add_rmsnorm, +) +from tensorrt_llm._torch.modeling_v2.catalog.norm.flashinfer_rmsnorm import flashinfer_rmsnorm +from tensorrt_llm._torch.modeling_v2.catalog.quantization.fp4_quantize import fp4_quantize +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.activation.gelu_erf import gelu_erf +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.activation.tanh_softcap import ( + tanh_softcap, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.convert_req_index_to_global import ( # noqa: E501 + convert_req_index_to_global, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_ds_mla_sparse_decode import ( # noqa: E501 + build as build_ds_mla_decode, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_ds_mla_sparse_decode import ( # noqa: E501 + cute_dsl_ds_mla_sparse_decode, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_fp8_paged_mqa_logits import ( # noqa: E501 + cute_dsl_fp8_paged_mqa_logits, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_gvr_topk_prefill import ( # noqa: E501 + cute_dsl_gvr_topk_prefill, + gvr_warmup, + gvr_workspace, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_indexer_k_cache_update import ( # noqa: E501 + build as build_ix_k_cache, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_indexer_k_cache_update import ( # noqa: E501 + cute_dsl_indexer_k_cache_update, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_indexer_q_side_fused import ( # noqa: E501 + build as build_ix_q_side, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_indexer_q_side_fused import ( # noqa: E501 + cute_dsl_indexer_q_side_fused, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_radix_topk_decode import ( # noqa: E501 + build as build_ix_topk_decode, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_radix_topk_decode import ( # noqa: E501 + build_split as build_ix_topk_verify, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_radix_topk_decode import ( # noqa: E501 + cute_dsl_radix_topk_decode, + split_radix_topk_decode, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ds_mla_dequant_gather import ( + ds_mla_dequant_gather, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ds_mla_rope_append import ( + ds_mla_rope_append, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.flash_mla_sparse_decode import ( + flash_mla_sparse_decode, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.flash_mla_sparse_prefill import ( + flash_mla_sparse_prefill, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.indexer_k_cache_gather_op import ( # noqa: E501 + indexer_k_cache_gather_op, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.indexer_topk_decode import ( + indexer_topk_decode, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ssmax_indexer_q_side_fused import ( # noqa: E501 + ssmax_indexer_q_side_fused, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ssmax_scaling import ( + ssmax_scaling, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ssmax_scaling_fused_q import ( + ssmax_scaling_fused_q, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.triton_fp8_mqa_logits import ( + triton_fp8_mqa_logits, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.comm.cute_dsl_allreduce_add_rmsnorm import ( # noqa: E501 + build_workspace as build_lamport_workspace, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.comm.cute_dsl_allreduce_add_rmsnorm import ( # noqa: E501 + cute_dsl_allreduce_add_rmsnorm, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.gemm.cute_dsl_fp8_block_scale_linear import ( # noqa: E501 + cute_dsl_fp8_block_scale_linear, + prepare_weight_scale, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.moe.renorm_moe_routing_op import ( + renorm_moe_routing_op, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.norm.layer_norm import layer_norm +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.vision.pixtral_rope_2d import ( + pixtral_rope_2d, +) +from tensorrt_llm._torch.models.modeling_multimodal_utils import ( + find_input_mm_embeds, + get_multimodal_embeddings, +) +from tensorrt_llm._torch.models.modeling_utils import DecoderModel, DecoderModelForCausalLM +from tensorrt_llm._torch.modules.multi_stream_utils import maybe_execute_in_parallel +from tensorrt_llm._torch.pyexecutor.guided_decoder import CapturableGuidedDecoder +from tensorrt_llm._torch.speculative import get_spec_worker +from tensorrt_llm._utils import mpi_disabled +from tensorrt_llm.flash_mla import get_mla_metadata +from tensorrt_llm.inputs.multimodal import MultimodalParams +from tensorrt_llm.logger import logger +from tensorrt_llm.models.modeling_utils import QuantConfig + +from . import weights as _weights +from .ops.gemm.cute_dsl_fused_fp8_block_scale_linear import cute_dsl_fused_fp8_block_scale_linear +from .ops.gemm.low_m_cublas_mm import low_m_cublas_mm as cublas_mm + +#: eps 1e-5, the reference's value; params.json's `vision_encoder` states none. +_VISION_EPS = 1e-5 + +#: The thop attention arguments the vision tower's call holds fixed: the no-KV-cache +#: mode (an encoder: the three cache-addressing arguments None), one context sequence +#: per image, the fused `[q | k | v]` rows already rotated (no rope in the op), +#: bidirectional within each sequence (mask_type 0), softmax scale head_dim^-0.5. +_VISION_ATTN = dict( + k=None, + v=None, + output_sf=None, + max_context_q_len_override=None, + kv_cache_block_offsets=None, + host_kv_cache_pool_pointers=None, + host_kv_cache_pool_mapping=None, + cache_indirection=None, + kv_scale_orig_quant=None, + kv_scale_quant_orig=None, + out_scale=None, + rotary_inv_freq=None, + rotary_cos_sin=None, + latent_cache=None, + q_pe=None, + block_ids_per_seq=None, + attention_sinks=None, + is_fused_qkv=True, + update_kv_cache=True, + predicted_tokens_per_seq=1, + local_layer_idx=0, + tokens_per_block=None, + beam_width=1, + mask_type=0, + quant_mode=0, + q_scaling=1.0, + position_embedding_type=0, + rope_dim=0, + rope_base=10000.0, + rope_scale_type=0, + rope_scale=1.0, + rope_short_m_scale=1.0, + rope_long_m_scale=1.0, + rope_max_positions=1024, + rope_original_max_positions=1024, + use_paged_context_fmha=False, + attention_input_type=0, + is_mla_enable=False, + chunked_prefill_buffer_batch_size=1, + q_lora_rank=None, + kv_lora_rank=None, + qk_nope_head_dim=None, + qk_rope_head_dim=None, + v_head_dim=None, + rope_append=None, + mrope_rotary_cos_sin=None, + mrope_position_deltas=None, + helix_position_offsets=None, + helix_is_inactive_rank=None, + attention_chunk_size=None, + softmax_stats_tensor=None, + is_spec_decoding_enabled=False, + use_spec_decoding=False, + is_spec_dec_tree=False, + spec_decoding_generation_lengths=None, + spec_decoding_position_offsets=None, + spec_decoding_packed_mask=None, + spec_decoding_bl_tree_mask_offset=None, + spec_decoding_bl_tree_mask=None, + spec_bl_tree_first_sparse_mask_offset_kv=None, + sparse_kv_indices=None, + sparse_kv_offsets=None, + sparse_attn_indices=None, + sparse_attn_offsets=None, + sparse_attn_indices_block_size=0, +) + +# The latent pool layout this target addresses (`MistralLarge4CacheManager.latent_layout`). +_LATENT_LAYOUT = "fp8_ds_mla" +# The query heads FlashMLA's sm100 sparse kernels run; this rank's heads go first. +_FLASH_MLA_HEADS = 64 +# Generation rows' sparse MLA. False: cute_dsl_ds_mla_sparse_decode at this rank's heads, the +# shipped path. True: flash_mla_sparse_decode over the 64-head padded query, the kernel this +# target ran before (bit-identical where the two split a row alike; see the entry's note) -- +# a switch for A/B verification, not a configuration. It sets the core's +# `gen_attn_flash_mla`, which every generation-row site reads; the native entry's cells cover +# the 32K and the 1M configs (key lengths to 1048576, the 1M pools' slots past 2**31 bytes). +_GEN_ATTN_FLASH_MLA = False +# The pre-routed MoE call: the runner's routing method is inert once ids and weights are +# given; 1 (renormalize) is the value its cells drive. +_MOE_ROUTING_METHOD = 1 +# The routed experts' SwiGLU, the runner's `act_type` 0. +_MOE_ACT_SWIGLU = 0 +# NVFP4: one e4m3 scale per 16 activations, laid out row-major (linear), the order the MoE +# runner reads; the 128x4-swizzled order is another buffer of the same length at 128-row +# multiples, and silently wrong there. +_NVFP4_SF_VEC = 16 +_NVFP4_SF_LINEAR = False +# triton_fp8_mqa_logits allocates a `[q_rows, kv_rows]` fp32 score matrix per call; the context +# queries are tiled to keep it under the engine's budget (`_prefill_logits_budget`). Context +# steps are not captured. +# capacity: fixed: the budget's ceiling, one 1 GiB fp32 tile (256 rows at 1048576 keys). +# Mainline's default reachable tile (2**31 logits) would take 8 GiB from the KV pool. +_PREFILL_LOGITS_CEILING = 1 << 28 +# The context path's crossover, chosen once per step on host ints, and the held context +# workspace's ceiling (`_ctx_workspace_rows`). A step whose context KV (every context request's +# cached and new rows) fits the held workspace has it gathered, dequantized, into it (bf16 +# [rows, 576]) for FlashMLA's sparse prefill; a longer one attends its context rows straight off +# the pool with the sparse decode kernel, whose cost does not grow with the prefix as the +# gather's does. +# capacity: fixed: 2**19 rows, 576 MiB, the gather-vs-decode crossover. +_CTX_WORKSPACE_ROWS = 1 << 19 +# The most context rows one sparse-decode call takes on that path: FlashMLA's decode scheduler +# exits the process, raising nothing, past 11622 one-row sequences on sm_103. +# capacity: fixed: the kernel's scheduler, not an engine limit. +_CTX_DECODE_ROWS = 8192 +# CuTe decode selectors: one CTA per row for next_n=1 at 4K..64K columns, and ten +# partitions plus a radix merge for DSpark verification from 20480 columns (the narrowest +# row the native op splits into ten blocks) to 1048584 (the split kernel's widest row). +# split_radix_topk_decode keeps the native op for a call the op would not split into ten +# blocks, or whose rows are not 32-byte aligned with a width that is a multiple of 8. +# Other shapes retain the native selector. +_TOPK_DECODE_CUTE_DSL = True +_TOPK_DECODE_COLUMNS = (4096, 65536) +_TOPK_VERIFY_COLUMNS = (20480, 1048584) +_TOPK_VERIFY_ROWS = (4, 8, 16) +# The attention block's glue as separate ops instead of the fused entries, for old-versus-new +# verification (both compute the same bits; the fused entries' tests hold each pair equal): +# ssmax_scaling plus the rope tail's copy or zero fill for ssmax_scaling_fused_q; +# ssmax_scaling, the head weights' fp32 copy and cute_dsl_indexer_q_side_fused for +# ssmax_indexer_q_side_fused; a fresh RMSNorm and torch.cat for the in-place latent; the copy +# out of FlashMLA's 64-head output for the in-place read. Set +# TRTLLM_MODELING_V2_UNFUSED_GLUE=1 in every rank's environment at launch; off by default. +# It sets the core's `unfused_glue`, which every site reads. +_UNFUSED_GLUE = os.environ.get("TRTLLM_MODELING_V2_UNFUSED_GLUE", "0") == "1" +# A generation step's shared expert on the main stream after the routed experts, instead of on +# the decode target's side stream beside the router, the NVFP4 quantization and the runner. +# Both orders compute the same bits. The side stream is used only under CUDA-graph capture +# (`maybe_execute_in_parallel`). Set TRTLLM_MODELING_V2_SERIAL_SHARED_EXPERT=1 in every rank's +# environment at launch; off by default. +_SERIAL_SHARED_EXPERT = os.environ.get("TRTLLM_MODELING_V2_SERIAL_SHARED_EXPERT", "0") == "1" + + +def _prefill_logits_budget(max_num_tokens: int, max_seq_len: int) -> int: + """The most fp32 logits one indexer prefill tile holds, from the engine's max_num_tokens and + max_seq_len: mainline's reachable MQA-logits tile (`get_indexer_mqa_logits_workspace_bytes`, + 4 bytes a logit; TLLM_INDEXER_MQA_LOGITS_ELEM_BUDGET lowers it), within + `_PREFILL_LOGITS_CEILING`, and within max_seq_len / 64 query rows at max_seq_len keys. + + The last bound grows the tile with the engine's longest request, which it scores in at + least 64 tiles: 2**24 logits (512 rows of 32768 keys) at max_seq_len 32768, the ceiling + (256 rows of 1048576 keys) from max_seq_len 131072 on. Those are the budgets the 32768 and + 1048576 serving configs were certified at, so their tiles, and the fp8_mqa_logits and + prefill top-k cells that cover them, stay as they were.""" + reachable = get_indexer_mqa_logits_workspace_bytes(max_num_tokens, max_seq_len) // 4 + return min(reachable, _PREFILL_LOGITS_CEILING, max_seq_len * (max_seq_len >> 6)) + + +def _most_ctx_kv(md) -> int: + """The most context KV (cached and new rows over a step's context requests) one step can + attend, from the attention metadata alone. `runtime_features` is built once by the executor + for the service (chunked prefill from enable_chunked_prefill, cache reuse from the + KvCacheConfig's enable_block_reuse). With neither, every context row is new: + max_num_tokens. Otherwise mainline's never-stall worst case for a workspace chunked prefill + does not bound (`get_mla_context_workspace_kv_len_cap` without an + fp8_context_mla_kv_len_cap override; the gather reads every attended row, cached ones + included): min(max_num_requests, max_num_tokens) * max_seq_len. The target does not read + fp8_context_mla_kv_len_cap: the KvCacheConfig is not reachable from the model, and for these + targets the engine installs no admission cap that would bound a step to it.""" + features = md.runtime_features + if not (features.chunked_prefill or features.cache_reuse): + return md.max_num_tokens + return min(md.max_num_requests, md.max_num_tokens) * md.max_seq_len + + +def _ctx_workspace_rows(md) -> int: + """The held context workspace's rows: the most context KV one step can attend + (`_most_ctx_kv`) twice over, a step's rows and a shared owner's (layer 35's) kept beside + them, up to `_CTX_WORKSPACE_ROWS`. At max_num_tokens 32768 with neither chunked prefill nor + block reuse that is 65536 rows, the two `[32768, 576]` gathers a step there makes; with either, + at 16 requests of max_seq_len 32768 or more, 2**19.""" + return min(2 * _most_ctx_kv(md), _CTX_WORKSPACE_ROWS) + + +def _step_common(md: TrtllmAttentionMetadata, indexer_layers: list[int]) -> dict: + """The batch state both targets read: KV lengths and pool addressing for the MLA ops, the + indexer's key pools and slot mappings, the top-k conversion tables and the decode scorer's + schedule. The indexer pool views are taken per step: the engine replaces its KV cache + manager during boot, and a view of the old one writes into a dead allocation.""" + mgr = md.kv_cache_manager + latent_pool = mgr.get_latent_pool() + return dict( + num_seqs=md.num_seqs, + num_tokens=md.num_tokens, + kv_lens_cuda=md.kv_lens_cuda_runtime, + kv_lens_host=md.kv_lens_runtime, + prompt_lens_host=md.prompt_lens_cpu_runtime, + kv_cache_block_offsets=md.kv_cache_block_offsets, + host_kv_cache_pool_pointers=md.host_kv_cache_pool_pointers, + host_kv_cache_pool_mapping=md.host_kv_cache_pool_mapping, + tokens_per_block=md.tokens_per_block, + max_seq_len=md.max_seq_len, + # [blocks, owners, tokens_per_block, 656] bytes, and the same bytes as FlashMLA's + # pages: page `block * owners + row`, slot `page * tokens_per_block + offset`. + latent_pool=latent_pool, + latent_kv=latent_pool.view(-1, latent_pool.shape[2], 1, latent_pool.shape[3]), + ix_pools={i: mgr.get_indexer_k_cache_buffers(i) for i in indexer_layers}, + block_table=md.block_table, + req_idx_per_token=md.req_idx_per_token, + slot_mapping_fp8=md.slot_mapping_fp8, + slot_mapping_scale=md.slot_mapping_scale, + skip_gen=bool(md.skip_indexer_for_gen_reqs), + # Allocated only while the engine's short-sequence indexer skip is on; the two skip + # flags are False whenever it is off. + topk_indices_buffer=md.topk_indices_buffer if md.enable_indexer_skip else None, + indexer_k_cache_block_offsets=md.indexer_k_cache_block_offsets, + scheduler_metadata_buffer=md.scheduler_metadata_buffer, + radix_aux_indices=md.radix_aux_indices, + radix_aux_logits=md.radix_aux_logits, + ) + + +def _ctx_kv_offsets(md: TrtllmAttentionMetadata, num_contexts: int) -> dict: + """The context rope/append op's per-request offsets: int64 prefix sums of the cached and + of the total (cached + new) lengths over the context requests, and the longest new run. + + The engine keeps them only under `enable_context_mla_with_cached_kv` (KV reuse or chunked + prefill). Without either, no context request has a cached prefix, so the cached sums are + zero and the totals are the new lengths: the int32 prefix sum of `seq_lens_cuda` the + metadata always builds, widened to the op's int64. + """ + if md.enable_context_mla_with_cached_kv: + return dict( + cu_ctx_cached=md.ctx_cached_token_indptr[: num_contexts + 1], + cu_ctx_kv=md.ctx_kv_indptr[: num_contexts + 1], + max_ctx_new=int(md.max_ctx_seq_len), + ctx_cached_tokens=int(md.num_ctx_cached_tokens), + ) + cu = md.mla_prepare_ctx_cu_seqlens() + return dict( + cu_ctx_cached=torch.zeros(num_contexts + 1, dtype=torch.int64, device=cu.device), + cu_ctx_kv=cu.to(torch.int64), + max_ctx_new=int(md.seq_lens[:num_contexts].max()), + ctx_cached_tokens=0, + ) + + +def _ctx_host_offsets(md: TrtllmAttentionMetadata, num_contexts: int) -> dict: + """The host lists over the context requests: the prefix sums of the new and of the total + (cached + new) lengths, which place each context request's rows in the step and its keys in + the indexer's full-KV gather mappings (`_indexer_blocks`). + + The engine's host copies under `enable_context_mla_with_cached_kv`; without it no request + has a cached prefix, so both are the prefix sum of the host `seq_lens`. Every source is a + host tensor the metadata fills on the host, so taking them each context step syncs nothing. + """ + if md.enable_context_mla_with_cached_kv: + return dict( + host_cu_ctx_new=md.host_ctx_uncached_token_indptr[: num_contexts + 1].tolist(), + host_cu_ctx_kv=md.host_ctx_kv_indptr[: num_contexts + 1].tolist(), + ) + host_cu = [0, *itertools.accumulate(md.seq_lens[:num_contexts].tolist())] + return dict(host_cu_ctx_new=host_cu, host_cu_ctx_kv=host_cu) + + +def _indexer_blocks(chunks, step: dict) -> list: + """The context indexer's scoring blocks for one step, in the engine's chunk order: rows + `[token_start, token_end)` of the context rows scored against keys + `[k_token_start, k_token_end)` of the full-KV gather mappings, each row's window + (`cu_seqlen_ks` / `cu_seqlen_ke`) counted from the block's first key. + + An engine prefill chunk is one block, as planned, while it holds at most `max_seq_len` keys, + the most one request holds. The engine packs consecutive context requests into one chunk + while (summed new tokens) x (summed KV) stays within its packed-logits budget (2^28), so + short last chunks of long requests pack past max_seq_len: a 100-token tail of a + 1048559-token request beside a 150-token prompt is one chunk of 1048709 keys, wider than + the key scorer's and the prefill top-k's cells. Such a chunk is cut at its request + boundaries (`_ctx_host_offsets`), each request's rows against its own keys with their + windows re-based to its first key: a row's window lies inside its own request's keys, so + each request selects as it would alone and no block holds more than max_seq_len keys. + """ + tc = step["num_ctx_tokens"] + if chunks is None: + # One chunk over every context row: the engine plans chunks only past its chunk + # size, or under KV reuse or chunked prefill. + chunks = [ + SimpleNamespace( + token_start=0, + token_end=tc, + k_token_start=0, + k_token_end=step["num_ctx_kv_tokens"], + cu_seqlen_ks=step["cu_seqlen_ks"][:tc], + cu_seqlen_ke=step["cu_seqlen_ke"][:tc], + ) + ] + max_keys = step["max_seq_len"] + cu_new, cu_kv = step["host_cu_ctx_new"], step["host_cu_ctx_kv"] + blocks = [] + for chunk in chunks: + if chunk.k_token_end - chunk.k_token_start <= max_keys: + blocks.append(chunk) + continue + for r in range(len(cu_new) - 1): + lo, hi = max(chunk.token_start, cu_new[r]), min(chunk.token_end, cu_new[r + 1]) + if lo >= hi: + continue + k_lo, k_hi = max(chunk.k_token_start, cu_kv[r]), min(chunk.k_token_end, cu_kv[r + 1]) + assert k_hi - k_lo <= max_keys, ( + f"context request {r} holds keys [{k_lo}, {k_hi}) of an indexer chunk, past " + f"max_seq_len {max_keys}" + ) + shift = k_lo - chunk.k_token_start + rows = slice(lo - chunk.token_start, hi - chunk.token_start) + blocks.append( + SimpleNamespace( + token_start=lo, + token_end=hi, + k_token_start=k_lo, + k_token_end=k_hi, + cu_seqlen_ks=torch.sub(chunk.cu_seqlen_ks[rows], shift), + cu_seqlen_ke=torch.sub(chunk.cu_seqlen_ke[rows], shift), + ) + ) + return blocks + + +def _yarn_lerope_inv_freq(cfg, alpha: torch.Tensor) -> torch.Tensor: + """The rotary inverse frequencies, fp32, as the reference implementation builds + them: YaRN's blend of the extrapolated and the interpolated frequency, both scaled by + LeRoPE's `exp(alpha)`, over a ramp taken on the frequencies without LeRoPE.""" + rope = cfg.rope_parameters + dim = cfg.qk_rope_head_dim + theta = float(rope["rope_theta"]) + factor = float(rope["factor"]) + original = float(rope["original_max_position_embeddings"]) + pos_freqs = theta ** (torch.arange(0, dim, 2, dtype=torch.float32) / dim) + lerope = torch.exp(alpha.detach().to(device="cpu", dtype=torch.float32)) + extrapolated = 1.0 / pos_freqs * lerope + interpolated = 1.0 / (factor * pos_freqs) * lerope + + def correction_dim(rotations: float) -> float: + return dim * math.log(original / (rotations * 2 * math.pi)) / (2 * math.log(theta)) + + low = max(math.floor(correction_dim(float(rope.get("beta_fast", 32)))), 0) + high = min(math.ceil(correction_dim(float(rope.get("beta_slow", 1)))), dim - 1) + if low == high: + high += 0.001 + ramp = ((torch.arange(dim // 2, dtype=torch.float32) - low) / (high - low)).clamp(0, 1) + extrapolation = 1 - ramp + return interpolated * (1 - extrapolation) + extrapolated * extrapolation + + +def _rope_tables(cfg, alpha: torch.Tensor, positions: int, dtype, device) -> SimpleNamespace: + """The tables the rope consumers read, built from one fp32 angle grid and rounded to the + activation dtype first, as the reference's cos/sin cache is held. + + `indexer` is `[rows, 64]`, `[cos(32) | sin(32)]`: the layout the indexer q side and + `ds_mla_rope_append` (the MLA query tail and the latent's rope key) read; the indexer key + side reads the same storage as `[rows, 2, 32]`. `identity` (cos 1, sin 0) is the table of a + NoPE indexer layer, whose kernels have no no-rope mode. Every row depends only on its own + index, so a longer table is the same table with more rows. + """ + inv = _yarn_lerope_inv_freq(cfg, alpha) + t = torch.arange(positions, dtype=torch.float32) + angle = torch.einsum("i,j -> ij", t, inv) + cos = angle.cos().to(dtype).float() + sin = angle.sin().to(dtype).float() + identity = torch.cat([torch.ones_like(cos), torch.zeros_like(sin)], dim=1) + return SimpleNamespace( + rows=positions, + indexer=torch.cat([cos, sin], dim=1).to(device).contiguous(), + identity=identity.to(device).contiguous(), + ) + + +def _moe_scalars(isc1, isc2, ws2_13, ws2_2) -> SimpleNamespace: + """The NVFP4 MoE's call scalars from the checkpoint's per-tensor ones, fp32. + + ModelOpt stores `input_scale` and `weight_scale_2` as the reciprocals of the global scales a + quantizer divides by (`amax / (448 * 6)`), one `isc1` / `isc2` per layer and one + `ws2_13[e]` / `ws2_2[e]` per local expert. So the activation quantizer's global scale is + `1 / isc1`; FC1's accumulator dequantizes by `isc1 * ws2_13[e]` (the gate half, + `output1_scale_gate_scalar`) and, because the FC1 epilogue requantizes the SwiGLU output to + NVFP4 under `1 / isc2`, the linear half by `isc1 * ws2_13[e] / isc2` + (`output1_scale_scalar`); FC2's accumulator dequantizes by `isc2 * ws2_2[e]` + (`output2_scale_scalar`). The reference's TRT-LLM-Gen path folds the same three + (`g1_alphas`, `g1_scale_c`, `g2_alphas`).""" + gate1 = (isc1 * ws2_13).contiguous() + return SimpleNamespace( + gscale=(1.0 / isc1).contiguous(), + out1=(gate1 / isc2).contiguous(), + out1_gate=gate1, + out2=(isc2 * ws2_2).contiguous(), + ) + + +def _layer_views(core: "MistralLarge4TpModelingV2") -> list[SimpleNamespace]: + """Every layer's operands, as the views the entries read: GEMM operands transposed (`.t()`, + the column-major `[K, N]` cuBLAS operand), FP8 scales converted once to + the Rubin R128c4 K32 layout, `wkv_b` split into the absorption pair, and the NVFP4 MoE's call + scalars (`_moe_scalars`). `next_norm` is the next layer's input norm, the final norm after + the last layer. A sharer's absorption pair is its own `wkv_b`; its latent is its source's.""" + w = core.w + heads, v_dim, kv_lora = core.heads, core.v_dim, core.kv_lora + layers = [] + for i in range(core.num_layers): + p = f"l{i}_" + nope = core.qk_dim if core.nope_layer[i] else core.nope + kvb = w[p + "wkv_b"] + hn = heads * nope + owner = core.kv_source[i] == i + ix = None + if core.full_indexer[i]: + ix = SimpleNamespace( + q_norm_w=w[p + "ix_q_norm_w"], + q_norm_b=w[p + "ix_q_norm_b"], + wq_b=w[p + "ix_wq_b"], + wq_b_s=prepare_weight_scale(w[p + "ix_wq_b_s"].t(), w[p + "ix_wq_b"].shape[1]), + wk_wp=w[p + "ix_wk_wp"].t(), + k_norm_w=w[p + "ix_k_norm_w"], + k_norm_b=w[p + "ix_k_norm_b"], + ssmax_s=w[p + "ix_ssmax_s"], + ssmax_b=w[p + "ix_ssmax_b"], + rope=not core.indexer_nope[i], + ) + moe = None + if i >= core.dense_layers: + moe = SimpleNamespace( + router=w[p + "router"].t(), + fc1_w=w[p + "fc1_w"], + fc1_s=w[p + "fc1_s"], + fc2_w=w[p + "fc2_w"], + fc2_s=w[p + "fc2_s"], + **vars(_moe_scalars(w[p + "isc1"], w[p + "isc2"], w[p + "ws2_13"], w[p + "ws2_2"])), + ) + layers.append( + SimpleNamespace( + index=i, + rope=not core.nope_layer[i], + nope=nope, + owner=owner, + kv_source=core.kv_source[i], + attn_norm=w[p + "attn_norm"], + wq=w[p + "wq"], + wq_s=prepare_weight_scale(w[p + "wq_s"].t(), w[p + "wq"].shape[1]), + ssmax_s=w[p + "ssmax_s"], + ssmax_b=w[p + "ssmax_b"], + wkv_a=w[p + "wkv_a"].t() if owner else None, + kv_norm=w[p + "kv_a_norm"] if owner else None, + k_b=kvb[:hn].reshape(heads, nope, kv_lora), + v_b_t=torch.transpose(kvb[hn:].reshape(heads, v_dim, kv_lora), 1, 2), + wo=w[p + "wo"], + wo_s=prepare_weight_scale(w[p + "wo_s"].t(), w[p + "wo"].shape[1]), + ffn_norm=w[p + "ffn_norm"], + mlp_gu=w[p + "mlp_gu"], + mlp_gu_s=prepare_weight_scale(w[p + "mlp_gu_s"].t(), w[p + "mlp_gu"].shape[1]), + mlp_dn=w[p + "mlp_dn"], + mlp_dn_s=prepare_weight_scale(w[p + "mlp_dn_s"].t(), w[p + "mlp_dn"].shape[1]), + indexer=ix, + moe=moe, + next_norm=( + w[f"l{i + 1}_attn_norm"] if i + 1 < core.num_layers else w["final_norm"] + ), + ) + ) + return layers + + +class MistralLarge4TpModelingV2(DecoderModel): + def __init__(self, model_config: ModelConfig): + super().__init__(model_config) + cfg = model_config.pretrained_config + mapping = model_config.mapping + self.tp_rank = mapping.tp_rank + self.tp_size = mapping.tp_size + # The context selection's gather group: the TP ranks, in the order the engine sizes + # their query shares and the gather concatenates them. + self.tp_group = list(mapping.tp_group) + # The attention block's glue: the fused entries (ssmax_scaling_fused_q, + # ssmax_indexer_q_side_fused, the latent RMSNorm in place, FlashMLA's decode output + # read in place) at every max_seq_len, whose cells cover the 32K and 1M configs' + # positions (to 1048575), rope tables and step shapes; the separate ops they replace + # under `_UNFUSED_GLUE`, at every site. + self.unfused_glue = _UNFUSED_GLUE + # The generation rows' sparse MLA: cute_dsl_ds_mla_sparse_decode at the rank's heads at + # every max_seq_len, whose cells cover the 32K and 1M configs' pools and key lengths; + # flash_mla_sparse_decode over the 64-head padded query under `_GEN_ATTN_FLASH_MLA`. + self.gen_attn_flash_mla = _GEN_ATTN_FLASH_MLA + + self.num_layers = cfg.num_hidden_layers + self.hidden = cfg.hidden_size + self.eps = cfg.rms_norm_eps + self.heads = cfg.num_attention_heads // mapping.tp_size + self.nope = cfg.qk_nope_head_dim + self.rope_dim = cfg.qk_rope_head_dim + # A NoPE layer spends the rope dims on nope: the query head stays this wide. + self.qk_dim = self.nope + self.rope_dim + self.v_dim = cfg.v_head_dim + self.kv_lora = cfg.kv_lora_rank + self.lat_dim = self.kv_lora + self.rope_dim + # Every layer's softmax scale, NoPE layers included: qk_head_dim**-0.5, no YaRN + # temperature (its magnitude is 1). + self.softmax_scale = self.qk_dim**-0.5 + + self.nope_layer = [bool(f) for f in cfg.nope_layer_flags] + self.indexer_nope = [bool(f) for f in cfg.indexer_nope_layer_flags] + self.kv_source = [int(s) for s in cfg.kv_source_layers] + # An owner some later layer attends over (layer 35): a step whose context KV fits the + # held workspace twice keeps its gather there for the sharers; a longer one has each + # sharer gather its source's rows again (`_ctx_workspace`). + self.shared_source = [ + any(s == i and j != i for j, s in enumerate(self.kv_source)) + for i in range(self.num_layers) + ] + self.full_indexer = [role == "full" for role in cfg.index_sharing] + # A 'shared' layer attends with the most recent full layer's selection: each target + # keeps one selection buffer per step, which every full layer overwrites in order. + self.indexer_layers = [i for i in range(self.num_layers) if self.full_indexer[i]] + + self.idx_heads = cfg.index_n_heads + self.idx_dim = cfg.index_head_dim + self.idx_topk = cfg.index_topk + self.idx_eps = cfg.indexer_norm_eps + # The indexer's softmax scale and head average: its scorers take none, so the q side + # folds both into the head weights together with the q quantization scale. + self.idx_weight_scale = self.idx_dim**-0.5 * self.idx_heads**-0.5 + + self.dense_layers = cfg.first_k_dense_replace + self.num_experts = cfg.n_routed_experts + self.topk = cfg.num_experts_per_tok + # The routed experts' tensor-parallel split: every expert this rank answers for (all of + # them at moe_ep_size 1, at offset 0), each at its moe_intermediate_size / moe_tp_size + # slice of the intermediate -- the width the weight table declares and the runner runs. + self.moe_inter = cfg.moe_intermediate_size // mapping.moe_tp_size + self.local_experts = self.num_experts // mapping.moe_ep_size + self.expert_offset = self.local_experts * mapping.moe_ep_rank + # capacity: max_num_tokens -- the collective's peer buffers and the MoE autotuner's + # largest bucket. + self.max_num_tokens = model_config.max_num_tokens + # The rope tables' first size; grown on the first step to the engine's max_seq_len. + self.rope_rows = model_config.max_seq_len or cfg.max_position_embeddings + + self.w = _weights.MODEL_WEIGHTS.declare(_weights.MODEL_WEIGHTS.dims(self)) + + self._rope: SimpleNamespace | None = None + self._derived_for = None + self._ix_k = None + self._ix_k_checked = False + self._pool_row: list[int] = [] + self._stride_factor = 0 + self._max_ctx_len = 0 + self.tp_pg = None + # The engine's sizes, fixed on the first step (`_derive_first_step`). + self.prefill_logits_budget = 0 + self.ctx_workspace_rows = 0 + # The decode top-k's handle, built on the first step at a scorer width it serves. + self._ix_topk = None + self._ix_topk_verify = None + self._prefill = self._decode = None + + def post_load_weights(self) -> None: + """Derive what the weights fix, build the kernels and the collectives, then the + targets. + + Named for the shell hook that drives it. Everything compiles here or on the first + (eager) step: a first compile inside a CUDA-graph capture breaks the graph. + """ + dev = torch.cuda.current_device() + self.build_kernels(dev) + # Collective: every rank builds it with the same arguments. The peer buffers hold + # the whole token budget, so one workspace serves every closing collective. + self._lamport = build_lamport_workspace( + self.hidden, self.eps, self.max_num_tokens, self.tp_rank, self.tp_size + ) + # The selection gather's communicator flavor, as the engine's own collectives pick it: + # the MPI session's, or the TP process group's when MPI is disabled. + mapping = self.model_config.mapping + self.tp_pg = mapping.tp_group_pg.boxed() if mpi_disabled() else None + self._prefill = PrefillTarget(self) + self._decode = DecodeTarget(self) + + def build_kernels(self, dev: int) -> None: + """The rank-local derivations: the rope tables, the indexer q side for the unfused glue, + and the generation rows' sparse decode at this rank's heads.""" + self._rope = _rope_tables( + self.model_config.pretrained_config, + self.w["lerope_alpha"], + self.rope_rows, + self.w["final_norm"].dtype, + self.w["final_norm"].device, + ) + if self.unfused_glue: + # The checkpoint's rope is interleaved (GPT-J) for the indexer too. + self._ix_q = build_ix_q_side(dev, is_neox=False) + self._mla_decode = ( + None if self.gen_attn_flash_mla else build_ds_mla_decode(dev, num_heads=self.heads) + ) + + def _derive_first_step(self, md) -> None: + """What depends on values the engine resolves, fixed on the first step it hands over + (eager, ahead of any capture) and again for a replaced KV cache manager.""" + if md.kv_cache_manager is self._derived_for: + return + dev = torch.cuda.current_device() + mgr = md.kv_cache_manager + # The pool layout is the KV cache manager's, which routing cannot see; every latent + # op here reads and writes fp8_ds_mla rows addressed by owner row. + if getattr(mgr, "latent_layout", None) != _LATENT_LAYOUT: + raise RuntimeError( + f"this target attends over an {_LATENT_LAYOUT} latent pool with owner-only " + f"pages (MistralLarge4CacheManager); the engine built {type(mgr).__name__} " + f"with layout {getattr(mgr, 'latent_layout', None)!r}. Use KV cache manager V2 " + "and kv_cache_config.dtype auto, fp8 or fp8_ds_mla." + ) + if list(mgr.kv_source_layers) != self.kv_source: + raise RuntimeError("the KV cache manager's kv_source_layers differ from the config's") + # A table shorter than max_seq_len is read past its end with no check. + if md.max_seq_len > self._rope.rows: + self.rope_rows = md.max_seq_len + self._rope = _rope_tables( + self.model_config.pretrained_config, + self.w["lerope_alpha"], + self.rope_rows, + self.w["final_norm"].dtype, + self.w["final_norm"].device, + ) + # Layer i reads owner row `_pool_row[i]` (its source's); a block holds every owner's + # page, so a global row is `block * owners * tpb + row * tpb + offset`. + self._pool_row = [mgr.latent_pool_row(i) for i in range(self.num_layers)] + self._stride_factor = len(mgr.kv_owner_layers) * md.tokens_per_block + # The decode scorer's column count: frozen into a capture, so an engine constant. + self._max_ctx_len = md.max_seq_len + # The indexer's prefill logits budget and the held context workspace's rows, from the + # engine's max_num_tokens, max_num_requests, max_seq_len and runtime features. + self.prefill_logits_budget = _prefill_logits_budget(md.max_num_tokens, md.max_seq_len) + self.ctx_workspace_rows = _ctx_workspace_rows(md) + logger.info( + f"modeling_v2 Mistral Large 4 NVFP4: sized from the engine: prefill logits budget " + f"{self.prefill_logits_budget}, context workspace {self.ctx_workspace_rows} rows" + ) + # The decode selector, by the scorer's column count: cute_dsl_radix_topk_decode's + # handle (built once) at a width it serves, else none and the trtllm op selects. + lo, hi = _TOPK_DECODE_COLUMNS + if not (_TOPK_DECODE_CUTE_DSL and lo <= self._max_ctx_len <= hi): + self._ix_topk = None + elif self._ix_topk is None: + self._ix_topk = build_ix_topk_decode(dev, next_n=1) + lo, hi = _TOPK_VERIFY_COLUMNS + if not (_TOPK_DECODE_CUTE_DSL and lo <= self._max_ctx_len <= hi and self.idx_topk == 2048): + self._ix_topk_verify = None + elif self._ix_topk_verify is None: + self._ix_topk_verify = build_ix_topk_verify( + dev, next_n=4, use_pdl=os.environ.get("TRTLLM_ENABLE_PDL") != "0" + ) + # Both key-pool layouts: a launcher compiled for a contiguous view addresses a strided + # one as if it were dense. + if self._ix_k is None: + self._ix_k = { + contiguous: build_ix_k_cache(dev, is_neox=False, cache_contiguous=contiguous) + for contiguous in (False, True) + } + self._derived_for = md.kv_cache_manager + + def _select_indexer_decode( + self, + logits: torch.Tensor, + kv_lens: torch.Tensor, + indices: torch.Tensor, + next_n: int, + aux_indices: torch.Tensor, + aux_logits: torch.Tensor, + ) -> torch.Tensor: + """Select generation rows with the compiled selector for their static shape.""" + lo, hi = _TOPK_VERIFY_COLUMNS + if ( + self._ix_topk_verify is not None + and next_n == 4 + and logits.shape[0] in _TOPK_VERIFY_ROWS + and lo <= logits.shape[1] <= hi + and logits.dtype == torch.float32 + ): + return split_radix_topk_decode( + logits, kv_lens, indices, aux_indices, aux_logits, compiled=self._ix_topk_verify + ) + if next_n != 1 or self._ix_topk is None: + return indexer_topk_decode( + logits, kv_lens, indices, next_n, self.idx_topk, 1, aux_indices, aux_logits + ) + return cute_dsl_radix_topk_decode( + logits, kv_lens, indices, next_n=1, compiled=self._ix_topk + ) + + def _select_target(self, attn_metadata) -> "PrefillTarget | DecodeTarget": + """Which target runs this step: one host-side integer read. A mixed batch goes to + prefill, so prefill is the general case and decode the specialization that may assume + it has no context rows.""" + return self._decode if attn_metadata.num_contexts == 0 else self._prefill + + def forward( + self, + attn_metadata: AttentionMetadata, + input_ids: torch.IntTensor | None = None, + position_ids: torch.IntTensor | None = None, + inputs_embeds: torch.FloatTensor | None = None, + lora_params: dict | None = None, + **kwargs, + ) -> torch.Tensor: + """Derive what the first step fixes, then route.""" + self._derive_first_step(attn_metadata) + return self._select_target(attn_metadata).forward( + attn_metadata, + input_ids=input_ids, + position_ids=position_ids, + inputs_embeds=inputs_embeds, + lora_params=lora_params, + **kwargs, + ) + + +class PrefillTarget: + """The general case: context rows, and possibly generation rows behind them.""" + + # The held context workspace, bf16 [core.ctx_workspace_rows, 576]: allocated on the first + # context step and held from then on (`_ctx_workspace`). + _ws_buf: torch.Tensor | None = None + # The indexer's prefill top-k scratch (`gvr_workspace`, 20 MiB): allocated on the first + # context step that selects and held, used on the main stream only (`_select_ctx`). + _gvr_ws: torch.Tensor | None = None + + def __init__(self, core: MistralLarge4TpModelingV2) -> None: + """Take the per-layer views once, off the real weights. Everything an engine value + fixes is read off `self.core` at the call.""" + self.core = core + # Off unless TRTLLM_MODELING_V2_VALIDATE asks for it. Each target owns its own flag: + # the two `step_args` projections differ, so probing one does not prove the other. + self._contract_pending = step_contract_enabled() + self._layers = _layer_views(core) + self._embed = core.w["embed"] + self._norm0 = core.w["l0_attn_norm"] + + def _check_step_contract(self, md) -> None: + """Opt-in first-forward fail-fast: calling `step_args` reads every metadata field this + target consumes (private trtllm surface). Fixed at engine construction, so once per + target instance, and off in a served engine.""" + if not self._contract_pending: + return + try: + self.step_args(md) + except AttributeError as exc: + raise AssertionError(f"attention metadata surface drifted: {exc}") from exc + self._contract_pending = False + + def step_args(self, md: TrtllmAttentionMetadata) -> dict: + """The common batch state, plus the context rows': their counts and lengths, the + indexer's prefill windows and scoring blocks, its query split over the TP ranks and its + key gather mappings, and the context rope/append op's offsets and their host prefix + sums.""" + step = _step_common(md, self.core.indexer_layers) + ctx = md.num_contexts + step.update( + num_contexts=ctx, + num_ctx_tokens=md.num_ctx_tokens, + seq_lens=md.seq_lens, + skip_ctx=bool(md.skip_indexer_for_ctx_reqs), + cu_seqlen_ks=md.cu_seqlen_ks, + cu_seqlen_ke=md.cu_seqlen_ke, + indexer_prefill_chunks=md.indexer_prefill_chunks, + # (q_begin, q_end, per-rank row counts) once a chunked-prefill step's context rows + # reach the engine's q_split_threshold, None below it: the chunks then cover this + # rank's rows [q_begin, q_end) only. + indexer_prefill_q_split=md.indexer_prefill_q_split, + num_ctx_kv_tokens=md.num_ctx_kv_tokens, + slot_mapping_fp8_fullkv=md.slot_mapping_fp8_fullkv, + slot_mapping_scale_fullkv=md.slot_mapping_scale_fullkv, + **_ctx_kv_offsets(md, ctx), + **_ctx_host_offsets(md, ctx), + ) + step["indexer_blocks"] = _indexer_blocks(md.indexer_prefill_chunks, step) + return step + + # ---- one layer's pieces, context and generation rows ---- + + def _linear_fp8(self, x, w, w_s): + """K128-quantized FP8 linear through the Rubin MXFP8 kernel.""" + return cute_dsl_fp8_block_scale_linear(x, w, w_s) + + def _latent(self, x, L, nope_lat): + """`[RMSNorm(c_kv) | k_pe]`, 576 wide: a RoPE owner's kv_a row, normalized in place; a + NoPE owner's 512-wide latent normalized into `nope_lat`, whose rope key stays zero.""" + core = self.core + kv = cublas_mm(x, L.wkv_a) + if core.unfused_glue: + c_kv = flashinfer_rmsnorm(kv[:, : core.kv_lora], L.kv_norm, core.eps) + if L.rope: + k_pe = kv[:, core.kv_lora :] + else: + k_pe = torch.zeros([kv.shape[0], core.rope_dim], dtype=kv.dtype, device=kv.device) + return torch.cat([c_kv, k_pe], dim=-1) + if L.rope: + c_kv = kv[:, : core.kv_lora] + flashinfer_rmsnorm(c_kv, L.kv_norm, core.eps, out=c_kv) + return kv + flashinfer_rmsnorm(kv, L.kv_norm, core.eps, out=nope_lat[:, : core.kv_lora]) + return nope_lat + + def _indexer(self, step, x, pos, L, topk_buf, tc: int, skip_ctx: bool, skip_gen: bool): + """Append this step's keys to the layer's key pool; select `index_topk` positions per + row into `topk_buf` for the rows whose phase the engine did not skip. Under the + engine's query split this rank selects for its share of the context rows, and the + shares are gathered into rows [0, tc) before any layer reads them.""" + core = self.core + ix = L.indexer + rows = x.shape[0] + table = core._rope.indexer if ix.rope else core._rope.identity + kw = cublas_mm(x, ix.wk_wp) + k = kw[:, : core.idx_dim].contiguous() + k_pool = step["ix_pools"][L.index] + k_pool = cute_dsl_indexer_k_cache_update( + k, ix.k_norm_w, ix.k_norm_b, pos, table.view(-1, 2, core.rope_dim // 2), k_pool, + step["slot_mapping_fp8"], step["slot_mapping_scale"], rows, + is_neox=False, compiled=core._ix_k[k_pool.is_contiguous()], + # Its range check is a host sync: once, on the first (eager) call. + validate=not core._ix_k_checked, + ) # fmt: skip + core._ix_k_checked = True + if skip_ctx and skip_gen: + return + qn = layer_norm(x, ix.q_norm_w, ix.q_norm_b, core.idx_eps) + q = torch.reshape(self._linear_fp8(qn, ix.wq_b, ix.wq_b_s), [rows, core.idx_heads, -1]) + q_fp8 = torch.empty_like(q, dtype=torch.float8_e4m3fn) + weights = torch.empty([rows, core.idx_heads], dtype=torch.float32, device=x.device) + if core.unfused_glue: + q = ssmax_scaling(q, pos, ix.ssmax_s, ix.ssmax_b) + q_fp8, weights = cute_dsl_indexer_q_side_fused( + q, pos, table, kw[:, core.idx_dim :].float().contiguous(), core.idx_weight_scale, + q_fp8, weights, is_neox=False, compiled=core._ix_q, + ) # fmt: skip + else: + # SSMax, rope, e4m3 and the head weights (read in place out of `kw`) in one launch. + q_fp8, weights = ssmax_indexer_q_side_fused( + q, pos, ix.ssmax_s, ix.ssmax_b, table, kw[:, core.idx_dim :], + core.idx_weight_scale, q_fp8, weights, + ) # fmt: skip + if tc and not skip_ctx: + self._select_ctx(step, k_pool, q_fp8, weights, topk_buf) + q_split = step["indexer_prefill_q_split"] + if q_split is not None: + # The chunks covered this rank's share only. The split depends on the step's + # counts alone, so every rank of the group makes this gather once per selecting + # layer, in layer order, at the same sizes: the communicator pairs calls by + # position. + q_begin, q_end, q_sizes = q_split + topk_buf[:tc].copy_( + allgather(topk_buf[q_begin:q_end], q_sizes, core.tp_group, core.tp_pg) + ) + gen = rows - tc + if gen and not skip_gen: + ctx = step["num_contexts"] + num_gens = step["num_seqs"] - ctx + assert gen % num_gens == 0, (gen, num_gens) + next_n = gen // num_gens + kv_lens = step["kv_lens_cuda"][ctx : ctx + num_gens] + logits = cute_dsl_fp8_paged_mqa_logits( + torch.reshape(q_fp8[tc:], [num_gens, next_n, core.idx_heads, core.idx_dim]), + k_pool, + weights[tc:], + kv_lens, + step["indexer_k_cache_block_offsets"][ctx : ctx + num_gens], + step["scheduler_metadata_buffer"], + core._max_ctx_len, + ) + core._select_indexer_decode( + logits, kv_lens, topk_buf[tc:], next_n, + step["radix_aux_indices"], step["radix_aux_logits"], + ) # fmt: skip + + def _ix_plan(self, step) -> list[SimpleNamespace]: + """Plan the context rows and key windows scored by each query tile. + + The engine's chunk and query-split mappings determine each block's rows, + keys and window offsets. Tiles stay within the prefill logits budget. + The plan is reused across indexer layers within this step.""" + core = self.core + plan = [] + for block in step["indexer_blocks"]: + n_k = block.k_token_end - block.k_token_start + if block.token_end <= block.token_start: + continue + tiles = [] + if n_k > 0: + tile = max(1, core.prefill_logits_budget // n_k) + for lo in range(block.token_start, block.token_end, tile): + hi = min(lo + tile, block.token_end) + # The windows are the block's own, indexed from its first row. + ks = block.cu_seqlen_ks[lo - block.token_start : hi - block.token_start] + ke = block.cu_seqlen_ke[lo - block.token_start : hi - block.token_start] + tiles.append((lo, hi, ks, ke)) + plan.append( + SimpleNamespace( + token_start=block.token_start, + token_end=block.token_end, + k_token_start=block.k_token_start, + n_k=n_k, + tiles=tiles, + ) + ) + return plan + + def _select_ctx(self, step, k_pool, q_fp8, weights, topk_buf) -> None: + """Select window-local top-k keys for the context rows assigned to this rank. + + Gather each block's FP8 keys, score each query tile with the Triton MQA + scorer, and pass its aligned logits to GVR top-k. Empty blocks retain + -1 indices. Only one tile's logits are live at a time.""" + core = self.core + plan = step.get("ix_plan") + if plan is None: + plan = step["ix_plan"] = self._ix_plan(step) + if self._gvr_ws is None: + # On the first context step, which the KV-capacity estimate's profiling forward + # is: the pool is sized with the top-k's scratch held, and every top-k launch this + # engine's tiles reach compiles here rather than on a served step (each tile holds + # at most the logits budget, one block at most max_seq_len keys). + self._gvr_ws = gvr_workspace(q_fp8.device) + t0 = time.monotonic() + calls = gvr_warmup( + core.idx_topk, core._max_ctx_len, core.prefill_logits_budget, q_fp8.device, + self._gvr_ws, + ) # fmt: skip + logger.info( + f"modeling_v2 Mistral Large 4 NVFP4: indexer prefill top-k compiled for this " + f"engine's tiles ({calls} warm calls, {time.monotonic() - t0:.1f} s)" + ) + for block in plan: + if block.n_k <= 0: + # No key to select from: every slot of the block's rows is empty. + topk_buf[block.token_start : block.token_end].fill_(-1) + continue + ck, cs = indexer_k_cache_gather_op( + k_pool, step["slot_mapping_fp8_fullkv"], step["slot_mapping_scale_fullkv"], + block.k_token_start, block.n_k, core.idx_dim, + ) # fmt: skip + cs = torch.reshape(cs, [-1]) + for lo, hi, ks, ke in block.tiles: + logits = triton_fp8_mqa_logits(q_fp8[lo:hi], (ck, cs), weights[lo:hi], ks, ke) + cute_dsl_gvr_topk_prefill( + logits, ks, ke, topk_buf[lo:hi], core.idx_topk, self._gvr_ws + ) + # Freed before the next tile's is allocated: one tile is live at a time. + del logits + + def _attn_ctx(self, step, L, fused_q, lat_out, req_idx, topk_buf, ctx): + """Sparse MLA over the context rows into `lat_out`, on the step's path (`ctx`). + + Workspace path: FlashMLA's bf16 sparse prefill over the context requests' latent rows + of the source layer, dequantized out of the pool (this step's rows included, so every + row is read FP8-rounded) into the held workspace's first rows. A shared source (layer + 35) is gathered once into the rows the step keeps beside them, when it has them, and + read there by every layer that shares it; otherwise each of those layers gathers it + again into the first rows. Decode path, a step past the held workspace: the selections + as pool slots, then FlashMLA's sparse fp8 decode over the pool with every context row a + one-row sequence, in the step's sub-batches.""" + core = self.core + tc = fused_q.shape[0] + src = L.kv_source + if ctx.ws_buf is None: + slots = convert_req_index_to_global( + req_idx[:tc], step["block_table"], topk_buf[:tc], step["tokens_per_block"], + core.idx_topk, core._stride_factor, core._pool_row[src], + ) # fmt: skip + for lo, hi in ctx.sub_batches: + # Past 11622 rows FlashMLA's scheduler exits the process instead of raising. + assert hi - lo <= _CTX_DECODE_ROWS, (lo, hi) + flash_mla_sparse_decode( + fused_q[lo:hi], step["latent_kv"], slots[lo:hi], ctx.sched[hi - lo], + core.softmax_scale, lat_out[lo:hi], + ) # fmt: skip + return + ws = ctx.ws.get(src) + if ws is None: + keep = core.shared_source[src] and ctx.kept is not None + ws = ds_mla_dequant_gather( + step["latent_pool"], core._pool_row[src], ctx.kv_req, ctx.kv_pos, + step["block_table"], ctx.kept if keep else ctx.ws_buf, + ) # fmt: skip + if keep: + ctx.ws[src] = ws + # Request-local selections to workspace rows: each request's run starts at its + # cumulative KV offset. + sel = topk_buf[:tc] + idx = torch.where(sel >= 0, sel + ctx.ws_off[req_idx[:tc]][:, None], sel) + flash_mla_sparse_prefill(fused_q, ws, idx.to(torch.int32), core.softmax_scale, lat_out) + + def _attn_gen(self, step, L, fused_q, lat_out, slots, sched): + """Sparse MLA over the generation rows behind the context rows into `lat_out`, over the + pool: this rank's heads read off the 64-head scratch the prefill kernel shares (or, under + `gen_attn_flash_mla`, FlashMLA's decode over all 64, one scheduler for the step).""" + core = self.core + if core.gen_attn_flash_mla: + flash_mla_sparse_decode( + fused_q, step["latent_kv"], slots, sched, core.softmax_scale, lat_out + ) + else: + cute_dsl_ds_mla_sparse_decode( + fused_q, step["latent_kv"], slots, core.softmax_scale, lat_out, + compiled=core._mla_decode, + ) # fmt: skip + + def _mlp(self, o, L): + """This rank's share of the layer's MLP, a partial sum: the dense SwiGLU slice on the + dense layer; on a MoE layer every routed expert's width slice plus the shared expert's + slice.""" + core = self.core + act = flashinfer_silu_and_mul(self._linear_fp8(o, L.mlp_gu, L.mlp_gu_s)) + shared = self._linear_fp8(act, L.mlp_dn, L.mlp_dn_s) + m = L.moe + if m is None: + return shared + logits = cublas_mm(o, m.router, None, torch.float32) + ids, weights = renorm_moe_routing_op(logits, core.topk, torch.bfloat16) + # One NVFP4 quantization under the layer's static scale serves every expert's slice on + # every rank: the width slices' partials sum. + xq, xs = fp4_quantize(o, m.gscale, _NVFP4_SF_VEC, False, _NVFP4_SF_LINEAR) + routed = fp4_block_scale_moe_runner( + None, None, xq, xs.view(torch.float8_e4m3fn), m.fc1_w, m.fc1_s, None, None, None, + None, m.fc2_w, m.fc2_s, None, m.out1, m.out1_gate, m.out2, core.num_experts, + core.topk, None, None, core.moe_inter, core.expert_offset, core.local_experts, None, + _MOE_ROUTING_METHOD, True, _MOE_ACT_SWIGLU, topk_weights=weights, topk_ids=ids, + tune_max_num_tokens=core.max_num_tokens, + )[0] # fmt: skip + return torch.add(routed, shared) + + def _topk_buf(self, step, rows: int, tc: int, skip_ctx: bool, skip_gen: bool): + """Where this step's selections land, request-local positions. A phase the engine + skipped (every sequence within index_topk) takes the dense causal index it wrote.""" + if skip_ctx and skip_gen: + return step["topk_indices_buffer"][:rows] + buf = torch.empty( + [rows, self.core.idx_topk], dtype=torch.int32, device=step["block_table"].device + ) + if skip_ctx and tc: + buf[:tc].copy_(step["topk_indices_buffer"][:tc]) + if skip_gen and rows > tc: + buf[tc:].copy_(step["topk_indices_buffer"][tc:rows]) + return buf + + def forward( + self, + attn_metadata: AttentionMetadata, + input_ids: torch.IntTensor | None = None, + position_ids: torch.IntTensor | None = None, + inputs_embeds: torch.FloatTensor | None = None, + lora_params: dict | None = None, + **kwargs, + ) -> torch.Tensor: + core = self.core + self._check_step_contract(attn_metadata) + step = self.step_args(attn_metadata) + h = ( + nn.functional.embedding(input_ids, self._embed) + if inputs_embeds is None + else inputs_embeds + ) + rows = h.shape[0] + dev = h.device + # The slot mappings and windows follow the prepared token order; a padded input + # would scatter keys into stale slots. + assert rows == step["num_tokens"], (rows, step["num_tokens"]) + tc = step["num_ctx_tokens"] + gen = rows - tc + skip_ctx = step["skip_ctx"] + skip_gen = step["skip_gen"] if gen else True + pos = torch.reshape(position_ids, [-1]) + req_idx = step["req_idx_per_token"][:rows] + topk_buf = self._topk_buf(step, rows, tc, skip_ctx, skip_gen) + scratch = self._scratch(rows, h.dtype, dev) + ctx = self._ctx_workspace(step, tc, dev) if tc else None + + x = flashinfer_rmsnorm(h, self._norm0, core.eps) + residual = h + spec_metadata = kwargs.get("spec_metadata") + for layer_idx, L in enumerate(self._layers): + attn_out = self._attention( + step, L, x, pos, req_idx, topk_buf, scratch, ctx, tc, skip_ctx, skip_gen + ) + partial = self._linear_fp8(attn_out, L.wo, L.wo_s) + o = cute_dsl_allreduce_add_rmsnorm( + partial, residual, L.ffn_norm, torch.empty_like(partial), core._lamport + ) + mlp = self._mlp(o, L) + x = cute_dsl_allreduce_add_rmsnorm( + mlp, residual, L.next_norm, torch.empty_like(mlp), core._lamport + ) + if spec_metadata is not None: + spec_metadata.maybe_capture_hidden_states(layer_idx, residual) + return x + + def _scratch(self, rows: int, dtype, dev) -> SimpleNamespace: + """Per-step scratch every layer rewrites in turn: the attention output, the absorbed + query at the prefill kernel's 64 heads (this rank's first, the rest zero for the step), + the latent attention output, a NoPE owner's 576-wide latent rows (the rope key zero for + the step), and, under `gen_attn_flash_mla`, the generation rows' FlashMLA scheduler, + filled by the step's first decode call and shared by the rest.""" + core = self.core + bf16 = dict(dtype=dtype, device=dev) + return SimpleNamespace( + attn_out=torch.empty([rows, core.heads * core.v_dim], **bf16), + fused_q=torch.zeros([rows, _FLASH_MLA_HEADS, core.lat_dim], **bf16), + lat_out=torch.empty([rows, core.heads, core.kv_lora], **bf16), + nope_lat=None if core.unfused_glue else torch.zeros([rows, core.lat_dim], **bf16), + sched=get_mla_metadata()[0] if core.gen_attn_flash_mla else None, + ) + + def _ctx_workspace(self, step, tc: int, dev) -> SimpleNamespace: + """The context rows' path for the step, chosen once on host ints by its context KV: + every context request's cached prefix and new tokens (the host + `ctx_kv_indptr[num_contexts]`; the engine's `num_ctx_kv_tokens` counts the new ones + only, and under chunked prefill every chunk after a prompt's first has a prefix). + + The held workspace, bf16 `[core.ctx_workspace_rows, 576]` (`_ctx_workspace_rows`), is + allocated on the first context step, whichever path that step takes, and held. The + engine's KV-capacity estimate runs context steps as its profiling forward and sizes the + pool after their peak memory (`KvCacheCreator.configure_kv_cache_capacity`), so the pool + is sized with the workspace held. A generation-only engine runs no context step and + holds none. + + The workspace path, a step whose context KV fits the held rows: request and position + per workspace row as the dequantizing gather and the prefill kernel address them, each + request's offset, the held workspace's first `n_kv` rows, which every layer's gather + rewrites, and, when the context KV fits twice, the next `n_kv` rows (`kept`), where a + shared owner's gather stays for the step (`ws`). + + The decode path, a step past the held rows, with no workspace: the context rows' + sub-batches of at most `_CTX_DECODE_ROWS`, and one FlashMLA scheduler per sub-batch + size, filled by its first call and shared by the step's layers.""" + # Under KV reuse a context request attends over its cached prefix and its new tokens. + n_kv = step["ctx_cached_tokens"] + step["num_ctx_tokens"] + if self._ws_buf is None: + # On the first context step: the KV-capacity estimate's profiling forward is one, + # so the peak it measures, which sizes the pool, holds this buffer. + self._ws_buf = torch.empty( + [self.core.ctx_workspace_rows, self.core.lat_dim], dtype=torch.bfloat16, device=dev + ) + logger.info( + f"modeling_v2 Mistral Large 4 NVFP4: context workspace held, {self._ws_buf.shape[0]} " + f"rows x {self.core.lat_dim} bf16 ({self._ws_buf.nbytes / 2**20:.0f} MiB)" + ) + held = self._ws_buf.shape[0] + if n_kv > held: + bounds = [(lo, min(lo + _CTX_DECODE_ROWS, tc)) for lo in range(0, tc, _CTX_DECODE_ROWS)] + logger.debug( + f"modeling_v2 Mistral Large 4 NVFP4: context KV {n_kv} past the held {held} rows: " + f"decode path, {len(bounds)} sub-batches" + ) + return SimpleNamespace( + ws_buf=None, + sub_batches=bounds, + sched={rows: get_mla_metadata()[0] for rows in {hi - lo for lo, hi in bounds}}, + ) + ctx = step["num_contexts"] + cu = step["cu_ctx_kv"] + lens = cu[1:] - cu[:-1] + kv_req = torch.repeat_interleave( + torch.arange(ctx, dtype=torch.int32, device=dev), lens, output_size=n_kv + ) + kv_pos = (torch.arange(n_kv, device=dev) - cu[:-1][kv_req]).to(torch.int32) + keep = 2 * n_kv <= held + logger.debug( + f"modeling_v2 Mistral Large 4 NVFP4: context KV {n_kv} of the held {held} rows: " + f"workspace path, the shared source {'kept' if keep else 'gathered again'}" + ) + return SimpleNamespace( + kv_req=kv_req, + kv_pos=kv_pos, + ws_off=cu[:-1], + ws_buf=self._ws_buf[:n_kv], + kept=self._ws_buf[n_kv : 2 * n_kv] if keep else None, + ws={}, + ) + + def _attention(self, step, L, x, pos, req_idx, topk_buf, s, ctx, tc, skip_ctx, skip_gen): + """One layer's attention block, both phases, into `s.attn_out` (returned): the query + with SSMax, the indexer's selection, the absorbed query, the owner's rope and + fp8_ds_mla append, sparse MLA over the context and the generation rows, and the V + up-projection.""" + core = self.core + rows = x.shape[0] + q = self._linear_fp8(x, L.wq, L.wq_s) + q3 = torch.reshape(q, [rows, core.heads, core.qk_dim]) + fq = s.fused_q[:, : core.heads] + if core.unfused_glue: + q3 = ssmax_scaling(q3, pos, L.ssmax_s, L.ssmax_b) + if L.rope: + fq[..., core.kv_lora :].copy_(q3[..., L.nope :]) + else: + fq[..., core.kv_lora :].zero_() + else: + # SSMax in place; its rope half (a NoPE layer's zero) into the absorbed query's tail. + ssmax_scaling_fused_q( + q3, pos, L.ssmax_s, L.ssmax_b, q3, fq[..., core.kv_lora :], L.rope + ) + if L.indexer is not None: + self._indexer(step, x, pos, L, topk_buf, tc, skip_ctx, skip_gen) + bmm_out( + torch.transpose(q3[..., : L.nope], 0, 1), L.k_b, + torch.transpose(fq[..., : core.kv_lora], 0, 1), + ) # fmt: skip + if L.owner: + # Every row's latent, context and generation, before either phase reads the pool; + # the query tail is roped in the same call. + ds_mla_rope_append( + self._latent(x, L, s.nope_lat), fq, pos, core._rope.indexer, req_idx, + step["block_table"], step["latent_pool"], core._pool_row[L.index], L.rope, + ) # fmt: skip + if tc: + self._attn_ctx(step, L, s.fused_q[:tc], s.lat_out[:tc], req_idx, topk_buf, ctx) + if rows > tc: + slots = convert_req_index_to_global( + req_idx[tc:], step["block_table"], topk_buf[tc:], step["tokens_per_block"], + core.idx_topk, core._stride_factor, core._pool_row[L.kv_source], + ) # fmt: skip + self._attn_gen(step, L, s.fused_q[tc:], s.lat_out[tc:], slots, s.sched) + bmm_out( + torch.transpose(s.lat_out, 0, 1), + L.v_b_t, + torch.transpose(torch.reshape(s.attn_out, [rows, core.heads, core.v_dim]), 0, 1), + ) + return s.attn_out + + def dspark_forward( + self, + draft, + noise_embedding, + query_positions, + num_ctx_per_req, + ctx_k_cache, + ctx_v_cache, + ctx_cache_batch_idx, + ctx_kv_cache=None, + ctx_page_table=None, + block_placement=None, + ): + # Only the worker's fused paged-latent preparation places the block + # (block_placement), and a drafter with embed_draft_block does not take + # it; this decode reads its context window through ctx_page_table. + del block_placement + draft._build_fused_kv_buffers() + draft._ensure_rope() + core = self.core + batch, block, hidden = noise_embedding.shape + rows = batch * block + # Gather a bounded window, not every page in the configured context. + # Invalid padding slots are masked even if their pages are allocated. + lengths = num_ctx_per_req[:batch].long() + offsets = torch.arange(draft.window, device=noise_embedding.device) + positions = (lengths - draft.window).clamp_min(0)[:, None] + offsets[None, :] + valid = positions < lengths[:, None] + slots = ctx_cache_batch_idx.long() + if ctx_page_table is not None: + page_size = ctx_kv_cache[0].shape[-2] + table = ctx_page_table.index_select(0, slots).long() + safe_positions = positions.clamp(max=table.shape[1] * page_size - 1) + pages = table.gather(1, safe_positions // page_size).clamp_min(0) + page_offsets = safe_positions % page_size + else: + safe_positions = positions.clamp(max=ctx_k_cache.shape[2] - 1) + key_valid = torch.cat( + (valid, torch.ones(batch, block, dtype=torch.bool, device=valid.device)), dim=1 + ) + query = torch.empty( + batch, + block, + core.heads, + core.lat_dim, + dtype=noise_embedding.dtype, + device=noise_embedding.device, + ) + values = torch.empty( + rows, core.heads, core.v_dim, dtype=noise_embedding.dtype, device=noise_embedding.device + ) + residual = noise_embedding.reshape(rows, hidden).clone() + normalized = torch.empty_like(residual) + x = flashinfer_rmsnorm(residual, draft._layers[0].attn_norm, core.eps) + for i, layer in enumerate(draft._layers): + q = cublas_mm(x, layer.q, None, None).view(batch, block, core.heads, core.qk_dim) + bmm_out( + q[..., : core.nope].reshape(rows, core.heads, core.nope).transpose(0, 1), + layer.k_b, + query[..., : core.kv_lora].view(rows, core.heads, core.kv_lora).transpose(0, 1), + ) + query[..., core.kv_lora :].copy_(draft._rotate(q[..., core.nope :], query_positions)) + kv = cublas_mm(x, layer.kv_a, None, None) + content = flashinfer_rmsnorm( + kv[:, : core.kv_lora].contiguous(), layer.kv_norm, core.eps + ) + block_kv = torch.cat( + ( + content.view(batch, block, core.kv_lora), + draft._rotate( + kv[:, core.kv_lora :].view(batch, block, core.rope_dim), query_positions + ), + ), + dim=-1, + ) + if ctx_page_table is not None: + context = ctx_kv_cache[i][:, 0, 0][pages, page_offsets] + else: + context = ctx_k_cache[slots[:, None], i, safe_positions, 0] + # Zero masked values as well as scores: 0 * an uninitialized + # NaN in the latent pool must not contaminate the weighted sum. + context = context.masked_fill(~valid.unsqueeze(-1), 0) + keys = torch.cat((context, block_kv), dim=1) + scores = torch.einsum("bshd,btd->bsht", query, keys).float() * core.softmax_scale + scores = scores.masked_fill(~key_valid[:, None, None, :], float("-inf")) + probabilities = torch.softmax(scores, dim=-1).to(keys.dtype) + latent = torch.einsum("bsht,btc->bshc", probabilities, keys[..., : core.kv_lora]) + bmm_out( + latent.reshape(rows, core.heads, core.kv_lora).transpose(0, 1), + layer.v_b.transpose(1, 2), + values.transpose(0, 1), + ) + partial = cublas_mm(values.reshape(rows, -1), layer.out, None, None) + o = cute_dsl_allreduce_add_rmsnorm( + partial, residual, layer.ffn_norm, normalized, core._lamport + ) + gate = cublas_mm(o, layer.gate, None, None) + up = cublas_mm(o, layer.up, None, None) + activated = flashinfer_silu_and_mul(torch.cat((gate, up), dim=-1)) + mlp = cublas_mm(activated, layer.down, None, None) + x = cute_dsl_allreduce_add_rmsnorm( + mlp, residual, layer.next_norm, normalized, core._lamport + ) + return x + + +class DecodeTarget: + """Generation-only steps, including uniform DSpark verification blocks; graph captured.""" + + def __init__(self, core: MistralLarge4TpModelingV2) -> None: + """Take the per-layer views once, off the real weights. Everything an engine value + fixes is read off `self.core` at the call.""" + self.core = core + self._contract_pending = step_contract_enabled() + self._layers = _layer_views(core) + self._embed = core.w["embed"] + self._norm0 = core.w["l0_attn_norm"] + # The shared expert's side stream (`_mlp`) and the events that fork it from and join it + # to the main stream; no stream keeps the shared expert on the main stream. + self._shared_stream = None if _SERIAL_SHARED_EXPERT else torch.cuda.Stream() + self._shared_fork = torch.cuda.Event() + self._shared_join = torch.cuda.Event() + + def _check_step_contract(self, md) -> None: + """Opt-in first-forward fail-fast: calling `step_args` reads every metadata field this + target consumes (private trtllm surface). Fixed at engine construction, so once per + target instance, and off in a served engine.""" + if not self._contract_pending: + return + try: + self.step_args(md) + except AttributeError as exc: + raise AssertionError(f"attention metadata surface drifted: {exc}") from exc + self._contract_pending = False + + def step_args(self, md: TrtllmAttentionMetadata) -> dict: + """The common batch state. States the two context counts instead of reading them + back: this target runs only when there are no context rows.""" + step = _step_common(md, self.core.indexer_layers) + step.update(num_contexts=0, num_ctx_tokens=0) + return step + + def _linear_fp8(self, x, w, w_s): + """K128-quantized FP8 linear: activation quantization fused into the GEMM at the + (M, N, K) shapes the entry has tuned, the Rubin MXFP8 kernel at every other.""" + return cute_dsl_fused_fp8_block_scale_linear(x, w, w_s) + + def _latent(self, x, L, nope_lat): + """`[RMSNorm(c_kv) | k_pe]`, 576 wide: a RoPE owner's kv_a row, normalized in place; a + NoPE owner's 512-wide latent normalized into `nope_lat`, whose rope key stays zero.""" + core = self.core + kv = cublas_mm(x, L.wkv_a) + if core.unfused_glue: + c_kv = flashinfer_rmsnorm(kv[:, : core.kv_lora], L.kv_norm, core.eps) + if L.rope: + k_pe = kv[:, core.kv_lora :] + else: + k_pe = torch.zeros([kv.shape[0], core.rope_dim], dtype=kv.dtype, device=kv.device) + return torch.cat([c_kv, k_pe], dim=-1) + if L.rope: + c_kv = kv[:, : core.kv_lora] + flashinfer_rmsnorm(c_kv, L.kv_norm, core.eps, out=c_kv) + return kv + flashinfer_rmsnorm(kv, L.kv_norm, core.eps, out=nope_lat[:, : core.kv_lora]) + return nope_lat + + def _indexer(self, step, x, pos, L, topk_buf, skip_gen: bool): + """Append this step's keys to the layer's key pool; unless the engine skipped the + indexer (every sequence within index_topk), score every sequence's paged keys and + select `index_topk` positions per row into `topk_buf`.""" + core = self.core + ix = L.indexer + rows = x.shape[0] + table = core._rope.indexer if ix.rope else core._rope.identity + kw = cublas_mm(x, ix.wk_wp) + k = kw[:, : core.idx_dim].contiguous() + k_pool = step["ix_pools"][L.index] + k_pool = cute_dsl_indexer_k_cache_update( + k, ix.k_norm_w, ix.k_norm_b, pos, table.view(-1, 2, core.rope_dim // 2), k_pool, + step["slot_mapping_fp8"], step["slot_mapping_scale"], rows, + is_neox=False, compiled=core._ix_k[k_pool.is_contiguous()], validate=False, + ) # fmt: skip + if skip_gen: + return + qn = layer_norm(x, ix.q_norm_w, ix.q_norm_b, core.idx_eps) + q = torch.reshape(self._linear_fp8(qn, ix.wq_b, ix.wq_b_s), [rows, core.idx_heads, -1]) + q_fp8 = torch.empty_like(q, dtype=torch.float8_e4m3fn) + weights = torch.empty([rows, core.idx_heads], dtype=torch.float32, device=x.device) + if core.unfused_glue: + q = ssmax_scaling(q, pos, ix.ssmax_s, ix.ssmax_b) + q_fp8, weights = cute_dsl_indexer_q_side_fused( + q, pos, table, kw[:, core.idx_dim :].float().contiguous(), core.idx_weight_scale, + q_fp8, weights, is_neox=False, compiled=core._ix_q, + ) # fmt: skip + else: + # SSMax, rope, e4m3 and the head weights (read in place out of `kw`) in one launch. + q_fp8, weights = ssmax_indexer_q_side_fused( + q, pos, ix.ssmax_s, ix.ssmax_b, table, kw[:, core.idx_dim :], + core.idx_weight_scale, q_fp8, weights, + ) # fmt: skip + num_gens = step["num_seqs"] + next_n = rows // num_gens + kv_lens = step["kv_lens_cuda"][:num_gens] + logits = cute_dsl_fp8_paged_mqa_logits( + torch.reshape(q_fp8, [num_gens, next_n, core.idx_heads, core.idx_dim]), + k_pool, + weights, + kv_lens, + step["indexer_k_cache_block_offsets"][:num_gens], + step["scheduler_metadata_buffer"], + core._max_ctx_len, + ) + core._select_indexer_decode( + logits, kv_lens, topk_buf, next_n, + step["radix_aux_indices"], step["radix_aux_logits"], + ) # fmt: skip + + def _mlp(self, o, L): + """This rank's share of the layer's MLP, a partial sum: the dense SwiGLU slice on the + dense layer; on a MoE layer every routed expert's width slice plus the shared expert's + slice. Under CUDA-graph capture the MoE layer's shared expert runs on the side stream + while the main stream runs the router, the quantization and the routed experts.""" + core = self.core + + def shared(): + act = flashinfer_silu_and_mul(self._linear_fp8(o, L.mlp_gu, L.mlp_gu_s)) + return self._linear_fp8(act, L.mlp_dn, L.mlp_dn_s) + + m = L.moe + if m is None: + return shared() + + def routed(): + logits = cublas_mm(o, m.router, None, torch.float32) + ids, weights = renorm_moe_routing_op(logits, core.topk, torch.bfloat16) + # One NVFP4 quantization under the layer's static scale serves every expert's slice + # on every rank: the width slices' partials sum. + xq, xs = fp4_quantize(o, m.gscale, _NVFP4_SF_VEC, False, _NVFP4_SF_LINEAR) + return fp4_block_scale_moe_runner( + None, None, xq, xs.view(torch.float8_e4m3fn), m.fc1_w, m.fc1_s, None, None, None, + None, m.fc2_w, m.fc2_s, None, m.out1, m.out1_gate, m.out2, core.num_experts, + core.topk, None, None, core.moe_inter, core.expert_offset, core.local_experts, + None, _MOE_ROUTING_METHOD, True, _MOE_ACT_SWIGLU, topk_weights=weights, + topk_ids=ids, tune_max_num_tokens=core.max_num_tokens, + )[0] # fmt: skip + + routed_out, shared_out = maybe_execute_in_parallel( + routed, shared, self._shared_fork, self._shared_join, self._shared_stream + ) + return torch.add(routed_out, shared_out) + + def forward( + self, + attn_metadata: AttentionMetadata, + input_ids: torch.IntTensor | None = None, + position_ids: torch.IntTensor | None = None, + inputs_embeds: torch.FloatTensor | None = None, + lora_params: dict | None = None, + **kwargs, + ) -> torch.Tensor: + core = self.core + self._check_step_contract(attn_metadata) + step = self.step_args(attn_metadata) + h = ( + nn.functional.embedding(input_ids, self._embed) + if inputs_embeds is None + else inputs_embeds + ) + rows = h.shape[0] + dev = h.device + assert rows == step["num_tokens"] and rows % step["num_seqs"] == 0, ( + rows, + step["num_tokens"], + step["num_seqs"], + ) + # A separate graph is captured for steps whose sequences all fit in index_topk, so + # this host branch is a per-capture constant. + skip_gen = step["skip_gen"] + pos = torch.reshape(position_ids, [-1]) + req_idx = step["req_idx_per_token"][:rows] + if skip_gen: + topk_buf = step["topk_indices_buffer"][:rows] + else: + topk_buf = torch.empty([rows, core.idx_topk], dtype=torch.int32, device=dev) + scratch = self._scratch(rows, h.dtype, dev) + + x = flashinfer_rmsnorm(h, self._norm0, core.eps) + residual = h + spec_metadata = kwargs.get("spec_metadata") + for layer_idx, L in enumerate(self._layers): + attn_out = self._attention(step, L, x, pos, req_idx, topk_buf, scratch, skip_gen) + partial = self._linear_fp8(attn_out, L.wo, L.wo_s) + o = cute_dsl_allreduce_add_rmsnorm( + partial, residual, L.ffn_norm, torch.empty_like(partial), core._lamport + ) + mlp = self._mlp(o, L) + x = cute_dsl_allreduce_add_rmsnorm( + mlp, residual, L.next_norm, torch.empty_like(mlp), core._lamport + ) + if spec_metadata is not None: + spec_metadata.maybe_capture_hidden_states(layer_idx, residual) + return x + + def _scratch(self, rows: int, dtype, dev) -> SimpleNamespace: + """Per-step scratch every layer rewrites in turn: the attention output, the absorbed + query at this rank's heads (every element written by each layer: the absorption BMM + and the SSMax op's rope tail or zeros), a NoPE owner's 576-wide latent rows (the rope + key zero for the step), and the latent attention output the native decode writes. + Under `gen_attn_flash_mla` the query is FlashMLA's 64 heads (this rank's first, the + rest zero for the step), the scratch carries the FlashMLA scheduler the step's first + layer fills and the rest share (captured with the step), and the latent output is kept + only for the unfused glue, which copies FlashMLA's live heads into it.""" + core = self.core + bf16 = dict(dtype=dtype, device=dev) + if core.gen_attn_flash_mla: + fused_q = torch.zeros([rows, _FLASH_MLA_HEADS, core.lat_dim], **bf16) + else: + fused_q = torch.empty([rows, core.heads, core.lat_dim], **bf16) + return SimpleNamespace( + attn_out=torch.empty([rows, core.heads * core.v_dim], **bf16), + fused_q=fused_q, + nope_lat=None if core.unfused_glue else torch.zeros([rows, core.lat_dim], **bf16), + lat_out=( + torch.empty([rows, core.heads, core.kv_lora], **bf16) + if core.unfused_glue or not core.gen_attn_flash_mla + else None + ), + sched=get_mla_metadata()[0] if core.gen_attn_flash_mla else None, + ) + + def _attention(self, step, L, x, pos, req_idx, topk_buf, s, skip_gen: bool): + """One layer's attention block into `s.attn_out` (returned): the query with SSMax, the + indexer's selection, the absorbed query, the owner's rope and fp8_ds_mla append, + sparse decode over the pool, and the V up-projection of the latent output (FlashMLA's + live heads read in place under `gen_attn_flash_mla` with the fused glue).""" + core = self.core + rows = x.shape[0] + q = self._linear_fp8(x, L.wq, L.wq_s) + q3 = torch.reshape(q, [rows, core.heads, core.qk_dim]) + fq = s.fused_q[:, : core.heads] + if core.unfused_glue: + q3 = ssmax_scaling(q3, pos, L.ssmax_s, L.ssmax_b) + if L.rope: + fq[..., core.kv_lora :].copy_(q3[..., L.nope :]) + else: + fq[..., core.kv_lora :].zero_() + else: + # SSMax in place; its rope half (a NoPE layer's zero) into the absorbed query's tail. + ssmax_scaling_fused_q( + q3, pos, L.ssmax_s, L.ssmax_b, q3, fq[..., core.kv_lora :], L.rope + ) + if L.indexer is not None: + self._indexer(step, x, pos, L, topk_buf, skip_gen) + slots = convert_req_index_to_global( + req_idx, step["block_table"], topk_buf, step["tokens_per_block"], core.idx_topk, + core._stride_factor, core._pool_row[L.kv_source], + ) # fmt: skip + bmm_out( + torch.transpose(q3[..., : L.nope], 0, 1), L.k_b, + torch.transpose(fq[..., : core.kv_lora], 0, 1), + ) # fmt: skip + if L.owner: + ds_mla_rope_append( + self._latent(x, L, s.nope_lat), fq, pos, core._rope.indexer, req_idx, + step["block_table"], step["latent_pool"], core._pool_row[L.index], L.rope, + ) # fmt: skip + if core.gen_attn_flash_mla and core.unfused_glue: + lat = flash_mla_sparse_decode( + s.fused_q, step["latent_kv"], slots, s.sched, core.softmax_scale, s.lat_out + ) + elif core.gen_attn_flash_mla: + lat = flash_mla_sparse_decode( + s.fused_q, step["latent_kv"], slots, s.sched, core.softmax_scale, None + )[:, : core.heads] + else: + # The rank's heads, unpadded, into the latent output. + cute_dsl_ds_mla_sparse_decode( + s.fused_q, step["latent_kv"], slots, core.softmax_scale, s.lat_out, + compiled=core._mla_decode, + ) # fmt: skip + lat = s.lat_out + bmm_out( + torch.transpose(lat, 0, 1), + L.v_b_t, + torch.transpose(torch.reshape(s.attn_out, [rows, core.heads, core.v_dim]), 0, 1), + ) + return s.attn_out + + def dspark_forward( + self, + draft, + noise_embedding, + query_positions, + num_ctx_per_req, + ctx_k_cache, + ctx_v_cache, + ctx_cache_batch_idx, + ctx_kv_cache=None, + ctx_page_table=None, + block_placement=None, + ): + # Only the worker's fused paged-latent preparation places the block + # (block_placement), and a drafter with embed_draft_block does not take + # it; this decode reads its context window through ctx_page_table. + del block_placement + draft._build_fused_kv_buffers() + draft._ensure_rope() + core = self.core + batch, block, hidden = noise_embedding.shape + rows = batch * block + # Gather a bounded window, not every page in the configured context. + # Invalid padding slots are masked even if their pages are allocated. + lengths = num_ctx_per_req[:batch].long() + offsets = torch.arange(draft.window, device=noise_embedding.device) + positions = (lengths - draft.window).clamp_min(0)[:, None] + offsets[None, :] + valid = positions < lengths[:, None] + slots = ctx_cache_batch_idx.long() + if ctx_page_table is not None: + page_size = ctx_kv_cache[0].shape[-2] + table = ctx_page_table.index_select(0, slots).long() + safe_positions = positions.clamp(max=table.shape[1] * page_size - 1) + pages = table.gather(1, safe_positions // page_size).clamp_min(0) + page_offsets = safe_positions % page_size + else: + safe_positions = positions.clamp(max=ctx_k_cache.shape[2] - 1) + key_valid = torch.cat( + (valid, torch.ones(batch, block, dtype=torch.bool, device=valid.device)), dim=1 + ) + query = torch.empty( + batch, + block, + core.heads, + core.lat_dim, + dtype=noise_embedding.dtype, + device=noise_embedding.device, + ) + values = torch.empty( + rows, core.heads, core.v_dim, dtype=noise_embedding.dtype, device=noise_embedding.device + ) + residual = noise_embedding.reshape(rows, hidden).clone() + normalized = torch.empty_like(residual) + x = flashinfer_rmsnorm(residual, draft._layers[0].attn_norm, core.eps) + for i, layer in enumerate(draft._layers): + q = cublas_mm(x, layer.q, None, None).view(batch, block, core.heads, core.qk_dim) + bmm_out( + q[..., : core.nope].reshape(rows, core.heads, core.nope).transpose(0, 1), + layer.k_b, + query[..., : core.kv_lora].view(rows, core.heads, core.kv_lora).transpose(0, 1), + ) + query[..., core.kv_lora :].copy_(draft._rotate(q[..., core.nope :], query_positions)) + kv = cublas_mm(x, layer.kv_a, None, None) + content = flashinfer_rmsnorm( + kv[:, : core.kv_lora].contiguous(), layer.kv_norm, core.eps + ) + block_kv = torch.cat( + ( + content.view(batch, block, core.kv_lora), + draft._rotate( + kv[:, core.kv_lora :].view(batch, block, core.rope_dim), query_positions + ), + ), + dim=-1, + ) + if ctx_page_table is not None: + context = ctx_kv_cache[i][:, 0, 0][pages, page_offsets] + else: + context = ctx_k_cache[slots[:, None], i, safe_positions, 0] + # Zero masked values as well as scores: 0 * an uninitialized + # NaN in the latent pool must not contaminate the weighted sum. + context = context.masked_fill(~valid.unsqueeze(-1), 0) + keys = torch.cat((context, block_kv), dim=1) + scores = torch.einsum("bshd,btd->bsht", query, keys).float() * core.softmax_scale + scores = scores.masked_fill(~key_valid[:, None, None, :], float("-inf")) + probabilities = torch.softmax(scores, dim=-1).to(keys.dtype) + latent = torch.einsum("bsht,btc->bshc", probabilities, keys[..., : core.kv_lora]) + bmm_out( + latent.reshape(rows, core.heads, core.kv_lora).transpose(0, 1), + layer.v_b.transpose(1, 2), + values.transpose(0, 1), + ) + partial = cublas_mm(values.reshape(rows, -1), layer.out, None, None) + o = cute_dsl_allreduce_add_rmsnorm( + partial, residual, layer.ffn_norm, normalized, core._lamport + ) + gate = cublas_mm(o, layer.gate, None, None) + up = cublas_mm(o, layer.up, None, None) + activated = flashinfer_silu_and_mul(torch.cat((gate, up), dim=-1)) + mlp = cublas_mm(activated, layer.down, None, None) + x = cute_dsl_allreduce_add_rmsnorm( + mlp, residual, layer.next_norm, normalized, core._lamport + ) + return x + + +class DraftModel(nn.Module): + """This specialization's BF16 DSpark backbone and recurrent head. + + Runtime owns accepted-token bookkeeping and the paged context pool. The + draft attends the last trained window of captured target features plus + its whole non-causal query block. All parameters live in the core table. + """ + + _uses_worker_attention_backend = False + _paged_ctx_cache = True + _kv_factor = 1 + _dspark_shift_label = True + has_markov_head = True + has_recurrent_markov_head = True + + def __init__(self, shell, draft_config): + super().__init__() + # Keep references without registering the parent as a child module. + object.__setattr__(self, "_shell", shell) + self.model_config = draft_config + self.config = draft_config.pretrained_config + core = shell.model + spec = shell.spec_config + native = spec._draft_model_config + self.block_size = spec.block_size + self.mask_token_id = spec.mask_token_id + self.target_layer_ids = tuple(spec.target_layer_ids) + self.window = int(native["ragged_attention"].split(",")[0]) + self._num_attn_layers = int(native["n_layers"]) + self._num_heads = core.heads + self._num_kv_heads = 1 + self._head_dim = core.lat_dim + self.dflash_attention_backend = "VANILLA" + self._runtime_position_ceiling = None + self._rope = None + self._layers = None + + @property + def core(self): + return self._shell.model + + @property + def lm_head(self): + return self._shell.lm_head + + @property + def markov_w2(self): + return self.core.w["draft_markov_output"] + + def logits_processor(self, hidden, lm_head, attn_metadata, gather): + logits = self._shell.logits_processor(hidden, lm_head, attn_metadata, gather) + # The shared-head logits are soft-capped before the RNN bias is added. + return tanh_softcap(logits, self._shell.softcap) + + def apply_draft_logit_transform(self, logits): + """Return ``logits`` unchanged: :meth:`logits_processor` already soft-caps them. + + The worker calls this on the block logits after ``logits_processor``. + This drafter declares no ``logit_multiplier`` or ``logit_softcap``, so the + cap is applied once, in ``logits_processor``. + """ + return logits + + def _build_fused_kv_buffers(self): + # The worker calls this before sizing its cache. Take views once after + # materialization; context projection is explicit per layer here. + if self._layers is not None: + return + w, core = self.core.w, self.core + self._layers = [] + for i in range(self._num_attn_layers): + prefix = f"l{i}_draft_" + kv_b = w[prefix + "wkv_b"].view(core.heads, core.nope + core.v_dim, core.kv_lora) + self._layers.append( + SimpleNamespace( + attn_norm=w[prefix + "attn_norm"], + q=w[prefix + "wq"].t(), + kv_a=w[prefix + "wkv_a"].t(), + kv_norm=w[prefix + "kv_a_norm"], + k_b=kv_b[:, : core.nope].contiguous(), + v_b=kv_b[:, core.nope :].contiguous(), + out=w[prefix + "wo"].t(), + ffn_norm=w[prefix + "ffn_norm"], + gate=w[prefix + "w1"].t(), + up=w[prefix + "w3"].t(), + down=w[prefix + "w2"].t(), + next_norm=( + w[f"l{i + 1}_draft_attn_norm"] + if i + 1 < self._num_attn_layers + else w["draft_final_norm"] + ), + ) + ) + + def _ensure_rope(self): + cap = self._runtime_position_ceiling or self.core.rope_rows + if self._rope is None or self._rope.rows != cap: + w = self.core.w["draft_final_norm"] + # alpha=0 removes LeRoPE; all draft layers use plain YaRN. + self._rope = _rope_tables( + self.core.model_config.pretrained_config, + torch.zeros(self.core.rope_dim // 2), + cap, + w.dtype, + w.device, + ) + + def _rotate(self, x, positions): + table = self._rope.indexer[positions.long()] + cos, sin = table.chunk(2, dim=-1) + # One head axis for Q, none for the shared latent K. + if x.ndim == positions.ndim + 2: + cos, sin = cos.unsqueeze(-2), sin.unsqueeze(-2) + even, odd = x.float()[..., 0::2], x.float()[..., 1::2] + return ( + torch.stack((even * cos - odd * sin, even * sin + odd * cos), -1) + .flatten(-2) + .to(x.dtype) + ) + + def embed_draft_block(self, anchors, block_size): + # The route pins the trained block width; no mask token is looked up + # in the vocabulary (its sentinel is deliberately out of range). + weight = self.core.w["embed"] + valid = (anchors >= 0) & (anchors < weight.shape[0]) + embedded = nn.functional.embedding(anchors.clamp(0, weight.shape[0] - 1), weight) + embedded = embedded.masked_fill(~valid.unsqueeze(-1), 0) + masks = self.core.w["draft_mask"].unsqueeze(0).expand(anchors.shape[0], -1, -1) + return torch.cat((embedded.unsqueeze(1), masks), dim=1) + + def project_target_hidden(self, hidden): + w = self.core.w + projected = cublas_mm(hidden, w["draft_target_projection"].t(), None, None) + return flashinfer_rmsnorm(projected, w["draft_target_norm"], self.core.eps) + + def precompute_context_kv(self, projected_hidden, positions): + self._build_fused_kv_buffers() + self._ensure_rope() + core = self.core + latent = [] + for layer in self._layers: + kv = cublas_mm(projected_hidden, layer.kv_a, None, None) + content = flashinfer_rmsnorm( + kv[:, : core.kv_lora].contiguous(), layer.kv_norm, core.eps + ) + rope = self._rotate(kv[:, core.kv_lora :], positions.reshape(-1)) + latent.append(torch.cat((content, rope), dim=-1)) + return torch.stack(latent, dim=1).unsqueeze(2), None + + def dflash_forward_with_metadata(self, *, attn_metadata, **kwargs): + # The phase belongs to this engine step, including mixed batches. + return self.core._select_target(attn_metadata).dspark_forward(self, **kwargs) + + def apply_markov_chain_logits(self, base_logits, first_prev_tokens, *, hidden_states): + """The recurrent Markov bias over full-vocab ``[B, K, vocab]`` block logits. + + Each position's greedy token (the argmax of its biased logits) is the + next position's previous token; ``hidden_states`` is the ``[B, K, + hidden]`` block output the logits were projected from. + """ + w = self.core.w + output = w["draft_markov_output"] + previous = first_prev_tokens.long() + state = None + refined = [] + for i in range(base_logits.shape[1]): + weight = w["draft_markov_embedding"] + valid = (previous >= 0) & (previous < weight.shape[0]) + embedding = nn.functional.embedding(previous.clamp(0, weight.shape[0] - 1), weight) + embedding = embedding.masked_fill(~valid.unsqueeze(-1), 0) + if state is None: + state = torch.zeros_like(embedding) + joint = cublas_mm( + torch.cat((state, embedding, hidden_states[:, i]), dim=-1), + w["draft_joint"].t(), + w["draft_joint_bias"], + None, + ) + gate, candidate, emitted = joint.chunk(3, dim=-1) + gate = torch.sigmoid(gate) + state = gate * state + (1 - gate) * torch.tanh(candidate) + bias = cublas_mm(torch.tanh(emitted), output.t(), None, None) + logits = base_logits[:, i] + bias + refined.append(logits) + previous = logits.argmax(dim=-1) + return torch.stack(refined, dim=1) + + +class _VisionBlock(NamedTuple): + """One vision transformer block's weights, as the views the ops read: the GEMM + operands column-major (`.t()` of the stored rows).""" + + attn_norm: torch.Tensor + qkv: torch.Tensor + wo: torch.Tensor + ffn_norm: torch.Tensor + w13: torch.Tensor + w2: torch.Tensor + + +def _vision_step(seg_lens: list[int], device, workspace: torch.Tensor) -> dict: + """The batch state of one vision attention call: one context sequence per image, + no KV cache. Every value is host-known from the images' patch counts.""" + total = sum(seg_lens) + longest = max(seg_lens) + lens_dev = torch.tensor(seg_lens, dtype=torch.int32, device=device) + lens_host = torch.tensor(seg_lens, dtype=torch.int32) + return dict( + workspace=workspace, + sequence_length=lens_dev, + host_past_key_value_lengths=lens_host, + host_total_kv_lens=torch.tensor([total, 0], dtype=torch.int32), + context_lengths=lens_dev, + host_context_lengths=lens_host, + host_request_types=torch.zeros(len(seg_lens), dtype=torch.int32), + max_num_requests=len(seg_lens), + max_context_length=longest, + max_seq_len=longest, + attention_window_size=longest, + num_contexts=len(seg_lens), + num_ctx_tokens=total, + ) + + +def _vision_attention( + x: torch.Tensor, + B: _VisionBlock, + positions: torch.Tensor, + step: dict, + heads: int, + head_dim: int, + theta: float, +) -> torch.Tensor: + """One vision block's attention over the patch rows of every image of the call: + `[wq ; wk ; wv]` as one GEMM, q and k rotated by each patch's (row, col) in its + own image, attention bidirectional within each image, then `wo`.""" + rows = x.shape[0] + width = heads * head_dim + qkv = cublas_mm(x, B.qkv) + q, k = pixtral_rope_2d( + torch.reshape(qkv[:, :width], [rows, heads, head_dim]), + torch.reshape(qkv[:, width : 2 * width], [rows, heads, head_dim]), + positions, + theta, + ) + qkv = torch.cat( + [torch.reshape(q, [rows, width]), torch.reshape(k, [rows, width]), qkv[:, 2 * width :]], -1 + ) + out = torch.empty([rows, width], dtype=x.dtype, device=x.device) + out = thop_attention( + q=qkv, + output=out, + num_heads=heads, + num_kv_heads=heads, + head_size=head_dim, + **step, + **_VISION_ATTN, + ) + return cublas_mm(out, B.wo) + + +def _merge_tiles(x: torch.Tensor, grid_h: int, grid_w: int, merge: int) -> torch.Tensor: + """One image's `[grid_h * grid_w, d]` patch rows -> `[grid_h * grid_w / merge^2, + d * merge^2]`: each merge x merge tile of the grid as one row, tiles row-major (the + order of the image's [IMG] tokens), each row laid out (channel, tile row, tile + column) as the merging layer's input columns are.""" + d = x.shape[-1] + grid = torch.permute(torch.reshape(x, [grid_h, grid_w, d]), [2, 0, 1]) + tiles = nn.functional.unfold(torch.unsqueeze(grid, 0), kernel_size=merge, stride=merge) + return torch.t(tiles[0]) + + +class VisionTower: + """The image input path: a request's pixels -> one row per [IMG] token, in the + language model's hidden width. Run by the shell, in context steps only, never + captured; every rank runs the whole tower (all heads) over its own requests. + + Per image: the 14x14 patch embedding as an unfold and one GEMM, `ln_pre`, the + transformer blocks (RMSNorm, attention with 2-D RoPE, RMSNorm, SwiGLU, each with + its residual), the pre-projector RMSNorm, the 2x2 patch merger and the adapter + (`w_in`, exact GELU, `w_out`). The images of one call are concatenated along the + rows; only the attention needs their boundaries.""" + + def __init__(self, core) -> None: + """Take the tower's weight views and geometry, once, off the real weights.""" + d = _weights.MODEL_WEIGHTS.dims(core) + w = core.w + self.hidden = d.v_hidden + self.heads = d.v_heads + self.head_dim = d.v_head_dim + self.patch = d.v_patch + self.merge = d.v_merge + self.theta = d.v_theta + self.dtype = d.dtype + self.out_hidden = d.hidden + self._validate = step_contract_enabled() + self._rank = core.model_config.mapping.rank + self.patch_w = w["vis_patch"].t() + self.ln_pre = w["vis_ln_pre"] + self.blocks = [ + _VisionBlock( + attn_norm=w[f"v{j}_attn_norm"], + qkv=w[f"v{j}_qkv"].t(), + wo=w[f"v{j}_wo"].t(), + ffn_norm=w[f"v{j}_ffn_norm"], + w13=w[f"v{j}_w13"].t(), + w2=w[f"v{j}_w2"].t(), + ) + for j in range(d.v_layers) + ] + self.proj_norm = w["vis_proj_norm"] + self.merge_w = w["vis_merge"].t() + self.w_in = w["vis_w_in"].t() + self.w_out = w["vis_w_out"].t() + # The prompt token the tower's rows replace, on the host: the executor finds a + # step's image positions with it before anything reaches the device. + self.image_token_ids = torch.tensor([d.image_token_id], dtype=torch.int32) + # The attention op sizes its workspace in place per call and never shrinks it. + self._workspace = torch.empty(0, dtype=torch.int8, device=self.ln_pre.device) + + def _images(self, multimodal_params: list[MultimodalParams]) -> list[tuple]: + """`(pixels [C, H, W], grid rows, grid cols)` of every image, in order. The + mistral-native processor pads the batch and preserves each image's true size.""" + images = [] + unit = self.patch * self.merge + for param in multimodal_params: + image = (param.multimodal_data or {}).get("image") + if image is None: + continue + pixels = image["pixel_values"] + sizes = image.get("image_sizes") + if pixels.ndim != 4 or pixels.shape[1] != 3: + raise ValueError("Image pixels must have shape [N, 3, H, W]") + if sizes is None or len(sizes) != pixels.shape[0]: + raise ValueError("Image sizes must describe every image in the padded batch") + for b in range(pixels.shape[0]): + if len(sizes[b]) != 2: + raise ValueError("Each image size must be [height, width]") + height, width = int(sizes[b][0]), int(sizes[b][1]) + if not (0 < height <= pixels.shape[-2] and 0 < width <= pixels.shape[-1]): + raise ValueError("Image sizes must fit within their padded pixels") + if height % unit or width % unit: + raise ValueError( + f"image {height}x{width} is not a whole number of {unit}-pixel " + "merge tiles; the processor resizes to them" + ) + images.append( + (pixels[b, :, :height, :width], height // self.patch, width // self.patch) + ) + return images + + def forward(self, multimodal_params: list[MultimodalParams]) -> torch.Tensor: + """The rows of every image of `multimodal_params`, request by request, image by + image: `[sum of merged tiles, text hidden]`.""" + images = self._images(multimodal_params) + dev = self.ln_pre.device + if not images: + return torch.empty([0, self.out_hidden], dtype=self.dtype, device=dev) + seg = [gh * gw for _, gh, gw in images] + if self._validate: + logger.info( + "modeling_v2 image forward target=%s rank=%d grids=%s merged_rows=%d", + __name__.rsplit(".", 1)[0], + self._rank, + [(gh, gw) for _, gh, gw in images], + sum(seg) // self.merge**2, + ) + # Each patch as one row, (channel, row, col) within it, patches row-major. + patches = torch.cat( + [ + torch.t( + nn.functional.unfold( + torch.unsqueeze(px.to(device=dev, dtype=self.dtype), 0), + kernel_size=self.patch, + stride=self.patch, + )[0] + ) + for px, _, _ in images + ], + 0, + ) + positions = torch.cat( + [ + torch.reshape( + torch.stack( + torch.meshgrid( + torch.arange(gh, device=dev), + torch.arange(gw, device=dev), + indexing="ij", + ), + -1, + ), + [-1, 2], + ) + for _, gh, gw in images + ], + 0, + ) + step = _vision_step(seg, dev, self._workspace) + + h = flashinfer_rmsnorm(cublas_mm(patches, self.patch_w), self.ln_pre, _VISION_EPS) + x = flashinfer_rmsnorm(h, self.blocks[0].attn_norm, _VISION_EPS) + residual = h + for j, B in enumerate(self.blocks): + attn = _vision_attention(x, B, positions, step, self.heads, self.head_dim, self.theta) + x, residual = flashinfer_fused_add_rmsnorm(attn, residual, B.ffn_norm, _VISION_EPS) + mlp = cublas_mm(flashinfer_silu_and_mul(cublas_mm(x, B.w13)), B.w2) + norm_next = self.blocks[j + 1].attn_norm if j + 1 < len(self.blocks) else self.proj_norm + x, residual = flashinfer_fused_add_rmsnorm(mlp, residual, norm_next, _VISION_EPS) + + # x is the pre-projector-normed tower output. + tiles = torch.cat( + [ + _merge_tiles(xi, gh, gw, self.merge) + for xi, (_, gh, gw) in zip(torch.split(x, seg, 0), images) + ], + 0, + ) + merged = cublas_mm(tiles, self.merge_w) + return cublas_mm(gelu_erf(cublas_mm(merged, self.w_in)), self.w_out) + + +def _image_rows( + vision: Optional[VisionTower], multimodal_params: list[MultimodalParams] +) -> Optional[torch.Tensor]: + """This step's image rows, one per [IMG] position its context chunks cover, in + the order those positions appear; None when the step covers none. + + The tower runs once per request, on the first step that needs it; its rows stay + on the request's multimodal data for the later chunks. A chunk -- under chunked + prefill, or resuming after a reused prefix -- takes rows `[cached, cached + in + chunk)` of its request's, counted by the request's [IMG] cumsum.""" + if vision is None: + if any((p.multimodal_data or {}).get("image") is not None for p in multimodal_params): + raise ValueError( + "an image reached a Mistral Large 4 target whose checkpoint has no vision encoder" + ) + return None + # Filter before encoding and gathering so inactive requests neither run the + # tower nor shift the embedding offsets of active requests in a mixed batch. + multimodal_params = [ + param + for param in multimodal_params + if param.multimodal_runtime is None or param.multimodal_runtime.num_mm_tokens_in_chunk > 0 + ] + embeds = find_input_mm_embeds( + get_multimodal_embeddings(vision.forward, multimodal_params), multimodal_params + ) + if not embeds: + return None + rows = embeds[0] if len(embeds) == 1 else torch.cat(embeds, 0) + return rows if rows.shape[0] else None + + +def _splice_image_rows( + embed: torch.Tensor, + input_ids: torch.Tensor, + rows: torch.Tensor, + mm_token_indices: Optional[torch.Tensor], +) -> torch.Tensor: + """The step's input embeddings with `rows` written over its [IMG] positions, + `mm_token_indices` (which the executor computes on the host from the shell's + `multimodal_token_ids`); every other position keeps its token embedding.""" + if mm_token_indices is None: + raise ValueError("an image step arrived without the executor's mm_token_indices") + if mm_token_indices.shape[0] != rows.shape[0]: + raise ValueError( + f"{mm_token_indices.shape[0]} [IMG] positions in this step but " + f"{rows.shape[0]} image rows for them" + ) + h = nn.functional.embedding(input_ids, embed) + return h.index_copy_(0, mm_token_indices, rows.to(h.dtype)) + + +class ModelingV2MistralLarge4Nvfp4Sm107Tp( + DecoderModelForCausalLM[MistralLarge4TpModelingV2, PretrainedConfig] +): + def __init__(self, model_config: ModelConfig): + cfg = model_config.pretrained_config + super().__init__( + MistralLarge4TpModelingV2(model_config), + config=model_config, + hidden_size=cfg.hidden_size, + vocab_size=cfg.vocab_size, + ) + self.softcap = float(cfg.softmax_tanh_gating) + self._vision: Optional[VisionTower] = None + vis = getattr(cfg, "vision_config", None) + self._image_token_ids = ( + torch.tensor([vis.image_token_id], dtype=torch.int32) if vis is not None else None + ) + + self.spec_config = model_config.spec_config + self.draft_model = None + self.draft_config = None + self.spec_worker = None + if self.spec_config is not None: + # This is allocation geometry for the independently paged BF16 + # draft, not a second registry-resolved model or target cache. + draft_cfg = PretrainedConfig( + num_hidden_layers=int(self.spec_config._draft_model_config["n_layers"]), + hidden_size=cfg.hidden_size, + num_attention_heads=cfg.num_attention_heads, + num_key_value_heads=1, + kv_lora_rank=cfg.kv_lora_rank, + qk_rope_head_dim=cfg.qk_rope_head_dim, + qk_nope_head_dim=cfg.qk_nope_head_dim, + v_head_dim=cfg.v_head_dim, + intermediate_size=cfg.intermediate_size, + vocab_size=cfg.vocab_size, + max_position_embeddings=cfg.max_position_embeddings, + torch_dtype=torch.bfloat16, + ) + self.draft_config = replace( + model_config, + pretrained_config=draft_cfg, + quant_config=QuantConfig(), + quant_config_dict=None, + sparse_attention_config=None, + spec_config=None, + modeling_v2_target=None, + ) + self.draft_model = DraftModel(self, self.draft_config) + self.spec_worker = get_spec_worker( + self.spec_config, + model_config, + model_config.mapping, + use_separate_draft_kv_cache=True, + ) + self.spec_worker.set_draft_model(self.draft_model) + self.epilogue.append(self.spec_worker) + + def set_guided_decoder(self, guided_decoder: CapturableGuidedDecoder) -> bool: + if self.spec_worker is None: + return False + return self.spec_worker.set_guided_decoder(guided_decoder) + + @property + def multimodal_token_ids(self) -> Optional[torch.Tensor]: + return self._image_token_ids + + def forward( + self, + attn_metadata: AttentionMetadata, + input_ids: torch.IntTensor | None = None, + position_ids: torch.IntTensor | None = None, + inputs_embeds: torch.FloatTensor | None = None, + return_context_logits: bool = False, + spec_metadata=None, + lora_params: dict | None = None, + **kwargs, + ) -> torch.Tensor: + """Soft-cap target logits before verification and return the worker result.""" + worker_input_ids = input_ids + multimodal_params = kwargs.get("multimodal_params") or [] + if multimodal_params: + rows = _image_rows(self._vision, multimodal_params) + if rows is not None: + inputs_embeds = _splice_image_rows( + self.model.w["embed"], input_ids, rows, kwargs.get("mm_token_indices") + ) + input_ids = None + if self.spec_worker is None: + logits = super().forward( + attn_metadata, + cast(torch.IntTensor, input_ids), + position_ids, + inputs_embeds, + return_context_logits, + spec_metadata, + lora_params, + **kwargs, + ) + return tanh_softcap(logits, self.softcap) + hidden = self.model( + attn_metadata=attn_metadata, + input_ids=input_ids, + position_ids=position_ids, + inputs_embeds=inputs_embeds, + spec_metadata=spec_metadata, + lora_params=lora_params, + **kwargs, + ) + logits = self.logits_processor( + nn.functional.embedding(spec_metadata.gather_ids, hidden), + self.lm_head, + attn_metadata, + True, + ) + return self.spec_worker( + input_ids=worker_input_ids, + position_ids=position_ids, + hidden_states=hidden, + logits=tanh_softcap(logits, self.softcap), + attn_metadata=attn_metadata, + spec_metadata=spec_metadata, + draft_model=self.draft_model, + resource_manager=kwargs.get("resource_manager"), + ) + + def load_weights(self, weights, *args, **kwargs): + _weights.MODEL_WEIGHTS.load(self, weights) + + def load_checkpoint(self, checkpoint_dir: str) -> None: + _weights.MODEL_WEIGHTS.load_checkpoint(self, checkpoint_dir) + + def post_load_weights(self): + super().post_load_weights() + self.model.post_load_weights() + if step_contract_enabled(): + logger.info( + "modeling_v2 loaded target=%s class=%s rank=%d world=%d vision=%s", + __name__.rsplit(".", 1)[0], + type(self).__name__, + self.model_config.mapping.rank, + self.model_config.mapping.world_size, + getattr(self.config, "vision_config", None) is not None, + ) + if getattr(self.config, "vision_config", None) is not None: + self._vision = VisionTower(self.model) + if self.draft_model is not None: + self.draft_model._build_fused_kv_buffers() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/ops/__init__.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/ops/__init__.py new file mode 100644 index 000000000000..1059445198db --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/ops/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Ops used only by this target. Shared ops stay in the catalog.""" diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/ops/gemm/__init__.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/ops/gemm/__init__.py new file mode 100644 index 000000000000..52a7a9daf028 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/ops/gemm/__init__.py @@ -0,0 +1,2 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/ops/gemm/_fused_fp8_quant_gemv.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/ops/gemm/_fused_fp8_quant_gemv.py new file mode 100644 index 000000000000..9ba37d41447c --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/ops/gemm/_fused_fp8_quant_gemv.py @@ -0,0 +1,468 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Low-M K128 quantization and FP8 GEMM without global activation scratch.""" + +from functools import lru_cache + +import cuda.bindings.driver as cuda +import cutlass +import cutlass.cute as cute +import cutlass.utils as utils +import torch +from cutlass._mlir.dialects import llvm +from cutlass.cute.runtime import from_dlpack +from cutlass.cutlass_dsl import T, dsl_user_op + +from tensorrt_llm._torch.cute_dsl_kernels.blackwell.cute_ptx_utils import mma_sync +from tensorrt_llm._torch.cute_dsl_kernels.blackwell.utils import ( + TRTLLM_ENABLE_PDL, + griddepcontrol_launch_dependents, + griddepcontrol_wait, +) + + +@dsl_user_op +def _exponent(amax: cutlass.Float32, *, loc=None, ip=None): + return cutlass.Int32( + llvm.inline_asm( + T.i32(), + [amax.ir_value(loc=loc, ip=ip)], + "{ .reg .f32 rcp, raw; .reg .b32 bits, mant, inc; .reg .pred p; " + "rcp.approx.ftz.f32 rcp, 0f43e00000; mul.f32 raw, $1, rcp; " + "mov.b32 bits, raw; and.b32 mant, bits, 0x7fffff; " + "setp.ne.u32 p, mant, 0; selp.u32 inc, 1, 0, p; " + "shr.u32 bits, bits, 23; add.u32 $0, bits, inc; }", + "=r,f", + has_side_effects=False, + loc=loc, + ip=ip, + ) + ) + + +@dsl_user_op +def _float_bits(bits: cutlass.Int32, *, loc=None, ip=None): + return cutlass.Float32( + llvm.inline_asm( + T.f32(), + [bits.ir_value(loc=loc, ip=ip)], + "mov.b32 $0, $1;", + "=f,r", + has_side_effects=False, + loc=loc, + ip=ip, + ) + ) + + +@dsl_user_op +def _scale_code(value: cutlass.Float32, *, loc=None, ip=None): + return cutlass.Int32( + llvm.inline_asm( + T.i32(), + [value.ir_value(loc=loc, ip=ip)], + "{ .reg .b32 bits; mov.b32 bits, $1; shr.u32 $0, bits, 23; }", + "=r,f", + has_side_effects=False, + loc=loc, + ip=ip, + ) + ) + + +# Weight loads are volatile so they stay ahead of griddepcontrol.wait. +@dsl_user_op +def _load_words(address: cutlass.Int64, *, loc=None, ip=None): + words = llvm.inline_asm( + llvm.StructType.get_literal([T.i32()] * 4), + [address.ir_value(loc=loc, ip=ip)], + "ld.global.nc.L1::no_allocate.v4.b32 {$0, $1, $2, $3}, [$4];", + "=r,=r,=r,=r,l", + has_side_effects=True, + is_align_stack=False, + loc=loc, + ip=ip, + ) + return [cutlass.Int32(llvm.extractvalue(T.i32(), words, [i], loc=loc, ip=ip)) for i in range(4)] + + +@dsl_user_op +def _load_byte(address: cutlass.Int64, *, loc=None, ip=None): + return cutlass.Int32( + llvm.inline_asm( + T.i32(), + [address.ir_value(loc=loc, ip=ip)], + "ld.global.nc.u8 $0, [$1];", + "=r,l", + has_side_effects=True, + loc=loc, + ip=ip, + ) + ) + + +class FusedFp8QuantGemv: + """Each CTA quantizes one K slice and computes a 16x64 output tile. + + Quantization is repeated across output tiles. Activation codes and K128 + scales stay in shared memory. Weight scales use R128c4 with each K128 + exponent replicated over four K32 slots, as in prepare_weight_scale. + Split-K writes disjoint FP32 partials, followed by one BF16 reduction. + + Each lane streams 2x16 bytes of one weight row per K128 slot: words + 4q..4q+3 and 16+4q..16+4q+3 for quad q. MMA step s pairs word 4q+s with + 16+4q+s, a K permutation applied identically to the activation fragment, + which therefore reads the same two 16-byte chunks from shared memory. Up to + ``prefetch`` slots of weights are issued before the PDL wait. + """ + + def __init__(self, m: int, n: int, k: int, split_k: int, warps: int = 8, prefetch: int = 8): + self.m, self.n, self.k = m, n, k + self.split_k = split_k + self.warps = warps + self.tile_n = warps * 8 + self.slots = k // 128 // split_k + self.prefetch = min(prefetch, self.slots) + self.buffers = max(self.prefetch, 1) + # MMA rows 8..15 are constant zero operands when M <= 8. + self.rows = 8 if m <= 8 else 16 + + def shared_chunk(self, sa, slot, row, chunk): + # Odd rows rotate by four 16-byte chunks so the two rows read by each + # quarter warp cover all banks. + offset = (slot * self.rows + row) * 32 + (chunk + row % 2 * 4) % 8 * 4 + return cute.make_tensor( + cute.make_ptr( + cutlass.Int32, + (sa.iterator + offset).toint(), + cute.AddressSpace.smem, + assumed_align=16, + ), + cute.make_layout((4,)), + ) + + def load_weights(self, w, ws, weights, scales, buffer, column, group, quad, kb): + row = cute.domain_offset((column + group, kb * 32 + quad * 4), w).iterator.toint() + low = _load_words(row) + high = _load_words(row + 64) + for i in range(4): + weights[buffer, i] = low[i] + weights[buffer, i + 4] = high[i] + for i in range(2): + nc = column + quad * 2 + i + sf_offset = ( + (nc // 128) * (self.k // 128) * 512 + + kb * 512 + + (nc % 32) * 16 + + (nc % 128 // 32) * 4 + ) + scales[buffer, i] = _load_byte(cute.domain_offset((sf_offset,), ws).iterator.toint()) + + @cute.jit + def quantize_tile(self, x, sa, ss, slot, kb, tid): + # Eight lanes own each complete K128 row; padded MMA rows are zero. + if tid < 128: + row = tid // 8 + chunk = tid % 8 + if row < ((self.m + 3) // 4) * 4: + values = cute.make_rmem_tensor((16,), cutlass.Float32) + values.fill(0.0) + if row < self.m: + source = cute.make_tensor( + cute.make_ptr( + cutlass.BFloat16, + cute.domain_offset((row, kb * 128 + chunk * 16), x).iterator.toint(), + cute.AddressSpace.gmem, + assumed_align=16, + ), + cute.make_layout((16,)), + ) + loaded = cute.make_rmem_tensor((16,), cutlass.BFloat16) + atom = cute.make_copy_atom( + cute.nvgpu.CopyUniversalOp(), cutlass.BFloat16, num_bits_per_copy=128 + ) + cute.copy(atom, source, loaded) + values.store(loaded.load().to(cutlass.Float32)) + amax = cutlass.Float32(1.0e-10) + for i in cutlass.range_constexpr(16): + amax = cute.arch.fmax(amax, cute.arch.fmax(values[i], -values[i])) + for shift in cutlass.range_constexpr(3): + amax = cute.arch.fmax( + amax, cute.arch.shuffle_sync_bfly(amax, offset=1 << shift) + ) + exponent = _exponent(amax) + dequant = _float_bits(exponent << 23) + quant = _float_bits((254 - exponent) << 23) + packed = cute.make_rmem_tensor((16,), cutlass.Float8E4M3FN) + for i in cutlass.range_constexpr(16): + packed[i] = (values[i] * quant).to(cutlass.Float8E4M3FN) + cute.autovec_copy( + cute.recast_tensor(packed, cutlass.Int32), + self.shared_chunk(sa, slot, row, chunk), + ) + if chunk == 0: + ss[row, slot] = dequant + else: + zeros = cute.make_rmem_tensor((4,), cutlass.Int32) + zeros.fill(0) + cute.autovec_copy(zeros, self.shared_chunk(sa, slot, row, chunk)) + if chunk == 0: + ss[row, slot] = cutlass.Float32(1.0) + + @cute.kernel + def kernel(self, x: cute.Tensor, w: cute.Tensor, ws: cute.Tensor, y: cute.Tensor): + tid, _, _ = cute.arch.thread_idx() + bn, sk, _ = cute.arch.block_idx() + lane = tid % 32 + warp = tid // 32 + group = lane // 4 + quad = lane % 4 + slots = self.slots + prefetch = self.prefetch + buffers = self.buffers + column = bn * self.tile_n + warp * 8 + weights = cute.make_rmem_tensor((buffers, 8), cutlass.Int32) + weight_scales = cute.make_rmem_tensor((buffers, 2), cutlass.Int32) + # Weights are not written by the preceding kernel, so their HBM latency + # overlaps its tail and the activation quantization below. + for slot in cutlass.range_constexpr(prefetch): + self.load_weights( + w, ws, weights, weight_scales, slot, column, group, quad, sk * slots + slot + ) + if cutlass.const_expr(TRTLLM_ENABLE_PDL): + griddepcontrol_wait() + allocator = utils.SmemAllocator() + sa = allocator.allocate_tensor( + cutlass.Int32, cute.make_layout((slots * self.rows * 32,)), byte_alignment=16 + ) + ss = allocator.allocate_tensor( + cutlass.Float32, cute.make_layout((self.rows, slots), stride=(1, self.rows)) + ) + total = cute.make_rmem_tensor((4,), cutlass.Float32) + total.fill(0.0) + # Assign consecutive warps to different K128 tiles so low-M rows + # quantize concurrently before the single consumer barrier. + for task in range(warp, self.rows // 4 * slots, self.warps): + slot = task % slots + row_group = task // slots + self.quantize_tile(x, sa, ss, slot, sk * slots + slot, row_group * 32 + lane) + cute.arch.sync_threads() + ar = cute.make_rmem_tensor((4,), cutlass.Int32) + ar.fill(0) + br = cute.make_rmem_tensor((2,), cutlass.Int32) + for slot in cutlass.range_constexpr(slots): + kb = sk * slots + slot + buffer = slot % buffers + if cutlass.const_expr(prefetch == 0): + self.load_weights(w, ws, weights, weight_scales, 0, column, group, quad, kb) + low = cute.make_rmem_tensor((4,), cutlass.Int32) + high = cute.make_rmem_tensor((4,), cutlass.Int32) + cute.autovec_copy(self.shared_chunk(sa, slot, group, quad), low) + cute.autovec_copy(self.shared_chunk(sa, slot, group, quad + 4), high) + if cutlass.const_expr(self.rows == 16): + low8 = cute.make_rmem_tensor((4,), cutlass.Int32) + high8 = cute.make_rmem_tensor((4,), cutlass.Int32) + cute.autovec_copy(self.shared_chunk(sa, slot, group + 8, quad), low8) + cute.autovec_copy(self.shared_chunk(sa, slot, group + 8, quad + 4), high8) + accum = cute.make_rmem_tensor((4,), cutlass.Float32) + accum.fill(0.0) + for step in cutlass.range_constexpr(4): + ar[0] = low[step] + ar[2] = high[step] + if cutlass.const_expr(self.rows == 16): + ar[1] = low8[step] + ar[3] = high8[step] + br[0] = weights[buffer, step] + br[1] = weights[buffer, step + 4] + accum.store( + mma_sync( + cute.recast_tensor(ar, cutlass.Float8E4M3FN), + cute.recast_tensor(br, cutlass.Float8E4M3FN), + accum, + ) + ) + for i in cutlass.range_constexpr(self.rows // 4): + mr = group + (i // 2) * 8 + combined_exponent = _scale_code(ss[mr, slot]) + weight_scales[buffer, i % 2] - 254 + half_exponent = combined_exponent // 2 + # Balanced powers avoid overflow in an intermediate scale product. + scale_lo = _float_bits((half_exponent + 127) << 23) + scale_hi = _float_bits((combined_exponent - half_exponent + 127) << 23) + total[i] += (accum[i] * scale_lo) * scale_hi + if cutlass.const_expr(prefetch > 0 and slot + prefetch < slots): + self.load_weights( + w, ws, weights, weight_scales, buffer, column, group, quad, kb + prefetch + ) + for i in cutlass.range_constexpr(self.rows // 4): + mr = group + (i // 2) * 8 + nc = column + quad * 2 + i % 2 + if mr < self.m: + if cutlass.const_expr(self.split_k == 1): + y[mr, nc] = total[i].to(cutlass.BFloat16) + else: + y[sk, mr, nc] = total[i] + + if cutlass.const_expr(TRTLLM_ENABLE_PDL): + griddepcontrol_launch_dependents() + + @cute.kernel + def debug_kernel(self, x: cute.Tensor, codes: cute.Tensor, scales: cute.Tensor): + if cutlass.const_expr(TRTLLM_ENABLE_PDL): + griddepcontrol_wait() + tid, _, _ = cute.arch.thread_idx() + kb, _, _ = cute.arch.block_idx() + allocator = utils.SmemAllocator() + sa = allocator.allocate_tensor( + cutlass.Int32, cute.make_layout((16 * 32,)), byte_alignment=16 + ) + ss = allocator.allocate_tensor(cutlass.Float32, cute.make_layout((16, 1), stride=(1, 16))) + self.quantize_tile(x, sa, ss, 0, kb, tid) + cute.arch.sync_threads() + row, word = tid // 32, tid % 32 + for group in cutlass.range_constexpr(4): + mr = row + group * 4 + if mr < self.m: + codes[mr, kb * 32 + word] = self.shared_chunk(sa, 0, mr, word // 4)[word % 4] + if word == 0: + scales[mr, kb] = _scale_code(ss[mr, 0]).to(cutlass.Uint8) + + if cutlass.const_expr(TRTLLM_ENABLE_PDL): + griddepcontrol_launch_dependents() + + @cute.jit + def debug(self, x: cute.Tensor, codes: cute.Tensor, scales: cute.Tensor, stream: cuda.CUstream): + self.debug_kernel(x, codes, scales).launch( + grid=(self.k // 128, 1, 1), block=(128, 1, 1), stream=stream, use_pdl=TRTLLM_ENABLE_PDL + ) + + @cute.kernel + def reduce(self, partial: cute.Tensor, y: cute.Tensor): + if cutlass.const_expr(TRTLLM_ENABLE_PDL): + griddepcontrol_wait() + tid, _, _ = cute.arch.thread_idx() + bid, _, _ = cute.arch.block_idx() + idx = bid * 256 + tid + if idx < self.m * self.n: + row, col = idx // self.n, idx % self.n + value = cutlass.Float32(0.0) + for sk in cutlass.range_constexpr(self.split_k): + value += partial[sk, row, col] + y[row, col] = value.to(cutlass.BFloat16) + + if cutlass.const_expr(TRTLLM_ENABLE_PDL): + griddepcontrol_launch_dependents() + + @cute.jit + def single( + self, x: cute.Tensor, w: cute.Tensor, ws: cute.Tensor, y: cute.Tensor, stream: cuda.CUstream + ): + self.kernel(x, w, ws, y).launch( + grid=(self.n // self.tile_n, 1, 1), + block=(32 * self.warps, 1, 1), + stream=stream, + use_pdl=TRTLLM_ENABLE_PDL, + ) + + @cute.jit + def __call__( + self, + x: cute.Tensor, + w: cute.Tensor, + ws: cute.Tensor, + partial: cute.Tensor, + y: cute.Tensor, + stream: cuda.CUstream, + ): + self.kernel(x, w, ws, partial).launch( + grid=(self.n // self.tile_n, self.split_k, 1), + block=(32 * self.warps, 1, 1), + stream=stream, + use_pdl=TRTLLM_ENABLE_PDL, + ) + self.reduce(partial, y).launch( + grid=((self.m * self.n + 255) // 256, 1, 1), + block=(256, 1, 1), + stream=stream, + use_pdl=TRTLLM_ENABLE_PDL, + ) + + +@lru_cache(maxsize=128) +def _compile(m: int, n: int, k: int, row_stride: int, split_k: int, warps: int, device: int): + # Setup allocations only describe tensor layouts; the executable owns no data. + device = torch.device("cuda", device) + x = torch.empty_strided((m, k), (row_stride, 1), dtype=torch.bfloat16, device=device) + w = torch.empty((n, k // 4), dtype=torch.int32, device=device) + ws = torch.empty(((n + 127) // 128 * 128 * k // 32,), dtype=torch.uint8, device=device) + y = torch.empty((m, n), dtype=torch.bfloat16, device=device) + kernel = FusedFp8QuantGemv(m, n, k, split_k, warps) + stream = cute.runtime.make_fake_stream(use_tvm_ffi_env_stream=True) + if split_k == 1: + return cute.compile( + kernel.single, + *(from_dlpack(t, assumed_align=16) for t in (x, w, ws, y)), + stream, + options="--enable-tvm-ffi", + ) + partial = torch.empty((split_k, m, n), dtype=torch.float32, device=device) + return cute.compile( + FusedFp8QuantGemv(m, n, k, split_k, warps), + *(from_dlpack(t, assumed_align=16) for t in (x, w, ws, partial, y)), + cute.runtime.make_fake_stream(use_tvm_ffi_env_stream=True), + options="--enable-tvm-ffi", + ) + + +def fused_fp8_quant_gemv( + input: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + *, + split_k: int | None = None, + warps: int = 8, +) -> torch.Tensor: + """SM107 BF16 to FP8 linear with replicated K128 weight scales. + + Weight scale storage is R128c4. Its four K32 scale bytes for every K128 + block must be identical; independent K32 weight scales are unsupported. + Weights and weight scales are read before the PDL wait, so they must not be + written by the immediately preceding kernel on the stream. + """ + m, k = input.shape + n = weight.shape[0] + if warps not in (4, 8): + raise ValueError("Fused FP8 GEMV supports four or eight warps per CTA") + if not (1 <= m <= 16 and k % 128 == 0 and n % (warps * 8) == 0): + raise ValueError("Fused FP8 GEMV requires M <= 16, K divisible by 128, and full N tiles") + if input.stride(-1) != 1 or not weight.is_contiguous(): + raise ValueError("Fused FP8 GEMV requires contiguous K dimensions") + if split_k is None: + split_k = 8 if k >= 3072 else 1 + if (k // 128) % split_k: + raise ValueError("Split-K must divide the number of K128 blocks") + out = torch.empty((m, n), dtype=torch.bfloat16, device=input.device) + fn = _compile(m, n, k, input.stride(0), split_k, warps, input.device.index) + if split_k == 1: + fn(input, weight.view(torch.int32), weight_scale.reshape(-1), out) + else: + partial = torch.empty((split_k, m, n), dtype=torch.float32, device=input.device) + fn(input, weight.view(torch.int32), weight_scale.reshape(-1), partial, out) + return out + + +def debug_quantization(input: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]: + """Export the exact shared-memory quantization for byte-level T0 checks.""" + m, k = input.shape + codes = torch.empty((m, k), dtype=torch.float8_e4m3fn, device=input.device) + scales = torch.empty((m, k // 128), dtype=torch.uint8, device=input.device) + kernel = FusedFp8QuantGemv(m, 64, k, 1) + fn = cute.compile( + kernel.debug, + from_dlpack(input, assumed_align=16), + from_dlpack(codes.view(torch.int32), assumed_align=16), + from_dlpack(scales, assumed_align=1), + cute.runtime.make_fake_stream(use_tvm_ffi_env_stream=True), + options="--enable-tvm-ffi", + ) + fn(input, codes.view(torch.int32), scales) + return codes, scales diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/ops/gemm/cute_dsl_fused_fp8_block_scale_linear.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/ops/gemm/cute_dsl_fused_fp8_block_scale_linear.py new file mode 100644 index 000000000000..f8011d807520 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/ops/gemm/cute_dsl_fused_fp8_block_scale_linear.py @@ -0,0 +1,132 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Experimental low-M fused K128 activation quantization and FP8 linear.""" + +import torch + +from tensorrt_llm._torch.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.gemm.cute_dsl_fp8_block_scale_linear import ( # noqa: E501 + cute_dsl_fp8_block_scale_linear as baseline, +) + +# (M, N, K) -> (Split-K slices, warps per CTA) for the decode target's per-rank FP8 projections +# at TP8 and TP4, M = 1, 2, 4, 8, 12 and 16. Each cell holds the fastest tactic at the kernel's +# default prefetch depth on VR200. A shape is absent at an M where no tactic beat the MXFP8 +# kernel by 3% with both L2-resident and streamed weights, and runs that kernel there: wq_b, +# dense down and 6144x6144 above M=8, and TP4 dense gate/up from M=4. +_TACTICS = { + # TP8 wq + (1, 1024, 6144): (12, 8), + (2, 1024, 6144): (12, 8), + (4, 1024, 6144): (24, 8), + (8, 1024, 6144): (24, 4), + (12, 1024, 6144): (24, 8), + (16, 1024, 6144): (24, 8), + # TP8 shared-expert gate/up + (1, 1536, 6144): (16, 4), + (2, 1536, 6144): (16, 4), + (4, 1536, 6144): (24, 4), + (8, 1536, 6144): (24, 4), + (12, 1536, 6144): (24, 8), + (16, 1536, 6144): (24, 8), + # TP8 and TP4 indexer wq_b + (1, 4096, 6144): (12, 4), + (2, 4096, 6144): (12, 4), + (4, 4096, 6144): (12, 4), + (8, 4096, 6144): (12, 8), + # TP8 shared-expert down + (1, 6144, 768): (1, 4), + (2, 6144, 768): (1, 4), + (4, 6144, 768): (1, 4), + (8, 6144, 768): (1, 4), + (12, 6144, 768): (6, 8), + (16, 6144, 768): (6, 4), + # TP8 wo + (1, 6144, 1024): (1, 4), + (2, 6144, 1024): (1, 4), + (4, 6144, 1024): (1, 4), + (8, 6144, 1024): (4, 4), + (12, 6144, 1024): (4, 8), + (16, 6144, 1024): (4, 8), + # TP8 dense down + (1, 6144, 3072): (4, 8), + (2, 6144, 3072): (4, 8), + (4, 6144, 3072): (4, 8), + (8, 6144, 3072): (6, 8), + # TP8 dense gate/up, TP4 dense down + (1, 6144, 6144): (12, 4), + (2, 6144, 6144): (12, 4), + (4, 6144, 6144): (12, 4), + (8, 6144, 6144): (12, 8), + # TP4 wq + (1, 2048, 6144): (16, 4), + (2, 2048, 6144): (16, 4), + (4, 2048, 6144): (16, 4), + (8, 2048, 6144): (12, 8), + (12, 2048, 6144): (24, 8), + (16, 2048, 6144): (24, 8), + # TP4 shared-expert gate/up + (1, 3072, 6144): (8, 8), + (2, 3072, 6144): (8, 8), + (4, 3072, 6144): (8, 8), + (8, 3072, 6144): (12, 8), + (12, 3072, 6144): (24, 8), + (16, 3072, 6144): (12, 8), + # TP4 shared-expert down + (1, 6144, 1536): (4, 4), + (2, 6144, 1536): (6, 4), + (4, 6144, 1536): (6, 4), + (8, 6144, 1536): (6, 4), + (12, 6144, 1536): (6, 8), + (16, 6144, 1536): (6, 8), + # TP4 wo + (1, 6144, 2048): (8, 4), + (2, 6144, 2048): (8, 8), + (4, 6144, 2048): (8, 8), + (8, 6144, 2048): (4, 8), + (12, 6144, 2048): (8, 8), + (16, 6144, 2048): (8, 8), + # TP4 dense gate/up + (1, 12288, 6144): (12, 4), + (2, 12288, 6144): (12, 4), +} + + +@torch.library.custom_op("trtllm::cute_dsl_fused_fp8_block_scale_linear", mutates_args=()) +def _fused(input: torch.Tensor, weight: torch.Tensor, weight_scale: torch.Tensor) -> torch.Tensor: + from ._fused_fp8_quant_gemv import fused_fp8_quant_gemv + + if input.data_ptr() % 16 or input.stride(0) % 8: + return baseline(input.contiguous(), weight, weight_scale) + split_k, warps = _TACTICS[(input.shape[0], weight.shape[0], input.shape[1])] + return fused_fp8_quant_gemv(input, weight, weight_scale, split_k=split_k, warps=warps) + + +@_fused.register_fake +def _fake(input: torch.Tensor, weight: torch.Tensor, weight_scale: torch.Tensor) -> torch.Tensor: + return input.new_empty((input.shape[0], weight.shape[0])) + + +class CuteDslFusedFp8BlockScaleLinear(OpWrapper): + """Low-M SM107 GEMM with K128 weight exponents replicated over K32 slots.""" + + ARCHS = frozenset({Arch.SM_107}) + CELLS = tuple( + Cell(f"SM107 fused FP8 M={m}, N={n}, K={k}", dict(m=m, n=n, k=k)) for m, n, k in _TACTICS + ) + + def __call__( + self, input: torch.Tensor, weight: torch.Tensor, weight_scale: torch.Tensor + ) -> torch.Tensor: + m, k = input.shape + if (m, weight.shape[0], k) in _TACTICS and input.stride(-1) == 1 and weight.is_contiguous(): + return _fused(input, weight, weight_scale) + return baseline(input, weight, weight_scale) + + def reference( + self, input: torch.Tensor, weight: torch.Tensor, weight_scale: torch.Tensor + ) -> torch.Tensor: + return baseline.reference(input, weight, weight_scale) + + +cute_dsl_fused_fp8_block_scale_linear = CuteDslFusedFp8BlockScaleLinear() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/ops/gemm/low_m_cublas_mm.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/ops/gemm/low_m_cublas_mm.py new file mode 100644 index 000000000000..c810452ca99d --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/ops/gemm/low_m_cublas_mm.py @@ -0,0 +1,107 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""SM107 narrow BF16 projections: fixed direct-kernel tactics, the cublas_mm entry otherwise.""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch.modeling_v2.catalog._op import Arch, Cell +from tensorrt_llm._torch.modeling_v2.catalog.gemm.cublas_mm import CublasMm + +# (M, N, K) -> DirectTactic(block_size, outputs_per_block, rows_per_block). M is the decode +# rows of a captured batch: 3 per sequence for the draft's kv_a, 4 per sequence for the +# target's kv_a and indexer projections. Every CTA re-reads all of A, so the direct kernel's +# cost grows with M x N while cuBLAS's stays nearly flat; 16x512, 16x576 and larger M stay on +# cuBLAS. +_LOW_M_DIRECT_TACTICS: dict[tuple[int, int, int], tuple[int, int, int]] = { + (3, 576, 6144): (128, 1, 3), + (4, 160, 6144): (128, 2, 4), + (4, 512, 6144): (128, 1, 4), + (4, 576, 6144): (128, 1, 4), + (6, 576, 6144): (128, 2, 6), + (8, 160, 6144): (128, 2, 8), + (8, 512, 6144): (128, 2, 8), + (8, 576, 6144): (128, 2, 8), + (12, 576, 6144): (64, 2, 4), + (16, 160, 6144): (128, 2, 8), +} + + +def _low_m_direct_tactic( + mat_a: torch.Tensor, + mat_b: torch.Tensor, + bias: torch.Tensor | None, + out_dtype: torch.dtype | None, + output_buffer_kind: int, + group: list[int] | None, +) -> tuple[int, int, int] | None: + if ( + bias is not None + or output_buffer_kind != 0 + or group is not None + or mat_a.dtype != torch.bfloat16 + or mat_b.dtype != torch.bfloat16 + or out_dtype not in (None, torch.bfloat16) + or mat_a.ndim != 2 + or mat_b.ndim != 2 + or not mat_a.is_cuda + or mat_a.device != mat_b.device + or not mat_a.is_contiguous() + or mat_b.stride() != (1, mat_b.shape[0]) + or mat_a.shape[1] != mat_b.shape[0] + or mat_a.data_ptr() % 32 != 0 + or mat_b.data_ptr() % 32 != 0 + ): + return None + tactic = _LOW_M_DIRECT_TACTICS.get((mat_a.shape[0], mat_b.shape[1], mat_a.shape[1])) + if tactic is None or torch.cuda.get_device_capability(mat_a.device) != (10, 7): + return None + return tactic + + +class LowMCublasMm(CublasMm): + """SM107 narrow BF16 projections, with the cuBLAS contract for other calls. + + Fixed direct-kernel tactics accumulate in FP32 and round once to BF16. + FP32 outputs, fused bias, collective buffers, operands below 32-byte alignment, other layouts/shapes, and other + architectures use the original cuBLAS op. The direct kernel uses ordinary + stream ordering, like the cuBLAS path it replaces. + """ + + ARCHS = frozenset({Arch.SM_107}) + note = """ + Exact finite BF16 direct tactics at the cells below. Other calls use + CublasMm, including FP32 router logits and collective output buffers. + There is no mutable tactic cache: dtype, layout, output role, architecture, + and exact matrix dimensions gate the fixed table on every call. + """ + CELLS = tuple( + Cell( + why=f"SM107 low-M direct projection {m}x{k}x{n}", + spec=dict(m=m, k=k, n=n, dtype=torch.bfloat16, bias=False, out_dtype=None), + ) + for m, n, k in _LOW_M_DIRECT_TACTICS + ) + + def __call__( + self, + mat_a: torch.Tensor, + mat_b: torch.Tensor, + bias: torch.Tensor | None = None, + out_dtype: torch.dtype | None = None, + output_buffer_kind: int = 0, + group: list[int] | None = None, + ) -> torch.Tensor: + tactic = _low_m_direct_tactic(mat_a, mat_b, bias, out_dtype, output_buffer_kind, group) + if tactic is None: + return torch.ops.trtllm.cublas_mm( + mat_a, mat_b, bias, out_dtype, output_buffer_kind, group + ) + from flashinfer.gemm.kernels.dense_bf16_gemm_direct import DirectTactic, run_direct_dense + + out = mat_a.new_empty((mat_a.shape[0], mat_b.shape[1])) + return run_direct_dense(mat_a, mat_b, out, False, DirectTactic(*tactic)) + + +low_m_cublas_mm = LowMCublasMm() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/weights.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/weights.py new file mode 100644 index 000000000000..61555f501476 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/weights.py @@ -0,0 +1,1378 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Weight table and loader: mistral-large-4-nvfp4 / sm_107 / tp, the language model. + +The checkpoint is Mistral-native (`consolidated-*.safetensors`, `layers.N.attention.wq`, ...) +and ModelOpt `MIXED_PRECISION`: the routed experts NVFP4, the other big linears FP8 128x128 +block (`FP8_PB_WO`), everything else bf16. Every rank is handed the whole of it, so every split +is this table's, every width read off the mapping (`tp_size` and `moe_tp_size` 4 on one node, or +8 over two nodes of four, and `moe_ep_size` 1): + +* **tensor-parallel slices** -- `wq` by output rows and `wo` by input columns (this rank's 16 + heads at 4 ranks, 8 heads -- 1024 rows or columns -- at 8); `wkv_b` by heads; the MLA SSMax + `s`/`b` by heads; the dense layer's and the shared expert's `w1`/`w3` by rows and `w2` by + columns (6144 and 1536 per rank at 4 ranks, 3072 and 768 at 8); the shell's `lm_head` by + vocabulary rows (51200 or 25600 per rank). Every slice is whole 128-blocks, and a block scale + is sliced on the same boundary. The slice is taken on the checkpoint entry before the copy to + the device, so a rank moves only its own bytes. That is the `shard` column of `W`. +* **routed-expert width slices** -- every rank loads all 128 routed experts (the window + `[expert_offset, expert_offset + local_experts)` that `moe_ep_size` 1 makes `[0, 128)`), each + cut along its intermediate width to `moe_inter = moe_intermediate_size / moe_tp_size` (1536 at + 4 ranks, 768 at 8) at `moe_tp_rank * moe_inter`: `w1` and `w3`, codes and block scales alike, + by output rows `[Ir, Ir + I)`; `w2`'s codes by input bytes `[Ir/2, Ir/2 + I/2)` (two e2m1 per + byte: 768 or 384 bytes) and its block scales by columns `[Ir/16, Ir/16 + I/16)` (one per 16 + inputs, 96 or 48 columns, so the cut splits no block). These are `shard`s too: a row read, and + for `w2` a strided column read, of the checkpoint entry. +* **replicated** -- `wkv_a_with_mqa` and `kv_a_norm`, the whole indexer (all 32 of its heads), + the router, the embedding, every norm, `lerope.alpha`, and the routed experts' per-tensor + scalars. + +Layer sets come from the config: `wkv_a_with_mqa` and `kv_a_norm` exist on the KV-owning layers +only (`kv_source_layers[i] == i`; the sharers read their source's latent), the indexer on the +`index_sharing == 'full'` layers only, the dense MLP on the first `first_k_dense_replace` +layers and the router, shared expert and routed experts on the rest. + +Load-time layouts, each the one its consumer reads: + +* `wkv_b` regrouped per head `[k_nope | v]` -> `[all k_nope ; all v]` over this rank's heads + (k_nope is 64 wide on RoPE layers and 128 on NoPE layers); +* the dense MLP and the shared expert `[w1 ; w3]`, gate rows first, the half order + `flashinfer_silu_and_mul` reads; +* FP8 linears (`wq`, `wo`, the indexer's `wq_b`, the dense and shared `w1`/`w3`/`w2`): e4m3 + `[N, K]` as stored, and the fp32 block scales -- stored as `.weight_scale` + `[N/128, 1, K/128, 1]`, squeezed here to the `[N/128, K/128]` grid -- in DeepGEMM's packed + UE8M0 operand, `transform_sf_into_required_layout(recipe=(1, 128, 128))`: what + `FP8BlockScalesLinearMethod.transform_weights` installs as `weight_scale` on sm_100f. That + operand is an MN-major `[N, ceil(K/512)]` int32 view; the parameter holds the dense + `[ceil(K/512), N]` behind it. The layer views convert `.t()` to Rubin R128c4 K32 scales once, + replicating each K128 exponent four times. Where K/128 is not a multiple of four + the last word of each row is partly padding: the shared expert's `w2` at K 768 (8 ranks) has 6 + blocks in 2 words, and bytes 2 and 3 of the second word are zero. The method's + `resmooth_to_fp8_e8m0` is not run: every scale is asserted an exact power of two, which makes + it a power-of-two re-exponent of the same values; +* the indexer's `wk` and `weights_proj` stacked into one `[128 + 32, hidden]` bf16 GEMM operand; + its `k_norm` and `lerope.alpha` widened to fp32. + +The routed experts are this rank's width slice of every expert, stacked on an expert axis, in +the NVFP4 MoE runner's layout: e2m1 codes, two per byte, `[E, 2I, H/2]` / `[E, H, I/2]` uint8 +and their e4m3 block scales, one per 16 inputs, `[E, 2I, H/16]` / `[E, H, I/16]`, with `E` 128 +and `I` this rank's 1536 (4 ranks) or 768 (8 ranks). Each sliced source lands as stored -- FC1's +rows `[w3 ; w1]`, up rows first, the order the runner's FC1 concatenates them in -- and once +every source of an expert stack has landed, its row's `layout` rearranges each expert's slice in +place (the runner reads nothing else, and a wrong layout costs 300-3400 bf16 ulp and raises +nothing): + +* FC1, codes and block scales alike: the halves interleaved (row `2i` up row `i`, row `2i + 1` + gate row `i`), then the 32-row block shuffle (within each 32 rows, row `4u + v` moves to + `8v + u`); +* FC2: the 32-row block shuffle; +* both block scales: then the 128x4 swizzle (`torch.ops.trtllm.block_scale_interleave`'s order, + written here in torch so the table also runs on meta tensors). + +The layout is a function of the slice's shape, so it runs on the slice and never on a whole +expert: the swizzle's byte order depends on the scale column count (FC2's 96 or 48 here, not +the stored 384), and the interleave pairs this rank's up and gate rows. At either per-rank width +every rule is met without padding: FC1's 2 x 1536 = 3072 or 2 x 768 = 1536 rows and FC2's 6144 +are multiples of the 32-row shuffle block and of the swizzle's 128 rows, and FC1's 384 and FC2's +96 or 48 scale columns of its 4. + +The runner's output scalars are not derived here: the four per-tensor scalars are handed over +as stored, unsliced -- `isc1` and `isc2`, the one FC1 and FC2 activation `input_scale` of the +layer, and per local expert `ws2_13` (`w1`'s `weight_scale_2`, which `w3` shares) and `ws2_2`. +ModelOpt stores these as reciprocals of the global scales a quantizer divides by. A per-tensor +scale holds for every slice of its tensor, so each rank's slice dequantizes with the whole +expert's scalars. + +Before any of that is read, `load` reads the per-tensor scalars of all the layer's experts and +asserts what the forward relies on: `w1` and `w3` store the same `input_scale` and +`weight_scale_2` on every expert, so one FC1 alpha serves the stacked `[w3 ; w1]`; and each +projection stores one `input_scale` for every expert, so one NVFP4 quantization of the +replicated activations is right for every expert on every rank, and the ranks' partial +outputs, each over its slice of every expert, sum. + +The BF16 vision encoder, pre-projector norm, patch merger and language adapter are +replicated on every rank when the configuration declares a vision encoder. + +`expected_unconsumed` accounts for off-rank routed experts and disabled DSpark draft weights. +Off-rank expert scalars are still read to check shared scales; their codes and block scales +are never read. +At `moe_ep_size` 1 every expert belongs to the rank and is read, sliced. +""" + +from __future__ import annotations + +import collections +import functools +import json +import os +from dataclasses import dataclass +from pathlib import Path +from types import SimpleNamespace +from typing import Any, Callable + +import torch +import torch.nn as nn +from safetensors import safe_open + +from tensorrt_llm.logger import logger +from tensorrt_llm.quantization.utils.fp8_utils import transform_sf_into_required_layout + +F8, F32, I32, U8 = torch.float8_e4m3fn, torch.float32, torch.int32, torch.uint8 + +#: The FP8 block edge on both weight axes (the checkpoint's `FP8_PB_WO` 128x128 blocks), and +#: the K granularity of the DeepGEMM scale recipe `(1, 128, 128)`. +BLOCK = 128 +#: UE8M0 bytes per int32 word of the DeepGEMM scale operand: four K blocks. +SF_PACK = 4 +#: NVFP4: one e4m3 block scale per 16 inputs (the checkpoint's `group_size`), two e2m1 codes +#: per byte. +NVFP4_BLOCK = 16 +E2M1_PER_BYTE = 2 + +#: A routed expert's three projections: `w1` gate, `w3` up, `w2` down. +PROJECTIONS = ("w1", "w2", "w3") +#: The per-tensor NVFP4 scalars each of them carries beside its codes and block scales. +EXPERT_SCALARS = ("weight_scale_2", "input_scale") + +#: The vision tower's checkpoint prefixes. +VISION_PREFIXES = ( + "vision_encoder.", + "patch_merger.", + "pre_mm_projector_norm.", + "vision_language_adapter.", +) + + +#: `layers=ALL_LAYERS` is the common case and the default. A sentinel rather +#: than a lambda so the table reads as data, and rather than `True` so the +#: field has one type of answer: which layers. +ALL_LAYERS = object() + + +@dataclass(frozen=True) +class W: + """One weight role: what it is, where its bytes come from, what happens on the way. + + `shape` takes the dimension bundle and returns the declared shape. + + `src` says where the bytes come from: a checkpoint key template with `{p}` + for the layer prefix (`layers.N`), or a callable `(d, i) -> [(key, index), ...]` + where `index` is where in the parameter that tensor lands (`None`: all of it). + A `key` may be a tuple of per-tensor scalars, which `transform` is handed together. + + `layers` says which layers carry this weight: `ALL_LAYERS`, `None` for a + weight that is not per-layer, or a callable `d -> iterable[int]`. Two rows + may share a name over disjoint layer sets, with different shapes. + + `shard` takes the dimension bundle and returns a function of the checkpoint + entry: this rank's tensor-parallel slice, applied to every source of the row. + It runs before the copy to the device and before `transform`, which is what + `wkv_b` relies on: the regroup reads this rank's head count off the slice. + + `transform` runs on the (sliced) source tensor, on the destination's device. + + `layout` runs once every source of the parameter has landed, on each slot of its leading + (expert) axis, and the slot is overwritten with the result: a rearrangement that mixes rows + several sources fill, which no single source's `transform` can make. + """ + + name: str + shape: Callable[[Any], tuple[int, ...]] + src: str | Callable + dtype: torch.dtype | None = None + transform: Callable | None = None + layers: object | None = ALL_LAYERS + shard: Callable | None = None + layout: Callable | None = None + when: Callable | None = None + tag: str = "l" + + def layer_indices(self, d: SimpleNamespace) -> list[int] | None: + """The layers this weight exists on, or None if it is not per-layer.""" + if self.layers is None: + return None + if self.layers is ALL_LAYERS: + return list(range(d.num_layers)) + return list(self.layers(d)) + + def sources(self, d: SimpleNamespace, i: int | None, prefix: str) -> list[tuple[Any, Any]]: + """`[(checkpoint key | key tuple, index into the parameter | None), ...]`.""" + if callable(self.src): + return list(self.src(d, i)) + return [(self.src.format(p=prefix), None)] + + +def _keys(ckpt) -> tuple[str, ...]: + """The checkpoint keys of one source: itself, or the members of a key tuple.""" + return ckpt if isinstance(ckpt, tuple) else (ckpt,) + + +def _rows(start, width): + return lambda d: (lambda e: e[start(d) : start(d) + width(d)]) + + +def _cols(start, width): + return lambda d: (lambda e: e[:, start(d) : start(d) + width(d)]) + + +def _sf_cols(start, width): + """`_cols` on an FP8 block scale as stored, `[N/128, 1, K/128, 1]`: its K blocks are axis 2.""" + return lambda d: (lambda e: e[:, :, start(d) : start(d) + width(d)]) + + +def _sf_words(k: int) -> int: + """int32 words per row of the DeepGEMM scale operand of a K-wide weight.""" + return -(-k // (BLOCK * SF_PACK)) + + +def _assert_ue8m0(s: torch.Tensor) -> None: + """Every block scale is a positive normal power of two, so its UE8M0 byte (the fp32 + exponent field) holds it exactly. A meta tensor carries no values to check.""" + if s.is_meta: + return + bits = s.view(torch.int32) + exp = (bits >> 23) & 0xFF + exact = ((bits & 0x807FFFFF) == 0) & (exp > 0) & (exp < 255) + assert bool(exact.all()), ( + f"weight_scale: {int((~exact).sum())} of {s.numel()} block scales are not a positive " + "normal power of two (UE8M0); the scale layouts below would change their values" + ) + + +def _deepgemm_sf(s: torch.Tensor, core) -> torch.Tensor: + """fp32 `[N/128, K/128]` block scales -> the dense `[ceil(K/512), N]` int32 storage of + DeepGEMM's MN-major packed UE8M0 operand, whose `.t()` is that operand. N is a multiple of + 128 for every role here, so the operand's row stride is N and its storage has no MN padding. + A K of a block count that is not a multiple of four (the shared expert's `w2` at K 768 on 8 + ranks: 6 blocks) leaves the high bytes of each row's last word zero.""" + _assert_ue8m0(s) + n_blk, k_blk = s.shape + sf = transform_sf_into_required_layout( + s, mn=n_blk * BLOCK, k=k_blk * BLOCK, recipe=(1, BLOCK, BLOCK), is_sfa=False + ) + return sf.t() + + +def _block_sf(s: torch.Tensor, core) -> torch.Tensor: + """An FP8 linear's block scales as stored, fp32 `[N/128, 1, K/128, 1]`: squeezed to the + `[N/128, K/128]` grid, then `_deepgemm_sf`.""" + assert s.dim() == 4 and s.shape[1] == 1 and s.shape[3] == 1, ( + f"FP8 block scale stored as {tuple(s.shape)}, not [N/128, 1, K/128, 1]" + ) + return _deepgemm_sf(s.reshape(s.shape[0], s.shape[2]), core) + + +def _shared_scalar(t: tuple[torch.Tensor, ...], core) -> torch.Tensor: + """One per-tensor scalar the checkpoint stores several times (`w1`'s and `w3`'s, or every + expert's of the window): asserted the same value everywhere, then that value.""" + first = t[0] + if not first.is_meta: + values = torch.stack(t) + assert bool((values == first).all()), ( + f"{int((values != first).sum())} of {len(t)} copies of one shared scale differ from " + f"the first, {first.item()!r}" + ) + return first + + +def _regroup_kv_b(t: torch.Tensor, core) -> torch.Tensor: + """This rank's `[heads * (k_nope + v), kv_lora]` rows, per-head `[k_nope | v]` -> + `[heads * k_nope ; heads * v]`: the K-B operand, then the V-B one. k_nope is read off the + slice: 64 on RoPE layers, 128 on NoPE layers.""" + cfg = core.model_config.pretrained_config + heads = cfg.num_attention_heads // core.model_config.mapping.tp_size + v = cfg.v_head_dim + t3 = t.reshape(heads, t.shape[0] // heads, t.shape[1]) + nope = t3.shape[1] - v + return torch.cat( + [t3[:, :nope, :].reshape(heads * nope, -1), t3[:, nope:, :].reshape(heads * v, -1)], dim=0 + ).contiguous() + + +def _to_fp32(t: torch.Tensor, core) -> torch.Tensor: + return t.to(torch.float32) + + +# --- the NVFP4 MoE runner's expert layout, one expert at a time --- + +#: The runner's row shuffle works on blocks of this many rows. +SHUFFLE_ROWS = 32 +#: The block-scale swizzle's tile: 128 rows by 4 scale columns. +SWIZZLE_ROWS, SWIZZLE_COLS = 128, 4 + + +@functools.lru_cache(maxsize=None) +def _block32_perm(rows: int, device) -> torch.Tensor: + """Gather index of the 32-row block shuffle: within each block of 32 rows, source row + `4u + v` lands at row `8v + u`.""" + assert rows % SHUFFLE_ROWS == 0, rows + j = torch.arange(SHUFFLE_ROWS) + one = torch.empty(SHUFFLE_ROWS, dtype=torch.long) + one[(j % 4) * 8 + j // 4] = j + blocks = torch.arange(rows // SHUFFLE_ROWS).repeat_interleave(SHUFFLE_ROWS) * SHUFFLE_ROWS + return (one.repeat(rows // SHUFFLE_ROWS) + blocks).to(device) + + +@functools.lru_cache(maxsize=None) +def _fc1_perm(rows: int, device) -> torch.Tensor: + """Gather index of FC1's `[up ; gate]` rows: interleaved (row `2i` up row `i`, row `2i + 1` + gate row `i`), then block-shuffled.""" + interleave = torch.empty(rows, dtype=torch.long) + interleave[0::2] = torch.arange(rows // 2) + interleave[1::2] = torch.arange(rows // 2, rows) + return interleave.to(device)[_block32_perm(rows, device)] + + +def _swizzle(s: torch.Tensor) -> torch.Tensor: + """One expert's `[M, C]` e4m3 block scales in the 128x4-swizzled byte order, kept `[M, C]`: + byte `(m, c)` lands at flat offset `(m // 128) * 128 * C + (c // 4) * 512 + (m % 32) * 16 + + ((m % 128) // 32) * 4 + c % 4`. Both of this target's scale stacks tile it exactly (M a + multiple of 128, C of 4), so there is no padding.""" + m, c = s.shape + assert m % SWIZZLE_ROWS == 0 and c % SWIZZLE_COLS == 0, (m, c) + tiles = s.view(torch.uint8).reshape(m // SWIZZLE_ROWS, 4, 32, c // SWIZZLE_COLS, SWIZZLE_COLS) + return tiles.permute(0, 3, 2, 1, 4).reshape(m, c).view(s.dtype) + + +def _fc1_codes(w: torch.Tensor) -> torch.Tensor: + return torch.index_select(w, 0, _fc1_perm(w.shape[0], w.device)) + + +def _fc1_scales(s: torch.Tensor) -> torch.Tensor: + rows = torch.index_select(s.view(torch.uint8), 0, _fc1_perm(s.shape[0], s.device)) + return _swizzle(rows.view(s.dtype)) + + +def _fc2_codes(w: torch.Tensor) -> torch.Tensor: + return torch.index_select(w, 0, _block32_perm(w.shape[0], w.device)) + + +def _fc2_scales(s: torch.Tensor) -> torch.Tensor: + rows = torch.index_select(s.view(torch.uint8), 0, _block32_perm(s.shape[0], s.device)) + return _swizzle(rows.view(s.dtype)) + + +def _materialize(entry, shard) -> torch.Tensor: + """This rank's bytes of one checkpoint entry: a tensor, or a lazy safetensors slice, which + reads only what the index selects. `[...]`, not `[:]`: the per-tensor scalars are 0-dim.""" + return entry[...] if shard is None else shard(entry) + + +#: The checkpoint's shard index, beside its `consolidated-*.safetensors`. +SHARD_INDEX = "consolidated.safetensors.index.json" + + +def _checkpoint_shards(checkpoint_dir) -> tuple[Path, dict[str, str]]: + """`(directory, key -> shard file)` of a checkpoint directory: its index when it ships + one, else the header of each `consolidated*.safetensors`.""" + directory = Path(checkpoint_dir) + index = directory / SHARD_INDEX + if index.is_file(): + return directory, json.loads(index.read_text())["weight_map"] + weight_map: dict[str, str] = {} + for path in sorted(directory.glob("consolidated*.safetensors")): + with safe_open(str(path), framework="pt", device="cpu") as handle: + weight_map.update(dict.fromkeys(handle.keys(), path.name)) + if not weight_map: + raise FileNotFoundError(f"no consolidated*.safetensors in {directory}") + return directory, weight_map + + +def _shard_files(core, weights) -> tuple[Path, dict[str, str]] | None: + """`(directory, key -> shard file)` of the checkpoint a dict of sources was read from, or + None when the sources are not the shard files of one (the directory the config records + lacks the index, or the index does not name every key `weights` holds).""" + cfg = getattr(core.model_config, "pretrained_config", None) + root = getattr(cfg, "_name_or_path", None) + if not root: + return None + index = Path(root) / SHARD_INDEX + if not index.is_file(): + return None + weight_map = json.loads(index.read_text())["weight_map"] + if not set(weights.keys()) <= set(weight_map): + return None + return Path(root), weight_map + + +def _drop_page_cache(path: Path) -> None: + """Advise the kernel to drop a consumed shard file's cached pages. Advice only: a + filesystem that ignores it costs memory pressure, not correctness. Every rank reads every + shard file, a node's ranks through that node's page cache, so the advice can also drop pages + a sibling rank has yet to read, which that rank then reads again from disk; the kernel keeps + the pages a process has mapped.""" + try: + fd = os.open(path, os.O_RDONLY) + except OSError: + return + try: + os.posix_fadvise(fd, 0, 0, os.POSIX_FADV_DONTNEED) + except OSError: + pass + finally: + os.close(fd) + + +# --- which layers carry a weight --- + + +def _owners(d): + """Layers that own a latent KV. The others read `kv_source_layers[i]`'s and project none.""" + return [i for i in range(d.num_layers) if d.kv_source[i] == i] + + +def _rope_owners(d): + return [i for i in _owners(d) if not d.nope_layer[i]] + + +def _nope_owners(d): + return [i for i in _owners(d) if d.nope_layer[i]] + + +def _rope_layers(d): + return [i for i in range(d.num_layers) if not d.nope_layer[i]] + + +def _nope_layers(d): + return [i for i in range(d.num_layers) if d.nope_layer[i]] + + +def _indexer_layers(d): + """Layers that own an indexer. The 'shared' ones reuse the most recent one's selection.""" + return [i for i in range(d.num_layers) if d.full_indexer[i]] + + +def _dense_layers(d): + return range(d.dense_layers) + + +def _moe_layers(d): + return range(d.dense_layers, d.num_layers) + + +# --- where the bytes come from --- + + +def _stack(*parts, prefix="layers"): + """Several checkpoint tensors into one parameter: each `(key template, index(d))`, the + index being where in the parameter that tensor lands.""" + return lambda d, i: [(t.format(p=f"{prefix}.{i}"), index(d)) for t, index in parts] + + +def _expert_key(i: int, e: int, suffix: str) -> str: + return f"layers.{i}.experts.{e}.{suffix}" + + +def _expert_window(*parts): + """This rank's routed experts, local slot `s` <- global expert `expert_offset + s`: each + `(checkpoint suffix | suffix tuple, index(d) after the expert axis)`.""" + + def key(i, e, suffix): + if isinstance(suffix, tuple): + return tuple(_expert_key(i, e, s) for s in suffix) + return _expert_key(i, e, suffix) + + def build(d, i): + return [ + (key(i, d.expert_offset + slot, suffix), (slot, *index(d))) + for slot in range(d.local_experts) + for suffix, index in parts + ] + + return build + + +def _window_scalar(*suffixes): + """One per-tensor scalar every expert of the window stores: all of the window's copies, one + key tuple, into element 0.""" + return lambda d, i: [ + ( + tuple( + _expert_key(i, d.expert_offset + slot, s) + for slot in range(d.local_experts) + for s in suffixes + ), + (0,), + ) + ] + + +def _expert_rows(): + """This rank's slice of an expert's intermediate output rows: `w1` and `w3`, codes and block + scales alike, `[moe_tp_rank * moe_inter, + moe_inter)`.""" + return _rows(lambda d: d.moe_tp_rank * d.moe_inter, lambda d: d.moe_inter) + + +def _expert_cols(per_column: int): + """This rank's slice of an expert's intermediate input axis, `w2`'s, as stored with + `per_column` inputs to a column: 2 for the e2m1 codes, 16 for the block scales.""" + width = lambda d: d.moe_inter // per_column # noqa: E731 + return _cols(lambda d: d.moe_tp_rank * width(d), width) + + +def _expert_scalar_keys(d) -> list[str]: + """Every routed expert's per-tensor scalars on every MoE layer: what `load` reads, and + checks, before anything else.""" + return [ + _expert_key(i, e, f"{proj}.{s}") + for i in _moe_layers(d) + for e in range(d.num_experts) + for proj in PROJECTIONS + for s in EXPERT_SCALARS + ] + + +def _check_expert_scalars(scalars: dict[str, torch.Tensor], d) -> None: + """Per MoE layer, over all of its experts: `w1` and `w3` store the same `input_scale` and + `weight_scale_2` on every expert; `w1` and `w2` each store one `input_scale` for every expert; + every scalar is a positive finite fp32. Meta tensors carry no values to check.""" + if scalars and next(iter(scalars.values())).is_meta: + return + experts = range(d.num_experts) + for i in _moe_layers(d): + v = { + (proj, s): torch.stack([scalars[_expert_key(i, e, f"{proj}.{s}")] for e in experts]) + for proj in PROJECTIONS + for s in EXPERT_SCALARS + } + for (proj, s), t in v.items(): + bad = (~(torch.isfinite(t) & (t > 0))).nonzero().flatten() + assert bad.numel() == 0, ( + f"layers.{i}: {proj}.{s} of expert {int(bad[0])} is {t[bad[0]].item()!r}, " + "not a positive finite scale" + ) + for s in EXPERT_SCALARS: + bad = (v["w1", s] != v["w3", s]).nonzero().flatten() + assert bad.numel() == 0, ( + f"layers.{i}: w1 and w3 {s} differ on {bad.numel()} of {len(experts)} experts, " + f"first expert {int(bad[0])}: {v['w1', s][bad[0]].item()!r} vs " + f"{v['w3', s][bad[0]].item()!r}" + ) + for proj in ("w1", "w2"): + isc = v[proj, "input_scale"] + bad = (isc != isc[0]).nonzero().flatten() + assert bad.numel() == 0, ( + f"layers.{i}: {proj}.input_scale is not one value over the experts: expert " + f"{int(bad[0])} stores {isc[bad[0]].item()!r}, expert 0 {isc[0].item()!r}" + ) + + +def _read_scalars(keys: list[str], weights, shards) -> dict[str, torch.Tensor]: + """The given per-tensor scalars, off the shard files one file at a time when they are + found, otherwise off the given dict.""" + if shards is None: + return {key: _materialize(weights[key], None) for key in keys} + directory, weight_map = shards + by_file: dict[str, list] = collections.defaultdict(list) + for key in keys: + by_file[weight_map[key]].append(key) + out = {} + for name in sorted(by_file): + with safe_open(str(directory / name), framework="pt", device="cpu") as handle: + for key in by_file[name]: + out[key] = handle.get_slice(key)[...].clone() + return out + + +def _fp8_linear(name, key, n, k, layers, w_shard=None, s_shard=None): + """An FP8 128x128-block linear: e4m3 `[N, K]` and its DeepGEMM scale operand's storage.""" + return ( + W(name, shape=lambda d: (n(d), k(d)), dtype=F8, src=key, layers=layers, shard=w_shard), + W( + f"{name}_s", + shape=lambda d: (_sf_words(k(d)), n(d)), + dtype=I32, + src=key.replace(".weight", ".weight_scale"), + transform=_block_sf, + layers=layers, + shard=s_shard, + ), + ) + + +def _tp_mlp(module, inter, layers): + """A SwiGLU MLP split over its intermediate dimension: `[w1 ; w3]` by output rows, gate + first (the half order `flashinfer_silu_and_mul` reads), `w2` by input columns. The same + row slice applies to `w1` and `w3` before they are stacked, and the scales follow in + 128-blocks; in the scale storage the stacked axis is the second one.""" + blocks = lambda d: inter(d) // BLOCK # noqa: E731 + start = lambda d: d.tp_rank * inter(d) # noqa: E731 + b_start = lambda d: d.tp_rank * blocks(d) # noqa: E731 + p = "{p}." + module + return ( + W( + "mlp_gu", + shape=lambda d: (2 * inter(d), d.hidden), + dtype=F8, + src=_stack( + (f"{p}.w1.weight", lambda d: (slice(0, inter(d)),)), + (f"{p}.w3.weight", lambda d: (slice(inter(d), 2 * inter(d)),)), + ), + layers=layers, + shard=_rows(start, inter), + ), + W( + "mlp_gu_s", + shape=lambda d: (_sf_words(d.hidden), 2 * inter(d)), + dtype=I32, + src=_stack( + (f"{p}.w1.weight_scale", lambda d: (slice(None), slice(0, inter(d)))), + (f"{p}.w3.weight_scale", lambda d: (slice(None), slice(inter(d), 2 * inter(d)))), + ), + transform=_block_sf, + layers=layers, + shard=_rows(b_start, blocks), + ), + *_fp8_linear( + "mlp_dn", + f"{p}.w2.weight", + lambda d: d.hidden, + inter, + layers, + w_shard=_cols(start, inter), + s_shard=_sf_cols(b_start, blocks), + ), + ) + + +def _kv_b(nope, layers): + """`wkv_b`, this rank's heads, regrouped. Per-head rows are `nope(d) + v` wide.""" + width = lambda d: d.heads * (nope(d) + d.v_dim) # noqa: E731 + return W( + "wkv_b", + shape=lambda d: (width(d), d.kv_lora), + src="{p}.attention.wkv_b.weight", + transform=_regroup_kv_b, + layers=layers, + shard=_rows(lambda d: d.tp_rank * width(d), width), + ) + + +def _heads(width): + """This rank's slice of a per-head axis of the given per-head width.""" + return _rows(lambda d: d.tp_rank * d.heads * width(d), lambda d: d.heads * width(d)) + + +def _vision_blocks(d): + return list(range(d.v_layers)) + + +def _has_vision(d): + return d.has_vision + + +def _flatten_patch_conv(tensor, core): + return tensor.flatten(1) + + +def _draft_layers(d): + return range(d.draft_layers) + + +def _draft_source(suffix): + return lambda d, i: [(f"mtp_dspark.backbone.layers.{i}.{suffix}", None)] + + +class MistralLarge4TpWeights: + """mistral-large-4-nvfp4 / sm_107 / tp.""" + + WEIGHTS: tuple[W, ...] = ( + # --- MLA: every layer, KV owners only where marked --- + W("attn_norm", shape=lambda d: (d.hidden,), src="{p}.attention_norm.weight"), + # Per head `[nope 64 | rope 64]` on RoPE layers, 128 nope on NoPE layers: 128 either way. + *_fp8_linear( + "wq", + "{p}.attention.wq.weight", + lambda d: d.heads * d.qk_dim, + lambda d: d.hidden, + ALL_LAYERS, + w_shard=_heads(lambda d: d.qk_dim), + s_shard=_heads(lambda d: d.qk_dim // BLOCK), + ), + W( + "ssmax_s", + shape=lambda d: (d.heads,), + src="{p}.attention.ssmax_scaling.s", + shard=_heads(lambda d: 1), + ), + W( + "ssmax_b", + shape=lambda d: (d.heads,), + src="{p}.attention.ssmax_scaling.b", + shard=_heads(lambda d: 1), + ), + # `[c_kv 512 | k_pe 64]` on RoPE owners; NoPE owners project the 512 latent alone. + W( + "wkv_a", + shape=lambda d: (d.lat_dim, d.hidden), + src="{p}.attention.wkv_a_with_mqa.weight", + layers=_rope_owners, + ), + W( + "wkv_a", + shape=lambda d: (d.kv_lora, d.hidden), + src="{p}.attention.wkv_a_with_mqa.weight", + layers=_nope_owners, + ), + W( + "kv_a_norm", + shape=lambda d: (d.kv_lora,), + src="{p}.attention.kv_a_norm.weight", + layers=_owners, + ), + # Every layer absorbs through its own `wkv_b`, KV sharers included. + _kv_b(lambda d: d.nope, _rope_layers), + _kv_b(lambda d: d.qk_dim, _nope_layers), + *_fp8_linear( + "wo", + "{p}.attention.wo.weight", + lambda d: d.hidden, + lambda d: d.heads * d.v_dim, + ALL_LAYERS, + w_shard=_cols(lambda d: d.tp_rank * d.heads * d.v_dim, lambda d: d.heads * d.v_dim), + s_shard=_sf_cols( + lambda d: d.tp_rank * d.heads * d.v_dim // BLOCK, + lambda d: d.heads * d.v_dim // BLOCK, + ), + ), + W("ffn_norm", shape=lambda d: (d.hidden,), src="{p}.ffn_norm.weight"), + # --- the DSA indexer: replicated, on the 'full' layers --- + W( + "ix_q_norm_w", + shape=lambda d: (d.hidden,), + src="{p}.attention.indexer.q_input_norm.weight", + layers=_indexer_layers, + ), + W( + "ix_q_norm_b", + shape=lambda d: (d.hidden,), + src="{p}.attention.indexer.q_input_norm.bias", + layers=_indexer_layers, + ), + # Per head `[rope 64 | nope 64]`, the reverse of MLA's. + *_fp8_linear( + "ix_wq_b", + "{p}.attention.indexer.wq_b.weight", + lambda d: d.idx_heads * d.idx_dim, + lambda d: d.hidden, + _indexer_layers, + ), + # `[wk ; weights_proj]`: the key and the head weights, one GEMM. + W( + "ix_wk_wp", + shape=lambda d: (d.idx_dim + d.idx_heads, d.hidden), + src=_stack( + ("{p}.attention.indexer.wk.weight", lambda d: (slice(0, d.idx_dim),)), + ( + "{p}.attention.indexer.weights_proj.weight", + lambda d: (slice(d.idx_dim, d.idx_dim + d.idx_heads),), + ), + ), + layers=_indexer_layers, + ), + # fp32, as the fused key entry (LayerNorm in fp32) takes them. + W( + "ix_k_norm_w", + shape=lambda d: (d.idx_dim,), + dtype=F32, + src="{p}.attention.indexer.k_norm.weight", + transform=_to_fp32, + layers=_indexer_layers, + ), + W( + "ix_k_norm_b", + shape=lambda d: (d.idx_dim,), + dtype=F32, + src="{p}.attention.indexer.k_norm.bias", + transform=_to_fp32, + layers=_indexer_layers, + ), + W( + "ix_ssmax_s", + shape=lambda d: (d.idx_heads,), + src="{p}.attention.indexer.ssmax_scaling.s", + layers=_indexer_layers, + ), + W( + "ix_ssmax_b", + shape=lambda d: (d.idx_heads,), + src="{p}.attention.indexer.ssmax_scaling.b", + layers=_indexer_layers, + ), + # --- the TP-sliced SwiGLU MLP: the dense layer's own, then each MoE layer's shared + # expert, one key --- + *_tp_mlp("feed_forward", lambda d: d.dense_inter, _dense_layers), + *_tp_mlp("shared_experts", lambda d: d.shared_inter, _moe_layers), + # --- routing --- + W( + "router", + shape=lambda d: (d.num_experts, d.hidden), + src="{p}.gate.weight", + layers=_moe_layers, + ), + # --- the routed experts, NVFP4: this rank's width slice of every expert, `[w3 ; w1]` as + # stored, then each expert's slice in the MoE runner's layout --- + W( + "fc1_w", + shape=lambda d: (d.local_experts, 2 * d.moe_inter, d.hidden // E2M1_PER_BYTE), + dtype=U8, + src=_expert_window( + ("w3.weight", lambda d: (slice(0, d.moe_inter),)), + ("w1.weight", lambda d: (slice(d.moe_inter, 2 * d.moe_inter),)), + ), + layers=_moe_layers, + shard=_expert_rows(), + layout=_fc1_codes, + ), + W( + "fc1_s", + shape=lambda d: (d.local_experts, 2 * d.moe_inter, d.hidden // NVFP4_BLOCK), + dtype=F8, + src=_expert_window( + ("w3.weight_scale", lambda d: (slice(0, d.moe_inter),)), + ("w1.weight_scale", lambda d: (slice(d.moe_inter, 2 * d.moe_inter),)), + ), + layers=_moe_layers, + shard=_expert_rows(), + layout=_fc1_scales, + ), + W( + "fc2_w", + shape=lambda d: (d.local_experts, d.hidden, d.moe_inter // E2M1_PER_BYTE), + dtype=U8, + src=_expert_window(("w2.weight", lambda d: ())), + layers=_moe_layers, + shard=_expert_cols(E2M1_PER_BYTE), + layout=_fc2_codes, + ), + W( + "fc2_s", + shape=lambda d: (d.local_experts, d.hidden, d.moe_inter // NVFP4_BLOCK), + dtype=F8, + src=_expert_window(("w2.weight_scale", lambda d: ())), + layers=_moe_layers, + shard=_expert_cols(NVFP4_BLOCK), + layout=_fc2_scales, + ), + # The per-tensor scalars, whole: the layer's one FC1 and one FC2 activation scale, and + # each local expert's global weight scales (`w3`'s are `w1`'s). + W( + "isc1", + shape=lambda d: (1,), + dtype=F32, + src=_window_scalar("w1.input_scale", "w3.input_scale"), + transform=_shared_scalar, + layers=_moe_layers, + ), + W( + "isc2", + shape=lambda d: (1,), + dtype=F32, + src=_window_scalar("w2.input_scale"), + transform=_shared_scalar, + layers=_moe_layers, + ), + W( + "ws2_13", + shape=lambda d: (d.local_experts,), + dtype=F32, + src=_expert_window((("w1.weight_scale_2", "w3.weight_scale_2"), lambda d: ())), + transform=_shared_scalar, + layers=_moe_layers, + ), + W( + "ws2_2", + shape=lambda d: (d.local_experts,), + dtype=F32, + src=_expert_window(("w2.weight_scale_2", lambda d: ())), + layers=_moe_layers, + ), + # --- not per-layer --- + W("final_norm", shape=lambda d: (d.hidden,), src="norm.weight", layers=None), + W("embed", shape=lambda d: (d.vocab, d.hidden), src="tok_embeddings.weight", layers=None), + # The LeRoPE per-frequency log-scales the rope tables are built from, fp32 as the + # reference widens them. + W( + "lerope_alpha", + shape=lambda d: (d.rope_dim // 2,), + dtype=F32, + src="lerope.alpha", + transform=_to_fp32, + layers=None, + ), + # BF16 vision encoder and projector, replicated on every rank. + W( + "vis_patch", + shape=lambda d: (d.v_hidden, d.v_channels * d.v_patch**2), + src="vision_encoder.patch_conv.weight", + transform=_flatten_patch_conv, + layers=None, + when=_has_vision, + ), + W( + "vis_ln_pre", + shape=lambda d: (d.v_hidden,), + src="vision_encoder.ln_pre.weight", + layers=None, + when=_has_vision, + ), + W( + "attn_norm", + shape=lambda d: (d.v_hidden,), + src="{p}.attention_norm.weight", + layers=_vision_blocks, + tag="v", + ), + W( + "qkv", + shape=lambda d: (3 * d.v_hidden, d.v_hidden), + src=_stack( + ("{p}.attention.wq.weight", lambda d: (slice(0, d.v_hidden),)), + ("{p}.attention.wk.weight", lambda d: (slice(d.v_hidden, 2 * d.v_hidden),)), + ("{p}.attention.wv.weight", lambda d: (slice(2 * d.v_hidden, 3 * d.v_hidden),)), + prefix="vision_encoder.transformer.layers", + ), + layers=_vision_blocks, + tag="v", + ), + W( + "wo", + shape=lambda d: (d.v_hidden, d.v_hidden), + src="{p}.attention.wo.weight", + layers=_vision_blocks, + tag="v", + ), + W( + "ffn_norm", + shape=lambda d: (d.v_hidden,), + src="{p}.ffn_norm.weight", + layers=_vision_blocks, + tag="v", + ), + W( + "w13", + shape=lambda d: (2 * d.v_inter, d.v_hidden), + src=_stack( + ("{p}.feed_forward.w1.weight", lambda d: (slice(0, d.v_inter),)), + ("{p}.feed_forward.w3.weight", lambda d: (slice(d.v_inter, 2 * d.v_inter),)), + prefix="vision_encoder.transformer.layers", + ), + layers=_vision_blocks, + tag="v", + ), + W( + "w2", + shape=lambda d: (d.v_hidden, d.v_inter), + src="{p}.feed_forward.w2.weight", + layers=_vision_blocks, + tag="v", + ), + W( + "vis_proj_norm", + shape=lambda d: (d.v_hidden,), + src="pre_mm_projector_norm.weight", + layers=None, + when=_has_vision, + ), + W( + "vis_merge", + shape=lambda d: (d.v_hidden, d.v_hidden * d.v_merge**2), + src="patch_merger.merging_layer.weight", + layers=None, + when=_has_vision, + ), + W( + "vis_w_in", + shape=lambda d: (d.hidden, d.v_hidden), + src="vision_language_adapter.w_in.weight", + layers=None, + when=_has_vision, + ), + W( + "vis_w_out", + shape=lambda d: (d.hidden, d.hidden), + src="vision_language_adapter.w_out.weight", + layers=None, + when=_has_vision, + ), + ) + + # BF16 draft weights use the same attention and dense-MLP partition as + # this target. The RNN and target-feature projection are replicated. + DRAFT_WEIGHTS: tuple[W, ...] = ( + W( + "draft_attn_norm", + lambda d: (d.hidden,), + _draft_source("attention_norm.weight"), + layers=_draft_layers, + ), + W( + "draft_wq", + lambda d: (d.heads * d.qk_dim, d.hidden), + _draft_source("attention.wq.weight"), + layers=_draft_layers, + shard=_rows( + lambda d: d.draft_tp_rank * d.heads * d.qk_dim, lambda d: d.heads * d.qk_dim + ), + ), + W( + "draft_wkv_a", + lambda d: (d.lat_dim, d.hidden), + _draft_source("attention.wkv_a_with_mqa.weight"), + layers=_draft_layers, + ), + W( + "draft_kv_a_norm", + lambda d: (d.kv_lora,), + _draft_source("attention.kv_a_norm.weight"), + layers=_draft_layers, + ), + W( + "draft_wkv_b", + lambda d: (d.heads * (d.nope + d.v_dim), d.kv_lora), + _draft_source("attention.wkv_b.weight"), + layers=_draft_layers, + shard=_rows( + lambda d: d.draft_tp_rank * d.heads * (d.nope + d.v_dim), + lambda d: d.heads * (d.nope + d.v_dim), + ), + ), + W( + "draft_wo", + lambda d: (d.hidden, d.heads * d.v_dim), + _draft_source("attention.wo.weight"), + layers=_draft_layers, + shard=_cols(lambda d: d.draft_tp_rank * d.heads * d.v_dim, lambda d: d.heads * d.v_dim), + ), + W( + "draft_ffn_norm", + lambda d: (d.hidden,), + _draft_source("ffn_norm.weight"), + layers=_draft_layers, + ), + W( + "draft_w1", + lambda d: (d.dense_inter, d.hidden), + _draft_source("feed_forward.w1.weight"), + layers=_draft_layers, + shard=_rows(lambda d: d.draft_tp_rank * d.dense_inter, lambda d: d.dense_inter), + ), + W( + "draft_w3", + lambda d: (d.dense_inter, d.hidden), + _draft_source("feed_forward.w3.weight"), + layers=_draft_layers, + shard=_rows(lambda d: d.draft_tp_rank * d.dense_inter, lambda d: d.dense_inter), + ), + W( + "draft_w2", + lambda d: (d.hidden, d.dense_inter), + _draft_source("feed_forward.w2.weight"), + layers=_draft_layers, + shard=_cols(lambda d: d.draft_tp_rank * d.dense_inter, lambda d: d.dense_inter), + ), + W( + "draft_final_norm", + lambda d: (d.hidden,), + "mtp_dspark.backbone.final_norm.weight", + layers=None, + ), + W( + "draft_target_norm", + lambda d: (d.hidden,), + "mtp_dspark.backbone.target_norm.weight", + layers=None, + ), + W( + "draft_target_projection", + lambda d: (d.hidden, d.hidden * d.draft_capture_layers), + "mtp_dspark.backbone.target_projection.weight", + layers=None, + ), + W( + "draft_mask", + lambda d: (d.draft_depth - 1, d.hidden), + "mtp_dspark.backbone.mask_embedding", + layers=None, + ), + W( + "draft_markov_embedding", + lambda d: (d.vocab, d.draft_rank), + "mtp_dspark.autoregressive_head.embedding.weight", + layers=None, + ), + W( + "draft_markov_output", + lambda d: (d.vocab, d.draft_rank), + "mtp_dspark.autoregressive_head.output.weight", + layers=None, + ), + W( + "draft_joint", + lambda d: (3 * d.draft_rank, d.hidden + 2 * d.draft_rank), + "mtp_dspark.autoregressive_head.joint_proj.weight", + layers=None, + ), + W( + "draft_joint_bias", + lambda d: (3 * d.draft_rank,), + "mtp_dspark.autoregressive_head.joint_proj.bias", + layers=None, + ), + ) + + def _entries(self, d): + return self.WEIGHTS + (self.DRAFT_WEIGHTS if d.draft_layers else ()) + + def declare(self, d: SimpleNamespace) -> nn.ParameterDict: + """Allocate every weight the table declares. + + Meta-init intercepts `torch.empty` here -- real CUDA storage arrives + when the engine materializes the registry. + """ + w = nn.ParameterDict() + for entry in self._entries(d): + dtype = entry.dtype or d.dtype + for key, _, _ in self._targets(entry, d): + assert key not in w, f"two rows declare {key}" + w[key] = nn.Parameter( + torch.empty(*entry.shape(d), dtype=dtype), requires_grad=False + ) + return w + + def _targets(self, entry: W, d: SimpleNamespace) -> list[tuple[str, int | None, str]]: + """`[(param key, layer index | None, checkpoint layer prefix), ...]`. + + One place decides which parameters an entry declares, so `declare` and + `manifest` cannot disagree about whether a weight exists. + """ + if entry.when is not None and not entry.when(d): + return [] + layers = entry.layer_indices(d) + if layers is None: + return [(entry.name, None, "")] + prefix = "vision_encoder.transformer.layers" if entry.tag == "v" else "layers" + return [(f"{entry.tag}{i}_{entry.name}", i, f"{prefix}.{i}") for i in layers] + + def manifest(self, core) -> dict: + """param key -> [(checkpoint key | key tuple, index into the param | None, transform, + shard)].""" + d = self.dims(core) + rows: dict = {} + for entry in self._entries(d): + shard = entry.shard(d) if entry.shard else None + for key, i, prefix in self._targets(entry, d): + assert key not in rows, f"two rows declare {key}" + rows[key] = [ + (ckpt, index, entry.transform, shard) + for ckpt, index in entry.sources(d, i, prefix) + ] + return rows + + def expected_unconsumed(self, core, weights) -> dict[str, str]: + """Declared unused keys: off-rank experts and disabled draft weights. Off-rank expert + scalars are still read to validate the layer's shared scales.""" + d = self.dims(core) + lo, hi = d.expert_offset, d.expert_offset + d.local_experts + keys = { + k: "DSpark draft disabled" + for k in weights.keys() + if not d.draft_layers and k.startswith("mtp_dspark.") + } + window = f"routed expert outside this rank's window [{lo}, {hi})" + never_read = f"{window}: never read" + checked = f"{window}: read only to check the layer's shared scales" + for i in _moe_layers(d): + for e in range(d.num_experts): + if lo <= e < hi: + continue + for proj in PROJECTIONS: + for s in ("weight", "weight_scale"): + key = _expert_key(i, e, f"{proj}.{s}") + if key in weights: + keys[key] = never_read + for s in EXPERT_SCALARS: + keys[_expert_key(i, e, f"{proj}.{s}")] = checked + return keys + + def load_checkpoint(self, model, checkpoint_dir) -> None: + """Fill every declared weight from a checkpoint directory, which is what the engine + loads: `load` over the directory's keys, every source read from its shard file.""" + directory, weight_map = _checkpoint_shards(checkpoint_dir) + self.load(model, weight_map, shards=(directory, weight_map)) + + def load(self, model, weights, shards=None) -> None: + """Fill every declared weight from the checkpoint: the shard column, multi-source rows, + the vocab-parallel `lm_head`, and coverage asserted in both directions. + + First every routed expert's per-tensor scalars are read and checked, so a checkpoint + that breaks what the forward relies on fails before any weight moves. When the + checkpoint's own shard files can be found, every source is read from them one file at a + time, this rank's slices only -- of the routed experts its width slice of each, never a + whole expert -- and each file's cached pages are dropped once its sources are copied. + Mapping every shard file on every rank and reading through the mappings instead + would fault a node's four ranks' slices into that node's host memory: on one + node of four, between them every expert, 539 GiB, the whole expert stack; on each of + two nodes, half of every expert's `w1` and `w3` and every page of every `w2`, which a + strided column read touches whole, 359 GiB. Last, each row's `layout` puts each + expert's slice into the MoE runner's layout, so a second `load` starts again from the + stored bytes. + """ + core = model.model + manifest = self.manifest(core) + consumed: set = set() + + def fill(param, ckpt, entry, index, transform, shard) -> None: + dst = param.data if index is None else param.data[index] + if isinstance(ckpt, tuple): + src = transform( + tuple(_materialize(e, shard).to(dst.device, non_blocking=True) for e in entry), + core, + ) + else: + src = _materialize(entry, shard) + if transform is not None: + # A column shard is a strided view: make it dense before a transform reads it. + src = transform(src.to(dst.device, non_blocking=True).contiguous(), core) + assert tuple(dst.shape) == tuple(src.shape), (ckpt, tuple(dst.shape), tuple(src.shape)) + assert src.dtype == dst.dtype, (ckpt, src.dtype, dst.dtype) + dst.copy_(src, non_blocking=True) + consumed.update(_keys(ckpt)) + + assert set(manifest.keys()) == set(core.w.keys()), ( + "manifest/parameter drift", + sorted(set(manifest.keys()) ^ set(core.w.keys()))[:12], + ) + # The shell builds lm_head vocab-parallel under TP: this rank's block of rows. + d = self.dims(core) + rows = d.vocab // d.tp_size + plan = [ + (core.w[param_key], ckpt, index, transform, shard) + for param_key, sources in manifest.items() + for ckpt, index, transform, shard in sources + ] + plan.append( + ( + model.lm_head.weight, + "output.weight", + None, + None, + _rows(lambda _: d.tp_rank * rows, lambda _: rows)(d), + ) + ) + scalar_keys = _expert_scalar_keys(d) + for key in (*(k for row in plan for k in _keys(row[1])), *scalar_keys): + assert key in weights, f"checkpoint key missing: {key}" + if shards is None: + shards = _shard_files(core, weights) + scalars = _read_scalars(scalar_keys, weights, shards) + _check_expert_scalars(scalars, d) + + streamed = [] + for param, ckpt, index, transform, shard in plan: + if all(k in scalars for k in _keys(ckpt)): + entry = tuple(scalars[k] for k in _keys(ckpt)) + entry = entry if isinstance(ckpt, tuple) else entry[0] + fill(param, ckpt, entry, index, transform, shard) + else: + assert isinstance(ckpt, str), f"a key tuple of other than expert scalars: {ckpt}" + streamed.append((param, ckpt, index, transform, shard)) + if shards is None: + for param, ckpt, index, transform, shard in streamed: + fill(param, ckpt, weights[ckpt], index, transform, shard) + else: + directory, weight_map = shards + by_file: dict[str, list] = collections.defaultdict(list) + for row in streamed: + by_file[weight_map[row[1]]].append(row) + on_device = model.lm_head.weight.device.type == "cuda" + for name in sorted(by_file): + with safe_open(str(directory / name), framework="pt", device="cpu") as handle: + for param, ckpt, index, transform, shard in by_file[name]: + fill(param, ckpt, handle.get_slice(ckpt), index, transform, shard) + if on_device: + torch.cuda.synchronize() + _drop_page_cache(directory / name) + # Every source has landed: rearrange each expert in place, one at a time. + for entry in self._entries(d): + if entry.layout is None: + continue + for key, _, _ in self._targets(entry, d): + param = core.w[key].data + for slot in range(param.shape[0]): + param[slot].copy_(entry.layout(param[slot])) + shell = {n for n, _ in model.named_parameters() if not n.startswith("model.w.")} + assert shell == {"lm_head.weight"}, f"unfed shell parameters: {sorted(shell)}" + + if model.lm_head.weight.device.type == "cuda": + torch.cuda.synchronize() + leftover = set(weights.keys()) - consumed + expected = self.expected_unconsumed(core, weights) + assert leftover == set(expected), ( + "checkpoint coverage: unexpected " + f"{sorted(leftover - set(expected))[:8]}, missed {sorted(set(expected) - leftover)[:8]}" + ) + not_loaded = collections.Counter(expected.values()) + logger.info( + f"modeling_v2 weights: {len(consumed)} checkpoint tensors loaded " + f"({'streamed per shard file' if shards else 'from the given dict'}), " + f"{len(scalars)} expert scalars checked; not loaded: " + + "; ".join(f"{n} x {reason}" for reason, n in sorted(not_loaded.items())) + ) + + def dims(self, core) -> SimpleNamespace: + cfg = core.model_config.pretrained_config + vis = getattr(cfg, "vision_config", None) + mapping = core.model_config.mapping + spec = getattr(core.model_config, "spec_config", None) + draft = getattr(spec, "_draft_model_config", None) or {} + tp = mapping.tp_size + local_experts = cfg.n_routed_experts // mapping.moe_ep_size + return SimpleNamespace( + draft_layers=int(draft.get("n_layers", 0)), + draft_rank=int((draft.get("dspark") or {}).get("markov_rank", 0)), + draft_depth=int(draft.get("depth", 0)), + draft_capture_layers=len(spec.target_layer_ids) if draft else 0, + draft_tp_rank=mapping.tp_rank, + num_layers=cfg.num_hidden_layers, + hidden=cfg.hidden_size, + heads=cfg.num_attention_heads // tp, + nope=cfg.qk_nope_head_dim, + rope_dim=cfg.qk_rope_head_dim, + qk_dim=cfg.qk_nope_head_dim + cfg.qk_rope_head_dim, + v_dim=cfg.v_head_dim, + kv_lora=cfg.kv_lora_rank, + lat_dim=cfg.kv_lora_rank + cfg.qk_rope_head_dim, + nope_layer=tuple(bool(f) for f in cfg.nope_layer_flags), + kv_source=tuple(int(s) for s in cfg.kv_source_layers), + full_indexer=tuple(t == "full" for t in cfg.index_sharing), + idx_heads=cfg.index_n_heads, + idx_dim=cfg.index_head_dim, + dense_layers=cfg.first_k_dense_replace, + num_experts=cfg.n_routed_experts, + moe_inter=cfg.moe_intermediate_size // mapping.moe_tp_size, + moe_tp_rank=mapping.moe_tp_rank, + local_experts=local_experts, + expert_offset=local_experts * mapping.moe_ep_rank, + shared_inter=cfg.moe_intermediate_size * cfg.n_shared_experts // tp, + dense_inter=cfg.intermediate_size // tp, + vocab=cfg.vocab_size, + tp_rank=mapping.tp_rank, + tp_size=tp, + dtype=core.model_config.torch_dtype, + # The vision tower (replicated; absent without a vision encoder). + has_vision=vis is not None, + v_layers=vis.num_hidden_layers if vis is not None else 0, + v_hidden=vis.hidden_size if vis is not None else 0, + v_heads=vis.num_attention_heads if vis is not None else 0, + v_head_dim=vis.hidden_size // vis.num_attention_heads if vis is not None else 0, + v_inter=vis.intermediate_size if vis is not None else 0, + v_channels=vis.num_channels if vis is not None else 0, + v_patch=vis.patch_size if vis is not None else 0, + v_merge=vis.spatial_merge_size if vis is not None else 0, + v_theta=float(vis.rope_theta) if vis is not None else 0.0, + image_token_id=vis.image_token_id if vis is not None else None, + ) + + +MODEL_WEIGHTS = MistralLarge4TpWeights() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/__init__.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/__init__.py new file mode 100644 index 000000000000..52a7a9daf028 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/__init__.py @@ -0,0 +1,2 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/activation/__init__.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/activation/__init__.py new file mode 100644 index 000000000000..52a7a9daf028 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/activation/__init__.py @@ -0,0 +1,2 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/activation/gelu_erf.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/activation/gelu_erf.py new file mode 100644 index 000000000000..efb31424cd0c --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/activation/gelu_erf.py @@ -0,0 +1,213 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Non-gated exact (erf) GELU -- what `torch.nn.GELU()` is. + +Two call sites, both in a vision tower: the qwen4_exp patch merger's +activation (`merger.linear_fc1` 4608 -> 4608, this GELU, `linear_fc2`), which +no config key names, and the Mistral Large 4 vision-language adapter's (`w_in` +2048 -> 6144, this GELU, `w_out` 6144 -> 6144, no biases). + +Tier 4, torch composed: one `torch.nn.functional.gelu(x)` call, +`approximate="none"`, which launches ATen's `GeluCUDAKernelImpl` elementwise +kernel once. The cheaper rungs were rejected: + +- tier 1: `torch.ops.trtllm.moe_gelu` computes this function bitwise but is + not a substitute. Its `tile_idx_to_mn_limit` / `num_non_exiting_tiles` + operands must be int32 *device* tensors (on the host it raises nothing and + returns garbage, nan, or an illegal memory access), so wrapping it means + launching two fills before the call. It also rejects fp32 and non-2-D + inputs. `cute_dsl_nvfp4_dense_gemm_gelu_blackwell` fuses a GELU into an + NVFP4 GEMM epilogue, which `_torch/modules/mlp.py` calls GELU(tanh); the + adapter's `w_in` is a BF16 weight. `flashinfer_gelu_tanh_and_mul` is gated, + and is the tanh form. +- tier 2: no module callable under `_torch/modules` computes a non-gated erf + GELU. `gelu_tanh` in `_torch/utils.py` is `F.gelu(approximate="tanh")`, the + other GELU. +- tier 3: `flashinfer.activation` offers only gated forms (`gelu_and_mul`, + `gelu_tanh_and_mul`). +""" + +from __future__ import annotations + +import math + +import torch +import torch.nn.functional as F + +from tensorrt_llm._torch.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: qwen4_exp: `merger.linear_fc1` is `[4608, 4608]`, spatial merge 2 over a +#: 1152-wide tower. +MERGER_WIDTH = 4608 + +#: qwen4_exp: rows the merger is asked for, `sum(t*h*w) / 4`. 64 is a 224x224 +#: image (grid [1, 16, 16]) and 140 a 336x448 one ([1, 20, 28]); neither is a +#: multiple of any MoE tile size. +MERGER_ROW_COUNTS = (1, 7, 64, 65, 140, 1024, 4096) + +#: Mistral Large 4: the vision-language adapter's `w_in` is `[6144, 2048]`, so the +#: GELU runs at width 6144 on `w_in`'s bf16 output. +ADAPTER_WIDTH = 6144 + +#: Mistral Large 4: rows the adapter is asked for, one per merged image token. +#: 256 is a 448x448 image (32x32 patches of 14, merged 2x2). 3025 is one image +#: at the 1540-pixel maximum (110x110 patches, merged to 55x55). 6050 is two +#: of those. 30250 is ten maximum-size images, leaving room for the native +#: row delimiters, text and output within the 32768-token envelope. +ADAPTER_ROW_COUNTS = (1, 48, 60, 108, 256, 3025, 6050, 8192, 30250) + +_SQRT_2 = math.sqrt(2.0) + + +class GeluErf(OpWrapper): + """`0.5 * x * (1 + erf(x / sqrt(2)))`, elementwise, any shape, new tensor + out in `x`'s dtype. + + Fusion boundary: the activation only. The projection before it (qwen4_exp + `merger.linear_fc1`, Mistral Large 4 `vision_language_adapter.w_in`) and the + one after it (`linear_fc2`, `w_out`) are the caller's; there is no gate + half and no multiply. This is the erf GELU and not the tanh one: qwen4_exp's + block MLPs use the tanh one, in the same tower. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # spec: shape, dtype; `dist` "uniform" over [lo, hi] (default) or "merger" + # (a tower hidden state scaled by the merger norm's measured gamma, grouped + # 4-to-1, through an fc1 at the checkpoint's measured weight/bias scale); + # `layout` "contiguous" (default), "column_slice" or "transposed". + CELLS: tuple[Cell, ...] = ( + tuple( + Cell( + why=f"merger activand, {rows} merged rows x 4608 bf16", + spec=dict(shape=(rows, MERGER_WIDTH), dtype=torch.bfloat16, lo=-16.0, hi=16.0), + ) + for rows in MERGER_ROW_COUNTS + ) + + ( + Cell( + why="activand built the way the merger builds it", + spec=dict(shape=(512, MERGER_WIDTH), dtype=torch.bfloat16, dist="merger"), + ), + Cell( + why="fp16 over [-16, 16]", + spec=dict(shape=(256, MERGER_WIDTH), dtype=torch.float16, lo=-16.0, hi=16.0), + ), + Cell( + why="fp32 over [-16, 16]", + spec=dict(shape=(256, MERGER_WIDTH), dtype=torch.float32, lo=-16.0, hi=16.0), + ), + Cell( + why="bfloat16 over [-3, 3], the erf/tanh divergence peak", + spec=dict(shape=(256, MERGER_WIDTH), dtype=torch.bfloat16, lo=-3.0, hi=3.0), + ), + Cell( + why="float16 over [-3, 3], the erf/tanh divergence peak", + spec=dict(shape=(256, MERGER_WIDTH), dtype=torch.float16, lo=-3.0, hi=3.0), + ), + Cell( + why="float32 over [-3, 3], the erf/tanh divergence peak", + spec=dict(shape=(256, MERGER_WIDTH), dtype=torch.float32, lo=-3.0, hi=3.0), + ), + Cell( + why="column slice of a wider buffer: row stride free", + spec=dict( + shape=(64, MERGER_WIDTH), + dtype=torch.bfloat16, + lo=-16.0, + hi=16.0, + layout="column_slice", + ), + ), + Cell( + why="transposed buffer: output keeps the input's strides", + spec=dict( + shape=(64, MERGER_WIDTH), + dtype=torch.bfloat16, + lo=-16.0, + hi=16.0, + layout="transposed", + ), + ), + Cell( + why="1-D input", + spec=dict(shape=(MERGER_WIDTH,), dtype=torch.bfloat16, lo=-16.0, hi=16.0), + ), + Cell( + why="3-D input", + spec=dict(shape=(2, 37, MERGER_WIDTH), dtype=torch.bfloat16, lo=-16.0, hi=16.0), + ), + Cell( + why="zero-row batch", + spec=dict(shape=(0, MERGER_WIDTH), dtype=torch.bfloat16, lo=-16.0, hi=16.0), + ), + ) + + tuple( + Cell( + why=f"Mistral Large 4 adapter, {rows} image tokens x 6144 bf16", + spec=dict(shape=(rows, ADAPTER_WIDTH), dtype=torch.bfloat16, lo=-16.0, hi=16.0), + ) + for rows in ADAPTER_ROW_COUNTS + ) + ) + + note = """ + Metadata consumed: none. Out-of-place; `x` is not mutated. CUDA-graph + capturable; the output is a fresh allocation. + + Call sites: the qwen4_exp vision merger at width 4608, and the Mistral Large 4 + vision-language adapter at width 6144 (`w_in`'s contiguous bf16 output, + eager, one row per merged image token). + + The output inherits `x`'s memory format: a transposed `x` returns a + non-contiguous result (bitwise equal to the call on a contiguous copy). + Rows are independent: two images' rows concatenated give, bitwise, the two + calls' outputs concatenated. + + Numerics. The kernel evaluates `1 + erf` in fp32 and rounds once to the + output dtype. + - bf16: every finite bf16 input with |x| <= 4 (33026 values, driven + exhaustively) returns the correctly rounded value, bitwise. Below -4 the + fp32 `1 + erf` cancels, and 198 of the bf16 inputs there are not + correctly rounded: the error stays under |x| * 2^-21 in absolute terms + but is many ulp of the tiny result. From x = -5.5625 down (exact value + -7.4e-8 there) the output is -0.0. + - fp16: 59 of the 34818 fp16 inputs with |x| <= 4 are not correctly + rounded; near the cancellation the output is up to 1.94 ulp off. + - fp32: up to 143 ulp relative error near x = -3, and a zero output deep + in the negative tail, from the same cancellation. + Every finite bf16 and fp16 input lies within one ulp + |x| * 2^-21 of the + exact value, at worst half of that allowance. + + Identity. Max |erf GELU - tanh GELU| is 4.73e-4 at x = -2.699. In fp32 the + tanh form lands about 1400x the correct path's deviation away, 31x outside + torch's default gate. In bf16 it lands 15x outside the test's per-element + ulp gate over [-3, 3] and 5.7x outside torch's default bf16 band, so a bf16 + match does tell the two apart on inputs that reach about [-3.5, -2]. The max-abs deviation alone cannot: at + the output's scale one bf16 ulp exceeds the gap. An erf scale of 1/1.414 + instead of 1/sqrt(2) stays inside one bf16 ulp. Only the bf16 + correctly-rounded count refuses it, at 3.87% of elements over [-3, 3]. + + Values: `-inf -> nan` (the closed form does the same); `+inf -> +inf`; + large negatives give `-0.0`; `finfo.max` comes back unchanged. + + Not certified: float8 operands; rows above 8192 at width 6144. + + Hard limits + numel no index-typed argument; the largest certified call is + 8192 x 6144 = 50331648 elements boundary 2^31 not probed + """ + + def __call__(self, x: torch.Tensor) -> torch.Tensor: + return F.gelu(x) + + def reference(self, x: torch.Tensor) -> torch.Tensor: + """The closed form in fp64, rounded once to `x.dtype`. Written with + `erfc(-x / sqrt(2))`: `1 + erf(x / sqrt(2))` cancels in fp64 too, and + returns 0 instead of the tiny negative value below about x = -8.3. An + fp64 `x` comes back unrounded, the exact value a ulp gate needs.""" + x64 = x.double() + return (0.5 * x64 * torch.erfc(-x64 / _SQRT_2)).to(x.dtype) + + +gelu_erf = GeluErf() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/activation/tanh_softcap.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/activation/tanh_softcap.py new file mode 100644 index 000000000000..b7a3245586ad --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/activation/tanh_softcap.py @@ -0,0 +1,177 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Final-logit tanh soft-cap, ``cap * tanh(x / cap)``: the Mistral Large 4 checkpoints' +``softmax_tanh_gating`` (30.0) applied to the gathered fp32 logits. + +Sourcing tier 4, composed from native torch: no engine kernel computes it. +Rejected at the cheaper rungs, each searched on this build: tier 1, +``torch.ops.trtllm`` registers no soft-cap op (its only tanh op, +``flashinfer_gelu_tanh_and_mul``, is a gated GELU); tier 2, no callable under +``tensorrt_llm/_torch/modules`` (``LogitsProcessor`` in +``modules/logits_processor.py`` has no cap, and ``modeling_gemma4.py`` applies +``torch.tanh(logits / cap) * cap`` inline rather than through a module); tier 3, +the ``flashinfer.logits_processor`` operators (``Temperature``, ``Softmax``, +``TopK``, ``TopP``, ``MinP``, ``Sample``) carry no soft-cap. Correct and slow by +design: three elementwise launches, which a tuning pass may fold into the +logits epilogue. +""" + +from __future__ import annotations + +import math + +import torch + +from tensorrt_llm._torch.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: The checkpoint's vocabulary; the call site holds every rank's gathered row. +VOCAB_SIZE = 204800 +#: ``softmax_tanh_gating`` in the checkpoint's config. +SOFTCAP = 30.0 +#: The Mistral Large 4 NVFP4 target's token budget (``max_num_tokens``; the FP8 +#: target's is 9216): the most rows a step can hand over when a context request +#: asks for its context logits. +MAX_NUM_TOKENS = 32768 +#: The most rows whose 204800-wide logits stay below 2^31 elements. +INT32_ROWS = (2**31 - 1) // VOCAB_SIZE + +_CKPT = dict(vocab=VOCAB_SIZE, dtype=torch.float32, cap=SOFTCAP) + + +class TanhSoftcap(OpWrapper): + """`cap * tanh(x / cap)`, elementwise, any shape, a new tensor in `x`'s dtype. + + The three torch ops the reference implementation's logits processor runs, + in its order, each rounding to `x.dtype`: `x / cap` (a Python-float divisor + is applied on CUDA as `x * fl(1 / cap)`, bitwise), `torch.tanh`, `* cap`. + + Fusion boundary: the cap only. The lm_head projection, the vocabulary + all-gather and the fp32 upcast before it are the caller's (the shell's + logits processor); temperature, softmax and sampling after it are the + sampler's. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # spec: rows, vocab, dtype, cap; values uniform over [lo, hi]. `graph` True + # captures the call in a CUDA graph and checks a replay on fresh logits + # written into the captured input, bitwise against an eager call on them; + # False runs it eagerly. `windows` are (start, stop) row ranges whose output + # must be bitwise the op run on those rows alone. A `large` cell reaches a + # bound past the per-cell budget and carries pytest.mark.large. + # + # Rows per call: one per sequence in the step (1..16, max_batch_size 16), + # or, when a context request asks for its context logits, every token of + # the step (up to max_num_tokens 32768). Decode steps are captured: the + # gates capture 1/2/4/8/16 padded, the LLM API default 1..16 unpadded. + CELLS: tuple[Cell, ...] = tuple( + Cell( + why=f"decode graph batch {rows}: captured, replayed; |x| <= 400 crosses saturation", + spec=dict(rows=rows, lo=-400.0, hi=400.0, graph=True, **_CKPT), + ) + for rows in range(1, 17) + ) + ( + Cell( + why="context step, 3 requests: eager, one last-token row per request", + spec=dict(rows=3, lo=-400.0, hi=400.0, graph=False, **_CKPT), + ), + Cell( + why="16 rows over [-60, 60]: the bend, |x / cap| <= 2, nothing saturates", + spec=dict(rows=16, lo=-60.0, hi=60.0, graph=False, **_CKPT), + ), + Cell( + why="16 rows over [-1e-3, 1e-3]: relative precision where the cap is identity", + spec=dict(rows=16, lo=-1e-3, hi=1e-3, graph=False, **_CKPT), + ), + Cell( + why=f"context logits, {INT32_ROWS} rows: the most below 2^31 elements, " + "leading and last rows bitwise; 26 GiB peak", + spec=dict( + rows=INT32_ROWS, + lo=-400.0, + hi=400.0, + graph=False, + windows=((0, 1), (0, 16), (0, 4096), (INT32_ROWS - 16, INT32_ROWS)), + **_CKPT, + ), + ), + Cell( + why=f"context logits, {MAX_NUM_TOKENS} rows = max_num_tokens (large): 6.7e9 " + "elements, past 2^31 and 2^32; 81 GiB peak", + spec=dict( + rows=MAX_NUM_TOKENS, + lo=-400.0, + hi=400.0, + graph=False, + windows=( + (0, 16), + (8184, 8200), + (10480, 10492), # element 2^31 is row 10485, column 155648 + (16376, 16392), + (20966, 20978), # element 2^32 is row 20971, column 106496 + (24568, 24584), + (MAX_NUM_TOKENS - 16, MAX_NUM_TOKENS), + ), + large=True, + **_CKPT, + ), + ), + ) + + note = """ + Metadata consumed: none. Out-of-place; `x` is not mutated. Three launches, + allocating the quotient, the tanh and the output, each of `x`'s size, at + most two live at once; the first two are freed on return. That is 12.5 MiB + each at 16 x 204800 fp32, but 25 GiB each at a 32768-row context-logits + step: the call then holds 50 GiB beyond its 25 GiB input at its peak + (measured), which a caller serving context logits at the token budget must + leave free. Under CUDA-graph capture they come from the graph's pool and + `cap` is baked into the captured kernels. Rows are independent: a padded + batch's dummy rows, nan or inf included, never reach the live ones, and a + row's result does not depend on how many rows the call holds (measured: + row windows of a 10485- and a 32768-row call are bitwise the op run on + those rows alone, and every captured batch 1..16 replays bitwise eager). + + `cap` is a Python float. A 0-d CUDA tensor `cap` would take torch's true + division instead of the reciprocal multiply: within one ulp of it, not + bitwise equal (measured); not a certified form. + + Numerics (fp32): at most 4 ulp from `reference` over every cell, under a + derived 7.5 ulp bound (the test states the derivation). |x| >= + 270.327392578125 returns exactly +-30.0 (the fp32 tanh rounds to 1); + +-inf -> +-30.0; nan -> nan; -0.0 -> -0.0. Inputs below 30 * 2^-126 + (about 3.5e-37) in magnitude pass through a subnormal quotient and lose + relative precision (1e-40 -> 1.0001e-40), absolute error below 1e-43. + + `cap` must be finite and non-zero: `is_valid` refuses 0, +-inf and nan, + which the op accepts and answers with zeros and nan (cap 0: every logit + becomes +-0 and an exact 0 becomes nan; an infinite or nan cap: nan + everywhere). A negative cap returns the |cap| result bitwise (tanh is odd). + + Hard limits (sm_103) + x.numel() elementwise TensorIterator launches boundary 2^31 elements + (crossed from 10486 rows at width 204800) probed at 32768 x 204800 = + 6710886400 elements, past 2^31 and 2^32 passed: every element + within the bound, and rows straddling elements 2^31 and 2^32 + and the quarter points bitwise the op run on those rows alone + + Not certified: bf16/fp16 input (each step would round to the input dtype), + non-contiguous input, a width other than 204800, torch.compile. + """ + + def __call__(self, x: torch.Tensor, cap: float) -> torch.Tensor: + return torch.tanh(x / cap) * cap + + def reference(self, x: torch.Tensor, cap: float) -> torch.Tensor: + """The closed form in fp64, rounded once to `x.dtype`.""" + return (cap * torch.tanh(x.double() / cap)).to(x.dtype) + + def is_valid(self, x: torch.Tensor, cap: float) -> None: + if not (math.isfinite(cap) and cap != 0.0): + raise ValueError( + f"tanh_softcap: cap must be finite and non-zero, got {cap!r}; " + "a zero cap zeroes every logit and an infinite or nan one makes them nan" + ) + + +tanh_softcap = TanhSoftcap() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/__init__.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/__init__.py new file mode 100644 index 000000000000..52a7a9daf028 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/__init__.py @@ -0,0 +1,2 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/convert_req_index_to_global.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/convert_req_index_to_global.py new file mode 100644 index 000000000000..75300c7c70d7 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/convert_req_index_to_global.py @@ -0,0 +1,1830 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Map a DSA request-local KV position to its row in the paged latent KV pool.""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: Mistral Large 4 FP8 at TEP4 under its serving envelope (max_seq_len = max_num_tokens = 9216, +#: max_batch_size 16): pages of 32 tokens, 2048 selected keys, a pool that interleaves all +#: 72 layers' pages (`stride_factor` 72 x 32 = 2304), and the cache manager's block table of +#: max_batch_size rows by 9216 / 32 = 288 columns. Page ids are drawn from an 18432-block +#: pool, more than a rank can hold: about 24.7 GiB is left per GB300 after weights and a +#: token costs about 47 KB (latent plus indexer keys), which is about 17.5K pages -- an +#: estimate, as the pool has not been measured yet. Rows of the table no request reaches +#: hold stale page ids rather than -1, as the engine leaves them. +_ML4 = dict( + block_size=32, topk=2048, pool_layers=72, table_rows=16, table_width=288, pool_blocks=18432 +) + +#: Mistral Large 4 NVFP4 at TEP4 under its serving envelope (max_seq_len = max_num_tokens = +#: 32768, no chunked prefill, max_batch_size 16): the same pages, top-k and 72-layer stride, +#: and the cache manager's block table of max_batch_size rows by 32768 / 32 = 1024 columns. +#: Page ids are drawn from a 107917-block pool, the most one GB300 rank could hold: the +#: device's 276.62 GiB less the target's 143.24 GiB of weights, all of it 72-layer fp8 +#: latent at 72 x 32 x 576 = 1327104 bytes a page -- a bound, as no gate has measured the +#: engine's pool, which also holds indexer keys and leaves room for activations. +_ML4_32K = dict( + block_size=32, topk=2048, pool_layers=72, table_rows=16, table_width=1024, pool_blocks=107917 +) + +#: The `layer_id` each of Mistral Large 4's 72 layers passes in one step: its own pool row, +#: except the KV sharers 39, 43, ..., 71, which read layer 35's latent (the checkpoint's +#: `kv_source_layers`). +_ML4_STEP = tuple(35 if i >= 39 and i % 4 == 3 else i for i in range(72)) + +#: The fp8_ds_mla pool (MistralLarge4CacheManager) holds pages for the 63 KV owners only: +#: stride 63 x tokens_per_block (2016 at 32-token pages), and layer i passes its source's +#: owner row -- the sharers 39, 43, ..., 71 read layer 35's, owner row 35. Both are plain +#: expressions, so the cells read without running the module: `pool_layers` 63 is +#: len(_ML4_OWNERS), and `_ML4_DS_STEP[i]` is `_ML4_OWNERS.index(_ML4_STEP[i])` in closed form +#: (an owner past 39 sits (i - 35) // 4 rows lower, one per sharer below it); the test +#: checks both against the owner list. +_ML4_OWNERS = tuple(i for i in range(72) if _ML4_STEP[i] == i) +_ML4_32K_DS = dict(_ML4_32K, pool_layers=63) +_ML4_DS_STEP = tuple( + 35 if i >= 39 and i % 4 == 3 else i - (i - 35) // 4 if i >= 39 else i for i in range(72) +) + +#: The same pool at 64-token pages: stride 63 x 64 = 4032, a block table of 32768 / 64 = 512 +#: columns, and 53958 blocks, the 107917-block bound's token capacity. Its last row, +#: 53957 x 4032 + 62 x 64 + 63 = 217558655, is 10.1% of 2**31, as at 32-token pages. +_ML4_32K_DS64 = dict(_ML4_32K_DS, block_size=64, table_width=512, pool_blocks=53958) + +#: Mistral Large 4 NVFP4 on eight GB300s over two nodes on the fp8_ds_mla pool -- tep8 and tp8 +#: (8 of the 64 heads a rank; the two shard attention alike) and dep8 (eight attention-DP +#: ranks, all 64 heads on each): the call unchanged, page ids from a 147456-block pool. A tep8 +#: or tp8 rank holds 73.61 GiB of weights and a dep8 rank 89.29 GiB, which leave 203.01 and +#: 187.33 GiB of a 276.62 GiB GB300; at 47136 bytes a token (63 owner rows of 656 bytes and 44 +#: indexer-key rows of 132), 1508352 bytes a 32-token page, that is 144519 and 133355 pages: +#: 147456 is above both -- a bound computed from the shard headers; the engine's pool, which +#: also leaves room for activations, is smaller. +_ML4_32K_DS_8GPU = dict(_ML4_32K_DS, pool_blocks=147456) + +#: The same pool behind the block table the engine's cache manager builds +#: (`MistralLarge4CacheManager`, a `DSACacheManagerV2`): `max_blocks_per_seq` columns, +#: (32768 + 1) / 32 rounded up to 1025 and padded to a multiple of 4. +_ML4_32K_DS_8GPU_V2 = dict(_ML4_32K_DS_8GPU, table_width=1028) + +#: 16 requests packed into 9216 tokens, the context step at max_batch_size. +_ML4_PACKED = [1, 7, 33, 100, 255, 257, 513, 1023, 1025, 2047, 2049, 931, 299, 301, 201, 174] + +#: 16 requests packed into 32768 tokens, the NVFP4 target's context step at max_batch_size. +_ML4_32K_PACKED = [ + 1, + 7, + 33, + 100, + 255, + 257, + 513, + 1023, + 1025, + 2047, + 2049, + 4095, + 4097, + 8191, + 8193, + 882, +] + +#: Mistral Large 4 NVFP4 at TEP4 at the checkpoint's own context (max_seq_len 1048576, chunked +#: prefill in steps of max_num_tokens 32768, max_batch_size 16): the same pages, top-k, +#: 72-layer stride and 107917-block pool bound, and a block table of max_batch_size rows by +#: 1048576 / 32 = 32768 columns. Positions reach 1048575, column 32767. +_ML4_1M = dict( + block_size=32, topk=2048, pool_layers=72, table_rows=16, table_width=32768, pool_blocks=107917 +) + +#: The width the engine's table has at 1048576: `KVCacheManagerV2` sizes `max_blocks_per_seq` +#: for max_seq_len plus one generation token of headroom, ceil(1048577 / 32) = 32769 pages, +#: padded to a multiple of 4 (`_torch/pyexecutor/kv_cache/kv_cache_manager_v2.py`), and the +#: DSA metadata's `block_table` is that wide. No position below max_seq_len reaches its last +#: four columns. +_ML4_1M_V2 = dict(_ML4_1M, table_width=32772) + +#: The tp4 and tep4 deployments on the fp8_ds_mla pool at 1048576: the engine's 32772-column table +#: and the 107917-block bound, at the 63 owners' stride 2016. Their boots at this envelope held +#: 69011 blocks (2208352 tokens; tp4) and 72757 blocks (2328224 tokens; tep4) per rank, inside +#: the bound. +_ML4_1M_DS = dict(_ML4_1M_V2, pool_layers=63) + +#: The tep8 deployment on the fp8_ds_mla pool at 1048576: the engine's 32772-column table and the +#: eight-rank 147456-block bound (`_ML4_32K_DS_8GPU`'s), at the 63 owners' stride 2016. Its boot +#: at this envelope held 3658432 tokens, 114326 blocks, per rank, inside the bound. +_ML4_1M_TEP8_DS = dict(_ML4_1M_DS, pool_blocks=147456) + +#: 16 chunk tails packed into one 32768-row context step, each request's query rows; its KV +#: lengths (cached prefix plus those rows) are the cells' `kv_lens`: the last token of a +#: 1048576-token prompt, fresh prompts, and tails behind prefixes of 1, 31, 32, 33, 2047, +#: 2048, 2049, 32767, 65535, 131072, 262143 and 500000 tokens. +_ML4_1M_TAIL_ROWS = [1, 7, 31, 32, 33, 100, 255, 257, 513, 1023, 1025, 2047, 2049, 8191, 8193, 9011] + +#: Mistral Large 4 NVFP4 at DEP4 at the checkpoint's own context (max_seq_len 1048576, chunked +#: prefill in steps of max_num_tokens 32768 per rank, max_batch_size 16 per rank): the same +#: pages, top-k, 72-layer stride and 107917-block pool bound (a dep4 rank's V2 pool holds about +#: 62K slots), and the DSA metadata's block table of max_num_sequences 16 rows by the V2 +#: manager's `max_blocks_per_seq`, 32772 columns: ceil(1048577 / 32) = 32769 pages (one +#: generation token of headroom), padded to a multiple of 4, as a `DSACacheManagerV2` built at +#: max_seq_len 1048576 reports it. Positions reach 1048575, column 32767; no position below +#: max_seq_len reaches the last four columns. +_ML4_1M_DEP4 = dict( + block_size=32, topk=2048, pool_layers=72, table_rows=16, table_width=32772, pool_blocks=107917 +) + +#: The dep4 deployment on the fp8_ds_mla pool at 1048576: the same 16 x 32772 table and +#: 107917-block bound at the 63 owners' stride 2016. The dep4 boot on this pool held 64633 +#: blocks (2068256 tokens) on every rank, inside the bound. +_ML4_1M_DEP4_DS = dict(_ML4_1M_DEP4, pool_layers=63) + + +class ConvertReqIndexToGlobal(OpWrapper): + """Turn the indexer's per-request top-k positions into pool rows. + + Entry `token_indices[i, j]` is a KV position `p` counted from the start of + request `req_id[i]`'s own sequence. The op returns + + block_table[req_id[i], p // block_size] * stride_factor + + layer_id * block_size + p % block_size + + which is the row of the latent pool viewed as `[-1, head_dim]` that holds + that token's latent KV for this layer. A position that cannot be resolved + comes back as -1: `p < 0` (the top-k padding), `p // block_size` at or past + `block_table.size(1)`, or a negative block-table entry (an unused page). + + Returns a fresh contiguous int32 tensor shaped like `token_indices`; no + argument is modified. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # It is called once per layer on the step's whole top-k buffer: one row per + # query token, `topk` columns. `pool_layers` sets `stride_factor = + # pool_layers * block_size`, the page stride of a pool that interleaves + # every layer's pages; `layer_id` picks the layer inside a page. + # + # Mistral Large 4 calls it the same way, at its envelope: decode rows are the + # sequences of a captured batch (1, 2, 4, 8, 16, padded), each `graph` cell + # capturing the step's 72 calls (`layer_ids`) and replaying them on new + # batch data; prefill rows reach 9216 per call (the FP8 target, "ML4") or + # 32768 (the NVFP4 target, "ML4 32768": DecodeTarget._attention and + # PrefillTarget._attention, one call per layer). On eight GPUs over the + # fp8_ds_mla pool ("ML4 fp8_ds_mla tep8/tp8/dep8", the calls of all three + # targets) the call is the same and only the pool grows, to page ids up to + # 147455; the "dep8 V2 table" cells add a rank's attention-DP padding (one + # live sequence replaying the batch-16 graph behind 15 dummies) and a packed + # context step over the 1028-column table. `kv_lens` are the requests' KV + # lengths and `seq_rows` their query rows, so row `j` of request `s` sees + # positions `[0, kv_lens[s] - seq_rows[s] + j + 1)`; `last_position` puts + # each row's newest position among its selections, so a context that fills + # the table reads its last column. + # + # The NVFP4 target at max_seq_len 1048576 ("ML4 1M") keeps those call sites + # and steps of at most 32768 rows; what grows is the position (to 1048575) + # and the table (32768 columns, and the 32772 the engine allocates). A + # chunk's rows sit behind the request's cached prefix, so a step packs chunk + # tails whose positions reach far past its own row count. + # + # The tp4 deployment ("ML4 1M tp4") makes the same calls at the same envelope: + # `md.block_table`, the stride of the pool mapping, the step's index-mode + # context rows and a mixed step's generation rows from + # `PrefillTarget._attention`, and the step's 72 calls from + # `DecodeTarget._attention`. `layer_ids="all"` is every layer id of the page, + # 0..pool_layers - 1, with one call each. `order="chunk"` splits each chunk + # row's selections between the chunk's own rows up to that row (half of + # top-k behind a cached prefix of half or more, all of them when there are + # fewer, more behind a shorter prefix) and the request's cached prefix (the + # rest). Each part is drawn by strata, and the columns come in random order. + # + # The tp4 and tep4 deployments on the fp8_ds_mla pool ("ML4 1M fp8_ds_mla" and + # "ML4 1M tep4 fp8_ds_mla") pass stride 63 x 32 = 2016 and each layer's + # `latent_pool_row(kv_source)` (`_ML4_DS_STEP`) at three sites: the step's 72 + # calls from `DecodeTarget._attention`; a mixed step's generation rows from + # `PrefillTarget._attention`, whose requests are the table's rows from + # `num_contexts` on (`first_seq`); and, on a step whose context KV exceeds + # 2**19 rows, every context row of the step in one call from + # `PrefillTarget._attn_ctx`. `layer_ids="owner_rows_0_62"` is every owner row + # of the 63-owner page, one call each. + # + # The NVFP4 dep4 deployment at max_seq_len 1048576 ("dep4 1M") keeps those call + # sites, each rank running its own requests, and steps of at most 32768 + # rows; what grows is the position (to 1048575) and the table (the 32772 + # columns the engine allocates). A chunk's rows sit behind the request's + # cached prefix, so a step's positions reach far past its own row count. + # The engine captures each decode graph at boot on its longest dummy + # request, max_seq_len - 1 = 1048575 keys, beside 1-key padding rows, and + # replays it on the step's data: `replay_kv_lens` names the replays' contexts + # (without it a graph cell replays the capture's contexts reversed). + # + # The dep4 deployment on the fp8_ds_mla pool ("dep4 1M fp8_ds_mla") passes stride + # 63 x 32 = 2016 and each layer's `latent_pool_row(kv_source)` (`_ML4_DS_STEP`) + # at three sites: the step's 72 calls from `DecodeTarget._attention`; a mixed + # step's generation rows from `PrefillTarget._attention`, whose requests are + # the table's rows from `num_contexts` on (`first_seq`); and every context row + # of a step whose context KV exceeds 2**19 rows in one call, from + # `PrefillTarget._attn_ctx` ("ctx rows"). `boot_capture` captures a graph on + # the boot's warm-up batch, the longest dummy (max_seq_len - 1 keys) first and + # one-key fillers, each a request of its own. Under attention DP a rank pads + # its live rows to the largest rank's graph batch with one CUDA-graph padding + # dummy repeated (one key, one page): `pad` (and per replay, `replay_pad`) + # makes the last that many requests that dummy, rows of the table of their own + # holding the same page. `order="chunk"` is the split above. + # `layer_ids="owner_rows_0_62"` is every owner row of the 63-owner page. + # + # The tep8 deployment on the fp8_ds_mla pool passes stride 63 x 32 = 2016 and + # each layer's `latent_pool_row(kv_source)` (`_ML4_DS_STEP`). At 32768 its + # calls are the "ML4 fp8_ds_mla tep8/tp8/dep8" cells': only its pool grows, + # to page ids up to 147455. At max_seq_len 1048576 ("ML4 1M tep8 + # fp8_ds_mla", `_ML4_1M_TEP8_DS`) it calls the op at the three sites tp4's + # and tep4's calls do: the step's 72 calls from `DecodeTarget._attention`; + # a mixed step's generation rows from `PrefillTarget._attention`, whose + # requests are the table's rows from `num_contexts` on (`first_seq`); and, + # on a step whose context KV exceeds 2**19 rows, every context row of the + # step in one call from `PrefillTarget._attn_ctx`. Its cells are tep4's on + # the 147456-block bound. `layer_ids="owner_rows_0_62"` is every owner row + # of the 63-owner page, one call each; `order="chunk"` is the split above. + CELLS: tuple[Cell, ...] = ( + Cell( + why="ML4 decode graph batch 1 (captured), 72 calls: one sequence at " + "max_seq_len 9216, its last position in the table's last column", + spec=dict(_ML4, graph=True, seq_rows=[1], kv_lens=[9216], layer_ids=_ML4_STEP), + ), + Cell( + why="ML4 decode graph batch 2 (captured), 72 calls: contexts of 1 and " + "exactly 2048 in the dense causal order the engine writes for a skipped indexer", + spec=dict( + _ML4, + graph=True, + order="causal", + seq_rows=[1, 1], + kv_lens=[1, 2048], + layer_ids=_ML4_STEP, + ), + ), + Cell( + why="ML4 decode graph batch 4 (captured), 72 calls: contexts of 2049 " + "(one position past top-k), 31 and 33 (one page and a page plus one), 5000", + spec=dict( + _ML4, + graph=True, + seq_rows=[1] * 4, + kv_lens=[2049, 31, 5000, 33], + layer_ids=_ML4_STEP, + ), + ), + Cell( + why="ML4 decode graph batch 8 (captured), 72 calls: contexts across " + "1..9216, several longer than top-k", + spec=dict( + _ML4, + graph=True, + seq_rows=[1] * 8, + kv_lens=[9216, 4097, 2047, 64, 100, 8000, 3000, 6145], + layer_ids=_ML4_STEP, + ), + ), + Cell( + why="ML4 decode graph batch 16 (captured), 72 calls: 13 sequences and 3 " + "padding rows (1-token dummies), pages at the far end of the pool", + spec=dict( + _ML4, + graph=True, + far_end=True, + seq_rows=[1] * 16, + kv_lens=[9216, 1, 2048, 2049, 512, 7777, 33, 9000, 4096, 1500, 300, 8191, 6000] + + [1] * 3, + layer_ids=_ML4_STEP, + ), + ), + Cell( + why="ML4 prefill at max_num_tokens: one 9216-token request, causal windows, " + "the last pool layer; every row count 1..9216 is a bitwise prefix of it (about 1.4 s)", + spec=dict(_ML4, seq_rows=[9216], kv_lens=[9216], layer_ids=(71,), prefix_sweep=True), + ), + Cell( + why="ML4 context step at max_batch_size: 16 requests packed into 9216 rows, " + "the KV sharers' pool row 35, pages at the far end of the pool", + spec=dict( + _ML4, far_end=True, seq_rows=_ML4_PACKED, kv_lens=_ML4_PACKED, layer_ids=(35,) + ), + ), + Cell( + why="ML4 mixed step: 3 context requests, then 13 generation rows with " + "contexts up to 9216, in one 6013-row call", + spec=dict( + _ML4, + seq_rows=[3000, 2000, 1000] + [1] * 13, + kv_lens=[3000, 2000, 1000] + + [9216, 2, 2048, 2049, 4500, 8000, 1, 700, 3333, 9100, 6000, 129, 5000], + layer_ids=(0,), + ), + ), + Cell( + why="ML4 every layer id 0..71 at stride 72 x 32 = 2304, eager, 16 decode " + "rows: a pool that gives the KV sharers rows of their own", + spec=dict( + _ML4, + seq_rows=[1] * 16, + kv_lens=[9216, 1, 2048, 2049, 512, 7777, 33, 9000, 4096, 1500, 300, 8191] + + [6000, 2, 64, 4097], + layer_ids=tuple(range(72)), + ), + ), + Cell( + why="ML4 32768 decode graph batch 1 (captured), 72 calls: one sequence at " + "max_seq_len 32768, its newest position selected, in the 1024-column table's " + "last column", + spec=dict( + _ML4_32K, + graph=True, + last_position=True, + seq_rows=[1], + kv_lens=[32768], + layer_ids=_ML4_STEP, + ), + ), + Cell( + why="ML4 32768 decode graph batch 2 (captured), 72 calls: contexts of 1 and " + "32768, swapped between the capture and the replay", + spec=dict( + _ML4_32K, + graph=True, + last_position=True, + seq_rows=[1, 1], + kv_lens=[1, 32768], + layer_ids=_ML4_STEP, + ), + ), + Cell( + why="ML4 32768 decode graph batch 4 (captured), 72 calls: contexts of 32737 " + "(one position in the last column), 32736 (the last column unused), 2049, 33", + spec=dict( + _ML4_32K, + graph=True, + last_position=True, + seq_rows=[1] * 4, + kv_lens=[32737, 33, 2049, 32736], + layer_ids=_ML4_STEP, + ), + ), + Cell( + why="ML4 32768 decode graph batch 8 (captured), 72 calls: contexts across " + "1..32768, most longer than top-k", + spec=dict( + _ML4_32K, + graph=True, + last_position=True, + seq_rows=[1] * 8, + kv_lens=[32768, 16384, 9217, 4097, 2047, 64, 24000, 30001], + layer_ids=_ML4_STEP, + ), + ), + Cell( + why="ML4 32768 decode graph batch 16 (captured), 72 calls: 13 sequences up to " + "32768 and 3 padding rows (1-token dummies), pages at the far end of the pool", + spec=dict( + _ML4_32K, + graph=True, + far_end=True, + last_position=True, + seq_rows=[1] * 16, + kv_lens=[32768, 1, 2048, 2049, 512, 31111, 33, 9216, 16385, 1500, 300, 32767] + + [20000] + + [1] * 3, + layer_ids=_ML4_STEP, + ), + ), + Cell( + why="ML4 32768 prefill at max_num_tokens: one 32768-token request, causal " + "windows, the last pool layer; every row count 1..32768 is a bitwise prefix of it " + "(about 8 s)", + spec=dict( + _ML4_32K, + last_position=True, + seq_rows=[32768], + kv_lens=[32768], + layer_ids=(71,), + prefix_sweep=True, + ), + ), + Cell( + why="ML4 32768 context step at max_batch_size: 16 requests packed into 32768 " + "rows, the KV sharers' pool row 35, pages at the far end of the pool", + spec=dict( + _ML4_32K, + far_end=True, + seq_rows=_ML4_32K_PACKED, + kv_lens=_ML4_32K_PACKED, + layer_ids=(35,), + ), + ), + Cell( + why="ML4 32768 mixed step: 3 context requests, then 13 generation rows with " + "contexts up to 32768, in one 32768-row call", + spec=dict( + _ML4_32K, + last_position=True, + seq_rows=[16000, 10000, 6755] + [1] * 13, + kv_lens=[16000, 10000, 6755] + + [32768, 2, 2048, 2049, 4500, 32767, 1, 700, 16385, 30000, 9216, 129, 32737], + layer_ids=(0,), + ), + ), + Cell( + why="ML4 32768 every layer id 0..71 at stride 72 x 32 = 2304, eager, 16 decode " + "rows with contexts up to 32768", + spec=dict( + _ML4_32K, + last_position=True, + seq_rows=[1] * 16, + kv_lens=[32768, 1, 2048, 2049, 512, 31111, 33, 9216, 16385, 1500, 300, 32767] + + [20000, 2, 64, 32737], + layer_ids=tuple(range(72)), + ), + ), + Cell( + why="ML4 1M decode graph batch 1 (captured), 72 calls: one sequence at max_seq_len " + "1048576, its newest position 1048575 selected, in the 32768-column table's last " + "column", + spec=dict( + _ML4_1M, + graph=True, + last_position=True, + seq_rows=[1], + kv_lens=[1048576], + layer_ids=_ML4_STEP, + ), + ), + Cell( + why="ML4 1M decode graph batch 2 (captured), 72 calls: contexts of 1 and 1048576, " + "swapped between the capture and the replay", + spec=dict( + _ML4_1M, + graph=True, + last_position=True, + seq_rows=[1, 1], + kv_lens=[1, 1048576], + layer_ids=_ML4_STEP, + ), + ), + Cell( + why="ML4 1M decode graph batch 4 (captured), 72 calls: contexts of 1048545 (one " + "position in the last column), 1048544 (the last column unused), 2049, 33", + spec=dict( + _ML4_1M, + graph=True, + last_position=True, + seq_rows=[1] * 4, + kv_lens=[1048545, 33, 2049, 1048544], + layer_ids=_ML4_STEP, + ), + ), + Cell( + why="ML4 1M decode graph batch 8 (captured), 72 calls: contexts across 1..1048576, " + "most longer than top-k", + spec=dict( + _ML4_1M, + graph=True, + last_position=True, + seq_rows=[1] * 8, + kv_lens=[1048576, 524288, 131073, 32769, 2047, 64, 700001, 1000000], + layer_ids=_ML4_STEP, + ), + ), + Cell( + why="ML4 1M decode graph batch 16 (captured), 72 calls: 13 sequences up to 1048576 " + "and 3 padding rows (1-token dummies), pages at the far end of the pool", + spec=dict( + _ML4_1M, + graph=True, + far_end=True, + last_position=True, + seq_rows=[1] * 16, + kv_lens=[1048576, 1, 2048, 2049, 512, 1000001, 33, 9216, 32769, 1500, 300] + + [1048575, 524288, 1, 1, 1], + layer_ids=_ML4_STEP, + ), + ), + Cell( + why="ML4 1M prefill: the last 32768-token chunk of a 1048576-token prompt, behind " + "its 1015808-token cached prefix, the last pool layer; every row count 1..32768 is " + "a bitwise prefix of it", + spec=dict( + _ML4_1M, + last_position=True, + seq_rows=[32768], + kv_lens=[1048576], + layer_ids=(71,), + prefix_sweep=True, + ), + ), + Cell( + why="ML4 1M context step at max_batch_size: 16 chunk tails packed into 32768 rows, " + "cached prefixes up to 1048575, the KV sharers' pool row 35, pages at the far end " + "of the pool", + spec=dict( + _ML4_1M, + far_end=True, + seq_rows=_ML4_1M_TAIL_ROWS, + kv_lens=[1048576, 7, 32, 63, 65, 133, 2302, 2305, 2562, 33790, 66560, 133119] + + [264192, 508191, 8193, 9011], + layer_ids=(35,), + ), + ), + Cell( + why="ML4 1M mixed step: the last 16000 rows of a 1M prompt, 10000 behind a 490000 " + "prefix and a fresh 6755, then 13 generation rows with contexts up to 1048576, in " + "one 32768-row call", + spec=dict( + _ML4_1M, + last_position=True, + seq_rows=[16000, 10000, 6755] + [1] * 13, + kv_lens=[1048576, 500000, 6755, 1048576, 2, 2048, 2049, 4500, 1048575, 1, 700] + + [32769, 1000000, 131072, 129, 1048545], + layer_ids=(0,), + ), + ), + Cell( + why="ML4 1M every layer id 0..71 at stride 72 x 32 = 2304, eager, 16 decode rows " + "with contexts up to 1048576", + spec=dict( + _ML4_1M, + last_position=True, + seq_rows=[1] * 16, + kv_lens=[1048576, 1, 2048, 2049, 512, 1000001, 33, 9216, 32769, 1500, 300] + + [1048575, 524288, 2, 64, 1048545], + layer_ids=tuple(range(72)), + ), + ), + Cell( + why="ML4 1M 32772-column V2 table, graph batch 1 (captured), 72 calls: the boot " + "capture's longest row, 1048576, its newest position in column 32767, four columns " + "short of the table's end", + spec=dict( + _ML4_1M_V2, + graph=True, + last_position=True, + seq_rows=[1], + kv_lens=[1048576], + layer_ids=_ML4_STEP, + ), + ), + Cell( + why="ML4 1M 32772-column V2 table, graph batch 16 (captured), 72 calls: contexts of " + "1048576, 1048545 and 524288 beside short ones and 8 one-token rows, swapped " + "end for end by the replay, pages at the far end of the pool", + spec=dict( + _ML4_1M_V2, + graph=True, + far_end=True, + last_position=True, + seq_rows=[1] * 16, + kv_lens=[1048576, 1, 2049, 1048545, 33, 2048, 524288, 9216] + [1] * 8, + layer_ids=_ML4_STEP, + ), + ), + Cell( + why="ML4 1M 32772-column V2 table, context step at max_batch_size: the 16 chunk " + "tails of 32768 rows, cached prefixes up to 1048575, pool row 35, pages at the far " + "end of the pool", + spec=dict( + _ML4_1M_V2, + far_end=True, + last_position=True, + seq_rows=_ML4_1M_TAIL_ROWS, + kv_lens=[1048576, 7, 32, 63, 65, 133, 2302, 2305, 2562, 33790, 66560, 133119] + + [264192, 508191, 8193, 9011], + layer_ids=(35,), + ), + ), + Cell( + why="ML4 1M 32772-column V2 table, mixed step: 3 context chunks behind cached " + "prefixes up to 1032576, then 13 generation rows with contexts up to 1048576, " + "in one 32768-row call, pool row 70", + spec=dict( + _ML4_1M_V2, + last_position=True, + seq_rows=[16000, 10000, 6755] + [1] * 13, + kv_lens=[1048576, 500000, 6755, 1048576, 2, 2048, 2049, 4500, 1048575, 1, 700] + + [32769, 1000000, 131072, 129, 1048545], + layer_ids=(70,), + ), + ), + Cell( + why="ML4 1M tp4 decode graph batch 1 (captured), all 72 layer ids 0..71: one " + "sequence at 1048576, its newest position 1048575 in the 32768-column table's last " + "column", + spec=dict( + _ML4_1M, + graph=True, + last_position=True, + seq_rows=[1], + kv_lens=[1048576], + layer_ids="all", + ), + ), + Cell( + why="ML4 1M tp4 decode graph batch 16 (captured), 72 calls: 13 sequences, 1048576 " + "halved to 65536, then 32769, 32768, 16385, 9217, 4097, 2049, 2048 and 257, and 3 " + "padding rows (1-token dummies), all 66609 pages the pool's top ones", + spec=dict( + _ML4_1M, + graph=True, + far_end=True, + last_position=True, + seq_rows=[1] * 16, + kv_lens=[1048576, 524288, 262144, 131072, 65536, 32769, 32768, 16385, 9217] + + [4097, 2049, 2048, 257, 1, 1, 1], + layer_ids=_ML4_STEP, + ), + ), + Cell( + why="ML4 1M tp4 prefill chunk order: the last 32768-token chunk of a 1048576-token " + "prompt, each row's selections split between the chunk's own rows up to it and the " + "1015808-token cached prefix, pool rows 0, 35 and 70", + spec=dict( + _ML4_1M, + order="chunk", + last_position=True, + seq_rows=[32768], + kv_lens=[1048576], + layer_ids=(0, 35, 70), + ), + ), + Cell( + why="ML4 1M tp4 packed chunk step: 16384, 8192, 4096 and 4096 rows behind cached " + "prefixes of 1032192, 524288, 0 and 2048 in one 32768-row call, pool row 70, all " + "49728 pages the pool's top ones", + spec=dict( + _ML4_1M, + far_end=True, + last_position=True, + seq_rows=[16384, 8192, 4096, 4096], + kv_lens=[1048576, 532480, 4096, 6144], + layer_ids=(70,), + ), + ), + *( + Cell( + why=f"dep4 1M boot capture, decode graph batch {b}, 72 calls: the warm-up's " + f"longest dummy, 1048575 keys, beside {b - 1} one-key padding rows; replayed " + "with one live row at 1048576 behind the padding, then the capture reversed", + spec=dict( + _ML4_1M_DEP4, + graph=True, + last_position=True, + seq_rows=[1] * b, + kv_lens=[1048575] + [1] * (b - 1), + layer_ids=_ML4_STEP, + replay_kv_lens=([1048576] + [1] * (b - 1), [1] * (b - 1) + [1048575]), + ), + ) + for b in (1, 2, 4, 8, 16) + ), + Cell( + why="dep4 1M decode graph batch 1 (captured), 72 calls: one sequence at max_seq_len " + "1048576, its newest position 1048575 selected, in column 32767 of the 32772-column " + "table", + spec=dict( + _ML4_1M_DEP4, + graph=True, + last_position=True, + seq_rows=[1], + kv_lens=[1048576], + layer_ids=_ML4_STEP, + ), + ), + Cell( + why="dep4 1M decode graph batch 2 (captured), 72 calls: contexts of 1 and 1048576, " + "swapped between the capture and the replay", + spec=dict( + _ML4_1M_DEP4, + graph=True, + last_position=True, + seq_rows=[1, 1], + kv_lens=[1, 1048576], + layer_ids=_ML4_STEP, + ), + ), + Cell( + why="dep4 1M decode graph batch 4 (captured), 72 calls: contexts of 1048545 (one " + "position in column 32767), 1048544 (column 32767 unused), 2049, 33", + spec=dict( + _ML4_1M_DEP4, + graph=True, + last_position=True, + seq_rows=[1] * 4, + kv_lens=[1048545, 33, 2049, 1048544], + layer_ids=_ML4_STEP, + ), + ), + Cell( + why="dep4 1M decode graph batch 8 (captured), 72 calls: contexts across 1..1048576, " + "past each power of two from 32768, most longer than top-k", + spec=dict( + _ML4_1M_DEP4, + graph=True, + last_position=True, + seq_rows=[1] * 8, + kv_lens=[1048576, 524288, 262145, 131073, 32769, 2047, 64, 1000001], + layer_ids=_ML4_STEP, + ), + ), + Cell( + why="dep4 1M decode graph batch 16 (captured), 72 calls: 13 sequences up to 1048576 " + "and 3 padding rows (1-token dummies); their 122596 pages outnumber the pool and run " + "through all of it, its top block included", + spec=dict( + _ML4_1M_DEP4, + graph=True, + far_end=True, + last_position=True, + seq_rows=[1] * 16, + kv_lens=[1048576, 1, 2048, 2049, 512, 1000000, 33, 32769, 524289, 1500, 300] + + [1048575, 262144, 1, 1, 1], + layer_ids=_ML4_STEP, + ), + ), + Cell( + why="dep4 1M prefill: the last 32768-token chunk of a 1048576-token prompt, behind " + "its 1015808-token cached prefix, the last pool layer; every row count 1..32768 is " + "a bitwise prefix of it", + spec=dict( + _ML4_1M_DEP4, + last_position=True, + seq_rows=[32768], + kv_lens=[1048576], + layer_ids=(71,), + prefix_sweep=True, + ), + ), + Cell( + why="dep4 1M context step at max_batch_size: 16 chunk tails of 16 rows, each behind " + "a 65520-token cached prefix, the KV sharers' pool row 35, pages at the far end of " + "the pool", + spec=dict( + _ML4_1M_DEP4, + far_end=True, + seq_rows=[16] * 16, + kv_lens=[65536] * 16, + layer_ids=(35,), + ), + ), + Cell( + why="dep4 1M mixed step: chunks of 16000 rows behind 500000 cached, 10000 fresh and " + "6755 behind 1000000 cached, then 13 generation rows with contexts up to 1048576, in " + "one 32768-row call", + spec=dict( + _ML4_1M_DEP4, + last_position=True, + seq_rows=[16000, 10000, 6755] + [1] * 13, + kv_lens=[516000, 10000, 1006755, 1048576, 2, 2048, 2049, 4500, 1048575, 1, 700] + + [524289, 30000, 9216, 129, 1048545], + layer_ids=(0,), + ), + ), + Cell( + why="dep4 1M every layer id 0..71 at stride 72 x 32 = 2304, eager, 16 decode rows " + "with contexts up to 1048576", + spec=dict( + _ML4_1M_DEP4, + last_position=True, + seq_rows=[1] * 16, + kv_lens=[1048576, 1, 2048, 2049, 512, 1000000, 33, 32769, 524289, 1500, 300] + + [1048575, 262144, 2, 64, 1048545], + layer_ids=tuple(range(72)), + ), + ), + Cell( + why="ML4 fp8_ds_mla decode graph batch 16 (captured), 72 calls over the 63-owner " + "pool: stride 2016, each layer at its source's owner row, contexts up to 32768", + spec=dict( + _ML4_32K_DS, + graph=True, + far_end=True, + seq_rows=[1] * 16, + kv_lens=[32768, 1, 2048, 2049, 512, 31111, 33, 9216, 16385, 1500, 300, 32767] + + [1] * 4, + layer_ids=_ML4_DS_STEP, + ), + ), + Cell( + why="ML4 fp8_ds_mla mixed step over the 63-owner pool: 2 context requests and 6 " + "generation rows, layer 35's owner row (the sharers' source) and the last", + spec=dict( + _ML4_32K_DS, + seq_rows=[20000, 4000] + [1] * 6, + kv_lens=[20000, 4000, 32768, 2, 2049, 9000, 30000, 777], + layer_ids=(_ML4_DS_STEP[35], _ML4_DS_STEP[71], 62), + ), + ), + Cell( + why="ML4 fp8_ds_mla 64-token pages, decode graph batch 16 (captured), 72 calls over " + "the 63-owner pool: stride 4032, each layer at its source's owner row, contexts up " + "to 32768 in the 512-column table, pages at the far end of the 53958-block pool " + "(last row 217558655, 10.1% of 2**31)", + spec=dict( + _ML4_32K_DS64, + graph=True, + far_end=True, + seq_rows=[1] * 16, + kv_lens=[32768, 1, 2048, 2049, 512, 31111, 33, 9216, 16385, 1500, 300, 32767] + + [1] * 4, + layer_ids=_ML4_DS_STEP, + ), + ), + Cell( + why="ML4 fp8_ds_mla 64-token pages, mixed step over the 63-owner pool: 2 context " + "requests and 6 generation rows, layer 35's owner row (the sharers' source) and " + "the last", + spec=dict( + _ML4_32K_DS64, + seq_rows=[20000, 4000] + [1] * 6, + kv_lens=[20000, 4000, 32768, 2, 2049, 9000, 30000, 777], + layer_ids=(_ML4_DS_STEP[35], _ML4_DS_STEP[71], 62), + ), + ), + Cell( + why="ML4 fp8_ds_mla tep8/tp8/dep8 decode graph batch 16 (captured), 72 calls over the " + "63-owner pool: 13 sequences up to 32768 and 3 padding rows (1-token dummies), " + "pages at the far end of a 147456-block pool", + spec=dict( + _ML4_32K_DS_8GPU, + graph=True, + far_end=True, + last_position=True, + seq_rows=[1] * 16, + kv_lens=[32768, 1, 2048, 2049, 512, 31111, 33, 9216, 16385, 1500, 300, 32767] + + [20000] + + [1] * 3, + layer_ids=_ML4_DS_STEP, + ), + ), + Cell( + why="ML4 fp8_ds_mla tep8/tp8/dep8 context step at max_batch_size: 16 requests packed " + "into 32768 rows, layer 35's owner row (the sharers' source) and the last, pages at " + "the far end of a 147456-block pool", + spec=dict( + _ML4_32K_DS_8GPU, + far_end=True, + seq_rows=_ML4_32K_PACKED, + kv_lens=_ML4_32K_PACKED, + layer_ids=(_ML4_DS_STEP[35], 62), + ), + ), + Cell( + why="ML4 fp8_ds_mla tep8/tp8/dep8 V2 table: decode graph batch 16 (captured), 72 " + "calls, over the 1028-column block table the cache manager builds, pages at the far " + "end of a 147456-block pool", + spec=dict( + _ML4_32K_DS_8GPU_V2, + graph=True, + far_end=True, + last_position=True, + seq_rows=[1] * 16, + kv_lens=[32768, 32767, 32737, 32736, 1, 2049, 16385, 9216, 2048, 33, 300, 31111] + + [20000] + + [1] * 3, + layer_ids=_ML4_DS_STEP, + ), + ), + Cell( + why="ML4 fp8_ds_mla dep8 context step at max_batch_size: 16 requests packed into " + "32768 rows, the KV sharers' source owner row 35 (as layer 35 and as layer 71, which " + "reads it) and the last owner row, pages at the far end of a 147456-block pool", + spec=dict( + _ML4_32K_DS_8GPU, + far_end=True, + seq_rows=_ML4_32K_PACKED, + kv_lens=_ML4_32K_PACKED, + layer_ids=(_ML4_DS_STEP[35], _ML4_DS_STEP[71], 62), + ), + ), + Cell( + why="ML4 fp8_ds_mla dep8 V2 table, attention-DP padding: decode graph batch 16 " + "(captured), 72 calls, one live sequence at 32768 behind 15 padding dummies, as a " + "rank with one request replays the largest rank's batch", + spec=dict( + _ML4_32K_DS_8GPU_V2, + graph=True, + far_end=True, + last_position=True, + seq_rows=[1] * 16, + kv_lens=[32768] + [1] * 15, + layer_ids=_ML4_DS_STEP, + ), + ), + Cell( + why="ML4 fp8_ds_mla dep8 V2 table: context step at max_batch_size, 16 requests " + "packed into 32768 rows over the 1028-column table, the sharers' source owner row " + "35, pages at the far end of a 147456-block pool", + spec=dict( + _ML4_32K_DS_8GPU_V2, + far_end=True, + seq_rows=_ML4_32K_PACKED, + kv_lens=_ML4_32K_PACKED, + layer_ids=(_ML4_DS_STEP[35],), + ), + ), + Cell( + why="ML4 1M fp8_ds_mla decode graph batch 1 (captured), 72 calls over the 63-owner " + "pool at stride 2016, each layer at its source's owner row: one sequence at " + "1048576, its newest position 1048575 in column 32767 of the 32772-column table", + spec=dict( + _ML4_1M_DS, + graph=True, + last_position=True, + seq_rows=[1], + kv_lens=[1048576], + layer_ids=_ML4_DS_STEP, + ), + ), + Cell( + why="ML4 1M fp8_ds_mla decode graph batch 2 (captured), 72 calls at stride 2016: " + "contexts of 1 and 1048576, swapped between the capture and the replay", + spec=dict( + _ML4_1M_DS, + graph=True, + last_position=True, + seq_rows=[1, 1], + kv_lens=[1, 1048576], + layer_ids=_ML4_DS_STEP, + ), + ), + Cell( + why="ML4 1M fp8_ds_mla decode graph batch 4 (captured), 72 calls at stride 2016: " + "contexts of 1048576, 1048545 (one position in column 32767), 2049 and 33", + spec=dict( + _ML4_1M_DS, + graph=True, + last_position=True, + seq_rows=[1] * 4, + kv_lens=[1048576, 1048545, 2049, 33], + layer_ids=_ML4_DS_STEP, + ), + ), + Cell( + why="ML4 1M fp8_ds_mla decode graph batch 8 (captured), 72 calls at stride 2016: " + "contexts across 1..1048576, most longer than top-k", + spec=dict( + _ML4_1M_DS, + graph=True, + last_position=True, + seq_rows=[1] * 8, + kv_lens=[1048576, 524288, 131073, 32769, 2047, 64, 700001, 1000000], + layer_ids=_ML4_DS_STEP, + ), + ), + Cell( + why="ML4 1M fp8_ds_mla decode graph batch 16 (captured), 72 calls at stride 2016: " + "13 sequences, 1048576 halved to 65536, then 32769 down to 257, and 3 padding rows " + "(1-token dummies), all 66609 pages the 107917-block pool's top ones", + spec=dict( + _ML4_1M_DS, + graph=True, + far_end=True, + last_position=True, + seq_rows=[1] * 16, + kv_lens=[1048576, 524288, 262144, 131072, 65536, 32769, 32768, 16385, 9217] + + [4097, 2049, 2048, 257, 1, 1, 1], + layer_ids=_ML4_DS_STEP, + ), + ), + Cell( + why="ML4 1M fp8_ds_mla mixed step's generation rows, eager: 13 rows whose requests " + "are table rows 3..15 (req_idx from num_contexts 3), contexts up to 1048576, owner " + "rows 35 (the sharers' source) and 62", + spec=dict( + _ML4_1M_DS, + last_position=True, + first_seq=3, + seq_rows=[1] * 13, + kv_lens=[1048576, 2, 2048, 2049, 4500, 1048575, 1, 700, 32769, 1000000, 131072] + + [129, 1048545], + layer_ids=(35, 62), + ), + ), + Cell( + why="ML4 1M fp8_ds_mla every owner row 0..62 at stride 2016, eager, 16 decode rows " + "with contexts up to 1048576", + spec=dict( + _ML4_1M_DS, + last_position=True, + seq_rows=[1] * 16, + kv_lens=[1048576, 1, 2048, 2049, 512, 1000001, 33, 9216, 32769, 1500, 300] + + [1048575, 524288, 2, 64, 1048545], + layer_ids="owner_rows_0_62", + ), + ), + Cell( + why="ML4 1M fp8_ds_mla decode path, chunk order: the last 32768-row chunk of a " + "1048576-token prompt in one call, each row's selections split between its chunk " + "and the 1015808-token cached prefix, owner rows 0, 35 and 62; every row count " + "1..32768 is a bitwise prefix of it (about 8.5 s)", + spec=dict( + _ML4_1M_DS, + order="chunk", + last_position=True, + seq_rows=[32768], + kv_lens=[1048576], + layer_ids=(0, 35, 62), + prefix_sweep=True, + ), + ), + Cell( + why="ML4 1M fp8_ds_mla decode path, packed chunks: 16384, 8192, 4096 and 4096 rows " + "behind cached prefixes of 1032192, 524288, 0 and 2048 in one 32768-row call, " + "owner row 62, all 49728 pages the pool's top ones", + spec=dict( + _ML4_1M_DS, + far_end=True, + last_position=True, + seq_rows=[16384, 8192, 4096, 4096], + kv_lens=[1048576, 532480, 4096, 6144], + layer_ids=(62,), + ), + ), + Cell( + why="ML4 1M fp8_ds_mla decode path, a 16-row tail beside a chunk: a 1048576-token " + "prompt's last 16 rows and 32752 rows behind a 524304-token prefix in one " + "32768-row call, owner row 35, all 50176 pages the pool's top ones", + spec=dict( + _ML4_1M_DS, + far_end=True, + last_position=True, + seq_rows=[16, 32752], + kv_lens=[1048576, 557056], + layer_ids=(35,), + ), + ), + # The decode path's steps of a 1M prompt before its last chunk: 32768 rows behind a + # cached prefix of 524288 (the first step past 2**19 context KV rows) to 983040; the + # chunk behind 1015808 is the chunk-order cell above. + *( + Cell( + why=f"ML4 1M fp8_ds_mla chunk behind {n - 32768} cached, decode path: a 1M " + f"prompt's 32768-row chunk in one call over {n} KV rows, each row's selections " + "split between its chunk and the cached prefix, owner rows 0, 35 and 62", + spec=dict( + _ML4_1M_DS, + order="chunk", + last_position=True, + seq_rows=[32768], + kv_lens=[n], + layer_ids=(0, 35, 62), + ), + ) + for n in range(557056, 1015809, 32768) + ), + Cell( + why="ML4 1M fp8_ds_mla chunk tail beside a fresh prompt, decode path: a 1040000-token " + "prompt's last 24192 rows behind its 1015808-token cached prefix and a fresh 7-row " + "prompt in one 24199-row call, in chunk order, owner rows 0, 35 and 62", + spec=dict( + _ML4_1M_DS, + order="chunk", + last_position=True, + seq_rows=[24192, 7], + kv_lens=[1040000, 7], + layer_ids=(0, 35, 62), + ), + ), + Cell( + why="ML4 1M tep4 fp8_ds_mla decode graph batch 1 (captured), 72 calls over the " + "63-owner pool at stride 2016, each layer at its source's owner row: one sequence " + "at 1048576, its newest position 1048575 in column 32767 of the 32772-column table", + spec=dict( + _ML4_1M_DS, + graph=True, + last_position=True, + seq_rows=[1], + kv_lens=[1048576], + layer_ids=_ML4_DS_STEP, + ), + ), + Cell( + why="ML4 1M tep4 fp8_ds_mla decode graph batch 2 (captured), 72 calls at stride " + "2016: contexts of 1 and 1048576, swapped between the capture and the replay", + spec=dict( + _ML4_1M_DS, + graph=True, + last_position=True, + seq_rows=[1, 1], + kv_lens=[1, 1048576], + layer_ids=_ML4_DS_STEP, + ), + ), + Cell( + why="ML4 1M tep4 fp8_ds_mla decode graph batch 4 (captured), 72 calls at stride " + "2016: contexts of 1048576, 1048545 (one position in column 32767), 2049 and 33", + spec=dict( + _ML4_1M_DS, + graph=True, + last_position=True, + seq_rows=[1] * 4, + kv_lens=[1048576, 1048545, 2049, 33], + layer_ids=_ML4_DS_STEP, + ), + ), + Cell( + why="ML4 1M tep4 fp8_ds_mla decode graph batch 8 (captured), 72 calls at stride " + "2016: contexts across 1..1048576, most longer than top-k", + spec=dict( + _ML4_1M_DS, + graph=True, + last_position=True, + seq_rows=[1] * 8, + kv_lens=[1048576, 524288, 131073, 32769, 2047, 64, 700001, 1000000], + layer_ids=_ML4_DS_STEP, + ), + ), + Cell( + why="ML4 1M tep4 fp8_ds_mla decode graph batch 16 (captured), 72 calls at stride " + "2016: 13 sequences, 1048576 halved down to 65536, then 32769 down to 257, and 3 " + "padding rows (1-token dummies), all 66609 pages the 107917-block pool's top ones", + spec=dict( + _ML4_1M_DS, + graph=True, + far_end=True, + last_position=True, + seq_rows=[1] * 16, + kv_lens=[1048576, 524288, 262144, 131072, 65536, 32769, 32768, 16385, 9217] + + [4097, 2049, 2048, 257, 1, 1, 1], + layer_ids=_ML4_DS_STEP, + ), + ), + Cell( + why="ML4 1M tep4 fp8_ds_mla mixed step's generation rows, eager: 13 rows whose " + "requests are table rows 3..15 (req_idx from num_contexts 3), contexts up to " + "1048576, owner rows 35 (the sharers' source) and 62", + spec=dict( + _ML4_1M_DS, + last_position=True, + first_seq=3, + seq_rows=[1] * 13, + kv_lens=[1048576, 2, 2048, 2049, 4500, 1048575, 1, 700, 32769, 1000000, 131072] + + [129, 1048545], + layer_ids=(35, 62), + ), + ), + Cell( + why="ML4 1M tep4 fp8_ds_mla every owner row 0..62 at stride 2016, eager, 16 decode " + "rows with contexts up to 1048576", + spec=dict( + _ML4_1M_DS, + last_position=True, + seq_rows=[1] * 16, + kv_lens=[1048576, 1, 2048, 2049, 512, 1000001, 33, 9216, 32769, 1500, 300] + + [1048575, 524288, 2, 64, 1048545], + layer_ids="owner_rows_0_62", + ), + ), + Cell( + why="ML4 1M tep4 fp8_ds_mla decode path, chunk order: the last 32768-row chunk of a " + "1048576-token prompt in one call, each row's selections split between its chunk " + "and the 1015808-token cached prefix, owner rows 0, 35 and 62; every row count " + "1..32768 is a bitwise prefix of it (about 8.5 s)", + spec=dict( + _ML4_1M_DS, + order="chunk", + last_position=True, + seq_rows=[32768], + kv_lens=[1048576], + layer_ids=(0, 35, 62), + prefix_sweep=True, + ), + ), + Cell( + why="ML4 1M tep4 fp8_ds_mla decode path, packed chunks: 16384, 8192, 4096 and 4096 " + "rows behind cached prefixes of 1032192, 524288, 0 and 2048 in one 32768-row call, " + "owner row 62, all 49728 pages the pool's top ones", + spec=dict( + _ML4_1M_DS, + far_end=True, + last_position=True, + seq_rows=[16384, 8192, 4096, 4096], + kv_lens=[1048576, 532480, 4096, 6144], + layer_ids=(62,), + ), + ), + Cell( + why="ML4 1M tep4 fp8_ds_mla decode path, a 16-row tail beside a chunk: a " + "1048576-token prompt's last 16 rows and 32752 rows behind a 524304-token prefix " + "in one 32768-row call, owner row 35, all 50176 pages the pool's top ones", + spec=dict( + _ML4_1M_DS, + far_end=True, + last_position=True, + seq_rows=[16, 32752], + kv_lens=[1048576, 557056], + layer_ids=(35,), + ), + ), + *( + Cell( + why=f"dep4 1M fp8_ds_mla boot graph {b} (captured), 72 calls at stride 2016, each " + "layer at its source's owner row: the warm-up's longest dummy, 1048575 keys, " + f"beside {b - 1} one-key fillers; replayed under attention DP with one live row at " + f"1048576 behind {b - 1} rows of the one padding dummy, then on an idle rank (its " + "attention-DP dummy behind the same padding)", + spec=dict( + _ML4_1M_DEP4_DS, + graph=True, + last_position=True, + boot_capture=True, + seq_rows=[1] * b, + kv_lens=[1048575] + [1] * (b - 1), + layer_ids=_ML4_DS_STEP, + replay_kv_lens=([1048576] + [1] * (b - 1), [1] * b), + replay_pad=(b - 1, b - 1), + ), + ) + for b in (1, 2, 4, 8, 16) + ), + Cell( + why="dep4 1M fp8_ds_mla decode graph 1 (captured), 72 calls at stride 2016, each layer " + "at its source's owner row: one sequence at 1048576, its newest position 1048575 in " + "column 32767 of the 32772-column table", + spec=dict( + _ML4_1M_DEP4_DS, + graph=True, + last_position=True, + seq_rows=[1], + kv_lens=[1048576], + layer_ids=_ML4_DS_STEP, + ), + ), + Cell( + why="dep4 1M fp8_ds_mla decode graph 2 (captured), 72 calls at stride 2016: contexts " + "of 1 and 1048576, swapped between the capture and the replay", + spec=dict( + _ML4_1M_DEP4_DS, + graph=True, + last_position=True, + seq_rows=[1, 1], + kv_lens=[1, 1048576], + layer_ids=_ML4_DS_STEP, + ), + ), + Cell( + why="dep4 1M fp8_ds_mla decode graph 4 (captured), 72 calls at stride 2016: contexts " + "of 1048545 (one position in column 32767), 33, 2049 and 1048544 (column 32767 " + "unused); replayed reversed, then with 3 live rows behind one row of the padding dummy", + spec=dict( + _ML4_1M_DEP4_DS, + graph=True, + last_position=True, + seq_rows=[1] * 4, + kv_lens=[1048545, 33, 2049, 1048544], + layer_ids=_ML4_DS_STEP, + replay_kv_lens=([1048544, 2049, 33, 1048545], [1048545, 2049, 33, 1]), + replay_pad=(0, 1), + ), + ), + Cell( + why="dep4 1M fp8_ds_mla decode graph 8 (captured), 72 calls at stride 2016: contexts " + "across 1..1048576, past each power of two from 32768; replayed reversed, then with 5 " + "live rows behind 3 rows of the padding dummy", + spec=dict( + _ML4_1M_DEP4_DS, + graph=True, + last_position=True, + seq_rows=[1] * 8, + kv_lens=[1048576, 524288, 262145, 131073, 32769, 2047, 64, 1000001], + layer_ids=_ML4_DS_STEP, + replay_kv_lens=( + [1000001, 64, 2047, 32769, 131073, 262145, 524288, 1048576], + [1000001, 64, 2047, 32769, 1048576, 1, 1, 1], + ), + replay_pad=(0, 3), + ), + ), + Cell( + why="dep4 1M fp8_ds_mla decode graph 16 (captured), 72 calls at stride 2016: 13 " + "sequences, 1048576 halved to 65536, then 32769 down to 257, behind 3 rows of the " + "padding dummy, all 66609 pages the 107917-block pool's top ones; replayed on the 13 " + "reversed, then with 9 live rows behind 7 rows of the padding dummy", + spec=dict( + _ML4_1M_DEP4_DS, + graph=True, + far_end=True, + last_position=True, + seq_rows=[1] * 16, + kv_lens=[1048576, 524288, 262144, 131072, 65536, 32769, 32768, 16385, 9217] + + [4097, 2049, 2048, 257, 1, 1, 1], + pad=3, + layer_ids=_ML4_DS_STEP, + replay_kv_lens=( + [257, 2048, 2049, 4097, 9217, 16385, 32768, 32769, 65536, 131072, 262144] + + [524288, 1048576, 1, 1, 1], + [1048576, 65536, 2049, 257, 524288, 32769, 9217, 131072, 2048] + [1] * 7, + ), + replay_pad=(3, 7), + ), + ), + Cell( + why="dep4 1M fp8_ds_mla mixed step's generation rows, eager: 13 rows whose requests " + "are table rows 3..15 (req_idx from num_contexts 3), contexts up to 1048576, owner " + "rows 35 (the sharers' source) and 62", + spec=dict( + _ML4_1M_DEP4_DS, + last_position=True, + first_seq=3, + seq_rows=[1] * 13, + kv_lens=[1048576, 2, 2048, 2049, 4500, 1048575, 1, 700, 32769, 1000000, 131072] + + [129, 1048545], + layer_ids=(35, 62), + ), + ), + Cell( + why="dep4 1M fp8_ds_mla every owner row 0..62 at stride 2016, eager, 16 decode rows " + "with contexts up to 1048576", + spec=dict( + _ML4_1M_DEP4_DS, + last_position=True, + seq_rows=[1] * 16, + kv_lens=[1048576, 1, 2048, 2049, 512, 1000001, 33, 9216, 32769, 1500, 300] + + [1048575, 524288, 2, 64, 1048545], + layer_ids="owner_rows_0_62", + ), + ), + Cell( + why="dep4 1M fp8_ds_mla ctx rows, chunk order: the last 32768-row chunk of a " + "1048576-token prompt in one call, each row's selections split between its chunk and " + "the 1015808-token cached prefix, owner rows 0, 35 and 62; every row count 1..32768 " + "is a bitwise prefix of it", + spec=dict( + _ML4_1M_DEP4_DS, + order="chunk", + last_position=True, + seq_rows=[32768], + kv_lens=[1048576], + layer_ids=(0, 35, 62), + prefix_sweep=True, + ), + ), + Cell( + why="dep4 1M fp8_ds_mla ctx rows, packed chunks: 16384, 8192, 4096 and 4096 rows " + "behind cached prefixes of 1032192, 524288, 0 and 2048 in one 32768-row call, owner " + "row 62, all 49728 pages the pool's top ones", + spec=dict( + _ML4_1M_DEP4_DS, + far_end=True, + last_position=True, + seq_rows=[16384, 8192, 4096, 4096], + kv_lens=[1048576, 532480, 4096, 6144], + layer_ids=(62,), + ), + ), + Cell( + why="dep4 1M fp8_ds_mla ctx rows, 16 chunk tails of 16 rows, each behind a 65520-token " + "cached prefix (1048576 context KV, past 2**19), owner row 35, all 32768 pages the " + "pool's top ones", + spec=dict( + _ML4_1M_DEP4_DS, + far_end=True, + last_position=True, + seq_rows=[16] * 16, + kv_lens=[65536] * 16, + layer_ids=(35,), + ), + ), + Cell( + why="dep4 1M fp8_ds_mla ctx rows, a 16-row tail beside a chunk: a 1048576-token " + "prompt's last 16 rows and 32752 rows behind a 524304-token prefix in one 32768-row " + "call, owner row 35, all 50176 pages the pool's top ones", + spec=dict( + _ML4_1M_DEP4_DS, + far_end=True, + last_position=True, + seq_rows=[16, 32752], + kv_lens=[1048576, 557056], + layer_ids=(35,), + ), + ), + Cell( + why="ML4 1M tep8 fp8_ds_mla decode graph batch 1 (captured), 72 calls over the " + "63-owner pool at stride 2016, each layer at its source's owner row: one sequence " + "at 1048576, its newest position 1048575 in column 32767 of the 32772-column table", + spec=dict( + _ML4_1M_TEP8_DS, + graph=True, + last_position=True, + seq_rows=[1], + kv_lens=[1048576], + layer_ids=_ML4_DS_STEP, + ), + ), + Cell( + why="ML4 1M tep8 fp8_ds_mla decode graph batch 2 (captured), 72 calls at stride " + "2016: contexts of 1 and 1048576, swapped between the capture and the replay", + spec=dict( + _ML4_1M_TEP8_DS, + graph=True, + last_position=True, + seq_rows=[1, 1], + kv_lens=[1, 1048576], + layer_ids=_ML4_DS_STEP, + ), + ), + Cell( + why="ML4 1M tep8 fp8_ds_mla decode graph batch 4 (captured), 72 calls at stride " + "2016: contexts of 1048576, 1048545 (one position in column 32767), 2049 and 33", + spec=dict( + _ML4_1M_TEP8_DS, + graph=True, + last_position=True, + seq_rows=[1] * 4, + kv_lens=[1048576, 1048545, 2049, 33], + layer_ids=_ML4_DS_STEP, + ), + ), + Cell( + why="ML4 1M tep8 fp8_ds_mla decode graph batch 8 (captured), 72 calls at stride " + "2016: contexts across 1..1048576, most longer than top-k", + spec=dict( + _ML4_1M_TEP8_DS, + graph=True, + last_position=True, + seq_rows=[1] * 8, + kv_lens=[1048576, 524288, 131073, 32769, 2047, 64, 700001, 1000000], + layer_ids=_ML4_DS_STEP, + ), + ), + Cell( + why="ML4 1M tep8 fp8_ds_mla decode graph batch 16 (captured), 72 calls at stride " + "2016: 13 sequences, 1048576 halved down to 65536, then 32769 down to 257, and 3 " + "padding rows (1-token dummies), all 66609 pages the 147456-block pool's top ones", + spec=dict( + _ML4_1M_TEP8_DS, + graph=True, + far_end=True, + last_position=True, + seq_rows=[1] * 16, + kv_lens=[1048576, 524288, 262144, 131072, 65536, 32769, 32768, 16385, 9217] + + [4097, 2049, 2048, 257, 1, 1, 1], + layer_ids=_ML4_DS_STEP, + ), + ), + Cell( + why="ML4 1M tep8 fp8_ds_mla mixed step's generation rows, eager: 13 rows whose " + "requests are table rows 3..15 (req_idx from num_contexts 3), contexts up to " + "1048576, owner rows 35 (the sharers' source) and 62", + spec=dict( + _ML4_1M_TEP8_DS, + last_position=True, + first_seq=3, + seq_rows=[1] * 13, + kv_lens=[1048576, 2, 2048, 2049, 4500, 1048575, 1, 700, 32769, 1000000, 131072] + + [129, 1048545], + layer_ids=(35, 62), + ), + ), + Cell( + why="ML4 1M tep8 fp8_ds_mla every owner row 0..62 at stride 2016, eager, 16 decode " + "rows with contexts up to 1048576, pages cycled through a 147456-block pool", + spec=dict( + _ML4_1M_TEP8_DS, + last_position=True, + seq_rows=[1] * 16, + kv_lens=[1048576, 1, 2048, 2049, 512, 1000001, 33, 9216, 32769, 1500, 300] + + [1048575, 524288, 2, 64, 1048545], + layer_ids="owner_rows_0_62", + ), + ), + Cell( + why="ML4 1M tep8 fp8_ds_mla decode path, chunk order: the last 32768-row chunk of a " + "1048576-token prompt in one call, each row's selections split between its chunk " + "and the 1015808-token cached prefix, owner rows 0, 35 and 62; every row count " + "1..32768 is a bitwise prefix of it (about 8.5 s)", + spec=dict( + _ML4_1M_TEP8_DS, + order="chunk", + last_position=True, + seq_rows=[32768], + kv_lens=[1048576], + layer_ids=(0, 35, 62), + prefix_sweep=True, + ), + ), + Cell( + why="ML4 1M tep8 fp8_ds_mla decode path, packed chunks: 16384, 8192, 4096 and 4096 " + "rows behind cached prefixes of 1032192, 524288, 0 and 2048 in one 32768-row call, " + "owner row 62, all 49728 pages the 147456-block pool's top ones", + spec=dict( + _ML4_1M_TEP8_DS, + far_end=True, + last_position=True, + seq_rows=[16384, 8192, 4096, 4096], + kv_lens=[1048576, 532480, 4096, 6144], + layer_ids=(62,), + ), + ), + Cell( + why="ML4 1M tep8 fp8_ds_mla decode path, a 16-row tail beside a chunk: a " + "1048576-token prompt's last 16 rows and 32752 rows behind a 524304-token prefix " + "in one 32768-row call, owner row 35, all 50176 pages the 147456-block pool's top " + "ones", + spec=dict( + _ML4_1M_TEP8_DS, + far_end=True, + last_position=True, + seq_rows=[16, 32752], + kv_lens=[1048576, 557056], + layer_ids=(35,), + ), + ), + ) + + note = """ + Metadata consumed: none. The op is a pure function of its arguments. + + The kernel launches one block row per `req_id` entry and writes + `num_topk_tokens` columns at the output's row stride, which is + `token_indices.size(1)`. Both counts are taken on trust; see `is_valid`. + Rows are independent: an m-row call is bitwise the leading m rows of a + longer one, driven at every m in 1..9216 and three times at every m in + 1..32768: the second time on the last chunk of a 1048576-token prompt, the + third on that chunk in chunk order at stride 2016. + + Preconditions, each refused by `is_valid` because the op accepts the input + and answers wrongly without raising (measured, except where marked read): + - `num_topk_tokens == token_indices.size(1)`: narrower leaves each row's + trailing columns unwritten; wider spills into the next row and past the + output (read from the kernel, not driven). + - `req_id.size(0) == token_indices.size(0)`: shorter leaves the trailing + output rows unwritten; longer reads and writes past them (read from the + kernel, not driven). + - every `req_id` value is a row of `block_table`: the kernel reads + `block_table[req_id[i]]` unchecked, so a request index past a sliced table + reads the parent tensor's next row, and -1 the row before the slice. + - `0 <= layer_id < stride_factor / block_size`: outside the page the row + lands in a neighbouring page (layer 72 at stride 2304 is the next page's + layer 0; -1 is the previous page's last layer). + - every pool row fits int32 (Hard limits). + The value checks read `req_id` and `block_table` on the host, which a CUDA + graph capture forbids; a captured call is checked for shapes only, so check + the data a replay reads eagerly. + + Mistral Large 4 passes `stride_factor = 72 * 32 = 2304` (a pool that + interleaves all 72 layers' pages) and, as `layer_id`, the pool row of the + layer whose latent it reads: a KV-sharing layer (39, 43, ..., 71) passes 35. + Its FP8 target serves 9216 tokens (a 288-column block table); its NVFP4 + targets serve max_seq_len = max_num_tokens = 32768 with no chunked prefill + (a 1024-column table, up to 32768 rows per call); on the fp8_ds_mla pool + (`MistralLarge4CacheManager`) the stride is 63 x 32 = 2016 and `layer_id` the + source's owner row, and a tep8, tp8 or dep8 rank's pool holds up to 147456 + pages. The cache manager pads the block table to 1028 columns at 32768, + which only the "V2 table" cells drive; the columns past 1023 are never read + below max_seq_len. Under attention DP (dep8) every rank replays the largest + rank's graph batch, its own requests followed by 1-token padding dummies + (`CUDAGraphRunner._get_padded_batch`, `_torch/pyexecutor/cuda_graph_runner.py`; + read, not driven: the engine repeats one dummy request, the cells give each + dummy a page of its own); rows are independent, and one live row behind 15 + dummies is a cell. + Its NVFP4 tep4 deployment also serves max_seq_len 1048576 with chunked + prefill: steps of up to 32768 rows, + chunks behind cached prefixes, positions to 1048575, and the engine's + 32772-column table (cells at 32768 and 32772 columns). + Its NVFP4 tp4 deployment makes the same calls at the same envelope. In index + mode a chunk step's context rows reach the op after the q-split's + all-gather has filled every row of the selection buffer, so one call covers + all of the step's context rows. When the indexer is skipped (dense causal), + the context rows read the block table instead, and only the generation + rows behind them reach the op. The tp4 cells add every layer id 0..71 in + one capture, a far-end batch-16 graph whose pages fit the pool, a 32768-row + chunk whose rows select from both their own chunk and the cached prefix, + and four chunks packed behind prefixes up to 1032192. Its NVFP4 dep4 + deployment serves max_seq_len 1048576 with chunked prefill too: steps of up to + 32768 rows per rank, chunks behind cached prefixes, positions to 1048575, + and the DSA metadata's 16 x 32772 table (`block_table`, `max_num_sequences` + rows by the V2 manager's `max_blocks_per_seq`). The metadata writes each + step's rows only up to the step's widest request, -1 past a request's own + pages, and the columns past that keep earlier steps' values (read in the + DSA metadata's `prepare_for_indexer_k_cache`, not driven), which the dep4 + cells model as stale page ids. The boot captures every decode graph on a + dummy of 1048575 keys beside 1-key padding rows + (`_capture_generation_cuda_graphs` with `add_longest_dummy_request`); the + dep4 cells capture those shapes and replay them on live data. + + On the fp8_ds_mla latent pool (`MistralLarge4CacheManager`: pages for the + 63 KV owners only) the tp4, tep4 and tep8 deployments pass `stride_factor = + len(kv_owner_layers) * tokens_per_block = 63 * 32 = 2016` and, as + `layer_id`, `latent_pool_row(kv_source)`: 0..38 for layers 0..38, 35 for + each sharer 39, 43, ..., 71, and 39..62 for the owners between (cells: + `_ML4_DS_STEP`). Its block table is `md.block_table`, 16 rows by the + engine's 32772 columns at max_seq_len 1048576 (1028 at 32768); positions + reach 1048575, column 32767; tep8's page ids reach 147455 (the eight-rank + bound; its 1M boot held 114326 blocks a rank). The "ML4 1M fp8_ds_mla" + (tp4), "ML4 1M tep4 fp8_ds_mla" and "ML4 1M tep8 fp8_ds_mla" cells certify + three call shapes: + - `DecodeTarget._attention`: the step's 72 calls, captured at the padded + batches 1/2/4/8/16 (cells at 1, 2, 4, 8 and 16 rows, replayed on new + data); + - `PrefillTarget._attention`: a mixed step's 1..16 generation rows, eager, + `req_idx[tc:]`, whose request indices start at `num_contexts`; + - the context rows of a step whose context KV (cached plus new over its + context requests) exceeds 2**19 rows, when `PrefillTarget._attn_ctx` + attends them straight off the pool rather than through a bf16 + workspace: every context row of the step in one call, up to 32768 rows + (in index mode after the q-split's all-gather has filled the selection + buffer), each chunk row selecting across its request's cached prefix + and its own chunk, several requests packed. Each step a 1M prompt + chunked at 32768 takes on this path is a tp4 cell: its chunks behind + cached prefixes of 524288, 557056, ..., 983040 and 1015808, and a + 1040000-token prompt's last 24192 rows behind 1015808 packed with a + fresh 7-row prompt. Context rows that address a bf16 workspace add + their request's workspace offset instead and do not call the op. + The dep4 deployment passes the same stride and owner rows over its own + `md.block_table` (16 x 32772); the "dep4 1M fp8_ds_mla" cells certify its + three call shapes: + - `DecodeTarget._attention`: the step's 72 calls, captured at the padded + batches 1/2/4/8/16 on the boot's warm-up batch (the 1048575-key longest + dummy beside one-key fillers) and on live batches, and replayed on new + data; eager (a peer rank prefilling) at any count 1..16, which the row + independence above covers; + - `PrefillTarget._attention`: a mixed step's 1..15 generation rows, eager, + `req_idx[tc:]`, whose request indices start at `num_contexts`; + - the context rows of a step whose context KV (cached plus new over its + context requests) exceeds 2**19 rows, attended straight off the pool + rather than through a bf16 workspace: every context row of the step in + one call, up to 32768 rows, each chunk row selecting across its + request's cached prefix and its own chunk, several requests packed. + Context rows that address a bf16 workspace add their request's + workspace offset instead and do not call the op. + Under attention DP every rank replays the largest rank's graph batch: the + rank's live rows first, then one CUDA-graph padding dummy (one key, + position 0) repeated to the batch (`CUDAGraphRunner._get_padded_batch` + appends the one request object; read, not driven in an engine), and an + idle rank's attention-DP dummy (one key) ahead of that padding. Each + padding row is a batch row of its own, so its request index is its own and + its table row holds the dummy's one page: every padding row maps to the + same pool rows. The cells replay that behind one, three, five, nine and + thirteen live rows and on an idle rank, every row bitwise the reference. + The table must be passed whole: a table sliced narrower than a request's + pages maps every position past its last column to -1 without raising (read + from the kernel, not driven), the same -1 the top-k padding carries. + The binding calls `.contiguous()` on each input before the launch + (`convertReqIndexToGlobal`, `cpp/tensorrt_llm/thop/convertReqIndexToGlobalOp.cpp`; + read, not driven): only contiguous inputs were driven. The output is + allocated per call: under CUDA-graph capture it comes from the graph's pool + and each replay rewrites it. + + Hard limits (sm_103) + pool row block * stride_factor + layer_id * block_size + p % block_size, + int32. Boundary 2**31: at stride 2304, block 932067 from layer 40, + every layer from block 932068. Probed at block 932067: layer 39 + is exact up to 2**31 - 1; layer 40 wraps to -2**31, and + `is_valid` refuses it. The FP8 envelope's 18432-block pool + reaches row 42467327, the NVFP4 target's 107917-block bound row + 248640767 (11.6% of 2**31), the tep8, tp8 and dep8 deployments' + 147456-block bound on the 2016-stride fp8_ds_mla pool row 297271295 + (13.8% of 2**31); the far-end cells hand requests each pool's top + pages: exact. Block 932067 of a 72-layer fp8 latent + pool sits past 1236949843968 bytes, 4.16 times a GB300's + 297020948480. The tp4 boot pools hold 74183 slots at 32768 + (2373856 tokens) and 73500 slots at 1048576 (2352000 tokens): + the top row is 74182 * 2304 + 71 * 32 + 31 = 170917631, 7.96% + of 2**31, which is inside the 107917-block cells. + At the 63 owners' stride 2016: block 1065220 from owner row 4, + every row from block 1065221. Probed at block 1065220: row 3 is + exact up to 2**31 - 1; row 4 wraps to -2**31, and `is_valid` + refuses it. The fp8_ds_mla tp4 boot at 1048576 held 69011 + blocks, top row 69010 * 2016 + 62 * 32 + 31 = 139126175 (6.48% of + 2**31), the tep4 boot 72757 blocks (2328224 tokens), top row + 72756 * 2016 + 62 * 32 + 31 = 146678111 (6.83%), the dep4 boot + 64633 blocks per rank, top row 64632 * 2016 + 62 * 32 + 31 = + 130300127 (6.07%), the tep8 boot 114326 blocks (3658432 + tokens), top row 114325 * 2016 + 62 * 32 + 31 = 230481215 + (10.73%); the 107917-block cells reach 107916 * 2016 + 62 * 32 + + 31 = 217560671 (10.13%) and the 147456-block cells 147455 * + 2016 + 62 * 32 + 31 = 297271295 (13.84%), and their far-end + cells hand requests the pool's top pages: exact. + fp32 no index passes through fp32. Every ML4 cell's output names pool + rows past 2**24 + 1 (from block 7282 at stride 2304, 8323 at + stride 2016), which its test asserts: exact. + offsets row * stride in int64 for `token_indices`, `block_table` and + the output: no 32-bit product (one would wrap at 2**31 / 2048 = + 1048576 rows). 32768 x 2048 = 2**26 elements in the envelope; + the table offset req * 32772 + column is at most 524347 at 16 + rows. + position p and p // block_size in int32, no product: boundary 2**31, + not reached, not probed. Positions 0..1048575 in the 1M + envelope, table columns 0..32767 of the engine's 32772; the + table's row stride enters in int64. Every position is below + 2**24 and none passes through fp32. + rows one grid row per `req_id` entry: 2**31 - 1 rows; 32768 in the + envelope. + + Not in any cell at stride 2016: a pool past 147456 blocks (past 107917 at + tp4, tep4 and dep4), and a captured call of the context-row shape (those + calls run eager). + + The grouped variant (`convert_req_index_to_global_grouped`, one launch for + several layers) is a different op and is not covered here. + """ + + def __call__( + self, + req_id: torch.Tensor, + block_table: torch.Tensor, + token_indices: torch.Tensor, + block_size: int, + num_topk_tokens: int, + stride_factor: int, + layer_id: int, + ) -> torch.Tensor: + return torch.ops.trtllm.convert_req_index_to_global( + req_id, + block_table, + token_indices, + block_size, + num_topk_tokens, + stride_factor, + layer_id, + ) + + def reference( + self, + req_id: torch.Tensor, + block_table: torch.Tensor, + token_indices: torch.Tensor, + block_size: int, + num_topk_tokens: int, + stride_factor: int, + layer_id: int, + ) -> torch.Tensor: + """The kernel's integer map, in int64 so it cannot wrap. + + `num_topk_tokens` is not read: the reference covers every column of + `token_indices`, which `is_valid` requires to be the same count. + """ + tok = token_indices.long() + max_blocks = block_table.shape[1] + block = torch.div(tok, block_size, rounding_mode="floor") + in_block = tok - block * block_size + layer_id * block_size + valid = (tok >= 0) & (block < max_blocks) + rows = req_id.long().unsqueeze(1).expand_as(tok) + base = block_table.long()[rows, block.clamp(0, max_blocks - 1)] + valid &= base >= 0 + out = torch.where(valid, base * stride_factor + in_block, torch.full_like(tok, -1)) + return out.to(torch.int32) + + def is_valid( + self, + req_id: torch.Tensor, + block_table: torch.Tensor, + token_indices: torch.Tensor, + block_size: int, + num_topk_tokens: int, + stride_factor: int, + layer_id: int, + ) -> None: + # The output's row stride is `token_indices.size(1)`, but the kernel + # walks `num_topk_tokens` columns. Wider spills each row's surplus into + # the next row and past the end of the output; narrower leaves the + # trailing columns uninitialized. Neither raises. + assert num_topk_tokens == token_indices.shape[1], ( + f"num_topk_tokens must equal token_indices.size(1); got {num_topk_tokens} " + f"against a {tuple(token_indices.shape)} operand" + ) + # The row count is `req_id.size(0)`, not `token_indices.size(0)`. A + # longer `req_id` reads past `token_indices` and writes past the output; + # a shorter one leaves the extra output rows uninitialized. + assert req_id.shape[0] == token_indices.shape[0], ( + f"req_id must hold exactly one entry per row of token_indices; got " + f"{req_id.shape[0]} for {token_indices.shape[0]} rows" + ) + # `layer_id * block_size` is added inside the page unchecked: a layer + # outside it addresses a neighbouring page's rows. + assert 0 <= layer_id and (layer_id + 1) * block_size <= stride_factor, ( + f"layer_id {layer_id} is outside a page of {stride_factor // block_size} layers " + f"(stride_factor {stride_factor}, block_size {block_size})" + ) + # The rest reads values off the device, which a graph capture forbids. + if torch.cuda.is_current_stream_capturing() or not req_id.numel(): + return + reads = [req_id.min(), req_id.max()] + if block_table.numel(): + reads.append(block_table.max()) + lo, hi, *top_block = torch.stack(reads).tolist() + # `block_table[req_id[i]]` is read unchecked: past a sliced table it + # reads the parent tensor's rows. + assert 0 <= lo and hi < block_table.shape[0], ( + f"req_id values {lo}..{hi} must be rows of block_table, which has " + f"{block_table.shape[0]} rows" + ) + # int32 arithmetic on the device: a pool row past 2**31 - 1 wraps silently. + if top_block: + top = top_block[0] * stride_factor + (layer_id + 1) * block_size + assert top <= 2**31, ( + f"pool row {top - 1} does not fit int32; the kernel wraps it silently" + ) + + +convert_req_index_to_global = ConvertReqIndexToGlobal() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/cute_dsl_ds_mla_sparse_decode.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/cute_dsl_ds_mla_sparse_decode.py new file mode 100644 index 000000000000..e33f4c9633e2 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/cute_dsl_ds_mla_sparse_decode.py @@ -0,0 +1,634 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Sparse MQA of one query row per sequence over an fp8_ds_mla pool, 8 or 16 query heads +unpadded: a CuTe DSL kernel that computes FlashMLA's sm100f sparse fp8 decode.""" + +from __future__ import annotations + +from collections.abc import Callable, Mapping +from dataclasses import dataclass + +import torch + +from tensorrt_llm._torch.modeling_v2.catalog._op import Arch, Cell, OpWrapper, true_fp32_matmul +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.kernels import ( + _ds_mla_sparse_decode_kernel as _kernel, +) + +ROW_BYTES = _kernel.ROW_BYTES +D_NOPE = _kernel.D_NOPE +D_QK = _kernel.D_QK +TOPK = _kernel.TOPK +TILE = 128 +#: The real head counts the kernel runs. +HEADS = _kernel.SUPPORTED_HEADS +#: The split-K counts (CTAs per row) a batch can select. +SPLITS = _kernel.SUPPORTED_SPLITS +#: Global-workspace variants, separate from the cluster functions in `compiled.fns`. +GLOBAL_SPLITS = (16,) +#: The most rows the global variant runs. A CTA holds a whole SM, so a call also needs its +#: rows x GLOBAL_SPLITS CTAs to fit in one wave; past it the cluster split is faster. +GLOBAL_MAX_ROWS = 8 + +#: The tp4 / tep4 (16 heads) and tp8 / tep8 (8 heads) targets at their 1M config (max_seq_len +#: 1048576, chunked prefill, 16 sequences): top-2048 selections over keys up to 1048576 in the +#: fp8_ds_mla pool's 32-token pages of 63 KV-owner rows per block. +_1M = dict(topk=2048, kv_len=1048576, page_size=32, owners=63) +#: The 1M pools by block count: the delivered 1M tp4 boot's 74183 (above the 72757 a 1M tep4 +#: rank's boot measured), the far end 107917 (the far cells of the 1M attention entries), and +#: the 147456-block bound (a 1M tep8 rank held 114326). Every one puts slots past 2**31 bytes. +_BOOT_BLOCKS, _FAR_BLOCKS, _BOUND_BLOCKS = 74183, 107917, 147456 + + +@dataclass(frozen=True) +class CompiledDsMlaSparseDecode: + """Every split variant a batch can select, for one head count on one device, and the SM + count the selection reads.""" + + device_index: int + num_heads: int + num_sms: int + fns: Mapping[int, Callable] + global_fns: tuple[Callable, Callable] | None = None + + +def build(device_id: int, *, num_heads: int) -> CompiledDsMlaSparseDecode: + """Compile the cluster variants and the supported low-row global variant. + + The compiled object owns code only; invocation scratch belongs to PyTorch's + allocator and CUDA graph pools. + """ + with torch.cuda.device(device_id): + fns = {ns: _kernel.compile_variant(num_heads, ns, device_id) for ns in SPLITS} + num_sms = torch.cuda.get_device_properties(device_id).multi_processor_count + global_fns = None + if torch.cuda.get_device_capability(device_id) == (10, 7): + global_fns = _kernel._compile_global_variant(num_heads, GLOBAL_SPLITS[0], device_id) + return CompiledDsMlaSparseDecode( + device_index=device_id, num_heads=num_heads, num_sms=num_sms, fns=fns, global_fns=global_fns + ) + + +def _runs_global(rows: int, compiled: CompiledDsMlaSparseDecode) -> bool: + """Whether a call with `rows` rows runs the global variant.""" + return ( + compiled.global_fns is not None + and 1 <= rows <= GLOBAL_MAX_ROWS + and rows * GLOBAL_SPLITS[0] <= compiled.num_sms + ) + + +def splits_for(rows: int, compiled: CompiledDsMlaSparseDecode) -> int: + """The split-K count a call with `rows` rows runs.""" + if _runs_global(rows, compiled): + return GLOBAL_SPLITS[0] + return _kernel.pick_splits(rows, compiled.num_sms) + + +class CuteDslDsMlaSparseDecode(OpWrapper): + """For every query row r and head h < `out.shape[1]`:: + + K[j] = [e4m3(row_j[:512]) * tile_scale_j (per 128) | bf16(row_j[528:656])] + p = softmax_j(softmax_scale * q[r, h] . K[j]), j in indices[r] (-1 skipped) + out[r, h] = sum_j p[j] * K[j][:512] + + `kv_cache`: uint8 `[pages, page_size, 1, 656]`, contiguous; a slot is + `page * page_size + offset`. `indices`: int32 `[rows, 2048]` slots, -1 padded, + contiguous. `q`: bf16 `[rows, h_q, 576]`, contiguous, the real heads first: h_q is the + caller's scratch width (the heads themselves, or a padded scratch another kernel shares) + and `out` (bf16 `[rows, heads, 512]`, contiguous) names how many heads are real, 8 or 16. + `compiled`: `build(device, num_heads=heads)`. Writes `out` and returns it. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # spec: `rows` sequences of one query row, `heads` real heads read from a `h_q`-head + # query, `topk` slots over `kv_len`-token sequences in a pool of `pages` pages of + # `page_size` x `owners` rows; `layers` calls; `captured` replays from a CUDA graph; + # `holes` scatters -1 through short rows; `nope` zeroes the rope halves. + # The "1M" cells build a pool of `blocks` blocks on the device, every byte 0xFF (e4m3, + # fp32 and bf16 NaN) but one owner row of the pages 16 decode sequences hold (keys + # `kv_len`, `kv_len` - 31, ... down to 1); a row r is a decode row of sequence r % 16; + # `far` puts those pages at the pool's top and the first sequence's last key on its top + # block; `graph_batches` captures each batch and replays it on new queries and slots; + # `q_offset` makes q and out tail views from that row of a 32768-row step scratch. + CELLS: tuple[Cell, ...] = ( + tuple( + Cell( + why=f"tep4/tp4 decode batch {b}: 16 heads, graph replay, 3 layers", + spec=dict( + rows=b, + heads=16, + h_q=16, + topk=2048, + kv_len=32768, + page_size=32, + owners=63, + pages=1100 * b // 16 + 64, + layers=3, + captured=True, + ), + ) + for b in (1, 2, 4, 8, 16) + ) + + tuple( + Cell( + why=f"tp4 decode batch {b}: 16 heads, graph replay ({s} CTAs per row)", + spec=dict( + rows=b, + heads=16, + h_q=16, + topk=2048, + kv_len=32768, + page_size=32, + owners=63, + pages=1100 * b // 16 + 64, + layers=2, + captured=True, + ), + ) + for b, s in ((32, 4), (64, 2), (128, 1), (256, 1)) + ) + + tuple( + Cell( + why=f"tep8/tp8 decode batch {b}: 8 heads, graph replay, 3 layers", + spec=dict( + rows=b, + heads=8, + h_q=8, + topk=2048, + kv_len=32768, + page_size=32, + owners=63, + pages=1100 * b // 16 + 64, + layers=3, + captured=True, + ), + ) + for b in (1, 2, 4, 8, 16, 64, 256) + ) + + ( + Cell( + why="mixed step: generation rows behind context rows read 16 of the " + "prefill scratch's 64 heads", + spec=dict( + rows=24, + heads=16, + h_q=64, + topk=2048, + kv_len=9000, + page_size=32, + owners=63, + pages=512, + layers=1, + ), + ), + Cell( + why="mixed step at world 8: 8 of the prefill scratch's 64 heads", + spec=dict( + rows=5, + heads=8, + h_q=64, + topk=2048, + kv_len=9000, + page_size=32, + owners=63, + pages=512, + layers=1, + ), + ), + Cell( + why="short sequences: -1 tails and scattered holes", + spec=dict( + rows=16, + heads=16, + h_q=16, + topk=2048, + kv_len=700, + page_size=32, + owners=63, + pages=512, + layers=1, + holes=True, + ), + ), + Cell( + why="short sequences at 8 heads: -1 tails and scattered holes", + spec=dict( + rows=40, + heads=8, + h_q=8, + topk=2048, + kv_len=700, + page_size=32, + owners=63, + pages=512, + layers=1, + holes=True, + ), + ), + Cell( + why="NoPE layer: zero query and key rope halves", + spec=dict( + rows=16, + heads=16, + h_q=16, + topk=2048, + kv_len=9000, + page_size=32, + owners=63, + pages=512, + layers=1, + nope=True, + ), + ), + ) + # The tep4/tp4 and tep8/tp8 cells at 64-token pages: the same rows, heads, top-k, + # lengths and graph replays, half the pages for the same token capacity. + + tuple( + Cell( + why=f"64-token pages: decode batch {b}: {h} heads, graph replay, 3 layers", + spec=dict( + rows=b, + heads=h, + h_q=h, + topk=2048, + kv_len=32768, + page_size=64, + owners=63, + pages=550 * b // 16 + 32, + layers=3, + captured=True, + ), + ) + for h in (16, 8) + for b in (1, 4, 16, 64) + ) + + ( + Cell( + why="64-token pages: short sequences: -1 tails and scattered holes", + spec=dict( + rows=16, + heads=16, + h_q=16, + topk=2048, + kv_len=700, + page_size=64, + owners=63, + pages=256, + layers=1, + holes=True, + ), + ), + Cell( + why="64-token pages: NoPE layer: zero query and key rope halves", + spec=dict( + rows=16, + heads=16, + h_q=16, + topk=2048, + kv_len=9000, + page_size=64, + owners=63, + pages=256, + layers=1, + nope=True, + ), + ), + ) + ) + # The 1M config's call sites: DecodeTarget's generation rows (captured at every batch the + # LLM API's default list reaches, 1..16) at 16 and 8 heads on the boot, far and bound pools, + # the larger batches every split variant runs, PrefillTarget._attn_gen's generation rows + # behind a chunk (16 / 8 of the prefill scratch's 64 heads, tail views of the step's + # scratch), a NoPE layer's shared row, and the boot captures' short keys. + CELLS += ( + Cell( + why="1M tp4/tep4 decode graphs at every batch 1..16, boot pool (large): 16 heads, " + "74183 blocks (98.1 GB), rows of 1048576 keys down to 1, the pool's last slot read, " + "3 layers", + spec=dict( + _1M, + rows=16, + heads=16, + h_q=16, + blocks=_BOOT_BLOCKS, + layers=3, + captured=True, + graph_batches=tuple(range(1, 17)), + far=True, + large=True, + ), + ), + Cell( + why="1M tp8/tep8 decode graphs at every batch 1..16, bound pool (large): 8 heads, " + "147456 blocks (195 GB), the top slot read, 3 layers", + spec=dict( + _1M, + rows=16, + heads=8, + h_q=8, + blocks=_BOUND_BLOCKS, + layers=3, + captured=True, + graph_batches=tuple(range(1, 17)), + far=True, + large=True, + ), + ), + Cell( + why="1M tp4/tep4 decode graph 16, far pool (large): 107917 blocks (142.7 GB), the " + "top slot read at byte 1.427e11, 3 layers", + spec=dict( + _1M, + rows=16, + heads=16, + h_q=16, + blocks=_FAR_BLOCKS, + layers=3, + captured=True, + far=True, + large=True, + ), + ), + Cell( + why="1M decode batches 32, 64, 128 and 256 at 16 heads (every split variant), keys to " + "1048576 on the boot pool (large), captured", + spec=dict( + _1M, + rows=256, + heads=16, + h_q=16, + blocks=_BOOT_BLOCKS, + layers=2, + captured=True, + graph_batches=(32, 64, 128, 256), + far=True, + large=True, + ), + ), + Cell( + why="1M decode batches 32, 64 and 256 at 8 heads, keys to 1048576 on the boot pool " + "(large), captured", + spec=dict( + _1M, + rows=256, + heads=8, + h_q=8, + blocks=_BOOT_BLOCKS, + layers=2, + captured=True, + graph_batches=(32, 64, 256), + far=True, + large=True, + ), + ), + Cell( + why="1M tp4/tep4 mixed step's 13 generation rows (large): 16 of the prefill scratch's " + "64 heads, q and out tail views from row 32755 of a 32768-row scratch, boot pool", + spec=dict( + _1M, + rows=13, + heads=16, + h_q=64, + blocks=_BOOT_BLOCKS, + layers=3, + q_offset=32755, + far=True, + large=True, + ), + ), + Cell( + why="1M tp8/tep8 mixed step's 16 generation rows (large): 8 of the prefill scratch's " + "64 heads, q and out tail views from row 32752, boot pool", + spec=dict( + _1M, + rows=16, + heads=8, + h_q=64, + blocks=_BOOT_BLOCKS, + layers=3, + q_offset=32752, + far=True, + large=True, + ), + ), + Cell( + why="1M NoPE decode graph 16 (large): owner row 35 as its nine sharers read it, zero " + "query and key rope halves, boot pool", + spec=dict( + _1M, + rows=16, + heads=16, + h_q=16, + blocks=_BOOT_BLOCKS, + layers=9, + captured=True, + nope=True, + far=True, + large=True, + ), + ), + Cell( + why="1M pre-merge decode graphs at every batch 1..16 to 2049 keys (2048 of 2049 " + "valid, -1 tails below 2048), 16 heads, a 512-block pool", + spec=dict( + _1M, + rows=16, + heads=16, + h_q=16, + kv_len=2049, + blocks=512, + layers=3, + captured=True, + graph_batches=tuple(range(1, 17)), + ), + ), + ) + + CELLS += tuple( + Cell( + why=f"DSpark 1M mixed step (large): 13 four-token verification blocks, {h} heads " + "in the prefill scratch's 64 heads, tail view after 32716 context rows", + spec=dict( + _1M, + rows=52, + heads=h, + h_q=64, + blocks=_BOOT_BLOCKS, + layers=3, + q_offset=32716, + far=True, + large=True, + ), + ) + for h in (8, 16) + ) + + note = """ + Call sites: the mistral_large4 NVFP4 tp4 and tep4 deployments (also run at world 8 as tp8 + and tep8), the generation rows of every layer, decode and mixed steps. Kernel (sm_103): + `_ds_mla_sparse_decode_kernel`, one launch, split-K partials merged in the cluster. + On SM107, rows 1 to 8 whose 16 CTAs per row fit in one wave of the SMs instead + run 16 independent split CTAs per row and one combine launch. The FP32 partial + workspace has 516 elements per (row, split, head), including the 512-wide + normalized output and LSE. Each invocation allocates its own workspace through + PyTorch, preserving stream and graph-pool lifetimes; other shapes and devices use + the original cluster path. + It replaces flash_mla_sparse_decode's head64 kernel on these targets, which needs the + query padded to 64 heads, a scheduler kernel and a combine kernel. + + Numerics: FlashMLA's (15f13e5, built --use_fast_math), operation for operation. Stage, + FlashMLA (csrc/sm100/decode/head64/kernel.cuh unless named) -> this kernel: + scale sparse_decode.h:387, common.h:12: fp32 sm_scale * 1.44269504f + -> scale_log2_of, the same fp32 product on the host. + K/V dequant :708 cvt.rz.ue8m0x2.f32 of the fp32 scales, :788 cvt.rn.bf16x2.ue8m0x2, + :802 helpers.h fp8x2_to_bf16x2_with_scale (bf16(e4m3) * bf16 scale) + -> the same two PTX conversions, cvt e4m3x2 -> bf16x2, mul.rn.bf16x2. + Rope bf16 as stored. Invalid slots: :703/:710 TMA coordinate -1 and + scale bytes 0 -> TMA zero-fills the row (codes and scales 0); the + token is masked below either way, so K and V of it never reach a sum. + Q bf16 as given (UTCCP into TMEM, :465) -> TMA into SMEM, bf16. + QK :542 rope (clear) then :552 NoPE, a dual GEMM: two fp32 accumulators, + half 0 = rope[0:32], NoPE chunks 0,2,4,6; half 1 = rope[32:64], chunks + 1,3,5,7; K = 16 per MMA in that order; :192 float2_add (add.rn.f32) + of the halves -> two TMEM accumulators fed in the same order, add.rn. + mask :215 p = -inf for an invalid token -> the same select. + row max :222 max of the raw scores, :224 * sm_scale_div_log2 (mul.ftz), :229 + max with the other 32 tokens -> max over the tile's 64 tokens, then + mul.ftz (max commutes with the monotone product). + real max :230 real_mi = max(real_mi, cur) -> the same. + rescale vote :231 __any_sync(cur - mi > 6.0f) over the warp's 32 heads (16 real + + 16 zero-query heads at 16 real; the zero heads never vote after the + first valid tile and vote with the real ones on it) -> OR over the + real heads of sub.ftz(cur, mi) > 6. + new max :240-244 1.0f / max(cur, mi), exp2f(mi - new_max) -> ex2.approx.ftz of + sub.ftz, exactly 1.0 when the max stays. + exponent :254 float2_fma (fma.rn.f32) of p, scale, -new_max; :255 exp2f + (ex2.approx.ftz) -> the same two instructions. + P :258 __float22bfloat162_rn -> cvt.rn.bf16x2.f32. + denominator :257 cur_sum += d (add.rn.f32, even and odd tokens of each 32-token + half in order), :260 li = fma(li, scale_for_old, x + y) (fma.ftz, + add.ftz), :311 li(tokens 0-31) + li(32-63) -> warps 4-5 sum the fp32 P + the softmax hands over in that order with those instructions. + O rescale :269-283 if block != first and voted: float2_mul (mul.rn.f32) in TMEM + -> the same condition and mul.rn.f32, before the tile's PV. + PV :580 bf16 P x bf16 V, fp32 accumulation in TMEM, K = 16 tokens per MMA + in token order, cleared on the first block -> the same, swapped + (O^T = V^T P^T). + no valid key :301-306 li = 0, mi = -inf -> the same. + no split :346 li == 0 ? 0 : __fdividef(1, li + exp2f(-inf - mi)) (= li), + :358 float2_mul, :359 __float22bfloat162_rn -> div.approx.ftz, + mul.rn.f32, cvt.rn.bf16x2.f32. + split :387 __fdividef(1, li), :400 float2_mul into fp32 o_accum, :321 + lse = log2f(li) + mi -> div.approx.ftz, mul.rn.f32 into the fp32 + partial in shared memory, lg2.approx.ftz + add.ftz. + combine combine.cu:84-88 max lse (0 when -inf), :93-96 sum of exp2f(lse - max) + over a xor butterfly, :98 global = sum == 0 ? +inf : log2f(sum) + max, + :117 w = exp2f(lse - global), :136 result += w * o (fma.ftz from 0, in + split order), :154 bf16 -> the same operations and order, through the + cluster's shared memory. + What is not FlashMLA's: the split partition. FlashMLA's scheduler gives every + partition ceil(37 B / 152) + 5 block-units of load (32 blocks + 5 overhead per row), so + a row's split count moves with the batch (32 one-block splits at 1-4 rows, 8 at 16, 4 + at 32, 2 at 64, 1 at 128); this kernel splits a row in 8, 4, 2 or 1 equal parts by + batch (`_kernel.pick_splits`). Where the two partitions coincide the output is + bit-identical to FlashMLA's (the GPU test checks it at 16/32/64/128 rows, 16 and 8 + heads, with holes and NoPE). Elsewhere the partition sets which tile maxima the online + softmax rescales against and which partials the combine merges -- a reordering of the + same fp32 sums, with P rounded to bf16 against a different running max. + + Addressing: a pool row is fetched whole by a TMA `tile::gather4` of the pool viewed + `[rows, 82]` int64 (row index int32, byte offset row * 656 in the TMA unit's 64-bit + address); the index ring, the query and the output are addressed per row from int32 + offsets under 2**31. At the 1M config's pools (74183 to 147456 blocks of 63 x 32 rows, + up to 2.97e8 rows and 1.95e11 bytes) the gathers read past 2**31 and 2**32 bytes; the + "1M" cells read the pool's top slot. Keys up to 1048576 change nothing else: the kernel + sees 2048 slots a row whatever the sequence. The 1M cells are bitwise FlashMLA's output + where the partitions coincide (16 / 32 / 64 / 128 rows, 16 and 8 heads, on the boot + pool), as the 32K ones are. + """ + + def __call__( + self, + q: torch.Tensor, + kv_cache: torch.Tensor, + indices: torch.Tensor, + softmax_scale: float, + out: torch.Tensor, + *, + compiled: CompiledDsMlaSparseDecode, + ) -> torch.Tensor: + if _runs_global(q.shape[0], compiled): + workspace = torch.empty( + (q.shape[0], GLOBAL_SPLITS[0], compiled.num_heads, 516), + dtype=torch.float32, + device=q.device, + ) + return _kernel._launch_global( + compiled.global_fns, q, kv_cache, indices, out, softmax_scale, workspace + ) + fn = compiled.fns[_kernel.pick_splits(q.shape[0], compiled.num_sms)] + return _kernel.launch(fn, q, kv_cache, indices, out, softmax_scale) + + def reference( + self, + q: torch.Tensor, + kv_cache: torch.Tensor, + indices: torch.Tensor, + softmax_scale: float, + out: torch.Tensor, + ) -> torch.Tensor: + """fp32, exact dequantization, softmax over the valid slots; a new tensor shaped + like `out`, rounded to bf16 once.""" + h_out = out.shape[1] + flat = kv_cache.view(torch.uint8).reshape(-1, ROW_BYTES) + res = torch.zeros(out.shape, dtype=torch.float32, device=q.device) + with true_fp32_matmul(): + for r in range(q.shape[0]): + idx = indices[r] + idx = idx[idx >= 0].long() + if idx.numel() == 0: + continue + rows = flat[idx] + codes = rows[:, :D_NOPE].contiguous().view(torch.float8_e4m3fn).float() + scale = rows[:, D_NOPE : D_NOPE + 16].contiguous().view(torch.float32) + nope = (codes.view(-1, D_NOPE // TILE, TILE) * scale.unsqueeze(-1)).view(-1, D_NOPE) + rope = rows[:, D_NOPE + 16 :].contiguous().view(torch.bfloat16).float() + k = torch.cat([nope, rope], 1) + s = (q[r, :h_out].float() @ k.T) * softmax_scale + res[r] = torch.softmax(s, -1) @ nope + return res.to(out.dtype) + + def is_valid( + self, + q: torch.Tensor, + kv_cache: torch.Tensor, + indices: torch.Tensor, + softmax_scale: float, + out: torch.Tensor, + *, + compiled: CompiledDsMlaSparseDecode, + ) -> None: + heads = out.shape[1] + if heads not in HEADS or heads != compiled.num_heads: + raise ValueError(f"{heads} heads: the kernel runs {HEADS}, compiled for " + f"{compiled.num_heads}") # fmt: skip + if q.dim() != 3 or q.shape[1] < heads or q.shape[2] != D_QK: + raise ValueError(f"q must be [rows, h_q >= {heads}, {D_QK}]") + if out.shape[0] != q.shape[0] or out.shape[2] != D_NOPE: + raise ValueError(f"out must be [rows, heads, {D_NOPE}]") + if indices.shape != (q.shape[0], TOPK) or indices.dtype != torch.int32: + raise ValueError(f"indices must be int32 [rows, {TOPK}] slots") + if kv_cache.shape[-1] != ROW_BYTES or kv_cache.dim() != 4 or kv_cache.shape[2] != 1: + raise ValueError("kv_cache must be [pages, page_size, 1, 656]") + if q.dtype != torch.bfloat16 or out.dtype != torch.bfloat16: + raise ValueError("q and out must be bf16") + for name, t in (("q", q), ("kv_cache", kv_cache), ("indices", indices), ("out", out)): + if not t.is_contiguous() or t.data_ptr() % 16: + raise ValueError(f"{name} must be contiguous and 16-byte aligned") + if t.device.index != compiled.device_index: + raise ValueError(f"{name} is not on the compiled device") + + +cute_dsl_ds_mla_sparse_decode = CuteDslDsMlaSparseDecode() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/cute_dsl_fp8_paged_mqa_logits.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/cute_dsl_fp8_paged_mqa_logits.py new file mode 100644 index 000000000000..d3d9353e3078 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/cute_dsl_fp8_paged_mqa_logits.py @@ -0,0 +1,1414 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Paged FP8 MQA logits: the DSA indexer's decode scores, in one CuTe DSL call.""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: Mistral Large 4's indexer: 32 heads of 128; the K cache holds 32 tokens per page, +#: each as 128 fp8 bytes plus one fp32 scale (132 bytes). It scores +#: one token per sequence (no speculation) into rows of `max_seq_len` columns: +#: 9216 for the FP8 target, 32768 for the NVFP4 target (`_ML4_32K` below), and +#: 1048576 for the NVFP4 target at the checkpoint's own context (`_ML4_1M`). In +#: its cells `layout` says how the cache view is made: "v2" and +#: "v1" rebuild the per-layer views the two DSA cache managers hand out (see +#: `note`), "manager_v2" and "manager_v1" take the view and the block ids from +#: a real manager at `layer` (built for `max_num_tokens`, 9216 unless the cell +#: says, and for `max_seq_len`, `max_context_len` unless the cell says). +#: `pool_layers` is the number of indexer rows in +#: the pool, `pool_row` the row read, `num_blocks` the pool's slot count, and +#: `tail` puts the sequences' pages in the pool's last blocks. The block table +#: is `table_rows` rows of `table_width` entries, the manager's +#: `max_blocks_per_seq` (measured: 292 for V2, 288 for V1, at 9216 tokens; +#: 1028 for V2, 1024 for V1, at 32768; 32772 for V2 at 1048576), the +#: call's sequences start at row `first_seq`, and its query and weight rows +#: come after `q_offset` rows of the same buffers, as the generation rows of a +#: mixed step do. `capture_lens` captures the call in a CUDA graph at those +#: lengths and replays it at `context_lens`. `padding` is the number of +#: trailing rows that repeat one dummy sequence's table row and length, as a +#: padded graph batch repeats one dummy request (its single key is read off +#: `add_dummy_requests`' default of one token, not measured). `past` is the +#: byte offset into the pool the highest page read must lie beyond. +_ML4 = dict(heads=32, head_dim=128, tokens_per_block=32, next_n=1, max_context_len=9216) + +#: The largest pools an FP8-target rank can hold, deliberate overestimates: +#: about 24.7 GiB is left per GB300 after the 252.84 GiB of weights, and both +#: DSA cache managers take 47280 bytes per token with 44 indexer rows and 50976 +#: with 72 (measured), so at most about 17.5K and 16.3K blocks of 32 tokens. +_ML4_POOL_44 = dict(layout="v2", pool_layers=44, num_blocks=18432, table_width=292) +_ML4_FAR_44 = dict(pool_layers=44, num_blocks=18432, tail=True, past=2**31) +_ML4_FAR_72 = dict(pool_layers=72, num_blocks=16384, tail=True, past=2**32) + +#: Every edge of a page (32 keys), the schedule block (64), a compute block +#: (128) and a two-block step (256), and the row's full width. +_ML4_EDGES = [1, 31, 32, 33, 63, 64, 65, 127, 128, 129, 255, 256, 257, 2049, 9215, 9216] + +#: The NVFP4 target: rows of 32768 columns, 1024 pages per sequence. +_ML4_32K = dict(heads=32, head_dim=128, tokens_per_block=32, next_n=1, max_context_len=32768) + +#: The NVFP4 target's pools, in the engine-default V2 layout, the table V2's +#: `max_blocks_per_seq` at 32768 tokens wide. A boot gate read 2373024 KV +#: tokens per rank, `max_num_blocks` x 32, and `max_num_blocks` is the pool's +#: slot count (measured: a V2 manager built for a 2373024-token quota held +#: 78051 slots and reported 78051), so that pool is 74157 blocks: the far +#: cell's. The large cell's 109568 is more than a GB300 can hold beside the +#: ~143 GiB of weights per tep4 rank (about 94660 blocks at 47280 bytes per +#: token). +#: Graph and edge cells take 32768 blocks, twice the 16384 pages that 16 +#: sequences of 32768 keys read. +_ML4_32K_POOL_44 = dict(layout="v2", pool_layers=44, num_blocks=32768, table_width=1028) +_ML4_32K_FAR_44 = dict(layout="v2", pool_layers=44, table_width=1028, pool_row=43, tail=True) + +#: The largest pool a rank of the eight-GPU targets can hold, a bound: a tep8 or +#: tp8 rank keeps 73.61 GiB of weights and a dep8 rank 89.29 GiB (its 16 routed +#: experts and the replicated rest), computed from the checkpoint's shard headers +#: under each target's split rules, so 203.01 and 187.33 GiB of a 276.62 GiB GB300 +#: are left, 144075 and 132949 V2 slots of 32 tokens at 47280 bytes per token. V2 +#: hands out about 5% more slots than its token quota, and 147456 clears both +#: readings. No boot has measured the pool at that bound. +_ML4_32K_8GPU_MAX_BLOCKS = 147456 + +#: The edges of the schedule block (64) and a two-block step (256), the +#: top-k's 2048, 4096, the FP8 envelope's 9216, half the row, the row's last +#: schedule block and its full width. +_ML4_32K_EDGES = [ + *(1, 64, 65, 255, 256, 257, 2048, 2049, 4097, 9216, 9217), + *(16384, 16385, 32704, 32767, 32768), +] + +#: The NVFP4 target at 64-token pages: 8448-byte pages (64 x 132), 512 pages per sequence, +#: and a V2 table of 516 columns (32768 / 64 rounded up past the manager's generation +#: headroom to a multiple of 4). Each pool holds its 32-token counterpart's tokens in half +#: the blocks: 16384 for the graph and mixed-step cells, 37078 for the measured 2373024-token +#: pool, 54784 for the large one. +_ML4_32K_P64 = dict(_ML4_32K, tokens_per_block=64) +_ML4_32K_P64_POOL_44 = dict(layout="v2", pool_layers=44, num_blocks=16384, table_width=516) +_ML4_32K_P64_FAR_44 = dict(layout="v2", pool_layers=44, table_width=516, pool_row=43, tail=True) + +#: The NVFP4 target at its checkpoint's context (max_seq_len 1048576, chunked +#: prefill in 32768-token steps, max_batch_size 16, per rank for dep4): the +#: target passes `max_context_len = md.max_seq_len` (its `_max_ctx_len`, +#: frozen into every capture), so every call scores rows of 1048576 columns, +#: 32768 pages per sequence. +_ML4_1M = dict(heads=32, head_dim=128, tokens_per_block=32, next_n=1, max_context_len=1048576) + +#: DSpark K=3 includes overlap/verification headroom at the 1M envelope. +#: Four causal rows per request, each padded to a 1048832-column stride. +_ML4_1M_DSPARK = dict(_ML4_1M, next_n=4, max_context_len=1048584) + +#: Its pools, in the V2 layout, the table the V2 manager's `max_blocks_per_seq` +#: at 1048576 tokens wide (32772, measured on the manager: ceil((1048576 + 1) / +#: 32) padded to a multiple of 4). The tp4 boot gate read 2373856 KV tokens per +#: rank, 74183 blocks: room for two sequences of 1048576 keys beside shorter +#: ones, so the populations stay within it. `padded_to` puts the `padding` +#: rows of a 1-key dummy after `context_lens` (and `capture_lens`). +_ML4_1M_POOL_44 = dict(layout="v2", pool_layers=44, num_blocks=74183, table_width=32772) +_ML4_1M_FAR_44 = dict(layout="v2", pool_layers=44, table_width=32772, pool_row=43, tail=True) + +#: tep4's pool, the delivered 74157 blocks in the V2 layout (13.8 GB of indexer +#: rows; two sequences of 1048576 keys take 65536 of its pages), read through +#: the table `KVCacheManagerV2` allocates at 1048576 tokens: ceil(1048577 / 32) +#: = 32769 pages, padded to a multiple of 4 (the manager cell measures it). +#: tep4's cells use it; it differs from `_ML4_1M_POOL_44` (tp4's boot pool of +#: 74183 blocks) only in `num_blocks`. +_ML4_1M_POOL_44_TEP4 = dict(layout="v2", pool_layers=44, num_blocks=74157, table_width=32772) + +#: A dep4 rank's pool at 1048576, rounded up: 65536 blocks in the V2 layout +#: (12.2 GB of indexer rows). The dep4 boot gate at 1048576 read 1983872 KV +#: tokens per rank, `max_num_blocks` x 32, and V2's `max_num_blocks` is its +#: slot count, so that pool is 61996 slots (58557 under the accuracy gate's +#: settings); the far cell's 74157 blocks bound it too. The table is the one +#: `KVCacheManagerV2` allocates at 1048576 tokens: ceil(1048577 / 32) = 32769 +#: pages, padded to a multiple of 4 (the manager cell measures it). +_ML4_1M_DEP4_POOL_44 = dict(layout="v2", pool_layers=44, num_blocks=65536, table_width=32772) + +#: The tep8 deployment makes tep4's call at 1048576 (its 32 indexer heads +#: replicated on every rank) on a rank's own pool: the delivered tep8 boot read +#: 3680512 KV tokens per rank, 115016 V2 slots of 32 tokens (`max_num_blocks` is +#: the slot count, as for tep4's 74157), and `_ML4_32K_8GPU_MAX_BLOCKS` bounds +#: it. Same view and table as `_ML4_1M_POOL_44_TEP4`, the slot count per cell. +_ML4_1M_TEP8_BOOT_BLOCKS = 115016 +_ML4_1M_TEP8_44 = dict(layout="v2", pool_layers=44, table_width=32772) + + +class CuteDslFp8PagedMqaLogits(OpWrapper): + """Score each decode query token against its own sequence's paged keys. + + `q` is `[B, next_n, H, D]` fp8; `kv_fused` is the indexer K cache + `[num_pages, page, 1, D + 4]` uint8, read through `block_table[b]`. Returns + `[B * next_n, max_context_len]` fp32 with + + logits[b * next_n + t, j] = k_scale[j] * sum_h weights[b * next_n + t, h] + * relu(q[b, t, h, :] . k[j, :]) + + where `k` / `k_scale` are sequence `b`'s keys. Only the causal part of + each row, `j < context_lens[b] - next_n + t + 1`, is defined; that is all + the decode top-k reads. As with `fp8_mqa_logits`, the q-side scale must be + folded into `weights`. + + The result is a view of a shared arena buffer, not a fresh tensor: the + next call returns the same storage and overwrites it. Consume the logits + before scoring again. + + Fusion boundary: the q.k products, relu, head weighting, head sum and key + scale happen inside the call. The caller owns the q-side quantization and + weight folding, the key cache writes, the schedule and the top-k after it. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # `max_context_len` is the engine's max sequence length, so most of each + # row is past the sequence's end. Context lengths are the sequences' KV + # lengths after this step. + # + # Mistral Large 4 (max_batch_size 16, no speculation) scores 1..16 sequences + # in CUDA-graph decode batches 1, 2, 4, 8 and 16 (padded), and the + # generation rows of a mixed step eagerly. The FP8 target's cells (max_seq_len + # 9216) capture every graph batch and replay it on new data, read the + # engine-default V2 view of the largest 44-row pool, take the view and the + # block ids from both real managers, and address the far end of the + # largest pool in both layouts at 44 and 72 rows. The NVFP4 target's + # ("NVFP4 32768", max_seq_len 32768) do the same for rows of 32768 keys in + # its V2 layout: every graph batch replayed at lengths up to 32768, a mixed + # step after 32765 prompt rows, the real V2 manager at 32768, and the far + # end of its measured pool (past 2^33 bytes) and of a pool no tep4 rank + # can hold (past 2^34, `large`). The "64-token pages" cells repeat those NVFP4 + # cells over 64-token pages: the same lengths, rows and replays, a 516-column + # table and pools of the same bytes in half the blocks. The eight-GPU cell + # ("NVFP4 32768 tep8/tp8/dep8") is the call of the tep8, tp8 and dep8 + # deployments, the 32 indexer heads whole on every rank (replicated at tep8 and + # tp8, data parallel at dep8), on the larger pool a rank holds beside its + # smaller share of the weights -- the far end of the 147456-block bound + # (past 2^34, `large`). The NVFP4 target at max_seq_len 1048576 with + # chunked prefill ("NVFP4 1M", tp4: the indexer is replicated, so every + # rank scores all 32 heads) freezes 1048576 into the capture: every graph + # batch replayed with one or two rows at 1M and the rest shorter, inside + # its boot pool of 74183 blocks; the 16-row batch padded after 11 live + # rows; a mixed step's 3 generation rows to 1M after 32765 prompt rows; the + # far end of the boot pool; and, `large`, the real V2 manager at 1048576 + # built for the boot's 2373856-token quota (V2 keeps max_seq_len 1048576 + # only for a quota of at least 32769 blocks, about 50 GiB) and the far end + # of the 109568-block pool. + # + # The NVFP4 tep4 deployment at max_seq_len 1048576 with chunked prefill + # ("NVFP4 1M") keeps both call sites (DecodeTarget._indexer, captured; + # PrefillTarget._indexer's generation rows of a mixed step, eager); what + # grows is the row (1048576 columns), the context lengths (to 1048576) and + # the table (32772 columns). Its cells replay every graph batch at lengths + # up to 1048576 on the 44-row V2 view of the delivered 74157-block pool, + # score a mixed step's generation rows after 32765 chunk rows, take the + # view and the table from the real V2 manager built at 1048576 (`large`: + # it took 115 GiB of the device), and read a 1048576-key sequence from the + # pool's last pages. The kernel is compiled once for every row and table + # width (both are symbolic in `CuteDSLPagedMQALogitsRunner._compile`), so + # the 1M rows run the kernel the 32768 rows do. The tep8 deployment at 1048576 + # ("NVFP4 1M tep8") makes the tep4 call; its cells are the tep4 cells plus + # a rank's own pools: boot's capture shape on the 115016-block pool the + # tep8 boot measured, and a 1048576-key sequence at the far end of the + # 147456-block bound (past 2^34), both `large` (pool and the test's copy + # of it pass 40 GiB). + # + # The NVFP4 dep4 deployment at max_seq_len 1048576 with chunked prefill + # ("dep4 1M") keeps both call sites (DecodeTarget._indexer, captured; + # PrefillTarget._indexer's generation rows of a mixed step, eager); what + # grows is the row (1048576 columns), the context lengths (to 1048576) and + # the table (32772 columns). Boot reaches it first: the engine's CUDA-graph + # warm-up captures each decode batch on its longest dummy request, 1048575 + # keys in row 0, beside 1-key fillers, and those shapes are replayed first + # ("dep4 1M boot capture"). The other cells replay every graph batch at + # lengths up to 1048576 on the 44-row V2 view of a 65536-block pool, + # attention DP's one live row behind 1-key padding rows at batches 2, 4, 8 + # and 16, score a mixed step's generation rows after 32765 and 32753 chunk + # rows, take the view and the table from the real V2 manager built at + # 1048576 (`large`), and read a 1048576-key sequence from the last pages of + # a 74157-block pool. The kernel is compiled once for every row and table + # width (both, and the logits row stride, are symbolic in + # `CuteDSLPagedMQALogitsRunner._compile`), so the 1M rows run the kernel + # the 32768 rows do. + CELLS: tuple[Cell, ...] = ( + *( + Cell( + why=f"DSpark 1M verify graph batch {batch}: boot to long causal/page ends", + spec=dict( + _ML4_1M_DSPARK, + **_ML4_1M_POOL_44, + pool_row=43, + capture_lens=([1048583] + [4] * 15)[:batch], + context_lens=( + [1048584, 1048576, 2049, 2048, 33, 32, 5, 4] + + [257, 65, 64, 31, 32769, 4097, 9217, 16385] + )[:batch], + ), + ) + for batch in (1, 2, 4, 8, 16) + ), + Cell( + why="DSpark 1M verify mixed: 60 rows after 32708 prompt rows", + spec=dict( + _ML4_1M_DSPARK, + **_ML4_1M_POOL_44, + pool_row=0, + first_seq=1, + table_rows=16, + q_offset=32708, + context_lens=[1048584, 2048, 2049, 32, 33] * 3, + ), + ), + Cell( + why="DSpark 1M verify DEP padding: one live request and 60 dummy rows", + spec=dict( + _ML4_1M_DSPARK, + **_ML4_1M_POOL_44, + pool_row=43, + capture_lens=[1048583] + [4] * 15, + context_lens=[1048575] + [4] * 15, + padding=15, + ), + ), + Cell( + why="DSpark 1M verify far pool: four causal ends past 2^34 bytes", + spec=dict( + _ML4_1M_DSPARK, + **_ML4_1M_FAR_44, + num_blocks=_ML4_32K_8GPU_MAX_BLOCKS, + context_lens=[1048584, 777], + past=2**34, + large=True, + ), + ), + *( + Cell( + why=f"DSpark verify 4 tokens, graph batch {batch}, 32776 columns and causal ends", + spec=dict( + _ML4_32K, + next_n=4, + max_context_len=32776, + **_ML4_32K_POOL_44, + pool_row=43, + capture_lens=[4] * batch, + context_lens=([32776, 2049, 33, 65, 257, 4097, 9217, 16385] * 2)[:batch], + ), + ) + for batch in (1, 2, 4, 8, 16) + ), + Cell( + why="DSpark verify mixed: 15 sequences after 32708 prompt rows, padded table rows", + spec=dict( + _ML4_32K, + next_n=4, + max_context_len=32776, + **_ML4_32K_POOL_44, + pool_row=0, + first_seq=1, + table_rows=16, + q_offset=32708, + context_lens=[32776, 2048, 2049, 32, 33] * 3, + ), + ), + Cell( + why="DSpark verify dummy padding at one-key context, each query's causal end", + spec=dict( + _ML4_32K, + next_n=4, + max_context_len=32776, + **_ML4_32K_POOL_44, + pool_row=43, + padding=7, + capture_lens=[4] * 9 + [1] * 7, + context_lens=[32776, 2049, 33, 65, 257, 4097, 9217, 16385, 32767] + [1] * 7, + ), + ), + Cell( + why="decode graph batch 1: 9216 keys, the row's full width; captured at 5000, replayed", + spec=dict(_ML4, **_ML4_POOL_44, pool_row=43, capture_lens=[5000], context_lens=[9216]), + ), + Cell( + why="decode graph batch 2: captured at 1 key each, replayed at 2049 and 9215", + spec=dict( + _ML4, **_ML4_POOL_44, pool_row=0, capture_lens=[1, 1], context_lens=[2049, 9215] + ), + ), + Cell( + why="decode graph batch 4: captured, replayed across a page and the top-k edge", + spec=dict( + _ML4, + **_ML4_POOL_44, + pool_row=17, + capture_lens=[9216, 9216, 9216, 9216], + context_lens=[33, 2048, 2049, 6000], + ), + ), + Cell( + why="decode graph batch 8: captured, replayed at compute-block edges and 9216", + spec=dict( + _ML4, + **_ML4_POOL_44, + pool_row=29, + capture_lens=[100] * 8, + context_lens=[1, 64, 65, 255, 256, 257, 4097, 9216], + ), + ), + Cell( + why="decode graph batch 16: every page / schedule / block edge, captured, replayed", + spec=dict( + _ML4, + **_ML4_POOL_44, + pool_row=31, + capture_lens=[100 + 500 * i for i in range(16)], + context_lens=list(_ML4_EDGES), + ), + ), + Cell( + why="decode graph batch 16 padded: 11 sequences + 5 rows of one 1-key dummy", + spec=dict( + _ML4, + **_ML4_POOL_44, + pool_row=40, + capture_lens=[1] * 16, + context_lens=[9216, 8191, 7000, 5555, 4096, 3001, 2049, 1024, 500, 77, 2] + [1] * 5, + padding=5, + ), + ), + Cell( + why="mixed step, eager: 3 generation rows after 2 prompts, rows 2..4 of the table", + spec=dict( + _ML4, + layout="v1", + pool_layers=44, + pool_row=12, + num_blocks=4096, + table_width=288, + table_rows=16, + first_seq=2, + q_offset=5000, + context_lens=[2049, 777, 9216], + ), + ), + Cell( + why="engine V2 manager (the default), 44 rows, layer 71: 3 sequences", + spec=dict( + _ML4, + layout="manager_v2", + pool_layers=44, + layer=71, + max_tokens=18432, + table_width=292, + table_rows=16, + context_lens=[9216, 2049, 1], + ), + ), + Cell( + why="engine V1 manager, 44 rows, layer 0: 3 sequences", + spec=dict( + _ML4, + layout="manager_v1", + pool_layers=44, + layer=0, + max_tokens=18432, + table_width=288, + table_rows=16, + context_lens=[5000, 3000, 1216], + ), + ), + Cell( + why="far pool V1, 44 rows x 18432 blocks (3.4 GB): last row, bytes past 2^31", + spec=dict( + _ML4, + **_ML4_FAR_44, + layout="v1", + table_width=288, + pool_row=43, + context_lens=[9216, 777], + ), + ), + Cell( + why="far pool V2, 44 rows x 18432 blocks (3.4 GB): last row, pages past 2^31 bytes", + spec=dict( + _ML4, + **_ML4_FAR_44, + layout="v2", + table_width=292, + pool_row=43, + context_lens=[9216, 777], + ), + ), + Cell( + why="far pool V1, 72 rows x 16384 blocks (5.0 GB): last row, bytes past 2^32", + spec=dict( + _ML4, + **_ML4_FAR_72, + layout="v1", + table_width=288, + pool_row=71, + context_lens=[9216, 777], + ), + ), + Cell( + why="far pool V2, 72 rows x 16384 blocks (5.0 GB): last row, pages past 2^32 bytes", + spec=dict( + _ML4, + **_ML4_FAR_72, + layout="v2", + table_width=292, + pool_row=71, + context_lens=[9216, 777], + ), + ), + Cell( + why="NVFP4 32768, decode graph batch 1: the row's full width, captured at 5000, replayed", + spec=dict( + _ML4_32K, **_ML4_32K_POOL_44, pool_row=43, capture_lens=[5000], context_lens=[32768] + ), + ), + Cell( + why="NVFP4 32768, decode graph batch 2: captured at 1 key each, replayed at 16385, 32767", + spec=dict( + _ML4_32K, + **_ML4_32K_POOL_44, + pool_row=0, + capture_lens=[1, 1], + context_lens=[16385, 32767], + ), + ), + Cell( + why="NVFP4 32768, decode graph batch 4: captured at 32768, replayed across 2048 and 9216", + spec=dict( + _ML4_32K, + **_ML4_32K_POOL_44, + pool_row=17, + capture_lens=[32768] * 4, + context_lens=[33, 2049, 9217, 30000], + ), + ), + Cell( + why="NVFP4 32768, decode graph batch 8: captured, replayed at compute-block edges to 32768", + spec=dict( + _ML4_32K, + **_ML4_32K_POOL_44, + pool_row=29, + capture_lens=[100] * 8, + context_lens=[127, 128, 129, 16383, 16384, 32640, 32641, 32768], + ), + ), + Cell( + why="NVFP4 32768, decode graph batch 16: every schedule / block edge to 32768, replayed", + spec=dict( + _ML4_32K, + **_ML4_32K_POOL_44, + pool_row=31, + capture_lens=[100 + 2000 * i for i in range(16)], + context_lens=list(_ML4_32K_EDGES), + ), + ), + Cell( + why="NVFP4 32768, decode graph batch 16 padded: 11 sequences + 5 rows of a 1-key dummy", + spec=dict( + _ML4_32K, + **_ML4_32K_POOL_44, + pool_row=40, + capture_lens=[1] * 16, + context_lens=[32768, 32767, 30001, 24576, 16385, 9217, 8191, 2049, 500, 77, 2] + + [1] * 5, + padding=5, + ), + ), + Cell( + why="NVFP4 32768, mixed step, eager: 3 generation rows after 32765 prompt rows", + spec=dict( + _ML4_32K, + **_ML4_32K_POOL_44, + pool_row=12, + table_rows=16, + first_seq=2, + q_offset=32765, + context_lens=[2049, 777, 32768], + ), + ), + Cell( + why="NVFP4 32768, engine V2 manager (the default), 44 rows, layer 71: 3 sequences", + spec=dict( + _ML4_32K, + layout="manager_v2", + pool_layers=44, + layer=71, + max_tokens=65536, + max_num_tokens=32768, + table_width=1028, + table_rows=16, + context_lens=[32768, 9217, 1], + ), + ), + Cell( + why="NVFP4 32768, far pool V2 at the measured 74157 blocks (13.8 GB): pages past 2^33", + spec=dict( + _ML4_32K, **_ML4_32K_FAR_44, num_blocks=74157, past=2**33, context_lens=[32768, 777] + ), + ), + Cell( + why="NVFP4 32768, far pool V2, 109568 blocks (20.4 GB, large): pages past 2^34 bytes", + spec=dict( + _ML4_32K, + **_ML4_32K_FAR_44, + num_blocks=109568, + past=2**34, + large=True, + context_lens=[32768, 777], + ), + ), + Cell( + why="64-token pages, NVFP4 32768, graph batch 1: the row's full width, captured at " + "5000, replayed", + spec=dict( + _ML4_32K_P64, + **_ML4_32K_P64_POOL_44, + pool_row=43, + capture_lens=[5000], + context_lens=[32768], + ), + ), + Cell( + why="64-token pages, NVFP4 32768, graph batch 2: captured at 1 key each, replayed at " + "16385, 32767", + spec=dict( + _ML4_32K_P64, + **_ML4_32K_P64_POOL_44, + pool_row=0, + capture_lens=[1, 1], + context_lens=[16385, 32767], + ), + ), + Cell( + why="64-token pages, NVFP4 32768, graph batch 4: captured at 32768, replayed across " + "2048 and 9216", + spec=dict( + _ML4_32K_P64, + **_ML4_32K_P64_POOL_44, + pool_row=17, + capture_lens=[32768] * 4, + context_lens=[33, 2049, 9217, 30000], + ), + ), + Cell( + why="64-token pages, NVFP4 32768, graph batch 8: captured, replayed at compute-block " + "edges to 32768", + spec=dict( + _ML4_32K_P64, + **_ML4_32K_P64_POOL_44, + pool_row=29, + capture_lens=[100] * 8, + context_lens=[127, 128, 129, 16383, 16384, 32640, 32641, 32768], + ), + ), + Cell( + why="64-token pages, NVFP4 32768, graph batch 16: every schedule / block edge to " + "32768, replayed", + spec=dict( + _ML4_32K_P64, + **_ML4_32K_P64_POOL_44, + pool_row=31, + capture_lens=[100 + 2000 * i for i in range(16)], + context_lens=list(_ML4_32K_EDGES), + ), + ), + Cell( + why="64-token pages, NVFP4 32768, graph batch 16 padded: 11 sequences + 5 rows of a " + "1-key dummy", + spec=dict( + _ML4_32K_P64, + **_ML4_32K_P64_POOL_44, + pool_row=40, + capture_lens=[1] * 16, + context_lens=[32768, 32767, 30001, 24576, 16385, 9217, 8191, 2049, 500, 77, 2] + + [1] * 5, + padding=5, + ), + ), + Cell( + why="64-token pages, NVFP4 32768, mixed step, eager: 3 generation rows after 32765 " + "prompt rows", + spec=dict( + _ML4_32K_P64, + **_ML4_32K_P64_POOL_44, + pool_row=12, + table_rows=16, + first_seq=2, + q_offset=32765, + context_lens=[2049, 777, 32768], + ), + ), + Cell( + why="64-token pages, NVFP4 32768, engine V2 manager (the default), 44 rows, layer 71: " + "3 sequences", + spec=dict( + _ML4_32K_P64, + layout="manager_v2", + pool_layers=44, + layer=71, + max_tokens=65536, + max_num_tokens=32768, + table_width=516, + table_rows=16, + context_lens=[32768, 9217, 1], + ), + ), + Cell( + why="64-token pages, NVFP4 32768, far pool V2 at 37078 blocks (13.8 GB, the measured " + "pool's tokens): pages past 2^33 bytes, page indices below 1631432 (0.08% of 2^31)", + spec=dict( + _ML4_32K_P64, + **_ML4_32K_P64_FAR_44, + num_blocks=37078, + past=2**33, + context_lens=[32768, 777], + ), + ), + Cell( + why="64-token pages, NVFP4 32768, far pool V2, 54784 blocks (20.4 GB, large): pages " + "past 2^34 bytes, page indices below 2410496 (0.11% of 2^31)", + spec=dict( + _ML4_32K_P64, + **_ML4_32K_P64_FAR_44, + num_blocks=54784, + past=2**34, + large=True, + context_lens=[32768, 777], + ), + ), + Cell( + why="NVFP4 32768 tep8/tp8/dep8, far pool V2 at a rank's bound, 147456 blocks (27.4 " + "GB, large): pages past 2^34 bytes", + spec=dict( + _ML4_32K, + **_ML4_32K_FAR_44, + num_blocks=_ML4_32K_8GPU_MAX_BLOCKS, + past=2**34, + large=True, + context_lens=[32768, 777], + ), + ), + Cell( + why="NVFP4 1M, decode graph batch 1: a 1048576-key row, captured at 5000, replayed", + spec=dict( + _ML4_1M, **_ML4_1M_POOL_44, pool_row=43, capture_lens=[5000], context_lens=[1048576] + ), + ), + Cell( + why="NVFP4 1M, decode graph batch 2: rows of 1048576 and 1048575 keys, captured at 1", + spec=dict( + _ML4_1M, + **_ML4_1M_POOL_44, + pool_row=0, + capture_lens=[1, 1], + context_lens=[1048576, 1048575], + ), + ), + Cell( + why="NVFP4 1M, decode graph batch 4: 65536/65537 and 524288 beside 1M, captured short", + spec=dict( + _ML4_1M, + **_ML4_1M_POOL_44, + pool_row=17, + capture_lens=[32768, 1, 2049, 9217], + context_lens=[65537, 1048576, 524288, 65536], + ), + ), + Cell( + why="NVFP4 1M, decode graph batch 8: two 1M rows beside 65535..65537, captured as " + "the engine captures (one row of 1048575 keys, 1-key fillers)", + spec=dict( + _ML4_1M, + **_ML4_1M_POOL_44, + pool_row=29, + capture_lens=[1048575] + [1] * 7, + context_lens=[65535, 1048576, 65536, 32769, 65537, 1048575, 129, 1], + ), + ), + Cell( + why="NVFP4 1M, decode graph batch 16: a row per edge from 1 to 1048576, replayed", + spec=dict( + _ML4_1M, + **_ML4_1M_POOL_44, + pool_row=31, + capture_lens=[1] * 16, + context_lens=[1048576, 524288, 262144, 131072, 65536, 32769, 32768, 16385] + + [9217, 4097, 2049, 2048, 257, 256, 65, 1], + ), + ), + Cell( + why="NVFP4 1M, decode graph batch 16 padded: 11 sequences to 1M + 5 rows of a 1-key " + "dummy", + spec=dict( + _ML4_1M, + **_ML4_1M_POOL_44, + pool_row=40, + capture_lens=[1] * 11, + context_lens=[1048576, 900000, 100000, 50000, 32768, 2049, 2048, 1000, 100, 2, 1], + padding=5, + padded_to=16, + ), + ), + Cell( + why="NVFP4 1M, mixed step, eager: 3 generation rows to 1M after 32765 prompt rows", + spec=dict( + _ML4_1M, + **_ML4_1M_POOL_44, + pool_row=12, + table_rows=16, + first_seq=2, + q_offset=32765, + context_lens=[1048576, 524288, 3], + ), + ), + Cell( + why="NVFP4 1M, far pool V2 at the boot's 74183 blocks (13.8 GB): a 1M row past 2^33", + spec=dict( + _ML4_1M, **_ML4_1M_FAR_44, num_blocks=74183, past=2**33, context_lens=[1048576, 777] + ), + ), + Cell( + why="NVFP4 1M, engine V2 manager at the boot's 2373856 tokens (large): layer 71, " + "two 1M rows + 32768 + 777 keys", + spec=dict( + _ML4_1M, + layout="manager_v2", + pool_layers=44, + layer=71, + max_tokens=2373856, + max_num_tokens=32768, + table_width=32772, + table_rows=16, + large=True, + context_lens=[1048576, 1048575, 32768, 777], + ), + ), + Cell( + why="NVFP4 1M, far pool V2, 109568 blocks (20.4 GB, large): a 1M row past 2^34", + spec=dict( + _ML4_1M, + **_ML4_1M_FAR_44, + num_blocks=109568, + past=2**34, + large=True, + context_lens=[1048576, 777], + ), + ), + Cell( + why="NVFP4 1M, decode graph batch 1: a 1048576-key row in full, captured at 5000, replayed", + spec=dict( + _ML4_1M, + **_ML4_1M_POOL_44_TEP4, + pool_row=43, + capture_lens=[5000], + context_lens=[1048576], + ), + ), + Cell( + why="NVFP4 1M, decode graph batch 2: captured at 1 key each, replayed at 524289, 1048575", + spec=dict( + _ML4_1M, + **_ML4_1M_POOL_44_TEP4, + pool_row=0, + capture_lens=[1, 1], + context_lens=[524289, 1048575], + ), + ), + Cell( + why="NVFP4 1M, decode graph batch 4: captured at 1 key, replayed at 33 to 1000000", + spec=dict( + _ML4_1M, + **_ML4_1M_POOL_44_TEP4, + pool_row=17, + capture_lens=[1, 1, 1, 1], + context_lens=[33, 2049, 32769, 1000000], + ), + ), + Cell( + why="NVFP4 1M, decode graph batch 8: block edges at 128, 32768 and 1048576, pages shared", + spec=dict( + _ML4_1M, + **_ML4_1M_POOL_44_TEP4, + pool_row=29, + capture_lens=[100] * 8, + context_lens=[127, 128, 129, 32768, 32769, 524288, 1048320, 1048576], + ), + ), + Cell( + why="NVFP4 1M, decode graph batch 16: one row per width from 1 key to 1048576, replayed", + spec=dict( + _ML4_1M, + **_ML4_1M_POOL_44_TEP4, + pool_row=31, + capture_lens=[100 + 2000 * i for i in range(16)], + context_lens=[ + *(1, 64, 65, 255, 256, 257, 2048, 2049, 9217, 32768, 32769), + *(65536, 131072, 262145, 524288, 1048576), + ], + ), + ), + Cell( + why="NVFP4 1M, decode graph batch 16 padded: 11 sequences to 1048576 + 5 rows of a dummy", + spec=dict( + _ML4_1M, + **_ML4_1M_POOL_44_TEP4, + pool_row=40, + capture_lens=[1] * 16, + context_lens=[1048576, 524289, 131073, 65536, 32769, 9217, 8191, 2049, 500, 77, 2] + + [1] * 5, + padding=5, + ), + ), + Cell( + why="NVFP4 1M, decode graph batch 16 at the boot capture: KV 1048576 beside 15 of KV 2", + spec=dict( + _ML4_1M, + **_ML4_1M_POOL_44_TEP4, + pool_row=7, + capture_lens=[1048576] + [2] * 15, + context_lens=[1048576] + [2] * 15, + ), + ), + Cell( + why="NVFP4 1M, mixed step, eager: 3 generation rows after 32765 chunk rows, to 1048576", + spec=dict( + _ML4_1M, + **_ML4_1M_POOL_44_TEP4, + pool_row=12, + table_rows=16, + first_seq=2, + q_offset=32765, + context_lens=[2049, 777, 1048576], + ), + ), + Cell( + why="NVFP4 1M, engine V2 manager built at 1048576 (large), layer 71: 3 sequences", + spec=dict( + _ML4_1M, + layout="manager_v2", + pool_layers=44, + layer=71, + max_tokens=2097152, + max_num_tokens=32768, + table_width=32772, + table_rows=16, + large=True, + context_lens=[1048576, 9217, 1], + ), + ), + Cell( + why="NVFP4 1M, far pool V2 at 74157 blocks: a 1048576-key sequence's pages past 2^33", + spec=dict( + _ML4_1M, + **_ML4_1M_POOL_44_TEP4, + pool_row=43, + tail=True, + past=2**33, + context_lens=[1048576, 777], + ), + ), + Cell( + why="dep4 1M boot capture, graph batch 1: the warm-up dummy's 1048575 keys, replayed", + spec=dict( + _ML4_1M, + **_ML4_1M_DEP4_POOL_44, + pool_row=43, + capture_lens=[1048575], + context_lens=[1048575], + ), + ), + Cell( + why="dep4 1M boot capture, graph batch 2: 1048575 keys + a 1-key filler, replayed", + spec=dict( + _ML4_1M, + **_ML4_1M_DEP4_POOL_44, + pool_row=0, + capture_lens=[1048575, 1], + context_lens=[1048575, 1], + ), + ), + Cell( + why="dep4 1M boot capture, graph batch 4: 1048575 keys + 3 1-key fillers, replayed", + spec=dict( + _ML4_1M, + **_ML4_1M_DEP4_POOL_44, + pool_row=21, + capture_lens=[1048575] + [1] * 3, + context_lens=[1048575] + [1] * 3, + ), + ), + Cell( + why="dep4 1M boot capture, graph batch 8: 1048575 keys + 7 1-key fillers, replayed", + spec=dict( + _ML4_1M, + **_ML4_1M_DEP4_POOL_44, + pool_row=36, + capture_lens=[1048575] + [1] * 7, + context_lens=[1048575] + [1] * 7, + ), + ), + Cell( + why="dep4 1M boot capture, graph batch 16: 1048575 keys + 15 1-key fillers, replayed", + spec=dict( + _ML4_1M, + **_ML4_1M_DEP4_POOL_44, + pool_row=7, + capture_lens=[1048575] + [1] * 15, + context_lens=[1048575] + [1] * 15, + ), + ), + Cell( + why="dep4 1M, decode graph batch 1: a 1048576-key row in full, captured at 5000, replayed", + spec=dict( + _ML4_1M, + **_ML4_1M_DEP4_POOL_44, + pool_row=43, + capture_lens=[5000], + context_lens=[1048576], + ), + ), + Cell( + why="dep4 1M, decode graph batch 2: captured at 1 key each, replayed at 524289, 1048575", + spec=dict( + _ML4_1M, + **_ML4_1M_DEP4_POOL_44, + pool_row=0, + capture_lens=[1, 1], + context_lens=[524289, 1048575], + ), + ), + Cell( + why="dep4 1M, decode graph batch 4: captured at 32768, replayed at 33 to 1048576", + spec=dict( + _ML4_1M, + **_ML4_1M_DEP4_POOL_44, + pool_row=17, + capture_lens=[32768] * 4, + context_lens=[33, 2049, 65537, 1048576], + ), + ), + Cell( + why="dep4 1M, decode graph batch 8: block edges at 128, 32768 and 1048576, pages shared", + spec=dict( + _ML4_1M, + **_ML4_1M_DEP4_POOL_44, + pool_row=29, + capture_lens=[100] * 8, + context_lens=[127, 128, 129, 32768, 32769, 262145, 1048320, 1048576], + ), + ), + Cell( + why="dep4 1M, decode graph batch 16: every edge from 1 key to 1048576, pages shared", + spec=dict( + _ML4_1M, + **_ML4_1M_DEP4_POOL_44, + pool_row=31, + capture_lens=[100 + 2000 * i for i in range(16)], + context_lens=[ + *(1, 64, 65, 255, 256, 257, 2049, 32768, 32769, 65537, 131073), + *(262145, 524289, 1048320, 1048575, 1048576), + ], + ), + ), + Cell( + why="dep4 1M, attention DP graph batch 2: one live row of 1048575 + a 1-key dummy", + spec=dict( + _ML4_1M, + **_ML4_1M_DEP4_POOL_44, + pool_row=3, + capture_lens=[1] * 2, + context_lens=[1048575, 1], + padding=1, + ), + ), + Cell( + why="dep4 1M, attention DP graph batch 4: one live row of 524289 + 3 rows of a 1-key dummy", + spec=dict( + _ML4_1M, + **_ML4_1M_DEP4_POOL_44, + pool_row=11, + capture_lens=[1] * 4, + context_lens=[524289] + [1] * 3, + padding=3, + ), + ), + Cell( + why="dep4 1M, attention DP graph batch 8: one live row of 1048576 + 7 rows of a 1-key dummy", + spec=dict( + _ML4_1M, + **_ML4_1M_DEP4_POOL_44, + pool_row=26, + capture_lens=[1] * 8, + context_lens=[1048576] + [1] * 7, + padding=7, + ), + ), + Cell( + why="dep4 1M, attention DP graph batch 16: one live row of 1048576 + 15 rows of a dummy", + spec=dict( + _ML4_1M, + **_ML4_1M_DEP4_POOL_44, + pool_row=40, + capture_lens=[1] * 16, + context_lens=[1048576] + [1] * 15, + padding=15, + ), + ), + Cell( + why="dep4 1M, mixed step, eager: 3 generation rows after 32765 chunk rows, to 1048576", + spec=dict( + _ML4_1M, + **_ML4_1M_DEP4_POOL_44, + pool_row=12, + table_rows=16, + first_seq=2, + q_offset=32765, + context_lens=[2049, 777, 1048576], + ), + ), + Cell( + why="dep4 1M, mixed step, eager: 15 generation rows after a 32753-row chunk, pages shared", + spec=dict( + _ML4_1M, + **_ML4_1M_DEP4_POOL_44, + pool_row=25, + table_rows=16, + first_seq=1, + q_offset=32753, + context_lens=[ + *(1048576, 1, 2049, 65537, 524289, 777, 33, 32769), + *(1048575, 2, 9217, 131073, 256, 257, 262145), + ], + ), + ), + Cell( + why="dep4 1M, engine V2 manager built at 1048576 (large), layer 71: 3 sequences", + spec=dict( + _ML4_1M, + layout="manager_v2", + pool_layers=44, + layer=71, + max_seq_len=1048576, + max_tokens=1081344, + max_num_tokens=32768, + table_width=32772, + table_rows=16, + large=True, + context_lens=[1048576, 9217, 1], + ), + ), + Cell( + why="dep4 1M, far pool V2 at 74157 blocks: a 1048576-key sequence's pages past 2^33", + spec=dict( + _ML4_1M, + layout="v2", + pool_layers=44, + table_width=32772, + pool_row=43, + tail=True, + num_blocks=74157, + past=2**33, + context_lens=[1048576, 777], + ), + ), + Cell( + why="NVFP4 1M tep8, decode graph batch 16 at the boot capture on the tep8 boot's " + "115016-block pool (large): KV 1048576 beside 15 of KV 2", + spec=dict( + _ML4_1M, + **_ML4_1M_TEP8_44, + num_blocks=_ML4_1M_TEP8_BOOT_BLOCKS, + pool_row=7, + capture_lens=[1048576] + [2] * 15, + large=True, + context_lens=[1048576] + [2] * 15, + ), + ), + Cell( + why="NVFP4 1M tep8, far pool V2 at a rank's bound, 147456 blocks (27.4 GB, large): " + "a 1048576-key sequence's pages past 2^34 bytes", + spec=dict( + _ML4_1M, + **_ML4_1M_TEP8_44, + num_blocks=_ML4_32K_8GPU_MAX_BLOCKS, + pool_row=43, + tail=True, + past=2**34, + large=True, + context_lens=[1048576, 777], + ), + ), + ) + + note = """ + DSpark adds next_n=4 at graph batches 1/2/4/8/16 and mixed-batch tails, + with max_context_len=32776 or 1048584, output stride 33024 or 1048832, + and the 44-owner V2 pool. The 1M cells use the 32772-column page table, + up to 64 verification rows, a 60-row tail after 32708 prompt rows, and + one live request beside 15 padding requests of four keys each. Boot + capture has one 1048583-key request; replay reaches 1048584 keys. + Replays change the lengths and keys, including padding at the low bound. + The 64-row output has 2113536 or 67125248 elements, below the 2^31-element + boundary at 65028 or 2048 rows for those strides. The 1M far-pool cell + reaches byte 27405582335 with all four causal ends; redirecting the last + page to poisoned or unrelated keys fails the accuracy gate. Pool bounds + are unchanged from below. + + Metadata consumed: `context_lens`, each sequence's KV length after this + step (cached keys plus this step's); `block_table`, the indexer K cache's + page indices, which the DSA metadata keeps as `indexer_k_cache_block_offsets` + (pool slot times the manager's `indexer_k_cache_page_scale`, one row of the + manager's `max_blocks_per_seq` entries per sequence: 292 for V2 and 288 for + V1 at 9216 tokens, 1028 and 1024 at 32768, 516 for V2 at 32768 with 64-token + pages, 32772 for V2 at 1048576, measured); and `schedule_meta`, + which is + `tensorrt_llm.deep_gemm.get_paged_mqa_logits_metadata` over + `context_lens.view(-1, 1)`, a 64-token schedule block and the device's SM + count -- how the DSA indexer builds its `scheduler_metadata_buffer`. The + schedule must come from the same `context_lens` the call passes. + + `kv_fused` is the view `get_indexer_k_cache_buffers(layer)` hands out. At + Mistral Large 4's geometry (page 32, 132 bytes per token, R indexer rows) the + two DSA cache managers give, measured: DSACacheManagerV2 (the engine + default) a contiguous [num_blocks * R - row, 32, 1, 132] view starting at + the row's first page, page indices scaled by R; DSACacheManager (V1) + [num_blocks, 32, 1, 132] with stride(0) R * 4224, page scale 1. The op + reads the block stride from the view, so both are cells, at R 44 and 72; + at 32768 and 1048576 keys, V2 at R 44, the NVFP4 target's, and at 32768 + keys also at 64-token pages ([num_blocks * 44 - row, 64, 1, 132], 8448-byte + pages). Its TMA loads take + the page index as an Int32 coordinate into a descriptor carrying that stride + (`FP8MQALogitsKernel` in + `attention/kernels/blackwell/paged_mqa_logits/fp8_paged_mqa_logits.py`); the + byte offsets the cells reach are under Hard limits. + + The output is a `[B * next_n, max_context_len]` view of the arena buffer + `cute_dsl_mqa_logits` (`memory_buffer_utils.get_memory_buffers`), whose + rows are `max_context_len` rounded up to 256 columns (9216, 32768 and + 1048576 stay as they are; 16 rows of 1048576 are 64 MiB). A + later call of the same shape or a smaller one returns the same storage; + under CUDA-graph capture the buffer is reserved, so the replay and any + eager call after it share it. For sequence b the kernel writes every column + below ceil(context_lens[b] / 256) * 256 of its rows (pinned at next_n 1, + on rows of 9216, 32768 and 1048576 columns): past the causal end those are + scores of whatever the cache holds there + (the slots after the last key, and the pages the block table names after + the sequence's last one); columns past that keep what the buffer held. + + Block-table entries after a sequence's last page are read too, up to the + end of its last 128-key block, so they must name pages the kernel can + load; the engine leaves stale in-range ones there, clamped at 0. They + change only columns past the causal end, and the cells point them at + poisoned pages to prove it. Out-of-range entries were not probed. + + At 1048576 tep4's delivered pool (74157 blocks) holds two full sequences, so + a step's sequences are at most about 2.37M keys together. One tep4 graph + cell (batch 8) replays at lengths summing to 2.6M keys, more pages than the + pool has: its sequences draw their pages from the same blocks in turn, a + shared page holding the later sequence's keys, which the call and the + reference both read. Its lengths, table and schedule are what it + certifies, not a pool state the engine reaches. + + At 1048576 a dep4 rank's pool (61996 slots at boot) holds one full + sequence and most of a second, so a step's sequences are at most about + 1.98M keys together. Three dep4 cells (graph + batches 8 and 16, the mixed step's 15 generation rows) score lengths + summing to 2.4M to 4.2M keys, more pages than their 65536-block pool has: + their sequences draw their pages from the same blocks in turn, a shared + page holding the later sequence's keys, which the call and the reference + both read. Their lengths, table and schedule are what they certify, not a + pool state the engine reaches. + + The op is SM100-family only: its wrapper checks `is_sm_100f()` and raises + ValueError elsewhere (only sm_103 was run). It does not validate its + inputs: the shapes are the caller's contract, and `is_valid` states the + ones a wrong value of which returns plausible logits or faults. + + Not certified: `next_n` other than 1 or 4, `num_epi_subtiles` other than 1, and + non-fp32 `epi_dtype` / `acc_dtype` / `output_dtype`. Measured on 16 + sequences: `epi_dtype=torch.float16` (an fp16 epilogue) lands 758 fp32 + ulps of the term magnitude from the reference, where the fp32 epilogue + lands at 0.13. Rows wider than 32768 keys are certified for DSpark's + 44-row V2 pools at 32776 and 1048584 (up to 64 verification rows) and the NVFP4 + target's 44-row V2 pools at + 1048576 (`_ML4_1M`, at most 16 rows: V2 views of 65536-, 74157-, 74183-, + 109568-, 115016- and 147456-block pools through a 32772-column table, and + the V2 manager built at 1048576); no `max_context_len` between 32776 and + 1048576 is a cell. At 1048576 the tep8 rank's pools are cells too: the + 115016 slots its boot measured (at max_seq_len 32768) and the 147456-block + bound. V1 views and 72-row pools only to 9216 keys, and 44-row V2 pools only to + 147456 blocks of 32 keys and 54784 of 64. 64-token pages are certified + only on the NVFP4 target's 44-row V2 pools, at 32768 keys. The tep8, tp8 + and dep8 bound itself is unmeasured: 147456 is computed from the weights + a rank holds. + + Hard limits (sm_103) + kv page offset page index x stride(0) bytes, the stride read from the view + boundary 2^31 / 2^32 / 2^33 / 2^34 bytes + probed at byte 4982833151, a pool's last (72 rows x 16384 + blocks, V1 and V2), 3425697791 (44 rows x 18432), and the + last bytes of 44-row V2 pools of 74157 blocks (13782523391, + the NVFP4 target's measured pool), 109568 blocks + (20363870207, `large`) and 147456 blocks (27405582335, the + eight-GPU bound, `large`), at 32768 keys; at 64-token + pages, 37078 blocks (13782337535) and 54784 (20363870207, + `large`); at 1048576 keys, the last bytes of pools of + 74183 blocks (13787355647, the tp4 boot pool) and 109568 + blocks (20363870207, `large`), and a sequence's 32768 + pages in the last blocks of the 74157-block pool, past 2^33, + and of the 147456-block eight-GPU bound (27405582335 again, + tep8, `large`), past 2^34; next_n=4 at 1048584 keys also + reaches byte 27405582335 in that bound + passed; 2^35, the next, takes 184873 blocks of 44 rows, + past the eight-GPU bound: not probed + block_table int32 page index, slot x 44 in a 44-row V2 pool + boundary 2^31 + largest certified 6488020 (V2, 44 rows x 147456 blocks); a + GB300 holds at most 70317459 pages of 4224 bytes: not reachable + kv_fused rows `kv_fused.shape[0]`, passed to the kernel as Int32 + boundary 2^31 + largest certified 6488021; not reachable, as above + logits B * next_n rows x the 256-aligned row; the row stride is a + 64-bit symbol of the compiled kernel (read, not probed) + boundary 2^31 elements = 233016 rows of 9216, 65536 of 32768, + 2048 of 1048576 or the DSpark stride 1048832 + 16 rows for Mistral Large 4 at 1048576 (2^24 elements), + 64 at DSpark's 32776/1048584 (stride 33024/1048832); + not reached + context_lens int32 key positions, at most max_context_len + boundary 2^31 + 1048584 (the 1M envelope with DSpark runtime headroom); + not reached + fp32 no index passes through fp32 (read, not probed); the page + indices certified and the key columns (to 1048583) stay + below 2^24 + 1 either way + """ + + def __call__( + self, + q: torch.Tensor, + kv_fused: torch.Tensor, + weights: torch.Tensor, + context_lens: torch.Tensor, + block_table: torch.Tensor, + schedule_meta: torch.Tensor, + max_context_len: int, + num_epi_subtiles: int = 1, + epi_dtype: torch.dtype = torch.float32, + acc_dtype: torch.dtype = torch.float32, + output_dtype: torch.dtype = torch.float32, + ) -> torch.Tensor: + return torch.ops.trtllm.cute_dsl_fp8_paged_mqa_logits( + q, + kv_fused, + weights, + context_lens, + block_table, + schedule_meta, + max_context_len, + num_epi_subtiles, + epi_dtype, + acc_dtype, + output_dtype, + ) + + @staticmethod + def unpack_cache(kv_fused: torch.Tensor, head_dim: int) -> tuple[torch.Tensor, torch.Tensor]: + """`(k [num_pages, page, D] uint8 codes, k_scale [num_pages, page] fp32)`. + + A page stores its tokens' data first, then their scales. The codes stay + uint8 so they can be gathered by page; view them as e4m3 afterwards. + """ + pages, page = kv_fused.shape[0], kv_fused.shape[1] + flat = kv_fused.view(torch.uint8).reshape(pages, -1) + k = flat[:, : page * head_dim].reshape(pages, page, head_dim) + k_scale = flat[:, page * head_dim : page * (head_dim + 4)].contiguous() + return k, k_scale.view(torch.float32).reshape(pages, page) + + def reference( + self, + q: torch.Tensor, + kv_fused: torch.Tensor, + weights: torch.Tensor, + context_lens: torch.Tensor, + block_table: torch.Tensor, + schedule_meta: torch.Tensor, + max_context_len: int, + num_epi_subtiles: int = 1, + epi_dtype: torch.dtype = torch.float32, + acc_dtype: torch.dtype = torch.float32, + output_dtype: torch.dtype = torch.float32, + ) -> torch.Tensor: + """The causal scores in fp64, rounded once; NaN where unspecified. + + Each sequence's pages are gathered from `kv_fused` first, so a strided + or offset view and a pool of any size cost only what the sequence + reads. `schedule_meta` and the three dtype/tiling knobs do not change + the value and are not read. + """ + batch, next_n, heads, head_dim = q.shape + page = kv_fused.shape[1] + out = torch.full( + (batch * next_n, max_context_len), float("nan"), dtype=torch.float64, device=q.device + ) + lens = context_lens.tolist() + for b in range(batch): + n = lens[b] + if n <= 0: + continue + pages = block_table[b, : (n + page - 1) // page].long() + k_pages, s_pages = self.unpack_cache(kv_fused[pages], head_dim) + codes = k_pages.reshape(-1, head_dim)[:n].contiguous() + k = codes.view(torch.float8_e4m3fn).float().double() + s = s_pages.reshape(-1)[:n].double() + for t in range(next_n): + row = b * next_n + t + end = n - next_n + t + 1 + if end <= 0: + continue + qd = q[b, t].float().double() + dots = (qd @ k[:end].t()).relu() + w = weights[row].double().unsqueeze(1) + out[row, :end] = (dots * w).sum(0) * s[:end] + return out.float() + + def is_valid( + self, + q: torch.Tensor, + kv_fused: torch.Tensor, + weights: torch.Tensor, + context_lens: torch.Tensor, + block_table: torch.Tensor, + schedule_meta: torch.Tensor, + max_context_len: int, + num_epi_subtiles: int = 1, + epi_dtype: torch.dtype = torch.float32, + acc_dtype: torch.dtype = torch.float32, + output_dtype: torch.dtype = torch.float32, + ) -> None: + # One schedule row per CTA plus one, and the op launches one CTA per + # SM. The schedule's producer takes its own SM count, so the two can + # disagree. Measured on 16 sequences of 9216 keys: a schedule built for + # twice the SMs leaves 69632 of the 147456 causal logits unwritten + # without raising (a light batch can fit the first SMs' partitions and + # come out right); one built for 8 fewer SMs faults the context with an + # illegal memory access. + expected = torch.cuda.get_device_properties(q.device).multi_processor_count + 1 + assert tuple(schedule_meta.shape) == (expected, 2), ( + f"schedule_meta must be [{expected}, 2] (SM count + 1 rows); got " + f"{tuple(schedule_meta.shape)}" + ) + # The output has `max_context_len` columns; the scores of a longer + # context's last keys land in the row's padding and are lost without + # an error. Reading the lengths is a host sync, which a CUDA-graph + # capture cannot take: under capture only the schedule is checked. + if context_lens.numel() and not torch.cuda.is_current_stream_capturing(): + assert int(context_lens.max()) <= max_context_len, ( + f"a context of {int(context_lens.max())} keys exceeds max_context_len " + f"{max_context_len}" + ) + + +cute_dsl_fp8_paged_mqa_logits = CuteDslFp8PagedMqaLogits() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/cute_dsl_gvr_topk_prefill.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/cute_dsl_gvr_topk_prefill.py new file mode 100644 index 000000000000..91e1ce42a6e6 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/cute_dsl_gvr_topk_prefill.py @@ -0,0 +1,378 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Prefill top-k of the DSA indexer, the self-sampling GVR engine: per query row, the best keys +of its window, in one streaming pass over the row.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch.kernels.blackwell.top_k import selfsampling_topk_run_prefill +from tensorrt_llm._torch.kernels.blackwell.top_k.gvr_topk_decode_self_sampling_host import WS_BYTES +from tensorrt_llm._torch.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: The DSA indexer of Mistral Large 4: 2048 positions per row. +_INDEXER = dict(top_k=2048) + +#: The bytes of scratch one call takes (`workspace`): the engine's per-device slab. +GVR_WORKSPACE_BYTES = WS_BYTES + + +#: The engine's row-count tiers (`_prefill_tier`): a launch of at most 148 rows (tier 0), of +#: 149..296 (tier 1) or of more (tier 2); the smallest row count of each. +_TIER_ROWS = (1, 149, 297) +#: The widths whose tier-0 launches the engine compiles apart (its power-of-two bucket of the +#: width, capped at 32768): (2048, 4096], (4096, 8192], (8192, 8448] (8449..16384 take tier 1's +#: plan at top_k 2048) and from 16385 on, one each. +_TIER0_WIDTHS = (4096, 8192, 8448, 32768) + + +def gvr_workspace(device: torch.device) -> torch.Tensor: + """A zeroed `GVR_WORKSPACE_BYTES` scratch slab for `cute_dsl_gvr_topk_prefill` on `device`. + + The kernel restores every word it consumes to zero before it returns, so one zeroing + serves every later call on the stream that owns the slab; calls on two streams at once + need a slab each.""" + return torch.zeros(GVR_WORKSPACE_BYTES // 4, dtype=torch.int32, device=device) + + +def gvr_warmup( + top_k: int, max_cols: int, max_elems: int, device: torch.device, workspace: torch.Tensor +) -> int: + """Compile now every engine launch a caller's `cute_dsl_gvr_topk_prefill` calls can reach, + for calls of at most `max_cols` columns whose `[rows, W]` blocks hold at most `max_elems` + scores (a logits budget), so that no served step compiles. Returns the warm calls made. + + The engine compiles on first use, about 2 s each on sm_103: one launcher per row-count tier + (and, at tier 0, per width bucket), and in its deterministic mode two variants of each, + chosen per launch by the row stride (a bitmap-assisted one up to 256 x the tier's block + size of columns, 65536 at tier 2, 131072 at tier 1 and 262144 at tier 0, a plain one past + it; `get_compiled` in `gvr_topk_decode_self_sampling.py`). So each tier is called once at a + narrow stride and once at the widest its calls reach: `max_cols` at tier 0 (one row), + `max_elems // rows` at tiers 1 and 2, within `max_cols` (a budget of 2**28 reaches tier 1's + 149 rows only at `max_cols` and below). Blocks of at most `top_k` columns need none + (`__call__` answers them without the engine). Each call scores zeros, `[rows, W]` within + `max(max_elems, max_cols)` elements, freed before the next; the calls run on the current + stream with `workspace`.""" + calls = {(1, w) for w in _TIER0_WIDTHS} + calls.add((1, max_cols)) + for rows in _TIER_ROWS[1:]: + calls.add((rows, 2 * top_k)) + calls.add((rows, min(max_elems // rows, max_cols))) + calls = sorted((r, w) for r, w in calls if top_k < w <= max_cols) + for rows, width in calls: + stride = (width + 255) // 256 * 256 + 256 + scores = torch.zeros(rows, stride, dtype=torch.float32, device=device) + starts = torch.zeros(rows, dtype=torch.int32, device=device) + ends = torch.full((rows,), width, dtype=torch.int32, device=device) + out = torch.empty(rows, top_k, dtype=torch.int32, device=device) + cute_dsl_gvr_topk_prefill(scores[:, :width], starts, ends, out, top_k, workspace) + del scores, starts, ends, out + torch.cuda.synchronize(device) + return len(calls) + + +class CuteDslGvrTopkPrefill(OpWrapper): + """Select each prefill row's `top_k` best keys inside its window, as window-local indices. + + The contract: `input_values` is `[rows, W]` fp32 scores + over one K block; row `r` sees columns `[row_starts[r], row_ends[r])`. With `len = + row_ends[r] - row_starts[r]`: when `len <= top_k`, `output_indices[r]` is `0..len-1` in + order, then -1 (a window with `len <= 0` is all -1); otherwise it holds the `top_k` + columns of the window with the largest scores, minus `row_starts[r]`, and no -1. + + One launch of the CuTe DSL self-sampling GVR top-k in its prefill form + (`selfsampling_topk_run_prefill`, `tensorrt_llm/_torch/kernels/blackwell/top_k/ + gvr_topk_decode_self_sampling_host.py`), the engine's `TopK` module's + `TopKImplementation.CUTE_DSL_GVR` prefill path, in its deterministic mode: one CTA per + row (three row-count tiers) samples the row, derives a ladder of candidate thresholds + from the sample's histogram and harvests the winners in one streaming pass. Exactness + rests on count-crossing invariants (the harvest checks the count above each rung), not + on the estimate. Deterministic: the selected set is the scores above the k-th score plus + the smallest columns among those equal to it, and the order is a fixed function of the + input. Selection only, fp32 compares throughout: the scores (`fp8_mqa_logits`) and the + mapping of local indices to pool rows are the caller's. + + `output_indices` (`[rows, top_k]` int32, contiguous, 16-byte aligned) is written in place + and returned; `workspace` is the caller's `gvr_workspace` slab, or None for the engine's + per-device one. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # The dep4 deployment's calls (dep8 alike), one per query tile of an indexer + # prefill block, on fp8_mqa_logits' row-padded output: at the 1M envelope + # (max_seq_len 1048576, chunked prefill in 32768-token chunks) its tiles are + # floor(2**26 / W) rows of a block of W keys, at max_seq_len 32768 + # floor(2**24 / W). `blocks` lists one call per K block; a block lists + # `(cached, new, rows_taken)` per prompt (`rows_taken` keeps that prompt's + # last rows, a row tile). The engine picks its plan by row count (tier 0 at + # up to 148 rows, 1 at 149..296, 2 above; tier 0 at widths of 8449..16384 + # takes tier 1's plan at k = 2048) and, at tier 0, by the power-of-two width + # bucket up to 32768; the cells drive each tier. `values` names the score pattern the test + # writes into the windows ("logits" scores them with fp8_mqa_logits over + # random e4m3 keys, as the target does). + CELLS: tuple[Cell, ...] = ( + Cell( + why="dep4 1M, the last 64-row tile of the 32nd chunk of a 1M prompt: tier 0, " + "bucket 32768, windows of 1048513..1048576 keys", + spec=dict(blocks=([(1015808, 32768, 64)],), values="distinct", **_INDEXER), + ), + Cell( + why="dep4 1M, the same tile scored by fp8_mqa_logits over random keys", + spec=dict(blocks=([(1015808, 32768, 64)],), values="logits", **_INDEXER), + ), + Cell( + why="dep4 1M, the 64-row tile with narrow scores: every key in one coarse bin", + spec=dict(blocks=([(1015808, 32768, 64)],), values="narrow", **_INDEXER), + ), + Cell( + why="dep4 1M, the 64-row tile with tied scores (4 values): the k-th score is " + "shared by about a quarter of each window", + spec=dict(blocks=([(1015808, 32768, 64)],), values="ties", **_INDEXER), + ), + Cell( + why="dep4 1M, a 15-row final chunk at 1048559 keys, alone: a short tile", + spec=dict(blocks=([(1048544, 15, None)],), values="distinct", **_INDEXER), + ), + Cell( + why="dep4 1M, the 128-row tile at 524288 keys: tier 0's widest row count", + spec=dict(blocks=([(491520, 32768, 128)],), values="logits", **_INDEXER), + ), + Cell( + why="dep4 1M, the 256-row tile at 262144 keys: tier 1", + spec=dict(blocks=([(229376, 32768, 256)],), values="logits", **_INDEXER), + ), + Cell( + why="dep4 1M, the 512-row tile at 131072 keys: tier 2", + spec=dict(blocks=([(98304, 32768, 512)],), values="distinct", **_INDEXER), + ), + Cell( + why="dep4 1M, the 1024-row tile at 65536 keys, windows crossing 2**16", + spec=dict(blocks=([(32768, 32768, 1024)],), values="spread", **_INDEXER), + ), + Cell( + why="dep4 1M, the first 2048-row tile of a fresh 32768-token chunk: windows of " + "1..2048 keys, every row short (identity, then -1)", + spec=dict(blocks=([(0, 2048, None)],), values="distinct", **_INDEXER), + ), + Cell( + why="dep4 1M, the second 2048-row tile of a fresh chunk: windows of 2049..4096, " + "the first ranked rows, a 4096-column block (tier 2)", + spec=dict(blocks=([(0, 4096, 2048)],), values="distinct", **_INDEXER), + ), + Cell( + why="dep4 1M, the last 2048-row tile of a fresh 32768-token chunk", + spec=dict(blocks=([(0, 32768, 2048)],), values="logits", **_INDEXER), + ), + Cell( + why="dep4 32K, the last 512-row tile of a whole 32768-token prompt (2**24)", + spec=dict(blocks=([(0, 32768, 512)],), values="distinct", **_INDEXER), + ), + Cell( + why="dep4 32K, 16 packed prompts in one 32768-column block, the last 142 rows of " + "the 8193-token prompt and the 882 of the last one: windows starting at 23693 and " + "31886", + spec=dict( + blocks=( + [ + (0, n, 0) + for n in ( + 1, + 7, + 33, + 100, + 255, + 257, + 513, + 1023, + 1025, + 2047, + 2049, + 4095, + 4097, + 8191, + ) + ] + + [(0, 8193, 142), (0, 882, None)], + ), + values="distinct", + **_INDEXER, + ), # fmt: skip + ), + Cell( + why="dep4 1M, 16 packed chunk tails in one 1048576-key block, its last 64 rows: " + "four requests' windows starting at multiples of 65536", + spec=dict( + blocks=([(65520, 16, 0)] * 12 + [(65520, 16, None)] * 4,), + values="distinct", + **_INDEXER, + ), + ), + Cell( + why="dep4 1M, a short tail beside a fresh prompt in one block: rows of a " + "1048400-key window and of windows of 1..155 keys in one call", + spec=dict( + blocks=([(1048400, 100, 40), (0, 155, None)],), values="distinct", **_INDEXER + ), + ), + # The tep4 deployment's calls (tep8 alike): its 2**28-element tiles at the 1M envelope, + # cut from the first row of the rank's share of a chunk under the engine's query split, + # and the pieces of a packed chunk cut per request past max_seq_len. + Cell( + why="tep4 1M, a q-split share's 256-row tile at 1048576 keys: tier 1", + spec=dict(blocks=([(1015808, 32768, 256)],), values="logits", **_INDEXER), + ), + Cell( + why="tep4 1M, q-split rank 0's 70-row last tile at 1024000 keys: tier 0", + spec=dict(blocks=([(1015808, 8192, 70)],), values="distinct", **_INDEXER), + ), + Cell( + why="tep4 1M, q-split rank 1's 260-row tile at 1032192 keys, narrow scores", + spec=dict(blocks=([(1015808, 16384, 260)],), values="narrow", **_INDEXER), + ), + Cell( + why="tep4 1M, a q-split share's 512-row tile at 524288 keys: tier 2", + spec=dict(blocks=([(491520, 32768, 512)],), values="logits", **_INDEXER), + ), + Cell( + why="tep4 1M, q-split rank 3's whole 8192-row share of a first chunk at 32768 keys", + spec=dict(blocks=([(0, 32768, 8192)],), values="spread", **_INDEXER), + ), + Cell( + why="tep8 1M, q-split rank 7's whole 4096-row share of a first chunk at 32768 keys", + spec=dict(blocks=([(0, 32768, 4096)],), values="distinct", **_INDEXER), + ), + Cell( + why="tep4 1M, an unsplit 8191-row tail chunk's last 255 rows at 1048576 keys", + spec=dict(blocks=([(1040385, 8191, 255)],), values="distinct", **_INDEXER), + ), + Cell( + why="tep4 1M, 16 packed chunk tails in one 983296-key block, 256 rows (2**28)", + spec=dict(blocks=([(61440, 16, None)] * 16,), values="distinct", **_INDEXER), + ), + Cell( + why="tep4 1M, the packed-chunk cut: a ~1M request's 100-token tail alone, 100 rows " + "against its own 1048500 keys", + spec=dict(blocks=([(1048400, 100, None)],), values="ties", **_INDEXER), + ), + ) + + note = """ + Metadata consumed: none. The windows are the caller's: an indexer prefill + chunk's `cu_seqlen_ks` / `cu_seqlen_ke`, sliced to the row tile, in the K + block's column coordinates. The engine reads no device value on the host + and trusts `0 <= row_starts <= row_ends <= W`; `is_valid` checks it. + + Preconditions, each raised by the engine's host checks: fp32 + `input_values` with unit column stride, a row stride that is a multiple of + 4 and a 16-byte aligned base (fp8_mqa_logits' row-padded view is read in + place, its stride align(W + 256, 256)); int32, contiguous, 1-D + `row_starts` / `row_ends` of `rows` entries; `output_indices` int32, + contiguous, `[rows, k]` with k a multiple of 4, 16-byte aligned (a row + slice of a `[T, k]` buffer is). `top_k` is not read: k is + `output_indices.shape[1]`, which `is_valid` holds to `top_k`. A call with + zero rows returns without a launch. + + The engine compiles one launcher per (tier, k, width bucket at tier 0, + deterministic) on the first call that needs it, and in deterministic mode + two variants of each by the call's row stride, and caches them for the + process, about 2 s per compile on sm_103 (measured: 4.4 s for the first, + 2.1 to 2.6 s for each later one); it refuses to compile under CUDA-graph + capture. Prefill steps are eager; `gvr_warmup` compiles every launch a + caller's tiles reach ahead of serving. A block of at most `top_k` columns + does not reach the engine: every row of it is short, and `__call__` + writes the identity rows itself (the `TopK` module routes such blocks + away from the engine too). + """ + + def __call__( + self, + input_values: torch.Tensor, + row_starts: torch.Tensor, + row_ends: torch.Tensor, + output_indices: torch.Tensor, + top_k: int, + workspace: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + if input_values.shape[1] <= top_k: + # Every window lies inside the block, so every row is short: the identity, then -1. + ident = torch.arange(top_k, dtype=torch.int32, device=output_indices.device) + empty = torch.full((), -1, dtype=torch.int32, device=output_indices.device) + lengths = (row_ends - row_starts)[:, None] + return torch.where(ident < lengths, ident, empty, out=output_indices) + selfsampling_topk_run_prefill( + input_values, + row_starts, + row_ends, + output_indices, + workspace=workspace, + deterministic=True, + ) + return output_indices + + def reference( + self, + input_values: torch.Tensor, + row_starts: torch.Tensor, + row_ends: torch.Tensor, + output_indices: torch.Tensor, + top_k: int, + workspace: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + """A fresh `[rows, top_k]` int32 selection. + + `output_indices` and `workspace` are not read. A long row holds `torch.topk`'s + winners in descending score order; the engine promises its own fixed order and keeps + the lowest columns among keys tied at the k-th score, so the set is exact only on + rows without such a tie. + """ + rows, width = input_values.shape + device = input_values.device + starts = row_starts.long()[:, None] + ends = row_ends.long()[:, None] + cols = torch.arange(width, device=device) + inside = (cols >= starts) & (cols < ends) + scores = torch.where(inside, input_values.float(), float("-inf")) + out = torch.full((rows, top_k), -1, dtype=torch.int64, device=device) + ranked = min(top_k, width) + if ranked: + out[:, :ranked] = scores.topk(ranked, dim=1).indices - starts + lengths = (ends - starts).squeeze(1) + short = lengths <= top_k + ident = torch.arange(top_k, device=device) + out[short] = torch.where(ident < lengths[short, None], ident, -1) + return out.to(torch.int32) + + def is_valid( + self, + input_values: torch.Tensor, + row_starts: torch.Tensor, + row_ends: torch.Tensor, + output_indices: torch.Tensor, + top_k: int, + workspace: Optional[torch.Tensor] = None, + ) -> None: + # k is the output's width: a wider buffer selects more than top_k per row. + assert output_indices.shape[1] == top_k, ( + f"output_indices must be exactly top_k = {top_k} wide; got " + f"{tuple(output_indices.shape)}, whose width the engine selects" + ) + # The engine trusts the windows: a start below zero or an end past W reads another + # row's scores and ranks them as this row's. + if row_starts.numel(): + assert int(row_starts.min()) >= 0, "a window starts before its row" + assert bool((row_starts <= row_ends).all()), "a window ends before it starts" + assert int(row_ends.max()) <= input_values.shape[1], ( + f"a window ends past the {input_values.shape[1]}-column K block" + ) + if workspace is not None: + assert workspace.numel() * workspace.element_size() >= GVR_WORKSPACE_BYTES, ( + f"workspace holds {workspace.numel() * workspace.element_size()} bytes; the " + f"engine needs {GVR_WORKSPACE_BYTES}" + ) + + +cute_dsl_gvr_topk_prefill = CuteDslGvrTopkPrefill() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/cute_dsl_indexer_k_cache_update.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/cute_dsl_indexer_k_cache_update.py new file mode 100644 index 000000000000..3e3b1cde156a --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/cute_dsl_indexer_k_cache_update.py @@ -0,0 +1,1064 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Norm, rope, fp8-pack and append the DSA indexer keys to the paged K cache, in one launch.""" + +from __future__ import annotations + +from collections.abc import Callable, Mapping +from dataclasses import dataclass + +import torch + +from tensorrt_llm._torch.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.kernels import ( + _indexer_k_cache_update_kernel as _kernel, +) + +_HEAD_DIM = 128 +_ROPE_DIM = 64 +_HALF = _ROPE_DIM // 2 +_SCALE_BYTES = 4 +#: The kernel's LayerNorm epsilon, a compile-time constant (not a parameter). +NORM_EPS = 1.0e-6 +_E4M3_MAX = 448.0 +_MIN_AMAX = 1.0e-12 + +#: The two launch shapes the kernel's own wrapper picks between: 2 rows per CTA +#: up to `_SMALL_TOKEN_THRESHOLD` tokens, 8 above. +_ROWS_PER_BLOCK = (_kernel._SMALL_TOKEN_ROWS_PER_BLOCK, _kernel._FUSED_CAT_SCATTER_ROWS_PER_BLOCK) + + +@dataclass(frozen=True) +class CompiledIndexerKCacheUpdate: + """One launcher per (rows per CTA, vectorised input loads), and the flags they share.""" + + fns: Mapping[tuple[int, bool], Callable] + is_neox: bool + cache_contiguous: bool + + +def build(device_id: int, *, is_neox: bool, cache_contiguous: bool) -> CompiledIndexerKCacheUpdate: + """Compile the four launchers a step can need: 2 or 8 rows per CTA, scalar or 8-byte loads. + + `cache_contiguous` must describe every cache view the launchers will write: + the contiguous variant addresses a strided per-layer view as if it were + dense. Four compiles of a few seconds each, cached per process by the kernel + module; allocates nothing. ue8m0 scales and the PDL launch are fixed. + """ + fns = {} + for rows_per_block in _ROWS_PER_BLOCK: + for vectorized in (False, True): + fns[(rows_per_block, vectorized)] = ( + _kernel.compile_indexer_k_norm_rope_fp8_cache_update( + device_id=device_id, + use_ue8m0=True, + is_neox=bool(is_neox), + rows_per_block=rows_per_block, + vectorized_loads=vectorized, + cache_contiguous=bool(cache_contiguous), + ) + ) + return CompiledIndexerKCacheUpdate( + fns=fns, is_neox=bool(is_neox), cache_contiguous=bool(cache_contiguous) + ) + + +#: What every Mistral Large 4 cell shares: interleaved rope over its YaRN + LeRoPE +#: table of max_seq_len 9216 rows, a 44-layer pool of 64 blocks (layer 3 written), +#: 8-byte-aligned keys, no padding rows. +_ML4 = dict(pool_layers=44, blocks=64, pad_rows=0, misalign=False, is_neox=False, table="yarn") +#: The Mistral Large 4 NVFP4 target's: the same, over its YaRN factor 2048 table of +#: max_seq_len 32768 rows. `layout` says which view the cache is: absent, the +#: strided per-layer view above (the V1 DSA cache manager's); "v2", the view +#: the engine's default `DSACacheManagerV2` hands out, rebuilt: contiguous +#: `[blocks * 44 - layer, 32, 1, 132]` from the layer's first page, a block's +#: page at `block * 44` (see `note`); "manager_v2", that view and the slot +#: mappings taken from a real manager whose quota is `blocks` pages, with one +#: request per `requests` entry (that many new tokens after `cached` cached +#: ones) and the view of model layer `model_layer`. +_ML4_32K = dict(_ML4, max_seq_len=32768, yarn_factor=2048.0) +_ML4_32K_V2 = dict(_ML4_32K, layout="v2") +_ML4_32K_MGR = dict(_ML4_32K, layout="manager_v2", blocks=1100) +#: The largest pool one rank of the NVFP4 target can hold, a deliberate +#: overestimate: 276.62 GiB per GB300 less 143.24 GiB of weights leaves room +#: for 94659 blocks of 32 tokens at the 47280 bytes a token takes in the 72 +#: latent and 44 indexer rows; the V2 manager's slot range runs about 5% past +#: its quota, and 110592 blocks clear both. +_ML4_NVFP4_MAX_BLOCKS = 110592 +#: The same bound for one rank of the NVFP4 eight-GPU targets, again an +#: overestimate: a tep8 or tp8 rank holds 73.61 GiB of weights and a dep8 rank +#: 89.29 GiB (computed from the shard headers), so 203.01 and 187.33 GiB of a +#: GB300 are left, device memory for 144075 and 132949 V2 slots at 1512960 bytes +#: a slot (32 tokens of 47280 bytes). The manager backs every slot of its range, +#: past its quota or not, with device memory, so 147456 blocks clear any pool a +#: tep8, tp8 or dep8 rank can allocate. +_ML4_NVFP4_8GPU_MAX_BLOCKS = 147456 +#: Max_batch_size 16 requests packed to 32768 new tokens, lengths on either side +#: of the page (32), the 2-row launcher's last count (1024) and index_topk (2048). +_ML4_32K_PACKED = (1, 31, 32, 33, 63, 64, 65, 1023, 1024, 1025, 2047, 2049, 4095, 4097, 8191, 8928) +#: The tp4 deployment at the checkpoint's own context: max_seq_len 1048576 with chunked +#: prefill on and max_num_tokens 32768 (the chunk), over its YaRN factor 2048 table of +#: 1048576 rows with its values rounded to bf16 (`table_dtype`), as the target's rope +#: tables hold them. `requests` and `cached` place a step's rows on every layout: +#: request i's `requests[i]` rows at positions `cached[i]..cached[i] + requests[i] - 1`. +#: The manager cells build it at max_num_tokens 32768 apart from max_seq_len. +#: Named `_TP4` (tp4's `_ML4_1M*`; only tp4's cells use them): tep4's and dep4's `_ML4_1M`, +#: `_ML4_1M_V2` and `_ML4_1M_MGR` hold no `table_dtype` (an fp32 table) and no +#: `max_num_tokens` key, and tep4's `_ML4_1M_PACKED*` and `_ML4_1M_MIXED_CACHED` are other +#: steps (`_ML4_1M_MIXED_TP4` equals tep4's `_ML4_1M_MIXED`, and keeps its partner's suffix). +_ML4_1M_TP4 = dict(_ML4, max_seq_len=1048576, yarn_factor=2048.0, table_dtype="bfloat16") +_ML4_1M_V2_TP4 = dict(_ML4_1M_TP4, layout="v2") +_ML4_1M_MGR_TP4 = dict(_ML4_1M_TP4, layout="manager_v2", max_num_tokens=32768) +#: One chunk packing four requests: the last 16384 rows of a 1048576-token request, +#: 8192 rows past a 524288-token prefix, a first chunk and a short one past 2048. +_ML4_1M_PACKED_TP4 = (16384, 8192, 4096, 4096) +_ML4_1M_PACKED_CACHED_TP4 = (1032192, 524288, 0, 2048) +#: A mixed chunk step of max_batch_size 16: three context chunks (one ending at position +#: 1048575) ahead of 13 decode rows across the 1M range. +_ML4_1M_MIXED_TP4 = (20000, 9000, 3755, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1) +_ML4_1M_MIXED_CACHED_TP4 = ( + 1028576, + 524288, + 0, + 1048575, + 1048544, + 1015808, + 1015807, + 524288, + 262144, + 131072, + 65536, + 32768, + 32767, + 16384, + 2048, + 1, +) +#: The NVFP4 target at the checkpoint's own context: max_seq_len 1048576 with +#: chunked prefill (max_num_tokens 32768), over the 1048576-row YaRN factor 2048 +#: table (or the 1048576-row identity table). On a "v2" view as on the manager, +#: `requests` and `cached` put each request's new rows at positions +#: cached..cached+new-1. tep4's cells use these `_ML4_1M*`, and dep4's `_ML4_1M`, +#: `_ML4_1M_V2` and `_ML4_1M_MGR` (tp4's are the `_TP4` ones above: a bf16-rounded +#: table, the manager's max_num_tokens named in the spec). +_ML4_1M = dict(_ML4, max_seq_len=1048576, yarn_factor=2048.0) +_ML4_1M_V2 = dict(_ML4_1M, layout="v2") +_ML4_1M_MGR = dict(_ML4_1M, layout="manager_v2") +#: Max_batch_size 16 chunk tails packed to 32768 rows (the new tokens, then each +#: one's cached prefix): a decode-sized tail at position 1048575, lengths across +#: the page, launcher and index_topk edges, prefixes across 32768 and 2^16..2^19. +_ML4_1M_PACKED = (1, 7, 31, 32, 33, 100, 255, 257, 513, 1023, 1025, 2047, 2049, 8191, 8193, 9011) +_ML4_1M_PACKED_CACHED = ( + 1048575, 0, 1, 31, 32, 33, 2047, 2048, 2049, 32767, 65535, 131072, 262143, 500000, 0, 0, +) # fmt: skip +#: A mixed step at 1M: three chunks (32755 rows) ahead of 13 decode rows whose +#: prefixes reach 1048575. +_ML4_1M_MIXED = (20000, 9000, 3755) + (1,) * 13 +_ML4_1M_MIXED_CACHED = ( + 1028576, 500000, 0, 1048575, 1048544, 524287, 131071, 32768, 32767, 2048, 2047, 64, 33, 32, + 2, 1, +) # fmt: skip +#: The dep4 deployment's mixed step at 1M: three chunks (32755 rows; one fresh, two +#: behind prefixes of 100000 and 65536) ahead of 13 decode rows whose prefixes +#: run from 1048575 down across 2^19..2^15, index_topk and the page to 1. +_ML4_1M_DEP4_MIXED = (20000, 9000, 3755) + (1,) * 13 +_ML4_1M_DEP4_MIXED_CACHED = ( + 100000, 0, 65536, 1048575, 524288, 262144, 131072, 65536, 32768, 32767, 2048, 2047, 33, 32, + 2, 1, +) # fmt: skip +#: The dep4 deployment's long-family decode graphs at 1M (`boot_capture` cells). The +#: engine captures each batch b of 1/2/4/8/16 on its CUDA-graph warm-up batch: +#: one dummy of max_seq_len - 1 = 1048575 keys (its new key at position +#: 1048574) ahead of b - 1 one-key fillers, each a request of its own (position +#: 0 in a page of its own). Attention DP then pads every rank to the largest +#: rank's batch with one padding dummy repeated (`pad_rows` rows of one request: +#: position 0, one slot pair, one key), so a rank replays the graph with its own +#: live rows ahead of them, an idle rank with its one-key attention-DP dummy. +#: The slot mappings are the metadata's max_num_tokens buffers. +_ML4_1M_DEP4_BOOT = dict( + _ML4_1M_V2, blocks=1100, graph=True, top_position=True, slot_rows=32768, boot_capture=True +) + + +class CuteDslIndexerKCacheUpdate(OpWrapper): + """Turn this step's pre-norm indexer keys into indexer K cache entries, in place. + + For each of the first `num_tokens` rows of `indexer_k` (`[T, 128]` bf16): + + 1. LayerNorm over the row (`eps = 1e-6`, fp32 `norm_weight` / `norm_bias`), + rounded to bf16; + 2. rope on the first 64 dims at `position_ids[t]`, from + `rotary_cos_sin[pos] = [[cos(32)], [sin(32)]]`, rounded to bf16: NeoX + halves (`is_neox=True`: dims `i` and `i + 32` rotate by column `i`) or + interleaved pairs (`is_neox=False`: dims `2i` and `2i + 1` rotate by + column `i`). The last 64 dims are not rotated; + 3. `scale = max(amax, 1e-12) / 448` rounded up to a power of two, and the + e4m3 codes `row / scale`; + 4. the 128 code bytes go to flat byte `slot_mapping_fp8[t]` of `k_cache` + and the fp32 scale to flat byte `slot_mapping_scale[t]`, where "flat" + is the logical row-major index of `k_cache`'s `[blocks, page, 1, D]` + shape, mapped through its strides. A row with either slot negative is + skipped. + + `k_cache` is written in place and returned. Nothing else is written. + + The call is `indexer_k_norm_rope_fp8_cache_update`, a CuTe DSL kernel's + Python launcher (no `torch.ops` registration), launched once. It lives in + the family's private kernel module `_indexer_k_cache_update_kernel`. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # The Mistral Large 4 FP8 target writes its 44 full indexer layers: decode + # batches 1..16 (captured at 1, 2, 4, 8, 16), prefill up to its max_num_tokens + # 9216 at positions 0..9215, interleaved rope, and its 18 NoPE indexer layers + # with an identity table. The NVFP4 target (`_ML4_32K` cells) is the same forward at + # max_seq_len = max_num_tokens = 32768 with chunked prefill off: a context + # step of up to 32768 rows (one request, or up to 16), decode rows at + # positions up to 32767, graphs 1/2/4/8/16 padded. Its tep8, tp8 and dep8 + # deployments make the same call on every rank (the indexer is whole on every + # rank: replicated at tep8 and tp8, data parallel at dep8), on a larger + # pool: a tep8 or tp8 rank holds 73.61 GiB of weights, a dep8 rank 89.29 + # GiB, a tep4 rank 143.24 GiB. In the cells without a + # `layout` the K pool is one block-major allocation of every indexer layer, + # so a layer's cache is a strided view whose blocks are dense; the engine's + # default manager hands out a contiguous view instead (`note`), and the + # target picks the launchers by `k_cache.is_contiguous()`, so its calls + # run the contiguous launchers. 32 tokens per page, 132 bytes per token. + # The NVFP4 tp4 deployment at max_seq_len 1048576 (`_ML4_1M_TP4` cells) is that forward + # with chunked prefill on: a step still holds at most 32768 rows + # (max_num_tokens, the chunk), but a context chunk's rows sit at positions + # cached..cached+new-1 after a cached prefix of up to 1048576 - new (one + # request, or up to 16 packed, decode rows behind them), decode rows sit + # anywhere up to 1048575, and the tables have 1048576 rows. Its keys and + # its table rows are the same per row at every length, so the `_ML4_32K` + # cells carry over what does not depend on the position. + # The tep4 and tep8 NVFP4 targets at the checkpoint's own context (`_ML4_1M` + # cells; the indexer is replicated at both, so the call is the same) run + # max_seq_len 1048576 with chunked prefill on and max_num_tokens 32768: + # a context step is still at most 32768 rows, but every chunk after a + # request's first sits behind its cached prefix (rows at positions + # cached..cached+new-1, up to 1048575), decode rows reach position 1048575, + # and both tables have 1048576 rows. Its manager cells hold a whole 1M + # request, so their pools pass the pre-merge budget and are `large`; the + # `v2` cells at the same positions are their pre-merge share. + # The dep4 NVFP4 target at the checkpoint's own context (`_ML4_1M` cells) + # runs max_seq_len 1048576 with chunked prefill on and max_num_tokens 32768 + # per rank, attention data parallel (each rank appends its own requests' + # keys to its own pool): a context step is still at most 32768 rows, but + # every chunk after a request's first sits behind its cached prefix (rows + # at positions cached..cached+new-1, up to 1048575), decode rows reach + # position 1048575, and both tables have 1048576 rows. Its decode graphs + # are captured at boot on the warm-up batch and replayed behind attention + # DP's padding dummy (`boot_capture`, `_ML4_1M_DEP4_BOOT`). Its manager cells + # hold a whole 1M request, so their pools pass the pre-merge budget and are + # `large` (device peaks 58.42 GiB at a 33792-page quota, 124.95 GiB at + # 73728); the `v2` cells at the same positions are their pre-merge share. + # A cell's spec is one of the bases above with its own keys over it. + CELLS: tuple[Cell, ...] = ( + Cell( + why="interleaved rope, decode batch 1 (captured): one row in a 2-row CTA", + spec=dict(_ML4, tokens=1, graph=True), + ), + Cell( + why="interleaved rope, decode batch 2 (captured)", + spec=dict(_ML4, tokens=2, graph=True), + ), + Cell( + why="interleaved rope, decode batch 3: an odd batch of the unpadded 1..16 captures", + spec=dict(_ML4, tokens=3, graph=True), + ), + Cell( + why="interleaved rope, decode batch 4 (captured)", + spec=dict(_ML4, tokens=4, graph=True), + ), + Cell( + why="interleaved rope, decode batch 8 (captured), slot buffers sized for batch 16", + spec=dict(_ML4, tokens=8, graph=True, slot_rows=16), + ), + Cell( + why="interleaved rope, decode batch 16 (captured), its last 3 rows padding at slot -1", + spec=dict(_ML4, tokens=16, graph=True, pad_rows=3), + ), + Cell( + why="a NoPE indexer layer at decode batch 16 (captured): identity table, no rotation", + spec=dict(_ML4, tokens=16, graph=True, table="identity"), + ), + Cell( + why="interleaved prefill at 1024 rows: the last count of the 2-row-per-CTA launcher", + spec=dict(_ML4, tokens=1024), + ), + Cell( + why="interleaved prefill at 1025 rows: the first count of the 8-row-per-CTA launcher", + spec=dict(_ML4, tokens=1025), + ), + Cell( + why="interleaved prefill at max_num_tokens 9216 rows, positions 0..9215", + spec=dict(_ML4, tokens=9216, blocks=320), + ), + Cell( + why="a NoPE indexer layer at 9216 rows: identity table over positions 0..9215", + spec=dict(_ML4, tokens=9216, blocks=320, table="identity"), + ), + Cell( + why="interleaved keys not 8-byte aligned: the scalar-load launcher", + spec=dict(_ML4, tokens=33, misalign=True), + ), + Cell( + why="a 72-layer pool (no indexer layer mask), rows in its last layer", + spec=dict(_ML4, tokens=16, pool_layers=72, layer=71), + ), + Cell( + why="the largest 44-layer pool a rank can hold (18432 blocks, 3.4 GB; 9.6 GiB peak): " + "rows in the last block of the last layer, past 2^31 bytes", + spec=dict(_ML4, tokens=16, blocks=18432, layer=43, far_end=True), + ), + Cell( + why="32768-row prefill on the strided view, positions 0..32767, YaRN factor 2048: " + "the strided 8-row launcher at the NVFP4 target's max_num_tokens", + spec=dict(_ML4_32K, tokens=32768, blocks=1100), + ), + Cell( + why="32768-row NoPE layer on the strided view: identity table over positions 0..32767", + spec=dict(_ML4_32K, tokens=32768, blocks=1100, table="identity"), + ), + Cell( + why="32768 rows, engine V2 manager, layer 68 (pool row 42): one request at positions " + "0..32767, the contiguous 8-row launcher", + spec=dict(_ML4_32K_MGR, tokens=32768, requests=(32768,), model_layer=68), + ), + Cell( + why="32768 rows, engine V2 manager, NoPE layer 71 (the last pool row, 43): identity " + "table over positions 0..32767", + spec=dict( + _ML4_32K_MGR, tokens=32768, requests=(32768,), model_layer=71, table="identity" + ), + ), + Cell( + why="32768 rows, engine V2 manager, layer 0: max_batch_size 16 requests packed, " + "lengths across page, launcher and index_topk edges", + spec=dict(_ML4_32K_MGR, tokens=32768, requests=_ML4_32K_PACKED, model_layer=0), + ), + Cell( + why="32768 rows, engine V2 manager, layer 20: a mixed step, 3 contexts (32755 rows) " + "ahead of 13 decode rows at positions up to 32767", + spec=dict( + _ML4_32K_MGR, + tokens=32768, + blocks=4096, + requests=(20000, 9000, 3755) + (1,) * 13, + cached=(0, 0, 0, 32767, 32736, 16383, 9215, 2048, 2047, 1024, 64, 33, 32, 31, 2, 1), + model_layer=20, + ), + ), + Cell( + why="32768-table decode batch 1 (captured), V2 view, last row: position 32767, " + "the contiguous 2-row launcher", + spec=dict(_ML4_32K_V2, tokens=1, graph=True, top_position=True, layer=43), + ), + Cell( + why="32768-table decode batch 2 (captured), V2 view, first row", + spec=dict(_ML4_32K_V2, tokens=2, graph=True, top_position=True, layer=0), + ), + Cell( + why="32768-table decode batch 4 (captured), V2 view", + spec=dict(_ML4_32K_V2, tokens=4, graph=True, top_position=True, layer=17), + ), + Cell( + why="32768-table decode batch 8 (captured), V2 view", + spec=dict(_ML4_32K_V2, tokens=8, graph=True, top_position=True, layer=30), + ), + Cell( + why="32768-table decode batch 16 (captured), V2 view, slot buffers of max_num_tokens " + "32768, its last 3 rows padding at slot -1", + spec=dict( + _ML4_32K_V2, + tokens=16, + blocks=1100, + graph=True, + top_position=True, + pad_rows=3, + slot_rows=32768, + layer=42, + ), + ), + Cell( + why="32768-table NoPE decode batch 16 (captured), V2 view: identity table", + spec=dict( + _ML4_32K_V2, tokens=16, graph=True, top_position=True, table="identity", layer=43 + ), + ), + Cell( + why="32768 rows over a 24576-block V2 view (4.6 GB): slots on both " + "sides of 2^31 and 2^32 bytes", + spec=dict(_ML4_32K_V2, tokens=32768, blocks=24576, layer=43), + ), + Cell( + why="32768 rows in the top blocks of the largest V2 pool a rank can hold (110592 " + "blocks, 20.6 GB; 39.6 GiB peak), descending as V2 hands them out: past 2^34 bytes", + spec=dict( + _ML4_32K_V2, + tokens=32768, + blocks=_ML4_NVFP4_MAX_BLOCKS, + layer=43, + far_end=True, + large=True, + ), + ), + Cell( + why="a tep8, tp8 or dep8 rank's largest V2 pool, 147456 blocks (27.4 GB; 51.4 GiB " + "peak): 32768 rows in its top blocks, descending as V2 hands them out, at pool row 43", + spec=dict( + _ML4_32K_V2, + tokens=32768, + blocks=_ML4_NVFP4_8GPU_MAX_BLOCKS, + layer=43, + far_end=True, + large=True, + ), + ), + Cell( + why="1M manager chunk, layer 68 (pool row 42): one request's last 32768 rows at " + "positions 1015808..1048575 over a 33900-block V2 quota, the contiguous 8-row " + "launcher (large: the quota's pool)", + spec=dict( + _ML4_1M_MGR_TP4, + tokens=32768, + blocks=33900, + requests=(32768,), + cached=(1015808,), + model_layer=68, + large=True, + ), + ), + Cell( + why="1M manager chunk, NoPE layer 71 (pool row 43): identity table over positions " + "1015808..1048575 (large: the quota's pool)", + spec=dict( + _ML4_1M_MGR_TP4, + tokens=32768, + blocks=33900, + requests=(32768,), + cached=(1015808,), + model_layer=71, + table="identity", + large=True, + ), + ), + Cell( + why="1M manager packed chunk, layer 0: 4 requests, to positions 1048575 and 532479 " + "beside a first chunk and a short one, over a 53000-block quota (large: its pool)", + spec=dict( + _ML4_1M_MGR_TP4, + tokens=32768, + blocks=53000, + requests=_ML4_1M_PACKED_TP4, + cached=_ML4_1M_PACKED_CACHED_TP4, + model_layer=0, + large=True, + ), + ), + Cell( + why="1M chunk on a V2 view, layer 42: 32768 rows at positions 1015808..1048575, the " + "contiguous 8-row launcher", + spec=dict( + _ML4_1M_V2_TP4, + tokens=32768, + blocks=1100, + requests=(32768,), + cached=(1015808,), + layer=42, + ), + ), + Cell( + why="1M chunk on a V2 view, NoPE layer 43: identity table over positions " + "1015808..1048575", + spec=dict( + _ML4_1M_V2_TP4, + tokens=32768, + blocks=1100, + requests=(32768,), + cached=(1015808,), + table="identity", + layer=43, + ), + ), + Cell( + why="1M packed chunk on a V2 view, layer 0: the manager cell's 4 requests at their " + "positions, slots drawn by the test", + spec=dict( + _ML4_1M_V2_TP4, + tokens=32768, + blocks=1100, + requests=_ML4_1M_PACKED_TP4, + cached=_ML4_1M_PACKED_CACHED_TP4, + layer=0, + ), + ), + Cell( + why="1M mixed step on a V2 view, layer 20: 3 context chunks (one ending at 1048575) " + "ahead of 13 decode rows across the 1M range", + spec=dict( + _ML4_1M_V2_TP4, + tokens=32768, + blocks=1100, + requests=_ML4_1M_MIXED_TP4, + cached=_ML4_1M_MIXED_CACHED_TP4, + layer=20, + ), + ), + Cell( + why="1M chunk on the strided view: 32768 rows at positions 1015808..1048575, the " + "strided 8-row launcher", + spec=dict(_ML4_1M_TP4, tokens=32768, blocks=1100, requests=(32768,), cached=(1015808,)), + ), + Cell( + why="1M-table decode batch 1 (captured), V2 view: position 1048575, the table's " + "last row", + spec=dict(_ML4_1M_V2_TP4, tokens=1, graph=True, top_position=True, layer=43), + ), + Cell( + why="1M-table decode batch 2 (captured), V2 view, first row", + spec=dict(_ML4_1M_V2_TP4, tokens=2, graph=True, top_position=True, layer=0), + ), + Cell( + why="1M-table decode batch 4 (captured), V2 view", + spec=dict(_ML4_1M_V2_TP4, tokens=4, graph=True, top_position=True, layer=17), + ), + Cell( + why="1M-table decode batch 8 (captured), V2 view, its last 3 rows padding at slot -1", + spec=dict( + _ML4_1M_V2_TP4, tokens=8, graph=True, top_position=True, pad_rows=3, layer=30 + ), + ), + Cell( + why="1M-table decode batch 16 (captured), V2 view, slot buffers of max_num_tokens " + "32768 over a 64-block view", + spec=dict(_ML4_1M_V2_TP4, tokens=16, graph=True, top_position=True, slot_rows=32768), + ), + Cell( + why="1M-table NoPE decode batch 16 (captured), V2 view: identity table", + spec=dict( + _ML4_1M_V2_TP4, tokens=16, graph=True, top_position=True, table="identity", layer=43 + ), + ), + Cell( + why="1M chunk on the engine V2 manager, layer 68 (pool row 42): the last 32768 rows " + "of a 1048576-token prompt, positions 1015808..1048575 behind its cached prefix " + "(a 33024-page quota, 57.1 GiB device peak: large)", + spec=dict( + _ML4_1M_MGR, + blocks=33024, + tokens=32768, + requests=(32768,), + cached=(1015808,), + model_layer=68, + large=True, + ), + ), + Cell( + why="1M chunk on the engine V2 manager, NoPE layer 71 (pool row 43): identity table " + "at positions 1015808..1048575 (57.1 GiB device peak: large)", + spec=dict( + _ML4_1M_MGR, + blocks=33024, + table="identity", + tokens=32768, + requests=(32768,), + cached=(1015808,), + model_layer=71, + large=True, + ), + ), + Cell( + why="1M packed chunk tails on the engine V2 manager, layer 0: 16 requests in 32768 " + "rows behind prefixes up to 1048575 (a 65234-page quota, 110.7 GiB device peak: " + "large)", + spec=dict( + _ML4_1M_MGR, + blocks=65234, + tokens=32768, + requests=_ML4_1M_PACKED, + cached=_ML4_1M_PACKED_CACHED, + model_layer=0, + large=True, + ), + ), + Cell( + why="1M mixed step on the engine V2 manager, layer 20 (pool row 18): 3 chunks behind " + "prefixes up to 1028576 ahead of 13 decode rows at positions up to 1048575 (a " + "137252-page quota, 144472 slots, slots past 2^34 bytes; 230.7 GiB device peak: " + "large)", + spec=dict( + _ML4_1M_MGR, + blocks=137252, + tokens=32768, + requests=_ML4_1M_MIXED, + cached=_ML4_1M_MIXED_CACHED, + model_layer=20, + large=True, + ), + ), + Cell( + why="1M-table decode batch 1 (captured), V2 view, last row: position 1048575", + spec=dict(_ML4_1M_V2, tokens=1, graph=True, top_position=True, layer=43), + ), + Cell( + why="1M-table decode batch 2 (captured), V2 view, first row: positions over the " + "whole table", + spec=dict(_ML4_1M_V2, tokens=2, graph=True, top_position=True, layer=0), + ), + Cell( + why="1M-table decode batch 4 (captured), V2 view", + spec=dict(_ML4_1M_V2, tokens=4, graph=True, top_position=True, layer=17), + ), + Cell( + why="1M-table decode batch 8 (captured), V2 view", + spec=dict(_ML4_1M_V2, tokens=8, graph=True, top_position=True, layer=30), + ), + Cell( + why="1M-table decode batch 16 (captured), V2 view", + spec=dict(_ML4_1M_V2, tokens=16, graph=True, top_position=True, layer=42), + ), + Cell( + why="1M-table NoPE decode batch 16 (captured), V2 view: identity table", + spec=dict( + _ML4_1M_V2, tokens=16, graph=True, top_position=True, table="identity", layer=43 + ), + ), + Cell( + why="1M chunk on a V2 view (pool row 42): positions 1015808..1048575, the large " + "manager cell's rows in pre-merge", + spec=dict( + _ML4_1M_V2, + blocks=1100, + tokens=32768, + requests=(32768,), + cached=(1015808,), + layer=42, + ), + ), + Cell( + why="1M NoPE chunk on a V2 view (pool row 43): identity table at positions " + "1015808..1048575", + spec=dict( + _ML4_1M_V2, + blocks=1100, + table="identity", + tokens=32768, + requests=(32768,), + cached=(1015808,), + layer=43, + ), + ), + Cell( + why="1M packed chunk tails on a V2 view (pool row 0): the large manager cell's 16 " + "requests and prefixes in pre-merge", + spec=dict( + _ML4_1M_V2, + blocks=1100, + tokens=32768, + requests=_ML4_1M_PACKED, + cached=_ML4_1M_PACKED_CACHED, + layer=0, + ), + ), + Cell( + why="1M mixed step on a V2 view (pool row 18): the large manager cell's 3 chunks and " + "13 decode rows in pre-merge", + spec=dict( + _ML4_1M_V2, + blocks=1100, + tokens=32768, + requests=_ML4_1M_MIXED, + cached=_ML4_1M_MIXED_CACHED, + layer=18, + ), + ), + Cell( + why="1M every position: 32 chunk calls of 32768 rows on a V2 view read table rows " + "0..1048575, each at its own position", + spec=dict(_ML4_1M_V2, blocks=1100, tokens=32768, layer=41, position_sweep=(0, 1048576)), + ), + Cell( + why="dep4 1M boot capture, batch 1 (V2 view): captured on the warm-up dummy's key at " + "position 1048574, replayed with the live row at 1048575, with attention DP's idle " + "dummy at 0 and at a drawn position", + spec=dict(_ML4_1M_DEP4_BOOT, tokens=1, layer=43), + ), + Cell( + why="dep4 1M boot capture, batch 2 (V2 view): the warm-up key at 1048574 ahead of a " + "one-key filler, replayed with one live row at 1048575 ahead of a padding-dummy row", + spec=dict(_ML4_1M_DEP4_BOOT, tokens=2, pad_rows=1, layer=0), + ), + Cell( + why="dep4 1M boot capture, batch 4 (V2 view): the warm-up key at 1048574 ahead of 3 " + "one-key fillers, replayed with one live row at 1048575 ahead of 3 padding-dummy rows", + spec=dict(_ML4_1M_DEP4_BOOT, tokens=4, pad_rows=3, layer=17), + ), + Cell( + why="dep4 1M boot capture, batch 8 (V2 view): the warm-up key at 1048574 ahead of 7 " + "one-key fillers, replayed with one live row at 1048575 ahead of 7 padding-dummy rows", + spec=dict(_ML4_1M_DEP4_BOOT, tokens=8, pad_rows=7, layer=30), + ), + Cell( + why="dep4 1M boot capture, batch 16 (V2 view), slot buffers of 32768: the warm-up key " + "at 1048574 ahead of 15 one-key fillers, replayed with attention DP's one live row " + "at 1048575 ahead of 15 padding-dummy rows on one slot", + spec=dict(_ML4_1M_DEP4_BOOT, tokens=16, pad_rows=15, layer=42), + ), + Cell( + why="dep4 1M mixed step on a V2 view (pool row 18): the large manager cell's 3 " + "chunks and 13 decode rows in pre-merge", + spec=dict( + _ML4_1M_V2, + blocks=1100, + tokens=32768, + requests=_ML4_1M_DEP4_MIXED, + cached=_ML4_1M_DEP4_MIXED_CACHED, + layer=18, + ), + ), + Cell( + why="dep4 1M chunk on the engine V2 manager, layer 68 (pool row 42): the last 32768 " + "rows of a 1048576-token prompt, positions 1015808..1048575 behind its cached " + "prefix (a 33792-page quota: large)", + spec=dict( + _ML4_1M_MGR, + blocks=33792, + tokens=32768, + requests=(32768,), + cached=(1015808,), + model_layer=68, + large=True, + ), + ), + Cell( + why="dep4 1M chunk on the engine V2 manager, NoPE layer 71 (pool row 43): identity " + "table at positions 1015808..1048575 (a 33792-page quota: large)", + spec=dict( + _ML4_1M_MGR, + blocks=33792, + table="identity", + tokens=32768, + requests=(32768,), + cached=(1015808,), + model_layer=71, + large=True, + ), + ), + Cell( + why="dep4 1M mixed step on the engine V2 manager, layer 20 (pool row 18): 3 chunks " + "(one fresh, two behind 100000 and 65536 cached) ahead of 13 decode rows at " + "positions up to 1048575 (a 73728-page quota: large)", + spec=dict( + _ML4_1M_MGR, + blocks=73728, + tokens=32768, + requests=_ML4_1M_DEP4_MIXED, + cached=_ML4_1M_DEP4_MIXED_CACHED, + model_layer=20, + large=True, + ), + ), + ) + + note = """ + `build(device_id, is_neox=..., cache_contiguous=...)` compiles the four + launchers once; a target builds them on its first forward (the pool + layout is an engine value) and passes the result as `compiled`. `__call__` + picks the launcher on the host: 2 rows per CTA up to 1024 tokens, 8 above, + and 8-byte loads when `indexer_k` is 8-byte aligned. There is no stock-op + fallback; every row count runs the kernel. Metadata consumed: none beyond + the arguments (the slot mappings and positions are the caller's). + + `validate=True` makes the kernel's wrapper check slot and position ranges + on the device, which is a host sync and breaks a CUDA-graph capture: pass + it on an eager call only. With it off those ranges are unchecked and + `is_valid` states them. Captured with interleaved rope (`validate=False`) + at decode batches 1, 2, 3, 4, 8 and 16 on a strided view, and at 1, 2, 4, + 8 and 16 on a V2 view (the contiguous launchers) at positions up to 32767 + over a 32768-row table and up to 1048575 over a 1048576-row one, a replay + on new data writes the same bytes as the eager call; a NeoX capture is not + driven. The dep4 engine's long-family graphs (`boot_capture` cells), each + batch captured on its warm-up batch (a key at position 1048574 ahead of + one-key fillers at 0) and replayed with attention DP's one live row at + 1048575, an idle rank's dummy at 0 or a batch at drawn positions, ahead of + the padding dummy repeated (b - 1 rows writing one slot pair with one key), + write the eager call's bytes; every code of those steps equals the + reference's. + + Pool views. `get_indexer_k_cache_buffers(layer)` hands out, at + Mistral Large 4's geometry (page 32, 132 bytes a token, 44 full indexer + layers; measured at max_seq_len = max_num_tokens 32768, and at + max_seq_len 1048576 with max_num_tokens 32768): + DSACacheManagerV2 (the engine's default): contiguous + `[slots * 44 - row, 32, 1, 132]`, starting `row * 4224` bytes into the + indexer pool and running to its end, where `row` is the layer's rank + among the full indexer layers. The page scale is 44: block b is page + `b * 44` of every row's view. The slot range runs past the quota + (2155 slots for a 2048-page quota; at max_seq_len 1048576, 34765 for + 33024, 68671 for 65234 and 144472 for 137252, and 35566 for 33792 and + 77621 for 73728), and a request's pages take descending slots, so the + last chunk of a 1M request alone in its pool writes slots 0..1023. At + max_seq_len 1048576 the block table + is 32772 columns wide and positions up to 1048575 address columns up to + 32767 of it. A caller therefore runs the contiguous launchers. + The pages of every later layer lie inside the view, so only slot + mappings built with the page scale keep a call in its own layer; the + op cannot see a mapping built without it. + DSACacheManager (V1): `[blocks, 32, 1, 132]` with stride(0) `44 * 4224`; + page scale 1, the strided launchers. + `is_valid` and `validate=True` bound the slots by the view, not by the + layer. At tep8, tp8 and dep8 a rank's weights leave room for about 144075 + and 132949 V2 slots (computed from the checkpoint's shard headers); no boot + has measured those pools. V2 views are certified to 147456 blocks (the + eight-GPU cell), rebuilt as above, not taken from a manager of that quota. + At max_seq_len 1048576 with max_num_tokens 32768 (measured): a 33900-block + quota gives 35675 slots and a 53000-block one 55786 (50.28 and 78.63 GiB of + device memory with the 72 latent rows, outside torch's allocator; the + manager cells read 57.97 and 89.79 GiB in use after their compare), and the + block table is 32772 columns wide. A 1048576-token request takes 32768 + pages in descending slots, so its last 32768-token chunk lands in the + lowest 1024 of them; the slot mappings are the engine's own arithmetic + over block-table columns up to 32767. + + Rope modes: interleaved (Mistral Large 4) is certified; NeoX rope is + accepted by the op but not certified here. The mode is compiled into the + launchers, so `is_neox` must match `compiled.is_neox`. The kernel has no + no-rope mode: a layer that takes no rope (Mistral Large 4's NoPE indexer + layers) passes an identity table, cos 1 and sin 0 in every row it reads, + so the rotation `x * 1 - y * 0` is exact and the keys are LayerNorm and + quantization only. The kernel's + LayerNorm sums in a different order and uses an approximate rsqrt, so a + normed element now and then rounds to the other bf16 neighbour of the + reference's (a handful of codes in 4194304 differ at 32768 rows). Rope + carries that ulp into the rotated pair, and where the pair cancels the + small result can land several e4m3 steps away (measured: 3 steps on one + code of a 32768-row step). The test bounds each code by that chain. + + Refused by the compiled launcher itself, before any launch and with + `validate` off (measured): `indexer_k` that is not row-contiguous, such as + the first 128 columns of a `[T, 160]` wk|weights_proj output, raises + `ValueError: Mismatched indexer_k.strides[0] on argument #0 ... expected to + be 128` -- copy the keys contiguous first; fp32 `norm_weight`, `norm_bias` + and `rotary_cos_sin`, int32 `position_ids` and int64 slot mappings are + enforced (`Mismatched Tensor on argument #N ... expected dtype=...`); + `position_ids` must have exactly `indexer_k`'s rows (`Mismatched + position_ids.shape[0] on argument #3 ... expected to match + indexer_k.shape[0]`), and so must the slot mappings after the wrapper + slices them to those rows (shorter raises the same way; longer is + accepted and certified). + + Cost: the call is bound by the host launch at every row count the targets + pass. + + Not certified: pools of other than 44 and 72 layers (the layer + count reaches the kernel only as the Int64 block stride, or as a V2 + view's start and length); interleaved positions past 1048575, or a table + of more than 1048576 rows; NeoX rope; more than 32768 + rows; strided pools past 18432 blocks; rebuilt V2 views past 147456 + blocks, and manager views past 144472 slots (dep4's manager cells reach + 77621); the scalar-load launchers on a V2 view (the target's keys are a + fresh contiguous copy, 8-byte aligned). + + Hard limits (sm_103) + slot offset contiguous launchers: int64 slot + int64 column, added to + the view's base pointer boundary 2^31 / 2^32 bytes + probed at a 24576-block 44-row V2 view (4567415424 bytes), + slots on both sides of 2^31 and 2^32 passed + and at a 110592-block V2 view (20554005120 bytes), rows in + its top 1024 blocks up to byte 20554004991 (past 2^34) passed + and at a 147456-block V2 view (27405400704 bytes, an + eight-GPU rank's largest pool), rows in its top 1024 blocks + up to byte 27405400575, scales to the view's last byte passed + and on the engine manager's view at a 137252-page quota + (144472 slots; pool row 18's view 26850912000 bytes), slots + up to byte 25461342848 passed + and on the engine manager's view at max_seq_len 1048576 and a + 73728-page quota (77621 slots; pool row 18's view + 14426252544 bytes), code slots up to byte 13358028416 passed + pool offset strided launchers: block * block stride, Int64 + boundary 2^31 bytes + probed at an 18432-block 44-layer pool (3425697792 bytes), + rows in the last block of the last layer passed + view pages k_cache dim 0, Int32 in the compiled launchers + boundary 2^31 pages = 48806446 blocks of 44 rows + the largest view certified has 6488021 pages (tp4's 1M + manager cells' 1569658 and 2454584); not reached + key row token * 128 elements, Int32 boundary 2^31 / 128 = 16777216 rows + not reached by the envelope (32768 rows: element 4194303), + not probed + position table row * 64, Int64 no 32-bit product; position_ids + are int32, the envelope reaches 1048575 (row offset 67108800 + elements, 268435200 bytes: under 2^31 either way); probed at + 1048575, the last row of a 1048576-row table (elements up to + 67108863) passed; the 2^31 boundary not probed; no index + passes through fp32 (and 1048575 < 2^24) + table rows rotary_cos_sin dim 0, Int32 in the compiled launchers + boundary 2^31 rows; 1048576 certified; not reached + slot int64 slot mappings, Int64 no 32-bit product; tp4's 1M + manager cells' engine-built slots reach byte 9230170496 + """ + + def __call__( + self, + indexer_k: torch.Tensor, + norm_weight: torch.Tensor, + norm_bias: torch.Tensor, + position_ids: torch.Tensor, + rotary_cos_sin: torch.Tensor, + k_cache: torch.Tensor, + slot_mapping_fp8: torch.Tensor, + slot_mapping_scale: torch.Tensor, + num_tokens: int, + *, + is_neox: bool, + compiled: CompiledIndexerKCacheUpdate, + validate: bool = False, + ) -> torch.Tensor: + rows_per_block = _kernel._select_rows_per_block(int(num_tokens)) + vectorized = _kernel._can_vectorize_loads(indexer_k) + _kernel.indexer_k_norm_rope_fp8_cache_update( + indexer_k, + norm_weight, + norm_bias, + position_ids, + rotary_cos_sin, + k_cache, + slot_mapping_fp8, + slot_mapping_scale, + num_tokens, + use_ue8m0=True, + is_neox=is_neox, + compiled_fn=compiled.fns[(rows_per_block, vectorized)], + validate=validate, + ) + return k_cache + + def reference( + self, + indexer_k: torch.Tensor, + norm_weight: torch.Tensor, + norm_bias: torch.Tensor, + position_ids: torch.Tensor, + rotary_cos_sin: torch.Tensor, + k_cache: torch.Tensor, + slot_mapping_fp8: torch.Tensor, + slot_mapping_scale: torch.Tensor, + num_tokens: int, + *, + is_neox: bool, + compiled: CompiledIndexerKCacheUpdate, + validate: bool = False, + ) -> torch.Tensor: + """The cache image after the write, as a fresh contiguous tensor shaped like `k_cache`. + + `k_cache` is read (the bytes not written keep their value), not modified. + LayerNorm and rope run in fp64 and are rounded to bf16 at the two points + the kernel materialises bf16; the scale and codes are then exact. + `compiled` and `validate` are not read. + """ + n = int(num_tokens) + dev = indexer_k.device + x = indexer_k[:n].double() + mean = x.mean(dim=-1, keepdim=True) + var = (x - mean).pow(2).mean(dim=-1, keepdim=True) + normed = (x - mean) * torch.rsqrt( + var + NORM_EPS + ) * norm_weight.double() + norm_bias.double() + normed = normed.float().to(torch.bfloat16).double() + + pe, nope = normed[:, :_ROPE_DIM], normed[:, _ROPE_DIM:] + table = rotary_cos_sin[position_ids[:n].long()].double() # [n, 2, 32] + cos, sin = table[:, 0], table[:, 1] + if is_neox: + partner = torch.cat([pe[:, _HALF:], pe[:, :_HALF]], dim=-1) + cos_e, sin_e = torch.cat([cos, cos], -1), torch.cat([sin, sin], -1) + sign = torch.cat([-torch.ones(_HALF), torch.ones(_HALF)]) + else: + partner = pe.reshape(n, _HALF, 2).flip(-1).reshape(n, _ROPE_DIM) + cos_e, sin_e = cos.repeat_interleave(2, -1), sin.repeat_interleave(2, -1) + sign = torch.tensor([-1.0, 1.0]).repeat(_HALF) + sign = sign.to(device=dev, dtype=torch.float64) + pe = (pe * cos_e + sign * partner * sin_e).float().to(torch.bfloat16) + row = torch.cat([pe.float(), nope.float()], dim=-1) # [n, 128] fp32, bf16 values + + amax = row.abs().amax(dim=-1) + floor = torch.tensor(_MIN_AMAX, dtype=torch.float32, device=dev) + inv_max = torch.tensor(1.0 / _E4M3_MAX, dtype=torch.float32, device=dev) + bits = (torch.maximum(amax, floor) * inv_max).contiguous().view(torch.int32) + exp_bits = bits & 0x7F800000 + exp_bits = torch.where((bits & 0x007FFFFF) != 0, exp_bits + 0x00800000, exp_bits) + scale = exp_bits.view(torch.float32) + inv_scale = (0x7F000000 - exp_bits).view(torch.float32) + codes = (row * inv_scale[:, None]).to(torch.float8_e4m3fn).view(torch.uint8) + scale_bytes = scale.contiguous().view(torch.uint8).reshape(n, _SCALE_BYTES) + + out = k_cache.clone(memory_format=torch.contiguous_format) + flat = out.view(-1) + fp8_slot = slot_mapping_fp8[:n].long() + scale_slot = slot_mapping_scale[:n].long() + live = (fp8_slot >= 0) & (scale_slot >= 0) + cols = torch.arange(_HEAD_DIM, device=dev) + flat[(fp8_slot[live].unsqueeze(1) + cols).reshape(-1)] = codes[live].reshape(-1) + cols = torch.arange(_SCALE_BYTES, device=dev) + flat[(scale_slot[live].unsqueeze(1) + cols).reshape(-1)] = scale_bytes[live].reshape(-1) + return out + + def is_valid( + self, + indexer_k: torch.Tensor, + norm_weight: torch.Tensor, + norm_bias: torch.Tensor, + position_ids: torch.Tensor, + rotary_cos_sin: torch.Tensor, + k_cache: torch.Tensor, + slot_mapping_fp8: torch.Tensor, + slot_mapping_scale: torch.Tensor, + num_tokens: int, + *, + is_neox: bool, + compiled: CompiledIndexerKCacheUpdate, + validate: bool = False, + ) -> None: + # Both are compile-time constants the wrapper does not check against the + # call: the other rope mode writes plausible wrong keys, and contiguous + # addressing on a strided per-layer view writes into other layers. + assert compiled.is_neox == bool(is_neox), ( + f"compiled for is_neox={compiled.is_neox}, called with is_neox={is_neox}" + ) + assert not compiled.cache_contiguous or k_cache.is_contiguous(), ( + "compiled with contiguous cache addressing but k_cache is a strided view " + f"(stride {tuple(k_cache.stride())}); the kernel writes outside it" + ) + # Everything below is what `validate=True` checks on the device; with it + # off (every call inside a CUDA graph) nothing does. + n = int(num_tokens) + assert 0 <= n <= indexer_k.shape[0], ( + f"num_tokens {n} exceeds the {indexer_k.shape[0]} rows of indexer_k" + ) + if n == 0: + return + pos = position_ids[:n] + assert int(pos.min()) >= 0 and int(pos.max()) < rotary_cos_sin.shape[0], ( + f"a position is outside the {rotary_cos_sin.shape[0]}-row rope table; " + "the kernel reads past it" + ) + fp8_slot, scale_slot = slot_mapping_fp8[:n], slot_mapping_scale[:n] + assert bool(((fp8_slot >= 0) == (scale_slot >= 0)).all()), ( + "slot_mapping_fp8 and slot_mapping_scale disagree on which rows are written" + ) + size = k_cache.numel() + live = fp8_slot >= 0 + if bool(live.any()): + f, s = fp8_slot[live], scale_slot[live] + assert bool(((f % 4 == 0) & (s % 4 == 0)).all()), ( + "slots must be 4-byte aligned: the kernel stores 32-bit words" + ) + assert int(f.max()) + _HEAD_DIM <= size and int(s.max()) + _SCALE_BYTES <= size, ( + f"a slot writes past the {size}-byte cache view" + ) + + +cute_dsl_indexer_k_cache_update = CuteDslIndexerKCacheUpdate() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/cute_dsl_indexer_q_side_fused.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/cute_dsl_indexer_q_side_fused.py new file mode 100644 index 000000000000..20e1830a7d31 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/cute_dsl_indexer_q_side_fused.py @@ -0,0 +1,622 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The DSA indexer's q side after its projection, in one CuTe DSL launch.""" + +from __future__ import annotations + +from collections.abc import Callable +from dataclasses import dataclass + +import torch + +from tensorrt_llm._torch.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.kernels import ( + _indexer_q_side_fused_kernel as _kernel, +) + +#: The indexer geometry the kernel is specialised for: 32 heads of 128, the +#: first 64 dims of each head rotated. +_HEADS = 32 +_HEAD_DIM = 128 +_ROPE_DIM = 64 +_HALF = _ROPE_DIM // 2 +_E4M3_MAX = 448.0 +_MIN_AMAX = 1.0e-12 + + +@dataclass(frozen=True) +class CompiledIndexerQSide: + """The compiled launcher and the rope mode it was compiled for.""" + + fn: Callable + is_neox: bool + + +def build(device_id: int, *, is_neox: bool) -> CompiledIndexerQSide: + """Compile the variant the target runs: ue8m0 row scales, PDL launch. + + `is_neox` is a compile-time constant of the kernel. One compile, a few + seconds, cached per process by the kernel module; allocates nothing. + """ + with torch.cuda.device(device_id): + fn = _kernel.compile_indexer_q_side_fused(device_id, bool(is_neox), True, use_pdl=True) + return CompiledIndexerQSide(fn=fn, is_neox=bool(is_neox)) + + +#: What every Mistral Large 4 FP8 cell shares: interleaved rope over its YaRN +#: (factor 512) + LeRoPE table of max_seq_len 9216 rows, positions drawn from +#: all of it. +_ML4 = dict(is_neox=False, table="yarn", max_pos=9216, positions="spread") +#: What every Mistral Large 4 NVFP4 cell shares: the same rope over that +#: checkpoint's YaRN factor 2048 + LeRoPE table of max_seq_len 32768 rows, +#: positions drawn from all of it. +_ML4_32K = dict(is_neox=False, table="yarn", factor=2048, max_pos=32768, positions="spread") +#: What every Mistral Large 4 NVFP4 cell at the checkpoint's own context shares (max_seq_len +#: 1048576, chunked prefill): the same rope over the factor 2048 + LeRoPE table at +#: 1048576 rows, positions drawn from all of it. +_ML4_1M = dict(is_neox=False, table="yarn", factor=2048, max_pos=1048576, positions="spread") + + +class CuteDslIndexerQSideFused(OpWrapper): + """Rope the indexer query, pack it to e4m3, and fold its scale into the head weights. + + `q` is the unrotated `[T, 32, 128]` bf16 indexer query, each head `[pe(64) | + nope(64)]`. For every (token, head) row the op + + 1. rotates `pe` at `position_ids[t]` from `cos_sin[pos] = [cos(32) | sin(32)]` + and rounds the result to bf16: NeoX halves (`is_neox=True`: dims `i` and + `i + 32` rotate by column `i`) or interleaved pairs (`is_neox=False`: + dims `2i` and `2i + 1` rotate by column `i`); + 2. takes `amax` over the rotated `pe` and the untouched `nope`, sets the row + scale to `max(amax, 1e-12) / 448` rounded up to a power of two, and writes + `q_fp8_out[t, h] = e4m3([pe | nope] / scale)`; + 3. writes `weights_out[t, h] = (wq[t, h] * scale) * weight_scale`. + + `q` is not modified. Both outputs are caller-owned and returned. What the + caller still owns: the query's projection and any scaling of it before + rope (Mistral Large 4's SSMax), and the head weights' projection and their + cast to a contiguous fp32 `[T, 32]`. + + The call is `indexer_q_side_fused`, a CuTe DSL kernel's Python launcher + (no `torch.ops` registration), launched once. It lives in the family's + private kernel module `_indexer_q_side_fused_kernel`. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # One compiled launcher per rope mode serves every row count. Rows are + # generation tokens on a decode step and the whole step on a prefill step; + # positions go up to the engine's max_seq_len. + # Mistral Large 4 runs interleaved rope on its 26 RoPE indexer layers and an + # identity table (cos 1, sin 0) on its 18 NoPE indexer layers: decode + # batches 1..16, CUDA-graph captured at 1, 2, 4, 8 and 16, and eager prefill + # steps (PrefillTarget._indexer: every row of a step, a mixed step's + # generation rows included). The FP8 target serves max_num_tokens 9216 at + # positions 0..9215 on its factor 512 table; the NVFP4 target serves + # max_seq_len = max_num_tokens 32768 with chunked prefill off, so one step + # carries up to 32768 rows at positions 0..32767 on its factor 2048 table, + # and its decode rows (CUDA graphs 1, 2, 4, 8, 16, padded) read positions up + # to 32767. At its own context (max_seq_len 1048576, chunked prefill on, + # max_num_tokens 32768) the NVFP4 tep4, dep4 and tep8 deployments build a 1048576-row + # table and a step still carries up to 32768 rows: a chunk's rows sit at cached .. + # cached + n - 1 behind the request's cached prefix (the last chunk of a 1M prompt + # at 1015808..1048575), a mixed step's rows at any position, and its decode rows + # read any position up to 1048575. The NVFP4 tp4 deployment makes the same call: its + # indexer is replicated (32 heads on every rank), its tables come from the same + # `_rope_tables`, and under the engine's query split every rank still ropes every + # row of the step (the split divides the selection, after this op). At dep4 + # (attention DP, no indexer query split) each rank runs every row of its own + # requests, so the step is the rank's: up to 32768 rows. Its boot captures each + # decode graph 1, 2, 4, 8 and 16 on the warm-up batch: the longest dummy + # (max_seq_len - 1 = 1048575 keys, so its decode row reads position 1048574) in + # row 0 beside one-key fillers at position 0; the graph then replays with + # attention DP's one live row behind one-key padding rows, or a full batch. A + # cell's spec is one of the bases above with its own keys over it: `cached` starts + # an `arange` step at that position, `position_sweep` (lo, hi) reads every row + # lo .. hi - 1 in calls of `tokens` rows, `graph` captures at the cell's own row + # count (True) or at each batch it lists, each replayed on new data, `first` is + # row 0's position under `positions="first"` (every other row at 0), and + # `tail_rows` is how many of the table's last rows `positions="tail"` draws from + # (64 unless given). + CELLS: tuple[Cell, ...] = ( + Cell( + why="interleaved rope, decode batch 1 (captured)", + spec=dict(_ML4, tokens=1, graph=True), + ), + Cell( + why="interleaved rope, decode batch 2 (captured)", + spec=dict(_ML4, tokens=2, graph=True), + ), + Cell( + why="interleaved rope, decode batch 3: an odd batch of the unpadded 1..16 captures", + spec=dict(_ML4, tokens=3, graph=True), + ), + Cell( + why="interleaved rope, decode batch 4 (captured)", + spec=dict(_ML4, tokens=4, graph=True), + ), + Cell( + why="interleaved rope, decode batch 8 (captured)", + spec=dict(_ML4, tokens=8, graph=True), + ), + Cell( + why="interleaved rope, decode batch 16 (captured)", + spec=dict(_ML4, tokens=16, graph=True), + ), + Cell( + why="a NoPE indexer layer at decode batch 16 (captured): identity table, no rotation", + spec=dict(_ML4, tokens=16, graph=True, table="identity"), + ), + Cell( + why="an interleaved prefill step of 300 rows at positions in the table's last 64 rows", + spec=dict(_ML4, tokens=300, positions="tail"), + ), + Cell( + why="interleaved prefill at max_num_tokens 9216 rows, positions 0..9215", + spec=dict(_ML4, tokens=9216, positions="arange"), + ), + Cell( + why="a NoPE indexer layer at 9216 rows: identity table over positions 0..9215", + spec=dict(_ML4, tokens=9216, positions="arange", table="identity"), + ), + Cell( + why="NVFP4 interleaved prefill at max_num_tokens 32768 rows, positions 0..32767", + spec=dict(_ML4_32K, tokens=32768, positions="arange"), + ), + Cell( + why="NVFP4 NoPE indexer layer at 32768 rows: identity table over positions 0..32767", + spec=dict(_ML4_32K, tokens=32768, positions="arange", table="identity"), + ), + Cell( + why="NVFP4 table as the target holds it, cos and sin rounded through bf16, " + "32768 rows at positions 0..32767", + spec=dict(_ML4_32K, tokens=32768, positions="arange", table_dtype="bfloat16"), + ), + Cell( + why="NVFP4 interleaved decode batch 1 (captured), positions up to 32767", + spec=dict(_ML4_32K, tokens=1, graph=True), + ), + Cell( + why="NVFP4 interleaved decode batch 2 (captured), positions up to 32767", + spec=dict(_ML4_32K, tokens=2, graph=True), + ), + Cell( + why="NVFP4 interleaved decode batch 4 (captured), positions up to 32767", + spec=dict(_ML4_32K, tokens=4, graph=True), + ), + Cell( + why="NVFP4 interleaved decode batch 8 (captured), positions up to 32767", + spec=dict(_ML4_32K, tokens=8, graph=True), + ), + Cell( + why="NVFP4 interleaved decode batch 16 (captured), positions up to 32767", + spec=dict(_ML4_32K, tokens=16, graph=True), + ), + Cell( + why="NVFP4 decode batch 16 (captured) at positions in the table's last 64 rows, " + "32704..32767", + spec=dict(_ML4_32K, tokens=16, graph=True, positions="tail"), + ), + Cell( + why="NVFP4 NoPE indexer layer at decode batch 16 (captured): identity table of " + "32768 rows", + spec=dict(_ML4_32K, tokens=16, graph=True, table="identity"), + ), + Cell( + why="NVFP4 1M last chunk: 32768 rows at positions 1015808..1048575 behind a " + "1015808-token cached prefix, table of 1048576 rows", + spec=dict(_ML4_1M, tokens=32768, positions="arange", cached=1015808), + ), + Cell( + why="NVFP4 1M NoPE indexer layer, last chunk: identity table of 1048576 rows at " + "positions 1015808..1048575", + spec=dict(_ML4_1M, tokens=32768, positions="arange", cached=1015808, table="identity"), + ), + Cell( + why="NVFP4 1M last chunk on the table as the target holds it, cos and sin rounded " + "through bf16, positions 1015808..1048575", + spec=dict( + _ML4_1M, tokens=32768, positions="arange", cached=1015808, table_dtype="bfloat16" + ), + ), + Cell( + why="NVFP4 1M every table row: positions 0..1048575 in 32 chunk calls of 32768 " + "rows, cos and sin rounded through bf16", + spec=dict( + _ML4_1M, + tokens=32768, + positions="arange", + position_sweep=(0, 1048576), + table_dtype="bfloat16", + ), + ), + Cell( + why="NVFP4 1M mixed step: 32768 rows at positions spread over 0..1048575, cos and " + "sin rounded through bf16", + spec=dict(_ML4_1M, tokens=32768, table_dtype="bfloat16"), + ), + Cell( + why="NVFP4 1M decode batch 1 (captured), positions spread over 0..1048575", + spec=dict(_ML4_1M, tokens=1, graph=True), + ), + Cell( + why="NVFP4 1M decode batch 2 (captured), positions spread over 0..1048575", + spec=dict(_ML4_1M, tokens=2, graph=True), + ), + Cell( + why="NVFP4 1M decode batch 4 (captured), positions spread over 0..1048575", + spec=dict(_ML4_1M, tokens=4, graph=True), + ), + Cell( + why="NVFP4 1M decode batch 8 (captured), positions spread over 0..1048575", + spec=dict(_ML4_1M, tokens=8, graph=True), + ), + Cell( + why="NVFP4 1M decode batch 16 (captured), positions spread over 0..1048575", + spec=dict(_ML4_1M, tokens=16, graph=True), + ), + Cell( + why="NVFP4 1M decode batch 16 (captured) at positions in the table's last 64 rows, " + "1048512..1048575", + spec=dict(_ML4_1M, tokens=16, graph=True, positions="tail"), + ), + Cell( + why="NVFP4 1M NoPE indexer layer at decode batch 16 (captured): identity table of " + "1048576 rows", + spec=dict(_ML4_1M, tokens=16, graph=True, table="identity"), + ), + Cell( + why="NVFP4 1M tp4 table tail at max_num_tokens: 32768 rows at positions in the " + "table's last 64 rows, 1048512..1048575", + spec=dict(_ML4_1M, tokens=32768, positions="tail"), + ), + Cell( + why="NVFP4 1M tp4 NoPE indexer layer, mixed step: 32768 rows at positions spread " + "over the identity table of 1048576 rows", + spec=dict(_ML4_1M, tokens=32768, table="identity"), + ), + Cell( + why="NVFP4 1M tp4 decode graphs at the engine's padded batches 1, 2, 4, 8 and 16 " + "(captured), positions in the table's last 64 rows, 1048512..1048575", + spec=dict(_ML4_1M, tokens=16, positions="tail", graph=(1, 2, 4, 8, 16)), + ), + Cell( + why="NVFP4 1M tp4 NoPE indexer layer, decode graph 16 (captured) at positions in " + "the identity table's last 64 rows", + spec=dict(_ML4_1M, tokens=16, positions="tail", graph=(16,), table="identity"), + ), + Cell( + why="NVFP4 1M dep4 boot capture at batch 1: the warm-up's longest dummy (1048575 " + "keys) at position 1048574; replayed at 1048575, then at a spread position", + spec=dict( + _ML4_1M, + tokens=1, + graph=True, + positions="first", + first=1048574, + table_dtype="bfloat16", + ), + ), + Cell( + why="NVFP4 1M dep4 boot capture at batch 2: the longest dummy at position 1048574 in " + "row 0 beside a one-key filler at 0; replayed with attention DP's one live row at " + "1048575 behind a padding row at 0, then at positions spread over 0..1048575", + spec=dict( + _ML4_1M, + tokens=2, + graph=True, + positions="first", + first=1048574, + table_dtype="bfloat16", + ), + ), + Cell( + why="NVFP4 1M dep4 boot capture at batch 4: the longest dummy at position 1048574 in " + "row 0 beside 3 one-key fillers at 0; replayed with one live row at 1048575 behind " + "padding rows at 0, then at positions spread over 0..1048575", + spec=dict( + _ML4_1M, + tokens=4, + graph=True, + positions="first", + first=1048574, + table_dtype="bfloat16", + ), + ), + Cell( + why="NVFP4 1M dep4 boot capture at batch 8: the longest dummy at position 1048574 in " + "row 0 beside 7 one-key fillers at 0; replayed with one live row at 1048575 behind " + "padding rows at 0, then at positions spread over 0..1048575", + spec=dict( + _ML4_1M, + tokens=8, + graph=True, + positions="first", + first=1048574, + table_dtype="bfloat16", + ), + ), + Cell( + why="NVFP4 1M dep4 boot capture at batch 16: the longest dummy at position 1048574 in " + "row 0 beside 15 one-key fillers at 0; replayed with one live row at 1048575 behind " + "padding rows at 0, then at positions spread over 0..1048575", + spec=dict( + _ML4_1M, + tokens=16, + graph=True, + positions="first", + first=1048574, + table_dtype="bfloat16", + ), + ), + Cell( + why="NVFP4 1M dep4 NoPE boot capture at batch 16: identity table of 1048576 rows, the " + "longest dummy at position 1048574 beside 15 fillers at 0, replayed as the RoPE ones", + spec=dict( + _ML4_1M, tokens=16, graph=True, positions="first", first=1048574, table="identity" + ), + ), + Cell( + why="NVFP4 1M dep4 tail step: 32768 rows at positions drawn from the last chunk's " + "1015808..1048575 in any order (a packed or mixed step's far rows)", + spec=dict(_ML4_1M, tokens=32768, positions="tail", tail_rows=32768), + ), + Cell( + why="NVFP4 1M dep4 tail step on the table as the target holds it, cos and sin rounded " + "through bf16: 32768 rows drawn from 1015808..1048575", + spec=dict( + _ML4_1M, tokens=32768, positions="tail", tail_rows=32768, table_dtype="bfloat16" + ), + ), + Cell( + why="NVFP4 1M dep4 NoPE tail step: identity table of 1048576 rows, 32768 rows drawn " + "from 1015808..1048575", + spec=dict(_ML4_1M, tokens=32768, positions="tail", tail_rows=32768, table="identity"), + ), + ) + + note = """ + `build(device_id, is_neox=...)` compiles the launcher once; the target calls + it at load and passes the result as `compiled`. Row count is a runtime + value, so one launcher serves decode and prefill and there is no stock-op + fallback at the call site. Metadata consumed: none beyond the arguments. + + Rope modes: interleaved (Mistral Large 4) is certified; NeoX rope is + accepted by the op but not certified here. The mode is compiled into the + launcher, so `is_neox` must match `compiled.is_neox`. The kernel has no + no-rope mode: a layer that takes no rope (Mistral Large 4's NoPE indexer + layers) passes an identity table, cos 1 and sin 0 in every row it reads, + with as many rows as the largest position. The rotation is then exact: + the weights bitwise, and the codes + value for value, are those of a plain per-(token, head) ue8m0 + quantization of `q` (measured). The one bitwise difference is the sign of + a zero code in the rope half: a negative bf16 subnormal or -0 there can + come out +0, which decodes to the same zero. + + Interleaved tables: the rope table is data, read as row `position_ids[t]` + and nowhere else, so a table is certified by the rows its cells read. Three + are: the FP8 Mistral Large 4 target's (YaRN factor 512, 9216 rows), the NVFP4 + target's at 32768 (YaRN factor 2048, 32768 rows, every position 0..32767 + read by one 32768-row step, the decode batches spread over all of it and + into its last 64 rows) and the NVFP4 target's at its own context (the same + factor 2048 table at 1048576 rows: every position 0..1048575 read by 32 + chunk calls of 32768 rows, the last chunk 1015808..1048575 behind its + cached prefix, a 32768-row step at positions spread over all of it, a + 32768-row step in its last 64 rows, 32768-row steps drawn out of order from + the last chunk's rows, the decode batches spread over all of it and into + its last 64 rows at each of 1, 2, 4, 8 and 16, and dep4's boot batches at + positions 1048574 and 0), all with a seeded LeRoPE alpha, and the NVFP4 + ones also with cos and sin rounded through bf16 as the target holds its + table. The identity table is certified at 9216, 32768 and 1048576 rows (at + 1048576: the last chunk, a 32768-row step spread over all of it, a + 32768-row step drawn from the last chunk's rows, decode batch 16 spread + over it and in its last 64 rows, and dep4's boot batch 16 at positions + 1048574 and 0). Measured beside the cells on sm_103: the table the FP8 + target's `_rope_tables` builds from the NVFP4 checkpoint's config and + `lerope.alpha` (alpha -0.35..0.92) is bitwise the reference at + 32768 rows, and on that table a read wrapped at 9216 rows or one position + early moves a rope code in every row from 9216 up, so the 32768-row cells + see a wrong row there (the test pins the same on the stand-in table). The + tables the NVFP4 tep4 and tep8 deployments' `_rope_tables` build at 1048576 + rows from the same config and alpha (indexer and identity) are bitwise the + reference at every position 0..1048575, read in 32 calls of 32768 rows, + and on the indexer table a read of the last chunk wrapped at 32768 rows or + one position early moves a rope code in every one of its 32768 rows (0.927 + and 0.303 of them; the test pins the same on the stand-in table). The NVFP4 + tp4 deployment's `_rope_tables` is the same code, and its tables at 1048576 + rows from that config and alpha measured the same: bitwise the reference + at every position 0..1048575 in 32 calls (indexer and identity), the same + 0.927 and 0.303 at the last chunk and at a 32768-row step in the last 64 + rows (no row unseen), and its decode graphs at 1, 2, 4, 8 and 16 replayed + as the engine pads them (live rows in the last 64 rows, padding rows at + position 0) bitwise equal to eager and to the reference. The NVFP4 + dep4 deployment builds its tables with the same `_rope_tables`, and on its + 1048576-row tables from the checkpoint the same was measured: every + position 0..1048575 bitwise the reference (indexer and identity), the + last chunk's wrapped and one-early reads moving 0.927 and 0.303 of the + rope codes with no row unseen, and the boot shapes below bitwise the + reference, eager and replayed. The test's + stand-in is built from fp64 angles and the target's table from fp32 angles, + as the reference's cos/sin cache is: with the checkpoint's alpha, both + rounded through bf16, they differ in 59206 of the 2097152 entries below + position 32768 and in 13122455 of the 65011712 from it (largest 0.0817). + The op reads a row as data, so either table certifies the same reads. + + The launch takes no host sync and allocates nothing: captured with + interleaved rope at decode batches 1, 2, 3, 4, 8 and 16 (FP8 table) and 1, + 2, 4, 8 and 16 at positions up to 32767 (NVFP4 table) and up to 1048575 + (NVFP4 1M table, spread and in its last 64 rows at each batch; the identity + table at 16), a replay on new data writes the same bytes as an eager call. + A NeoX capture is not driven. dep4's boot captures each batch 1, 2, + 4, 8 and 16 on its warm-up batch, row 0 at 1048574 (the max_seq_len - 1 + key dummy's decode position) and the other rows at 0; the cells capture + that batch (its eager call gated against the reference), then replay it + with one live row at 1048575 behind rows at 0 (attention DP's padding) and + with a batch spread over the table, each replay bitwise the eager call on + its data, on the indexer table and at 16 on the identity table. + + Refused by the launcher itself, before any launch (measured; the kernel's + `validate_abi`, metadata only, also on a captured call): `wq` that is not + fp32, such as the bf16 head-weight columns of a `[T, 160]` wk|weights_proj + output (`TypeError: wq must be torch.float32, got torch.bfloat16`), or not + contiguous, such as the same columns of an fp32 `[T, 160]` (`ValueError: + wq must be contiguous, got stride (160, 1)`) -- pass a contiguous fp32 + copy (`kw[:, 128:].float()` is accepted); a bf16 `cos_sin` (`TypeError: + cos_sin must be torch.float32, got torch.bfloat16`); int64 `position_ids` + (`TypeError: position_ids must be torch.int32, got torch.int64`). + + `position_ids` is read on the device with no range check: a position at or + past `cos_sin.shape[0]` reads past the table. `is_valid` states it. + + Bit-exact against the reference: every rounding the kernel does (bf16 after + the rope FMA, the power-of-two scale, the two fp32 products) is reproduced. + A bf16 subnormal in the rope half is flushed to signed zero first. + + Not certified: interleaved positions past 1048575; more than 32768 rows + with interleaved rope; NeoX rope; tables other than the + ones named above. + + Hard limits (sm_103), at the envelope of 32768 tokens per step and + positions to 1048575 over a 1048576-row table + q and q_fp8_out offset row * 128 elements, row = token * 32 + head, Int32 + boundary 2^31 / 4096 = 524288 tokens + not reached (largest offset 134217727), not probed + row count and grid tokens * 32 rows in Int32, one CTA per 16 rows + boundary 2^31 / 32 = 67108864 tokens + not reached (65536 CTAs), not probed + wq and weights_out row, Int32 + offset boundary 2^31 / 32 = 67108864 tokens, not reached + table offset position * 64, Int64 no 32-bit product + (largest 1048575 * 64 + 63 = 67108863) + table rows the compiled launcher's row count is an Int32 + symbol: boundary 2^31 rows + not reached (1048576 rows), not probed + position_ids int32 (the launcher refuses int64): boundary + 2^31, not reached (1048575); no index passes + through fp32 + KV pool none: the op reads the step's rows and the rope + table, so the pool size changes nothing here + """ + + def __call__( + self, + q: torch.Tensor, + position_ids: torch.Tensor, + cos_sin: torch.Tensor, + wq: torch.Tensor, + weight_scale: float, + q_fp8_out: torch.Tensor, + weights_out: torch.Tensor, + *, + is_neox: bool, + compiled: CompiledIndexerQSide, + ) -> tuple[torch.Tensor, torch.Tensor]: + return _kernel.indexer_q_side_fused( + q, + position_ids, + cos_sin, + wq, + weight_scale, + q_fp8_out, + weights_out, + is_neox=is_neox, + use_ue8m0=True, + compiled_fn=compiled.fn, + use_pdl=True, + ) + + def reference( + self, + q: torch.Tensor, + position_ids: torch.Tensor, + cos_sin: torch.Tensor, + wq: torch.Tensor, + weight_scale: float, + q_fp8_out: torch.Tensor, + weights_out: torch.Tensor, + *, + is_neox: bool, + compiled: CompiledIndexerQSide, + ) -> tuple[torch.Tensor, torch.Tensor]: + """Fresh `(q_fp8, weights)`; `q_fp8_out`, `weights_out` and `compiled` are not read. + + The rope FMA is evaluated in fp64 and rounded to fp32 once, which is what + a fused multiply-add does; every later step is an exact fp32 or bf16 + rounding, so the result is bit-exact rather than "close". + """ + tokens = q.shape[0] + pe_bf16 = q[..., :_ROPE_DIM] + x = pe_bf16.float() + bits = pe_bf16.contiguous().view(torch.int16) + subnormal = (bits & 0x7F80) == 0 + x = torch.where(subnormal, torch.copysign(torch.zeros_like(x), x), x) + nope = q[..., _ROPE_DIM:].float() + + table = cos_sin[position_ids.long()] # [T, 64] + cos, sin = table[:, :_HALF], table[:, _HALF:] + if is_neox: + partner = torch.cat([x[..., _HALF:], x[..., :_HALF]], dim=-1) + cos_e = torch.cat([cos, cos], dim=-1) + sin_e = torch.cat([sin, sin], dim=-1) + sign = torch.cat([-torch.ones(_HALF), torch.ones(_HALF)]) + else: + pairs = x.reshape(tokens, _HEADS, _HALF, 2) + partner = pairs.flip(-1).reshape(tokens, _HEADS, _ROPE_DIM) + cos_e = cos.repeat_interleave(2, dim=-1) + sin_e = sin.repeat_interleave(2, dim=-1) + sign = torch.tensor([-1.0, 1.0]).repeat(_HALF) + sign = sign.to(device=q.device, dtype=torch.float32) + cos_e, sin_e = cos_e[:, None, :], sin_e[:, None, :] + partner_term = (partner * sin_e) * sign # one fp32 rounding, then an exact sign flip + rotated = (x.double() * cos_e.double() + partner_term.double()).float() + rotated = rotated.to(torch.bfloat16).float() + + row = torch.cat([rotated, nope], dim=-1) # [T, 32, 128] + amax = row.abs().amax(dim=-1) + floor = torch.tensor(_MIN_AMAX, dtype=torch.float32, device=q.device) + inv_max = torch.tensor(1.0 / _E4M3_MAX, dtype=torch.float32, device=q.device) + raw_scale = torch.maximum(amax, floor) * inv_max + scale, inv_scale = _ue8m0_round_up(raw_scale) + q_fp8 = (row * inv_scale[..., None]).to(torch.float8_e4m3fn) + c = torch.tensor(weight_scale, dtype=torch.float32, device=q.device) + weights = (wq * scale) * c + return q_fp8, weights + + def is_valid( + self, + q: torch.Tensor, + position_ids: torch.Tensor, + cos_sin: torch.Tensor, + wq: torch.Tensor, + weight_scale: float, + q_fp8_out: torch.Tensor, + weights_out: torch.Tensor, + *, + is_neox: bool, + compiled: CompiledIndexerQSide, + ) -> None: + # The launcher's rope mode is fixed at compile time and the wrapper does + # not check it against `is_neox`: the other mode returns plausible wrong + # codes (max abs ~7.8 against 2.4e-4). + assert compiled.is_neox == bool(is_neox), ( + f"compiled for is_neox={compiled.is_neox}, called with is_neox={is_neox}" + ) + # Positions are read on the device with no bound. + if position_ids.numel(): + lo, hi = int(position_ids.min()), int(position_ids.max()) + assert 0 <= lo and hi < cos_sin.shape[0], ( + f"position_ids span [{lo}, {hi}] but the rope table has " + f"{cos_sin.shape[0]} rows; the kernel reads past it" + ) + + +def _ue8m0_round_up(scale: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]: + """`(2**ceil(log2 scale), 2**-ceil(log2 scale))` by exponent arithmetic, as the kernel does.""" + bits = scale.contiguous().view(torch.int32) + exp_bits = bits & 0x7F800000 + exp_bits = torch.where((bits & 0x007FFFFF) != 0, exp_bits + 0x00800000, exp_bits) + inv_bits = 0x7F000000 - exp_bits + return exp_bits.view(torch.float32), inv_bits.view(torch.float32) + + +cute_dsl_indexer_q_side_fused = CuteDslIndexerQSideFused() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/cute_dsl_radix_topk_decode.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/cute_dsl_radix_topk_decode.py new file mode 100644 index 000000000000..5491f726bf0a --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/cute_dsl_radix_topk_decode.py @@ -0,0 +1,657 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Decode top-k of the DSA indexer in one CuTe DSL launch: the trtllm op's selection, per row.""" + +from __future__ import annotations + +from dataclasses import dataclass + +import cutlass +import cutlass.cute as cute +import torch +from cutlass.cute.runtime import make_fake_compact_tensor, make_fake_stream, make_fake_tensor + +from tensorrt_llm._torch.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.kernels import ( + _radix_topk_decode as _kernel, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.indexer_topk_decode import ( + indexer_topk_decode, +) + +TOP_K = _kernel.TOP_K +_PAD_KEY = _kernel.PAD_KEY +_OP_FINAL_ITEMS = _kernel.OP_FINAL_ITEMS +#: fp16(-FLT_MAX) is -inf, 0xFC00, whose 11-bit ordered key is 0x7E0. +_PAD_FP16_BIN = 0x7E0 + +#: The trtllm op's split-work dispatch (`computeIndexerTopKDecodeBlocksPerRow`, +#: `kernels/indexerTopK.cu`): its column threshold for 10 blocks per row, its cap on blocks +#: per row, and the fewest columns per block. +_SPLIT_WORK_COLUMNS = 200_000 +_MAX_BLOCKS_PER_ROW = 10 +_MIN_COLUMNS_PER_BLOCK = 2048 + +#: The Mistral Large 4 NVFP4 targets' decode call: next_n 1, and their max_seq_len of 32768 as +#: the paged scorer's column count (a multiple of its 256-column alignment, so contiguous). +_ML4 = dict(next_n=1, width=32768) +#: KV lengths in the 8K/1K decode window: 8192-token prompts plus up to 1024 generated tokens. +_ML4_DECODE_WINDOW = [8600 + (37 * i) % 617 for i in range(256)] +#: The targets' other scorer widths: the decode scorer's column count is the engine's +#: max_seq_len, and a target builds this selector at any width from 4096 (where the op splits a +#: row over 2 blocks at every row count) to 65536 (its 10 blocks of 6553 columns at 1..15 rows). +_OTHER_WIDTHS = (4096, 8192, 16384, 65536) + + +def _lens_within(width: int, n: int) -> list[int]: + """`n` KV lengths a `width`-column scorer row holds: one key, the top-k edge (2047, 2048, + 2049), the full row and one key short of it, half of it, then a spread over 1..width.""" + edges = [1, 2047, 2048, 2049, width, width - 1, width // 2 + 1, 3] + return (edges + [1 + (7919 * i) % width for i in range(n)])[:n] + + +def blocks_per_row(num_rows: int, num_columns: int, sm_count: int) -> int: + """How many blocks the trtllm op splits each row across, for this call's shape. + + The op's own host dispatch, line for line: a pure function of the row count, the column + count and the device's SM count, so a captured graph keeps its capture's value. The + selection (see `_radix_topk_decode`) depends on it through the op's padding.""" + if num_rows <= 0: + return 1 + if num_columns >= _SPLIT_WORK_COLUMNS: + return _MAX_BLOCKS_PER_ROW + max_bp = min(max(1, num_columns // _MIN_COLUMNS_PER_BLOCK), _MAX_BLOCKS_PER_ROW) + if num_rows < sm_count // 2: + best_bp, best_waves = 1, 1 + for bp in range(2, max_bp + 1): + waves = (num_rows * bp + sm_count - 1) // sm_count + if waves * best_bp < best_waves * bp: + best_bp, best_waves = bp, waves + return max(1, best_bp) + return max(1, min(2, max_bp)) + + +@dataclass(frozen=True) +class CompiledRadixTopkDecode: + """A compiled launcher with the facts its call needs: device, next_n and SM count.""" + + fn: object + device_id: int + next_n: int + sm_count: int + + +_COMPILED: dict[tuple[int, int], CompiledRadixTopkDecode] = {} + + +def build(device_id: int, *, next_n: int = 1) -> CompiledRadixTopkDecode: + """Compile the kernel for `next_n` and launch it once, ahead of any capture. + + The row count, the column count and the logits row stride are symbolic, so one handle + serves every batch and width. A few seconds of compile, cached per process.""" + key = (int(device_id), int(next_n)) + if key in _COMPILED: + return _COMPILED[key] + with torch.cuda.device(device_id): + rows, cols, batch, stride = cute.sym_int(), cute.sym_int(), cute.sym_int(), cute.sym_int64() + fake_logits = make_fake_tensor( + cutlass.Float32, (rows, cols), stride=(stride, 1), assumed_align=4 + ) + fake_lens = make_fake_compact_tensor(cutlass.Int32, (batch,), assumed_align=4) + fake_out = make_fake_compact_tensor( + cutlass.Int32, (rows, TOP_K), stride_order=(1, 0), assumed_align=4 + ) + fn = cute.compile( + _kernel.RadixTopkDecodeKernel(int(next_n)), + fake_logits, + fake_lens, + fake_out, + cutlass.Int32(2), + make_fake_stream(use_tvm_ffi_env_stream=True), + options="--enable-tvm-ffi", + ) + sm_count = torch.cuda.get_device_properties(device_id).multi_processor_count + handle = CompiledRadixTopkDecode(fn, int(device_id), int(next_n), int(sm_count)) + logits = torch.zeros((next_n, 4096), dtype=torch.float32, device="cuda") + lens = torch.full((1,), 4096, dtype=torch.int32, device="cuda") + out = torch.empty((next_n, TOP_K), dtype=torch.int32, device="cuda") + fn(logits, lens, out, blocks_per_row(next_n, 4096, sm_count)) + torch.cuda.synchronize() + _COMPILED[key] = handle + return handle + + +def _keys(values: torch.Tensor) -> torch.Tensor: + """fp32 scores as int64 keys of the op's order: ascending key is descending score.""" + bits = values.contiguous().view(torch.int32).long() & 0xFFFFFFFF + return torch.where(bits >= 0x80000000, bits, (~bits) & 0x7FFFFFFF) + + +def _fp16_bins(values: torch.Tensor) -> torch.Tensor: + """The op's first-pass bins: the 11-bit ordered key of each score's fp16 image, with + every NaN in the top bin (the device conversion yields the canonical positive NaN).""" + bits = values.half().view(torch.int16).long() & 0xFFFF + key = torch.where(bits >= 0x8000, bits, (~bits) & 0x7FFF) + return torch.where(torch.isnan(values), 0, key >> 5) + + +def _top(keys: torch.Tensor, pad: int, need: int) -> tuple[torch.Tensor, int]: + """Positions of the `need` best of `keys` and `pad` pads at key(-FLT_MAX): by key, a + column before a pad, then by position. Returns the column positions and the pads taken.""" + rank = torch.cat([keys * 2, torch.full((pad,), _PAD_KEY * 2 + 1, device=keys.device)]) + order = torch.sort(rank, stable=True).indices[:need] + cols = order[order < keys.numel()] + return cols, need - cols.numel() + + +def _stage(values: torch.Tensor, keys: torch.Tensor, pad: int) -> tuple[torch.Tensor, int]: + """One selection of the op's kernel over `values` and `pad` pads: its fp16 first pass + decides when its threshold bin holds at most 2048 entries, its key passes otherwise. + Returns the selected positions and the pads taken.""" + bins = torch.cat([_fp16_bins(values), torch.full((pad,), _PAD_FP16_BIN, device=keys.device)]) + counts = torch.bincount(bins, minlength=2048) + t0 = int(torch.searchsorted(torch.cumsum(counts, 0), TOP_K)) + before, count = int(counts[:t0].sum()), int(counts[t0]) + if count > _OP_FINAL_ITEMS: + return _top(keys, pad, TOP_K) + vbins = bins[: values.numel()] + above = (vbins < t0).nonzero().flatten() + inbin = (vbins == t0).nonzero().flatten() + picked, pads = _top(keys[inbin], pad if _PAD_FP16_BIN == t0 else 0, TOP_K - before) + pads += pad if _PAD_FP16_BIN < t0 else 0 + return torch.cat([above, inbin[picked]]), pads + + +class CuteDslRadixTopkDecode(OpWrapper): + """Select each decode row's `top_k` (2048) entries the way the trtllm op does, in one launch. + + Row `r = b * next_n + t` is sequence `b`'s `t`-th new token; its columns are `[0, end)` + with `end = seq_lens[b] - next_n + t + 1`. The row's selection is the trtllm + `indexer_topk_decode` op's on its split path: scores ranked by fp32 bit pattern + (`key(x) = bits` for a negative `x`, else `~bits & 0x7FFFFFFF`, ascending), over the + row's columns plus the op's padding entries -- `pad = sum(max(0, top_k - len))` entries + at -FLT_MAX for the blocks of `len <= top_k` columns it cuts the row into + (`blocks_per_row` of the call's shape), each standing for a -1. Selected columns come + first in `indices[r]`, in no particular order (in column order when `end <= top_k`); + the selected pads follow as -1. Where scores tie at the cut, which of them win is the + op's freedom too; here a column wins a tie against a pad. + + So for scores above -FLT_MAX (every finite relu-weighted sum the paged scorer writes) + the row holds its top `min(end, top_k)` columns and then -1s. Below that, the op's + padding decides: a column scored -inf (and -FLT_MAX, which ties with a pad) is selected + only where fewer than `top_k` columns and pads rank above it. + + One score is ranked by the op's fp16 first pass apart from its key: a negative NaN, + whose fp16 image is the canonical positive NaN, the best score. A selection the op + settles in that pass (its threshold bin holds at most 2048 entries) treats a -NaN as + the best score, and one it settles by key as the worst. A row holding a negative NaN is + therefore selected the op's way stage by stage: each of its blocks of more than top_k + columns, then the merge of their winners, the shorter blocks' columns and the pads. + + `indices` (`[rows, top_k]` int32, contiguous) is written in place and returned. + `next_n` and `compiled` are keyword-only. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # The Mistral Large 4 NVFP4 tp4 / tep4 / dep4 deployments (`_ML4`) score every generation token + # of a step with the paged scorer, fp32 rows 32768 wide (their max_seq_len), and select + # here: from their decode steps (CUDA-graph captured, padded to the engine's batch list + # 1, 2, 4, 8, 16, 24, ..., 128, 192, 256; dep4 on each rank's own sequences) and from the + # generation rows of a mixed step, which sit at the tail of the step's selection buffer. + # At 32768 columns the op splits a row over 10 blocks at 1..15 rows, 9 at 16, and 2 from + # 64 rows on; `scores` names the pattern the test writes into the rows. + CELLS: tuple[Cell, ...] = ( + Cell( + why="Mistral Large 4 32768, 16 rows: ends 1..32768, the 2048/2049 top-k edge, the 8K/1K " + "decode window and full rows", + spec=dict( + _ML4, + seq_lens=[1, 2, 2047, 2048, 2049, 4096, 8600, 8960] + + [9216, 12345, 16384, 20481, 30000, 32767, 32768, 9000], + ), + ), + Cell( + why="Mistral Large 4 32768, row counts 1..16 and 17..256 (every blocks-per-row the op " + "picks, 10 down to 2): leading rows of one 256-row batch in the decode window", + spec=dict( + _ML4, + batches=tuple(range(1, 17)) + (17, 24, 31, 33, 64, 75, 76, 100, 128, 192, 256), + seq_lens=[1, 2049, 32768, 2048] + _ML4_DECODE_WINDOW[4:], + ), + ), + Cell( + why="Mistral Large 4 32768, captured decode batches 1..256 of the engine's padded list: " + "replayed on new scores and seq_lens up to 32768, the padding dummy's row at 1", + spec=dict( + _ML4, + graph_batches=(1, 2, 4, 8, 16, 24, 64, 128, 192, 256), + seq_lens=_ML4_DECODE_WINDOW, + replays=3, + ), + ), + Cell( + why="Mistral Large 4 32768, indexer-like scores (head-weighted relu sums, many exact " + "zeros) at 1, 16, 64 and 256 rows", + spec=dict( + _ML4, + scores="indexer", + batches=(1, 16, 64, 256), + seq_lens=[9216, 32768, 2049, 20481] + _ML4_DECODE_WINDOW[4:], + ), + ), + Cell( + why="Mistral Large 4 32768, tied scores at the cut at 16 and 256 rows: 16 levels, all " + "+0.0 (relu's floor), mixed +-0.0", + spec=dict( + _ML4, + scores="ties", + batches=(16, 256), + seq_lens=[32768, 32768, 32768, 9216, 9216, 9216, 20481, 20481] + + [20481, 2049, 2049, 2049, 2048, 100, 1, 30000] + + _ML4_DECODE_WINDOW[16:], + ), + ), + Cell( + why="Mistral Large 4 32768, scores past fp16's range and under its resolution at 16 " + "and 256 rows, both signs", + spec=dict( + _ML4, + scores="wide", + batches=(16, 256), + seq_lens=[32768, 32767, 9216, 8600, 20481, 2049, 2048, 1] + + [16385, 4097, 30000, 12345, 9000, 8192, 3000, 32768] + + _ML4_DECODE_WINDOW[16:], + ), + ), + Cell( + why="Mistral Large 4 32768, special values: +-NaN, +-inf, -FLT_MAX, -0.0 and denormals " + "among few finite scores, short and long rows, at 1, 15, 16 and 64 rows", + spec=dict( + _ML4, + scores="special", + batches=(1, 15, 16, 64), + seq_lens=[2100, 1000, 2048, 4000, 9216, 32768, 20481, 2049] + + [3000, 12000, 1, 2, 5000, 6000, 30000, 2500] + + _ML4_DECODE_WINDOW[16:64], + ), + ), + Cell( + why="Mistral Large 4 32768, negative NaNs, the one score the op's fp16 pass ranks " + "differently: among distinct scores (the pass decides) and in a crowd of +0.0 (it " + "does not), short and long rows, at 1, 16 and 64 rows", + spec=dict( + _ML4, + scores="negnan", + batches=(1, 16, 64), + seq_lens=[9216, 2048, 4000, 32768, 20481, 1500, 12000, 2049] + + [30000, 5000, 2100, 16384, 8600, 3000, 25000, 7000] + + _ML4_DECODE_WINDOW[16:64], + ), + ), + Cell( + why="Mistral Large 4 32768, threshold groups past the 4096-entry list: up to 30000 " + "equal scores at the cut, and 6000 sharing the top 11 key bits", + spec=dict( + _ML4, + scores="crowded", + batches=(1, 16), + seq_lens=[32768, 9216, 30000, 20481, 12000, 8600, 32767, 4097] + + [16384, 2049, 6145, 9000, 32768, 10000, 25000, 7000], + ), + ), + Cell( + why="Mistral Large 4 32768, mixed step: 15 generation rows at rows 32753..32767 of the " + "step's 32768-row selection buffer; the context rows untouched", + spec=dict( + _ML4, + tail_of=32768, + seq_lens=[32768, 1, 2049, 20481, 2048, 8600, 32767, 3] + + [9216, 7777, 4096, 1234, 30000, 6145, 16385], + ), + ), + ) + # The targets' other scorer widths (an engine's max_seq_len of 4096, 8192, 16384 or 65536): + # every row count 1..16 and up to 256, the captured batches replayed, ties at the cut and + # special values, KV lengths within the width. + CELLS += tuple( + Cell( + why=f"width {w}, row counts 1..16 and 17..256 (every blocks-per-row the op picks): " + f"ends 1..{w}, the 2048/2049 top-k edge, full rows", + spec=dict( + next_n=1, + width=w, + batches=tuple(range(1, 17)) + (17, 33, 64, 76, 77, 128, 256), + seq_lens=_lens_within(w, 256), + ), + ) + for w in _OTHER_WIDTHS + ) + CELLS += tuple( + Cell( + why=f"width {w}, captured decode batches 1..256 of the engine's padded list: " + f"replayed on new scores and seq_lens up to {w}, the padding dummy's row at 1", + spec=dict( + next_n=1, + width=w, + graph_batches=(1, 2, 4, 8, 16, 64, 256), + seq_lens=_lens_within(w, 256), + replays=2, + ), + ) + for w in _OTHER_WIDTHS + ) + CELLS += tuple( + Cell( + why=f"width {w}, {what}", + spec=dict( + next_n=1, width=w, scores=kind, batches=batches, seq_lens=_lens_within(w, 256) + ), + ) + for w in _OTHER_WIDTHS + for kind, what, batches in ( + ( + "ties", + "tied scores at the cut at 16 and 256 rows: 16 levels, all +0.0, mixed +-0.0", + (16, 256), + ), + ( + "special", + "special values: +-NaN, +-inf, -FLT_MAX, -0.0 and denormals among few " + "finite scores, at 1, 15, 16 and 64 rows", + (1, 15, 16, 64), + ), + ) + ) + + note = """ + `build(device_id, next_n=)` compiles and warms the kernel; a target builds it on its + first forward, ahead of any capture, and passes the handle as `compiled`. Its row and + column counts are symbolic, so one handle serves every batch, eager or captured, with + nothing compiled later. No scratch: one 1024-thread CTA owns one row and works in about + 66 KB of shared memory. The call's host work is `blocks_per_row` of its shape, which a + captured graph freezes as the op does. The call takes fp32 logits whose rows start + 32-byte aligned, a multiple of 8 columns wide and at most 65536 (the scorer's output at + the Mistral Large 4 width is), and refuses any other. + + The op it reproduces is `indexer_topk_decode` (the family entry and + torch.ops.trtllm.indexer_topk_decode) on its split path, which it takes for every + Mistral Large 4 call: fp32 logits at least 4096 and under 200000 columns wide. The call + refuses a shape the op would run as one block per row (`blocks_per_row` 1), whose + single-block kernel orders scores differently (an insertion sort on float compares, and + a short row listed without reading its scores). + + How the two compute the same set. The op ranks by the fp32 key above. Its first pass + bins fp16(x), and every later comparison (radix digits 31..21, 20..10, 9..0 and a radix + sort of the final bin) is on the full 32-bit key. fp16(x) is a monotone image of the key + for every score but a negative NaN, so wherever that pass settles a selection it settles + it as the key would. Each of its blocks keeps its own top_k, or all its columns plus + pads, and a merge takes the top_k of those: the top_k of the row's columns and the pads, + since a column a block drops has top_k better ones beside it. This kernel ranks by the + same key (its digits are bits 31..20, 19..10 and 9..0, no narrower type) over the row's + columns and the same number of pads, so the two select the same multiset of keys: the + same columns wherever no two scores share the cut's bit pattern, and among the ones that + do, each takes as many as the other. A row holding a negative NaN is selected through + the op's own stages (see the class docstring), fp16 pass included. + + On the tp4 deployment's decode scorer logits (264 calls: 6 steps of its 44 indexer layers, + KV lengths about 8200, rows tiled to the batch at 1, 16, 64 and 256 rows) no row is + selected differently from the op. Cost: one CTA reads its whole row, so the time follows + the KV length. + + Hard limits (sm_103) + rows grid x, one CTA per row envelope 256 + columns, end int32; end clamped to [0, columns] envelope 4096..65536 (cells at + 4096, 8192, 16384, 32768 and + 65536); is_valid refuses + an end past the row; at most + 65536 columns (the emulation + path's column mask) + logits stride(0) symbolic int64, stride(1) 1, rows 32-byte aligned + threshold group 4096-entry shared-memory list; a larger group is narrowed over the + row in global memory (one pass per digit) + """ + + def __call__( + self, + logits: torch.Tensor, + seq_lens: torch.Tensor, + indices: torch.Tensor, + *, + next_n: int, + compiled: CompiledRadixTopkDecode, + ) -> torch.Tensor: + rows, columns = logits.shape + if compiled.next_n != next_n: + raise ValueError(f"compiled for next_n={compiled.next_n}, called with {next_n}") + if indices.shape != (rows, TOP_K) or rows != seq_lens.shape[0] * next_n: + raise ValueError( + f"indices must be [{rows}, {TOP_K}] and seq_lens hold {rows} / next_n entries; " + f"got {tuple(indices.shape)} and {tuple(seq_lens.shape)}" + ) + if columns > _kernel.MAX_COLUMNS: + raise ValueError(f"logits rows wider than {_kernel.MAX_COLUMNS} columns are not served") + # The kernel reads each row in 32-byte vectors from its start. + if logits.data_ptr() % 32 or logits.stride(0) % 8 or columns % 8 or logits.stride(1) != 1: + raise ValueError( + "logits rows must start 32-byte aligned and be a multiple of 8 unit-stride " + f"columns; got shape {tuple(logits.shape)}, stride {logits.stride()}" + ) + bp = blocks_per_row(rows, columns, compiled.sm_count) + if bp < 2: + raise ValueError( + f"the trtllm op runs [{rows}, {columns}] as one block per row; that path is not " + "the one reproduced here" + ) + if rows: + compiled.fn(logits, seq_lens, indices, bp) + return indices + + def reference( + self, + logits: torch.Tensor, + seq_lens: torch.Tensor, + indices: torch.Tensor, + *, + next_n: int, + compiled: CompiledRadixTopkDecode, + ) -> torch.Tensor: + """The op's selection, as a fresh `[rows, top_k]` int32 tensor; `indices` is not read. + + Columns ranked by key, a column before a pad at the same key, then by column: one + answer among the ones the op allows. Only `compiled.sm_count` is read off the handle, + for the op's blocks per row.""" + rows, columns = logits.shape + bp = blocks_per_row(rows, columns, compiled.sm_count) + out = torch.full((rows, TOP_K), -1, dtype=torch.int32, device=logits.device) + lens = seq_lens.tolist() + keys = _keys(logits.float()) + for r in range(rows): + end = max(0, min(columns, lens[r // next_n] - next_n + r % next_n + 1)) + block = end // bp + last = end - block * (bp - 1) + pad = (bp - 1) * max(0, TOP_K - block) + max(0, TOP_K - last) + if not bool((keys[r, :end] > _kernel.NEG_INF_KEY).any()): + won, _ = _top(keys[r, :end], pad, TOP_K) + else: + # A negative NaN: the op's own two launches, block by block, then the merge. + passed = [] + for j in range(bp): + lo, hi = j * block, (end if j == bp - 1 else (j + 1) * block) + if hi - lo <= TOP_K: + passed.append(torch.arange(lo, hi, device=logits.device)) + else: + picked, _ = _stage(logits[r, lo:hi], keys[r, lo:hi], 0) + passed.append(lo + picked) + cand = torch.cat(passed) + picked, _ = _stage(logits[r, cand], keys[r, cand], pad) + won = cand[picked] + won = won.sort().values + out[r, : won.numel()] = won.to(torch.int32) + return out + + def is_valid( + self, + logits: torch.Tensor, + seq_lens: torch.Tensor, + indices: torch.Tensor, + *, + next_n: int, + compiled: CompiledRadixTopkDecode, + ) -> None: + # The kernel clamps a causal end past the row to the row's width and answers for the + # truncated row; the op reads past it. The lengths are device data that a captured + # call reads only at replay, so the check needs an eager call. + if seq_lens.numel() and not torch.cuda.is_current_stream_capturing(): + widest = int(seq_lens.max()) + assert widest <= logits.shape[1], ( + f"a causal end of {widest} exceeds the {logits.shape[1]}-column logits" + ) + + +cute_dsl_radix_topk_decode = CuteDslRadixTopkDecode() + + +@dataclass(frozen=True) +class CompiledSplitRadixTopkDecode: + """Precompiled partition and merge launches; scratch is supplied by the caller.""" + + stage1: object + stage2: object + device_id: int + next_n: int + partitions: int + use_pdl: bool + sm_count: int + + +_SPLIT_COMPILED: dict[tuple[int, int, int, bool], CompiledSplitRadixTopkDecode] = {} + + +def build_split( + device_id: int, *, next_n: int = 4, partitions: int = 10, use_pdl: bool = True +) -> CompiledSplitRadixTopkDecode: + """Compile a two-launch selector, before capture, for SM107 component experiments. + + Rows, width and row stride are symbolic. The selector retains the long-row native + split's ten logical partition boundaries and sentinel padding. + """ + if next_n not in (1, 4) or partitions != 10: + raise ValueError("split radix requires next_n 1/4 and exactly 10 native partitions") + key = (int(device_id), int(next_n), int(partitions), bool(use_pdl)) + if key in _SPLIT_COMPILED: + return _SPLIT_COMPILED[key] + with torch.cuda.device(device_id): + rows, cols, batch, size = (cute.sym_int() for _ in range(4)) + stride = cute.sym_int64() + fake_logits = make_fake_tensor( + cutlass.Float32, (rows, cols), stride=(stride, 1), assumed_align=4 + ) + fake_lens = make_fake_compact_tensor(cutlass.Int32, (batch,), assumed_align=4) + fake_aux_i = make_fake_compact_tensor(cutlass.Int32, (size,), assumed_align=4) + fake_aux_l = make_fake_compact_tensor(cutlass.Float32, (size,), assumed_align=4) + fake_out = make_fake_compact_tensor( + cutlass.Int32, (rows, TOP_K), stride_order=(1, 0), assumed_align=4 + ) + stream = make_fake_stream(use_tvm_ffi_env_stream=True) + kernel = _kernel.SplitRadixTopkDecodeKernel(next_n, partitions, use_pdl) + stage1 = cute.compile( + kernel.split, + fake_logits, + fake_lens, + fake_aux_i, + fake_aux_l, + stream, + options="--enable-tvm-ffi", + ) + stage2 = cute.compile( + kernel.merge, + fake_aux_i, + fake_aux_l, + fake_out, + stream, + options="--enable-tvm-ffi", + ) + sm_count = torch.cuda.get_device_properties(device_id).multi_processor_count + handle = CompiledSplitRadixTopkDecode(stage1, stage2, *key, int(sm_count)) + logits = torch.zeros((next_n, 4096), dtype=torch.float32, device="cuda") + lens = torch.full((1,), 4096, dtype=torch.int32, device="cuda") + out = torch.empty((next_n, TOP_K), dtype=torch.int32, device="cuda") + aux_i = torch.empty(next_n * partitions * TOP_K, dtype=torch.int32, device="cuda") + aux_l = torch.empty_like(aux_i, dtype=torch.float32) + stage1(logits, lens, aux_i, aux_l) + stage2(aux_i, aux_l, out) + torch.cuda.synchronize() + _SPLIT_COMPILED[key] = handle + return handle + + +def split_radix_topk_decode( + logits: torch.Tensor, + seq_lens: torch.Tensor, + indices: torch.Tensor, + aux_indices: torch.Tensor, + aux_logits: torch.Tensor, + *, + compiled: CompiledSplitRadixTopkDecode, +) -> torch.Tensor: + """Select K=2048 from fp32 ``logits[batch * next_n, width]`` into int32 ``indices``. + + ``seq_lens[batch]`` is int32 and read on device for each replay. Both contiguous scratch + buffers hold at least ``rows * partitions * 2048`` elements; their initial contents are + ignored. Unit-stride logits may have padded rows and nonzero storage offsets. This + selector uses the native op when its shape would select fewer than ten partitions, + preserving that shape's sentinel and nonfinite behavior, and when its rows are not + 32-byte aligned with a width that is a multiple of 8, which its vector reads require. + """ + if logits.ndim != 2 or logits.dtype != torch.float32 or not logits.is_cuda: + raise ValueError("logits must be a two-dimensional CUDA float32 tensor") + rows, width = logits.shape + if not 4096 <= width <= 1048584 or rows > 64: + raise ValueError("split radix requires width 4096..1048584 and at most 64 rows") + if logits.stride(1) != 1 or logits.stride(0) < width: + raise ValueError("logits must have unit column stride and non-overlapping rows") + if seq_lens.shape != (rows // compiled.next_n,) or rows % compiled.next_n: + raise ValueError("seq_lens must contain one length per request") + if indices.shape != (rows, TOP_K): + raise ValueError(f"indices must have shape ({rows}, {TOP_K})") + native_partitions = blocks_per_row(rows, width, compiled.sm_count) + needed = rows * native_partitions * TOP_K + for name, tensor, dtype in ( + ("seq_lens", seq_lens, torch.int32), + ("indices", indices, torch.int32), + ("aux_indices", aux_indices, torch.int32), + ("aux_logits", aux_logits, torch.float32), + ): + if tensor.dtype != dtype or tensor.device != logits.device or not tensor.is_contiguous(): + raise ValueError(f"{name} must be contiguous {dtype} on {logits.device}") + if logits.device.index != compiled.device_id: + raise ValueError("compiled selector and logits must be on the same CUDA device") + if aux_indices.numel() < needed or aux_logits.numel() < needed: + raise ValueError(f"scratch must hold at least {needed} elements") + vector_rows = logits.data_ptr() % 32 == 0 and logits.stride(0) % 8 == 0 and width % 8 == 0 + if rows: + if native_partitions != compiled.partitions or not vector_rows: + return indexer_topk_decode( + logits, seq_lens, indices, compiled.next_n, TOP_K, 1, aux_indices, aux_logits + ) + flat_i, flat_l = aux_indices.view(-1), aux_logits.view(-1) + compiled.stage1(logits, seq_lens, flat_i, flat_l) + compiled.stage2(flat_i, flat_l, indices) + return indices + + +def validate_split_radix_topk_lengths( + logits: torch.Tensor, seq_lens: torch.Tensor, *, next_n: int +) -> None: + """Eager test guard for device metadata; never call from the production launch path. + + A legal request contains the preceding ``next_n - 1`` tokens, with its last causal end + within the score row. CUDA graph capture must use already validated legal metadata. + """ + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError("validate split radix lengths before CUDA graph capture") + if seq_lens.numel(): + shortest, longest = torch.aminmax(seq_lens) + if int(shortest) < next_n - 1 or int(longest) > logits.shape[1]: + raise ValueError("seq_lens must be within [next_n - 1, logits width]") diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/ds_mla_dequant_gather.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/ds_mla_dequant_gather.py new file mode 100644 index 000000000000..678686347123 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/ds_mla_dequant_gather.py @@ -0,0 +1,674 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Gather context requests' fp8_ds_mla latent rows out of the paged pool, dequantized to bf16.""" + +from __future__ import annotations + +import torch +import triton +import triton.language as tl + +from tensorrt_llm._torch.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +DIM_NOPE = 512 +DIM_ROPE = 64 +TILE = 128 +ROW_BYTES = 656 + + +@triton.jit +def _gather_kernel( + pool_ptr, # uint8 [blocks, owners, tpb, 656] + owners, + row, + kv_req_ptr, # int32 [N] + kv_pos_ptr, # int32 [N] + block_table_ptr, + bt_stride, + out_ptr, # bf16 [N, 576] (row stride out_stride) + out_stride, + TPB: tl.constexpr, +): + w = tl.program_id(0) + part = tl.program_id(1) + req = tl.load(kv_req_ptr + w).to(tl.int64) + pos = tl.load(kv_pos_ptr + w).to(tl.int64) + block = tl.load(block_table_ptr + req * bt_stride + pos // TPB).to(tl.int64) + src = pool_ptr + ((block * owners + row) * TPB + pos % TPB) * 656 + dst = out_ptr + w.to(tl.int64) * out_stride + if part < 4: + cols = part * 128 + tl.arange(0, 128) + codes = tl.load(src + cols).to(tl.float8e4nv, bitcast=True).to(tl.float32) + scale = tl.load((src + 512 + part * 4).to(tl.pointer_type(tl.float32))) + tl.store(dst + cols, (codes * scale).to(tl.bfloat16)) + else: + r = tl.arange(0, 64) + rope = tl.load((src + 528).to(tl.pointer_type(tl.bfloat16)) + r) + tl.store(dst + 512 + r, rope) + + +class DsMlaDequantGather(OpWrapper): + """`out[w] = dequant(pool[block_table[kv_req[w], kv_pos[w] // tpb], row, kv_pos[w] % tpb])`:: + + out[w, :512] = e4m3(code) * tile_scale (exact in bf16: 3 mantissa bits times 2^k) + out[w, 512:576] = the stored bf16 rope + + The bf16 workspace a dense-key sparse prefill kernel reads: the context requests' + whole KV (cached prefix and this step's tokens), each request's run contiguous. + `pool`: uint8 `[blocks, owners, tokens_per_block, 656]`; `kv_req` / `kv_pos`: + int32 `[N]`; `out`: bf16 `[N, 576]` (row stride free). Returns `out`. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # spec: context requests with `cached` + `new` tokens each, a pool of `blocks` x + # `owners` pages of `tokens_per_block`, owner `row`. The "1M tp4", "1M tep4", "1M dep4" + # and "1M tep8" cells add `out_rows` (`out` leads a buffer of that many rows, the rest + # left alone), the block table's `table_rows` (one per request when absent) x + # `table_width`, `nope` (zero rope: the NoPE source's rows), `far` (the requests' pages + # are the pool's top ones) and `prefixes` (every N below the cell's is driven too, + # each bitwise the leading rows of the full call). + CELLS: tuple[Cell, ...] = ( + Cell( + why="one 32768-token request (the unchunked envelope)", + spec=dict(cached=[0], new=[32768], blocks=1100, owners=63, tokens_per_block=32, row=31), + ), + Cell( + why="three requests, cached prefixes mid-page, the last owner row", + spec=dict( + cached=[0, 1029, 5185], + new=[300, 2047, 1], + blocks=600, + owners=63, + tokens_per_block=32, + row=62, + ), + ), + Cell( + why="sixteen 2048-token requests (a full prefill batch)", + spec=dict( + cached=[0] * 16, new=[2048] * 16, blocks=1100, owners=63, tokens_per_block=32, row=0 + ), + ), + Cell( + why="64-token pages", + spec=dict(cached=[37], new=[1000], blocks=64, owners=63, tokens_per_block=64, row=2), + ), + # The 32-token cells above at 64-token pages: the same requests and owner rows, half + # the blocks for the same token capacity. + Cell( + why="64-token pages: one 32768-token request (the unchunked envelope)", + spec=dict(cached=[0], new=[32768], blocks=550, owners=63, tokens_per_block=64, row=31), + ), + Cell( + why="64-token pages: three requests, cached prefixes mid-page, the last owner row", + spec=dict( + cached=[0, 1029, 5185], + new=[300, 2047, 1], + blocks=300, + owners=63, + tokens_per_block=64, + row=62, + ), + ), + Cell( + why="64-token pages: sixteen 2048-token requests (a full prefill batch)", + spec=dict( + cached=[0] * 16, new=[2048] * 16, blocks=550, owners=63, tokens_per_block=64, row=0 + ), + ), + # The tp4 deployment at the 1M envelope, workspace path: a step whose context KV (cached + + # new over its context requests) is at most B = 2**19 rows gathers it into rows [0, N) + # of one bf16 [B, 576] buffer, out = buf[:N]; md.block_table is [16, 32772]. + Cell( + why="1M tp4 workspace at N = B = 2**19: a 32768-row chunk behind a 491520-token " + "prefix, owner row 62, its 16384 pages anywhere in a 16500-block pool (20.3 GiB, " + "byte offsets past 2**34); every N 1..B a bitwise prefix, rows past N untouched", + spec=dict( + cached=[491520], + new=[32768], + blocks=16500, + owners=63, + tokens_per_block=32, + row=62, + out_rows=524288, + table_width=32772, + prefixes=True, + ), + ), + Cell( + why="1M tp4 workspace at N = B, NoPE source: the nine sharers re-gather layer 35's " + "row 35 (zero rope) into the same buffer, 491520 cached + 32768 new", + spec=dict( + cached=[491520], + new=[32768], + blocks=16500, + owners=63, + tokens_per_block=32, + row=35, + out_rows=524288, + table_width=32772, + nope=True, + ), + ), + Cell( + why="1M tp4 packed chunk step: four requests cached 262144/131072/0/2048 + " + "16384/8192/4096/4096 (N = 428032 of the 2**19-row buffer), owner row 1", + spec=dict( + cached=[262144, 131072, 0, 2048], + new=[16384, 8192, 4096, 4096], + blocks=13500, + owners=63, + tokens_per_block=32, + row=1, + out_rows=524288, + table_width=32772, + ), + ), + Cell( + why="1M tp4 short step in the stale 2**19-row buffer: three requests, prefixes " + "mid-page (N = 9580), the 32772-column table", + spec=dict( + cached=[0, 2047, 5185], + new=[300, 1, 2047], + blocks=600, + owners=63, + tokens_per_block=32, + row=7, + out_rows=524288, + table_width=32772, + ), + ), + Cell( + why="1M tp4 sixteen requests filling the 2**19-row buffer: 30720 cached + 2048 new " + "each, kv_req 0..15 over all 16 rows of the 32772-column table", + spec=dict( + cached=[30720] * 16, + new=[2048] * 16, + blocks=16500, + owners=63, + tokens_per_block=32, + row=0, + out_rows=524288, + table_width=32772, + ), + ), + Cell( + why="1M tp4 boot-sized pool, large: 74183 blocks (91.4 GiB), the 491520 + 32768 " + "request's 16384 pages the pool's top ones (byte offsets to 98.1e9), N = B", + spec=dict( + cached=[491520], + new=[32768], + blocks=74183, + owners=63, + tokens_per_block=32, + row=62, + out_rows=524288, + table_width=32772, + far=True, + large=True, + ), + ), + # A 1M prompt's chunk steps on the workspace path: chunk k is 32768 new rows behind a + # 32768 * (k - 1)-token prefix, N = 32768 * k, for k = 1..15 (chunk 16 is the N = B + # cell above; past it the decode path). Every owner layer gathers each chunk at its + # own row; the chunks here take rows 2, 6, .., 58. + *( + Cell( + why=f"1M tp4 chunk {k} of a 1M prompt at N = {32768 * k}: 32768 new rows behind " + f"a {32768 * (k - 1)}-token prefix into the 2**19-row buffer, owner row " + f"{4 * k - 2}, its pages anywhere in a 16500-block pool", + spec=dict( + cached=[32768 * (k - 1)], + new=[32768], + blocks=16500, + owners=63, + tokens_per_block=32, + row=4 * k - 2, + out_rows=524288, + table_width=32772, + ), + ) + for k in range(1, 16) + ), + # A fresh request's dense step: the engine's warmup steps (1 and 2 rows) and a short + # request (7 rows), into the first rows of the held buffer. + *( + Cell( + why=f"1M tp4 fresh {n}-row step (the engine warmup steps, a short request): " + f"N = {n} of the stale 2**19-row buffer, its page the pool's last one (the " + "16500-block pool's top block, owner row 62)", + spec=dict( + cached=[0], + new=[n], + blocks=16500, + owners=63, + tokens_per_block=32, + row=62, + out_rows=524288, + table_width=32772, + far=True, + ), + ) + for n in (1, 2, 7) + ), + # The Mistral Large 4 NVFP4 tep4 deployment's workspace path at max_seq_len 1048576 with + # chunked prefill: a step whose context KV (cached + new over its context requests) is + # at most B = 2**19 rows gathers it into rows [0, N) of one bf16 [B, 576] buffer, + # out = buf[:N], through md.block_table [16, 32772]; owners 63, 32 tokens per block. + Cell( + why="1M tep4 workspace at N = B = 2**19: a 32768-row chunk behind a 491520-token " + "prefix, owner row 62, its 16384 pages anywhere in a 16500-block pool (20.3 GiB, " + "byte offsets to 2.18e10); every N below B a bitwise prefix of the full call", + spec=dict( + cached=[491520], + new=[32768], + blocks=16500, + owners=63, + tokens_per_block=32, + row=62, + out_rows=524288, + table_rows=16, + table_width=32772, + prefixes=True, + ), + ), + Cell( + why="1M tep4 NoPE source at N = B: the nine sharers re-gather layer 35's row 35 " + "(zero rope) into the same 2**19-row buffer, 491520 cached + 32768 new", + spec=dict( + cached=[491520], + new=[32768], + blocks=16500, + owners=63, + tokens_per_block=32, + row=35, + out_rows=524288, + table_rows=16, + table_width=32772, + nope=True, + ), + ), + Cell( + why="1M tep4 packed step, N = 428032: four requests cached 262144/131072/0/2048 " + "+ new 16384/8192/4096/4096 over a 13500-block pool, owner row 1", + spec=dict( + cached=[262144, 131072, 0, 2048], + new=[16384, 8192, 4096, 4096], + blocks=13500, + owners=63, + tokens_per_block=32, + row=1, + out_rows=524288, + table_rows=16, + table_width=32772, + ), + ), + Cell( + why="1M tep4 short step, N = 9580, leading the 2**19-row buffer: three requests " + "with prefixes mid-page, owner row 7", + spec=dict( + cached=[0, 2047, 5185], + new=[300, 1, 2047], + blocks=600, + owners=63, + tokens_per_block=32, + row=7, + out_rows=524288, + table_rows=16, + table_width=32772, + ), + ), + Cell( + why="1M tep4 sixteen requests at N = B: 30720 cached + 2048 new each (a full " + "32768-row step), kv_req 0..15 over every row of the table, owner row 0", + spec=dict( + cached=[30720] * 16, + new=[2048] * 16, + blocks=16500, + owners=63, + tokens_per_block=32, + row=0, + out_rows=524288, + table_rows=16, + table_width=32772, + ), + ), + Cell( + why="1M tep4 boot-sized pool, large: 74183 blocks (91.4 GiB), the 491520 + 32768 " + "request's 16384 pages its top ones (byte offsets to 9.81e10), N = B, row 62", + spec=dict( + cached=[491520], + new=[32768], + blocks=74183, + owners=63, + tokens_per_block=32, + row=62, + out_rows=524288, + table_rows=16, + table_width=32772, + far=True, + large=True, + ), + ), + # The Mistral Large 4 NVFP4 dep4 deployment's workspace path at max_seq_len 1048576 with + # chunked prefill, under attention DP (a rank's own context requests, up to 16): a step + # whose context KV on the rank (cached + new over its context requests) is at most + # B = 2**19 rows gathers it into rows [0, N) of one bf16 [B, 576] buffer, out = buf[:N], + # through md.block_table [16, 32772]; owners 63, 32 tokens per block. + Cell( + why="1M dep4 workspace at N = B = 2**19: a 32768-row chunk behind a 491520-token " + "prefix, owner row 62, its 16384 pages anywhere in a 16500-block pool (20.3 GiB, " + "byte offsets to 2.18e10); every N below B a bitwise prefix of the full call", + spec=dict( + cached=[491520], + new=[32768], + blocks=16500, + owners=63, + tokens_per_block=32, + row=62, + out_rows=524288, + table_rows=16, + table_width=32772, + prefixes=True, + ), + ), + Cell( + why="1M dep4 NoPE source at N = B: the nine sharers re-gather layer 35's row 35 " + "(zero rope) into the same 2**19-row buffer, 491520 cached + 32768 new", + spec=dict( + cached=[491520], + new=[32768], + blocks=16500, + owners=63, + tokens_per_block=32, + row=35, + out_rows=524288, + table_rows=16, + table_width=32772, + nope=True, + ), + ), + Cell( + why="1M dep4 packed step, N = 428032: four of the rank's requests cached " + "262144/131072/0/2048 + new 16384/8192/4096/4096, a 13500-block pool, owner row 1", + spec=dict( + cached=[262144, 131072, 0, 2048], + new=[16384, 8192, 4096, 4096], + blocks=13500, + owners=63, + tokens_per_block=32, + row=1, + out_rows=524288, + table_rows=16, + table_width=32772, + ), + ), + Cell( + why="1M dep4 short step, N = 9580, leading the 2**19-row buffer: three requests " + "with prefixes mid-page, owner row 7", + spec=dict( + cached=[0, 2047, 5185], + new=[300, 1, 2047], + blocks=600, + owners=63, + tokens_per_block=32, + row=7, + out_rows=524288, + table_rows=16, + table_width=32772, + ), + ), + Cell( + why="1M dep4 sixteen requests at N = B: a rank's 16 own requests, 30720 cached + " + "2048 new each (a full 32768-row step), kv_req 0..15 over every table row, owner row 0", + spec=dict( + cached=[30720] * 16, + new=[2048] * 16, + blocks=16500, + owners=63, + tokens_per_block=32, + row=0, + out_rows=524288, + table_rows=16, + table_width=32772, + ), + ), + Cell( + why="1M dep4 boot-sized pool, large: 74183 blocks (91.4 GiB), the 491520 + 32768 " + "request's 16384 pages its top ones (byte offsets to 9.81e10), N = B, row 62", + spec=dict( + cached=[491520], + new=[32768], + blocks=74183, + owners=63, + tokens_per_block=32, + row=62, + out_rows=524288, + table_rows=16, + table_width=32772, + far=True, + large=True, + ), + ), + # The Mistral Large 4 NVFP4 tep8 deployment's workspace path at max_seq_len 1048576 with + # chunked prefill: a step whose context KV (cached + new over its context requests) is + # at most B = 2**19 rows gathers it into rows [0, N) of one bf16 [B, 576] buffer, + # out = buf[:N], through md.block_table [16, 32772]; owners 63, 32 tokens per block. + Cell( + why="1M tep8 workspace at N = B = 2**19: a 32768-row chunk behind a 491520-token " + "prefix, owner row 62, its 16384 pages anywhere in a 16500-block pool (20.3 GiB, " + "byte offsets to 2.18e10); every N below B a bitwise prefix of the full call", + spec=dict( + cached=[491520], + new=[32768], + blocks=16500, + owners=63, + tokens_per_block=32, + row=62, + out_rows=524288, + table_rows=16, + table_width=32772, + prefixes=True, + ), + ), + Cell( + why="1M tep8 NoPE source at N = B: the nine sharers re-gather layer 35's row 35 " + "(zero rope) into the same 2**19-row buffer, 491520 cached + 32768 new", + spec=dict( + cached=[491520], + new=[32768], + blocks=16500, + owners=63, + tokens_per_block=32, + row=35, + out_rows=524288, + table_rows=16, + table_width=32772, + nope=True, + ), + ), + Cell( + why="1M tep8 packed step, N = 428032: four requests cached 262144/131072/0/2048 " + "+ new 16384/8192/4096/4096 over a 13500-block pool, owner row 1", + spec=dict( + cached=[262144, 131072, 0, 2048], + new=[16384, 8192, 4096, 4096], + blocks=13500, + owners=63, + tokens_per_block=32, + row=1, + out_rows=524288, + table_rows=16, + table_width=32772, + ), + ), + Cell( + why="1M tep8 short step, N = 9580, leading the 2**19-row buffer: three requests " + "with prefixes mid-page, owner row 7", + spec=dict( + cached=[0, 2047, 5185], + new=[300, 1, 2047], + blocks=600, + owners=63, + tokens_per_block=32, + row=7, + out_rows=524288, + table_rows=16, + table_width=32772, + ), + ), + Cell( + why="1M tep8 sixteen requests at N = B: 30720 cached + 2048 new each (a full " + "32768-row step), kv_req 0..15 over every row of the table, owner row 0", + spec=dict( + cached=[30720] * 16, + new=[2048] * 16, + blocks=16500, + owners=63, + tokens_per_block=32, + row=0, + out_rows=524288, + table_rows=16, + table_width=32772, + ), + ), + Cell( + why="1M tep8 bound-sized pool, large: 147456 blocks (181.6 GiB), the 491520 + 32768 " + "request's 16384 pages its top ones (byte offsets to 1.95e11), N = B, row 62", + spec=dict( + cached=[491520], + new=[32768], + blocks=147456, + owners=63, + tokens_per_block=32, + row=62, + out_rows=524288, + table_rows=16, + table_width=32772, + far=True, + large=True, + ), + ), + ) + + note = """ + Call sites: mistral_large4 targets' context rows, once per owner layer and once + per step for the NoPE source (its sharers reuse that workspace). Bitwise equal to + `reference` (the cells). Launch: Triton, grid (N, 5); not captured (context steps). + A call writes 37.7 MB per 32768 rows and 604 MB at 2**19 rows; the 147456-block cell + peaks at 185.65 GiB allocated. + + The "1M tp4", "1M tep4", "1M dep4" and "1M tep8" cells are the Mistral Large 4 NVFP4 tp4, + tep4, dep4 and tep8 deployments' workspace path at max_seq_len 1048576 with chunked + prefill (steps of up to 32768 new rows and 16 requests; under dep4's attention DP, a + rank's own requests): a step whose context KV (on the rank), cached + new summed over + its context requests, is at most B = 2**19 rows gathers it into rows [0, N) of one + bf16 [B, 576] buffer (576 MiB), out = buf[:N], its later rows stale; the owner layers + at their own row and the NoPE sharers at row 35 again, into the same buffer. A step + above B gathers nothing (the sparse decode kernel attends it off the pool), so no cell + passes N > 2**19. Each step a 1M prompt takes on this path is a "1M tp4" cell: its + chunks k = 1..16, 32768 new rows behind 32768 (k - 1) cached, N = 32768 k (k = 16 the + N = B cell), and the fresh dense steps of 1, 2 and 7 rows (the engine's warmup steps + and a short request). The kernel writes rows [0, N) of `out` and nothing past them: + the cells hold the buffer's later rows to a sentinel, and the N = B cells hold every N + below B to the full call's leading rows (bitwise). Pages a cell's requests do not + hold are filled with 0xFF (NaN in every field), so a read from a wrong page, owner + row or offset changes the bits the cells compare. The tp4, tep4 and dep4 large cells' + 74183-block pool lies above the dep4 boot's fp8_ds_mla pool (64292 blocks per rank, + measured at fraction 0.9: 2057344 tokens), with the request's pages at its top. The + tep8 pool at the 1M boot is 114021 blocks per rank (3,648,672 tokens, read back from + the 1M tep8 boot on fp8_ds_mla at fraction 0.9) and cannot reach 147456 blocks + (147456 x 32 x 47,136 B = 222.4 GB, more than a rank's whole free memory): the tep8 + large cell's pool is 147456 blocks, the request's pages its top ones. + + Hard limits (sm_103) + pool byte offset ((block * owners + row) * tpb + pos % tpb) * 656, int64 in the + kernel; past 2**31 bytes from block 1623 row 51, every row from + block 1624 on (owners 63, tpb 32); probed at 2.18e10 bytes (16500 + blocks), 9.81e10 bytes (74183 blocks) and 1.95e11 bytes (147456 + blocks), the pages the pool's top ones in the large cells passed + out row offset w * out.stride(0), int64; the int32 boundary 2**31 / 576 = + 3728270 rows lies past every cell (N <= 2**19) not reached + kv_req, kv_pos int32 tensors, widened to int64 in the kernel; kv_req probed to + 15, kv_pos to 524287 passed + block_table req * block_table.stride(0) + pos // tpb, int64; a [16, 32772] + int32 table read to row 15 and column 16383 passed + N grid x; every N 1..64, 2**k - 1 / 2**k / 2**k + 1 from 2**7, every + 4093rd and every 4099th, every multiple of 32768 and 2**19 - 1, + each a prefix of the 2**19 call passed + no index passes through fp32. + + is_valid refuses an `out` with fewer rows than `kv_req` (a shape check, no sync): the + kernel writes a row for every kv_req entry, past out's end, silently (the guard tests + show it). It reads kv_pos at every kv_req row as well; nothing compares kv_pos's + length with kv_req's. + """ + + def __call__( + self, + pool: torch.Tensor, + row: int, + kv_req: torch.Tensor, + kv_pos: torch.Tensor, + block_table: torch.Tensor, + out: torch.Tensor, + ) -> torch.Tensor: + n = kv_req.shape[0] + if n: + _gather_kernel[(n, 5)]( + pool, pool.shape[1], row, kv_req, kv_pos, block_table, block_table.stride(0), + out, out.stride(0), TPB=pool.shape[2], + ) # fmt: skip + return out + + def reference( + self, + pool: torch.Tensor, + row: int, + kv_req: torch.Tensor, + kv_pos: torch.Tensor, + block_table: torch.Tensor, + out: torch.Tensor, + ) -> torch.Tensor: + """A new tensor shaped like `out`.""" + tpb = pool.shape[2] + p = kv_pos.long() + block = block_table[kv_req.long(), p // tpb].long() + rows = pool[block, row, p % tpb] + codes = rows[:, :DIM_NOPE].contiguous().view(torch.float8_e4m3fn).float() + scale = rows[:, DIM_NOPE : DIM_NOPE + 16].contiguous().view(torch.float32) + nope = (codes.view(-1, DIM_NOPE // TILE, TILE) * scale.unsqueeze(-1)).view(-1, DIM_NOPE) + rope = rows[:, DIM_NOPE + 16 :].contiguous().view(torch.bfloat16) + res = torch.empty_like(out) + res[:, :DIM_NOPE] = nope.to(torch.bfloat16) + res[:, DIM_NOPE:] = rope + return res + + def is_valid( + self, + pool: torch.Tensor, + row: int, + kv_req: torch.Tensor, + kv_pos: torch.Tensor, + block_table: torch.Tensor, + out: torch.Tensor, + ) -> None: + if pool.dtype != torch.uint8 or pool.dim() != 4 or pool.shape[-1] != ROW_BYTES: + raise ValueError("pool must be uint8 [blocks, owners, tpb, 656]") + if not pool.is_contiguous() or not 0 <= row < pool.shape[1]: + raise ValueError("pool must be contiguous and row one of its owner rows") + if ( + out.dtype != torch.bfloat16 + or out.shape[-1] != DIM_NOPE + DIM_ROPE + or out.stride(-1) != 1 + ): + raise ValueError("out must be bf16 [N, 576] with contiguous rows") + if out.shape[0] < kv_req.shape[0]: + raise ValueError( + f"out has {out.shape[0]} rows for {kv_req.shape[0]} kv_req entries: the kernel " + "writes a row per entry, past out's end" + ) + + +ds_mla_dequant_gather = DsMlaDequantGather() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/ds_mla_rope_append.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/ds_mla_rope_append.py new file mode 100644 index 000000000000..7fb1bdbd66ae --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/ds_mla_rope_append.py @@ -0,0 +1,1337 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Rope the query and key tails and append each token's latent to an fp8_ds_mla pool.""" + +from __future__ import annotations + +from typing import Optional + +import torch +import triton +import triton.language as tl + +from tensorrt_llm._torch.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: One fp8_ds_mla latent row: 512 e4m3 codes | 4 fp32 tile scales | 64 bf16 rope values. +DIM_NOPE = 512 +DIM_ROPE = 64 +TILE = 128 +NUM_TILES = DIM_NOPE // TILE +SCALE_OFFSET = DIM_NOPE +ROPE_OFFSET = DIM_NOPE + 4 * NUM_TILES +TOKEN_BYTES = ROPE_OFFSET + 2 * DIM_ROPE +FP8_MAX = 448.0 +#: The floor on a tile's amax / 448 before it is rounded up to a power of two. +SCALE_FLOOR = 1e-4 + +#: What every `_ML4_1M` cell shares: the tep4 and tp4 deployments at max_seq_len 1048576 -- 16 query +#: heads (the first of the 64-head fused-query scratch), the 63 KV-owner rows of +#: MistralLarge4CacheManager's latent pool, 32-token pages, the 1048576-row `_rope.indexer` +#: table and md.block_table's 32772 columns (the V2 manager's max_blocks_per_seq at 1M). +_ML4_1M = dict(heads=16, owners=63, tokens_per_block=32, max_pos=1048576, table_width=32772) + +#: What every `_ML4_1M_DEP4` cell shares: the dep4 deployment at max_seq_len 1048576 -- all 64 +#: query heads of a rank's fused-query scratch (attention DP, no padding heads), 63 KV +#: owners, 32-token pages, the 1048576-row `_rope.indexer` table and the V2 manager's +#: 32772-column `md.block_table`. +_ML4_1M_DEP4 = dict(heads=64, owners=63, tokens_per_block=32, max_pos=1048576, table_width=32772) + +#: What every `_ML4_1M_TEP8` cell shares: the tep8 deployment at max_seq_len 1048576 -- 8 query +#: heads (the first of its 64-head fused-query scratch), 63 KV owners, 32-token pages, the +#: 1048576-row `_rope.indexer` table and the V2 manager's 32772-column `md.block_table`. +_ML4_1M_TEP8 = dict(heads=8, owners=63, tokens_per_block=32, max_pos=1048576, table_width=32772) + + +@triton.jit +def _pow2_ceil(x): + """2^ceil(log2(x)) for a positive normal fp32 x, exact: from the exponent bits.""" + bits = x.to(tl.int32, bitcast=True) + exp = (bits >> 23) & 0xFF + exp = exp + ((bits & 0x7FFFFF) != 0).to(tl.int32) + return (exp << 23).to(tl.float32, bitcast=True) + + +@triton.jit +def _rope_append_kernel( + latent_ptr, # bf16 [T, 576]: RMSNorm(c_kv) | k_pe (unroped) + lat_stride, + q_ptr, # bf16 query rows; its rope tail [.., 512:576] per head, roped in place + q_stride_t, + q_stride_h, + num_q_heads, + pos_ptr, # int32/int64 [T] + table_ptr, # fp32 [positions, 64]: cos of the 32 pairs, then their sin + req_ptr, # int32 [T]: the token's request row in block_table + block_table_ptr, + bt_stride, + pool_ptr, # uint8 [blocks, owners, tokens_per_block, 656] + owners, + row, + TPB: tl.constexpr, + ROPE: tl.constexpr, + HAS_Q: tl.constexpr, + H_BLOCK: tl.constexpr, +): + t = tl.program_id(0) + part = tl.program_id(1) + pos = tl.load(pos_ptr + t).to(tl.int64) + req = tl.load(req_ptr + t).to(tl.int64) + block = tl.load(block_table_ptr + req * bt_stride + pos // TPB).to(tl.int64) + base = ((block * owners + row) * TPB + pos % TPB) * 656 + lat = latent_ptr + t.to(tl.int64) * lat_stride + half = tl.arange(0, 32) + if part < 4: + cols = part * 128 + tl.arange(0, 128) + xt = tl.load(lat + cols).to(tl.float32) + # IEEE quotients: an approximate amax / 448 can cross a power of two and double the + # scale. A power-of-two scale divides exactly; |x / scale| <= 448 by construction. + ratio = tl.maximum(tl.fdiv(tl.max(tl.abs(xt)), 448.0, ieee_rounding=True), 1e-4) + scale = _pow2_ceil(ratio) + codes = tl.fdiv(xt, scale, ieee_rounding=True).to(tl.float8e4nv) + tl.store(pool_ptr + base + cols, codes.to(tl.uint8, bitcast=True)) + sp = (pool_ptr + base + 512 + part * 4).to(tl.pointer_type(tl.float32)) + tl.store(sp, scale) + elif part == 4: + k0 = tl.load(lat + 512 + 2 * half).to(tl.float32) + k1 = tl.load(lat + 513 + 2 * half).to(tl.float32) + if ROPE: + kc = tl.load(table_ptr + pos * 64 + half) + ks = tl.load(table_ptr + pos * 64 + 32 + half) + y0 = k0 * kc - k1 * ks + y1 = k0 * ks + k1 * kc + else: + y0 = k0 + y1 = k1 + rp = (pool_ptr + base + 528).to(tl.pointer_type(tl.bfloat16)) + tl.store(rp + 2 * half, y0.to(tl.bfloat16)) + tl.store(rp + 2 * half + 1, y1.to(tl.bfloat16)) + else: + if HAS_Q and ROPE: + qc = tl.load(table_ptr + pos * 64 + half) + qs = tl.load(table_ptr + pos * 64 + 32 + half) + heads = tl.arange(0, H_BLOCK) + hmask = heads[:, None] < num_q_heads + qp = ( + q_ptr + + t.to(tl.int64) * q_stride_t + + heads[:, None].to(tl.int64) * q_stride_h + + 512 + + 2 * half[None, :] + ) + q0 = tl.load(qp, mask=hmask).to(tl.float32) + q1 = tl.load(qp + 1, mask=hmask).to(tl.float32) + tl.store(qp, (q0 * qc[None, :] - q1 * qs[None, :]).to(tl.bfloat16), mask=hmask) + tl.store(qp + 1, (q0 * qs[None, :] + q1 * qc[None, :]).to(tl.bfloat16), mask=hmask) + + +def _pow2_ceil_torch(ratio: torch.Tensor) -> torch.Tensor: + mantissa, exponent = torch.frexp(ratio) + return torch.exp2((exponent - (mantissa == 0.5).to(exponent.dtype)).float()) + + +class DsMlaRopeAppend(OpWrapper): + """Rope `q`'s rope tail in place and write every token's fp8_ds_mla latent row:: + + rope(x)[2i:2i+2] = (x0 c_i - x1 s_i, x0 s_i + x1 c_i) in fp32, rounded to bf16 once + (GPT-J interleaved pairs, c_i / s_i = table[pos, i] / [pos, 32 + i]) + scale[j] = 2^ceil(log2(max(max|c_kv[128j:128j+128]| / 448, 1e-4))) (fp32) + row = [e4m3_rn(c_kv / scale) | scale[0..3] | bf16 rope(k_pe)] (656 B) + + The row recipe is the reference implementation's fp8_ds_mla writer + (`concat_and_cache_ds_mla`); a power-of-two scale makes the quotient exact and lets + a reader that holds the scale as E8M0 read it exactly. `rope=False` (a NoPE layer) + copies the key tail and leaves `q` alone. The token at position p of the request in + block-table row r goes to `pool[block_table[r, p // tpb], row, p % tpb]`. + + `latent`: bf16 `[T, 576]`, rows contiguous. `q`: bf16 `[T, H, >= 576]` (the absorbed + query; its `[..., 512:576]` is roped) or None. `table`: fp32 `[>= max pos + 1, 64]`. + `pool`: uint8 `[blocks, owners, tokens_per_block, 656]`, contiguous. Returns `pool`. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # spec: `tokens` rows of `phase` "decode" (one per request, random positions) or + # "context" (one request's consecutive positions from `cached`), `heads` query + # heads (0: no query), `rope`, a pool of `blocks` x `owners` pages of + # `tokens_per_block`, written at owner `row`; `captured` replays a CUDA graph on new + # data; `amax` "every_bf16" gives the rows' tiles every finite bf16 magnitude. + # The `_ML4_1M`, `_ML4_1M_DEP4` and `_ML4_1M_TEP8` cells (`max_pos` set) run on the device, + # over a table of `max_pos` rows and a `[16, table_width]` block table whose request i + # is row i: phase "context" (one request of `tokens` rows from `cached`, or + # `sweep_positions` [lo, hi) in calls of `tokens` rows), "packed" (`contexts`, (cached, + # new) per request) and "mixed" (`contexts`, then one generation row per + # `decode_positions` entry); a "decode" cell is captured at each of `graph_batches` + # (default `tokens`) and replayed on new data, with `positions` "tail" (rows drawn from + # the table's last 64 positions, one at the last) or "boot" (captured on the engine's + # warm-up batch, replayed with `live` rows ahead of attention DP's padding dummy, + # `padding` "one_slot": rows of one request at position 0 whose latents are one row + # repeated). `far` puts every written page at the top of the pool's `blocks`, the + # step's last position on its last block. + CELLS: tuple[Cell, ...] = ( + tuple( + Cell( + why=f"decode, {t} rows from a CUDA graph, 16 query heads (tep4/tp4)", + spec=dict( + phase="decode", + tokens=t, + heads=16, + rope=True, + blocks=256, + owners=63, + tokens_per_block=32, + row=1, + captured=True, + ), + ) + for t in (1, 16) + ) + + ( + Cell( + why="decode, 64 rows, 64 query heads (dep4), the last owner row", + spec=dict( + phase="decode", + tokens=64, + heads=64, + rope=True, + blocks=256, + owners=63, + tokens_per_block=32, + row=62, + captured=True, + ), + ), + Cell( + why="NoPE owner (layer 35's row): no rope, query untouched", + spec=dict( + phase="decode", + tokens=16, + heads=16, + rope=False, + blocks=256, + owners=63, + tokens_per_block=32, + row=31, + ), + ), + Cell( + why="context of 32768 rows from position 0 (the unchunked envelope), 16 heads", + spec=dict( + phase="context", + tokens=32768, + cached=0, + heads=16, + rope=True, + blocks=1100, + owners=63, + tokens_per_block=32, + row=0, + ), + ), + Cell( + why="context chunk of 2047 rows after a 5185-token cached prefix, mid-page", + spec=dict( + phase="context", + tokens=2047, + cached=5185, + heads=64, + rope=True, + blocks=400, + owners=63, + tokens_per_block=32, + row=7, + ), + ), + Cell( + why="every finite bf16 tile amax once: zero, subnormals, the 1e-4 floor, powers " + "of two, the largest", + spec=dict( + phase="context", + tokens=8160, + cached=0, + heads=0, + rope=True, + blocks=300, + owners=63, + tokens_per_block=32, + row=5, + amax="every_bf16", + ), + ), + Cell( + why="64-token pages", + spec=dict( + phase="context", + tokens=1000, + cached=37, + heads=16, + rope=True, + blocks=64, + owners=63, + tokens_per_block=64, + row=2, + ), + ), + # The 32-token cells above at 64-token pages: the same rows, heads, positions and owner + # rows, half the blocks for the same token capacity. + *( + Cell( + why=f"64-token pages: decode, {t} rows from a CUDA graph, 16 query heads " + "(tep4/tp4)", + spec=dict( + phase="decode", + tokens=t, + heads=16, + rope=True, + blocks=128, + owners=63, + tokens_per_block=64, + row=1, + captured=True, + ), + ) + for t in (1, 16) + ), + Cell( + why="64-token pages: decode, 64 rows, 64 query heads (dep4), the last owner row", + spec=dict( + phase="decode", + tokens=64, + heads=64, + rope=True, + blocks=128, + owners=63, + tokens_per_block=64, + row=62, + captured=True, + ), + ), + Cell( + why="64-token pages: NoPE owner (layer 35's row): no rope, query untouched", + spec=dict( + phase="decode", + tokens=16, + heads=16, + rope=False, + blocks=128, + owners=63, + tokens_per_block=64, + row=31, + ), + ), + Cell( + why="64-token pages: context of 32768 rows from position 0 (the unchunked envelope), " + "16 heads", + spec=dict( + phase="context", + tokens=32768, + cached=0, + heads=16, + rope=True, + blocks=550, + owners=63, + tokens_per_block=64, + row=0, + ), + ), + Cell( + why="64-token pages: context chunk of 2047 rows after a 5185-token cached prefix, " + "mid-page", + spec=dict( + phase="context", + tokens=2047, + cached=5185, + heads=64, + rope=True, + blocks=200, + owners=63, + tokens_per_block=64, + row=7, + ), + ), + Cell( + why="64-token pages: every finite bf16 tile amax once: zero, subnormals, the 1e-4 " + "floor, powers of two, the largest", + spec=dict( + phase="context", + tokens=8160, + cached=0, + heads=0, + rope=True, + blocks=150, + owners=63, + tokens_per_block=64, + row=5, + amax="every_bf16", + ), + ), + ) + + tuple( + Cell( + why=f"decode, {t} rows from a CUDA graph, 8 query heads (tep8)", + spec=dict( + phase="decode", + tokens=t, + heads=8, + rope=True, + blocks=256, + owners=63, + tokens_per_block=32, + row=1, + captured=True, + ), + ) + for t in (1, 16) + ) + + ( + Cell( + why="context of 32768 rows from position 0 (the unchunked envelope), 8 heads (tep8)", + spec=dict( + phase="context", + tokens=32768, + cached=0, + heads=8, + rope=True, + blocks=1100, + owners=63, + tokens_per_block=32, + row=0, + ), + ), + ) + + ( + Cell( + why="1M last chunk, RoPE owner row 62: 32768 rows at 1015808..1048575 behind a " + "1015808-token cached prefix, 16 of the 64-head scratch (PrefillTarget._attention)", + spec=dict( + _ML4_1M, + phase="context", + tokens=32768, + cached=1015808, + rope=True, + blocks=1100, + row=62, + ), + ), + Cell( + why="1M last chunk, NoPE owner (layer 35, row 35): no rope, the query untouched, " + "32768 rows at 1015808..1048575", + spec=dict( + _ML4_1M, + phase="context", + tokens=32768, + cached=1015808, + rope=False, + blocks=1100, + row=35, + ), + ), + Cell( + why="1M packed step, 4 chunks cached 1032192 / 524288 / 0 / 2048 (16384 + 8192 + " + "4096 + 4096 rows), req_idx 0..3 in one call, a 2.9 GB pool written past 2^31 bytes", + spec=dict( + _ML4_1M, + phase="packed", + contexts=((1032192, 16384), (524288, 8192), (0, 4096), (2048, 4096)), + rope=True, + blocks=2200, + row=1, + ), + ), + Cell( + why="1M mixed step, 3 chunks cached to 1015808 (one mid-page at 490000) and 13 " + "generation rows at 0..1048575: req_idx 0..15, 32768 rows in one call", + spec=dict( + _ML4_1M, + phase="mixed", + contexts=((1015808, 16000), (490000, 10000), (0, 6755)), + decode_positions=( + 1048575, + 0, + 2047, + 2048, + 4499, + 1048574, + 1, + 699, + 32768, + 999999, + 131071, + 128, + 1048544, + ), + rope=True, + blocks=2200, + row=0, + ), + ), + Cell( + why="1M decode graphs at every batch 1..16 (captured; the engine pads to 1/2/4/8/16): " + "positions in the table's last 64 rows, 1048575 in each, replays bitwise equal to " + "eager (DecodeTarget._attention)", + spec=dict( + _ML4_1M, + phase="decode", + tokens=16, + rope=True, + blocks=256, + row=1, + captured=True, + graph_batches=tuple(range(1, 17)), + positions="tail", + ), + ), + Cell( + why="1M NoPE decode graph 16 (captured, layer 35's row 35): positions in the " + "table's last 64 rows, replayed", + spec=dict( + _ML4_1M, + phase="decode", + tokens=16, + rope=False, + blocks=256, + row=35, + captured=True, + positions="tail", + ), + ), + Cell( + why="1M every table row: positions 0..1048575 in 32 chunk calls of 32768 rows, " + "owner row 2", + spec=dict( + _ML4_1M, + phase="context", + tokens=32768, + rope=True, + blocks=1100, + row=2, + sweep_positions=(0, 1048576), + ), + ), + Cell( + why="1M far pool, decode graph 16 (captured, large): 107917 blocks (142.7 GB), every " + "page at the pool's top, owner row 62, byte offsets to 142,719,800,176", + spec=dict( + _ML4_1M, + phase="decode", + tokens=16, + rope=True, + blocks=107917, + row=62, + captured=True, + positions="tail", + far=True, + large=True, + ), + ), + Cell( + why="1M boot pool, last chunk (large): 32768 rows behind 1015808 cached on 74183 " + "blocks (98.1 GB), every page at the pool's top, byte offsets to 98,106,720,112", + spec=dict( + _ML4_1M, + phase="context", + tokens=32768, + cached=1015808, + rope=True, + blocks=74183, + row=62, + far=True, + large=True, + ), + ), + Cell( + why="1M last chunk, RoPE owner (row 62): 32768 rows at 1015808..1048575 behind a " + "1015808-token cached prefix, the first 16 heads of the 64-head scratch " + "(PrefillTarget._attention)", + spec=dict( + _ML4_1M, + phase="context", + tokens=32768, + cached=1015808, + rope=True, + blocks=1100, + row=62, + ), + ), + Cell( + why="1M last chunk, NoPE owner (layer 35, row 35): rope False, 32768 rows at " + "1015808..1048575, the query untouched", + spec=dict( + _ML4_1M, + phase="context", + tokens=32768, + cached=1015808, + rope=False, + blocks=1100, + row=35, + ), + ), + Cell( + why="1M packed step of 4 chunks cached 1032192 / 524288 / 0 / 2048 (16384 + 8192 + " + "4096 + 4096 rows), req_idx 0..3 in one call, rows past 2^31 bytes of a 2200-block " + "pool", + spec=dict( + _ML4_1M, + phase="packed", + contexts=((1032192, 16384), (524288, 8192), (0, 4096), (2048, 4096)), + rope=True, + blocks=2200, + row=1, + ), + ), + Cell( + why="1M packed step of 16 chunk tails, 2048 rows each behind cached prefixes " + "1046528 - 65537 i (15 of them mid-page), req_idx 0..15 in one call", + spec=dict( + _ML4_1M, + phase="packed", + contexts=tuple((1046528 - 65537 * i, 2048) for i in range(16)), + rope=True, + blocks=1100, + row=40, + ), + ), + Cell( + why="1M mixed step: 3 chunks cached to 1015808 (one from mid-page 490000), then 13 " + "generation rows at 0..1048575, req_idx 0..15, 32768 rows in one call", + spec=dict( + _ML4_1M, + phase="mixed", + contexts=((1015808, 16000), (490000, 10000), (0, 6755)), + decode_positions=( + 1048575, + 0, + 2047, + 2048, + 4499, + 1048574, + 1, + 699, + 32768, + 999999, + 131071, + 128, + 1048544, + ), + rope=True, + blocks=2200, + row=0, + ), + ), + Cell( + why="1M decode graphs at every batch 1..16 (captured; the engine pads to " + "1/2/4/8/16): positions in the table's last 64 rows, 1048575 in each, every replay " + "bitwise equal to eager (DecodeTarget._attention)", + spec=dict( + _ML4_1M, + phase="decode", + tokens=16, + rope=True, + blocks=256, + row=1, + captured=True, + graph_batches=tuple(range(1, 17)), + positions="tail", + ), + ), + Cell( + why="1M NoPE decode graph 16 (captured, layer 35's row 35): positions in the " + "table's last 64 rows, the query untouched", + spec=dict( + _ML4_1M, + phase="decode", + tokens=16, + rope=False, + blocks=256, + row=35, + captured=True, + positions="tail", + ), + ), + Cell( + why="1M boot-scale pool, decode graph 16 (captured, large): 74183 blocks (98.1 GB), " + "every page at the pool's top, its last row written at byte 98,106,720,112", + spec=dict( + _ML4_1M, + phase="decode", + tokens=16, + rope=True, + blocks=74183, + row=62, + captured=True, + positions="tail", + far=True, + large=True, + ), + ), + Cell( + why="1M boot-scale pool, last chunk (large): 32768 rows at 1015808..1048575 on " + "74183 blocks (98.1 GB), every page at the pool's top, its last row written", + spec=dict( + _ML4_1M, + phase="context", + tokens=32768, + cached=1015808, + rope=True, + blocks=74183, + row=62, + far=True, + large=True, + ), + ), + Cell( + why="dep4 1M last chunk, RoPE owner (row 62): 32768 rows at 1015808..1048575 behind " + "a 1015808-token cached prefix, all 64 query heads roped (PrefillTarget._attention)", + spec=dict( + _ML4_1M_DEP4, + phase="context", + tokens=32768, + cached=1015808, + rope=True, + blocks=1100, + row=62, + ), + ), + Cell( + why="dep4 1M last chunk, NoPE owner (layer 35, row 35): rope False, 32768 rows at " + "1015808..1048575, the 64-head query untouched", + spec=dict( + _ML4_1M_DEP4, + phase="context", + tokens=32768, + cached=1015808, + rope=False, + blocks=1100, + row=35, + ), + ), + Cell( + why="dep4 1M packed step of 4 chunks cached 1032192 / 524288 / 0 / 2048 (16384 + " + "8192 + 4096 + 4096 rows), req_idx 0..3 in one call, rows past 2^31 bytes of a " + "2200-block pool", + spec=dict( + _ML4_1M_DEP4, + phase="packed", + contexts=((1032192, 16384), (524288, 8192), (0, 4096), (2048, 4096)), + rope=True, + blocks=2200, + row=1, + ), + ), + Cell( + why="dep4 1M packed step of 16 chunk tails, 2048 rows each behind cached prefixes " + "1046528 - 65537 i (15 of them mid-page), req_idx 0..15 in one call", + spec=dict( + _ML4_1M_DEP4, + phase="packed", + contexts=tuple((1046528 - 65537 * i, 2048) for i in range(16)), + rope=True, + blocks=1100, + row=40, + ), + ), + Cell( + why="dep4 1M mixed step: 3 chunks cached to 1015808 (one from mid-page 490000), then " + "13 generation rows at 0..1048575, req_idx 0..15, 32768 rows in one call", + spec=dict( + _ML4_1M_DEP4, + phase="mixed", + contexts=((1015808, 16000), (490000, 10000), (0, 6755)), + decode_positions=( + 1048575, + 0, + 2047, + 2048, + 4499, + 1048574, + 1, + 699, + 32768, + 999999, + 131071, + 128, + 1048544, + ), + rope=True, + blocks=2200, + row=0, + ), + ), + Cell( + why="dep4 1M decode graphs at every batch 1..16 (captured; the engine pads to " + "1/2/4/8/16, and runs any batch eagerly beside a prefilling peer): positions in the " + "table's last 64 rows, 1048575 in each, every replay bitwise equal to eager " + "(DecodeTarget._attention)", + spec=dict( + _ML4_1M_DEP4, + phase="decode", + tokens=16, + rope=True, + blocks=256, + row=1, + captured=True, + graph_batches=tuple(range(1, 17)), + positions="tail", + ), + ), + Cell( + why="dep4 1M boot captures 1/2/4/8/16 under attention DP's padding: captured on the " + "warm-up batch (a key at 1048574 ahead of one-key fillers at 0, the warm-up call " + "gated), replayed with one live row at 1048575, at 0 (an idle rank's dummy) and at a " + "drawn position ahead of the padding dummy repeated on one slot; every replay " + "bitwise the eager call and the call with the padding cut to one row", + spec=dict( + _ML4_1M_DEP4, + phase="decode", + tokens=16, + rope=True, + blocks=256, + row=1, + captured=True, + graph_batches=(1, 2, 4, 8, 16), + live=1, + padding="one_slot", + positions="boot", + ), + ), + Cell( + why="dep4 1M boot captures at owner row 0, Triton's multiple-of-16 class of `row` " + "(DecodeTarget's rows 0/16/32/48, layers 0/16/32/52): graphs 1/2/4/8/16 captured on " + "the warm-up batch and replayed as the row-1 boot cell replays them", + spec=dict( + _ML4_1M_DEP4, + phase="decode", + tokens=16, + rope=True, + blocks=256, + row=0, + captured=True, + graph_batches=(1, 2, 4, 8, 16), + live=1, + padding="one_slot", + positions="boot", + ), + ), + Cell( + why="dep4 1M NoPE decode graph 16 (captured, layer 35's row 35): positions in the " + "table's last 64 rows, the 64-head query untouched", + spec=dict( + _ML4_1M_DEP4, + phase="decode", + tokens=16, + rope=False, + blocks=256, + row=35, + captured=True, + positions="tail", + ), + ), + Cell( + why="dep4 1M every table row: positions 0..1048575 in 32 chunk calls of 32768 rows, " + "all 64 heads, owner row 2", + spec=dict( + _ML4_1M_DEP4, + phase="context", + tokens=32768, + rope=True, + blocks=1100, + row=2, + sweep_positions=(0, 1048576), + ), + ), + Cell( + why="dep4 1M far pool, decode graph 16 (captured, large): 107917 blocks (142.7 GB), " + "every page at the pool's top, its last row written at byte 142,719,800,176", + spec=dict( + _ML4_1M_DEP4, + phase="decode", + tokens=16, + rope=True, + blocks=107917, + row=62, + captured=True, + positions="tail", + far=True, + large=True, + ), + ), + Cell( + why="dep4 1M boot-scale pool, last chunk (large): 32768 rows at 1015808..1048575 on " + "74183 blocks (98.1 GB), every page at the pool's top, its last row written at byte " + "98,106,720,112", + spec=dict( + _ML4_1M_DEP4, + phase="context", + tokens=32768, + cached=1015808, + rope=True, + blocks=74183, + row=62, + far=True, + large=True, + ), + ), + ) + + ( + Cell( + why="tep8 1M last chunk, RoPE owner (row 62): 32768 rows at 1015808..1048575 " + "behind a 1015808-token cached prefix, the first 8 heads of the 64-head scratch " + "(PrefillTarget._attention)", + spec=dict( + _ML4_1M_TEP8, + phase="context", + tokens=32768, + cached=1015808, + rope=True, + blocks=1100, + row=62, + ), + ), + Cell( + why="tep8 1M last chunk, NoPE owner (layer 35, row 35): rope False, 32768 rows at " + "1015808..1048575, the query untouched", + spec=dict( + _ML4_1M_TEP8, + phase="context", + tokens=32768, + cached=1015808, + rope=False, + blocks=1100, + row=35, + ), + ), + Cell( + why="tep8 1M packed step of 4 chunks cached 1032192 / 524288 / 0 / 2048 (16384 + " + "8192 + 4096 + 4096 rows), req_idx 0..3 in one call, rows past 2^31 bytes of a " + "2200-block pool", + spec=dict( + _ML4_1M_TEP8, + phase="packed", + contexts=((1032192, 16384), (524288, 8192), (0, 4096), (2048, 4096)), + rope=True, + blocks=2200, + row=1, + ), + ), + Cell( + why="tep8 1M packed step of 16 chunk tails, 2048 rows each behind cached prefixes " + "1046528 - 65537 i (15 of them mid-page), req_idx 0..15 in one call", + spec=dict( + _ML4_1M_TEP8, + phase="packed", + contexts=tuple((1046528 - 65537 * i, 2048) for i in range(16)), + rope=True, + blocks=1100, + row=40, + ), + ), + Cell( + why="tep8 1M mixed step: 3 chunks cached to 1015808 (one from mid-page 490000), " + "then 13 generation rows at 0..1048575, req_idx 0..15, 32768 rows in one call", + spec=dict( + _ML4_1M_TEP8, + phase="mixed", + contexts=((1015808, 16000), (490000, 10000), (0, 6755)), + decode_positions=( + 1048575, + 0, + 2047, + 2048, + 4499, + 1048574, + 1, + 699, + 32768, + 999999, + 131071, + 128, + 1048544, + ), + rope=True, + blocks=2200, + row=0, + ), + ), + Cell( + why="tep8 1M decode graphs at every batch 1..16 (captured; the engine pads to " + "1/2/4/8/16): positions in the table's last 64 rows, 1048575 in each, every replay " + "bitwise equal to eager (DecodeTarget._attention)", + spec=dict( + _ML4_1M_TEP8, + phase="decode", + tokens=16, + rope=True, + blocks=256, + row=1, + captured=True, + graph_batches=tuple(range(1, 17)), + positions="tail", + ), + ), + Cell( + why="tep8 1M boot captures at owner row 0, Triton's multiple-of-16 class of `row` " + "(DecodeTarget._attention's rows 0/16/32/48, layers 0/16/32/52): decode graphs at " + "1/2/4/8/16, positions in the table's last 64 rows, every replay bitwise equal to " + "eager", + spec=dict( + _ML4_1M_TEP8, + phase="decode", + tokens=16, + rope=True, + blocks=256, + row=0, + captured=True, + graph_batches=(1, 2, 4, 8, 16), + positions="tail", + ), + ), + Cell( + why="tep8 1M NoPE decode graph 16 (captured, layer 35's row 35): positions in the " + "table's last 64 rows, the query untouched", + spec=dict( + _ML4_1M_TEP8, + phase="decode", + tokens=16, + rope=False, + blocks=256, + row=35, + captured=True, + positions="tail", + ), + ), + Cell( + why="tep8 1M every table row: positions 0..1048575 in 32 chunk calls of 32768 " + "rows, owner row 2", + spec=dict( + _ML4_1M_TEP8, + phase="context", + tokens=32768, + rope=True, + blocks=1100, + row=2, + sweep_positions=(0, 1048576), + ), + ), + Cell( + why="tep8 1M pool bound, decode graph 16 (captured, large): 147456 blocks (195.0 " + "GB), every page at the pool's top, its last row written at byte 195,009,969,520", + spec=dict( + _ML4_1M_TEP8, + phase="decode", + tokens=16, + rope=True, + blocks=147456, + row=62, + captured=True, + positions="tail", + far=True, + large=True, + ), + ), + Cell( + why="tep8 1M pool bound, last chunk (large): 32768 rows at 1015808..1048575 on " + "147456 blocks (195.0 GB), every page at the pool's top, its last row written", + spec=dict( + _ML4_1M_TEP8, + phase="context", + tokens=32768, + cached=1015808, + rope=True, + blocks=147456, + row=62, + far=True, + large=True, + ), + ), + ) + ) + + note = """ + Call sites: every mistral_large4 target's owner layers, context and generation + rows in one call, before the layer's attention reads the pool. A sharing layer + (NoPE) writes nothing and its query tail is zero. + + The Mistral Large 4 NVFP4 tp4 deployment at max_seq_len 1048576 (chunked prefill, + max_num_tokens 32768, max_batch_size 16, decode captured at 1/2/4/8/16 padded): + PrefillTarget._attention (eager) and DecodeTarget._attention (captured) on the 63 KV + owners, one call per owner layer over every row of the step. `q` is the first 16 + heads of the step's [T, 64, 576] fused-query scratch (heads 16..63 are never + touched), `table` the target's `_rope.indexer` (fp32 [1048576, 64], cos | sin rounded + through bf16), `pos` the step's position_ids and `req_idx` md.req_idx_per_token (both + int32; a generation row's request row is num_contexts + its index), `block_table` + md.block_table [16, 32772] int32, `pool` the manager's get_latent_pool() at `row` = + latent_pool_row(layer), and `rope` False on the NoPE owners. The `_ML4_1M` cells cover + that envelope: a 32768-row chunk at 1015808..1048575 on a RoPE and on the NoPE + owner, every position 0..1048575 in 32 chunk calls, a packed step of 4 chunks, a + mixed step of 3 chunks and 13 generation rows (16 requests: block-table rows 0..15, + column 32767 of row 15), decode graphs captured at every batch 1..16 with a row at + 1048575 (each replay bitwise equal to an eager call on the same inputs), and two + pools at the far end (large): 74183 blocks, the delivered 1M tp4 boot's (this pool's + own 1M tp4 boot measured 72757 blocks, 2328224 tokens), and 107917, each with the + pool's last slot written. Cost on sm_103: 0.02 to 6.7 s per pre-merge cell, most of + it Triton's first compile of a specialization (the 32-call sweep, compiled already, + 1.6 s); the large cells 1.2 s and 0.2 s. + + The Mistral Large 4 NVFP4 tep4 deployment at max_seq_len 1048576 (chunked prefill, + max_num_tokens 32768, max_batch_size 16, decode captured at 1/2/4/8/16 padded): + PrefillTarget._attention (eager) and DecodeTarget._attention (captured) on the 63 KV + owners, one call per owner layer over every row of the step, a step's context chunk + rows first and its generation rows after them. `latent` is `[RMSNorm(c_kv) | k_pe]` + (k_pe zero on a NoPE owner); `q` the first 16 heads of the step's `[T, 64, 576]` + fused-query scratch, whose heads 16..63 the call leaves as they are; `table` the + target's `_rope.indexer` (fp32 `[1048576, 64]`, cos | sin rounded through bf16); + `pos` the position_ids and `req_idx` md.req_idx_per_token (both int32; a generation + row's request is num_contexts + its index); `block_table` md.block_table (int32 + `[16, 32772]`, the V2 manager's max_blocks_per_seq at 1M); `pool` the manager's + get_latent_pool() at `row` = latent_pool_row(layer); `rope` False on the NoPE owners + (layer 35 at row 35 and the `i % 4 == 3` owners below it). The `_ML4_1M` cells cover + that envelope: a 32768-row chunk at 1015808..1048575 on a RoPE owner and on the NoPE + owner; every position 0..1048575 in 32 chunk calls; packed steps of 4 chunks and of + 16 chunk tails; a mixed step of 3 chunks and 13 generation rows (16 requests, block + table rows 0..15, column 32767 of rows 3, 8 and 15); decode graphs captured at every + batch 1..16 with a row at 1048575, each replay bitwise equal to an eager call on the + same inputs; and, large, a pool of 74183 blocks with its last row written, by a + decode graph and by a last chunk. 74183 is above the tep4 deployment's 1M boot pool (72757 + blocks per rank, 2328224 tokens, as the target's boot measured it on this + pool). Cost on sm_103: 0.03 to 1.7 s per pre-merge `_ML4_1M` cell (the first + one builds the 1048576-row table; the 32-call sweep 1.7 s), pools to 2.9 GB; the two + large cells 1.4 s together on a 98.1 GB pool. + + The Mistral Large 4 NVFP4 dep4 deployment at max_seq_len 1048576 (attention DP over 4 + ranks, each running all 64 heads on its own requests; chunked prefill, + max_num_tokens 32768 and max_batch_size 16 per rank, decode captured at + 1/2/4/8/16 padded): PrefillTarget._attention (eager) and DecodeTarget._attention + (captured; eager on a rank whose peer prefills) on the 63 KV owners, one call per + owner layer over every row of the rank's step, its context chunk rows first and + its generation rows after them. `latent` is `[RMSNorm(c_kv) | k_pe]` (k_pe zero on + a NoPE owner); `q` the whole `[T, 64, 576]` fused-query scratch, every head + roped; `table` the target's `_rope.indexer` (fp32 `[1048576, 64]`, cos | sin + rounded through bf16); `pos` the position_ids and `req_idx` + md.req_idx_per_token (both int32; a generation row's request is num_contexts + + its index); `block_table` md.block_table (int32 `[16, 32772]`, the V2 manager's + max_blocks_per_seq at 1M); `pool` the manager's get_latent_pool() at `row` = + latent_pool_row(layer); `rope` False on the NoPE owners (layer 35 at row 35 and + the `i % 4 == 3` owners below it). The `_ML4_1M_DEP4` cells cover that envelope: + a 32768-row chunk at 1015808..1048575 on a RoPE owner and on the NoPE owner; + every position 0..1048575 in 32 chunk calls; packed steps of 4 chunks and of 16 + chunk tails; a mixed step of 3 chunks and 13 generation rows (16 requests, + block-table column 32767 of rows 3, 8 and 15); decode graphs captured at every + batch 1..16 with a row at 1048575, each replay bitwise equal to an eager call on + the same inputs; the NoPE decode graph at 16; the boot captures under attention + DP's padding (below), at owner row 1 and again at row 0; and, large, a + 107917-block pool written by a decode graph + and a 74183-block pool written by a last chunk, each up to its last row. Both + are above this target's 1M boot pool (64292 blocks per rank, measured by its + fp8_ds_mla boot at fraction 0.9: 2057344 tokens). Cost + on sm_103: 0.02 to 1.44 s per pre-merge `_ML4_1M_DEP4` cell (the first one builds + the 1048576-row table; the 32-call sweep the longest), pools to 2.9 GB; the two + large cells 1.4 s together on pools of 142.7 and 98.1 GB. + + Attention DP's CUDA-graph padding (dep4): `CUDAGraphRunner._get_padded_batch` + pads every rank's decode batch to the largest rank's with one padding dummy + repeated, so a graph replays a rank's live rows ahead of rows of one request at + position 0, each with its own req_idx and its own block-table row holding the + dummy's page: they all write one slot. The engine zero-fills a CUDA-graph + dummy's input id (`_zero_fill_row_spans` in + `pyexecutor/engine/runners/decoder/inputs.py`), so the target hands this call + one latent row repeated for them (inferred: the layers before the write compute + each row on its own). Rows with bitwise-identical latents on one slot leave it + as one of them alone does: certified at batches 1/2/4/8/16, captured on the + warm-up batch (the longest dummy's new key at 1048574 ahead of one-key fillers + at 0, each on a page of its own; the warm-up call gated) and replayed with the + live row at 1048575, at 0 (an idle rank's dummy) and at a drawn position, each + replay bitwise the eager call and the eager call with the padding cut to one + row; probed once more over 500 eager calls and 500 replays at batch 16, every + one bitwise one row's write. Rows with different latents on one slot race: + probed over 1000 calls with 15 such rows, the slot held no one row's whole + write in any of them, its code tiles, scales and rope bytes coming from several + rows, and its rope bytes were themselves a byte mix of rows in 32. + + The Mistral Large 4 NVFP4 tep8 deployment at max_seq_len 1048576 (chunked prefill, + max_num_tokens 32768, max_batch_size 16, decode captured at 1/2/4/8/16 padded): + PrefillTarget._attention (eager) and DecodeTarget._attention (captured) on the 63 KV + owners, one call per owner layer over every row of the step, a step's context chunk + rows first and its generation rows after them. `latent` is `[RMSNorm(c_kv) | k_pe]` + (k_pe zero on a NoPE owner); `q` the first 8 heads of the step's `[T, 64, 576]` + fused-query scratch, whose heads 8..63 the call leaves as they are (the kernel's + 16-head block masks heads 8..15); `table` the target's `_rope.indexer` (fp32 + `[1048576, 64]`, cos | sin rounded through bf16); `pos` the position_ids and `req_idx` + md.req_idx_per_token (both int32; a generation row's request is num_contexts + its + index); `block_table` md.block_table (int32 `[16, 32772]`, the V2 manager's + max_blocks_per_seq at 1M); `pool` the manager's get_latent_pool() at `row` = + latent_pool_row(layer); `rope` False on the NoPE owners (layer 35 at row 35 and the + `i % 4 == 3` owners below it). The `_ML4_1M_TEP8` cells cover that envelope: a + 32768-row chunk at 1015808..1048575 on a RoPE owner and on the NoPE owner; every + position 0..1048575 in 32 chunk calls; packed steps of 4 chunks and of 16 chunk tails; + a mixed step of 3 chunks and 13 generation rows (16 requests, block table rows 0..15, + column 32767 of rows 3, 8 and 15); decode graphs captured at every batch 1..16 with a + row at 1048575, each replay bitwise equal to an eager call on the same inputs, at owner + row 1 and again at row 0 for the captured batches 1/2/4/8/16; and, large, a pool of + 147456 blocks with its last row written, by a decode graph and by a last chunk. + 147456 is the tep8 pool bound, above this target's 1M boot pool (114021 blocks per + rank, 3,648,672 tokens, read back from its fp8_ds_mla boot at fraction 0.9). Cost on + sm_103: 0.02 to 1.5 s per pre-merge `_ML4_1M_TEP8` cell (the first one builds the + 1048576-row table; the 32-call sweep 1.5 s), at most 14.6 GiB of device memory, + pools to 2.9 GB; the two large cells 0.3 s each, peaks of 184.2 and 193.4 GiB on a + 195.0 GB pool. + + Bytes: the cells gate the codes and the scales bitwise against `reference`, the + rope halves (key and query) within one bf16 ulp (an FMA in the rotation may + round once less; none did on sm_103 so far), and every other pool byte bitwise. + With `rope=False` the row is bitwise the reference implementation's writer + (`concat_and_cache_mla`, fp8_ds_mla) on the same input: 20,480 rows, tile amaxes + 2^-30..2^15 and zero tiles, measured once on sm_103. + Launch: Triton, grid (T, 6) -- four quantized tiles, the key rope, the query + rope; CUDA-graph capturable. Triton specializes the int `row` three ways (equal + to 1, a multiple of 16, any other), so the dep4 and the tep8 DecodeTarget's captured + calls run three compiled kernels with rope (owner row 1; rows 0, 16, 32 and 48; the + rest) and one without (every NoPE owner row, 3, 7, ..., 35, is in the third class). + A captured 64-head cell holds each: the boot captures and decode graphs at row 1, + the boot captures at row 0, the far decode graph at 62, the NoPE graph at 35. So + does a captured 8-head cell: the decode graphs at row 1 and at row 0, the far decode + graph at 62, the NoPE graph at 35. + + Preconditions the op does not check: positions below the table's rows, every + token's block-table entry allocated, `row < owners`, no two tokens on one slot + unless their latent rows are bitwise identical (attention DP's padding, above). + + Hard limits (sm_103). The kernel widens pos, req and block to int64 before any + address arithmetic, so every offset below is int64. + pool byte offset ((block * owners + row) * tpb + pos % tpb) * 656: at 63 owners + x 32 a row passes 2^31 bytes from slot 3,273,604 (block 1623) + and 2^37 from block 103,923; probed at 2200 blocks (rows to + byte 2,906,887,536), at 74183 (the last row at byte + 98,106,720,112), at 107917 (the last row at byte + 142,719,800,176) and, at tep8's 8 heads, at 147456 (the last + row at byte 195,009,969,520) passed + table offset pos * 64 + 63 <= 67,108,863 at position 1048575, 2^31 not + reached; probed at every position 0..1048575 passed + block table req * 32772 + pos // 32 <= 524,347 (row 15, column 32767), + 2^31 not reached; probed at 524,347 passed + q offset t * 36864 + h * 576 + 575 elements in the 64-head scratch + passes 2^31 at 58,255 rows (row 58,254 at 64 heads; at h <= 7, + row 58,255), past max_num_tokens 32768 (1,207,959,552 elements + there, 56% of 2^31); not reached (rows to 32767 driven at 8, at + 16 and at all 64 heads) + latent offset t * 576 <= 18,874,368 elements at 32768 rows; not reached + fp32 index none: positions and slots never pass through fp32 + Not certified: positions at or past the table's rows (the kernel reads past its + end unchecked), pools past 147456 blocks (past 107917 at 16 and 64 heads), more + than 32768 rows in one call, query widths other than 0, 8, 16 and 64 heads, pages + other than 32 at 1M positions, owner counts other than 63 (and 1, in the scale + test), rows with different latents on one slot (they race, above). + """ + + def __call__( + self, + latent: torch.Tensor, + q: Optional[torch.Tensor], + pos: torch.Tensor, + table: torch.Tensor, + req_idx: torch.Tensor, + block_table: torch.Tensor, + pool: torch.Tensor, + row: int, + rope: bool, + ) -> torch.Tensor: + tokens = latent.shape[0] + if tokens == 0: + return pool + heads = 0 if q is None else q.shape[1] + _rope_append_kernel[(tokens, 6)]( + latent, latent.stride(0), + latent if q is None else q, 0 if q is None else q.stride(0), + 0 if q is None else q.stride(1), heads, + pos, table, req_idx, block_table, block_table.stride(0), + pool, pool.shape[1], row, + TPB=pool.shape[2], ROPE=bool(rope), HAS_Q=q is not None, + H_BLOCK=max(16, triton.next_power_of_2(max(heads, 1))), + ) # fmt: skip + return pool + + def reference( + self, + latent: torch.Tensor, + q: Optional[torch.Tensor], + pos: torch.Tensor, + table: torch.Tensor, + req_idx: torch.Tensor, + block_table: torch.Tensor, + pool: torch.Tensor, + row: int, + rope: bool, + ) -> tuple[torch.Tensor, Optional[torch.Tensor]]: + """`(pool, q)` after the call, as new tensors; the inputs are not written.""" + p = pos.long() + tpb = pool.shape[2] + x = latent[:, :DIM_NOPE].float().view(-1, NUM_TILES, TILE) + ratio = (x.abs().amax(-1) / FP8_MAX).clamp_min(SCALE_FLOOR) + scale = _pow2_ceil_torch(ratio) + codes = (x / scale.unsqueeze(-1)).to(torch.float8_e4m3fn) + + def rotate(tail): + pair = tail.float().reshape(*tail.shape[:-1], DIM_ROPE // 2, 2) + c = table[p, : DIM_ROPE // 2].view(-1, *([1] * (pair.dim() - 3)), DIM_ROPE // 2) + s = table[p, DIM_ROPE // 2 :].view_as(c) + out = torch.stack( + [pair[..., 0] * c - pair[..., 1] * s, pair[..., 0] * s + pair[..., 1] * c], -1 + ) + return out.reshape(tail.shape).to(torch.bfloat16) + + k_pe = latent[:, DIM_NOPE:] + k_pe = rotate(k_pe) if rope else k_pe.to(torch.bfloat16) + packed = torch.cat( + [ + codes.reshape(-1, DIM_NOPE).view(torch.uint8), + scale.contiguous().view(torch.uint8), + k_pe.contiguous().view(torch.uint8), + ], + dim=1, + ) + out = pool.clone() + block = block_table[req_idx.long(), p // tpb].long() + out[block, row, p % tpb] = packed + q_out = None + if q is not None: + q_out = q.clone() + if rope: + q_out[..., DIM_NOPE : DIM_NOPE + DIM_ROPE] = rotate( + q[..., DIM_NOPE : DIM_NOPE + DIM_ROPE] + ) + return out, q_out + + def is_valid( + self, + latent: torch.Tensor, + q: Optional[torch.Tensor], + pos: torch.Tensor, + table: torch.Tensor, + req_idx: torch.Tensor, + block_table: torch.Tensor, + pool: torch.Tensor, + row: int, + rope: bool, + ) -> None: + # Each would address the wrong bytes without an error. + if latent.dtype != torch.bfloat16 or latent.shape[-1] != DIM_NOPE + DIM_ROPE: + raise ValueError( + f"latent must be bf16 [T, 576], got {latent.dtype} {tuple(latent.shape)}" + ) + if latent.stride(1) != 1: + raise ValueError("latent rows must be contiguous") + if pool.dtype != torch.uint8 or pool.dim() != 4 or pool.shape[-1] != TOKEN_BYTES: + raise ValueError( + f"pool must be uint8 [blocks, owners, tpb, 656], got {tuple(pool.shape)}" + ) + if not pool.is_contiguous(): + raise ValueError("pool must be contiguous") + if not 0 <= row < pool.shape[1]: + raise ValueError(f"row {row} outside the pool's {pool.shape[1]} owner rows") + if table.dtype != torch.float32 or table.shape[-1] != DIM_ROPE: + raise ValueError("table must be fp32 [positions, 64]: 32 cos then 32 sin") + if q is not None and ( + q.dtype != torch.bfloat16 or q.shape[-1] < DIM_NOPE + DIM_ROPE or q.stride(2) != 1 + ): + raise ValueError("q must be bf16 [T, H, >= 576] with contiguous heads") + + +ds_mla_rope_append = DsMlaRopeAppend() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/flash_mla_sparse_decode.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/flash_mla_sparse_decode.py new file mode 100644 index 000000000000..3c765c9a7733 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/flash_mla_sparse_decode.py @@ -0,0 +1,1641 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Sparse MQA of one query row per sequence over an fp8_ds_mla pool: FlashMLA's sm100f +sparse fp8 decode kernel.""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch.modeling_v2.catalog._op import Arch, Cell, OpWrapper, true_fp32_matmul +from tensorrt_llm.flash_mla import flash_mla_with_kvcache as _flash_mla_with_kvcache + +ROW_BYTES = 656 +D_NOPE = 512 +D_ROPE = 64 +TILE = 128 +#: The query head counts FlashMLA's sm100 sparse decode runs (64, or 64 twice). +KERNEL_HEADS = (64, 128) +#: The most rows (one-row sequences) one call takes: the largest the cells drive. FlashMLA's +#: scheduler exits the process past 11622 on sm_103 (the note's Hard limits). +MAX_ROWS = 8192 + +#: The tp4 deployment at the 1M envelope: 16 live heads of the 64-head scratch, sequences to +#: 1048576 keys, the fp8_ds_mla pool's 32-token pages of 63 KV-owner rows per block. +_ML4_1M = dict(h_q=64, h_out=16, topk=2048, kv_len=1048576, page_size=32, owners=63) +#: The tep4 deployment at the 1M envelope: 16 live heads of its 64-head scratch, top-2048 +#: selections, sequences to 1048576 keys, 32-token pages of 63 KV-owner rows per block. +_TEP4_1M = dict(h_q=64, h_out=16, topk=2048, kv_len=1048576, page_size=32, owners=63) +#: The dep4 deployment at the 1M envelope: all 64 heads of a rank (its scratch is the kernel's +#: 64 heads, no padding), top-2048 selections, sequences to 1048576 keys, 32-token pages of +#: 63 KV-owner rows per block. +_DEP4_1M = dict(h_q=64, h_out=64, topk=2048, kv_len=1048576, page_size=32, owners=63) +#: `pages` (blocks x 63) of a 74183-block pool (the delivered 1M tp4 boot's, above the +#: 72757 blocks a 1M tep4 rank's boot measured and the 64292 the 1M dep4 boot measured per +#: rank at fraction 0.9, on fp8_ds_mla), of the 107917-block far end (the far cells of the +#: 1M attention entries), of the 1100-block pool the dep4 pre-merge eager and context cells +#: spread 1M sequences over, and of the 2048-block pool the tep4, dep4 and tep8 pre-merge +#: cells read past 2**31 bytes. +_BOOT_PAGES = 74183 * 63 +_FAR_PAGES = 107917 * 63 +_SMALL_PAGES = 1100 * 63 +_PRE_PAGES = 2048 * 63 +#: The batches the engine captures (padded): an attention-DP rank replays the largest rank's. +_CAPTURED = (1, 2, 4, 8, 16) +#: The pre-merge share of the ragged-tail sweep's row counts: 1..65, either side of every +#: power of two up to 4096, and the top 33. +_TAIL_SHARE = ( + tuple(range(1, 66)) + + tuple(r for k in range(7, 13) for r in (2**k - 1, 2**k, 2**k + 1)) + + tuple(range(8160, 8193)) +) +#: The ragged-tail sweep's pre-merge share of the last sub-batch's row counts r: 1..65, 96, +#: 2**k - 1 / 2**k / 2**k + 1 up to 4097, 8160 and 8175..8192. +_RAGGED_SHARE = ( + tuple(range(1, 66)) + + (96,) + + tuple(r for k in range(7, 14) for r in (2**k - 1, 2**k, 2**k + 1) if r < 8175) + + (8160,) + + tuple(range(8175, 8193)) +) +#: The dep4 ragged-tail sweep's pre-merge share of the last sub-batch's row counts r: 1..65, +#: 2**k - 1 / 2**k / 2**k + 1 to 4097, r017-land2's 6445, the witness's 8096 and 8175..8192. +_DEP4_TAIL_SHARE = ( + tuple(range(1, 66)) + + tuple(r for k in range(7, 13) for r in (2**k - 1, 2**k, 2**k + 1)) + + (6445, 8096) + + tuple(range(8175, 8193)) +) +#: The tep8 deployment at the 1M envelope: 8 live heads of its 64-head scratch, top-2048 +#: selections, sequences to 1048576 keys, 32-token pages of 63 KV-owner rows per block. +_TEP8_1M = dict(h_q=64, h_out=8, topk=2048, kv_len=1048576, page_size=32, owners=63) +#: `pages` (blocks x 63) of the 147456-block pool bound, the "1M tep8" cells' far pool (a 1M +#: tep8 rank holds 114326 blocks on fp8_ds_mla at fraction 0.9); their pre-merge cells read +#: the 2048-block `_PRE_PAGES` past 2**31 bytes and run the `_RAGGED_SHARE` row counts. +_BOUND_PAGES = 147456 * 63 + + +class FlashMlaSparseDecode(OpWrapper): + """For every query row r and head h < `out.shape[1]`:: + + K[j] = [e4m3(row_j[:512]) * tile_scale_j (per 128) | bf16(row_j[528:656])] + p = softmax_j(softmax_scale * q[r, h] . K[j]), j in indices[r] (-1 skipped) + out[r, h] = sum_j p[j] * K[j][:512] + + `kv_cache`: uint8 `[pages, page_size, 1, 656]`, a page's rows contiguous; a slot is + `page * page_size + offset`. `indices`: int32 `[rows, topk]` slots, -1 padded. + `q`: bf16 `[rows, h_q, 576]` with h_q 64 or 128 -- the kernel's head counts; a rank + with fewer heads passes them first and zeros after (the caller's scratch), and + names how many it keeps through `out`. `sched_meta`: a FlashMLA scheduler object + (`tensorrt_llm.flash_mla.get_mla_metadata()[0]`), filled on its first call and + shared by every call of one step with the same rows, h_q and topk. Writes `out` + (bf16 `[rows, h_out, 512]`, any strides) and returns it; with `out` None, returns the + kernel's own `[rows, h_q, 512]` output (a view, no copy), whose first heads a caller + reads in place. At most `MAX_ROWS` rows. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # spec: `rows` sequences of one query row, `h_q` kernel heads of which `h_out` are + # real, `topk` slots over `kv_len`-token sequences in a pool of `pages` pages of + # `page_size` x `owners` rows; `layers` calls share one sched meta; `captured` + # replays from a CUDA graph; `holes` scatters -1 through short rows; `view` passes + # `out` None and gates the returned kernel output. + # The "1M tp4" cells: `kv_len` is the longest sequence, the others spread below it; + # `graph_batches` the captured batches (each replay on new slots bitwise equal to an + # eager call); `far` builds the pool at `pages` / `owners` blocks exactly with the + # requests' pages its top ones and the first request's last page its top block; + # `q_offset` makes q and out tail views from that row of a 32768-row step scratch; + # `context` makes the rows a step's context rows (`chunk_rows` behind `cached`, or the + # packed `contexts` of (cached, new)), each a one-row sequence, driven in sub-batches of + # `sub_batch` rows with one sched meta per sub-batch size (`rows`: the largest). + # `ragged_tail` drives a chunk's last sub-batch at every row count r in 1..`sub_batch` + # (`tail_rows`: those r only): rows `lead_rows` .. `lead_rows` + r - 1 of the chunk behind + # `cached`, q and out views from that row of the step's scratch, a sched meta per r. + # The "1M tep4", "1M dep4" and "1M tep8" cells build the pool on the device, every byte + # 0xFF (e4m3, fp32 and bf16 NaN) but one owner row of the pages the requests hold. + # `kv_len` is the longest decode sequence, the others spread below it; `graph_batches` + # the batches captured (each replay on new queries and new slots in the same pages, + # bitwise an eager call); `live: 1` with `boot_capture`: each batch captured on the boot + # warm-up's rows (its longest dummy at kv_len - 1 keys beside one-key fillers), then + # replayed as attention DP replays it, one live row at kv_len keys behind one-key padding + # dummies that all read one slot, and an idle rank's one-key dummy behind the same + # padding; + # `row_counts`: eager decode steps at each count, a fresh sched meta per step; `far`: the + # pool is `pages` / `owners` blocks, the requests' pages distinct and its top ones, the + # first request's last page on the top block; `top_blocks`: the requests' pages drawn + # from the pool's top that many blocks; `q_offset`: q and out tail views from that row + # of a 32768-row step scratch; `context`: the rows are a step's context rows + # (`chunk_rows` behind `cached`, or the packed `contexts` of (cached, new)), each a + # one-row sequence, in sub-batches of `sub_batch` rows with one sched meta per sub-batch + # size shared by every layer (`rows`: the largest sub-batch); `ragged_tail`: only a + # chunk's last sub-batch, each r one call on a sched meta of its own -- a "1M dep4" + # cell's rows `lead_rows` + [0, r) of the chunk behind `cached`, at every r in + # 1..`sub_batch` (`tail_rows`: at those r only), a "1M tep4" or "1M tep8" cell's the + # last sub-batch of a 32768-row chunk behind `cached`, its first r rows at each r in + # `tails` (every r in 1..`sub_batch` without it; a "1M tep8" cell's the largest first). + # The test drives the "1M tp4", the "1M tep4", the "1M dep4" and the "1M tep8" cells each + # with the harness that certified them. + CELLS: tuple[Cell, ...] = ( + tuple( + Cell( + why=f"tep4/tp4 decode batch {b}: 16 real heads padded to 64, graph replay, " + "3 layers share a sched meta", + spec=dict( + rows=b, + h_q=64, + h_out=16, + topk=2048, + kv_len=32768, + page_size=32, + owners=63, + pages=1100 * b // 16 + 64, + layers=3, + captured=True, + ), + ) + for b in (1, 2, 4, 8, 16) + ) + + tuple( + Cell( + why=f"dep4 decode batch {b}: 64 heads, graph replay", + spec=dict( + rows=b, + h_q=64, + h_out=64, + topk=2048, + kv_len=32768, + page_size=32, + owners=63, + pages=1100 * max(1, b // 16) + 64, + layers=2, + captured=True, + ), + ) + for b in (1, 16, 64) + ) + + ( + Cell( + why="short sequences: -1 tails and scattered holes", + spec=dict( + rows=16, + h_q=64, + h_out=16, + topk=2048, + kv_len=700, + page_size=32, + owners=63, + pages=512, + layers=1, + holes=True, + ), + ), + Cell( + why="NoPE layer: zero query and key rope halves", + spec=dict( + rows=16, + h_q=64, + h_out=16, + topk=2048, + kv_len=9000, + page_size=32, + owners=63, + pages=512, + layers=1, + nope=True, + ), + ), + Cell( + why="tp8-style 8 real heads padded to 64", + spec=dict( + rows=8, + h_q=64, + h_out=8, + topk=2048, + kv_len=4096, + page_size=32, + owners=63, + pages=512, + layers=1, + ), + ), + ) + # The cells above at 64-token pages: the same rows, heads, top-k, lengths and graph + # replays, half the pages for the same token capacity. + + tuple( + Cell( + why=f"64-token pages: tep4/tp4 decode batch {b}: 16 real heads padded to 64, " + "graph replay, 3 layers share a sched meta", + spec=dict( + rows=b, + h_q=64, + h_out=16, + topk=2048, + kv_len=32768, + page_size=64, + owners=63, + pages=550 * b // 16 + 32, + layers=3, + captured=True, + ), + ) + for b in (1, 2, 4, 8, 16) + ) + + tuple( + Cell( + why=f"64-token pages: dep4 decode batch {b}: 64 heads, graph replay", + spec=dict( + rows=b, + h_q=64, + h_out=64, + topk=2048, + kv_len=32768, + page_size=64, + owners=63, + pages=550 * max(1, b // 16) + 32, + layers=2, + captured=True, + ), + ) + for b in (1, 16, 64) + ) + + ( + Cell( + why="64-token pages: short sequences: -1 tails and scattered holes", + spec=dict( + rows=16, + h_q=64, + h_out=16, + topk=2048, + kv_len=700, + page_size=64, + owners=63, + pages=256, + layers=1, + holes=True, + ), + ), + Cell( + why="64-token pages: NoPE layer: zero query and key rope halves", + spec=dict( + rows=16, + h_q=64, + h_out=16, + topk=2048, + kv_len=9000, + page_size=64, + owners=63, + pages=256, + layers=1, + nope=True, + ), + ), + Cell( + why="64-token pages: tp8-style 8 real heads padded to 64", + spec=dict( + rows=8, + h_q=64, + h_out=8, + topk=2048, + kv_len=4096, + page_size=64, + owners=63, + pages=256, + layers=1, + ), + ), + ) + # `out` None, the NVFP4 decode targets' call: the V up-projection reads the live heads + # of the kernel's 64-head output in place. The decode batches the targets capture, + # up to a max_batch_size-256 engine's. + + tuple( + Cell( + why=f"out None, tp4/tep4 decode batch {b}: the kernel's 64-head output as " + "returned, graph replay, 3 layers share a sched meta", + spec=dict( + rows=b, + h_q=64, + h_out=16, + topk=2048, + kv_len=32768, + page_size=32, + owners=63, + pages=1100 * min(4, max(1, b // 16)) + 64, + layers=3, + captured=True, + view=True, + ), + ) + for b in (1, 16, 64, 256) + ) + + ( + Cell( + why="out None, dep4 decode batch 64: all 64 heads as returned, graph replay", + spec=dict( + rows=64, + h_q=64, + h_out=64, + topk=2048, + kv_len=32768, + page_size=32, + owners=63, + pages=1100 * 4 + 64, + layers=2, + captured=True, + view=True, + ), + ), + Cell( + why="out None, NoPE layer at 64-token pages: zero query and key rope halves", + spec=dict( + rows=16, + h_q=64, + h_out=16, + topk=2048, + kv_len=9000, + page_size=64, + owners=63, + pages=256, + layers=1, + nope=True, + view=True, + ), + ), + ) + + tuple( + Cell( + why=f"tep8 decode batch {b}: 8 real heads padded to 64, graph replay, " + "3 layers share a sched meta", + spec=dict( + rows=b, + h_q=64, + h_out=8, + topk=2048, + kv_len=32768, + page_size=32, + owners=63, + pages=1100 * b // 16 + 64, + layers=3, + captured=True, + ), + ) + for b in (1, 2, 4, 8, 16) + ) + # The tp4 deployment at the 1M envelope. Decode rows (DecodeTarget._attention, captured at + # 1/2/4/8/16 padded; the LLM API's default list reaches every batch 1..16) and a mixed + # step's generation rows (PrefillTarget._attn_gen) are one query each at kv_len - 1; + # context rows on the decode path (PrefillTarget._attn_ctx on a step whose context KV + # exceeds 2**19 rows) are a chunk's rows, each one sequence of one row. + + ( + Cell( + why="1M tp4 decode graphs at every batch 1..16, boot pool (large): 72 layers share " + "one sched meta, rows of 1048576 keys down to 1 on the pool's top 73398 blocks " + "(98.1 GB), the pool's last slot read", + spec=dict( + _ML4_1M, + rows=16, + pages=_BOOT_PAGES, + layers=72, + captured=True, + graph_batches=tuple(range(1, 17)), + far=True, + large=True, + ), + ), + Cell( + why="1M tp4 decode graph 16, far pool (large): 107917 blocks (142.7 GB), the top " + "slot 217560671 read at byte 142.7e9, 72 layers", + spec=dict( + _ML4_1M, + rows=16, + pages=_FAR_PAGES, + layers=72, + captured=True, + far=True, + large=True, + ), + ), + Cell( + why="1M tp4 mixed step's 13 generation rows (large): q and out tail views from row " + "32755 of the step's 32768-row scratch, eager, a fresh sched meta, boot pool", + spec=dict( + _ML4_1M, + rows=13, + pages=_BOOT_PAGES, + layers=3, + q_offset=32755, + far=True, + large=True, + ), + ), + Cell( + why="1M tp4 NoPE decode graph 16 (large): layer 35's row 35 as its nine sharers " + "read it, zero rope halves, boot pool", + spec=dict( + _ML4_1M, + rows=16, + pages=_BOOT_PAGES, + layers=9, + captured=True, + nope=True, + far=True, + large=True, + ), + ), + Cell( + why="1M tp4 graphs at every batch 1..16 to 2049 keys (2048 of 2049 valid), 72 " + "layers on one sched meta: the boot-pool cell's captures in pre-merge", + spec=dict( + _ML4_1M, + rows=16, + kv_len=2049, + pages=512, + layers=72, + captured=True, + graph_batches=tuple(range(1, 17)), + ), + ), + Cell( + why="1M tp4 context rows on the decode path (large): the last 32768-row chunk " + "behind 1015808 cached keys, 4 sub-batches of 8192 one-row sequences, 72 layers", + spec=dict( + _ML4_1M, + rows=8192, + pages=_BOOT_PAGES, + layers=72, + context=True, + chunk_rows=32768, + cached=1015808, + sub_batch=8192, + far=True, + large=True, + ), + ), + Cell( + why="1M tp4 context rows, ragged chunk (large): 32751 rows behind 1015808, " + "sub-batches 8192 x 3 and 8175 on two sched metas", + spec=dict( + _ML4_1M, + rows=8192, + pages=_BOOT_PAGES, + layers=3, + context=True, + chunk_rows=32751, + cached=1015808, + sub_batch=8192, + far=True, + large=True, + ), + ), + Cell( + why="1M tp4 context rows, packed step (large): chunks of 16384/8192/4096/4096 " + "behind 1032192/524288/0/2048 on the pool's top 49728 blocks, sub-batches of 8192", + spec=dict( + _ML4_1M, + rows=8192, + pages=_BOOT_PAGES, + layers=3, + context=True, + contexts=((1032192, 16384), (524288, 8192), (0, 4096), (2048, 4096)), + sub_batch=8192, + far=True, + large=True, + ), + ), + Cell( + why="1M tp4 context rows, 16-row chunk tail (large): positions 1048560..1048575 " + "in one sub-batch, 72 layers", + spec=dict( + _ML4_1M, + rows=16, + pages=_BOOT_PAGES, + layers=72, + context=True, + chunk_rows=16, + cached=1048560, + sub_batch=8192, + far=True, + large=True, + ), + ), + Cell( + why="1M tp4 context sub-batch at the 8192-row limit: a chunk's last 8192 rows " + "behind 1040384 cached keys over a 1100-block pool, 2 layers", + spec=dict( + _ML4_1M, + rows=8192, + pages=1100 * 63, + layers=2, + context=True, + chunk_rows=8192, + cached=1040384, + sub_batch=8192, + ), + ), + Cell( + why="1M tp4 mixed step's tail view: 13 generation rows from row 32755 of a " + "32768-row scratch, keys to 1048576 over a 1100-block pool", + spec=dict(_ML4_1M, rows=13, pages=1100 * 63, layers=3, q_offset=32755), + ), + ) + # The decode-path steps of the envelope prompt's landing (r019-land2: a 1040000-token + # prompt in 32768-row chunks, the 7-token request packed into its last step) as the + # target's switch ran them, and the last sub-batch of a chunk at every row count it + # can take. The kernel reads keys by slot alone, so these run over a 1100-block pool + # whose pages repeat; the cells above take the slots to the pool's far end. + + ( + Cell( + why="1M tp4 r019-land2 last step: 24192 rows behind 1015808 packed with a 7-row " + "fresh request, sub-batches 8192, 8192, 7815 on two sched metas, 72 layers", + spec=dict( + _ML4_1M, + rows=8192, + pages=1100 * 63, + layers=72, + context=True, + contexts=((1015808, 24192), (0, 7)), + sub_batch=8192, + ), + ), + *( + Cell( + why=f"1M tp4 r019-land2 chunk behind {cached}: 32768 rows in 4 sub-batches " + "of 8192 on one sched meta, 2 layers", + spec=dict( + _ML4_1M, + rows=8192, + pages=1100 * 63, + layers=2, + context=True, + chunk_rows=32768, + cached=cached, + sub_batch=8192, + ), + ) + for cached in range(524288, 1015808, 32768) + ), + Cell( + why="1M tp4 r019-land2 ragged tail (large): a chunk's last sub-batch at every r " + "in 1..8192, rows 24576..24576 + r - 1 behind 1015808, a sched meta each (22 s)", + spec=dict( + _ML4_1M, + rows=8192, + pages=1100 * 63, + layers=1, + context=True, + cached=1015808, + sub_batch=8192, + lead_rows=24576, + ragged_tail=True, + large=True, + ), + ), + Cell( + why="1M tp4 r019-land2 ragged tail, pre-merge share: r in 1..65, 2**k - 1, 2**k, " + "2**k + 1 for k 7..12, and 8160..8192", + spec=dict( + _ML4_1M, + rows=8192, + pages=1100 * 63, + layers=1, + context=True, + cached=1015808, + sub_batch=8192, + lead_rows=24576, + tail_rows=_TAIL_SHARE, + ), + ), + ) + # The tep4 deployment at the 1M envelope, its three call sites: DecodeTarget._attention + # (captured at 1/2/4/8/16 padded, every batch 1..16 under the LLM API's default list, + # 72 calls on one sched meta made per step), PrefillTarget._attn_gen (a mixed step's + # 1..16 generation rows, eager, q and out tail views of the step's scratch, a fresh + # sched meta), and PrefillTarget._attn_ctx on a step whose context KV exceeds 2**19 + # rows (every context row a one-row sequence, sub-batches of at most 8192 rows, the + # last one tc mod 8192 rows: any r in 1..8192). + + ( + Cell( + why="1M tep4 decode graphs, every batch 1..16, boot pool (large): 72 layers on " + "one sched meta, rows of 1048576 keys down to 1 in the top 73898 of 74183 " + "blocks (98.1 GB), the pool's last slot read", + spec=dict( + _TEP4_1M, + rows=16, + pages=_BOOT_PAGES, + layers=72, + captured=True, + graph_batches=tuple(range(1, 17)), + far=True, + large=True, + ), + ), + Cell( + why="1M tep4 decode graph 16, far pool (large): 107917 blocks (142.7 GB), slot " + "217560671 at byte 1.427e11 read, 72 layers", + spec=dict( + _TEP4_1M, + rows=16, + pages=_FAR_PAGES, + layers=72, + captured=True, + far=True, + large=True, + ), + ), + Cell( + why="1M tep4 mixed-step generation tail (large): 13 rows, q and out tail views " + "from row 32755 of the step's 32768-row scratch, eager, a fresh sched meta, " + "boot pool", + spec=dict( + _TEP4_1M, + rows=13, + pages=_BOOT_PAGES, + layers=3, + q_offset=32755, + far=True, + large=True, + ), + ), + Cell( + why="1M tep4 NoPE decode graph 16 (large): owner row 35 as layer 35 and its nine " + "sharers read it, zero rope halves, boot pool, 9 layers", + spec=dict( + _TEP4_1M, + rows=16, + pages=_BOOT_PAGES, + layers=9, + captured=True, + nope=True, + far=True, + large=True, + ), + ), + Cell( + why="1M tep4 pre-merge decode graphs at every batch 1..16 to 2049 keys (2048 of " + "2049 valid, -1 tails below 2048), 72 layers on one sched meta", + spec=dict( + _TEP4_1M, + rows=16, + kv_len=2049, + pages=512, + layers=72, + captured=True, + graph_batches=tuple(range(1, 17)), + ), + ), + Cell( + why="1M tep4 context rows, full chunk (large): 32768 rows behind 1015808 cached " + "keys, 4 sub-batches of 8192 one-row sequences on one sched meta, 72 layers, " + "boot pool", + spec=dict( + _TEP4_1M, + rows=8192, + pages=_BOOT_PAGES, + layers=72, + context=True, + chunk_rows=32768, + cached=1015808, + sub_batch=8192, + far=True, + large=True, + ), + ), + Cell( + why="1M tep4 context rows, ragged chunk (large): 32751 rows behind 1015808, " + "sub-batches 8192 x 3 and 8175 on two sched metas, boot pool", + spec=dict( + _TEP4_1M, + rows=8192, + pages=_BOOT_PAGES, + layers=3, + context=True, + chunk_rows=32751, + cached=1015808, + sub_batch=8192, + far=True, + large=True, + ), + ), + Cell( + why="1M tep4 context rows, packed step (large): chunks of 16384/8192/4096/4096 " + "behind 1032192/524288/0/2048, sub-batches across request edges, boot pool", + spec=dict( + _TEP4_1M, + rows=8192, + pages=_BOOT_PAGES, + layers=3, + context=True, + contexts=((1032192, 16384), (524288, 8192), (0, 4096), (2048, 4096)), + sub_batch=8192, + far=True, + large=True, + ), + ), + Cell( + why="1M tep4 context rows, 16-row chunk tail (large): positions 1048560..1048575 " + "in one sub-batch, 72 layers, boot pool", + spec=dict( + _TEP4_1M, + rows=16, + pages=_BOOT_PAGES, + layers=72, + context=True, + chunk_rows=16, + cached=1048560, + sub_batch=8192, + far=True, + large=True, + ), + ), + Cell( + why="1M tep4 pre-merge context sub-batches past 2**31 pool bytes: a chunk's last " + "12000 rows behind 1036576 cached keys, 8192 + 3808 on two sched metas, every " + "slot above 3273604, 2 layers", + spec=dict( + _TEP4_1M, + rows=8192, + pages=_PRE_PAGES, + top_blocks=400, + layers=2, + context=True, + chunk_rows=12000, + cached=1036576, + sub_batch=8192, + ), + ), + Cell( + why="1M tep4 pre-merge generation tail past 2**31 pool bytes: 13 rows of keys to " + "1048576, q and out tail views from row 32755 of a 32768-row scratch, 3 layers", + spec=dict( + _TEP4_1M, + rows=13, + pages=_PRE_PAGES, + top_blocks=400, + layers=3, + q_offset=32755, + ), + ), + Cell( + why="1M tep4 pre-merge r019-land1 last step: 24672 context rows behind 1015328 " + "cached keys, sub-batches 8192 x 3 + 96 on two sched metas, 72 layers, every " + "slot above 3273604", + spec=dict( + _TEP4_1M, + rows=8192, + pages=_PRE_PAGES, + top_blocks=400, + layers=72, + context=True, + chunk_rows=24672, + cached=1015328, + sub_batch=8192, + ), + ), + Cell( + why="1M tep4 context rows, ragged last sub-batch at every r in 1..8192 (large): " + "rows 24576 + [0, r) of a 32768-row chunk behind 1015328 cached keys (positions " + "to 1048095), each r one call on a sched meta of its own, every slot above " + "3273604", + spec=dict( + _TEP4_1M, + rows=8192, + pages=_PRE_PAGES, + top_blocks=400, + layers=1, + context=True, + cached=1015328, + sub_batch=8192, + ragged_tail=True, + large=True, + ), + ), + Cell( + why="1M tep4 pre-merge ragged last sub-batch, the sweep's share: r in 1..65, 96, " + "2**k - 1 / 2**k / 2**k + 1 to 4097, 8160 and 8175..8192 behind 1015328 cached " + "keys", + spec=dict( + _TEP4_1M, + rows=8192, + pages=_PRE_PAGES, + top_blocks=400, + layers=1, + context=True, + cached=1015328, + sub_batch=8192, + ragged_tail=True, + tails=_RAGGED_SHARE, + ), + ), + ) + # The dep4 deployment at the 1M envelope, its three call sites: DecodeTarget._attention + # (captured at 1/2/4/8/16 padded, every batch 1..16 under the LLM API's default list, + # 72 calls on one sched meta made per step; under attention DP every rank replays the + # largest rank's batch, one live row behind one-key padding dummies; eager at any + # count 1..16 beside a prefilling peer), PrefillTarget._attn_gen (a mixed step's 1..15 + # generation rows, eager, q and out tail views of the step's scratch, a fresh sched + # meta), and PrefillTarget._attn_ctx on a step whose context KV exceeds 2**19 rows + # (every context row a one-row sequence, sub-batches of at most 8192 rows, the last one + # tc - 8192 x (ceil(tc / 8192) - 1) rows: any r in 1..8192). + + ( + Cell( + why="1M dep4 decode graphs, every batch 1..16, boot pool (large): 72 layers on " + "one sched meta, rows of 1048576 keys down to 1 in the top 73898 of 74183 " + "blocks (98.1 GB), the pool's last slot read", + spec=dict( + _DEP4_1M, + rows=16, + pages=_BOOT_PAGES, + layers=72, + captured=True, + graph_batches=tuple(range(1, 17)), + far=True, + large=True, + ), + ), + Cell( + why="1M dep4 ADP boot capture at 1/2/4/8/16, boot pool (large): captured on " + "the warm-up's 1048575-key longest dummy beside one-key fillers, replayed with " + "one live row of 1048576 keys behind one-key padding dummies on one slot, then " + "an idle rank's dummy behind them, 72 layers", + spec=dict( + _DEP4_1M, + rows=16, + pages=_BOOT_PAGES, + layers=72, + captured=True, + graph_batches=_CAPTURED, + live=1, + boot_capture=True, + far=True, + large=True, + ), + ), + Cell( + why="1M dep4 decode graph 16, far pool (large): 107917 blocks (142.7 GB), slot " + "217560671 at byte 1.427e11 read, 72 layers", + spec=dict( + _DEP4_1M, + rows=16, + pages=_FAR_PAGES, + layers=72, + captured=True, + far=True, + large=True, + ), + ), + Cell( + why="1M dep4 mixed-step generation tail (large): 13 rows, q and out tail views " + "from row 32755 of the step's 32768-row scratch, eager, a fresh sched meta, " + "boot pool", + spec=dict( + _DEP4_1M, + rows=13, + pages=_BOOT_PAGES, + layers=3, + q_offset=32755, + far=True, + large=True, + ), + ), + Cell( + why="1M dep4 NoPE decode graph 16 (large): owner row 35 as layer 35 and its " + "sharers read it, zero rope halves, boot pool, 9 layers", + spec=dict( + _DEP4_1M, + rows=16, + pages=_BOOT_PAGES, + layers=9, + captured=True, + nope=True, + far=True, + large=True, + ), + ), + Cell( + why="1M dep4 pre-merge decode graphs at every batch 1..16 to 2049 keys (2048 of " + "2049 valid, -1 tails below 2048), 72 layers on one sched meta", + spec=dict( + _DEP4_1M, + rows=16, + kv_len=2049, + pages=512, + layers=72, + captured=True, + graph_batches=tuple(range(1, 17)), + ), + ), + Cell( + why="1M dep4 pre-merge eager decode steps at every row count 1..16 beside a " + "prefilling peer: a fresh sched meta each, 72 layers, keys to 1048576 over an " + "1100-block pool", + spec=dict( + _DEP4_1M, + rows=16, + pages=_SMALL_PAGES, + layers=72, + row_counts=tuple(range(1, 17)), + ), + ), + Cell( + why="1M dep4 context rows, full chunk (large): 32768 rows behind 1015808 cached " + "keys, 4 sub-batches of 8192 one-row sequences on one sched meta, 72 layers, " + "boot pool", + spec=dict( + _DEP4_1M, + rows=8192, + pages=_BOOT_PAGES, + layers=72, + context=True, + chunk_rows=32768, + cached=1015808, + sub_batch=8192, + far=True, + large=True, + ), + ), + Cell( + why="1M dep4 context rows, ragged chunk (large): 32751 rows behind 1015808, " + "sub-batches 8192 x 3 and 8175 on two sched metas, boot pool", + spec=dict( + _DEP4_1M, + rows=8192, + pages=_BOOT_PAGES, + layers=3, + context=True, + chunk_rows=32751, + cached=1015808, + sub_batch=8192, + far=True, + large=True, + ), + ), + Cell( + why="1M dep4 context rows, packed step (large): chunks of 16384/8192/4096/4096 " + "behind 1032192/524288/0/2048, sub-batches across request edges, boot pool", + spec=dict( + _DEP4_1M, + rows=8192, + pages=_BOOT_PAGES, + layers=3, + context=True, + contexts=((1032192, 16384), (524288, 8192), (0, 4096), (2048, 4096)), + sub_batch=8192, + far=True, + large=True, + ), + ), + Cell( + why="1M dep4 context rows, 16-row chunk tail (large): positions 1048560..1048575 " + "in one sub-batch, 72 layers, boot pool", + spec=dict( + _DEP4_1M, + rows=16, + pages=_BOOT_PAGES, + layers=72, + context=True, + chunk_rows=16, + cached=1048560, + sub_batch=8192, + far=True, + large=True, + ), + ), + Cell( + why="1M dep4 pre-merge context rows, one 8192-row sub-batch: the last chunk of a " + "1M prompt behind 1040384 cached keys over an 1100-block pool, 2 layers", + spec=dict( + _DEP4_1M, + rows=8192, + pages=_SMALL_PAGES, + layers=2, + context=True, + chunk_rows=8192, + cached=1040384, + sub_batch=8192, + ), + ), + Cell( + why="1M dep4 pre-merge context sub-batches past 2**31 pool bytes: a chunk's last " + "12000 rows behind 1036576 cached keys, 8192 + 3808 on two sched metas, every " + "slot above 3322368, 2 layers", + spec=dict( + _DEP4_1M, + rows=8192, + pages=_PRE_PAGES, + top_blocks=400, + layers=2, + context=True, + chunk_rows=12000, + cached=1036576, + sub_batch=8192, + ), + ), + Cell( + why="1M dep4 pre-merge ADP boot capture past 2**31 pool bytes: graphs 1/2/4/8/16 " + "captured on the 1048575-key dummy and one-key fillers, replayed with one live " + "row behind padding on one slot and with an idle rank's dummy, 72 layers", + spec=dict( + _DEP4_1M, + rows=16, + pages=_PRE_PAGES, + top_blocks=400, + layers=72, + captured=True, + graph_batches=_CAPTURED, + live=1, + boot_capture=True, + ), + ), + Cell( + why="1M dep4 pre-merge generation tail past 2**31 pool bytes: 13 rows of keys to " + "1048576, q and out tail views from row 32755 of a 32768-row scratch, 3 layers", + spec=dict( + _DEP4_1M, + rows=13, + pages=_PRE_PAGES, + top_blocks=400, + layers=3, + q_offset=32755, + ), + ), + Cell( + why="1M dep4 pre-merge r017-land2 last step: 22829 context rows behind 1015808 " + "cached keys (positions to 1038636), sub-batches 8192 x 2 + 6445 on two sched " + "metas, 72 layers, every slot above 3322368", + spec=dict( + _DEP4_1M, + rows=8192, + pages=_PRE_PAGES, + top_blocks=400, + layers=72, + context=True, + chunk_rows=22829, + cached=1015808, + sub_batch=8192, + ), + ), + Cell( + why="1M dep4 context rows, ragged last sub-batch at every r in 1..8192 (large): " + "rows 16384 + [0, r) of a chunk behind 1015808 cached keys (positions to " + "1040383), each r one call on a sched meta of its own, every slot above 3322368", + spec=dict( + _DEP4_1M, + rows=8192, + pages=_PRE_PAGES, + top_blocks=400, + layers=1, + context=True, + cached=1015808, + lead_rows=16384, + sub_batch=8192, + ragged_tail=True, + large=True, + ), + ), + Cell( + why="1M dep4 pre-merge ragged last sub-batch, the sweep's share: r in 1..65, " + "2**k - 1 / 2**k / 2**k + 1 to 4097, 6445, 8096 and 8175..8192, rows 16384 + " + "[0, r) behind 1015808 cached keys", + spec=dict( + _DEP4_1M, + rows=8192, + pages=_PRE_PAGES, + top_blocks=400, + layers=1, + context=True, + cached=1015808, + lead_rows=16384, + sub_batch=8192, + tail_rows=_DEP4_TAIL_SHARE, + ), + ), + ) + # The tep8 deployment at the 1M envelope, its three call sites: DecodeTarget._attention + # (captured at 1/2/4/8/16 padded, every batch 1..16 under the LLM API's default list, + # 72 calls on one sched meta made per step), PrefillTarget._attn_gen (a mixed step's + # 1..16 generation rows, eager, q and out tail views of the step's scratch, a fresh + # sched meta), and PrefillTarget._attn_ctx on a step whose context KV exceeds 2**19 + # rows (every context row a one-row sequence, sub-batches of at most 8192 rows, the + # last one tc - 8192 x (ceil(tc / 8192) - 1) rows: any r in 1..8192). + + ( + Cell( + why="1M tep8 decode graphs, every batch 1..16, far pool (large): 72 layers on " + "one sched meta, rows of 1048576 keys down to 1 in the top 73898 of 147456 " + "blocks (195.0 GB), slot 297271295 at byte 1.950e11 read", + spec=dict( + _TEP8_1M, + rows=16, + pages=_BOUND_PAGES, + layers=72, + captured=True, + graph_batches=tuple(range(1, 17)), + far=True, + large=True, + ), + ), + Cell( + why="1M tep8 mixed-step generation tail (large): 13 rows, q and out tail views " + "from row 32755 of the step's 32768-row scratch, eager, a fresh sched meta, " + "far pool", + spec=dict( + _TEP8_1M, + rows=13, + pages=_BOUND_PAGES, + layers=3, + q_offset=32755, + far=True, + large=True, + ), + ), + Cell( + why="1M tep8 NoPE decode graph 16 (large): owner row 35, which layer 35 and its " + "nine sharers read, zero rope halves, far pool, 9 layers", + spec=dict( + _TEP8_1M, + rows=16, + pages=_BOUND_PAGES, + layers=9, + captured=True, + nope=True, + far=True, + large=True, + ), + ), + Cell( + why="1M tep8 pre-merge decode graphs at every batch 1..16 to 2049 keys (2048 of " + "2049 valid, -1 tails below 2048), 72 layers on one sched meta", + spec=dict( + _TEP8_1M, + rows=16, + kv_len=2049, + pages=512, + layers=72, + captured=True, + graph_batches=tuple(range(1, 17)), + ), + ), + Cell( + why="1M tep8 context rows, full chunk (large): 32768 rows behind 1015808 cached " + "keys, 4 sub-batches of 8192 one-row sequences on one sched meta, 72 layers, " + "far pool", + spec=dict( + _TEP8_1M, + rows=8192, + pages=_BOUND_PAGES, + layers=72, + context=True, + chunk_rows=32768, + cached=1015808, + sub_batch=8192, + far=True, + large=True, + ), + ), + Cell( + why="1M tep8 context rows, ragged chunk (large): 32751 rows behind 1015808, " + "sub-batches 8192 x 3 and 8175 on two sched metas, far pool", + spec=dict( + _TEP8_1M, + rows=8192, + pages=_BOUND_PAGES, + layers=3, + context=True, + chunk_rows=32751, + cached=1015808, + sub_batch=8192, + far=True, + large=True, + ), + ), + Cell( + why="1M tep8 context rows, packed step (large): chunks of 16384/8192/4096/4096 " + "behind 1032192/524288/0/2048, sub-batches across request edges, far pool", + spec=dict( + _TEP8_1M, + rows=8192, + pages=_BOUND_PAGES, + layers=3, + context=True, + contexts=((1032192, 16384), (524288, 8192), (0, 4096), (2048, 4096)), + sub_batch=8192, + far=True, + large=True, + ), + ), + Cell( + why="1M tep8 context rows, 16-row chunk tail (large): positions 1048560..1048575 " + "in one sub-batch, 72 layers, far pool", + spec=dict( + _TEP8_1M, + rows=16, + pages=_BOUND_PAGES, + layers=72, + context=True, + chunk_rows=16, + cached=1048560, + sub_batch=8192, + far=True, + large=True, + ), + ), + Cell( + why="1M tep8 pre-merge context sub-batches past 2**31 pool bytes: a chunk's last " + "12000 rows behind 1036576 cached keys, 8192 + 3808 on two sched metas, every " + "slot above 3273604, 2 layers", + spec=dict( + _TEP8_1M, + rows=8192, + pages=_PRE_PAGES, + top_blocks=400, + layers=2, + context=True, + chunk_rows=12000, + cached=1036576, + sub_batch=8192, + ), + ), + Cell( + why="1M tep8 pre-merge generation tail past 2**31 pool bytes: 13 rows of keys to " + "1048576, q and out tail views from row 32755 of a 32768-row scratch, 3 layers", + spec=dict( + _TEP8_1M, + rows=13, + pages=_PRE_PAGES, + top_blocks=400, + layers=3, + q_offset=32755, + ), + ), + Cell( + why="1M tep8 pre-merge r018-land1 last step: 24672 context rows behind 1015328 " + "cached keys, sub-batches 8192 x 3 + 96 on two sched metas, 72 layers, every " + "slot above 3273604", + spec=dict( + _TEP8_1M, + rows=8192, + pages=_PRE_PAGES, + top_blocks=400, + layers=72, + context=True, + chunk_rows=24672, + cached=1015328, + sub_batch=8192, + ), + ), + Cell( + why="1M tep8 context rows, ragged last sub-batch at every r in 1..8192 (large): " + "rows 24576 + [0, r) of a 32768-row chunk behind 1015328 cached keys (positions " + "to 1048095), each r one call on a sched meta of its own, every slot above " + "3273604", + spec=dict( + _TEP8_1M, + rows=8192, + pages=_PRE_PAGES, + top_blocks=400, + layers=1, + context=True, + cached=1015328, + sub_batch=8192, + ragged_tail=True, + large=True, + ), + ), + Cell( + why="1M tep8 pre-merge ragged last sub-batch, the sweep's share: r in 1..65, 96, " + "2**k - 1 / 2**k / 2**k + 1 to 4097, 8160 and 8175..8192 behind 1015328 cached " + "keys", + spec=dict( + _TEP8_1M, + rows=8192, + pages=_PRE_PAGES, + top_blocks=400, + layers=1, + context=True, + cached=1015328, + sub_batch=8192, + ragged_tail=True, + tails=_RAGGED_SHARE, + ), + ), + ) + ) + # `out` None at the 1M config, the decode call the targets read in place (dep4's always, + # tp4 / tep4's under `_GEN_ATTN_FLASH_MLA` with the fused glue): the "1M tep4" / "1M dep4" + # harness's pools and rows, every layer's call returning the kernel's 64-head output, its + # live heads bitwise the `out` call's copy on the same inputs and gated to the reference. + CELLS += ( + Cell( + why="1M view tp4/tep4 decode graphs at every batch 1..16, boot pool (large): out " + "None, the 16 live heads of the 64-head output read in place, 3 layers on one " + "sched meta, rows of 1048576 keys down to 1, the pool's last slot read", + spec=dict( + _TEP4_1M, + rows=16, + pages=_BOOT_PAGES, + layers=3, + captured=True, + graph_batches=tuple(range(1, 17)), + far=True, + view=True, + large=True, + ), + ), + Cell( + why="1M view tp4/tep4 decode graph 16, far pool (large): 107917 blocks (142.7 GB), " + "out None, the top slot read at byte 1.427e11", + spec=dict( + _TEP4_1M, + rows=16, + pages=_FAR_PAGES, + layers=3, + captured=True, + far=True, + view=True, + large=True, + ), + ), + Cell( + why="1M view dep4 decode graphs at every batch 1..16, boot pool (large): out None, " + "all 64 heads as returned, 3 layers on one sched meta", + spec=dict( + _DEP4_1M, + rows=16, + pages=_BOOT_PAGES, + layers=3, + captured=True, + graph_batches=tuple(range(1, 17)), + far=True, + view=True, + large=True, + ), + ), + Cell( + why="1M view dep4 NoPE decode graph 16 (large): owner row 35, zero rope halves, out " + "None, boot pool", + spec=dict( + _DEP4_1M, + rows=16, + pages=_BOOT_PAGES, + layers=3, + captured=True, + nope=True, + far=True, + view=True, + large=True, + ), + ), + ) + + note = """ + Call sites: mistral_large4 targets, the generation rows of every layer. Kernels + (sm_103): smxx get_mla_metadata (first call of a step), sm100 head64 + flash_fwd_splitkv_mla_fp8_sparse (V32 format), the split combine. The kernel holds + each tile scale as E8M0 rounding toward zero: exact for the power-of-two scales the + fp8_ds_mla writer stores, off by up to one binade for any other. + + `out` None launches the same kernels on the same arguments and returns their output + instead of copying its first `h_out` heads (an exact copy): the live heads are bitwise + the `out` call's (tested), and a V up-projection BMM over the strided view returns the + bits it returns over the copy (tested). Nothing in it depends on the sequence length or + the pool: its cells run the 32K decode batches and the 1M config's (`1M view`, keys to + 1048576 on the boot and far pools, 16 and 64 live heads). + + Cost: a call that makes a fresh sched meta pays a fixed cost over one that shares the + step's sched meta. + + The "1M tp4" cells are the Mistral Large 4 NVFP4 tp4 deployment at max_seq_len 1048576 with + chunked prefill (32768-row steps, up to 16 requests), 16 live heads of its 64-head + scratch, `kv_cache` the whole latent pool viewed [blocks * 63, 32, 1, 656] (FlashMLA + requires the cache contiguously valid from its first byte to its last on sm100), + slots (block * 63 + latent_pool_row) * 32 + offset. Three call sites: the decode step + (captured, 72 calls on one sched meta made per step), a mixed step's generation rows + (eager, q and out tail views of the step's scratch, a fresh sched meta), and the + context rows of a step whose context KV exceeds 2**19 rows, each row a one-row + sequence, in sub-batches of at most 8192 rows with one sched meta per sub-batch size + shared by the 72 layers. The kernel reads keys by slot alone: a 1M sequence reaches it + only as slot magnitude, which the cells take to the pool's last slot. Each graph replay + in the cells matches an eager call bitwise. + + The "r019-land2" cells are the decode-path steps of a 1040000-token prompt landing at + the envelope, as the target's switch ran them: 15 chunks of 32768 rows behind 524288 .. + 983040 cached keys, 4 sub-batches of 8192 each, and the last step, 24192 rows behind + 1015808 packed with a 7-token request (sub-batches 8192, 8192, 7815). A step's last + sub-batch is r = (its context rows mod 8192) rows, any r in 1..8192 (generation rows + behind a chunk, any prompt's last chunk): the ragged-tail cell drives every r on a sched + meta of its own, and the scheduler took each. The split-KV schedule follows the row + count, so a ragged sub-batch's rows are not bitwise a full one's (on one draw, 6857 of + the 8192 r differ from the 8192-row call on up to 67 rows). Every r lands inside the + band: per-(row, head) rel L2 at most 5.33e-3 and whole call 2.55e-4 on the cell's draw, + 5.51e-3 and 2.44e-4 on another. The sweep's 8192 kernel calls make it large; pre-merge + drives r in 1..65, either side of each power of two to 4096, and 8160..8192. + + The "1M tep4" cells are the Mistral Large 4 NVFP4 tep4 deployment at max_seq_len 1048576 with + chunked prefill (steps of up to 32768 rows and 16 requests), 16 live heads of its + 64-head scratch, `kv_cache` the whole latent pool viewed [blocks * 63, 32, 1, 656] + (FlashMLA requires the cache contiguously valid from its first byte to its last on + sm100), slots (block * 63 + latent_pool_row) * 32 + offset. Its decode step, a mixed + step's generation rows, and the context rows of a step whose context KV exceeds 2**19 + rows, each row then a one-row sequence in sub-batches of at most 8192 rows with one + sched meta per sub-batch size shared by the 72 layers. The kernel reads keys by slot + alone, so a 1M sequence reaches it as slot magnitude: the cells read the pool's top + blocks. Every graph replay matches an eager call bitwise. + + The decode path's last sub-batch is tc mod 8192 rows, any r in 1..8192, and FlashMLA's + scheduler splits some rows of a call over their keys (`num_splits` > 1, the parts merged + in fp32 by the combine kernel), which ones depending on r and the device's sm parts + alone. Swept at every r in 1..8192 on a sched meta each (rows 24576 + [0, r) of a chunk + behind 1015328 keys, positions to 1048095), one process: every r ran, every row finite + and in the band (one seed: per (row, head) at most 5.25e-3, whole call 2.40e-4), every row + the scheduler kept in one part bitwise the 8192-row call's (which splits none). Rows + split per call on sm_103 (one sm part per SM): every row at r 1..127 (all 96 of r019's + 96-row call), none at 1334 r (128..152, 4096, 8184..8192 among them), 71 to 147 at the + other 6731 (121 at 8175); 828080 split rows over the sweep, about 40% of them not bitwise + the 8192-row call's (327794 and 327350 at two seeds). Its 8192 calls took 18.2 s, so the + sweep is large and a pre-merge share runs the 103 row counts of `_RAGGED_SHARE`. + + The "1M dep4" cells are the Mistral Large 4 NVFP4 dep4 deployment at max_seq_len 1048576 with + chunked prefill (steps of up to 32768 rows and 16 requests per rank) under attention DP: + a rank calls with all 64 heads of its own rows (q its [*, 64, 576] scratch, no padding + heads; out [*, 64, 512]), `kv_cache` the whole latent pool viewed [blocks * 63, 32, 1, + 656] (FlashMLA requires the cache contiguously valid from its first byte to its last on + sm100), slots (block * 63 + latent_pool_row) * 32 + offset. Its decode step: graphs + captured at 1/2/4/8/16 on the boot warm-up's rows (the 1048575-key longest dummy beside + one-key fillers) and replayed as attention DP pads them, one live row behind one-key + padding dummies whose rows repeat one slot and one query, or an idle rank's one-key dummy + behind them; graphs at every batch 1..16; eager steps at every count 1..16 beside a + prefilling peer, a fresh sched meta each. A mixed step's generation rows, tail views of + the step's scratch. The context rows of a step whose context KV exceeds 2**19 rows, each + then a one-row sequence in sub-batches of at most 8192 rows with one sched meta per + sub-batch size shared by the 72 layers. Every graph replay matches an eager call + bitwise. The kernel reads keys by slot alone, so a 1M sequence reaches it as slot + magnitude: the cells read the pool's top blocks. + + The decode path's last sub-batch is tc - 8192 x (ceil(tc / 8192) - 1) rows, any r in + 1..8192 (r017-land2's 1,038,637-token prompt ended on 6445). FlashMLA's scheduler splits + some rows of a call over their keys (`num_splits` > 1, the parts merged in fp32 by the + combine kernel), which ones depending on r and the device's sm parts alone. Swept at the + dep4 call at every r in 1..8192, largest first, one process, a sched meta each (rows + 16384 + [0, r) of a chunk behind 1015808 keys, positions to 1040383, q, slots and out + views of the step's buffers): every r ran, every row finite and in the band (two seeds: + per (row, head) at most 5.65e-3 and 6.03e-3, a call's rel L2 at most 1.92e-4 and + 1.93e-4), every row neither that call nor the 8192-row call split bitwise the 8192-row + call's (which splits none). Rows split per call on sm_103 (one sm part per SM), the + same at both seeds: every row at r 1..127, none at 1334 r (128..152, 4096, 8184..8192 + among them), 71 to 147 at the other 6731 (139 at 6445, 130 at 8096, 121 at 8175); 828080 + split rows over the sweep, 675860 and 680683 of them not bitwise the 8192-row call's at + the two seeds. Its 8192 calls took 19.1 s and the cell 25.7 s, so the sweep is large and + a pre-merge share runs the 103 row counts of `_DEP4_TAIL_SHARE`; largest first, the sweep + peaked at 9.44 GiB allocated, 11.10 GiB reserved. + + The "1M tep8" cells are the Mistral Large 4 NVFP4 tep8 deployment at max_seq_len 1048576 with + chunked prefill (steps of up to 32768 rows and 16 requests), 8 live heads of its + 64-head scratch, `kv_cache` the whole latent pool viewed [blocks * 63, 32, 1, 656] + (FlashMLA requires the cache contiguously valid from its first byte to its last on + sm100), slots (block * 63 + latent_pool_row) * 32 + offset. It calls the op for its + decode step, a mixed step's generation rows, and the context rows of a step whose + context KV exceeds 2**19 rows, each context row then a one-row sequence in sub-batches + of at most 8192 rows with one sched meta per sub-batch size shared by the 72 layers. + The kernel reads keys by slot alone, so a 1M sequence reaches it as slot magnitude: the + cells read the top blocks of a pool at the 147456-block bound, above the 114326 blocks + (top slot 230,481,215) a 1M tep8 rank holds. Every graph replay matches an eager call + bitwise. A 147456-block cell peaks at 182.8-185.9 GiB of device memory (its pool 181.6 + GiB). + + At the tep8 call the decode path's last sub-batch is likewise tc - 8192 x (ceil(tc / + 8192) - 1) rows, any r in 1..8192: get_mla_metadata deals the call's 37 r blocks (per + row 32 of 64 keys and 5 of overhead) in row order to the device's sm parts, ceil(37 r / + parts) + 5 each, so which rows split depends on r and the SM count alone. Swept at + every r in 1..8192 on a sched meta each (rows 24576 + [0, r) of a chunk behind 1015328 + keys, positions to 1048095, heads 8..63 zero), one process: every r ran, every row + finite and in the band (one draw: per (row, head) at most 5.12e-3, whole call + 2.82e-4), every row neither scheduler split bitwise the 8192-row call's (which splits + none). Rows split per call on sm_103 (one sm part per SM): every row at r 1..127 + (all 96 of r018-land1's 96-row call), none at 1334 r (128..152, 4096, 8184..8192 among + them), 71 to 147 at the other 6731 (121 at 8175, 126 at 8176), r for r the same as at + 16 live heads; 828080 split rows over the sweep, about 22% of them not bitwise the + 8192-row call's (185190 and 180132 at two draws). Its 8192 calls took 17.5 s, so the + sweep is large and a pre-merge share runs the 103 row counts of `_RAGGED_SHARE`; walked + largest or smallest r first it reserved 7.91 GiB either way. + + Hard limits (sm_103) + rows get_decoding_sched_meta takes 4 * (5 * rows + 1) bytes of dynamic + shared memory, against the device's opt-in 232448, which caps one + call at 11622 rows; past it the scheduler prints "CUDA error + (.../flashmla-src/csrc/smxx/decode/get_decoding_sched_meta/ + get_decoding_sched_meta.cu:111): invalid argument" and exits the + process (rc 1), no exception. Probed one process each, at the tp4 + and tep4 calls (16 heads out), at the dep4 call (64 heads in and + out, top-2048 behind a 1M prefix) and at the tep8 call (8 heads + out): 8192 and 11622 rows run (finite, rel L2 2.5e-4 and 2.4e-4 on + 64 rows at 16 heads, 2.1e-4 and 1.9e-4 at 64, 2.9e-4 and 2.5e-4 at + 8), 11623 and 16384 exit; every r in 1..8192 ran, one process, a + fresh sched meta each. is_valid refuses rows > 8192, the most the + cells drive + indices int32 slots; the kernel's TMA row coordinate is block * 32 + offset + = the slot, in int32 (656-byte rows of a [pages * 32, 656] tensor + map): 2**31 slots = 67,108,864 pages = 1,065,220 blocks of 63, not + reached (107917 blocks: top slot 217,560,671; 147456 blocks: top + slot 297,271,295) + slot bytes slot * 656 is the TMA unit's address (64-bit) and the tile scales' + load offset is int64; 2**31 bytes at slot 3,273,604, 2**37 at slot + 209,510,600: probed at the pools' last slots, bytes 98,106,720,112 + (74183 blocks), 142,719,800,176 (107917 blocks) and + 195,009,969,520 (147456 blocks), large; in pre-merge every key past + 2**31 bytes (slots from 3,322,368) passed + indices row row * topk in int32 (the kernel's indices offset): 2**31 at + 1,048,576 rows of 2048, past the row cap not reached + q / o_accum rows * 64 * 576 = rows * 36864 q elements pass 2**31 at 58,255 + rows; the split accumulator (rows + sm parts) * 64 * 512 fp32, its + int32 offset n_split * 32768 at rows + sm parts = 65536: both past + the row cap not reached + fp32 no index passes through fp32 + Not certified: more than 8192 rows per call (11622 ran once, in the probe), pools + past 147456 blocks (past 107917 at 16 and 64 heads out), h_q 128, kv_len past + 1048576, page sizes other than 32 at 1M; which rows a call splits on a device with + another SM count (the band holds either way). + """ + + def __call__( + self, + q: torch.Tensor, + kv_cache: torch.Tensor, + indices: torch.Tensor, + sched_meta, + softmax_scale: float, + out: torch.Tensor | None, + ) -> torch.Tensor: + rows, h_q = q.shape[0], q.shape[1] + o, _ = _flash_mla_with_kvcache( + q.view(rows, 1, h_q, q.shape[-1]), + kv_cache, + None, + None, + D_NOPE, + sched_meta, + softmax_scale=softmax_scale, + is_fp8_kvcache=True, + indices=indices.view(rows, 1, indices.shape[-1]), + ) + if out is None: + return o[:, 0] + out.copy_(o[:, 0, : out.shape[1]]) + return out + + def reference( + self, + q: torch.Tensor, + kv_cache: torch.Tensor, + indices: torch.Tensor, + sched_meta, + softmax_scale: float, + out: torch.Tensor | None, + ) -> torch.Tensor: + """fp32, exact dequantization, softmax over the valid slots; a new tensor shaped + like `out` (`[rows, h_q, 512]` when `out` is None), rounded to bf16 once. Ignores + `sched_meta`.""" + shape = (q.shape[0], q.shape[1], D_NOPE) if out is None else tuple(out.shape) + h_out = shape[1] + flat = kv_cache.view(torch.uint8).reshape(-1, ROW_BYTES) + res = torch.zeros(shape, dtype=torch.float32, device=q.device) + with true_fp32_matmul(): + for r in range(q.shape[0]): + idx = indices[r] + idx = idx[idx >= 0].long() + if idx.numel() == 0: + continue + rows = flat[idx] + codes = rows[:, :D_NOPE].contiguous().view(torch.float8_e4m3fn).float() + scale = rows[:, D_NOPE : D_NOPE + 16].contiguous().view(torch.float32) + nope = (codes.view(-1, D_NOPE // TILE, TILE) * scale.unsqueeze(-1)).view(-1, D_NOPE) + rope = rows[:, D_NOPE + 16 :].contiguous().view(torch.bfloat16).float() + k = torch.cat([nope, rope], 1) + s = (q[r, :h_out].float() @ k.T) * softmax_scale + res[r] = torch.softmax(s, -1) @ nope + return res.to(torch.bfloat16 if out is None else out.dtype) + + def is_valid( + self, + q: torch.Tensor, + kv_cache: torch.Tensor, + indices: torch.Tensor, + sched_meta, + softmax_scale: float, + out: torch.Tensor | None, + ) -> None: + if q.shape[1] not in KERNEL_HEADS: + raise ValueError(f"h_q {q.shape[1]} is not one of the kernel's {KERNEL_HEADS}") + if out is not None and out.shape[1] > q.shape[1]: + raise ValueError("out names more heads than q carries") + if indices.dtype != torch.int32: + raise ValueError("indices must be int32 slots") + if kv_cache.shape[-1] != ROW_BYTES or kv_cache.dim() != 4 or kv_cache.shape[2] != 1: + raise ValueError("kv_cache must be [pages, page_size, 1, 656]") + if q.shape[0] > MAX_ROWS: + raise ValueError( + f"{q.shape[0]} rows in one call, more than {MAX_ROWS}: FlashMLA's decode " + "scheduler exits the process past its shared-memory row limit (11622 rows on " + "sm_103); split the rows into sub-batches" + ) + + +flash_mla_sparse_decode = FlashMlaSparseDecode() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/flash_mla_sparse_prefill.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/flash_mla_sparse_prefill.py new file mode 100644 index 000000000000..0a9bcf22bf1d --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/flash_mla_sparse_prefill.py @@ -0,0 +1,949 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Sparse MQA of context rows over a bf16 latent workspace: FlashMLA's sm100f sparse prefill.""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch.modeling_v2.catalog._op import Arch, Cell, OpWrapper, true_fp32_matmul +from tensorrt_llm.flash_mla import flash_mla_sparse_fwd as _flash_mla_sparse_fwd + +D_NOPE = 512 +D_QK = 576 +KERNEL_HEADS = (64, 128) +#: The tep4, dep4 and tep8 deployments' bounded context workspace at max_seq_len 1048576: +#: B = 2**19 bf16 rows. +_WS_ROWS = 2**19 +#: The row sweep's pre-merge share of the 32768 query row counts: 1..65, 2**k - 1, 2**k and +#: 2**k + 1 up to 32768, every 4093rd count, and 32736 (a chunk with generation rows behind). +_ROW_SAMPLE = tuple( + sorted( + [ + *range(1, 66), + *[2**k + d for k in range(7, 16) for d in (-1, 0, 1) if 2**k + d <= 32768], + *range(4093, 32768, 4093), + 32736, + ] + ) +) +#: The dep4 row sweep's pre-merge share of the 32768 query row counts: 1..65, 2**k - 1, +#: 2**k and 2**k + 1 up to 32768, every 4093rd count, 29127 and 29128 (q's byte offset +#: passes 2**31 inside row 29127) and 32767. +_DEP4_ROW_SAMPLE = tuple( + sorted( + [ + *range(1, 66), + *[2**k + d for k in range(7, 16) for d in (-1, 0, 1) if 2**k + d <= 32768], + *range(4093, 32768, 4093), + 29127, + 29128, + ] + ) +) +#: The tep8 row sweep's pre-merge share of the 32768 query row counts: 1..65, 2**k - 1, 2**k +#: and 2**k + 1 up to 32768, every 4093rd count, 29127 and 29128 (q's byte offset passes +#: 2**31 inside row 29127) and 32736 (a chunk with a generation row behind it). +_TEP8_ROW_SAMPLE = tuple( + sorted( + [ + *range(1, 66), + *[2**k + d for k in range(7, 16) for d in (-1, 0, 1) if 2**k + d <= 32768], + *range(4093, 32768, 4093), + 29127, + 29128, + 32736, + ] + ) +) + + +class FlashMlaSparsePrefill(OpWrapper): + """For every query row r and head h < `out.shape[1]`:: + + p = softmax_j(softmax_scale * q[r, h] . kv[j]), j in indices[r] (-1 skipped) + out[r, h] = sum_j p[j] * kv[j][:512] + + `kv`: bf16 `[N, 576]` (the dequantized fp8_ds_mla rows, `ds_mla_dequant_gather`). + `indices`: int32 `[rows, topk]` rows of `kv`, -1 padded. `q`: bf16 `[rows, h_q, 576]`, + h_q 64 or 128 (fewer real heads first, zeros after; `out` names how many are kept). + Writes `out` (bf16 `[rows, h_out, 512]`) and returns it. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # spec: `rows` query rows of one or more requests of `kv_len` keys (the last `new` positions + # of each, all of them without `new`), `h_q` kernel heads with `h_out` real, `topk` causal + # selections; `holes` scatters -1. The "1M tp4" cells add `kv_rows`: `kv` is the first + # N = sum(kv_len) rows of a workspace buffer of that many, its later rows stale (NaN), and + # `out` leads a longer buffer whose tail stays untouched; `selection="topk"`: each row's + # request-local selection as the indexer's top-k returns it (positions 0..p in order then + # -1 while p + 1 <= topk, else topk distinct positions spread over [0, p], 0 and p among + # them, in shuffled order); `dense`: the engine's dense causal index for a skipped indexer + # (0..p, then -1). Both are offset to the request's run in the workspace, -1 kept. With + # `kv_rows`, a "tep4 1M", "dep4 1M" or "tep8 1M" cell is the tep4, dep4 or tep8 deployment's + # bounded context workspace: kv is the first sum(kv_len) rows of a `kv_rows`-row buffer + # whose later rows are stale, each request's run at its cumulative offset, and the + # selections are the indexer's top-`topk` shape (`selection="topk"`, which the "dep4 1M" + # cells name: distinct, spread over [0, p], 0 and p among them; a row that sees fewer + # keys takes all of them, then -1) or with `dense` DSA's dense causal index. + # `row_sweep` also calls q[:m], indices[:m] and out[:m] for every m in 1..rows, and + # `row_counts` for the m it lists: each m's rows bitwise the full call's leading rows, and + # out's rows past m unwritten. + CELLS: tuple[Cell, ...] = ( + Cell( + why="tep4/tp4: 32768 rows of one request, 16 heads padded to 64", + spec=dict(rows=32768, kv_len=[32768], h_q=64, h_out=16, topk=2048), + ), + Cell( + why="dep4: 8192 rows, 64 heads", + spec=dict(rows=8192, kv_len=[8192], h_q=64, h_out=64, topk=2048), + ), + Cell( + why="three requests with cached prefixes, 16 of 64 heads", + spec=dict( + rows=2348, kv_len=[300, 3076, 5186], new=[300, 2047, 1], h_q=64, h_out=16, topk=2048 + ), + ), + Cell( + why="short rows with scattered -1 holes", + spec=dict(rows=700, kv_len=[700], h_q=64, h_out=16, topk=2048, holes=True), + ), + Cell( + why="tep8: 32768 rows of one request, 8 heads padded to 64", + spec=dict(rows=32768, kv_len=[32768], h_q=64, h_out=8, topk=2048), + ), + # The tp4 deployment at the 1M envelope, workspace path (PrefillTarget._attn_ctx, every + # layer, eager): a step whose context KV (cached + new over its context requests) is + # at most B = 2**19 rows attends over kv = buf[:N] of one bf16 [B, 576] buffer; q is + # the context rows of the [T, 64, 576] fused_q scratch, this rank's 16 heads first. + Cell( + why="1M tp4 index mode at N = B = 2**19: a 32768-row chunk behind a 491520-token " + "prefix, each row's 2048 distinct selections spread over its whole prefix, row 0 " + "and its own among them (workspace rows to 524287)", + spec=dict( + rows=32768, + kv_len=[524288], + new=[32768], + h_q=64, + h_out=16, + topk=2048, + kv_rows=524288, + selection="topk", + ), + ), + Cell( + why="1M tp4 index mode, packed chunk step: four requests cached 262144/131072/0/" + "2048 + 16384/8192/4096/4096, runs at workspace offsets 0/278528/417792/421888 " + "(N = 428032 of the 2**19-row buffer), short rows 0..p then -1", + spec=dict( + rows=32768, + kv_len=[278528, 139264, 4096, 6144], + new=[16384, 8192, 4096, 4096], + h_q=64, + h_out=16, + topk=2048, + kv_rows=524288, + selection="topk", + ), + ), + Cell( + why="1M tp4 index mode, a 16-row chunk tail behind a 524272-token prefix (the " + "workspace path's largest), selections to workspace row 524287", + spec=dict( + rows=16, + kv_len=[524288], + new=[16], + h_q=64, + h_out=16, + topk=2048, + kv_rows=524288, + selection="topk", + ), + ), + Cell( + why="1M tp4 short step in the stale 2**19-row buffer: kv = buf[:8192] of one " + "8192-token prompt, every row past it NaN", + spec=dict( + rows=8192, + kv_len=[8192], + new=[8192], + kv_rows=524288, + h_q=64, + h_out=16, + topk=2048, + selection="topk", + ), + ), + Cell( + why="1M tp4 dense mode (the engine skipped the context indexer): three requests " + "2048/1777/700 cached 1748/1700/0, the dense causal index 0..p then -1 at " + "workspace offsets 0/2048/3825", + spec=dict( + rows=1077, + kv_len=[2048, 1777, 700], + new=[300, 77, 700], + h_q=64, + h_out=16, + topk=2048, + dense=True, + kv_rows=524288, + ), + ), + Cell( + why="1M tp4 dense mode: sixteen 2048-token prompts in one 32768-row step, the " + "dense causal index over 16 runs of the workspace", + spec=dict( + rows=32768, + kv_len=[2048] * 16, + new=[2048] * 16, + h_q=64, + h_out=16, + topk=2048, + dense=True, + kv_rows=524288, + ), + ), + # The workspace path's steps of a 1M prompt: chunk k is 32768 rows behind a + # 32768 * (k - 1)-token prefix, N = 32768 * k, for k = 2..15 (chunk 1 is the shared + # 32768-row cell's N, chunk 16 the N = B cell above; past it the decode path). + *( + Cell( + why=f"1M tp4 index mode at N = {n}: a 1M prompt's chunk {n // 32768}, 32768 " + f"rows behind a {n - 32768}-token prefix, each row's 2048 distinct selections " + "spread over its whole prefix, row 0 and its own among them", + spec=dict( + rows=32768, + kv_len=[n], + new=[32768], + h_q=64, + h_out=16, + topk=2048, + kv_rows=524288, + selection="topk", + ), + ) + for n in range(65536, 524288, 32768) + ), + # A fresh request's dense step under 16 rows: the engine's warmup steps (1 and 2 + # rows) and a short request (7), the request's own new tokens its whole KV. + *( + Cell( + why=f"1M tp4 dense mode, a fresh {n}-row step (the engine's warmup steps, a " + f"short request): the dense causal index 0..p over N = {n} workspace rows", + spec=dict( + rows=n, + kv_len=[n], + new=[n], + h_q=64, + h_out=16, + topk=2048, + dense=True, + kv_rows=524288, + ), + ) + for n in (1, 2, 7) + ), + Cell( + why="tep4 1M workspace: a 32768-row chunk at cached 491520 selects over all 2**19 rows", + spec=dict( + rows=32768, + kv_len=[524288], + new=[32768], + h_q=64, + h_out=16, + topk=2048, + kv_rows=_WS_ROWS, + ), + ), + Cell( + why="tep4 1M workspace: a packed step, four runs, the last at offsets 421888..428031", + spec=dict( + rows=32768, + kv_len=[278528, 139264, 4096, 6144], + new=[16384, 8192, 4096, 4096], + h_q=64, + h_out=16, + topk=2048, + kv_rows=_WS_ROWS, + ), + ), + Cell( + why="tep4 1M workspace: a 16-row chunk tail over the whole 2**19-row prefix", + spec=dict( + rows=16, + kv_len=[524288], + new=[16], + h_q=64, + h_out=16, + topk=2048, + kv_rows=_WS_ROWS, + ), + ), + Cell( + why="tep4 1M workspace: an 8192-row first chunk, the buffer's later rows stale", + spec=dict( + rows=8192, + kv_len=[8192], + new=[8192], + h_q=64, + h_out=16, + topk=2048, + kv_rows=_WS_ROWS, + ), + ), + Cell( + why="tep4 1M workspace dense: three requests' causal index over cached prefixes", + spec=dict( + rows=1077, + kv_len=[2048, 1777, 700], + new=[300, 77, 700], + h_q=64, + h_out=16, + topk=2048, + kv_rows=_WS_ROWS, + dense=True, + ), + ), + Cell( + why="tep4 1M workspace dense: sixteen 2048-token prompts in one 32768-row step", + spec=dict( + rows=32768, + kv_len=[2048] * 16, + new=[2048] * 16, + h_q=64, + h_out=16, + topk=2048, + kv_rows=_WS_ROWS, + dense=True, + ), + ), + # The 1M landing's workspace steps outside the cells above: chunks 1..15 of a 1M + # prompt cut to 32736 rows by the generation row behind each (the last at N 491040), + # and the dense fresh steps of 1, 2 and 7 rows (the engine's warmup, a short prompt). + # Any other count in 1..32768 is a prompt's last chunk: the row sweep. + Cell( + why="tep4 1M workspace: a 32736-row chunk at cached 458304, N 491040, a generation row " + "behind it", + spec=dict( + rows=32736, + kv_len=[491040], + new=[32736], + h_q=64, + h_out=16, + topk=2048, + kv_rows=_WS_ROWS, + ), + ), + Cell( + why="tep4 1M workspace dense: a fresh 1-row step, N 1 (the engine's warmup)", + spec=dict( + rows=1, + kv_len=[1], + new=[1], + h_q=64, + h_out=16, + topk=2048, + kv_rows=_WS_ROWS, + dense=True, + ), + ), + Cell( + why="tep4 1M workspace dense: a fresh 2-row step, N 2 (the engine's warmup)", + spec=dict( + rows=2, + kv_len=[2], + new=[2], + h_q=64, + h_out=16, + topk=2048, + kv_rows=_WS_ROWS, + dense=True, + ), + ), + Cell( + why="tep4 1M workspace dense: a fresh 7-row step, N 7 (a short prompt)", + spec=dict( + rows=7, + kv_len=[7], + new=[7], + h_q=64, + h_out=16, + topk=2048, + kv_rows=_WS_ROWS, + dense=True, + ), + ), + Cell( + why="tep4 1M workspace row sweep, large: every m 1..32768 of a 32768-row chunk over " + "all 2**19 rows, each m bitwise the full call's leading rows (195 s)", + spec=dict( + rows=32768, + kv_len=[524288], + new=[32768], + h_q=64, + h_out=16, + topk=2048, + kv_rows=_WS_ROWS, + row_sweep=True, + large=True, + ), + ), + Cell( + why="tep4 1M workspace row sample: the row sweep's pre-merge share, m 1..65, " + "2**k +- 1, every 4093rd, 32736 and 32767 (0.3 s)", + spec=dict( + rows=32768, + kv_len=[524288], + new=[32768], + h_q=64, + h_out=16, + topk=2048, + kv_rows=_WS_ROWS, + row_counts=_ROW_SAMPLE, + ), + ), + Cell( + why="dep4 1M workspace, 64 live heads: a fresh prompt's first 32768-row chunk " + "(cached 0, the KV-capacity estimate's step)", + spec=dict( + rows=32768, + kv_len=[32768], + new=[32768], + h_q=64, + h_out=64, + topk=2048, + kv_rows=_WS_ROWS, + selection="topk", + ), + ), + Cell( + why="dep4 1M workspace, 64 live heads: a 32768-row chunk at cached 491520 selects " + "over all 2**19 rows", + spec=dict( + rows=32768, + kv_len=[524288], + new=[32768], + h_q=64, + h_out=64, + topk=2048, + kv_rows=_WS_ROWS, + selection="topk", + ), + ), + Cell( + why="dep4 1M workspace, 64 live heads: a packed step of four of the rank's requests, " + "the last run at offsets 421888..428031", + spec=dict( + rows=32768, + kv_len=[278528, 139264, 4096, 6144], + new=[16384, 8192, 4096, 4096], + h_q=64, + h_out=64, + topk=2048, + kv_rows=_WS_ROWS, + selection="topk", + ), + ), + Cell( + why="dep4 1M workspace, 64 live heads: a 16-row chunk tail over the whole 2**19-row " + "prefix", + spec=dict( + rows=16, + kv_len=[524288], + new=[16], + h_q=64, + h_out=64, + topk=2048, + kv_rows=_WS_ROWS, + selection="topk", + ), + ), + Cell( + why="dep4 1M workspace, 64 live heads: an 8192-row first chunk, the buffer's later " + "rows stale", + spec=dict( + rows=8192, + kv_len=[8192], + new=[8192], + h_q=64, + h_out=64, + topk=2048, + kv_rows=_WS_ROWS, + selection="topk", + ), + ), + Cell( + why="dep4 1M workspace dense, 64 live heads: three requests' causal index over " + "cached prefixes", + spec=dict( + rows=1077, + kv_len=[2048, 1777, 700], + new=[300, 77, 700], + h_q=64, + h_out=64, + topk=2048, + kv_rows=_WS_ROWS, + dense=True, + ), + ), + Cell( + why="dep4 1M workspace dense, 64 live heads: sixteen 2048-token prompts of the rank " + "in one 32768-row step", + spec=dict( + rows=32768, + kv_len=[2048] * 16, + new=[2048] * 16, + h_q=64, + h_out=64, + topk=2048, + kv_rows=_WS_ROWS, + dense=True, + ), + ), + # The 1M landing's workspace steps outside the cells above: chunk k of a 1M prompt is + # 32768 rows behind 32768 * (k - 1) cached keys over N = 32768 * k rows, k = 2..15 + # (chunk 1 and chunk 16 at N = 2**19 are cells above; past 2**19 the decode path). + *( + Cell( + why=f"dep4 1M workspace, 64 live heads: chunk {n // 32768} of a 1M prompt, 32768 " + f"rows behind {n - 32768} cached keys over N = {n} workspace rows", + spec=dict( + rows=32768, + kv_len=[n], + new=[32768], + h_q=64, + h_out=64, + topk=2048, + kv_rows=_WS_ROWS, + selection="topk", + ), + ) + for n in range(65536, 524288, 32768) + ), + # Fresh dense steps of every rank: the engine's warm-up steps of 1 and 2 rows (and the + # mixed warm-up step's one context row beside a generation row) and a 7-token prompt, + # the request's own new tokens its whole KV. + *( + Cell( + why=f"dep4 1M workspace dense, 64 live heads: {n}-row fresh step over N = {n} " + "workspace rows (the warm-up steps, a short prompt)", + spec=dict( + rows=n, + kv_len=[n], + new=[n], + h_q=64, + h_out=64, + topk=2048, + kv_rows=_WS_ROWS, + dense=True, + ), + ) + for n in (1, 2, 7) + ), + # Every other row count in 1..32768 is a prompt's last chunk, a short prompt, or a chunk + # cut short by the rank's generation rows behind it: the row sweep, at N = 2**19. + Cell( + why="dep4 1M workspace, 64 live heads, row sweep (large): every m 1..32768 of a " + "32768-row chunk over all 2**19 rows, each m bitwise the full call's first m rows", + spec=dict( + rows=32768, + kv_len=[524288], + new=[32768], + h_q=64, + h_out=64, + topk=2048, + kv_rows=_WS_ROWS, + selection="topk", + row_sweep=True, + large=True, + ), + ), + Cell( + why="dep4 1M workspace, 64 live heads, row sample: the row sweep's pre-merge share, " + "m 1..65, 2**k - 1 / 2**k / 2**k + 1, every 4093rd, 29127, 29128 and 32767", + spec=dict( + rows=32768, + kv_len=[524288], + new=[32768], + h_q=64, + h_out=64, + topk=2048, + kv_rows=_WS_ROWS, + selection="topk", + row_counts=_DEP4_ROW_SAMPLE, + ), + ), + Cell( + why="tep8 1M workspace: a 32768-row chunk at cached 491520 selects over all 2**19 rows", + spec=dict( + rows=32768, + kv_len=[524288], + new=[32768], + h_q=64, + h_out=8, + topk=2048, + kv_rows=_WS_ROWS, + ), + ), + Cell( + why="tep8 1M workspace: a packed step, four runs, the last at offsets 421888..428031", + spec=dict( + rows=32768, + kv_len=[278528, 139264, 4096, 6144], + new=[16384, 8192, 4096, 4096], + h_q=64, + h_out=8, + topk=2048, + kv_rows=_WS_ROWS, + ), + ), + Cell( + why="tep8 1M workspace: a 16-row chunk tail over the whole 2**19-row prefix", + spec=dict( + rows=16, + kv_len=[524288], + new=[16], + h_q=64, + h_out=8, + topk=2048, + kv_rows=_WS_ROWS, + ), + ), + Cell( + why="tep8 1M workspace: an 8192-row first chunk, the buffer's later rows stale", + spec=dict( + rows=8192, + kv_len=[8192], + new=[8192], + h_q=64, + h_out=8, + topk=2048, + kv_rows=_WS_ROWS, + ), + ), + Cell( + why="tep8 1M workspace dense: three requests' causal index over cached prefixes", + spec=dict( + rows=1077, + kv_len=[2048, 1777, 700], + new=[300, 77, 700], + h_q=64, + h_out=8, + topk=2048, + kv_rows=_WS_ROWS, + dense=True, + ), + ), + Cell( + why="tep8 1M workspace dense: sixteen 2048-token prompts in one 32768-row step", + spec=dict( + rows=32768, + kv_len=[2048] * 16, + new=[2048] * 16, + h_q=64, + h_out=8, + topk=2048, + kv_rows=_WS_ROWS, + dense=True, + ), + ), + # The 1M landing's workspace steps outside the cells above, as they ran: chunk k of a + # 1,040,000-token prompt is 32736 rows (cut from 32768 by a 7-token prompt's generation + # row riding behind it) behind 32736 * (k - 1) cached keys over N = 32736 * k rows, + # k = 1..15; chunk 16 is 32768 rows behind 491040 over N = 523808; past 2**19 the + # decode path. + *( + Cell( + why=f"tep8 1M workspace: chunk {n // 32736} of a 1M prompt, 32736 rows (a " + f"generation row behind) after {n - 32736} cached keys over N = {n} rows", + spec=dict( + rows=32736, + kv_len=[n], + new=[32736], + h_q=64, + h_out=8, + topk=2048, + kv_rows=_WS_ROWS, + ), + ) + for n in range(32736, 491041, 32736) + ), + Cell( + why="tep8 1M workspace: chunk 16 of a 1M prompt, 32768 rows after 491040 cached keys " + "over N = 523808 rows", + spec=dict( + rows=32768, + kv_len=[523808], + new=[32768], + h_q=64, + h_out=8, + topk=2048, + kv_rows=_WS_ROWS, + ), + ), + # Fresh dense steps (the engine skipped the indexer): the warm-up's 1-row and 2-row + # steps, the mixed warm-up step's one context row beside a generation row, and a 7-token + # prompt, the request's own new tokens its whole KV. + *( + Cell( + why=f"tep8 1M workspace dense: a fresh {n}-row step, N = {n} (the engine's " + "warm-up steps, a short prompt)", + spec=dict( + rows=n, + kv_len=[n], + new=[n], + h_q=64, + h_out=8, + topk=2048, + kv_rows=_WS_ROWS, + dense=True, + ), + ) + for n in (1, 2, 7) + ), + # Every other row count in 1..32768 is a prompt's last chunk, a short prompt, or a chunk + # cut short by the generation rows behind it: the row sweep, at N = 2**19. + Cell( + why="tep8 1M workspace row sweep (large): every m 1..32768 of a 32768-row chunk over " + "all 2**19 rows, each m bitwise the full call's first m rows", + spec=dict( + rows=32768, + kv_len=[524288], + new=[32768], + h_q=64, + h_out=8, + topk=2048, + kv_rows=_WS_ROWS, + row_sweep=True, + large=True, + ), + ), + Cell( + why="tep8 1M workspace row sample: the row sweep's pre-merge share, m 1..65, " + "2**k - 1 / 2**k / 2**k + 1, every 4093rd, 29127, 29128 and 32736", + spec=dict( + rows=32768, + kv_len=[524288], + new=[32768], + h_q=64, + h_out=8, + topk=2048, + kv_rows=_WS_ROWS, + row_counts=_TEP8_ROW_SAMPLE, + ), + ), + ) + + note = """ + Call sites: mistral_large4 targets' context rows, every layer. Kernel (sm_103): + FlashMLA sm100 sparse prefill (`sparse_prefill_fwd`). + + Mistral Large 4 NVFP4 tp4 at 1M (PrefillTarget._attn_ctx, every layer, eager, on a step + whose context KV, cached + new over its context requests, is at most B = 2**19 rows): + q = the context rows of the [T, 64, 576] fused_q scratch (the rank's 16 heads first, + the rest zero), kv = buf[:N] of the bf16 [B, 576] workspace `ds_mla_dequant_gather` + fills (each request's run contiguous, the buffer's later rows stale), indices = the + indexer's request-local top-2048, or the engine's dense causal index when it skipped + the indexer, plus the request's run offset (-1 kept), softmax_scale 128**-0.5, out = + lat_out[:tc] [tc, 16, 512]. h_q 64 runs FlashMLA's sm100 head64 sparse prefill kernel. + A 1M prompt's 32768-row chunks on this path, N = 32768 k for k = 2..16, are each a + cell; a chunk's time rises with N. A row's answer is bitwise the same wherever it sits + in the call (each 4096-row window re-run). + Memory: every call allocates the kernel's own [rows, h_q, 512] bf16 output and two + [rows, h_q] fp32 (max_logits, lse), dropped on return: measured as the call's peak + over its inputs, 2,164,260,864 B (2.02 GiB) at 32768 rows x 64 heads, 541,065,216 B at + 8192. A caller budgeting a 1M step counts it beside the workspace and fused_q. + An index at or past kv.shape[0] is skipped as -1 is (FlashMLA's contract for its + indices) and the row it names is not read: bitwise the answer with -1 in its place, + with NaN rows behind the view. So a kv view shorter than the step's workspace drops + the selections past it without an error; the test's gate sees that defect (the + step's KV counted as its new tokens only) at a whole-call relative L2 of 1.46 on the + N = B cell, against a band of 4e-3. + + tep4 at max_seq_len 1048576 (the `kv_rows` cells): `PrefillTarget._attn_ctx` calls it once + per layer, eager, on a step whose context KV fits the bounded workspace of 2**19 rows. q is + the step's context rows of the [T, 64, 576] fused_q scratch (this rank's 16 heads first, + zeros after), out `lat_out[:tc]` [tc, 16, 512], kv the workspace's first N = cached + new + rows of the step's context requests (`ds_mla_dequant_gather`), and indices the + request-local selections plus each request's offset in the workspace, -1 kept: the + indexer's top-2048, or on a step whose context indexer the engine skipped DSA metadata's + dense causal index (`_get_dense_topk_indices`). On sm_103 the call runs FlashMLA's + `sm100::fwd::head64::sparse_attn_fwd_kernel` (h_q 64 selects it in + `sparse_attn_prefill_interface`, FlashMLA's `api/sparse_fwd.h`; seen in torch.profiler) + and a copy into out. A row's answer is bitwise the same in any call that holds it + (windows of 4093 rows re-run alone), and every leading row count m in 1..32768 of a + 32768-row call over N = 2**19 answers bitwise as that call's first m rows, writing no row + of out past m (the `row_sweep` cell, large, 195 s; pre-merge runs `_ROW_SAMPLE`'s 100 + counts). The kernel is launched one CTA per query row (FlashMLA's sm100 head64 phase1), so + the row count moves only the grid and q's and out's tensor-map extents: a wave of one row + per SM, the next row starting a second. Besides the cells' row counts the 1M + landing (r019-land1) made 32736-row chunks, a generation row behind each, at N = 32736 x j + to 491040, and dense fresh steps of 1, 2 and 7 rows; a prompt's last chunk is any count. + Every call allocates the kernel's own [rows, h_q, 512] bf16 output and fp32 max_logits + and lse [rows, h_q]: 2164260864 B (2.02 GiB) at 32768 x 64, 541065216 B at 8192 x 64; a + caller budgets it beside the workspace and the scratch. + The kernel takes s_kv from kv.shape[0], and an index at or past it answers bitwise as -1 + does: the row is skipped and not read (measured at N, at the buffer's last row and at + 2**31 - 1, with NaN rows past N). A kv view cut short of the step's KV (N from the new + tokens only, say) therefore drops selections with no error and no NaN; the call cannot + refuse it without a device sync, so `is_valid` does not. + + dep4 at max_seq_len 1048576 (the `kv_rows` cells): attention DP, so a rank runs all 64 heads + on its own requests. `PrefillTarget._attn_ctx` calls it once per layer, eager, on a step + whose context KV on the rank fits the bounded workspace of 2**19 rows. q is the step's + context rows of the rank's [T, 64, 576] fused_q scratch (every head live, no padding), out + `lat_out[:tc]` [tc, 64, 512], kv the workspace's first N = cached + new rows of the rank's + context requests (`ds_mla_dequant_gather`, up to 16 of them), and indices the request-local + selections plus each request's offset in the workspace, -1 kept: the indexer's top-2048, or + on a step whose context indexer the engine skipped (every context within 2048 keys) DSA + metadata's dense causal index (`_get_dense_topk_indices`). On sm_103 the call runs FlashMLA's + `sm100::fwd::head64::sparse_attn_fwd_kernel` (h_q 64 selects it in + `sparse_attn_prefill_interface`, FlashMLA's `api/sparse_fwd.h`; seen in torch.profiler) + and a device-to-device copy into out. A row's answer is bitwise the same in any call that + holds it (windows of 4093 rows re-run alone) and from call to call. Heads are not bitwise + independent: heads 0..15 of a 64-live-head call differ from the same call with heads 16..63 + zero (the tep4 / tp4 shape) in 11-14% of elements, relative L2 1.3e-3 to 1.5e-3 between + the two, both equally close to the reference; the 16-live-head cells do not stand in for + this shape. + Besides those cells' row counts the 1M landing (r017-land2) made on this path chunks 2..15 + of a 1,038,637-token prompt (32768 rows behind 32768 x (k - 1) cached keys over N = 32768 k, + to 491520) and every rank's fresh dense steps of 1, 2 and 7 rows (the warm-up's 1-row and + 2-row steps, the mixed warm-up step's one context row, a 7-token prompt), each now a cell. + A prompt's last chunk at or under 2**19, a short prompt and a chunk cut short by the rank's + generation rows make any other count in 1..32768: every leading row count m of a 32768-row + call over N = 2**19 answers bitwise as that call's first m rows and writes no row of out + past m (the `row_sweep` cell, large, 211 s; pre-merge runs `_DEP4_ROW_SAMPLE`'s 101 counts, + 0.16 s). The kernel is launched one CTA of 384 threads per query row (grid = rows, 123 + registers a thread; kineto trace), so the row count moves only the grid and q's and out's + extents: a wave is one row per SM, the next row starting a second, and the op alone + over every m 1..32768 takes 196 s. + Every call allocates the kernel's own [rows, h_q, 512] bf16 output and fp32 max_logits + and lse [rows, h_q]: 2164260864 B (2.02 GiB) at 32768 x 64, 541065216 B at 8192 x 64; a + caller budgets it beside the workspace, the scratch and its own out (another 2 GiB at + 32768 x 64 x 512). + The kernel takes s_kv from kv.shape[0], and an index at or past it answers bitwise as -1 + does: the row is skipped and not read (measured at N, at the buffer's last row and at + 2**31 - 1, with NaN rows past N). A kv view cut short of the step's KV (N from the new + tokens only, say) therefore drops selections with no error and no NaN; the call cannot + refuse it without a device sync, so `is_valid` does not. + + tep8 at max_seq_len 1048576 (the `kv_rows` cells): `PrefillTarget._attn_ctx` calls it + once per layer, eager, on a step whose context KV fits the bounded workspace of 2**19 + rows. q is the step's context rows of the [T, 64, 576] fused_q scratch (this rank's 8 + heads first, zeros after), out `lat_out[:tc]` [tc, 8, 512], kv the workspace's first N = + cached + new rows of the step's context requests (`ds_mla_dequant_gather`), and indices + the request-local selections plus each request's offset in the workspace, -1 kept: the + indexer's top-2048 (every context row's, after the q-split all-gather), or on a step + whose context indexer the engine skipped DSA metadata's dense causal index + (`_get_dense_topk_indices`). On sm_103 the call runs FlashMLA's + `sm100::fwd::head64::sparse_attn_fwd_kernel` (h_q 64 selects it in + `sparse_attn_prefill_interface`, FlashMLA's `api/sparse_fwd.h`; seen in torch.profiler) + and a copy of heads 0..7 into out. A row's answer is bitwise the same in any call that + holds it (windows of 4093 rows re-run alone), and every leading row count m in 1..32768 + of a 32768-row call over N = 2**19 answers bitwise as that call's first m rows and writes + no row of out past m (the `row_sweep` cell, large, 190 s; pre-merge runs + `_TEP8_ROW_SAMPLE`'s 102 counts, 0.15 s). The kernel is launched one CTA of 384 threads + per query row with all 64 heads in it (grid = rows, 123 registers a thread; kineto + trace), and takes s_kv only as the index bound and the extent of kv's tensor map + (`sparse_attn_fwd_kernel`, FlashMLA's `sm100/prefill/sparse/fwd/head64/phase1.cuh`), so + the row count moves only the grid and q's and out's extents: a wave is 152 rows on 152 + SMs, the 153rd row starting a second. + Besides those cells' row counts the 1M landing (r018-land1) made on this path chunks + 1..15 of a 1,040,000-token prompt at 32736 rows (a 7-token prompt's generation row + behind each) behind 32736 x (k - 1) cached keys over N = 32736 k, to 491040, chunk 16 at + 32768 rows over N = 523808, and dense fresh steps of 1, 2 and 7 rows (the engine's + warm-up steps, the mixed warm-up step's one context row, the 7-token prompt), each now a + cell. A prompt's last chunk at or under 2**19, a short prompt and a chunk cut short by + the generation rows behind it make any other count in 1..32768: the sweep. + Every call allocates the kernel's own [rows, h_q, 512] bf16 output and fp32 max_logits + and lse [rows, h_q], whatever h_out is: 2164260864 B (2.02 GiB) at 32768 x 64, + 541065216 B at 8192 x 64; a caller budgets it beside the workspace and the scratch. + The kernel takes s_kv from kv.shape[0], and an index at or past it answers bitwise as -1 + does: the row is skipped and not read (measured at N, at the buffer's last row and at + 2**31 - 1, with NaN rows past N). A kv view cut short of the step's KV (N from the new + tokens only, say) therefore drops selections with no error and no NaN; the call cannot + refuse it without a device sync, so `is_valid` does not. + + Hard limits (sm_103) + q rows x 64 x 576 x 2 bytes 2**31 bytes inside row 29127 probed at + (TMA coordinates) 32768 rows: + rows 29119..29134 gated, every row + window-checked, and every m 29128..32768 of + the tep4, dep4 and tep8 row sweeps (16, 64 + and 8 live heads); passed + q rows x h_q x 576 elements 2**31 at 58255 rows (h_q 64) not reached + (1.21e9 at 32768) + kernel out rows x 64 x 512 x 2 bytes 2**31 bytes at 32768 rows reached at + 32768 rows (its last byte), passed + indices rows x topk int32 offset 2**31 at 1048576 rows not reached + kv index x 1152 bytes 2**31 at index 1864136 not reached: + index x 576 elements 2**31 at index 3728271 the workspace + path stops at N = 2**19, indices to 524287 + read and passed + indices through fp32 2**24 + 1 not reached + (largest 524287) + Not certified: h_q 128 at these shapes, kv past 2**19 rows (the target takes the + decode path there), topk other than 2048, the kernel's topk_length and attn_sink + arguments (the call passes neither), CUDA-graph capture (the call site is eager), + selections from the live indexer (the cells' have its output's shape), a row count no + cell names at an N other than 2**19 (the sweep's; s_kv enters the kernel only as the + index bound and the kv tensor map's extent), a packed step of several requests at a + row count other than 32768. + """ + + def __call__( + self, + q: torch.Tensor, + kv: torch.Tensor, + indices: torch.Tensor, + softmax_scale: float, + out: torch.Tensor, + ) -> torch.Tensor: + rows = q.shape[0] + o = _flash_mla_sparse_fwd( + q, kv.view(-1, 1, D_QK), indices.view(rows, 1, indices.shape[-1]), softmax_scale, D_NOPE + )[0] + out.copy_(o[:, : out.shape[1]]) + return out + + def reference( + self, + q: torch.Tensor, + kv: torch.Tensor, + indices: torch.Tensor, + softmax_scale: float, + out: torch.Tensor, + ) -> torch.Tensor: + """fp32 softmax over the valid rows; a new tensor shaped like `out`.""" + h_out = out.shape[1] + res = torch.zeros(out.shape, dtype=torch.float32, device=q.device) + k_all = kv.float() + with true_fp32_matmul(): + for r in range(q.shape[0]): + idx = indices[r] + idx = idx[idx >= 0].long() + if idx.numel() == 0: + continue + k = k_all[idx] + s = (q[r, :h_out].float() @ k.T) * softmax_scale + res[r] = torch.softmax(s, -1) @ k[:, :D_NOPE] + return res.to(out.dtype) + + def is_valid( + self, + q: torch.Tensor, + kv: torch.Tensor, + indices: torch.Tensor, + softmax_scale: float, + out: torch.Tensor, + ) -> None: + if q.shape[1] not in KERNEL_HEADS or out.shape[1] > q.shape[1]: + raise ValueError(f"h_q must be one of {KERNEL_HEADS} and cover out's heads") + if indices.dtype != torch.int32: + raise ValueError("indices must be int32") + if kv.dtype != torch.bfloat16 or kv.shape[-1] != D_QK: + raise ValueError("kv must be bf16 [N, 576]") + + +flash_mla_sparse_prefill = FlashMlaSparsePrefill() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/fp8_mqa_logits.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/fp8_mqa_logits.py new file mode 100644 index 000000000000..e21cd76d5d76 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/fp8_mqa_logits.py @@ -0,0 +1,974 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Ragged FP8 MQA logits: the DSA indexer's prefill scores, in one DeepGEMM call. + +Wraps the pybind `tensorrt_llm.deep_gemm.fp8_mqa_logits`, not a +`torch.ops.trtllm.*` op: the ragged prefill scorer has no registered-op form, +and the engine's DSA indexer (`Indexer._call_mqa_logits` in +`tensorrt_llm/_torch/attention/backends/sparse/dsa/indexer.py`) calls the +pybind directly. A call given a split-KV schedule (`mqa_logits_schedule`) goes +through `fp8_fp4_mqa_logits`, of which the legacy pybind is the schedule-less +form: the same kernel, its keys spread over every SM. +""" + +from __future__ import annotations + +from typing import Optional, Tuple + +import torch + +from tensorrt_llm._torch.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm.deep_gemm import fp8_fp4_mqa_logits as _deep_gemm_fp8_fp4_mqa_logits +from tensorrt_llm.deep_gemm import fp8_mqa_logits as _deep_gemm_fp8_mqa_logits +from tensorrt_llm.deep_gemm import get_mqa_logits_metadata as _deep_gemm_mqa_logits_metadata +from tensorrt_llm.deep_gemm import get_num_sms as _deep_gemm_num_sms + + +def mqa_logits_schedule( + cu_seq_len_k_start: torch.Tensor, + cu_seq_len_k_end: torch.Tensor, + num_kv_tokens: int, + num_heads: int, +) -> torch.Tensor: + """The split-KV schedule of one `fp8_mqa_logits` call over these windows and keys. + + DeepGEMM's `get_mqa_logits_metadata`: one single-CTA kernel on the current stream + (no host sync) that prices each 4-row query block (`128 / num_heads` rows) by the + 256-key splits of its window span and cuts the summed splits into one contiguous run + per SM, so a block's keys may be spread over several SMs. It returns an int32 tensor + the call takes as `schedule_meta`. The schedule is a function of the windows, the + key count, the head count and the device's SM count only: a caller that scores the + same rows and windows against several layers' keys builds it once.""" + return _deep_gemm_mqa_logits_metadata( + cu_seq_len_k_start, cu_seq_len_k_end, num_kv_tokens, num_heads + ) + + +#: The DSA indexer of Mistral Large 4: 32 heads of 128, fp8 keys with one fp32 +#: scale per key. +_INDEXER = dict(heads=32, head_dim=128) + +#: The Mistral Large 4 FP8 target's fullest context step: max_batch_size 16 +#: requests packed into its 9216-token budget (no KV reuse, no chunked +#: prefill, so no cached prefix). Lengths are chosen so request boundaries fall +#: inside the kernel's 4-row query blocks, and the first request is a single +#: token. +_PACKED_16 = [ + (0, n) for n in (1, 7, 33, 100, 255, 257, 513, 1023, 1025, 2047, 2049, 931, 299, 301, 201, 174) +] + +#: The Mistral Large 4 NVFP4 target's fullest context step: 16 requests packed into +#: its 32768-token budget, the same way. Request starts (and so window starts) +#: at keys 0, 1, 8, 41, 141, 396, 653, 1166, 2189, 3214, 5261, 7310, 11405, +#: 15502, 23693 and 31886; windows up to 8193 keys long. +_PACKED_16_32K = [ + (0, n) + for n in (1, 7, 33, 100, 255, 257, 513, 1023, 1025, 2047, 2049, 4095, 4097, 8191, 8193, 882) +] + +#: The NVFP4 tep4, tp4 and tep8 deployments at their 1M envelope (max_seq_len 1048576, +#: chunked prefill in 32768-token steps): the last chunk of a 1048576-token +#: prompt, behind its 1015808-token cached prefix. Each chunk is scored against +#: the request's whole KV, cached + new, so this step's keys are 1048576. The +#: dep4 deployment's cells use it too: the 32nd and last indexer chunk of a +#: 1048576-token prompt at its 1M envelope (chunked prefill in 32768-token +#: chunks), 32768 new tokens over 1015808 cached, so the chunk gathers +#: n_k = 1048576 keys and its row t sees keys [0, 1015809 + t). +_LAST_CHUNK_1M = [(1015808, 32768)] + +#: Under the engine's query split (tep4 and tp4, a step of 8192 or more context +#: query tokens) rank 0 scores the first 8192 rows of that chunk against the +#: keys up to its share's end, cached + 8192 = 1024000. +_RANK0_SHARE_1M = [(1015808, 8192)] + +#: Rank 1's share of the same chunk: rows 8192..16383 against the keys up to +#: its share's end, cached + 16384 = 1032192. +_RANK1_SHARE_1M = [(1015808, 16384)] + +#: 16 requests' 16-token chunk tails behind 61440-token prefixes, packed into +#: one indexer chunk as the engine packs them while sum(q) x sum(kv) stays +#: within its 2**28 packed-logits budget: 256 x 983296 = 251,723,776. +_TAILS_16_1M = [(61440, 16) for _ in range(16)] + +#: Two 1048576-token prompts' 1-token final chunks, packed into one indexer +#: chunk: 2 x 2097152 = 4,194,304 packed logits. The second request's keys and +#: window start at 1048576 = 2**20. +_FINAL_TOKENS_2X1M = [(1048575, 1), (1048575, 1)] + +#: Three requests' final chunks packed up to the 2**28 budget: two near-1M +#: prompts' 50-token tails and a 277613-token prompt's 13-token tail, +#: 113 x 2373713 = 268,229,569 packed logits. 2373713 keys sit just under the +#: 2,373,856-token per-rank pool of the tp4 deployment's delivered boot, which bounds +#: a packed chunk's keys while block reuse is off. +_FINAL_CHUNKS_POOL = [(1048000, 50), (1048000, 50), (277600, 13)] + +#: A packed indexer chunk at the dep4 deployment's 1M envelope: the engine groups +#: whole requests into one chunk while sum(new) x sum(cached + new) <= 2**28, +#: so 16 requests' 16-token last chunks over 65520 cached keys each are one +#: 1048576-key block of 256 rows, each request's windows inside its own 65536 +#: keys. +_PACKED_16_TAILS_1M = [(65520, 16) for _ in range(16)] + +#: Under the tep8 deployment's query split (eight ranks, a step of 8192 or more +#: context query tokens) rank r scores rows [4096 r, 4096 (r + 1)) of a +#: 32768-token chunk against the keys up to its share's end, cached + +#: 4096 (r + 1): the request's rows up to there are the step. Rank 3's share of +#: the last chunk of a 1048576-token prompt sees 1032192 keys, rank 0's 1019904. +_TEP8_RANK3_SHARE_1M = [(1015808, 16384)] +_TEP8_RANK0_SHARE_1M = [(1015808, 4096)] + + +class Fp8MqaLogits(OpWrapper): + """Score every query row against every key of one ragged prefill chunk. + + Returns `[T, N]` fp32 logits (a row-padded view, not contiguous) with + + logits[i, j] = k_scale[j] * sum_h weights[i, h] * relu(q[i, h, :] . k[j, :]) + + for `cu_seq_len_k_start[i] <= j < cu_seq_len_k_end[i]`, and -inf + elsewhere when `clean_logits`. `kv` is the pair `(k [N, D] fp8, k_scale [N] + fp32)`. The fp8 codes of `q` enter the dot product raw: there is no q-side + scale, so the caller folds it, with the indexer's `D**-0.5 * H**-0.5`, into + `weights`. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # `seqs` lists each request of one context step as (cached keys, new + # tokens); token `t` of a request sees its own keys up to `cached + t`. + # `rows`, when present, is the query tile `[lo, hi)` the call scores, + # passed as views into the step's buffers the way a tiling caller slices + # them; without it the call scores the whole step. The "Mistral Large 4 1M" + # cells are the tep4 and tp4 deployments' calls at their 1M envelope: each + # scores every indexer prefill chunk in query tiles of floor(2**28 / n_k) + # rows, cut from the chunk's first row. The "Mistral Large 4 tep8 1M" cells are + # the tep8 deployment's, which tiles the same way from the first row of its + # rank's share of the chunk under the engine's query split; `share`, when + # present, is that share `[lo, hi)` of the step's rows (the whole step below + # the split threshold), the rows a tile is cut from. Rank 7's last tile at + # 1048576, 524288, 262144, 131072 and 65536 keys and the 16 packed chunk + # tails are the "Mistral Large 4 1M" cells of the same calls. `budget`, when + # present (the dep4 deployment's 1M cells), is the logits budget whose + # floor(budget / keys)-row tiles the call is one of (the whole step when it + # has no `rows`). + CELLS: tuple[Cell, ...] = ( + Cell( + why="one request, no cache: causal windows over the chunk's own keys", + spec=dict(**_INDEXER, seqs=[(0, 2048)]), + ), + Cell( + why="a cached prefix: every window starts at 0 and runs past the chunk", + spec=dict(**_INDEXER, seqs=[(4096, 1024)]), + ), + Cell( + why="two requests in one call: the second's windows start mid-row", + spec=dict(**_INDEXER, seqs=[(0, 300), (1700, 500)]), + ), + Cell( + why="a query count that is not a multiple of the 4-row query block", + spec=dict(**_INDEXER, seqs=[(100, 37)]), + ), + Cell( + why="Mistral Large 4 9216 keys in one call, T = N = 9216: the largest [T, N] the FP8 " + "target's envelope (max_num_tokens = max_seq_len = 9216) admits, for a caller that " + "does not tile", + spec=dict(**_INDEXER, seqs=[(0, 9216)]), + ), + Cell( + why="Mistral Large 4 9216 keys, the 116-row tail tile of the floor(2**24 / 9216) = " + "1820-row query tiles: views 9100 rows into q and weights, windows reaching the " + "last key", + spec=dict(**_INDEXER, seqs=[(0, 9216)], rows=(9100, 9216)), + ), + Cell( + why="Mistral Large 4 16 packed requests (max_batch_size) filling the FP8 target's 9216 " + "tokens in one call: a 1-token request, request boundaries inside 4-row query blocks", + spec=dict(**_INDEXER, seqs=_PACKED_16), + ), + Cell( + why="Mistral Large 4 16 packed requests, the 1820-row tile at rows 7280..9099: " + "windows start at keys 5261, 7310, 8241, 8540, 8841, 9042, mostly off the " + "kernel's 4-key alignment", + spec=dict(**_INDEXER, seqs=_PACKED_16, rows=(7280, 9100)), + ), + Cell( + why="Mistral Large 4 32768 keys in one call, T = N = 32768: the largest [T, N] the NVFP4 " + "target's envelope (max_num_tokens = max_seq_len = indexer_max_chunk_size = 32768) " + "admits, for a caller that does not tile; its [32768, 33024]-element fp32 rows span " + "4,328,521,728 bytes, rows 16257 on past 2**31 bytes and 32514 on past 2**32", + spec=dict(**_INDEXER, seqs=[(0, 32768)]), + ), + Cell( + why="Mistral Large 4 32768 keys, the last of the floor(2**24 / 32768) = 512-row query " + "tiles the NVFP4 target cuts: views 32256 rows into q and weights, windows of " + "32257..32768 keys reaching the last one", + spec=dict(**_INDEXER, seqs=[(0, 32768)], rows=(32256, 32768)), + ), + Cell( + why="Mistral Large 4 32768 tokens in one call, 16 packed requests (max_batch_size): a " + "1-token request, windows up to 8193 keys, request boundaries inside 4-row query " + "blocks", + spec=dict(**_INDEXER, seqs=_PACKED_16_32K), + ), + Cell( + why="Mistral Large 4 32768 tokens, the 512-row tile at rows 31744..32255 of the 16 " + "packed requests: two requests' windows, starting at keys 23693 and 31886, off the " + "kernel's 4-key alignment, the second request's first row mid query block", + spec=dict(**_INDEXER, seqs=_PACKED_16_32K, rows=(31744, 32256)), + ), + Cell( + why="Mistral Large 4 1M, first 256-row tile at 1048576 keys: rows 0..255 of the last " + "32768-token chunk of a 1048576-token prompt, behind its 1015808-token cached " + "prefix, windows of 1015809..1016064 keys; floor(2**28 / 1048576) = 256 rows, a " + "256 x 1048832 fp32 tile of 1,074,003,968 bytes", + spec=dict(**_INDEXER, seqs=_LAST_CHUNK_1M, rows=(0, 256)), + ), + Cell( + why="Mistral Large 4 1M, last 256-row tile at 1048576 keys: rows 32512..32767 of the " + "last chunk, windows of 1048321..1048576 keys reaching the last key at " + "max_seq_len", + spec=dict(**_INDEXER, seqs=_LAST_CHUNK_1M, rows=(32512, 32768)), + ), + Cell( + why="Mistral Large 4 1M, q-split rank 3's first tile: rows 24576..24831 of the last " + "chunk; under the engine's query split rank 3 scores rows 24576..32767 against " + "all 1048576 keys, in 256-row tiles from its share's first row", + spec=dict(**_INDEXER, seqs=_LAST_CHUNK_1M, rows=(24576, 24832)), + ), + Cell( + why="Mistral Large 4 1M, q-split rank 0's last tile: the 70 rows 8122..8191 of rank " + "0's share of the last chunk, against its share's 1024000 keys; floor(2**28 / " + "1024000) = 262-row tiles, 31 of them and this tail, its last query block partial", + spec=dict(**_INDEXER, seqs=_RANK0_SHARE_1M, rows=(8122, 8192)), + ), + Cell( + why="Mistral Large 4 1M, last 512-row tile at 524288 keys: rows 32256..32767 of a " + "chunk behind a 491520-token prefix, floor(2**28 / 524288) = 512 rows", + spec=dict(**_INDEXER, seqs=[(491520, 32768)], rows=(32256, 32768)), + ), + Cell( + why="Mistral Large 4 1M, last 1024-row tile at 262144 keys: rows 31744..32767 of a " + "chunk behind a 229376-token prefix, floor(2**28 / 262144) = 1024 rows", + spec=dict(**_INDEXER, seqs=[(229376, 32768)], rows=(31744, 32768)), + ), + Cell( + why="Mistral Large 4 1M, last 2048-row tile at 131072 keys: rows 30720..32767 of a " + "chunk behind a 98304-token prefix, floor(2**28 / 131072) = 2048 rows", + spec=dict(**_INDEXER, seqs=[(98304, 32768)], rows=(30720, 32768)), + ), + Cell( + why="Mistral Large 4 1M, last 4096-row tile at 65536 keys: rows 28672..32767 of the " + "second chunk of a prompt, behind a 32768-token prefix, windows of 61441..65536 " + "keys crossing 2**16", + spec=dict(**_INDEXER, seqs=[(32768, 32768)], rows=(28672, 32768)), + ), + Cell( + why="Mistral Large 4 1M, 4095 rows at 65537 keys: a floor(2**28 / 65537) = 4095-row " + "tile ending at the step's last row, rows 28673..32767 behind a 32769-token " + "prefix, windows of 61443..65537 keys, an odd key count, its last query block " + "partial", + spec=dict(**_INDEXER, seqs=[(32769, 32768)], rows=(28673, 32768)), + ), + Cell( + why="Mistral Large 4 1M, last 8192-row tile at 32768 keys: rows 24576..32767 of a " + "32768-token first chunk, floor(2**28 / 32768) = 8192 rows; the tile spans " + "8192 x 33024 fp32, 1,082,130,432 bytes", + spec=dict(**_INDEXER, seqs=[(0, 32768)], rows=(24576, 32768)), + ), + Cell( + why="Mistral Large 4 1M, 16 packed chunk tails in one call: 16 requests of 16 new " + "tokens behind 61440-token prefixes, 256 rows against 983296 keys, windows " + "starting at the multiples of 61456 up to 921840", + spec=dict(**_INDEXER, seqs=_TAILS_16_1M), + ), + Cell( + why="Mistral Large 4 1M, a 15-token final chunk at 1048559 keys: the envelope gate's " + "1048559-token prompt (max_seq_len - 16 - 1, 15 tokens past its last full " + "32-token page) scored alone, 15 rows, a key count off the kernel's 4-key " + "alignment", + spec=dict(**_INDEXER, seqs=[(1048544, 15)]), + ), + Cell( + why="Mistral Large 4 1M, the 256-row tile at rows 8192..8447 of the last chunk at " + "1048576 keys, the 33rd tile of the unsplit chunk and the row the q-split's rank 1 " + "share starts at: windows of 1024001..1024256 keys", + spec=dict(**_INDEXER, seqs=_LAST_CHUNK_1M, rows=(8192, 8448)), + ), + Cell( + why="Mistral Large 4 1M, q-split rank 1's first tile: rows 8192..8451 of its share " + "8192..16383 of the last chunk, against the share's 1032192 keys (cached + 16384); " + "floor(2**28 / 1032192) = 260-row tiles cut from the share's first row", + spec=dict(**_INDEXER, seqs=_RANK1_SHARE_1M, rows=(8192, 8452)), + ), + Cell( + why="Mistral Large 4 1M packed, two 1M prompts' 1-token final chunks in one call: 2 rows " + "against 2097152 keys, the second window [1048576, 2097152) starting at 2**20 and " + "ending at 2**21; floor(2**28 / 2097152) = 128 rows, one tile", + spec=dict(**_INDEXER, seqs=_FINAL_TOKENS_2X1M, rows=(0, 2)), + ), + Cell( + why="Mistral Large 4 1M packed, three final chunks at the 2**28 packing budget: 50 + 50 + " + "13 rows against 2373713 keys (cached 1048000, 1048000, 277600), windows starting at " + "0, 1048050 and 2096100; floor(2**28 / 2373713) = 113 rows, one tile of 113 x 2374144 " + "fp32", + spec=dict(**_INDEXER, seqs=_FINAL_CHUNKS_POOL, rows=(0, 113)), + ), + Cell( + why="Mistral Large 4 1048576 keys, the last 256-row tile of the 2**28 logits budget " + "(floor(2**28 / 1048576)): the 32nd chunk of a 1M prompt, 32768 new over 1015808 " + "cached, windows of 1048321..1048576 keys; rows 1048832 fp32 wide", + spec=dict(**_INDEXER, seqs=_LAST_CHUNK_1M, rows=(32512, 32768), budget=2**28), + ), + Cell( + why="Mistral Large 4 1048576 keys, the first 256-row tile of the 2**28 budget: the " + "32nd chunk's rows 0..255, windows of 1015809..1016064 keys", + spec=dict(**_INDEXER, seqs=_LAST_CHUNK_1M, rows=(0, 256), budget=2**28), + ), + Cell( + why="Mistral Large 4 1048576 keys, the last 64-row tile of the 2**26 budget (the " + "fallback): the 32nd chunk's rows 32704..32767", + spec=dict(**_INDEXER, seqs=_LAST_CHUNK_1M, rows=(32704, 32768), budget=2**26), + ), + Cell( + why="Mistral Large 4 1048576 keys, the last 16-row tile of the 2**24 budget (the " + "target's before the 1M envelope): the 32nd chunk's rows 32752..32767", + spec=dict(**_INDEXER, seqs=_LAST_CHUNK_1M, rows=(32752, 32768), budget=2**24), + ), + Cell( + why="Mistral Large 4 524288 keys, the last 512-row tile of the 2**28 budget: the 16th " + "chunk of a 1M prompt, 32768 new over 491520 cached", + spec=dict(**_INDEXER, seqs=[(491520, 32768)], rows=(32256, 32768), budget=2**28), + ), + Cell( + why="Mistral Large 4 131072 keys, the last 2048-row tile of the 2**28 budget: the 4th " + "chunk of a 1M prompt, 32768 new over 98304 cached", + spec=dict(**_INDEXER, seqs=[(98304, 32768)], rows=(30720, 32768), budget=2**28), + ), + Cell( + why="Mistral Large 4 98304 keys, a 2730-row tile of the 2**28 budget (floor(2**28 / " + "98304), not a multiple of the 4-row query block): the 3rd chunk's rows " + "30030..32759", + spec=dict(**_INDEXER, seqs=[(65536, 32768)], rows=(30030, 32760), budget=2**28), + ), + Cell( + why="Mistral Large 4 65536 keys, the last 4096-row tile of the 2**28 budget: the 2nd " + "chunk of a 1M prompt, 32768 new over 32768 cached", + spec=dict(**_INDEXER, seqs=[(32768, 32768)], rows=(28672, 32768), budget=2**28), + ), + Cell( + why="Mistral Large 4 32768 keys, the last 8192-row tile of the 2**28 budget: a 1M " + "prompt's first chunk, no cache, windows of 24577..32768 keys", + spec=dict(**_INDEXER, seqs=[(0, 32768)], rows=(24576, 32768), budget=2**28), + ), + Cell( + why="Mistral Large 4 16 packed chunk tails, 16 tokens over 65520 cached each: one " + "1048576-key block of 256 rows (2**28 elements, the engine's packed-logits cap), " + "windows starting at multiples of 65536", + spec=dict(**_INDEXER, seqs=_PACKED_16_TAILS_1M, budget=2**28), + ), + Cell( + why="Mistral Large 4 2 packed chunk tails, 144 over 262000 cached and 112 over 500000: " + "762256 keys in one call, the second request's windows starting at key 262144", + spec=dict(**_INDEXER, seqs=[(262000, 144), (500000, 112)], budget=2**28), + ), + Cell( + why="Mistral Large 4 packed chunk split per request: a 5-token last chunk over 983040 " + "cached, 983045 keys from key 0, the call each request of the engine's packed " + "1966090-key chunk takes once the target splits it", + spec=dict(**_INDEXER, seqs=[(983040, 5)], budget=2**28), + ), + Cell( + why="Mistral Large 4 2 packed 5-token last chunks over 983040 cached each, unsplit: " + "1966090 keys in one call, past max_seq_len, the second request's first row inside " + "the first's 4-row query block", + spec=dict(**_INDEXER, seqs=[(983040, 5), (983040, 5)], budget=2**28), + ), + # The dep4 deployment's calls (dep8 alike) with the split-KV schedule + # (`split_kv`): its 2**26-element tiles at the 1M envelope and its + # 2**24-element tiles at max_seq_len 32768, each scored with the schedule + # `mqa_logits_schedule` builds over the tile's own windows and keys. + Cell( + why="dep4 split-KV, the last 64-row tile of the 2**26 budget at 1048576 keys: the " + "32nd chunk of a 1M prompt, rows 32704..32767, 16 query blocks spread over every SM", + spec=dict( + **_INDEXER, seqs=_LAST_CHUNK_1M, rows=(32704, 32768), budget=2**26, split_kv=True + ), # fmt: skip + ), + Cell( + why="dep4 split-KV, the first 64-row tile at 1048576 keys: the 32nd chunk's rows " + "0..63, windows of 1015809..1015872 keys", + spec=dict(**_INDEXER, seqs=_LAST_CHUNK_1M, rows=(0, 64), budget=2**26, split_kv=True), + ), + Cell( + why="dep4 split-KV, a 15-row final chunk at 1048559 keys: the envelope prompt's " + "15-token tail alone, a partial query block, a key count off the 4-key alignment", + spec=dict(**_INDEXER, seqs=[(1048544, 15)], budget=2**26, split_kv=True), + ), + Cell( + why="dep4 split-KV, the last 128-row tile at 524288 keys: a chunk behind a " + "491520-token prefix, floor(2**26 / 524288) = 128 rows", + spec=dict( + **_INDEXER, seqs=[(491520, 32768)], rows=(32640, 32768), budget=2**26, split_kv=True + ), # fmt: skip + ), + Cell( + why="dep4 split-KV, the last 256-row tile at 262144 keys: behind a 229376-token prefix", + spec=dict( + **_INDEXER, seqs=[(229376, 32768)], rows=(32512, 32768), budget=2**26, split_kv=True + ), # fmt: skip + ), + Cell( + why="dep4 split-KV, a 682-row tile at 98304 keys (floor(2**26 / 98304), not a " + "multiple of the 4-row query block): the 3rd chunk's rows 32054..32735", + spec=dict( + **_INDEXER, seqs=[(65536, 32768)], rows=(32054, 32736), budget=2**26, split_kv=True + ), # fmt: skip + ), + Cell( + why="dep4 split-KV, the last 1024-row tile at 65536 keys: the 2nd chunk of a 1M " + "prompt, windows crossing 2**16", + spec=dict( + **_INDEXER, seqs=[(32768, 32768)], rows=(31744, 32768), budget=2**26, split_kv=True + ), # fmt: skip + ), + Cell( + why="dep4 split-KV, the first 2048-row tile at 32768 keys: a 1M prompt's first " + "chunk, causal windows of 1..2048 keys, rows whose whole window is one split", + spec=dict(**_INDEXER, seqs=[(0, 32768)], rows=(0, 2048), budget=2**26, split_kv=True), + ), + Cell( + why="dep4 split-KV, the last 512-row tile of the 2**24 budget at 32768 keys: the " + "32K config's whole-prompt step, windows of 32257..32768 keys", + spec=dict( + **_INDEXER, seqs=[(0, 32768)], rows=(32256, 32768), budget=2**24, split_kv=True + ), # fmt: skip + ), + Cell( + why="dep4 split-KV, 16 packed requests at 32768 keys, the 512-row tile at rows " + "31744..32255 (2**24): two requests' windows, request boundaries inside query blocks", + spec=dict( + **_INDEXER, seqs=_PACKED_16_32K, rows=(31744, 32256), budget=2**24, split_kv=True + ), # fmt: skip + ), + Cell( + why="dep4 split-KV, 16 packed chunk tails in one 1048576-key block, the 64-row tile " + "at rows 192..255 (2**26): four requests' windows starting at multiples of 65536", + spec=dict( + **_INDEXER, seqs=_PACKED_16_TAILS_1M, rows=(192, 256), budget=2**26, split_kv=True + ), # fmt: skip + ), + Cell( + why="dep4 split-KV, 2 packed tails over 762256 keys, the 88-row tile at rows " + "88..175 (2**26) across the request boundary at row 144", + spec=dict( + **_INDEXER, + seqs=[(262000, 144), (500000, 112)], + rows=(88, 176), + budget=2**26, + split_kv=True, + ), # fmt: skip + ), + Cell( + why="dep4 split-KV, one request's part of the packed 1966090-key chunk: 5 rows " + "against its own 983045 keys from key 0", + spec=dict(**_INDEXER, seqs=[(983040, 5)], budget=2**26, split_kv=True), + ), + # The tep4 deployment's calls (tep8 alike) with the split-KV schedule: its 2**28-element + # tiles at the 1M envelope, cut from the first row of the rank's share of a chunk under + # the engine's query split, and the pieces of a packed chunk cut per request + # past max_seq_len. At max_seq_len 32768 its 2**24 tiles are the dep4 cells' above. + Cell( + why="tep4 split-KV, q-split rank 3's first 256-row tile at 1048576 keys: rows " + "24576..24831 of the last chunk of a 1M prompt", + spec=dict( + **_INDEXER, seqs=_LAST_CHUNK_1M, rows=(24576, 24832), budget=2**28, split_kv=True + ), # fmt: skip + ), + Cell( + why="tep4 split-KV, q-split rank 0's 70-row last tile at 1024000 keys: a partial " + "query block", + spec=dict( + **_INDEXER, seqs=_RANK0_SHARE_1M, rows=(8122, 8192), budget=2**28, split_kv=True + ), # fmt: skip + ), + Cell( + why="tep4 split-KV, q-split rank 1's first 260-row tile at 1032192 keys", + spec=dict( + **_INDEXER, seqs=_RANK1_SHARE_1M, rows=(8192, 8452), budget=2**28, split_kv=True + ), # fmt: skip + ), + Cell( + why="tep4 split-KV, q-split rank 3's first 512-row tile at 524288 keys: a chunk " + "behind a 491520-token prefix", + spec=dict( + **_INDEXER, seqs=[(491520, 32768)], rows=(24576, 25088), budget=2**28, split_kv=True + ), # fmt: skip + ), + Cell( + why="tep4 split-KV, q-split rank 3's whole share of a 1M prompt's first chunk: 8192 " + "rows against 32768 keys in one tile", + spec=dict( + **_INDEXER, seqs=[(0, 32768)], rows=(24576, 32768), budget=2**28, split_kv=True + ), # fmt: skip + ), + Cell( + why="tep4 split-KV, q-split rank 0's whole share of a first chunk: 8192 rows " + "against its share's 8192 keys, causal windows", + spec=dict( + **_INDEXER, seqs=[(0, 8192)], rows=(0, 8192), budget=2**28, split_kv=True + ), # fmt: skip + ), + Cell( + why="tep8 split-KV, q-split rank 0's 151-row last tile at 1019904 keys", + spec=dict( + **_INDEXER, + seqs=_TEP8_RANK0_SHARE_1M, + rows=(3945, 4096), + budget=2**28, + split_kv=True, + ), # fmt: skip + ), + Cell( + why="tep8 split-KV, q-split rank 7's whole 4096-row share of a first chunk at " + "32768 keys", + spec=dict( + **_INDEXER, seqs=[(0, 32768)], rows=(28672, 32768), budget=2**28, split_kv=True + ), # fmt: skip + ), + Cell( + why="tep4 split-KV, an unsplit 8191-row tail chunk's last 255 rows at 1048576 " + "keys, below the split threshold", + spec=dict( + **_INDEXER, seqs=[(1040385, 8191)], rows=(7936, 8191), budget=2**28, split_kv=True + ), # fmt: skip + ), + Cell( + why="tep4 split-KV, 16 packed chunk tails, 256 rows against 983296 keys in one " + "2**28 tile, windows starting at multiples of 61456", + spec=dict(**_INDEXER, seqs=_TAILS_16_1M, budget=2**28, split_kv=True), + ), + Cell( + why="tep4 split-KV, the packed-chunk cut: a ~1M request's 100-token tail against its " + "own 1048500 keys, the piece of a 1048655-key chunk the target scores alone", + spec=dict(**_INDEXER, seqs=[(1048400, 100)], budget=2**28, split_kv=True), + ), + Cell( + why="Mistral Large 4 tep8 1M, rank 7 first tile at 1048576 keys: rows 28672..28927 of " + "the last 32768-token chunk of a 1048576-token prompt behind its 1015808-token " + "cached prefix, windows of 1044481..1044736 keys; under tep8's query split rank 7 " + "scores rows 28672..32767 against all 1048576 keys in 256-row tiles", + spec=dict(**_INDEXER, seqs=_LAST_CHUNK_1M, rows=(28672, 28928), share=(28672, 32768)), + ), + Cell( + why="Mistral Large 4 tep8 1M, rank 3 last tile at 1032192 keys: the 196 rows " + "16188..16383 of rank 3's share 12288..16383 of the last chunk, against the " + "share's cached + 16384 keys; floor(2**28 / 1032192) = 260-row tiles, 15 of them " + "and this tail, windows of 1031997..1032192 keys reaching the share's last key", + spec=dict( + **_INDEXER, seqs=_TEP8_RANK3_SHARE_1M, rows=(16188, 16384), share=(12288, 16384) + ), + ), + Cell( + why="Mistral Large 4 tep8 1M, rank 3 first tile at 1032192 keys: rows 12288..12547 of " + "the last chunk, the first 260-row tile of rank 3's share, windows of " + "1028097..1028356 keys", + spec=dict( + **_INDEXER, seqs=_TEP8_RANK3_SHARE_1M, rows=(12288, 12548), share=(12288, 16384) + ), + ), + Cell( + why="Mistral Large 4 tep8 1M, rank 0 last tile at 1019904 keys: the 151 rows 3945..4095 " + "of rank 0's share 0..4095 of the last chunk, against cached + 4096 keys; " + "floor(2**28 / 1019904) = 263-row tiles, 15 of them and this tail, its last query " + "block partial", + spec=dict(**_INDEXER, seqs=_TEP8_RANK0_SHARE_1M, rows=(3945, 4096), share=(0, 4096)), + ), + Cell( + why="Mistral Large 4 tep8 1M, rank 0 first tile at 1019904 keys: rows 0..262 of the " + "last chunk, an odd 263-row tile, windows of 1015809..1016071 keys", + spec=dict(**_INDEXER, seqs=_TEP8_RANK0_SHARE_1M, rows=(0, 263), share=(0, 4096)), + ), + Cell( + why="Mistral Large 4 tep8 1M, rank 7 first tile at 524288 keys: rows 28672..29183 of a " + "chunk behind a 491520-token prefix, the first 512-row tile of rank 7's share, " + "windows of 520193..520704 keys", + spec=dict( + **_INDEXER, seqs=[(491520, 32768)], rows=(28672, 29184), share=(28672, 32768) + ), + ), + Cell( + why="Mistral Large 4 tep8 1M, rank 0 last tile at 495616 keys: the 309 rows 3787..4095 " + "of rank 0's share of a chunk behind a 491520-token prefix, against cached + 4096 " + "keys; floor(2**28 / 495616) = 541-row tiles, 7 of them and this tail", + spec=dict(**_INDEXER, seqs=[(491520, 4096)], rows=(3787, 4096), share=(0, 4096)), + ), + Cell( + why="Mistral Large 4 tep8 1M, rank 0 first tile at 495616 keys: rows 0..540 of that " + "chunk, an odd 541-row tile, windows of 491521..492061 keys", + spec=dict(**_INDEXER, seqs=[(491520, 4096)], rows=(0, 541), share=(0, 4096)), + ), + Cell( + why="Mistral Large 4 tep8 1M, rank 7 first tile at 262144 keys: rows 28672..29695 of a " + "chunk behind a 229376-token prefix, the first 1024-row tile of rank 7's share, " + "windows of 258049..259072 keys", + spec=dict( + **_INDEXER, seqs=[(229376, 32768)], rows=(28672, 29696), share=(28672, 32768) + ), + ), + Cell( + why="Mistral Large 4 tep8 1M, rank 7 first tile at 131072 keys: rows 28672..30719 of a " + "chunk behind a 98304-token prefix, the first 2048-row tile of rank 7's share, " + "windows of 126977..129024 keys", + spec=dict(**_INDEXER, seqs=[(98304, 32768)], rows=(28672, 30720), share=(28672, 32768)), + ), + Cell( + why="Mistral Large 4 tep8 1M, rank 7 whole share at 32768 keys: rows 28672..32767 of a " + "prompt's 32768-token first chunk in one call (floor(2**28 / 32768) = 8192 rows " + "exceed the 4096-row share), windows of 28673..32768 keys", + spec=dict(**_INDEXER, seqs=[(0, 32768)], rows=(28672, 32768), share=(28672, 32768)), + ), + Cell( + why="Mistral Large 4 tep8 1M, rank 0 whole share at 4096 keys: rows 0..4095 of a " + "prompt's 32768-token first chunk, against the share's 4096 keys in one call, " + "causal windows of 1..4096 keys", + spec=dict(**_INDEXER, seqs=[(0, 4096)], rows=(0, 4096), share=(0, 4096)), + ), + Cell( + why="Mistral Large 4 tep8 1M, the envelope prompt's last chunk: the 1048559-token " + "prompt's 32751-token last chunk behind 1015808 cached, split 4093 + 7 x 4094; " + "rank 7's share 28657..32750 in floor(2**28 / 1048559) = 256-row tiles, this its " + "254-row tail 32497..32750, windows of 1048306..1048559 keys", + spec=dict( + **_INDEXER, seqs=[(1015808, 32751)], rows=(32497, 32751), share=(28657, 32751) + ), + ), + Cell( + why="Mistral Large 4 tep8 1M, an unsplit 8191-row tail chunk: 8191 new tokens behind " + "1040385 cached, below the 8192-row split threshold, so one rank scores every row " + "against 1048576 keys; 31 256-row tiles and this 255-row tail 7936..8190, windows " + "of 1048322..1048576 keys", + spec=dict(**_INDEXER, seqs=[(1040385, 8191)], rows=(7936, 8191), share=(0, 8191)), + ), + Cell( + why="Mistral Large 4 tep8 1M, one request's part of a packed chunk past 1M keys: two " + "5-token final chunks over 983040-token prefixes pack into one engine chunk of " + "1966090 keys, which the tep8 deployment scores per request, each part 5 rows against " + "its request's own 983045 keys, windows of 983041..983045", + spec=dict(**_INDEXER, seqs=[(983040, 5)]), + ), + ) + + note = """ + Metadata consumed: none. `cu_seq_len_k_start` / `cu_seq_len_k_end` are int32 + [T], one window per query row, in the row coordinates of `k`; packed requests + are expressed through them alone. `q` [T, H, D] e4m3, `k` [N, D] e4m3 and + `k_scale` [N] fp32 are contiguous (the op asserts it); `weights` [T, H] fp32. + The output is the op's own allocation, returned as a [T, N] view of rows + whose stride is `align(N + 256, 256)` fp32 (measured 33024 at N = 32768, + 9472 at N = 9216, 2304 at N = 2048, 512 at N = 137, 1048832 at N = 1048576, + 1024256 at N = 1024000, 66048 at N = 65537, 1032448 at N = 1032192, 2097408 + at N = 2097152, 2374144 at N = 2373713, 1966592 at N = 1966090, 1020160 at + N = 1019904, 495872 at N = 495616, 1048832 at N = 1048559, 983552 at + N = 983045, 4352 at N = 4096; every cell asserts it), so an untiled call at + T = N = 32768 allocates 4,328,521,728 bytes. + + Only `clean_logits=True` with `max_seqlen_k=0` is certified, which is what + Mistral Large 4 passes. `clean_logits=False` leaves out-of-window entries + unspecified, and `max_seqlen_k > 0` switches to a compressed layout where + column `j` of row `i` is key `cu_seq_len_k_start[i] + j`; `reference` + implements both, no cell drives them. + + At the NVFP4 tep4 and tp4 deployments' 1M envelope (max_seq_len 1048576, + chunked prefill in 32768-token steps) each indexer prefill chunk is scored + against its requests' whole KV, cached + new, so one request's N reaches + 1048576, in query tiles of floor(2**28 / N) rows cut from the chunk's first + row: 256 rows at 1048576 keys, 262 at 1024000, 260 at 1032192, 512 at + 524288, 1024 at 262144, 2048 at 131072, 4096 at 65536, 4095 at 65537 and + 8192 at 32768, each about 1 GiB of fp32 logits. Under the engine's query + split (a step of 8192 or more context query tokens; tep4 and tp4 alike, + attention DP off) rank r scores the chunk's rows [8192 r, 8192 (r + 1)) + against the keys up to its share's end (`_split_prefill_queries` and + `plan_mla_chunked_prefill_chunks` in + `_torch/attention/backends/sparse/dsa/indexer.py`); the cells hold rank 1's + and rank 3's first tiles and rank 0's 70-row tail, rank 2 (1040384 keys, + 258-row tiles) only through the uniformity that follows. The engine packs + consecutive context requests into one chunk while sum(q) x sum(kv) <= 2**28 + (`group_prefill_requests_by_logits_budget`, same file), so a packed chunk's + rows never exceed floor(2**28 / N) and the target scores it in one call; its + later requests' windows start past key 0, at the sum of the earlier + requests' KV. The cells hold two 1M prompts' 1-token final chunks (2 rows, + 2097152 keys, the second window starting at 2**20) and three final chunks + at the packing budget (113 rows, 2373713 keys, windows from 0, 1048050 and + 2096100). With block reuse off a packed chunk's keys are bounded by the + rank's KV pool, 2,373,856 tokens at the tp4 deployment's delivered boot, 143 + above the widest cell; with block reuse on, requests sharing a cached + prefix each count it, and a chunk can reach 16 x 1048576 = 2**24 keys at + the packing cap, outside the cells. The kernel reads each window bound and + key index as an unsigned 32-bit integer, scans each 4-row query block's keys + from its smallest window start, and forms every logit's offset as row x + stride in 64 bits (`get_mqa_logits_kv_span`, `get_logits_row` and + `get_logits_col` in DeepGEMM's `scheduler/sm100_mqa_logits.cuh`): N moves + which keys a block scans and how wide the rows are, not the arithmetic. Not + certified: N above 2373713 (tep4's and tep8's cells alone reach N = 1048576, + dep4's 1966090, tp4's 2373713). The short final chunks of two near-1M + prompts can share a chunk of about 2**21 keys (two 5-token tails over + 983040-token prefixes: 1,966,090, scored whole and split per request in + dep4's cells, below, and split per request by the tep8 deployment); tep4's and + tep8's cells score no packed chunk whole, tp4's two above do, at 2097152 + and 2373713 keys. + + The 1M envelope (Mistral Large 4 NVFP4 dep4: max_seq_len 1048576, chunked prefill + in 32768-token chunks). Each engine indexer chunk is scored over its + gathered keys, cached + new, up to n_k = 1048576 (a 1M prompt's 32nd chunk), + in query tiles of floor(budget / n_k) rows. The `budget` cells are 2**28 + tiles at 32768 (8192 rows), 65536 (4096), 98304 (2730), 131072 (2048), + 524288 (512) and 1048576 keys (256, the first and the last tile), and the + last 2**26 (64-row) and 2**24 (16-row) tiles at 1048576 keys. Every 2**26 + and 2**24 tile inside a 2**28 tile cell is bitwise its rows of it, so those + budgets' tiles are certified at the same widths. A call's only device + allocation is its output: 1,074,003,968 bytes at 256 x 1048576 and + 1,082,130,432 at 8192 x 32768 (measured peak); a caller that holds the + previous tile's logits while the next call allocates peaks at two tiles, + 2,148,007,936 bytes at 2**28 and 1048576 keys. + Packed chunks: the engine groups whole requests into one chunk while + sum(new) x sum(cached + new) <= 2**28. dep4's cells hold 16 requests' 16-token + tails over 65520 cached keys each in one 1048576-key block, two tails over + 762256 keys, and two 5-token tails over 983040 cached keys each, 1966090 + keys in one call, past max_seq_len. Split per request, each request's own + call over its 983045 keys from key 0 (a cell) is bitwise its rows and keys + of the packed call. A split must give each call a 16-byte-aligned + `k_scale`: a view of the packed scales at a key offset that is not a + multiple of 4 raises (below). So gather each request's keys into buffers of + their own, or copy the scales. + + The NVFP4 tep8 deployment at the same envelope tiles the same way, at eight + ranks: under the engine's query split (`_split_prefill_queries` in + `_torch/attention/backends/sparse/dsa/indexer.py`, a step of 8192 or more + context query tokens) rank r scores the chunk's rows [count r / 8, + count (r + 1) / 8) against the keys up to its share's end (cached + + 4096 (r + 1) for a 32768-token chunk), in floor(2**28 / N)-row tiles cut + from the share's first row, so a call holds at most 4096 rows of a split + step. The "Mistral Large 4 tep8 1M" cells hold each share's first and last tile + behind cached prefixes of 1015808 (ranks 0, 3 and 7: 263-row tiles at + 1019904 keys, 260 at 1032192, 256 at 1048576), 491520 (ranks 0 and 7: 541 + at 495616, 512 at 524288), 229376 and 98304 (rank 7), rank 7's and rank 0's + whole shares of a first chunk (4096 rows at 32768 and at 4096 keys), rank + 7's 254-row tail of the 1048559-token envelope prompt's 32751-token last + chunk (shares 4093 + 7 x 4094), the 255-row tail of an unsplit 8191-row + chunk at 1048576 keys, and one request's 5 rows against its 983045 keys, + the part of the packed 1966090-key chunk above that the tep8 deployment scores + per request. Every tile of each of those shares is bitwise its rows of a + call over up to four tiles of the share, and every rank's first and last + tile of the 1M prompt's last chunk (32768 and 32751 tokens) over its + share's keys is bitwise its rows and keys of the same rows scored against + all the chunk's keys, which hold -inf past the share's last key: the split + changes no score. + + Measured on sm_103, on the path the cells drive (the SM100 DeepGEMM kernel, + fp32 weights, no schedule metadata, no autotuning), each kept true by a test: + - repeated calls are bitwise equal, and each query tile is bitwise the same + rows of one untiled call, so a caller may tile the queries at any row count + (proved at the 1820-row tiles a 2**24-element logits budget cuts at 9216 keys + and the 512-row tiles it cuts at 32768; from 32768 to 1048576 keys, where + no untiled chunk fits a call, against the `budget` cells' 2**28 tiles as + above); at the 2**28 budget each tile is bitwise the same rows of a call + over the step's last four tiles, at every N of the "Mistral Large 4 1M" cells, + tiles that start inside that call's 4-row query blocks included (262 rows + at 1024000 keys, 4095 at 65537), and at 32768 keys that call is the whole + step; + - a window that ends past the last key is clamped to the keys passed and + answers as `reference` does; + - row-padded `weights` (stride(0) 64 at 32 heads) score bitwise like packed + ones. + Measured to raise: a `weights` row stride that is not a multiple of 16 + bytes, with `RuntimeError: Assertion error (.../jit_kernels/impls/runtime_utils.hpp:130): + num_gmem_outer_stride_bytes % 16u == 0u`; bf16 `weights`, with + `RuntimeError: Assertion error (.../csrc/apis/attention.hpp:144): + weights.scalar_type() != torch::kBFloat16 or logits_dtype == torch::kBFloat16`; + a `k_scale` whose data pointer is not 16-byte aligned (a view starting at + a key offset of 1, 2 or 3 mod 4), with `RuntimeError: Assertion error + (.../csrc/apis/../jit_kernels/impls/runtime_utils.hpp:129): + reinterpret_cast(t.data_ptr()) % 16u == 0u`. + `is_valid` refuses fp16 weights, a negative window start and an inverted + window: the op accepts each and answers wrongly (see `is_valid`). + + Cost on sm_103: without schedule metadata the scheduler gives each SM whole + 4-row query blocks and never splits a block's keys (read in + `scheduler/sm100_mqa_logits.cuh`), so a 256-row tile at 1048576 keys keeps + 64 SMs busy, each scanning its block's window to the end, and a + tile of fewer rows at about 1M keys takes as long as 256 rows. A 4-row + query block holding rows of two near-1M requests (a packed final chunk) is + scanned by one SM from its smallest window start to its largest window + end, about 2**21 keys, so such a call takes longer than a 1M tile. + + Hard limits (sm_103) + logits row offset: row index x row stride, formed in 64 bits (the scheduler's + `get_logits_row` times a uint64 stride, DeepGEMM `sm100_mqa_logits.cuh`); + the allocation's element count is torch's int64 + 2**31 and 2**32 bytes: T = N = 32768 in one call (a cell) puts + rows 16257 on past 2**31 bytes and 32514 on past 2**32 passed + 2**31 elements: crossed in row 65027 at N = 32768, outside the + envelope (its largest call spans 1,082,130,176 elements); + probed once at 65100 rows x 32768 keys (2,149,862,400 + elements), rows 63488..65099 against `reference` passed + at the 1M envelope (stride 1048832 at N = 1048576, 2**31 + elements inside row 2047) a 2**28-budget tile, whose rows + never exceed its keys, spans at most 2**28 + 511 x 16384 = + 276,807,680 elements, 1,107,230,720 bytes; the 256-row tiles + at 1048576 keys span 268,500,992 (1,074,003,968 bytes), the + 113-row packed chunk at 2373713 keys 268,278,272 + (1,073,113,088 bytes) not reached + the stride itself is a uint32 (`logits_stride`): 2**32 at + N >= 2**32 - 511 not reached + the 2**28 tile cells span at most 270,532,608 elements + (8192 x 33024 at 32768 keys; 268,500,992 at 256 x 1048576) no boundary + at the 1M envelope (stride 1048832 at N = 1048576, 2**31 + elements inside row 2047) a 2**28-budget tile, whose rows + never exceed its keys, spans at most 2**28 + 511 x 16384 = + 276,807,680 elements, 1,107,230,720 bytes; the 256-row tiles + at 1048576 keys span 268,500,992 (1,074,003,968 bytes) not reached + q T * H rows of D bytes, a 32-bit TMA row coordinate 2**31 rows + envelope max 32768 * 32 = 1,048,576 rows untiled, 16384 * 32 = + 524,288 in a 2**28-budget tile not reached, unprobed + k N rows of D bytes, a 32-bit TMA row coordinate (`get_kv_tma_offset`) + 2**31 rows (key indices, spans and columns are uint32 in the + kernel's scheduler); cells reach N = 2373713 (303,835,264 bytes + of keys, under 2**31 bytes until N = 2**24), dep4's cells + N = 1966090 not reached, unprobed + windows int32 key indices, < N <= 2373713 (cells reach N = 2373713, dep4's + N <= 1966090; the per-rank pool bounds a packed chunk at 2,373,856 + with block reuse off), read as unsigned 32-bit by the kernel; none + passes through fp32 (dep4's cells: 2**21 < 2**24 in any case), and + every index stays below 2**24 + 1 regardless + pool none: the op reads the keys its caller gathered, not the KV + pool, so no pool size reaches it + + Not certified: N above 2373713 (dep4's cells alone reach 1966090, see + above); at 65536 keys and more, a query tile above 2**28 elements (no + untiled chunk there is a cell); budgets other than 2**24, 2**26 and 2**28 + above 32768 keys; `k_scale` views (only buffers of their own and views at + key offsets that are multiples of 4 were driven). + """ + + def __call__( + self, + q: torch.Tensor, + kv: Tuple[torch.Tensor, torch.Tensor], + weights: torch.Tensor, + cu_seq_len_k_start: torch.Tensor, + cu_seq_len_k_end: torch.Tensor, + clean_logits: bool = True, + max_seqlen_k: int = 0, + schedule_meta: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + if schedule_meta is None: + return _deep_gemm_fp8_mqa_logits( + q, + kv, + weights, + cu_seq_len_k_start, + cu_seq_len_k_end, + clean_logits, + max_seqlen_k, + ) + # The legacy pybind is exactly this call without the schedule (DeepGEMM + # `csrc/apis/attention.hpp`): q unscaled, fp32 logits. + return _deep_gemm_fp8_fp4_mqa_logits( + (q, None), + kv, + weights, + cu_seq_len_k_start, + cu_seq_len_k_end, + clean_logits, + max_seqlen_k, + torch.float32, + schedule_meta, + ) + + def reference( + self, + q: torch.Tensor, + kv: Tuple[torch.Tensor, torch.Tensor], + weights: torch.Tensor, + cu_seq_len_k_start: torch.Tensor, + cu_seq_len_k_end: torch.Tensor, + clean_logits: bool = True, + max_seqlen_k: int = 0, + schedule_meta: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + """The scores in fp64, rounded once to fp32. + + Outside each row's window the entry is -inf when `clean_logits`, else + NaN, which the test's gate reads as "unspecified". `schedule_meta` is + not read: it moves which SM computes a logit, not its value. + """ + k, k_scale = kv + qd, kd, wd = q.float().double(), k.float().double(), weights.double() + n = k.shape[0] + # One head at a time keeps the transient at [T, N] rather than [T, H, N]. + full = torch.zeros(q.shape[0], n, dtype=torch.float64, device=q.device) + for h in range(q.shape[1]): + full += (qd[:, h] @ kd.t()).relu() * wd[:, h : h + 1] + full *= k_scale.double().unsqueeze(0) + ks = cu_seq_len_k_start.long().unsqueeze(1) + ke = cu_seq_len_k_end.long().unsqueeze(1) + fill = float("-inf") if clean_logits else float("nan") + if max_seqlen_k == 0: + cols = torch.arange(n, device=q.device).unsqueeze(0) + out = full.masked_fill((cols < ks) | (cols >= ke), fill) + else: + cols = torch.arange(max_seqlen_k, device=q.device).unsqueeze(0) + src = (ks + cols).clamp(max=n - 1) + out = torch.gather(full, 1, src).masked_fill(cols >= ke - ks, fill) + return out.float() + + def is_valid( + self, + q: torch.Tensor, + kv: Tuple[torch.Tensor, torch.Tensor], + weights: torch.Tensor, + cu_seq_len_k_start: torch.Tensor, + cu_seq_len_k_end: torch.Tensor, + clean_logits: bool = True, + max_seqlen_k: int = 0, + schedule_meta: Optional[torch.Tensor] = None, + ) -> None: + if schedule_meta is not None: + # The kernel trusts the schedule: one built for other windows, another key + # count or another SM count scores the wrong spans and answers wrongly. Its + # live words (per-SM starts and split counts, then each query block's span) + # must be the ones this call's own schedule holds; the pad between them is + # never written. + fresh = mqa_logits_schedule( + cu_seq_len_k_start, cu_seq_len_k_end, kv[0].shape[0], q.shape[1] + ) + sms = _deep_gemm_num_sms() + span_off = (3 * sms + 1) // 2 * 2 + blocks = -(-q.shape[0] // (128 // q.shape[1])) + live = [slice(0, 3 * sms), slice(span_off, span_off + 2 * blocks)] + assert schedule_meta.dtype == torch.int32 and schedule_meta.numel() >= fresh.numel(), ( + f"schedule_meta {schedule_meta.dtype} [{schedule_meta.numel()}] is not this " + f"call's int32 [{fresh.numel()}] schedule" + ) + assert all(torch.equal(schedule_meta[s], fresh[s]) for s in live), ( + "schedule_meta was not built for this call's windows, keys and SM count" + ) + # fp16 weights are accepted and select a second kernel, which answers + # at half precision: measured relative RMS error 4.8e-4 against the + # reference, where the fp32 path measures 6.8e-8. + assert weights.dtype == torch.float32, ( + f"weights must be fp32; got {weights.dtype}, which selects a kernel that answers " + f"at half precision" + ) + if cu_seq_len_k_start.numel(): + # A negative start is not read as 0: with every start at -1, every + # logit of the call came back -inf. + assert int(cu_seq_len_k_start.min()) >= 0, "a window starts before key 0" + # An inverted window is not read as empty: its row came back with + # finite logits where an empty window gives -inf. + assert bool((cu_seq_len_k_start <= cu_seq_len_k_end).all()), ( + "a window ends before it starts" + ) + + +fp8_mqa_logits = Fp8MqaLogits() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/indexer_k_cache_gather_op.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/indexer_k_cache_gather_op.py new file mode 100644 index 000000000000..c827385c44a6 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/indexer_k_cache_gather_op.py @@ -0,0 +1,1271 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Gather one prefill chunk's DSA indexer keys back out of the indexer K cache.""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: Mistral Large 4's indexer cache: 128 fp8 bytes plus one fp32 scale per token, 32 +#: tokens per page. A page stores all 32 tokens' data first, then their scales. In its cells +#: `layout` says how the view is made: "v2" and "v1" rebuild the per-layer +#: views the two DSA cache managers hand out (see `note`), "manager_v2" and +#: "manager_v1" take the view and the block ids from a real manager. +#: `pool_layers` is the number of indexer rows in the pool, `pool_row` the row +#: gathered from, `num_blocks` the pool's slot count; `tail` puts the step's +#: pages in the pool's last blocks, and `cycle` lets the step's pages outnumber +#: the pool (they are drawn as consecutive fresh permutations of it, so pages +#: repeat but no stretch of the mapping repeats another). `decode_rows` appends +#: that many decode sequences' new-token slots to the mapping after the +#: prompts', as a mixed step does, at drawn positions; `decode_pos` at the +#: positions it lists. `sweep=True` reads every key count of the window as its +#: prefix and its suffix; `sweep="windows"` reads, besides the window, each +#: q-split rank's share of every request of 32768 keys or more, and windows of +#: 2^k - 1, 2^k and 2^k + 1 keys (k = 15..21) at the mapping's start and end and +#: from keys 2^20 and 2^21 (each +-1); `qsplit_ranks` is the number of ranks the +#: split cuts a chunk over (4 unless given). `max_seq_len` sizes a manager +#: (default 9216); `large` marks a cell pre-merge deselects. `cached` gives each +#: prompt a cached prefix ahead of its `seq_lens` new keys: the mapping then holds +#: the prompt's cached and new keys (positions 0..cached + new - 1), as the full-KV +#: mapping of a chunked-prefill step does, and a decode row owns only the page +#: its new key sits on. `sweep_counts` reads each listed key count of the window +#: as its prefix and its suffix. +_ML4 = dict(head_dim=128, tokens_per_block=32) + +#: 16 prompts, the most one step schedules (max_batch_size 16), filling the FP8 +#: target's 9216-token budget; the first is the shortest prompt the indexer +#: runs on (one key past index_topk 2048). +_ML4_16_PROMPTS = (2049, 1500, 1024, 977, 700, 600, 512, 400, 333, 300, 250, 200, 150, 100, 64, 57) + +#: The largest pools a rank can hold, deliberate overestimates: about 24.7 GiB +#: is left per GB300 after the 252.84 GiB of weights, and the managers measure +#: 47280 bytes per token with 44 indexer rows and 50976 with 72, so at most +#: about 17.5K and 16.3K blocks of 32 tokens. +_ML4_FAR_44 = dict(_ML4, pool_layers=44, num_blocks=18432, tail=True) +_ML4_FAR_72 = dict(_ML4, pool_layers=72, num_blocks=16384, tail=True) + +#: The NVFP4 target serves max_seq_len = max_num_tokens = 32768 with chunked +#: prefill off: 16 prompts packed to 32768 keys, the first again one key past +#: index_topk, the others across page edges (4097, 4096, 1023, 255, 31). +_ML4_32K_16 = ( + 2049, + 9000, + 4097, + 4096, + 3000, + 2500, + 2048, + 2047, + 1024, + 1023, + 777, + 513, + 255, + 64, + 31, + 244, +) +#: Its manager cells: a quota of two full prompts, at its sequence and token limits. +_ML4_32K_MANAGER = dict(_ML4, pool_layers=44, max_tokens=65536, max_seq_len=32768) +#: The largest pool an NVFP4 tep4 rank can hold, a deliberate overestimate: the +#: GB300's 276.62 GiB less 143.24 GiB of weights is 133.38 GiB, 94661 blocks +#: at the manager's 47280 bytes per token (44 indexer rows), and V2 hands out +#: about 5.3% more slots than its quota; 109568 slots clears both. Its 44-row +#: indexer pool is 20.4 GB, so these cells are `large`. +_ML4_32K_FAR = dict(_ML4, pool_layers=44, num_blocks=109568, tail=True, large=True) +#: The largest pool an NVFP4 tep8, tp8 or dep8 rank can hold, a deliberate +#: overestimate: a tep8 or tp8 rank holds 73.61 GiB of weights and a dep8 rank +#: 89.29 GiB (its 16 routed experts and the replicated rest; computed from the +#: shard headers), which leave 203.01 and 187.33 GiB, 144075 and 132949 slots at +#: the manager's 1512960 bytes per 32-token slot. V2 backs every slot it hands +#: out, the ~5% past its quota included, with device memory, so 147456 slots +#: clears both readings. Its 44-row indexer pool is 27.4 GB, so this cell is +#: `large`. +_ML4_32K_8GPU_FAR = dict(_ML4, pool_layers=44, num_blocks=147456, tail=True, large=True) + +#: The NVFP4 target's cells at 64-token pages: 8448-byte pages, data at `(pos % 64) * 128`, +#: scales from byte 8192. The same prompts and windows; each pool holds its 32-token +#: counterpart's tokens in half the blocks (1024, 4096, and 54784 for the far pool, the +#: same 20.4 GB). +_ML4_64 = dict(head_dim=128, tokens_per_block=64) +_ML4_32K_P64_MANAGER = dict(_ML4_64, pool_layers=44, max_tokens=65536, max_seq_len=32768) +_ML4_32K_P64_FAR = dict(_ML4_64, pool_layers=44, num_blocks=54784, tail=True, large=True) + +#: The NVFP4 tep4 deployment at its own context ("1M" cells): max_seq_len 1048576 with +#: chunked prefill in 32768-token steps, on V2 views of the 44-row indexer pool. +_ML4_1M_V2 = dict(_ML4, layout="v2", pool_layers=44) +#: The most keys one step's full-KV mapping holds: every token of the delivered +#: tep4 boot's pool (2373024 tokens, 74157 pages), as two 1M prompts' KV and a +#: third's. V2 hands out 78051 slots for that quota. +_ML4_1M_POOL_LENS = (1048576, 1048576, 275872) +#: The NVFP4 tp4 deployment at the same context ("1M tp4" cells): its delivered boot left a +#: rank 2373856 tokens (74183 pages; its 1M boot, 2352000), so its packed corner is two +#: 1M prompts' KV and a third's 276704 keys, every page of a 74183-block pool. +_ML4_1M_TP4_POOL_LENS = (1048576, 1048576, 276704) + +#: The NVFP4 dep4 deployment at its own context (the "dep4 1M" cells): max_seq_len +#: 1048576 with chunked prefill in 32768-token steps, on V2 views of the 44-row +#: indexer pool. 36864 blocks hold a 1M prompt's 32768 pages and a mixed step's +#: decode pages (6.8 GB of indexer pool). +_DEP4_1M_V2 = dict(_ML4, layout="v2", pool_layers=44, num_blocks=36864) +#: Its packed corner, the engine-config receipt's case (d): two 5-token last +#: chunks over 983040-token prefixes on one rank pack into one indexer chunk of +#: 1966090 keys; the pool holds both requests' 61442 pages (11.4 GB; a cell peaks +#: at 24.6 GiB). +_DEP4_1M_CASE_D = dict( + _ML4, layout="v2", pool_layers=44, num_blocks=61442, seq_lens=[5, 5], cached=[983040, 983040] +) + +#: The NVFP4 tep8 deployment at its own context ("1M tep8" cells), on the "1M" cells' views. +#: The most keys one step's full-KV mapping holds on a tep8 rank: every token of +#: the delivered tep8 boot's pool (3680512 tokens, 115016 pages, at max_seq_len +#: 32768), as three 1M prompts' KV and a fourth's. +_ML4_1M_TEP8_POOL_LENS = (1048576, 1048576, 1048576, 534784) + + +class IndexerKCacheGatherOp(OpWrapper): + """Read `num_tokens` cached indexer keys into two fresh tensors. + + Output row `i` is the `head_dim` bytes at logical flat byte offset + `slot_mapping_fp8[k_token_start + i]` of `k_cache`, and its scale the four + bytes at `slot_mapping_scale[k_token_start + i]`. "Logical" means the + offset indexes `k_cache` as if it were contiguous: the kernel unravels it + over `k_cache`'s shape and applies its real strides, so a layer view of an + interleaved pool works, strided or not. + + Returns `(data [num_tokens, head_dim] float8_e4m3fn, scale [num_tokens, 1] + float32)`. `k_cache` is not modified. + + Fusion boundary: a byte gather and nothing else. The caller owns the slot + mappings, the window, and the scoring that reads the keys. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # The FP8 Mistral Large 4 target's prefill (max_num_tokens 9216, max_batch_size + # 16, no chunked prefill, no reuse) gathers a window of one step's prompt + # keys from one of its 44 indexer layers. Its cells take that view from both + # cache managers, pack the token budget, start the window mid-mapping, end it + # at the mapping's end, drive every key count, and address the far end of + # the largest pool in both layouts and at both row counts. + # + # The NVFP4 Mistral Large 4 target (the "32768" cells) runs the same call at + # max_seq_len = max_num_tokens = 32768, so every one of those cells has a + # 32768 analogue, and its weights leave a pool of up to ~95K blocks rather + # than ~17K. The "32768, 64-token pages" cells repeat the 32768 ones over + # 64-token pages: the same prompts and windows, pools of the same bytes in + # half the blocks. The tep8, tp8 and dep8 deployments (the "tep8/tp8/dep8" + # cell; the indexer is whole on every rank, replicated at tep8 and tp8, data + # parallel at dep8) make the same call, a rank holding a smaller share of + # those weights, on a V2 pool of up to ~144K slots. + # + # The NVFP4 tep4 deployment at max_seq_len 1048576 (the "1M" cells) runs it with + # chunked prefill: once per indexer prefill chunk, over the chunk's whole KV + # (cached + new) in the step's full-KV mappings. Its cells gather a 1M + # prompt's KV, start a window at key 2^20 behind one, take each query-split + # rank's share, pack 16 chunk tails, use a real V2 manager built at 1048576, + # address the far pool, and gather as many keys as the delivered pool holds. + # + # The NVFP4 tp4 deployment (the "1M tp4" cells) makes the same call at the same + # context. Its cells add real V2 managers at max_seq_len 1048576 whose quota + # is exactly their prompts' pages (one 1M prompt; a 1M prompt behind a + # 32768-key one, gathered from key 32768) and every token of its own boot + # pool in one window. + # + # The NVFP4 dep4 deployment at max_seq_len 1048576 (the "dep4 1M" cells) runs it + # with chunked prefill: once per indexer prefill chunk, over the chunk's whole + # KV (cached + new) in the step's full-KV mapping, each rank on its own + # requests. Its cells take a 1M chunk's view from a real V2 manager built at + # 1048576 (and a 524288 share in pre-merge), start a request's window + # mid-mapping, put decode slots behind a 1M window, pack 16 chunk tails into + # one window, read the 1M window's edge key counts, address the far pool, and + # gather the packed corner past max_seq_len both whole and split per request. + # + # The NVFP4 tep8 deployment at max_seq_len 1048576 (the "1M tep8" cells) makes the + # same call, but cuts a chunk the engine packs past max_seq_len keys at its + # request boundaries, so one call gathers at most one request's keys, from + # that request's offset in the full-KV mappings. Its cells take q-split ranks + # 0 and 3's shares at eight ranks, the second request of such a packed chunk, + # a 1M prompt at the far end of a tep8 rank's pool bound, and every rank's + # share of each request in a mapping as long as the delivered tep8 boot's pool. + CELLS: tuple[Cell, ...] = ( + Cell( + why="engine V2 manager (the default), 44 rows: 3 prompts = 9216 keys, layer 71", + spec=dict( + _ML4, + layout="manager_v2", + pool_layers=44, + layer=71, + max_tokens=18432, + seq_lens=[5000, 3000, 1216], + k_token_start=0, + num_tokens=9216, + ), + ), + Cell( + why="engine V1 manager, 44 rows: the second prompt's 3000 keys, layer 0", + spec=dict( + _ML4, + layout="manager_v1", + pool_layers=44, + layer=0, + max_tokens=18432, + seq_lens=[5000, 3000, 1216], + k_token_start=5000, + num_tokens=3000, + ), + ), + Cell( + why="V2 view, 44 rows: 16 prompts packed to 9216 keys, one window over all", + spec=dict( + _ML4, + layout="v2", + pool_layers=44, + pool_row=20, + num_blocks=2048, + seq_lens=list(_ML4_16_PROMPTS), + k_token_start=0, + num_tokens=9216, + ), + ), + Cell( + why="V1 view, 44 rows: 16 prompts packed to 9216 keys, one window over all", + spec=dict( + _ML4, + layout="v1", + pool_layers=44, + pool_row=20, + num_blocks=2048, + seq_lens=list(_ML4_16_PROMPTS), + k_token_start=0, + num_tokens=9216, + ), + ), + Cell( + why="mixed step: the third of 4 prompts (2049 keys), 12 decode rows after it", + spec=dict( + _ML4, + layout="v2", + pool_layers=44, + pool_row=43, + num_blocks=2048, + seq_lens=[3000, 2500, 2049, 1655], + decode_rows=12, + k_token_start=5500, + num_tokens=2049, + ), + ), + Cell( + why="a 1-key window ending exactly at the mapping's end, V1 row 0", + spec=dict( + _ML4, + layout="v1", + pool_layers=44, + pool_row=0, + num_blocks=2048, + seq_lens=[9215, 1], + k_token_start=9215, + num_tokens=1, + ), + ), + Cell( + why="every key count 1..9216 as prefix and suffix of one window (18432 calls, ~4 s)", + spec=dict( + _ML4, + layout="v2", + pool_layers=44, + pool_row=7, + num_blocks=2048, + seq_lens=[9216], + k_token_start=0, + num_tokens=9216, + sweep=True, + ), + ), + Cell( + why="far pool V1, 44 rows x 18432 blocks (3.4 GB): last row, bytes past 2^31", + spec=dict( + _ML4_FAR_44, + layout="v1", + pool_row=43, + seq_lens=[9216], + k_token_start=0, + num_tokens=9216, + ), + ), + Cell( + why="far pool V2, 44 rows x 18432 blocks (3.4 GB): last row, slots past 2^31", + spec=dict( + _ML4_FAR_44, + layout="v2", + pool_row=43, + seq_lens=[9216], + k_token_start=0, + num_tokens=9216, + ), + ), + Cell( + why="far pool V1, 72 rows x 16384 blocks (5.0 GB): last row, bytes past 2^32", + spec=dict( + _ML4_FAR_72, + layout="v1", + pool_row=71, + seq_lens=[9216], + k_token_start=0, + num_tokens=9216, + ), + ), + Cell( + why="far pool V2, 72 rows x 16384 blocks (5.0 GB): last row, slots past 2^32", + spec=dict( + _ML4_FAR_72, + layout="v2", + pool_row=71, + seq_lens=[9216], + k_token_start=0, + num_tokens=9216, + ), + ), + Cell( + why="32768: engine V2 manager, 44 rows: one 32768-key prompt, the max_num_tokens " + "step's whole window, layer 71", + spec=dict( + _ML4_32K_MANAGER, + layout="manager_v2", + layer=71, + seq_lens=[32768], + k_token_start=0, + num_tokens=32768, + ), + ), + Cell( + why="32768: engine V1 manager, 44 rows: the second of 3 prompts (10000 keys from " + "key 20000), layer 0", + spec=dict( + _ML4_32K_MANAGER, + layout="manager_v1", + layer=0, + seq_lens=[20000, 10000, 2768], + k_token_start=20000, + num_tokens=10000, + ), + ), + Cell( + why="32768: V2 view, 44 rows: 16 prompts (max_batch_size) packed to 32768 keys, one " + "window over all", + spec=dict( + _ML4, + layout="v2", + pool_layers=44, + pool_row=20, + num_blocks=2048, + seq_lens=list(_ML4_32K_16), + k_token_start=0, + num_tokens=32768, + ), + ), + Cell( + why="32768: V1 view, 44 rows: 16 prompts (max_batch_size) packed to 32768 keys, one " + "window over all", + spec=dict( + _ML4, + layout="v1", + pool_layers=44, + pool_row=20, + num_blocks=2048, + seq_lens=list(_ML4_32K_16), + k_token_start=0, + num_tokens=32768, + ), + ), + Cell( + why="32768: mixed step, the call site's window: 4 prompts' 32756 keys, then 12 decode " + "rows' new-token slots at positions up to 32767", + spec=dict( + _ML4, + layout="v2", + pool_layers=44, + pool_row=43, + num_blocks=8192, + seq_lens=[12000, 10000, 8000, 2756], + decode_pos=[32767, 1, 31, 32, 33, 1023, 2048, 9216, 16384, 20000, 30000, 32766], + k_token_start=0, + num_tokens=32756, + ), + ), + Cell( + why="32768: a 1-key window ending exactly at the mapping's end, V1 row 0", + spec=dict( + _ML4, + layout="v1", + pool_layers=44, + pool_row=0, + num_blocks=2048, + seq_lens=[32767, 1], + k_token_start=32767, + num_tokens=1, + ), + ), + Cell( + why="32768: every key count 1..32768 as prefix and suffix of one window (65536 " + "calls, ~5 s)", + spec=dict( + _ML4, + layout="v2", + pool_layers=44, + pool_row=7, + num_blocks=2048, + seq_lens=[32768], + k_token_start=0, + num_tokens=32768, + sweep=True, + ), + ), + Cell( + why="32768: far pool V2, 44 rows x 109568 slots (20.4 GB, large): last row, past 2^34", + spec=dict( + _ML4_32K_FAR, + layout="v2", + pool_row=43, + seq_lens=[32768], + k_token_start=0, + num_tokens=32768, + ), + ), + Cell( + why="32768: far pool V1, 44 rows x 109568 slots (20.4 GB, large): last row, past 2^34", + spec=dict( + _ML4_32K_FAR, + layout="v1", + pool_row=43, + seq_lens=[32768], + k_token_start=0, + num_tokens=32768, + ), + ), + Cell( + why="32768, 64-token pages: engine V2 manager, 44 rows: one 32768-key prompt, the " + "max_num_tokens step's whole window, layer 71", + spec=dict( + _ML4_32K_P64_MANAGER, + layout="manager_v2", + layer=71, + seq_lens=[32768], + k_token_start=0, + num_tokens=32768, + ), + ), + Cell( + why="32768, 64-token pages: engine V1 manager, 44 rows: the second of 3 prompts " + "(10000 keys from key 20000), layer 0", + spec=dict( + _ML4_32K_P64_MANAGER, + layout="manager_v1", + layer=0, + seq_lens=[20000, 10000, 2768], + k_token_start=20000, + num_tokens=10000, + ), + ), + Cell( + why="32768, 64-token pages: V2 view, 44 rows: 16 prompts (max_batch_size) packed to " + "32768 keys, one window over all", + spec=dict( + _ML4_64, + layout="v2", + pool_layers=44, + pool_row=20, + num_blocks=1024, + seq_lens=list(_ML4_32K_16), + k_token_start=0, + num_tokens=32768, + ), + ), + Cell( + why="32768, 64-token pages: V1 view, 44 rows: 16 prompts (max_batch_size) packed to " + "32768 keys, one window over all", + spec=dict( + _ML4_64, + layout="v1", + pool_layers=44, + pool_row=20, + num_blocks=1024, + seq_lens=list(_ML4_32K_16), + k_token_start=0, + num_tokens=32768, + ), + ), + Cell( + why="32768, 64-token pages: mixed step, the call site's window: 4 prompts' 32756 " + "keys, then 12 decode rows' new-token slots at positions up to 32767", + spec=dict( + _ML4_64, + layout="v2", + pool_layers=44, + pool_row=43, + num_blocks=4096, + seq_lens=[12000, 10000, 8000, 2756], + decode_pos=[32767, 1, 31, 32, 33, 1023, 2048, 9216, 16384, 20000, 30000, 32766], + k_token_start=0, + num_tokens=32756, + ), + ), + Cell( + why="32768, 64-token pages: a 1-key window ending exactly at the mapping's end, V1 " + "row 0", + spec=dict( + _ML4_64, + layout="v1", + pool_layers=44, + pool_row=0, + num_blocks=1024, + seq_lens=[32767, 1], + k_token_start=32767, + num_tokens=1, + ), + ), + Cell( + why="32768, 64-token pages: every key count 1..32768 as prefix and suffix of one " + "window (65536 calls, ~5 s)", + spec=dict( + _ML4_64, + layout="v2", + pool_layers=44, + pool_row=7, + num_blocks=1024, + seq_lens=[32768], + k_token_start=0, + num_tokens=32768, + sweep=True, + ), + ), + Cell( + why="32768, 64-token pages: far pool V2, 44 rows x 54784 slots (20.4 GB, large): last " + "row, past 2^34 bytes, view pages below 2410496 (0.11% of 2^31)", + spec=dict( + _ML4_32K_P64_FAR, + layout="v2", + pool_row=43, + seq_lens=[32768], + k_token_start=0, + num_tokens=32768, + ), + ), + Cell( + why="32768, 64-token pages: far pool V1, 44 rows x 54784 slots (20.4 GB, large): last " + "row, past 2^34 bytes", + spec=dict( + _ML4_32K_P64_FAR, + layout="v1", + pool_row=43, + seq_lens=[32768], + k_token_start=0, + num_tokens=32768, + ), + ), + Cell( + why="tep8/tp8/dep8: a rank's largest V2 pool, 44 rows x 147456 slots (27.4 GB, 54.2 " + "GiB peak, large): the 32768-key window from the last row's top 1024 blocks, to the " + "view's last byte", + spec=dict( + _ML4_32K_8GPU_FAR, + layout="v2", + pool_row=43, + seq_lens=[32768], + k_token_start=0, + num_tokens=32768, + ), + ), + Cell( + why="1M: one chunk's whole KV, a 1048576-key prompt's last chunk (1015808 cached + " + "32768 new), V2 row 43 of 32800 blocks (6.1 GB)", + spec=dict( + _ML4_1M_V2, + pool_row=43, + num_blocks=32800, + seq_lens=[1048576], + k_token_start=0, + num_tokens=1048576, + ), + ), + Cell( + why="1M: the second of two 1M prompts' chunks, its whole KV from key 1048576 = 2^20, " + "V2 row 20 of 65568 blocks (12.2 GB; 26.2 GiB peak, large)", + spec=dict( + _ML4_1M_V2, + pool_row=20, + num_blocks=65568, + seq_lens=[1048576, 1048576], + k_token_start=1048576, + num_tokens=1048576, + large=True, + ), + ), + Cell( + why="1M: q-split rank 0's share of a 1M prompt's last chunk: 1015808 cached + 8192 = " + "1024000 keys, V2 row 7", + spec=dict( + _ML4_1M_V2, + pool_row=7, + num_blocks=32800, + seq_lens=[1048576], + k_token_start=0, + num_tokens=1024000, + ), + ), + Cell( + why="1M: a 2049-key prompt behind a 1M one, from key 1048576, V2 row 0 (its 32833 " + "pages cycled over 32800 blocks)", + spec=dict( + _ML4_1M_V2, + pool_row=0, + num_blocks=32800, + cycle=True, + seq_lens=[1048576, 2049], + k_token_start=1048576, + num_tokens=2049, + ), + ), + Cell( + why="1M: 16 packed chunk tails' whole KV, 16 x 61456 = 983296 keys in one window, " + "V2 row 31", + spec=dict( + _ML4_1M_V2, + pool_row=31, + num_blocks=30768, + seq_lens=[61456] * 16, + k_token_start=0, + num_tokens=983296, + ), + ), + Cell( + why="1M: engine V2 manager at max_seq_len 1048576 (quota 2097152 tokens; 112.8 GiB on " + "device, large): one 1M prompt's whole KV, layer 71", + spec=dict( + _ML4, + layout="manager_v2", + pool_layers=44, + layer=71, + max_tokens=2097152, + max_seq_len=1048576, + seq_lens=[1048576], + k_token_start=0, + num_tokens=1048576, + large=True, + ), + ), + Cell( + why="1M: far pool V2, 44 rows x 109568 slots (20.4 GB; 42.4 GiB peak, large): a 1M " + "prompt's whole KV in the last 32768 blocks, past 2^34", + spec=dict( + _ML4_32K_FAR, + layout="v2", + pool_row=43, + seq_lens=[1048576], + k_token_start=0, + num_tokens=1048576, + ), + ), + Cell( + why="1M packed corner: as many keys as the boot pool holds (2373024) in one window, " + "V2 row 43 of 78051 slots (14.5 GB; 30.7 GiB peak, large)", + spec=dict( + _ML4_1M_V2, + pool_row=43, + num_blocks=78051, + tail=True, + seq_lens=list(_ML4_1M_POOL_LENS), + k_token_start=0, + num_tokens=2373024, + large=True, + ), + ), + Cell( + why="1M packed corner on 4096 blocks, pages cycled: 2373024 keys in one window", + spec=dict( + _ML4_1M_V2, + pool_row=43, + num_blocks=4096, + cycle=True, + seq_lens=list(_ML4_1M_POOL_LENS), + k_token_start=0, + num_tokens=2373024, + ), + ), + Cell( + why="1M windows: q-split shares and 2^k - 1, 2^k, 2^k + 1 keys to 2^21 + 1 over a " + "2373024-key mapping, pages cycled", + spec=dict( + _ML4_1M_V2, + pool_row=20, + num_blocks=4096, + cycle=True, + seq_lens=list(_ML4_1M_POOL_LENS), + k_token_start=0, + num_tokens=2373024, + sweep="windows", + ), + ), + Cell( + why="1M tp4: engine V2 manager at a 1048576-token quota (32768 blocks, 34487 slots; " + "56.7 GiB device peak, large): one 1M prompt's whole KV, layer 71", + spec=dict( + _ML4, + layout="manager_v2", + pool_layers=44, + layer=71, + max_tokens=1048576, + max_seq_len=1048576, + seq_lens=[1048576], + k_token_start=0, + num_tokens=1048576, + large=True, + ), + ), + Cell( + why="1M tp4: engine V2 manager at a 1081344-token quota (33792 blocks, 35566 slots; " + "58.4 GiB device peak, large): a 1M prompt behind a 32768-key one, its whole KV from " + "key 32768, layer 0", + spec=dict( + _ML4, + layout="manager_v2", + pool_layers=44, + layer=0, + max_tokens=1081344, + max_seq_len=1048576, + seq_lens=[32768, 1048576], + k_token_start=32768, + num_tokens=1048576, + large=True, + ), + ), + Cell( + why="1M tp4 packed corner: every token of the tp4 boot pool (2373856) in one window, " + "V2 row 43 of 74183 blocks, each page once (13.8 GB; 29.4 GiB peak, large)", + spec=dict( + _ML4_1M_V2, + pool_row=43, + num_blocks=74183, + seq_lens=list(_ML4_1M_TP4_POOL_LENS), + k_token_start=0, + num_tokens=2373856, + large=True, + ), + ), + Cell( + why="1M tp4 packed corner on 4096 blocks, pages cycled: 2373856 keys in one window", + spec=dict( + _ML4_1M_V2, + pool_row=43, + num_blocks=4096, + cycle=True, + seq_lens=list(_ML4_1M_TP4_POOL_LENS), + k_token_start=0, + num_tokens=2373856, + ), + ), + Cell( + why="dep4 1M manager, a 1M prompt's last chunk: 1015808 cached + 32768 new, layer 71, " + "engine V2 manager at max_seq_len 1048576 (quota 1050624 tokens; 56.8 GiB peak, large)", + spec=dict( + _ML4, + layout="manager_v2", + pool_layers=44, + layer=71, + max_tokens=1050624, + max_seq_len=1048576, + seq_lens=[32768], + cached=[1015808], + k_token_start=0, + num_tokens=1048576, + large=True, + ), + ), + Cell( + why="dep4 1M manager, a later request's whole KV: 1000000 cached + 30720 new from key " + "502048, layer 0, engine V2 manager at max_seq_len 1048576 (quota 1572864 tokens; " + "85.1 GiB peak, large)", + spec=dict( + _ML4, + layout="manager_v2", + pool_layers=44, + layer=0, + max_tokens=1572864, + max_seq_len=1048576, + seq_lens=[2048, 30720], + cached=[500000, 1000000], + k_token_start=502048, + num_tokens=1030720, + large=True, + ), + ), + Cell( + why="dep4 1M manager share at max_seq_len 524288 (quota 526336 tokens; 28.9 GiB peak): " + "a 524288-key prompt's last chunk, 491520 cached + 32768 new, layer 71", + spec=dict( + _ML4, + layout="manager_v2", + pool_layers=44, + layer=71, + max_tokens=526336, + max_seq_len=524288, + seq_lens=[32768], + cached=[491520], + k_token_start=0, + num_tokens=524288, + ), + ), + Cell( + why="dep4 1M mixed step: a 1M prompt's last chunk (1015808 cached + 32768 new), then " + "12 decode slots at positions to 1048575, V2 row 43", + spec=dict( + _DEP4_1M_V2, + pool_row=43, + seq_lens=[32768], + cached=[1015808], + decode_pos=[ + 1048575, + 1, + 31, + 32, + 33, + 2048, + 32768, + 65536, + 131072, + 262144, + 524288, + 1048574, + ], + k_token_start=0, + num_tokens=1048576, + ), + ), + Cell( + why="dep4 1M packed chunk: 16 requests' tails, 16 x (65520 cached + 16 new) = 1048576 " + "keys in one window, V2 row 20", + spec=dict( + _DEP4_1M_V2, + pool_row=20, + seq_lens=[16] * 16, + cached=[65520] * 16, + k_token_start=0, + num_tokens=1048576, + ), + ), + Cell( + why="dep4 1M edge key counts 1, 2048, 32768, 32769, 2^k + 1 (k = 16..19), 1048575 and " + "1048576 as the 1M window's prefix and suffix, V2 row 7", + spec=dict( + _DEP4_1M_V2, + pool_row=7, + seq_lens=[32768], + cached=[1015808], + k_token_start=0, + num_tokens=1048576, + sweep_counts=[ + 1, + 2048, + 32768, + 32769, + 65537, + 131073, + 262145, + 524289, + 1048575, + 1048576, + ], + ), + ), + Cell( + why="dep4 1M far pool V2, 44 rows x 109568 slots (20.4 GB; 42.4 GiB peak, large): a 1M " + "prompt's whole KV in the last 32768 blocks, past 2^34", + spec=dict( + _DEP4_1M_V2, + pool_row=43, + num_blocks=109568, + tail=True, + large=True, + seq_lens=[32768], + cached=[1015808], + k_token_start=0, + num_tokens=1048576, + ), + ), + Cell( + why="dep4 1M split packed chunk, request 1 of case (d): its whole KV (983040 cached + " + "5 new) from key 0 of the 1966090-key mapping, V2 row 0", + spec=dict(_DEP4_1M_CASE_D, pool_row=0, k_token_start=0, num_tokens=983045), + ), + Cell( + why="dep4 1M split packed chunk, request 2 of case (d): its whole KV (983040 cached + " + "5 new) from key 983045 of the 1966090-key mapping, V2 row 43", + spec=dict(_DEP4_1M_CASE_D, pool_row=43, k_token_start=983045, num_tokens=983045), + ), + Cell( + why="dep4 1M packed chunk past max_seq_len, unsplit: case (d)'s 1966090 keys in one " + "window, V2 row 20", + spec=dict(_DEP4_1M_CASE_D, pool_row=20, k_token_start=0, num_tokens=1966090), + ), + Cell( + why="1M tep8: q-split rank 0's share of a 1M prompt's last chunk at eight ranks: " + "1015808 cached + 4096 = 1019904 keys, V2 row 11", + spec=dict( + _ML4_1M_V2, + pool_row=11, + num_blocks=32800, + seq_lens=[1048576], + k_token_start=0, + num_tokens=1019904, + ), + ), + Cell( + why="1M tep8: q-split rank 3's share of a 1M prompt's last chunk at eight ranks: " + "1015808 cached + 16384 = 1032192 keys, V2 row 30", + spec=dict( + _ML4_1M_V2, + pool_row=30, + num_blocks=32800, + seq_lens=[1048576], + k_token_start=0, + num_tokens=1032192, + ), + ), + Cell( + why="1M tep8: a packed chunk past 1M keys cut per request, two 983045-key requests' " + "5-token tails: the second's whole KV from key 983045, V2 row 5 of 61448 blocks", + spec=dict( + _ML4_1M_V2, + pool_row=5, + num_blocks=61448, + seq_lens=[983045, 983045], + k_token_start=983045, + num_tokens=983045, + ), + ), + Cell( + why="1M tep8: far pool V2, 44 rows x 147456 slots (27.4 GB, large): a 1M prompt's " + "whole KV in the last 32768 blocks, to the view's last byte", + spec=dict( + _ML4_32K_8GPU_FAR, + layout="v2", + pool_row=43, + seq_lens=[1048576], + k_token_start=0, + num_tokens=1048576, + ), + ), + Cell( + why="1M tep8 windows: eight ranks' q-split shares of every request, and 2^k - 1, 2^k, " + "2^k + 1 keys over a 3680512-key mapping, pages cycled", + spec=dict( + _ML4_1M_V2, + pool_row=37, + num_blocks=4096, + cycle=True, + seq_lens=list(_ML4_1M_TEP8_POOL_LENS), + k_token_start=2097152, + num_tokens=1048576, + sweep="windows", + qsplit_ranks=8, + ), + ), + ) + + note = """ + Metadata consumed: none. The op reads `k_cache`, the two slot mappings and + the three integers. A caller builds the mappings the way the DSA indexer + does (`_compute_slot_mappings` in + `_torch/attention/backends/sparse/dsa/indexer.py`): page = block id times + the manager's `indexer_k_cache_page_scale`, a token's data at + `page * tpb * 132 + (pos % tpb) * 128`, its scale at + `page * tpb * 132 + tpb * 128 + (pos % tpb) * 4` for `tpb` tokens per + page (4224 and 4096 at 32, 8448 and 8192 at 64). + + The views `get_indexer_k_cache_buffers` hands out at Mistral Large 4's geometry + (page 32, 132 bytes per token, R indexer rows), measured on both managers: + DSACacheManagerV2 (the engine default): contiguous + [num_blocks * R - row, 32, 1, 132], starting at row * 4224 bytes into + the indexer pool; page scale R. + DSACacheManager (V1): [num_blocks, 32, 1, 132] with stride(0) R * 4224, + starting at row * 4224; page scale 1. + R is 44 when the engine's pretrained config carries `index_topk_pattern`. + It is 72 when it does not, because `_is_full_indexer_layer` does not read the + checkpoint's own spelling, `index_sharing`. Both are cells; the 72-row ones + stop at the FP8 target's pool (16384 blocks). The NVFP4 target's engine + config, `MistralLarge4Config`, derives `index_topk_pattern` from + `index_sharing`, so it runs at R = 44. A request's blocks take + descending slots on V2 and ascending ones on V1; the op reads whatever the + mappings name. At the NVFP4 target's max_seq_len = max_num_tokens = 32768, + a V2 quota of 65536 tokens gave 2155 slots (2048 blocks of quota), and the + views of layers 0 and 71 were [94820, ...] and [94777, ...], as above. + The 64-token manager cells gather through both managers' views at 64 + tokens per page, built for the same 32768-token limits. + At tep8 and tp8 a rank holds 73.61 GiB of weights and at dep8 89.29 GiB + (computed from the checkpoint's shard headers), which leave room for about + 144075 and 132949 V2 slots; no boot has measured those pools. Views are + certified to 147456 slots on V2 (the eight-GPU cell) and to 109568 on V1; + past those they are not certified. + + Call site (the Mistral Large 4 targets' `PrefillTarget._indexer`): one call per + indexer layer of a context step whose longest prompt passes index_topk + 2048, on that layer's view and the step's slot mappings. With chunked + prefill and reuse off the engine plans no indexer chunk at these sizes + (`indexer_max_chunk_size` defaults to 32768), so `k_token_start` is 0 and + `num_tokens` is every context key of the step, up to 32768; a mixed step's + decode slots follow them in the mapping. + + With chunked prefill on (the NVFP4 tep4, tp4 and dep4 deployments at + max_seq_len 1048576, max_num_tokens 32768) every chunk after a request's + first has a cached prefix. `Indexer.prepare_for_chunked_prefill` then plans + the indexer's chunks with `plan_mla_chunked_prefill_chunks`, and once any + context request of the step has cached keys, + `recompute_context_kv_gather_mappings` builds the full-KV mappings + (`slot_mapping_fp8_fullkv` / `slot_mapping_scale_fullkv`): every context + request's cached and new keys, request after request, and no decode slots + (a step whose contexts have no cached keys keeps the regular mappings, + decode slots after the context keys). The target calls the op once per + indexer layer and chunk, from the chunk's `k_token_start` (its first + request's offset in that concatenation) over its + `k_token_end - k_token_start` keys: + one request: its cached + new keys, up to 1048576 (the 1M and dep4 1M + cells); + a group packed while (sum of new) x (sum of KV) stays within the 2^28 + packed-logits budget: every key of the group, which the pool bounds, + not max_seq_len (two 1M prompts' short tails pack into one chunk); the + cells reach 2373024 keys, every token of the delivered tep4 boot's pool, + and 2373856, every token of the delivered tp4 boot's (74183 pages; the + tp4 boot at max_seq_len 1048576 left 2352000). Two 5-token last chunks + over 983040-token prefixes pack into one chunk of 1966090 keys (the + dep4 boot's pool holds 1983872 tokens per rank); split at the request + boundary, as the dep4 deployment's indexer may do, it is two calls, + (0, 983045) and (983045, 983045); + a request cut by a rank's share under the engine's query split: from its + offset to its cached keys plus the share's end, cached + 8192 (r + 1) + keys for rank r of a 32768-row chunk. + Under attention DP (the dep4 deployment) the engine does not split the + indexer's prefill queries across ranks, so a rank's calls cover its own + requests whole. + The 1M cells' V2 views are rebuilt as above, but one: a real V2 manager at + max_seq_len 1048576 with a quota of 2097152 tokens kept max_seq_len + 1048576, handed out a [1, 32772] block table and a layer-71 view of + [3034813, 32, 1, 132] (68974 slots), and put a 1048576-key prompt's 32768 + pages in slots 0..32767 (data bytes 0..6089947520 of the view). So did a + quota of exactly the prompt's pages, 1048576 tokens (32768 blocks): V2 + kept max_seq_len 1048576 and the [1, 32772] table, handed out 34487 + slots (layer 71's view [1517385, 32, 1, 132]) and took the prompt in + slots 32767..0. A 1081344-token quota (33792 blocks) took a + 32768-token prompt in slots 1023..0 and a 1M prompt behind it in slots + 33791..1024, again exactly the quota, with 35566 slots (layer 0's view + [1564904, 32, 1, 132]); a window from key 32768 gathers the second. The + "cycled" cells reach these mapping lengths on a small pool: their pages + outnumber the pool and are drawn as consecutive fresh permutations of it; + a window started one key early, or wrapped at key 2^20, reads other bytes + there (a test proves it). + For the "dep4 1M" cells, a real V2 manager at max_seq_len 1048576 with a + quota of 1050624 tokens kept max_seq_len 1048576, handed out 34563 slots + (49.4 GiB on device), a [1, 32772] block table and a layer-71 view of + [1520729, 32, 1, 132], and put a 1048576-key prompt's 32768 pages in + slots 32767..0, descending; a quota of 1572864 tokens gave 51730 slots + (73.6 GiB) and put a 502048-key and a 1030720-key prompt in slots 15688..0 + and 47898..15689. A quota too small to hold max_seq_len tokens lowers the + manager's max_seq_len to what it holds, so the pre-merge "dep4 1M" cells + rebuild V2 views instead, and a share runs a real manager at max_seq_len + 524288 (quota 526336 tokens: 17319 slots, a [1, 16388] table, 25.1 GiB). + In the "dep4 1M" mixed-step cell the decode slots stand behind a 1M window + as in the regular mapping. + + The NVFP4 tep8 deployment at max_seq_len 1048576 makes tep4's calls but one: + a chunk the engine packs past max_seq_len keys is cut at its request + boundaries (`_per_request_chunks` in the target), one call per request over + that request's own keys from its offset in the full-KV mappings, so a call + gathers at most 1048576 keys. Two 983045-key requests' 5-token tails make + one chunk of 1966090 keys, gathered as (0, 983045) and (983045, 983045). + The offsets still reach the pool's size: the "1M tep8" windows run over a + 3680512-key mapping, every token of the delivered tep8 boot's pool. Under + the engine's query split over the eight ranks, rank r's chunk of a + request's last 32768 rows stops at cached + 4096 (r + 1) keys + (`_split_prefill_queries`, `plan_mla_chunked_prefill_chunks` and + `Indexer.prepare_one_prefill_chunk` in `dsa/indexer.py`; a test checks the + shares and the packed chunk against them). Device peaks: 13.4 GiB for the + shares on 32800 blocks, 24.6 GiB for the packed chunk's 61448 blocks, 56.3 + GiB for the 147456-slot far view (`large`). + + Preconditions the op enforces itself: + k_cache uint8, 4-D, dim 2 == 1; + both mappings int64, 1-D and contiguous; + every tensor on the device; + k_token_start + num_tokens within both mappings. + It raises on each, with these messages: + "expected scalar type Byte but found Float8_e4m3fn" + "k_cache must be a 4D Tensor [num_blocks, block_size, 1, per_token_size], got %d dimensions3" + "k_cache dimension 2 must be 1 for DeepSeek-V3.2, got %d2" + "slot_mapping_fp8 must be int64" + "slot_mapping_fp8 must be a 1D Tensor" + "slot_mapping_fp8 must be contiguous" + "All tensors must be CUDA tensors" + "slot_mapping_fp8 too short for k_token_start + num_tokens" + "indexer_k_cache_gather_op head_dim must be 128 (FP8) or 64 (FP4 packed), got %d96" + The literal "%d" is in the engine's text. + + Alignment: every slot must be a multiple of 4 and the view's data pointer + 4-byte aligned. A misaligned data slot, scale slot or base pointer faults + the CUDA context, which is not recoverable in that process: + "torch.AcceleratorError: CUDA error: misaligned address". + + `num_tokens = 0` checks `head_dim`, then returns a (0, head_dim) + float8_e4m3fn and a (0, 1) float32 tensor without checking or reading any + other argument. + + The outputs are fresh, contiguous tensors. No CUDA-graph capture is + certified: the call site runs in eager context steps. + + Cost: a call is launch-bound at every key count from 1 to 32768, on V1 + and V2 views and at the far ends of the 109568-slot pool and the + 147456-slot V2 view: 32768 keys are 4.1 MiB. + Past that the time grows with the key count (the grid is one 32-thread + block per key). + + Memory: a call allocates its two outputs and nothing else, + num_tokens * (head_dim + 4) bytes. At num_tokens 1048576 that is 134217728 + bytes of keys and 4194304 of scales, 138412032 bytes (132 MiB): torch's + allocated peak over one such call rose by exactly 138412032 bytes. + + The FP4 cache (head_dim 64, 68-byte rows) uses the same kernel with half the + bytes per token. No target uses it, and it is not certified. + + Hard limits (sm_103) + slot int64 logical byte offset, unravelled over k_cache's + shape, then multiplied by the int64 strides + boundary 2^31 / 2^32 bytes + probed at 4982833151 (72 rows x 16384 blocks, V1 and V2) + and at 20363870207 (44 rows x 109568 slots, the NVFP4 + tep4 deployment's largest pool, V1 and V2: past 2^34), there + also with a 1M prompt's 1048576 keys (V2, dep4 1M), the + same bytes at 64-token pages (54784 slots, V1 and V2), + and at slot 27405400700, the last scale of a 147456-slot + V2 view of 27405400704 bytes (44 rows, row 43, a tep8, + tp8 or dep8 rank's largest pool: past 2^34, below 2^35) + passed + page index int32 after the unravel; k_cache.size(0) is also cast + to int32 + boundary 2^31 pages + largest certified view 6488021 pages (V2, 44 rows x + 147456 slots); a whole 276.62 GiB GB300 holds 70317459 + pages of 4224 bytes: not reachable + window index k_token_start + i in int32 (the op narrows k_token_start + and num_tokens to int32 before the launch) + boundary 2^31 + mapping <= 32768 entries without chunked prefill; with + it, every context key of the step, which the pool + bounds (1983872 tokens at the dep4 boot): certified to + 3680512 entries (k_token_start + i up to 3680511, the + tep8 windows), on tp4's packed corner to 2373856 + (k_token_start + i up to 2373855), and on dep4's packed + corner to 1966090 entries (k_token_start up to 983045, + k_token_start + i up to 1966089); not reached + num_tokens int32 grid, one block per key; output offsets row * 128 + in int64 + boundary 2^31 + 32768 without chunked prefill; certified to 3680512 + with it (a tep8 call gathers at most 1048576), to + 2373856 on tp4's packed corner (a grid of 2373856 + blocks, 303853568 output bytes), on dep4's packed corner + to 1966090; not reached + fp32 no index passes through fp32 + """ + + def __call__( + self, + k_cache: torch.Tensor, + slot_mapping_fp8: torch.Tensor, + slot_mapping_scale: torch.Tensor, + k_token_start: int, + num_tokens: int, + head_dim: int, + ) -> tuple[torch.Tensor, torch.Tensor]: + return torch.ops.trtllm.indexer_k_cache_gather_op( + k_cache, + slot_mapping_fp8, + slot_mapping_scale, + k_token_start, + num_tokens, + head_dim, + ) + + def reference( + self, + k_cache: torch.Tensor, + slot_mapping_fp8: torch.Tensor, + slot_mapping_scale: torch.Tensor, + k_token_start: int, + num_tokens: int, + head_dim: int, + ) -> tuple[torch.Tensor, torch.Tensor]: + """A byte gather over the cache's logical (contiguous) order.""" + flat = k_cache.contiguous().view(torch.uint8).reshape(-1) + window = slice(k_token_start, k_token_start + num_tokens) + data_at = slot_mapping_fp8[window].long().unsqueeze(1) + scale_at = slot_mapping_scale[window].long().unsqueeze(1) + data = flat[data_at + torch.arange(head_dim, device=flat.device)] + scale = flat[scale_at + torch.arange(4, device=flat.device)] + return ( + data.view(torch.float8_e4m3fn), + scale.contiguous().view(torch.float32).reshape(num_tokens, 1), + ) + + def is_valid( + self, + k_cache: torch.Tensor, + slot_mapping_fp8: torch.Tensor, + slot_mapping_scale: torch.Tensor, + k_token_start: int, + num_tokens: int, + head_dim: int, + ) -> None: + if num_tokens == 0: + return # two empty tensors, and nothing is read + # The op checks only the upper end of the window. A negative start reads the + # int64 words in front of the mapping and uses them as slots. + assert k_token_start >= 0, f"k_token_start must be >= 0, got {k_token_start}" + # Width and range are checked only where the op does not refuse loudly itself. + cache_ok = k_cache.dtype == torch.uint8 and k_cache.dim() == 4 + if cache_ok and head_dim in (64, 128): + # head_dim is the gather width, and the op admits either width on any + # cache: 64 on a 132-byte row returns the first half of every key. + assert k_cache.shape[-1] == head_dim + 4, ( + f"head_dim {head_dim} does not match the cache's {k_cache.shape[-1]}-byte rows " + f"(head_dim key bytes plus a 4-byte scale)" + ) + window = slice(k_token_start, k_token_start + num_tokens) + data_at, scale_at = slot_mapping_fp8[window], slot_mapping_scale[window] + # The op refuses these itself: a window past a mapping's end, a mapping that + # is not 1-D int64, mappings on two devices. + if ( + data_at.shape != (num_tokens,) + or scale_at.shape != (num_tokens,) + or data_at.dtype != torch.int64 + or scale_at.dtype != torch.int64 + or data_at.device != scale_at.device + ): + return + # Both windows' bounds in one host read: a test runs this before every call. + lo, hi = torch.aminmax(torch.stack([data_at, scale_at]), dim=1) + lo_data, lo_scale, hi_data, hi_scale = torch.cat([lo, hi]).tolist() + # A negative slot makes the kernel skip that row's data and scale, and the + # output is torch.empty: the row comes back as whatever the allocator held. + assert lo_data >= 0 and lo_scale >= 0, ( + "every slot in the window must be >= 0; a negative slot leaves its row uninitialized" + ) + # Nothing bounds a slot from above: one past the view reads the memory after it. + if cache_ok: + extent = k_cache.numel() + assert hi_data <= extent - head_dim and hi_scale <= extent - 4, ( + f"every slot in the window must address bytes inside k_cache's {extent} bytes" + ) + + +indexer_k_cache_gather_op = IndexerKCacheGatherOp() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/indexer_topk_decode.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/indexer_topk_decode.py new file mode 100644 index 000000000000..5f310fdea1fd --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/indexer_topk_decode.py @@ -0,0 +1,799 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Decode top-k of the DSA indexer: per query row, the best keys up to its causal end.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: With fp32 logits, a row of 4096 or more columns is split across blocks and +#: merged in a second launch; below that one block owns a row. The split path +#: needs the caller's `radix_aux_*` scratch. +SPLIT_WORK_COLUMNS = 4096 + +#: Largest `index_topk` that launches in each regime: the kernels take +#: `4 * index_topk` (one block) or `8 * index_topk` (split) bytes of dynamic +#: shared memory on top of their static use, against the 48 KB default. Above +#: it the launch fails at configuration and `indices` keeps its old contents. +#: The call raises nothing, and torch does not report the failure either: the +#: launch goes through the CUDA runtime that libth_common links statically, so +#: its error never reaches torch's own runtime. Found by bisection on +#: sm_100 and again on sm_103 (the split ceiling at 4096 and 9216 columns, +#: and at 32768). +MAX_INDEX_TOPK_SINGLE_BLOCK = 8188 +MAX_INDEX_TOPK_SPLIT_WORK = 3020 + +#: Mistral Large 4's decode shape: next_n 1 (no speculation), its max_seq_len as +#: the scorer's column count, index_topk 2048, and the DSA metadata's radix +#: scratch, `[max_num_sequences = 16, 10, index_topk]`, whatever the batch. +#: `_ML4` is the FP8 target's, whose envelope is 9216. +_ML4 = dict(next_n=1, width=9216, index_topk=2048, aux_rows=16) +#: The NVFP4 target's: the same call at its max_seq_len of 32768, a multiple +#: of the scorer's 256-column alignment, so the scorer's rows are 32768 wide. +_ML4_32K = dict(next_n=1, width=32768, index_topk=2048, aux_rows=16) +#: The NVFP4 tep4 deployment at its 1M envelope (max_seq_len 1048576, chunked +#: prefill): the scorer's rows are 1048576 wide at every decode step, the +#: short prompts' included, since its column count is frozen at max_seq_len. +#: The NVFP4 tp4 deployment at the same envelope makes the same call (scorer +#: width, decode batches, the 16-sequence arena), so these cells cover it too. +#: So does the NVFP4 tep8 deployment (the tep4 deployment at eight ranks, the indexer +#: replicated) at its 1M envelope: the same call on every rank. +_ML4_1M = dict(next_n=1, width=1048576, index_topk=2048, aux_rows=16) +#: The NVFP4 dep4 deployment at the checkpoint's own context: max_seq_len 1048576, +#: the scorer's column count (a multiple of 256, so its rows are contiguous). +#: From 200000 columns every row is split across 10 blocks whatever the row +#: count, so 16 rows fill the 16-sequence scratch exactly; `blocks_per_row` is +#: that split, which the test reads back off the scratch. Only the dep4 +#: deployment's cells use it (its own file named it `_ML4_1M`): it is `_ML4_1M` plus +#: `blocks_per_row=10`, the key by which the test routes them to the dep4 +#: deployment's drivers. +_ML4_1M_DEP4 = dict(next_n=1, width=1048576, index_topk=2048, aux_rows=16, blocks_per_row=10) +#: DSpark K=3 at the 1M envelope, including overlap/verification headroom. +#: The scorer pads each row to 1048832 columns; 16 requests verify 64 rows. +_ML4_1M_DSPARK = dict( + next_n=4, width=1048584, row_pad=248, index_topk=2048, aux_rows=64, blocks_per_row=10 +) + + +class IndexerTopkDecode(OpWrapper): + """Select each decode row's `index_topk` best columns below its causal end. + + Row `b * next_n + t` belongs to sequence `b`'s `t`-th new token. Its + candidates are columns `[0, end)` with `end = (seq_lens[b] - next_n + t + + 1) // compress_ratio`; nothing past `end` is read. With `n = min(index_topk, + end)`, the `n` winners land in `indices[r, :n]` as plain column indices and + `indices[r, n:]` is -1 (a score of -inf is the exception; see the note). + + Each row is a set. The winners are the top-n by score; where scores tie at + the n-th value, which of the tied columns win is unspecified, but the + selected scores are the reference's. Their order within `indices[r, :n]` + is unspecified; on the split path it changes from call to call on + identical input (20 calls, 20 orders, the set unchanged). Only on the + single-block path (fp32 rows narrower than `SPLIT_WORK_COLUMNS`) does a + row with `end <= index_topk` list `0..end-1` in order. Split rows, every + Mistral Large 4 row included, list the same columns in no fixed order. + + `indices` is a caller-allocated int32 out-parameter written in place; the + call returns it. The scorer's logits are consumed as they are: the call + owns the selection, the caller owns the scoring and what reads the indices. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # The decode feeds it the paged scorer's output: fp32, one row per + # generation token, as wide as the engine's max sequence length, + # row-padded to a multiple of 256 columns. `width` is that column count; + # `seq_lens` the sequences' KV lengths after this step. + # + # Mistral Large 4's (`_ML4`, the FP8 target's 9216 envelope): next_n 1, 1..16 + # sequences, every row split across 4 blocks at width 9216, its scratch the + # engine's 16-sequence arena. The NVFP4 target's (`_ML4_32K`, its 32768 + # envelope): the same call at width 32768, where a row is split across 10 + # blocks at 1..15 rows and 9 at 16, so its cells drive 15 and 16 rows. At + # its 1M envelope (`_ML4_1M`, the tep4 deployment's and, at eight ranks, the tep8 + # deployment's): width 1048576, where every row count 1..16 splits 10 ways, so + # 16 rows fill the 16-sequence arena exactly; seq_lens up to 1048576, the + # decode graph captured at full length. The tp4 deployment at that envelope + # adds ends at the per-block spans, the blocks' boundary columns as winners + # and the graphs captured at the engine's own capture lengths. + # The dep4 deployment at max_seq_len 1048576 (`_ML4_1M_DEP4`): width 1048576, 10 + # blocks per row at every row count 1..16, the scratch the arena's exact + # 327680 elements at 16 rows. Its boot captures the long-family decode + # graphs on a warm-up dummy of max_seq_len - 1 keys beside 1-key padding + # rows, the first `_ML4_1M_DEP4` cell. + CELLS: tuple[Cell, ...] = ( + Cell( + why="DSpark 1M verify causal/page ends, 64 rows and exact radix arena", + spec=dict( + _ML4_1M_DSPARK, + seq_lens=[ + 3, + 4, + 5, + 31, + 32, + 33, + 2047, + 2048, + 2049, + 32768, + 65536, + 131072, + 524289, + 1048575, + 1048576, + 1048584, + ], + batches=[1, 2, 4, 8, 15, 16], + ), + ), + *( + Cell( + why=f"DSpark 1M verify graph batch {batch}: boot, long replay and DEP padding", + spec=dict( + _ML4_1M_DSPARK, + graph_batches=[batch], + seq_lens=[1048583] + [4] * 15, + replay_lens=( + [ + 1048584, + 1048576, + 1048575, + 524289, + 131073, + 65537, + 32769, + 2049, + 2048, + 257, + 65, + 33, + 32, + 5, + 4, + 4, + ], + [1048576] + [4] * 15, + ), + replays=2, + ), + ) + for batch in (1, 2, 4, 8, 16) + ), + Cell( + why="DSpark 1M verify mixed tail: 60 rows after 32708 context rows", + spec=dict( + _ML4_1M_DSPARK, + tail_of=32768, + seq_lens=[1048584, 2048, 2049, 32, 33] * 3, + ), + ), + Cell( + why="DSpark 1M verify split-edge winners: all 64 rows reach the last block", + spec=dict(_ML4_1M_DSPARK, seq_lens=[1048584] * 16, scores="edges"), + ), + Cell( + why="DSpark 1M verify tied winners and signed zero across four causal ends", + spec=dict(_ML4_1M_DSPARK, seq_lens=[1048584, 2049, 4], scores="ties"), + ), + Cell( + why="DSpark verify four causal rows per request, 64 rows and scorer stride 33024", + spec=dict( + next_n=4, + width=32776, + row_pad=248, + index_topk=2048, + aux_rows=64, + seq_lens=[ + 3, + 4, + 5, + 6, + 31, + 32, + 33, + 2047, + 2048, + 2049, + 2050, + 4096, + 16385, + 32767, + 32768, + 32776, + ], + ), + ), + Cell( + why="DSpark verify graph batches 1..16 x 4, independent causal ends after replay", + spec=dict( + next_n=4, + width=32776, + row_pad=248, + index_topk=2048, + aux_rows=64, + graph_batches=[1, 2, 4, 8, 16], + seq_lens=[32776] * 16, + replays=3, + ), + ), + Cell( + why="DSpark verify mixed topk: 60 rows after 32708 prompt rows", + spec=dict( + next_n=4, + width=32776, + row_pad=248, + index_topk=2048, + aux_rows=64, + tail_of=32768, + seq_lens=[32776, 2048, 2049, 32, 33] * 3, + ), + ), + Cell( + why="Mistral Large 4 batch 16 at width 9216, 4 blocks per row: seq_lens 1..9216, " + "the 2048/2049 top-k edge, the 4-block edges 8192..8196", + spec=dict( + _ML4, + seq_lens=[1, 2, 5, 2047, 2048, 2049, 4097, 6145] + + [8191, 8192, 8193, 8195, 8196, 9000, 9215, 9216], + ), + ), + Cell( + why="Mistral Large 4 every row count 1..16 (mixed steps' generation rows; the default " + "graph list 1..16, unpadded): the leading b rows of one batch", + spec=dict( + _ML4, + batches=tuple(range(1, 17)), + seq_lens=[9216, 2049, 8193, 1, 5000, 2048, 8196, 777] + + [9215, 4097, 8192, 3, 6144, 2047, 8195, 7000], + ), + ), + Cell( + why="Mistral Large 4 captured decode batches 1, 2, 4, 8, 16: replayed on new logits " + "and seq_lens, the padding dummy's row at seq_len 1", + spec=dict(_ML4, graph_batches=(1, 2, 4, 8, 16), seq_lens=[9216] * 16, replays=3), + ), + Cell( + why="Mistral Large 4 tied scores at the k-th value: 16 levels, all +0.0 (relu's floor), " + "mixed +-0.0; gated on the selected scores", + spec=dict( + _ML4, + scores="ties", + seq_lens=[9216, 9216, 9216, 8196, 8196, 8196, 5000, 5000] + + [5000, 2049, 2049, 2049, 2048, 100, 1, 9000], + ), + ), + Cell( + why="Mistral Large 4 scores past fp16's range and under its resolution, both signs: " + "the fp16 first pass bins them together", + spec=dict( + _ML4, + scores="wide", + seq_lens=[9216, 9216, 9215, 8196, 8195, 8192, 6000, 4097] + + [2049, 2048, 1000, 1, 9216, 7000, 3000, 9100], + ), + ), + Cell( + why="Mistral Large 4 mixed step: 15 generation rows at rows 9201..9215 of the " + "step's 9216-row selection buffer; the context rows untouched", + spec=dict( + _ML4, + tail_of=9216, + seq_lens=[9216, 1, 2049, 8196, 2048, 5000, 9215, 3] + + [8192, 7777, 4096, 1234, 9000, 6145, 8193], + ), + ), + Cell( + why="Mistral Large 4 32768, batch 16 at width 32768, 9 blocks per row: seq_lens " + "1..32768, the 2048/2049 top-k edge, the 9216 and 16384 marks", + spec=dict( + _ML4_32K, + seq_lens=[1, 2, 5, 2047, 2048, 2049, 8192, 8196] + + [9216, 16384, 16385, 24576, 30000, 32767, 32768, 12345], + ), + ), + Cell( + why="Mistral Large 4 32768, every row count 1..16 (10 blocks per row to 15 rows, 9 " + "at 16; mixed steps' generation rows, the default graph list unpadded): the " + "per-block edges 20480/20481/20489/20490 and 18432/18433/18440/18441", + spec=dict( + _ML4_32K, + batches=tuple(range(1, 17)), + seq_lens=[20480, 20481, 20489, 20490, 32768, 2049, 1, 18432] + + [18433, 18440, 18441, 2048, 32767, 16385, 3, 30000], + ), + ), + Cell( + why="Mistral Large 4 32768, captured decode batches 1, 2, 4, 8, 16: replayed on new " + "logits and seq_lens up to 32768, the padding dummy's row at seq_len 1", + spec=dict(_ML4_32K, graph_batches=[1, 2, 4, 8, 16], seq_lens=[32768] * 16, replays=3), + ), + Cell( + why="Mistral Large 4 32768, tied scores at the k-th value at 15 and 16 rows: 16 " + "levels, all +0.0 (relu's floor), mixed +-0.0; gated on the selected scores", + spec=dict( + _ML4_32K, + scores="ties", + batches=(15, 16), + seq_lens=[32768, 32768, 32768, 20490, 20490, 20490, 18441, 18441] + + [18441, 2049, 2049, 2049, 2048, 100, 1, 30000], + ), + ), + Cell( + why="Mistral Large 4 32768, scores past fp16's range and under its resolution at 15 " + "and 16 rows, both signs: the fp16 first pass bins them together", + spec=dict( + _ML4_32K, + scores="wide", + batches=(15, 16), + seq_lens=[32768, 32768, 32767, 20490, 20489, 20481, 20480, 18441] + + [18433, 18432, 2049, 2048, 1000, 1, 16385, 9216], + ), + ), + Cell( + why="Mistral Large 4 32768, mixed step: 15 generation rows at rows 32753..32767 of " + "the step's 32768-row selection buffer (10 blocks per row); the context rows " + "untouched", + spec=dict( + _ML4_32K, + tail_of=32768, + seq_lens=[32768, 1, 2049, 20481, 2048, 20490, 32767, 3] + + [18441, 7777, 4096, 1234, 30000, 6145, 16385], + ), + ), + Cell( + why="Mistral Large 4 1M, batch 16 at width 1048576, 10 blocks per row filling the " + "16-sequence arena: seq_lens 1..1048576, the 2048/2049 top-k edge, the 10-block " + "edges 20480/20481/20490, the 32768 and 200000 marks", + spec=dict( + _ML4_1M, + seq_lens=[1, 2, 2047, 2048, 2049, 20480, 20481, 20490] + + [32768, 32769, 131072, 200000, 524288, 1000000, 1048575, 1048576], + ), + ), + Cell( + why="Mistral Large 4 1M, every row count 1..16 at width 1048576 (10 blocks per row at " + "each; mixed steps' generation rows, the default graph list unpadded): full-length " + "rows beside short ones", + spec=dict( + _ML4_1M, + batches=tuple(range(1, 17)), + seq_lens=[1048576, 2049, 20481, 1, 524288, 2048, 20490, 777] + + [1048575, 32769, 200001, 3, 131072, 2047, 1000000, 65536], + ), + ), + Cell( + why="Mistral Large 4 1M, captured decode batches 1, 2, 4, 8, 16 at width 1048576, " + "captured at full length: replayed on new logits and seq_lens up to 1048576, the " + "padding dummy's row at seq_len 1", + spec=dict(_ML4_1M, graph_batches=[1, 2, 4, 8, 16], seq_lens=[1048576] * 16, replays=3), + ), + Cell( + why="Mistral Large 4 1M, tied scores at the k-th value at 15 and 16 rows: 16 levels, " + "all +0.0 (relu's floor), mixed +-0.0, up to 104863 tied columns a block; gated on " + "the selected scores", + spec=dict( + _ML4_1M, + scores="ties", + batches=(15, 16), + seq_lens=[1048576, 1048576, 1048576, 20490, 20490, 20490, 20481, 20481] + + [20481, 2049, 2049, 2049, 2048, 100, 1, 524288], + ), + ), + Cell( + why="Mistral Large 4 1M, scores past fp16's range and under its resolution at 15 and 16 " + "rows, both signs, at width 1048576: the fp16 first pass bins them together", + spec=dict( + _ML4_1M, + scores="wide", + batches=(15, 16), + seq_lens=[1048576, 1048576, 1048575, 524289, 200000, 20490, 20481, 20480] + + [32769, 2049, 2048, 1000, 1, 131073, 9216, 65537], + ), + ), + Cell( + why="Mistral Large 4 1M, mixed step: 15 generation rows at rows 32753..32767 of the " + "step's 32768-row selection buffer, at width 1048576 (10 blocks per row); the " + "context rows untouched", + spec=dict( + _ML4_1M, + tail_of=32768, + seq_lens=[1048576, 1, 2049, 20481, 2048, 20490, 1048575, 3] + + [524288, 7777, 4096, 1234, 1000000, 6145, 131073], + ), + ), + Cell( + why="Mistral Large 4 1M, both sides of each 10-block edge at 16 rows: ends 20480/20481 " + "(the last block starts selecting) and 20489/20490 (all 10 do), beside full-length " + "rows whose 104857-column blocks start off the 16-byte grid", + spec=dict( + _ML4_1M, + seq_lens=[20479, 20480, 20481, 20482, 20488, 20489, 20490, 20491] + + [1048576, 1048575, 1048570, 1048569, 104857, 209715, 2048, 2049], + ), + ), + Cell( + why="Mistral Large 4 1M, the per-block spans at batch 16, width 1048576: ends 104857/104858 " + "(a full-length row's block width) and 943713 (where its last block starts), 12345 " + "(every block under 2048 columns), 199999/200000 as ends, the 2048/2049 top-k edge, " + "1048575/1048576", + spec=dict( + _ML4_1M, + seq_lens=[1, 2047, 2048, 2049, 32768, 32769, 65536, 104857] + + [104858, 199999, 200000, 524288, 943713, 1048575, 1048576, 12345], + ), + ), + Cell( + why="Mistral Large 4 1M, the 10 blocks' boundary columns as winners at 16 rows: each " + "block's first and last 4 columns scored at the top or straddling the 2048th score, " + "blocks of floor(end / 10) = 0..3 mod 4 columns, so starts on and off the 16-byte grid", + spec=dict( + _ML4_1M, + scores="edges", + seq_lens=[1048576, 1048575, 1048570, 1048569, 1048560, 1048559, 1048550, 1048549] + + [524288, 200000, 104857, 209715, 32769, 20490, 20491, 1000003], + ), + ), + Cell( + why="Mistral Large 4 1M, graphs captured as the engine captures them: batches 1, 2, 4, " + "8, 16 with one row at 1048575 and the rest at 1, replayed at the 2048/2049 and " + "10-block edges, 1048575 and 1048576, the padding dummy's row at seq_len 1", + spec=dict( + _ML4_1M, + graph_batches=[1, 2, 4, 8, 16], + seq_lens=[1048575] + [1] * 15, + replay_lens=( + [2049, 1048576, 2048, 20481, 20490, 1048575, 2047, 20480] + + [104857, 200000, 524288, 20489, 1, 32768, 1000000, 1048576], + [1048576, 2048, 1048575, 2049, 20490, 1, 20481, 20480] + + [1048576, 1048575, 1048576, 2049, 2048, 20489, 20491, 1048576], + ), + replays=2, + ), + ), + Cell( + why="Mistral Large 4 1M boot: decode graphs 1, 2, 4, 8, 16 captured on the warm-up " + "dummy at max_seq_len - 1 = 1048575 keys ahead of 1-key padding rows (the " + "long-family capture), the warm-up call gated; replayed on new logits and " + "seq_lens up to 1048576", + spec=dict( + _ML4_1M_DEP4, + graph_batches=(1, 2, 4, 8, 16), + seq_lens=[1048575] + [1] * 15, + replays=3, + ), + ), + Cell( + why="Mistral Large 4 1M, batch 16 at width 1048576, 10 blocks per row (the arena " + "full): seq_lens 1..1048576, the 2048/2049 top-k edge, 32768/32769, " + "199999/200000, 1048575/1048576", + spec=dict( + _ML4_1M_DEP4, + seq_lens=[1, 2, 5, 2047, 2048, 2049, 32768, 32769] + + [131072, 199999, 200000, 262145, 524288, 777777, 1048575, 1048576], + ), + ), + Cell( + why="Mistral Large 4 1M, every row count 1..16 (10 blocks per row at each, 16 " + "included; mixed steps' generation rows, the default graph list unpadded): the " + "per-block edges 20480/20481/20489/20490 and 204800/204801/204810/204811", + spec=dict( + _ML4_1M_DEP4, + batches=tuple(range(1, 17)), + seq_lens=[20480, 20481, 20489, 20490, 1048576, 2049, 1, 204800] + + [204801, 204810, 204811, 2048, 1048575, 524289, 3, 1000000], + ), + ), + Cell( + why="Mistral Large 4 1M, captured decode batches 1, 2, 4, 8, 16 at 1048576 keys: " + "replayed on new logits and seq_lens up to 1048576, the padding dummy's row at " + "seq_len 1", + spec=dict( + _ML4_1M_DEP4, graph_batches=[1, 2, 4, 8, 16], seq_lens=[1048576] * 16, replays=3 + ), + ), + Cell( + why="Mistral Large 4 1M, attention DP's one live row: graphs 1, 2, 4, 8, 16 captured " + "at one 1048576-key row ahead of 1-key dummies, replayed with the live row at " + "1048576, 1048575 and a drawn length, the rest of the batch seq_len-1 dummies", + spec=dict( + _ML4_1M_DEP4, + graph_batches=[1, 2, 4, 8, 16], + live=1, + seq_lens=[1048576] + [1] * 15, + replays=3, + ), + ), + Cell( + why="Mistral Large 4 1M, tied scores at the k-th value at 15 and 16 rows: 16 levels, " + "all +0.0 (relu's floor), mixed +-0.0; gated on the selected scores", + spec=dict( + _ML4_1M_DEP4, + scores="ties", + batches=(15, 16), + seq_lens=[1048576, 1048576, 1048576, 20490, 20490, 20490, 204811, 204811] + + [204811, 2049, 2049, 2049, 2048, 100, 1, 524289], + ), + ), + Cell( + why="Mistral Large 4 1M, scores past fp16's range and under its resolution at 15 and " + "16 rows, both signs: the fp16 first pass bins them together", + spec=dict( + _ML4_1M_DEP4, + scores="wide", + batches=(15, 16), + seq_lens=[1048576, 1048576, 1048575, 20490, 20489, 20481, 20480, 204811] + + [204801, 204800, 2049, 2048, 1000, 1, 524289, 262145], + ), + ), + Cell( + why="Mistral Large 4 1M, mixed step: 15 generation rows at rows 32753..32767 of the " + "step's 32768-row selection buffer (10 blocks per row); the context rows " + "untouched", + spec=dict( + _ML4_1M_DEP4, + tail_of=32768, + seq_lens=[1048576, 1, 2049, 20481, 2048, 20490, 1048575, 3] + + [524289, 7777, 4096, 1234, 1000000, 6145, 262145], + ), + ), + ) + + note = """ + DSpark's 1M cells verify four tokens per request at width 1048584 and + stride 1048832, up to 64 rows. Every row splits 10 ways, including graph + batches 1/2/4/8/16 and the 60-row tail after 32708 prompt rows. Boot + capture starts at one 1048583-key request beside four-key dummies; + replays reach 1048584 and return the eager call's sets on new logits. + The exact scratch is [64, 10, 2048] (1310720 elements per buffer): every + gated launch writes only its leading footprint and leaves the canary + untouched. One element short is refused before any write. The cells + cover causal/page ends, split-edge winners, tied scores and signed zero. + + Only fp32 logits are certified. bf16/fp16 logits take another path through + the op, with no split work, that no cell drives. Nor does any cell drive + fp32 rows narrower than `SPLIT_WORK_COLUMNS` (the single-block path): the + op accepts them, but they are not certified here. + + `compress_ratio` other than 1 and `row_kv_lens` (ragged rows) belong to + other DSA models; Mistral Large 4 passes neither and neither is certified + here. + + Logits are read at `logits.stride(0)`, so the paged scorer's row-padded + view is a legal operand without a copy. At the Mistral Large 4 targets' + max_context_len, 9216 (FP8), 32768 (NVFP4) and 1048576 (NVFP4 at its 1M + envelope, for dep4 the checkpoint's own context), all multiples of the + scorer's 256-column alignment, that view is contiguous. + + Metadata consumed: none beyond the arguments. Preconditions: + - How many blocks a row is split across is chosen on the host per call + from the row count, the column count and the device's SM count + (`computeIndexerTopKDecodeBlocksPerRow`, `kernels/indexerTopK.cu`), + not from the lengths, so a captured graph keeps its capture's count. + Measured from the scratch footprint on a 152-SM GB300 at 1..16 rows: + 4096 columns give 2 blocks; 8192 and 9216 give 4 (the FP8 target's + width: every row at 4 blocks); 12288 give 6, 16384 give 8, 18432 give + 9; 20480, 24576 and 32768 give 10 at 1..15 rows and 9 at 16 rows (the + NVFP4 target's width), and so do 131072, 199936 and 199999. From + 200000 columns (`kDefaultSplitWorkThreshold`) the host returns 10 + blocks before it reads the SM count, so every row count splits 10 + ways: 200000, 262144, 524288 and 1048576 (the NVFP4 target at its 1M + envelope) give 10 at 1..16 rows. A test pins 200000 and 1048576, and + every `_ML4_1M_DEP4` cell reads that split back off the scratch. + - Each block takes floor(end / blocks) of the row's columns and the last + block the rest. A block of at most index_topk columns hands all of + them to the merge, and a wider one selects its own top index_topk + (`topKPerRowDecode`). The edges depend on the block count, not the + width: with 10 blocks (width 32768 at 1..15 rows, width 1048576 at + every row count) the blocks start selecting at ends 20481 (the last + block) and 20490 (all 10), with 9 at 18433 and 18441. The cells drive + both sides of each edge, at width 32768 (10 and 9 blocks) and at + 1048576 (10). One-off sweeps (not cells) with distinct scores selected + the reference's set in every row: every end 1..32768 at width 32768, + at 15 and at 16 rows; and at width 1048576, 16 rows a call, every end + 1..32768, every end 1015809..1048576 and 32768 random ends between, + plus 1024 random ends one row a call; and, in the dep4 deployment's sweep + at width 1048576, 16 rows a call, every end 1..40960, 16384 drawn ends + past it and the last 1024 ends to 1048576. + - A block's columns are read in 16-byte vectors from its own start + (`vectorized_process`, `kernels/indexerTopK.cu`), so a block that + starts off that grid (a block width of 104857 columns, odd, puts most + of a full-length row's starts there) reads its first and last few + columns one at a time. At width 1048576 a cell scores each block's + first and last 4 columns at the top of the row or either side of its + 2048th score, over block widths of every residue mod 4 (68 of its 160 + block starts off the grid), and every planted column lands on its side + of the cut. One-off sweeps of the same planting at 16 rows a call (40 + ends 1048537..1048576, 512 ends 20490..21001 and 2048 random ends + between) selected the reference's set in every row. + - From 4096 fp32 columns the call needs `radix_aux_indices` (int32) and + `radix_aux_logits` (fp32), contiguous, at least rows x blocks x + index_topk elements each. At width 32768 that is 307200 at 15 rows and + 294912 at 16, inside the 16-sequence arena's 327680. At width 1048576 + it is the whole arena at 16 rows: every element of `[16, 10, 2048]` is + written and nothing past it, and an arena one element short is + refused (`got 327679 / 327679, need 327680`). At every call a + `_ML4_1M_DEP4` cell gates, the first launch writes exactly rows x blocks + x index_topk elements, a leading prefix, and nothing past it: those + cells pass the arena at its exact size, `[16, 10, 2048]` of each, as + the leading view of a buffer whose tail stayed untouched. A stable + arena keeps a captured graph valid: the DSA metadata's `radix_aux_*` + reserves one row per verification token, or one per sequence without + speculation. Without them the op raises + `RuntimeError: radix_aux_{indices,logits} must be pre-allocated by the caller when + blocks_per_row > 1 (got blocks_per_row=4). Required for CUDA Graph + safety.`; with too few elements, `RuntimeError: + radix_aux_{indices,logits} must hold at least + num_rows*blocks_per_row*index_topk elements (got 24576 / 24576, need + 32768)`. + - `seq_lens` must be int32; the op refuses int64 itself with + `RuntimeError: expected scalar type Int but found Long`. + - Uniform `next_n` needs `seq_lens >= next_n - 1`: a negative causal end + triggers an illegal address in the split kernel (measured at next_n=4, + seq_lens=1/2, 32776 columns and stride 33024). The eager guard refuses + this input at 32776 and 1048584 columns; captured calls must preserve + the bound on replay. Runtime + padding dummies have at least next_n tokens, including their drafts. + - Treat each row as a set (see the class docstring). A captured graph's + replay selects the same set as an eager call on the same data, in a + different order. The lengths are read at replay: a graph captured as + the engine captures it at width 1048576 (one row at 1048575, the + others at 1) and replayed at ends from 1 to 1048576 (the 2048/2049 + and 10-block edges among them) selected an eager call's sets. + - On the split path, a row with end <= index_topk returns a column scored + -inf or -FLT_MAX as -1: it loses to the merge's -FLT_MAX padding + (measured: 99 of 100 columns back). The paged scorer's finite + relu-weighted sums reach neither value. A row whose end is 0 returns + all -1. Measured again at width 32768, at 10 blocks and at 9, and at + width 1048576 (10 blocks, 2 and 16 rows): 99 of 100 columns back for + each value. + + Cost: the rows' ends, more than the width, set the cost. At width 1048576 + with every row at full length, 16 rows launch 160 blocks, more than one + wave on sm_103. + + Hard limits (sm_103) + logits row offset rowIdx * stride0, int64 no 32-bit product + indices offset rowIdx * index_topk, int64 no 32-bit product. A mixed step + passes the tail rows of a + [32768, 2048] selection buffer as + a view, whose base offset torch + forms in 64 bits + scratch offset rowIdx * blocks * index_topk, int64 no 32-bit product + rows, seq_lens int32 envelope: 16 rows at 1048576; + 64 at 32776/1048584 (DSpark) + column index int32, below the column count boundary 2^31 columns; envelope + 1048584, not probed + logits stride(0) cast to int32 on the host boundary 2^31; envelope 1048576, + DSpark 33024/1048832; not probed + logits elements rows x stride(0) envelope 16 x 1048576 = 2^24, far + below 2^31; probed one-off (not a + cell) at 2049 x 1048576, past + 2^31: every row's set right, row + 2048 (from element 2^31) included; + DSpark 64 x 1048832 = 67125248, + below 2^31 (2048 such rows) + scratch elements rows x blocks x index_topk envelope 16 x 10 x 2048 = 327680, + the arena exactly at width + 1048576; DSpark 64 x 10 x 2048 + = 1310720 at width 1048584: driven + top-k indices int32 throughout, never through fp32 (the split path carries each + candidate's fp32 score beside its int32 column) + index_topk launch ceiling 8188 (one block), probed at 8188/8189, and + 3020 (split) 3020/3021 at 9216, 32768 and + 1048576 columns: is_valid + refuses past it + KV pool not read the op takes the scorer's logits, + so the pool's size does not bound + it + """ + + def __call__( + self, + logits: torch.Tensor, + seq_lens: torch.Tensor, + indices: torch.Tensor, + next_n: int, + index_topk: int = 2048, + compress_ratio: int = 1, + radix_aux_indices: Optional[torch.Tensor] = None, + radix_aux_logits: Optional[torch.Tensor] = None, + row_kv_lens: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + torch.ops.trtllm.indexer_topk_decode( + logits, + seq_lens, + indices, + next_n, + index_topk, + compress_ratio, + radix_aux_indices, + radix_aux_logits, + row_kv_lens, + ) + return indices + + def reference( + self, + logits: torch.Tensor, + seq_lens: torch.Tensor, + indices: torch.Tensor, + next_n: int, + index_topk: int = 2048, + compress_ratio: int = 1, + radix_aux_indices: Optional[torch.Tensor] = None, + radix_aux_logits: Optional[torch.Tensor] = None, + row_kv_lens: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + """The winners per row, as a fresh `[rows, index_topk]` int32 tensor. + + `indices` and the two scratch buffers are not read. Rows are judged on + fp32 values with `torch.topk`, so the answer is exact only when a row + has no tie at its k-th value; under a tie the selected scores, not the + columns, are what the reference fixes. + """ + rows = logits.shape[0] + out = torch.full((rows, index_topk), -1, dtype=torch.int32, device=logits.device) + lens = seq_lens.tolist() + per_row = None if row_kv_lens is None else row_kv_lens.tolist() + for r in range(rows): + kv = per_row[r] if per_row is not None else lens[r // next_n] - next_n + r % next_n + 1 + end = max(0, kv // compress_ratio) + if end <= index_topk: + out[r, :end] = torch.arange(end, dtype=torch.int32, device=logits.device) + else: + out[r] = torch.topk(logits[r, :end].float(), index_topk).indices.to(torch.int32) + return out + + def is_valid( + self, + logits: torch.Tensor, + seq_lens: torch.Tensor, + indices: torch.Tensor, + next_n: int, + index_topk: int = 2048, + compress_ratio: int = 1, + radix_aux_indices: Optional[torch.Tensor] = None, + radix_aux_logits: Optional[torch.Tensor] = None, + row_kv_lens: Optional[torch.Tensor] = None, + ) -> None: + split = logits.dtype == torch.float32 and logits.shape[1] >= SPLIT_WORK_COLUMNS + ceiling = MAX_INDEX_TOPK_SPLIT_WORK if split else MAX_INDEX_TOPK_SINGLE_BLOCK + assert 1 <= index_topk <= ceiling, ( + f"index_topk must be in [1, {ceiling}] for a {logits.shape[1]}-column operand; " + f"got {index_topk}, which fails to launch and raises nothing" + ) + # The op checks `indices.size(1) >= index_topk`, but the kernel writes at + # a row stride of `index_topk`: a wider buffer gets its rows packed into + # the first few and the rest left at their old contents. + assert indices.shape[1] == index_topk, ( + f"indices must be exactly {index_topk} columns wide (the kernel writes it " + f"flat at a row stride of index_topk); got shape {tuple(indices.shape)}" + ) + # Only the column stride of 1 is driven. On the split path another stride + # selects a wrong set, with indices past the row's end. + assert logits.stride(1) == 1, ( + f"logits rows must be unit-stride in the column dimension; got {logits.stride()}" + ) + # A causal end past the logits width reads beyond the row. The lengths are + # device data that a captured call reads only at replay, and reading them + # here would sync inside the capture, so the check needs an eager call. + capturing = torch.cuda.is_current_stream_capturing() + if row_kv_lens is None and seq_lens.numel() and not capturing: + assert int(seq_lens.min()) >= next_n - 1, ( + "uniform next_n produces a negative causal end; " + "seq_lens must be at least next_n - 1" + ) + widest = int(seq_lens.max()) // compress_ratio + assert widest <= logits.shape[1], ( + f"a causal end of {widest} exceeds the {logits.shape[1]}-column logits; " + "the kernel reads past the row" + ) + + +indexer_topk_decode = IndexerTopkDecode() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/ssmax_indexer_q_side_fused.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/ssmax_indexer_q_side_fused.py new file mode 100644 index 000000000000..a30b9aaf03c6 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/ssmax_indexer_q_side_fused.py @@ -0,0 +1,451 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The Mistral Large 4 DSA indexer's q side after its projection, SSMax included, in one Triton +launch: ``ssmax_scaling`` followed by ``cute_dsl_indexer_q_side_fused`` (interleaved rope), bit +for bit, reading the head weights straight out of the bf16 ``wk | weights_proj`` output.""" + +from __future__ import annotations + +import torch +import triton +import triton.language as tl +from triton.language.extra import libdevice +from triton.language.extra.cuda import gdc_wait + +from tensorrt_llm._torch.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_indexer_q_side_fused import ( # noqa: E501 + cute_dsl_indexer_q_side_fused, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ssmax_scaling import ( + ssmax_scaling, +) + +HEADS = 32 +HEAD_DIM = 128 +ROPE_DIM = 64 + +#: max_seq_len of the NVFP4 targets' 32K config: every position, and the rope table's rows. +MAX_SEQ_LEN = 32768 +#: Their 1M config (the checkpoint's own context, chunked prefill in steps of at most +#: MAX_NUM_TOKENS rows): a 1048576-row table, a chunk's rows behind its request's cached +#: prefix, decode rows at any position up to 1048575. +LONG_MAX_SEQ_LEN = 1048576 +MAX_NUM_TOKENS = 32768 +#: Decode batches the targets capture (see `ssmax_scaling_fused_q.GRAPH_BATCHES`). +GRAPH_BATCHES = (1, 2, 3, 4, 8, 13, 16, 32, 64, 128, 256) + +#: What every cell shares: the NVFP4 checkpoint's indexer s and b (44 layers x 32 heads, read +#: on disk as bf16: s in [-0.0437012, 0.0664062], b in [0.976562, 1.04688]) inside the draw +#: ranges with the four corners pinned; its interleaved YaRN (factor 2048) + LeRoPE table of +#: 32768 rows with cos and sin rounded through bf16, as the targets hold it; the head weights +#: as the targets pass them, the last 32 columns of the bf16 `[T, 160]` wk | weights_proj +#: output. +_ML4 = dict( + s_range=(-0.05, 0.075), + b_range=(0.97, 1.06), + q="normal", + table="yarn", + wq="kw_bf16", + max_pos=MAX_SEQ_LEN, +) +#: The same at the 1M config: the YaRN (factor 2048) + LeRoPE table of 1048576 rows. +_ML4_1M = dict(_ML4, max_pos=LONG_MAX_SEQ_LEN) + + +@triton.jit +def _round_bf16(x): + """fp32 rounded to bf16 (to nearest, ties to even) and widened back. The rounding is an + inline `cvt.rn.bf16.f32`: the compiler folds a bare `.to(bf16).to(fp32)` pair into + nothing, which skips a rounding the chain specifies.""" + r = tl.inline_asm_elementwise( + "cvt.rn.bf16.f32 $0, $1;", "=h,r", [x], dtype=tl.bfloat16, is_pure=True, pack=1 + ) + return r.to(tl.float32) + + +@triton.jit +def _round_bf16_ftz(x): + """`_round_bf16`, then a bf16 subnormal (or zero) flushed to the zero of its sign: the + q side's fast-math widening of its rope-half input.""" + bits = _round_bf16(x).to(tl.int32, bitcast=True) + # bf16 shares fp32's exponent field: a bf16 subnormal widens to a zero exponent. + flushed = tl.where((bits & 0x7F800000) == 0, bits & -2147483648, bits) + return flushed.to(tl.float32, bitcast=True) + + +@triton.jit +def _max_nan(a, b): + """max that returns NaN when either side is NaN, as the q side's amax and its floor do.""" + return tl.maximum(a, b, propagate_nan=tl.PropagateNan.ALL) + + +@triton.jit +def _ssmax_indexer_q_side_kernel( + q_ptr, # bf16 [T, 32, 128], unit stride over the head dim + q_st, + q_sh, + pos_ptr, # int32 [T] + s_ptr, # [32] + s_st, + b_ptr, # [32] + b_st, + cs_ptr, # fp32 [positions, 64]: cos(32) | sin(32), contiguous + w_ptr, # bf16 or fp32 [T, 32], unit stride over the heads + w_st, + weight_scale, # fp32 + qf_ptr, # e4m3 [T, 32, 128], contiguous + wo_ptr, # fp32 [T, 32], contiguous + USE_PDL: tl.constexpr, +): + t = tl.program_id(0).to(tl.int64) + h = tl.arange(0, 32) + i = tl.arange(0, 32) + j = tl.arange(0, 64) + if USE_PDL: + gdc_wait() + # SSMax's scale, `ssmax_scaling`'s chain: the position to fp32, + 1, logf rounded to + # bf16; * s and + b, each rounded to bf16. + pos = tl.load(pos_ptr + t) + lp = _round_bf16(libdevice.log(pos.to(tl.float32) + 1.0)) + s = _round_bf16(tl.load(s_ptr + h * s_st).to(tl.float32)) + b = _round_bf16(tl.load(b_ptr + h * b_st).to(tl.float32)) + scale = _round_bf16(_round_bf16(lp * s) + b)[:, None] + row = q_ptr + t * q_st + h[:, None].to(tl.int64) * q_sh + # The scaled query, rounded to bf16 (SSMax's output); the rope half then enters the + # q side through its subnormal flush. Interleaved pairs: dims 2i and 2i + 1. + xe = _round_bf16_ftz(tl.load(row + 2 * i[None, :]).to(tl.float32) * scale) + xo = _round_bf16_ftz(tl.load(row + 2 * i[None, :] + 1).to(tl.float32) * scale) + xn = _round_bf16(tl.load(row + 64 + j[None, :]).to(tl.float32) * scale) + # The rope: fma(x, cos, (-/+ partner) * sin), rounded to bf16. + tab = cs_ptr + pos.to(tl.int64) * 64 + c = tl.load(tab + i)[None, :] + sn = tl.load(tab + 32 + i)[None, :] + ye = _round_bf16(tl.fma(xe, c, -(xo * sn))) + yo = _round_bf16(tl.fma(xo, c, xe * sn)) + # One power-of-two scale per (token, head) over [rope | nope]: max(amax, 1e-12) / 448 + # rounded up from its exponent bits; its inverse is exact. A NaN in the row (only an + # SSMax overflow makes one) propagates through the amax and the floor, as there. + amax = _max_nan(tl.reduce(tl.abs(ye), 1, _max_nan), tl.reduce(tl.abs(yo), 1, _max_nan)) + amax = _max_nan(_max_nan(amax, tl.reduce(tl.abs(xn), 1, _max_nan)), 1.0e-12) + # fp32(1 / 448), the q side's constant, and one fp32 product. + raw = (amax * tl.full([1], 1.0 / 448.0, tl.float32)).to(tl.int32, bitcast=True) + exp_bits = raw & 0x7F800000 + exp_bits = tl.where((raw & 0x007FFFFF) != 0, exp_bits + 0x00800000, exp_bits) + q_scale = exp_bits.to(tl.float32, bitcast=True) + inv = (0x7F000000 - exp_bits).to(tl.float32, bitcast=True)[:, None] + out = qf_ptr + t * (32 * 128) + h[:, None] * 128 + tl.store(out + 2 * i[None, :], (ye * inv).to(tl.float8e4nv)) + tl.store(out + 2 * i[None, :] + 1, (yo * inv).to(tl.float8e4nv)) + tl.store(out + 64 + j[None, :], (xn * inv).to(tl.float8e4nv)) + # The head weight: two fp32 products, left-associated. + w = tl.load(w_ptr + t * w_st + h).to(tl.float32) + tl.store(wo_ptr + t * 32 + h, (w * q_scale) * weight_scale) + + +class SSMaxIndexerQSideFused(OpWrapper): + """SSMax-scale the indexer query, rope it, pack it to e4m3, fold its scale into the head + weights. + + Computes ``cute_dsl_indexer_q_side_fused(ssmax_scaling(q, positions, s, b), positions, + cos_sin, wq.float(), weight_scale, q_fp8_out, weights_out, is_neox=False)``: `q` is the + unscaled, unrotated ``[T, 32, 128]`` bf16 indexer query, each head ``[pe(64) | + nope(64)]``; `s` and `b` the layer's ``[32]`` SSMax parameters; `cos_sin` the fp32 + ``[positions, 64]`` ``[cos(32) | sin(32)]`` table, rotating interleaved pairs (dims + ``2i``, ``2i + 1`` by column ``i``; an identity table on a NoPE indexer layer); `wq` the + ``[T, 32]`` head weights, bf16 or fp32, any row stride (the targets pass the last 32 + columns of the bf16 ``[T, 160]`` wk | weights_proj output). Writes `q_fp8_out` (e4m3 + ``[T, 32, 128]``) and `weights_out` (fp32 ``[T, 32]``), both contiguous and + caller-owned, and returns them. `q` is not modified. + + Fusion boundary: everything between the `wq_b` GEMM (and the wk | weights_proj GEMM) and + the logits scorer. The projections, and the scorer's reads of the outputs, are the + caller's. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # spec: `tokens` rows; `positions` "spread" (uniform over the table), "tail" (its last 64 + # rows) or "arange" (0..T - 1, or `cached`..`cached` + T - 1 behind a cached prefix); + # `position_sweep` (lo, hi) drives every position lo..hi - 1 in calls of `tokens` rows, + # each on fresh inputs; `table` "yarn" (the NVFP4 checkpoint's indexer table, a + # seeded LeRoPE alpha, cos and sin rounded through bf16) or "identity"; `wq` "kw_bf16" + # (the [T, 160] output's last 32 columns) or "fp32" (contiguous); s and b drawn over + # [s_range] / [b_range] with the four corners on heads 0..3; `q` "normal" is 2 N(0, 1), + # "wide" every bf16 binade from the subnormals up; `graph` captures the call and replays + # it on new inputs. + CELLS: tuple[Cell, ...] = tuple( + Cell( + why=f"decode, graph batch {n:>3}, rope layer: captured, replayed on new inputs", + spec=dict(_ML4, tokens=n, positions="spread", graph=True), + ) + for n in GRAPH_BATCHES + ) + ( + Cell( + why="decode, graph batch 16 at positions in the table's last 64 rows", + spec=dict(_ML4, tokens=16, positions="tail", graph=True), + ), + Cell( + why="decode, graph batch 16, NoPE indexer layer: identity table, captured", + spec=dict(_ML4, tokens=16, positions="spread", graph=True, table="identity"), + ), + Cell( + why="decode, graph batch 256, NoPE indexer layer: identity table, captured", + spec=dict(_ML4, tokens=256, positions="spread", graph=True, table="identity"), + ), + Cell( + why="prefill of 32768 rows at positions 0..32767, rope layer: every table row", + spec=dict(_ML4, tokens=32768, positions="arange", graph=False), + ), + Cell( + why="prefill of 32768 rows, NoPE indexer layer: identity table over 0..32767", + spec=dict(_ML4, tokens=32768, positions="arange", graph=False, table="identity"), + ), + Cell( + why="an odd step of 300 rows, contiguous fp32 head weights", + spec=dict(_ML4, tokens=300, positions="spread", graph=False, wq="fp32"), + ), + Cell( + why="stress at 8192 rows, s in [-1, 1] and b in [-2, 2], q over every bf16 " + "binade: subnormal flushes, zero rows, overflow to the e4m3 range", + spec=dict( + _ML4, + tokens=8192, + positions="spread", + graph=False, + q="wide", + s_range=(-1.0, 1.0), + b_range=(-2.0, 2.0), + ), + ), + ) + + # The 1M config (max_seq_len 1048576, chunked prefill, max_num_tokens 32768, 16 sequences): + # the 1048576-row table, decode rows at any position, a chunk's rows behind a cached prefix, + # and every table row swept in calls of 32768 rows. + CELLS += tuple( + Cell( + why=f"1M decode, graph batch {n:>2}, rope layer: positions spread over the " + "1048576-row table, captured, replayed on new inputs", + spec=dict(_ML4_1M, tokens=n, positions="spread", graph=True), + ) + for n in (1, 2, 3, 4, 8, 13, 16) + ) + CELLS += ( + Cell( + why="1M decode, graph batch 16 at positions in the 1048576-row table's last 64 rows", + spec=dict(_ML4_1M, tokens=16, positions="tail", graph=True), + ), + Cell( + why="1M decode, graph batch 16, NoPE indexer layer: identity table of 1048576 rows", + spec=dict(_ML4_1M, tokens=16, positions="spread", graph=True, table="identity"), + ), + Cell( + why="1M last chunk: 32768 rows at positions 1015808..1048575 behind a " + "1015808-token cached prefix, rope layer", + spec=dict( + _ML4_1M, + tokens=MAX_NUM_TOKENS, + positions="arange", + cached=LONG_MAX_SEQ_LEN - MAX_NUM_TOKENS, + graph=False, + ), + ), + Cell( + why="1M last chunk, NoPE indexer layer: identity table, positions 1015808..1048575", + spec=dict( + _ML4_1M, + tokens=MAX_NUM_TOKENS, + positions="arange", + cached=LONG_MAX_SEQ_LEN - MAX_NUM_TOKENS, + graph=False, + table="identity", + ), + ), + Cell( + why="1M mixed step of 32768 rows at positions spread over 0..1048575", + spec=dict(_ML4_1M, tokens=MAX_NUM_TOKENS, positions="spread", graph=False), + ), + Cell( + why="1M stress at 8192 rows spread over the table, s in [-1, 1] and b in [-2, 2], " + "q over every bf16 binade", + spec=dict( + _ML4_1M, + tokens=8192, + positions="spread", + graph=False, + q="wide", + s_range=(-1.0, 1.0), + b_range=(-2.0, 2.0), + ), + ), + Cell( + why="1M every table row: positions 0..1048575 in 32 calls of 32768 rows, rope layer", + spec=dict( + _ML4_1M, + tokens=MAX_NUM_TOKENS, + positions="arange", + graph=False, + position_sweep=(0, LONG_MAX_SEQ_LEN), + ), + ), + ) + + CELLS += tuple( + Cell( + why=f"DSpark 1M verification: {n} rows at the table tail, {table}, captured", + spec=dict(_ML4_1M, tokens=n, positions="tail", table=table, graph=True), + ) + for n in (4, 52, 64) + for table in ("yarn", "identity") + ) + + note = """ + Call sites: the mistral_large4 NVFP4 targets (tp4, tep4, dep4), every full-indexer layer + (44 per step), decode and prefill, right after the `wq_b` GEMM. Replaces `ssmax_scaling` + (seven torch launches), the head weights' fp32 copy (one) and + `cute_dsl_indexer_q_side_fused` (one) with one launch. + + Numerics: the same operations as the composition it replaces, in the same order, at the + same types, rounded at the same points; codes and weights are bitwise that composition's + over every cell, on tensors captured from a served decode step, and over the NVFP4 + checkpoint's 44 indexer layers (their s and b, the 1048576-row table or the identity) at + every position 0..1048575 (the test runs all three), and bitwise the composed references + of the two entries. Stage by stage (the unfused op, then this kernel): + ssmax_scaling its seven ops (`ssmax_scaling_fused_q`'s note), bf16 output: + the same stages, the same roundings + .float().contiguous() bf16 head weight widened to fp32 (exact): the same widening, + at the load + q side, rope half bf16 to fp32, a subnormal flushed to its signed zero: the + same flush on the widened value (zero exponent field) + q side, nope half bf16 to fp32 (exact): the same + q side, rope fma(x, cos, -(x' * sin)), fma(x', cos, x * sin) in fp32 with + the partner product rounded first: tl.fma, the same operands + q side, rope output rounded to bf16 (round to nearest even): cvt.rn.bf16.f32 + q side, amax max of |.| over the row, NaN propagating: the same max (exact) + q side, scale max(amax, 1e-12) * fp32(1 / 448), rounded up to a power of + two from its exponent bits: the same fp32 product, the same + integer arithmetic + q side, codes e4m3(x * 2^-k), cvt.rn.satfinite: the same conversion + q side, weights (wq * scale) * weight_scale, two fp32 products: the same + Every intermediate rounding is an inline `cvt.rn.bf16.f32` (the compiler folds a bare + bf16 round trip); the kernel is compiled without implicit FMA contraction + (`enable_fp_fusion=False`), so the only fused multiply-adds are the rope's explicit ones, + as in the q side. No fast-math function is used. + + Launch: Triton, grid (T,), one program per token over its 32 heads; programmatic + dependent launch (waits on its predecessor before the first load). No host + synchronization and no allocation: CUDA-graph capturable, a replay on new data writes + what an eager call writes. + + `positions` is read on the device with no range check: a position at or past + `cos_sin.shape[0]` reads past the table (`is_valid` states it). + + Hard limits (sm_103, T <= 32768): token offsets are int64; the table offset is + position * 64 in int64; positions pass through fp32 for the log, exact below 2^24 (at most + 1048575 at the 1M config, its table's last row). None reached. + + Not certified: NeoX rope (no mode here), tables other than the NVFP4 checkpoint's indexer + table and the identity table, positions past 1048575 (tables of more than 1048576 rows), + more than 32768 rows, a q dtype other than bf16. + """ + + def __call__( + self, + q: torch.Tensor, + positions: torch.Tensor, + s: torch.Tensor, + b: torch.Tensor, + cos_sin: torch.Tensor, + wq: torch.Tensor, + weight_scale: float, + q_fp8_out: torch.Tensor, + weights_out: torch.Tensor, + ) -> tuple[torch.Tensor, torch.Tensor]: + tokens = q.shape[0] + if tokens == 0: + return q_fp8_out, weights_out + _ssmax_indexer_q_side_kernel[(tokens,)]( + q, q.stride(0), q.stride(1), positions, s, s.stride(0), b, b.stride(0), cos_sin, + wq, wq.stride(0), float(weight_scale), q_fp8_out, weights_out, + USE_PDL=True, num_warps=4, launch_pdl=True, enable_fp_fusion=False, + ) # fmt: skip + return q_fp8_out, weights_out + + def reference( + self, + q: torch.Tensor, + positions: torch.Tensor, + s: torch.Tensor, + b: torch.Tensor, + cos_sin: torch.Tensor, + wq: torch.Tensor, + weight_scale: float, + q_fp8_out: torch.Tensor, + weights_out: torch.Tensor, + ) -> tuple[torch.Tensor, torch.Tensor]: + """Fresh ``(q_fp8, weights)``: `ssmax_scaling`'s reference chain, then + `cute_dsl_indexer_q_side_fused`'s reference (interleaved) on its bf16 output and the + fp32 head weights; `q_fp8_out` and `weights_out` are not read.""" + scaled = ssmax_scaling.reference(q, positions, s, b) + return cute_dsl_indexer_q_side_fused.reference( + scaled, positions, cos_sin, wq.float(), weight_scale, q_fp8_out, weights_out, + is_neox=False, compiled=None, + ) # fmt: skip + + def is_valid( + self, + q: torch.Tensor, + positions: torch.Tensor, + s: torch.Tensor, + b: torch.Tensor, + cos_sin: torch.Tensor, + wq: torch.Tensor, + weight_scale: float, + q_fp8_out: torch.Tensor, + weights_out: torch.Tensor, + ) -> None: + # Each is addressed through a layout the kernel assumes and answers wrongly otherwise. + name = "ssmax_indexer_q_side_fused" + if q.dtype != torch.bfloat16 or q.dim() != 3 or tuple(q.shape[1:]) != (HEADS, HEAD_DIM): + raise ValueError(f"{name}: q must be bf16 [T, 32, 128], got {tuple(q.shape)}") + tokens = q.shape[0] + if q.stride(2) != 1: + raise ValueError(f"{name}: q needs a unit stride over the head dim") + if tuple(positions.shape) != (tokens,) or positions.dtype != torch.int32: + raise ValueError(f"{name}: positions must be int32 [T] = [{tokens}]") + for pname, t in (("s", s), ("b", b)): + if tuple(t.shape) != (HEADS,): + raise ValueError(f"{name}: {pname} must be [32]") + if cos_sin.dtype != torch.float32 or cos_sin.dim() != 2 or cos_sin.shape[1] != 64: + raise ValueError(f"{name}: cos_sin must be fp32 [positions, 64]") + if not cos_sin.is_contiguous(): + raise ValueError(f"{name}: cos_sin must be contiguous") + if ( + tuple(wq.shape) != (tokens, HEADS) + or wq.dtype not in (torch.bfloat16, torch.float32) + or wq.stride(1) != 1 + ): + raise ValueError(f"{name}: wq must be bf16 or fp32 [T, 32] with a unit head stride") + if ( + q_fp8_out.dtype != torch.float8_e4m3fn + or tuple(q_fp8_out.shape) != (tokens, HEADS, HEAD_DIM) + or not q_fp8_out.is_contiguous() + ): + raise ValueError(f"{name}: q_fp8_out must be contiguous e4m3 [T, 32, 128]") + if ( + weights_out.dtype != torch.float32 + or tuple(weights_out.shape) != (tokens, HEADS) + or not weights_out.is_contiguous() + ): + raise ValueError(f"{name}: weights_out must be contiguous fp32 [T, 32]") + # Positions are read on the device with no bound. + if positions.numel(): + lo, hi = int(positions.min()), int(positions.max()) + if lo < 0 or hi >= cos_sin.shape[0]: + raise ValueError( + f"{name}: position_ids span [{lo}, {hi}] but the rope table has " + f"{cos_sin.shape[0]} rows; the kernel reads past it" + ) + + +ssmax_indexer_q_side_fused = SSMaxIndexerQSideFused() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/ssmax_scaling.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/ssmax_scaling.py new file mode 100644 index 000000000000..856e4cdd702f --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/ssmax_scaling.py @@ -0,0 +1,1155 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""SSMax (scalable-softmax) query scaling, ``q[t, h, :] * (s[h] * log(pos[t] + 1) + b[h])``, +in the reference implementation's bf16 rounding: the MLA query of all 72 layers and the DSA +indexer query of the 44 layers that carry an indexer. The cells cover both Mistral Large 4 +checkpoints' s and b, every decode batch 1..16 a CUDA graph is captured at, and positions up +to the NVFP4 target's max_seq_len of 32768 beside the FP8 target's 9216. The MLA cells drive +one rank's 16 of the 64 heads (tep4), one rank's 8 (tep8 and tp8, which shard attention +alike) and all 64 heads on one rank (dep4, attention data parallel, where every rank holds +every head and the unsliced [64] s and b). At tep4's and tp4's 16 MLA and 32 indexer heads per +rank (tensor parallel 4 at tp4: the same slices as at tep4) they also cover the checkpoint's +own context, max_seq_len 1048576 with chunked prefill: decode rows at positions up to 1048575, +chunk rows behind a cached prefix, packed chunks of several requests beside decode rows, and +every position 0..1048575, on drawn heads and on the checkpoint's own heads whose factor +crosses zero. At tep8's 8 MLA heads per rank (each rank r's [8] slice s[8 r : 8 r + 8], +b[8 r : 8 r + 8]) and the 32 replicated indexer heads they also cover the checkpoint's own +context, max_seq_len 1048576 with chunked prefill: decode rows at positions up to 1048575, +chunk rows behind a cached prefix, mixed and packed steps of several requests' chunks beside +decode rows, and every position 0..1048575, on drawn heads and on the checkpoint's own heads +whose factor crosses zero. At dep4's 64 MLA and 32 indexer heads they also cover the +checkpoint's own context, max_seq_len 1048576 with chunked prefill: decode rows at positions +up to 1048575, chunk rows behind a cached prefix, and a whole 1048576-token prompt as its 32 +chunks, every position 0..1048575. + +Sourcing tier 4, composed from native torch: no engine kernel computes it. +Rejected at the cheaper rungs, each searched on this build: tier 1, ``torch.ops.trtllm`` +registers no op that scales a query per (token, head) by a log-position affine +(``mla_rope_generation`` takes one scalar ``q_scaling``; ``fused_qk_norm_rope`` and +``fused_qk_norm_rope_to_fp8`` carry one scalar ``attention_factor``, nothing per position; +``llama4_fp8_bf16_gemm`` folds Llama-4's attention temperature, one scalar with a floor of +the position, into one FP8 GEMM); tier 2, no callable under ``tensorrt_llm/_torch/modules`` +or ``_torch/attention`` (the nearest relative, ``MLA._attention_scaling`` in +``_torch/attention/mla.py``, is that same Llama-4 temperature written inline in fp32, with no +per-head terms); tier 3, FlashInfer's ``rope`` module only rotates, and no FlashInfer callable +carries a per-token query scale. Correct and slow by design: seven elementwise launches, +which a tuning pass may fold into the rope or quantization kernel beside it. +""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: Per-rank query heads at tep4: the MLA's 64 split four ways, the indexer's 32 replicated. +#: At dep4 (attention data parallel) every rank holds all MLA_PARAM_HEADS of the MLA; at +#: tep8 and tp8 each of the eight ranks holds MLA_TEP8_HEADS of them. +MLA_HEADS = 16 +MLA_TEP8_HEADS = 8 +TEP8_RANKS = 8 +MLA_PARAM_HEADS = 64 +INDEXER_HEADS = 32 +HEAD_DIM = 128 + +#: max_seq_len, which bounds every position: the NVFP4 targets serve 32768 (max_num_tokens +#: 32768 per rank, no chunked prefill, so one context step holds a whole 32768-token +#: prompt); the FP8 target's 9216 cells are kept as they were. +MAX_SEQ_LEN = 32768 +FP8_MAX_SEQ_LEN = 9216 + +#: The checkpoint's own context (max_position_embeddings), which the tep4, tp4, dep4 and tep8 +#: deployments serve with chunked prefill: a step still carries at most MAX_NUM_TOKENS rows per +#: rank, a chunk's rows sit at positions cached .. cached + n - 1 behind its request's cached +#: prefix, and decode rows reach position LONG_MAX_SEQ_LEN - 1, where log(pos + 1) is 13.86 +#: and a bf16 step of it is 0.0625. +LONG_MAX_SEQ_LEN = 1048576 +MAX_NUM_TOKENS = 32768 + +#: MLA layer 0's s and b on tp4 rank 1 (heads 16..31 of 64), as the NVFP4 checkpoint stores +#: them (bf16, read on disk). Every s is negative, and two heads' factors cross zero below +#: LONG_MAX_SEQ_LEN, the only two of the checkpoint's 72 x 64 MLA heads that do: local head 4 +#: (head 20, s -0.0693359, b 0.914063) near position 531308 and local head 10 (head 26, +#: s -0.078125, b 0.90625) near 109097. No indexer head crosses below 1048576. +NVFP4_MLA_LAYER0_RANK1 = ( + ( + -0.0135498046875, -0.036865234375, -0.0289306640625, -0.05322265625, + -0.0693359375, -0.039306640625, -0.0106201171875, -0.026123046875, + -0.025390625, -0.052490234375, -0.078125, -0.04052734375, + -0.0400390625, -0.0322265625, -0.0301513671875, -0.05810546875, + ), + ( + 0.9609375, 0.9453125, 0.9453125, 0.93359375, 0.9140625, 0.9453125, 0.96875, 0.96875, + 0.94921875, 0.921875, 0.90625, 0.9609375, 0.953125, 0.95703125, 0.9453125, 0.92578125, + ), +) # fmt: skip + +#: MLA layer 0's s and b on tep8 ranks 2 and 3 (heads 16..23 and 24..31 of 64, the two halves +#: of NVFP4_MLA_LAYER0_RANK1), as the NVFP4 checkpoint stores them (bf16, read on disk). Every +#: s is negative, and two heads' factors cross zero below LONG_MAX_SEQ_LEN, the only two of +#: the checkpoint's 72 x 64 MLA heads that do, one on each slice: rank 2's local head 4 (head +#: 20, s -0.0693359, b 0.914063) near position 531308 and rank 3's local head 2 (head 26, +#: s -0.078125, b 0.90625) near 109097. No indexer head crosses below 1048576. +NVFP4_MLA_LAYER0_TEP8_RANK2 = ( + ( + -0.0135498046875, -0.036865234375, -0.0289306640625, -0.05322265625, + -0.0693359375, -0.039306640625, -0.0106201171875, -0.026123046875, + ), + (0.9609375, 0.9453125, 0.9453125, 0.93359375, 0.9140625, 0.9453125, 0.96875, 0.96875), +) # fmt: skip +NVFP4_MLA_LAYER0_TEP8_RANK3 = ( + ( + -0.025390625, -0.052490234375, -0.078125, -0.04052734375, + -0.0400390625, -0.0322265625, -0.0301513671875, -0.05810546875, + ), + (0.94921875, 0.921875, 0.90625, 0.9609375, 0.953125, 0.95703125, 0.9453125, 0.92578125), +) # fmt: skip + +#: Decode batches a CUDA graph is captured at: the gates' padded 1/2/4/8/16, then the LLM +#: API default's unpadded batches between them (the engine cuts its 1..32 to 1..16 at +#: max_batch_size 16). On the MLA at tep4, each batch drives one rank's slice of the 64 +#: heads; at dep4, a rank's own batch drives all 64. +GRAPH_BATCHES_PADDED = ((1, 0), (2, 1), (4, 2), (8, 3), (16, 3)) +GRAPH_BATCHES_UNPADDED = tuple((n, n % 4) for n in (3, 5, 6, 7, 9, 10, 11, 12, 13, 14, 15)) + +#: The draw ranges hold both Mistral Large 4 checkpoints' s and b over every layer that carries +#: them. NVFP4 (read on disk, bf16): MLA (72 layers x 64 heads) s in [-0.078125, 0.574219], +#: b in [0.902344, 1.57031]; indexer (44 layers x 32 heads) s in [-0.0437012, 0.0664062], +#: b in [0.976562, 1.04688]. FP8, rounded outward: MLA s in [-0.0767, 0.574], b in +#: [0.902, 1.578]; indexer s in [-0.0486, 0.0703], b in [0.973, 1.055]. +_BF16 = dict( + head_dim=HEAD_DIM, dtype=torch.bfloat16, positions_dtype=torch.int32, q="normal", corners=False +) +_MLA = dict( + _BF16, + site="mla", + heads=MLA_HEADS, + param_heads=MLA_PARAM_HEADS, + s_range=(-0.08, 0.58), + b_range=(0.9, 1.6), +) +_INDEXER = dict( + _BF16, + site="indexer", + heads=INDEXER_HEADS, + param_heads=INDEXER_HEADS, + rank=0, + s_range=(-0.05, 0.075), + b_range=(0.97, 1.06), +) +#: The MLA at dep4: rank 0 of a 64-head draw holding all 64, so s and b are the whole +#: parameter, with the four corners on heads 0..3. +_MLA_DEP4 = dict( + _MLA, heads=MLA_PARAM_HEADS, rank=0, max_seq_len=MAX_SEQ_LEN, corners=True, graph=False +) +#: The MLA at tep8 and tp8: rank r's 8 heads, 8 r .. 8 r + 7 of a 64-head draw, so s and b are views +#: at offset 8 r into the parameter, with the four corners on four of the slice's heads. +_MLA_TEP8 = dict(_MLA, heads=MLA_TEP8_HEADS, max_seq_len=MAX_SEQ_LEN, corners=True) +#: dep4 at the checkpoint's own context: the MLA's 64 heads as above, and the indexer's 32 +#: (replicated, the same at every parallel layout), with the corners, at max_seq_len 1048576. +_MLA_DEP4_1M = dict(_MLA_DEP4, max_seq_len=LONG_MAX_SEQ_LEN) +_INDEXER_1M = dict(_INDEXER, max_seq_len=LONG_MAX_SEQ_LEN, corners=True, graph=False) +#: The same tep8 slices at the checkpoint's own context, max_seq_len 1048576. +_MLA_TEP8_1M = dict(_MLA_TEP8, max_seq_len=LONG_MAX_SEQ_LEN) + + +class SSMaxScaling(OpWrapper): + """``q * (s * log(positions + 1) + b)`` per (token, head), a new tensor in `q`'s dtype. + + `q` is ``[T, H, D]``, `positions` the step's ``[T]`` position ids (int32, as the engine + hands them over), `s` and `b` ``[H]``: this rank's slice of the MLA's 64 per-head + parameters (heads ``16 r .. 16 r + 15`` on rank ``r`` at tep4, ``8 r .. 8 r + 7`` at + tep8 and tp8; all 64, unsliced, under attention data parallel at dep4), or the indexer's + 32, replicated. + + The rounding is the specification, not an implementation detail: the log is taken in + fp32 and rounded to `q`'s dtype, then ``* s`` and ``+ b`` are each rounded to it, then + ``q * scale`` is. Those are the reference implementation's unfused steps, which its + fused kernels reproduce bit for bit, and these are the same torch calls in the same + order. An fp32 scale with no intermediate rounding is closer to the closed form and is + not this op (the test measures how far apart they land). + + Fusion boundary: the log of the positions and the scaling. Everything around the query + is the caller's. The reference implementation scales the MLA query after its rope (on + the rope's bf16 output; on NoPE layers, the whole 128-wide query), and the indexer + query before its rope and fp8 quantization. A caller whose rope runs inside a later op + gets the same product with the rounding at a different step. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # spec: q is [tokens, heads, head_dim]; positions are each context's 0..n-1 in + # `contexts` order, then `decode_rows` rows at random positions in 1..max_seq_len - 1 + # (the first at max_seq_len - 1). s and b are drawn over [s_range] / [b_range] for + # `param_heads` heads and sliced to rank `rank`'s `heads`, so a rank > 0 slice is a + # view at an offset. `corners` pins four of the slice's heads, (4 * rank) % heads .. + # + 3 (4 * rank .. 4 * rank + 3 at 16 heads: no two ranks' at the same heads; at 8 + # heads, the first four on even ranks and the last four on odd ones), to the four + # (s, b) corners of the ranges: the largest factor and the deepest cancellation the + # ranges allow at the cell's positions. + # `q` "normal" is 4 * N(0, 1); "wide" draws every bf16 binade, subnormals included. + # `graph` captures the call in a CUDA graph and replays it on new q and positions + # written into the captured inputs. `cached` (one per context, 0 when absent) starts + # each context at its cached prefix, at positions cached .. cached + n - 1, as a chunk + # after a request's first runs. `position_sweep` (lo, hi) drives every position + # lo .. hi - 1, `tokens` rows per call, each call on a fresh q. `slice_values` (s, b) + # replaces the draw on the rank's slice with those per-head values. `chunks` drives a + # whole prompt of chunks * tokens positions as that many calls of `tokens` rows, chunk k + # at positions k * tokens onward, each call on a fresh q. + CELLS: tuple[Cell, ...] = ( + tuple( + Cell( + why=f"MLA decode, graph batch {n:>2}, rank {r}'s heads: " + "captured, replayed on new q and positions", + spec=dict( + _MLA, + max_seq_len=FP8_MAX_SEQ_LEN, + rank=r, + tokens=n, + contexts=(), + decode_rows=n, + graph=True, + ), + ) + for n, r in GRAPH_BATCHES_PADDED + GRAPH_BATCHES_UNPADDED + ) + + tuple( + Cell( + why=f"indexer decode, graph batch {n:>2}: " + "captured, replayed on new q and positions", + spec=dict( + _INDEXER, + max_seq_len=FP8_MAX_SEQ_LEN, + tokens=n, + contexts=(), + decode_rows=n, + graph=True, + ), + ) + for n, _ in GRAPH_BATCHES_PADDED + GRAPH_BATCHES_UNPADDED + ) + + ( + Cell( + why="MLA context of 9216 rows at positions 0..9215: " + "every position the FP8 target's envelope holds, rank 1's heads", + spec=dict( + _MLA, + max_seq_len=FP8_MAX_SEQ_LEN, + rank=1, + tokens=9216, + contexts=(9216,), + decode_rows=0, + graph=False, + ), + ), + Cell( + why="indexer context of 9216 rows at positions 0..9215: " + "every position the FP8 target's envelope holds", + spec=dict( + _INDEXER, + max_seq_len=FP8_MAX_SEQ_LEN, + tokens=9216, + contexts=(9216,), + decode_rows=0, + graph=False, + ), + ), + Cell( + why="MLA mixed step, 3 contexts + 13 decode rows (16 requests, 9216 rows): " + "positions are not the row index", + spec=dict( + _MLA, + max_seq_len=FP8_MAX_SEQ_LEN, + rank=2, + tokens=9216, + contexts=(5000, 3000, 1203), + decode_rows=13, + graph=False, + ), + ), + Cell( + why="stress, s in [-1, 1] and b in [-2, 2], q over every bf16 binade: " + "cancellation, overflow, subnormals", + spec=dict( + _MLA, + max_seq_len=FP8_MAX_SEQ_LEN, + rank=3, + s_range=(-1.0, 1.0), + b_range=(-2.0, 2.0), + q="wide", + tokens=9216, + contexts=(9216,), + decode_rows=0, + graph=False, + ), + ), + ) + # The NVFP4 target's envelope, max_seq_len = max_num_tokens = 32768 with no chunked + # prefill: positions up to 32767 (log 10.4, so the MLA factor reaches 7.53 at the + # checkpoint's largest s and b), one context step of up to 32768 rows. + + tuple( + Cell( + why=f"MLA decode 32K, graph batch {n:>2}, rank {r}'s heads: positions up to " + "32767, the range corners, captured, replayed on new q and positions", + spec=dict( + _MLA, + max_seq_len=MAX_SEQ_LEN, + corners=True, + rank=r, + tokens=n, + contexts=(), + decode_rows=n, + graph=True, + ), + ) + for n, r in GRAPH_BATCHES_PADDED + GRAPH_BATCHES_UNPADDED + ) + + tuple( + Cell( + why=f"indexer decode 32K, graph batch {n:>2}: positions up to 32767, " + "the range corners, captured, replayed on new q and positions", + spec=dict( + _INDEXER, + max_seq_len=MAX_SEQ_LEN, + corners=True, + tokens=n, + contexts=(), + decode_rows=n, + graph=True, + ), + ) + for n, _ in GRAPH_BATCHES_PADDED + GRAPH_BATCHES_UNPADDED + ) + + ( + Cell( + why="MLA context of 32768 rows at positions 0..32767: every position the " + "NVFP4 target's envelope holds, one whole-prompt step, rank 1's heads, the corners", + spec=dict( + _MLA, + max_seq_len=MAX_SEQ_LEN, + corners=True, + rank=1, + tokens=32768, + contexts=(32768,), + decode_rows=0, + graph=False, + ), + ), + Cell( + why="indexer context of 32768 rows at positions 0..32767: every position " + "the NVFP4 target's envelope holds, one whole-prompt step, the corners", + spec=dict( + _INDEXER, + max_seq_len=MAX_SEQ_LEN, + corners=True, + tokens=32768, + contexts=(32768,), + decode_rows=0, + graph=False, + ), + ), + Cell( + why="MLA mixed step of 32768 rows, 3 contexts + 13 decode rows (16 requests, " + "max_num_tokens): positions are not the row index, decode rows up to 32767", + spec=dict( + _MLA, + max_seq_len=MAX_SEQ_LEN, + corners=True, + rank=2, + tokens=32768, + contexts=(17000, 11000, 4755), + decode_rows=13, + graph=False, + ), + ), + Cell( + why="stress at 32768 rows, s in [-1, 1] and b in [-2, 2], q over every bf16 " + "binade, positions 0..32767: cancellation, overflow, subnormals", + spec=dict( + _MLA, + max_seq_len=MAX_SEQ_LEN, + corners=True, + rank=3, + s_range=(-1.0, 1.0), + b_range=(-2.0, 2.0), + q="wide", + tokens=32768, + contexts=(32768,), + decode_rows=0, + graph=False, + ), + ), + ) + # dep4, attention data parallel: each rank runs its own requests with all 64 MLA heads + # and the unsliced [64] s and b, over the same per-rank envelope (positions up to + # 32767, steps of up to 32768 rows, decode batches 1..16 captured). + + tuple( + Cell( + why=f"MLA dep4 decode 32K, graph batch {n:>2}, all 64 heads: positions up to " + "32767, the range corners, captured, replayed on new q and positions", + spec=dict(_MLA_DEP4, tokens=n, contexts=(), decode_rows=n, graph=True), + ) + for n, _ in GRAPH_BATCHES_PADDED + GRAPH_BATCHES_UNPADDED + ) + + ( + Cell( + why="MLA dep4 context of 32768 rows, all 64 heads, positions 0..32767: every " + "position of a rank's envelope in one whole-prompt step, the corners", + spec=dict(_MLA_DEP4, tokens=32768, contexts=(32768,), decode_rows=0), + ), + Cell( + why="MLA dep4 mixed step of 32768 rows, all 64 heads, 3 contexts + 13 decode " + "rows (16 requests, max_num_tokens): positions are not the row index", + spec=dict(_MLA_DEP4, tokens=32768, contexts=(17000, 11000, 4755), decode_rows=13), + ), + # `sweep_rows` (lo, hi, step) ranges and `sweep_pow2` (2^k - 1, 2^k, 2^k + 1) + # name step sizes T, each driven as its own call on the context's leading T rows + # and held bitwise to them; `large` puts a cell past the pre-merge budget. + Cell( + why="MLA dep4 step sizes T 1..4096, the top 1024 and a stride of 61 between, " + "all 64 heads: each T bitwise the leading T rows of the 32768-row context " + "call, every last-block tail of every launch", + spec=dict( + _MLA_DEP4, + tokens=32768, + contexts=(32768,), + decode_rows=0, + sweep_rows=((1, 4096, 1), (4097, 31744, 61), (31745, 32768, 1)), + sweep_pow2=True, + ), + ), + Cell( + why="MLA dep4 every step size T 1..32768, all 64 heads (large, ~14 s): each T " + "bitwise the leading T rows of the 32768-row context call", + spec=dict( + _MLA_DEP4, + tokens=32768, + contexts=(32768,), + decode_rows=0, + sweep_rows=((1, 32768, 1),), + large=True, + ), + ), + ) + # tep8 and tp8 (the "MLA tep8" cells; the two targets shard attention alike), the 64 + # heads over eight ranks with attention data parallel off: each rank holds 8 heads and + # the [8] slices s[8 r : 8 r + 8], b[8 r : 8 r + 8], over the same per-rank envelope + # (positions up to 32767, steps of up to 32768 rows, decode batches 1..16 captured). + # The graph batches cycle the rank over 0..7, so every slice is driven. + + tuple( + Cell( + why=f"MLA tep8 decode 32K, graph batch {n:>2}, rank {n % TEP8_RANKS}'s 8 heads: " + "positions up to 32767, the range corners, captured, replayed on new q and " + "positions", + spec=dict( + _MLA_TEP8, + rank=n % TEP8_RANKS, + tokens=n, + contexts=(), + decode_rows=n, + graph=True, + ), + ) + for n in range(1, 17) + ) + + ( + Cell( + why="MLA tep8 context of 32768 rows at positions 0..32767, rank 1's 8 heads: " + "every position of a rank's envelope in one whole-prompt step, the corners", + spec=dict( + _MLA_TEP8, rank=1, tokens=32768, contexts=(32768,), decode_rows=0, graph=False + ), + ), + Cell( + why="MLA tep8 mixed step of 32768 rows, rank 2's 8 heads, 3 contexts + 13 " + "decode rows (16 requests, max_num_tokens): positions are not the row index", + spec=dict( + _MLA_TEP8, + rank=2, + tokens=32768, + contexts=(17000, 11000, 4755), + decode_rows=13, + graph=False, + ), + ), + Cell( + why="MLA tep8 last slice, rank 7's heads 56..63, a 32768-row context at " + "positions 0..32767, the corners on heads 60..63; step sizes T 1..4096, the " + "top 1024 and a stride of 61 between, each bitwise the leading T rows of that " + "call: every last-block tail of every launch at 8 heads", + spec=dict( + _MLA_TEP8, + rank=7, + tokens=32768, + contexts=(32768,), + decode_rows=0, + graph=False, + sweep_rows=((1, 4096, 1), (4097, 31744, 61), (31745, 32768, 1)), + sweep_pow2=True, + ), + ), + ) + # tep4 and tp4 at the checkpoint's own context, max_seq_len 1048576 with chunked + # prefill (max_num_tokens 32768), on the 16 MLA heads of a rank and the 32 replicated + # indexer heads: decode rows up to position 1048575 (log 13.86, so the MLA factor + # reaches 9.53 at the checkpoint's largest s and b), and steps of up to 32768 rows + # whose contexts start at their cached prefix. + + tuple( + Cell( + why=f"MLA decode 1M, graph batch {n:>2}, rank {r}'s heads: positions up to " + "1048575, the range corners, captured, replayed on new q and positions", + spec=dict( + _MLA, + max_seq_len=LONG_MAX_SEQ_LEN, + corners=True, + rank=r, + tokens=n, + contexts=(), + decode_rows=n, + graph=True, + ), + ) + for n, r in GRAPH_BATCHES_PADDED + GRAPH_BATCHES_UNPADDED + ) + + tuple( + Cell( + why=f"indexer decode 1M, graph batch {n:>2}: positions up to 1048575, " + "the range corners, captured, replayed on new q and positions", + spec=dict( + _INDEXER, + max_seq_len=LONG_MAX_SEQ_LEN, + corners=True, + tokens=n, + contexts=(), + decode_rows=n, + graph=True, + ), + ) + for n, _ in GRAPH_BATCHES_PADDED + GRAPH_BATCHES_UNPADDED + ) + + ( + Cell( + why="MLA 1M last chunk, 32768 rows behind a 1015808-token cached prefix: " + "positions 1015808..1048575, rank 1's heads, the corners", + spec=dict( + _MLA, + max_seq_len=LONG_MAX_SEQ_LEN, + corners=True, + rank=1, + tokens=MAX_NUM_TOKENS, + contexts=(MAX_NUM_TOKENS,), + cached=(LONG_MAX_SEQ_LEN - MAX_NUM_TOKENS,), + decode_rows=0, + graph=False, + ), + ), + Cell( + why="indexer 1M last chunk, 32768 rows behind a 1015808-token cached prefix: " + "positions 1015808..1048575, the corners", + spec=dict( + _INDEXER, + max_seq_len=LONG_MAX_SEQ_LEN, + corners=True, + tokens=MAX_NUM_TOKENS, + contexts=(MAX_NUM_TOKENS,), + cached=(LONG_MAX_SEQ_LEN - MAX_NUM_TOKENS,), + decode_rows=0, + graph=False, + ), + ), + Cell( + why="MLA 1M mixed step of 32768 rows, 3 chunk tails + 13 decode rows: chunks " + "at positions 1031576.., 500000.. and 0.., decode rows up to 1048575", + spec=dict( + _MLA, + max_seq_len=LONG_MAX_SEQ_LEN, + corners=True, + rank=2, + tokens=MAX_NUM_TOKENS, + contexts=(17000, 11000, 4755), + cached=(LONG_MAX_SEQ_LEN - 17000, 500000, 0), + decode_rows=13, + graph=False, + ), + ), + Cell( + why="indexer 1M mixed step of 32768 rows, 3 chunk tails + 13 decode rows: " + "chunks at positions 1031576.., 500000.. and 0.., decode rows up to 1048575", + spec=dict( + _INDEXER, + max_seq_len=LONG_MAX_SEQ_LEN, + corners=True, + tokens=MAX_NUM_TOKENS, + contexts=(17000, 11000, 4755), + cached=(LONG_MAX_SEQ_LEN - 17000, 500000, 0), + decode_rows=13, + graph=False, + ), + ), + Cell( + why="MLA 1M packed step of 32768 rows, 4 chunks + 13 decode rows: chunks of " + "16384, 8192, 4096 and 4083 rows at positions 1032192.., 524288.., 98304.. " + "and 0.., rank 3's heads, decode rows up to 1048575", + spec=dict( + _MLA, + max_seq_len=LONG_MAX_SEQ_LEN, + corners=True, + rank=3, + tokens=MAX_NUM_TOKENS, + contexts=(16384, 8192, 4096, 4083), + cached=(LONG_MAX_SEQ_LEN - 16384, 524288, 98304, 0), + decode_rows=13, + graph=False, + ), + ), + Cell( + why="indexer 1M packed step of 32768 rows, 4 chunks + 13 decode rows: chunks " + "of 16384, 8192, 4096 and 4083 rows at positions 1032192.., 524288.., " + "98304.. and 0.., decode rows up to 1048575", + spec=dict( + _INDEXER, + max_seq_len=LONG_MAX_SEQ_LEN, + corners=True, + tokens=MAX_NUM_TOKENS, + contexts=(16384, 8192, 4096, 4083), + cached=(LONG_MAX_SEQ_LEN - 16384, 524288, 98304, 0), + decode_rows=13, + graph=False, + ), + ), + Cell( + why="stress at the 1M last chunk, s in [-1, 1] and b in [-2, 2], q over every " + "bf16 binade, positions 1015808..1048575: cancellation, overflow, subnormals", + spec=dict( + _MLA, + max_seq_len=LONG_MAX_SEQ_LEN, + corners=True, + rank=3, + s_range=(-1.0, 1.0), + b_range=(-2.0, 2.0), + q="wide", + tokens=MAX_NUM_TOKENS, + contexts=(MAX_NUM_TOKENS,), + cached=(LONG_MAX_SEQ_LEN - MAX_NUM_TOKENS,), + decode_rows=0, + graph=False, + ), + ), + Cell( + why="MLA every position 0..1048575 in 32 chunk calls of 32768 rows, rank 0's " + "heads, the corners: the device log rounds to the correctly rounded log's " + "bf16 at every position below 2^20", + spec=dict( + _MLA, + max_seq_len=LONG_MAX_SEQ_LEN, + corners=True, + rank=0, + tokens=MAX_NUM_TOKENS, + contexts=(MAX_NUM_TOKENS,), + decode_rows=0, + graph=False, + position_sweep=(0, LONG_MAX_SEQ_LEN), + ), + ), + Cell( + why="indexer every position 0..1048575 in 32 chunk calls of 32768 rows, the " + "corners: the device log rounds to the correctly rounded log's bf16 at every " + "position below 2^20", + spec=dict( + _INDEXER, + max_seq_len=LONG_MAX_SEQ_LEN, + corners=True, + tokens=MAX_NUM_TOKENS, + contexts=(MAX_NUM_TOKENS,), + decode_rows=0, + graph=False, + position_sweep=(0, LONG_MAX_SEQ_LEN), + ), + ), + Cell( + why="MLA layer 0 on rank 1, the NVFP4 checkpoint's own s and b, every position " + "0..1048575 in 32 chunk calls: two heads' factors cross zero and end negative", + spec=dict( + _MLA, + max_seq_len=LONG_MAX_SEQ_LEN, + rank=1, + tokens=MAX_NUM_TOKENS, + contexts=(MAX_NUM_TOKENS,), + decode_rows=0, + graph=False, + position_sweep=(0, LONG_MAX_SEQ_LEN), + slice_values=NVFP4_MLA_LAYER0_RANK1, + ), + ), + ) + # dep4 at the checkpoint's own context, max_seq_len 1048576 with chunked prefill + # (max_num_tokens 32768 per rank): all 64 MLA heads with the unsliced [64] s and b, and + # the indexer's 32, at decode rows up to position 1048575 (log 13.86, so the MLA factor + # reaches 9.53 at the checkpoint's largest s and b) and steps of up to 32768 rows whose + # contexts start at their cached prefix. The indexer's 32 heads are replicated, the same + # at every parallel layout, so dep4's indexer decode, last-chunk and mixed-step cells + # are the tep4 and tp4 ones above (the same why and spec). + + tuple( + Cell( + why=f"MLA dep4 decode 1M, graph batch {n:>2}, all 64 heads: positions up to " + "1048575, the range corners, captured, replayed on new q and positions", + spec=dict(_MLA_DEP4_1M, tokens=n, contexts=(), decode_rows=n, graph=True), + ) + for n, _ in GRAPH_BATCHES_PADDED + GRAPH_BATCHES_UNPADDED + ) + + ( + Cell( + why="MLA dep4 1M last chunk, 32768 rows behind a 1015808-token cached prefix, " + "all 64 heads: positions 1015808..1048575, the corners", + spec=dict( + _MLA_DEP4_1M, + tokens=MAX_NUM_TOKENS, + contexts=(MAX_NUM_TOKENS,), + cached=(LONG_MAX_SEQ_LEN - MAX_NUM_TOKENS,), + decode_rows=0, + ), + ), + Cell( + why="MLA dep4 1M mixed step of 32768 rows, all 64 heads: a 32755-row chunk at " + "positions 1015808..1048562 behind its cached prefix, 13 decode rows up to 1048575", + spec=dict( + _MLA_DEP4_1M, + tokens=MAX_NUM_TOKENS, + contexts=(MAX_NUM_TOKENS - 13,), + cached=(LONG_MAX_SEQ_LEN - MAX_NUM_TOKENS,), + decode_rows=13, + ), + ), + Cell( + why="MLA dep4 1M mixed step, 3 chunk tails + 13 decode rows (32768 rows, all 64 " + "heads): chunks at positions 1031576.., 500000.. and 0.., decode rows up to " + "1048575", + spec=dict( + _MLA_DEP4_1M, + tokens=MAX_NUM_TOKENS, + contexts=(17000, 11000, 4755), + cached=(LONG_MAX_SEQ_LEN - 17000, 500000, 0), + decode_rows=13, + ), + ), + Cell( + why="stress at the dep4 1M last chunk, all 64 heads, s in [-1, 1] and b in " + "[-2, 2], q over every bf16 binade, positions 1015808..1048575: cancellation, " + "overflow, subnormals", + spec=dict( + _MLA_DEP4_1M, + s_range=(-1.0, 1.0), + b_range=(-2.0, 2.0), + q="wide", + tokens=MAX_NUM_TOKENS, + contexts=(MAX_NUM_TOKENS,), + cached=(LONG_MAX_SEQ_LEN - MAX_NUM_TOKENS,), + decode_rows=0, + ), + ), + Cell( + why="MLA dep4 1M prompt as its 32 chunks of 32768 rows, all 64 heads, the " + "corners: every position 0..1048575, the device log rounding to the correctly " + "rounded log's bf16 at each", + spec=dict( + _MLA_DEP4_1M, + tokens=MAX_NUM_TOKENS, + contexts=(MAX_NUM_TOKENS,), + chunks=LONG_MAX_SEQ_LEN // MAX_NUM_TOKENS, + decode_rows=0, + ), + ), + Cell( + why="indexer 1M prompt as its 32 chunks of 32768 rows, the corners: every " + "position 0..1048575, the device log rounding to the correctly rounded log's " + "bf16 at each", + spec=dict( + _INDEXER_1M, + tokens=MAX_NUM_TOKENS, + contexts=(MAX_NUM_TOKENS,), + chunks=LONG_MAX_SEQ_LEN // MAX_NUM_TOKENS, + decode_rows=0, + ), + ), + ) + # tep8 at the checkpoint's own context, max_seq_len 1048576 with chunked prefill + # (max_num_tokens 32768), on a rank's 8 MLA heads (rank r's [8] slices of s and b) and + # the 32 replicated indexer heads: decode rows up to position 1048575 (log 13.86, so + # the MLA factor reaches 9.53 at the checkpoint's largest s and b), and steps of up to + # 32768 rows whose contexts start at their cached prefix. The graph batches cycle the + # rank over 0..7, so every slice is driven at decode. The indexer's 32 heads are + # replicated, the same at every parallel layout, so tep8's indexer decode, last-chunk, + # mixed-step, packed-step and every-position cells are the tep4 and tp4 ones above (the + # same why and spec). + + tuple( + Cell( + why=f"MLA tep8 decode 1M, graph batch {n:>2}, rank {n % TEP8_RANKS}'s 8 heads: " + "positions up to 1048575, the range corners, captured, replayed on new q and " + "positions", + spec=dict( + _MLA_TEP8_1M, + rank=n % TEP8_RANKS, + tokens=n, + contexts=(), + decode_rows=n, + graph=True, + ), + ) + for n in range(1, 17) + ) + + ( + Cell( + why="MLA tep8 1M last chunk, 32768 rows behind a 1015808-token cached prefix: " + "positions 1015808..1048575, rank 1's 8 heads, the corners", + spec=dict( + _MLA_TEP8_1M, + rank=1, + tokens=MAX_NUM_TOKENS, + contexts=(MAX_NUM_TOKENS,), + cached=(LONG_MAX_SEQ_LEN - MAX_NUM_TOKENS,), + decode_rows=0, + graph=False, + ), + ), + Cell( + why="MLA tep8 1M mixed step of 32768 rows, rank 2's 8 heads, 3 chunk tails + 13 " + "decode rows: chunks at positions 1031576.., 500000.. and 0.., decode rows up " + "to 1048575", + spec=dict( + _MLA_TEP8_1M, + rank=2, + tokens=MAX_NUM_TOKENS, + contexts=(17000, 11000, 4755), + cached=(LONG_MAX_SEQ_LEN - 17000, 500000, 0), + decode_rows=13, + graph=False, + ), + ), + Cell( + why="MLA tep8 1M packed step of 32768 rows, rank 3's 8 heads, 4 chunks + 13 " + "decode rows: chunks of 16384, 8192, 4096 and 4083 rows at positions " + "1032192.., 524288.., 98304.. and 0.., decode rows up to 1048575", + spec=dict( + _MLA_TEP8_1M, + rank=3, + tokens=MAX_NUM_TOKENS, + contexts=(16384, 8192, 4096, 4083), + cached=(LONG_MAX_SEQ_LEN - 16384, 524288, 98304, 0), + decode_rows=13, + graph=False, + ), + ), + Cell( + why="MLA tep8 stress at the 1M last chunk, rank 5's 8 heads, s in [-1, 1] and b " + "in [-2, 2], q over every bf16 binade, positions 1015808..1048575: " + "cancellation, overflow, subnormals", + spec=dict( + _MLA_TEP8_1M, + rank=5, + s_range=(-1.0, 1.0), + b_range=(-2.0, 2.0), + q="wide", + tokens=MAX_NUM_TOKENS, + contexts=(MAX_NUM_TOKENS,), + cached=(LONG_MAX_SEQ_LEN - MAX_NUM_TOKENS,), + decode_rows=0, + graph=False, + ), + ), + Cell( + why="MLA tep8 every position 0..1048575 in 32 chunk calls of 32768 rows, rank " + "7's 8 heads, the corners: the device log rounds to the correctly rounded " + "log's bf16 at every position below 2^20", + spec=dict( + _MLA_TEP8_1M, + rank=7, + tokens=MAX_NUM_TOKENS, + contexts=(MAX_NUM_TOKENS,), + decode_rows=0, + graph=False, + position_sweep=(0, LONG_MAX_SEQ_LEN), + ), + ), + Cell( + why="MLA tep8 layer 0 on rank 2, the NVFP4 checkpoint's own s and b on heads " + "16..23, every position 0..1048575 in 32 chunk calls: head 20 (local head 4) " + "crosses zero and ends negative", + spec=dict( + _MLA_TEP8_1M, + corners=False, + rank=2, + tokens=MAX_NUM_TOKENS, + contexts=(MAX_NUM_TOKENS,), + decode_rows=0, + graph=False, + position_sweep=(0, LONG_MAX_SEQ_LEN), + slice_values=NVFP4_MLA_LAYER0_TEP8_RANK2, + ), + ), + Cell( + why="MLA tep8 layer 0 on rank 3, the NVFP4 checkpoint's own s and b on heads " + "24..31, every position 0..1048575 in 32 chunk calls: head 26 (local head 2) " + "crosses zero and ends negative", + spec=dict( + _MLA_TEP8_1M, + corners=False, + rank=3, + tokens=MAX_NUM_TOKENS, + contexts=(MAX_NUM_TOKENS,), + decode_rows=0, + graph=False, + position_sweep=(0, LONG_MAX_SEQ_LEN), + slice_values=NVFP4_MLA_LAYER0_TEP8_RANK3, + ), + ), + ) + ) + + note = """ + Metadata consumed: the step's position ids, flattened to [T] (`position_ids`); nothing + else. Out-of-place: `q`, `positions`, `s` and `b` are not mutated. Seven launches: + positions to fp32, + 1, log, round to q's dtype, * s, + b, q * scale. `s` and `b` pass + through `.to(q.dtype)`, which is free when they are already bf16 (the checkpoint's + dtype); an fp32 s or b holding bf16 values gives the bf16 result bitwise (tested). + `positions` is converted to fp32 by name, so a bf16 default dtype never reaches the + log (tested). No host synchronization: a CUDA-graph replay on new q and positions is + bitwise equal to the eager call at every captured batch 1..16, at 16 and 32 heads, at + all 64 (dep4) and at 8 (tep8 and tp8, every rank's slice), and at tep8's 8 MLA and 32 + indexer heads at decode positions up to 1048575 (max_seq_len 1048576). Under capture the + intermediates come from the graph's pool. + + Cost, measured on sm_103: at T <= 16 an eager call is launch-bound and a CUDA-graph + replay of the call is cheaper than the eager call; at T 32768 the two cost about the + same. + + Launches, profiled on sm_103 at 64 heads for every T 1..32768: seven per call, 128 + threads each, none with a grid-stride loop. positions to fp32 is an unrolled copy over + [T], grid ceil(T / 512); + 1, the log and the rounding to bf16 are vectorized over [T], + grid ceil(T / 1024); * s and + b run the offset-calculator kernel over [T, H, 1], grid + ceil(T * H / 512) (+ b takes the vectorized kernel at T = 1 and the offset-calculator + one from T = 2, the only kernel that changes with T); q * scale runs the + offset-calculator kernel, grid T * H * D / 512. What T moves is each grid and its last + block's tail, and T 1..4096 holds every tail of every launch: the dep4 step-size cell + drives those, the top 1024 and a stride between, each bitwise the leading rows of the + 32768-row call, and its large cell drives every T 1..32768 the same way. At tep8's 8 + heads, profiled the same way: the same seven kernels and blocks; * s and + b take grid + ceil(T / 64), and q * scale 2 T blocks, a row's 1024 elements filling two blocks whole, + so that launch has no partial block; + b still changes kernel at T = 2, and nothing + else changes with T. Every tail repeats with a period of 1024 in T, so T 1..4096 holds + them all: the tep8 last-slice cell drives those, the top 1024 and a stride between, + each bitwise the leading rows of its 32768-row call, and every T 1..32768 at 8 heads + was driven the same way once (measured, 0 of 32768 step sizes differ; not a cell). + + Numerics: bitwise equal to `reference`, the specified bf16 chain, over every cell. + The chain itself sits up to 2.8 bf16 ulp from the closed form at the MLA cells' drawn + s and b over positions 0..9215, and up to 15.5 ulp over positions 0..32767 with the + ranges' corners pinned (measured), where the cancellation corner (s -0.08, b 0.9) + shrinks the factor to 0.068 at position 32767 (computed); both inside the bound the + test derives. An fp32 scale with no intermediate rounding lands within 0.5 ulp of the + closed form and differs from this op in about a third of the elements (37.4% at 9216 + rows, 36.4% at 32768, 35.5% at 32768 x 64 heads, measured): a replacement kernel has + to reproduce the roundings, not the math. At 64 heads, with the corners on heads 0..3, + the chain sits up to 15.5 bf16 ulp from the closed form over positions 0..32767, as at + 16 (measured), and so it does at 8 heads with the corners on four of a rank's 8 (15.494 + at rank 2's slice, 15.531 at rank 7's, where an fp32 scale rounded once differs from the + op in 43.6% and 39.1% of the elements; measured). The device's fp32 log (CUDA logf, not + correctly rounded) differs from the correctly rounded log in the last fp32 bit at 935 of + the positions 0..32767 and at 39331 of the positions 0..1048575, and rounds to the same + bf16 at every one of them: the context cells drive every position below 32768 one by one, + and the six sweep cells (two at 16 MLA heads, the indexer's, and three at tep8's 8 MLA + heads) and dep4's two 32-chunk cells every position below 2^20, where the test also holds + the two logs' bf16 equal directly. Below 32768 they cannot part: logf is within 1 fp32 + ulp of the exact log, and the exact log(pos + 1) of every position 1..32767 lies at least + 1.748 fp32 ulp from a bf16 rounding midpoint (measured; the nearest is position 6510). + Above it there is no such margin: 38 of the positions 32768..1048575 lie within 1 fp32 + ulp of a midpoint, the nearest 0.0458 ulp from it (position 1028641, in the last chunk), + so their agreement is measured, not implied, and the sweep and 32-chunk cells are what + hold it. Measured beyond every envelope, not a cell: the first position where the two + round apart is 1695942, and 5 of the positions below 2^24 do. + + At the NVFP4 target's envelope the factor reaches 7.53125 at position 32767 with the + checkpoint's largest s and b (0.574219, 1.57031; the closed form is 7.5406). Its + largest error is at the checkpoint's deepest cancellation, s -0.078125 with b + 0.902344, where the factor falls to 0.0898 at position 32767: there the chain's factor + lands up to 8.997 bf16 ulp (of the closed form's factor) from it over positions + 0..32767 (measured), because s * log(pos + 1) is rounded at its own magnitude, about + nine times the factor's. That is what the chain specifies. (That s and b are the + corner of the checkpoint's ranges; no one head holds both: the head with s -0.078125, + MLA layer 0 head 26, has b 0.90625 and a closed-form factor of 0.0940 at position + 32767, computed.) + + At tep4, tp4 and tep8 over max_seq_len 1048576, the closed-form factor at position + 1048575 reaches 9.53067 on the checkpoint's largest head (MLA layer 19 head 15, rank 0's + local head 15 at tep4 and tp4 and tep8 rank 1's local head 7, which holds both range + tops, s 0.574219 and b 1.57031; read on disk). log(pos + 1) is 13.86 there and a bf16 + step of it is 0.0625: the chain's log takes two values over the last chunk + 1015808..1048575 (13.8125 and 13.875), and log(pos + 1) and log(pos + 2) round to + different bf16 at only 276 of the positions 0..1048575, 56 of them above 32767 and one + in the last chunk (position 1028640; measured). A position off by one therefore leaves + this op's output bitwise unchanged at most far positions, where neither gate of the test + can see it; the cells' near positions can. On the tep8 last chunk (rank 1's 8 heads) it + moves one row of 32768, the one at 1028640 (or 1028641 for log(pos) in place of log(pos + + 1)), and stays inside the closed form's bound (0.680 of it), so neither gate sees it on + the other rows. There, s and b swapped land at 1723 times the bound and rank 2's 8 heads + on rank 1 at 94.2 times it, and the rounding variants move 9.0% to 43.2% of the elements + (measured). Over every position 0..1048575 and every head of the NVFP4 checkpoint, the + chain's factor lands within 0.0558 of the closed form's on the MLA (0.0411 over + 0..32767) and within 0.00793 on the indexer, and within 2.249 bf16 ulp of it wherever the + factor is at least 0.25 (measured). + + Two MLA heads of the NVFP4 checkpoint, and no other of its 72 x 64, have a factor that + crosses zero below 1048576, both on layer 0: head 20 (s -0.0693359, b 0.914063) near + position 531308 and head 26 (s -0.078125, b 0.90625) near 109097 (computed from the + closed form). Both are in tp4 rank 1's slice (`NVFP4_MLA_LAYER0_RANK1`, local heads 4 + and 10), and one is in each of tep8 ranks 2 and 3's slices (`NVFP4_MLA_LAYER0_TEP8_RANK2`, + rank 2's local head 4; `NVFP4_MLA_LAYER0_TEP8_RANK3`, rank 3's local head 2). There the + chain's factor is exactly 0, a zeroed query, over 33359 positions (517233..550591) and + 6992 positions (108418..115409), never has the opposite sign to the closed form, and + ends at -0.046875 and -0.179688 at position 1048575 (the closed form -0.0471377 and + -0.176792; measured). That is what the chain specifies: the checkpoint-slice cells, tp4 + rank 1's and tep8 ranks 2 and 3's, drive both crossings at every position and the test + holds those runs. No indexer head crosses below 1048576 (the earliest near position + 7.2e9, computed). The ranges' cancellation corner crosses zero too (s -0.080078, b + 0.898438 in bf16, near position 74570), its chain factor stepping from +2^-8 to -2^-8 + where the bf16 log steps from 11.1875 to 11.25, with no exactly-zero value between + (computed; the MLA corner sweeps, at 16 heads and at tep8's 8, zero no query, measured): + the MLA corner cells past it (the last chunks, the packed and mixed steps, the MLA + sweeps through the crossing, at 16 heads and at tep8's 8) drive the negative factor, and + the test's closed-form bound, which scales with |s * log(pos + 1)| and not with the + factor, holds there. + + At dep4 over max_seq_len 1048576 every rank holds all 64 MLA heads, so every MLA head + above, the two whose factor crosses zero included, is on every rank. At position + 1048575 the chain gives 9.5625 on the largest head (the closed form 9.53067), and the + indexer's closed-form factor reaches 1.90887 (layer 1 head 19). One of the 276 + positions where log(pos + 1) and log(pos + 2) round to different bf16 lies in the last + chunk: over the MLA's 1M last chunk at 64 heads, either off-by-one moves one row of the + 32768, at position 1028640 or 1028641, and stays inside the closed form's bound + (tested). On the indexer the chain's factor lands within 1.645 bf16 ulp of the closed + form's wherever the factor is at least 0.25 (measured). Over dep4's 1M cells the output + lands at most 0.965 of the test's bound from the closed form (measured). + + The cells' range corners over positions 0..1048575, the factor alone (measured at dep4): + on the MLA (s -0.080078 or 0.578125, b 0.898438 or 1.60156 in bf16) the chain's factor + lands within 0.0571 of the closed form's and within 2.969 bf16 ulp of it wherever the + factor is at least 0.25, at most 0.8005 of the test's bound on the factor (dC at + `_closed_form_ratio`; 0.3881 over the last chunk); on the indexer (s -0.050049 or + 0.075195, b 0.96875 or 1.0625) within 0.0138, 1.746 ulp and 0.8538 of dC. At position + 1048575 the MLA corners' factor is 9.625 at the top corner (closed form 9.61608) and + -0.210938 at the cancellation corner (closed form -0.211681). That corner (s -0.080078, + b 0.898438) crosses zero near position 74570 (computed): the chain's factor is never 0 + there, and is -0.00390625 at the 57 positions 74514..74570, where the closed form is + positive and at most 6.06e-05 (measured). The test's bound scales with + |s * log(pos + 1)|, not with the factor, so it holds through the crossing, and dep4's MLA + 32-chunk cell drives it there, as the MLA sweep cells (at 16 heads and at tep8's 8) do. + + Rows are independent: a padded row cannot reach a live one. A negative position + gives a non-finite scale on its own row only (-1: log 0 = -inf, an infinite scale; + -2: nan). + + `positions` must be [T] and `s`, `b` [H]: `is_valid` refuses a q that is not 3-D, + positions of any other shape, and s or b of any other shape. These broadcast silently + in the op: s or b of length 1 scales every head by head 0's terms; positions of length + 1 scales every row at position 0; T = 1 with positions of length P returns P rows; + when T == H, s and b as [H, 1] scale row t by head t's terms, and a 2-D q [T, H * D] + returns [T, H, H * D]. Mismatches the broadcast cannot absorb raise from torch: an + unsliced [64] s or b against 16 heads raises `RuntimeError: The size of tensor a (16) + must match the size of tensor b (64) at non-singleton dimension 1`, and a rank's + 16-head slice against all 64 heads (a tep4 slice at dep4) raises `RuntimeError: The + size of tensor a (64) must match the size of tensor b (16) at non-singleton dimension + 1`. At tep8's 8 heads, the unsliced [64] s or b raises `RuntimeError: The size of + tensor a (8) must match the size of tensor b (64) at non-singleton dimension 1`, and a + tep4 rank's 16-head slice `RuntimeError: The size of tensor a (8) must match the size + of tensor b (16) at non-singleton dimension 1`; s and b as [8, 1] pair rows with heads + silently at decode batch 8, a captured batch where T == H (all tested). + + Hard limits (sm_103), computed at the widest envelope the cells cover: positions <= + 1048575 (tep4, tp4, dep4 and tep8 at max_seq_len 1048576 with chunked prefill; 32767 at + the other NVFP4 targets) and T <= 32768 per step (max_num_tokens 32768 per rank, chunked + prefill or not); none is reached, none was probed: + positions -> fp32 exact below 2^24 boundary 2^24 + 1 not reached + (positions + 1 <= 2^20) + q.numel() elementwise TensorIterator boundary 2^31 elements not reached + (32768 x 64 x 128 = 268435456 at dep4; 32768 x 32 x 128 = + 134217728 on the indexer, 32768 x 16 x 128 = 67108864 on the + tp4 MLA, 32768 x 8 x 128 = 33554432 on the tep8 MLA) + row t of q t * H * D, H * D = 1024 boundary 2^31 / 8192 = not reached + (MLA at tep8), 2048 (MLA 262144 rows at dep4, (T <= 32768) + at tep4 and tp4), 4096 2^31 / 1024 = 2097152 + (indexer), 8192 (MLA, dep4) at tep8 (2^20 rows at + 16 heads) + grid of q * scale T * H * D / 512 blocks boundary 2^31 - 1 not reached + (524288 at 32768 x 64) + positions int32 values, no table no pool or slot index - + The op holds no KV pool, page or slot table, so the pool size the engine allocates + does not reach it, and the context length does not size any operand: a 1048576-token + prompt reaches it as chunks of at most 32768 rows. One call over every position of a + 1M request would be [1048576, 16, 128] at 16 heads, exactly 2^31 elements, and + [1048576, 32, 128] on the indexer, 2^32 elements, past the 2^31 boundary; no step of + the envelope is either, and the sweep cells drive every position in 32768-row calls. + + Not certified: a q dtype other than bf16 (each step would round to that dtype instead); + int64 positions (tested equal to int32 bitwise, in no cell); head_dim other than 128, + heads other than 8, 16, 32 and 64; at 8 and 64 heads, s and b outside the + checkpoint-shaped draw ranges and q over every bf16 binade below position 1015808 (the + stress cells run at 16 heads, and at 64 and at 8 over dep4's and tep8's 1M last chunk); + positions above 1048575 (the 1M cells at 16 MLA heads are tep4's and tp4's, at all 64 + dep4's, at 8 tep8's, and the indexer's serve all four); steps above 32768 rows; decode + batches above 16; a non-contiguous q; torch.compile. + """ + + def __call__( + self, q: torch.Tensor, positions: torch.Tensor, s: torch.Tensor, b: torch.Tensor + ) -> torch.Tensor: + log_pos = torch.log(positions.to(torch.float32) + 1).to(q.dtype) + scale = log_pos[:, None, None] * s.to(q.dtype)[:, None] + b.to(q.dtype)[:, None] + return q * scale + + def reference( + self, q: torch.Tensor, positions: torch.Tensor, s: torch.Tensor, b: torch.Tensor + ) -> torch.Tensor: + """The specified chain, each step evaluated in fp64 and rounded to fp32, then to + `q.dtype`: ``lp = rnd(log(pos + 1))``, ``m = rnd(lp * s)``, ``scale = rnd(m + b)``, + ``out = rnd(q * scale)``. + + fp64 holds a product of two bf16 values exactly, and a sum of two to far less than + half a bf16 ulp of the larger, so the two roundings land where torch's bf16 + arithmetic (fp32 math, rounded to bf16) lands: this is the chain, not an + approximation of it. Its fp32 log is the correctly rounded one; the device's is + not, and the test states why the two agree once rounded to bf16. + """ + dt = q.dtype + + def rnd(x: torch.Tensor) -> torch.Tensor: + return x.to(torch.float32).to(dt).double() + + lp = rnd(torch.log(positions.double() + 1)) + m = rnd(lp[:, None] * rnd(s.double())[None, :]) + scale = rnd(m + rnd(b.double())[None, :]) + return (q.double() * scale[:, :, None]).to(torch.float32).to(dt) + + def is_valid( + self, q: torch.Tensor, positions: torch.Tensor, s: torch.Tensor, b: torch.Tensor + ) -> None: + if q.dim() != 3: + raise ValueError( + f"ssmax_scaling: q must be [T, H, D], got shape {tuple(q.shape)}; a 2-D q " + "broadcasts against the [T, H, 1] scale and pairs rows with heads when T == H" + ) + tokens, heads = q.shape[0], q.shape[1] + if tuple(positions.shape) != (tokens,): + raise ValueError( + f"ssmax_scaling: positions must be [T] = [{tokens}], got shape " + f"{tuple(positions.shape)}; a length-1 positions scales every row at one " + "position, and a q of one row is broadcast to every position" + ) + for name, t in (("s", s), ("b", b)): + if tuple(t.shape) != (heads,): + raise ValueError( + f"ssmax_scaling: {name} must be [H] = [{heads}], this rank's heads, got " + f"shape {tuple(t.shape)}; a length-1 {name} is broadcast over every head, " + f"and an [H, 1] {name} pairs rows with heads when T == H" + ) + + +ssmax_scaling = SSMaxScaling() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/ssmax_scaling_fused_q.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/ssmax_scaling_fused_q.py new file mode 100644 index 000000000000..cc7fc6aec2ff --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/ssmax_scaling_fused_q.py @@ -0,0 +1,669 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""SSMax query scaling of the MLA query, with its rope half written into the absorbed query's +tail, in one Triton launch: ``ssmax_scaling``'s bf16 chain bit for bit, plus the copy (or, on a +NoPE layer, the zero fill) of the ``[T, H, 64]`` tail of the 576-wide fused query FlashMLA +reads.""" + +from __future__ import annotations + +import torch +import triton +import triton.language as tl +from triton.language.extra import libdevice +from triton.language.extra.cuda import gdc_wait + +from tensorrt_llm._torch.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ssmax_scaling import ( + ssmax_scaling, +) + +HEAD_DIM = 128 +#: The query's two halves: [nope(64) | rope(64)] on a rope layer; a NoPE layer's 128 are all +#: nope and its tail is zero. +HALF = 64 +#: Heads one program scales: a rank's 16 (tp4/tep4) or 8 (tp8/tep8) in one, dep4's 64 in four. +_HEADS_PER_PROGRAM = 16 + +#: max_seq_len of the NVFP4 targets' 32K config, which bounds every position there. +MAX_SEQ_LEN = 32768 +#: Their 1M config: the checkpoint's own context (max_seq_len 1048576), prefilled in chunked +#: steps of at most MAX_NUM_TOKENS rows, a chunk's rows at cached .. cached + n - 1 behind its +#: request's cached prefix, decode rows at any position up to 1048575 (log 13.86). +LONG_MAX_SEQ_LEN = 1048576 +MAX_NUM_TOKENS = 32768 +#: The 1M config's decode batches (max_batch_size 16): the padded 1/2/4/8/16 and unpadded ones. +LONG_GRAPH_BATCHES = (1, 2, 3, 4, 5, 7, 8, 13, 16) +#: Decode batches the targets capture: the padded 1/2/4/8/16 and the unpadded 3..15 of a +#: max_batch_size-16 engine, then the larger padded batches a max_batch_size-256 engine +#: captures (the Pareto and decode-profile configurations). +GRAPH_BATCHES = (1, 2, 3, 4, 5, 7, 8, 13, 16, 32, 64, 128, 256) + +#: The draw ranges hold the NVFP4 checkpoint's MLA s and b over all 72 layers x 64 heads +#: (read on disk, bf16): s in [-0.078125, 0.574219], b in [0.902344, 1.57031]. +_MLA = dict( + dtype=torch.bfloat16, + positions_dtype=torch.int32, + param_heads=64, + s_range=(-0.08, 0.58), + b_range=(0.9, 1.6), + max_seq_len=MAX_SEQ_LEN, + q="normal", + corners=True, + inplace=True, +) + + +@triton.jit +def _round_bf16(x): + """fp32 rounded to bf16 (to nearest, ties to even) and widened back. The rounding is an + inline `cvt.rn.bf16.f32`: the compiler folds a bare `.to(bf16).to(fp32)` pair into + nothing, which skips the rounding the chain specifies.""" + r = tl.inline_asm_elementwise( + "cvt.rn.bf16.f32 $0, $1;", "=h,r", [x], dtype=tl.bfloat16, is_pure=True, pack=1 + ) + return r.to(tl.float32) + + +@triton.jit +def _ssmax_fused_q_kernel( + q_ptr, # bf16 [T, H, 128], unit stride over the head dim + q_st, + q_sh, + pos_ptr, # int32/int64 [T] + s_ptr, # [H] + s_st, + b_ptr, # [H] + b_st, + out_ptr, # bf16 [T, H, 128], unit stride over the head dim; may be q itself + o_st, + o_sh, + tail_ptr, # bf16 [T, H, 64], unit stride over the last dim + t_st, + t_sh, + heads, + ROPE: tl.constexpr, + HB: tl.constexpr, + USE_PDL: tl.constexpr, +): + t = tl.program_id(0).to(tl.int64) + h = tl.program_id(1) * HB + tl.arange(0, HB) + hmask = h < heads + d = tl.arange(0, 64) + m2 = hmask[:, None] + if USE_PDL: + gdc_wait() + # The chain of `ssmax_scaling`, one rounding per torch op: the position to fp32, + 1, + # logf, rounded to bf16; then * s and + b, each rounded to bf16; then q * scale. + pos = tl.load(pos_ptr + t).to(tl.float32) + lp = _round_bf16(libdevice.log(pos + 1.0)) + s = _round_bf16(tl.load(s_ptr + h * s_st, mask=hmask, other=0.0).to(tl.float32)) + b = _round_bf16(tl.load(b_ptr + h * b_st, mask=hmask, other=0.0).to(tl.float32)) + m = _round_bf16(lp * s) + scale = _round_bf16(m + b)[:, None] + q_row = q_ptr + t * q_st + h[:, None].to(tl.int64) * q_sh + d[None, :] + lo = tl.load(q_row, mask=m2, other=0.0).to(tl.float32) + hi = tl.load(q_row + 64, mask=m2, other=0.0).to(tl.float32) + lo = (lo * scale).to(tl.bfloat16) + hi = (hi * scale).to(tl.bfloat16) + o_row = out_ptr + t * o_st + h[:, None].to(tl.int64) * o_sh + d[None, :] + tl.store(o_row, lo, mask=m2) + tl.store(o_row + 64, hi, mask=m2) + t_row = tail_ptr + t * t_st + h[:, None].to(tl.int64) * t_sh + d[None, :] + if ROPE: + tl.store(t_row, hi, mask=m2) + else: + tl.store(t_row, tl.zeros([HB, 64], dtype=tl.bfloat16), mask=m2) + + +class SSMaxScalingFusedQ(OpWrapper): + """``out = ssmax_scaling(q, positions, s, b)``, and the absorbed query's rope tail from it. + + `q` is the MLA query ``[T, H, 128]`` bf16, `positions` the step's ``[T]`` position ids, + `s` and `b` this rank's ``[H]`` slice of the 64 per-head parameters (all 64 at dep4). + `out` (``[T, H, 128]`` bf16, may be `q` itself) receives the scaled query, bitwise + `ssmax_scaling`'s, and `tail` (``[T, H, 64]`` bf16, the ``[..., 512:576]`` view of this + rank's heads of the 576-wide fused query) receives ``out[..., 64:]`` when `rope`, zeros + otherwise (a NoPE layer, whose 128 dims are all nope and whose query tail is zero). + Returns `out`. + + Fusion boundary: the scaling, the rope half's copy into the fused query and a NoPE + layer's zero tail. The K-B absorption of ``out[..., :nope]`` into the fused query's first + 512 columns, and the rope rotation of the tail (``ds_mla_rope_append`` on owner layers), + are the caller's; nothing here reads or writes those columns or the other heads. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # spec: `heads` of a `param_heads` draw at rank `rank`'s slice, `tokens` rows at + # positions: each context's 0..n-1 in `contexts` order (c..c + n - 1 behind a `cached` + # prefix c), then `decode_rows` random positions in 1..max_seq_len - 1 (the first at + # max_seq_len - 1); `position_sweep` (lo, hi) drives every position lo..hi - 1 in calls of + # `tokens` rows, each on a fresh q, the same s and b. `corners` pins four of + # the slice's heads to the (s, b) corners of the ranges. `rope` picks the tail's + # contents; `inplace` passes `q` as `out` (the targets' call); `graph` captures the + # call and replays it on new q and positions. `q` "wide" draws every bf16 binade. + # `fq_heads` (64 when absent, FlashMLA's padded query) is the head count of the fused + # query the tail is a view of: `heads` is the tp4 / tep4 decode targets' own + # `[T, heads, 576]` scratch for cute_dsl_ds_mla_sparse_decode. + CELLS: tuple[Cell, ...] = ( + tuple( + Cell( + why=f"tp4/tep4 decode, graph batch {n:>3}, rank {n % 4}'s 16 heads, rope layer: " + "captured, replayed on new q and positions", + spec=dict( + _MLA, + heads=16, + rank=n % 4, + tokens=n, + contexts=(), + decode_rows=n, + rope=True, + graph=True, + ), + ) + for n in GRAPH_BATCHES + ) + + tuple( + Cell( + why=f"tp4/tep4 decode, graph batch {n:>3}, NoPE layer: zero tail, captured", + spec=dict( + _MLA, + heads=16, + rank=1, + tokens=n, + contexts=(), + decode_rows=n, + rope=False, + graph=True, + ), + ) + for n in (1, 16, 256) + ) + + tuple( + Cell( + why=f"dep4 decode, graph batch {n:>3}, all 64 heads (four programs per row), " + f"{'rope' if rope else 'NoPE'} layer, captured", + spec=dict( + _MLA, + heads=64, + rank=0, + tokens=n, + contexts=(), + decode_rows=n, + rope=rope, + graph=True, + ), + ) + for n in (1, 16, 64, 256) + for rope in (True, False) + ) + + tuple( + Cell( + why=f"tp8/tep8 decode, graph batch {n:>3}, rank {n % 8}'s 8 heads (a half-masked " + "program), rope layer, captured", + spec=dict( + _MLA, + heads=8, + rank=n % 8, + tokens=n, + contexts=(), + decode_rows=n, + rope=True, + graph=True, + ), + ) + for n in (1, 7, 16, 256) + ) + + ( + Cell( + why="tp4 context of 32768 rows at positions 0..32767, rank 1's heads, rope layer: " + "every position of the envelope", + spec=dict( + _MLA, + heads=16, + rank=1, + tokens=32768, + contexts=(32768,), + decode_rows=0, + rope=True, + graph=False, + ), + ), + Cell( + why="tp4 mixed step of 32768 rows, 3 contexts + 13 decode rows, NoPE layer, " + "out-of-place", + spec=dict( + _MLA, + heads=16, + rank=2, + tokens=32768, + contexts=(17000, 11000, 4755), + decode_rows=13, + rope=False, + graph=False, + inplace=False, + ), + ), + Cell( + why="dep4 context of 32768 rows, all 64 heads, rope layer", + spec=dict( + _MLA, + heads=64, + rank=0, + tokens=32768, + contexts=(32768,), + decode_rows=0, + rope=True, + graph=False, + ), + ), + Cell( + why="tp8 context of 32768 rows, rank 7's 8 heads, rope layer", + spec=dict( + _MLA, + heads=8, + rank=7, + tokens=32768, + contexts=(32768,), + decode_rows=0, + rope=True, + graph=False, + ), + ), + Cell( + why="stress at 32768 rows, s in [-1, 1] and b in [-2, 2], q over every bf16 " + "binade: cancellation, overflow, subnormals, out-of-place", + spec=dict( + _MLA, + heads=16, + rank=3, + s_range=(-1.0, 1.0), + b_range=(-2.0, 2.0), + q="wide", + tokens=32768, + contexts=(32768,), + decode_rows=0, + rope=True, + graph=False, + inplace=False, + ), + ), + ) + # The tp4 / tep4 decode step on cute_dsl_ds_mla_sparse_decode (16 heads, 8 at tp8 / + # tep8): the tail is a view of the rank's unpadded `[T, heads, 576]` query scratch (row + # stride heads x 576), every element of which the absorption BMM and this tail write + # rewrite each layer. + + tuple( + Cell( + why=f"{'tp4/tep4' if h == 16 else 'tp8/tep8'} decode {n:>3} " + f"{'rope' if rope else 'NoPE'}, native [T, {h}, 576] scratch: the tail at row " + f"stride {h} x 576, captured", + spec=dict( + _MLA, + heads=h, + rank=n % (64 // h), + tokens=n, + contexts=(), + decode_rows=n, + rope=rope, + graph=True, + fq_heads=h, + ), + ) + for h in (16, 8) + for n in (1, 16, 64, 256) + for rope in (True, False) + ) + ) + # The 1M config (max_seq_len 1048576, chunked prefill, max_num_tokens 32768, 16 sequences): + # decode rows at positions up to 1048575, chunk rows behind a cached prefix, mixed and packed + # steps, and every position 0..1048575 swept in calls of 32768 rows (`position_sweep`), at + # the rank's 16 heads (tp4 / tep4), 8 (tp8 / tep8) and all 64 (dep4), into FlashMLA's + # 64-head padded fused query. + CELLS += tuple( + Cell( + why=f"1M tp4/tep4 decode, graph batch {n:>2}, rank {n % 4}'s 16 heads, rope layer: " + "positions up to 1048575, captured, replayed on new q and positions", + spec=dict( + _MLA, + max_seq_len=LONG_MAX_SEQ_LEN, + heads=16, + rank=n % 4, + tokens=n, + contexts=(), + decode_rows=n, + rope=True, + graph=True, + ), + ) + for n in LONG_GRAPH_BATCHES + ) + CELLS += tuple( + Cell( + why=f"1M {name} decode, graph batch {n:>2}, {heads} heads, " + f"{'rope' if rope else 'NoPE'} layer: positions up to 1048575, captured", + spec=dict( + _MLA, + max_seq_len=LONG_MAX_SEQ_LEN, + heads=heads, + rank=n % (64 // heads), + tokens=n, + contexts=(), + decode_rows=n, + rope=rope, + graph=True, + ), + ) + for name, heads in (("tp4/tep4", 16), ("tp8/tep8", 8), ("dep4", 64)) + for n in (1, 16) + for rope in (True, False) + if not (heads == 16 and rope) + ) + CELLS += ( + Cell( + why="1M tp4 last chunk: 32768 rows behind a 1015808-token cached prefix, positions " + "1015808..1048575, rank 1's heads, rope layer", + spec=dict( + _MLA, + max_seq_len=LONG_MAX_SEQ_LEN, + heads=16, + rank=1, + tokens=MAX_NUM_TOKENS, + contexts=(MAX_NUM_TOKENS,), + cached=(LONG_MAX_SEQ_LEN - MAX_NUM_TOKENS,), + decode_rows=0, + rope=True, + graph=False, + ), + ), + Cell( + why="1M dep4 last chunk, all 64 heads, positions 1015808..1048575, NoPE layer", + spec=dict( + _MLA, + max_seq_len=LONG_MAX_SEQ_LEN, + heads=64, + rank=0, + tokens=MAX_NUM_TOKENS, + contexts=(MAX_NUM_TOKENS,), + cached=(LONG_MAX_SEQ_LEN - MAX_NUM_TOKENS,), + decode_rows=0, + rope=False, + graph=False, + ), + ), + Cell( + why="1M tp4 mixed step of 32768 rows, 3 chunk tails + 13 decode rows: chunks at " + "positions 1031576.., 500000.. and 0.., decode rows up to 1048575, out-of-place", + spec=dict( + _MLA, + max_seq_len=LONG_MAX_SEQ_LEN, + heads=16, + rank=2, + tokens=MAX_NUM_TOKENS, + contexts=(17000, 11000, 4755), + cached=(LONG_MAX_SEQ_LEN - 17000, 500000, 0), + decode_rows=13, + rope=True, + graph=False, + inplace=False, + ), + ), + Cell( + why="1M tp8 packed step of 32768 rows, 4 chunks + 13 decode rows: chunks at " + "1032192.., 524288.., 98304.. and 0.., rank 7's 8 heads, NoPE layer", + spec=dict( + _MLA, + max_seq_len=LONG_MAX_SEQ_LEN, + heads=8, + rank=7, + tokens=MAX_NUM_TOKENS, + contexts=(16384, 8192, 4096, 4083), + cached=(LONG_MAX_SEQ_LEN - 16384, 524288, 98304, 0), + decode_rows=13, + rope=False, + graph=False, + ), + ), + Cell( + why="stress at the 1M last chunk, s in [-1, 1] and b in [-2, 2], q over every bf16 " + "binade, positions 1015808..1048575: cancellation, overflow, subnormals", + spec=dict( + _MLA, + max_seq_len=LONG_MAX_SEQ_LEN, + heads=16, + rank=3, + s_range=(-1.0, 1.0), + b_range=(-2.0, 2.0), + q="wide", + tokens=MAX_NUM_TOKENS, + contexts=(MAX_NUM_TOKENS,), + cached=(LONG_MAX_SEQ_LEN - MAX_NUM_TOKENS,), + decode_rows=0, + rope=True, + graph=False, + inplace=False, + ), + ), + ) + CELLS += tuple( + Cell( + why=f"1M every position 0..1048575 in 32 calls of 32768 rows, {heads} heads " + f"(rank {rank}), {'rope' if rope else 'NoPE'} layer, the range corners", + spec=dict( + _MLA, + max_seq_len=LONG_MAX_SEQ_LEN, + heads=heads, + rank=rank, + tokens=MAX_NUM_TOKENS, + contexts=(MAX_NUM_TOKENS,), + decode_rows=0, + rope=rope, + graph=False, + position_sweep=(0, LONG_MAX_SEQ_LEN), + ), + ) + for heads, rank, rope in ((16, 0, True), (8, 5, False), (64, 0, True)) + ) + # The 1M config's decode step on cute_dsl_ds_mla_sparse_decode: the tail a view of the + # rank's unpadded `[T, heads, 576]` query scratch (row stride heads x 576), positions up to + # 1048575, and every position swept in that layout. + CELLS += tuple( + Cell( + why=f"1M {'tp4/tep4' if h == 16 else 'tp8/tep8'} decode {n:>2} " + f"{'rope' if rope else 'NoPE'}, native [T, {h}, 576] scratch: positions up to " + "1048575, captured", + spec=dict( + _MLA, + max_seq_len=LONG_MAX_SEQ_LEN, + heads=h, + rank=n % (64 // h), + tokens=n, + contexts=(), + decode_rows=n, + rope=rope, + graph=True, + fq_heads=h, + ), + ) + for h in (16, 8) + for n in (1, 13, 16) + for rope in (True, False) + ) + CELLS += tuple( + Cell( + why=f"1M every position 0..1048575 in 32 calls, native [T, {h}, 576] scratch, rank " + f"{rank}'s {h} heads, {'rope' if rope else 'NoPE'} layer", + spec=dict( + _MLA, + max_seq_len=LONG_MAX_SEQ_LEN, + heads=h, + rank=rank, + tokens=MAX_NUM_TOKENS, + contexts=(MAX_NUM_TOKENS,), + decode_rows=0, + rope=rope, + graph=False, + position_sweep=(0, LONG_MAX_SEQ_LEN), + fq_heads=h, + ), + ) + for h, rank, rope in ((16, 2, True), (8, 6, False)) + ) + + CELLS += tuple( + Cell( + why=f"DSpark 1M verification: {n} rows, {h} heads, rope={rope}, captured", + spec=dict( + _MLA, + max_seq_len=LONG_MAX_SEQ_LEN, + heads=h, + rank=0, + tokens=n, + contexts=(), + decode_rows=n, + rope=rope, + graph=True, + fq_heads=h, + ), + ) + for h in (8, 16, 64) + for n in (52, 64) + for rope in (True, False) + ) + + note = """ + Call sites: the mistral_large4 NVFP4 targets (tp4, tep4, dep4), every layer, context and + generation rows, right after the `wq` GEMM; the targets pass `q` as `out`. Replaces + `ssmax_scaling` (seven torch launches) plus the tail's strided copy or zero fill (one + more) with one launch. The tail is a view of FlashMLA's 64-head padded fused query, or, + on the tp4 / tep4 decode step that runs cute_dsl_ds_mla_sparse_decode, of the rank's own + `[T, heads, 576]` query (the kernel takes the tail's row and head strides). + + Numerics: the same operations as the unfused ops, in the same order, at the same types, + rounded at the same points; `out` and `tail` are bitwise those ops' results over every + cell, on tensors captured from a served decode step, and over the NVFP4 checkpoint's own + s and b of all 72 layers at every position 0..1048575 in every call site's layout of the + fused query (the test runs all three). Stage by stage (the unfused op, then this + kernel): + positions.to(float32) int to fp32 (exact below 2^24) the same conversion + + 1 fp32 add fp32 add + torch.log fp32 logf (libdevice __nv_logf) libdevice.log, the same + function: fp32-bitwise at + every pos + 1 below 2^24 + and 2^24 random fp32 (tested) + .to(bf16) round to nearest even cvt.rn.bf16.f32 + s.to(bf16), b.to(bf16) no-op on the bf16 parameters cvt.rn.bf16.f32 (no-op) + log_pos * s fp32 product, rounded to bf16 fp32 product, cvt.rn.bf16 + + b fp32 sum, rounded to bf16 fp32 sum, cvt.rn.bf16 + q * scale fp32 product, rounded to bf16 fp32 product, cvt.rn.bf16 + tail copy_ / zero_ exact copy / zero fill store of the same bf16 / 0 + Every intermediate rounding is an inline `cvt.rn.bf16.f32`, because the compiler folds a + bare `.to(bf16).to(fp32)` pair into nothing; the kernel is compiled without implicit FMA + contraction (`enable_fp_fusion=False`), so no product and sum are fused. No fast-math + function is used. `ssmax_scaling`'s note derives where the chain sits from the closed + form; nothing here changes it. + + Launch: Triton, grid (T, ceil(H / 16)), 16 heads (8 at tp8/tep8) per program, two + 64-wide halves per head; programmatic dependent launch (waits on its predecessor before + the first load). No host synchronization and no allocation: CUDA-graph capturable, a + replay on new q and positions writes what an eager call writes. + + Rows and heads are independent: a padded decode row reaches no live one. In place is + safe: every element is loaded before the program stores it, and no program reads + another's elements. `tail` must not overlap `q` or `out`. + + Hard limits (sm_103, T <= 32768 rows a call): row offsets are int64; positions pass + through fp32, exact below 2^24 (positions <= 1048575 at the 1M config); the grid's T + dimension is the x dimension (2^31 - 1). None reached. + + Not certified: a q dtype other than bf16, head_dim other than 128, heads other than 8, + 16 and 64, positions above 1048575, a non-unit stride over the head dim of q, out or + tail. + """ + + def __call__( + self, + q: torch.Tensor, + positions: torch.Tensor, + s: torch.Tensor, + b: torch.Tensor, + out: torch.Tensor, + tail: torch.Tensor, + rope: bool, + ) -> torch.Tensor: + tokens, heads = q.shape[0], q.shape[1] + if tokens == 0: + return out + hb = min(_HEADS_PER_PROGRAM, triton.next_power_of_2(heads)) + _ssmax_fused_q_kernel[(tokens, triton.cdiv(heads, hb))]( + q, q.stride(0), q.stride(1), positions, s, s.stride(0), b, b.stride(0), + out, out.stride(0), out.stride(1), tail, tail.stride(0), tail.stride(1), heads, + ROPE=bool(rope), HB=hb, USE_PDL=True, num_warps=4, launch_pdl=True, + enable_fp_fusion=False, + ) # fmt: skip + return out + + def reference( + self, + q: torch.Tensor, + positions: torch.Tensor, + s: torch.Tensor, + b: torch.Tensor, + out: torch.Tensor, + tail: torch.Tensor, + rope: bool, + ) -> tuple[torch.Tensor, torch.Tensor]: + """Fresh ``(out, tail)``: `ssmax_scaling`'s reference chain, then its rope half or + zeros; `out` and `tail` are not read.""" + scaled = ssmax_scaling.reference(q, positions, s, b) + new_tail = scaled[..., HALF:].clone() if rope else torch.zeros_like(scaled[..., HALF:]) + return scaled, new_tail + + def is_valid( + self, + q: torch.Tensor, + positions: torch.Tensor, + s: torch.Tensor, + b: torch.Tensor, + out: torch.Tensor, + tail: torch.Tensor, + rope: bool, + ) -> None: + # Each is addressed through strides the kernel assumes, and answers wrongly otherwise. + if q.dim() != 3 or q.shape[2] != HEAD_DIM or q.dtype != torch.bfloat16: + raise ValueError(f"ssmax_scaling_fused_q: q must be bf16 [T, H, 128], got {q.shape}") + tokens, heads = q.shape[0], q.shape[1] + if tuple(positions.shape) != (tokens,): + raise ValueError(f"ssmax_scaling_fused_q: positions must be [T] = [{tokens}]") + for name, t in (("s", s), ("b", b)): + if tuple(t.shape) != (heads,): + raise ValueError(f"ssmax_scaling_fused_q: {name} must be [H] = [{heads}]") + if tuple(out.shape) != tuple(q.shape) or out.dtype != torch.bfloat16: + raise ValueError("ssmax_scaling_fused_q: out must be bf16 shaped like q") + if tuple(tail.shape) != (tokens, heads, HALF) or tail.dtype != torch.bfloat16: + raise ValueError(f"ssmax_scaling_fused_q: tail must be bf16 [T, H, {HALF}]") + for name, t in (("q", q), ("out", out), ("tail", tail)): + if t.stride(2) != 1: + raise ValueError(f"ssmax_scaling_fused_q: {name} needs a unit last stride") + if out.data_ptr() != q.data_ptr() and _overlap(out, q): + raise ValueError("ssmax_scaling_fused_q: out must be q itself or not overlap it") + if _overlap(tail, q) or _overlap(tail, out): + raise ValueError("ssmax_scaling_fused_q: tail must not overlap q or out") + + +def _overlap(a: torch.Tensor, b: torch.Tensor) -> bool: + """Whether the byte spans of `a` and `b` intersect.""" + a0, b0 = a.data_ptr(), b.data_ptr() + return a0 < b0 + _span(b) and b0 < a0 + _span(a) + + +def _span(t: torch.Tensor) -> int: + """Bytes from `t`'s first element to one past its last.""" + if t.numel() == 0: + return 0 + last = sum((n - 1) * st for n, st in zip(t.shape, t.stride())) + return (last + 1) * t.element_size() + + +ssmax_scaling_fused_q = SSMaxScalingFusedQ() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/triton_fp8_mqa_logits.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/triton_fp8_mqa_logits.py new file mode 100644 index 000000000000..aa7010d25240 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/attention/triton_fp8_mqa_logits.py @@ -0,0 +1,143 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""SM107 DSA prefill scoring over FP8 queries and gathered FP8 keys.""" + +from __future__ import annotations + +import torch +import triton +import triton.language as tl + +from tensorrt_llm._torch.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.fp8_mqa_logits import ( # noqa: E501 + fp8_mqa_logits, +) + + +@triton.jit +def _score( + Q, + K, + SCALE, + WEIGHT, + START, + END, + OUTPUT, + M, + N, + OUTPUT_ROW_STRIDE, + Q0: tl.constexpr, + Q1: tl.constexpr, + Q2: tl.constexpr, + K0: tl.constexpr, + K1: tl.constexpr, + W0: tl.constexpr, + W1: tl.constexpr, + S0: tl.constexpr, + START0: tl.constexpr, + END0: tl.constexpr, + BLOCK_M: tl.constexpr, + BLOCK_N: tl.constexpr, +): + qh = tl.arange(0, BLOCK_M * 32) + rows = tl.program_id(0) * BLOCK_M + qh // 32 + heads = qh % 32 + keys = tl.program_id(1) * BLOCK_N + tl.arange(0, BLOCK_N) + dim = tl.arange(0, 128) + query = tl.load( + Q + rows[:, None] * Q0 + heads[:, None] * Q1 + dim[None, :] * Q2, + rows[:, None] < M, + other=0.0, + ) + key = tl.load(K + dim[:, None] * K1 + keys[None, :] * K0, keys[None, :] < N, other=0.0) + dots = tl.dot(query, key, out_dtype=tl.float32, max_num_imprecise_acc=0) + weight = tl.load(WEIGHT + rows * W0 + heads * W1, rows < M, other=0) + weighted = tl.maximum(dots, 0) * weight[:, None] + score = tl.sum(weighted.reshape(BLOCK_M, 32, BLOCK_N), axis=1) + scale = tl.load(SCALE + keys * S0, keys < N, other=0) + score *= scale[None, :] + output_rows = tl.program_id(0) * BLOCK_M + tl.arange(0, BLOCK_M) + start = tl.load(START + output_rows * START0, output_rows < M, other=0) + end = tl.load(END + output_rows * END0, output_rows < M, other=0) + valid = (keys[None, :] < N) & (keys[None, :] >= start[:, None]) & (keys[None, :] < end[:, None]) + score = tl.where(valid, score, -float("inf")) + tl.store( + OUTPUT + output_rows[:, None].to(tl.int64) * OUTPUT_ROW_STRIDE + keys[None, :], + score, + (output_rows[:, None] < M) & (keys[None, :] < OUTPUT_ROW_STRIDE), + ) + + +class TritonFp8MqaLogits(OpWrapper): + """FP32 sum of weighted, rectified FP8 dot products, masked by each query's window. + + The key's FP32 scale is applied after the head reduction. Inputs retain + their original FP8 codes; no BF16 dequantization precedes the dot product. + Rows are padded to 16-byte alignment for the prefill top-k consumer. + """ + + ARCHS = frozenset({Arch.SM_107}) + CELLS = tuple( + Cell(f"SM107 indexer rows={m}, keys={n}", dict(m=m, n=n)) + for m in (1, 17, 129, 2048) + for n in (33, 2051, 4096) + ) + ( + Cell("SM107 strided scales and windows", dict(m=17, n=2051, strided=True)), + Cell("SM107 32K context tile", dict(m=512, n=32768)), + Cell("SM107 1M context tile", dict(m=64, n=1048576)), + ) + + def __call__( + self, + q: torch.Tensor, + kv: tuple[torch.Tensor, torch.Tensor], + weights: torch.Tensor, + cu_seq_len_k_start: torch.Tensor, + cu_seq_len_k_end: torch.Tensor, + ) -> torch.Tensor: + k, scale = kv + if q.shape[1:] != (32, 128) or k.ndim != 2 or k.shape[1] != 128: + raise ValueError("SM107 MQA scorer requires 32 query heads and head dimension 128") + if q.dtype != torch.float8_e4m3fn or k.dtype != torch.float8_e4m3fn: + raise ValueError("SM107 MQA scorer requires FP8 E4M3 queries and keys") + padded_n = triton.cdiv(k.shape[0], 4) * 4 + output = torch.empty((q.shape[0], padded_n), dtype=torch.float32, device=q.device)[ + :, : k.shape[0] + ] + if not output.numel(): + return output + _score[(triton.cdiv(q.shape[0], 4), triton.cdiv(k.shape[0], 128))]( + q, + k, + scale, + weights, + cu_seq_len_k_start, + cu_seq_len_k_end, + output, + q.shape[0], + k.shape[0], + output.stride(0), + *q.stride(), + *k.stride(), + *weights.stride(), + scale.stride(0), + cu_seq_len_k_start.stride(0), + cu_seq_len_k_end.stride(0), + 4, + 128, + num_warps=8, + ) + return output + + def reference( + self, + q: torch.Tensor, + kv: tuple[torch.Tensor, torch.Tensor], + weights: torch.Tensor, + cu_seq_len_k_start: torch.Tensor, + cu_seq_len_k_end: torch.Tensor, + ) -> torch.Tensor: + return fp8_mqa_logits.reference(q, kv, weights, cu_seq_len_k_start, cu_seq_len_k_end) + + +triton_fp8_mqa_logits = TritonFp8MqaLogits() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/comm/__init__.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/comm/__init__.py new file mode 100644 index 000000000000..52a7a9daf028 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/comm/__init__.py @@ -0,0 +1,2 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/comm/cute_dsl_allreduce_add_rmsnorm.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/comm/cute_dsl_allreduce_add_rmsnorm.py new file mode 100644 index 000000000000..e940bc7b11e6 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/comm/cute_dsl_allreduce_add_rmsnorm.py @@ -0,0 +1,370 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""TP4 and TP8 all-reduce fused with the residual add and RMSNorm that consume it.""" + +from __future__ import annotations + +from collections.abc import Sequence + +import torch + +from tensorrt_llm._torch.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.kernels._lamport import ( + TWOSHOT_ABOVE, + LamportWorkspace, + check_rows, + tail, +) + +# A target builds the workspace this entry is handed through the entry itself, +# so `_lamport` stays private to the family. +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.kernels._lamport import ( + build_workspace as build_workspace, +) + +# Mistral Large 4's configuration: TP4, hidden 6144, rms eps 1e-5; the workspace holds +# max_num_tokens, 9216 rows. +_ML4 = dict(world_size=4, hidden=6144, eps=1e-5, max_rows=9216) +# Mistral Large 4 NVFP4's: the same TP width, hidden and eps; its max_num_tokens, and so its +# workspace, is 32768 rows (max_seq_len 32768, no chunked prefill). +_ML4_32 = dict(world_size=4, hidden=6144, eps=1e-5, max_rows=32768) +# Mistral Large 4 NVFP4 at tep8 and tp8: the same hidden, eps and 32768-row workspace over 8 ranks +# on two GB300 nodes of 4 (one NVL72 segment, MNNVL), started as MPI ranks as the engine +# starts them. Both targets make the same two calls per layer at the same rows; the second +# sums tep8's expert-window partials, tp8's expert width-slice partials. +_ML4_32_TEP8 = dict(world_size=8, nodes=2, hidden=6144, eps=1e-5, max_rows=32768) + + +class CuteDslAllreduceAddRmsnorm(OpWrapper): + """`residual += sum over ranks of partial`; `out = rmsnorm(residual) * weight`. + + `partial`, `residual` and `out` are contiguous bf16 `[rows, hidden]`, + `weight` bf16 `[hidden]`, `rows` up to `workspace.max_rows`. `residual` is + updated in place, `out` is written and returned. The sum is fp32 in rank + order on every rank; the residual add is fp32 and rounds once to bf16, and + the norm reads that bf16 residual. `eps` is the workspace's, fixed at build. + + Up to `TWOSHOT_ABOVE` rows the kernel is one-shot: every rank pulls every + peer's whole partial. Above it, two-shot: each rank reduces 1/world of the + rows (a quarter at TP4, an eighth at TP8) and the reduced rows are + gathered back. The two-shot gather moves the reduced rows as bf16, so the + residual add there rounds twice (sum, then sum + residual). The choice is + a function of the row count alone and is made here, identically on every + rank. + + The kernels are `models/mistral_large4/kernels/_tp4_allreduce_fused_rms_kernel.py` + (CuTe DSL), compiled by `build_workspace` (`models/mistral_large4/kernels/_lamport.py`); + `fused` is the one-shot launcher, `fused_twoshot` the two-shot one. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # Mistral Large 4 FP8 and NVFP4: both closing collectives of each of their 72 layers + # (o_proj into the post-attention norm; the MoE or dense MLP into the next layer's input + # norm or the final norm), on captured decode graphs (batches 1/2/4/8/16, padded) and + # eager prefill steps of one workspace. NVFP4 runs without chunked prefill, so one step + # carries up to max_num_tokens, 32768 rows (one prompt, or up to 16 packed). NVFP4 tep8 + # and tp8 make the same two calls per layer over 8 ranks on two nodes (the "NVFP4 tep8" + # cells are both targets'); their engines capture decode graphs at 1/2/4/8/16 padded, and + # the cells replay a step at every batch 1..16. + CELLS: tuple[Cell, ...] = ( + *( + Cell( + why=f"DSpark graph {geometry['world_size']} ranks, {rows} verification/draft rows", + spec=dict(geometry, rows=rows, graph=True), + ) + for geometry in (_ML4_32, _ML4_32_TEP8) + for rows in range(17, 65) + ), + Cell( + why="Mistral Large 4 decode batch 1: a captured 144-launch step, replayed", + spec=dict(_ML4, rows=1, graph=True), + ), + Cell( + why="Mistral Large 4 decode batch 2: a captured 144-launch step, replayed", + spec=dict(_ML4, rows=2, graph=True), + ), + Cell( + why="Mistral Large 4 decode batch 4: a captured 144-launch step, replayed", + spec=dict(_ML4, rows=4, graph=True), + ), + Cell( + why="Mistral Large 4 decode batch 8: a captured 144-launch step, replayed", + spec=dict(_ML4, rows=8, graph=True), + ), + Cell( + why="Mistral Large 4 decode batch 16, max_batch_size: a captured step, replayed", + spec=dict(_ML4, rows=16, graph=True), + ), + Cell( + why="Mistral Large 4 prefill at the last one-shot row count, 3072", + spec=dict(_ML4, rows=3072), + ), + Cell( + why="Mistral Large 4 prefill at the first two-shot row count, 3073", + spec=dict(_ML4, rows=3073), + ), + Cell( + why="Mistral Large 4 prefill at 9215 rows: rank 3's last row block holds 3 rows", + spec=dict(_ML4, rows=9215), + ), + Cell( + why="Mistral Large 4 max_num_tokens, 9216 rows: the whole 9216-row workspace", + spec=dict(_ML4, rows=9216), + ), + Cell( + why="Mistral Large 4 NVFP4 decode batch 1, 32768-row workspace: a captured step, replayed", + spec=dict(_ML4_32, rows=1, graph=True), + ), + Cell( + why="Mistral Large 4 NVFP4 decode batch 2, 32768-row workspace: a captured step, replayed", + spec=dict(_ML4_32, rows=2, graph=True), + ), + Cell( + why="Mistral Large 4 NVFP4 decode batch 4, 32768-row workspace: a captured step, replayed", + spec=dict(_ML4_32, rows=4, graph=True), + ), + Cell( + why="Mistral Large 4 NVFP4 decode batch 8, 32768-row workspace: a captured step, replayed", + spec=dict(_ML4_32, rows=8, graph=True), + ), + Cell( + why="Mistral Large 4 NVFP4 decode batch 16, max_batch_size, 32768-row workspace: replayed", + spec=dict(_ML4_32, rows=16, graph=True), + ), + Cell( + why="Mistral Large 4 NVFP4 prefill at the last one-shot row count, 3072", + spec=dict(_ML4_32, rows=3072), + ), + Cell( + why="Mistral Large 4 NVFP4 prefill at the first two-shot row count, 3073", + spec=dict(_ML4_32, rows=3073), + ), + Cell( + why="Mistral Large 4 NVFP4 prefill at 4096 rows: 256 row groups, one per two-shot CTA", + spec=dict(_ML4_32, rows=4096), + ), + Cell( + why="Mistral Large 4 NVFP4 prefill at 4097 rows: 257 row groups, the two-shot CTAs loop", + spec=dict(_ML4_32, rows=4097), + ), + Cell( + why="Mistral Large 4 NVFP4 prefill at 32767 rows: rank 3's last row block holds 3 rows", + spec=dict(_ML4_32, rows=32767), + ), + Cell( + why="Mistral Large 4 NVFP4 max_num_tokens, 32768 rows: the whole 32768-row workspace", + spec=dict(_ML4_32, rows=32768), + ), + *( + Cell( + why=f"NVFP4 tep8 decode batch {rows}, 8 ranks on 2 nodes: a captured step, replayed", + spec=dict(_ML4_32_TEP8, rows=rows, graph=True), + ) + for rows in range(1, 17) + ), + Cell( + why="NVFP4 tep8 prefill at the last one-shot row count, 3072: 7 peers' partials, 4 off-node", + spec=dict(_ML4_32_TEP8, rows=3072), + ), + Cell( + why="NVFP4 tep8 prefill at the first two-shot row count, 3073: a one-row last group", + spec=dict(_ML4_32_TEP8, rows=3073), + ), + Cell( + why="NVFP4 tep8 prefill at 4096 rows: 128 row groups of 32, one per two-shot CTA", + spec=dict(_ML4_32_TEP8, rows=4096), + ), + Cell( + why="NVFP4 tep8 prefill at 4097 rows: 129 row groups, the last holding one row", + spec=dict(_ML4_32_TEP8, rows=4097), + ), + Cell( + why="NVFP4 tep8 prefill at 8192 rows: 256 row groups, one per two-shot CTA (the cap)", + spec=dict(_ML4_32_TEP8, rows=8192), + ), + Cell( + why="NVFP4 tep8 prefill at 8193 rows: 257 row groups, the two-shot CTAs loop", + spec=dict(_ML4_32_TEP8, rows=8193), + ), + Cell( + why="NVFP4 tep8 prefill at 32767 rows: rank 7's last row block holds 3 rows", + spec=dict(_ML4_32_TEP8, rows=32767), + ), + Cell( + why="NVFP4 tep8 max_num_tokens, 32768 rows: the whole 32768-row workspace over 8 ranks", + spec=dict(_ML4_32_TEP8, rows=32768), + ), + ) + + note = """ + DSpark adds captured steps at every row count 17..64 for the 4-rank and + 8-rank 32768-row workspaces. Together with the existing 1..16 cells, + this covers the target's four verification rows and draft's three rows + per request through batch 16. Eager prefill bounds stay at 32768 rows. + + A target builds one workspace with `max_rows` = its token budget (Mistral + Large 4 FP8: max_num_tokens, 9216; Mistral Large 4 NVFP4 at world + 4 and 8: max_num_tokens, 32768) through `build_workspace(hidden, eps, + max_rows, rank, world)`, imported from this module, and calls this entry + at every row count up to it; there is no stock fallback. + + World size is a cell axis, and so is the topology under it. The + `world_size` 4 cells ran on 4 GB300 of one node, as spawned processes + whose group came up from an exported rendezvous address. The `world_size` + 8 cells ran on 8 MPI ranks over two GB300 nodes of 4 (`nodes=2`, one + NVL72 segment: every peer on NVLink, 4 of the 7 on the other node), whose + group `ensure_process_group` brought up from rank 0's address broadcast + over `MPI.COMM_WORLD`. Neither set says anything about the other world. + + Metadata consumed: none. The row count is the input's. The slot is read + from the workspace's device counter and advanced by the kernel itself, so + a captured graph replays onto whichever slot is current: the decode cells + capture one Mistral Large 4 step (144 launches, one residual carried through + them in place) and replay it between eager prefill calls on one workspace. + + Preconditions, measured on the cells' path (sm_103): + - `build_workspace` is collective (every rank, the same arguments), runs + outside any graph capture and compiles four CuTe DSL kernels (0.5, 0.7, + 0.6 and 1.7 s with the DSL's caches off). The peer buffers are + 3 x world x max_rows x hidden bf16 plus a quarter of that in int32 + flags, in symmetric memory: 1.58 GiB per rank at 9216 rows, 5.63 GiB at + 32768, 11.25 GiB at 32768 and world 8. A first build in a process + measured 2.14 GiB of device memory taken at 9216 rows; a first build + at 32768 rows 6.18 GiB, in 4.7 s; at world 8, with the group already + up, 11.80 GiB in 6.6 s. + - Under MPI, every rank must bring the group up the same way: + `ensure_process_group` takes the rendezvous address from the + environment when `MASTER_ADDR`, `MASTER_PORT`, `RANK` and `WORLD_SIZE` + are all set, and broadcasts rank 0's otherwise. The two-node session + the world-8 cells ran on sets all four in each node's environment (on + the first node: `MASTER_ADDR` that node, `MASTER_PORT` 64710, `RANK` + 0, `WORLD_SIZE` 2). Measured: with them removed on the first node + only, its ranks took the broadcast and the other node's dialled the + environment's address until their 600 s timeout; rank 0 failed after + 601 s with "4/8 clients joined". A launcher that forwards one node's + environment to every rank keeps the ranks in one case; the world-8 + cells removed the four on every rank and ran the broadcast. + - The one-shot publish is a multicast store when the build finds a + multicast address on every rank's buffer (agreed across the group), + and per-peer stores otherwise. Every cell ran with multicast on, which + the GB300 nodes they ran on provide, at world 8 across the two nodes + too; the per-peer path is in no cell. + - Every rank calls with the same row count. With three ranks at 16 rows + and one at 8, the three had not completed after 20 s (the fourth had); + at 4096 rows against 2048, none had. A disagreeing rank hangs the group + and nothing raises. + - `partial`, `residual` and `out` contiguous bf16 `[rows, hidden]`, + `weight` bf16 `[hidden]`, all 16 B aligned. A row window of a larger + contiguous buffer is accepted and answered correctly (one-shot at 16 + rows: bit-exact; two-shot at 4096 rows: inside the two-shot band). The + launcher refuses the rest before anything launches, leaving the slot + counter as it was (probed at 16 rows), each with a `ValueError: ... + when calling: `: a strided view (`Mismatched + input_tensor.strides[0] on argument #0`; `norm_output_tensor` on #3), + another dtype (`Mismatched Tensor on argument #0`; #2 for an fp32 + weight), another hidden (`Mismatched input_tensor.shape[1] on argument + #0`), operands of different row counts (`Mismatched + residual_tensor.shape[0] on argument #1`), a pointer off 16 B + (`Misaligned Tensor data on argument #0`). + - The kernels launch with PDL and wait on their predecessor before any + load; the one-shot kernel releases its dependents at entry (read in + the kernel source, not measured). So a PDL-launched successor must + wait on it before reading `out` or `residual`. + + Two-shot schedule: `min(ceil(rows / (4 x world)), 256)` producer CTAs, + each owning (4 x world)-row groups (4 rows per rank, block g x world + r + rank r's), plus 1024 consumer CTAs. At world 4, 16-row groups: up to + 4096 rows each producer takes one group, from 4097 they loop. At world + 8, 32-row groups: up to 8192 rows one group each, from 8193 they loop. + The block owner sums every rank's rows in fp32 in rank order 0..world-1; + at world 8 the two-shot residual was bit-exact to that arithmetic + (bf16(bf16(sum) + residual)) at every two-shot cell, 3073 to 32768 rows, + and the one-shot residual (3072 rows, every replayed decode step) to the + reference's. + + Cost after a large call: each call clears the slot the next call will + use, as many vectors as that slot last held, so the second call after a + large call clears that call's rows' worth, and a small call there costs + more than it does otherwise. + + Hard limits (sm_103; the kernels' index arithmetic read, and probed where + the 32768-row workspace reaches it) + peer-buffer offset (slot x world + region) x max_rows x hidden + row x + hidden + col, Int64 in every kernel. The buffer + passes 2^31 elements from max_rows 29,128 at world + 4 and 14,564 at world 8; at 32768 it holds + 2,415,919,104 (world 4) and 4,831,838,208 (world 8, + past 2^32). Probed at 32768, world 4: rank 3's + one-shot publish on slot 2 (from element + 2,214,592,512, past 2^32 bytes too) lands in place + and is cleared, and a two-shot launch clears slot 2 + to element 2,415,919,103: passed; two-shot moves + data through regions 0 and 1 only, below + 2,013,265,920. World 8: rank 7's one-shot publish on + slot 2 (from element 4,630,511,616, past 2^32 + elements) lands in place and is cleared, a 32768-row + two-shot call on slot 2 (regions 0 and 1 from + element 3,221,225,472, past 2^31) passes its gate, + and a two-shot launch clears slot 2 to element + 4,831,838,207: passed. + workspace numel max_rows x hidden, the compiled Int32 extent: + passes 2^31 at 349,526 rows at 6144, not probed + buffer mode-0 world x max_rows x hidden: passes 2^31 at 87,382 + stride rows at world 4, 43,691 at world 8 (hidden 6144), + computed, not probed + rows x hidden input offsets and the vector counts in Int32: pass + 2^31 at 349,526 rows, not probed (rows <= max_rows) + rows > max_rows `is_valid` refuses: the kernel would write past every + peer's buffer (not driven unguarded) + """ + + def __call__( + self, + partial: torch.Tensor, + residual: torch.Tensor, + weight: torch.Tensor, + out: torch.Tensor, + workspace: LamportWorkspace, + ) -> torch.Tensor: + compiled = workspace.fused_twoshot if partial.shape[0] > TWOSHOT_ABOVE else workspace.fused + compiled(partial, residual, weight, out, residual, *tail(workspace, fused=True)) + return out + + def reference( + self, + partials: Sequence[torch.Tensor], + residual: torch.Tensor, + weight: torch.Tensor, + out: torch.Tensor, + workspace: LamportWorkspace, + ) -> tuple[torch.Tensor, torch.Tensor]: + """`(norm, new_residual)`, both bf16; `residual` is not modified. + + `partials` is the group's partials in rank order (the collective + exception to mirroring `__call__`). The sum is fp32 in rank order. + `new_residual` is `bf16(sum + residual)`, rounded once. The norm is + computed in fp64 from that bf16 residual with `workspace.eps` and + rounded once. `out` is ignored. + """ + acc = torch.zeros(residual.shape, dtype=torch.float32, device=residual.device) + for p in partials: + acc += p.float() + new_residual = (acc + residual.float()).to(torch.bfloat16) + x = new_residual.double() + inv_rms = torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + workspace.eps) + norm = (x * inv_rms * weight.double()).to(torch.bfloat16) + return norm, new_residual + + def is_valid( + self, + partial: torch.Tensor, + residual: torch.Tensor, + weight: torch.Tensor, + out: torch.Tensor, + workspace: LamportWorkspace, + ) -> None: + check_rows(int(partial.shape[0]), workspace) + + +cute_dsl_allreduce_add_rmsnorm = CuteDslAllreduceAddRmsnorm() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/gemm/__init__.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/gemm/__init__.py new file mode 100644 index 000000000000..52a7a9daf028 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/gemm/__init__.py @@ -0,0 +1,2 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/gemm/cute_dsl_fp8_block_scale_linear.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/gemm/cute_dsl_fp8_block_scale_linear.py new file mode 100644 index 000000000000..874c32820cac --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/gemm/cute_dsl_fp8_block_scale_linear.py @@ -0,0 +1,74 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""SM107 FP8 linear with K128 quantization and replicated K32 UE8M0 scales.""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch.modeling_v2.catalog._op import Arch, Cell, OpWrapper, true_fp32_matmul +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.gemm.fp8_prequantized_swap_ab_gemm import ( # noqa: E501 + PROJECTIONS, + _per_element_scale, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.quantization.fp8_quantize_1x128_packed_ue8m0 import ( # noqa: E501 + fp8_quantize_1x128_packed_ue8m0 as quantize, +) + + +def prepare_weight_scale(packed: torch.Tensor, k: int) -> torch.Tensor: + """MN-major int32 K128 scales -> R128c4 uint8 K32 scales, without requantization.""" + rows = packed.shape[0] + exponents = packed.reshape(-1).view(torch.uint8).reshape(rows, -1)[:, : k // 128] + return torch.ops.trtllm.block_scale_interleave(exponents.repeat_interleave(4, dim=1)) + + +def _weight_scales(scale: torch.Tensor, n: int, k: int) -> torch.Tensor: + """Unswizzle R128c4 into logical row/K32 scales using tensor indexing only.""" + padded_n = (n + 127) // 128 * 128 + columns = k // 32 + exponent = ( + scale.reshape(padded_n // 128, columns // 4, 32, 4, 4) + .permute(0, 3, 2, 1, 4) + .reshape(padded_n, columns)[:n] + ) + return (exponent.to(torch.int32) << 23).contiguous().view(torch.float32) + + +class CuteDslFp8BlockScaleLinear(OpWrapper): + """Quantize BF16 input per 128 values, then apply FP8 weights with K128 scales. + + Every K128 exponent is replicated over its four K32 slots. The Rubin MXFP8 + kernel therefore consumes the original FP8 values and scales exactly. + Weight scales are prepared once when the model constructs its layer views. + """ + + ARCHS = frozenset({Arch.SM_107}) + CELLS = tuple( + Cell(f"SM107 {name}, M={m}", dict(m=m, n=n, k=k)) + for name, n, k in PROJECTIONS + for m in (1, 4, 16, 64, 129, 2048, 32768) + ) + + def __call__( + self, input: torch.Tensor, weight: torch.Tensor, weight_scale: torch.Tensor + ) -> torch.Tensor: + codes, scale = torch.ops.trtllm.fp8_quantize_1x128_packed_ue8m0( + input, use_r128c4_layout=True + ) + return torch.ops.trtllm.cute_dsl_mxfp8_gemm_rubin(codes, weight, scale, weight_scale) + + def reference( + self, input: torch.Tensor, weight: torch.Tensor, weight_scale: torch.Tensor + ) -> torch.Tensor: + codes, scale = quantize.reference(input) + m, k = input.shape + n = weight.shape[0] + activation = codes.float() * _per_element_scale(scale, m, k) + weights = weight.float() * _weight_scales(weight_scale, n, k).repeat_interleave(32, dim=1) + with true_fp32_matmul(): + output = activation @ weights.t() + return output.to(torch.bfloat16) + + +cute_dsl_fp8_block_scale_linear = CuteDslFp8BlockScaleLinear() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/gemm/fp8_prequantized_swap_ab_gemm.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/gemm/fp8_prequantized_swap_ab_gemm.py new file mode 100644 index 000000000000..71074050f224 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/gemm/fp8_prequantized_swap_ab_gemm.py @@ -0,0 +1,391 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""FP8 128x128 block-scale linear on operands that are already quantized: one DeepGEMM +``fp8_gemm_nt`` through ``torch.ops.trtllm.fp8_prequantized_swap_ab_gemm``. + +Sourcing tier 1: a ``torch.library.custom_op`` in +``tensorrt_llm/_torch/custom_ops/torch_custom_ops.py``, the call +``FP8BlockScalesLinearMethod.apply`` (``_torch/modules/linear.py``) makes on the sm_100 +family when its input arrives as an ``(e4m3, int32 scale)`` pair. Its body is a +single-tactic ``AutoTuner.choose_one`` that exists only to drive DeepGEMM's JIT warmup, the +output ``torch.empty``, and one ``deep_gemm.fp8_gemm_nt`` (plus one copy for an activation +scale in another layout than the quantizer's; see `note`). + +This is the GEMM half of ``torch.ops.trtllm.fp8_swap_ab_gemm``, which quantizes its bf16 +input itself first. The quantize is a separate op, +``torch.ops.trtllm.fp8_quantize_1x128_packed_ue8m0(x, use_r128c4_layout=False)``, whose +``(codes, scale)`` this entry takes as they come. The engine's ``GatedMLP`` +(``_torch/modules/gated_mlp.py``) feeds its down projection the same way from +``torch.ops.trtllm.silu_and_mul_fp8_quantize_1x128_packed_ue8m0``. + +The Mistral Large 4 NVFP4 checkpoint's FP8 block linears are its ``FP8_PB_WO`` layers: wq, wo, +indexer wq_b, the shared experts' and layer 0's w1/w2/w3. Each stores its block scale as +``.weight_scale``, fp32 ``[N/128, 1, K/128, 1]`` (one per 128 x 128 block, every value a +power of two); viewed ``[N/128, K/128]`` it is the fp32 block scale the weight preparation +in `note` takes, so the operands this entry sees are those of any 128 x 128 block-FP8 +checkpoint. + +At dep4 (attention data parallel) every rank holds all 64 attention heads, so wq is the +whole ``[8192, 6144]`` and wo the whole ``[6144, 8192]``, and each call carries only the +rank's own tokens. The dense layer-0 MLP runs replicated on each rank's own tokens there, so +its gate_up is the whole ``[w1 ; w3]``, ``[49152, 6144]``, and its down the whole +``[6144, 24576]``. + +At tep8 and tp8 (tensor parallel 8, attention data parallel off; tp8 differs from tep8 only +in the routed experts, which do not call this entry) every rank holds 8 of the 64 heads, so +wq is the rank's ``[1024, 6144]`` row slice and wo its ``[6144, 1024]`` K slice; the dense +layer-0 MLP and the shared experts shard eight ways, so the dense down is ``[6144, 3072]``, +the shared gate_up the stacked ``[w1 ; w3]`` slices, ``[1536, 6144]``, and the shared down +``[6144, 768]``, whose K of six 128-wide blocks leaves the last int32 of each scale row half +empty (see `note`). Every call carries every token of the step. The dense gate_up there is +``[6144, 6144]``, the tep4 dense down's shape. The projections named "at tep8" are both +targets' shards. +""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch.modeling_v2.catalog._op import Arch, Cell, OpWrapper, true_fp32_matmul + +#: Every FP8 block-scale projection of the Mistral Large 4 targets, as the weight's [N, K]: per rank +#: at tep4 (FP8 and NVFP4 alike), then the NVFP4 target's attention projections at dep4, where +#: attention data parallel leaves the 64 heads unsplit, and its dense layer-0 MLP at dep4, which +#: runs replicated on each rank's own tokens; then the shards of the NVFP4 tep8 and tp8 +#: deployments, eight ways (their dense gate_up, [6144, 6144], is the tep4 dense down's shape above). +PROJECTIONS = ( + ("wq (q_proj, 16 heads x 128)", 2048, 6144), + ("wo (o_proj, row-parallel K slice of 8192)", 6144, 2048), + ("indexer wq_b (32 heads x 128, replicated)", 4096, 6144), + ("dense layer-0 gate_up (TP4 of 2 x 24576)", 12288, 6144), + ("dense layer-0 down (TP4 of 24576)", 6144, 6144), + ("shared-expert gate_up (TP4 of 2 x 6144)", 3072, 6144), + ("shared-expert down (TP4 of 6144)", 6144, 1536), + ("wq at dep4 (q_proj, all 64 heads x 128)", 8192, 6144), + ("wo at dep4 (o_proj, the whole K of 64 heads x 128)", 6144, 8192), + ("dense layer-0 gate_up at dep4 (replicated, 2 x 24576)", 49152, 6144), + ("dense layer-0 down at dep4 (replicated, the whole K of 24576)", 6144, 24576), + ("wq at tep8 (q_proj, 8 heads x 128)", 1024, 6144), + ("wo at tep8 (o_proj, row-parallel K slice of 8 heads x 128)", 6144, 1024), + ("dense layer-0 down at tep8 (TP8 of 24576)", 6144, 3072), + ("shared-expert gate_up at tep8 (TP8 of 2 x 6144)", 1536, 6144), + ("shared-expert down at tep8 (TP8 of 6144)", 6144, 768), +) + +#: The NVFP4 target's token budget (max_num_tokens; with chunked prefill off one context step +#: carries up to that many rows; per rank at dep4, where wq, wo and the replicated dense layer 0 +#: see the rank's own tokens only; at tep8 and tp8 every rank's call carries all of the step's +#: tokens), the row count up to which the eager cells drive every m +#: (the FP8 target's max_num_tokens), +#: the window DeepGEMM's tile layout is constant on (see `note`), which the cells above +#: EVERY_M_TO drive at its edges and one interior row, and every decode batch a CUDA graph +#: can be captured at: max_batch_size 16 bounds both the padded 1/2/4/8/16 the gates capture +#: and the LLM API default's unpadded 1..16. +MAX_NUM_TOKENS = 32768 +EVERY_M_TO = 9216 +LAYOUT_WINDOW = 16 +GRAPH_BATCHES = tuple(range(1, 17)) + +#: The [N, K] whose every-m sweep (LARGE_EVERY_M) or windowed sweep (LARGE_WINDOWED) runs past +#: the 10 s cell budget: warm bodies of 10.0 s (49152 x 6144, every m), 21.7 s (49152 x 6144, +#: windowed) and 9.8 s (6144 x 24576, windowed) on sm_103. Their cells carry `large`, which +#: pre-merge deselects; the pre-merge cells after the graph cells keep a share of each there. +LARGE_EVERY_M = ((49152, 6144),) +LARGE_WINDOWED = ((49152, 6144), (6144, 24576)) + +_AT_CALL_SITE = dict(output_dtype=torch.bfloat16, disable_ue8m0_cast=True) +_LARGE_WHY = "; large: past the 10 s cell budget at this N x K" + + +def _per_element_scale(scale: torch.Tensor, rows: int, k: int) -> torch.Tensor: + """Unpack an int32 [rows, ceil(k / 512)] UE8M0 scale into fp32 [rows, k]. + + Byte ``j`` (little-endian) of int32 column ``c`` is the biased exponent ``e`` of the + scale of K-block ``4 c + j``, worth ``2 ** (e - 127)``. Read by logical index, so any + strides are fine. Exact for ``e`` in 1..254 (every scale a 1x128 quantizer with an amax + floor writes); 0 would decode to 0.0 rather than 2^-127, 255 to inf rather than NaN. + """ + shifts = torch.tensor((0, 8, 16, 24), dtype=torch.int32, device=scale.device) + exponent = ((scale.unsqueeze(-1) >> shifts) & 0xFF).reshape(rows, -1)[:, : k // 128] + return (exponent << 23).view(torch.float32).repeat_interleave(128, dim=1) + + +class Fp8PrequantizedSwapAbGemm(OpWrapper): + """``(activation * a_scale) @ (weight * w_scale).T`` in fp32, rounded to `output_dtype`. + + `activation` is e4m3 ``[M, K]`` with one UE8M0 scale per row and 128-wide K block; + `weight` is e4m3 ``[N, K]`` with one per 128 x 128 block, repeated over each block's 128 + rows. Both scales are int32, four exponent bytes per element, MN-major (see `note`). + The GEMM and the scaling are inside the call; quantizing the activation and preparing + the weight are not. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # Sixteen projections call it, and N and K are compiled into DeepGEMM's kernel, so each is + # a kernel set of its own. Each is driven three times, every m its own call (DeepGEMM + # picks its tile layout by M): eagerly at every m in 1..9216; eagerly over 9217..32768 + # at three rows of each 16-row window, both of its edges among them, which reach every + # layout picked there and both ends of each run of one layout; and captured in a CUDA + # graph at every decode batch, 1..16. Three of those sweeps are `large` (LARGE_EVERY_M, + # LARGE_WINDOWED); four cells after the graph cells drive a share of them in pre-merge. + CELLS: tuple[Cell, ...] = ( + *( + Cell( + why=f"eager {n}x{k} {site}: every m in 1..{EVERY_M_TO}, each its own " + f"call, bitwise the leading rows of the {EVERY_M_TO}-row call, which " + f"meets the reference{_LARGE_WHY if (n, k) in LARGE_EVERY_M else ''}", + spec=dict( + n=n, + k=k, + m_range=(1, EVERY_M_TO), + **_AT_CALL_SITE, + **(dict(large=True) if (n, k) in LARGE_EVERY_M else {}), + ), + ) + for site, n, k in PROJECTIONS + ), + *( + Cell( + why=f"eager m {EVERY_M_TO + 1}..{MAX_NUM_TOKENS} {n}x{k} {site}: both edges " + f"and one interior row of every {LAYOUT_WINDOW}-row window, each its own " + f"call, bitwise the leading rows of the {MAX_NUM_TOKENS}-row call, which " + f"meets the reference{_LARGE_WHY if (n, k) in LARGE_WINDOWED else ''}", + spec=dict( + n=n, + k=k, + m_range=(EVERY_M_TO + 1, MAX_NUM_TOKENS), + m_window=LAYOUT_WINDOW, + **_AT_CALL_SITE, + **(dict(large=True) if (n, k) in LARGE_WINDOWED else {}), + ), + ) + for site, n, k in PROJECTIONS + ), + *( + Cell( + why=f"graphs {n}x{k} {site}: every decode batch 1..16, each captured, then " + "replayed on new operands bitwise equal to eager", + spec=dict(n=n, k=k, graph_m=GRAPH_BATCHES, **_AT_CALL_SITE), + ) + for site, n, k in PROJECTIONS + ), + Cell( + why="eager m 1..1024 49152x6144 dense layer-0 gate_up at dep4, pre-merge's share " + "of its large every-m sweep: every m its own call, 17 of the 18 tile layouts that " + "shape selects on 152 SMs (not the swap A/B block_m 256 one, first at m 2161), " + "bitwise the leading rows of the 1024-row call, which meets the reference", + spec=dict(n=49152, k=6144, m_range=(1, 1024), **_AT_CALL_SITE), + ), + Cell( + why="eager m 21761..22016 49152x6144 dense layer-0 gate_up at dep4, pre-merge's " + "share of its large windowed sweep: every m from 21846 on writes past byte 2^31 of " + "the bf16 output; both edges and one interior row of every 16-row window, across " + "a switch between the two tile layouts of 9217..32768, each its own call, bitwise " + "the leading rows of the 22016-row call, which meets the reference", + spec=dict( + n=49152, + k=6144, + m_range=(21761, 22016), + m_window=LAYOUT_WINDOW, + **_AT_CALL_SITE, + ), + ), + *( + Cell( + why=f"eager m {MAX_NUM_TOKENS - 255}..{MAX_NUM_TOKENS} {n}x{k} {site}, " + "pre-merge's share of its large windowed sweep: the top of the envelope, both " + f"edges and one interior row of every {LAYOUT_WINDOW}-row window, across a " + "switch between the tile layouts of 9217..32768, each its own call, bitwise " + f"the leading rows of the {MAX_NUM_TOKENS}-row call, which meets the reference", + spec=dict( + n=n, + k=k, + m_range=(MAX_NUM_TOKENS - 255, MAX_NUM_TOKENS), + m_window=LAYOUT_WINDOW, + **_AT_CALL_SITE, + ), + ) + for site, n, k in PROJECTIONS + if (n, k) in LARGE_WINDOWED + ), + Cell( + why="default disable_ue8m0_cast=False on int32 scales, wq 2048x6144: bitwise " + "equal to True, the call site's value", + spec=dict( + n=2048, + k=6144, + m_points=(1, 5, 16, 300, EVERY_M_TO, MAX_NUM_TOKENS), + output_dtype=torch.bfloat16, + disable_ue8m0_cast=False, + ), + ), + ) + + note = """ + Operands (sm_103, the layouts the cells drive). + activation e4m3 [M, K], unit stride along K. What + `fp8_quantize_1x128_packed_ue8m0(x, use_r128c4_layout=False)` returns. + activation_scale int32 [M, ceil(K/512)], strides (1, align(M, 4)): MN-major, each + int32 holding the exponent bytes of four consecutive 1x128 K blocks + (`_per_element_scale`). The storage the quantizer allocates ends at + row M of the last column; the gaps between columns are never read + for a valid row (the cells fill them with 0xFF bytes, UE8M0 NaN). + At K 768 (six blocks) bytes 2 and 3 of each row's second int32 are + empty: the quantizer writes 0 there, the cells 0xFF, and 0x00 or + 0xFF there (in this scale or the weight's) gave bitwise one result + at ten m from 1 to 32768, while 0xFF in a live byte (block 5's) made + every output element NaN; a test pins both. + weight e4m3 [N, K] and weight_scale int32 [N, ceil(K/512)], strides (1, N), + prepared once at load the way `FP8BlockScalesLinearMethod. + transform_weights` does on the DeepGEMM path: + `resmooth_to_fp8_e8m0(weight, scale_fp32)` (re-derives each 128 x 128 + block's power-of-two scale from its amax), then + `transform_sf_into_required_layout(scale, mn=N, k=K, + recipe=(1, 128, 128), is_sfa=False)` (repeats each block's scale per + row and packs it, 0 in K 768's empty bytes). `scale_fp32` is the NVFP4 + checkpoint's `.weight_scale` [N/128, 1, K/128, 1] viewed + [N/128, K/128], or the FP8 checkpoint's `.qscale_weight` + [N/128, K/128]; the cells draw block scales 2^-15..2^-8, the range the + NVFP4 checkpoint's 404 FP8 block scales span. The tep4 and dep4 + deployments' loaders (`_block_sf` in their `weights.py`) assert the stored + scales are powers of two and pack them with the same + `transform_sf_into_required_layout` call, without the resmooth. + output a new [M, N] tensor; DeepGEMM's bf16 output is its fp32 result + rounded once (a float32 output, rounded to bf16, was bitwise equal + to it in a probe). + Metadata consumed: none. + + Inside the call, besides the GEMM. An activation scale whose strides are not + (1, align(M, 4)) is first copied into that layout by the runner, one extra copy kernel + (a row-major int32 scale gave a bitwise-equal result). With int32 scales + `disable_ue8m0_cast` selects nothing: DeepGEMM takes the same packed-UE8M0 path either + way (a cell pins False against True). A row-strided activation (row stride K + 128, + the TMA descriptor takes the stride) gave a bitwise-equal result too; a test pins it. + + Refused loudly (each pinned by a test): + - the quantizer's default layout, `use_r128c4_layout=True` (a 1-D uint8 scale): + `RuntimeError: dimensionality of sizes (1) must match dimensionality of strides (2)`, + from the runner's re-stride; + - an fp32 activation scale with `disable_ue8m0_cast=True`: DeepGEMM's + `Unsupported architecture or scaling factor types`; + - a weight scale that is not MN-major: DeepGEMM's `check_sf_layout` assertion + `sf.stride(-2) == 1 or mn == 1`. + An fp32 [M, K/128] activation scale with `disable_ue8m0_cast=False` is packed inside the + call by DeepGEMM (its JIT kernel `pack_fp32_into_ue8m0`): powers of two gave the int32 + path's result bitwise; a scale that is not one faulted the device (`CUDA error: + unspecified launch failure`), which leaves the context unusable -- the packer asserts a + power of two on the device (`smxx_layout.cuh`). Probed only, at no cell. + + Batch invariance. Every eager cell asserts it: each m-row call's output is bitwise the + leading m rows of the cell's top call on the same operands -- the 9216-row call at every + m in 1..9216, the 32768-row call at both edges and one interior row of every 16-row + window of 9217..32768 -- at every shape, and the pre-merge cells against their own top + calls of 1024, 22016 and 32768 rows. A row's result does not depend on how many rows + follow it. + + Tile layout by M. DeepGEMM picks one config (swap A/B or not, block M and N, cluster, + stages) per call from M, N, K and the SM count (`get_layout_candidates` and `compare` in + DeepGEMM's `csrc/jit_kernels/heuristics/sm100.hpp`). It reads M only as + ceil(M / block_m) and, for the unswapped block_m, as M <= 32 and M <= 64; every candidate + block_m is a multiple of 16, so the choice is constant on each window + [16 j + 1, 16 j + 16]. Measured with `DG_PRINT_CONFIGS=1` at + every M in 1..32768 and the sixteen shapes, on sm_103: constant on every window; 18 to + 25 configs per shape over 1..9216, and 2 to 8 over 9217..32768 (swap A/B with block_m 176 + to 256 and cluster 2 along N, or no swap with 128 x 224 or 128 x 256 tiles and cluster 2 + along M), each of which 1..9216 selects too. Over 9217..32768 they alternate in 82 to 188 + runs (every run 128 rows long at 49152 x 6144), and both edges of every run are rows the + windowed cells drive. These are sm_103's configs; the heuristic takes the SM count, so another + count may select others. + + DeepGEMM JIT. Each (N, K, config) is a kernel compiled with nvcc on its first use, about + 2.5 to 3 s each, and cached on disk (`$HOME/.dj`, or `DG_JIT_CACHE_DIR`): over M 1..32768 + the sixteen shapes select 323 kernels (19 at each of 8192 x 6144 and 6144 x 8192, which the + warmup grid below compiled in 48.9 and 48.2 s, 18 at 49152 x 6144 and 19 at 6144 x 24576, + in 47.7 and 49.1 s; at the tep8 shards 25 at 1024 x 6144, 19 at 6144 x 1024, 19 at + 6144 x 3072, 24 at 1536 x 6144 and 19 at 6144 x 768, in 72.4, 55.2, 56.5, 70.5 and + 55.3 s), so on a cold cache an every-m cell took 50 to 65 s, against 1.4 to + 3.1 s warm (4.7 s at 6144 x 24576, 10.0 s at 49152 x 6144; a windowed cell's body: 0.7 to + 5.7 s warm, 9.8 and 21.7 s at the two dense layer-0 dep4 shapes, peak 1.6 to 14.4 GiB + above what the process held before). Calling the op at every M of the grid + its single tactic tunes over (`deep_gemm_jit_warmup_buckets` in `_torch/utils.py`, 520 + values up to 8192; the grid is capped at 8192, so it is the same at max_num_tokens 9216 + and 32768) compiles all 323: after it, every M in 1..32768 at each shape compiled + nothing. A first call outside a warm cache pays the compile; never time one. + + Uncertified: output_dtype float32 (DeepGEMM accepts it; probed at 32768 x 49152, whose + 6442450944 output bytes pass 2^32: 86.8 fp32 ulp of the row's scale from the fp32 + reference, 245.4 at 32768 x 6144 with K 24576, 20.1 to 80.6 at the tep8 shards); M above + 32768; the rows of + 9217..32768 the windowed cells skip, 13 of every 16 -- each runs its window's config + (measured) and differs from the rows driven only in how full its last M tile is, which + the driven rows vary across windows, so uniform behaviour there is inferred, not + measured; any N x K not in `PROJECTIONS`; K not a multiple of 512 other than 768, where + the last int32 of a scale row is partly padding too; exponent bytes 0 + and 255 in a live slot (no quantizer with an amax floor writes them); captured batches + above 16; a + CUDA-graph capture with no call before it on the capture stream (the cells warm each + shape first, as the engine does). + + Hard limits (sm_103). The envelope reaches one boundary, byte 2^31 of the bf16 output at + N 49152; the others are computed, not probed. The tep8 and tp8 shards reach none (their + largest output, 32768 x 6144 bf16, is 402653184 bytes). + M, N, K int in DeepGEMM's host code 2^31 unprobed + activation elems M * K, largest 32768 * 24576 805306368 < 2^31 unprobed + weight elems N * K, largest 49152 * 6144 301989888 < 2^31 unprobed + output elems M * N, largest 32768 * 49152 1610612736 < 2^31 unprobed + row offsets m * K, m * N in 32 bits 2^31 / 24576 = 87381 unprobed + and 2^31 / 49152 = + 43690 rows > 32768 + output bytes M * N * 2 (bf16), largest 32768 * at N 49152 every m probed at 22016 + 49152 * 2 = 3221225472 > 2^31 from 21846 on and 32768 rows: + writes past 2^31 passed, every row + in the band + TMA descriptors carry 64-bit extents and strides. No argument is an index carried + through fp32. + """ + + def __call__( + self, + activation: torch.Tensor, + activation_scale: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + output_dtype: torch.dtype = torch.bfloat16, + disable_ue8m0_cast: bool = False, + ) -> torch.Tensor: + return torch.ops.trtllm.fp8_prequantized_swap_ab_gemm( + activation, + activation_scale, + weight, + weight_scale, + output_dtype=output_dtype, + disable_ue8m0_cast=disable_ue8m0_cast, + ) + + def reference( + self, + activation: torch.Tensor, + activation_scale: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + output_dtype: torch.dtype = torch.bfloat16, + disable_ue8m0_cast: bool = False, + ) -> torch.Tensor: + """Dequantize both operands to fp32 (exact: a power-of-two scale times an e4m3 + code), multiply in fp32 with TF32 off (`true_fp32_matmul`), round once. + + Takes int32 packed scales only. `disable_ue8m0_cast` is ignored: on int32 scales it + changes nothing. + """ + m, k = activation.shape + n = weight.shape[0] + a = activation.float() * _per_element_scale(activation_scale, m, k) + w = weight.float() * _per_element_scale(weight_scale, n, k) + with true_fp32_matmul(): + out = a @ w.t() + return out.to(output_dtype) + + +fp8_prequantized_swap_ab_gemm = Fp8PrequantizedSwapAbGemm() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/moe/__init__.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/moe/__init__.py new file mode 100644 index 000000000000..52a7a9daf028 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/moe/__init__.py @@ -0,0 +1,2 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/moe/renorm_moe_routing_op.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/moe/renorm_moe_routing_op.py new file mode 100644 index 000000000000..c0d7ff8782a6 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/moe/renorm_moe_routing_op.py @@ -0,0 +1,369 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Renormalized MoE routing: top-k over the raw router logits, then a softmax +over just those k, in one trtllm call. + +The cells, built by `_ml4` and on `_ML4_32K` and `_ML4_131K`, are the Mistral Large 4 +router's: 128 experts, top-4, fp32 logits, bf16 weights -- `_ml4` at the FP8 target's +max_num_tokens 9216, `_ML4_32K` at the NVFP4 target's 32768 (and the NVFP4 dep8 +deployment's chunks of the 8-rank gathered rows, with its decode graphs' gathered +8 x 1..16), `_ML4_131K` at the NVFP4 dep4 deployment's gathered 4 x 32768. +""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +def _ml4(**overrides) -> dict: + """The Mistral Large 4 router: 128 experts, top-4, the fp32 logits of its + replicated router GEMM, bf16 weights for the routed experts, at the FP8 + target's max_num_tokens 9216.""" + return dict( + dict( + tokens=9216, + num_experts=128, + topk=4, + logits_dtype=torch.float32, + output_dtype=torch.bfloat16, + dist="randn", + magnitude=3.0, + ), + **overrides, + ) + + +# The Mistral Large 4 router at the NVFP4 target's envelope: max_num_tokens 32768 +# with chunked prefill off, so one step routes up to 32768 rows. A literal, so +# the cells built on it read without running this module. +_ML4_32K = dict( + tokens=32768, + num_experts=128, + topk=4, + logits_dtype=torch.float32, + output_dtype=torch.bfloat16, + dist="randn", + magnitude=3.0, +) + +# The Mistral Large 4 router at the NVFP4 dep4 deployment: attention data parallel, so +# each of the 4 ranks zero-pads its rows to T_max = max(all_rank_num_tokens) +# and the router runs over the gathered 4 x T_max rows, up to 4 x 32768. +# A literal, like `_ML4_32K`. +_ML4_131K = dict( + tokens=131072, + num_experts=128, + topk=4, + logits_dtype=torch.float32, + output_dtype=torch.bfloat16, + dist="randn", + magnitude=3.0, +) + + +class RenormMoeRoutingOp(OpWrapper): + """`norm_topk_prob=True` routing. Returns `(topk_ids, topk_weights)` -- IDS FIRST. + + Per token: `ids` = the `topk` largest `router_logits`, ranked in the + logits' own dtype, by decreasing value, ties to the LOWER expert index + (a stable descending sort -- `torch.topk` breaks ties the other way, and at + 512 bf16 experts ties are common: it disagrees on ~30% of rows); + `weights` = an fp32 softmax over just those `topk` logits, cast to + `output_dtype` afterwards. Equal to HF's softmax-then-top-k-then-divide up + to rounding (5.96e-08 at 512 / top-10; at 128 / top-4 on fp32 logits the + bf16 weights are bitwise those of the softmax over all 128). + + `ids` is `[T, topk]` int32; `weights` `[T, topk]` in `output_dtype`, which + defaults to fp32 -- not the logits' dtype. Sibling routing ops return + `(weights, ids)`; both are `[T, topk]`, so a swapped unpack is no shape + error. + + Fusion boundary: routing only. The router GEMM is the caller's, and so is + expert parallelism -- every column of the row is ranked, so a TEP target + passes the full 128-expert row and its expert runner keeps the ids in its + own window. A DEP target gathers every rank's rows, zero padding + included, before the router, and routes all of them on every rank. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why=( + "Mistral Large 4 decode, CUDA graphs 1/2/4/8/16 (the captured, padded " + "batches): each captured, replayed on new logits bitwise equal to eager" + ), + spec=_ml4(tokens=16, graph_tokens=(1, 2, 4, 8, 16)), + ), + Cell( + why=( + "Mistral Large 4 eager T 1..9216: every row count up to max_num_tokens, " + "1 to 2304 blocks, each a call on the leading T rows of one set (~1 s)" + ), + spec=_ml4(tokens_range=(1, 9216)), + ), + Cell( + why=( + "Mistral Large 4 ties: integer fp32 logits 0..2 at 9216 tokens, every " + "row ties inside its top-4 and most at the 4th/5th boundary" + ), + spec=_ml4(dist="coarse"), + ), + Cell( + why=( + "Mistral Large 4 wide logits (sigma 100) at 9216 tokens: the lower " + "weights underflow through bf16 subnormals to 0" + ), + spec=_ml4(magnitude=100.0), + ), + Cell( + why=( + "Mistral Large 4 32768 eager T 1..32768: every row count of the NVFP4 " + "target's max_num_tokens (a whole 32768-token prompt is one step), 1 " + "to 8192 blocks -- the launcher's cap, reached at T 32765..32768 " + "(8191 blocks at 32764) with one token per warp -- each a call on " + "the leading T rows of one set (~2 s)" + ), + spec=dict(_ML4_32K, tokens_range=(1, 32768)), + ), + Cell( + why=( + "Mistral Large 4 32768 ties: integer fp32 logits 0..2 at 32768 tokens, " + "every row ties inside its top-4 and at the 4th/5th boundary" + ), + spec=dict(_ML4_32K, dist="coarse"), + ), + Cell( + why=( + "Mistral Large 4 32768 wide logits (sigma 100) at 32768 tokens: the " + "lower weights underflow through bf16 subnormals to 0" + ), + spec=dict(_ML4_32K, magnitude=100.0), + ), + Cell( + why=( + "Mistral Large 4 32768 past the block cap: 65537 rows on 8192 blocks, " + "each warp looping over tokens 32768 apart (warp 0 three times); " + "outside the NVFP4 envelope, for a step past 32768 rows" + ), + spec=dict(_ML4_32K, tokens=65537), + ), + Cell( + why=( + "Mistral Large 4 dep4 eager T 32769..131072: every row count of the " + "dep4 deployment's gathered step (4 x T_max, T_max up to max_num_tokens " + "32768) past the NVFP4 sweep, all on the 8192-block cap with each " + "warp looping over up to four tokens 32768 apart (four at 131072) " + "-- each a call on the leading T rows of one set (~5 s)" + ), + spec=dict(_ML4_131K, tokens_range=(32769, 131072)), + ), + Cell( + why=( + "Mistral Large 4 dep4 ties: integer fp32 logits 0..2 at 131072 gathered " + "tokens, every row ties inside its top-4 and at the 4th/5th boundary" + ), + spec=dict(_ML4_131K, dist="coarse"), + ), + Cell( + why=( + "Mistral Large 4 dep4 wide logits (sigma 100) at 131072 gathered tokens: " + "the lower weights underflow through bf16 subnormals to 0" + ), + spec=dict(_ML4_131K, magnitude=100.0), + ), + Cell( + why=( + "Mistral Large 4 dep4 padding: T_max 32768 with one rank at 32768 tokens " + "and three at 1, so 98301 of the 131072 gathered rows are the all-+0.0 " + "logits of a zero hidden row, each an exact 128-way tie routed to " + "experts 0..3 at weight 0.25" + ), + spec=dict(_ML4_131K, rank_tokens=(32768, 1, 1, 1)), + ), + Cell( + why=( + "Mistral Large 4 dep4 decode, CUDA graphs 4/8/16/32/64: the captured, " + "padded per-rank batches 1/2/4/8/16 gathered over 4 ranks, each " + "captured, replayed on new logits bitwise equal to eager" + ), + spec=dict(_ML4_131K, tokens=64, graph_tokens=(4, 8, 16, 32, 64)), + ), + Cell( + why=( + "Mistral Large 4 dep8 decode, CUDA graphs 8..128 step 8: the per-rank " + "batches 1..16 (the captured, padded 1/2/4/8/16 and every size of the " + "default list) gathered over 8 ranks, 2 to 32 blocks, each captured, " + "replayed on new logits bitwise equal to eager" + ), + spec=dict( + _ML4_32K, + tokens=128, + graph_tokens=(8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128), + ), + ), + ) + + note = """ + Metadata consumed: none; stateless, deterministic, runs on the ambient + stream. The launch (`invokeCustomMoeRouting`, in + `kernels/moe/routing/customMoeRoutingKernels.cu`) picks its kernel + instance by `nextPowerOfTwo(num_experts)` and `nextPowerOfTwo(topk)` -- + Mistral Large 4's 128 / top-4 is the (128, 4) instance with a 128-thread + block, one warp per token, so ceil(T / 4) blocks: 1 to 2304 over the FP8 + target's T 1..9216 and 1 to 8192 over the NVFP4 target's T 1..32768. + 8192 is the launcher's block cap, and 32768 rows reach it exactly: the + profiled grid is 8191 blocks at T 32764 and 8192 at T 32765 and 32768, so + up to 32768 every warp routes one token. Past it the grid stays at 8192 + blocks and each warp loops over tokens 32768 apart (8192 blocks measured + at T 32769, 65536, 65537 and 131072). The dep4 deployment's gathered steps, + T 32769..131072, all run there, each warp routing up to four tokens + (exactly four at 131072). That path is bit-exact at every T in + 32769..131072 and at 131073; its cells are 65537 and the dep4 sweep. PDL + (`TRTLLM_ENABLE_PDL`) was measured bitwise inert at 128 / top-4 on 32768 + rows and on T 32769, 65537, 98305, 131071 and 131072 (0 and 1 both equal + to unset); the Mistral Large 4 cells run with it unset, which + `getEnvEnablePDL` reads as on for sm_90 and later. + + Preconditions, enforced loudly by the op: `router_logits` 2-D CUDA bf16 / + fp16 / fp32; `output_dtype` fp32, bf16 or None (fp16 raises even for fp16 + logits); `topk <= 16`; `num_experts <= 512`. `topk == 0` and `num_tokens == 0` + return empties. The outputs are allocated by the op, from a graph's pool + under capture. + + Silent, refused by `is_valid`: a non-contiguous `router_logits` (read as a + dense `[T, E]` buffer from `data_ptr()`) and `topk > num_experts` (the + surplus slots come back as expert 0 at weight 0 -- duplicated ids). A + contiguous row window (a nonzero storage offset) is read correctly. + + Non-finite logits poison the whole row to NaN (`+inf`, `NaN`, an all + `-inf` row); a `-inf` logit among finite ones is a soft mask, at weight 0 + once `topk` exceeds the finite experts. At 128 experts every id stays in + [0, 128), distinct within its row, whatever the row holds, and a row never + reaches its neighbours: a padded batch's garbage rows cost only their own + outputs. + + Zero padding rows, which the dep4 deployment appends to each rank's rows up + to T_max before the gather: a zero hidden row through + `torch.ops.trtllm.cublas_mm` (bf16, 6144 -> 128, fp32 out) gives +0.0 in + every column (measured at m 4, 64, 32768, 32772, 65536 and 131072 with + every other row zero), and an all-+0.0 row is an exact 128-way tie that + the op and `reference` both route to experts 0..3 at weight exactly 0.25. + With one rank at 32768 tokens and three at 1, 98301 of the 131072 + gathered rows are padding: in one draw, experts 0..3 took 99333 to 99350 + rows each and no other expert more than 1098, a skew the expert runner + carries, not this op. + + The dep8 deployment gathers 8 x T_max rows and routes them in chunks of at + most 32768, each a call on a fresh contiguous logits tensor inside the + T 1..32768 sweep; with one rank at 32768 tokens and seven at 1, each of + chunks 1..7 is one live row and 32767 padding rows. Its decode graphs + route the 8 x b gathered rows of per-rank batch b in one call: 8, 16, 32, + 64 and 128 at the captured, padded 1/2/4/8/16, every multiple of 8 to 128 + under the default batch list, 2 to 32 blocks. + + Where the op and `reference` part, both from the kernel ranking by the + float's bit pattern: + - signed zeros: +0.0 ranks above -0.0, where `reference` (and torch) + holds them equal and keeps the lower index first. A row whose top-k + boundary falls inside a run of mixed-sign zeros gets different ids, and + equal weights. A zero hidden row through `torch.ops.trtllm.cublas_mm` + (bf16, 6144 -> 128, fp32 out) gives +0.0 throughout. + - a NaN with its sign bit set ranks below -inf: the op skips it and + answers finite weights over the rest of the row, where `reference` ranks + it first and answers NaN. CUDA arithmetic produces the positive NaN + (`0 * inf` and `-nan` both gave 0x7fffffff). + + `torch.ops.trtllm.default_moe_routing_op` has the identical signature and + softmaxes before the top-k: same ids, un-renormalized weights (rows sum to + 0.09-0.27 at 512 / top-10). `RenormalizeMoeRoutingMethod`'s PyTorch + fallback (used past 512 / 16) is not bit-compatible: it uses `torch.topk` + and casts before the softmax. + + Bit-exact against `reference` over 2040 configurations (in dtype x out + dtype x num_experts {1..512} x topk {0..16} x tokens {0..8192}); the + test's sweep re-drives the boundaries of that space. At 128 / top-4 + on fp32 logits: bit-exact at 9216, 32768 and 131072 tokens for logits of + sigma 1e-3 to 100 and integer ties, in both output dtypes, and at every T + in 1..131072 with bf16 weights. At sigma 100, 14012 of the 131072 bf16 + weights on 32768 rows are 0 and 2490 subnormal; 55908 of the 524288 on + 131072 rows are 0 and 9986 subnormal. The cells cover only 128 / top-4 on + fp32 logits with bf16 weights: bf16 and fp16 logits, fp32 weights, other + expert counts and other topk are accepted by the op but not certified here. + + Cost: back to back, a call's time is linear in T past the block cap. The + stateless op reads no KV pool, so the pool's size sets none of its limits. + + Hard limits (sm_103); the offsets are `customMoeRoutingKernel`'s, from a + uint32 token index + num_experts <= 512 (raises past it) + topk <= 16 (raises past it); > num_experts silent, guarded + logits offset token * num_experts in uint32; boundary 2^32 elements = + 33554432 tokens at 128 experts (8388608 at 512); unprobed: + the dep4 deployment's largest step is 131072 x 128 = 16777216 + = 2^24 elements (the NVFP4 tep4 deployment's and the dep8 + deployment's 32768-row chunk's 32768 x 128 = 4194304, the FP8 + target's 9216 x 128 = 1179648), and the + largest probed 131073 x 128 = 16777344, bit-exact + out offset token * topk in uint32; 2^30 tokens at top-4; unprobed + (the dep4 step's 131072 x 4 = 524288) + num_tokens passed to the kernel as int32; 2^31 tokens; unprobed + grid min(ceil(T / 4), 8192) blocks at 128 experts; the cap is + reached at T 32765 and is not a limit: past it warps loop + (certified at every T to 131072, probed at 131073, + bit-exact) + no index passes through fp32; an expert id rides in a 16-bit field of + the ranking key (`TopKRedType`, `kernels/moeTopKFuncs.cuh`), far above + the 512-expert ceiling. The dep4 step's largest flat logits index is + 2^24 - 1, and the 131073-row probe crosses 2^24 bit-exact + T certified every T in 1..131072 at 128 / top-4, captured at + 1/2/4/8/16, 4/8/16/32/64 and 8..128 step 8 + """ + + def __call__( + self, + router_logits: torch.Tensor, + topk: int, + output_dtype: Optional[torch.dtype] = None, + ) -> tuple[torch.Tensor, torch.Tensor]: + return torch.ops.trtllm.renorm_moe_routing_op(router_logits, topk, output_dtype) + + def reference( + self, + router_logits: torch.Tensor, + topk: int, + output_dtype: Optional[torch.dtype] = None, + ) -> tuple[torch.Tensor, torch.Tensor]: + """Stable descending sort in the logits' dtype, fp32 softmax over the + selected k, then the cast.""" + order = torch.sort(router_logits, dim=-1, descending=True, stable=True).indices[:, :topk] + selected = torch.gather(router_logits, 1, order).float() + e = torch.exp(selected - selected.max(-1, keepdim=True).values) + weights = e / e.sum(-1, keepdim=True) + return order.to(torch.int32), weights.to(output_dtype or torch.float32) + + def is_valid( + self, + router_logits: torch.Tensor, + topk: int, + output_dtype: Optional[torch.dtype] = None, + ) -> None: + # The kernel addresses router_logits from data_ptr() as dense + # [T, num_experts]: a column slice routes on the wider buffer's bytes. + assert router_logits.is_contiguous(), ( + "router_logits must be contiguous; a strided view is read as a dense " + "[num_tokens, num_experts] buffer and silently gives wrong routing" + ) + # Surplus slots come back as duplicated expert 0 at weight 0. + assert topk <= router_logits.shape[-1], ( + f"topk ({topk}) exceeds num_experts ({router_logits.shape[-1]}); the " + "surplus slots come back as duplicates of expert 0 at weight 0" + ) + + +renorm_moe_routing_op = RenormMoeRoutingOp() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/norm/__init__.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/norm/__init__.py new file mode 100644 index 000000000000..52a7a9daf028 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/norm/__init__.py @@ -0,0 +1,2 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/norm/layer_norm.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/norm/layer_norm.py new file mode 100644 index 000000000000..2f1e5c3b7187 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/norm/layer_norm.py @@ -0,0 +1,201 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""LayerNorm over the last dim -- mean subtraction, biased variance, weight and +bias -- in one Triton kernel, through the FLA module callable +`tensorrt_llm._torch.modules.fla.layernorm_gated.layernorm_fn`. + +One call site: the Mistral Large 4 DSA indexer's q_input_norm (hidden 6144, +bf16 weight and bias, eps 1e-6, rows 1..9216 on the FP8 checkpoint's target, +1..32768 on the NVFP4 checkpoint's). + +Tier-2 module callable, not a `torch.ops.trtllm.*` op: the registered norm +family is RMS-norm throughout, and `fused_adaptive_layernorm` refuses every +width but 5120. One call reshapes to 2-D, allocates the output and launches +`_layer_norm_fwd_1pass_kernel` once. +""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.modules.fla.layernorm_gated import layernorm_fn as _layernorm_fn + +#: The hidden-6144 MLA + DSA target (tep4): its lightning indexer's +#: q_input_norm, a LayerNorm(6144) with a bias at the checkpoint's +#: indexer_norm_eps 1e-6, on each of the 44 indexer layers. Its input is the +#: layer's attention-normed hidden state, one contiguous bf16 row per token: +#: every row of the step, up to max_num_tokens (9216 for the FP8 checkpoint's +#: target, 32768 for the NVFP4 checkpoint's, whose chunked prefill is off so +#: one context step carries all 32768 rows). +HIDDEN = 6144 + + +class LayerNorm(OpWrapper): + """`(x - mean) / sqrt(var + eps) * weight + bias` per row of the last dim, + variance over `N` (not `N - 1`), new contiguous tensor in `x`'s dtype. + + Every operand is widened to fp32 on load, the reduction and the affine run + in fp32, and only the store rounds. `bias=None` drops the additive term (a + compile-time branch of the kernel, not a zero bias). + + Fusion boundary: the norm and its affine only -- no residual, no gate, no + output quantization. The callable's `z`, `group_size`, `norm_before_gate` + and `is_rms_norm` are fixed at "no gate, one group, LayerNorm"; the gated + and RMS forms are different computations, not options of this entry. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # spec: shape, dtype; `eps` (default 1e-6). `site` "indexer_q" (rows shaped + # like the attention-normed hidden state, a quarter of them each at 1e-6, + # 1e-4, 1e-2 and 1 times its variance -- 1e-6 of it is about 2 eps -- so + # the gate sees eps); weight and bias are drawn over that site's checkpoint + # ranges. `checkpoint` "fp8" (default) or "nvfp4": whose q_input_norm + # weight and bias ranges are drawn (two training snapshots). + # `graph_rows`: each row count captured in a CUDA graph and replayed on new + # data. `sweep_rows` (lo, hi): every row count lo..hi, eagerly. + # + # The kernel runs one program per row with the whole row in registers: + # BLOCK_N is the width's next power of two (8192 at 6144, 2048 lanes of it + # masked) at 8 warps. Triton also specializes the row count, compiling + # M == 1, M divisible by 16 and any other M separately; the graph batches + # reach all three (1; 2, 4 and 8; 16) and so does the sweep. + # + # The indexer's eager cells drive every row count up to each target's + # max_num_tokens, each as its own call: 1..9216 for the FP8 checkpoint's + # target, 1..32768 for the NVFP4 checkpoint's. On sm_103 the 1..32768 + # sweep costs ~6 s, about half of it the bitwise compare; 1..9216 costs + # ~1.5 s. + CELLS: tuple[Cell, ...] = ( + Cell( + why="indexer q_input_norm, CUDA graphs 1/2/4/8/16: hidden 6144 bf16 at eps " + "1e-6, the captured decode batches, each replayed on new data bitwise equal " + "to eager", + spec=dict( + shape=(16, HIDDEN), + dtype=torch.bfloat16, + eps=1e-6, + site="indexer_q", + graph_rows=(1, 2, 4, 8, 16), + ), + ), + Cell( + why="indexer q_input_norm, eager rows 1..9216: hidden 6144 bf16 at eps 1e-6, " + "every row count up to the token budget, each bitwise the leading rows of the " + "9216-row call", + spec=dict( + shape=(9216, HIDDEN), + dtype=torch.bfloat16, + eps=1e-6, + site="indexer_q", + sweep_rows=(1, 9216), + ), + ), + Cell( + why="indexer q_input_norm, eager rows 1..32768: hidden 6144 bf16 at eps 1e-6, " + "the NVFP4 target's max_num_tokens 32768 with chunked prefill off (one 32768-row " + "prompt, or up to 16 packed), every row count, each bitwise the leading rows of " + "the 32768-row call, at the NVFP4 checkpoint's q_input_norm ranges", + spec=dict( + shape=(32768, HIDDEN), + dtype=torch.bfloat16, + eps=1e-6, + site="indexer_q", + checkpoint="nvfp4", + sweep_rows=(1, 32768), + ), + ), + ) + + note = """ + Metadata consumed: none. Inputs are not mutated; two scratch fp32 [M] + tensors (mean, rstd) are allocated per call and discarded. + + Layout: `x.stride(-1) == 1` is read in place through its row stride; any + other `x`, and non-contiguous `weight`/`bias`, are silently `.contiguous()`d + first (one extra copy, correct result). `weight`/`bias` dtype is free. + + Loud failures (left to the callable): weight or bias length != N raises a + bare AssertionError with no message; `next_pow2(N) * x.element_size() > + 64KB` raises "This layer norm doesn't support feature dim >= 64KB" (N = + 65536 in bf16); CPU tensors fail before or at the Triton launch. + + Accuracy: at the indexer's affine (weights 0.89..1.14, biases within 0.09) + over 9216 hidden-state rows of 6144 the bf16 result is the correctly + rounded value on all but 1.6e-5..1.8e-5 of elements, and one bf16 step off + at most except where `x_hat * weight` and `bias` cancel: 21 elements of + the sweep cell's input sit more than one step off (up to 26; 92 in another + draw), each a result within 2e-6 of zero and off by at most 1.2e-8. + torch's `F.layer_norm` on the same bf16 operands (fp32 inside, one + rounding) measures the same and differs from this op on 1.9e-5 of + elements. Over 32768 such rows at the NVFP4 checkpoint's affine (weights + 0.89..1.13, biases -0.087..0.090): all but 1.77e-5 of elements (3,562 of + 201,326,592) correctly rounded, 117 more than one step off, each a result + within 1.9e-6 of zero and off by at most 1.5e-8; `F.layer_norm` 1.93e-5 + off, differing from this op on 1.94e-5. A constant row (zero included) + returns `bias` bitwise. `eps` sits inside the sqrt. + + Rows are computed independently: at 6144, every row count in 1..32768 is + bitwise the leading rows of the 32768-row call (and every one in 1..9216 + of the 9216-row call), and a CUDA-graph replay at each of 1..16 rows is + bitwise equal to an eager call on the same data. One 32768-row call is one + launch; back to back, an eager call of 1 or 16 rows takes as long as the + host's launch path, not the kernel. + + Compilation: each (dtype, width, row-count class) is compiled by Triton + on its first call in a process, or loaded from Triton's on-disk cache; a + cold compile measured 0.3-0.9 s. The graph cells capture after one + warm-up call at the same row count, as the engine's warm-up does; capture + with no warm-up was not measured. + + Accepted but not certified: an fp8-e4m3 `x` returns fp8-e4m3 (output is + `empty_like(x)`), within one e4m3 ulp of the exact value; fp16, fp32 and + fp64 `x`; `weight`/`bias` in a dtype other than `x`'s; `bias=None`; an + `x` of more than two dims, or with a row stride other than N; widths + other than 6144; more than 32768 rows at 6144. Forward only: the autograd + Function has no backward. Runs unchanged under `inference_mode`, and the + graph cells capture and replay inside it. + + Hard limits (sm_103). Computed, not probed: the envelopes' largest call, + 32768 x 6144 on the NVFP4 target (9216 x 6144 on the FP8 target), is + 201,326,592 elements, and its last row starts at 201,320,448, 9.4% of + 2^31. The op reads no KV pool, so no pool size bounds it. + row offset `row * stride_x_row` in int32: the program id is 32-bit + and the compiled signature types the row stride i32. + Wraps at row * stride >= 2^31: 349,526 rows at 6144 + columns one block of next_pow2(N) lanes; N is bounded by the 64KB + check above, not by an index width + rows the grid's first dimension and an i32 argument: 2^31 - 1 + fp32 no index passes through fp32 + """ + + def __call__( + self, + x: torch.Tensor, + weight: torch.Tensor, + bias: Optional[torch.Tensor], + eps: float, + ) -> torch.Tensor: + return _layernorm_fn(x, weight, bias, z=None, eps=eps) + + def reference( + self, + x: torch.Tensor, + weight: torch.Tensor, + bias: Optional[torch.Tensor], + eps: float, + ) -> torch.Tensor: + """fp64 from primitive ops (not `F.layer_norm`), rounded once to `x.dtype`.""" + xd = x.double() + mean = xd.mean(dim=-1, keepdim=True) + var = (xd - mean).square().mean(dim=-1, keepdim=True) + out = (xd - mean) / torch.sqrt(var + eps) * weight.double() + if bias is not None: + out = out + bias.double() + return out.to(x.dtype) + + +layer_norm = LayerNorm() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/quantization/__init__.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/quantization/__init__.py new file mode 100644 index 000000000000..52a7a9daf028 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/quantization/__init__.py @@ -0,0 +1,2 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/quantization/fp8_quantize_1x128_packed_ue8m0.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/quantization/fp8_quantize_1x128_packed_ue8m0.py new file mode 100644 index 000000000000..3d5c77d655b4 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/quantization/fp8_quantize_1x128_packed_ue8m0.py @@ -0,0 +1,344 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Dynamic 1x128 FP8 quantization of a bf16 activation, with packed UE8M0 block scales. + +One power-of-two scale per 1x128 block, packed four to an int32 in the MN-major layout +DeepGEMM's FP8 GEMM reads. +""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: Largest finite e4m3 magnitude. +E4M3_MAX = 448.0 + +#: The kernel's amax floor, compared in fp32: an all-zero block takes the scale 2**-42. +AMAX_FLOOR = 1e-10 + +#: A UE8M0 byte is a bare fp32 exponent: byte `b` encodes 2**(b - 127). +UE8M0_BIAS = 127 + +#: The UE8M0 cast saturates at the largest finite exponent, 2**127 (byte 254). Only an +#: infinite amax reaches it: the largest bf16 over 448 is below 2**120. +UE8M0_MAX_EXP = 127 + +#: Scale bytes per int32 word: the word in column `c` holds K blocks 4c .. 4c + 3. +BLOCKS_PER_WORD = 4 + + +def scale_leading_dim(m: int) -> int: + """The scale's column stride: M rounded up to a multiple of 4 (16-byte TMA alignment).""" + return (m + 3) // 4 * 4 + + +def block_exponents(input: torch.Tensor) -> torch.Tensor: + """`e [M, ceil(K/128)]` int64: block `b` of row `t` is quantized with the scale `2**e`. + + `e = ceil(log2(amax / 448))`, exactly: `amax` is the block's largest `|x|` with NaN + skipped (a short last block counts only its own columns), floored at `AMAX_FLOOR`, and + `e` saturates at `UE8M0_MAX_EXP`. `frexp` of the fp64 quotient finds the ceiling without + a `log2`, so a block whose amax is exactly 448 * 2**j takes `e = j`. + """ + m, k = input.shape + nb = (k + 127) // 128 + mag = input.float().abs() + mag = torch.where(mag.isnan(), torch.zeros_like(mag), mag) + mag = torch.nn.functional.pad(mag, (0, nb * 128 - k)) + amax = mag.reshape(m, nb, 128).amax(dim=2).clamp(min=AMAX_FLOOR) + mant, exp = torch.frexp(amax.double() / E4M3_MAX) + e = torch.where(mant == 0.5, exp - 1, exp).long() + return torch.where(amax.isinf(), UE8M0_MAX_EXP, e).clamp(max=UE8M0_MAX_EXP) + + +#: The widths Mistral Large 4's FP8 128x128 block-scale linears quantize at tep4, and their +#: sites (the same per-rank shapes in the FP8 and the NVFP4 checkpoints): hidden 6144 feeds +#: wq, the indexer's wq_b and the dense and shared-expert gate_up; the dense layer-0 down +#: reads 6144 too (TP4 of 24576); wo reads 2048 (16 heads x 128 per rank); the shared-expert +#: down reads 1536 (TP4 of 6144). +_SITES = { + 6144: "wq, indexer wq_b, gate_up, dense down", + 2048: "wo", + 1536: "shared-expert down", +} + +#: The widths Mistral Large 4 adds at dep4. Attention DP keeps all 64 heads and the dense +#: layer-0 MLP whole on every rank, each over the rank's own tokens: wo reads the whole +#: 64 x 128 attention output and the dense layer-0 down the whole SwiGLU output of +#: intermediate 24576. dep4's other sites (wq, indexer wq_b, the dense and shared-expert +#: gate_up, the shared-expert down) read hidden 6144. +_DEP4_SITES = { + 8192: "wo at dep4, all 64 heads", + 24576: "dense down at dep4, replicated", +} + +#: The widths Mistral Large 4 adds at tep8 and tp8, which shard everything but the routed experts +#: alike. Tensor parallel 8 (attention DP off) slices eight ways: wo reads this rank's 8 of the +#: 64 heads x 128, the dense layer-0 down 24576 / 8 and the shared-expert down 6144 / 8. 768 is +#: six blocks: its second scale word holds two, and its bytes 2 and 3 are the empty slots. The +#: two targets' other sites (wq, indexer wq_b, the dense and shared-expert gate_up) read hidden +#: 6144. +_TEP8_SITES = { + 1024: "wo at tep8 and tp8, 8 heads", + 3072: "dense down at tep8 and tp8, 24576 / 8", + 768: "shared-expert down at tep8 and tp8, 6144 / 8", +} + +#: Every width with cells, and its sites. +_WIDTHS = {**_SITES, **_DEP4_SITES, **_TEP8_SITES} + +#: The CUDA-graph decode batches (max_batch_size 16, one row per request): the gates +#: capture 1/2/4/8/16 padded; the LLM API's unpadded default captures every batch 1..16. +_GRAPH_BATCHES = tuple(range(1, 17)) +_PADDED_BATCHES = (1, 2, 4, 8, 16) + +#: max_num_tokens of the NVFP4 target: chunked prefill is off, so one context step carries +#: up to 32768 rows (one whole 32K prompt, or up to 16 packed ones). +_MAX_TOKENS = 32768 + +#: max_num_tokens of the FP8 Mistral Large 4 target, which calls this op at the same widths. +_FP8_TARGET_MAX_TOKENS = 9216 + +#: The eager row counts each dep4 width is driven at, up to max_num_tokens (per rank under +#: attention DP), and what each one reaches. +_DEP4_EAGER_ROWS = ( + (3, "inside one 4-row pad unit, scale stride 4"), + (5, "spills into a second pad unit, scale stride 8"), + (6, "two zeroed pad slots per scale column"), + (1023, "M % 4 == 3, one short prompt"), + (4097, "M % 4 == 1 past 4096 rows"), + (9215, "M % 4 == 3 past 8192 rows"), + (9216, "M % 4 == 0 past 8192 rows"), + (16385, "M % 4 == 1 past 16384 rows"), + (_MAX_TOKENS - 1, "one row under max_num_tokens, a pad slot per scale column"), + (_MAX_TOKENS, "max_num_tokens, one whole 32K prompt on the rank in one step"), +) + +#: The eager row counts each tep8 and tp8 width is driven at, up to max_num_tokens (every rank sees +#: every token of the step), and what each one reaches. +_TEP8_EAGER_ROWS = ( + (3, "inside one 4-row pad unit, scale stride 4"), + (5, "spills into a second pad unit, scale stride 8"), + (1023, "M % 4 == 3, one short prompt"), + (4097, "M % 4 == 1 past 4096 rows"), + (_MAX_TOKENS - 1, "one row under max_num_tokens, a pad slot per scale column"), + (_MAX_TOKENS, "max_num_tokens, one whole 32K prompt in one step"), +) + + +class Fp8Quantize1x128PackedUe8m0(OpWrapper): + """Quantize a bf16 `[M, K]` activation to e4m3, one power-of-two scale per 1x128 row block. + + Per row `t` and block `b` of 128 consecutive columns: `s = 2**ceil(log2(amax / 448))`, + `amax = max |x[t, block]|` floored at 1e-10, and `codes[t, block] = e4m3(x / s)`. As `s` + is a power of two at or above `amax / 448`, the division is exact and every code lies in + +-448: e4m3's round-to-nearest-even is the only rounding. Returns `(codes, scale)`: + + * `codes`: `[M, K]` float8_e4m3fn, contiguous -- a leading-row view of an allocation + padded to a multiple of 4 rows. + * `scale`: `[M, ceil(K / 512)]` int32 with strides `(1, align(M, 4))`, the MN-major + layout `tensorrt_llm.deep_gemm.fp8_gemm_nt` takes for the activation scale (what + `tensorrt_llm.deep_gemm.get_mn_major_tma_aligned_packed_ue8m0_tensor` makes of fp32 + power-of-two scales). Byte `j` (little-endian) of `scale[t, c]` is the biased exponent + `e + 127` of K block `4c + j`; the scale is `2**(byte - 127)`, and byte slots past the + last block hold 0. The tensor is not dense: its storage holds + `(cols - 1) * align(M, 4) + M` int32s, so read it through its own strides (index it, + or `.contiguous()`); an `as_strided` over the full `cols * align(M, 4)` rectangle is + out of bounds whenever `M % 4 != 0`. The padding slots inside the storage hold 0. + + The pair feeds `torch.ops.trtllm.fp8_prequantized_swap_ab_gemm(codes, scale, weight, + weight_scale, ..., disable_ue8m0_cast=True)`. `torch.ops.trtllm.fp8_swap_ab_gemm` makes + this same call itself on the sm_100 family before its GEMM, when its autotuner picks the + CUDA quantize tactic and `TRTLLM_FUSED_FP8_QUANT_PACK` is at its default `1` + (`_fp8_quantize_1x128_ue8m0` in `_torch/custom_ops/torch_custom_ops.py`). + + The call passes `use_r128c4_layout=False`. The op's schema default, True, is the sm_107 + CuTe layout: a 1-D uint8 R128c4 buffer that `fp8_prequantized_swap_ab_gemm` refuses + ("dimensionality of sizes (1) must match dimensionality of strides (2)"), so this entry + does not expose the flag. + + Fusion boundary: the amax reduction, the UE8M0 rounding, the e4m3 cast and the scale + packing are inside; nothing before (norm, SwiGLU) or after (the GEMM) is fused. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = tuple( + Cell( + why=f"graph batch {m}, K {k}: " + f"{'padded' if m in _PADDED_BATCHES else 'unpadded default'} capture, replayed on " + f"new rows ({_WIDTHS[k]})", + spec=dict(m=m, k=k, graph=True), + ) + for k in _WIDTHS + for m in _GRAPH_BATCHES + ) + ( + Cell( + why="eager M 3: inside one 4-row pad unit, scale stride 4", + spec=dict(m=3, k=6144, graph=False), + ), + Cell( + why="eager M 5: spills into a second pad unit, scale stride 8", + spec=dict(m=5, k=2048, graph=False), + ), + Cell( + why="eager M 6: two zeroed pad slots per scale column", + spec=dict(m=6, k=1536, graph=False), + ), + Cell( + why="eager M 1023: a prefill chunk, M % 4 == 3", + spec=dict(m=1023, k=6144, graph=False), + ), + Cell( + why="eager M 4097: M % 4 == 1 past 4096 rows", + spec=dict(m=4097, k=1536, graph=False), + ), + Cell( + why="eager M 9215: one row under the FP8 target's max_num_tokens", + spec=dict(m=_FP8_TARGET_MAX_TOKENS - 1, k=2048, graph=False), + ), + *( + Cell( + why=f"eager M 9216 at K {k}: the FP8 target's max_num_tokens ({_SITES[k]})", + spec=dict(m=_FP8_TARGET_MAX_TOKENS, k=k, graph=False), + ) + for k in _SITES + ), + Cell( + why="eager M 16385: M % 4 == 1 past 16384 rows", + spec=dict(m=16385, k=1536, graph=False), + ), + *( + Cell( + why=f"eager M 32767 at K {k}: one row under max_num_tokens, a pad slot per " + "scale column", + spec=dict(m=_MAX_TOKENS - 1, k=k, graph=False), + ) + for k in _SITES + ), + *( + Cell( + why=f"eager M 32768 at K {k}: max_num_tokens, one whole 32K prompt in one " + f"step ({_SITES[k]})", + spec=dict(m=_MAX_TOKENS, k=k, graph=False), + ) + for k in _SITES + ), + *( + Cell( + why=f"eager M {m} at K {k}: {what} ({_DEP4_SITES[k]})", + spec=dict(m=m, k=k, graph=False), + ) + for k in _DEP4_SITES + for m, what in _DEP4_EAGER_ROWS + ), + *( + Cell( + why=f"eager M {m} at K {k}: {what} ({_TEP8_SITES[k]})", + spec=dict(m=m, k=k, graph=False), + ) + for k in _TEP8_SITES + for m, what in _TEP8_EAGER_ROWS + ), + ) + + note = """ + Metadata consumed: none. Runs on the current stream. The launch uses programmatic + dependent launch unless `TRTLLM_ENABLE_PDL=0` (`getEnvEnablePDL` in + `cpp/tensorrt_llm/common/envUtils.cpp` turns it on from sm_90); the codes and scales + were measured byte-identical with it off (again at K 1024, 3072, 768 and 6144, at eight + M from 1 to 32768). + + Preconditions, each refused loudly by the op (the test drives every one): `input` bf16 + ("Input matrix dtype must be BF16."), 2-D ("input must be a matrix"; flatten [B, S, H] + yourself), contiguous ("self must be contiguous" -- a column slice of a fused + projection is refused, not read through its stride), on CUDA, and `K % 16 == 0` + ("self.sizes()[1] must be a multiple of 16, but got ..."). `M == 0` raises as well: + "The specified pointer resides on host memory and is not registered with any CUDA + device." (inferred: the empty scale buffer has no device pointer to wrap), so a caller + with no rows skips the call. The op also checks for an sm_100-family device ("currently + only supports SM100-family GPUs"), read in `thop/fp8Quantize.cpp`, not probed. + + Alignment: the kernel loads the input in 32-byte vectors (read in + `fp8_blockscale_quant_packed.cu`). A contiguous view 8 elements (16 bytes) into its + allocation was measured correct; one 1 element in faults with "CUDA error: misaligned + address", which leaves the CUDA context unusable. A fresh allocation, or a row slice of + a contiguous `[*, K]` tensor with `K % 16 == 0`, is always aligned. + + Edge behaviour, measured, and reproduced by `reference`: an all-zero block takes byte 85 + (2**-42, the floor) and zero codes; NaN is skipped by the amax and its own code is NaN + (0x7f); an inf saturates its block to byte 254 (2**127), its own code to +-448 and every + finite code in the block to +-0; -0.0 keeps its sign (0x80). + + A K that is not a multiple of 512 ends in a partial scale word; K 768 (six blocks, two + words) is the one with cells. In each row's second word the lanes past K load zeros, store + no codes, and have their bytes masked out of the word (read in + `fp8_blockscale_quant_packed.cu`), so bytes 2 and 3 of `scale[:, 1]` are 0. Measured over + an output allocation filled with 0xFF first: the empty bytes and the pad slots read 0, + while the code rows past M, which the kernel never writes, kept 0xFF. + + Not certified: K outside {6144, 2048, 1536, 8192, 24576, 1024, 3072, 768} -- the op + takes any `K % 16 == 0`, with a short last block and zero byte slots past it, but no + cell holds a K with a short last block, and of the K whose last scale word is partial + only 768 has cells; M above 32768 (two probes past it, below); the + `use_r128c4_layout=True` layout, which this entry never requests. + + Hard limits (sm_103). Read in `fp8_blockscale_quant_packed.cu` and `thop/fp8Quantize.cpp`: + the kernel forms the row index, the input and code offsets `row * K + col` and the + scale offset `col * align(M, 4) + row` in int64; `M`, `K`, `align(M, 4)` (the scale + stride) and the grid are int32. One warp per row and 512 columns (one scale word), four + rows per block; K sets only grid x, one block column per scale word (12 at K 6144, 16 at + K 8192, 48 at K 24576; 2 at K 1024 and 768, 6 at K 3072); the row blocks fold into grid + z past 65535 blocks (M > 262140). + row * K input and code offsets, int64 + boundary 2^31 elements: 2^31 / K = 349525.3 rows at K 6144, + 1048576 at K 2048, 1398101.3 at K 1536, 262144 at K 8192, + 87381.3 at K 24576, 2097152 at K 1024, 699050.7 at K 3072 and + 2796202.7 at K 768; the envelope's largest is + 32768 * 24576 = 805,306,368 elements (tep8's and tp8's 32768 * + 3072 = 100,663,296), not reached + probed at M 349526, K 6144 (2,147,487,744 elements, grid z 2) and at + M 87382, K 24576 (2,147,500,032 elements, grid (48, 21846, 1)): passed. + Every row bitwise equal to `reference`; the rows straddling element + 2^31 (and at K 6144 the grid-z seam) bitwise equal to the op on those + rows alone. Probes outside the envelope, not cells + col * align(M, 4) + row + scale word offset, int64. At M 32768 the largest is + 11 * 32768 + 32767 = 393,215 (K 6144), 131,071 (K 2048), + 98,303 (K 1536), 524,287 (K 8192), 1,572,863 (K 24576), 65,535 + (K 1024 and 768) and 196,607 (K 3072). In int32 it would first wrap + at M 178,956,969 (K 6144), 536,870,913 (K 2048), 715,827,881 + (K 1536), 134,217,729 (K 8192), 44,739,241 (K 24576), 1,073,741,825 + (K 1024 and 768) and 357,913,941 (K 3072): not reached. Probed at + M 349526, K 6144: largest offset 4,194,333, and at M 87382, K 24576: + largest offset 4,194,429, passed (above) + align(M, 4) the int32 scale stride: wraps for M > 2^31 - 4, which the op's + "M must be within int32" check (read, not probed) lets through for + M 2^31 - 3 .. 2^31 - 1. Computed, not probed: past any envelope + """ + + def __call__(self, input: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]: + return torch.ops.trtllm.fp8_quantize_1x128_packed_ue8m0(input, use_r128c4_layout=False) + + def reference(self, input: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]: + """`(codes, scale)` from the exact power-of-two scales, laid out as the op lays them out.""" + m, k = input.shape + e = block_exponents(input) + nb = e.shape[1] + cols = (nb + BLOCKS_PER_WORD - 1) // BLOCKS_PER_WORD + s = torch.ldexp(torch.ones_like(e, dtype=torch.float64), e) + q = (input.double() / s.repeat_interleave(128, dim=1)[:, :k]).float() + codes = q.clamp(-E4M3_MAX, E4M3_MAX).to(torch.float8_e4m3fn) + byte = torch.nn.functional.pad(e + UE8M0_BIAS, (0, cols * BLOCKS_PER_WORD - nb)) + words = byte.to(torch.uint8).reshape(m, cols, BLOCKS_PER_WORD).view(torch.int32) + scale = torch.empty_strided( + (m, cols), (1, scale_leading_dim(m)), dtype=torch.int32, device=input.device + ) + scale.copy_(words.squeeze(-1)) + return codes, scale + + +fp8_quantize_1x128_packed_ue8m0 = Fp8Quantize1x128PackedUe8m0() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/vision/__init__.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/vision/__init__.py new file mode 100644 index 000000000000..52a7a9daf028 --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/vision/__init__.py @@ -0,0 +1,2 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/vision/pixtral_rope_2d.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/vision/pixtral_rope_2d.py new file mode 100644 index 000000000000..b4b35e1b9d7b --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/ops/vision/pixtral_rope_2d.py @@ -0,0 +1,292 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The vision tower's 2-D rotary embedding: q and k of every patch rotated by +angles taken from the patch's (row, column) in its image. + +Tier 4, composed by this run from native torch (frequency vector, angle +table, cos/sin rotor, one complex multiply per tensor); no engine kernel +computes it. Rejected at the cheaper rungs: every registered rope op that +takes positions (`mla_rope_inplace`, +`flashinfer_apply_rope_with_cos_sin_cache_inplace`, `fused_qk_norm_rope`, +`fused_dit_qk_norm_rope`) takes 1-D positions into one cos/sin table, so a +per-patch 2-D table would still have to be composed first; the one that takes +per-token cos/sin, `fused_dit_split_norm_rope`, fuses a full-width RMSNorm the +tower's q/k do not have. The in-tree Pixtral tower +(`_torch/models/modeling_pixtral.py`) calls transformers' Pixtral rotary +embedding, out of tree and in the HF half-split pairing over permuted +weights. FlashInfer's `rope` module has no 2-D variant. A single-kernel form +(for instance a per-patch table fed to one of the table-indexed ops above in +its non-NeoX mode) is the tuning phase's to supply. +""" + +from __future__ import annotations + +from typing import Tuple + +import torch + +from tensorrt_llm._torch.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: params.json `vision_encoder`: hidden 2048 over 16 heads, rope_theta 1e4. +HEADS = 16 +HEAD_DIM = 128 +THETA = 10000.0 + + +class PixtralRope2d(OpWrapper): + """Rotate `q` [N, Hq, D] and `k` [N, Hk, D] of N patches by their 2-D + angles; return fresh `(q_out, k_out)` in the inputs' shapes and dtypes. + + f_i = theta ** (-2 i / D) i = 0 .. D/2 - 1 + angle[n, j] = row_n * f_{2j} j = 0 .. D/4 - 1 + angle[n, j] = col_n * f_{2(j - D/4) + 1} j = D/4 .. D/2 - 1 + (x[2j], x[2j+1]) -> (x[2j] cos a_j - x[2j+1] sin a_j, + x[2j] sin a_j + x[2j+1] cos a_j) + + GPT-J interleaved pairs `(2j, 2j+1)` over the checkpoint's own (mistral + native, unpermuted) wq/wk rows: row angles on the first D/4 pairs with the + even frequencies, column angles on the last D/4 with the odd ones. fp32 + math throughout: the frequency vector `1 / theta ** (2i / D)` is computed + in fp32 on the device, each angle is `fp32(position) * f` rounded once, + the rotor is `torch.polar(1, angle)` (fp32 cos/sin), and each head is + rotated as one complex64 multiply of `x.float()` viewed as D/2 complex + pairs, rounded once to the input dtype on the way out. + + `positions` is [N, 2] integer (row, col) of each patch within its own + image; patches of several images are concatenated along N, each image's + positions starting at (0, 0). Order along N is the caller's (the + reference walks each image row-major); the op is per patch. + + Fusion boundary: the angle table and the rotation. The qkv projection, the + position grid of each image, the unrotated `v` and the attention are the + caller's. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # spec: `grid` one image's (rows, cols) or a tuple of them (several images + # in one call); `heads` / `head_dim` for q and k; `theta`; `dtype` of q/k; + # `layout` "packed_qkv" (q and k are head-reshaped column chunks of one + # [N, 3 * heads * head_dim] projection, the tower's own layout) or + # "separate" (contiguous q and k). Positions are int64 (row, col), each + # image row-major from (0, 0). + CELLS: tuple[Cell, ...] = ( + Cell( + why="one-patch image 1x1: angle 0, the rotor is exactly 1", + spec=dict( + grid=(1, 1), + heads=HEADS, + head_dim=HEAD_DIM, + theta=THETA, + dtype=torch.bfloat16, + layout="packed_qkv", + ), + ), + Cell( + why="32x32 image: 1024 patches", + spec=dict( + grid=(32, 32), + heads=HEADS, + head_dim=HEAD_DIM, + theta=THETA, + dtype=torch.bfloat16, + layout="packed_qkv", + ), + ), + Cell( + why="110x110 image (1540 px, the largest): 12100 patches, positions 0..109", + spec=dict( + grid=(110, 110), + heads=HEADS, + head_dim=HEAD_DIM, + theta=THETA, + dtype=torch.bfloat16, + layout="packed_qkv", + ), + ), + Cell( + why="two images in one call, 110x110 + 40x60 (non-square): 14500 patches", + spec=dict( + grid=((110, 110), (40, 60)), + heads=HEADS, + head_dim=HEAD_DIM, + theta=THETA, + dtype=torch.bfloat16, + layout="packed_qkv", + ), + ), + Cell( + why="envelope: two max-size images in one call, 24200 patches", + spec=dict( + grid=((110, 110), (110, 110)), + heads=HEADS, + head_dim=HEAD_DIM, + theta=THETA, + dtype=torch.bfloat16, + layout="packed_qkv", + ), + ), + Cell( + why="32K image envelope: ten maximum-size images, 121000 patches", + spec=dict( + grid=((110, 110),) * 10, + heads=HEADS, + head_dim=HEAD_DIM, + theta=THETA, + dtype=torch.bfloat16, + layout="packed_qkv", + ), + ), + Cell( + why="110x110 with q and k as separate contiguous tensors", + spec=dict( + grid=(110, 110), + heads=HEADS, + head_dim=HEAD_DIM, + theta=THETA, + dtype=torch.bfloat16, + layout="separate", + ), + ), + Cell( + why="110x110 fp32 q/k: the fp32 math's own error, not hidden by bf16", + spec=dict( + grid=(110, 110), + heads=HEADS, + head_dim=HEAD_DIM, + theta=THETA, + dtype=torch.float32, + layout="packed_qkv", + ), + ), + ) + + note = """ + Metadata consumed: none. Out-of-place; `q`, `k` and `positions` are not + mutated, and `q_out` / `k_out` are fresh contiguous tensors. `q` and `k` + need last-dim stride 1 only (head-reshaped chunks of one qkv projection are + taken as they are; `.float()` makes the fp32 copy the rotation runs on). + `positions` sits on q's device. No host synchronization; capture under a + CUDA graph was not measured (the call site runs eagerly). + + Weight layout: the pairing is GPT-J interleaved over the checkpoint's + unpermuted wq/wk. The engine's mistral loader has `permute_qk` + (`_torch/models/checkpoints/mistral/weight_mapper.py`), which reorders a + vision tower's wq/wk into the HF half-split layout; a target that applies + it to these weights needs the half-split pairing instead, and this entry + is then silently wrong. + + Against the reference implementation: its mistral-format vision encoder + computes the same arithmetic (fp32 frequencies, fp32 angle = position + times frequency, complex64 rotor and multiply) but builds the frequency + vector and the cos/sin table on the host and moves them to the device. + Run in its own environment on the same bf16 inputs (not in CI), the + entry's outputs fell inside this test's band at every cell grid, with 749 + of 49561600 elements per tensor differing at two 110x110 images: the + device's fp32 pow, cos and sin sit a few ulp from the host's (4 of the 64 + frequencies differ by one ulp). The test's host-built-table alternative + reproduces that recipe and lands inside the band. + + Accuracy (test-derived band, per element): within half a unit of the + output dtype at both the result and the fp64 closed form, plus the fp32 + math's own budget times the element's pair magnitude; the cells spend at + most 0.107 of that budget. + + Cost (eager, not gated): a call is a chain of small torch kernels; the + tower calls it once per block. + + Guarded by `is_valid` (each accepted without an error and answered wrongly + when unguarded): `positions` not [N, 2] with N the token count of q (a + one-row `positions` broadcasts the first patch's angles over every + patch); `k` with a different token count from `q` (a one-token `k` + broadcasts to N tokens); `q` or `k` not 3-D (a 2-D [N, D] broadcasts to + [N, N, D]). + + Certified at heads 16, head_dim 128, theta 1e4, int64 positions with rows + and columns 0..109, bf16 q/k (fp32 at 110x110), up to 24200 patches per + call. Uncertified: other head counts or head dims (head_dim must be a + multiple of 4: the angle table's width is head_dim/2 only then), other + theta, fp16 q/k, int32 positions, positions above 109, more than 24200 + patches per call, a q/k dtype mismatch. + + Hard limits (sm_103) + positions fp32(position), torch elementwise boundary 2^24 + 1 not + reached: positions stop at 109 (a 1540-px side over 14-px + patches) -- unprobed + numel N * heads * head_dim, torch elementwise boundary 2^31 + not reached: the envelope's largest call is 24200 * 16 * 128 + = 49561600 elements -- unprobed + """ + + def __call__( + self, + q: torch.Tensor, + k: torch.Tensor, + positions: torch.Tensor, + theta: float, + ) -> Tuple[torch.Tensor, torch.Tensor]: + head_dim = q.shape[-1] + freqs = 1.0 / ( + theta ** (torch.arange(0, head_dim, 2, device=q.device, dtype=torch.float32) / head_dim) + ) + pos = positions.to(torch.float32) + angles = torch.cat((pos[:, 0:1] * freqs[0::2], pos[:, 1:2] * freqs[1::2]), dim=-1) + rotor = torch.polar(torch.ones_like(angles), angles).unsqueeze(-2) + q_pairs = torch.view_as_complex(q.float().reshape(*q.shape[:-1], -1, 2)) + k_pairs = torch.view_as_complex(k.float().reshape(*k.shape[:-1], -1, 2)) + q_out = torch.view_as_real(q_pairs * rotor).flatten(-2).to(q.dtype) + k_out = torch.view_as_real(k_pairs * rotor).flatten(-2).to(k.dtype) + return q_out, k_out + + def reference( + self, + q: torch.Tensor, + k: torch.Tensor, + positions: torch.Tensor, + theta: float, + ) -> Tuple[torch.Tensor, torch.Tensor]: + """The closed form in fp64 from the pair index, with explicit real + arithmetic: frequency, axis and angle per pair, then the 2x2 rotation, + rounded once to the input dtype.""" + head_dim = q.shape[-1] + pairs = head_dim // 2 + quarter = pairs // 2 + j = torch.arange(pairs, dtype=torch.float64, device=q.device) + on_col = j >= quarter + freq_index = torch.where(on_col, 2.0 * (j - quarter) + 1.0, 2.0 * j) + freq = torch.pow(torch.tensor(theta, dtype=torch.float64), -2.0 * freq_index / head_dim) + pos = positions.to(torch.float64) + axis_pos = torch.where(on_col, pos[:, 1:2], pos[:, 0:1]) + angle = (axis_pos * freq.to(q.device)).unsqueeze(-2) + cos, sin = torch.cos(angle), torch.sin(angle) + + def rotate(x: torch.Tensor) -> torch.Tensor: + x64 = x.double() + even, odd = x64[..., 0::2], x64[..., 1::2] + out = torch.empty_like(x64) + out[..., 0::2] = even * cos - odd * sin + out[..., 1::2] = even * sin + odd * cos + return out.to(x.dtype) + + return rotate(q), rotate(k) + + def is_valid( + self, + q: torch.Tensor, + k: torch.Tensor, + positions: torch.Tensor, + theta: float, + ) -> None: + assert q.dim() == 3 and k.dim() == 3, ( + f"q and k must be [tokens, heads, head_dim]; got q {tuple(q.shape)}, k {tuple(k.shape)}" + ) + assert k.shape[0] == q.shape[0], ( + f"q and k must carry the same tokens; got q {tuple(q.shape)}, k {tuple(k.shape)}" + ) + assert positions.dim() == 2 and tuple(positions.shape) == (q.shape[0], 2), ( + f"positions must be [tokens, 2] (row, col) for the {q.shape[0]} tokens of q; " + f"got {tuple(positions.shape)}" + ) + + +pixtral_rope_2d = PixtralRope2d() diff --git a/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/routing.py b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/routing.py new file mode 100644 index 000000000000..a3eca3627bfd --- /dev/null +++ b/tensorrt_llm/_torch/modeling_v2/models/mistral_large4/routing.py @@ -0,0 +1,366 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Where a MistralLarge4ForCausalLM config lands. Read this file and you know. + +``route`` is one forward-reading decision: the facts that fix which code runs, compared +against the deployments the targets below were written for. When none fits it raises +``NoTarget`` stating those facts. There is no built-in implementation of this architecture, +so ``auto`` then has nothing to fall back to and ``require`` raises the same message. +""" + +from __future__ import annotations + +import math +from typing import TYPE_CHECKING, Any, Tuple + +from ..._router_index import NoTarget, routes + +if TYPE_CHECKING: + from tensorrt_llm._torch.model_config import ModelConfig + from tensorrt_llm.llmapi.llm_args import TorchLlmArgs + +_LAYERS = 72 + +# The per-layer structure the forward dispatches on, as this checkpoint states it. Every +# fourth layer takes no rope, in MLA and in its indexer; the NoPE layers past 35 own no latent +# and read layer 35's (itself NoPE); the first 17 layers run their own indexer and after that +# the two layers following each indexer pair reuse its selection. The forward is written +# against exactly this pattern -- every KV sharer NoPE, its source NoPE, layer 0 an indexer +# layer -- so another pattern is another forward. +_NOPE = [i % 4 == 3 for i in range(_LAYERS)] +_KV_SOURCE = [35 if i > 35 and i % 4 == 3 else i for i in range(_LAYERS)] +_INDEX_SHARING = ["full" if i < 17 or i % 4 in (0, 3) else "shared" for i in range(_LAYERS)] + +# (layers, hidden, heads, q_lora, kv_lora, qk_nope, qk_rope, v, routed experts, top-k, shared +# experts, expert width, dense width, dense layers, indexer heads, indexer head dim, index_topk, +# vocab), in `route`'s order. +_SHAPE = (72, 6144, 64, None, 512, 64, 64, 128, 128, 4, 1, 6144, 24576, 1, 32, 128, 2048, 204800) + +# The ModelOpt MIXED_PRECISION export's per-module quantization, as the checkpoint states it, +# by Mistral module name: the routed experts of every MoE layer NVFP4 W4A4 (16-input blocks); +# `wq`, `wo`, each indexer's `wq_b`, the shared experts and layer 0's dense MLP FP8 128x128 +# blocks with dynamic 1x128 activations (FP8_PB_WO); every other module bf16. Which kernel each +# projection runs follows from it, so the whole map is compared. +_NVFP4_EXPERTS = {"quant_algo": "NVFP4", "group_size": 16} +_FP8_PB_WO = {"quant_algo": "FP8_PB_WO"} +_MIXED_PRECISION_LAYERS = { + **{f"layers.{i}.experts": _NVFP4_EXPERTS for i in range(1, _LAYERS)}, + **{f"layers.{i}.attention.{m}": _FP8_PB_WO for i in range(_LAYERS) for m in ("wq", "wo")}, + **{ + f"layers.{i}.attention.indexer.wq_b": _FP8_PB_WO + for i in range(_LAYERS) + if _INDEX_SHARING[i] == "full" + }, + **{f"layers.{i}.shared_experts.w{j}": _FP8_PB_WO for i in range(1, _LAYERS) for j in (1, 2, 3)}, + **{f"layers.0.feed_forward.w{j}": _FP8_PB_WO for j in (1, 2, 3)}, +} + + +def _native_dspark(spec: Any) -> bool: + """The checkpoint's trained DSpark block and the runtime this target implements.""" + if spec is None: + return True + native = getattr(spec, "_draft_model_config", None) or {} + head = native.get("dspark") or {} + return ( + spec.spec_dec_mode.name == "DSPARK" + and spec.max_draft_len == spec.block_size == 3 + and spec.target_layer_ids == [1, 18, 36, 54, 71] + and spec.mask_token_id == _SHAPE[-1] + and spec.markov_rank == 512 + and spec.markov_head_type == "rnn" + and spec.attention_backend in ("AUTO", "VANILLA") + and spec.draft_len_schedule is None + and not spec._use_shared_kv_cache + and spec._allow_separate_draft_kv_cache + and native.get("n_layers") == 3 + and native.get("depth") == 3 + and native.get("ragged_attention") == "2048,2048,2048" + and native.get("norm_strategy") == "eagle" + and head.get("draft_input") == "position_specific" + and head.get("head_type") == "rnn" + and head.get("markov_rank") == 512 + ) + + +def _yarn_mscale(factor: float, mscale: float) -> float: + return 1.0 if factor <= 1 else 0.1 * mscale * math.log(factor) + 1.0 + + +def _per_layer_structure(cfg: Any) -> bool: + return ( + [bool(f) for f in getattr(cfg, "nope_layer_flags", None) or []] == _NOPE + and [bool(f) for f in getattr(cfg, "indexer_nope_layer_flags", None) or []] == _NOPE + and [int(s) for s in getattr(cfg, "kv_source_layers", None) or []] == _KV_SOURCE + and list(getattr(cfg, "index_sharing", None) or []) == _INDEX_SHARING + ) + + +def _math_switches(cfg: Any) -> bool: + """The operations the forward contains beyond the shape and the weight formats: SSMax on + both queries, LeRoPE, the logit soft-cap, softmax top-k routing renormalised without + groups, interleaved YaRN with no magnitude scaling, and an fp8 latent cache.""" + rope = getattr(cfg, "rope_parameters", None) or {} + factor = float(rope.get("factor", 1.0)) + return ( + getattr(cfg, "ssmax_scaling", False) is True + and getattr(cfg, "indexer_ssmax_scaling", False) is True + and getattr(cfg, "lerope", False) is True + and getattr(cfg, "softmax_tanh_gating", None) is not None + and getattr(cfg, "scoring_func", None) == "softmax" + and getattr(cfg, "norm_topk_prob", False) is True + and getattr(cfg, "n_group", 1) == 1 + and getattr(cfg, "topk_group", 1) == 1 + and getattr(cfg, "topk_method", None) is None + and rope.get("rope_type") == "yarn" + and rope.get("attention_factor") == 1.0 + and _yarn_mscale(factor, float(rope.get("mscale", 1))) + == _yarn_mscale(factor, float(rope.get("mscale_all_dim", 0))) + and getattr(cfg, "fp8_kv_cache", False) is True + ) + + +def _nvfp4_mixed_precision(cfg: Any) -> bool: + """The NVFP4 export: ModelOpt MIXED_PRECISION, 16-input NVFP4 blocks, and exactly the + per-module map the forward is written against.""" + quant = getattr(cfg, "quantization_config", None) or {} + return ( + quant.get("quant_method") == "modelopt" + and quant.get("quant_algo") == "MIXED_PRECISION" + and quant.get("group_size") == 16 + and quant.get("quantized_layers") == _MIXED_PRECISION_LAYERS + ) + + +def serves_images(pretrained_config: Any) -> bool: + """Whether this config's target encodes images: the export carries a vision encoder.""" + return getattr(pretrained_config, "vision_config", None) is not None + + +@routes("MistralLarge4ForCausalLM") +def route( + model_config: ModelConfig, llm_args: TorchLlmArgs, compute_capability: Tuple[int, int] +) -> type: + """Return the target class for this deployment, or raise ``NoTarget`` saying why none fits. + + The target is imported where it is returned, so declining costs no import of modeling code. + """ + cfg = model_config.pretrained_config + mapping = model_config.mapping + spec_config = model_config.spec_config + # The targets encode images with exactly this vision tower. + vision = cfg.vision_config if serves_images(cfg) else None + if vision is not None: + expected_vision = { + "num_hidden_layers": 30, + "hidden_size": 2048, + "num_attention_heads": 16, + "intermediate_size": 6144, + "num_channels": 3, + "image_size": 1540, + "patch_size": 14, + "rope_theta": 10000.0, + "spatial_merge_size": 2, + "mm_projector_id": "patch_merge", + "adapter_bias": False, + "add_pre_mm_projector_layer_norm": True, + "image_token_id": 10, + "image_break_token_id": 12, + "image_end_token_id": 13, + } + if any(getattr(vision, key, None) != value for key, value in expected_vision.items()): + raise NoTarget("unsupported Mistral-Large-4 vision encoder or projector geometry") + if getattr(vision, "rms_norm_eps", 1e-5) != 1e-5: + raise NoTarget("Mistral-Large-4 vision RMSNorm requires eps=1e-5") + # The checkpoint is recognized by its config shape: the trunk, the MLA head split, the + # MoE geometry and the indexer, every width the declared weights are written against. + shape = ( + cfg.num_hidden_layers, + cfg.hidden_size, + cfg.num_attention_heads, + getattr(cfg, "q_lora_rank", None), + getattr(cfg, "kv_lora_rank", None), + getattr(cfg, "qk_nope_head_dim", None), + getattr(cfg, "qk_rope_head_dim", None), + getattr(cfg, "v_head_dim", None), + getattr(cfg, "n_routed_experts", None), + getattr(cfg, "num_experts_per_tok", None), + getattr(cfg, "n_shared_experts", None), + getattr(cfg, "moe_intermediate_size", None), + getattr(cfg, "intermediate_size", None), + getattr(cfg, "first_k_dense_replace", None), + getattr(cfg, "index_n_heads", None), + getattr(cfg, "index_head_dim", None), + getattr(cfg, "index_topk", None), + cfg.vocab_size, + ) + # The targets serve 4 ranks on one node or 8 ranks over two nodes of four: every per-rank + # width -- heads, expert windows, shard widths -- is read off the mapping, so the two sizes + # are one weight layout and one forward. + nvfp4_world = mapping.world_size in (4, 8) + # tep: the attention heads, the dense and shared MLPs and lm_head sharded over every rank, + # the routed experts too, by expert (32 per rank at 4 ranks, 16 at 8). Attention DP would + # replicate attention: another weight layout. At 4 ranks this is the tep4 deployment, at 8 + # the tep8 one; tensor_parallel_size 8 alone shards every expert eight ways instead (moe_tp + # 8, moe_ep 1), the tp branch's layout. + tep = ( + mapping.tp_size == mapping.world_size + and mapping.pp_size == 1 + and mapping.moe_ep_size == mapping.world_size + and mapping.moe_tp_size == 1 + and not mapping.enable_attention_dp + ) + # dep: attention data parallel over the ranks -- every rank holds the whole attention, the + # dense and shared MLPs and the full-vocabulary lm_head and serves its own requests -- and + # the routed experts split by expert over every rank (32 per rank at 4 ranks, 16 at 8), + # gathered over the ranks' tokens. lm_head tensor parallelism under attention DP would make + # the shell's lm_head vocab-parallel: another weight layout. At 4 ranks this is the dep4 + # deployment, at 8 the dep8 one. + dep = ( + mapping.tp_size == mapping.world_size + and mapping.pp_size == 1 + and mapping.moe_ep_size == mapping.world_size + and mapping.moe_tp_size == 1 + and mapping.enable_attention_dp + and not mapping.enable_lm_head_tp_in_adp + ) + # tp: tensor parallel throughout -- the attention heads, the dense and shared MLPs and + # lm_head sharded over every rank as at tep -- and the routed experts tensor parallel too: + # every rank holds all 128 experts, each cut to its 1/world_size of the intermediate width + # (1536 at 4 ranks, 768 at 8), and the ranks' partial sums close on the same all-reduce. At 4 + # ranks this is the tp4 deployment; at 8 the tp8 one, and what tensor_parallel_size 8 alone + # resolves to. + # Expert parallelism, or a mixed split of the experts, would be another weight layout; + # attention DP another attention. + tp = ( + mapping.tp_size == mapping.world_size + and mapping.pp_size == 1 + and mapping.moe_tp_size == mapping.world_size + and mapping.moe_ep_size == 1 + and not mapping.enable_attention_dp + ) + layers = _per_layer_structure(cfg) + math_ok = _math_switches(cfg) + nvfp4 = _nvfp4_mixed_precision(cfg) + draft = _native_dspark(spec_config) + # The NVFP4 checkpoint carries a BF16 DSpark block; the target and draft + # follow the same parallel topology, on 4 ranks or 8 over two nodes. + if ( + compute_capability in ((10, 3), (10, 7)) + and shape == _SHAPE + and nvfp4_world + and tep + and draft + and layers + and math_ok + and nvfp4 + ): + if compute_capability == (10, 7): + # fmt: off + # isort: off + from .mistral_large_4_nvfp4__sm_107__tep.modeling import ModelingV2MistralLarge4Nvfp4Sm107Tep + # isort: on + # fmt: on + + return ModelingV2MistralLarge4Nvfp4Sm107Tep + + # One line on purpose: the claims suite reads the routed targets off + # `from ..modeling import ` lines. + # fmt: off + # isort: off + from .mistral_large_4_nvfp4__sm_103__tep.modeling import ModelingV2MistralLarge4Nvfp4Sm103Tep + # isort: on + # fmt: on + + return ModelingV2MistralLarge4Nvfp4Sm103Tep + + # DEP replicates the attention and dense draft for each rank's requests. + if ( + compute_capability in ((10, 3), (10, 7)) + and shape == _SHAPE + and nvfp4_world + and dep + and draft + and layers + and math_ok + and nvfp4 + ): + if compute_capability == (10, 7): + # fmt: off + # isort: off + from .mistral_large_4_nvfp4__sm_107__dep.modeling import ModelingV2MistralLarge4Nvfp4Sm107Dep + # isort: on + # fmt: on + + return ModelingV2MistralLarge4Nvfp4Sm107Dep + + # One line on purpose: the claims suite reads the routed targets off + # `from ..modeling import ` lines. + # fmt: off + # isort: off + from .mistral_large_4_nvfp4__sm_103__dep.modeling import ModelingV2MistralLarge4Nvfp4Sm103Dep + # isort: on + # fmt: on + + return ModelingV2MistralLarge4Nvfp4Sm103Dep + + # TP shards the draft attention and dense MLP like the target's dense layer. + if ( + compute_capability in ((10, 3), (10, 7)) + and shape == _SHAPE + and nvfp4_world + and tp + and draft + and layers + and math_ok + and nvfp4 + ): + if compute_capability == (10, 7): + # fmt: off + # isort: off + from .mistral_large_4_nvfp4__sm_107__tp.modeling import ModelingV2MistralLarge4Nvfp4Sm107Tp + # isort: on + # fmt: on + + return ModelingV2MistralLarge4Nvfp4Sm107Tp + + # One line on purpose: the claims suite reads the routed targets off + # `from ..modeling import ` lines. + # fmt: off + # isort: off + from .mistral_large_4_nvfp4__sm_103__tp.modeling import ModelingV2MistralLarge4Nvfp4Sm103Tp + # isort: on + # fmt: on + + return ModelingV2MistralLarge4Nvfp4Sm103Tp + + if nvfp4: + quant = "ModelOpt MIXED_PRECISION (NVFP4 experts, FP8_PB_WO linears)" + else: + quant = "differs" + spec = "off" if spec_config is None else spec_config.spec_dec_mode.name + raise NoTarget( + f"compute capability {compute_capability}, config shape {shape}, " + f"world_size={mapping.world_size} tp={mapping.tp_size} pp={mapping.pp_size} " + f"moe_ep={mapping.moe_ep_size} moe_tp={mapping.moe_tp_size} " + f"attention_dp={mapping.enable_attention_dp} " + f"lm_head_tp_in_adp={mapping.enable_lm_head_tp_in_adp}, speculative decoding {spec}, " + f"per-layer NoPE/KV-sharing/indexer-sharing pattern " + f"{'as written' if layers else 'differs'}, math switches " + f"(SSMax, LeRoPE, soft-cap, softmax routing, YaRN magnitude 1, fp8 KV) " + f"{'as written' if math_ok else 'differ'}, quantization {quant}; the mistral_large4 " + "targets are Mistral-Large-4-1T-A52B-NVFP4 (ModelOpt MIXED_PRECISION: NVFP4 routed " + "experts, FP8_PB_WO linears, group 16, its 475-module map) on sm103 or sm107 in the tep, " + "dep and tp layouts; each target serves 4 ranks on one node or 8 ranks over two nodes of " + "four (world_size=tp=4 or 8, pp=1); tep is tensor-parallel attention, expert-parallel " + "MoE, attention DP off (world_size=4 tp=4 pp=1 moe_ep=4 moe_tp=1, or world_size=8 tp=8 " + "pp=1 moe_ep=8 moe_tp=1: tep8); dep is attention data parallel with lm_head TP off, " + "expert-parallel MoE (world_size=4 tp=4 pp=1 moe_ep=4 moe_tp=1, or world_size=8 tp=8 " + "pp=1 moe_ep=8 moe_tp=1: dep8); tp is tensor-parallel attention, tensor-parallel MoE " + "(every expert on every rank, cut by width), attention DP off (world_size=4 tp=4 pp=1 " + "moe_ep=1 moe_tp=4, or world_size=8 tp=8 pp=1 moe_ep=1 moe_tp=8: tp8). " + "Speculative decoding is off or native DSpark: fixed block/depth 3, layers " + "1,18,36,54,71, three dense draft layers, window 2048, eagle norm, " + "position-specific masks and rank-512 RNN, AUTO/VANILLA attention with " + "separate draft KV" + ) diff --git a/tensorrt_llm/_torch/models/_arch_index.py b/tensorrt_llm/_torch/models/_arch_index.py index ce451df18cd1..503cca991c4d 100644 --- a/tensorrt_llm/_torch/models/_arch_index.py +++ b/tensorrt_llm/_torch/models/_arch_index.py @@ -241,6 +241,7 @@ def is_builtin_zoo_module(module_name: str) -> bool: "mistral3": "modeling_mistral", "mistral_common": "modeling_mistral", "mistral_large_3": "modeling_mistral", + "mistral_large_4": "modeling_mistral", "muse_spark": "modeling_musespark_open_omni", "nemotron_h_omni": "modeling_nemotron_h_multimodal", "phi4mm": "modeling_phi4mm", diff --git a/tensorrt_llm/_torch/models/checkpoints/dspark_config.py b/tensorrt_llm/_torch/models/checkpoints/dspark_config.py new file mode 100644 index 000000000000..c5ea28a78c25 --- /dev/null +++ b/tensorrt_llm/_torch/models/checkpoints/dspark_config.py @@ -0,0 +1,116 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Resolve DSpark checkpoint settings before model or weight construction.""" + +import json +import os +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + from tensorrt_llm.llmapi.llm_args import DSparkDecodingConfig + + +def resolve_dspark_checkpoint(spec_cfg: "DSparkDecodingConfig", target_model: str) -> None: + from tensorrt_llm.llmapi.llm_args import translate_speculators_dspark_config + + from .mistral.config_loader import resolve_native_dspark_config + + # Re-resolve after model defaults reconstruct a config from public fields. + spec_cfg._draft_weights_in_target = False + spec_cfg._draft_model_config = None + checkpoint = spec_cfg.speculative_model or target_model + params_path = os.path.join(checkpoint, "params.json") + if os.path.isfile(params_path): + with open(params_path, encoding="utf-8") as stream: + params = json.load(stream) + resolve_native_dspark_config(params, spec_cfg, checkpoint, target_model) + + # An omitted draft path means the draft lives in the target checkpoint. + if spec_cfg.speculative_model is None: + spec_cfg.speculative_model = target_model + if not spec_cfg.draft_is_embedded_in_target: + raise ValueError( + "speculative_config.speculative_model is unset, " + "which means 'load the draft from the target " + "checkpoint', but the target has no mtp.* draft " + "weights. Point speculative_model at a standalone " + "DSpark drafter checkpoint directory." + ) + # Warm the embedded-vs-standalone probe here, while we are in + # the main process and the checkpoint path is known local. The + # flag then travels with the config, so no rank repeats the + # filesystem read and nothing probes at forward time. + if spec_cfg.draft_is_embedded_in_target and spec_cfg.use_ancestral_draft_sampling: + raise ValueError( + "use_ancestral_draft_sampling=True is not supported with a " + "drafter embedded in the target checkpoint." + ) + # Resolve target_layer_ids / mask_token_id / block_size / + # markov_rank from the draft (or main) model config if not set. + # Several checkpoint spellings are in the wild for the same knobs + # and all are accepted here, because a key the reader misses is + # not an error -- it silently falls back to a default and the + # drafter degrades (a markov_rank read as 0 skips the Markov + # head entirely, costing acceptance with no warning): + # - top-level ``dspark_*`` (DeepSeek-V4-Pro) + # - nested ``dflash_config`` (SpecForge / RadixArk drafters) + # - nested ``dspark_config`` (forward compatibility) + # - plain top-level keys (TorchSpec drafters) + # V4.1 wraps the language model settings in ``text_config``; + # prefer that scope, retaining the top-level legacy fallback. + draft_config_path = os.path.join(spec_cfg.speculative_model, "config.json") + if os.path.exists(draft_config_path): + with open(draft_config_path) as f: + draft_cfg = translate_speculators_dspark_config(json.load(f)) + text_cfg = draft_cfg.get("text_config") or {} + + def _dspark_get(key: str, top_level_key: str) -> list[int] | int | None: + for scope in (text_cfg, draft_cfg): + for source, name in ( + (scope.get("dspark_config") or {}, key), + (scope.get("dflash_config") or {}, key), + (scope, top_level_key), + (scope, key), + ): + value = source.get(name) + if value is not None: + return value + return None + + # The checkpoint's ``dspark_target_layer_ids`` is + # authoritative: it fixes both which target hidden states + # were captured during draft training and the input width of + # ``main_proj`` (hidden_size * num_capture_layers). If the + # user leaves ``target_layer_ids`` unset we copy it verbatim + # (order preserved, since the projection columns are + # order-dependent). An explicit override that does not match + # the checkpoint list exactly would either mismatch the + # projection shape at runtime (different count) or feed the + # draft hidden states it was not trained on (same count, + # different layers), so reject it during validation. + ckpt_layer_ids = _dspark_get("target_layer_ids", "dspark_target_layer_ids") + if spec_cfg.target_layer_ids is None: + if ckpt_layer_ids is not None: + spec_cfg.target_layer_ids = list(ckpt_layer_ids) + elif ckpt_layer_ids is not None and list(spec_cfg.target_layer_ids) != list(ckpt_layer_ids): + raise ValueError( + "DSpark target_layer_ids must match the checkpoint's " + "dspark_target_layer_ids exactly (the draft " + "projection weights are trained for that specific " + f"layer set); got override {spec_cfg.target_layer_ids} " + f"but the checkpoint specifies {list(ckpt_layer_ids)}. " + "Leave target_layer_ids unset to use the checkpoint " + "value." + ) + if spec_cfg.mask_token_id is None: + mask_id = _dspark_get("mask_token_id", "dspark_noise_token_id") + if mask_id is not None: + spec_cfg.mask_token_id = mask_id + if spec_cfg.block_size is None: + block_size = _dspark_get("block_size", "dspark_block_size") + if block_size is not None: + spec_cfg.block_size = int(block_size) + if spec_cfg.markov_rank is None: + markov_rank = _dspark_get("markov_rank", "dspark_markov_rank") + if markov_rank is not None: + spec_cfg.markov_rank = markov_rank diff --git a/tensorrt_llm/_torch/models/checkpoints/mistral/config_loader.py b/tensorrt_llm/_torch/models/checkpoints/mistral/config_loader.py index 00779ca022d9..f9f058b28abf 100644 --- a/tensorrt_llm/_torch/models/checkpoints/mistral/config_loader.py +++ b/tensorrt_llm/_torch/models/checkpoints/mistral/config_loader.py @@ -13,14 +13,19 @@ # See the License for the specific language governing permissions and # limitations under the License. +import importlib.metadata import json +import os +from copy import deepcopy from pathlib import Path -from typing import Any +from typing import TYPE_CHECKING, Any import torch +from packaging.version import Version from transformers import PretrainedConfig, WhisperConfig from tensorrt_llm._errors import UnsupportedError +from tensorrt_llm._torch.configs.mistral_large4 import MistralLarge4Config from tensorrt_llm._torch.model_config import ModelConfig from tensorrt_llm._torch.models.checkpoints.base_config_loader import BaseConfigLoader from tensorrt_llm._torch.models.modeling_utils import register_config_loader @@ -29,6 +34,53 @@ from tensorrt_llm.models.modeling_utils import QuantConfig from tensorrt_llm.quantization.mode import QuantAlgo +if TYPE_CHECKING: + from tensorrt_llm.llmapi.llm_args import DSparkDecodingConfig + + +def resolve_native_dspark_config( + params: dict, spec: "DSparkDecodingConfig", checkpoint: str, target_model: str +) -> None: + """Adapt the native draft's trained geometry and checkpoint ownership.""" + mtp = params.get("mtp") or {} + dspark = mtp.get("dspark") + if dspark is None: + return + if os.path.realpath(checkpoint) != os.path.realpath(target_model): + raise ValueError("Native Mistral DSpark must use the target checkpoint") + if spec.use_ancestral_draft_sampling: + raise ValueError( + "use_ancestral_draft_sampling=True is not supported with the native " + "Mistral DSpark drafter, whose Markov head is recurrent." + ) + layers = [int(layer) for layer in mtp["target_activation_layers"].split(",")] + trained = dict( + target_layer_ids=layers, + block_size=int(mtp["depth"]), + markov_rank=int(dspark["markov_rank"]), + markov_head_type=dspark["head_type"], + mask_token_id=int(params["vocab_size"]), + ) + for field, value in trained.items(): + override = getattr(spec, field) + if override is not None and override != value: + raise ValueError( + f"Mistral DSpark {field} must match the checkpoint: {value}, got {override}" + ) + if spec.max_draft_len != trained["block_size"]: + raise ValueError( + "Mistral DSpark max_draft_len must equal the trained block depth " + f"{trained['block_size']}" + ) + for field, value in trained.items(): + setattr(spec, field, value) + spec._draft_model_config = mtp + # Native draft weights are part of the target's manifest, while its paged + # draft runtime and KV cache remain independent of the target's cache. + spec._draft_weights_in_target = True + spec.speculative_model = checkpoint + + ################### # vllm code here # https://github.com/vllm-project/vllm/blob/48a5fff66e78985a634abac0d8d7f271da744000/vllm/transformers_utils/configs/mistral.py @@ -67,6 +119,15 @@ def adapt_config_dict( config_dict: dict[str, Any], defaults: dict[str, Any] = {}, ) -> PretrainedConfig: + if _is_mistral_large4(config_dict): + _require_mistral_common_for_mistral_large4() + config_dict = deepcopy(config_dict) + config_dict = _remap_general_mistral_args(config_dict) + config_dict = _remap_moe_args(config_dict) + for key, value in defaults.items(): + config_dict.setdefault(key, value) + return _adapt_mistral_large4_config(config_dict) + config_dict = _remap_general_mistral_args(config_dict) if bool(config_dict.get("quantization")): @@ -122,6 +183,81 @@ def adapt_config_dict( return config +def _is_mistral_large4(config: dict) -> bool: + return bool(config.get("dsa") and config.get("kv_lora_rank") and config.get("lerope")) + + +def is_mistral_large4_checkpoint(checkpoint_dir: str | Path) -> bool: + """Whether a directory's params.json describes a Mistral-Large-4 model.""" + params_path = Path(checkpoint_dir) / "params.json" + if not params_path.is_file(): + return False + try: + with open(params_path) as f: + params = json.load(f) + except (OSError, ValueError): + return False + return isinstance(params, dict) and _is_mistral_large4(params) + + +_MISTRAL_LARGE4_MISTRAL_COMMON = "1.13.0" + + +def _require_mistral_common_for_mistral_large4() -> None: + """Mistral-Large-4's tekken.json and chat settings need mistral-common 1.13.0 or later.""" + installed = importlib.metadata.version("mistral-common") + if Version(installed) < Version(_MISTRAL_LARGE4_MISTRAL_COMMON): + raise ImportError( + f"Mistral-Large-4 requires mistral-common>={_MISTRAL_LARGE4_MISTRAL_COMMON} " + f"(Python 3.11 or later); the installed version is {installed}." + ) + + +def _adapt_mistral_large4_config(config: dict) -> MistralLarge4Config: + """Adapt the native MLA/DSA language model, preserving the quantized-layer policy.""" + dsa = config.pop("dsa") + config.update( + model_type="mistral_large_4", + architectures=["MistralLarge4ForCausalLM"], + dtype=config.get("dtype") or "bfloat16", + index_n_heads=dsa["n_heads"], + index_head_dim=dsa["head_dim"], + index_topk=dsa["topk"], + indexer_norm_eps=dsa["norm_eps"], + indexer_ssmax_scaling=dsa["ssmax_scaling"], + index_sharing=dsa["sharing"].split(","), + fp8_kv_cache=dsa["fp8_kv_cache"], + ) + if dsa["fp8_kv_cache"] and config.get("quantization_config"): + config["quantization_config"].setdefault("kv_cache_quant_algo", "FP8") + num_layers = config["num_hidden_layers"] + pattern = config.pop("nope").split(",") + if set(pattern) - {"rope", "nope"} or num_layers % len(pattern): + raise ValueError("nope must be a rope/nope pattern dividing n_layers") + config["nope_layer_flags"] = [ + pattern[layer % len(pattern)] == "nope" for layer in range(num_layers) + ] + config["indexer_nope_layer_flags"] = ( + list(config["nope_layer_flags"]) if dsa["match_nope"] else [False] * num_layers + ) + config["kv_source_layers"] = [ + int(layer) for layer in config.pop("kv_sharing")["source_layers_str"].split(",") + ] + config = _remap_mistral_yarn_args(config) + rope = config.pop("rope_scaling") + if rope.pop("apply_yarn_scaling", False): + raise ValueError("Mistral-Large-4 requires YaRN without magnitude scaling") + config["rope_parameters"] = { + **rope, + "rope_theta": config["rope_theta"], + "attention_factor": 1.0, + } + vision = config.pop("vision_encoder", None) + if vision is not None: + config["vision_config"] = vision + return MistralLarge4Config.from_dict(config) + + def _remap_mistral_vision_args(config: dict) -> dict: if config.get("multimodal"): vision_config = config.pop("multimodal") @@ -274,7 +410,19 @@ def _remap_moe_args(config: dict) -> dict: @register_config_loader(CheckpointFormat.MISTRAL) @register_config_loader(CheckpointFormat.MISTRAL_LARGE_3) class MistralConfigLoader(BaseConfigLoader): - def _load_mistral_config_dict(self, checkpoint_dir: str, config_file_name: str) -> dict | None: + def load_pretrained_config(self, checkpoint_dir: str) -> PretrainedConfig: + """The checkpoint's params.json as the `PretrainedConfig` its model reads.""" + _, config = self._parse_mistral_config(checkpoint_dir) + if isinstance(config, MistralLarge4Config): + # `load` builds this model's ModelConfig with ModelConfig.from_pretrained, + # which reads the adapted config back from here. + config._name_or_path = str(checkpoint_dir) + return config + return self.load(checkpoint_dir).pretrained_config + + def _load_mistral_config_dict( + self, checkpoint_dir: str | Path, config_file_name: str + ) -> dict | None: file_path = Path(checkpoint_dir) / Path(config_file_name) if file_path.exists() and file_path.is_file(): @@ -323,6 +471,13 @@ def load(self, checkpoint_dir: str, **kwargs) -> ModelConfig: config_dict, pretrained_config = self._parse_mistral_config(checkpoint_dir) + if isinstance(pretrained_config, MistralLarge4Config): + # The shared builder derives DSA metadata and per-layer ModelOpt quantization + # from the native config returned by load_pretrained_config. + return ModelConfig.from_pretrained( + checkpoint_dir, checkpoint_format=CheckpointFormat.MISTRAL, **kwargs + ) + # Some checkpoints lack torch_dtype, populate with dtype pretrained_config.torch_dtype = ( getattr(pretrained_config, "dtype", torch.bfloat16) or torch.bfloat16 @@ -406,10 +561,11 @@ def load(self, checkpoint_dir: str, **kwargs) -> ModelConfig: # Native (mistral-format) checkpoints are served through the # mistral-common tokenizer/processor, which applies its own chat # template. Tag the config with a dedicated serving model_type so the - # serving layer resolves the PASSTHROUGH placeholder/chat-template - # metadata (registered under "mistral_common" in modeling_mistral.py) - # via the normal config path - i.e. resolve_top_level_model_type() - - # instead of having to inspect the live input processor. + # serving layer resolves the mistral-common renderer and the + # placeholder metadata (registered under "mistral_common" in + # modeling_mistral.py) via the normal config path - i.e. + # resolve_top_level_model_type() - instead of having to inspect the + # live input processor. model_config.pretrained_config.input_processor_type = "mistral_common" model_config.pretrained_config.model_type = "mistral_common" model_config._frozen = True diff --git a/tensorrt_llm/_torch/models/checkpoints/mistral/tokenizer.py b/tensorrt_llm/_torch/models/checkpoints/mistral/tokenizer.py index 6a9b76b10f54..8b933b7ebe31 100644 --- a/tensorrt_llm/_torch/models/checkpoints/mistral/tokenizer.py +++ b/tensorrt_llm/_torch/models/checkpoints/mistral/tokenizer.py @@ -1,7 +1,17 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +import copy +import hashlib +import json +from importlib.metadata import version as package_version from pathlib import Path -from typing import Union +from typing import TYPE_CHECKING, Optional, Union +import numpy as np +from mistral_common.protocol.instruct.chunk import ImageChunk, ImageURLChunk from mistral_common.tokens.tokenizers.base import SpecialTokenPolicy, SpecialTokens +from mistral_common.tokens.tokenizers.image import ImageEncoder, ImageEncoding, image_from_chunk from mistral_common.tokens.tokenizers.sentencepiece import SentencePieceTokenizer from mistral_common.tokens.tokenizers.tekken import Tekkenizer from transformers.tokenization_mistral_common import ( @@ -11,6 +21,28 @@ from tensorrt_llm.llmapi.tokenizer import TransformersTokenizer from tensorrt_llm.logger import logger +if TYPE_CHECKING: + import llguidance + from mistral_common.guidance.grammar_factory import GrammarFactory + from mistral_common.tokens.tokenizers.mistral import MistralTokenizer as MistralCommonTokenizer + + +class _ImageGridEncoder(ImageEncoder): + """mistral-common's image encoder reduced to each image's token grid. + + The grid of ``[IMG]``, ``[IMG_BREAK]`` and ``[IMG_END]`` tokens depends on the image size + alone, so a chat encodes to the same ids as with the full encoder, and no pixel is resized + or normalized. + """ + + def __call__(self, content: Union[ImageChunk, ImageURLChunk]) -> ImageEncoding: + width_tokens, height_tokens = self._image_to_num_tokens(image_from_chunk(content)) + tokens = ( + [self.special_ids.img] * width_tokens + [self.special_ids.img_break] + ) * height_tokens + tokens[-1] = self.special_ids.img_end + return ImageEncoding(tokens=tokens, image=np.empty((0, 0, 0), dtype=np.float32)) + # Adapted from: # https://github.com/vllm-project/vllm/blob/8e67b2557aae7204c697d7a5c61e00754da465be/vllm/transformers_utils/tokenizers/mistral.py#L166 @@ -29,9 +61,10 @@ def __init__(self, tokenizer: "TransformersMistralTokenizer"): if not (self.is_tekken or self.is_spm): raise TypeError(f"Unsupported tokenizer: {type(self.tokenizer)}") - # Reverse order to ensure that the lowest token id is kept. + # Reverse order to ensure that the lowest token id is kept. The vocab keys each id + # by its piece as written (convert_ids_to_tokens may return bytes for streaming). self._vocab_dict = { - self.convert_ids_to_tokens([i], skip_special_tokens=False)[0]: i + self.tokenizer.id_to_piece(i): i for i in range(self.transformers_tokenizer.vocab_size - 1, -1, -1) } # Sort the dict for convenience @@ -49,6 +82,63 @@ def __init__(self, tokenizer: "TransformersMistralTokenizer"): self._all_special_tokens_set = set(self.all_special_tokens) + self._encoding_identity: Optional[dict] = None + self._grid_only_mistral: Optional["MistralCommonTokenizer"] = None + self._grammar_factory_and_template: Optional[tuple["GrammarFactory", str]] = None + + def __reduce__(self): + # TransformersTokenizer.__reduce__ would rebuild a TransformersTokenizer + # around ``self.tokenizer``, which here is the raw mistral-common + # tokenizer. Rebuild this class from the transformers wrapper instead; + # mistral-common pickles the tokenizer by the path of its file. + return (self.__class__, (self.transformers_tokenizer,)) + + @property + def encoding_identity(self) -> Optional[dict]: + """What turns text, images and chats into ids, for the rendering fingerprint. + + mistral-common reads all of it from the tokenizer file (vocabulary, pattern, + control tokens, image and model settings) and renders chats with its own code, + so the identity is the file's digest and mistral-common's version. ``None`` + when the file is not known. The file is hashed on first use. + """ + if self._encoding_identity is None: + path = getattr(self.tokenizer, "file_path", None) + if path is None or not Path(path).is_file(): + return None + self._encoding_identity = { + "kind": "mistral_common", + "file_sha256": hashlib.sha256(Path(path).read_bytes()).hexdigest(), + "mistral_common": package_version("mistral_common"), + } + return self._encoding_identity + + @property + def grid_only_mistral(self) -> "MistralCommonTokenizer": + """``self.mistral`` with an image encoder that makes each image's token grid only. + + It encodes a chat to the same ids as ``self.mistral``, and an image needs only its + size: no pixel is resized or normalized. Chat rendering uses it; the input + processor encodes the pixels. Built on first use. + """ + if self._grid_only_mistral is None: + image_encoder = getattr(self.instruct, "image_encoder", None) + if image_encoder is None: + self._grid_only_mistral = self.mistral + else: + instruct = copy.copy(self.instruct) + instruct.image_encoder = _ImageGridEncoder( + image_encoder.image_config, image_encoder.special_ids + ) + # The constructor, not copy.copy: mistral-common pickles (and so copies) a + # tokenizer by reloading its file. + self._grid_only_mistral = type(self.mistral)( + instruct, + self.mistral._chat_completion_request_validator, + self.mistral._instruct_request_normalizer, + ) + return self._grid_only_mistral + def _get_special_tokens(self, all_special_ids: list[int]) -> list[str]: from mistral_common.tokens.tokenizers.base import SpecialTokenPolicy @@ -79,6 +169,127 @@ def from_pretrained(cls, pretrained_model_dir: str, **kwargs): tokenizer = TransformersMistralTokenizer.from_pretrained(pretrained_model_dir) return cls(tokenizer) + def xgrammar_tokenizer_info(self) -> dict: + """The vocabulary xgrammar matches grammars against, for a Tekken tokenizer. + + Each token is its raw bytes, which need not be valid UTF-8. Control tokens and + the tokens xgrammar reserves (those starting with a NUL byte) are empty strings, + which xgrammar treats as special: a grammar never selects them, and the + end-of-sequence id is the only stop token. + + Returns: + The ``encoded_vocab`` and ``stop_token_ids`` of a guided decoding config. + """ + if not self.is_tekken: + raise ValueError("Guided decoding supports Tekken Mistral tokenizers only.") + num_special = self.tokenizer.num_special_tokens + encoded_vocab: list[str | bytes] = [] + for token_id in range(self.tokenizer.n_words): + piece = b"" if token_id < num_special else self.tokenizer.id_to_byte_piece(token_id) + encoded_vocab.append("" if piece.startswith(b"\x00") or not piece else piece) + return {"encoded_vocab": encoded_vocab, "stop_token_ids": [self.eos_token_id]} + + def llguidance_tokenizer(self, n_vocab: int) -> "llguidance.LLTokenizer": + """The llguidance tokenizer of a Tekken vocabulary, as mistral-common builds it. + + Ordinary tokens are their raw bytes; control tokens are special, so a grammar + selects one only by reference (``<[9]>`` for ``[TOOL_CALLS]``, as + mistral-common's grammar factory writes them), never as text. Text is + tokenized with Tekken itself. + + Args: + n_vocab: The vocabulary size the token bitmask covers (the model's padded + vocabulary size); at least the Tekken vocabulary size. + + Returns: + The tokenizer an llguidance matcher factory compiles grammars against. + """ + if not self.is_tekken: + raise ValueError("Guided decoding supports Tekken Mistral tokenizers only.") + import llguidance + from mistral_common.guidance.tokenizer import MistralLLGTokenizer + + wrapper = llguidance.TokenizerWrapper(MistralLLGTokenizer(self.tokenizer)) + # llguidance sizes the vocabulary of a wrapped tokenizer from its token list, not + # n_vocab: pad it with empty tokens, which a grammar never selects. + wrapper.tokens = wrapper.tokens + [b""] * (n_vocab - len(wrapper.tokens)) + return llguidance.LLTokenizer(wrapper, n_vocab=n_vocab, eos_token=self.eos_token_id) + + def llguidance_json_schema_lark(self, json_schema: str) -> str: + """The Lark grammar of a JSON response, as mistral-common's grammar factory writes it. + + With the ``[THINK]`` and ``[/THINK]`` control tokens in the vocabulary, the + response may open with one reasoning section, ``[THINK]`` text ``[/THINK]``, + before the JSON: the model reasons there when the request turns reasoning on. + Otherwise the response is the JSON alone. + + Args: + json_schema: The JSON schema of the response, serialized. + + Returns: + The Lark grammar an llguidance matcher compiles against + `llguidance_tokenizer`, which references the control tokens by id. + """ + factory, template = self._grammar_factory + return factory.get_lark_for_json_schema(template, json.loads(json_schema)) + + def llguidance_tool_grammar( + self, tools: list[dict], tool_choice, parallel_tool_calls=None + ) -> str: + """Render native tool calls, including their reasoning and control tokens. + + A named tool choice is resolved as mistral-common's + `GrammarFactory.get_grammar_for_chat_request` resolves it: the grammar + offers that tool alone, requires a call, and pins the call to its name. + """ + from mistral_common.protocol.instruct.tool_calls import ( + NamedToolChoice, + Tool, + ToolChoiceEnum, + ) + + native_tools = [Tool.from_openai(tool) for tool in tools] + if isinstance(tool_choice, dict): + name = NamedToolChoice.model_validate(tool_choice).function.name + native_tools = [tool for tool in native_tools if tool.function.name == name] + if not native_tools: + raise ValueError(f"Named tool {name!r} is not declared in tools.") + mode, force_tool_names = ToolChoiceEnum.required, True + else: + mode, force_tool_names = ToolChoiceEnum(tool_choice), False + factory, template = self._grammar_factory + return factory.get_lark_from_jinja( + template=template, + mode=mode, + tools=native_tools, + json_schema=None, + parallel_tool_calls=parallel_tool_calls, + json_only=False, + force_tool_names=force_tool_names, + ) + + @property + def renders_grammars(self) -> bool: + """Whether mistral-common's grammar factory supports this tokenizer.""" + from mistral_common.guidance.grammar_factory import GrammarFactory + + return GrammarFactory.is_supported(self.mistral) + + @property + def _grammar_factory(self) -> tuple["GrammarFactory", str]: + """mistral-common's grammar factory and its chat template, built on first use.""" + if self._grammar_factory_and_template is None: + from mistral_common.guidance.grammar_factory import GrammarFactory + + if not self.renders_grammars: + raise ValueError( + "mistral-common renders no grammars for this tokenizer " + f"(version v{self.version})." + ) + factory = GrammarFactory(self.mistral) + self._grammar_factory_and_template = (factory, factory.select_jinja_template()) + return self._grammar_factory_and_template + def _get_special_token_ids(self) -> list[int]: from mistral_common.tokens.tokenizers.sentencepiece import SentencePieceTokenizer from mistral_common.tokens.tokenizers.tekken import Tekkenizer @@ -165,11 +376,14 @@ def convert_tokens_to_string( to_decode_special_tokens = {SpecialTokens.tool_calls} if self.is_tekken: assert isinstance(self.tokenizer, Tekkenizer), type(self.tokenizer) - tokens = [ - t - for t in tokens - if (t in to_decode_special_tokens or t not in self._special_tokens_set) - ] + # Without skip_special_tokens every control token stays, as decode() keeps + # them, so the streamed text equals decode()'s. + if skip_special_tokens: + tokens = [ + t + for t in tokens + if (t in to_decode_special_tokens or t not in self._special_tokens_set) + ] if any(isinstance(t, bytes) for t in tokens): # we need to encode and decode all tokens again @@ -244,21 +458,23 @@ def convert_ids_to_tokens( from mistral_common.tokens.tokenizers.base import SpecialTokenPolicy, SpecialTokens from mistral_common.tokens.tokenizers.instruct import InstructTokenizerV13 - if not skip_special_tokens: - return [self.tokenizer.id_to_piece(token_id) for token_id in ids] - - non_skip_special_tokens_ids = { - self.tokenizer.get_control_token(SpecialTokens.tool_calls), - } - if isinstance(self.instruct, InstructTokenizerV13): - if self.instruct.BEGIN_THINK: - non_skip_special_tokens_ids.add(self.instruct.BEGIN_THINK) - if self.instruct.END_THINK: - non_skip_special_tokens_ids.add(self.instruct.END_THINK) - - ids_kept = [ - i for i in ids if i in non_skip_special_tokens_ids or not self._is_special_token_id(i) - ] + if skip_special_tokens: + non_skip_special_tokens_ids = { + self.tokenizer.get_control_token(SpecialTokens.tool_calls), + } + if isinstance(self.instruct, InstructTokenizerV13): + if self.instruct.BEGIN_THINK: + non_skip_special_tokens_ids.add(self.instruct.BEGIN_THINK) + if self.instruct.END_THINK: + non_skip_special_tokens_ids.add(self.instruct.END_THINK) + + ids_kept = [ + i + for i in ids + if i in non_skip_special_tokens_ids or not self._is_special_token_id(i) + ] + else: + ids_kept = list(ids) # We filtered unwanted special tokens so we can decode the rest. tokens = [self.tokenizer.id_to_piece(token_id) for token_id in ids_kept] diff --git a/tensorrt_llm/_torch/models/modeling_mistral.py b/tensorrt_llm/_torch/models/modeling_mistral.py index eb11968a2f78..2eb31f0c49d0 100644 --- a/tensorrt_llm/_torch/models/modeling_mistral.py +++ b/tensorrt_llm/_torch/models/modeling_mistral.py @@ -5,6 +5,7 @@ import dataclasses from typing import Any, Dict, List, Mapping, Optional, Sequence, Tuple +import numpy as np import torch import torchvision from mistral_common.tokens.tokenizers.multimodal import ImageEncoder @@ -50,10 +51,10 @@ MultimodalPlaceholderMetadata, MultimodalPlaceholderPlacement, TextPrompt, register_input_processor) +from tensorrt_llm.inputs.media_io import convert_image_mode from tensorrt_llm.inputs.multimodal import (MultimodalItemMetadata, MultimodalParams) from tensorrt_llm.inputs.registry import MULTIMODAL_PLACEHOLDER_REGISTRY -from tensorrt_llm.inputs.utils import encode_base64_image from tensorrt_llm.llmapi import SamplingParams from tensorrt_llm.logger import logger @@ -326,46 +327,101 @@ def get_num_tokens_per_image(self, image_size): Image.new("RGB", (w, h))) return ncols * nrows + nrows + @staticmethod + def to_pil_image(image) -> Image.Image: + """An image as the media loaders hand it over -- a PIL image, a ``(C, H, W)`` + tensor (float in [0, 1] for the ``"pt"`` format, or uint8) or a uint8 + ``(H, W, C)`` array (``"np"``) -- as the PIL image mistral-common encodes. + + A float tensor's values are 8-bit levels divided by 255; they are + rounded back to those levels. + """ + if isinstance(image, Image.Image): + return image + if isinstance(image, torch.Tensor): + pixels = image.detach().cpu() + if pixels.is_floating_point(): + pixels = (pixels.float() * 255.0).round_().clamp_(0, 255) + array = pixels.to(torch.uint8).permute(1, 2, 0).numpy() + elif isinstance(image, np.ndarray): + array = image.astype(np.uint8, copy=False) + else: + raise TypeError( + f"unsupported image type {type(image).__name__}; expected a PIL " + "image, a (C, H, W) tensor or a (H, W, C) uint8 array") + if array.ndim == 3 and array.shape[-1] == 1: + array = array[..., 0] + return Image.fromarray(np.ascontiguousarray(array)) + + @staticmethod + def pil_image_size(image: Any) -> Tuple[int, int]: + """The ``(width, height)`` of an image in any form ``to_pil_image`` takes.""" + if isinstance(image, Image.Image): + return image.size + if isinstance(image, torch.Tensor) and image.ndim == 3: + return int(image.shape[2]), int(image.shape[1]) + if isinstance(image, np.ndarray) and image.ndim in (2, 3): + return int(image.shape[1]), int(image.shape[0]) + raise TypeError( + f"unsupported image {type(image).__name__}; expected a PIL image, " + "a (C, H, W) tensor or a (H, W, C) array") + + @staticmethod + def image_list(images: Any) -> List[Any]: + """``images`` as a list: one image, a list or tuple of them, or none.""" + if images is None: + return [] + return list(images) if isinstance(images, (list, tuple)) else [images] + def __call__(self, text: str, - images: list[Image.Image] | None = None, - **kwargs: Any) -> dict[str, torch.Tensor]: - """Tokenize a native request and batch images without losing their sizes.""" - mm_items = [{ - "type": "image", - "base64": encode_base64_image(image) - } for image in (images or [])] - - conversation = [{ - "role": "user", - "content": [{ - "type": "text", - "text": text - }, *mm_items] - }] - - encoded = self.tokenizer.transformers_tokenizer.apply_chat_template( - conversation, tokenize=True, return_dict=True, return_tensors=None) - - processed = { - "input_ids": torch.tensor([encoded.input_ids]), - } - - # text-only mode for VLM - if "pixel_values" in encoded: - # Native images may have different shapes. Preserve their unpadded - # sizes for item scheduling and the vision encoder's patch slicing. + images: Optional[Sequence[Any]] = None, + **kwargs: Any) -> Dict[str, torch.Tensor]: + """Tokenize one user turn of ``text`` and its images with mistral-common. + + The images go where ``text`` has ``[IMG]`` markers, one per image, or + after the text when it has none. mistral-common resizes and + normalizes each image as loaded. Images of different sizes are padded + to one batch; ``image_sizes`` keeps each one's size. + """ + from mistral_common.protocol.instruct.chunk import ImageChunk, TextChunk + from mistral_common.protocol.instruct.messages import UserMessage + from mistral_common.protocol.instruct.request import \ + ChatCompletionRequest + + images = self.image_list(images) + image_chunks = [ + ImageChunk( + image=convert_image_mode(self.to_pil_image(image), "RGB")) + for image in images + ] + if "[IMG]" in text: + parts = text.split("[IMG]") + if len(parts) != len(images) + 1: + raise ValueError( + "Image placeholder count does not match images") + content = [] + for index, part in enumerate(parts): + if part: + content.append(TextChunk(text=part)) + if index < len(image_chunks): + content.append(image_chunks[index]) + else: + content = [TextChunk(text=text), *image_chunks] + encoded = self.tokenizer.mistral.encode_chat_completion( + ChatCompletionRequest(messages=[UserMessage(content=content)])) + processed = {"input_ids": torch.tensor([encoded.tokens])} + if encoded.images: pixel_values, image_sizes = Mistral3VLM.batch_pixel_values([ torch.as_tensor(image, dtype=self.dtype).unsqueeze(0) - for image in encoded.pixel_values - ], [torch.tensor([size]) for size in encoded.image_sizes]) + for image in encoded.images + ], [ + torch.tensor([list(image.shape[-2:])]) + for image in encoded.images + ]) processed.update({ - "pixel_values": - pixel_values, - "attention_mask": - torch.tensor([encoded.attention_mask]), - "image_sizes": - image_sizes, + "pixel_values": pixel_values, + "image_sizes": image_sizes, }) return processed @@ -695,7 +751,7 @@ def get_vocab_size(self) -> int: """Return the vocab size of the model.""" # Unlike some other VLMs, mistral3's vocab size is stored in its `text_config`, not the top-level # config. - return self.config.text_config.vocab_size + return self.config.get_text_config().vocab_size def get_mm_token_ids(self) -> torch.Tensor: return torch.tensor([ @@ -723,8 +779,15 @@ class MistralNativeInputProcessor(MistralPixtralEncoderContract, Unlike the HF frontend, this one serves *every* mistral-native checkpoint (text-only, MoE, mamba, audio and vision), which is why the inherited encoder contract self-gates on the presence of a vision encoder. + + The server renders a chat to token ids with mistral-common, image grids + included (``tensorrt_llm.inputs.utils.apply_chat_template``); token ids + with images are checked against the grids mistral-common makes of the + images. """ + supports_token_id_mm_expansion = True + def __init__( self, model_path: str, @@ -769,7 +832,12 @@ def dtype(self) -> torch.dtype: return self._dtype def get_vocab_size(self) -> int: - return self.config.text_config.vocab_size + return self.config.get_text_config().vocab_size + + def get_preferred_media_io_kwargs(self) -> Dict[str, Dict[str, Any]]: + # mistral-common encodes PIL images, so the decoded image is passed on + # as loaded; other formats are converted back to PIL first. + return {"image": {"format": "pil"}} def get_mm_token_ids(self) -> torch.Tensor: return torch.tensor([ @@ -784,25 +852,100 @@ def get_mm_special_token_ids(self) -> torch.Tensor: self.processor.image_end_token_id, ]) + def _image_blocks(self, token_ids): + """Locate complete native image grids, including their row delimiters.""" + blocks = [] + begin = None + for index, token in enumerate(token_ids): + if token == self.processor.image_token_id: + if begin is None: + begin = index + elif token == self.processor.image_end_id: + if begin is None: + raise ValueError("Image end token without an image grid") + blocks.append((begin, index + 1)) + begin = None + elif token == self.processor.image_break_id: + if begin is None: + raise ValueError( + "Image row delimiter without an image grid") + elif begin is not None: + raise ValueError("Text token inside an incomplete image grid") + if begin is not None: + raise ValueError("Image grid is missing its end token") + return blocks + + @staticmethod + def _image_payload(processed): + processed.pop("input_ids") + processed.pop("attention_mask", None) + processed["image_sizes"] = processed["image_sizes"].tolist() + return {"multimodal_data": {"image": processed}} + + def requires_input_placeholders(self, *, tokenized: bool = False) -> bool: + """No caller-side check: this processor checks the images against the prompt. + + Text places the images at its ``[IMG]`` markers, or after itself without + markers; token ids carry a whole image grid per image. + """ + return False + + @torch.inference_mode() + def call_with_token_ids(self, inputs, sampling_params): + # Keep native chat/control tokens intact; decoding and re-encoding the + # prompt would turn some control tokens into ordinary text. + token_ids = list(inputs["prompt_token_ids"]) + images = self.processor.image_list( + inputs.get("multi_modal_data", {}).get("image")) + if not images: + if self._has_vision_encoder() and not set( + self.get_mm_token_ids().tolist()).isdisjoint(token_ids): + raise ValueError("Image tokens require image data") + return token_ids, None + processed = self.processor(text="", images=images) + encoded = processed["input_ids"][0].tolist() + expected = [encoded[a:b] for a, b in self._image_blocks(encoded)] + if len(expected) != len(images): + raise ValueError( + "Native tokenizer image grid count does not match images") + delimiters = (self.processor.image_break_id, + self.processor.image_end_id) + if (getattr(self._config, "model_type", None) == "mistral_large_4" + and not any(token in delimiters for token in token_ids)): + # Mistral Large 4 takes one [IMG] per image, as a renderer that + # leaves the grids to the server writes it: each expands to its + # image's grid. + if token_ids.count(self.processor.image_token_id) != len(images): + raise ValueError( + "Image placeholder count does not match images") + grids = iter(expected) + expanded = [] + for token in token_ids: + if token == self.processor.image_token_id: + expanded.extend(next(grids)) + else: + expanded.append(token) + return expanded, self._image_payload(processed) + actual = [token_ids[a:b] for a, b in self._image_blocks(token_ids)] + if actual != expected: + raise ValueError( + "Image token grids do not match image sizes and order") + return token_ids, self._image_payload(processed) + @torch.inference_mode() def call_with_text_prompt( self, inputs: TextPrompt, sampling_params: SamplingParams ) -> Tuple[List[int], ExtraProcessedInputs | None]: - images = inputs.get("multi_modal_data", {}).get("image") + if inputs.get("prompt_token_ids") is not None: + return self.call_with_token_ids(inputs, sampling_params) + images = self.processor.image_list( + inputs.get("multi_modal_data", {}).get("image")) if not images: - # Text-only: tokenize directly without wrapping in a chat template. - # The chat template is either already applied by the caller (serve - # path) or intentionally absent (e.g. raw few-shot eval like MMLU). - input_ids = self.tokenizer.transformers_tokenizer.encode( - inputs["prompt"]) - return input_ids, None - # Multimodal: MistralCommonImageProcessor builds the full conversation - # and applies the mistral-common chat template with image tokens. + return self.tokenizer.transformers_tokenizer.encode( + inputs["prompt"]), None processed = self.processor(text=inputs["prompt"], images=images) - input_ids = processed.pop("input_ids").tolist()[0] - processed.pop("attention_mask", None) - processed["image_sizes"] = processed["image_sizes"].tolist() - return input_ids, {"multimodal_data": {"image": {**processed}}} + input_ids = processed["input_ids"][0].tolist() + return input_ids, self._image_payload(processed) # Register the native processor's content-format metadata. We do this @@ -811,12 +954,22 @@ def call_with_text_prompt( # overwrite the MistralHFInputProcessor entry for Mistral3VLM. # Mistral is the only supported case where HF preprocessor can be used with # non-HF checkpoints, so this hack is preferred to changing the registry itself. +# mistral-common renders the chat from the messages with their image parts in +# place (OPENAI content), not from text with placeholders. MULTIMODAL_PLACEHOLDER_REGISTRY.set_placeholder_metadata( "mistral_common", MultimodalPlaceholderMetadata( placeholder_map={"image": "[IMG]"}, placeholder_placement=MultimodalPlaceholderPlacement.BEFORE_TEXT, - content_format=ContentFormat.PASSTHROUGH, + content_format=ContentFormat.OPENAI, + ), + registrant_module=__name__) +MULTIMODAL_PLACEHOLDER_REGISTRY.set_placeholder_metadata( + "mistral_large_4", + MultimodalPlaceholderMetadata( + placeholder_map={"image": "[IMG]"}, + placeholder_placement=MultimodalPlaceholderPlacement.BEFORE_TEXT, + content_format=ContentFormat.OPENAI, ), registrant_module=__name__) MistralNativeInputProcessor._registered_model_type = "mistral_common" diff --git a/tensorrt_llm/_torch/pyexecutor/_util.py b/tensorrt_llm/_torch/pyexecutor/_util.py index b77df9ef8fe1..25d4931a693b 100644 --- a/tensorrt_llm/_torch/pyexecutor/_util.py +++ b/tensorrt_llm/_torch/pyexecutor/_util.py @@ -561,7 +561,8 @@ def get_kv_cache_manager_cls( "QSA sparse attention currently requires a hybrid Mamba / " "linear-attention model.") return get_sparse_attn_kv_cache_manager(sparse_attn_config, - use_kv_cache_manager_v2=use_v2) + use_kv_cache_manager_v2=use_v2, + pretrained_config=config) else: return _non_hybrid_kv_cache_manager_cls(model_config, kv_cache_config) diff --git a/tensorrt_llm/_torch/pyexecutor/config_utils.py b/tensorrt_llm/_torch/pyexecutor/config_utils.py index 05f494bec37a..b186128f72ac 100644 --- a/tensorrt_llm/_torch/pyexecutor/config_utils.py +++ b/tensorrt_llm/_torch/pyexecutor/config_utils.py @@ -1097,6 +1097,20 @@ def build_minimax_dspark_draft_config( return config +def _require_native_format_for_mistral_large_4(model_name_or_path: str) -> None: + """Name the format a Mistral-Large-4 checkpoint needs. + + It ships Mistral-native files only (params.json, tekken.json, consolidated + safetensors), which checkpoint_format="mistral" reads. + """ + from tensorrt_llm._torch.models.checkpoints.mistral.config_loader import \ + is_mistral_large4_checkpoint + if is_mistral_large4_checkpoint(model_name_or_path): + raise ValueError( + f"{model_name_or_path} is a Mistral-Large-4 checkpoint in the " + "Mistral-native format; load it with checkpoint_format='mistral'") + + def load_pretrained_config(model_name_or_path: str, trust_remote_code: bool = False, checkpoint_format: CheckpointFormat | str @@ -1106,8 +1120,9 @@ def load_pretrained_config(model_name_or_path: str, CheckpointFormat.MISTRAL_LARGE_3): from tensorrt_llm._torch.models.checkpoints.mistral.config_loader import \ MistralConfigLoader - return MistralConfigLoader().load(model_name_or_path).pretrained_config + return MistralConfigLoader().load_pretrained_config(model_name_or_path) + _require_native_format_for_mistral_large_4(model_name_or_path) config_dict, _ = transformers.PretrainedConfig.get_config_dict( model_name_or_path, **kwargs) model_type = config_dict.get("model_type") @@ -1115,13 +1130,7 @@ def load_pretrained_config(model_name_or_path: str, model_kwargs = kwargs.get("model_kwargs") or {} requested_architectures = model_kwargs.get("architectures") or architectures - if checkpoint_format in (CheckpointFormat.MISTRAL, - CheckpointFormat.MISTRAL_LARGE_3): - from tensorrt_llm._torch.models.checkpoints.mistral.config_loader import \ - MistralConfigLoader - model_config = MistralConfigLoader().load( - model_name_or_path).pretrained_config - elif is_speculators_dspark_config(config_dict): + if is_speculators_dspark_config(config_dict): # A DSpark drafter in the vLLM speculators format: no top-level # model_type, and its auto_map needs the speculators package. Build # the backbone's config from the translation the spec config uses. diff --git a/tensorrt_llm/_torch/pyexecutor/grammar_matcher.py b/tensorrt_llm/_torch/pyexecutor/grammar_matcher.py index 3de8d6f2b40c..91c09a49be60 100644 --- a/tensorrt_llm/_torch/pyexecutor/grammar_matcher.py +++ b/tensorrt_llm/_torch/pyexecutor/grammar_matcher.py @@ -1,5 +1,8 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 import os from abc import ABC, abstractmethod +from typing import Callable, Optional import llguidance import llguidance.torch @@ -165,9 +168,30 @@ def _check_err(self) -> None: class LLGuidanceMatcherFactory(GrammarMatcherFactory): - def __init__(self, guided_decoding_config: GuidedDecodingConfig, - vocab_size_padded: int): + def __init__(self, + guided_decoding_config: GuidedDecodingConfig, + vocab_size_padded: int, + tokenizer: Optional[llguidance.LLTokenizer] = None, + json_schema_lark: Optional[Callable[[str], str]] = None): + """Build the factory's llguidance tokenizer. + + Args: + guided_decoding_config: The guided decoding config; its + `tokenizer_str` is the Hugging Face tokenizer JSON to build the + llguidance tokenizer from, unless `tokenizer` is given. + vocab_size_padded: The vocabulary size the token bitmask covers. + tokenizer: A ready llguidance tokenizer, for vocabularies with no + Hugging Face tokenizer JSON (Tekken). + json_schema_lark: Renders the Lark grammar of a JSON response from + its serialized schema, for models whose structured responses + may open with a reasoning section (Mistral-Large-4). Without + it, a JSON response is constrained from its first token. + """ super().__init__() + self._json_schema_lark = json_schema_lark + if tokenizer is not None: + self._tokenizer = tokenizer + return tokenizer_str = guided_decoding_config.tokenizer_str stop_token_ids = guided_decoding_config.stop_token_ids @@ -193,10 +217,9 @@ def create( grammar = None match guide_type: case GuidedDecodingParams.GuideType.JSON: - grammar = llguidance.LLMatcher.grammar_from_json_schema( - '{"type": "object"}') + grammar = self._json_grammar('{"type": "object"}') case GuidedDecodingParams.GuideType.JSON_SCHEMA: - grammar = llguidance.LLMatcher.grammar_from_json_schema(guide) + grammar = self._json_grammar(guide) case GuidedDecodingParams.GuideType.REGEX: grammar = llguidance.LLMatcher.grammar_from_regex(guide) case GuidedDecodingParams.GuideType.EBNF_GRAMMAR: @@ -212,3 +235,9 @@ def create( raise ValueError(f"LLGuidance matcher error: {matcher.get_error()}") return LLGuidanceMatcher(matcher, self._tokenizer.eos_token) + + def _json_grammar(self, json_schema: str) -> str: + if self._json_schema_lark is None: + return llguidance.LLMatcher.grammar_from_json_schema(json_schema) + return llguidance.LLMatcher.grammar_from_lark( + self._json_schema_lark(json_schema)) diff --git a/tensorrt_llm/_torch/pyexecutor/guided_decoder.py b/tensorrt_llm/_torch/pyexecutor/guided_decoder.py index 81f6049ace4b..b4fa9c32b416 100644 --- a/tensorrt_llm/_torch/pyexecutor/guided_decoder.py +++ b/tensorrt_llm/_torch/pyexecutor/guided_decoder.py @@ -4,8 +4,9 @@ import math from dataclasses import dataclass from queue import Empty, Queue -from typing import Iterable, List, Optional, Tuple +from typing import Callable, Iterable, List, Optional, Tuple +import llguidance import torch from tensorrt_llm.llmapi.llm_args import GuidedDecodingConfig @@ -169,7 +170,10 @@ def __init__(self, max_num_sequences: int, vocab_size_padded: int, max_num_draft_tokens: int = 0, - rank: int = 0): + rank: int = 0, + llguidance_tokenizer: Optional[llguidance.LLTokenizer] = None, + llguidance_json_schema_lark: Optional[Callable[[str], + str]] = None): self.guided_decoding_backend = guided_decoding_config.backend self.max_num_sequences = max_num_sequences self.vocab_size_padded = vocab_size_padded @@ -183,7 +187,10 @@ def __init__(self, max_num_draft_tokens=max_num_draft_tokens) elif self.guided_decoding_backend == GuidedDecodingConfig.GuidedDecodingBackend.LLGUIDANCE: self.grammar_matcher_factory = LLGuidanceMatcherFactory( - guided_decoding_config, vocab_size_padded) + guided_decoding_config, + vocab_size_padded, + tokenizer=llguidance_tokenizer, + json_schema_lark=llguidance_json_schema_lark) else: raise ValueError( f"Invalid guided decoding backend: {self.guided_decoding_backend}" @@ -528,12 +535,18 @@ def __init__(self, max_num_sequences: int, vocab_size_padded: int, max_num_draft_tokens: int = 0, - rank: int = 0): - super().__init__(guided_decoding_config=guided_decoding_config, - max_num_sequences=max_num_sequences, - vocab_size_padded=vocab_size_padded, - max_num_draft_tokens=max_num_draft_tokens, - rank=rank) + rank: int = 0, + llguidance_tokenizer: Optional[llguidance.LLTokenizer] = None, + llguidance_json_schema_lark: Optional[Callable[[str], + str]] = None): + super().__init__( + guided_decoding_config=guided_decoding_config, + max_num_sequences=max_num_sequences, + vocab_size_padded=vocab_size_padded, + max_num_draft_tokens=max_num_draft_tokens, + rank=rank, + llguidance_tokenizer=llguidance_tokenizer, + llguidance_json_schema_lark=llguidance_json_schema_lark) # self.requests should be accessed by normal host code; # self.requests_hostfunc should be accessed by hostfunc (CUDA callback). self.requests_hostfunc: Optional[GuidedRequests] = None diff --git a/tensorrt_llm/_torch/pyexecutor/kv_cache/kv_cache_manager_v2.py b/tensorrt_llm/_torch/pyexecutor/kv_cache/kv_cache_manager_v2.py index 6610cbed005b..fae4db5c1c89 100644 --- a/tensorrt_llm/_torch/pyexecutor/kv_cache/kv_cache_manager_v2.py +++ b/tensorrt_llm/_torch/pyexecutor/kv_cache/kv_cache_manager_v2.py @@ -3442,11 +3442,13 @@ def create_cold_page_codec(cache_config: object) -> Optional[object]: # layout representative for each role from the public pool descriptors. self.num_pools = len(self._layer_ids_by_group) self._pool_layer_ids_by_role: Dict[Tuple[int, DataRole], LayerId] = {} + self._buffer_roles: set[tuple[int, DataRole]] = set() for pool_group in self.impl.pool_group_descs: for variant in pool_group.slot_desc.variants: pool_id = int(variant.layer_group_id) for coalesced in variant.coalesced_buffers: for buffer_id in coalesced.buffer_ids: + self._buffer_roles.add((int(buffer_id.layer_id), buffer_id.role)) self._pool_layer_ids_by_role.setdefault( (pool_id, buffer_id.role), buffer_id.layer_id ) @@ -4114,6 +4116,7 @@ def _prepare_page_table_tensor(self, index_mapper_capacity: int) -> None: { self.impl.get_page_stride(layer_id, self._get_pool_roles(pool_id)[0]) for layer_id in layer_ids + if self.has_buffer(layer_id, self._get_pool_roles(pool_id)[0]) } ) > 1 @@ -6100,6 +6103,10 @@ def is_attention_layer(self, layer_idx: int) -> bool: self.kv_cache_manager_py_config.layers[layer_offset], AttentionLayerConfig ) + def has_buffer(self, local_layer_idx: int, role: DataRole) -> bool: + """Whether the local layer physically owns a buffer with this role.""" + return (int(local_layer_idx), role) in self._buffer_roles + def get_buffers( self, layer_idx: int, @@ -6107,8 +6114,13 @@ def get_buffers( *, mixed_precision_pool: Literal["low", "high"] = "low", ) -> Optional[torch.Tensor]: - """Return a zero-copy KV view; its slot dimension may be strided.""" + """Return a zero-copy KV view; its slot dimension may be strided. + + None for a layer that owns no KEY buffer. + """ layer_offset = self.layer_offsets[layer_idx] + if not self.has_buffer(layer_offset, Role.KEY): + return None if layer_offset in self.mixed_precision_high_layers: return self._get_mixed_precision_buffers(layer_offset, kv_layout, mixed_precision_pool) addr_key = self.impl.get_mem_pool_base_address(layer_offset, Role.KEY, PageIndexMode.SHARED) @@ -6443,6 +6455,7 @@ def get_num_free_blocks(self) -> int: [ self.impl.get_page_index_upper_bound(layer_id, Role.KEY) for layer_id in typed_range(LayerId(self.num_local_layers)) + if self.has_buffer(layer_id, Role.KEY) ] ) # Subtract the pages the reserved sequences hold, not the count of diff --git a/tensorrt_llm/_torch/pyexecutor/model_loader.py b/tensorrt_llm/_torch/pyexecutor/model_loader.py index 8b512a4446ea..b138afa7ba1a 100644 --- a/tensorrt_llm/_torch/pyexecutor/model_loader.py +++ b/tensorrt_llm/_torch/pyexecutor/model_loader.py @@ -974,6 +974,13 @@ def _copy_weights_into_model(model_load_weights: Callable[..., None], torch.cuda.synchronize() +def _model_reads_own_checkpoint(model: Any) -> bool: + """Whether the model defines `load_checkpoint(checkpoint_dir)` and reads its + checkpoint's files itself, so the checkpoint loader opens nothing and no + weight mapper applies.""" + return getattr(type(model), "load_checkpoint", None) is not None + + def load_model_from_auto_load_format( checkpoint_loader: BaseCheckpointLoader, checkpoint_dir: str, @@ -987,7 +994,7 @@ def load_model_from_auto_load_format( qualification: PostTransformQualificationDecision, metrics: dict[str, float], startup_metadata: _CheckpointStartupMetadata, -) -> tuple[BaseWeightMapper, bool]: +) -> tuple[Optional[BaseWeightMapper], bool]: """Load the target and any separate draft checkpoint for ``LoadFormat.AUTO``. The model is already built and has CUDA memory; post-load hooks run @@ -995,6 +1002,8 @@ def load_model_from_auto_load_format( Built-in and custom checkpoint loaders, including custom weight loaders, read the checkpoint here. + A model that defines ``load_checkpoint(checkpoint_dir)`` reads its target + checkpoint itself; the checkpoint loader opens none of its files. Args: checkpoint_loader: checkpoint loader of this load @@ -1011,8 +1020,9 @@ def load_model_from_auto_load_format( policy of the target and draft reads is recorded in place Returns: - The target weight mapper, and whether the checkpoint loader wrote - the target weights into the model directly (MX P2P). + The target weight mapper (None for a model that reads its checkpoint + itself), and whether the checkpoint loader wrote the target weights + into the model directly (MX P2P). """ # load target weights weights_checkpoint_dir = (model.llm_checkpoint_dir if hasattr( @@ -1026,20 +1036,31 @@ def load_model_from_auto_load_format( mx_loading.get_mx_load_weights_kwargs(model_config=model_config, llm_args=llm_args, qualification=qualification)) - weight_mapper = checkpoint_loader.get_initialized_weight_mapper( - model, model_config) - startup_metadata["checkpoint_io_policy"] = _load_checkpoint_into_model( - checkpoint_loader, - weights_checkpoint_dir, - model=model, - is_draft_model=False, - mapping=mapping, - weight_mapper=weight_mapper, - extra_load_weights_kwargs=target_load_weights_kwargs, - metrics=metrics) - # Read before the draft load: an MX draft read without MX state resets - # this flag. - weights_preloaded = checkpoint_loader.is_weights_preloaded() + if _model_reads_own_checkpoint(model): + weight_mapper = None + startup_metadata["checkpoint_io_policy"] = None + with timing_metric( + ModelLoaderMetricNames.WEIGHT_POPULATION_SECONDS.value, + metrics): + model.load_checkpoint(weights_checkpoint_dir) + if torch.cuda.is_available(): # CPU guard + torch.cuda.synchronize() + weights_preloaded = False + else: + weight_mapper = checkpoint_loader.get_initialized_weight_mapper( + model, model_config) + startup_metadata["checkpoint_io_policy"] = _load_checkpoint_into_model( + checkpoint_loader, + weights_checkpoint_dir, + model=model, + is_draft_model=False, + mapping=mapping, + weight_mapper=weight_mapper, + extra_load_weights_kwargs=target_load_weights_kwargs, + metrics=metrics) + # Read before the draft load: an MX draft read without MX state + # resets this flag. + weights_preloaded = checkpoint_loader.is_weights_preloaded() # load separate draft weights if spec_config is not None and spec_config.needs_separate_draft_weights: @@ -1590,11 +1611,8 @@ def _timed_load( self.spec_config.speculative_model) elif load_format == LoadFormat.DUMMY: - self.weight_mapper = checkpoint_loader.get_initialized_weight_mapper( - model, model_config) - initialize_dummy_weights(model) - if loads_draft_weights: - model.draft_model.load_weights_from_target_model(model) + self._initialize_dummy_weights(checkpoint_loader, model, + model_config, loads_draft_weights) elif load_format == LoadFormat.VISION_ONLY: # Vision weights are already loaded within the model. @@ -1774,6 +1792,16 @@ def cleanup(self) -> None: finally: self._checkpoint_loader = None + def _initialize_dummy_weights(self, checkpoint_loader, model, config, + loads_draft_weights: bool) -> None: + """Fill the model with dummy weights instead of reading a checkpoint.""" + if not _model_reads_own_checkpoint(model): + self.weight_mapper = checkpoint_loader.get_initialized_weight_mapper( + model, config) + initialize_dummy_weights(model) + if loads_draft_weights: + model.draft_model.load_weights_from_target_model(model) + def _load_and_validate_config( self, checkpoint_dir: str, checkpoint_loader: BaseCheckpointLoader) -> ModelConfig: diff --git a/tensorrt_llm/_torch/pyexecutor/py_executor_creator.py b/tensorrt_llm/_torch/pyexecutor/py_executor_creator.py index f2ea4985623b..2cc5bab4daa5 100644 --- a/tensorrt_llm/_torch/pyexecutor/py_executor_creator.py +++ b/tensorrt_llm/_torch/pyexecutor/py_executor_creator.py @@ -27,6 +27,7 @@ from tensorrt_llm._torch.pyexecutor.resource_manager import ( BaseResourceManager, ResourceManagerType) from tensorrt_llm._utils import get_sm_version +from tensorrt_llm.checkpoint_format import CheckpointFormat from tensorrt_llm.llmapi.llm_args import (CapacitySchedulerPolicy, ContextChunkingPolicy, ExecutorMemoryType, @@ -585,7 +586,26 @@ def _extend_full_attention_windows_for_spec_decode( def get_guided_decoding_config(guided_decoding_backend: str, tokenizer: Optional[TokenizerBase] = None): guided_decoding_config = None - if guided_decoding_backend == 'xgrammar': + if (guided_decoding_backend is not None and getattr( + type(tokenizer), "xgrammar_tokenizer_info", None) is not None): + # A tokenizer with no Hugging Face tokenizer JSON describes its own + # vocabulary (Mistral's Tekken: bytes, with separate control tokens). + # xgrammar takes it as the byte vocabulary; llguidance takes the + # tokenizer itself, which the guided decoder gets from + # `_llguidance_tokenizer`. + if guided_decoding_backend == 'xgrammar': + guided_decoding_config = GuidedDecodingConfig( + backend=GuidedDecodingConfig.GuidedDecodingBackend.XGRAMMAR, + **tokenizer.xgrammar_tokenizer_info()) + elif guided_decoding_backend == 'llguidance': + guided_decoding_config = GuidedDecodingConfig( + backend=GuidedDecodingConfig.GuidedDecodingBackend.LLGUIDANCE, + stop_token_ids=[tokenizer.eos_token_id]) + else: + raise ValueError( + f"Unsupported guided decoding backend {guided_decoding_backend}" + ) + elif guided_decoding_backend == 'xgrammar': assert tokenizer is not None guided_decoding_config = GuidedDecodingConfig( backend=GuidedDecodingConfig.GuidedDecodingBackend.XGRAMMAR, @@ -601,6 +621,66 @@ def get_guided_decoding_config(guided_decoding_backend: str, return guided_decoding_config +def _llguidance_tokenizer(guided_decoding_config: GuidedDecodingConfig, + tokenizer: Optional[TokenizerBase], + vocab_size_padded: int): + """The ready llguidance tokenizer of a vocabulary with no Hugging Face + tokenizer JSON (Mistral's Tekken), or None to build it from the config's + `tokenizer_str`.""" + if (guided_decoding_config.backend + != GuidedDecodingConfig.GuidedDecodingBackend.LLGUIDANCE + or getattr(type(tokenizer), "llguidance_tokenizer", None) is None): + return None + return tokenizer.llguidance_tokenizer(vocab_size_padded) + + +def _llguidance_json_schema_lark(guided_decoding_config: GuidedDecodingConfig, + tokenizer: Optional[TokenizerBase]): + """Renders the llguidance grammar of a JSON response for a tokenizer whose + structured responses may open with a reasoning section (Mistral's [THINK]), + or None to constrain a JSON response from its first token.""" + if (guided_decoding_config.backend + != GuidedDecodingConfig.GuidedDecodingBackend.LLGUIDANCE + or getattr(tokenizer, "renders_grammars", False) is not True): + return None + return tokenizer.llguidance_json_schema_lark + + +def _load_guided_decoding_tokenizer(llm_args: TorchLlmArgs, + checkpoint_dir: str) -> TokenizerBase: + """Load the tokenizer whose vocabulary guided decoding masks, from the checkpoint. + + A Mistral-native checkpoint (``checkpoint_format`` ``mistral`` or + ``mistral_large_3``) with a Tekken tokenizer (``tekken.json``) is read + through mistral-common, as its input processor reads it, whatever Hugging + Face tokenizer files it also ships; guided decoding supports Tekken + vocabularies only, so other checkpoints keep their Hugging Face tokenizer. + An explicit ``custom_tokenizer`` takes precedence. + + Args: + llm_args: The LLM arguments, for ``custom_tokenizer``, + ``checkpoint_format`` and ``trust_remote_code``. + checkpoint_dir: The checkpoint directory. + + Returns: + The loaded tokenizer. + """ + custom_tokenizer = llm_args.custom_tokenizer + if (custom_tokenizer is None and llm_args.checkpoint_format + in (CheckpointFormat.MISTRAL, CheckpointFormat.MISTRAL_LARGE_3) + and os.path.isfile(os.path.join(checkpoint_dir, "tekken.json"))): + custom_tokenizer = "mistral_common" + if custom_tokenizer: + from tensorrt_llm.tokenizer import load_custom_tokenizer + return load_custom_tokenizer( + custom_tokenizer, + checkpoint_dir, + trust_remote_code=llm_args.trust_remote_code) + from tensorrt_llm.tokenizer import TransformersTokenizer + return TransformersTokenizer.from_pretrained( + checkpoint_dir, trust_remote_code=llm_args.trust_remote_code) + + def _load_config_and_create_checkpoint_loader( llm_args: TorchLlmArgs, checkpoint_dir: Optional[str] = None @@ -771,19 +851,10 @@ def _create_py_executor( # when guided decoding needs it. if (tokenizer is None and llm_args.guided_decoding_backend is not None and checkpoint_dir is not None): + tokenizer = _load_guided_decoding_tokenizer(llm_args, checkpoint_dir) logger.info( - "Tokenizer not provided; loading from checkpoint for guided decoding" - ) - if llm_args.custom_tokenizer: - from tensorrt_llm.tokenizer import load_custom_tokenizer - tokenizer = load_custom_tokenizer( - llm_args.custom_tokenizer, - checkpoint_dir, - trust_remote_code=llm_args.trust_remote_code) - else: - from tensorrt_llm.tokenizer import TransformersTokenizer - tokenizer = TransformersTokenizer.from_pretrained( - checkpoint_dir, trust_remote_code=llm_args.trust_remote_code) + "Tokenizer not provided; loaded " + f"{type(tokenizer).__name__} from checkpoint for guided decoding") guided_decoding_config = get_guided_decoding_config( llm_args.guided_decoding_backend, tokenizer) @@ -1245,6 +1316,12 @@ def allocation_scope(current_stage: ExecutorMemoryType): "vocab_size_padded": model_engine.model.vocab_size_padded, "rank": mapping.rank, } + kwargs["llguidance_tokenizer"] = _llguidance_tokenizer( + guided_decoding_config, tokenizer, + model_engine.model.vocab_size_padded) + kwargs[ + "llguidance_json_schema_lark"] = _llguidance_json_schema_lark( + guided_decoding_config, tokenizer) if spec_config is not None: kwargs[ "max_num_draft_tokens"] = spec_config.tokens_per_gen_step - 1 diff --git a/tensorrt_llm/_torch/speculative/dflash.py b/tensorrt_llm/_torch/speculative/dflash.py index 3cffaa9a5a71..67b0610b19b5 100644 --- a/tensorrt_llm/_torch/speculative/dflash.py +++ b/tensorrt_llm/_torch/speculative/dflash.py @@ -532,6 +532,47 @@ def dflash_noise_block_embedding( return noise_embed_2d +def dflash_draft_block_embedding( + draft_model: nn.Module, + bonus: torch.Tensor, + mask_token_id: int, + block_size: int, + trained_mask_embedding: Optional[torch.Tensor] = None, +) -> torch.Tensor: + """The drafter's input block per request, [len(bonus), block_size, hidden]. + + A drafter that embeds its own block, for example with learned + position-specific mask embeddings, implements ``embed_draft_block(bonus, + block_size)``; the worker supplies only the accepted anchor tokens and the + block width. Any other drafter gets :func:`dflash_noise_block_embedding` + over its token embedding. + """ + embed_block = getattr(draft_model, "embed_draft_block", None) + if embed_block is not None: + return embed_block(bonus, block_size) + return dflash_noise_block_embedding( + draft_model.draft_model_full.model.embed_tokens, + bonus, + mask_token_id, + block_size, + trained_mask_embedding, + ) + + +def dflash_block_forward(draft_model: nn.Module, attn_metadata, **block_inputs) -> torch.Tensor: + """Run the drafter's block forward over this step's generation requests. + + A drafter whose block forward depends on the step's attention metadata, for + example one that selects a prefill or a decode implementation per step, + implements ``dflash_forward_with_metadata(*, attn_metadata, **block_inputs)``. + Any other drafter implements ``dflash_forward(**block_inputs)``. + """ + forward_with_metadata = getattr(draft_model, "dflash_forward_with_metadata", None) + if forward_with_metadata is None: + return draft_model.dflash_forward(**block_inputs) + return forward_with_metadata(attn_metadata=attn_metadata, **block_inputs) + + def dflash_position_ceiling(max_ctx: int, block_size: int, max_draft_len: int) -> int: """Positions a drafter that indexes absolute positions must be able to encode. @@ -870,6 +911,19 @@ class DFlashWorker(SpecWorkerBase): projected hidden states to a per-request buffer, giving the draft model the full history of target features. + The draft model supplies the model math through methods the worker calls: + + * ``project_target_hidden(hidden)`` projects captured target features + into context entries. A drafter without it drafts without target + context. + * ``embed_draft_block(bonus, block_size)`` (optional) builds the input + block itself; see :func:`dflash_draft_block_embedding`. Such a drafter + keeps the eager context preparation, because the fused paged-latent + preparation embeds the block from the token embedding. + * ``dflash_forward_with_metadata(*, attn_metadata, **block_inputs)`` + (optional) replaces ``dflash_forward`` for a block forward that depends + on the step's attention metadata; see :func:`dflash_block_forward`. + Reference: https://arxiv.org/pdf/2602.06036 """ @@ -1670,8 +1724,14 @@ def _lazy_init_ctx_buffers( # The fused preparation stores single-latent context entries # projected from the captured target features through fc and # hidden_norm (which every loaded DFlash drafter owns), as the - # eager path does; LiLiCorr drafters keep the eager path. - self._paged_latent_prepare = kv_factor == 1 and not draft_model._uses_lilicorr + # eager path does, and embeds the block from the drafter's token + # embedding. LiLiCorr drafters and drafters that embed their own + # block (embed_draft_block) keep the eager path. + self._paged_latent_prepare = ( + kv_factor == 1 + and getattr(draft_model, "embed_draft_block", None) is None + and not draft_model._uses_lilicorr + ) # Only a manager that publishes and matches its own blocks can # hand back a reused prefix's drafter K/V; the private arena and # an unpaired draft pool both start every request at 0. @@ -2149,7 +2209,7 @@ def _store_prefill_context( and store them per-request so the draft model can use the full prompt as cross-attention context on subsequent gen steps. """ - if not hasattr(draft_model, "fc") or not hasattr(draft_model, "hidden_norm"): + if not callable(getattr(draft_model, "project_target_hidden", None)): return num_ctx_tokens = attn_metadata.num_ctx_tokens @@ -2470,7 +2530,9 @@ def _forward_impl( if num_gens > 0: with self.draft_kv_cache_context(attn_metadata, draft_kv_cache_manager): - hidden_states_out = draft_model.dflash_forward( + hidden_states_out = dflash_block_forward( + draft_model, + attn_metadata, noise_embedding=inputs["noise_embedding"], query_positions=inputs["query_positions"], num_ctx_per_req=inputs["num_ctx_per_req"], @@ -2632,13 +2694,16 @@ def _refine_block_logits( gen_logits: torch.Tensor, inputs: dict, spec_metadata, + draft_hidden_states: Optional[torch.Tensor] = None, ) -> torch.Tensor: """Refine the block logits between the draft forward and sampling. Plain DFlash proposes the backbone's logits unchanged. Drafter families carrying extra heads override this — see :meth:`DSparkWorker._refine_block_logits` for the Markov - intra-block bias. + intra-block bias. ``draft_hidden_states`` is the ``[num_gens, K, + hidden]`` block output the logits were projected from, for heads that + condition on it. """ return gen_logits @@ -2661,7 +2726,13 @@ def _sample_block_draft_tokens( draft overrides this (``DSparkWorker``). """ num_gens, K = gen_logits.shape[:2] - gen_logits = self._refine_block_logits(draft_model, gen_logits, inputs, spec_metadata) + gen_logits = self._refine_block_logits( + draft_model, + gen_logits, + inputs, + spec_metadata, + None if gen_hidden_states is None else gen_hidden_states.reshape(num_gens, K, -1), + ) # DFlash 2: replace the independent per-position picks with one # coherent path through the block (absent for plain DFlash). @@ -2845,8 +2916,6 @@ def prepare_1st_drafter_inputs( ) mask_token_id = self._resolved_mask_token_id - # Get the embed_tokens layer from the draft model - embed_tokens = draft_model.draft_model_full.model.embed_tokens hidden_dim = ( spec_metadata.hidden_size if spec_metadata.hidden_size > 0 else hidden_states.shape[-1] ) @@ -2879,10 +2948,8 @@ def prepare_1st_drafter_inputs( # Get captured multi-layer hidden states from spec_metadata captured_hs = spec_metadata.get_hidden_states(total_target_tokens) - has_target_features = ( - captured_hs is not None - and hasattr(draft_model, "fc") - and hasattr(draft_model, "hidden_norm") + has_target_features = captured_hs is not None and callable( + getattr(draft_model, "project_target_hidden", None) ) # Use cached block_size (resolved once on first call) @@ -2913,8 +2980,8 @@ def prepare_1st_drafter_inputs( query_position_ids = ctx_len_now.unsqueeze(1) + j_block.unsqueeze(0) ctx_position_ids = ctx_len_gen.unsqueeze(1) + offsets_kp1.unsqueeze(0) - noise_embed_2d = dflash_noise_block_embedding( - embed_tokens, bonus, mask_token_id, query_tokens_per_req, trained_mask_embedding + noise_embed_2d = dflash_draft_block_embedding( + draft_model, bonus, mask_token_id, query_tokens_per_req, trained_mask_embedding ) # Accumulate new accepted features into context buffers diff --git a/tensorrt_llm/_torch/speculative/dspark.py b/tensorrt_llm/_torch/speculative/dspark.py index 3805514fe019..4da068516ab2 100644 --- a/tensorrt_llm/_torch/speculative/dspark.py +++ b/tensorrt_llm/_torch/speculative/dspark.py @@ -1948,7 +1948,11 @@ class DSparkWorker(DFlashWorker): drafter therefore reuses this class unchanged; its differences (fused-QKV assumptions, a 576-latent K/V layout) land in its own draft-model subclass. Naming workers by backbone would produce N classes with - identical bodies. + identical bodies. A drafter whose Markov head is recurrent over the draft + hidden states sets ``has_recurrent_markov_head``: its + ``apply_markov_chain_logits`` then receives the block's draft hidden states + as ``hidden_states`` and chains the bias over the full-vocab block logits + itself (:meth:`_refine_block_logits`). Deployment form is the axis the runtime state actually splits on: paged draft K/V here, a worker-owned rolling window in @@ -2028,7 +2032,14 @@ def _draft_slot_ids( ) def _vocab_parallel_draft_tokens(self, draft_model) -> Optional[VocabParallelDraftTokens]: - """A drafter with a Markov head chains the greedy block through it.""" + """A drafter with a Markov head chains the greedy block through it. + + The shard chain computes the vanilla Markov head only, so a drafter with + a recurrent head keeps the gathered block logits, which + :meth:`_refine_block_logits` passes to the drafter's own chain. + """ + if getattr(draft_model, "has_recurrent_markov_head", False): + return None draft_tokens = super()._vocab_parallel_draft_tokens(draft_model) if draft_tokens is None or not getattr(draft_model, "has_markov_head", False): return draft_tokens @@ -2063,10 +2074,12 @@ def _sample_block_draft_tokens( the sampling filters (the ``NO_TOPK_NO_TOPP`` sampling mode) drafts take the fused :func:`markov_sampled_chain`; other sampled drafts take a per-position sampler loop and only with ``use_ancestral_draft_sampling``. Otherwise, - without a Markov head, for argmax drafts, or for a drafter that also - carries a DFlash 2 candidate scorer (LiLiCorr or the selector), the DFlash path - (greedy-chained :meth:`_refine_block_logits`, then the scorer and the - block sampler) applies. + without a Markov head, for argmax drafts, for a drafter that also + carries a DFlash 2 candidate scorer (LiLiCorr or the selector), or for a + recurrent Markov head (``has_recurrent_markov_head``), which the chains + here do not compute, the DFlash path (greedy-chained + :meth:`_refine_block_logits`, then the scorer and the block sampler) + applies. """ has_candidate_scorer = getattr(draft_model, "lilicorr", None) is not None or getattr( draft_model, "has_candidate_selector", False @@ -2082,6 +2095,7 @@ def _sample_block_draft_tokens( and getattr(draft_model, "has_markov_head", False) and spec_metadata.wants_advanced_draft_sampling and not has_candidate_scorer + and not getattr(draft_model, "has_recurrent_markov_head", False) ): return super()._sample_block_draft_tokens( draft_model, @@ -2182,15 +2196,24 @@ def _refine_block_logits( gen_logits: torch.Tensor, inputs: dict, spec_metadata, + draft_hidden_states: Optional[torch.Tensor] = None, ) -> torch.Tensor: """Add the greedy-chained Markov intra-block bias to the block logits. A DSpark drafter checkpoint may omit the Markov head (``markov_rank`` 0), which loads as a drafter without one; that case falls through to - the unmodified backbone logits. + the unmodified backbone logits. A drafter with a recurrent Markov head + (``has_recurrent_markov_head``) chains its own bias over the full-vocab + block logits, conditioned on ``draft_hidden_states``. """ if not getattr(draft_model, "has_markov_head", False): return gen_logits + if getattr(draft_model, "has_recurrent_markov_head", False): + # The recurrent head's state reads each position's draft hidden + # state as well, so it is not the shared vanilla chain. + return draft_model.apply_markov_chain_logits( + gen_logits, inputs["first_prev_tokens"], hidden_states=draft_hidden_states + ) return self._apply_dspark_markov_bias(draft_model, gen_logits, inputs["first_prev_tokens"]) def _apply_dspark_markov_bias( diff --git a/tensorrt_llm/bench/benchmark/__init__.py b/tensorrt_llm/bench/benchmark/__init__.py index fc59fe7c6ed8..b6b47cda5d10 100644 --- a/tensorrt_llm/bench/benchmark/__init__.py +++ b/tensorrt_llm/bench/benchmark/__init__.py @@ -1,16 +1,23 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + import json from pathlib import Path from typing import Callable, Dict, Optional, Set +import yaml from pydantic import AliasChoices, BaseModel, Field from tensorrt_llm import LLM as PyTorchLLM +from tensorrt_llm._torch.pyexecutor.config_utils import load_pretrained_config from tensorrt_llm.bench.benchmark.utils.processes import IterationWriter from tensorrt_llm.bench.dataclasses.configuration import RuntimeConfig from tensorrt_llm.bench.dataclasses.general import BenchmarkEnvironment from tensorrt_llm.bench.tuning.settings import get_model_config +from tensorrt_llm.checkpoint_format import CheckpointFormat from tensorrt_llm.commands.utils import \ collect_explicit_cli_keys as _collect_explicit_cli_keys +from tensorrt_llm.llmapi.utils import download_hf_model from tensorrt_llm.logger import logger # Map trtllm-bench Click parameter names to the LlmArgs field name (or @@ -49,6 +56,9 @@ class GeneralExecSettings(BaseModel): beam_width: int = Field(default=1, description="Number of search beams") model_path: Optional[Path] = Field(default=None, description="Path to model checkpoint") + checkpoint_format: Optional[str] = Field( + default=None, + description="Checkpoint format that extra_llm_api_options names") concurrency: int = Field( default=-1, description="Desired concurrency rate, <=0 for no limit") dataset_path: Optional[Path] = Field(default=None, @@ -93,6 +103,14 @@ def iteration_writer(self) -> IterationWriter: @property def model_type(self) -> str: + # A Mistral-native checkpoint keeps its configuration in params.json, which + # the Hugging Face config lookup does not read. + if self.checkpoint_format in (CheckpointFormat.MISTRAL, + CheckpointFormat.MISTRAL_LARGE_3): + return load_pretrained_config( + self.checkpoint_path, + trust_remote_code=True, + checkpoint_format=self.checkpoint_format).model_type return get_model_config(self.model, self.checkpoint_path).model_type @property @@ -126,6 +144,10 @@ def get_general_cli_options( params: Dict, bench_env: BenchmarkEnvironment) -> GeneralExecSettings: """Get general execution settings from command line parameters. + A Mistral-native checkpoint is read from its local files, as the LLM API reads + it. Without --model_path, its model name is therefore downloaded here, and + `bench_env.checkpoint_path` is set to the snapshot directory. + Args: params: Dictionary of command line parameters. bench_env: Benchmark environment containing model and checkpoint information. @@ -145,6 +167,21 @@ def get_general_cli_options( "model": model, }) + # The checkpoint format extra_llm_api_options names decides how the model's + # config is read. + extra_llm_api_options = params.get("extra_llm_api_options") + if extra_llm_api_options: + with open(extra_llm_api_options) as f: + llm_args_dict = yaml.safe_load(f) + if isinstance(llm_args_dict, dict): + settings_dict["checkpoint_format"] = llm_args_dict.get( + "checkpoint_format") + + if model_path is None and settings_dict.get("checkpoint_format") in ( + CheckpointFormat.MISTRAL, CheckpointFormat.MISTRAL_LARGE_3): + bench_env.checkpoint_path = download_hf_model(model, bench_env.revision) + settings_dict["model_path"] = bench_env.checkpoint_path + # Create and return the settings object, ignoring any extra fields return GeneralExecSettings(**settings_dict) diff --git a/tensorrt_llm/bench/benchmark/utils/general.py b/tensorrt_llm/bench/benchmark/utils/general.py index d2a133a9be40..f0eaa6c83bae 100755 --- a/tensorrt_llm/bench/benchmark/utils/general.py +++ b/tensorrt_llm/bench/benchmark/utils/general.py @@ -1,3 +1,6 @@ +# SPDX-FileCopyrightText: Copyright (c) 2024-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + from __future__ import annotations from pathlib import Path @@ -14,7 +17,9 @@ from tensorrt_llm.bench.tuning.dataclasses import (NemotronHybridConfig, Qwen3HybridConfig) from tensorrt_llm.bench.tuning.settings import (get_benchmark_engine_settings, - get_model_config) + get_model_config, + load_runtime_model_config) +from tensorrt_llm.checkpoint_format import CheckpointFormat from tensorrt_llm.logger import logger from tensorrt_llm.quantization.mode import QuantAlgo from tensorrt_llm.version import __version__ @@ -27,6 +32,42 @@ ALL_SUPPORTED_BACKENDS = ["pytorch"] +def _tuning_model_configs(model: str, model_path: Union[Path, None], + checkpoint_format: Union[str, None]): + """Read the model configurations the tuning heuristic needs. + + A Mistral-native checkpoint is read the way the runtime loads it: params.json + and the consolidated safetensors files. When those cannot be read, its + config.json is read instead. + + Returns: + The benchmark model configuration and the runtime model configuration. + + Raises: + ValueError: When a local Mistral-native checkpoint can be read neither + from its native files nor from a config.json. + """ + checkpoint = model_path or model + if checkpoint_format in (CheckpointFormat.MISTRAL, + CheckpointFormat.MISTRAL_LARGE_3): + try: + return (get_model_config(model, model_path, checkpoint_format), + load_runtime_model_config(checkpoint, checkpoint_format)) + except (ValueError, RuntimeError) as e: + if (Path(checkpoint).is_dir() + and not (Path(checkpoint) / "config.json").is_file()): + raise ValueError( + f"checkpoint_format={checkpoint_format}: the tuning heuristic " + f"cannot read {checkpoint} ({e}), and it has no config.json; " + "pass --max_batch_size and --max_num_tokens") from e + logger.warning( + f"checkpoint_format={checkpoint_format}: the tuning heuristic " + f"cannot read the native files of {checkpoint} ({e}); reading " + "its config.json instead") + model_config = get_model_config(model, model_path) + return model_config, load_runtime_model_config(checkpoint) + + def get_settings(params: dict, dataset_metadata: DatasetMetadata, model: str, model_path: Union[Path, None]) -> Dict[str, Union[str, int]]: """Retrieve basic runtime config for pytorch backend path @@ -41,6 +82,7 @@ def get_settings(params: dict, dataset_metadata: DatasetMetadata, model: str, extra_llm_api_options = params.get("extra_llm_api_options") enable_chunked_prefill = params.get("enable_chunked_prefill", False) enable_attention_dp = False + checkpoint_format = None kv_cache_dtype = "auto" mamba_ssm_cache_dtype = params.get("mamba_ssm_cache_dtype", "auto") @@ -62,6 +104,7 @@ def get_settings(params: dict, dataset_metadata: DatasetMetadata, model: str, enable_chunked_prefill = llm_args_dict.get("enable_chunked_prefill", enable_chunked_prefill) enable_attention_dp = llm_args_dict.get("enable_attention_dp", False) + checkpoint_format = llm_args_dict.get("checkpoint_format") mapping = { "pp_size": params.get("pp"), @@ -81,18 +124,14 @@ def get_settings(params: dict, dataset_metadata: DatasetMetadata, model: str, f"Initial settings: max_batch_size={max_batch_size} (user-provided), " f"max_num_tokens={max_num_tokens} (user-provided).") else: - model_config = get_model_config(model, model_path) + model_config, tllm_model_config = _tuning_model_configs( + model, model_path, checkpoint_format) if isinstance(model_config, (NemotronHybridConfig, Qwen3HybridConfig )) and mamba_ssm_cache_dtype not in (None, "auto"): model_config.set_mamba_ssm_cache_dtype(mamba_ssm_cache_dtype) - from tensorrt_llm._torch.model_config import ModelConfig - model = model_path or model - tllm_model_config = ModelConfig.from_pretrained(model, - trust_remote_code=True) - if (kv_cache_dtype is None and tllm_model_config.quant_config.kv_cache_quant_algo is None): kv_cache_dtype = _KV_CACHE_MAP.get( diff --git a/tensorrt_llm/bench/dataclasses/reporting.py b/tensorrt_llm/bench/dataclasses/reporting.py index 0f3e6954d96d..87288e6839df 100755 --- a/tensorrt_llm/bench/dataclasses/reporting.py +++ b/tensorrt_llm/bench/dataclasses/reporting.py @@ -467,12 +467,13 @@ def get_statistics_dict(self) -> Dict[str, Any]: if kv_cache_mem_percent is not None else None # Engine/Backend details - from tensorrt_llm._torch.model_config import ModelConfig from tensorrt_llm._utils import torch_dtype_to_str + from tensorrt_llm.bench.tuning.settings import load_runtime_model_config + # Read the configuration in the checkpoint format the LLM loaded. model = self.rt_cfg.model_path or self.rt_cfg.model - model_config = ModelConfig.from_pretrained(model, - trust_remote_code=True) + model_config = load_runtime_model_config( + model, self.kwargs.get("checkpoint_format")) validate_and_set_kv_cache_quant(model_config, kv_cache_dtype) diff --git a/tensorrt_llm/bench/tuning/dataclasses.py b/tensorrt_llm/bench/tuning/dataclasses.py index fc0283e1d835..be0f2aa3dd09 100755 --- a/tensorrt_llm/bench/tuning/dataclasses.py +++ b/tensorrt_llm/bench/tuning/dataclasses.py @@ -1,3 +1,6 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + import json import os import struct @@ -55,6 +58,10 @@ "F8_E4M3": 1, } +# Weight files of a Mistral-native checkpoint, next to its params.json. +CONSOLIDATED_SAFETENSORS_SINGLE_FILE = "consolidated.safetensors" +CONSOLIDATED_SAFETENSORS_INDEX_FILE = "consolidated.safetensors.index.json" + def parse_safetensors_file_metadata(model_path, filename): with open(os.path.join(model_path, filename), "rb") as f: @@ -98,24 +105,34 @@ def parse_safetensors_file_metadata(model_path, filename): ) from e -def get_safetensors_metadata(model_name_or_path): - """Read the safetensors metadata from HF model.""" +def get_safetensors_metadata(model_name_or_path, consolidated: bool = False): + """Read the safetensors metadata from HF model. + + Args: + model_name_or_path: Hugging Face model name or local checkpoint directory. + consolidated: Read the consolidated*.safetensors files of a local Mistral-native + checkpoint instead of the model*.safetensors files. + """ + single_file, index_file = ( + (CONSOLIDATED_SAFETENSORS_SINGLE_FILE, CONSOLIDATED_SAFETENSORS_INDEX_FILE) + if consolidated + else (SAFETENSORS_SINGLE_FILE, SAFETENSORS_INDEX_FILE) + ) if os.path.isdir(model_name_or_path): - if os.path.exists(os.path.join(model_name_or_path, SAFETENSORS_SINGLE_FILE)): + if os.path.exists(os.path.join(model_name_or_path, single_file)): file_metadata = parse_safetensors_file_metadata( - model_path=model_name_or_path, filename=SAFETENSORS_SINGLE_FILE + model_path=model_name_or_path, filename=single_file ) return SafetensorsRepoMetadata( metadata=None, sharded=False, weight_map={ - tensor_name: SAFETENSORS_SINGLE_FILE - for tensor_name in file_metadata.tensors.keys() + tensor_name: single_file for tensor_name in file_metadata.tensors.keys() }, - files_metadata={SAFETENSORS_SINGLE_FILE: file_metadata}, + files_metadata={single_file: file_metadata}, ) - elif os.path.exists(os.path.join(model_name_or_path, SAFETENSORS_INDEX_FILE)): - with open(os.path.join(model_name_or_path, SAFETENSORS_INDEX_FILE)) as f: + elif os.path.exists(os.path.join(model_name_or_path, index_file)): + with open(os.path.join(model_name_or_path, index_file)) as f: index = json.load(f) weight_map = index.get("weight_map", {}) @@ -145,8 +162,13 @@ def _parse(filename: str) -> None: # Not a safetensors repo raise RuntimeError( f"'{model_name_or_path}' is not a safetensors repo. Couldn't find " - f"'{SAFETENSORS_INDEX_FILE}' or '{SAFETENSORS_SINGLE_FILE}' files." + f"'{index_file}' or '{single_file}' files." ) + elif consolidated: + raise RuntimeError( + f"'{model_name_or_path}' is not a local directory; consolidated safetensors " + "files are read from local checkpoints only." + ) else: return huggingface_hub.get_safetensors_metadata(model_name_or_path) @@ -266,9 +288,15 @@ def is_mla(self) -> bool: return bool(self.kv_lora_rank and self.qk_rope_head_dim) @classmethod - def get_param_count_and_checkpoint_size(cls, model_hf_name, hf_model_path): + def get_param_count_and_checkpoint_size(cls, model_hf_name, hf_model_path, consolidated=False): """Read parameter count and checkpoint size from safetensors metadata. + Args: + model_hf_name: Hugging Face model name. + hf_model_path: Local checkpoint directory, or None to read model_hf_name. + consolidated: Read the consolidated*.safetensors files of a Mistral-native + checkpoint. + Returns: Tuple[int, float]: (param_count, checkpoint_size_in_gb). """ @@ -282,7 +310,7 @@ def get_param_count_and_checkpoint_size(cls, model_hf_name, hf_model_path): checkpoint_size_in_gb = param_count / (1024**3) else: model_name_or_path = hf_model_path or model_hf_name - metadata = get_safetensors_metadata(model_name_or_path) + metadata = get_safetensors_metadata(model_name_or_path, consolidated=consolidated) param_count = sum(metadata.parameter_count.values()) # Compute actual checkpoint size using per-dtype byte sizes. # This is critical for quantized models (e.g. NVFP4) where @@ -317,6 +345,34 @@ def from_hf(cls, model_hf_name, hf_model_path): **hf_config, ) + @classmethod + def from_mistral(cls, model_name, model_path, checkpoint_format): + """Read a Mistral-native checkpoint the way the runtime loads it. + + The configuration comes from params.json through the Mistral config loader, and the + parameter count and checkpoint size from the consolidated safetensors files. + + Args: + model_name: Model name. + model_path: Local checkpoint directory, or None to read model_name. + checkpoint_format: The Mistral-native checkpoint format. + """ + pretrained_config = load_pretrained_config( + model_path or model_name, checkpoint_format=checkpoint_format + ) + config = pretrained_config.to_dict() + # to_dict() writes the model_type of the config class, not the one the loader sets. + config["model_type"] = pretrained_config.model_type + param_count, checkpoint_size_in_gb = cls.get_param_count_and_checkpoint_size( + model_name, model_path, consolidated=True + ) + return cls( + name=model_name, + param_count=param_count, + checkpoint_size_in_gb=checkpoint_size_in_gb, + **config, + ) + def extra_model_cache_in_gb(self, bytes_per_elem, target_seq_len=None): return 0 diff --git a/tensorrt_llm/bench/tuning/settings.py b/tensorrt_llm/bench/tuning/settings.py index 9f47292975ac..84e2161487d1 100644 --- a/tensorrt_llm/bench/tuning/settings.py +++ b/tensorrt_llm/bench/tuning/settings.py @@ -1,3 +1,6 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + from __future__ import annotations from pathlib import Path @@ -14,6 +17,7 @@ Qwen3HybridConfig, ) from tensorrt_llm.bench.tuning.heuristics import calc_engine_setting +from tensorrt_llm.checkpoint_format import CheckpointFormat from tensorrt_llm.llmapi.llm_args import TorchLlmArgs from tensorrt_llm.llmapi.llm_utils import QuantConfig from tensorrt_llm.logger import logger @@ -89,16 +93,45 @@ def get_benchmark_engine_settings( return max_batch_size, max_num_tokens -def get_model_config(model_name: str, model_path: Path = None) -> ModelConfig: +def load_runtime_model_config(checkpoint: str | Path, checkpoint_format: str | None = None): + """Read a checkpoint's runtime model configuration with the LLM API's config loader. + + The loader is the one the LLM API registers for `checkpoint_format`: a Mistral-native + format reads params.json, the Hugging Face format config.json. + + Args: + checkpoint (str | Path): Checkpoint directory or Hugging Face model name. + checkpoint_format (str | None): Checkpoint format the LLM API loads; None is the + Hugging Face format, the LLM API's default. + + Returns: + tensorrt_llm._torch.model_config.ModelConfig: The runtime model configuration. + """ + # Importing the checkpoints package registers the built-in config loaders. + import tensorrt_llm._torch.models.checkpoints # noqa: F401 + from tensorrt_llm._torch.models.modeling_utils import get_config_loader + + config_loader = get_config_loader(checkpoint_format or CheckpointFormat.HF)() + return config_loader.load(str(checkpoint), trust_remote_code=True) + + +def get_model_config( + model_name: str, model_path: Path = None, checkpoint_format: str | None = None +) -> ModelConfig: """Obtain the model-related parameters from Hugging Face. Args: model_name (str): Huggingface model name. model_path (Path): Path to a local Huggingface checkpoint. + checkpoint_format (str | None): Checkpoint format the LLM API loads. A + Mistral-native format reads params.json and the consolidated + safetensors files instead of the Hugging Face files. Raises: ValueError: When model is not supported. """ + if checkpoint_format in (CheckpointFormat.MISTRAL, CheckpointFormat.MISTRAL_LARGE_3): + return ModelConfig.from_mistral(model_name, model_path, checkpoint_format) pretrained_config = load_pretrained_config(model_path or model_name, trust_remote_code=True) if is_nemotron_hybrid(pretrained_config): return NemotronHybridConfig.from_hf(model_name, model_path) diff --git a/tensorrt_llm/evaluate/interface.py b/tensorrt_llm/evaluate/interface.py index c27ba88d6327..3c95aefc467d 100644 --- a/tensorrt_llm/evaluate/interface.py +++ b/tensorrt_llm/evaluate/interface.py @@ -176,8 +176,14 @@ def dump_inference_results(output_dir: str, results: List[dict], for task_id, result in enumerate(results): output_ids = result.outputs[0].token_ids output_text = result.outputs[0].text.strip() - input_text = result.prompt.strip() - input_ids = tokenizer.encode(input_text) + if result.prompt is None: + # A token-id prompt has no text: dump the ids the engine received, + # which keep native chat control tokens that text cannot. + input_ids = result.prompt_token_ids + input_text = tokenizer.decode(input_ids).strip() + else: + input_text = result.prompt.strip() + input_ids = tokenizer.encode(input_text) results_list.append({ "task_id": task_id, "input_ids": input_ids, diff --git a/tensorrt_llm/evaluate/lm_eval.py b/tensorrt_llm/evaluate/lm_eval.py index bf1aa8f5a64b..05721f90e7c0 100644 --- a/tensorrt_llm/evaluate/lm_eval.py +++ b/tensorrt_llm/evaluate/lm_eval.py @@ -42,10 +42,11 @@ add_multimodal_placeholders, convert_image_mode) from ..inputs.content_format import ContentFormat from ..inputs.registry import MULTIMODAL_PLACEHOLDER_REGISTRY -from ..inputs.utils import _resolve_content_format +from ..inputs.utils import NATIVE_MISTRAL_MODEL_TYPES, _resolve_content_format from ..inputs.utils import apply_chat_template as trtllm_apply_chat_template from ..inputs.utils import interleave_mm_placeholders, resolve_hf_chat_template from ..llmapi import RequestOutput +from ..llmapi.reasoning_parser import ReasoningParserFactory from ..logger import logger from ..sampling_params import SamplingParams from .interface import (Evaluator, dump_inference_results, @@ -194,6 +195,16 @@ def _parse_partial_scores_env() -> Optional[int]: return value if value > 0 else None +def _model_type(llm) -> Optional[str]: + """The model type of the config the LLM read from its checkpoint, if any.""" + return getattr(getattr(llm, "_hf_model_config", None), "model_type", None) + + +# lm-eval caches contexts as strings. Keep native control IDs in a lossless, +# versioned envelope rather than decoding them to text and encoding again. +_NATIVE_CHAT_TOKENS_PREFIX = "__trtllm_native_chat_tokens_v1__:" + + class LmEvalWrapper(TemplateLM): def __init__(self, @@ -210,6 +221,7 @@ def __init__(self, partial_scoring_task_dict: Optional[dict] = None): super().__init__() self.llm = llm + self._native_chat_tokens = _model_type(llm) == "mistral_large_4" self.sampling_params = sampling_params self.streaming = streaming self.chat_template_kwargs = chat_template_kwargs @@ -257,6 +269,16 @@ def apply_chat_template(self, """ chat_template_kwargs = get_chat_template_kwargs( self.llm.tokenizer, self.chat_template_kwargs) + if self._native_chat_tokens: + token_ids = self.llm.tokenizer.apply_chat_template( + chat_history, + tokenize=True, + return_dict=False, + add_generation_prompt=add_generation_prompt, + continue_final_message=not add_generation_prompt, + **chat_template_kwargs, + ) + return _NATIVE_CHAT_TOKENS_PREFIX + json.dumps(token_ids) return self.llm.tokenizer.apply_chat_template( chat_history, tokenize=False, @@ -269,8 +291,17 @@ def apply_chat_template(self, def tokenizer_name(self) -> str: return self.llm.tokenizer.name_or_path.replace("/", "__") + def _prepare_prompt(self, prompt): + if (self._native_chat_tokens and isinstance(prompt, str) + and prompt.startswith(_NATIVE_CHAT_TOKENS_PREFIX)): + return json.loads(prompt[len(_NATIVE_CHAT_TOKENS_PREFIX):]) + return prompt + def tok_encode(self, string: str, **kwargs) -> List[int]: - return self.llm.tokenizer.encode(string, **kwargs) + prompt = self._prepare_prompt(string) + if isinstance(prompt, list): + return prompt + return self.llm.tokenizer.encode(prompt, **kwargs) def _loglikelihood_tokens(self, requests, **kwargs) -> List[Tuple[float, bool]]: @@ -329,8 +360,28 @@ def _get_sampling_params(self, gen_kwargs: dict) -> SamplingParams: # populated when perf metrics are requested. Small per-request # overhead, opted into together with the stats themselves. sampling_params.return_perf_metrics = True + if self._native_chat_tokens: + # Mistral-Large-4's reasoning delimiters are control tokens: keep + # them in the text for _response_text, as trtllm-serve does for its + # mistral_large_4 reasoning parser. + sampling_params.skip_special_tokens = False return sampling_params + def _response_text(self, output: RequestOutput) -> str: + """The text lm-eval scores for one output. + + Mistral-Large-4 reasons between ``[THINK]`` and ``[/THINK]`` before it + answers. Its output is scored on the content the mistral_large_4 + reasoning parser separates from the reasoning, the response a + trtllm-serve chat client reads; a reasoning block that never closes + leaves no response. Every other model is scored on its text. + """ + text = output.outputs[0].text + if not self._native_chat_tokens: + return text + return ReasoningParserFactory.create_reasoning_parser( + "mistral_large_4").parse(text).content + def _log_spec_stats(self, outputs: List[RequestOutput]) -> None: """Log corpus-aggregate speculative-decoding stats (TLLM_EVAL_SPEC_STATS=1). @@ -427,7 +478,7 @@ def _submit_next(pool): next_idx += 1 prompt, gen_kwargs = requests[idx].args sampling_params = self._get_sampling_params(gen_kwargs) - output = self.llm.generate_async(prompt, + output = self.llm.generate_async(self._prepare_prompt(prompt), sampling_params=sampling_params, streaming=self.streaming) return pool.submit(lambda: (idx, output.result())) @@ -456,7 +507,8 @@ def _submit_next(pool): done_count += 1 pbar.update(1) if scorer is not None: - scorer.update(requests[idx], output.outputs[0].text) + scorer.update(requests[idx], + self._response_text(output)) scorer.maybe_log(done_count, total) if next_idx < total: pending.add(_submit_next(pool)) @@ -500,7 +552,7 @@ def generate_until(self, requests, disable_tqdm: bool = False) -> List[str]: prompt, gen_kwargs = request.args sampling_params = self._get_sampling_params(gen_kwargs) output = self.llm.generate_async( - prompt, + self._prepare_prompt(prompt), sampling_params=sampling_params, streaming=self.streaming) results.append(output) @@ -512,7 +564,7 @@ def generate_until(self, requests, disable_tqdm: bool = False) -> List[str]: disable=disable_tqdm), requests): outputs.append(output.result()) if scorer is not None: - scorer.update(request, outputs[-1].outputs[0].text) + scorer.update(request, self._response_text(outputs[-1])) scorer.maybe_log(len(outputs), len(requests)) if self.output_dir: @@ -527,7 +579,7 @@ def generate_until(self, requests, disable_tqdm: bool = False) -> List[str]: logger.info(f"TRTLLM execution time: {elapsed_time:.3f} seconds.") profiler.reset("trtllm exec") - return [output.outputs[0].text for output in outputs] + return [self._response_text(output) for output in outputs] class MultimodalLmEvalWrapper(LmEvalWrapper): @@ -614,6 +666,8 @@ def __init__(self, def _get_model_type(self, llm: PyTorchLLM) -> str: """Extract model type from the model configuration.""" + if _model_type(llm) == "mistral_large_4": + return "mistral_large_4" config_path = os.path.join(llm._hf_model_dir, 'config.json') if not os.path.exists(config_path): @@ -645,6 +699,15 @@ def apply_chat_template(self, Adapted from: https://github.com/EleutherAI/lm-evaluation-harness/blob/7f04db12d2f8e7a99a0830d99eb78130e1ba2122/lm_eval/models/hf_vlms.py#L225 """ + if self.model_type in NATIVE_MISTRAL_MODEL_TYPES: + # lm-eval supplies image pixels only in generate_until, and + # mistral-common sizes each image's tokens from its pixels. Its + # cached context is a string, so keep the structured chat there + # and render it once the images are loaded. + return json.dumps({ + "conversation": chat_history, + "add_generation_prompt": add_generation_prompt + }) # Resolve content format once to decide whether to pre-insert # placeholders. OPENAI templates handle media natively, so we must # NOT pre-insert or the template will produce duplicates. @@ -738,6 +801,57 @@ def apply_chat_template(self, })) return output + def _render_native_mistral_chat(self, prompt: str, images) -> List[int]: + """Render a deferred chat with mistral-common, its images where the text has them. + + The image placeholders take the images in order, across the conversation; + placeholders past the last image (``is_force_single_image``) are dropped, + and images past the last placeholder follow the last message's text. + """ + deferred = json.loads(prompt) + conversation, counts = [], [] + placed = 0 + for message in deferred["conversation"]: + parts = message["content"].split(LM_EVAL_DEFAULT_IMAGE_PLACEHOLDER) + content_parts = [] + for index, text in enumerate(parts): + if text: + content_parts.append(text) + if index < len(parts) - 1 and placed < len(images): + content_parts.append({ + "type": "image", + "media_index": placed + }) + placed += 1 + conversation.append( + ConversationMessage(role=message.get("role", "user"), + content="".join(parts), + content_parts=content_parts)) + counts.append(sum(isinstance(part, dict) for part in content_parts)) + if conversation and placed < len(images): + for index in range(placed, len(images)): + conversation[-1]["content_parts"].append({ + "type": "image", + "media_index": index + }) + counts[-1] += len(images) - placed + return trtllm_apply_chat_template( + model_type=self.model_type, + tokenizer=self.llm.tokenizer, + processor=None, + conversation=conversation, + add_generation_prompt=deferred["add_generation_prompt"], + mm_placeholder_counts=[{ + "[IMG]": count + } if count else {} for count in counts], + mm_data={"image": images}, + chat_template_kwargs={ + **(self.chat_template_kwargs or {}), + # A gen_prefix ends the chat with the assistant turn to continue. + "continue_final_message": + not deferred["add_generation_prompt"], + }) + def generate_until(self, requests, disable_tqdm: bool = False) -> List[str]: """ Generate text responses for multimodal requests. @@ -764,7 +878,6 @@ def generate_until(self, requests, disable_tqdm: bool = False) -> List[str]: # NOTE: For now, only this part is different from the original generate_until prompt, gen_kwargs, media_data = request.args - prompt = prompt_inputs(prompt) # NOTE: Convert RGBA format to RGB format if self.is_force_single_image: @@ -775,6 +888,9 @@ def generate_until(self, requests, disable_tqdm: bool = False) -> List[str]: convert_image_mode(img, "RGB") for img in media_data["visual"] ] + if self.model_type in NATIVE_MISTRAL_MODEL_TYPES: + prompt = self._render_native_mistral_chat(prompt, images) + prompt = prompt_inputs(prompt) prompt["multi_modal_data"] = {"image": images} sampling_params = self._get_sampling_params(gen_kwargs) @@ -793,7 +909,7 @@ def generate_until(self, requests, disable_tqdm: bool = False) -> List[str]: # Kimi K2.5 passes strip_thinking_and_extract_mmmu_answer to # recover answers from ...-wrapped outputs that # lm-eval's default extractor cannot parse. - raw = outputs[-1].outputs[0].text + raw = self._response_text(outputs[-1]) text = self.post_process_fn( raw) if self.post_process_fn is not None else raw results_text.append(text) diff --git a/tensorrt_llm/inputs/utils.py b/tensorrt_llm/inputs/utils.py index 0aa629d990a4..68c98b2e5904 100644 --- a/tensorrt_llm/inputs/utils.py +++ b/tensorrt_llm/inputs/utils.py @@ -6,8 +6,8 @@ from collections import defaultdict from io import BytesIO from pathlib import Path -from typing import (Any, Collection, Coroutine, Dict, List, NamedTuple, - Optional, Tuple, TypedDict, Union) +from typing import (TYPE_CHECKING, Any, Collection, Coroutine, Dict, List, + NamedTuple, Optional, Tuple, TypedDict, Union) from urllib.parse import urlparse import numpy as np @@ -43,6 +43,10 @@ from tensorrt_llm.tokenizer.deepseek_v32 import DeepseekV32Tokenizer from tensorrt_llm.tokenizer.deepseek_v41 import DeepseekV41Tokenizer +if TYPE_CHECKING: + from tensorrt_llm._torch.models.checkpoints.mistral.tokenizer import \ + MistralTokenizer + logger = logging.get_logger(__name__) @@ -663,10 +667,18 @@ def resolve_hf_chat_template( return None +# Model types whose chats mistral-common renders: the checkpoints the +# Mistral-native format (checkpoint_format "mistral" or "mistral_large_3") +# loads, and Mistral Large 4, whose config names its own model type. Their +# tokenizer is a MistralTokenizer, and the renderer returns token ids. +NATIVE_MISTRAL_MODEL_TYPES = frozenset(("mistral_common", "mistral_large_4")) + + class _RenderDispatch(NamedTuple): """The renderer `apply_chat_template` uses for a set of inputs. - `kind` is `"deepseek_v41"`, `"deepseek"` (the other DeepSeek + `kind` is `"mistral_native"` (mistral-common renders the chat), + `"deepseek_v41"`, `"deepseek"` (the other DeepSeek tokenizers), `"passthrough"` (the model skips chat templates), `"native"` (a tokenizer-native python template) or `"jinja"`. `tokenizer` is the object that renderer is called on, and `template` @@ -686,6 +698,8 @@ def _resolve_render_dispatch( tools: Optional[list[dict[str, Any]]], ) -> _RenderDispatch: """Pick the renderer, in the order `apply_chat_template` checks them.""" + if model_type in NATIVE_MISTRAL_MODEL_TYPES: + return _RenderDispatch("mistral_native", tokenizer) if isinstance(tokenizer, DeepseekV41Tokenizer): return _RenderDispatch("deepseek_v41", tokenizer) if isinstance(tokenizer, (DeepseekV32Tokenizer, DeepseekV4Tokenizer)): @@ -719,8 +733,9 @@ def rendered_hf_chat_template_source( """The Jinja source `apply_chat_template` will render for these inputs. Returns `None` when `apply_chat_template` dispatches to a renderer - with no Jinja source to analyze (the DeepSeek custom tokenizers, a model - that skips chat templates, a tokenizer-native python template) or when no + with no Jinja source to analyze (mistral-common for Mistral-native checkpoints, + the DeepSeek custom tokenizers, a model that skips chat templates, a + tokenizer-native python template) or when no template resolves. Callers that construct chat_template_kwargs use this to run the same referenced-kwargs analysis the render-site guard applies, so they can avoid injecting a kwarg the active template never reads. @@ -795,6 +810,188 @@ def _build_openai_content( return content_list +# The `chat_template_kwargs` keys mistral-common's chat encoder reads. It has +# no Jinja template, so other keys have nothing to apply to. +_NATIVE_MISTRAL_CHAT_KWARGS = frozenset( + ("continue_final_message", "reasoning_effort", "response_format")) + + +def _native_mistral_response_format(response_format): + """An OpenAI `response_format` as mistral-common's, or None if it renders nothing. + + `json_object` and a named `json_schema` render into the prompt's model + settings; other formats only constrain decoding. mistral-common renders + response formats from 1.13.0 on, which needs Python 3.11; with an older + release a response format only constrains decoding. + """ + if not isinstance(response_format, dict): + return None + kind = response_format.get("type") + named = response_format.get("json_schema") + if kind not in ("json_object", "json_schema") or ( + kind == "json_schema" and + (not isinstance(named, dict) or named.get("schema") is None + or not named.get("name"))): + return None + from mistral_common.protocol.instruct import request as mistral_request + if not hasattr(mistral_request, "JsonSchema"): + logger.warning_once( + "mistral-common renders response formats from 1.13.0 on (Python " + "3.11 or later); the prompt leaves the response format out, and it " + "only constrains decoding.") + return None + ResponseFormat = mistral_request.ResponseFormat + ResponseFormats = mistral_request.ResponseFormats + JsonSchema = mistral_request.JsonSchema + if kind == "json_object": + return ResponseFormat(type=ResponseFormats.json_object) + return ResponseFormat(type=ResponseFormats.json_schema, + json_schema=JsonSchema(name=named["name"], + schema=named["schema"], + strict=bool( + named.get("strict")))) + + +# The URL of each image part in the messages `_native_mistral_messages` builds; +# `_encode_native_mistral_chat` puts an image of the loaded image's size there. +_NATIVE_MISTRAL_IMAGE_SLOT = "trtllm-loaded-image" + + +def _encode_native_mistral_chat(tokenizer: "MistralTokenizer", + messages: list[dict[str, Any]], + image_sizes: list[tuple[int, int]], *, + tools: Optional[list[dict[str, Any]]], + continue_final_message: bool, + reasoning_effort: Optional[str], + response_format: Optional[Any]) -> list[int]: + """Token ids of a chat, encoded by mistral-common. + + The image parts are slots that take, in order, a blank image of each + `image_sizes` entry (width, height): mistral-common makes an image's token + grid from its size alone, and the tokenizer's `grid_only_mistral` makes + the grid without processing pixels, which the input processor does. + `response_format` is mistral-common's, or None. + """ + from mistral_common.protocol.instruct.chunk import ImageChunk, ImageURLChunk + from mistral_common.protocol.instruct.request import ChatCompletionRequest + + # mistral-common before 1.13.0 has no response formats. + request_kwargs = ({} if response_format is None else { + "response_format": response_format + }) + request = ChatCompletionRequest.from_openai( + messages=messages, + tools=tools, + continue_final_message=continue_final_message, + reasoning_effort=reasoning_effort, + **request_kwargs) + filled = 0 + for message in request.messages: + content = getattr(message, "content", None) + if not isinstance(content, list): + continue + chunks = [] + for chunk in content: + if isinstance(chunk, ImageURLChunk) and chunk.get_url( + ) == _NATIVE_MISTRAL_IMAGE_SLOT: + chunk = ImageChunk(image=Image.new("1", image_sizes[filled])) + filled += 1 + chunks.append(chunk) + message.content = chunks + if filled != len(image_sizes): + raise ValueError(f"The chat has {filled} image slots for " + f"{len(image_sizes)} images.") + return tokenizer.grid_only_mistral.encode_chat_completion(request).tokens + + +def _native_mistral_tools(tools): + """OpenAI tool dicts in the form mistral-common validates: no null `strict`.""" + if tools is None: + return None + normalized = [] + for tool in tools: + function = tool.get("function") + if isinstance(function, dict) and function.get("strict", False) is None: + function = { + key: value + for key, value in function.items() if key != "strict" + } + tool = {**tool, "function": function} + normalized.append(tool) + return normalized + + +def _validate_native_mistral_chat_args(messages, add_generation_prompt, + continue_final_message): + """Reject generation-prompt flags that contradict the final message.""" + if add_generation_prompt and continue_final_message: + raise ValueError("Cannot set both `add_generation_prompt` and " + "`continue_final_message` to True.") + last_role = messages[-1].get("role") if messages else None + if add_generation_prompt and last_role == "assistant": + raise ValueError( + "Cannot set `add_generation_prompt` to True when the last message " + "is from the assistant. Consider using `continue_final_message` " + "instead.") + if continue_final_message and last_role != "assistant": + raise ValueError("Cannot set `continue_final_message` to True when " + "the last message is not from the assistant.") + + +def _native_mistral_messages( + conversation: list[ConversationMessage], + mm_placeholder_counts: list[dict[str, int]], + mm_data: Optional[Dict[str, Any]], +) -> tuple[list[dict[str, Any]], list[tuple[int, int]]]: + """The chat as OpenAI messages, each image part in its place, and the image sizes. + + Each image part is a slot (see `_encode_native_mistral_chat`); the sizes, + (width, height) in the order of the slots, are those of the loaded images. + """ + from tensorrt_llm._torch.models.modeling_mistral import \ + MistralCommonImageProcessor + + mm_data = mm_data or {} + if set(mm_data) - {"image"}: + raise ValueError( + "Mistral-native checkpoints currently support image inputs only") + images = MistralCommonImageProcessor.image_list(mm_data.get("image")) + num_placeholders = sum( + sum(counts.values()) for counts in mm_placeholder_counts) + if num_placeholders != len(images): + # mistral-common sizes each image's tokens from the decoded image, so + # the images must be loaded before the chat is rendered. + raise ValueError( + f"The chat has {num_placeholders} images, but {len(images)} were " + "loaded for rendering; this route does not take images for " + "Mistral-native checkpoints.") + image_sizes = [] + messages = [] + for conv, counts in zip(conversation, mm_placeholder_counts): + message = { + key: value + for key, value in conv.items() + if key not in ("media", "content_parts") + } + if counts or conv.get("content_parts"): + content = _build_openai_content(conv, counts) + for index, part in enumerate(content): + if part["type"] == "text": + continue + content[index] = { + "type": "image_url", + "image_url": { + "url": _NATIVE_MISTRAL_IMAGE_SLOT + } + } + image_sizes.append( + MistralCommonImageProcessor.pil_image_size( + images[len(image_sizes)])) + message["content"] = content + messages.append(message) + return messages, image_sizes + + def apply_chat_template( *, model_type: str, @@ -809,7 +1006,8 @@ def apply_chat_template( chat_template_kwargs: Optional[dict[str, Any]] = None, enable_tokenize: bool = False, injected_chat_template_kwargs: Optional[Collection[str]] = None, -) -> (str | List[str]): + mm_data: Optional[Dict[str, Any]] = None, +) -> (str | List[str] | List[int]): """Apply chat template to the conversation. `injected_chat_template_kwargs` names the keys of `chat_template_kwargs` @@ -829,6 +1027,50 @@ def apply_chat_template( chat_template=chat_template, tools=tools) + if dispatch.kind == "mistral_native": + from tensorrt_llm._torch.models.checkpoints.mistral.tokenizer import \ + MistralTokenizer + if isinstance(tokenizer, TransformersTokenizer) and not isinstance( + tokenizer, MistralTokenizer): + raise ValueError( + "The Mistral-native format renders chats with " + "mistral-common's MistralTokenizer, not an HF tokenizer " + f"({type(tokenizer).__name__})") + messages, image_sizes = _native_mistral_messages( + conversation, mm_placeholder_counts, mm_data) + requested = chat_template_kwargs or {} + continue_final_message = bool( + requested.get("continue_final_message", False)) + _validate_native_mistral_chat_args(messages, add_generation_prompt, + continue_final_message) + ignored = sorted(set(requested) - _NATIVE_MISTRAL_CHAT_KWARGS) + if documents is not None: + ignored.append("documents") + if ignored: + logger.warning_once( + f"Mistral-native checkpoints render chats with mistral-common, which " + f"does not use {ignored}; ignoring them.") + from mistral_common.exceptions import MistralCommonException + + response_format = _native_mistral_response_format( + requested.get("response_format")) + # mistral-common validates the request against the checkpoint (for + # example a reasoning effort its tekken.json does not declare); its + # errors are invalid requests. add_generation_prompt only selects the + # validation above: the encoder always opens the assistant turn after + # a final user or tool message. + try: + return _encode_native_mistral_chat( + tokenizer, + messages, + image_sizes, + tools=_native_mistral_tools(tools), + continue_final_message=continue_final_message, + reasoning_effort=requested.get("reasoning_effort"), + response_format=response_format) + except MistralCommonException as error: + raise ValueError(str(error)) from error + if dispatch.kind == "deepseek_v41": # The native renderer deep-copies messages; omit pending media coroutines # and retain flattened content with its interleaved image placeholders. @@ -920,7 +1162,8 @@ async def async_apply_chat_template( chat_template_kwargs: Optional[dict[str, Any]] = None, enable_tokenize: bool = False, injected_chat_template_kwargs: Optional[Collection[str]] = None, -) -> (str | List[str]): + mm_data: Optional[Dict[str, Any]] = None, +) -> (str | List[str] | List[int]): """Apply chat template without blocking the event loop.""" with _time_input_processing("chat_template"): return await asyncio.to_thread( @@ -937,9 +1180,23 @@ async def async_apply_chat_template( chat_template_kwargs=chat_template_kwargs, enable_tokenize=enable_tokenize, injected_chat_template_kwargs=injected_chat_template_kwargs, + mm_data=mm_data, ) +async def async_apply_chat_template_with_mm(mm_coroutines, **kwargs): + """Resolve native image geometry before encoding its chat token stream.""" + if kwargs["model_type"] in NATIVE_MISTRAL_MODEL_TYPES: + mm_data, mm_embeddings = await mm_coroutines + if mm_embeddings: + raise ValueError( + "Native Mistral chat requires images, not external embeddings") + prompt = await async_apply_chat_template(**kwargs, mm_data=mm_data) + return prompt, (mm_data, mm_embeddings) + return await asyncio.gather(async_apply_chat_template(**kwargs), + mm_coroutines) + + def apply_mm_placeholders( model_type: str, message: ConversationMessage, @@ -1043,7 +1300,8 @@ async def async_build_multimodal_prompt( apply_mm_placeholders(model_type, message, placeholder_counts, mm_data_tracker) - prompt_task = async_apply_chat_template( + rendered, (mm_data, _) = await async_apply_chat_template_with_mm( + mm_data_tracker.retrieve_all_async(), model_type=model_type, tokenizer=tokenizer, processor=processor, @@ -1053,9 +1311,6 @@ async def async_build_multimodal_prompt( chat_template=chat_template, chat_template_kwargs=chat_template_kwargs, ) - rendered, (mm_data, - _) = await asyncio.gather(prompt_task, - mm_data_tracker.retrieve_all_async()) inputs = prompt_inputs(rendered) if mm_data: @@ -1198,12 +1453,16 @@ def convert_to_conversation_message( is_passthrough = (MULTIMODAL_PLACEHOLDER_REGISTRY.get_content_format( model_type) == ContentFormat.PASSTHROUGH) - if tokenizer is None and not is_passthrough: + if tokenizer is None and model_type in NATIVE_MISTRAL_MODEL_TYPES: + from tensorrt_llm._torch.models.checkpoints.mistral.tokenizer import \ + MistralTokenizer + tokenizer = MistralTokenizer.from_pretrained(model_dir) + elif tokenizer is None and not is_passthrough: tokenizer = ModelLoader.load_hf_tokenizer( model_dir, trust_remote_code=trust_remote_code, use_fast=True) processor = None - if not is_passthrough: + if not is_passthrough and model_type not in NATIVE_MISTRAL_MODEL_TYPES: processor = AutoProcessor.from_pretrained( model_dir, use_fast=True, trust_remote_code=trust_remote_code) @@ -1222,7 +1481,7 @@ def convert_to_conversation_message( is_embedding=is_embedding) mm_placeholder_counts = mm_data_tracker.placeholder_counts() prompt = conv["content"] - if mm_placeholder_counts: + if mm_placeholder_counts and model_type not in NATIVE_MISTRAL_MODEL_TYPES: # Resolve content format to decide whether to pre-insert # placeholders. OPENAI templates handle media natively (e.g. # numbered tags), so we must NOT pre-insert or the @@ -1234,14 +1493,22 @@ def convert_to_conversation_message( if content_format != ContentFormat.OPENAI: conv["content"] = add_multimodal_placeholders( model_type, conv["content"], mm_placeholder_counts) + native_mm_data = None + if model_type in NATIVE_MISTRAL_MODEL_TYPES: + if is_embedding: + raise ValueError( + "Native Mistral chat requires images, not external embeddings" + ) + native_mm_data, _ = mm_data_tracker.retrieve_all_sync() prompt = apply_chat_template( model_type=model_type, tokenizer=tokenizer, processor=processor, conversation=[conv], add_generation_prompt=True, - mm_placeholder_counts=[mm_placeholder_counts]) - input = {"prompt": prompt} + mm_placeholder_counts=[mm_placeholder_counts], + mm_data=native_mm_data) + input = prompt_inputs(prompt) if mm_placeholder_counts: input["chat_template_applied"] = True @@ -1250,9 +1517,12 @@ def convert_to_conversation_message( "multi_modal_embeddings"] = mm_data_tracker.retrieve_all_sync( ) else: - input[ - "multi_modal_data"], _ = mm_data_tracker.retrieve_all_sync( - ) + if native_mm_data is not None: + input["multi_modal_data"] = native_mm_data + else: + input[ + "multi_modal_data"], _ = mm_data_tracker.retrieve_all_sync( + ) item_order = mm_data_tracker.item_order() if item_order: input["mm_item_order"] = item_order diff --git a/tensorrt_llm/llmapi/disagg_utils.py b/tensorrt_llm/llmapi/disagg_utils.py index 29c74bac68a2..d5fa64f05a41 100644 --- a/tensorrt_llm/llmapi/disagg_utils.py +++ b/tensorrt_llm/llmapi/disagg_utils.py @@ -673,6 +673,12 @@ def extract_router_config(server_cfg: dict) -> RouterConfig: model_path = server_cfg.get("model") if model_path is not None: args["model_path"] = model_path + # The cache-aware router loads the servers' tokenizer and prompt renderer + # the way they do, which depends on the checkpoint format. + if router_type == "kv_cache_aware" and "checkpoint_format" not in args: + checkpoint_format = server_cfg.get("checkpoint_format") + if checkpoint_format is not None: + args["checkpoint_format"] = checkpoint_format # add fields that are not specific to router extract_keys = [ diff --git a/tensorrt_llm/llmapi/llm.py b/tensorrt_llm/llmapi/llm.py index 2feda38b6b1d..e2698eac4517 100644 --- a/tensorrt_llm/llmapi/llm.py +++ b/tensorrt_llm/llmapi/llm.py @@ -350,6 +350,22 @@ def _multimodal_params_have_encoder_features( """ +def _default_guided_decoding_backend( + args: Any, model_type: Optional[str]) -> Optional[str]: + """The guided decoding backend to serve with when the arguments set none. + + Mistral Large 4 serves structured outputs (``response_format``) by default, + with xgrammar. A speculative decoding config keeps the given value: how it + combines with guided decoding is settled when the arguments are validated. + A multimodal-encoder-only instance generates nothing and keeps it too. + """ + if (args.guided_decoding_backend is None and model_type == "mistral_large_4" + and args.speculative_config is None + and not getattr(args, "mm_encoder_only", False)): + return "xgrammar" + return args.guided_decoding_backend + + @dataclass class PreprocessedInputs: """Light structure for holding preprocessed inputs. @@ -1756,8 +1772,21 @@ def _try_load_generation_config_explicit_values(self) -> dict[str, Any]: def _try_load_hf_model_config( self) -> Optional[transformers.PretrainedConfig]: + kwargs = {} + # A Mistral-Large-4 checkpoint keeps its config in params.json, which + # the Mistral-native format reads; every other checkpoint keeps the HF + # config lookup. + checkpoint_format = getattr(self.args, "checkpoint_format", None) + if checkpoint_format in (CheckpointFormat.MISTRAL, + CheckpointFormat.MISTRAL_LARGE_3): + from .._torch.models.checkpoints.mistral.config_loader import \ + is_mistral_large4_checkpoint + if is_mistral_large4_checkpoint(self.args.model): + kwargs["checkpoint_format"] = checkpoint_format return ModelLoader.load_hf_model_config( - self.args.model, trust_remote_code=self.args.trust_remote_code) + self.args.model, + trust_remote_code=self.args.trust_remote_code, + **kwargs) @set_api_status("prototype") def start_profile(self, @@ -2219,6 +2248,22 @@ def _build_model(self): "llm.encode() for optimized batch-forward inference that " "bypasses the decoder scheduler.", architectures[0]) + model_type = getattr(self._hf_model_config, "model_type", None) + guided_decoding_backend = _default_guided_decoding_backend( + self.args, model_type) + if guided_decoding_backend != self.args.guided_decoding_backend: + logger.info( + f"Guided decoding backend {guided_decoding_backend} enabled " + "for structured outputs.") + self.args.guided_decoding_backend = guided_decoding_backend + elif (model_type == "mistral_large_4" + and guided_decoding_backend is None + and self.args.speculative_config is not None): + logger.info( + "Guided decoding stays off with speculative decoding; set " + "guided_decoding_backend to 'xgrammar' to serve structured " + "outputs.") + # Create the standard executor for generate()/generate_async() # TODO: revisit gather_context_logits return_logits = self.args.gather_generation_logits diff --git a/tensorrt_llm/llmapi/llm_args.py b/tensorrt_llm/llmapi/llm_args.py index d7cef8029182..8083a854f772 100644 --- a/tensorrt_llm/llmapi/llm_args.py +++ b/tensorrt_llm/llmapi/llm_args.py @@ -1012,9 +1012,11 @@ class GuidedDecodingBackend(Enum): backend: GuidedDecodingBackend = Field( default=GuidedDecodingBackend.XGRAMMAR, description="The backend for guided decoding config.") - encoded_vocab: Optional[List[str]] = Field( + encoded_vocab: Optional[List[Union[str, bytes]]] = Field( default=None, - description="The encoded vocab for guided decoding config.") + description= + "The encoded vocab for guided decoding config: each token's text, or " + "its raw bytes for tokenizers whose tokens need not be valid UTF-8.") tokenizer_str: Optional[str] = Field( default=None, description="The tokenizer string for guided decoding config.") @@ -4685,6 +4687,16 @@ class DSparkDecodingConfig(DecodingBaseConfig): "decode pass over context and block, on SM100/SM103/SM107. Which " "kernel each name selects: MLADSparkForCausalLM.") + # Checkpoint adapters own the model-specific draft metadata. These runtime + # attributes travel with the resolved config but are not public arguments. + _draft_model_config: Optional[dict] = PrivateAttr(default=None) + _draft_weights_in_target: bool = PrivateAttr(default=False) + + @property + def needs_separate_draft_weights(self) -> bool: + # Weight ownership is independent of the worker and its KV-cache layout. + return not self._draft_weights_in_target + @model_validator(mode="after") def validate_ancestral_draft_sampling(self) -> "DSparkDecodingConfig": """Ancestral drafting acts only on rejection-sampled drafts.""" @@ -9783,108 +9795,9 @@ def validate_speculative_config(self): if not spec_cfg.max_draft_len: raise ValueError("DSpark max_draft_len must be > 0; got " f"{spec_cfg.max_draft_len}") - # Same convention as MTP (see resolve_mtp_checkpoint_source): - # an unset speculative_model means "the draft lives in the - # target checkpoint". For DSpark that is the embedded - # DeepSeek-V4-Pro flavour, whose draft is the target's mtp.* - # namespace. Defaulting rather than rejecting keeps the - # spec-dec API consistent across algorithms; pointing - # speculative_model at the target explicitly is equivalent. - if spec_cfg.speculative_model is None: - spec_cfg.speculative_model = self.model - if not spec_cfg.draft_is_embedded_in_target: - raise ValueError( - "speculative_config.speculative_model is unset, " - "which means 'load the draft from the target " - "checkpoint', but the target has no mtp.* draft " - "weights. Point speculative_model at a standalone " - "DSpark drafter checkpoint directory.") - # Warm the embedded-vs-standalone probe here, while we are in - # the main process and the checkpoint path is known local. The - # flag then travels with the config, so no rank repeats the - # filesystem read and nothing probes at forward time. - if (spec_cfg.draft_is_embedded_in_target - and spec_cfg.use_ancestral_draft_sampling): - raise ValueError( - "use_ancestral_draft_sampling=True is not supported with a " - "drafter embedded in the target checkpoint.") - # Resolve target_layer_ids / mask_token_id / block_size / - # markov_rank from the draft (or main) model config if not set. - # Several checkpoint spellings are in the wild for the same knobs - # and all are accepted here, because a key the reader misses is - # not an error -- it silently falls back to a default and the - # drafter degrades (a markov_rank read as 0 skips the Markov - # head entirely, costing acceptance with no warning): - # - top-level ``dspark_*`` (DeepSeek-V4-Pro) - # - nested ``dflash_config`` (SpecForge / RadixArk drafters) - # - nested ``dspark_config`` (forward compatibility) - # - plain top-level keys (TorchSpec drafters) - # V4.1 wraps the language model settings in ``text_config``; - # prefer that scope, retaining the top-level legacy fallback. - draft_config_path = os.path.join(spec_cfg.speculative_model, - "config.json") - if os.path.exists(draft_config_path): - with open(draft_config_path) as f: - draft_cfg = translate_speculators_dspark_config( - json.load(f)) - text_cfg = draft_cfg.get("text_config") or {} - - def _dspark_get( - key: str, top_level_key: str - ) -> Optional[Union[List[int], int]]: - for scope in (text_cfg, draft_cfg): - for source, name in ((scope.get("dspark_config") - or {}, key), - (scope.get("dflash_config") - or {}, key), - (scope, top_level_key), (scope, - key)): - value = source.get(name) - if value is not None: - return value - return None - - # The checkpoint's ``dspark_target_layer_ids`` is - # authoritative: it fixes both which target hidden states - # were captured during draft training and the input width of - # ``main_proj`` (hidden_size * num_capture_layers). If the - # user leaves ``target_layer_ids`` unset we copy it verbatim - # (order preserved, since the projection columns are - # order-dependent). An explicit override that does not match - # the checkpoint list exactly would either mismatch the - # projection shape at runtime (different count) or feed the - # draft hidden states it was not trained on (same count, - # different layers), so reject it during validation. - ckpt_layer_ids = _dspark_get("target_layer_ids", - "dspark_target_layer_ids") - if spec_cfg.target_layer_ids is None: - if ckpt_layer_ids is not None: - spec_cfg.target_layer_ids = list(ckpt_layer_ids) - elif ckpt_layer_ids is not None and list( - spec_cfg.target_layer_ids) != list(ckpt_layer_ids): - raise ValueError( - "DSpark target_layer_ids must match the checkpoint's " - "dspark_target_layer_ids exactly (the draft " - "projection weights are trained for that specific " - f"layer set); got override {spec_cfg.target_layer_ids} " - f"but the checkpoint specifies {list(ckpt_layer_ids)}. " - "Leave target_layer_ids unset to use the checkpoint " - "value.") - if spec_cfg.mask_token_id is None: - mask_id = _dspark_get("mask_token_id", - "dspark_noise_token_id") - if mask_id is not None: - spec_cfg.mask_token_id = mask_id - if spec_cfg.block_size is None: - block_size = _dspark_get("block_size", - "dspark_block_size") - if block_size is not None: - spec_cfg.block_size = int(block_size) - if spec_cfg.markov_rank is None: - markov_rank = _dspark_get("markov_rank", - "dspark_markov_rank") - if markov_rank is not None: - spec_cfg.markov_rank = markov_rank + from .._torch.models.checkpoints.dspark_config import \ + resolve_dspark_checkpoint + resolve_dspark_checkpoint(spec_cfg, self.model) if spec_cfg.block_size is None: spec_cfg.block_size = spec_cfg.max_draft_len if spec_cfg.block_size <= 0: diff --git a/tensorrt_llm/llmapi/llm_utils.py b/tensorrt_llm/llmapi/llm_utils.py index ed158e87b1a1..61fc064162e4 100644 --- a/tensorrt_llm/llmapi/llm_utils.py +++ b/tensorrt_llm/llmapi/llm_utils.py @@ -14,6 +14,7 @@ # yapf: disable from ..bindings.executor import (CapacitySchedulerPolicy, ContextChunkingPolicy, KvCacheRetentionConfig) +from ..checkpoint_format import CheckpointFormat # yapf: enable from ..logger import logger from ..models.modeling_utils import QuantAlgo, QuantConfig @@ -136,6 +137,15 @@ def _apply_modelopt_quant_config(self, hf_quant_config: Dict[str, Any], setattr(quant_config, key, value) self.llm_args.quant_config = quant_config + def _is_mistral_large4_checkpoint(self) -> bool: + """Whether the model is a Mistral-Large-4 checkpoint read in the Mistral-native format.""" + if self.llm_args.checkpoint_format not in ( + CheckpointFormat.MISTRAL, CheckpointFormat.MISTRAL_LARGE_3): + return False + from .._torch.models.checkpoints.mistral.config_loader import \ + is_mistral_large4_checkpoint + return is_mistral_large4_checkpoint(self._model_dir) + def _update_from_hf_quant_config(self) -> bool: """Update quant_config from the config file of pre-quantized HF checkpoint. @@ -187,14 +197,24 @@ def _update_from_hf_quant_config(self) -> bool: hf_config_path = f"{self._model_dir}/config.json" hf_quant_config = None - if os.path.exists(hf_config_path): + hf_config = None + if self._is_mistral_large4_checkpoint(): + # Its params.json states the ModelOpt quantization a config.json + # would. + from .._torch.models.checkpoints.mistral.config_loader import \ + MistralConfigLoader + hf_config = MistralConfigLoader().load_pretrained_config( + self._model_dir).to_dict() + hf_config_path = f"{self._model_dir}/params.json" + elif os.path.exists(hf_config_path): with open(hf_config_path, "r") as f: hf_config = json.load(f) - hf_quant_config = hf_config.get("quantization_config", None) - if hf_quant_config is not None: - logger.info( - f"Found quantization_config field in {hf_config_path}, pre-quantized checkpoint is used." - ) + if hf_config is not None: + hf_quant_config = hf_config.get("quantization_config", None) + if hf_quant_config is not None: + logger.info( + f"Found quantization_config field in {hf_config_path}, pre-quantized checkpoint is used." + ) if self.llm_args.model_kwargs is not None and "quantization_config" in self.llm_args.model_kwargs: logger.info( f"Update hf_quant_config from model_kwargs: quantization_config={self.llm_args.model_kwargs['quantization_config']} (previous value: {hf_quant_config})" diff --git a/tensorrt_llm/llmapi/reasoning_parser.py b/tensorrt_llm/llmapi/reasoning_parser.py index 66c59d61577e..d6b7e6967b19 100644 --- a/tensorrt_llm/llmapi/reasoning_parser.py +++ b/tensorrt_llm/llmapi/reasoning_parser.py @@ -724,6 +724,7 @@ def parse_delta(self, delta_text: str) -> ReasoningParserResult: "kimi_k3": "kimi_k3", "minimax_m3": "minimax_m3", "minimax_m3_vl": "minimax_m3", + "mistral_large_4": "mistral_large_4", "muse_spark": "musespark", } @@ -1397,6 +1398,108 @@ def finish(self) -> ReasoningParserResult: return ReasoningParserResult() +@register_reasoning_parser("mistral_large_4") +class MistralLarge4ReasoningParser(BaseReasoningParser): + """Reasoning parser for Mistral-Large-4. + + Mistral-Large-4 (mistral-common Tekken v15) writes its reasoning between + the control tokens ``[THINK]`` and ``[/THINK]``, and starts its tool calls + with ``[TOOL_CALLS]``. The rules follow vLLM's Mistral parser + (``vllm/parser/mistral.py``): + + * the output starts as content; ``[THINK]`` opens reasoning and + ``[/THINK]`` closes it, so text around a block stays content and + several blocks are joined; + * a ``[THINK]`` inside reasoning and a ``[/THINK]`` outside it are + dropped, and a block that never closes is reasoning to the end; + * ``[TOOL_CALLS]`` also ends reasoning; it and everything after it are + content, for the tool parser; + * reasoning loses its trailing whitespace. + + The delimiters reach the parser only with special tokens kept in the + decoded text, which also keeps the end-of-sequence ````; it is + dropped. + """ + + needs_raw_special_tokens = True + + THINK_START = "[THINK]" + THINK_END = "[/THINK]" + TOOL_CALLS = "[TOOL_CALLS]" + EOS = "" + + def __init__(self, + *, + chat_template_kwargs: Optional[dict[str, Any]] = None) -> None: + super().__init__(chat_template_kwargs=chat_template_kwargs) + self._buffer = "" + self._in_reasoning = False + self._in_tool_calls = False + # Trailing reasoning whitespace, sent only if more reasoning follows. + self._held_space = "" + + def parse(self, text: str) -> ReasoningParserResult: + parser = MistralLarge4ReasoningParser() + result = parser._feed(text, final=True) + return ReasoningParserResult(content=result.content, + reasoning_content=result.reasoning_content) + + def parse_delta(self, delta_text: str) -> ReasoningParserResult: + return self._feed(delta_text, final=False) + + def finish(self) -> ReasoningParserResult: + return self._feed("", final=True) + + def _feed(self, text: str, final: bool) -> ReasoningParserResult: + self._buffer += text + content: list[str] = [] + reasoning: list[str] = [] + while self._step(content, reasoning, final): + pass + return ReasoningParserResult(content="".join(content), + reasoning_content="".join(reasoning)) + + def _step(self, content: list[str], reasoning: list[str], + final: bool) -> bool: + """Consume the buffer up to the next marker; False when it needs more.""" + buffer = self._buffer + markers = ((self.EOS, ) if self._in_tool_calls else + (self.THINK_START, self.THINK_END, self.TOOL_CALLS, + self.EOS)) + found = [(buffer.find(marker), marker) for marker in markers] + found = [(index, marker) for index, marker in found if index != -1] + if not found: + keep = 0 if final else _trailing_partial_marker(buffer, *markers) + self._emit(buffer[:len(buffer) - keep], content, reasoning) + self._buffer = buffer[len(buffer) - keep:] + return False + index, marker = min(found) + self._emit(buffer[:index], content, reasoning) + if marker == self.TOOL_CALLS: + # The tool parser reads the calls from the content. + self._in_reasoning = False + self._in_tool_calls = True + self._buffer = buffer[index:] + return True + self._buffer = buffer[index + len(marker):] + if marker == self.THINK_START: + self._in_reasoning = True + elif marker == self.THINK_END: + self._in_reasoning = False + return True + + def _emit(self, text: str, content: list[str], + reasoning: list[str]) -> None: + if not self._in_reasoning: + content.append(text) + return + combined = self._held_space + text + body = combined.rstrip() + self._held_space = combined[len(body):] + if body: + reasoning.append(body) + + # Model-specific parsers live in tensorrt_llm.llmapi.reasoning; importing a # module runs its register_reasoning_parser decorators. It is imported by # module rather than by name: a model module imported first imports this one, diff --git a/tensorrt_llm/serve/chat_utils.py b/tensorrt_llm/serve/chat_utils.py index 75f30edbf863..512e54d458e5 100644 --- a/tensorrt_llm/serve/chat_utils.py +++ b/tensorrt_llm/serve/chat_utils.py @@ -1,4 +1,9 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + import json +import secrets +import string import uuid from functools import lru_cache, partial from pathlib import Path @@ -23,7 +28,8 @@ from tensorrt_llm.inputs.model_config import resolve_top_level_model_type from tensorrt_llm.inputs.multimodal import MultimodalServerConfig from tensorrt_llm.inputs.registry import MULTIMODAL_PLACEHOLDER_REGISTRY -from tensorrt_llm.inputs.utils import apply_mm_placeholders +from tensorrt_llm.inputs.utils import (NATIVE_MISTRAL_MODEL_TYPES, + apply_mm_placeholders) from tensorrt_llm.logger import logger @@ -224,11 +230,13 @@ def parse_chat_message_content_parts( role: str, parts: Iterable[ChatCompletionContentPartParam], mm_data_tracker: MultimodalDataTracker, + keep_content_parts: bool = False, ) -> ConversationMessage: """Parse multiple parts of a chat message. Builds both the flattened text (`content`) for backward compatibility and an ordered `content_parts` list that preserves the interleaved positions of text and media items. + `content_parts` is kept for text-only messages too when `keep_content_parts` is set. """ text_parts: list[str] = [] media_parts: list[MultimodalData] = [] @@ -256,7 +264,7 @@ def parse_chat_message_content_parts( media=media_parts) # Only include content_parts when media is present (to preserve # interleaved ordering for multimodal dispatch). - if media_parts: + if media_parts or keep_content_parts: result["content_parts"] = content_parts return result @@ -265,10 +273,22 @@ def parse_chat_message_content( message: ChatCompletionMessageParam, mm_data_tracker: MultimodalDataTracker, lenient_tool_call_arguments: bool = False, - keep_message_tools: bool = False) -> ConversationMessage: - """Parse the content of a chat message.""" + keep_message_tools: bool = False, + keep_text_parts: bool = False, + raw_tool_call_arguments: bool = False) -> ConversationMessage: + """Parse the content of a chat message. + + With `keep_text_parts`, a content list keeps its text parts as separate + `content_parts` entries even without media, for renderers that encode each + part on its own rather than the newline-joined `content`. + + With `raw_tool_call_arguments`, an assistant tool call keeps its + `function.arguments` as the client sent them, for a renderer that parses + them itself. + """ role = message["role"] content = message.get("content") + is_part_list = content is not None and not isinstance(content, str) if content is None: content = [] @@ -281,11 +301,13 @@ def parse_chat_message_content( role, content, mm_data_tracker, + keep_content_parts=keep_text_parts and is_part_list, ) if role == "assistant": result.update( _parse_assistant_message_content(message, - lenient_tool_call_arguments)) + lenient_tool_call_arguments, + raw_tool_call_arguments)) elif role == "tool": result.update(_parse_tool_message_content(message)) elif keep_message_tools and role == "system" and message.get("tools"): @@ -357,11 +379,17 @@ def _validate_fallback_tool_calls( def _normalize_tool_call_arguments(index: int, item: Any, - lenient_json: bool = False - ) -> dict[str, Any]: - """Normalize `function.arguments` to the internal dict form.""" + lenient_json: bool = False, + keep_raw: bool = False) -> dict[str, Any]: + """Normalize `function.arguments` to the internal dict form. + + `keep_raw` leaves the arguments as given, for a renderer that parses them + itself (mistral-common); `lenient_json` keeps an unparsable string as is. + """ item = dict(item) item["function"] = dict(item["function"]) + if keep_raw: + return item arguments = item["function"].get("arguments") if arguments is None: @@ -391,9 +419,9 @@ def _normalize_tool_call_arguments(index: int, return item -def _parse_fallback_tool_calls( - tool_calls: list[Any], - lenient_json: bool = False) -> list[dict[str, Any]]: +def _parse_fallback_tool_calls(tool_calls: list[Any], + lenient_json: bool = False, + keep_raw: bool = False) -> list[dict[str, Any]]: """Parse raw tool-call lists accepted only by the tau2-bench fallback path. `openai_server.py` first attempts strict OpenAI request validation. Some tau2-bench requests @@ -405,15 +433,15 @@ def _parse_fallback_tool_calls( """ tool_calls = _validate_fallback_tool_calls(tool_calls) return [ - _normalize_tool_call_arguments(index, item, lenient_json) + _normalize_tool_call_arguments(index, item, lenient_json, keep_raw) for index, item in enumerate(tool_calls) ] # Adapted from: https://github.com/vllm-project/vllm/blob/4574d48bab9c4e38b7c0a830eeefc8f0980e8c58/vllm/entrypoints/chat_utils.py#L1406 def _parse_assistant_message_content(message: Dict[str, Any], - lenient_json: bool = False - ) -> Dict[str, Any]: + lenient_json: bool = False, + keep_raw: bool = False) -> Dict[str, Any]: result = {} # Include reasoning if present for interleaved thinking. reasoning_content = message.get("reasoning") @@ -426,7 +454,7 @@ def _parse_assistant_message_content(message: Dict[str, Any], if tool_calls is not None: if isinstance(tool_calls, list): result["tool_calls"] = _parse_fallback_tool_calls( - tool_calls, lenient_json) + tool_calls, lenient_json, keep_raw) else: # The strict parse path delivers tool_calls as a single-use Pydantic `ValidatorIterator` # of already-validated OpenAI tool calls, so only materialize and normalize arguments. @@ -436,7 +464,8 @@ def _parse_assistant_message_content(message: Dict[str, Any], # surfaces from the `list()` above, so the raw-body fallback is unchanged. message["tool_calls"] = tool_calls result["tool_calls"] = [ - _normalize_tool_call_arguments(index, item, lenient_json) + _normalize_tool_call_arguments(index, item, lenient_json, + keep_raw) for index, item in enumerate(tool_calls) ] @@ -518,11 +547,17 @@ def parse_chat_messages_coroutines( for msg in messages: # kimi_k3's reference renderer keeps unparsable tool-call argument # strings verbatim; other templates expect the strict dict contract. + # mistral-common (the Mistral-native renderer) encodes each text part + # of a content list on its own, and parses a history tool call's + # arguments itself. + native_mistral = model_type in NATIVE_MISTRAL_MODEL_TYPES parsed_msg = parse_chat_message_content( msg, mm_data_tracker, lenient_tool_call_arguments=(model_type == "kimi_k3"), - keep_message_tools=(model_type == "kimi_k3")) + keep_message_tools=(model_type == "kimi_k3"), + keep_text_parts=native_mistral, + raw_tool_call_arguments=native_mistral) conversation.append(parsed_msg) # Track placeholders added for this message only. @@ -560,6 +595,10 @@ def parse_chat_messages_coroutines( def make_tool_call_id(id_type: str = "random", func_name=None, idx=None): if id_type == "kimi_k2": return f"functions.{func_name}:{idx}" + elif id_type == "mistral": + # The ids mistral-common writes: nine ASCII letters or digits. + alphabet = string.ascii_letters + string.digits + return "".join(secrets.choice(alphabet) for _ in range(9)) else: # by default return random return f"chatcmpl-tool-{uuid.uuid4().hex}" diff --git a/tensorrt_llm/serve/extensions/__init__.py b/tensorrt_llm/serve/extensions/__init__.py index 9388bc5b7cee..a54490b5d01a 100644 --- a/tensorrt_llm/serve/extensions/__init__.py +++ b/tensorrt_llm/serve/extensions/__init__.py @@ -25,6 +25,14 @@ turn imports the registry module. Nothing here may import ``openai_server``. """ -from . import deepseek_v41, gpt_oss, kimi_k3, musespark, nemotron +from . import deepseek_v41, gpt_oss, kimi_k3, mistral_large_4, mistral_native, musespark, nemotron -__all__ = ["deepseek_v41", "gpt_oss", "kimi_k3", "musespark", "nemotron"] +__all__ = [ + "deepseek_v41", + "gpt_oss", + "kimi_k3", + "mistral_large_4", + "mistral_native", + "musespark", + "nemotron", +] diff --git a/tensorrt_llm/serve/extensions/mistral_large_4.py b/tensorrt_llm/serve/extensions/mistral_large_4.py new file mode 100644 index 000000000000..3e4b91bf66e6 --- /dev/null +++ b/tensorrt_llm/serve/extensions/mistral_large_4.py @@ -0,0 +1,120 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Serving extension for Mistral Large 4 checkpoints (``model_type == "mistral_large_4"``). + +Mistral Large 4 follows the Mistral-native chat rules (``mistral_native``). On top of +them, mistral-common builds the grammar that forces a required or named tool call. +""" + +from typing import Any, Dict, Optional + +from tensorrt_llm.serve.extensions.mistral_native import MistralNativeServingExtension +from tensorrt_llm.serve.openai_protocol import ( + ChatCompletionNamedToolChoiceParam, + ChatCompletionRequest, +) +from tensorrt_llm.serve.serving_extensions import RequiredToolChoiceMode, register_serving_extension + + +def native_tool_guided_decoding(request, tokenizer): + """Constrain a required or named call using mistral-common's native grammar.""" + from tensorrt_llm.sampling_params import GuidedDecodingParams + + choice = request.tool_choice + parallel_tool_calls = choice == "required" and request.parallel_tool_calls is not False + if not isinstance(choice, str): + name = choice.function.name + if not any(tool.function.name == name for tool in request.tools or []): + raise ValueError(f"Named tool {name!r} is not declared in tools.") + choice = choice.model_dump() + if not hasattr(tokenizer, "llguidance_tool_grammar"): + tokenizer = tokenizer.tokenizer + grammar = tokenizer.llguidance_tool_grammar( + [tool.model_dump(exclude_none=True) for tool in request.tools or []], + choice, + parallel_tool_calls, + ) + return GuidedDecodingParams(grammar=grammar) + + +@register_serving_extension( + model_types=("mistral_large_4",), reasoning_parsers=("mistral_large_4",) +) +class MistralLarge4ServingExtension(MistralNativeServingExtension): + """Mistral Large 4's hooks on top of the Mistral-native chat rules. + + It adds the output-budget check, the forced tool-call grammar, and where + structured outputs start. + """ + + def validate_chat_request( + self, request: ChatCompletionRequest, *, max_seq_len: Optional[int] + ) -> None: + """Refuse an output budget that alone exceeds the engine's maximum sequence length.""" + requested_tokens = request.max_completion_tokens + if ( + max_seq_len is not None + and requested_tokens is not None + and requested_tokens > max_seq_len + ): + raise ValueError( + f"Requested {requested_tokens} output tokens exceeds the " + f"maximum context length of {max_seq_len} tokens." + ) + + def required_tool_choice_mode(self) -> RequiredToolChoiceMode: + """Required calls are forced by mistral-common's tool grammar.""" + return "native" + + def forced_tool_call_decoding( + self, + request: ChatCompletionRequest, + tokenizer: Any, + *, + tool_parser: Optional[str], + guided_decoding_backend: Optional[str], + ): + """mistral-common's grammar for a required or named call. + + It applies when the server parses Mistral tool calls + (``--tool_parser mistral_large_4``); with another parser a named call takes + the generic path and a required call is refused. The grammar is an + llguidance grammar, so the server must run that backend. + """ + if tool_parser != "mistral_large_4": + return None + if request.tool_choice != "required" and not isinstance( + request.tool_choice, ChatCompletionNamedToolChoiceParam + ): + return None + if guided_decoding_backend != "llguidance": + raise ValueError( + "Mistral required and named tool calls require guided_decoding_backend='llguidance'." + ) + return native_tool_guided_decoding(request, tokenizer) + + def structured_output_format( + self, content: Dict[str, Any], chat_template_kwargs: Optional[Dict[str, Any]] + ) -> Optional[Dict[str, Any]]: + """Apply the raw grammar, with no structural tag around it. + + A structural tag cannot open a ``[THINK]`` block before the constraint: + ``[THINK]`` and ``[/THINK]`` are control tokens, which the + Mistral-Large-4 xgrammar vocabulary holds as empty strings. Under + xgrammar the raw grammar constrains the response from the first + generated token, so it carries no reasoning; under llguidance a JSON + response may open with a reasoning section. + """ + return None diff --git a/tensorrt_llm/serve/extensions/mistral_native.py b/tensorrt_llm/serve/extensions/mistral_native.py new file mode 100644 index 000000000000..8542395e3c81 --- /dev/null +++ b/tensorrt_llm/serve/extensions/mistral_native.py @@ -0,0 +1,78 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Serving extension for Mistral-native checkpoints (``model_type == "mistral_common"``). + +The Mistral-native format (``checkpoint_format`` ``mistral`` or ``mistral_large_3``) +renders chats with mistral-common, not a Jinja template. mistral-common takes the +reasoning effort and the response format as model settings (Tekken v15 and later), +and counts each image's tokens from the decoded image. +""" + +from typing import Any + +from tensorrt_llm.serve.openai_protocol import ChatCompletionRequest +from tensorrt_llm.serve.serving_extensions import ServingExtension, register_serving_extension + + +def _reasoning_effort(value: Any) -> Any: + """The effort name of an enum member, or ``value`` itself. + + mistral-common checks the effort when the chat is rendered, against the efforts + the checkpoint's tekken.json declares. + """ + return getattr(value, "value", value) + + +@register_serving_extension(model_types=("mistral_common",)) +class MistralNativeServingExtension(ServingExtension): + """Chat-request rules for the mistral-common renderer.""" + + def render_needs_media(self) -> bool: + """mistral-common derives each image's token count from the decoded image.""" + return True + + def apply_chat_extensions(self, request: ChatCompletionRequest) -> None: + """Validate the template controls and resolve the model settings. + + A request-level ``chat_template``, top-level or in ``chat_template_kwargs``, + cannot apply to mistral-common and is rejected. An explicit top-level + ``reasoning_effort`` is passed to the renderer + through ``chat_template_kwargs``; a ``reasoning_effort`` already in + ``chat_template_kwargs`` is accepted as well, and the two must agree when both + are given. Unset, the checkpoint's default effort applies; a checkpoint that + declares no efforts rejects one. A ``json_object`` or ``json_schema`` response + format is passed on as well, for the renderer to state in the prompt; it + constrains decoding either way. + """ + kwargs = dict(request.chat_template_kwargs or {}) + if request.chat_template is not None or kwargs.get("chat_template") is not None: + raise ValueError("chat_template is not supported for Mistral-native checkpoints.") + from_kwargs = kwargs.get("reasoning_effort") + if from_kwargs is not None: + from_kwargs = _reasoning_effort(from_kwargs) + if "reasoning_effort" in request.model_fields_set and request.reasoning_effort is not None: + effort = _reasoning_effort(request.reasoning_effort) + if from_kwargs is not None and from_kwargs != effort: + raise ValueError( + f"reasoning_effort={effort!r} conflicts with " + f"chat_template_kwargs['reasoning_effort']={from_kwargs!r}." + ) + kwargs["reasoning_effort"] = effort + kwargs.pop("response_format", None) + response_format = request.response_format + if response_format is not None and response_format.type in ("json_object", "json_schema"): + kwargs["response_format"] = response_format.model_dump(exclude_none=True) + if kwargs != (request.chat_template_kwargs or {}): + request.chat_template_kwargs = kwargs diff --git a/tensorrt_llm/serve/openai_protocol.py b/tensorrt_llm/serve/openai_protocol.py index 9806bf3c977a..d4fdab51d6ae 100644 --- a/tensorrt_llm/serve/openai_protocol.py +++ b/tensorrt_llm/serve/openai_protocol.py @@ -1366,6 +1366,11 @@ class ChatCompletionRequest(OpenAIBaseModel): "This is a parameter used by chat template in tokenizer config of the " "model."), ) + continue_final_message: bool = Field( + default=False, + description="Continue the final assistant message without closing it. " + "Cannot be combined with add_generation_prompt.", + ) add_special_tokens: bool = Field( default=False, description=( @@ -1550,6 +1555,20 @@ def check_tool_choice(cls, data): "When using `tool_choice`, `tools` must be set.") return data + @model_validator(mode="after") + def check_continue_final_message(self): + if "continue_final_message" in self.model_fields_set: + if self.continue_final_message and self.add_generation_prompt: + raise ValueError( + "Cannot set both add_generation_prompt and continue_final_message to True." + ) + self.chat_template_kwargs = { + **(self.chat_template_kwargs or {}), + "continue_final_message": + self.continue_final_message, + } + return self + @model_validator(mode="before") @classmethod def check_message_tools_role(cls, data): diff --git a/tensorrt_llm/serve/openai_server.py b/tensorrt_llm/serve/openai_server.py index b43f511556db..a465e67a9d5b 100644 --- a/tensorrt_llm/serve/openai_server.py +++ b/tensorrt_llm/serve/openai_server.py @@ -141,7 +141,7 @@ mount_render_endpoints, prepare_chat_request, render_chat, render_chat_for_length, - render_conversation) + render_conversation_with_media) from tensorrt_llm.serve.reranker import (Qwen3RerankInputBuilder, rerank_probability) from tensorrt_llm.serve.responses_utils import (ClosingStreamingResponse, @@ -1056,6 +1056,12 @@ def _init_llm(self, chat_template: Optional[str] = None): self.model_config = None self.chat_template = load_chat_template(chat_template) + if (getattr(self.model_config, "model_type", None) == "mistral_large_4" + and self.chat_template): + logger.warning( + "Mistral Large 4 renders chats with mistral-common; ignoring " + "the server chat template.") + self.chat_template = None # Enable response storage for Responses API self.enable_store = (len( @@ -1089,6 +1095,8 @@ def _init_llm(self, chat_template: Optional[str] = None): self.tool_call_id_type = "kimi_k2" elif self.model_config.model_type == "deepseek_v32": self.tool_call_id_type = "deepseek_v32" + elif self.model_config.model_type == "mistral_large_4": + self.tool_call_id_type = "mistral" if self.generator.args.return_perf_metrics: set_prometheus_multiproc_dir() @@ -2310,6 +2318,10 @@ async def chat_stream_generator( extension = render_resources.extension tool_dicts = prepare_chat_request(request, render_resources).tool_dicts + extension.validate_chat_request(request, + max_seq_len=getattr( + self.generator.args, + "max_seq_len", None)) required_mode = extension.required_tool_choice_mode() required_tool_grammar = (request.tool_choice == "required" and required_mode == "grammar") @@ -2333,6 +2345,30 @@ async def chat_stream_generator( guided_decoding_backend, tool_parser=self.tool_parser, reasoning_parser=self.generator.args.reasoning_parser) + # A model whose own grammar forces a required or named call takes + # it here, in place of the begin prefix and the required-tool modes + # below. + forced_tool_decoding = None + if request.tool_choice == "required" or isinstance( + request.tool_choice, ChatCompletionNamedToolChoiceParam): + forced_tool_decoding = extension.forced_tool_call_decoding( + request, + self.tokenizer, + tool_parser=self.tool_parser, + guided_decoding_backend=self.generator.args. + guided_decoding_backend) + if forced_tool_decoding is not None: + if sampling_params.guided_decoding is not None: + raise ValueError( + "Required or named tool calls cannot be combined with " + "response_format.") + sampling_params.guided_decoding = forced_tool_decoding + elif (request.tool_choice == "required" + and required_mode == "native"): + # Only the extension's grammar enforces "required" here; without + # it the request would silently degrade to "auto". + raise ValueError( + "tool_choice='required' is not supported for this model.") # Pre-render, so the mode may be unresolved. Safe because the # boundary lookup reads the parser class attributes, not the # branch `__init__` picked. @@ -2355,8 +2391,8 @@ async def chat_stream_generator( # the tool call. forced_tool_name = None forced_tool_begin_prefix = None - if isinstance(request.tool_choice, - ChatCompletionNamedToolChoiceParam): + if forced_tool_decoding is None and isinstance( + request.tool_choice, ChatCompletionNamedToolChoiceParam): if not request.tools: return self.create_error_response( message=("tool_choice with a named function requires " @@ -2565,24 +2601,33 @@ async def chat_stream_generator( prompt = request.prompt_token_ids else: # Render (and append the forced named-tool prefix, see - # ``_build_forced_tool_call_decoding``) while the media loads. + # ``_build_forced_tool_call_decoding``) while the media loads, + # or once it has loaded for a renderer that needs it. # Tokenization stays with the engine's input processor. - render_task = render_conversation( - render_resources, - conversation=conversation, - mm_placeholder_counts=mm_placeholder_counts, - add_generation_prompt=request.add_generation_prompt, - tools=tool_dicts, - documents=request.documents, - chat_template=request.chat_template, - chat_template_kwargs=request.chat_template_kwargs, - injected_chat_template_kwargs=request. - injected_chat_template_kwargs, - forced_prefix=forced_tool_begin_prefix, - ) - rendered, (mm_data, mm_embeddings) = await asyncio.gather( - render_task, mm_coroutines) + rendered, ( + mm_data, + mm_embeddings) = await render_conversation_with_media( + render_resources, + mm_coroutines, + conversation=conversation, + mm_placeholder_counts=mm_placeholder_counts, + add_generation_prompt=request.add_generation_prompt, + tools=tool_dicts, + documents=request.documents, + chat_template=request.chat_template, + chat_template_kwargs=request.chat_template_kwargs, + injected_chat_template_kwargs=request. + injected_chat_template_kwargs, + forced_prefix=forced_tool_begin_prefix, + ) if rendered.text is None: + if forced_tool_begin_prefix is not None: + # The prefix is text; appending it to rendered token + # ids would need a re-encode that loses control tokens. + raise ValueError( + "tool_choice with a named function is not " + "supported for this model, whose chat template " + "renders token ids.") prompt = rendered.token_ids else: prompt = rendered.text @@ -3143,20 +3188,21 @@ async def create_mm_embedding_response(promise: RequestOutput): if request.prompt_token_ids is not None: prompt = request.prompt_token_ids else: - render_task = render_conversation( - render_resources, - conversation=conversation, - mm_placeholder_counts=mm_placeholder_counts, - add_generation_prompt=request.add_generation_prompt, - tools=tool_dicts, - documents=request.documents, - chat_template=request.chat_template, - chat_template_kwargs=request.chat_template_kwargs, - injected_chat_template_kwargs=request. - injected_chat_template_kwargs, - ) - rendered, (mm_data, mm_embeddings) = await asyncio.gather( - render_task, mm_coroutines) + rendered, ( + mm_data, + mm_embeddings) = await render_conversation_with_media( + render_resources, + mm_coroutines, + conversation=conversation, + mm_placeholder_counts=mm_placeholder_counts, + add_generation_prompt=request.add_generation_prompt, + tools=tool_dicts, + documents=request.documents, + chat_template=request.chat_template, + chat_template_kwargs=request.chat_template_kwargs, + injected_chat_template_kwargs=request. + injected_chat_template_kwargs, + ) prompt = (rendered.text if rendered.text is not None else rendered.token_ids) prompt = prompt_inputs(prompt) diff --git a/tensorrt_llm/serve/postprocess_handlers.py b/tensorrt_llm/serve/postprocess_handlers.py index 7fa9dc9eadc6..2306ad2f9140 100644 --- a/tensorrt_llm/serve/postprocess_handlers.py +++ b/tensorrt_llm/serve/postprocess_handlers.py @@ -562,15 +562,19 @@ def yield_first_chat(num_tokens: int, finished=(output.finish_reason is not None)) if forced_tool and calls: - forced_name = _forced_call_name(calls, - forced_tool.function.name) - calls = calls[:1] - if calls[0].name: - calls[0].name = forced_name + # One chunk can contain several argument deltas for the first + # call. Keep those together while dropping subsequent calls. + calls = [call for call in calls if call.tool_index == 0] + for call in calls: + if call.name is not None: + call.name = _forced_call_name([call], + forced_tool.function.name) tool_calls = [] for call_item in calls: - # Tool call ID should be generated only once per tool call - if call_item.name: + # Tool call ID should be generated only once per tool call. A + # call may be named "" (mistral_large_4 keeps unnamed calls), so + # only an absent name marks an argument delta. + if call_item.name is not None: # First chunk: include ID and function name tool_call_id = make_tool_call_id( id_type=args.tool_call_id_type, @@ -767,6 +771,12 @@ def chat_response_post_processor( arguments=call.parameters)) for call in calls ] + if args.tool_call_id_type == "mistral": + # The streaming path draws every id type from + # make_tool_call_id; here only the Mistral format replaces the + # default random id. + for tool_call in tool_calls: + tool_call.id = make_tool_call_id(id_type="mistral") # Only the non-forced path builds ``message`` here; each forced # branch above builds its own. Keeping this outside the ``else`` # would clobber those and read ``tool_calls`` unbound. diff --git a/tensorrt_llm/serve/render/__init__.py b/tensorrt_llm/serve/render/__init__.py index 9729715f60b8..391750a838bc 100644 --- a/tensorrt_llm/serve/render/__init__.py +++ b/tensorrt_llm/serve/render/__init__.py @@ -28,6 +28,7 @@ render_chat, render_chat_for_length, render_conversation, + render_conversation_with_media, tokenize_rendered, ) from .fingerprint import ( @@ -73,6 +74,7 @@ "render_chat", "render_chat_for_length", "render_conversation", + "render_conversation_with_media", "render_endpoints_enabled", "tokenize_rendered", "worker_resolves_thinking", diff --git a/tensorrt_llm/serve/render/chat.py b/tensorrt_llm/serve/render/chat.py index b53e5b4bbacd..ebae1980cd70 100644 --- a/tensorrt_llm/serve/render/chat.py +++ b/tensorrt_llm/serve/render/chat.py @@ -33,7 +33,7 @@ import asyncio import dataclasses from dataclasses import dataclass -from typing import TYPE_CHECKING, Any, Callable, Dict, List, Optional +from typing import TYPE_CHECKING, Any, Awaitable, Callable, Dict, List, Optional, Tuple from pydantic import ValidationError @@ -104,8 +104,9 @@ def _template_kwargs( chat_template_kwargs: Optional[Dict[str, Any]], injected_chat_template_kwargs: Any, enable_tokenize: bool, + mm_data: Optional[Dict[str, Any]] = None, ) -> Dict[str, Any]: - return dict( + kwargs = dict( model_type=res.model_type or "", tokenizer=res.tokenizer, processor=res.processor, @@ -119,6 +120,9 @@ def _template_kwargs( injected_chat_template_kwargs=injected_chat_template_kwargs, enable_tokenize=enable_tokenize, ) + if mm_data is not None: + kwargs["mm_data"] = mm_data + return kwargs def tokenize_rendered( @@ -213,8 +217,13 @@ async def render_conversation( add_special_tokens: bool = False, truncate_prompt_tokens: Optional[int] = None, encode_rendered: Optional[Callable[[str, Any], List[int]]] = None, + mm_data: Optional[Dict[str, Any]] = None, ) -> RenderedPrompt: - """Render an already-parsed conversation without blocking the event loop.""" + """Render an already-parsed conversation without blocking the event loop. + + ``mm_data`` is the loaded media, for a renderer that needs it (see + ``ServingExtension.render_needs_media``); others render without it. + """ # A tokenizer-native template tokenizes segment by segment, so control-token strings inside # user or tool text stay text. Encoding the finished string would turn them into special # tokens, so keep the tokenizer's own ids when nothing is added after the template. @@ -236,6 +245,7 @@ async def render_conversation( chat_template_kwargs=chat_template_kwargs, injected_chat_template_kwargs=injected_chat_template_kwargs, enable_tokenize=native_ids, + mm_data=mm_data, ) ) if tokenize: @@ -261,6 +271,32 @@ async def render_conversation( ) +async def render_conversation_with_media( + res: RenderResources, mm_coroutines: Awaitable[Tuple[Any, Any]], **render_kwargs: Any +) -> Tuple[RenderedPrompt, Tuple[Any, Any]]: + """Render a parsed conversation and load its media. + + Returns ``(rendered, (mm_data, mm_embeddings))``. Rendering overlaps the media + loading, unless the model's renderer needs the loaded media (see + ``ServingExtension.render_needs_media``); it then renders afterwards, from the + loaded media, which rules out precomputed embeddings. ``render_kwargs`` are + those of :func:`render_conversation`. + """ + if not res.extension.render_needs_media(): + rendered, media = await asyncio.gather( + render_conversation(res, **render_kwargs), mm_coroutines + ) + return rendered, media + mm_data, mm_embeddings = await mm_coroutines + if mm_embeddings: + raise ValueError( + "This model renders its prompt from the loaded media, so it cannot take " + "multimodal embeddings." + ) + rendered = await render_conversation(res, mm_data=mm_data, **render_kwargs) + return rendered, (mm_data, mm_embeddings) + + _MEDIA_PART_MARKERS = ("image", "video", "audio") diff --git a/tensorrt_llm/serve/render/fingerprint.py b/tensorrt_llm/serve/render/fingerprint.py index e7becc932f67..bdcc085dbe55 100644 --- a/tensorrt_llm/serve/render/fingerprint.py +++ b/tensorrt_llm/serve/render/fingerprint.py @@ -136,7 +136,11 @@ def _tokenizer_digest(tokenizer: Any) -> Dict[str, Any]: and k in init_kwargs } ) - encoding = _encoding_digest(inner) + # A tokenizer that renders chats with its own code identifies its encoding itself + # (MistralTokenizer: its tokenizer file and mistral-common's version). + encoding = getattr(tokenizer, "encoding_identity", None) + if not isinstance(encoding, dict): + encoding = _encoding_digest(inner) info["encoding"] = encoding info["complete"] = encoding is not None return info diff --git a/tensorrt_llm/serve/resource_governor.py b/tensorrt_llm/serve/resource_governor.py index d924c8053e1b..90b27550dcc8 100644 --- a/tensorrt_llm/serve/resource_governor.py +++ b/tensorrt_llm/serve/resource_governor.py @@ -19,7 +19,6 @@ LLM/Proxy/Worker chain. """ -import asyncio import traceback from http import HTTPStatus from typing import Callable, List, Optional @@ -39,7 +38,7 @@ KVCacheTruncateTokensRequest, ensure_request_chat_template_allowed, ) -from tensorrt_llm.serve.render import RenderResources, render_conversation +from tensorrt_llm.serve.render import RenderResources, render_conversation_with_media class ResourceGovernor: @@ -144,8 +143,9 @@ async def _convert_messages( conversation, mm_coroutines, mm_placeholder_counts, _ = parse_chat_messages_coroutines( messages, self.model_config, None ) - render_task = render_conversation( + rendered, _ = await render_conversation_with_media( self._render_resources(), + mm_coroutines, conversation=conversation, mm_placeholder_counts=mm_placeholder_counts, add_generation_prompt=add_generation_prompt, @@ -155,7 +155,6 @@ async def _convert_messages( chat_template_kwargs=chat_template_kwargs, tokenize=True, ) - rendered, _ = await asyncio.gather(render_task, mm_coroutines) return rendered.token_ids async def _truncate_kv_cache(self, request: KVCacheTruncateRequest) -> Response: diff --git a/tensorrt_llm/serve/responses_utils.py b/tensorrt_llm/serve/responses_utils.py index 6f37bc51b27d..7fa54d86c88c 100644 --- a/tensorrt_llm/serve/responses_utils.py +++ b/tensorrt_llm/serve/responses_utils.py @@ -94,7 +94,8 @@ UCompletionRequest, UCompletionResponse, to_disaggregated_params) -from tensorrt_llm.serve.render import RenderResources, render_conversation +from tensorrt_llm.serve.render import (RenderResources, + render_conversation_with_media) from tensorrt_llm.serve.responses_web_search import is_web_search_tool from tensorrt_llm.serve.serving_extensions import normalize_reasoning_effort from tensorrt_llm.serve.tool_parser.base_tool_parser import ( @@ -1659,8 +1660,9 @@ async def _create_input_tokens( model_type=model_type, ) - render_task = render_conversation( + rendered, (mm_data, mm_embeddings) = await render_conversation_with_media( render_resources, + mm_coroutines, conversation=conversation, mm_placeholder_counts=mm_placeholder_counts, add_generation_prompt=True, @@ -1668,11 +1670,10 @@ async def _create_input_tokens( chat_template_kwargs=chat_template_kwargs or None, injected_chat_template_kwargs=reasoning_injected_chat_template_keys( request), - # Media is rendered to text for the model's input processor to expand. + # Media is rendered to text for the model's input processor to expand, + # unless the renderer tokenizes it itself. tokenize=not has_media, ) - rendered, (mm_data, - mm_embeddings) = await asyncio.gather(render_task, mm_coroutines) if not has_media: return rendered.token_ids @@ -1680,7 +1681,8 @@ async def _create_input_tokens( raise ValueError( "Passing 'multi_modal_data' and 'multi_modal_embeddings' at the " "same time is not supported.") - inputs = prompt_inputs(rendered.text) + inputs = prompt_inputs( + rendered.text if rendered.text is not None else rendered.token_ids) if mm_data: inputs["multi_modal_data"] = mm_data if mm_item_order: diff --git a/tensorrt_llm/serve/router.py b/tensorrt_llm/serve/router.py index e53762a6a7ee..f6ce2f13a2a4 100644 --- a/tensorrt_llm/serve/router.py +++ b/tensorrt_llm/serve/router.py @@ -1287,6 +1287,7 @@ def __init__(self, load_weight: float = 0.25, load_cap: float = float("inf"), enable_tokenization_cache: bool = False, + checkpoint_format: Optional[str] = None, **kwargs) -> None: super().__init__(server_role, servers, metadata_server_cfg, metadata_server, **kwargs) @@ -1296,7 +1297,8 @@ def __init__(self, tokenizer_dir, use_harmony=use_harmony, model_path=model_path, - enable_tokenization_cache=enable_tokenization_cache) + enable_tokenization_cache=enable_tokenization_cache, + checkpoint_format=checkpoint_format) self._init_load_balancing(servers, use_tokens) # TODO: use max_num_tokens? per server? self._max_batch_size = max_batch_size diff --git a/tensorrt_llm/serve/router_utils.py b/tensorrt_llm/serve/router_utils.py index f8a3a3a4aadc..4cb60e4fddfd 100644 --- a/tensorrt_llm/serve/router_utils.py +++ b/tensorrt_llm/serve/router_utils.py @@ -18,12 +18,14 @@ """ import os +import sys from collections import OrderedDict from dataclasses import dataclass, field from typing import Iterable, Optional, Union from tensorrt_llm.bindings.internal.batch_manager import BlockKey as _NativeBlockKey from tensorrt_llm.bindings.internal.batch_manager import BlockKeyHasher as _NativeBlockKeyHasher +from tensorrt_llm.checkpoint_format import CheckpointFormat from tensorrt_llm.inputs.prefix_token_cache import PrefixTokenCache, create_prefix_token_cache from tensorrt_llm.logger import logger from tensorrt_llm.runtime import kv_cache_hash @@ -70,6 +72,12 @@ OpenAIRequest = Union[CompletionRequest, ChatCompletionRequest, ResponsesRequest] BlockHash = Union[int, str] +# Checkpoint formats whose servers take the tokenizer, the input processor and the model +# configuration from the checkpoint's Mistral-native files (mistral-common's tekken.json and +# params.json), not from its Hugging Face files. +_MISTRAL_NATIVE_FORMATS = frozenset({CheckpointFormat.MISTRAL, CheckpointFormat.MISTRAL_LARGE_3}) +_MISTRAL_TOKENIZER_MODULE = "tensorrt_llm._torch.models.checkpoints.mistral.tokenizer" + __all__ = [ "KV_CACHE_HASH_ALGO_DEFAULT", "KV_CACHE_HASH_ALGO_V1", @@ -175,7 +183,15 @@ def _init_block_hashing( use_harmony: Optional[bool] = None, model_path: Optional[str] = None, enable_tokenization_cache: bool = False, + checkpoint_format: Optional[str] = None, ) -> None: + """Set up tokenization for block hashing. + + ``checkpoint_format`` is the servers' checkpoint format. For a Mistral-native one + (``mistral``, ``mistral_large_3``) the router loads the tokenizer and the rendering + resources from the checkpoint as the servers do (``RenderResources.load``); for any + other it loads the Hugging Face tokenizer. + """ env_tokens_per_block = os.environ.get("TRTLLM_KVCACHE_AWARE_ROUTER_HASH_TOKENS_PER_BLOCK") if env_tokens_per_block is not None: tokens_per_block = int(env_tokens_per_block) @@ -187,6 +203,7 @@ def _init_block_hashing( self._tokenizer_dir = tokenizer_dir self._model_path = model_path self._use_harmony = use_harmony + self._checkpoint_format = checkpoint_format self._tok_prefix_cache: OrderedDict = OrderedDict() self._enable_tokenization_cache = enable_tokenization_cache # id(tokenizer) -> (tokenizer, its prefix token cache or None) @@ -196,16 +213,27 @@ def _init_block_hashing( f"{' (auto, adopts worker)' if self._tpb_auto else ''}" f", custom_tokenizer={self._custom_tokenizer}" f", model_path={self._model_path}" + f", checkpoint_format={self._checkpoint_format}" f", use_harmony={self._use_harmony}" f", enable_tokenization_cache={self._enable_tokenization_cache}" ) + @property + def _mistral_native(self) -> bool: + """Whether the servers load a Mistral-native checkpoint (see ``_init_block_hashing``).""" + return getattr(self, "_checkpoint_format", None) in _MISTRAL_NATIVE_FORMATS + def _get_tokenizer(self, model: str): if model not in self._tokenizers: model_path = self._tokenizer_dir or model # Only an operator-configured tokenizer directory may run remote # tokenizer code; request model values are client-controlled. trust_remote_code = self._tokenizer_dir is not None + if self._mistral_native: + # The servers' own tokenizer wrapper (mistral-common's, for a Mistral-native + # checkpoint), which encodes as they do; its ``.tokenizer`` is not a + # Hugging Face tokenizer. Shared by every model name (see _render_key). + return self._render_resources(model).tokenizer if self._custom_tokenizer: from tensorrt_llm.tokenizer import load_custom_tokenizer @@ -220,7 +248,24 @@ def _get_tokenizer(self, model: str): tokenizer = load_hf_tokenizer(model_path, trust_remote_code=trust_remote_code) if tokenizer is None: raise ValueError(f"Could not load a tokenizer from {model_path!r}.") - self._tokenizers[model] = tokenizer.tokenizer + # load_hf_tokenizer imports mistral-common's wrapper before it returns one; + # importing it here for every other tokenizer would load the model packages. + mistral = sys.modules.get(_MISTRAL_TOKENIZER_MODULE) + if mistral is None or not isinstance(tokenizer, mistral.MistralTokenizer): + self._tokenizers[model] = tokenizer.tokenizer + else: + # A Mistral-native directory loads as mistral-common's wrapper, whose + # ``.tokenizer`` is not a Hugging Face tokenizer: keep the wrapper, as for + # a custom tokenizer. Without the format there is no renderer for its chats. + logger.warning( + f"Router reads {model_path!r} as a Mistral-native checkpoint but has no " + "Mistral-native checkpoint_format: it tokenizes completions as the " + "servers do and routes chats on no ids. Give it the servers' " + "checkpoint_format (mistral or mistral_large_3) to route chats on " + "their ids." + ) + self.__dict__.setdefault("_mistral_native_dirs", set()).add(model) + self._tokenizers[model] = tokenizer return self._tokenizers[model] def _prefix_token_cache(self, tokenizer) -> Optional[PrefixTokenCache]: @@ -289,7 +334,10 @@ def _length_chat_ids(self, request: ChatCompletionRequest) -> list[int]: """Routing ids from :func:`render_chat_for_length`; never forwarded to a worker. A request the request model rejects, or Harmony cannot convert, gets no ids. One the chat - template rejects raises. + template rejects raises, except for a Mistral-native checkpoint: mistral-common's + rejection also gets no ids, and the worker answers the request. So do the chats of a + Mistral-native directory read without the servers' checkpoint format, which this + router cannot render. """ try: return render_chat_for_length( @@ -300,11 +348,41 @@ def _length_chat_ids(self, request: ChatCompletionRequest) -> list[int]: except (UnsupportedRenderError, HarmonyConversionError) as error: logger.debug(f"Router has no length for the request ({error}); routing on no ids.") return [] + except ValueError as error: + if not self._mistral_native and request.model not in self.__dict__.get( + "_mistral_native_dirs", () + ): + raise + logger.debug(f"Router could not render the request ({error}); routing on no ids.") + return [] + + def _render_key(self, model: str) -> str: + """What ``model``'s rendering resources are cached under. + + A Mistral-native router loads the configured checkpoint whatever model name a + client sends, so it keys them by that checkpoint; any other router by the name. + """ + if self._mistral_native: + return self._model_path or self._tokenizer_dir or model + return model def _render_resources(self, model: str) -> RenderResources: """Rendering resources of this router for ``model``, built once.""" cache = self.__dict__.setdefault("_render_resources_cache", {}) - resources = cache.get(model) + key = self._render_key(model) + resources = cache.get(key) + if resources is None and self._mistral_native: + # What a server of this checkpoint loads: the tokenizer and input processor of + # the checkpoint format, and the model type from its native configuration. + resources = cache[key] = RenderResources.load( + key, + tokenizer=self._tokenizer_dir, + trust_remote_code=self._tokenizer_dir is not None, + custom_tokenizer=self._custom_tokenizer, + checkpoint_format=self._checkpoint_format, + # As for from_tokenizer: the worker polices request-level templates. + allow_request_chat_template=True, + ) if resources is None: try: model_type = self._get_model_type() @@ -331,7 +409,7 @@ def _render_resources(self, model: str) -> RenderResources: use_harmony=use_harmony, custom_tokenizer=self._custom_tokenizer, ) - cache[model] = resources + cache[key] = resources return resources def _write_back_decision( @@ -367,9 +445,10 @@ def _write_back_decision( if not remote: return WriteBackDecision(allowed=False) cache = self.__dict__.setdefault("_render_fingerprint_cache", {}) - local = cache.get(model) + key = self._render_key(model) + local = cache.get(key) if local is None: - local = cache[model] = self._render_resources(model).fingerprint() + local = cache[key] = self._render_resources(model).fingerprint() mismatched: dict[str, str] = {} for server, fingerprint in zip(servers, remote): reason = fingerprint_mismatch(local, fingerprint) @@ -397,10 +476,13 @@ def _tokenize_chat(self, request: ChatCompletionRequest) -> list[int]: """Render a chat request with the shared core and decide on write-back.""" if request.prompt_token_ids is not None: return list(request.prompt_token_ids) + # Outside the fallbacks below: resources that cannot be loaded are a configuration + # error, not a request the router cannot render. + resources = self._render_resources(request.model) try: rendered = render_chat( request, - self._render_resources(request.model), + resources, tokenize=True, encode_rendered=self._make_encode_rendered(request), ) diff --git a/tensorrt_llm/serve/serving_extensions.py b/tensorrt_llm/serve/serving_extensions.py index dd0a72d4e017..b0d3cd4f42cd 100644 --- a/tensorrt_llm/serve/serving_extensions.py +++ b/tensorrt_llm/serve/serving_extensions.py @@ -37,10 +37,14 @@ from typing import TYPE_CHECKING, Any, Callable, Dict, Iterable, List, Literal, Optional if TYPE_CHECKING: + from tensorrt_llm.sampling_params import GuidedDecodingParams from tensorrt_llm.serve.openai_protocol import ChatCompletionRequest -RequiredToolChoiceMode = Literal["template", "grammar"] +# "template": the chat template enforces the call; "grammar": a mandatory +# structural tag built by the tool parser; "native": the extension's own +# grammar (see ``ServingExtension.forced_tool_call_decoding``). +RequiredToolChoiceMode = Literal["template", "grammar", "native"] class OutputMode(enum.Enum): @@ -123,6 +127,15 @@ def apply_chat_extensions(self, request) -> None: ``model_type`` this extension is registered for. """ + def validate_chat_request( + self, request: "ChatCompletionRequest", *, max_seq_len: Optional[int] + ) -> None: + """Refuse a chat request the model cannot serve within the engine's limits. + + Raises ``ValueError`` (a 400 on the routes). The default accepts every + request and leaves the limits to the engine. + """ + def required_tool_choice_mode(self) -> Optional[RequiredToolChoiceMode]: """Return the required-tool policy; boolean permission selects template enforcement.""" return "template" if self.allows_required_tool_choice() else None @@ -138,6 +151,35 @@ def normalize_reasoning_effort(self, effort: str) -> Optional[str]: """ return effort + def forced_tool_call_decoding( + self, + request: "ChatCompletionRequest", + tokenizer: Any, + *, + tool_parser: Optional[str], + guided_decoding_backend: Optional[str], + ) -> Optional["GuidedDecodingParams"]: + """Guided decoding that forces the call a required or named ``tool_choice`` asks for. + + ``None`` leaves the request to the generic paths: the named-tool begin + prefix and the mode :meth:`required_tool_choice_mode` returns. A model + whose own grammar describes its tool-call format returns that grammar + instead; it replaces both, and cannot be combined with a client-supplied + guided-decoding setting. Raises ``ValueError`` for a server configuration + the grammar cannot be built for. + """ + return None + + def render_needs_media(self) -> bool: + """Whether the prompt renderer needs the request's media loaded first. + + The default renders the chat template while the media loads. A renderer + that tokenizes the media itself (it derives the media token count from + the decoded item) is handed the loaded ``mm_data`` instead, and cannot + take precomputed multimodal embeddings. + """ + return False + def structured_output_format( self, content: Dict[str, Any], chat_template_kwargs: Optional[Dict[str, Any]] ) -> Optional[Dict[str, Any]]: diff --git a/tensorrt_llm/serve/tool_parser/mistral_large_4_parser.py b/tensorrt_llm/serve/tool_parser/mistral_large_4_parser.py new file mode 100644 index 000000000000..a093ffc7ec95 --- /dev/null +++ b/tensorrt_llm/serve/tool_parser/mistral_large_4_parser.py @@ -0,0 +1,274 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Tool-call parser for Mistral-Large-4 output. + +Mistral-Large-4 (mistral-common Tekken v15) writes a call as the control tokens +``[TOOL_CALLS]`` and ``[ARGS]`` around the function name, followed by the JSON arguments. +Parallel calls repeat the block, and nothing closes the last one but the end of sequence:: + + optional content[TOOL_CALLS]get_weather[ARGS]{"city": "Paris"}[TOOL_CALLS]... + +The rules follow vLLM's Mistral parser for v11+ tokenizers +(``vllm/parser/mistral.py``): + +- content is the text before the first ``[TOOL_CALLS]``, stripped once a call follows; +- a name runs to ``[ARGS]``, or to the first ``{`` when ``[ARGS]`` is missing, and is + returned even when it is empty or not a declared tool; +- the arguments are the first complete JSON value after the name, verbatim, and the text + after that value, up to the next ``[TOOL_CALLS]``, is dropped; arguments that never + complete are returned as their stripped raw text, and missing ones as ``{}``. + +The control tokens reach the parser only while the request keeps special tokens in its +decoded text (``needs_raw_special_tokens``), which also keeps the end-of-sequence +````; the parser drops it. + +Streaming returns each name once it is complete and then the arguments as they arrive, +and yields exactly the calls the one-shot parse finds, however the text is split. +""" + +import json +from typing import List + +from tensorrt_llm.logger import logger + +from ..openai_protocol import ChatCompletionToolsParam as Tool +from .base_tool_parser import BaseToolParser +from .core_types import StreamingParseResult, StructureInfo, ToolCallItem, _GetInfoFunc + +TOOL_CALLS = "[TOOL_CALLS]" +ARGS = "[ARGS]" +EOS = "" + +_CONTENT, _NAME, _ARGUMENTS, _END = "content", "name", "arguments", "end" +_DECODER = json.JSONDecoder() + + +def _partial_marker_length(text: str, markers) -> int: + """Length of the longest suffix of ``text`` that is a proper prefix of a marker.""" + best = 0 + for marker in markers: + for length in range(min(len(text), len(marker) - 1), best, -1): + if marker.startswith(text[-length:]): + best = length + break + return best + + +class _Arguments: + """The arguments of one call: raw text up to the end of the first JSON value.""" + + def __init__(self): + self.text = "" + self.sent = 0 + self.structured = False + self.complete = False + self._depth = 0 + self._in_string = False + self._escaped = False + + def feed(self, chunk: str) -> None: + if self.complete: + return + if not self.text: + chunk = chunk.lstrip() + if not chunk: + return + self.structured = chunk[0] in "{[" + if not self.structured: + self.text += chunk + return + for index, char in enumerate(chunk): + if self._in_string: + if self._escaped: + self._escaped = False + elif char == "\\": + self._escaped = True + elif char == '"': + self._in_string = False + elif char == '"': + self._in_string = True + elif char in "{[": + self._depth += 1 + elif char in "}]": + self._depth -= 1 + if self._depth == 0: + self.text += chunk[: index + 1] + self.complete = True + return + self.text += chunk + + def ready(self) -> str: + """The part that may be streamed while more text can still arrive. + + Trailing whitespace of an open value is held back, since a value that never + completes loses it. Unstructured arguments are held until the call ends. + """ + if not self.structured: + return "" + return self.text if self.complete else self.text.rstrip() + + def value(self) -> str: + """The arguments once the call has ended.""" + if not self.text: + return "{}" + if self.structured: + return self.ready() + stripped = self.text.strip() + try: + _, end = _DECODER.raw_decode(stripped) + except ValueError: + return stripped + return stripped[:end] + + +class MistralLarge4ToolParser(BaseToolParser): + """Parses ``[TOOL_CALLS]name[ARGS]{...}`` calls of Mistral-Large-4.""" + + needs_raw_special_tokens = True + streaming_matches_whole_parse = True + extracts_forced_tool_calls = True + + def __init__(self): + super().__init__() + self.bot_token = TOOL_CALLS + self._state = _CONTENT + self._held_space = "" + self._arguments = _Arguments() + + def has_tool_call(self, text: str) -> bool: + return TOOL_CALLS in text + + def detect_and_parse(self, text: str, tools: List[Tool]) -> StreamingParseResult: + result = MistralLarge4ToolParser()._feed(text, final=True) + calls: List[ToolCallItem] = [] + for item in result.calls: + if item.name is None: + calls[-1].parameters += item.parameters + else: + calls.append(item.model_copy()) + declared = self._get_tool_indices(tools) + for call in calls: + if call.name not in declared: + logger.warning(f"Model attempted to call undefined function: {call.name!r}") + normal_text = result.normal_text.strip() if calls else result.normal_text + return StreamingParseResult(normal_text=normal_text, calls=calls) + + def parse_streaming_increment(self, new_text: str, tools: List[Tool]) -> StreamingParseResult: + return self._feed(new_text, final=False) + + def finish(self, tools: List[Tool]) -> StreamingParseResult: + return self._feed("", final=True) + + def supports_structural_tag(self) -> bool: + # The xgrammar vocabulary of Mistral-Large-4 cannot produce control tokens as + # text, so a grammar could not open a call (vLLM builds Mistral tool + # grammars with mistral-common on llguidance instead). + return False + + def structure_info(self) -> _GetInfoFunc: + return lambda name: StructureInfo( + begin=f"{TOOL_CALLS}{name}{ARGS}", end="", trigger=TOOL_CALLS + ) + + def _feed(self, text: str, final: bool) -> StreamingParseResult: + self._buffer += text + content: List[str] = [] + calls: List[ToolCallItem] = [] + while self._step(content, calls, final): + pass + return StreamingParseResult(normal_text="".join(content), calls=calls) + + def _step(self, content: List[str], calls: List[ToolCallItem], final: bool) -> bool: + """Consume as much of the buffer as the state allows; False when it needs more.""" + buffer = self._buffer + if self._state == _END: + self._buffer = "" + return False + + if self._state == _CONTENT: + index = buffer.find(TOOL_CALLS) + if index == -1: + keep = 0 if final else _partial_marker_length(buffer, (TOOL_CALLS, EOS)) + self._content(buffer[: len(buffer) - keep], content) + self._buffer = buffer[len(buffer) - keep :] + if final: + content.append(self._held_space) + self._held_space = "" + return False + self._content(buffer[:index], content) + # Content before a call is stripped, so its trailing whitespace is dropped. + self._held_space = "" + self._buffer = buffer[index + len(TOOL_CALLS) :] + self._state = _NAME + return True + + if self._state == _NAME: + found = [(buffer.find(marker), marker) for marker in (ARGS, "{", TOOL_CALLS, EOS)] + found = [(index, marker) for index, marker in found if index != -1] + if not found: + if final: + self._open_call(buffer, calls) + self._close_call(calls) + self._state = _END + return False + index, marker = min(found) + self._open_call(buffer[:index], calls) + if marker == "{": + self._buffer = buffer[index:] + self._state = _ARGUMENTS + elif marker == ARGS: + self._buffer = buffer[index + len(ARGS) :] + self._state = _ARGUMENTS + else: + self._close_call(calls) + self._buffer = buffer[index + len(marker) :] + self._state = _NAME if marker == TOOL_CALLS else _END + return True + + found = [(buffer.find(marker), marker) for marker in (TOOL_CALLS, EOS)] + found = [(index, marker) for index, marker in found if index != -1] + if not found: + keep = 0 if final else _partial_marker_length(buffer, (TOOL_CALLS, EOS)) + self._arguments.feed(buffer[: len(buffer) - keep]) + self._buffer = buffer[len(buffer) - keep :] + if final: + self._close_call(calls) + self._state = _END + else: + self._send_arguments(self._arguments.ready(), calls) + return False + index, marker = min(found) + self._arguments.feed(buffer[:index]) + self._close_call(calls) + self._buffer = buffer[index + len(marker) :] + self._state = _NAME if marker == TOOL_CALLS else _END + return True + + def _content(self, text: str, content: List[str]) -> None: + """Pass content on, holding back trailing whitespace a call would strip.""" + combined = self._held_space + text.replace(EOS, "") + body = combined.rstrip() + self._held_space = combined[len(body) :] + if body: + content.append(body) + + def _open_call(self, name: str, calls: List[ToolCallItem]) -> None: + self.current_tool_id += 1 + self._arguments = _Arguments() + calls.append( + ToolCallItem(tool_index=self.current_tool_id, name=name.strip(), parameters="") + ) + + def _close_call(self, calls: List[ToolCallItem]) -> None: + self._send_arguments(self._arguments.value(), calls) + + def _send_arguments(self, ready: str, calls: List[ToolCallItem]) -> None: + delta = ready[self._arguments.sent :] + if not delta: + return + self._arguments.sent = len(ready) + last = calls[-1] if calls else None + if last is not None and last.tool_index == self.current_tool_id and last.name is None: + last.parameters += delta + else: + calls.append(ToolCallItem(tool_index=self.current_tool_id, parameters=delta)) diff --git a/tensorrt_llm/serve/tool_parser/tool_parser_factory.py b/tensorrt_llm/serve/tool_parser/tool_parser_factory.py index 272ae49036d5..1a46bc62c274 100644 --- a/tensorrt_llm/serve/tool_parser/tool_parser_factory.py +++ b/tensorrt_llm/serve/tool_parser/tool_parser_factory.py @@ -18,6 +18,7 @@ from .kimi_k3_tool_parser import KimiK3ToolParser from .minimax_m2_parser import MiniMaxM2ToolParser from .minimax_m3_parser import MiniMaxM3ToolParser +from .mistral_large_4_parser import MistralLarge4ToolParser from .musespark_parser import MuseSparkToolParser from .poolside_v1_parser import PoolsideV1ToolParser from .qwen3_coder_parser import Qwen3CoderToolParser @@ -46,6 +47,7 @@ "gemma4_text": "gemma4", "laguna": "poolside_v1", "minimax_m3_vl": "minimax_m3", + "mistral_large_4": "mistral_large_4", "muse_spark": "musespark", # Nemotron 3.5 Super VL. Its chat template instructs the model to emit # the Qwen3-Coder XML shape, @@ -83,6 +85,7 @@ class ToolParserFactory: "glm47": Glm47ToolParser, "minimax_m2": MiniMaxM2ToolParser, "minimax_m3": MiniMaxM3ToolParser, + "mistral_large_4": MistralLarge4ToolParser, "musespark": MuseSparkToolParser, "poolside_v1": PoolsideV1ToolParser, } diff --git a/tensorrt_llm/tokenizer/tokenizer.py b/tensorrt_llm/tokenizer/tokenizer.py index 8d884a2b02bb..f1316605cf74 100644 --- a/tensorrt_llm/tokenizer/tokenizer.py +++ b/tensorrt_llm/tokenizer/tokenizer.py @@ -635,6 +635,42 @@ def _decode_pending_tail(self, last_decoded_id: Optional[int], return self.tokenizer.decode(pending_ids, **kwargs) +def _load_mistral_common_tokenizer( + model_dir: Union[str, Path], + trust_remote_code: bool = True, + use_fast: bool = True) -> Optional[TokenizerBase]: + """Load a Mistral-native checkpoint directory through ``MistralTokenizer``. + + A directory that ships mistral-common's ``tekken.json`` is Mistral-native + unless it is also a complete HF checkpoint, with both ``config.json`` and + ``tokenizer.json``. A native directory is read through the + ``mistral_common`` alias, exactly as ``custom_tokenizer="mistral_common"`` + would read it, even when it also ships HF tokenizer files, so that every + reader of the directory tokenizes as mistral-common does. AutoTokenizer + would convert a bare ``tekken.json`` into a fast tokenizer whose ids lack + the special-token offset and whose ``encode`` prepends no BOS. + + Args: + model_dir: The checkpoint or tokenizer directory. + trust_remote_code: Whether to trust remote code. + use_fast: Whether to use the fast tokenizer. + + Returns: + The ``MistralTokenizer`` of a Mistral-native directory, or None for any + other path, which keeps the AutoTokenizer route. + """ + model_dir = Path(model_dir) + if not (model_dir / "tekken.json").is_file(): + return None + if ((model_dir / "config.json").is_file() + and (model_dir / "tokenizer.json").is_file()): + return None + return load_custom_tokenizer("mistral_common", + str(model_dir), + trust_remote_code=trust_remote_code, + use_fast=use_fast) + + def tokenizer_factory(obj: Optional[Union[str, Path, PreTrainedTokenizerBase, TokenizerBase]] = None, **kwargs) -> Optional[TokenizerBase]: @@ -649,6 +685,11 @@ def tokenizer_factory(obj: Optional[Union[str, Path, PreTrainedTokenizerBase, 'use_fast': True, } default_kwargs.update(kwargs) + if (tokenizer := _load_mistral_common_tokenizer( + obj, + trust_remote_code=default_kwargs['trust_remote_code'], + use_fast=default_kwargs['use_fast'])) is not None: + return tokenizer return TransformersTokenizer.from_pretrained(obj, **default_kwargs) elif isinstance(obj, TokenizerBase): return obj @@ -766,10 +807,19 @@ def load_hf_tokenizer(model_dir: str, Returns: A TransformersTokenizer object if the tokenizer is loaded successfully. + A Mistral-native directory (tekken.json, and not both config.json and + tokenizer.json) loads as MistralTokenizer, a TransformersTokenizer + subclass. ''' _ensure_gpt2_bytes_to_unicode_compat() + if (tokenizer := + _load_mistral_common_tokenizer(model_dir, + trust_remote_code=trust_remote_code, + use_fast=use_fast)) is not None: + return tokenizer + try: tokenizer = TransformersTokenizer.from_pretrained( model_dir, diff --git a/tensorrt_llm/usage/architecture_allowlist.py b/tensorrt_llm/usage/architecture_allowlist.py index 3f0cfd8da5db..6f918497bc71 100644 --- a/tensorrt_llm/usage/architecture_allowlist.py +++ b/tensorrt_llm/usage/architecture_allowlist.py @@ -87,6 +87,7 @@ "MistralForCausalLM", "MistralLarge3EagleForCausalLM", "MistralLarge3ForCausalLM", + "MistralLarge4ForCausalLM", "MixtralForCausalLM", "MllamaForConditionalGeneration", "MuseSparkForConditionalGeneration", diff --git a/tests/integration/defs/accuracy/accuracy_core.py b/tests/integration/defs/accuracy/accuracy_core.py index 385e21ef0496..654ea5e91936 100644 --- a/tests/integration/defs/accuracy/accuracy_core.py +++ b/tests/integration/defs/accuracy/accuracy_core.py @@ -371,6 +371,10 @@ def get_hypothesis_testing_params(self, self.HIGHER_IS_BETTER), threshold=entry.get("threshold")) + def evaluator_num_samples(self, num_samples: int) -> int: + """The evaluator's num_samples for a total sample count.""" + return num_samples + def create_evaluator( self, num_samples: int, @@ -379,7 +383,8 @@ def create_evaluator( Creating it up front loads the dataset and evaluation harness while the GPUs are still idle; pass the result to `evaluate` as - `prepared_evaluator`. + `prepared_evaluator`. `num_samples` is the total sample count; the + evaluator receives `evaluator_num_samples(num_samples)`. """ assert self.EVALUATOR_CLS is not None @@ -388,7 +393,9 @@ def create_evaluator( evaluator_kwargs.update(self.EVALUATOR_KWARGS) if extra_evaluator_kwargs is not None: evaluator_kwargs.update(extra_evaluator_kwargs) - return self.EVALUATOR_CLS(num_samples=num_samples, **evaluator_kwargs) + return self.EVALUATOR_CLS( + num_samples=self.evaluator_num_samples(num_samples), + **evaluator_kwargs) def evaluate(self, llm: PyTorchLLM, @@ -440,19 +447,20 @@ def evaluate(self, if sampling_params.truncate_prompt_tokens is None: sampling_params.truncate_prompt_tokens = self.MAX_INPUT_LEN + evaluator_num_samples = self.evaluator_num_samples( + hypothesis_testing_params.num_samples) if prepared_evaluator is None: prepared_evaluator = self.create_evaluator( hypothesis_testing_params.num_samples, extra_evaluator_kwargs) elif (hasattr(prepared_evaluator, "num_samples") - and prepared_evaluator.num_samples - != hypothesis_testing_params.num_samples): + and prepared_evaluator.num_samples != evaluator_num_samples): # E.g. a full-size evaluator in INTEGRATION_TEST mode would run every # sample while the hypothesis test assumes one. Evaluators that do # not expose num_samples cannot be checked here. raise ValueError( f"prepared_evaluator has num_samples=" f"{prepared_evaluator.num_samples}, but this evaluation uses " - f"{hypothesis_testing_params.num_samples}") + f"{evaluator_num_samples}") evaluate_kwargs = {} if hasattr(self, 'EVALUATE_KWARGS'): evaluate_kwargs.update(self.EVALUATE_KWARGS) @@ -702,6 +710,17 @@ class MMMU(AccuracyTask): EVALUATE_KWARGS = dict(model_type=None, is_force_single_image=False) + # mmmu_val groups 30 subjects and lm-eval applies the evaluator's + # num_samples to each of them; references count questions in total. + NUM_SUBJECTS = 30 + + def evaluator_num_samples(self, num_samples: int) -> int: + # INTEGRATION_TEST asks for one sample: one question per subject. + assert num_samples == 1 or num_samples % self.NUM_SUBJECTS == 0, ( + f"MMMU num_samples={num_samples} must split evenly across " + f"{self.NUM_SUBJECTS} subjects") + return max(num_samples // self.NUM_SUBJECTS, 1) + class PassKeyRetrieval64k(AccuracyTask): DATASET = "passkey_retrieval_64k" diff --git a/tests/integration/defs/accuracy/modeling_v2/RULER.md b/tests/integration/defs/accuracy/modeling_v2/RULER.md new file mode 100644 index 000000000000..1436fbc23073 --- /dev/null +++ b/tests/integration/defs/accuracy/modeling_v2/RULER.md @@ -0,0 +1,104 @@ +# RULER data and references + +The 32K specialization test reads `datasets/ruler` under `LLM_MODELS_ROOT`. +`TRTLLM_RULER_DATA` can point to the same two JSONL files elsewhere. No network +access or external output files are needed during a test. + +Unlike GSM8K and MMMU, this RULER selection is not currently supplied by the +shared CI dataset mount. Provision it once from the v1 data archive, +`ruler-v1.tar.gz`, which is distributed separately from this repository: + +```bash +mkdir -p "${LLM_MODELS_ROOT}/datasets/ruler" +tar -xzf ruler-v1.tar.gz -C "${LLM_MODELS_ROOT}/datasets/ruler" +``` + +The archive SHA256 is +`7fbfa1e3ab6ad6a890545aee403faaf2e451eee69017c3037178f76262de02a2`. +It contains only `ruler_lite.jsonl` (144 samples) and `ruler_full.jsonl` +(1,800 samples), byte-identical to the inputs used for the existing baseline. +Tests check these files and the tokenizer before model initialization. + +The engineering baseline uses lm-eval 0.4.13 RULER generators and the checkpoint's Tekken tokenizer: +nine tasks at 4K, 8K, 16K and 32K. Full uses 50 questions per task/length; +quick uses the existing four-question selection, including the A/B cache +pairs. The v1 prompts include the chat template and answer prefix as token +IDs. Every v1 and long-v1 prompt was rendered at reasoning effort `none` and +starts with the model settings `{"reasoning_effort": "none"}`. The released +checkpoint does not declare that effort, and a chat request that asks for it +is rejected; the tests feed the stored IDs unchanged. +Keep these IDs, gold answers and generation budgets when comparing with +the recorded reference scores; regenerating a different selection needs its +own reference measurement. + +`references/ruler.yaml` stores the four full references and explicit floors +in the accuracy reference format. Quick checks NIAH/VT retrieval but does +not use full's aggregate floors. Both profiles call the upstream RULER scorer. + +The same LLM runs accuracy and cache checks. `accuracy_failures` reports +scoring failures; `reuse_failures` reports missing retrieval/cache witnesses. +Both contribute to the final verdict. The salted cold/partial/warm/isolated +panel and mixed batch require cache metrics; ordinary benchmark requests +only require valid, untruncated completions. A replay never replaces an +original answer in the accuracy score. Envelope evidence remains in the +specialization test's iteration statistics. + +GSM8K continues to use the shared CI dataset and `GSM8K.evaluate(llm)`. +Its NVFP4 floor of 90.43 is specified once in `references/gsm8k.yaml` and +applies to both the standalone CI case and the consolidated quick/full case. + +The consolidated profiles use the NVFP4 checkpoint with its native DSpark K=3. +`full` and `long_full` close that engine and start a fresh engine of the same +specialization with speculation disabled, for GSM8K only. Both arms must pass +90.43 and their absolute difference must not exceed 1.5 percentage points. +GSM8K retains its independently calibrated acceptance-length floor; +the 1M specialization uses the same topology's reference and frozen protocol. + +## The 1M profiles + +`long_lite` and `long_full` run on the 1M specialization. Both check the +cache witnesses over shared prefixes of 128K and 1M tokens, a completed +prompt of at least 1M tokens, every step within max_num_tokens, exact +niah / ruler_vt retrieval and GSM8K. They score a subset of the same frozen +rows: + +- `long_lite` (about 10 minutes): niah_single_1 and ruler_vt, two rows each at + 64K and 128K, and the 1M witness pair. Its means are reported only. Under + attention DP it stops at 128K: one rank prefills a 1M request alone. +- `long_full`: every task's ten rows at 64K and 128K, the 1M witness pair and + one ruler_vt row at 1M. + Its 64K and 128K means are gated on `references/ruler.yaml`; the 1M mean is + reported only. + +Both long profiles run speculation and retain per-request counts in +`ruler_predictions.jsonl`: actual prompt length, request ID, phase, cache hit, +drafted/accepted/rejected tokens, and counts by draft position. Acceptance length +is reported by length bin and phase, separately from GSM8K. It has no new +numeric floor without an independent long-context calibration. + +`spec-trace/rank-*.jsonl` records scheduled requests before each forward through +a test-only MPI worker observer. `spec_evidence.json` joins these records to +completed RULER requests and requires four-token verification on a long request, +successful warm/cached-prefix retrieval, successful retrieval after a rejection, +and a step with both long verification and a cold request longer than the context +token budget, so a cached-prefix tail alone cannot satisfy chunked prefill. Attention DP also +requires aligned rank-local live and dummy verification records. Missing +evidence is INCONCLUSIVE. The attention-DP `long_lite` exception above is +explicitly a 128K witness; only `dep4_1m/long_full` supplies its actual 1M E2E +evidence in the baseline matrix. Separate real-cache tests force acceptance +lengths 0/1/2/3 around the 1M page boundary for target and draft cache updates. + +Only `test_accuracy[long_lite-*_1m]` and `test_accuracy[long_full-*_1m]` read +this data. Provision it once from the long-v1 data archive, `ruler-long-v1.tar.gz` +(distributed separately, like v1), into the same directory, beside the v1 files: + +```bash +tar -xzf ruler-long-v1.tar.gz -C "${LLM_MODELS_ROOT}/datasets/ruler" +``` + +The archive (32,907,229 bytes) SHA256 is +`c9e57776357643639e5824ad144fca7e0b60341034ff331417809f78ed92de1f`. +It contains only `ruler_long.jsonl` (175,244,589 bytes, SHA256 +`5873d7313db5ee6ee31fcbf2b966647aa15fe7cbd39589018f9d5f6a5278eb7e`): +198 rows built from the frozen RULER sets of the reference runs, 10 per +task at 64K and 128K and 2 per task in the 1M bin. diff --git a/tests/integration/defs/accuracy/modeling_v2/llguidance_panel.py b/tests/integration/defs/accuracy/modeling_v2/llguidance_panel.py new file mode 100644 index 000000000000..95e0e6fd49e5 --- /dev/null +++ b/tests/integration/defs/accuracy/modeling_v2/llguidance_panel.py @@ -0,0 +1,240 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Guided decoding with llguidance over chat token ids a serving frontend renders itself. + +A frontend that renders Mistral chats itself hands the engine token ids with either a JSON +schema (``response_format``) or the Lark tool-call grammar mistral-common's grammar factory writes +(tool_choice auto / required / named). Each case sends the same, in one batch, and reads the +generated token ids. + +Output the grammar forbids fails the case: a value outside a one-value enum, a missing reasoning +section under reasoning effort high, a reply with no call under a forced tool choice, a call to +another tool. A reply cut off by ``max_tokens`` before its answer, and the model's choice not to call +under ``auto``, are INCONCLUSIVE. +""" + +import json +import time +from pathlib import Path + +from tensorrt_llm import SamplingParams +from tensorrt_llm.sampling_params import GuidedDecodingParams + +# A value no prompt asks for: only the grammar writes it. +_KEY = "zq-7f3k" +_KEY_SCHEMA = { + "type": "object", + "properties": {"value": {"type": "string", "enum": [_KEY]}}, + "required": ["value"], + "additionalProperties": False, +} +_CITY_SCHEMA = { + "type": "object", + "properties": {"city": {"type": "string"}, "population_millions": {"type": "number"}}, + "required": ["city", "population_millions"], + "additionalProperties": False, +} +_WEATHER_TOOL = { + "type": "function", + "function": { + "name": "get_weather", + "description": "Current weather in a city.", + "parameters": { + "type": "object", + "properties": {"city": {"type": "string"}, "unit": {"enum": ["c", "f"]}}, + "required": ["city", "unit"], + "additionalProperties": False, + }, + "strict": True, + }, +} +_RECORD_TOOL = { + "type": "function", + "function": { + "name": "record_key", + "description": "Store the session key.", + "parameters": { + "type": "object", + "properties": {"key": {"type": "string", "enum": [_KEY]}}, + "required": ["key"], + "additionalProperties": False, + }, + "strict": True, + }, +} +_WEATHER = "What is the weather in Paris right now, in celsius?" + +# name, user message, tools, reasoning effort, constraint (a JSON schema, or a tool choice), +# max_tokens. +_CASES = ( + ("json_low", "Name the capital of France. Answer in JSON.", None, "low", _KEY_SCHEMA, 2048), + ( + "json_after_reasoning", + "Which city has the largest metropolitan population? Answer in JSON.", + None, + "high", + _CITY_SCHEMA, + 8192, + ), + ("tool_required", "Store the session key now.", [_RECORD_TOOL], "low", "required", 2048), + ("tool_named", _WEATHER, [_WEATHER_TOOL, _RECORD_TOOL], "low", "record_key", 2048), + ("tool_auto", _WEATHER, [_WEATHER_TOOL], "high", "auto", 4096), +) + + +def _prompt(llm, message, tools, effort): + from mistral_common.protocol.instruct.request import ChatCompletionRequest + + request = ChatCompletionRequest.from_openai( + messages=[{"role": "user", "content": message}], tools=tools, reasoning_effort=effort + ) + return {"prompt_token_ids": llm.tokenizer.mistral.encode_chat_completion(request).tokens} + + +def _tool_grammar(llm, tools, choice): + from mistral_common.guidance.grammar_factory import GrammarFactory + from mistral_common.protocol.instruct.tool_calls import NamedToolChoice, Tool, ToolChoiceEnum + + factory = GrammarFactory(llm.tokenizer.mistral) + native_tools = [Tool.from_openai(tool) for tool in tools] + if choice in ("auto", "required"): + mode, force_tool_names = ToolChoiceEnum(choice), False + else: + # A named choice, as mistral-common's get_grammar_for_chat_request resolves it. + name = NamedToolChoice.model_validate( + {"type": "function", "function": {"name": choice}} + ).function.name + native_tools = [tool for tool in native_tools if tool.function.name == name] + mode, force_tool_names = ToolChoiceEnum.required, True + # A named choice is exactly one call, as the serving extension resolves it. + return factory.get_lark_from_jinja( + factory.select_jinja_template(), + mode=mode, + tools=native_tools, + json_schema=None, + parallel_tool_calls=choice in ("auto", "required"), + json_only=False, + force_tool_names=force_tool_names, + ) + + +def _split(llm, ids): + """The reasoning, content and calls of generated ids, or an error string.""" + control = llm.tokenizer.tokenizer.get_control_token + think, end_think = control("[THINK]"), control("[/THINK]") + tool_calls, args = control("[TOOL_CALLS]"), control("[ARGS]") + ids = [i for i in ids if i != llm.tokenizer.eos_token_id] + reasoning = None + if ids and ids[0] == think: + if end_think not in ids: + return None, "unterminated reasoning section" + end = ids.index(end_think) + reasoning, ids = ids[1:end], ids[end + 1 :] + if think in ids or end_think in ids: + return None, "reasoning marker outside a leading reasoning section" + starts = [n for n, i in enumerate(ids) if i == tool_calls] + content = ids[: starts[0]] if starts else ids + calls = [] + for start, stop in zip(starts, starts[1:] + [len(ids)]): + call = ids[start + 1 : stop] + if args not in call: + return None, "a call without [ARGS]" + cut = call.index(args) + # The tool grammar allows whitespace before the name, as the model writes it under a forced + # call to a tool the prompt does not ask for. + name = llm.tokenizer.decode(call[:cut]).strip() + calls.append((name, llm.tokenizer.decode(call[cut + 1 :]))) + decode = llm.tokenizer.decode + return { + "reasoning": None if reasoning is None else decode(reasoning), + "content": decode(content), + "calls": calls, + }, None + + +def _check(name, constraint, parts, finish_reason): + """FAIL, INCONCLUSIVE or None (pass) for one case.""" + cut = finish_reason == "length" + if isinstance(constraint, dict): + if name == "json_after_reasoning" and parts["reasoning"] is None: + return "FAIL: no reasoning section under reasoning effort high" + try: + answer = json.loads(parts["content"]) + except json.JSONDecodeError: + if cut: + return "INCONCLUSIVE: cut off by max_tokens before the JSON" + return f"FAIL: not JSON: {parts['content']!r}" + if constraint is _KEY_SCHEMA: + return None if answer == {"value": _KEY} else f"FAIL: outside the schema: {answer!r}" + ok = ( + isinstance(answer, dict) + and set(answer) == {"city", "population_millions"} + and isinstance(answer["city"], str) + and isinstance(answer["population_millions"], (int, float)) + ) + return None if ok else f"FAIL: outside the schema: {answer!r}" + if not parts["calls"]: + if constraint == "auto": + return "INCONCLUSIVE: no call under auto" + if cut: + return "INCONCLUSIVE: cut off by max_tokens before the forced call" + return f"FAIL: no call under tool_choice {constraint}" + for tool, arguments in parts["calls"]: + try: + arguments = json.loads(arguments) + except json.JSONDecodeError: + return f"FAIL: {tool} arguments are not JSON: {arguments!r}" + if constraint == "auto": + ok = ( + tool == "get_weather" + and isinstance(arguments, dict) + and set(arguments) == {"city", "unit"} + and isinstance(arguments["city"], str) + and arguments["unit"] in ("c", "f") + ) + else: + ok = tool == "record_key" and arguments == {"key": _KEY} + if not ok: + return f"FAIL: call outside the grammar: {tool} {arguments!r}" + return None + + +def evaluate(llm, out: Path): + started = time.monotonic() + prompts, params = [], [] + for name, message, tools, effort, constraint, max_tokens in _CASES: + prompts.append(_prompt(llm, message, tools, effort)) + guided = ( + GuidedDecodingParams(json=constraint) + if isinstance(constraint, dict) + else GuidedDecodingParams(grammar=_tool_grammar(llm, tools, constraint)) + ) + params.append(SamplingParams(max_tokens=max_tokens, temperature=0, guided_decoding=guided)) + outputs = llm.generate(prompts, params, use_tqdm=False) + failures = [] + with (out / "llguidance.jsonl").open("w") as stream: + for (name, _, _, _, constraint, _), output in zip(_CASES, outputs): + completion = output.outputs[0] + parts, error = _split(llm, list(completion.token_ids)) + verdict = ( + f"FAIL: {error}" + if error + else _check(name, constraint, parts, completion.finish_reason) + ) + if verdict: + failures.append(f"{verdict.split(':')[0]}: {name}: {verdict.split(': ', 1)[1]}") + stream.write( + json.dumps( + { + "name": name, + "finish_reason": completion.finish_reason, + "verdict": verdict or "PASS", + "parts": parts, + "text": llm.tokenizer.decode( + list(completion.token_ids), skip_special_tokens=False + ), + } + ) + + "\n" + ) + return {"seconds": time.monotonic() - started, "cases": len(_CASES), "failures": failures} diff --git a/tests/integration/defs/accuracy/modeling_v2/mistral_large4/__init__.py b/tests/integration/defs/accuracy/modeling_v2/mistral_large4/__init__.py new file mode 100644 index 000000000000..467079831e16 --- /dev/null +++ b/tests/integration/defs/accuracy/modeling_v2/mistral_large4/__init__.py @@ -0,0 +1,14 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/tests/integration/defs/accuracy/modeling_v2/mistral_large4/_common.py b/tests/integration/defs/accuracy/modeling_v2/mistral_large4/_common.py new file mode 100644 index 000000000000..9e744f343cb0 --- /dev/null +++ b/tests/integration/defs/accuracy/modeling_v2/mistral_large4/_common.py @@ -0,0 +1,700 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Shared harness for the mistral_large4 modeling_v2 accuracy targets. + +Each target's gate class is in ``/test_accuracy.py``. It names the target's +architecture and layout and lists its tests, which run the ``_test_*`` implementations here; +the layouts and the deployments are here too. + +Every test passes ``modeling_v2="require"`` to ``LLM``. Under ``"auto"`` a +configuration that missed a target's criteria -- or fell outside its bounds -- +would quietly fall back to the built-in implementation, pass, and report the +built-in's numbers as the target's. + +These targets are multi-rank, and the mode has to hold on every rank rather +than only on the one running the test body. The switch is an LLM API argument, +so it reaches the ranks the way the rest of the arguments do. +""" + +import contextlib +import json +import os +import threading +import time +from functools import partial +from pathlib import Path + +import pytest + +from tensorrt_llm import LLM, SamplingParams +from tensorrt_llm._torch.modeling_v2 import MODELING_V2_VALIDATE_ENV +from tensorrt_llm._utils import get_sm_version +from tensorrt_llm.llmapi import CudaGraphConfig, DSparkDecodingConfig, KvCacheConfig + +from ....conftest import llm_models_root +from ...accuracy_core import ( + GSM8K, + MMMU, + GPQADiamond, + LlmapiAccuracyTestHarness, + acceptance_length_from_iteration_stats, + assert_acceptance_length, +) +from .. import llguidance_panel, multimodal, ruler, spec_evidence +from .._env import scoped_env_overrides + +# Certification is per GPU architecture, and a receipt from another one says nothing here: +# each gate class runs only on its target's architecture. +skip_not_sm103 = pytest.mark.skipif( + get_sm_version() != 103, + reason="the mistral_large4 sm_103 modeling_v2 targets are certified on sm_103 only", +) +skip_not_sm107 = pytest.mark.skipif( + get_sm_version() != 107, + reason="the mistral_large4 sm_107 modeling_v2 targets are certified on sm_107 only", +) + + +# The eight-rank deployments: eight ranks over two nodes of four (one NVL72 segment). Each node +# sees four devices, so the gate is the MPI world the launcher started (one node of eight, +# unlaunched, counts its devices); the tests' four-device gate holds on every node. +_EIGHT_RANKS = pytest.mark.skip_less_mpi_world_size(8) + +# The engine bounds every target serves (per rank under attention DP): 16 sequences, unchunked +# steps of up to 32768 tokens (a whole 32K prompt in one step), CUDA graphs padded over batches +# 1-16. The certified recipe: max_seq_len = max_num_tokens = 32768, chunked prefill off. +_BOUNDS = dict( + max_batch_size=16, + max_num_tokens=32768, + max_seq_len=32768, + enable_chunked_prefill=False, + cuda_graph_config=CudaGraphConfig(max_batch_size=16, enable_padding=True), +) + +# The 1M envelope: the same target and topology at the checkpoint's own context +# (max_position_embeddings 1048576; per rank under attention DP), prefilled in chunked steps of +# up to max_num_tokens 32768. +_BOUNDS_1M = dict(_BOUNDS, max_seq_len=1048576, enable_chunked_prefill=True) + +# The latent KV cache at its default: the checkpoint's fp8_kv_cache makes it the fp8_ds_mla +# cache the reference implementation serves (656 B per token and owner layer, +# MistralLarge4CacheManager, a pool per rank under attention DP), read as kv_cache_quant_algo +# FP8; the checkpoint's quant_algo is MIXED_PRECISION, so the harness reads the +# MIXED_PRECISION / FP8 row. +_KV = KvCacheConfig(enable_block_reuse=False, free_gpu_memory_fraction=0.85) + + +# The profiles ``test_accuracy`` runs, and the ones ``test_accuracy_without_dspark`` runs. +PROFILES = ["quick", "full", "long_lite", "long_full"] +PROFILES_WITHOUT_DSPARK = ["full", "long_full"] + + +def pytest_generate_tests(metafunc): + """Run each gate test at every deployment its target serves. + + A deployment is the target's layout on four ranks of one node or on eight over two nodes of + four, at the 32K recipe or at the 1M envelope: ``tep4``, ``tep8``, ``tep4_1m``, ``tep8_1m``. + """ + if "deployment" not in metafunc.fixturenames: + return + metafunc.parametrize( + "deployment", + [ + pytest.param(name, marks=_EIGHT_RANKS) if name.split("_")[0].endswith("8") else name + for name in metafunc.cls.deployments() + ], + ) + + +@contextlib.contextmanager +def _iteration_coverage(llm: LLM, path: Path, coverage: dict): + """One stats consumer for raw evidence, rank coverage and this workload's AL.""" + stop = threading.Event() + errors, spec_stats = [], [] + llm.get_stats(timeout=0.1) + + def drain(stream): + for item in llm.get_stats(timeout=0.1): + row = json.loads(item) if isinstance(item, str) else item + stream.write(json.dumps(row) + "\n") + if row.get("specDecodingStats"): + spec_stats.append({"specDecodingStats": row["specDecodingStats"]}) + rank = row.get("attentionDpRank", 0 if not llm.args.enable_attention_dp else None) + stats = row.get("inflightBatchingStats") or {} + shape = coverage.setdefault(rank, {"context_tokens": 0, "decode_batch": 0}) + shape["context_tokens"] = max(shape["context_tokens"], stats.get("numCtxTokens", 0)) + shape["decode_batch"] = max(shape["decode_batch"], stats.get("numGenRequests", 0)) + stream.flush() + + def collect(): + try: + with path.open("w") as stream: + while not stop.is_set(): + drain(stream) + stop.wait(0.2) + drain(stream) + except Exception as error: + errors.append(str(error)) + + worker = threading.Thread(target=collect, name="accuracy-iterations", daemon=True) + worker.start() + try: + yield errors, spec_stats + finally: + stop.set() + worker.join(timeout=15) + assert not worker.is_alive(), "Iteration reader did not stop" + + +@contextlib.contextmanager +def _phase_timing(timings: dict, phase: str): + started = time.monotonic() + print(f"Accuracy phase {phase}: started", flush=True) + try: + yield + finally: + timings[phase] = time.monotonic() - started + print(f"Accuracy phase {phase}: {timings[phase]:.3f} s", flush=True) + + +class _Nvfp4Target(LlmapiAccuracyTestHarness): + """The gate of one target: every test runs at each deployment the target serves. + + A subclass names its layout and its parallel arguments at four and at eight ranks; + `pytest_generate_tests` runs each test at both, at the 32K recipe and at the 1M envelope. + """ + + MODEL_NAME = "mistralai/Mistral-Large-4-1T-A52B" + # The Mistral-native ModelOpt MIXED_PRECISION export as published (params.json, + # tekken.json, consolidated safetensors: NVFP4 routed experts, FP8 block-scale linears). + MODEL_PATH = f"{llm_models_root()}/Mistral-Large-4-1T-A52B-NVFP4" + KV = _KV + # The layout's short name, and its parallel arguments by rank count. + LAYOUT = "" + LAYOUTS = {} + + DRAFT_LENGTH = 3 + # MMMU's output budget: at reasoning effort low the answer follows a reasoning block, which + # the task yaml's 512 tokens cut off in about a third of the samples. + MMMU_OUTPUT_LEN = 8192 + # The allowed GSM8K gap between the two arms. + GSM8K_MAX_DELTA = 2 * 0.75 + + @classmethod + def deployments(cls): + """Every deployment the target serves, named ``[_1m]``.""" + return [f"{cls.LAYOUT}{ranks}{ctx}" for ctx in ("", "_1m") for ranks in sorted(cls.LAYOUTS)] + + @classmethod + def serving(cls, deployment): + """A deployment's parallel arguments and engine bounds.""" + name, _, ctx = deployment.partition("_") + layout = cls.LAYOUTS[int(name.removeprefix(cls.LAYOUT))] + return layout, (_BOUNDS_1M if ctx == "1m" else _BOUNDS) + + def _test_gsm8k(self, deployment): + """The identity accuracy gate, at the harness default protocol. + + That is the protocol the registered reference was set under, so the anchor and the + measurement read the same metric. The engine also checks its step contract on its + first forward. + """ + layout, bounds = self.serving(deployment) + task = GSM8K(self.MODEL_NAME) + with ( + scoped_env_overrides({MODELING_V2_VALIDATE_ENV: "1"}) as rank_env, + LLM( + self.MODEL_PATH, + checkpoint_format="mistral", + kv_cache_config=self.KV, + modeling_v2="require", + env_overrides=rank_env, + **layout, + **bounds, + ) as llm, + ): + task.evaluate(llm) + + # GSM8K with 24 exemplars: every prompt is 3152-5115 tokens, past index_topk, so the + # indexer's selection prunes context on every sample (5-shot prompts stay under 2048, + # where the engine skips the indexer). A key of its own in gsm8k.yaml, scored by the + # reference implementation on its fp8_ds_mla cache. + MODEL_NAME_24SHOT = "mistralai/Mistral-Large-4-1T-A52B-gsm8k-24shot" + + class GSM8K24Shot(GSM8K): + MAX_INPUT_LEN = 8192 + + def _test_gsm8k_24shot(self, deployment): + """24-shot GSM8K, where the indexer's selection prunes every prompt. + + `trtllm-eval gsm8k --num_fewshot 24 --max_input_length 8192`, otherwise the harness + protocol of the 5-shot gate. + """ + layout, bounds = self.serving(deployment) + task = self.GSM8K24Shot(self.MODEL_NAME_24SHOT) + with ( + scoped_env_overrides({}) as rank_env, + LLM( + self.MODEL_PATH, + checkpoint_format="mistral", + kv_cache_config=self.KV, + modeling_v2="require", + env_overrides=rank_env, + **layout, + **bounds, + ) as llm, + ): + task.evaluate(llm, extra_evaluator_kwargs=dict(num_fewshot=24)) + + # A passkey hidden in filler text, asked for at the end. GSM8K's prompts stay under the + # indexer's index_topk (2048), where the engine skips the indexer; these prompts are about + # 30000 tokens -- one context step, within max_num_tokens 32768 -- so the indexer scores and + # selects for every query row past 2048, in the context step and in each decode step. + PASSKEY_TOKENS = 30000 + PASSKEY_FILLER = ( + "The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again. " + ) + + def _test_passkey_retrieval(self, deployment): + """The long-context witness: a passkey recalled from a ~30000-token prompt. + + At three depths of the prompt, greedy decoding must reproduce the passkey. A wrong + selection, rope or SSMax position past 2048 reads the wrong positions and loses it. + """ + layout, bounds = self.serving(deployment) + params = SamplingParams(max_tokens=12, temperature=0.0) + with ( + scoped_env_overrides({}) as rank_env, + LLM( + self.MODEL_PATH, + checkpoint_format="mistral", + kv_cache_config=self.KV, + modeling_v2="require", + env_overrides=rank_env, + **layout, + **bounds, + ) as llm, + ): + unit = len(llm.tokenizer.encode(self.PASSKEY_FILLER * 10)) / 10 + reps = int(self.PASSKEY_TOKENS / unit) + for depth, key in ((0.2, 71432), (0.5, 58219), (0.8, 30476)): + before = int(reps * depth) + prompt = ( + "There is an important piece of information hidden inside a lot of " + "irrelevant text. Find it and memorize it. I will quiz you about it.\n\n" + + self.PASSKEY_FILLER * before + + f"\nThe pass key is {key}. Remember it. {key} is the pass key.\n" + + self.PASSKEY_FILLER * (reps - before) + + "\nWhat is the pass key? The pass key is" + ) + tokens = len(llm.tokenizer.encode(prompt)) + assert 2048 < tokens <= bounds["max_seq_len"] - params.max_tokens, tokens + answer = llm.generate([prompt], params)[0].outputs[0].text + assert str(key) in answer, (depth, tokens, answer) + + def _test_accuracy(self, deployment, profile, tmp_path, monkeypatch, record_property): + """Every profile runs native DSpark; full profiles add fresh spec OFF GSM8K. + + ``long_lite`` and ``long_full`` run on the 1M specializations only: RULER rows at 64K, + 128K and the 1M bin (cache witnesses over prefixes of up to 1M tokens, the mixed phase + and exact niah / ruler_vt retrieval), a completed prompt of at least + ruler.LONG_WITNESS_TOKENS and every step's context tokens within max_num_tokens, then the + multimodal panel and GSM8K as in quick / full. long_full also gates its 64K / 128K RULER + means on references/ruler.yaml. + """ + self._run_accuracy( + deployment, profile, tmp_path, monkeypatch, record_property, use_dspark=True + ) + + def _test_accuracy_without_dspark( + self, deployment, profile, tmp_path, monkeypatch, record_property + ): + """The complete accuracy and cache protocol with native DSpark disabled. + + ``full`` covers the 32K datasets; ``long_full`` covers the long-context datasets and + 1M witnesses. Only draft acceptance and speculative execution evidence are omitted. + """ + self._run_accuracy( + deployment, profile, tmp_path, monkeypatch, record_property, use_dspark=False + ) + + def _run_accuracy( + self, deployment, profile, tmp_path, monkeypatch, record_property, *, use_dspark: bool + ): + run_started = time.monotonic() + layout, bounds = self.serving(deployment) + envelope = max(ruler.cell_counts(profile)) + if bounds["max_seq_len"] < envelope: + pytest.skip( + f"{profile} needs max_seq_len {envelope}; {deployment} serves " + f"{bounds['max_seq_len']}" + ) + model_path = self.MODEL_PATH + is_long = profile.startswith("long") + world = layout["tensor_parallel_size"] + attention_dp = layout["enable_attention_dp"] + for name in ( + "INTEGRATION_TEST", + "TRTLLM_ACCURACY_NO_REFERENCE", + "TRTLLM_POPULATE_ACCEPTANCE_LENGTH", + ): + assert os.environ.get(name, "0") == "0", f"Disable {name} for accuracy validation" + data = Path(os.environ.get("TRTLLM_RULER_DATA", f"{llm_models_root()}/datasets/ruler")) + # Every profile serves images and runs the multimodal panel; the 32K ones also score MMMU. + scores_mmmu = profile in ("quick", "full") + # Every profile loads the checkpoint as published, in its Mistral-native format; + # MISTRAL_LARGE4_CHECKPOINT_DIR names a copy of it elsewhere. + model_path = os.environ.get("MISTRAL_LARGE4_CHECKPOINT_DIR", model_path) + # The panel's structured-output case needs xgrammar. + checkpoint = {"checkpoint_format": "mistral", "guided_decoding_backend": "xgrammar"} + rows = ruler.load_inputs(data, profile, Path(model_path) / "tekken.json") + if profile == "long_lite" and attention_dp: + # One attention-DP rank prefills a 1M request alone, about 15 minutes per row: the + # quick 1M case stops at 128K there, and long_full keeps the 1M rows. + rows = [row for row in rows if row["max_length"] <= 131072] + gpu_arch = f"sm_{get_sm_version()}" + root = ( + Path(os.environ.get("TRTLLM_ACCURACY_OUTPUT_DIR", tmp_path)) + / gpu_arch + / f"{deployment}-{profile}{'' if use_dspark else '-spec-off'}" + ) + root.mkdir(parents=True, exist_ok=False) + record_property("artifacts", str(root)) + record_property("gpu_arch", gpu_arch) + record_property("dspark", use_dspark) + # 16 sequences per rank under attention DP (each rank batches its own), 16 in all + # otherwise. + concurrency = 16 * world if attention_dp else 16 + monkeypatch.setenv("TLLM_EVAL_MAX_IN_FLIGHT", str(concurrency)) + failures = [] + spec_config = ( + DSparkDecodingConfig( + max_draft_len=self.DRAFT_LENGTH, + speculative_model=model_path, + ) + if use_dspark + else None + ) + timings = {} + summary = { + "gpu_arch": gpu_arch, + "deployment": deployment, + "profile": profile, + "reuse": True, + "dspark": use_dspark, + "speculative_config": spec_config.model_dump(mode="json") if spec_config else None, + "phase_wall_s": timings, + } + llm_kwargs = dict( + kv_cache_config=self.KV.model_copy(update={"enable_block_reuse": True}), + modeling_v2="require", + enable_iter_perf_stats=True, + env_overrides={MODELING_V2_VALIDATE_ENV: "1", "TLLM_LOG_LEVEL": "info"}, + **checkpoint, + **layout, + **bounds, + ) + coverage = {} + started = time.monotonic() + try: + factory = ( + partial(spec_evidence.traced_llm, trace=root / "spec-trace") + if is_long and use_dspark + else LLM + ) + with ( + scoped_env_overrides(llm_kwargs["env_overrides"]) as rank_env, + factory( + model_path, + speculative_config=spec_config, + **dict(llm_kwargs, env_overrides=rank_env), + ) as llm, + ): + ready = time.monotonic() + assert (llm.args.speculative_config is not None) == use_dspark + summary["speculative_config"] = ( + llm.args.speculative_config.model_dump(mode="json") if use_dspark else None + ) + summary["startup_s"] = ready - started + assert llm.args.kv_cache_config.enable_block_reuse + with _iteration_coverage(llm, root / "ruler-iterations.jsonl", coverage) as ( + errors, + _, + ): + try: + with _phase_timing(timings, "ruler"): + summary["ruler"] = ruler.evaluate( + llm, + rows, + profile, + root, + concurrency, + self.MODEL_NAME, + collect_spec=is_long and use_dspark, + ) + except AssertionError as error: + failures.append(str(error)) + try: + with _phase_timing(timings, "multimodal"): + summary["multimodal"] = multimodal.evaluate(llm, profile, root) + failures.extend( + f"multimodal: {failure}" + for failure in summary["multimodal"]["failures"] + ) + except AssertionError as error: + failures.append(f"multimodal: {error}") + failures.extend(f"RULER iteration reader: {error}" for error in errors) + tasks = [GSM8K(self.MODEL_NAME)] + if scores_mmmu: + tasks.append(MMMU(self.MODEL_NAME)) + if profile == "full": + tasks.append(GPQADiamond(self.MODEL_NAME)) + for task in tasks: + with _iteration_coverage( + llm, root / f"{task.DATASET}-iterations.jsonl", coverage + ) as (errors, spec_stats): + kwargs = dict( + log_samples=True, + output_path=str(root / f"{task.DATASET}.json"), + output_dir=str(root / task.DATASET), + ) + evaluate_kwargs = {} + if isinstance(task, GSM8K): + kwargs["num_fewshot"] = 5 + elif isinstance(task, MMMU): + kwargs.update( + dataset_path=os.environ.get("TRTLLM_MMMU_DATA", MMMU.DATASET_DIR), + chat_template_kwargs={"reasoning_effort": "low"}, + preserve_caller_max_tokens=True, + ) + # The task yaml sets greedy decoding and the stop string; its 512 + # tokens yield to MMMU_OUTPUT_LEN. The reference baseline truncates + # no prompt. + budget = bounds["max_seq_len"] - self.MMMU_OUTPUT_LEN + evaluate_kwargs = dict( + extra_acc_spec=f"native-mm-{profile}", + sampling_params=SamplingParams( + max_tokens=self.MMMU_OUTPUT_LEN, + truncate_prompt_tokens=budget, + ), + ) + elif isinstance(task, GPQADiamond): + kwargs.update( + apply_chat_template=True, + shuffle_dataset=False, + chat_template_kwargs={"reasoning_effort": "low"}, + ) + try: + with _phase_timing(timings, task.DATASET): + summary[task.DATASET] = task.evaluate( + llm, extra_evaluator_kwargs=kwargs, **evaluate_kwargs + ) + except AssertionError as error: + failures.append(f"{task.DATASET}: {error}") + failures.extend(f"{task.DATASET} iteration reader: {error}" for error in errors) + if isinstance(task, GSM8K) and use_dspark: + try: + al = acceptance_length_from_iteration_stats(spec_stats) + summary["gsm8k_acceptance_length"] = al + assert_acceptance_length( + f"ModelingV2MistralLarge4Nvfp4::{deployment.removesuffix('_1m')}::dspark3", + al, + ) + except (AssertionError, KeyError, ValueError) as error: + failures.append(f"GSM8K acceptance length: {error}") + summary["coverage"] = coverage + if "multimodal" in summary: + with (root / "ruler-iterations.jsonl").open() as stream: + failures.extend( + f"multimodal: {failure}" + for failure in multimodal.rank_evidence( + map(json.loads, stream), + summary["multimodal"]["windows"], + attention_dp, + ) + ) + ranks = range(world if attention_dp else 1) + if any( + coverage.get(rank, {}).get("context_tokens", 0) < 32112 + or coverage.get(rank, {}).get("decode_batch", 0) < 16 + for rank in ranks + ): + failures.append(f"INCONCLUSIVE: envelope {coverage}") + if profile.startswith("long"): + longest = ruler.longest_completed(root, rows) + summary["longest_prompt_tokens"] = longest + need = min(ruler.LONG_WITNESS_TOKENS, max(len(r["prompt_ids"]) for r in rows)) + if longest < need: + failures.append( + f"INCONCLUSIVE: no completed {need}-token prompt ({longest})" + ) + step = bounds["max_num_tokens"] + if any(shape["context_tokens"] > step for shape in coverage.values()): + failures.append( + f"FAIL: a step's context tokens exceeded {step}: {coverage}" + ) + summary["test_wall_s"] = time.monotonic() - ready + summary["teardown_s"] = time.monotonic() - ready - summary["test_wall_s"] + if is_long and use_dspark: + longest_bin = max(row["max_length"] for row in rows) + need = min( + ruler.LONG_WITNESS_TOKENS, + min(len(row["prompt_ids"]) for row in rows if row["max_length"] == longest_bin), + ) + summary["spec_evidence"] = spec_evidence.evaluate(root, need, world, attention_dp) + failures.extend(summary["spec_evidence"]["failures"]) + if use_dspark and profile in ("full", "long_full"): + # Release the target and draft completely before starting the other arm. + # Same specialization, reuse, bounds and evaluator; only speculation differs. + origin = root / "spec-off" + origin.mkdir() + task = GSM8K(self.MODEL_NAME) + spec_off_started = time.monotonic() + with ( + scoped_env_overrides(llm_kwargs["env_overrides"]) as rank_env, + LLM(model_path, **dict(llm_kwargs, env_overrides=rank_env)) as llm, + ): + spec_off_ready = time.monotonic() + summary["spec_off_startup_s"] = spec_off_ready - spec_off_started + with _iteration_coverage(llm, origin / "iterations.jsonl", {}) as ( + errors, + _, + ): + try: + with _phase_timing(timings, "gsm8k_spec_off"): + score = task.evaluate( + llm, + extra_evaluator_kwargs=dict( + num_fewshot=5, + log_samples=True, + output_path=str(origin / f"{task.DATASET}.json"), + output_dir=str(origin / task.DATASET), + ), + ) + summary["gsm8k_spec_off"] = score + except AssertionError as error: + failures.append(f"GSM8K spec OFF: {error}") + failures.extend(f"Spec OFF iteration reader: {error}" for error in errors) + spec_off_completed = time.monotonic() + summary["spec_off_teardown_s"] = time.monotonic() - spec_off_completed + if task.DATASET in summary and "gsm8k_spec_off" in summary: + delta = summary[task.DATASET] - summary["gsm8k_spec_off"] + summary["gsm8k_delta_pp"] = delta + summary["gsm8k_max_delta_pp"] = self.GSM8K_MAX_DELTA + if abs(delta) > self.GSM8K_MAX_DELTA: + failures.append( + f"GSM8K spec ON/OFF delta {delta:+.4f} pp exceeds " + f"{self.GSM8K_MAX_DELTA:.2f} pp" + ) + except Exception as error: + failures.append(f"{type(error).__name__}: {error}") + raise + finally: + summary["total_wall_s"] = time.monotonic() - run_started + summary["failures"] = list(failures) + (root / "summary.json").write_text(json.dumps(summary, indent=2) + "\n") + assert not failures, failures + + +class _TepTarget(_Nvfp4Target): + """The tep layout, the language model.""" + + # Tensor-parallel attention with expert-parallel experts: the attention heads and the routed + # experts both sharded over every rank, attention DP off -- four ways on one node, or eight + # over two nodes of four. Routing derives the target *from* these: attention DP would be a + # different weight layout (the dep target), and tensor_parallel_size alone shards every + # expert by width instead (the tp target). + LAYOUT = "tep" + LAYOUTS = { + 4: dict(tensor_parallel_size=4, moe_expert_parallel_size=4, enable_attention_dp=False), + 8: dict(tensor_parallel_size=8, moe_expert_parallel_size=8, enable_attention_dp=False), + } + + +class _DepTarget(_Nvfp4Target): + """The dep layout, the language model.""" + + # Attention data parallel over every rank (each holds the whole attention and serves its + # own requests) with the routed experts split by expert over every rank, four on one node + # or eight over two nodes of four. Routing derives the target *from* these: attention DP + # off is the tep target, and tensor_parallel_size alone another weight layout. + LAYOUT = "dep" + LAYOUTS = { + 4: dict(tensor_parallel_size=4, moe_expert_parallel_size=4, enable_attention_dp=True), + 8: dict(tensor_parallel_size=8, moe_expert_parallel_size=8, enable_attention_dp=True), + } + + +class _TpTarget(_Nvfp4Target): + """The tp layout, the language model.""" + + # Tensor parallel throughout: the attention heads sharded over every rank and every routed + # expert on every rank, cut to its share of the width, attention DP off -- four ways on one + # node, or eight over two nodes of four. Routing derives the target *from* these: expert + # parallelism is the tep target. A bare tensor_parallel_size resolves to the same split. + LAYOUT = "tp" + LAYOUTS = { + 4: dict( + tensor_parallel_size=4, + moe_tensor_parallel_size=4, + moe_expert_parallel_size=1, + enable_attention_dp=False, + ), + 8: dict( + tensor_parallel_size=8, + moe_tensor_parallel_size=8, + moe_expert_parallel_size=1, + enable_attention_dp=False, + ), + } + + def _test_llguidance(self, tmp_path, record_property): + """Guided decoding with llguidance on tp4 with native DSpark, over frontend-rendered chat ids. + + Cases (llguidance_panel): JSON, JSON after a reasoning section, and mistral-common's tool + grammars under tool_choice auto, required and named. + """ + layout, bounds = self.serving("tp4") + model_path = os.environ.get("MISTRAL_LARGE4_CHECKPOINT_DIR", self.MODEL_PATH) + gpu_arch = f"sm_{get_sm_version()}" + root = ( + Path(os.environ.get("TRTLLM_ACCURACY_OUTPUT_DIR", tmp_path)) + / gpu_arch + / "tp4-llguidance" + ) + root.mkdir(parents=True, exist_ok=False) + record_property("artifacts", str(root)) + record_property("gpu_arch", gpu_arch) + with ( + scoped_env_overrides({MODELING_V2_VALIDATE_ENV: "1"}) as rank_env, + LLM( + model_path, + speculative_config=DSparkDecodingConfig( + max_draft_len=self.DRAFT_LENGTH, speculative_model=model_path + ), + checkpoint_format="mistral", + guided_decoding_backend="llguidance", + kv_cache_config=self.KV.model_copy(update={"enable_block_reuse": True}), + modeling_v2="require", + env_overrides=rank_env, + **layout, + **bounds, + ) as llm, + ): + summary = llguidance_panel.evaluate(llm, root) + summary["gpu_arch"] = gpu_arch + (root / "summary.json").write_text(json.dumps(summary, indent=2) + "\n") + failures = [failure for failure in summary["failures"] if failure.startswith("FAIL")] + assert not failures, failures diff --git a/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/__init__.py b/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/__init__.py new file mode 100644 index 000000000000..467079831e16 --- /dev/null +++ b/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/__init__.py @@ -0,0 +1,14 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_accuracy.py b/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_accuracy.py new file mode 100644 index 000000000000..cf303e99c80b --- /dev/null +++ b/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_accuracy.py @@ -0,0 +1,61 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Whole-model gate for the mistral_large4 modeling_v2 target mistral_large_4_nvfp4__sm_103__dep. + +The tests run the implementations in ``../_common.py``, which also holds the layout and its +deployments. +""" + +import pytest + +from .._common import ( + PROFILES, + PROFILES_WITHOUT_DSPARK, + _DepTarget, + pytest_generate_tests, # noqa: F401 (the hook parametrizes the deployments) + skip_not_sm103, +) + + +class TestModelingV2MistralLarge4Nvfp4Sm103Dep(_DepTarget): + """mistral-large-4-nvfp4 / sm_103 / dep, the language model.""" + + pytestmark = skip_not_sm103 + + @pytest.mark.skip_less_device(4) + def test_gsm8k(self, deployment): + self._test_gsm8k(deployment) + + @pytest.mark.skip_less_device(4) + def test_gsm8k_24shot(self, deployment): + self._test_gsm8k_24shot(deployment) + + @pytest.mark.skip_less_device(4) + def test_passkey_retrieval(self, deployment): + self._test_passkey_retrieval(deployment) + + @pytest.mark.skip_less_device(4) + @pytest.mark.parametrize("profile", PROFILES) + def test_accuracy(self, deployment, profile, tmp_path, monkeypatch, record_property): + self._test_accuracy(deployment, profile, tmp_path, monkeypatch, record_property) + + @pytest.mark.skip_less_device(4) + @pytest.mark.parametrize("profile", PROFILES_WITHOUT_DSPARK) + def test_accuracy_without_dspark( + self, deployment, profile, tmp_path, monkeypatch, record_property + ): + self._test_accuracy_without_dspark( + deployment, profile, tmp_path, monkeypatch, record_property + ) diff --git a/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/__init__.py b/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/__init__.py new file mode 100644 index 000000000000..467079831e16 --- /dev/null +++ b/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/__init__.py @@ -0,0 +1,14 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_accuracy.py b/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_accuracy.py new file mode 100644 index 000000000000..b1bcc7813377 --- /dev/null +++ b/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_accuracy.py @@ -0,0 +1,61 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Whole-model gate for the mistral_large4 modeling_v2 target mistral_large_4_nvfp4__sm_103__tep. + +The tests run the implementations in ``../_common.py``, which also holds the layout and its +deployments. +""" + +import pytest + +from .._common import ( + PROFILES, + PROFILES_WITHOUT_DSPARK, + _TepTarget, + pytest_generate_tests, # noqa: F401 (the hook parametrizes the deployments) + skip_not_sm103, +) + + +class TestModelingV2MistralLarge4Nvfp4Sm103Tep(_TepTarget): + """mistral-large-4-nvfp4 / sm_103 / tep, the language model.""" + + pytestmark = skip_not_sm103 + + @pytest.mark.skip_less_device(4) + def test_gsm8k(self, deployment): + self._test_gsm8k(deployment) + + @pytest.mark.skip_less_device(4) + def test_gsm8k_24shot(self, deployment): + self._test_gsm8k_24shot(deployment) + + @pytest.mark.skip_less_device(4) + def test_passkey_retrieval(self, deployment): + self._test_passkey_retrieval(deployment) + + @pytest.mark.skip_less_device(4) + @pytest.mark.parametrize("profile", PROFILES) + def test_accuracy(self, deployment, profile, tmp_path, monkeypatch, record_property): + self._test_accuracy(deployment, profile, tmp_path, monkeypatch, record_property) + + @pytest.mark.skip_less_device(4) + @pytest.mark.parametrize("profile", PROFILES_WITHOUT_DSPARK) + def test_accuracy_without_dspark( + self, deployment, profile, tmp_path, monkeypatch, record_property + ): + self._test_accuracy_without_dspark( + deployment, profile, tmp_path, monkeypatch, record_property + ) diff --git a/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/__init__.py b/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/__init__.py new file mode 100644 index 000000000000..467079831e16 --- /dev/null +++ b/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/__init__.py @@ -0,0 +1,14 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_accuracy.py b/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_accuracy.py new file mode 100644 index 000000000000..a31c67bdb6dc --- /dev/null +++ b/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_accuracy.py @@ -0,0 +1,65 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Whole-model gate for the mistral_large4 modeling_v2 target mistral_large_4_nvfp4__sm_103__tp. + +The tests run the implementations in ``../_common.py``, which also holds the layout and its +deployments. +""" + +import pytest + +from .._common import ( + PROFILES, + PROFILES_WITHOUT_DSPARK, + _TpTarget, + pytest_generate_tests, # noqa: F401 (the hook parametrizes the deployments) + skip_not_sm103, +) + + +class TestModelingV2MistralLarge4Nvfp4Sm103Tp(_TpTarget): + """mistral-large-4-nvfp4 / sm_103 / tp, the language model.""" + + pytestmark = skip_not_sm103 + + @pytest.mark.skip_less_device(4) + def test_gsm8k(self, deployment): + self._test_gsm8k(deployment) + + @pytest.mark.skip_less_device(4) + def test_gsm8k_24shot(self, deployment): + self._test_gsm8k_24shot(deployment) + + @pytest.mark.skip_less_device(4) + def test_passkey_retrieval(self, deployment): + self._test_passkey_retrieval(deployment) + + @pytest.mark.skip_less_device(4) + @pytest.mark.parametrize("profile", PROFILES) + def test_accuracy(self, deployment, profile, tmp_path, monkeypatch, record_property): + self._test_accuracy(deployment, profile, tmp_path, monkeypatch, record_property) + + @pytest.mark.skip_less_device(4) + @pytest.mark.parametrize("profile", PROFILES_WITHOUT_DSPARK) + def test_accuracy_without_dspark( + self, deployment, profile, tmp_path, monkeypatch, record_property + ): + self._test_accuracy_without_dspark( + deployment, profile, tmp_path, monkeypatch, record_property + ) + + @pytest.mark.skip_less_device(4) + def test_llguidance(self, tmp_path, record_property): + self._test_llguidance(tmp_path, record_property) diff --git a/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_107__dep/__init__.py b/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_107__dep/__init__.py new file mode 100644 index 000000000000..467079831e16 --- /dev/null +++ b/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_107__dep/__init__.py @@ -0,0 +1,14 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_107__dep/test_accuracy.py b/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_107__dep/test_accuracy.py new file mode 100644 index 000000000000..3deeac8c760b --- /dev/null +++ b/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_107__dep/test_accuracy.py @@ -0,0 +1,61 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Whole-model gate for the mistral_large4 modeling_v2 target mistral_large_4_nvfp4__sm_107__dep. + +The tests run the implementations in ``../_common.py``, which also holds the layout and its +deployments. +""" + +import pytest + +from .._common import ( + PROFILES, + PROFILES_WITHOUT_DSPARK, + _DepTarget, + pytest_generate_tests, # noqa: F401 (the hook parametrizes the deployments) + skip_not_sm107, +) + + +class TestModelingV2MistralLarge4Nvfp4Sm107Dep(_DepTarget): + """mistral-large-4-nvfp4 / sm_107 / dep, the language model.""" + + pytestmark = skip_not_sm107 + + @pytest.mark.skip_less_device(4) + def test_gsm8k(self, deployment): + self._test_gsm8k(deployment) + + @pytest.mark.skip_less_device(4) + def test_gsm8k_24shot(self, deployment): + self._test_gsm8k_24shot(deployment) + + @pytest.mark.skip_less_device(4) + def test_passkey_retrieval(self, deployment): + self._test_passkey_retrieval(deployment) + + @pytest.mark.skip_less_device(4) + @pytest.mark.parametrize("profile", PROFILES) + def test_accuracy(self, deployment, profile, tmp_path, monkeypatch, record_property): + self._test_accuracy(deployment, profile, tmp_path, monkeypatch, record_property) + + @pytest.mark.skip_less_device(4) + @pytest.mark.parametrize("profile", PROFILES_WITHOUT_DSPARK) + def test_accuracy_without_dspark( + self, deployment, profile, tmp_path, monkeypatch, record_property + ): + self._test_accuracy_without_dspark( + deployment, profile, tmp_path, monkeypatch, record_property + ) diff --git a/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_107__tep/__init__.py b/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_107__tep/__init__.py new file mode 100644 index 000000000000..467079831e16 --- /dev/null +++ b/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_107__tep/__init__.py @@ -0,0 +1,14 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_107__tep/test_accuracy.py b/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_107__tep/test_accuracy.py new file mode 100644 index 000000000000..fbb6ff5d8e6a --- /dev/null +++ b/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_107__tep/test_accuracy.py @@ -0,0 +1,61 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Whole-model gate for the mistral_large4 modeling_v2 target mistral_large_4_nvfp4__sm_107__tep. + +The tests run the implementations in ``../_common.py``, which also holds the layout and its +deployments. +""" + +import pytest + +from .._common import ( + PROFILES, + PROFILES_WITHOUT_DSPARK, + _TepTarget, + pytest_generate_tests, # noqa: F401 (the hook parametrizes the deployments) + skip_not_sm107, +) + + +class TestModelingV2MistralLarge4Nvfp4Sm107Tep(_TepTarget): + """mistral-large-4-nvfp4 / sm_107 / tep, the language model.""" + + pytestmark = skip_not_sm107 + + @pytest.mark.skip_less_device(4) + def test_gsm8k(self, deployment): + self._test_gsm8k(deployment) + + @pytest.mark.skip_less_device(4) + def test_gsm8k_24shot(self, deployment): + self._test_gsm8k_24shot(deployment) + + @pytest.mark.skip_less_device(4) + def test_passkey_retrieval(self, deployment): + self._test_passkey_retrieval(deployment) + + @pytest.mark.skip_less_device(4) + @pytest.mark.parametrize("profile", PROFILES) + def test_accuracy(self, deployment, profile, tmp_path, monkeypatch, record_property): + self._test_accuracy(deployment, profile, tmp_path, monkeypatch, record_property) + + @pytest.mark.skip_less_device(4) + @pytest.mark.parametrize("profile", PROFILES_WITHOUT_DSPARK) + def test_accuracy_without_dspark( + self, deployment, profile, tmp_path, monkeypatch, record_property + ): + self._test_accuracy_without_dspark( + deployment, profile, tmp_path, monkeypatch, record_property + ) diff --git a/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/__init__.py b/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/__init__.py new file mode 100644 index 000000000000..467079831e16 --- /dev/null +++ b/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/__init__.py @@ -0,0 +1,14 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/test_accuracy.py b/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/test_accuracy.py new file mode 100644 index 000000000000..20452b37db2f --- /dev/null +++ b/tests/integration/defs/accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/test_accuracy.py @@ -0,0 +1,65 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Whole-model gate for the mistral_large4 modeling_v2 target mistral_large_4_nvfp4__sm_107__tp. + +The tests run the implementations in ``../_common.py``, which also holds the layout and its +deployments. +""" + +import pytest + +from .._common import ( + PROFILES, + PROFILES_WITHOUT_DSPARK, + _TpTarget, + pytest_generate_tests, # noqa: F401 (the hook parametrizes the deployments) + skip_not_sm107, +) + + +class TestModelingV2MistralLarge4Nvfp4Sm107Tp(_TpTarget): + """mistral-large-4-nvfp4 / sm_107 / tp, the language model.""" + + pytestmark = skip_not_sm107 + + @pytest.mark.skip_less_device(4) + def test_gsm8k(self, deployment): + self._test_gsm8k(deployment) + + @pytest.mark.skip_less_device(4) + def test_gsm8k_24shot(self, deployment): + self._test_gsm8k_24shot(deployment) + + @pytest.mark.skip_less_device(4) + def test_passkey_retrieval(self, deployment): + self._test_passkey_retrieval(deployment) + + @pytest.mark.skip_less_device(4) + @pytest.mark.parametrize("profile", PROFILES) + def test_accuracy(self, deployment, profile, tmp_path, monkeypatch, record_property): + self._test_accuracy(deployment, profile, tmp_path, monkeypatch, record_property) + + @pytest.mark.skip_less_device(4) + @pytest.mark.parametrize("profile", PROFILES_WITHOUT_DSPARK) + def test_accuracy_without_dspark( + self, deployment, profile, tmp_path, monkeypatch, record_property + ): + self._test_accuracy_without_dspark( + deployment, profile, tmp_path, monkeypatch, record_property + ) + + @pytest.mark.skip_less_device(4) + def test_llguidance(self, tmp_path, record_property): + self._test_llguidance(tmp_path, record_property) diff --git a/tests/integration/defs/accuracy/modeling_v2/multimodal.py b/tests/integration/defs/accuracy/modeling_v2/multimodal.py new file mode 100644 index 000000000000..099bdc2953b2 --- /dev/null +++ b/tests/integration/defs/accuracy/modeling_v2/multimodal.py @@ -0,0 +1,595 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Native image checks within one accuracy-test engine: cache construction, content and ranks. + +Path invariants fail the case: the prompt round trip, a fresh cache-salt namespace with no hit, +and hits that stay before the first image that differs. Missing cache telemetry, a missing warm +witness, a rank without image prefill evidence and a wrong answer about image content or order +are INCONCLUSIVE: greedy text may change across kernels or scheduling. +""" + +import base64 +import hashlib +import json +import re +import time +from contextlib import closing, contextmanager +from datetime import datetime +from io import BytesIO +from pathlib import Path + +from PIL import Image, ImageDraw + +from tensorrt_llm import SamplingParams +from tensorrt_llm.llmapi import SchedulingParams +from tensorrt_llm.llmapi.reasoning_parser import ReasoningParserFactory +from tensorrt_llm.sampling_params import GuidedDecodingParams + +# Output tokens per question: room for the reasoning block that precedes every answer at +# reasoning effort low. +ANSWER_TOKENS = 1024 +# The cache pairs use one image size, so their prompts are token-identical and only the images +# differ: a hit may then extend no further than the first image that differs. +PAIR_SIZE = (224, 168) +# The structured-output case: one color from the panel's palette, as a JSON object. +_COLOR_SCHEMA = { + "type": "object", + "properties": {"color": {"type": "string", "enum": ["red", "blue", "green"]}}, + "required": ["color"], + "additionalProperties": False, +} +# Shapes are drawn on white, so white and black are not answers. +_COLORS = re.compile(r"\b(red|blue|green|yellow|orange|purple|pink|brown|gray|grey|paris)\b") +_ORDER = re.compile(r"\b(first|second)\b") +_REFUSAL = re.compile(r"can't|cannot|don't|didn't|unable|blank|if you|not see") +_PREFIX = ( + "You are checking a test image. Look at the picture carefully, ignore the white background " + "and answer only the question that follows the image, with no explanation." +) + + +def image_part(image): + buffer = BytesIO() + image.save(buffer, "PNG") + return { + "type": "image_url", + "image_url": { + "url": "data:image/png;base64," + base64.b64encode(buffer.getvalue()).decode() + }, + } + + +def text_part(text): + return {"type": "text", "text": text} + + +def shape(kind, size, color): + image = Image.new("RGB", size, "white") + draw = ImageDraw.Draw(image) + width, height = size + margin = max(8, min(width, height) // 8) + bounds = (margin, margin, width - margin - 1, height - margin - 1) + if kind == "circle": + draw.ellipse(bounds, fill=color) + elif kind == "square": + draw.rectangle(bounds, fill=color) + else: + draw.polygon( + [ + (width // 2, margin), + (width - margin - 1, height - margin - 1), + (margin, height - margin - 1), + ], + fill=color, + ) + return image + + +def native_prompt(llm, messages, images): + ids = llm.tokenizer.apply_chat_template( + messages, + tokenize=True, + return_dict=False, + return_tensors=None, + reasoning_effort="low", + ) + assert len(ids) + ANSWER_TOKENS <= llm.args.max_seq_len, ( + "Image expansion exceeded the sequence budget" + ) + prompt = {"prompt_token_ids": ids} + if images: + prompt["multi_modal_data"] = {"image": images} + return prompt + + +def final_answer(text): + """The answer in ``text`` (decoded with special tokens kept), without the reasoning before it.""" + return ReasoningParserFactory.create_reasoning_parser("mistral_large_4").parse(text).content + + +def image_tokens(llm): + """The [IMG] and [IMG_END] token ids of the native tokenizer.""" + tekken = llm.tokenizer.tokenizer + lookup = getattr(tekken, "get_special_token", None) or tekken.get_control_token + return lookup("[IMG]"), lookup("[IMG_END]") + + +def image_spans(ids, img, img_end): + """(first [IMG], [IMG_END]) positions of each image, in prompt order.""" + spans, start = [], None + for position, token in enumerate(ids): + if token == img and start is None: + start = position + elif token == img_end: + assert start is not None, "[IMG_END] without an image" + spans.append((start, position)) + start = None + assert start is None, "Image without [IMG_END]" + return spans + + +def answer_matches(text, expected): + """Whether the answer names the expected colors (or first / second) in order.""" + answer = text.lower().replace("’", "'") + # A refusal can repeat the expected words as a hypothetical example. + if _REFUSAL.search(answer): + return False + if expected in (["first"], ["second"]): + return _ORDER.findall(answer)[:1] == expected + words = _COLORS.findall(answer) + # Repeated descriptions and a repeated final answer must keep the same order. + runs = [word for index, word in enumerate(words) if index == 0 or word != words[index - 1]] + return ( + bool(runs) + and len(runs) % len(expected) == 0 + and runs == expected * (len(runs) // len(expected)) + ) + + +def pair_failures(hits, bounds, witnesses): + """Cache checks of one salted construction. + + hits: cached tokens per case name (None when telemetry was missing). bounds: the largest + legal hit per case. witnesses: the hit a case must exceed to show the cache reused image + blocks, i.e. the start of its first image: hits are whole blocks. + """ + failures = [] + for name, hit in hits.items(): + if hit is None: + failures.append(f"INCONCLUSIVE: {name}: missing cache telemetry") + elif hit > bounds[name]: + failures.append(f"FAIL: {name}: illegal cache hit {hit} > {bounds[name]}") + for name, floor in witnesses.items(): + if hits[name] is not None and hits[name] <= floor: + failures.append(f"INCONCLUSIVE: {name}: no warm image-block witness ({hits[name]})") + return failures + + +def _timestamp(row): + return datetime.strptime(row["timestamp"], "%m-%d-%Y %H:%M:%S.%f").timestamp() + + +def rank_evidence(iterations, windows, enable_attention_dp, margin=2.0): + """Require each rank to prefill its cold image prompt inside the window it was pinned to. + + iterations: rows of the engine's iteration stats. windows: rank -> (start, end, prompt + tokens of the cold request sent alone at the start of the window). + """ + seen = set() + for row in iterations: + rank = row.get("attentionDpRank", None if enable_attention_dp else 0) + if rank not in windows or "timestamp" not in row: + continue + start, end, tokens = windows[rank] + stats = row.get("inflightBatchingStats") or {} + if ( + start - margin <= _timestamp(row) <= end + margin + and stats.get("numContextRequests", 0) >= 1 + and stats.get("numCtxTokens", 0) >= tokens + ): + seen.add(rank) + return [ + f"INCONCLUSIVE: rank {rank} shows no image prefill in its pinned window" + for rank in sorted(set(windows) - seen) + ] + + +@contextmanager +def _openai_client(llm): + """Borrow the test's engine and own only the server's client and thread pools.""" + from fastapi.testclient import TestClient + + from tensorrt_llm.inputs.media_io import BaseMediaIO + from tensorrt_llm.serve.openai_server import OpenAIServer + + previous_executor = BaseMediaIO._executor + server = OpenAIServer( + llm, + model=str(llm.args.model), + tool_parser=None, + server_role=None, + metadata_server_cfg=None, + ) + try: + # The ASGI lifespan owns engine shutdown; this smoke borrows an engine + # that the remaining accuracy stages still need. + with closing(TestClient(server.app)) as client: + yield server, client + finally: + server._input_proc_executor.shutdown(wait=True) + server._media_load_executor.shutdown(wait=True) + BaseMediaIO._executor = previous_executor + + +def check_structured(name, text, failures): + """Fail unless ``text`` is a JSON object of ``_COLOR_SCHEMA``; a wrong color is inconclusive.""" + try: + answer = json.loads(text) + except json.JSONDecodeError: + failures.append(f"FAIL: {name}: not JSON: {text!r}") + return + if ( + not isinstance(answer, dict) + or set(answer) != {"color"} + or answer["color"] not in _COLOR_SCHEMA["properties"]["color"]["enum"] + ): + failures.append(f"FAIL: {name}: outside the schema: {text!r}") + elif answer["color"] != "red": + failures.append(f"INCONCLUSIVE: {name}: {text!r}") + + +def evaluate(llm, profile, out: Path): + """Run the panel on every attention-DP rank and return its result with failures labelled.""" + img, img_end = image_tokens(llm) + red = shape("circle", PAIR_SIZE, "red") + blue = shape("square", PAIR_SIZE, "blue") + green = shape("triangle", PAIR_SIZE, "green") + tall_blue = shape("square", (168, 280), "blue") + fixtures = Path(__file__).parents[3] / "test_input_files" + merlion = Image.open(fixtures / "merlion.png").convert("RGB") + hockey = Image.open(fixtures / "pexels-ron-lach-8975010.jpg").convert("RGB") + full = profile == "full" + boundary = ( + [ + ("large", shape("circle", (1540, 1540), "red"), "red"), + # Not blue: on this bar blue nearly ties with purple, for the reference implementation too. + ("landscape", shape("square", (1540, 112), "yellow"), "yellow"), + ("portrait", shape("triangle", (112, 1540), "green"), "green"), + ] + if full + else [] + ) + aggregate = ( + [shape("circle", (1540, 1540), "red") for _ in range(9)] + + [shape("square", (1540, 1540), "blue")] + if full + else [] + ) + named = [("red", red), ("blue", blue), ("green", green), ("tall-blue", tall_blue)] + named += [("merlion", merlion), ("hockey", hockey)] + named += [(name, image) for name, image, _ in boundary] + named += [(f"aggregate-{index}", image) for index, image in enumerate(aggregate)] + manifest = { + name: {"size": list(image.size), "rgb_sha256": hashlib.sha256(image.tobytes()).hexdigest()} + for name, image in named + } + (out / "mm_images.json").write_text(json.dumps(manifest, indent=2) + "\n") + + def single(image, question="What color is the large shape? Answer one color word."): + messages = [ + { + "role": "user", + "content": [text_part(_PREFIX), image_part(image), text_part(question)], + } + ] + return native_prompt(llm, messages, [image]) + + def pair(first, second, question=" Name the colors of the large shapes in order, two words."): + content = [text_part("First image: "), image_part(first)] + content += [text_part(" Second image: "), image_part(second), text_part(question)] + messages = [{"role": "user", "content": content}] + return messages, native_prompt(llm, messages, [first, second]) + + a, b = single(red), single(blue) + pair_messages, red_blue = pair(red, blue) + _, red_green = pair(red, green) + _, blue_red = pair(blue, red) + # The constructions below rely on token-identical prompts. + assert a["prompt_token_ids"] == b["prompt_token_ids"] + assert red_blue["prompt_token_ids"] == red_green["prompt_token_ids"] + assert red_blue["prompt_token_ids"] == blue_red["prompt_token_ids"] + (single_span,) = image_spans(a["prompt_token_ids"], img, img_end) + first_span, second_span = image_spans(red_blue["prompt_token_ids"], img, img_end) + _, sized = pair(red, tall_blue) + conversation = pair_messages + [ + {"role": "assistant", "content": "Red blue."}, + { + "role": "user", + "content": [ + image_part(blue), + text_part( + "Which previous image has a shape of this color, first or second? " + "Answer one word." + ), + ], + }, + ] + multiturn = native_prompt(llm, conversation, [red, blue, blue]) + text = native_prompt(llm, [{"role": "user", "content": "Reply with Paris only."}], []) + natural = [ + ( + name, + pair(first, second, " Which image shows hockey players, first or second? One word.")[1], + expected, + ) + for name, first, second, expected in ( + ("natural-merlion-hockey", merlion, hockey, ["second"]), + ("natural-hockey-merlion", hockey, merlion, ["first"]), + ) + ] + + attention_dp = llm.args.enable_attention_dp + ranks = range(llm.args.tensor_parallel_size if attention_dp else 1) + params = SamplingParams( + max_tokens=ANSWER_TOKENS, + temperature=0, + return_perf_metrics=True, + skip_special_tokens=False, + ) + failures, windows, cases = [], {}, 0 + started = time.monotonic() + with (out / "mm_predictions.jsonl").open("w") as stream: + + def run(batch, rank, phase, salts): + """batch: (name, prompt, expected answer or None). Returns cached tokens per case.""" + nonlocal cases + outputs = llm.generate( + [prompt for _, prompt, _ in batch], + params, + cache_salt=salts, + scheduling_params=( + [SchedulingParams(attention_dp_rank=rank, attention_dp_relax=False)] + * len(batch) + if attention_dp + else None + ), + use_tqdm=False, + ) + hits = [] + for (name, prompt, expected), salt, output in zip(batch, salts, outputs, strict=True): + completion = output.outputs[0] + if list(output.prompt_token_ids) != prompt["prompt_token_ids"]: + failures.append(f"FAIL: rank {rank} {name}: prompt changed or truncated") + hit = output.cached_tokens if completion.request_perf_metrics else None + passed = expected is None or answer_matches(final_answer(completion.text), expected) + if not passed: + failures.append(f"INCONCLUSIVE: rank {rank} {name}: {completion.text!r}") + record = { + "name": name, + "phase": phase, + "attention_dp_rank": rank, + "cache_salt": salt, + "prompt_tokens": len(output.prompt_token_ids), + "cached_tokens": hit, + "text": completion.text, + "output_token_ids": completion.token_ids, + "expected": expected, + "passed": passed, + } + stream.write(json.dumps(record) + "\n") + stream.flush() + hits.append(hit) + cases += 1 + return hits + + for rank in ranks: + window_start = time.time() + salt = f"mm/{rank}" + # A(s1) -> B(s1) -> A(s1) warm -> A(s2): B differs from A only in its image. + hits = {} + for name, prompt, color, namespace in ( + ("red-cold", a, "red", f"{salt}/pair"), + ("blue-same-text", b, "blue", f"{salt}/pair"), + ("red-warm", a, "red", f"{salt}/pair"), + ("red-fresh-salt", a, "red", f"{salt}/fresh"), + ): + hits[name] = run([(name, prompt, [color])], rank, "pair", [namespace])[0] + length = len(a["prompt_token_ids"]) + failures.extend( + f"rank {rank} {failure}" + for failure in pair_failures( + hits, + bounds={ + "red-cold": 0, + "blue-same-text": single_span[0], + "red-warm": length - 1, + "red-fresh-salt": 0, + }, + witnesses={"red-warm": single_span[0]}, + ) + ) + # Same first image, then a different companion or the swapped order. + companion, order = {}, f"{salt}/order" + for name, prompt, expected in ( + ("pair-red-blue", red_blue, ["red", "blue"]), + ("pair-red-green", red_green, ["red", "green"]), + ("pair-blue-red", blue_red, ["blue", "red"]), + ): + (companion[name],) = run([(name, prompt, expected)], rank, "pair", [order]) + failures.extend( + f"rank {rank} {failure}" + for failure in pair_failures( + companion, + bounds={ + "pair-red-blue": 0, + "pair-red-green": second_span[0], + "pair-blue-red": first_span[0], + }, + witnesses={"pair-red-green": first_span[0]}, + ) + ) + # Content and order with sizes, turns and photos, each in its own namespace. + for name, prompt, expected in [ + ("pair-different-sizes", sized, ["red", "blue"]), + ("multiturn", multiturn, ["second"]), + ("text", text, ["paris"]), + *natural, + ]: + run([(name, prompt, expected)], rank, "content", [f"{salt}/{name}"]) + # Mixed batch: warm anchors keep their namespace, every new request gets its own. + new = [("text", text, ["paris"]), ("pair-different-sizes", sized, ["red", "blue"])] + new += natural + batch, salts = [], [] + for index, case in enumerate(new): + anchor = ( + ("red-warm", a, ["red"], f"{salt}/pair") + if index % 2 == 0 + else ("pair-red-blue", red_blue, ["red", "blue"], order) + ) + batch += [anchor[:3], case] + salts += [anchor[3], f"{salt}/mixed/{index}"] + mixed = run(batch, rank, "mixed", salts) + if any(hit is None for hit in mixed): + failures.append(f"INCONCLUSIVE: rank {rank} mixed: missing cache telemetry") + elif any(mixed[1::2]): + failures.append(f"FAIL: rank {rank} mixed: new requests reused cached prefixes") + elif not any(mixed[::2]): + failures.append(f"INCONCLUSIVE: rank {rank} mixed: no warm hits") + if full: + for name, image, color in boundary: + run([(name, single(image), [color])], rank, "boundary", [f"{salt}/{name}"]) + messages = [ + { + "role": "user", + "content": [ + *[image_part(image) for image in aggregate], + text_part( + "What color is the large shape in the LAST image? " + "Answer one color word." + ), + ], + } + ] + prompt = native_prompt(llm, messages, aggregate) + assert len(prompt["prompt_token_ids"]) >= 30000 + run([("aggregate-32k", prompt, ["blue"])], rank, "boundary", [f"{salt}/aggregate"]) + # Sustained batches of 16, each request in its own namespace. + for iteration in range(3): + namespaces = [f"{salt}/sustained/{iteration}/{index}" for index in range(16)] + hits = run((new * 4)[:16], rank, f"sustained-{iteration}", namespaces) + if any(hit for hit in hits if hit is not None): + failures.append(f"FAIL: rank {rank} sustained-{iteration}: reused prefixes") + windows[rank] = (window_start, time.time(), length) + + if full: + with _openai_client(llm) as (server, client): + for name, messages, expected in [ + ( + "openai-single", + [ + { + "role": "user", + "content": [ + image_part(red), + text_part("What color is the large shape? One color word."), + ], + } + ], + ["red"], + ), + ("openai-pair", pair_messages, ["red", "blue"]), + ]: + response = client.post( + "/v1/chat/completions", + json={ + "model": server.model, + "messages": messages, + "max_tokens": ANSWER_TOKENS, + "temperature": 0, + "skip_special_tokens": False, + "chat_template_kwargs": {"reasoning_effort": "low"}, + }, + ) + row = { + "name": name, + "status_code": response.status_code, + "body": response.json(), + } + stream.write(json.dumps(row) + "\n") + if response.status_code != 200: + failures.append(f"FAIL: {name}: HTTP {response.status_code}") + continue + answer = final_answer(row["body"]["choices"][0]["message"]["content"]) + if not answer_matches(answer, expected): + failures.append(f"INCONCLUSIVE: {name}: {answer!r}") + # response_format is stated in the prompt's model settings and enforced. + response = client.post( + "/v1/chat/completions", + json={ + "model": server.model, + "messages": [ + { + "role": "user", + "content": [ + image_part(red), + text_part("What color is the large shape?"), + ], + } + ], + "max_tokens": 64, + "temperature": 0, + "reasoning_effort": "low", + "response_format": { + "type": "json_schema", + "json_schema": { + "name": "color", + "schema": _COLOR_SCHEMA, + "strict": True, + }, + }, + }, + ) + row = { + "name": "openai-structured", + "status_code": response.status_code, + "body": response.json(), + } + stream.write(json.dumps(row) + "\n") + if response.status_code != 200: + failures.append(f"FAIL: openai-structured: HTTP {response.status_code}") + else: + check_structured( + "openai-structured", + row["body"]["choices"][0]["message"]["content"], + failures, + ) + + # Structured outputs are on by default: guided decoding holds the answer to the schema. + if llm.args.guided_decoding_backend != "xgrammar": + failures.append( + f"FAIL: guided decoding backend {llm.args.guided_decoding_backend!r}, not xgrammar" + ) + else: + output = llm.generate( + single(red, question="What color is the large shape? Answer in JSON."), + SamplingParams( + max_tokens=64, + temperature=0, + guided_decoding=GuidedDecodingParams(json=_COLOR_SCHEMA), + ), + use_tqdm=False, + ) + text = output.outputs[0].text + stream.write(json.dumps({"name": "structured", "text": text}) + "\n") + check_structured("structured", text, failures) + cases += 1 + + result = { + "seconds": time.monotonic() - started, + "cases": cases, + "ranks": list(ranks), + "windows": windows, + "failures": failures, + } + (out / "mm_functional_summary.json").write_text(json.dumps(result, indent=2) + "\n") + return result diff --git a/tests/integration/defs/accuracy/modeling_v2/ruler.py b/tests/integration/defs/accuracy/modeling_v2/ruler.py new file mode 100644 index 000000000000..eb46f13e24e6 --- /dev/null +++ b/tests/integration/defs/accuracy/modeling_v2/ruler.py @@ -0,0 +1,425 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Frozen RULER inputs and the upstream scorer for the 32K accuracy test and the 1M witness.""" + +import hashlib +import json +from collections import Counter, defaultdict +from collections.abc import Callable +from pathlib import Path +from typing import TYPE_CHECKING + +import yaml + +if TYPE_CHECKING: + from tensorrt_llm import LLM + +TASKS = ( + "niah_single_1", + "niah_multikey_2", + "niah_multikey_3", + "niah_multiquery", + "niah_multivalue", + "ruler_vt", + "ruler_fwe", + "ruler_qa_squad", + "ruler_qa_hotpot", +) +LENGTHS = (4096, 8192, 16384, 32768) +PROFILES = { + "quick": ( + "ruler_lite.jsonl", + "2ea8232784b07631deb7157ce5cca6cf6b4a079af4cabcf5a666df3cd68f7452", + 4, + ), + "full": ( + "ruler_full.jsonl", + "23bb0ad7cf2422148d82fca3b651506b8829ebce161643dc3b6ae939dba32007", + 50, + ), +} +# The 1M specialization's frozen rows, per cell by length: rows 0-9 of each task at 64K and 128K +# and two rows of the 1M bin (0 and 1; niah_single_1 keeps 7 and 9, its cache-witness pair), taken +# from the frozen long-context sets of the reference runs. The 1M bin keeps its label where a +# task's prompts are shorter (niah_multiquery / niah_multivalue about 654.5K tokens, +# ruler_qa_squad about 206.7K); n_prompt_tokens is the length. long_lite and long_full each score a +# subset of them (LONG_ROWS). +PROFILES["long_lite"] = PROFILES["long_full"] = ( + "ruler_long.jsonl", + "5873d7313db5ee6ee31fcbf2b966647aa15fe7cbd39589018f9d5f6a5278eb7e", + {65536: 10, 131072: 10, 1048576: 2}, +) +# The rows each 1M profile scores, by cell. Both hold the cache-witness pairs (LONG_PAIRS). +# long_lite adds two niah_single_1 / ruler_vt rows per cell at 64K and 128K, so one full 1M +# prefill; long_full every task's ten rows there, its gated means, and one ruler_vt row at 1M. +LONG_ROWS = { + "long_lite": { + ("niah_single_1", 65536): (0, 1), + ("ruler_vt", 65536): (0, 1), + ("niah_single_1", 131072): (1, 8), + ("ruler_vt", 131072): (0, 1), + ("niah_single_1", 1048576): (7, 9), + }, + "long_full": { + **{(task, length): range(10) for task in TASKS for length in (65536, 131072)}, + ("niah_single_1", 1048576): (7, 9), + ("ruler_vt", 1048576): (0,), + }, +} +# The tokenizer of every profile's prompt_ids: the checkpoint's tekken.json as tokenizer_digest +# reads it. +TOKENIZER_SHA256 = "e8e1f1927d59af6c41d5c0e8773f527199b89d46a7d38f8317ba39555d50319b" +PAIRS = ((4096, 0, 12), (8192, 13, 29), (16384, 15, 29), (32768, 12, 39)) +# The long profile's niah_single_1 pairs, the rows of each cell sharing the longest prefix: +# 127337 of about 130.9K tokens at 128K and 952097 of about 1048.3K in the 1M bin, so the second +# request of a pair prefills its last chunks over a cached context of up to 1M tokens. +LONG_PAIRS = ((131072, 1, 8), (1048576, 7, 9)) +# A fresh-salt replay of a pair's first row only up to this length: at 1M it would be one more +# full 1M prefill, and the 128K pair and the mixed phase's new requests already show isolation. +COLD_REPLAY_MAX = 131072 +# The lengths whose RULER mean has a floor in references/ruler.yaml, by profile. +GATED_LENGTHS = {"full": LENGTHS, "long_full": (65536, 131072)} +# A completed request at least this long witnesses the 1M envelope. +LONG_WITNESS_TOKENS = 1_000_000 + + +def cell_counts(profile: str) -> dict[int, int]: + """Rows per (task, length) cell of a profile, by length.""" + per_cell = PROFILES[profile][2] + return per_cell if isinstance(per_cell, dict) else dict.fromkeys(LENGTHS, per_cell) + + +def tokenizer_digest(tokenizer: Path) -> str: + """SHA-256 of tekken.json without its model settings, in canonical JSON. + + The vocabulary, special tokens, pre-tokenizer pattern and image settings fix every token id; + the model settings (the reasoning efforts the checkpoint declares) fix none. + """ + tekken = json.loads(tokenizer.read_bytes()) + tekken.pop("model_settings_builder", None) + return hashlib.sha256( + json.dumps(tekken, sort_keys=True, separators=(",", ":")).encode() + ).hexdigest() + + +def load_inputs(directory: Path, profile: str, tokenizer: Path) -> list[dict]: + """Read the prepared v1 inputs, whose token IDs require the matching tokenizer.""" + filename, digest, _ = PROFILES[profile] + path = directory / filename + if not path.is_file(): + raise FileNotFoundError(f"Missing {path}; see modeling_v2/RULER.md for data setup") + content = path.read_bytes() + assert hashlib.sha256(content).hexdigest() == digest, "Frozen RULER inputs changed" + assert tokenizer_digest(tokenizer) == TOKENIZER_SHA256, ( + "RULER v1 token IDs require the baseline tekken.json vocabulary" + ) + rows = [json.loads(line) for line in content.splitlines()] + assert Counter((r["task"], r["max_length"]) for r in rows) == { + (task, length): count for task in TASKS for length, count in cell_counts(profile).items() + } + if profile in LONG_ROWS: + cells = LONG_ROWS[profile] + rows = [r for r in rows if r["idx"] in cells.get((r["task"], r["max_length"]), ())] + return rows + + +def shared_prefix(left: list[int], right: list[int]) -> int: + return next( + (i for i, (x, y) in enumerate(zip(left, right)) if x != y), + min(len(left), len(right)), + ) + + +def score(text: str, row: dict) -> float: + from lm_eval.tasks.ruler.common_utils import ( + postprocess_pred, + string_match_all, + string_match_part, + ) + + matcher = string_match_part if row["metric"] == "part" else string_match_all + return float(matcher(postprocess_pred([text]), [row["outputs"]])) + + +def validate_output(result, row: dict) -> None: + assert list(result.prompt_token_ids) == row["prompt_ids"], "Prompt changed or truncated" + assert len(result.outputs) == 1 + output = result.outputs[0] + assert output.finish_reason in ("stop", "length"), output.finish_reason + assert len(output.token_ids) <= row["max_gen_toks"] + + +def cached_tokens(result, row: dict) -> int: + """Require actual cache telemetry for the reuse panel, including cold requests.""" + assert result.outputs[0].request_perf_metrics is not None, ( + "INCONCLUSIVE: missing cache telemetry" + ) + hit = result.cached_tokens + assert type(hit) is int and 0 <= hit <= len(row["prompt_ids"]), hit + return hit + + +def _key(row: dict) -> tuple: + return row["task"], row["max_length"], row["idx"] + + +def mixed_rows(remaining: list[dict], profile: str, concurrency: int) -> list[dict]: + """The new requests of the mixed phase, which must each retrieve exactly. + + quick / full take the first rows left. The 1M profiles take them from their exact-retrieval + cells below their longest length, so the 1M rows stay in the scoring phase. + """ + pool = remaining + if profile.startswith("long"): + top = max(row["max_length"] for row in remaining) + pool = [ + row for row in remaining if row["max_length"] < top and exact_task(profile, row["task"]) + ] + return pool[: concurrency // 2] + + +def _reuse_panel( + rows: list[dict], + generate: Callable[[list[dict], list[str | None], str], list], + concurrency: int, + profile: str, + *, + attention_dp_size: int = 1, +) -> list[str]: + by_id = {_key(row): row for row in rows} + anchors, failures = [], [] + panel_ids = set() + for length, a, b in LONG_PAIRS if profile.startswith("long") else PAIRS: + if ("niah_single_1", length, a) not in by_id: # rows the caller left out + continue + left, right = [by_id[("niah_single_1", length, idx)] for idx in (a, b)] + salt = f"ruler/{length}" + shared = shared_prefix(left["prompt_ids"], right["prompt_ids"]) + cold = generate([left], [salt], "first_a")[0] + suffix = generate([right], [salt], "first_b")[0] + warm = generate([left], [salt], "warm_a")[0] + isolated = ( + generate([left], [salt + "/cold"], "cold_a")[0] if length <= COLD_REPLAY_MAX else 0 + ) + assert cold == isolated == 0 and suffix <= shared, "Illegal cache hit" + # The long pairs share most of their prompt: the hit has to cover most of it. + short = length > 4096 and suffix <= 2048 + if profile.startswith("long"): + short = short or 2 * suffix < shared + if warm == 0 or short: + failures.append(f"INCONCLUSIVE: missing warm/cached-context witness at {length}") + anchors.append((left, salt)) + panel_ids.update((_key(left), _key(right))) + + remaining = [row for row in rows if _key(row) not in panel_ids] + cold_rows = mixed_rows(remaining, profile, concurrency) + if attention_dp_size > 1 and cold_rows: + # One rank receives the pinned pair; leave enough requests for every other rank + # to exercise a full prefill chunk. Replayed cold rows get distinct salts below. + cold_rows = [ + cold_rows[i % len(cold_rows)] + for i in range(max(len(cold_rows), attention_dp_size // 2 + 1)) + ] + mixed, salts = [], [] + for i, row in enumerate(cold_rows): + anchor, salt = anchors[i % len(anchors)] + mixed.extend((anchor, row)) + salts.extend((salt, f"ruler/mixed/{i}")) + hits = generate(mixed, salts, "mixed") + assert not any(hits[1::2]), "Isolated new requests reused cached prefixes" + if not any(hits[::2]): + failures.append("INCONCLUSIVE: mixed workload had no warm hits") + return failures + + +def load_references( + model_name: str, quant_algo: str, kv_cache_quant_algo: str, profile: str = "full" +) -> dict: + """Read explicit engineering floors in the accuracy reference YAML format.""" + path = Path(__file__).parents[1] / "references/ruler.yaml" + with path.open() as stream: + entries = yaml.safe_load(stream)[model_name] + return { + str(length): next( + entry + for entry in entries + if entry.get("quant_algo") == quant_algo + and entry.get("kv_cache_quant_algo") == kv_cache_quant_algo + and entry.get("extra_acc_spec") == f"{profile}-{length}" + ) + for length in GATED_LENGTHS[profile] + } + + +def exact_task(profile: str, task: str) -> bool: + """Whether every row of the task's cells must be right: retrieval, not a scored benchmark.""" + return task.startswith("niah_") or task == "ruler_vt" + + +def summarize(first: dict, profile: str, references: dict) -> dict: + """Score first responses only; cache replays never replace an original answer.""" + cells = defaultdict(list) + for (task, length, _), value in first.items(): + cells[(task, length)].append(value) + means = {} + for length in cell_counts(profile): + tasks = [task for task in TASKS if (task, length) in cells] + if not tasks: + continue + means[str(length)] = ( + 100 * sum(sum(cells[(task, length)]) / len(cells[(task, length)]) for task in tasks) + ) / len(tasks) + failures = [] + for (task, length), values in cells.items(): + if exact_task(profile, task) and min(values) != 1: + label = "INCONCLUSIVE" if profile in ("quick", "long_lite") else "FAIL" + failures.append(f"{label}: {task}@{length}") + thresholds = {} + for length, reference in references.items(): + assert ( + sum(len(cells.get((task, int(length)), ())) for task in TASKS) + == (reference["num_samples"]) + ) + thresholds[length] = reference["threshold"] + if means[length] < reference["threshold"]: + failures.append(f"FAIL: RULER mean at {length} below {reference['threshold']}") + result = { + "n": len(first), + "mean_by_length": means, + "threshold_by_length": thresholds, + "accuracy_failures": failures, + } + if profile.startswith("long"): + result["mean_by_cell"] = { + f"{task}@{length}": 100 * sum(values) / len(values) + for (task, length), values in cells.items() + } + return result + + +def evaluate( + llm: "LLM", + rows: list[dict], + profile: str, + out: Path, + concurrency: int, + model_name: str, + *, + collect_spec: bool = False, +) -> dict: + """Score each original question once on a block-reuse LLM; replays check the cache.""" + from tensorrt_llm import SamplingParams + from tensorrt_llm.scheduling_params import SchedulingParams + + from .spec_evidence import request_metrics + + references = ( + load_references( + model_name, + llm.args.quant_config.quant_algo, + llm.args.quant_config.kv_cache_quant_algo, + profile, + ) + if profile in GATED_LENGTHS + else {} + ) + first = {} + reuse_failures = [] + with (out / "ruler_predictions.jsonl").open("w") as stream: + + def generate(batch: list[dict], salts: list[str | None], phase: str) -> list: + check_reuse = phase != "benchmark" + scheduling = None + if profile.startswith("long") and llm.args.enable_attention_dp and check_reuse: + # Keep the longest warm request and its cold partner on the rank that ran + # the cache probes. Otherwise a small panel can put every request on a + # different rank, never exercising decode and prefill together. + pair = max(range(0, len(batch), 2), key=lambda i: len(batch[i]["prompt_ids"])) + scheduling = [ + SchedulingParams(attention_dp_rank=0, attention_dp_relax=False) + if pair <= i < pair + 2 + else SchedulingParams() + for i in range(len(batch)) + ] + outputs = llm.generate( + [row["prompt_ids"] for row in batch], + [ + SamplingParams( + max_tokens=row["max_gen_toks"], + temperature=0, + return_perf_metrics=check_reuse or collect_spec, + ) + for row in batch + ], + cache_salt=salts, + scheduling_params=scheduling, + ) + hits = [] + for row, salt, result in zip(batch, salts, outputs, strict=True): + validate_output(result, row) + hit = cached_tokens(result, row) if check_reuse else None + value = score(result.outputs[0].text, row) + first.setdefault(_key(row), value) + stream.write( + json.dumps( + { + "id": _key(row), + "request_id": result.request_id, + "prompt_tokens": len(row["prompt_ids"]), + "spec_decoding": request_metrics(result) if collect_spec else None, + "phase": phase, + "cache_salt": salt, + "cached_tokens": hit, + "score": value, + "text": result.outputs[0].text, + "output_ids": list(result.outputs[0].token_ids), + "finish_reason": result.outputs[0].finish_reason, + } + ) + + "\n" + ) + stream.flush() + if check_reuse and value != 1: + reuse_failures.append(f"INCONCLUSIVE: retrieval miss {_key(row)} {phase}") + hits.append(hit) + return hits + + reuse_failures.extend( + _reuse_panel( + rows, + generate, + concurrency, + profile, + attention_dp_size=( + llm.args.tensor_parallel_size + if profile.startswith("long") and llm.args.enable_attention_dp + else 1 + ), + ) + ) + remaining = [row for row in rows if _key(row) not in first] + for start in range(0, len(remaining), concurrency): + batch = remaining[start : start + concurrency] + generate(batch, [None] * len(batch), "benchmark") + print(f"RULER {len(first)}/{len(rows)} scored", flush=True) + + result = summarize(first, profile, references) + result["reuse_failures"] = reuse_failures + result["failures"] = result["accuracy_failures"] + reuse_failures + (out / "ruler_summary.json").write_text(json.dumps(result, indent=2) + "\n") + assert len(first) == len(rows) and not result["failures"], result + return result + + +def longest_completed(out: Path, rows: list[dict]) -> int: + """Prompt tokens of the longest request evaluate() recorded, 0 if none. + + It records an output only after validate_output held its prompt to the row's prompt_ids and + its finish to stop or length, so the row's length is the completed request's. + """ + lengths = {(r["task"], r["max_length"], r["idx"]): len(r["prompt_ids"]) for r in rows} + path = out / "ruler_predictions.jsonl" + if not path.exists(): + return 0 + records = (json.loads(line) for line in path.read_text().splitlines()) + return max((lengths[tuple(record["id"])] for record in records), default=0) diff --git a/tests/integration/defs/accuracy/modeling_v2/spec_evidence.py b/tests/integration/defs/accuracy/modeling_v2/spec_evidence.py new file mode 100644 index 000000000000..02babe26c320 --- /dev/null +++ b/tests/integration/defs/accuracy/modeling_v2/spec_evidence.py @@ -0,0 +1,247 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Request-linked evidence for long speculative decoding, collected only by the test.""" + +import contextlib +import json +from collections import defaultdict +from pathlib import Path + +_original_collect = None + + +def request_snapshot(req, phase: str) -> dict: + """Results expose client IDs; executor IDs are a separate rank-local namespace.""" + return dict( + id=req.py_client_id, + executor_id=req.request_id, + phase=phase, + dummy=bool(req.is_dummy), + prompt_tokens=req.orig_prompt_len, + tokens=req.get_num_tokens(0), + draft_tokens=len(req.py_draft_tokens), + ) + + +def _install_trace(directory: str | None) -> None: + from tensorrt_llm._torch.pyexecutor.py_executor import PyExecutor + from tensorrt_llm._utils import mpi_barrier, mpi_rank + + global _original_collect + # A barrier makes each submitted task occupy a different MPI worker. + mpi_barrier() + if directory is None: + PyExecutor._collect_scheduled_batch_stats = _original_collect + _original_collect = None + return + assert _original_collect is None + _original_collect = PyExecutor._collect_scheduled_batch_stats + path = Path(directory) / f"rank-{mpi_rank()}.jsonl" + + def collect(executor, batch): + stats = _original_collect(executor, batch) + requests = [] + for phase, items in ( + ("context", batch.context_requests), + ("generation", batch.generation_requests), + ): + for req in items: + requests.append(request_snapshot(req, phase)) + with path.open("a") as stream: + stream.write( + json.dumps( + dict( + iteration=executor.iter_counter, + rank=executor.dist.rank, + context_tokens=stats.num_ctx_tokens, + requests=requests, + ) + ) + + "\n" + ) + return stats + + PyExecutor._collect_scheduled_batch_stats = collect + + +@contextlib.contextmanager +def traced_llm(model: str, trace: Path, **kwargs): + """Use the normal MPI substrate, with a scoped observer on every worker. + + ``env_overrides`` is required: the engine is built inside its environment scope. + """ + from tensorrt_llm import LLM + from tensorrt_llm.executor.utils import create_mpi_comm_session, get_spawn_proxy_process_env + from tensorrt_llm.llmapi.mpi_session import MpiPoolSession + + from ._env import scoped_env_overrides + + trace.mkdir() + world = kwargs["tensor_parallel_size"] + session = ( + create_mpi_comm_session(world) + if get_spawn_proxy_process_env() + else MpiPoolSession(n_workers=world) + ) + installed = False + try: + session.submit_sync(_install_trace, str(trace)) + installed = True + with ( + scoped_env_overrides(kwargs.pop("env_overrides")) as rank_env, + LLM(model, _mpi_session=session, env_overrides=rank_env, **kwargs) as llm, + ): + yield llm + finally: + try: + if installed: + session.submit_sync(_install_trace, None) + finally: + session.shutdown() + + +def request_metrics(result) -> dict: + """Keep cumulative counts and positional counts; zero/missing counts are not proof.""" + totals = result.spec_dec_totals + accepted, drafted = totals if totals is not None else (0, 0) + return dict( + accepted=accepted, + drafted=drafted, + rejected=drafted - accepted, + drafted_per_position=result.per_pos_drafted, + accepted_per_position=result.per_pos_accepted, + ) + + +def summarize( + predictions: list[dict], + steps: list[dict], + need: int, + world: int, + attention_dp: bool, + max_context_tokens: int = 32768, +) -> dict: + """Require actual long verification, reuse, rejection, mixed steps and DEP padding.""" + by_request = {row["request_id"]: row for row in predictions} + failures = [] + grouped = defaultdict(lambda: dict(accepted=0, drafted=0, iterations=0, requests=0)) + long_requests, rejected = set(), set() + reused = defaultdict(set) + for row in predictions: + metrics = row["spec_decoding"] + accepted, drafted = metrics["accepted"], metrics["drafted"] + assert 0 <= accepted <= drafted, metrics + positions = metrics["drafted_per_position"] or [] + group = grouped[f"{row['id'][1]}/{row['phase']}"] + group["accepted"] += accepted + group["drafted"] += drafted + group["iterations"] += positions[0] if positions else 0 + group["requests"] += 1 + if ( + row["prompt_tokens"] >= need + and len(positions) >= 3 + and min(positions[:3]) > 0 + and not any(positions[3:]) + ): + long_requests.add(row["request_id"]) + if ( + row["phase"] in ("first_b", "warm_a") + and row["cached_tokens"] > 0 + and row["score"] == 1 + and accepted > 0 + ): + reused[row["phase"]].add(row["request_id"]) + if drafted > accepted and row["score"] == 1: + rejected.add(row["request_id"]) + for group in grouped.values(): + group["acceptance_length"] = ( + 1 + group["accepted"] / group["iterations"] if group["iterations"] else None + ) + + verified, mixed, iterations = set(), [], defaultdict(dict) + for step in steps: + iterations[step["iteration"]][step["rank"]] = step + assert 0 <= step["context_tokens"] <= max_context_tokens, step + live_long = [ + req + for req in step["requests"] + if not req["dummy"] + and req["phase"] == "generation" + and req["id"] in long_requests + and req["draft_tokens"] == 3 + and req["prompt_tokens"] == by_request[req["id"]]["prompt_tokens"] + ] + verified.update(req["id"] for req in live_long) + cold_chunks = [ + req["id"] + for req in step["requests"] + if not req["dummy"] + and req["phase"] == "context" + and req["id"] in by_request + and req["prompt_tokens"] == by_request[req["id"]]["prompt_tokens"] + and req["prompt_tokens"] > max_context_tokens + and by_request[req["id"]]["cached_tokens"] == 0 + ] + if live_long and cold_chunks and step["context_tokens"] > 0: + mixed.append( + dict(iteration=step["iteration"], rank=step["rank"], cold_requests=cold_chunks) + ) + padding = [] + if attention_dp: + for iteration, ranks in iterations.items(): + if set(ranks) != set(range(world)): + continue + live = [ + rank + for rank, step in ranks.items() + if any( + not req["dummy"] + and req["id"] in verified + and req["phase"] == "generation" + and req["draft_tokens"] == 3 + for req in step["requests"] + ) + ] + dummy = [ + rank + for rank, step in ranks.items() + if any( + req["dummy"] and req["phase"] == "generation" and req["draft_tokens"] == 3 + for req in step["requests"] + ) + ] + if live and set(dummy) - set(live): + padding.append(dict(iteration=iteration, live_ranks=live, dummy_ranks=dummy)) + if not padding: + failures.append("INCONCLUSIVE: no rank-local long speculative DEP dummy witness") + if not verified: + failures.append(f"INCONCLUSIVE: no completed {need}-token four-token verification") + if any(not reused[phase] & verified for phase in ("first_b", "warm_a")): + failures.append("INCONCLUSIVE: no long speculative warm/cached-prefix retrieval") + if not rejected & verified: + failures.append("INCONCLUSIVE: no successful long retrieval following draft rejection") + if not mixed: + failures.append("INCONCLUSIVE: no same-step long verification and chunked prefill") + return dict( + required_prompt_tokens=need, + verified_requests=sorted(verified), + rejected_requests=sorted(rejected & verified), + by_length_phase=dict(grouped), + mixed_steps=mixed, + dep_padding=padding, + failures=failures, + ) + + +def evaluate(root: Path, need: int, world: int, attention_dp: bool) -> dict: + predictions = [ + json.loads(line) for line in (root / "ruler_predictions.jsonl").read_text().splitlines() + ] + steps = [ + json.loads(line) + for path in sorted((root / "spec-trace").glob("rank-*.jsonl")) + for line in path.read_text().splitlines() + ] + result = summarize(predictions, steps, need, world, attention_dp) + (root / "spec_evidence.json").write_text(json.dumps(result, indent=2) + "\n") + return result diff --git a/tests/integration/defs/accuracy/references/acceptance_length.yaml b/tests/integration/defs/accuracy/references/acceptance_length.yaml index 60d2589eb5a5..0c3b95692a66 100644 --- a/tests/integration/defs/accuracy/references/acceptance_length.yaml +++ b/tests/integration/defs/accuracy/references/acceptance_length.yaml @@ -1,5 +1,28 @@ # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 +# Mistral Large 4 NVFP4, DSpark K=3: an independent reference implementation, +# TP4/GB300, CUDA 13.4. Full 1319-sample, 5-shot GSM8K with the quick/full test's saved +# prompt token IDs, no chat template, greedy decoding, 256 output tokens, +# concurrency 16. AL = 1 + 93126 accepted drafts / 42248 verification steps. +# All six specializations use this independent reference; min_al is 95%. +ModelingV2MistralLarge4Nvfp4::tep4::dspark3: + ref_al: 3.20427002461655 + min_al: 3.044056523385722 +ModelingV2MistralLarge4Nvfp4::dep4::dspark3: + ref_al: 3.20427002461655 + min_al: 3.044056523385722 +ModelingV2MistralLarge4Nvfp4::tp4::dspark3: + ref_al: 3.20427002461655 + min_al: 3.044056523385722 +ModelingV2MistralLarge4Nvfp4::tep8::dspark3: + ref_al: 3.20427002461655 + min_al: 3.044056523385722 +ModelingV2MistralLarge4Nvfp4::dep8::dspark3: + ref_al: 3.20427002461655 + min_al: 3.044056523385722 +ModelingV2MistralLarge4Nvfp4::tp8::dspark3: + ref_al: 3.20427002461655 + min_al: 3.044056523385722 TestQwen3_4B::test_eagle3: ref_al: 2.02809899228679 min_al: 1.9266940426724504 diff --git a/tests/integration/defs/accuracy/references/gpqa_diamond.yaml b/tests/integration/defs/accuracy/references/gpqa_diamond.yaml index 7f31760a6c69..1a9058cb2cdd 100644 --- a/tests/integration/defs/accuracy/references/gpqa_diamond.yaml +++ b/tests/integration/defs/accuracy/references/gpqa_diamond.yaml @@ -1,3 +1,16 @@ +mistralai/Mistral-Large-4-1T-A52B: + # NVFP4, AA zero-shot, greedy, 32K total context budget, set on an earlier NVFP4 checkpoint + # at reasoning effort none. The released checkpoint declares no `none` effort; the test runs + # at `low` against this value. + - quant_algo: MIXED_PRECISION + kv_cache_quant_algo: FP8 + accuracy: 82.83 + threshold: 75.1 + - quant_algo: MIXED_PRECISION + kv_cache_quant_algo: FP8 + spec_dec_algo: DSpark + accuracy: 82.83 + threshold: 75.1 deepseek-ai/DeepSeek-R1: - quant_algo: NVFP4 accuracy: 70.45 diff --git a/tests/integration/defs/accuracy/references/gsm8k.yaml b/tests/integration/defs/accuracy/references/gsm8k.yaml index 675e7ab36dbd..b2f0adf04dab 100644 --- a/tests/integration/defs/accuracy/references/gsm8k.yaml +++ b/tests/integration/defs/accuracy/references/gsm8k.yaml @@ -680,3 +680,31 @@ zai-org/GLM-5.3-Flash: kv_cache_quant_algo: FP8 spec_dec_algo: MTP accuracy: 92.87 +mistralai/Mistral-Large-4-1T-A52B: + # The ModelOpt MIXED_PRECISION export (NVFP4 routed experts, FP8 128x128 block linears), FP8 + # latent KV cache, set on an earlier NVFP4 checkpoint than the released one. Measured on an + # independent reference implementation, TP4 on 4 GB300s, harness protocol (5-shot, no chat + # template, greedy, 256 new tokens): 91.888 strict-match, 91.964 flexible-extract over the + # 1319-sample split. + - quant_algo: MIXED_PRECISION + kv_cache_quant_algo: FP8 + accuracy: 91.926 + threshold: 90.43 + # DSpark target verification keeps the same target accuracy requirement. + - quant_algo: MIXED_PRECISION + kv_cache_quant_algo: FP8 + spec_dec_algo: DSpark + accuracy: 91.926 + threshold: 90.43 +mistralai/Mistral-Large-4-1T-A52B-gsm8k-24shot: + # The NVFP4 (MIXED_PRECISION) checkpoint on a long-context GSM8K protocol: 24 exemplars, so + # every prompt is 3152-5115 tokens and the DSA indexer's top-2048 selection prunes context on + # every sample. `trtllm-eval gsm8k --num_fewshot 24 --max_input_length 8192`, otherwise the + # 5-shot protocol above. Scored on 2026-10-21, on an earlier NVFP4 checkpoint than the released + # one, by an independent reference implementation (TP4 on 4 GB300s) through + # tensorrt_llm.evaluate.GSM8K's task code, all 1319 samples, mean of strict-match and + # flexible-extract, on its fp8_ds_mla latent cache: 91.96 (strict 91.89, flexible 92.04); bf16 + # latent cache 92.23. + - quant_algo: MIXED_PRECISION + kv_cache_quant_algo: FP8 + accuracy: 91.96 diff --git a/tests/integration/defs/accuracy/references/mmmu.yaml b/tests/integration/defs/accuracy/references/mmmu.yaml index 36f1a0aeaa99..8fba9b49c1d1 100644 --- a/tests/integration/defs/accuracy/references/mmmu.yaml +++ b/tests/integration/defs/accuracy/references/mmmu.yaml @@ -79,3 +79,34 @@ moonshotai/Kimi-K3: zai-org/GLM-5.3-Flash: - quant_algo: FP8_BLOCK_SCALES accuracy: 77.22 +# An independent reference implementation on the CI MMMU dataset. +# Native NVFP4 (an earlier checkpoint); reasoning=none, greedy, max_tokens=512, seed=0. +# The released checkpoint declares no `none` effort; the test runs at `low` against these +# values. +# Seed-0 selection: 3 questions per subject (90 total), or all 900 validation questions. +# Two runs in different batch orders: full 581/900 both times; quick 57/90 and 62/90, and +# the quick reference is their mean. +mistralai/Mistral-Large-4-1T-A52B: + - quant_algo: MIXED_PRECISION + kv_cache_quant_algo: FP8 + extra_acc_spec: native-mm-quick + num_samples: 90 + accuracy: 66.11111111111111 + - quant_algo: MIXED_PRECISION + kv_cache_quant_algo: FP8 + extra_acc_spec: native-mm-full + num_samples: 900 + accuracy: 64.55555555555556 + # DSpark target verification keeps the same quality and sample-count gates. + - quant_algo: MIXED_PRECISION + kv_cache_quant_algo: FP8 + spec_dec_algo: DSpark + extra_acc_spec: native-mm-quick + num_samples: 90 + accuracy: 66.11111111111111 + - quant_algo: MIXED_PRECISION + kv_cache_quant_algo: FP8 + spec_dec_algo: DSpark + extra_acc_spec: native-mm-full + num_samples: 900 + accuracy: 64.55555555555556 diff --git a/tests/integration/defs/accuracy/references/ruler.yaml b/tests/integration/defs/accuracy/references/ruler.yaml new file mode 100644 index 000000000000..71dce9a9e7d0 --- /dev/null +++ b/tests/integration/defs/accuracy/references/ruler.yaml @@ -0,0 +1,47 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +mistralai/Mistral-Large-4-1T-A52B: + # An independent reference implementation, NVFP4, TP4 on 4 GB300s. + # RULER v1: 9 tasks x 50 samples per length; minima of three baseline runs. + # The explicit floors use the engineering acceptance margin of 1.5 points. + - quant_algo: MIXED_PRECISION + kv_cache_quant_algo: FP8 + extra_acc_spec: full-4096 + num_samples: 450 + accuracy: 95.67 + threshold: 94.17 + - quant_algo: MIXED_PRECISION + kv_cache_quant_algo: FP8 + extra_acc_spec: full-8192 + num_samples: 450 + accuracy: 95.37 + threshold: 93.87 + - quant_algo: MIXED_PRECISION + kv_cache_quant_algo: FP8 + extra_acc_spec: full-16384 + num_samples: 450 + accuracy: 94.26 + threshold: 92.76 + - quant_algo: MIXED_PRECISION + kv_cache_quant_algo: FP8 + extra_acc_spec: full-32768 + num_samples: 450 + accuracy: 94.78 + threshold: 93.28 + # long_full: RULER at 64K and 128K, 9 tasks x 10 samples per length on the 1M specialization. + # References are the reference implementation's on an earlier NVFP4 checkpoint than the + # released one (the full references above are the released checkpoint's), over 50 samples per + # cell on the same long-context sets; the floors use an engineering margin of 3 points for the + # smaller sample. + - quant_algo: MIXED_PRECISION + kv_cache_quant_algo: FP8 + extra_acc_spec: long_full-65536 + num_samples: 90 + accuracy: 96.03 + threshold: 93.03 + - quant_algo: MIXED_PRECISION + kv_cache_quant_algo: FP8 + extra_acc_spec: long_full-131072 + num_samples: 90 + accuracy: 95.22 + threshold: 92.22 diff --git a/tests/integration/defs/accuracy/test_mistral_large4_disaggregated.py b/tests/integration/defs/accuracy/test_mistral_large4_disaggregated.py new file mode 100644 index 000000000000..86361020f5c7 --- /dev/null +++ b/tests/integration/defs/accuracy/test_mistral_large4_disaggregated.py @@ -0,0 +1,244 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Two-node Mistral Large 4 TEP4 serving with the existing GSM8K accuracy gate.""" + +import json +import os +import socket +import time +from pathlib import Path + +import openai +import pytest +import requests +import test_common.magic_import # noqa: F401 +from mpi4py import MPI +from test_common import cluster_env +from test_common.http_utils import wait_for_endpoint_down, wait_for_endpoint_ready + +__extra_import_path__ = ["~/tests/unittest/llmapi/apps"] +from openai_server import RemoteDisaggOpenAIServer, RemoteOpenAIServer + +from defs.accuracy.accuracy_core import GSM8K +from defs.accuracy.test_disaggregated_serving import ( + DuckLLM, + MyThreadPoolExecutor, + Result, + run_accuracy_test, +) +from defs.conftest import llm_models_root +from tensorrt_llm._torch.modeling_v2 import MODELING_V2_VALIDATE_ENV +from tensorrt_llm._utils import get_free_port, get_sm_version +from tensorrt_llm.executor.utils import LlmLauncherEnvs, create_mpi_comm_session +from tensorrt_llm.llmapi import CompletionOutput, RequestOutput, SamplingParams +from tensorrt_llm.llmapi.disagg_utils import ServerRole +from tensorrt_llm.llmapi.llm_args import LlmArgs +from tensorrt_llm.llmapi.mpi_session import get_mpi_world_size, split_mpi_env +from tensorrt_llm.llmapi.tokenizer import load_hf_tokenizer + + +def _check_serving(url: str, model: str, output: Path) -> None: + """Exercise sparse context, prefix reuse, then the unchanged full GSM8K gate.""" + for name in ("INTEGRATION_TEST", "TRTLLM_ACCURACY_NO_REFERENCE"): + assert os.environ.get(name, "0") == "0", f"Disable {name} for baseline validation" + with ( + openai.OpenAI( + api_key="1234567890", base_url=f"{url}/v1", timeout=600, max_retries=0 + ) as client, + MyThreadPoolExecutor(max_workers=16) as pool, + ): + + def send_request(prompt: str, sampling_params: SamplingParams, streaming: bool): + assert not streaming and not sampling_params.use_beam_search + assert sampling_params.guided_decoding is None + response = client.completions.create( + model=model, + prompt=prompt, + stream=False, + max_tokens=sampling_params.max_tokens, + n=sampling_params.n, + best_of=sampling_params.best_of, + temperature=sampling_params.temperature if sampling_params.top_p is not None else 0, + top_p=sampling_params.top_p, + stop=sampling_params.stop, + seed=sampling_params.seed, + ) + result = Result( + id=0, + sampling_params=sampling_params, + outputs=[ + CompletionOutput(text=choice.text, index=index) + for index, choice in enumerate(response.choices) + ], + ) + request_output = RequestOutput._from_generation_result(result, prompt=prompt) + setattr(request_output, "result", result.result) + return request_output + + def generate_async(prompt: str, sampling_params: SamplingParams, streaming: bool = False): + future = pool.submit(send_request, prompt, sampling_params, streaming) + pool.futures.append(future) + return future + + llm = DuckLLM( + LlmArgs(model=model, checkpoint_format="mistral"), + load_hf_tokenizer(model), + generate_async, + url, + ) + llm.args.quant_config.quant_algo = "MIXED_PRECISION" + llm.args.quant_config.kv_cache_quant_algo = "FP8" + filler = "The grass is green. The sky is blue. There and back again. " + prompt = ( + filler * 600 + + "\nRemember this pass key: 71432.\n" + + filler * 600 + + "\nWhat is the pass key? Answer with just the number:" + ) + tokens = llm.tokenizer.encode(prompt) + assert 2048 < len(tokens) < 32700 + # One cold/warm pair also catches loss of the shared latent or indexer history. + with (output / "requests.jsonl").open("w") as evidence: + for warm in (False, True): + response = requests.post( + f"{url}/v1/completions", + json=dict(model=model, prompt=tokens, temperature=0, max_tokens=16), + timeout=600, + ) + response.raise_for_status() + result = response.json() + evidence.write(json.dumps(result) + "\n") + evidence.flush() + assert "71432" in result["choices"][0]["text"], result + usage = result["usage"] + assert usage["prompt_tokens"] == len(tokens), usage + assert usage["completion_tokens"] > 1, usage + cached = usage["prompt_tokens_details"]["cached_tokens"] + assert cached > 0 if warm else cached == 0, usage + run_accuracy_test( + llm, + "mistralai/Mistral-Large-4-1T-A52B", + [GSM8K], + extra_evaluator_kwargs={ + GSM8K: dict(num_fewshot=5, log_samples=True, output_path=str(output / "gsm8k")) + }, + ) + + +def _worker_info() -> tuple[int, str, int]: + MPI.COMM_WORLD.Barrier() + return MPI.COMM_WORLD.rank, socket.gethostname(), get_free_port() + + +def _serve_worker( + workers: list[tuple[int, str, int]], model: str, config: dict, output: Path, url: str +) -> dict: + rank = MPI.COMM_WORLD.rank + result = {"rank": rank} + if rank not in (0, 4): + return result + role = ServerRole.CONTEXT if rank == 0 else ServerRole.GENERATION + metrics_dir = output / f"rank{rank}" / "metrics" + # Keep node-local MPI instances separate from the outer CI worker pool. + env, _ = split_mpi_env( + [ + *LlmLauncherEnvs, + "tllm_mpi_size", + "MASTER_ADDR", + "MASTER_PORT", + "RANK", + "WORLD_SIZE", + "LOCAL_RANK", + "LOCAL_WORLD_SIZE", + ] + ) + cluster_env.apply_ucx_env(env, cluster_name=os.environ.get("SLURM_CLUSTER_NAME")) + with RemoteOpenAIServer( + model, + host=workers[rank][1], + port=workers[rank][2], + rank=0, + role=role, + extra_config={**config, "perf_metrics_output_dir": str(metrics_dir)}, + env=env, + wait=False, + ) as server: + wait_for_endpoint_ready(f"{url}/health", timeout=1800, server_proc=server.proc) + wait_for_endpoint_down(f"{url}/health", timeout=7200) + if role is ServerRole.GENERATION: + records = [ + json.loads(line) + for path in metrics_dir.glob("perf_metrics-generation-*.jsonl") + for line in path.read_text().splitlines() + if line.strip() + ] + assert len(records) == GSM8K.NUM_SAMPLES + 2 + assert all(record["status"] == "complete" for record in records) + for record in records: + timing = record["perf_metrics"]["timing_metrics"] + assert timing["kv_cache_size"] > 0 + assert 0 < timing["kv_cache_transfer_start"] <= timing["kv_cache_transfer_end"] + result["transfers"] = len(records) + return result + + +@pytest.mark.timeout(7200) +def test_mistral_large4_disagg(tmp_path: Path, capfd: pytest.CaptureFixture[str]) -> None: + assert get_mpi_world_size() == 8 and MPI.COMM_WORLD.size == 1, ( + "Run under the CI trtllm-llmapi-launch with two nodes and four workers per node" + ) + assert get_sm_version() == 103 + session = create_mpi_comm_session(8) + + workers = sorted(session.submit_sync(_worker_info)) + assert len(workers) == 8 and [rank for rank, _, _ in workers] == list(range(8)), workers + hosts = [host for _, host, _ in workers] + assert len(set(hosts)) == 2 and len(set(hosts[:4])) == len(set(hosts[4:])) == 1, hosts + model = f"{llm_models_root()}/Mistral-Large-4-1T-A52B-NVFP4" + output = Path(os.environ.get("TRTLLM_ACCURACY_OUTPUT_DIR", tmp_path)) + output.mkdir(parents=True, exist_ok=True) + proxy_port = get_free_port() + url = f"http://{hosts[0]}:{proxy_port}" + + worker_config = dict( + checkpoint_format="mistral", + tensor_parallel_size=4, + moe_expert_parallel_size=4, + enable_attention_dp=False, + modeling_v2="require", + env_overrides={MODELING_V2_VALIDATE_ENV: "1"}, + max_batch_size=16, + max_num_tokens=32768, + max_seq_len=32768, + enable_chunked_prefill=False, + cuda_graph_config=dict(max_batch_size=16, enable_padding=True), + kv_cache_config=dict(enable_block_reuse=True, free_gpu_memory_fraction=0.85), + cache_transceiver_config=dict(backend="NIXL", transceiver_runtime="PYTHON"), + return_perf_metrics=True, + ) + with ( + capfd.disabled(), + RemoteDisaggOpenAIServer( + ctx_servers=[f"{hosts[0]}:{workers[0][2]}"], + gen_servers=[f"{hosts[4]}:{workers[4][2]}"], + port=proxy_port, + disagg_config={"hostname": hosts[0]}, + wait_ready=False, + server_start_timeout=1800, + ) as proxy, + ): + session.submit(_serve_worker, workers, model, worker_config, output, url, sync=True) + try: + wait_for_endpoint_ready(f"{url}/health", timeout=1800, server_proc=proxy.proc) + _check_serving(url, model, output) + finally: + proxy.terminate() + deadline = time.monotonic() + 180 + while not (results := session.poll()): + assert time.monotonic() < deadline, "CI MPI workers did not stop" + time.sleep(1) + if isinstance(results, BaseException): + raise results + assert len(results) == 8, results + (output / "handoff.json").write_text(json.dumps(results) + "\n") + assert len(session.submit_sync(_worker_info)) == 8 diff --git a/tests/integration/defs/perf/_model_paths.py b/tests/integration/defs/perf/_model_paths.py index aff79ba87872..44d471b3714c 100644 --- a/tests/integration/defs/perf/_model_paths.py +++ b/tests/integration/defs/perf/_model_paths.py @@ -22,6 +22,7 @@ "llama_v3.1_8b_instruct_fp4": "modelopt-hf-model-hub/Llama-3.1-8B-Instruct-fp4", "gemma_4_26b_a4b_nvfp4": "gemma/nvidia-Gemma-4-26B-A4B-NVFP4", "gemma_4_31b_it_nvfp4": "gemma/nvidia-Gemma-4-31B-IT-NVFP4", + "mistral_large_4_nvfp4": "Mistral-Large-4-1T-A52B-NVFP4", "deepseek_r1_0528_fp8": "DeepSeek-R1/DeepSeek-R1-0528/", "deepseek_r1_0528_fp4": "DeepSeek-R1/DeepSeek-R1-0528-FP4/", "deepseek_r1_0528_fp4_v2": "DeepSeek-R1/DeepSeek-R1-0528-FP4-v2/", diff --git a/tests/integration/test_lists/qa/functional/llm_function_core.txt b/tests/integration/test_lists/qa/functional/llm_function_core.txt index 7fcf107df36e..c978bdbcfe96 100644 --- a/tests/integration/test_lists/qa/functional/llm_function_core.txt +++ b/tests/integration/test_lists/qa/functional/llm_function_core.txt @@ -483,6 +483,14 @@ accuracy/test_llm_api_pytorch.py::TestMinistral8BInstruct::test_fp8 accuracy/test_llm_api_pytorch.py::TestMistralLarge3_675B::test_fp8[latency_moe_deepgemm] accuracy/test_llm_api_pytorch.py::TestMistralLarge3_675B::test_nvfp4_4gpus[latency_moe_trtllm] accuracy/test_llm_api_pytorch.py::TestMistralLarge3_675B::test_nvfp4_4gpus[latency_moe_trtllm_eagle] +# ModelingV2 Mistral Large 4: four Rubin GPUs (SM107), the tep, dep and tp targets: the GSM8K +# gate, then the quick DSpark profile. +accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_107__tep/test_accuracy.py::TestModelingV2MistralLarge4Nvfp4Sm107Tep::test_gsm8k[tep4] TIMEOUT (180) +accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_107__dep/test_accuracy.py::TestModelingV2MistralLarge4Nvfp4Sm107Dep::test_gsm8k[dep4] TIMEOUT (180) +accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/test_accuracy.py::TestModelingV2MistralLarge4Nvfp4Sm107Tp::test_gsm8k[tp4] TIMEOUT (180) +accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_107__tep/test_accuracy.py::TestModelingV2MistralLarge4Nvfp4Sm107Tep::test_accuracy[tep4-quick] TIMEOUT (180) +accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_107__dep/test_accuracy.py::TestModelingV2MistralLarge4Nvfp4Sm107Dep::test_accuracy[dep4-quick] TIMEOUT (180) +accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/test_accuracy.py::TestModelingV2MistralLarge4Nvfp4Sm107Tp::test_accuracy[tp4-quick] TIMEOUT (180) accuracy/test_llm_api_pytorch.py::TestNemotron35Lightning::test_nvfp4_marlin_mtp3_chunked_prefill accuracy/test_llm_api_pytorch.py::TestNemotronV3Nano::test_auto_dtype accuracy/test_llm_api_pytorch.py::TestNemotronV3Nano::test_fp8 diff --git a/tests/integration/test_lists/test-db/l0_b300.yml b/tests/integration/test_lists/test-db/l0_b300.yml index d1cce8cd6858..a63774de4c19 100644 --- a/tests/integration/test_lists/test-db/l0_b300.yml +++ b/tests/integration/test_lists/test-db/l0_b300.yml @@ -197,6 +197,19 @@ l0_b300: - unittest/_torch/modeling_v2/models/minimax_m3/test_shared_kernel_contracts.py --import-mode=importlib TIMEOUT (30) - unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_103__tp4 --import-mode=importlib --ignore-glob=unittest/_torch/modeling_v2/models/minimax_m3/**/comm/* TIMEOUT (30) - unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_103__tp8 --import-mode=importlib --ignore-glob=unittest/_torch/modeling_v2/models/minimax_m3/**/comm/* TIMEOUT (30) + # mistral_large4: its GPU-run top-level file, op categories (ops/comm needs four GPUs and + # runs in l0_gb300_multi_gpus) and sm_103 targets. + - unittest/_torch/modeling_v2/models/mistral_large4/test_multimodal_vision.py TIMEOUT (30) + - unittest/_torch/modeling_v2/models/mistral_large4/ops/activation TIMEOUT (30) + - unittest/_torch/modeling_v2/models/mistral_large4/ops/attention TIMEOUT (30) + - unittest/_torch/modeling_v2/models/mistral_large4/ops/gemm TIMEOUT (30) + - unittest/_torch/modeling_v2/models/mistral_large4/ops/moe TIMEOUT (30) + - unittest/_torch/modeling_v2/models/mistral_large4/ops/norm TIMEOUT (30) + - unittest/_torch/modeling_v2/models/mistral_large4/ops/quantization TIMEOUT (30) + - unittest/_torch/modeling_v2/models/mistral_large4/ops/vision TIMEOUT (30) + - unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep TIMEOUT (30) + - unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep TIMEOUT (30) + - unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp TIMEOUT (30) # K3 single-GPU kernel and wrapper checks. - unittest/_torch/modeling_v2/models/kimi_k3_vl/kernels/k3_ctm_gemv/test_k3_ctm_gemv.py - unittest/_torch/modeling_v2/models/kimi_k3_vl/kimi_k3_mxfp4__sm_100__tp16_moetp4ep4/kernels/k3_ctx_kv/test_k3_ctx_kv.py diff --git a/tests/integration/test_lists/test-db/l0_cpu.yml b/tests/integration/test_lists/test-db/l0_cpu.yml index bc8c7edd934a..86f03648b171 100644 --- a/tests/integration/test_lists/test-db/l0_cpu.yml +++ b/tests/integration/test_lists/test-db/l0_cpu.yml @@ -65,6 +65,9 @@ l0_cpu: # catches a pruned reference before the post-merge canary does. - model_express/test_model_express_accuracy.py::test_mx_accuracy_cases_have_references - unittest/test_s3_output.py + - unittest/test_ruler_accuracy.py + - unittest/test_mmmu_accuracy.py + - unittest/test_multimodal_accuracy.py - unittest/_torch/visual_gen/multi_gpu/test_worker_pool.py # Covers the cpu_only cases of tests/unittest/_torch/attention/. On this CPU-only list, # pytest_ignore_collect in tests/unittest/conftest.py drops every file without a @@ -199,6 +202,11 @@ l0_cpu: - unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_n_valid_contract.py - unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp2/spec/test_modeling_v2_m3_wide_draft_logits.py - unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp2/test_modeling_v2_m3_draft_attention_bcg_route.py + # mistral_large4 targets' cpu_only tests (weight manifests, engine configs, tiny-config + # slicing and loading). Run with -m cpu_only, this entry collects only the marked files; the + # rest of unittest/_torch/modeling_v2 needs a GPU. Cases that read the checkpoint skip when it + # is not under LLM_MODELS_ROOT. + - unittest/_torch/modeling_v2/models/mistral_large4 - unittest/_torch/models/test_minimax_m3.py - unittest/_torch/models/test_qwen3_next_eager_fusion.py::test_sparse_moe_keeps_shared_expert_tokens_unpadded # MiniMax-M3 VL input processor and wrapper forward; the CPU stage's -m cpu_only @@ -209,6 +217,7 @@ l0_cpu: # what runs it. - unittest/_torch/misc/test_xgrammar_structural_tag.py - unittest/_torch/misc/test_grammar_matcher.py + - unittest/_torch/misc/test_tekken_guided_decoding.py - unittest/_torch/models/checkpoints - unittest/_torch/modules - unittest/_torch/moe @@ -352,6 +361,7 @@ l0_cpu: - unittest/llmapi/test_deepseek_v4_tokenizer.py - unittest/llmapi/test_deepseek_v41_tokenizer.py - unittest/llmapi/test_disagg_worker_role.py + - unittest/llmapi/test_tokenizer_factory_mistral.py - unittest/llmapi/test_session_reuse.py - unittest/llmapi/apps/test_anthropic_adapter.py - unittest/llmapi/apps/test_anthropic_batches.py diff --git a/tests/integration/test_lists/test-db/l0_gb300.yml b/tests/integration/test_lists/test-db/l0_gb300.yml index 41e57831f75b..439c38468011 100644 --- a/tests/integration/test_lists/test-db/l0_gb300.yml +++ b/tests/integration/test_lists/test-db/l0_gb300.yml @@ -41,6 +41,7 @@ l0_gb300: - unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_103__tp8/test_modeling_v2_m3_dispatch.py - unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_103__tp8/test_modeling_v2_m3_prebuild.py - unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_103__tp8/test_modeling_v2_m3_rank_agreement.py + - unittest/disaggregated/test_mistral_large4_kv_transfer.py # ------------- PyTorch tests --------------- # MLA context head slices, including FP8 scale-block boundaries. - unittest/_torch/modules/test_linear_output_slice.py diff --git a/tests/integration/test_lists/test-db/l0_gb300_multi_gpus.yml b/tests/integration/test_lists/test-db/l0_gb300_multi_gpus.yml index 0fce8e3a1dff..e15c87b998a2 100644 --- a/tests/integration/test_lists/test-db/l0_gb300_multi_gpus.yml +++ b/tests/integration/test_lists/test-db/l0_gb300_multi_gpus.yml @@ -25,6 +25,7 @@ l0_gb300_multi_gpus: - unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_103__tp8/ops/comm # One ordered sequence in one job: the cases share module-global rank state. - unittest/_torch/modeling_v2/comm + - unittest/_torch/modeling_v2/models/mistral_large4/ops/comm - accuracy/modeling_v2/deepseek_v3/r1_0528_nvfp4__sm_10x__default/test_accuracy.py::TestModelingV2DeepseekR10528Nvfp4Sm10xDefault::test_gsm8k_identity_vs_mtp3[dep4] - accuracy/modeling_v2/deepseek_v3/r1_0528_nvfp4__sm_10x__default/test_accuracy.py::TestModelingV2DeepseekR10528Nvfp4Sm10xDefault::test_mtp3_acceptance[modeling_v2-dep4] - accuracy/modeling_v2/deepseek_v3/r1_0528_nvfp4__sm_10x__default/test_accuracy.py::TestModelingV2DeepseekR10528Nvfp4Sm10xDefault::test_mtp3_acceptance[stock-dep4] @@ -61,6 +62,21 @@ l0_gb300_multi_gpus: # sampling, the breakable prefill graph, every opt-in projection) on one context and one # generation GPU. - accuracy/modeling_v2/qwen4_exp/qwen3_8_flash_next_nvfp4__sm_103__default/test_accuracy.py::TestModelingV2Qwen38FlashNextNvfp4Sm103Default::test_tp1_disaggregated_mtp2 TIMEOUT (90) + # modeling_v2 Mistral-Large-4-1T-A52B-NVFP4 sm_103 tep target: the GSM8K identity gate at tep4. + - accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_accuracy.py::TestModelingV2MistralLarge4Nvfp4Sm103Tep::test_gsm8k[tep4] TIMEOUT (180) + # modeling_v2 Mistral-Large-4-1T-A52B-NVFP4 sm_103 dep target: the GSM8K identity gate at dep4. + - accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_accuracy.py::TestModelingV2MistralLarge4Nvfp4Sm103Dep::test_gsm8k[dep4] TIMEOUT (180) + # modeling_v2 Mistral-Large-4-1T-A52B-NVFP4 sm_103 tp target: the GSM8K identity gate at tp4. + - accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_accuracy.py::TestModelingV2MistralLarge4Nvfp4Sm103Tp::test_gsm8k[tp4] TIMEOUT (180) + # Native DSpark: GSM8K precision/acceptance, RULER-lite and reuse/envelope. + - accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_accuracy.py::TestModelingV2MistralLarge4Nvfp4Sm103Tep::test_accuracy[tep4-quick] TIMEOUT (180) + - accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_accuracy.py::TestModelingV2MistralLarge4Nvfp4Sm103Dep::test_accuracy[dep4-quick] TIMEOUT (180) + - accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_accuracy.py::TestModelingV2MistralLarge4Nvfp4Sm103Tp::test_accuracy[tp4-quick] TIMEOUT (180) + # llguidance guided decoding over frontend-rendered chat ids: JSON, JSON after reasoning, tool grammars. + - accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_accuracy.py::TestModelingV2MistralLarge4Nvfp4Sm103Tp::test_llguidance TIMEOUT (60) + # Its expert path's attention-DP round trip over four ranks with unequal token counts; the + # single-GPU directory entry skips it. + - unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_moe_round_trip.py # Covers the GPU cases of tests/unittest/_torch/attention/ on GB300, kernels/ and # multi_gpu/ included; its cpu_only cases run on CPU (l0_cpu). The GPU cases of # kernels/parallel_hw_agnostic/ run here too, although l0_b200 and l0_b300 ignore that diff --git a/tests/integration/test_lists/test-db/l0_gb300_multi_gpus_perf_sanity.yml b/tests/integration/test_lists/test-db/l0_gb300_multi_gpus_perf_sanity.yml index 093965357be4..682f235eff1d 100644 --- a/tests/integration/test_lists/test-db/l0_gb300_multi_gpus_perf_sanity.yml +++ b/tests/integration/test_lists/test-db/l0_gb300_multi_gpus_perf_sanity.yml @@ -75,8 +75,19 @@ l0_gb300_multi_gpus_perf_sanity_key_models: # glm-5.3-flash-nvfp4 tp4 mtp3 8k1k (aggregated) - perf/test_perf_sanity.py::test_e2e[aggr_upload-gb300_glm53_flash_nvfp4_grace_blackwell-glm53_flash_nvfp4_tp4_mtp3_8k1k] TIMEOUT (30) - perf/test_perf_sanity.py::test_e2e[aggr_upload-gb300_glm53_flash_nvfp4_grace_blackwell-glm53_flash_nvfp4_tp4_mtp3_isl13312_osl33] TIMEOUT (35) + # mistral-large-4-nvfp4 8k1k (aggregated) + - perf/test_perf_sanity.py::test_e2e[aggr_upload-mistral_large_4_nvfp4_grace_blackwell-mistral_large_4_nvfp4_tp4_dspark3_con1_8k1k] TIMEOUT (90) + - perf/test_perf_sanity.py::test_e2e[aggr_upload-mistral_large_4_nvfp4_grace_blackwell-mistral_large_4_nvfp4_tp4_dspark3_con16_8k1k] TIMEOUT (90) + # mistral-large-4-nvfp4 8k1k (ctx_only) + - perf/test_perf_sanity.py::test_e2e[aggr_upload-ctx_only-gb300_mistral-large-4-nvfp4_8k1k_con64_ctx1_dep4_gen1_dep4_dspark3_ccb-NIXL] TIMEOUT (90) + - perf/test_perf_sanity.py::test_e2e[aggr_upload-ctx_only-gb300_mistral-large-4-nvfp4_8k1k_con256_ctx1_dep4_gen1_dep4_dspark3_ccb-NIXL] TIMEOUT (90) + - perf/test_perf_sanity.py::test_e2e[aggr_upload-ctx_only-gb300_mistral-large-4-nvfp4_8k1k_con1024_ctx1_dep4_gen1_dep4_dspark3_ccb-NIXL] TIMEOUT (90) # glm-5.3-flash-fp8 8k1k con16 - perf/test_perf_sanity.py::test_e2e[aggr_upload-gen_only_no_context-gb300_glm53-flash-fp8_8k1k_con16_ctx1_tep4_gen1_tep4_eplb0_mtp3_ccb-NIXL] TIMEOUT (30) # minimax-m3-fp4 eagle3 150k1k con1 and con64, TP4 gen worker - perf/test_perf_sanity.py::test_e2e[aggr_upload-gen_only_no_context-gb300_minimax-m3-fp4_150k1k_con1_ctx1_tp2_gen1_tp4_eplb0_eagle3_ccb-NIXL] TIMEOUT (35) - perf/test_perf_sanity.py::test_e2e[aggr_upload-gen_only_no_context-gb300_minimax-m3-fp4_150k1k_con64_ctx1_tp2_gen1_tp4_eplb0_eagle3_ccb-NIXL] TIMEOUT (45) + # mistral-large-4-nvfp4 8k1k (gen_only_no_context) + - perf/test_perf_sanity.py::test_e2e[aggr_upload-gen_only_no_context-gb300_mistral-large-4-nvfp4_8k1k_con64_ctx1_dep4_gen1_dep4_dspark3_ccb-NIXL] TIMEOUT (90) + - perf/test_perf_sanity.py::test_e2e[aggr_upload-gen_only_no_context-gb300_mistral-large-4-nvfp4_8k1k_con256_ctx1_dep4_gen1_dep4_dspark3_ccb-NIXL] TIMEOUT (90) + - perf/test_perf_sanity.py::test_e2e[aggr_upload-gen_only_no_context-gb300_mistral-large-4-nvfp4_8k1k_con1024_ctx1_dep4_gen1_dep4_dspark3_ccb-NIXL] TIMEOUT (90) diff --git a/tests/integration/test_lists/test-db/l0_gb300_multi_nodes_node2_gpu8.yml b/tests/integration/test_lists/test-db/l0_gb300_multi_nodes_node2_gpu8.yml index 5dfeffd5b1dc..ba139bf89d3b 100644 --- a/tests/integration/test_lists/test-db/l0_gb300_multi_nodes_node2_gpu8.yml +++ b/tests/integration/test_lists/test-db/l0_gb300_multi_nodes_node2_gpu8.yml @@ -1,3 +1,5 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 version: 0.0.1 l0_gb300_multi_nodes_node2_gpu8: - condition: @@ -13,6 +15,7 @@ l0_gb300_multi_nodes_node2_gpu8: stage: post_merge backend: pytorch tests: + - accuracy/test_mistral_large4_disaggregated.py::test_mistral_large4_disagg TIMEOUT (130) - accuracy/test_llm_api_pytorch.py::TestQwen3_8_2_4T_A95B::test_nvfp4_tp8_mtp3_trtllm TIMEOUT (60) # Kimi K3 GSM8K accuracy plus an acceptance-length gate on the TEP8 DSpark # recipe. Measured 25m45s on 2x4 GB300. @@ -31,3 +34,10 @@ l0_gb300_multi_nodes_node2_gpu8: # GLM-5.2 NVFP4 modeling_v2 sm_103 tp8 target, GSM8K under modeling_v2="require". # Measured 34 min on 2x4 GB300 (batch 1). - accuracy/modeling_v2/glm_moe_dsa/glm_5_2_nvfp4__sm_103__tp8/test_accuracy.py::TestModelingV2Glm52Nvfp4Sm103Tp8::test_gsm8k_tp8 TIMEOUT (90) + - accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_accuracy.py::TestModelingV2MistralLarge4Nvfp4Sm103Tep::test_gsm8k[tep8] TIMEOUT (60) + - accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_accuracy.py::TestModelingV2MistralLarge4Nvfp4Sm103Tp::test_gsm8k[tp8] TIMEOUT (60) + - accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_accuracy.py::TestModelingV2MistralLarge4Nvfp4Sm103Dep::test_gsm8k[dep8] TIMEOUT (60) + # Native DSpark quick coverage; full adds GPQA/RULER and spec-off GSM8K. + - accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_accuracy.py::TestModelingV2MistralLarge4Nvfp4Sm103Tep::test_accuracy[tep8-quick] TIMEOUT (180) + - accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_accuracy.py::TestModelingV2MistralLarge4Nvfp4Sm103Dep::test_accuracy[dep8-quick] TIMEOUT (180) + - accuracy/modeling_v2/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_accuracy.py::TestModelingV2MistralLarge4Nvfp4Sm103Tp::test_accuracy[tp8-quick] TIMEOUT (180) diff --git a/tests/integration/test_lists/test-db/l0_gb300_multi_nodes_perf_sanity.yml b/tests/integration/test_lists/test-db/l0_gb300_multi_nodes_perf_sanity.yml index 076538a57e79..8d3c41b1a77a 100644 --- a/tests/integration/test_lists/test-db/l0_gb300_multi_nodes_perf_sanity.yml +++ b/tests/integration/test_lists/test-db/l0_gb300_multi_nodes_perf_sanity.yml @@ -125,6 +125,10 @@ l0_gb300_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen1_node1_gpu4_key_models: tests: # glm-5.3-flash-fp8 8k1k con16 - perf/test_perf_sanity.py::test_e2e[disagg_upload-e2e-gb300_glm53-flash-fp8_8k1k_con16_ctx1_tep4_gen1_tep4_eplb0_mtp3_ccb-NIXL] TIMEOUT (30) + # mistral-large-4-nvfp4 8k1k (e2e) + - perf/test_perf_sanity.py::test_e2e[disagg_upload-e2e-gb300_mistral-large-4-nvfp4_8k1k_con64_ctx1_dep4_gen1_dep4_dspark3_ccb-NIXL] TIMEOUT (90) + - perf/test_perf_sanity.py::test_e2e[disagg_upload-e2e-gb300_mistral-large-4-nvfp4_8k1k_con256_ctx1_dep4_gen1_dep4_dspark3_ccb-NIXL] TIMEOUT (90) + - perf/test_perf_sanity.py::test_e2e[disagg_upload-e2e-gb300_mistral-large-4-nvfp4_8k1k_con1024_ctx1_dep4_gen1_dep4_dspark3_ccb-NIXL] TIMEOUT (90) l0_gb300_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen4_node2_gpu8: - condition: ranges: diff --git a/tests/integration/test_lists/test-db/l0_gr100.yml b/tests/integration/test_lists/test-db/l0_gr100.yml index 3fd53ced3c5e..13559b935f93 100644 --- a/tests/integration/test_lists/test-db/l0_gr100.yml +++ b/tests/integration/test_lists/test-db/l0_gr100.yml @@ -113,6 +113,16 @@ l0_gr100: - unittest/_torch/modeling_v2/models/glm_moe_dsa/glm_5_2_nvfp4__sm_107__low_latency/ops/moe - unittest/_torch/modeling_v2/models/glm_moe_dsa/glm_5_2_nvfp4__sm_107__low_latency/ops/quantization - unittest/_torch/modeling_v2/models/glm_moe_dsa/glm_5_2_nvfp4__sm_107__low_latency/spec + # The Mistral Large 4 sm_107 targets' own entries: the Rubin MXFP8 linear, the Triton + # indexer scorer, the split radix top-k of DSpark verification and sparse MLA decode's + # low-row global split. The B300 list reaches the family's op tests too, where the sm_107 + # cases skip. The sm_107 tp target's own ops (narrow BF16 projections, fused FP8 GEMV) + # follow. + - unittest/_torch/modeling_v2/models/mistral_large4/ops/gemm/test_modeling_v2_cute_dsl_fp8_block_scale_linear.py + - unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_triton_fp8_mqa_logits.py + - unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_split_radix_topk_decode.py + - unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_cute_dsl_ds_mla_sparse_decode.py + - unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp - unittest/_torch/visual_gen/kernels/parallel/test_fmha_dense_fp8.py - unittest/_torch/visual_gen/kernels/parallel/test_fused_dit_qk_norm_rope_fp8.py - unittest/_torch/modeling_v2/models/glm_moe_dsa/glm_5_2_nvfp4__sm_107__high_throughput diff --git a/tests/scripts/perf-sanity/aggregated/mistral_large_4_nvfp4_grace_blackwell.yaml b/tests/scripts/perf-sanity/aggregated/mistral_large_4_nvfp4_grace_blackwell.yaml new file mode 100644 index 000000000000..4208bf848304 --- /dev/null +++ b/tests/scripts/perf-sanity/aggregated/mistral_large_4_nvfp4_grace_blackwell.yaml @@ -0,0 +1,86 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# DSpark uses 1.772 accepted draft tokens on average (total acceptance length 2.772). +metadata: + model_name: mistral_large_4_nvfp4 + precision: fp4 + model_dir_name: Mistral-Large-4-1T-A52B-NVFP4 + supported_gpus: + - GB300 + benchmark_type: 8k1k +hardware: + gpus_per_node: 4 +server_configs: +- name: mistral_large_4_nvfp4_tp4_dspark3_con1_8k1k + server_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS=1.772 + gpus: 4 + checkpoint_format: mistral + modeling_v2: require + tensor_parallel_size: 4 + moe_tensor_parallel_size: 4 + moe_expert_parallel_size: 1 + pipeline_parallel_size: 1 + enable_attention_dp: false + max_batch_size: 1 + max_num_tokens: 16384 + max_seq_len: 32768 + enable_chunked_prefill: true + print_iter_log: true + stream_interval: 1 + cuda_graph_config: + enable_padding: true + max_batch_size: 1 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.85 + dtype: fp8_ds_mla + speculative_config: + decoding_type: DSpark + max_draft_len: 3 + speculative_model: Mistral-Large-4-1T-A52B-NVFP4 + client_configs: + - name: con1_iter10_8k1k + concurrency: 1 + iterations: 10 + isl: 8192 + osl: 1024 + random_range_ratio: 0.0 + backend: openai + streaming: true +- name: mistral_large_4_nvfp4_tp4_dspark3_con16_8k1k + server_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS=1.772 + gpus: 4 + checkpoint_format: mistral + modeling_v2: require + tensor_parallel_size: 4 + moe_tensor_parallel_size: 4 + moe_expert_parallel_size: 1 + pipeline_parallel_size: 1 + enable_attention_dp: false + max_batch_size: 16 + max_num_tokens: 16384 + max_seq_len: 32768 + enable_chunked_prefill: true + print_iter_log: true + stream_interval: 1 + cuda_graph_config: + enable_padding: true + max_batch_size: 16 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.85 + dtype: fp8_ds_mla + speculative_config: + decoding_type: DSpark + max_draft_len: 3 + speculative_model: Mistral-Large-4-1T-A52B-NVFP4 + client_configs: + - name: con16_iter10_8k1k + concurrency: 16 + iterations: 10 + isl: 8192 + osl: 1024 + random_range_ratio: 0.0 + backend: openai + streaming: true diff --git a/tests/scripts/perf-sanity/disaggregated/gb300_mistral-large-4-nvfp4_8k1k_con1024_ctx1_dep4_gen1_dep4_dspark3_ccb-NIXL.yaml b/tests/scripts/perf-sanity/disaggregated/gb300_mistral-large-4-nvfp4_8k1k_con1024_ctx1_dep4_gen1_dep4_dspark3_ccb-NIXL.yaml new file mode 100644 index 000000000000..3beda6469d7e --- /dev/null +++ b/tests/scripts/perf-sanity/disaggregated/gb300_mistral-large-4-nvfp4_8k1k_con1024_ctx1_dep4_gen1_dep4_dspark3_ccb-NIXL.yaml @@ -0,0 +1,106 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# DEP4 batch and token limits apply per attention-DP rank. +# DSpark uses 1.772 accepted draft tokens on average (total acceptance length 2.772). +metadata: + model_name: mistral_large_4_nvfp4 + precision: fp4 + model_dir_name: Mistral-Large-4-1T-A52B-NVFP4 + supported_gpus: + - GB300 + script_file: disaggr_torch.slurm + benchmark_type: 8k1k +slurm: + script_file: disaggr_torch.slurm + partition: + account: + job_time: 02:00:00 + job_name: unified-benchmark + extra_args: --gres=gpu:4 + numa_bind: true +benchmark: + mode: e2e + use_nv_sa_benchmark: false + multi_round: 10 + benchmark_ratio: 0.0 + streaming: true + concurrency_list: '1024' + input_length: 8192 + output_length: 1024 + dataset_file: '' +hardware: + gpus_per_node: 4 + num_ctx_servers: 1 + num_gen_servers: 1 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 ENROOT_ALLOW_DEV=yes TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS=1.772 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +worker_config: + gen: + checkpoint_format: mistral + modeling_v2: require + tensor_parallel_size: 4 + moe_tensor_parallel_size: 1 + moe_expert_parallel_size: 4 + pipeline_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: false + attention_dp_config: + enable_balance: true + batching_wait_iters: 0 + timeout_iters: 60 + max_batch_size: 256 + max_num_tokens: 1024 + max_seq_len: 32768 + print_iter_log: true + stream_interval: 1 + cuda_graph_config: + enable_padding: true + max_batch_size: 256 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.85 + dtype: fp8_ds_mla + cache_transceiver_config: + backend: NIXL + transceiver_runtime: PYTHON + speculative_config: &dspark + decoding_type: DSpark + max_draft_len: 3 + speculative_model: Mistral-Large-4-1T-A52B-NVFP4 + ctx: + checkpoint_format: mistral + modeling_v2: require + tensor_parallel_size: 4 + moe_tensor_parallel_size: 1 + moe_expert_parallel_size: 4 + pipeline_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: false + attention_dp_config: + enable_balance: true + batching_wait_iters: 0 + timeout_iters: 60 + max_batch_size: 16 + max_num_tokens: 16384 + max_seq_len: 32768 + enable_chunked_prefill: true + print_iter_log: true + cuda_graph_config: null + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.85 + dtype: fp8_ds_mla + cache_transceiver_config: + backend: NIXL + transceiver_runtime: PYTHON + speculative_config: *dspark diff --git a/tests/scripts/perf-sanity/disaggregated/gb300_mistral-large-4-nvfp4_8k1k_con256_ctx1_dep4_gen1_dep4_dspark3_ccb-NIXL.yaml b/tests/scripts/perf-sanity/disaggregated/gb300_mistral-large-4-nvfp4_8k1k_con256_ctx1_dep4_gen1_dep4_dspark3_ccb-NIXL.yaml new file mode 100644 index 000000000000..71593cfa3197 --- /dev/null +++ b/tests/scripts/perf-sanity/disaggregated/gb300_mistral-large-4-nvfp4_8k1k_con256_ctx1_dep4_gen1_dep4_dspark3_ccb-NIXL.yaml @@ -0,0 +1,106 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# DEP4 batch and token limits apply per attention-DP rank. +# DSpark uses 1.772 accepted draft tokens on average (total acceptance length 2.772). +metadata: + model_name: mistral_large_4_nvfp4 + precision: fp4 + model_dir_name: Mistral-Large-4-1T-A52B-NVFP4 + supported_gpus: + - GB300 + script_file: disaggr_torch.slurm + benchmark_type: 8k1k +slurm: + script_file: disaggr_torch.slurm + partition: + account: + job_time: 02:00:00 + job_name: unified-benchmark + extra_args: --gres=gpu:4 + numa_bind: true +benchmark: + mode: e2e + use_nv_sa_benchmark: false + multi_round: 10 + benchmark_ratio: 0.0 + streaming: true + concurrency_list: '256' + input_length: 8192 + output_length: 1024 + dataset_file: '' +hardware: + gpus_per_node: 4 + num_ctx_servers: 1 + num_gen_servers: 1 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 ENROOT_ALLOW_DEV=yes TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS=1.772 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +worker_config: + gen: + checkpoint_format: mistral + modeling_v2: require + tensor_parallel_size: 4 + moe_tensor_parallel_size: 1 + moe_expert_parallel_size: 4 + pipeline_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: false + attention_dp_config: + enable_balance: true + batching_wait_iters: 0 + timeout_iters: 60 + max_batch_size: 64 + max_num_tokens: 256 + max_seq_len: 32768 + print_iter_log: true + stream_interval: 1 + cuda_graph_config: + enable_padding: true + max_batch_size: 64 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.85 + dtype: fp8_ds_mla + cache_transceiver_config: + backend: NIXL + transceiver_runtime: PYTHON + speculative_config: &dspark + decoding_type: DSpark + max_draft_len: 3 + speculative_model: Mistral-Large-4-1T-A52B-NVFP4 + ctx: + checkpoint_format: mistral + modeling_v2: require + tensor_parallel_size: 4 + moe_tensor_parallel_size: 1 + moe_expert_parallel_size: 4 + pipeline_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: false + attention_dp_config: + enable_balance: true + batching_wait_iters: 0 + timeout_iters: 60 + max_batch_size: 16 + max_num_tokens: 16384 + max_seq_len: 32768 + enable_chunked_prefill: true + print_iter_log: true + cuda_graph_config: null + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.85 + dtype: fp8_ds_mla + cache_transceiver_config: + backend: NIXL + transceiver_runtime: PYTHON + speculative_config: *dspark diff --git a/tests/scripts/perf-sanity/disaggregated/gb300_mistral-large-4-nvfp4_8k1k_con64_ctx1_dep4_gen1_dep4_dspark3_ccb-NIXL.yaml b/tests/scripts/perf-sanity/disaggregated/gb300_mistral-large-4-nvfp4_8k1k_con64_ctx1_dep4_gen1_dep4_dspark3_ccb-NIXL.yaml new file mode 100644 index 000000000000..ae10e8264ba4 --- /dev/null +++ b/tests/scripts/perf-sanity/disaggregated/gb300_mistral-large-4-nvfp4_8k1k_con64_ctx1_dep4_gen1_dep4_dspark3_ccb-NIXL.yaml @@ -0,0 +1,106 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# DEP4 batch and token limits apply per attention-DP rank. +# DSpark uses 1.772 accepted draft tokens on average (total acceptance length 2.772). +metadata: + model_name: mistral_large_4_nvfp4 + precision: fp4 + model_dir_name: Mistral-Large-4-1T-A52B-NVFP4 + supported_gpus: + - GB300 + script_file: disaggr_torch.slurm + benchmark_type: 8k1k +slurm: + script_file: disaggr_torch.slurm + partition: + account: + job_time: 02:00:00 + job_name: unified-benchmark + extra_args: --gres=gpu:4 + numa_bind: true +benchmark: + mode: e2e + use_nv_sa_benchmark: false + multi_round: 10 + benchmark_ratio: 0.0 + streaming: true + concurrency_list: '64' + input_length: 8192 + output_length: 1024 + dataset_file: '' +hardware: + gpus_per_node: 4 + num_ctx_servers: 1 + num_gen_servers: 1 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 ENROOT_ALLOW_DEV=yes TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS=1.772 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +worker_config: + gen: + checkpoint_format: mistral + modeling_v2: require + tensor_parallel_size: 4 + moe_tensor_parallel_size: 1 + moe_expert_parallel_size: 4 + pipeline_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: false + attention_dp_config: + enable_balance: true + batching_wait_iters: 0 + timeout_iters: 60 + max_batch_size: 16 + max_num_tokens: 64 + max_seq_len: 32768 + print_iter_log: true + stream_interval: 1 + cuda_graph_config: + enable_padding: true + max_batch_size: 16 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.85 + dtype: fp8_ds_mla + cache_transceiver_config: + backend: NIXL + transceiver_runtime: PYTHON + speculative_config: &dspark + decoding_type: DSpark + max_draft_len: 3 + speculative_model: Mistral-Large-4-1T-A52B-NVFP4 + ctx: + checkpoint_format: mistral + modeling_v2: require + tensor_parallel_size: 4 + moe_tensor_parallel_size: 1 + moe_expert_parallel_size: 4 + pipeline_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: false + attention_dp_config: + enable_balance: true + batching_wait_iters: 0 + timeout_iters: 60 + max_batch_size: 16 + max_num_tokens: 16384 + max_seq_len: 32768 + enable_chunked_prefill: true + print_iter_log: true + cuda_graph_config: null + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.85 + dtype: fp8_ds_mla + cache_transceiver_config: + backend: NIXL + transceiver_runtime: PYTHON + speculative_config: *dspark diff --git a/tests/test_common/mistral_tekken.py b/tests/test_common/mistral_tekken.py new file mode 100644 index 000000000000..2f1594a5f514 --- /dev/null +++ b/tests/test_common/mistral_tekken.py @@ -0,0 +1,78 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Mistral Tekken tokenizers for tests. + +``bundled`` is the newest Tekken file mistral-common ships (v3, no reasoning tokens); ``v13`` +and ``v15`` are the files of Mistral-Large-3 and Mistral-Small-4 under LLM_MODELS_ROOT, with +``[THINK]``; v15 also declares model settings (reasoning effort, response format). +""" + +import json +from pathlib import Path + +import pytest + +from test_common.llm_data import llm_models_root + +TEKKEN_SOURCES = ("bundled", "v13", "v15") +TEKKEN_CHECKPOINT_FILES = { + "v13": "Mistral-Large-3-675B/Mistral-Large-3-675B-Instruct-2512/tekken.json", + "v15": "Mistral-Small-4-119B-2603/Mistral-Small-4-119B-2603/tekken.json", +} + + +HF_TOKENIZER_TEXT = "the capital of france is" +"""Text the tokenizer of ``write_hf_tokenizer`` encodes as ``HF_TOKENIZER_IDS``.""" +HF_TOKENIZER_IDS = [1, 2, 3, 4, 5] + + +def bundled_tekken_file() -> Path: + """The newest Tekken file mistral-common ships; skips when it ships none.""" + mistral_common = pytest.importorskip("mistral_common") + files = sorted((Path(mistral_common.__file__).parent / "data").glob("tekken*.json")) + if not files: + pytest.skip("this mistral_common ships no Tekken file") + return files[-1] + + +def write_hf_tokenizer(directory: Path) -> None: + """Write a word-level Hugging Face tokenizer (``tokenizer.json``, ``tokenizer_config.json``). + + Its vocabulary differs from every Tekken one, so a test can tell which of the two files of + a directory a loader read. + """ + tokenizers = pytest.importorskip("tokenizers") + vocab = {"[UNK]": 0, **{word: i for i, word in enumerate(HF_TOKENIZER_TEXT.split(), 1)}} + word_level = tokenizers.Tokenizer(tokenizers.models.WordLevel(vocab=vocab, unk_token="[UNK]")) + word_level.pre_tokenizer = tokenizers.pre_tokenizers.Whitespace() + word_level.save(str(directory / "tokenizer.json")) + (directory / "tokenizer_config.json").write_text( + json.dumps({"tokenizer_class": "PreTrainedTokenizerFast", "unk_token": "[UNK]"}) + ) + + +def load_tekken(source: str): + """A ``MistralTokenizer`` over the ``source`` Tekken file; skips when it is not there.""" + pytest.importorskip("mistral_common") + from tensorrt_llm._torch.models.checkpoints.mistral.tokenizer import MistralTokenizer + + if source == "bundled": + path = bundled_tekken_file() + else: + root = llm_models_root() + path = None if root is None else root / TEKKEN_CHECKPOINT_FILES[source] + if path is None or not path.is_file(): + pytest.skip(f"no {source} Tekken file") + return MistralTokenizer.from_pretrained(str(path)) diff --git a/tests/unittest/_torch/attention/sparse/mistral_large4/__init__.py b/tests/unittest/_torch/attention/sparse/mistral_large4/__init__.py new file mode 100644 index 000000000000..52a7a9daf028 --- /dev/null +++ b/tests/unittest/_torch/attention/sparse/mistral_large4/__init__.py @@ -0,0 +1,2 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 diff --git a/tests/unittest/_torch/attention/sparse/mistral_large4/test_mistral_large4_cache_layout.py b/tests/unittest/_torch/attention/sparse/mistral_large4/test_mistral_large4_cache_layout.py new file mode 100644 index 000000000000..31687867e847 --- /dev/null +++ b/tests/unittest/_torch/attention/sparse/mistral_large4/test_mistral_large4_cache_layout.py @@ -0,0 +1,275 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Small real-manager checks for logical layers and physical KV ownership.""" + +from dataclasses import replace +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch.attention.backends.sparse.dsa import DSACacheManagerV2 +from tensorrt_llm._torch.attention.backends.sparse.mistral_large4 import MistralLarge4CacheManager +from tensorrt_llm._torch.pyexecutor.kv_cache import kv_cache_manager_v2 as wrapper +from tensorrt_llm._torch.pyexecutor.kv_cache.kv_cache_manager_v2 import Role +from tensorrt_llm.bindings import DataType +from tensorrt_llm.bindings.internal.batch_manager import CacheType +from tensorrt_llm.llmapi.llm_args import ( + DeepSeekSparseAttentionConfig, + DSparkDecodingConfig, + KvCacheConfig, +) +from tensorrt_llm.mapping import Mapping +from tensorrt_llm.runtime.kv_cache_manager_v2 import PageIndexMode + +from .test_mistral_large4_cache_manager import _SHARING, _SOURCES, _model_config + + +def make_manager( + kind: str, + *, + enable_stats: bool = False, + spec_config: DSparkDecodingConfig | None = None, + layer_mask: list[bool] | None = None, +) -> DSACacheManagerV2: + """Use a few slots on device and host; no model weights are needed.""" + if kind == "ml4": + cls = MistralLarge4CacheManager + config = _model_config().pretrained_config + num_layers = 72 + else: + cls = DSACacheManagerV2 + num_layers = 4 + config = SimpleNamespace(num_hidden_layers=num_layers) + if kind == "glm": + config.index_topk_pattern = ["F", "S", "F", "S"] + return cls( + kv_cache_config=KvCacheConfig( + max_tokens=256, + host_cache_size=32 << 20, + enable_block_reuse=True, + ), + kv_cache_type=CacheType.SELFKONLY, + num_layers=num_layers, + num_kv_heads=1, + head_dim=576, + tokens_per_block=32, + max_seq_len=128, + max_num_tokens=64, + max_batch_size=2, + mapping=Mapping(world_size=1, rank=0, tp_size=1, pp_size=1), + dtype=DataType.FP8, + sparse_attention_config=DeepSeekSparseAttentionConfig( + index_n_heads=32, + index_head_dim=128, + index_topk=2048, + ), + pretrained_config=config, + enable_stats=enable_stats, + spec_config=spec_config, + layer_mask=layer_mask, + ) + + +@pytest.mark.parametrize("draft_len", [1, 3, 7]) +@pytest.mark.parametrize("layer_mask", [None, [True] * 72]) +def test_ml4_speculation_reserves_tokens_without_adding_target_layers( + draft_len: int, layer_mask: list[bool] | None +) -> None: + """Verification reserves draft capacity while preserving the shared latent layout.""" + manager = make_manager( + "ml4", spec_config=DSparkDecodingConfig(max_draft_len=draft_len), layer_mask=layer_mask + ) + try: + assert manager._kv_reserve_draft_tokens == draft_len + assert manager._generation_kv_capacity_headroom == 2 * draft_len + assert manager.pp_layers == list(range(72)) + assert manager.get_cache_bytes_per_token() == 63 * 656 + 44 * 132 + assert manager.get_latent_pool().shape[1:] == (63, 32, 656) + for layer, source in enumerate(_SOURCES): + assert manager.latent_pool_row(layer) == manager.latent_pool_row(source) + assert manager.has_buffer(layer, Role.KEY) == (layer == source) + finally: + manager.shutdown() + + +@pytest.mark.parametrize("layer_mask", [[True] * 71, [True] * 73, [False] + [True] * 71]) +def test_ml4_rejects_masks_that_change_logical_target_layers(layer_mask: list[bool]) -> None: + with pytest.raises(NotImplementedError, match="derives its layer mask"): + make_manager("ml4", layer_mask=layer_mask) + + +@pytest.mark.parametrize("kind", ["deepseek", "glm", "ml4"]) +def test_physical_roles_and_layer_views(kind: str) -> None: + manager = make_manager(kind) + try: + ml4 = kind == "ml4" + sources = _SOURCES if ml4 else list(range(4)) + owners = [i for i, source in enumerate(sources) if i == source] + index_layers = ( + [i for i, role in enumerate(_SHARING) if role == "full"] + if ml4 + else ([0, 2] if kind == "glm" else list(range(4))) + ) + token_bytes = 656 if ml4 else 576 + pool = manager.get_unique_primary_pool() + assert pool.shape == (manager.blocks_in_primary_pool, len(owners), 1, 32 * token_bytes) + assert manager.pp_layers == list(range(len(sources))) + assert ( + manager.get_cache_bytes_per_token() + == len(owners) * token_bytes + len(index_layers) * 132 + ) + assert manager.indexer_k_cache_page_scale == len(index_layers) + if ml4: + assert manager.kv_owner_layers == owners + assert manager.kv_source_layers == sources + assert manager.get_latent_pool().stride(0) // 656 == 2016 + inventory = {(int(buf.layer_id), buf.role) for buf in manager.impl.all_buffer_ids} + for layer, source in enumerate(sources): + assert manager.has_buffer(layer, Role.KEY) == ((layer, Role.KEY) in inventory) + assert manager.has_buffer(layer, Role.INDEX_KEY) == ( + (layer, Role.INDEX_KEY) in inventory + ) + assert manager.has_buffer(layer, Role.KEY) == (layer in owners) + assert manager.has_buffer(layer, Role.INDEX_KEY) == (layer in index_layers) + if layer in owners: + ordinal = owners.index(layer) + assert manager.get_primary_pool_page_index_params(layer) == (len(owners), ordinal) + view = manager.get_buffers(layer) + assert view.data_ptr() - pool.data_ptr() == ordinal * 32 * token_bytes + view.view(torch.uint8)[0, 0, 0, 0, 0] = layer + else: + assert manager.get_buffers(layer) is None + assert manager.get_primary_pool_page_index_params(layer) is None + if ml4: + assert manager.latent_pool_row(layer) == owners.index(source) + + index_base = manager.impl.get_mem_pool_base_address( + index_layers[0], Role.INDEX_KEY, PageIndexMode.SHARED + ) + for ordinal, layer in enumerate(index_layers): + view = manager.get_indexer_k_cache_buffers(layer) + assert view.shape[1:] == (32, 1, 132) + assert view.stride() == (32 * 132, 132, 132, 1) + assert view.data_ptr() - index_base == ordinal * 32 * 132 + converter = manager.impl.get_page_index_converter(layer, Role.INDEX_KEY) + assert (int(converter.scale), int(converter.layer_offset)) == ( + len(index_layers), + ordinal, + ) + view[0, 0, 0, 0] = layer + for ordinal, layer in enumerate(owners): + assert pool.view(torch.uint8)[0, ordinal, 0, 0].item() == layer + for layer in index_layers: + assert manager.get_indexer_k_cache_buffers(layer)[0, 0, 0, 0].item() == layer + if ml4: + assert manager.get_latent_pool()[0, manager.latent_pool_row(39), 0, 0].item() == 35 + assert manager.get_indexer_k_cache_buffers(39)[0, 0, 0, 0].item() == 39 + assert manager.get_indexer_k_cache_buffers(35)[0, 0, 0, 0].item() == 35 + + # The existing batch API reads stable slots in request order. + slots = [manager.index_mapper.add_new_sequence(i) for i in (101, 202)] + try: + for slot, blocks in zip(slots, ([1, 2, 0, 0], [2, 3, 0, 0])): + manager.host_kv_cache_block_offsets[0, slot, 0, :4] = torch.tensor(blocks) + torch.testing.assert_close( + manager.get_pool_block_indices(2, request_ids=[202, 101], num_contexts=2)[:, :4], + torch.tensor([[2, 3, 0, 0], [1, 2, 0, 0]], dtype=torch.int32), + ) + finally: + for request_id in (101, 202): + manager.index_mapper.remove_sequence(request_id) + finally: + manager.shutdown() + + +def test_ml4_host_slots_preserve_latent_and_index_pages() -> None: + """Force real host transfers and check every byte of both physical pools.""" + manager = make_manager("ml4", enable_stats=True) + cache = manager.impl.create_kv_cache() + pressure = None + try: + stream = torch.cuda.current_stream().cuda_stream + assert cache.resume(stream) + cache.capacity = 192 + blocks = list(cache.get_base_page_indices(0)) + assert len(blocks) == 6 + latent = manager.get_latent_pool() + indexes = [i for i, role in enumerate(_SHARING) if role == "full"] + expected = [] + for ordinal, block in enumerate(blocks): + latent[block].copy_( + (torch.arange(latent[block].numel(), device="cuda") + ordinal) + .to(torch.uint8) + .view_as(latent[block]) + ) + expected.append(latent[block].clone()) + for layer in indexes: + manager.get_indexer_k_cache_buffers(layer)[block * 44].fill_(layer + ordinal) + torch.cuda.synchronize() + cache.commit(list(range(192))) + cache.suspend() + manager.impl.get_and_reset_iteration_stats() + # Another live request forces suspended pages into the host tier. + pressure = manager.impl.create_kv_cache() + assert pressure.resume(stream) + pressure.capacity = 192 + offload_stats = manager.impl.get_and_reset_iteration_stats() + offload = dict( + iterOffloadBlocks=sum(s.iter_offload_blocks for s in offload_stats.values()), + iterOffloadBytes=sum(s.iter_offload_bytes for s in offload_stats.values()), + ) + assert offload["iterOffloadBlocks"] > 0 + assert offload["iterOffloadBytes"] == offload["iterOffloadBlocks"] * 32 * 47136 + pressure.close() + pressure = None + assert cache.resume(stream) + torch.cuda.synchronize() + onboard_stats = manager.impl.get_and_reset_iteration_stats() + onboard = dict( + iterOnboardBlocks=sum(s.iter_onboard_blocks for s in onboard_stats.values()), + iterOnboardBytes=sum(s.iter_onboard_bytes for s in onboard_stats.values()), + ) + assert onboard["iterOnboardBlocks"] > 0 + assert onboard["iterOnboardBytes"] == onboard["iterOnboardBlocks"] * 32 * 47136 + for ordinal, block in enumerate(cache.get_base_page_indices(0)): + torch.testing.assert_close(latent[block], expected[ordinal]) + for layer in indexes: + page = manager.get_indexer_k_cache_buffers(layer)[block * 44] + assert torch.all(page == layer + ordinal).item() + print("ML4_HOST_SLOT_ROUNDTRIP", dict(offload=offload, onboard=onboard)) + finally: + if pressure is not None: + pressure.close() + cache.close() + manager.shutdown() + + +def test_ml4_pool_rows_follow_physical_order(monkeypatch) -> None: + """Swap real native allocations without changing the model's logical layers.""" + native_manager = wrapper.KVCacheManagerPy + + def swapped_manager(config, **kwargs): + layers = list(config.layers) + layers[0], layers[1] = layers[1], layers[0] + return native_manager(replace(config, layers=layers), **kwargs) + + monkeypatch.setattr(wrapper, "KVCacheManagerPy", swapped_manager) + manager = make_manager("ml4") + try: + pool = manager.get_latent_pool() + assert manager.kv_owner_layers[:2] == [0, 1] + assert manager.latent_pool_row(0) == 1 + assert manager.latent_pool_row(1) == 0 + assert pool.is_contiguous() + assert pool.stride(0) // 656 == 2016 + for layer in manager.kv_owner_layers: + row = manager.latent_pool_row(layer) + view = manager.get_buffers(layer) + assert view.data_ptr() - pool.data_ptr() == row * 32 * 656 + view.view(torch.uint8)[0, 0, 0, 0, 0] = layer + for layer, source in enumerate(manager.kv_source_layers): + assert pool[0, manager.latent_pool_row(layer), 0, 0].item() == source + assert manager.latent_pool_row(39) == manager.latent_pool_row(35) + finally: + manager.shutdown() diff --git a/tests/unittest/_torch/attention/sparse/mistral_large4/test_mistral_large4_cache_manager.py b/tests/unittest/_torch/attention/sparse/mistral_large4/test_mistral_large4_cache_manager.py new file mode 100644 index 000000000000..ab8b460e961e --- /dev/null +++ b/tests/unittest/_torch/attention/sparse/mistral_large4/test_mistral_large4_cache_manager.py @@ -0,0 +1,67 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The Mistral-Large-4 latent cache's sizing and selection, from the checkpoint's own layer +structure. No device: the per-token bytes the engine budgets with and the registry branch.""" + +from types import SimpleNamespace + +import pytest + +from tensorrt_llm._torch.attention.backends.sparse import get_sparse_attn_kv_cache_manager +from tensorrt_llm._torch.attention.backends.sparse.dsa import DSACacheManagerV2 +from tensorrt_llm._torch.attention.backends.sparse.mistral_large4 import MistralLarge4CacheManager +from tensorrt_llm._torch.attention.backends.sparse.mistral_large4.cache_manager import TOKEN_BYTES +from tensorrt_llm._torch.configs.mistral_large4 import MistralLarge4Config +from tensorrt_llm.llmapi.llm_args import DeepSeekSparseAttentionConfig + +pytestmark = pytest.mark.cpu_only + +# The Mistral-Large-4-1T-A52B checkpoints' layer structure (params.json): 72 layers; every +# fourth from 39 on attends over layer 35's latent; the indexer is 'full' on layers 0-16 and +# then on two of every four. +_SOURCES = [i for i in range(72)] +for _i in range(39, 72, 4): + _SOURCES[_i] = 35 +_SHARING = ["full"] * 17 + [ + ("shared", "shared", "full", "full")[(i - 17) % 4] for i in range(17, 72) +] + + +def _model_config(sources=_SOURCES, sharing=_SHARING): + cfg = MistralLarge4Config( + num_hidden_layers=len(sources), + kv_source_layers=sources, + index_sharing=sharing, + index_n_heads=32, + index_head_dim=128, + index_topk=2048, + kv_lora_rank=512, + qk_rope_head_dim=64, + ) + sparse = DeepSeekSparseAttentionConfig(index_n_heads=32, index_head_dim=128, index_topk=2048) + return SimpleNamespace(pretrained_config=cfg, sparse_attention_config=sparse) + + +def test_bytes_per_token_are_owners_times_656_plus_full_indexers_times_132(): + """63 latent owners and 44 full-indexer layers: 63 x 656 + 44 x 132 = 47,136 B/token, + against 72 x 576 + 44 x 132 = 47,280 for the 576-byte pool on every layer.""" + mc = _model_config() + owners = sum(1 for i, s in enumerate(_SOURCES) if s == i) + full = _SHARING.count("full") + assert (owners, full, TOKEN_BYTES) == (63, 44, 656) + mapping = SimpleNamespace(pp_size=1) + assert MistralLarge4CacheManager.get_cache_size_per_token(mc, mapping) == 63 * 656 + 44 * 132 + + +def test_registry_selects_the_manager_from_the_config_class(): + """The DSA branch hands a MistralLarge4Config checkpoint to this manager (V2 only) and + every other DSA checkpoint to the generic one, whatever attributes it carries.""" + mc = _model_config() + sparse, cfg = mc.sparse_attention_config, mc.pretrained_config + assert get_sparse_attn_kv_cache_manager(sparse, True, cfg) is MistralLarge4CacheManager + assert get_sparse_attn_kv_cache_manager(sparse, True, SimpleNamespace()) is DSACacheManagerV2 + other = SimpleNamespace(kv_source_layers=_SOURCES) + assert get_sparse_attn_kv_cache_manager(sparse, True, other) is DSACacheManagerV2 + assert get_sparse_attn_kv_cache_manager(sparse, True) is DSACacheManagerV2 + with pytest.raises(ValueError, match="V2"): + get_sparse_attn_kv_cache_manager(sparse, False, cfg) diff --git a/tests/unittest/_torch/attention/sparse/test_flash_mla_gil.py b/tests/unittest/_torch/attention/sparse/test_flash_mla_gil.py new file mode 100644 index 000000000000..53d97b4f8104 --- /dev/null +++ b/tests/unittest/_torch/attention/sparse/test_flash_mla_gil.py @@ -0,0 +1,51 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""FlashMLA's launch wrappers release the GIL. + +A CUDA host function that takes the GIL (the capturable guided decoder's, for one) can be queued ahead of +FlashMLA's kernels on the same stream. Once the launch queue is full, a FlashMLA call that kept the GIL while +it blocks in a launch would wait on the host function, and the host function on the GIL. +""" + +import subprocess +import sys +import textwrap + +import pytest +import torch + +_CHILD = textwrap.dedent(""" + import time + + import torch + + import tensorrt_llm.flash_mla_cpp_tllm as flash_mla + from tensorrt_llm._torch.hostfunc import launch_hostfunc + + q = torch.randn(4, 64, 576, device="cuda", dtype=torch.bfloat16) + kv = torch.randn(256, 1, 576, device="cuda", dtype=torch.bfloat16) + indices = torch.randint(0, 256, (4, 1, 64), device="cuda", dtype=torch.int32) + flash_mla.sparse_prefill_fwd(q, kv, indices, 0.05, 512, None, None) + torch.cuda.synchronize() + + ran = [] + torch.cuda._sleep(int(3e9)) # the host function runs only after the device works through this + launch_hostfunc(lambda: ran.append(time.time())) + for _ in range(20000): # far more launches than the queue holds + flash_mla.sparse_prefill_fwd(q, kv, indices, 0.05, 512, None, None) + torch.cuda.synchronize() + print("host function ran", len(ran)) +""") + + +@pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability()[0] != 10, + reason="FlashMLA sparse prefill on SM100-family GPUs", +) +def test_sparse_prefill_does_not_hold_the_gil_while_launching(): + result = subprocess.run( + [sys.executable, "-c", _CHILD], capture_output=True, text=True, timeout=120 + ) + + assert result.returncode == 0, result.stderr[-2000:] + assert "host function ran 1" in result.stdout diff --git a/tests/unittest/_torch/misc/test_grammar_matcher.py b/tests/unittest/_torch/misc/test_grammar_matcher.py index 5a30fe91ccc7..924b19cc833b 100644 --- a/tests/unittest/_torch/misc/test_grammar_matcher.py +++ b/tests/unittest/_torch/misc/test_grammar_matcher.py @@ -2,9 +2,16 @@ # SPDX-License-Identifier: Apache-2.0 """Draft tokens from the padded part of the vocabulary reaching the guided-decoding matcher.""" +import json + import pytest +from utils.llm_data import llm_models_root -from tensorrt_llm._torch.pyexecutor.grammar_matcher import XGrammarMatcherFactory +from tensorrt_llm._torch.pyexecutor.grammar_matcher import ( + LLGuidanceMatcherFactory, + XGrammarMatcherFactory, +) +from tensorrt_llm._torch.pyexecutor.py_executor_creator import get_guided_decoding_config from tensorrt_llm.bindings.executor import GuidedDecodingParams from tensorrt_llm.llmapi.llm_args import GuidedDecodingConfig @@ -41,3 +48,27 @@ def test_vocabulary_ids_are_decided_by_the_grammar(): matcher = _json_matcher() assert not matcher.accept_token(VOCAB.index("}")) assert matcher.accept_token(VOCAB.index("[")) + + +def test_llguidance_builds_its_tokenizer_from_the_hugging_face_tokenizer_json(): + """Given no ready tokenizer, the llguidance factory builds one from the config's JSON.""" + from tensorrt_llm.tokenizer import TransformersTokenizer + + root = llm_models_root() + model_dir = None if root is None else root / "Qwen3" / "Qwen3-0.6B" + if model_dir is None or not model_dir.is_dir(): + pytest.skip("needs Qwen3/Qwen3-0.6B under LLM_MODELS_ROOT") + tokenizer = TransformersTokenizer.from_pretrained(str(model_dir)) + vocab_size = len(tokenizer.tokenizer) + vocab_size_padded = (vocab_size + 127) // 128 * 128 + 128 + config = get_guided_decoding_config("llguidance", tokenizer) + assert config.backend == GuidedDecodingConfig.GuidedDecodingBackend.LLGUIDANCE + factory = LLGuidanceMatcherFactory(config, vocab_size_padded) + schema = {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]} + params = GuidedDecodingParams(GuidedDecodingParams.GuideType.JSON_SCHEMA, json.dumps(schema)) + + matcher = factory.create(params) + for token_id in tokenizer.encode('{"city": "Paris"}', add_special_tokens=False): + assert matcher.accept_token(token_id), token_id + assert matcher.accept_token(tokenizer.eos_token_id) + assert matcher.is_terminated() diff --git a/tests/unittest/_torch/misc/test_tekken_guided_decoding.py b/tests/unittest/_torch/misc/test_tekken_guided_decoding.py new file mode 100644 index 000000000000..e3cf4874e07c --- /dev/null +++ b/tests/unittest/_torch/misc/test_tekken_guided_decoding.py @@ -0,0 +1,213 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Guided decoding over Mistral Tekken vocabularies, with xgrammar and llguidance. + +xgrammar takes each token's raw bytes; control tokens are empty strings it never selects. +llguidance takes the tokenizer mistral-common wraps for it, whose control tokens are +special, so a JSON grammar never selects them either. +""" + +import json +import shutil +from types import SimpleNamespace + +import pytest +from test_common.mistral_tekken import ( + HF_TOKENIZER_IDS, + HF_TOKENIZER_TEXT, + TEKKEN_SOURCES, + bundled_tekken_file, + load_tekken, + write_hf_tokenizer, +) + +from tensorrt_llm._torch.models.checkpoints.mistral.tokenizer import MistralTokenizer +from tensorrt_llm._torch.pyexecutor.grammar_matcher import ( + LLGuidanceMatcherFactory, + XGrammarMatcherFactory, +) +from tensorrt_llm._torch.pyexecutor.py_executor_creator import ( + _llguidance_tokenizer, + _load_guided_decoding_tokenizer, + get_guided_decoding_config, +) +from tensorrt_llm.bindings.executor import GuidedDecodingParams +from tensorrt_llm.tokenizer import TransformersTokenizer + +pytestmark = pytest.mark.cpu_only + +_SCHEMA = json.dumps( + {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]} +) +_GUIDES = [("JSON_SCHEMA", _SCHEMA), ("JSON", None)] + + +@pytest.fixture(scope="module", params=TEKKEN_SOURCES) +def tokenizer(request): + return load_tekken(request.param) + + +@pytest.fixture(scope="module") +def llg(tokenizer): + return tokenizer.llguidance_tokenizer(tokenizer.tokenizer.n_words) + + +def _text(tokenizer, text): + return tokenizer.tokenizer.encode(text, bos=False, eos=False) + + +def _consume(matcher, ids): + return all(matcher.accept_token(token_id) for token_id in ids) + + +def _allowed(matcher, n_words): + import torch + + bitmask = torch.zeros(1, (n_words + 31) // 32, dtype=torch.int32) + matcher.fill_next_token_bitmask(bitmask, 0) + return {i for i in range(n_words) if (int(bitmask[0, i // 32]) >> (i % 32)) & 1} + + +def _matcher(tokenizer, llg, backend, guide_type, guide): + """A matcher built as the executor builds it.""" + config = get_guided_decoding_config(backend, tokenizer) + n_words = tokenizer.tokenizer.n_words + if backend == "xgrammar": + factory = XGrammarMatcherFactory(config, n_words, max_num_draft_tokens=8) + else: + factory = LLGuidanceMatcherFactory(config, n_words, tokenizer=llg) + return factory.create( + GuidedDecodingParams(getattr(GuidedDecodingParams.GuideType, guide_type), guide) + ) + + +def test_guided_decoding_configs(tokenizer): + n_words = tokenizer.tokenizer.n_words + xgrammar = get_guided_decoding_config("xgrammar", tokenizer) + assert xgrammar.tokenizer_str is None + assert xgrammar.stop_token_ids == [tokenizer.eos_token_id] + assert len(xgrammar.encoded_vocab) == n_words + + llguidance = get_guided_decoding_config("llguidance", tokenizer) + assert llguidance.tokenizer_str is None and llguidance.encoded_vocab is None + assert llguidance.stop_token_ids == [tokenizer.eos_token_id] + assert _llguidance_tokenizer(llguidance, tokenizer, n_words) is not None + assert _llguidance_tokenizer(xgrammar, tokenizer, n_words) is None + assert get_guided_decoding_config(None, tokenizer) is None + with pytest.raises(ValueError, match="Unsupported guided decoding backend"): + get_guided_decoding_config("outlines", tokenizer) + + +@pytest.fixture +def dual_format_dir(tmp_path): + """A checkpoint directory with a Tekken file beside a complete HF checkpoint's + config.json and tokenizer.json, as Ministral-3 ships them.""" + shutil.copy(bundled_tekken_file(), tmp_path / "tekken.json") + write_hf_tokenizer(tmp_path) + (tmp_path / "config.json").write_text(json.dumps({"model_type": "mistral"})) + return tmp_path + + +def _llm_args(checkpoint_format=None, custom_tokenizer=None): + return SimpleNamespace( + checkpoint_format=checkpoint_format, + custom_tokenizer=custom_tokenizer, + trust_remote_code=False, + ) + + +_HF_TOKENIZER = "tensorrt_llm.tokenizer.TransformersTokenizer" + + +@pytest.mark.parametrize( + "llm_args,keep_tekken,expected", + [ + (_llm_args("mistral"), True, MistralTokenizer), + (_llm_args("mistral_large_3"), True, MistralTokenizer), + # Guided decoding masks Tekken vocabularies only; a SentencePiece checkpoint keeps HF's. + (_llm_args("mistral"), False, TransformersTokenizer), + (_llm_args("mistral_large_3"), False, TransformersTokenizer), + (_llm_args(), True, TransformersTokenizer), + (_llm_args("HF"), True, TransformersTokenizer), + # An explicit custom tokenizer takes precedence over the format's. + (_llm_args("mistral", custom_tokenizer=_HF_TOKENIZER), True, TransformersTokenizer), + (_llm_args(custom_tokenizer="mistral_common"), True, MistralTokenizer), + ], + ids=[ + "mistral", + "mistral_large_3", + "mistral_without_tekken", + "mistral_large_3_without_tekken", + "default", + "HF", + "custom_hf_tokenizer", + "custom_mistral_common", + ], +) +def test_a_rank_reloads_the_tokenizer_its_input_processor_encodes_with( + dual_format_dir, llm_args, keep_tekken, expected +): + if not keep_tekken: + (dual_format_dir / "tekken.json").unlink() + tokenizer = _load_guided_decoding_tokenizer(llm_args, str(dual_format_dir)) + assert type(tokenizer) is expected + if expected is TransformersTokenizer: + assert tokenizer.encode(HF_TOKENIZER_TEXT) == HF_TOKENIZER_IDS + + +def test_llguidance_masks_a_padded_vocabulary(tokenizer): + """The bitmask covers the model's padded vocabulary (a TP size that does not divide the + vocabulary pads it); a padded id is never allowed and is refused without an error.""" + n_words = tokenizer.tokenizer.n_words + padded = n_words + 96 + llg = tokenizer.llguidance_tokenizer(padded) + assert llg.vocab_size == padded + config = get_guided_decoding_config("llguidance", tokenizer) + factory = LLGuidanceMatcherFactory(config, padded, tokenizer=llg) + matcher = factory.create( + GuidedDecodingParams(GuidedDecodingParams.GuideType.JSON_SCHEMA, _SCHEMA) + ) + allowed = _allowed(matcher, padded) + assert allowed and max(allowed) < n_words + assert not matcher.accept_token(padded - 1) + assert _consume(matcher, _text(tokenizer, '{"city": "Paris"}')) + + +@pytest.mark.parametrize("backend", ["xgrammar", "llguidance"]) +@pytest.mark.parametrize("guide_type, guide", _GUIDES) +def test_json_response(tokenizer, llg, backend, guide_type, guide): + """A valid answer is accepted and prose is not; a rejected draft rolls back; neither the + response's first token nor a JSON string is a control token.""" + tekken = tokenizer.tokenizer + + def make(): + return _matcher(tokenizer, llg, backend, guide_type, guide) + + matcher = make() + assert min(_allowed(matcher, tekken.n_words)) >= tekken.num_special_tokens + assert _consume(matcher, _text(tokenizer, '{"city": "巴黎 São Paulo 🙂"}')) + assert matcher.accept_token(tokenizer.eos_token_id) + assert not _consume(make(), _text(tokenizer, 'Sure: {"city": "Paris"}')) + + matcher = make() + assert _consume(matcher, _text(tokenizer, '{"city": ')) + before = _allowed(matcher, tekken.n_words) + draft = _text(tokenizer, '"Paris"}') + assert _consume(matcher, draft) + matcher.rollback(len(draft)) + assert _allowed(matcher, tekken.n_words) == before + assert _consume(matcher, _text(tokenizer, '"')) + allowed = _allowed(matcher, tekken.n_words) + assert allowed and min(allowed) >= tekken.num_special_tokens diff --git a/tests/unittest/_torch/modeling/test_modeling_mistral.py b/tests/unittest/_torch/modeling/test_modeling_mistral.py index 62f22e2df0f7..7ce5901f058e 100644 --- a/tests/unittest/_torch/modeling/test_modeling_mistral.py +++ b/tests/unittest/_torch/modeling/test_modeling_mistral.py @@ -782,16 +782,9 @@ def test_native_processed_images_preserve_item_sizes( pixels = [ torch.full((3, height, width), float(i + 1)) for i, (height, width) in enumerate(sizes) ] - encoded = transformers.BatchEncoding( - { - "input_ids": [1, 2, 3], - "attention_mask": [1, 1, 1], - "pixel_values": [pixel.numpy() for pixel in pixels], - "image_sizes": sizes, - } - ) + encoded = SimpleNamespace(tokens=[1, 2, 3], images=[pixel.numpy() for pixel in pixels]) tokenizer = SimpleNamespace( - transformers_tokenizer=mock.Mock(), + mistral=mock.Mock(), instruct=SimpleNamespace( mm_encoder=mock.Mock( spec=modeling_mistral.ImageEncoder, @@ -799,14 +792,7 @@ def test_native_processed_images_preserve_item_sizes( ) ), ) - - def apply_chat_template( - *args: Any, return_tensors: str | None, **kwargs: Any - ) -> transformers.BatchEncoding: - assert return_tensors is None - return encoded - - tokenizer.transformers_tokenizer.apply_chat_template.side_effect = apply_chat_template + tokenizer.mistral.encode_chat_completion.return_value = encoded proc._processor = modeling_mistral.MistralCommonImageProcessor(tokenizer, proc.dtype) input_ids, extra = proc.call_with_text_prompt( { diff --git a/tests/unittest/_torch/modeling_v2/activation/test_modeling_v2_flashinfer_silu_and_mul.py b/tests/unittest/_torch/modeling_v2/activation/test_modeling_v2_flashinfer_silu_and_mul.py index 52a89bcfcb80..e121d4a8c9cd 100644 --- a/tests/unittest/_torch/modeling_v2/activation/test_modeling_v2_flashinfer_silu_and_mul.py +++ b/tests/unittest/_torch/modeling_v2/activation/test_modeling_v2_flashinfer_silu_and_mul.py @@ -3,7 +3,10 @@ """GPU test for the flashinfer_silu_and_mul catalog entry. The entry owns what is certified. This file turns a cell's spec into tensors -and drives the inputs the guard exists to refuse. +and drives the inputs the guard exists to refuse. A cell whose spec says +`large` is the every-count sweep at dep4's width 49152, past the per-cell +budget: `test_large_certified_cells` drives it, which pre-merge +(`-m "not large"`) deselects, and pre-merge drives its `sampled` cell instead. """ import zlib @@ -18,7 +21,7 @@ __extra_import_path__ = [".."] from _cells import certified_cells # noqa: E402 — needs the path declared above -from _validating import validating # noqa: E402 — needs the path declared above +from _validating import certifying, validating # noqa: E402,F401 — needs the path declared above assert torch.cuda.is_available(), "flashinfer_silu_and_mul requires a CUDA device" @@ -30,6 +33,151 @@ def _build(spec, seed): # The shapes each shipped target calls this entry with, by target directory # under `models/`. The entry's own `CELLS` are the op's whoever calls it. +# +# Mistral Large 4 NVFP4 at tep4 and tp4 (tp4 differs from tep4 only in its +# routed experts, which do not call this entry) calls it on two gate_up outputs, +# each TP4-sharded to [gate | up] per rank: the dense layer 0 (intermediate +# 24576, so 2 x 6144) and the one shared expert of layers 1..71 (6144, so +# 2 x 1536). Each call site passes one row per token: decode is captured at the +# batches the engine pads to (1, 2, 4, 8, 16), prefill is eager at any row count +# up to the token budget -- 32768 (`max_num_tokens` 32768 with chunked prefill +# off, so one context step carries every row of up to 16 packed prompts); the +# 9216-row cells are the smaller budget the same widths ran at first. `sweep` +# drives every row count from 1 to `shape[0]`, each as its own call: "eager" +# against the call at `shape` bit for bit, "graph" each captured in its own CUDA +# graph and replayed on new data, bit for bit against eager. The two 32768-row +# sweeps are the file's costliest cells: 6.2 s and 2.1 s on sm_103 at widths +# 12288 and 3072, peaking 4.1 and 1.0 GiB of device memory. +_ML4_WIDTH_3072 = ( + Cell( + why="eager rows 1..9216 at width 3072: the shared expert's gate and up, " + "2 x 1536 per rank at tep4, every prefill row count up to the token budget", + spec=dict(shape=(9216, 3072), dtype=torch.bfloat16, sweep="eager"), + ), + Cell( + why="eager rows 1..32768 at width 3072: the shared expert's gate and up, " + "2 x 1536 per rank at tep4, every prefill row count up to max_num_tokens 32768", + spec=dict(shape=(32768, 3072), dtype=torch.bfloat16, sweep="eager"), + ), + Cell( + why="CUDA graphs at rows 1..16, width 3072: the decode batches the engine " + "captures (1, 2, 4, 8, 16, padded) and the rest an unpadded capture adds", + spec=dict(shape=(16, 3072), dtype=torch.bfloat16, sweep="graph"), + ), +) +# Width 12288 is also dep4's replicated shared expert, at the same rows. +_ML4_WIDTH_12288 = ( + Cell( + why="eager rows 1..32768 at width 12288: dense layer 0's gate and up, " + "2 x 6144 per rank at tep4, every prefill row count up to max_num_tokens 32768", + spec=dict(shape=(32768, 12288), dtype=torch.bfloat16, sweep="eager"), + ), + Cell( + why="CUDA graphs at rows 1..16, width 12288: the decode batches the engine " + "captures (1, 2, 4, 8, 16, padded) and the rest an unpadded capture adds", + spec=dict(shape=(16, 12288), dtype=torch.bfloat16, sweep="graph"), + ), +) +_ML4_TP_WORLD4 = ( + Cell( + why="eager rows 1..9216 at width 12288: dense layer 0's gate and up, " + "2 x 6144 per rank at tep4, every prefill row count up to the token budget", + spec=dict(shape=(9216, 12288), dtype=torch.bfloat16, sweep="eager"), + ), + *_ML4_WIDTH_12288, + *_ML4_WIDTH_3072, +) + +# The same two targets over eight ranks (tep8 and tp8, attention DP off) slice +# both call sites eight ways: dense layer 0's gate_up is 2 x 3072 = 6144 per +# rank and the shared expert's 2 x 768 = 1536, at the four-rank rows (1..32768 +# eager, the padded decode batches 1/2/4/8/16 captured). Each width launches a +# block size no other cell runs (384 and 96 threads, see the entry's `note`), so +# each has its own every-count sweep and graph cell. On sm_103 the two +# 32768-row sweeps took 3.5 s and 1.9 s, peaking 2.1 and 0.5 GiB of device +# memory. +_ML4_TP_WORLD8 = ( + Cell( + why="eager rows 1..32768 at width 6144: dense layer 0's gate and up, " + "2 x 3072 per rank at tep8 and tp8, every prefill row count up to max_num_tokens " + "32768", + spec=dict(shape=(32768, 6144), dtype=torch.bfloat16, sweep="eager"), + ), + Cell( + why="eager rows 1..32768 at width 1536: the shared expert's gate and up, " + "2 x 768 per rank at tep8 and tp8, every prefill row count up to max_num_tokens " + "32768", + spec=dict(shape=(32768, 1536), dtype=torch.bfloat16, sweep="eager"), + ), + Cell( + why="CUDA graphs at rows 1..16, width 6144: the decode batches the engine " + "captures at tep8 and tp8 (1, 2, 4, 8, 16, padded) and the rest an unpadded " + "capture adds", + spec=dict(shape=(16, 6144), dtype=torch.bfloat16, sweep="graph"), + ), + Cell( + why="CUDA graphs at rows 1..16, width 1536: the decode batches the engine " + "captures at tep8 and tp8 (1, 2, 4, 8, 16, padded) and the rest an unpadded " + "capture adds", + spec=dict(shape=(16, 1536), dtype=torch.bfloat16, sweep="graph"), + ), +) + +# Mistral Large 4 NVFP4 at dep4 and dep8 (attention data parallel) runs dense +# layer 0 replicated on each rank's own tokens, so its gate_up output is the +# whole [gate | up], 2 x 24576 = 49152, at the same per-rank rows: 1..32768 +# eager, the padded decode batches 1/2/4/8/16 captured. Its replicated shared +# expert keeps width 12288, `_ML4_WIDTH_12288`. Every row count at +# 49152 moves four times the bytes of 12288, so that sweep is `large`, past the +# 10 s budget, and pre-merge drives "sampled" instead: the 32768-row call gated +# against `reference`, and each count in 1..`dense_to`, either side of every +# multiple of `period` and the last `tail` bitwise its leading rows. The period +# is one wave of resident CTAs on sm_103 (two 1024-thread CTAs on each SM). On +# sm_103 the `large` cell took 21.0 s and the sampled one 0.3 s, each reserving +# 18.0 GiB (the reference gate on the 32768 x 49152 call); the 49152 graphs +# 0.1 s. +_ML4_DP = ( + Cell( + why="eager rows 1..32768 at width 49152: dense layer 0's gate and up, 2 x 24576 " + "replicated on each rank's own tokens at dep4, every prefill row count up to " + "max_num_tokens 32768", + spec=dict(shape=(32768, 49152), dtype=torch.bfloat16, sweep="eager", large=True), + ), + Cell( + why="eager rows sampled to 32768 at width 49152, pre-merge's share of the large " + "every-count cell: every count through two waves of resident CTAs, both sides " + "of each later wave, the last 16", + spec=dict( + shape=(32768, 49152), + dtype=torch.bfloat16, + sweep="sampled", + dense_to=608, + period=304, + tail=16, + ), + ), + Cell( + why="CUDA graphs at rows 1..16, width 49152: the decode batches the engine " + "captures at dep4 (1, 2, 4, 8, 16, padded) and the rest an unpadded capture adds", + spec=dict(shape=(16, 49152), dtype=torch.bfloat16, sweep="graph"), + ), +) + + +def _ml4_dspark(*widths: int) -> tuple[Cell, ...]: + """DSpark's dense draft and target MLPs at verification batch sizes, graph rows 1..64. + + Their index bounds are below the eager envelope of the cells above. + """ + return tuple( + Cell( + why=f"DSpark SwiGLU width {width}, graph rows 1..64", + spec=dict(shape=(64, width), dtype=torch.bfloat16, sweep="graph"), + ) + for width in widths + ) + + TARGET_CELLS: dict[str, tuple[Cell, ...]] = { "deepseek_v3/r1_0528_nvfp4__sm_10x__default": ( Cell( @@ -37,21 +185,187 @@ def _build(spec, seed): spec=dict(shape=(16, 512), dtype=torch.bfloat16), ), ), + "mistral_large4/mistral_large_4_nvfp4__sm_103__tep": ( + *_ML4_TP_WORLD4, + *_ML4_TP_WORLD8, + *_ml4_dspark(12288, 3072, 6144, 1536), + ), + "mistral_large4/mistral_large_4_nvfp4__sm_103__tp": ( + *_ML4_TP_WORLD4, + *_ML4_TP_WORLD8, + *_ml4_dspark(12288, 3072, 6144, 1536), + ), + "mistral_large4/mistral_large_4_nvfp4__sm_103__dep": ( + *_ML4_DP, + *_ML4_WIDTH_12288, + *_ml4_dspark(49152, 12288), + ), } CELLS = certified_cells(op, TARGET_CELLS) -@pytest.mark.parametrize("cell", CELLS, ids=[c.why[:40] for c in CELLS]) -def test_certified_cells(cell) -> None: - x = _build(cell.spec, seed=zlib.crc32(cell.why.encode()) % 2**31) - with validating(op): - out = op(x) +def _check(x: torch.Tensor, out: torch.Tensor) -> None: + """Shape, a fresh output, and torch's dtype-aware default band. + + The band is torch's bf16 default (rtol 1.6e-2, atol 1e-5); the test adds + none of its own. Kernel and reference both form silu(gate) * up in fp32 + and round once to the input dtype; they differ only in how the fp32 exp + and quotient are evaluated (the kernel's `silu` uses `__expf`), which + surfaces as one bf16 rounding flip, or as +-0 where a gate near -88 puts + the reference below 2e-35. Measured at widths 12288 and 3072 at 9216 rows, + on the cells' own draws: bitwise equal for randn inputs; at 16x that + scale, one step on 1730 of 56623104 outputs and three +-0 at width 12288, + one step on 405 of 14155776 at 3072. Wrong variants at 9216 rows land far + outside: gelu in place of silu at 62x the band, the halves swapped at + 8105-9383x, sigmoid in place of silu at 1.7e5x. Rounding silu(gate) to bf16 + before the multiply stays inside (0.49 of the band, 1 ulp): that is below + what a single bf16 result can show. At 32768 rows, on the cells' own + draws: one output per draw one step off, 0.11 of the band at worst; at 16x, + 5886 of 201326592 one step off and one +-0 where the reference is 8.2e-36, + 0.072 of the band. Outputs right on rows below 9216 and wrong after fail + the band in every row they change: rows read wrapped at 9216 by up to + 1.0e6x, rows read one row early by up to 9.1e5x, rows from 16384 left at + zero by 62.5x. + + Width 49152, where each thread makes three vector passes per half: at + 32768 rows the two eager cells' draws put 2 and 4 outputs one step off, + 0.071 of the band; at 16x, 0.072. Wrong variants at 4096 x 49152: the third + pass skipped (columns 16384.. left zero) 62.5x the band, one thread's last + vector dropped 62.5x, the third pass re-reading the second's vectors + 8.1e5x, up read one vector early 5.1e5x, the halves swapped 1.2e4x, rows + shifted by one 8.5e5x; silu(gate) rounded to bf16 first stays inside, 0.49. + + Widths 6144 and 1536 (tep8 and tp8), one vector pass per half at 384 and 96 + threads: at 32768 rows the cells' draws put 1 and 0 outputs one step off, + 0.011 of the band; at 16x, 3033 and 712 one step off plus 3 and 0 of +-0, + 0.072. Wrong variants on those draws at 32768 rows (6144, then 1536): gelu + in place of silu 62x the band, the last vector of each row dropped and the + rows from 16384 left zero 62.5x, the halves swapped 1.5e4x and 7.8e3x, up + read one vector early 6.8e5x and 4.4e5x, rows shifted by one 8.6e5x and + 8.5e5x, sigmoid in place of silu 1.9e5x and 1.5e5x; silu(gate) rounded to + bf16 first stays inside, 0.49. + """ assert out.shape == (*x.shape[:-1], x.shape[-1] // 2) assert out.data_ptr() != x.data_ptr(), "the op must return a new tensor" torch.testing.assert_close(out, op.reference(x)) +def _words(t: torch.Tensor) -> torch.Tensor: + """`t`'s bits as int64 words over whole contiguous rows, for the sweep's compare.""" + assert t.is_contiguous() and (t.shape[-1] * t.element_size()) % 8 == 0 + return t.view(torch.int64) + + +def _sampled_row_counts(rows, dense_to, period, tail) -> list[int]: + """Every count in 1..dense_to, either side of each multiple of `period`, the last `tail`. + + All below `rows`, whose own call is the one the others are compared with. + """ + counts = set(range(1, dense_to + 1)) + counts.update(m for k in range(period, rows + 1, period) for m in (k - 1, k, k + 1)) + counts.update(range(rows - tail + 1, rows + 1)) + return sorted(m for m in counts if m < rows) + + +def _eager_row_counts(spec, seed, counts) -> None: + """Each row count in `counts` as its own call, on the leading rows of one input. + + The call at `shape` is gated against the reference; each shorter call has + to equal its leading rows bit for bit. A CTA reads only its own row, so a + shorter call that differed would be a launch-shape defect, not rounding. + + Each call is compared as whole rows of int64 words (the same bits, four + bf16 to a word) and reduced with `any()` on the device; the verdicts are + stacked and read once, and the smallest failing row count is re-run to + count its elements. Measured on sm_103 over 1..32768 at width 12288: 6.0 s + this way, 3.2 s for the op calls alone, and 18.6 s with bf16 `!=` and + `.sum()` per call, a form whose peak reserved device memory reached 255 GiB. + At 49152, every count took 20.9 s (11.0 s the op calls, 10.7 GiB + reserved) and the 939 sampled counts 0.27 s; with one bit flipped in the + compared rows at row 20000, the sample flagged exactly its 141 counts past + that row. + """ + x = _build(spec, seed) + full = op(x) + _check(x, full) + rows = x.shape[0] + full_words = _words(full) + differs = torch.stack([torch.ne(_words(op(x[:m])), full_words[:m]).any() for m in counts]) + bad = [counts[i] for i in torch.nonzero(differs).flatten().tolist()] + if bad: + count = int((op(x[: bad[0]]) != full[: bad[0]]).sum()) + raise AssertionError( + f"rows={bad[0]}: {count} elements differ from the leading rows of the {rows}-row " + f"call; {len(bad)} row counts differ, the smallest {bad[:16]}" + ) + + +def _every_captured_row_count(spec, seed) -> None: + """Each row count 1..rows captured in its own CUDA graph, replayed on new data. + + The replay has to equal an eager call on the same data bit for bit -- a + graph that froze anything of the capture would differ -- and is gated + against the reference like any other output. One eager call on the capture + stream precedes each capture, as an engine's warm-up does. + """ + rows, width = spec["shape"] + stream = torch.cuda.Stream() + for m in range(1, rows + 1): + shape = dict(spec, shape=(m, width)) + x = _build(shape, seed + m) + stream.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(stream): + op(x) + torch.cuda.current_stream().wait_stream(stream) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=stream): + out = op(x) + fresh = _build(shape, seed + rows + m) + x.copy_(fresh) + graph.replay() + eager = op(fresh) + assert torch.equal(out, eager), f"rows={m}: graph replay differs from eager" + _check(fresh, out) + + +def _drive(cell) -> None: + seed = zlib.crc32(cell.why.encode()) % 2**31 + sweep = cell.spec.get("sweep") + rows = cell.spec["shape"][0] + if sweep == "eager": + _eager_row_counts(cell.spec, seed, list(range(1, rows))) + elif sweep == "sampled": + counts = _sampled_row_counts( + rows, cell.spec["dense_to"], cell.spec["period"], cell.spec["tail"] + ) + _eager_row_counts(cell.spec, seed, counts) + elif sweep == "graph": + _every_captured_row_count(cell.spec, seed) + else: + assert sweep is None, f"unknown sweep {sweep!r}" + x = _build(cell.spec, seed) + _check(x, op(x)) + + +@pytest.mark.parametrize("cell", CELLS, ids=[c.why[:44] for c in CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 + if cell.spec.get("large"): + pytest.skip("past the per-cell budget; test_large_certified_cells drives it") + certifying(op) + _drive(cell) + + +_LARGE_CELLS = tuple(c for c in CELLS if c.spec.get("large")) + + +@pytest.mark.large +@pytest.mark.parametrize("cell", _LARGE_CELLS, ids=[c.why[:44] for c in _LARGE_CELLS]) +def test_large_certified_cells(cell, certifying) -> None: # noqa: F811 + certifying(op) + _drive(cell) + + @pytest.mark.parametrize( "width,match", [ @@ -76,3 +390,56 @@ def test_guard_refuses_widths_the_kernel_faults_on(width, match) -> None: with pytest.raises(AssertionError, match=match): with validating(op): op(x) + + +@pytest.mark.parametrize("width", [12288, 3072, 6144, 1536]) +@pytest.mark.parametrize("layout", ["column slice", "every other row"]) +def test_guard_refuses_a_strided_input(layout, width) -> None: + """The kernel reads row r at `r * width` from the base and ignores strides. + + Shown unguarded first: the op accepts a view whose rows are not packed and + computes every row but the first from the wrong elements. Both views keep + the rows the kernel reads inside the buffer this test allocated, and the + output is the op's own allocation. + """ + if layout == "column slice": + x = torch.randn(16, width + 256, device="cuda").to(torch.bfloat16)[:, :width] + else: + x = torch.randn(32, width, device="cuda").to(torch.bfloat16)[::2] + assert not x.is_contiguous() + out, ref = op(x), op.reference(x) + assert torch.equal(out[0], ref[0]), "row 0 sits at the base, where both read it" + wrong = (out[1:] != ref[1:]).float().mean().item() + assert wrong > 0.5, f"only {wrong:.3f} of rows 1.. differ; the strided read went unseen" + with pytest.raises(AssertionError, match="must be contiguous"): + with validating(op): + op(x) + + +def test_guard_refuses_a_misaligned_base() -> None: + """A packed view that starts off a 16-byte boundary faults the vector loads. + + Not driven unguarded: the fault is `CUDA error: misaligned address`, which + is sticky and would take every later test in the process with it. + """ + buf = torch.randn(16 * 3072 + 8, device="cuda").to(torch.bfloat16) + x = buf[1 : 1 + 16 * 3072].view(16, 3072) + assert x.is_contiguous() and x.data_ptr() % 16 == 2 + with pytest.raises(AssertionError, match="16-byte aligned"): + with validating(op): + op(x) + + +@pytest.mark.parametrize( + "shape,dtype,match", + [ + ((16, 3072), torch.float32, "failed to dispatch data type"), + ((0, 3072), torch.bfloat16, "Failed to launch kernel: invalid argument"), + ], + ids=["fp32", "zero rows"], +) +def test_the_op_raises_itself(shape, dtype, match) -> None: + """What the `note` says the op refuses loudly, kept true by running it.""" + x = torch.randn(shape, device="cuda").to(dtype) + with pytest.raises(RuntimeError, match=match): + op(x) diff --git a/tests/unittest/_torch/modeling_v2/comm/_allgather_op_matrix.py b/tests/unittest/_torch/modeling_v2/comm/_allgather_op_matrix.py index 9dc9d8aef579..ad738c81442b 100644 --- a/tests/unittest/_torch/modeling_v2/comm/_allgather_op_matrix.py +++ b/tests/unittest/_torch/modeling_v2/comm/_allgather_op_matrix.py @@ -3,39 +3,69 @@ """GPU certification matrix for the allgather catalog entry. A collective cannot be exercised in one process, so this script is its own -launcher: run it plainly and it spawns one worker process per device, with one -rank per device named in CUDA_VISIBLE_DEVICES, under a deadline the parent -enforces by killing the whole process group. A wedged collective hangs -rather than raising — and this op wedges rather than raising for every -rank-argument-disagreement case probed — so the deadline is what keeps a -broken kernel from taking the calling run down with it. +launcher: run it plainly and it starts one rank per device named in +CUDA_VISIBLE_DEVICES, under a deadline the parent enforces by killing the +ranks. A wedged collective hangs rather than raising — and this op wedges +rather than raising for every rank-argument-disagreement case probed — so the +deadline is what keeps a broken kernel from taking the calling run down with +it. Beyond the lockstep surface (uniform / ragged / dtypes / CUDA graphs) the -file covers what a serving engine adds: the engine's own attention-DP +`_pg` job covers what a serving engine adds: the engine's own attention-DP synchronisation interleaved with the op, the engine's stream switching, and what disagreeing on call order actually does. CUDA_VISIBLE_DEVICES=0,1,2,3 python _allgather_op_matrix.py +That runs two jobs, in this order, and both must exit 0: + +- the `_pg` job: plain worker processes that rendezvous through + torch.distributed's env://, every `CHECKS` entry on `world_size` ranks. It + drives the cells that do not state `process_group`, through the + `allgather_pg` symbol on a boxed c10d group; +- the MPI-session job: the same number of ranks started by `mpirun`, every + `MPI_CHECKS` entry. It drives the cells that state `process_group=None` at + that world size, through `torch.ops.trtllm.allgather` on the communicator + the MPI session builds -- the symbol and communicator a dep4, tep4 or tp4 + world launched under mpirun reaches. Its `mpirun` gets an environment with + the launching allocation's PMI/PMIx/Open MPI/Slurm variables removed, so it + starts its own ranks on this node rather than joining the allocation's step. + +The world-8 cells -- the dep8 gather and the tep8 q-split gather, eight ranks +over two nodes of four -- are a third job, the MPI-session checks again on +ranks an MPI launcher starts across both nodes, one per device, as the dep8 +and tep8 engines' ranks are. One process cannot place ranks on another node, +so the launcher is the environment's (the collected test takes it from +`MODELING_V2_MPI_LAUNCH`): + + python _allgather_op_matrix.py --mpi-rank-worker + +Each MPI-session job drives the cells of its own world size and node count, +and runs each check on the kind of cell it holds: the four-rank job the dep4 +gather's and the tep4 and tp4 q-split's, the eight-rank job the dep8 gather's +and the tep8 q-split's. A rank that wedges inside a collective is ended by +faulthandler at `MPI_DEADLINE_S`, and the launcher ends its siblings. + Not a pytest module, despite the `check_*` bodies. They are one fixed -sequence inside a single 4-rank job rather than independent cases: each reads -module-global rank state that only `_run_one_rank` binds, and several assert +sequence inside a single job rather than independent cases: each reads +module-global rank state that only the rank body binds, and several assert on communicator state the previous one left behind. Collected as tests they would run at world size 1 against unbound globals — so neither this file's name nor its function names match pytest's collection patterns, which is what keeps it uncollectable however pytest is pointed at this tree. -The collected entry point is -`tests/unittest/_torch/modeling_v2/comm/test_modeling_v2_allgather_op_matrix.py`: -it starts this job and turns its exit code into an assertion. Both halves are -started by file path: the launcher must not import `tensorrt_llm` (that calls -CUDA nor torch.distributed state of its own), and the +The collected entry point is `test_modeling_v2_allgather_op_matrix.py` beside +this file: it starts these jobs and turns their exit codes into assertions. +Both halves are started by file path: the launcher must not import +`tensorrt_llm` (that would give it CUDA and MPI state of its own), and the ranks reach the catalog by absolute import, so neither needs a package. """ +import faulthandler import functools import os import random +import shutil import signal import socket import subprocess @@ -53,7 +83,13 @@ assert torch.cuda.is_available(), "allgather requires CUDA devices" _WORKER_FLAG = "--rank-worker" -DEADLINE_S = 1200 +_MPI_WORKER_FLAG = "--mpi-rank-worker" +# Each job's own deadline; `_rank_job` reads their sum off DEADLINE_S. Each +# MPI-session rank also ends itself at MPI_DEADLINE_S, and the collected test +# allows the world-8 job's launcher that plus a grace. +PG_DEADLINE_S = 1200 +MPI_DEADLINE_S = 600 +DEADLINE_S = PG_DEADLINE_S + MPI_DEADLINE_S HIDDEN = 2560 # DeepSeek-V3-Lite hidden size, the caller that motivates this @@ -73,11 +109,15 @@ SITES_PER_DECODE_GRAPH = 29 # Bound inside the rank body, never in the launcher: importing the entry -# pulls in tensorrt_llm, which calls MPI_Init at import, and an +# pulls in tensorrt_llm, which calls MPI_Init at import, and the launcher has +# to stay free of MPI and CUDA state. allgather: Any = None entry: Any = None +MPI_COMM: Any = None RANK = 0 WORLD = 1 +#: Distinct hosts the MPI-session job's ranks run on. +NODES = 1 GROUP: List[int] = [0] # The engine's own cross-rank synchronisation object, built from the real # Mapping a serving engine builds under attention data parallelism. Bound in @@ -353,16 +393,25 @@ def check_cuda_graph_captures_a_cold_first_call() -> None: dist.barrier() +def _on_mpi_session(cell: Any) -> bool: + """Whether `cell` drives the MPI-session symbol: it states `process_group=None`.""" + return "process_group" in cell.spec and cell.spec["process_group"] is None + + def check_certified_cells() -> None: - """Drive the entry's own cell list, inside `validating`. + """Drive the entry's `_pg` cells, inside `validating`. - The other checks in this file say things no cell can -- CUDA graph + The other checks in this job say things no cell can -- CUDA graph behaviour, call-order disagreement, which stream the call lands on. This one - says the plain thing: for every configuration the entry claims to be - certified over, the gather is the concatenation, and the guard admits the - input a shipped target passes. + says the plain thing: for every configuration these cells claim, the + gather is the concatenation, and the guard admits the input a shipped + target passes. """ + driven = 0 for i, cell in enumerate(entry.allgather.CELLS): + if _on_mpi_session(cell): + continue # the MPI-session job drives it (`MPI_CHECKS`) + driven += 1 spec = cell.spec rows, trailing, dtype = spec["rows"], spec["trailing"], spec["dtype"] seed = 9000 + 13 * i @@ -372,6 +421,7 @@ def check_certified_cells() -> None: assert out.shape == (rows * WORLD, *trailing), (cell.why, out.shape) _assert_bitwise(out, _gather_ref([rows] * WORLD, seed, dtype, trailing), cell.why) dist.barrier() + assert driven > 0, "no cell drives the _pg symbol" def check_uniform_gather() -> None: @@ -1014,11 +1064,1258 @@ def _run_one_rank(device: Optional[int] = None) -> int: os._exit(0) +# ---- the MPI-session job ------------------------------------------------------ +# +# Everything from here to `MPI_CHECKS` drives the cells that state +# `process_group=None`, on the symbol and the communicator a dep4, tep4, tp4, +# dep8 or tep8 serving world reaches: ranks started by mpirun, +# `torch.ops.trtllm.allgather`, and the NCCL communicator tensorrt_llm's MPI +# session builds for `group` at its first call. These ranks have no +# torch.distributed group; they synchronize through the MPI session itself. One +# job drives the cells of its own world size: four ranks of one node +# (`_mpirun_ranks`), or eight over two nodes (started by the environment's +# launcher). + +#: The entry's own `__call__` and `reference`, counted (`_count_entry_calls`). +SEEN: Dict[str, int] = {"__call__": 0, "reference": 0} +#: What each MPI-session cell ran through the entry: [calls, references]. +DRIVEN: Dict[int, List[int]] = {} + + +def _mpi_cells() -> List[Tuple[int, Any]]: + """(index, cell) of every cell this MPI-session job drives: its world size's.""" + return [ + (i, c) + for i, c in enumerate(entry.allgather.CELLS) + if _on_mpi_session(c) and c.spec["world_size"] == WORLD + ] + + +def _mpi_sync() -> None: + """Every rank's device work done, then every rank here.""" + torch.cuda.synchronize() + MPI_COMM.Barrier() + + +def _count_entry_calls() -> None: + """What `_validating.certifying` does under pytest, for this rank body. + + The entry's own `__call__` (with `is_valid` interposed first, as + `validating` does) and its own `reference` are wrapped to count, so every + cell can assert that both ran for it rather than a local copy. + """ + cls = type(entry.allgather) + call, ref = cls.__dict__["__call__"], cls.__dict__["reference"] + + def counted_call(self, *args, **kwargs): + SEEN["__call__"] += 1 + self.is_valid(*args, **kwargs) + return call(self, *args, **kwargs) + + def counted_reference(self, *args, **kwargs): + SEEN["reference"] += 1 + return ref(self, *args, **kwargs) + + cls.__call__ = counted_call + cls.reference = counted_reference + + +def _account(i: int, before: Dict[str, int]) -> None: + """Add what the entry ran since `before` to cell `i`.""" + got = DRIVEN.setdefault(i, [0, 0]) + got[0] += SEEN["__call__"] - before["__call__"] + got[1] += SEEN["reference"] - before["reference"] + + +def _gather(x: torch.Tensor, spec: Dict[str, Any]) -> torch.Tensor: + """The call the dep4 and dep8 deployments make, with the arguments the cell states.""" + return entry.allgather(x, spec["sizes"], GROUP, process_group=spec["process_group"]) + + +def _width(spec: Dict[str, Any]) -> int: + assert len(spec["trailing"]) == 1 and spec["dtype"] is torch.bfloat16, spec + assert spec["world_size"] == WORLD and spec["sizes"] is None, (spec, WORLD) + # A cell without `nodes` is one node's, as the dep4 cells are. + assert spec.get("nodes", 1) == NODES, (spec, NODES) + return spec["trailing"][0] + + +def _rows(rank: int, shape: Tuple[int, ...], seed: int) -> torch.Tensor: + """`rank`'s rows of `shape`, regenerable on every rank from the shared seed. + + Multiples of 1/8 with |x| <= 31/8, 63 values: two ranks' rows agree in about + one element in 63, so a block from the wrong rank or at the wrong offset + changes nearly every element it covers. + """ + gen = torch.Generator(device="cuda").manual_seed((seed * 977 + rank) * 131 + 7) + raw = torch.randint(-31, 32, shape, generator=gen, device="cuda", dtype=torch.int8) + return raw.to(torch.bfloat16).div_(8.0) + + +def _bits(rank: int, shape: Tuple[int, ...], seed: int) -> torch.Tensor: + """Every bf16 bit pattern equally likely: NaNs, infinities, -0.0, subnormals.""" + gen = torch.Generator(device="cuda").manual_seed((seed * 977 + rank) * 131 + 11) + raw = torch.randint(-(2**15), 2**15, shape, generator=gen, device="cuda", dtype=torch.int32) + return raw.to(torch.int16).view(torch.bfloat16) + + +def _differs(a: torch.Tensor, b: torch.Tensor) -> torch.Tensor: + """Whether two bf16 tensors differ in any bit: a device bool, compared as int64 words.""" + assert a.shape == b.shape and a.dtype == b.dtype == torch.bfloat16, (a.shape, b.shape) + return torch.ne(a.reshape(-1).view(torch.int64), b.reshape(-1).view(torch.int64)).any() + + +def _fraction_moved(a: torch.Tensor, b: torch.Tensor) -> float: + """The fraction of elements whose bits differ between two bf16 tensors.""" + return (a.view(torch.int16) != b.view(torch.int16)).float().mean().item() + + +def _sweep_rows(spec: Dict[str, Any]) -> List[int]: + """The row counts a sweep cell names, largest first.""" + lo, hi = spec["sweep_rows"] + rows = set(range(lo, hi + 1, spec["sweep_step"])) | set(spec.get("sweep_also", ())) | {hi} + assert spec["rows"] == hi and all(lo <= r <= hi for r in rows), spec + # Largest first, so the caching allocator sizes its blocks for the top once. + return sorted(rows, reverse=True) + + +def check_mpi_session_is_the_launchers_world() -> None: + """The ranks are mpirun's, one device each, and the MPI session spans them all. + + `getComm` builds the communicator through the MPI session, so a session of + one process per rank -- what a launch without the ranks in place gives -- + would wedge the first call instead of failing it. Over several nodes each + node's ranks hold that node's devices 0..n-1, in rank order, as the + launcher places them. + """ + from tensorrt_llm._utils import mpi_rank, mpi_world_size + + assert os.environ.get("TLLM_DISABLE_MPI") != "1", "the MPI-session job runs with MPI" + assert mpi_world_size() == MPI_COMM.Get_size() == WORLD, (mpi_world_size(), WORLD) + assert mpi_rank() == MPI_COMM.Get_rank() == RANK, (mpi_rank(), RANK) + placed = MPI_COMM.allgather((socket.gethostname(), torch.cuda.current_device())) + hosts = list(dict.fromkeys(host for host, _ in placed)) + assert len(hosts) == NODES and WORLD % NODES == 0, placed + per_node = WORLD // NODES + assert [hosts.index(h) for h, _ in placed] == [r // per_node for r in range(WORLD)], placed + assert [d for _, d in placed] == [r % per_node for r in range(WORLD)], placed + if RANK == 0: + print( + f"[mpi] {torch.cuda.get_device_name()} sm_" + f"{''.join(map(str, torch.cuda.get_device_capability()))}, NCCL " + f"{torch.cuda.nccl.version()}; {WORLD} ranks on {NODES} node(s): {placed}", + flush=True, + ) + cells = _mpi_cells() + assert cells, "no cell states process_group=None" + for _, cell in cells: + if _on_split(cell): + _split_width(cell.spec) + else: + _width(cell.spec) + _mpi_sync() + + +def check_mpi_eager_sweep_cells() -> None: + """Every sweep cell: at each row count it names, the gather is the reference's, bitwise. + + One full-height block per rank, regenerated for every rank of the world + from the shared seeds. A row count passes the leading `rows` rows of this rank's + block, as a rank whose rows were padded to that T_max does. These are the + group's first calls, so the first of them builds the communicator, eagerly. + A mismatch is agreed across the ranks before any rank raises, so the + collective that counts it is made by all of them. + """ + for i, cell in _mpi_cells(): + spec = cell.spec + if "sweep_rows" not in spec: + continue + width = _width(spec) + rows_list = _sweep_rows(spec) + top = rows_list[0] + seed = 20000 + 13 * i + full = [_rows(r, (top, width), seed) for r in range(WORLD)] + mine = full[RANK] + bad = torch.zeros(len(rows_list), dtype=torch.bool, device="cuda") + before = dict(SEEN) + _mpi_sync() + for k, rows in enumerate(rows_list): + out = _gather(mine[:rows], spec) + assert out.shape == (WORLD * rows, width), (cell.why, rows, out.shape) + bad[k] = _differs(out, entry.allgather.reference([block[:rows] for block in full])) + torch.cuda.synchronize() + everyone = MPI_COMM.allgather(bad.nonzero().flatten().tolist()) + failing = sorted({rows_list[k] for got in everyone for k in got}) + if failing: + rows = failing[0] + out = _gather(mine[:rows], spec) + ref = entry.allgather.reference([block[:rows] for block in full]) + raise AssertionError( + f"{cell.why}: {len(failing)} of {len(rows_list)} row counts differ from the " + f"reference on some rank, the smallest {rows} here in " + f"{_fraction_moved(out, ref):.4f} of the elements" + ) + _account(i, before) + del full, mine + _mpi_sync() + + +def check_mpi_the_gate_discriminates_a_wrong_gather() -> None: + """The bitwise gate, on the sweep cells' payload, rejects every plausible wrong gather. + + At the top of the first sweep cell: rank blocks in the wrong order, rank 0's + block standing in for the last rank's, the last rank never contributing + (its block left zero), and every block taken one row late. Each has the + right shape, so only the payload can tell them apart; each moves nearly + every element of the blocks it gets wrong, which is at most 1/WORLD of the + result when it gets one block wrong. A job holding only q-split cells skips + it: the q-split's own discrimination check is in `check_mpi_split_cells`. + """ + if _only_split_cells(): + return + i, cell = next((i, c) for i, c in _mpi_cells() if "sweep_rows" in c.spec) + spec = cell.spec + width, rows = _width(spec), spec["rows"] + before = dict(SEEN) + blocks = [_rows(r, (rows + 1, width), 25000) for r in range(WORLD)] + got = [block[:rows] for block in blocks] + out = _gather(got[RANK], spec) + assert not bool(_differs(out, entry.allgather.reference(got))), "baseline differs" + one_block = 0.9 / WORLD + last = WORLD - 1 + fractions = [] + for name, wrong, floor in ( + ("rank blocks reversed", torch.cat(got[::-1]), 0.9), + (f"rank 0's block in rank {last}'s place", torch.cat(got[:-1] + got[:1]), one_block), + ( + f"rank {last} never contributed", + torch.cat(got[:-1] + [torch.zeros_like(got[-1])]), + one_block, + ), + ("every block one row late", torch.cat([block[1:] for block in blocks]), 0.9), + ): + assert wrong.shape == out.shape, (name, wrong.shape, out.shape) + fraction = _fraction_moved(out, wrong) + assert fraction > floor, f"{name}: only {fraction:.4f} of elements differ" + fractions.append(round(fraction, 4)) + _account(i, before) + everyone = MPI_COMM.allgather(fractions) + if RANK == 0: + print( + f"[mpi] wrong gathers at rows {rows} move (reversed, substituted, missing, late) " + f"{everyone[0]} of the elements on rank 0, min over ranks " + f"{[min(f[k] for f in everyone) for k in range(4)]}", + flush=True, + ) + _mpi_sync() + + +def check_mpi_bits_payload_cells() -> None: + """Every `payload="bits"` cell: the dep4 or dep8 padding skew at its row count, bitwise. + + Rank r's leading `valid_rows[r]` rows are arbitrary bf16 bit patterns and + the rest is +0.0, the zero padding the target writes up to T_max. The op + moves bytes, so the gather is the concatenation bit for bit -- NaN payloads + and -0.0 included, which a float comparison could not show. Four calls + return the same bits in four distinct tensors, none of them the input, and + the input is unchanged. + """ + for i, cell in _mpi_cells(): + spec = cell.spec + if spec.get("payload") != "bits" or _on_split(cell): + continue # the q-split's bits cell runs in `check_mpi_split_cells` + width, rows, valid = _width(spec), spec["rows"], tuple(spec["valid_rows"]) + assert len(valid) == WORLD and max(valid) == rows, (cell.why, valid) + seed = 30000 + 13 * i + blocks = [] + for r in range(WORLD): + block = torch.zeros(rows, width, dtype=torch.bfloat16, device="cuda") + block[: valid[r]] = _bits(r, (valid[r], width), seed) + blocks.append(block) + mine = blocks[RANK] + kept = mine.clone() + before = dict(SEEN) + _mpi_sync() + outs = [_gather(mine, spec) for _ in range(4)] + ref = entry.allgather.reference(blocks) + for k, out in enumerate(outs): + assert out.shape == (WORLD * rows, width) and out.is_contiguous(), (cell.why, k) + assert not bool(_differs(out, ref)), f"{cell.why}: call {k} differs from the reference" + pointers = {out.data_ptr() for out in outs} + assert len(pointers) == len(outs) and mine.data_ptr() not in pointers, cell.why + assert not bool(_differs(mine, kept)), f"{cell.why}: the input was written" + # The payload holds every class of bit pattern the cell names. + full = blocks[valid.index(rows)] + tiny = torch.finfo(torch.bfloat16).tiny + counts = [ + int(torch.isnan(full).sum()), + int(torch.isinf(full).sum()), + int(((full == 0) & torch.signbit(full)).sum()), + int(((full != 0) & (full.abs() < tiny)).sum()), + ] + assert min(counts) > 0, f"{cell.why}: (NaN, inf, -0.0, subnormal) counts {counts}" + if RANK == 0: + print( + f"[mpi] {cell.why[:44]}: (NaN, inf, -0.0, subnormal) elements in the full " + f"block {counts}, moved bitwise", + flush=True, + ) + _account(i, before) + del blocks, mine, kept, outs, ref, full + _mpi_sync() + + +def check_mpi_graph_cells() -> None: + """The graph cells together: the decode graphs a dep4 or dep8 engine holds, one pool. + + One graph per captured batch, largest first as the engine's graph runner + captures, each with `sites` gathers that read their own persistent input + and copy into their own persistent output (outside the pool, whose blocks a + later capture reuses). The group is warmed eagerly at each batch before its + capture, as the engine's warmup does. Then four replay orders, each twice: + one replay at a time on fresh payloads checked bitwise against the + reference, then every graph's payload staged and all replayed back to back + with no synchronization. Last, arbitrary bit patterns: each replay bitwise + the reference and the same calls made eagerly. A job holding only q-split + cells (context steps, never captured) has no graph and skips it. + """ + if _only_split_cells(): + return + cells = [(i, c) for i, c in _mpi_cells() if "graph_rows" in c.spec] + assert cells, "no graph cell" + spec0 = cells[0][1].spec + pool_rows, sites, width = tuple(spec0["graph_rows"]), spec0["sites"], _width(spec0) + by_rows = {c.spec["rows"]: i for i, c in cells} + assert sorted(by_rows) == sorted(pool_rows), (sorted(by_rows), pool_rows) + for _, c in cells: + assert c.spec["sites"] == sites and tuple(c.spec["graph_rows"]) == pool_rows, c.why + + inputs: Dict[int, torch.Tensor] = {} + outputs: Dict[int, torch.Tensor] = {} + graphs: Dict[int, Any] = {} + pool = None + for rows in sorted(pool_rows, reverse=True): + spec = entry.allgather.CELLS[by_rows[rows]].spec + before = dict(SEEN) + x = _rows(RANK, (sites, rows, width), 40000 + rows) + out = torch.empty(sites, WORLD * rows, width, dtype=torch.bfloat16, device="cuda") + + def body(x: torch.Tensor = x, out: torch.Tensor = out, spec: Dict[str, Any] = spec) -> None: + for s in range(sites): + out[s].copy_(_gather(x[s], spec)) + + side = torch.cuda.Stream() + side.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(side): + for _ in range(2): + body() + torch.cuda.current_stream().wait_stream(side) + _mpi_sync() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, pool=pool): + body() + pool = graph.pool() if pool is None else pool + inputs[rows], outputs[rows], graphs[rows] = x, out, graph + _account(by_rows[rows], before) + _mpi_sync() + + def payload(rank: int, rows: int, seed: int, bits: bool) -> torch.Tensor: + return (_bits if bits else _rows)(rank, (sites, rows, width), seed) + + def stage(rows: int, seed: int, bits: bool = False) -> None: + inputs[rows].copy_(payload(RANK, rows, seed, bits)) + + def verify(rows: int, seed: int, where: str, bits: bool = False) -> None: + before = dict(SEEN) + everyone = [payload(r, rows, seed, bits) for r in range(WORLD)] + ref = torch.stack( + [entry.allgather.reference([p[s] for p in everyone]) for s in range(sites)] + ) + _account(by_rows[rows], before) + if bool(_differs(outputs[rows], ref)): + bad = [s for s in range(sites) if bool(_differs(outputs[rows][s], ref[s]))] + raise AssertionError(f"{where} rows={rows}: sites {bad} differ from the reference") + + asc = sorted(pool_rows) + shuffled = list(asc) + random.Random(1234).shuffle(shuffled) + extremes = [r for pair in zip(reversed(asc), asc) for r in pair][: len(asc)] + seed = 50000 + for name, order in ( + ("ascending", asc), + ("descending", list(reversed(asc))), + ("shuffled", shuffled), + ("extremes", extremes), + ): + assert sorted(order) == asc, (name, order) + for pos, rows in enumerate(order): + seed += 1 + stage(rows, seed) + _mpi_sync() + graphs[rows].replay() + torch.cuda.synchronize() + verify(rows, seed, f"{name}/checked@{pos}") + _mpi_sync() + staged: Dict[int, int] = {} + for rows in order: + seed += 1 + staged[rows] = seed + stage(rows, seed) + _mpi_sync() + for rows in order: + graphs[rows].replay() + torch.cuda.synchronize() + for pos, rows in enumerate(order): + verify(rows, staged[rows], f"{name}/burst@{pos}") + _mpi_sync() + + # A replay that did not happen leaves the previous payload's gather; the + # gate of exactly zero sees it in every element where consecutive payloads + # differ. + stale = _fraction_moved(payload(RANK, 16, seed - 1, False), payload(RANK, 16, seed, False)) + assert stale > 0.9, f"consecutive payloads differ in only {stale:.4f} of elements" + + for rows in shuffled: + seed += 1 + stage(rows, seed, bits=True) + _mpi_sync() + graphs[rows].replay() + torch.cuda.synchronize() + verify(rows, seed, "bit patterns", bits=True) + before = dict(SEEN) + spec = entry.allgather.CELLS[by_rows[rows]].spec + eager = torch.stack([_gather(inputs[rows][s], spec) for s in range(sites)]) + _account(by_rows[rows], before) + if bool(_differs(outputs[rows], eager)): + raise AssertionError(f"bit patterns, rows={rows}: a replay differs from eager") + _mpi_sync() + graphs.clear() + _mpi_sync() + + +def check_mpi_round_trip_through_a_combine_in_one_graph() -> None: + """The dep4 or dep8 decode step's collective sequence, captured: gather, window, combine. + + Per MoE layer the target gathers every rank's rows, lets this rank's expert + window act on all of them and combines the world's partials, all on the one + communicator `getComm` caches for the group. Here each of the `sites` + layers gathers through the entry, scales by a per-rank factor standing in + for the window, and combines with `torch.ops.trtllm.reducescatter` -- a + fixture standing in for the combine, never the reference -- every batch's + graph in one pool. The gathered rows are checked against the entry's + reference, and rank i gets back its own rows times the sum of the world's + factors, computed locally, so a wrong combine cannot hide behind a right + gather or the reverse. A job holding only q-split cells has no graph and + skips it. + """ + if _only_split_cells(): + return + cells = [(i, c) for i, c in _mpi_cells() if "graph_rows" in c.spec] + spec = cells[0][1].spec + by_rows = {c.spec["rows"]: i for i, c in cells} + sites, width, pool_rows = spec["sites"], _width(spec), spec["graph_rows"] + # Payload k/8 with |k| <= 7 and factors (r + 1) / 4: every product and every + # partial sum is exact in bf16, so both gates are bitwise. At world 8 a + # partial sum is k * s / 32 with s at most 1 + 2 + ... + 8 = 36, so |k * s| + # stays at or below 252, inside bf16's exact integers. + factor = (RANK + 1) / 4.0 + total = sum((r + 1) / 4.0 for r in range(WORLD)) + pool = None + for rows in sorted(pool_rows, reverse=True): + own = torch.empty(sites, rows, width, dtype=torch.bfloat16, device="cuda") + gathered = torch.empty(sites, WORLD * rows, width, dtype=torch.bfloat16, device="cuda") + back = torch.empty(sites, rows, width, dtype=torch.bfloat16, device="cuda") + + def body() -> None: + for s in range(sites): + g = _gather(own[s], spec) + gathered[s].copy_(g) + back[s].copy_(torch.ops.trtllm.reducescatter(torch.mul(g, factor), None, GROUP)) + + def payload(rank: int, seed: int) -> torch.Tensor: + gen = torch.Generator(device="cuda").manual_seed(seed * 131 + rank) + raw = torch.randint(-7, 8, (sites, rows, width), generator=gen, device="cuda") + return raw.to(torch.bfloat16).div_(8.0) + + before = dict(SEEN) + own.copy_(payload(RANK, 70000 + rows)) + side = torch.cuda.Stream() + side.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(side): + body() + torch.cuda.current_stream().wait_stream(side) + _mpi_sync() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, pool=pool): + body() + pool = graph.pool() if pool is None else pool + for seed in (71000 + rows, 72000 + rows): + own.copy_(payload(RANK, seed)) + _mpi_sync() + graph.replay() + torch.cuda.synchronize() + everyone = [payload(r, seed) for r in range(WORLD)] + ref = torch.stack( + [entry.allgather.reference([p[s] for p in everyone]) for s in range(sites)] + ) + assert not bool(_differs(gathered, ref)), f"rows={rows}: the gather is wrong" + expected = (own.float() * total).to(torch.bfloat16) + assert not bool(_differs(back, expected)), f"rows={rows}: the round trip is wrong" + _mpi_sync() + _account(by_rows[rows], before) + del graph + _mpi_sync() + + +# ---- the q-split selection gather (the cells stating `q_split`) --------------- +# +# Mistral Large 4 NVFP4 tep4 and tp4 (four ranks of one node) and tep8 (eight ranks +# over two nodes) under chunked prefill: once a step's tc context rows reach the +# engine's q_split_threshold, TP rank r of WORLD selects for rows +# [q_begin, q_end) = [tc * r // WORLD, tc * (r + 1) // WORLD) only, and every +# selecting indexer layer gathers the shares -- `topk_buf[q_begin:q_end]`, sizes +# = q_sizes -- into rows [0, tc) of the step's selection buffer, over the nodes +# the ranks span. A selection row holds key positions, so the payloads are int32 +# in [-1, 2^20) unless a cell asks for every bit pattern (`bits`) or for rows +# shaped as the top-k writes them (`indices`). Each job drives the q-split cells +# of its own world size: `_q_split` and every share below follow WORLD. + +#: The engine's default q_split_threshold: the smallest tc it splits. +Q_SPLIT_THRESHOLD = 8192 +#: A value no selection holds (positions are >= -1, salts stay below 2^30): a +#: row still holding it was never written. +SPLIT_SENTINEL = -(2**31) + + +def _on_split(cell: Any) -> bool: + """Whether `cell` is one of the q-split gather's: it states `q_split`.""" + return bool(cell.spec.get("q_split")) + + +def _split_cells() -> List[Tuple[int, Any]]: + """(index, cell) of every q-split cell this job drives, all on the MPI-session symbol.""" + return [(i, c) for i, c in _mpi_cells() if _on_split(c)] + + +def _only_split_cells() -> bool: + """Whether this job's cells are all the q-split's (then it has no gather sweep or graph).""" + return all(_on_split(c) for _, c in _mpi_cells()) + + +def _q_split(tc: int, rank: int) -> Tuple[int, int, List[int]]: + """`rank`'s (q_begin, q_end) of `tc` context rows and the q_sizes, as the engine splits.""" + sizes = [(r + 1) * tc // WORLD - r * tc // WORLD for r in range(WORLD)] + return tc * rank // WORLD, tc * (rank + 1) // WORLD, sizes + + +def _split_width(spec: Dict[str, Any]) -> int: + assert len(spec["trailing"]) == 1 and spec["dtype"] is torch.int32, spec + assert spec["world_size"] == WORLD and spec["process_group"] is None, (spec, WORLD) + # A cell without `nodes` is one node's, as the tep4 and tp4 cells are. + assert spec.get("nodes", 1) == NODES, (spec, NODES) + return spec["trailing"][0] + + +def _split_tokens(spec: Dict[str, Any]) -> List[int]: + """The tc values a q-split cell names, largest first, checked against its keys. + + A sweep names its totals as `sweep_tokens` or `sweep_totals`; its sizes are + the floor split at each (`sizes="floor_split"`, or no `sizes`). + """ + if "sweep_tokens" in spec or "sweep_totals" in spec: + lo, hi = spec["sweep_tokens"] if "sweep_tokens" in spec else spec["sweep_totals"] + tcs = set(range(lo, hi + 1, spec["sweep_step"])) | set(spec.get("sweep_also", ())) | {hi} + assert all(lo <= tc <= hi for tc in tcs), spec + assert spec.get("sizes", "floor_split") == "floor_split", spec + else: + tcs = {spec["tokens"]} + assert list(spec["sizes"]) == _q_split(spec["tokens"], RANK)[2], spec + assert min(tcs) >= Q_SPLIT_THRESHOLD, spec + assert spec["rows"] == max(max(_q_split(tc, RANK)[2]) for tc in tcs), spec + return sorted(tcs, reverse=True) + + +def _selections(tc: int, seed: int, width: int, bits: bool = False) -> torch.Tensor: + """A step's [tc, width] selection rows, the same on every rank for one seed. + + Rank r's input is its rows [q_begin, q_end) of these, so the gather is + supposed to give back exactly this tensor. Key positions in [-1, 2^20): two + rows agree in about one element in a million, so a block from the wrong rank + or at the wrong offset changes nearly every element it covers. With `bits`, + every int32 bit pattern equally likely. + """ + gen = torch.Generator(device="cuda").manual_seed(seed * 7919 + 3) + if bits: + shape = (tc, width) + raw = torch.randint(-(2**31), 2**31, shape, generator=gen, device="cuda", dtype=torch.int64) + return raw.to(torch.int32) + return torch.randint(-1, 2**20, (tc, width), generator=gen, device="cuda", dtype=torch.int32) + + +def _shares(rows: torch.Tensor, tc: int) -> List[torch.Tensor]: + """Every rank's share of a step's `tc` selection rows, in rank order.""" + return [rows[_q_split(tc, r)[0] : _q_split(tc, r)[1]] for r in range(WORLD)] + + +def _salted(shares: List[torch.Tensor], salt: int) -> List[torch.Tensor]: + return [torch.bitwise_xor(share, salt) for share in shares] + + +def _words_differ(a: torch.Tensor, b: torch.Tensor) -> torch.Tensor: + """Whether two int32 tensors differ in any bit: a device bool, compared as int64 words.""" + assert a.shape == b.shape and a.dtype == b.dtype == torch.int32, (a.shape, b.shape, a.dtype) + return torch.ne(a.reshape(-1).view(torch.int64), b.reshape(-1).view(torch.int64)).any() + + +def _int_fraction_moved(a: torch.Tensor, b: torch.Tensor) -> float: + """The fraction of elements that differ between two int32 tensors.""" + return torch.ne(a, b).float().mean().item() + + +def _split_gather(x: torch.Tensor, sizes: List[int]) -> torch.Tensor: + """The call the tep4, tp4 and tep8 deployments make: the share, q_sizes, the TP group, MPI.""" + return entry.allgather(x, sizes, GROUP, process_group=None) + + +def check_mpi_split_sizes_are_the_engines() -> None: + """Every q-split cell's tc splits as the engine splits it, by the engine's own function. + + `_split_prefill_queries` (the DSA indexer's) at the default threshold, on + this rank's Mappings at tp WORLD, each with attention DP off and the job's + GROUP as its TP group: the default one and the one with the MoE + tensor-parallel over the same ranks (tep4's and tp4's at four ranks), and + the one with moe_ep over every rank and moe_tp 1 (tep8's at eight ranks, as + the tep8 deployment builds it). Unsplit below 8192, and at every tc a cell names + the (q_begin, q_end, q_sizes) `_q_split` gives. So the cells' shares are the + targets', and a change to the engine's split fails here first. + """ + if not _split_cells(): + return + from tensorrt_llm._torch.attention.backends.sparse.dsa.indexer import _split_prefill_queries + from tensorrt_llm.mapping import Mapping + + checked = 0 + for mapping in ( + Mapping(world_size=WORLD, rank=RANK, tp_size=WORLD), + Mapping(world_size=WORLD, rank=RANK, tp_size=WORLD, moe_tp_size=WORLD, moe_ep_size=1), + Mapping(world_size=WORLD, rank=RANK, tp_size=WORLD, moe_ep_size=WORLD, moe_tp_size=1), + ): + assert mapping.tp_rank == RANK and not mapping.enable_attention_dp, mapping.tp_rank + assert list(mapping.tp_group) == GROUP, (list(mapping.tp_group), GROUP) + for tc in (1, Q_SPLIT_THRESHOLD - 1): + assert _split_prefill_queries(tc, mapping, Q_SPLIT_THRESHOLD) == (0, tc, None), tc + for _, cell in _split_cells(): + for tc in _split_tokens(cell.spec): + got = _split_prefill_queries(tc, mapping, Q_SPLIT_THRESHOLD) + assert tuple(got) == _q_split(tc, RANK), (cell.why[:44], tc, got) + checked += 1 + assert checked > 0, "no q-split cell" + _mpi_sync() + + +def check_mpi_split_cells() -> None: + """Every q-split cell at one tc: the target's call on its share, bitwise the reference. + + Each rank stages its share of a step's selection rows at [q_begin, q_end) of + a sentinel-filled [tc, width] buffer and passes that view, as the target + passes `topk_buf[q_begin:q_end]`. Three calls on three payloads (a `bits` + cell: four calls on one payload of every int32 bit pattern), each the + reference's concatenation of the shares bit for bit, in a fresh tensor + outside the buffer, with the buffer left as staged. + """ + for i, cell in _split_cells(): + spec = cell.spec + if "tokens" not in spec or "sites" in spec: + continue + width = _split_width(spec) + (tc,) = _split_tokens(spec) + begin, end, sizes = _q_split(tc, RANK) + bits = spec.get("payload") == "bits" + seeds = [60000 + 13 * i] * 4 if bits else [60000 + 13 * i + k for k in range(3)] + before = dict(SEEN) + outs = [] + rows = None + for k, seed in enumerate(seeds): + prev, rows = rows, _selections(tc, seed, width, bits) + buf = torch.full((tc, width), SPLIT_SENTINEL, dtype=torch.int32, device="cuda") + buf[begin:end] = rows[begin:end] + kept = buf.clone() + _mpi_sync() + out = _split_gather(buf[begin:end], sizes) + where = f"{cell.why[:44]}: call {k}" + assert out.shape == (tc, width) and out.is_contiguous(), (where, out.shape) + ref = entry.allgather.reference(_shares(rows, tc)) + assert not bool(_words_differ(out, ref)), f"{where} differs from the reference" + assert not bool(_words_differ(buf, kept)), f"{where} wrote its input buffer" + span = (buf.data_ptr(), buf.data_ptr() + buf.numel() * buf.element_size()) + assert not span[0] <= out.data_ptr() < span[1], f"{where}: the result is in the buffer" + outs.append(out) + assert len({out.data_ptr() for out in outs}) == len(outs), cell.why[:44] + if bits: + assert bool(((rows < -1) | (rows >= 2**20)).any()), ( + "the bits payload holds no wide value" + ) + if tc == Q_SPLIT_THRESHOLD + 1: + _split_gate_discriminates(rows, prev, tc, out) + _account(i, before) + del outs, rows, prev, buf, kept + _mpi_sync() + + +def _split_gate_discriminates( + rows: torch.Tensor, stale: torch.Tensor, tc: int, out: torch.Tensor +) -> None: + """The bitwise gate, on a ragged split's payload, rejects every plausible wrong gather. + + Shares in reversed rank order, the last rank's share never arriving (left + zero), every share one row late, and the previous call's rows (a call that + did not happen). Each has the right shape; each moves nearly every element + of the blocks it gets wrong, at most 1/WORLD of the result when it gets one + block wrong. + """ + shares = _shares(rows, tc) + late = [] + for r in range(WORLD): + begin, end, _ = _q_split(tc, r) + late.append(torch.cat([rows[begin + 1 : end], rows[end % tc : end % tc + 1]])) + missing = shares[:-1] + [torch.zeros_like(shares[-1])] + fractions = [] + for name, wrong, floor in ( + ("shares in reversed rank order", torch.cat(shares[::-1]), 0.9), + (f"rank {WORLD - 1}'s share never arrived", torch.cat(missing), 0.9 / WORLD), + ("every share one row late", torch.cat(late), 0.9), + ("the previous call's rows", stale, 0.9), + ): + assert wrong.shape == out.shape, (name, wrong.shape, out.shape) + fraction = _int_fraction_moved(out, wrong) + assert fraction > floor, f"{name}: only {fraction:.4f} of elements differ" + fractions.append(round(fraction, 4)) + everyone = MPI_COMM.allgather(fractions) + if RANK == 0: + print( + f"[mpi] q-split gate at tc {tc}: wrong gathers move (reversed, missing, late, " + f"stale) {everyone[0]} of the elements on rank 0, min over ranks " + f"{[min(f[k] for f in everyone) for k in range(4)]}", + flush=True, + ) + + +def check_mpi_split_sweep_cells() -> None: + """Every q-split sweep cell: at each tc it names, the target's call, bitwise the reference. + + One [top, width] payload regenerated on every rank, and at each tc a salt + XORed into it -- a different one per call, below 2^30 so no selection becomes + the sentinel -- so a call that did not happen, leaving a block the caching + allocator hands back with an earlier call's rows, differs in every element. + Rank r stages its share of the salted rows at [q_begin, q_end) of a buffer + and passes that view. Verdicts stay on the device and are read once; a + mismatch is agreed across the ranks before any rank raises. + """ + for i, cell in _split_cells(): + spec = cell.spec + if "sweep_tokens" not in spec: + continue + width = _split_width(spec) + tcs = _split_tokens(spec) + top = tcs[0] + base = _selections(top, 61000 + 13 * i, width) + buf = torch.empty(top, width, dtype=torch.int32, device="cuda") + bad = torch.zeros(len(tcs), dtype=torch.bool, device="cuda") + before = dict(SEEN) + _mpi_sync() + for k, tc in enumerate(tcs): + begin, end, sizes = _q_split(tc, RANK) + salt = (k + 1) * 40503 % 2**30 + torch.bitwise_xor(base[begin:end], salt, out=buf[begin:end]) + out = _split_gather(buf[begin:end], sizes) + assert out.shape == (tc, width), (cell.why[:44], tc, out.shape) + ref = entry.allgather.reference(_salted(_shares(base[:tc], tc), salt)) + bad[k] = _words_differ(out, ref) + torch.cuda.synchronize() + everyone = MPI_COMM.allgather(bad.nonzero().flatten().tolist()) + failing = sorted({tcs[k] for got in everyone for k in got}) + if failing: + raise AssertionError( + f"{cell.why[:44]}: {len(failing)} of {len(tcs)} tc differ from the reference " + f"on some rank, the smallest {failing[0]}" + ) + if RANK == 0: + print( + f"[mpi] q-split sweep: {len(tcs)} tc {tcs[-1]}..{tcs[0]}, each bitwise", flush=True + ) + _account(i, before) + del base, buf, bad + _mpi_sync() + + +def check_mpi_split_step_cells() -> None: + """Every q-split step cell: `sites` gathers in layer order, as the target issues them. + + One selection buffer per rank, as the target keeps one per step. Per layer + its rows [0, tc) are set to the sentinel, the rank's share of that layer's + selections is written at [q_begin, q_end), and the view is gathered with the + result copied into rows [0, tc) -- the target's `topk_buf[:tc].copy_( + allgather(topk_buf[q_begin:q_end], q_sizes, ...))` -- then the rows a layer + reads are kept. Nothing synchronizes between layers. Afterwards every layer's + rows are checked against the reference bitwise, and for the sentinel: a row + the gather did not write would still hold it. + """ + for i, cell in _split_cells(): + spec = cell.spec + if "sites" not in spec or spec.get("payload") == "indices": + continue # an `indices` step runs in `check_mpi_split_copy_back_cells` + width, sites = _split_width(spec), spec["sites"] + (tc,) = _split_tokens(spec) + begin, end, sizes = _q_split(tc, RANK) + base = _selections(tc, 62000 + 13 * i, width) + salts = [(997 * i + layer + 1) * 40503 % 2**30 for layer in range(sites)] + topk = torch.empty(tc, width, dtype=torch.int32, device="cuda") + read = torch.empty(sites, tc, width, dtype=torch.int32, device="cuda") + before = dict(SEEN) + _mpi_sync() + for layer, salt in enumerate(salts): + topk.fill_(SPLIT_SENTINEL) + torch.bitwise_xor(base[begin:end], salt, out=topk[begin:end]) + topk[:tc].copy_(_split_gather(topk[begin:end], sizes)) + read[layer].copy_(topk) + bad = torch.zeros(sites, dtype=torch.bool, device="cuda") + holes = torch.zeros(sites, dtype=torch.bool, device="cuda") + for layer, salt in enumerate(salts): + ref = entry.allgather.reference(_salted(_shares(base, tc), salt)) + bad[layer] = _words_differ(read[layer], ref) + holes[layer] = torch.eq(read[layer], SPLIT_SENTINEL).any() + torch.cuda.synchronize() + everyone = MPI_COMM.allgather( + (bad.nonzero().flatten().tolist(), holes.nonzero().flatten().tolist()) + ) + wrong = sorted({layer for got, _ in everyone for layer in got}) + unwritten = sorted({layer for _, got in everyone for layer in got}) + if wrong or unwritten: + raise AssertionError( + f"{cell.why[:44]}: layers {wrong} differ from the reference and layers " + f"{unwritten} hold unwritten rows, on some rank" + ) + _account(i, before) + del read, topk, base, bad, holes + _mpi_sync() + + +def _indices(tc: int, seed: int, width: int, top: int) -> torch.Tensor: + """A step's [tc, width] selection rows as the top-k leaves them, the same on every rank. + + Row j holds key positions in [0, top] in its first `seen` slots and -1 in + the rest: a query that sees fewer than `width` keys -- early in a request, + or in a chunk with no key at all -- leaves a -1 tail. Every third row sees + `width` keys or more and holds both 0 and `top`, every 97th from row 1 sees + none, and the others see 0..width keys, uniformly. Two rows' positions agree + in about one slot in a million, so a block from the wrong rank or at the + wrong offset changes every slot where either holds a position. + """ + gen = torch.Generator(device="cuda").manual_seed(seed * 7919 + 5) + rows = torch.randint(0, top + 1, (tc, width), generator=gen, device="cuda", dtype=torch.int32) + seen = torch.randint(0, width + 1, (tc, 1), generator=gen, device="cuda", dtype=torch.int32) + seen[::3] = width + seen[1::97] = 0 + rows[::3, 0] = 0 + rows[::3, -1] = top + slots = torch.arange(width, device="cuda", dtype=torch.int32) + return rows.masked_fill_(slots >= seen, -1) + + +def _indices_payload_and_gate(rows: torch.Tensor, other: torch.Tensor, tc: int, top: int) -> None: + """An `indices` payload holds what it claims, and the bitwise gate sees a wrong gather on it. + + The payload: positions 0 and `top` present, nothing outside [-1, top], rows + with -1 tails, rows of -1 only and rows without a tail. The gate: shares in + reversed rank order, every share one row late, another call's rows at this + tc, and the last rank's share never arriving (left zero) each move more than + half the slots of the blocks they get wrong (1/WORLD of the result for one + block); the rows that agree are the -1 tails both sides share. + """ + assert int(rows.min()) == -1 and int(rows.max()) == top, (int(rows.min()), int(rows.max())) + assert bool((rows == 0).any()), "no position 0" + tails = rows[:, -1] == -1 + empty = rows[:, 0] == -1 + counts = [int((~tails).sum()), int((tails & ~empty).sum()), int(empty.sum())] + assert min(counts) > 0, f"(full, tailed, empty) rows {counts}" + shares = _shares(rows, tc) + late = [] + for r in range(WORLD): + begin, end, _ = _q_split(tc, r) + late.append(torch.cat([rows[begin + 1 : end], rows[end % tc : end % tc + 1]])) + fractions = [] + for name, wrong, floor in ( + ("shares in reversed rank order", torch.cat(shares[::-1]), 0.5), + ("every share one row late", torch.cat(late), 0.5), + ("another call's rows", other, 0.5), + ( + f"rank {WORLD - 1}'s share never arrived", + torch.cat(shares[:-1] + [torch.zeros_like(shares[-1])]), + 0.5 / WORLD, + ), + ): + assert wrong.shape == rows.shape, (name, wrong.shape, rows.shape) + fraction = _int_fraction_moved(rows, wrong) + assert fraction > floor, f"{name}: only {fraction:.4f} of slots differ" + fractions.append(round(fraction, 4)) + everyone = MPI_COMM.allgather(fractions) + if RANK == 0: + print( + f"[mpi] indices payload at tc {tc}: (full, tailed, empty) rows {counts}, positions " + f"-1..{top}; wrong gathers move (reversed, late, other call, missing) " + f"{everyone[0]} of the slots on rank 0, min over ranks " + f"{[min(f[k] for f in everyone) for k in range(4)]}", + flush=True, + ) + + +def check_mpi_split_copy_back_cells() -> None: + """Every `payload="indices"` q-split cell, through the target's copy-back. + + The tp4 deployment's own form of the call, on rows shaped as its top-k leaves + them (`_indices`). One selection buffer per rank, as the target keeps one + per step: per call its rows [0, tc) are set to the sentinel, the rank's + share of a fresh payload is written at [q_begin, q_end), and the view is + gathered with the result copied into rows [0, tc) -- `topk_buf[:tc].copy_( + allgather(topk_buf[q_begin:q_end], q_sizes, ...))`. A `sites` cell makes + that many calls at its tc in layer order with nothing synchronizing between + them, keeps the rows each layer reads and checks them afterwards; a sweep + cell makes one call per tc, largest first, each checked as it goes. Every + call's rows must be bitwise the reference's concatenation of the shares and + hold no sentinel. Verdicts stay on the device and are read once; a mismatch + is agreed across the ranks before any rank raises. + """ + for i, cell in _split_cells(): + spec = cell.spec + if spec.get("payload") != "indices": + continue + started = time.monotonic() + torch.cuda.reset_peak_memory_stats() + width, top = _split_width(spec), spec["top_position"] + tcs = _split_tokens(spec) + sites = spec.get("sites", 1) + calls = [(tc, layer) for tc in tcs for layer in range(sites)] + seed = (64000 + 13 * i) * 4096 + topk = torch.empty(tcs[0], width, dtype=torch.int32, device="cuda") + read = None + if sites > 1: + read = torch.empty(sites, tcs[0], width, dtype=torch.int32, device="cuda") + bad = torch.zeros(len(calls), dtype=torch.bool, device="cuda") + holes = torch.zeros(len(calls), dtype=torch.bool, device="cuda") + _indices_payload_and_gate( + _indices(tcs[0], seed, width, top), _indices(tcs[0], seed - 1, width, top), tcs[0], top + ) + # The control: with the gather withheld, the other ranks' rows still hold the sentinel. + begin, end, _ = _q_split(tcs[0], RANK) + topk.fill_(SPLIT_SENTINEL) + topk[begin:end].copy_(_indices(tcs[0], seed - 1, width, top)[begin:end]) + assert bool(torch.eq(topk, SPLIT_SENTINEL).any()), "a withheld gather left no sentinel" + before = dict(SEEN) + _mpi_sync() + for k, (tc, layer) in enumerate(calls): + begin, end, sizes = _q_split(tc, RANK) + rows = _indices(tc, seed + k, width, top) + topk[:tc].fill_(SPLIT_SENTINEL) + topk[begin:end].copy_(rows[begin:end]) + topk[:tc].copy_(_split_gather(topk[begin:end], sizes)) + if read is not None: + read[layer, :tc].copy_(topk[:tc]) + continue + bad[k] = _words_differ(topk[:tc], entry.allgather.reference(_shares(rows, tc))) + holes[k] = torch.eq(topk[:tc], SPLIT_SENTINEL).any() + if read is not None: + for k, (tc, layer) in enumerate(calls): + ref = entry.allgather.reference(_shares(_indices(tc, seed + k, width, top), tc)) + bad[k] = _words_differ(read[layer, :tc], ref) + holes[k] = torch.eq(read[layer, :tc], SPLIT_SENTINEL).any() + torch.cuda.synchronize() + everyone = MPI_COMM.allgather( + (bad.nonzero().flatten().tolist(), holes.nonzero().flatten().tolist()) + ) + wrong = sorted({calls[k] for got, _ in everyone for k in got}) + unwritten = sorted({calls[k] for _, got in everyone for k in got}) + if wrong or unwritten: + raise AssertionError( + f"{cell.why[:44]}: {len(wrong)} of {len(calls)} (tc, layer) calls differ from " + f"the reference, the first {wrong[:4]}, and {len(unwritten)} hold unwritten " + f"rows, the first {unwritten[:4]}, on some rank" + ) + if RANK == 0: + print( + f"[mpi] {cell.why[:44]}: {len(calls)} copy-back calls at tc " + f"{tcs[-1]}..{tcs[0]}, each bitwise, none unwritten; " + f"{time.monotonic() - started:.2f}s, peak allocated " + f"{torch.cuda.max_memory_allocated() / 2**30:.2f} GiB", + flush=True, + ) + _account(i, before) + del topk, read, bad, holes + _mpi_sync() + + +def check_mpi_split_guards_a_share_that_is_not_its_sizes_entry() -> None: + """`sizes[i]` must be the i-th rank's input height, and the op never checks it. + + Run in a job that holds q-split cells, on its world. With `per` = 8192 // + WORLD rows (2048 at four ranks, 1024 at eight), unguarded -- the op itself, + at tc 8193's q_sizes [per, ..., per, per + 1] ([2048, 2048, 2048, 2049] at + four ranks): the last rank's input one row short of its entry, a leading + view of a buffer whose next row is a marker row, gives a result whose last + row is the marker, a row no rank passed; rank 0's input one row long, its + share then the marker, gives the shares' concatenation with the marker + nowhere in it. With every entry per + 1 and every input `per` rows (equal + entries: one ncclAllGather of `input.numel()`), the blocks land at + `per`-row offsets rather than where `sizes` puts them, and the result's + last WORLD rows are never written (the caching allocator's previous block + there, sentinel-filled, shows through when it is reused). Every rank gets + the same wrong result, and nothing raises. + + Guarded: with every rank's input one row short of its entry, and with every + entry per + 1 over `per`-row inputs, the entry's `is_valid` refuses the call + on every rank, so no rank enters the collective. Each refusal is first asked + of `is_valid` alone and agreed across the ranks, so a rank it admitted + cannot sit in the collective alone. + """ + if not _split_cells(): + return + width, tc = 2048, Q_SPLIT_THRESHOLD + 1 + per, last = Q_SPLIT_THRESHOLD // WORLD, WORLD - 1 + rows = _selections(tc, 63000, width) + begin, end, sizes = _q_split(tc, RANK) + assert sizes == [per] * last + [per + 1], sizes + marker = torch.full((1, width), 0x5A5A5A5A, dtype=torch.int32, device="cuda") + op = torch.ops.trtllm.allgather + + # One row short on the last rank: the op reads the row after its input. + share = rows[begin : end - 1] if RANK == last else rows[begin:end] + x = torch.cat([share, marker])[: share.shape[0]] + _mpi_sync() + out = op(x, sizes, GROUP) + torch.cuda.synchronize() + assert out.shape == (tc, width), out.shape + assert not bool(_words_differ(out[: tc - 1], rows[: tc - 1])), "short: the passed rows moved" + assert bool((out[tc - 1] == marker[0]).all()), "short: the result's last row is not the marker" + + # One row long on rank 0: its last row is dropped. + x = torch.cat([rows[begin:end], marker]) if RANK == 0 else rows[begin:end].clone() + _mpi_sync() + out = op(x, sizes, GROUP) + torch.cuda.synchronize() + assert not bool(_words_differ(out, rows)), "long: the result is not the shares" + assert not bool((out == marker[0]).all(dim=1).any()), "long: the extra row arrived" + + # Every entry per + 1, every input `per` rows: one ncclAllGather of input.numel(). + equal = [per + 1] * WORLD + x = rows[per * RANK : per * RANK + per].clone() + bait = torch.full((sum(equal), width), SPLIT_SENTINEL, dtype=torch.int32, device="cuda") + bait_at = bait.data_ptr() + del bait + _mpi_sync() + out = op(x, equal, GROUP) + torch.cuda.synchronize() + assert out.shape == (sum(equal), width), out.shape + packed = WORLD * per + assert not bool(_words_differ(out[:packed], rows[:packed])), "equal: not packed" + assert bool(_words_differ(out[per + 1 : 2 * per + 1], rows[per : 2 * per])), ( + "equal: rank 1 placed" + ) + reused = out.data_ptr() == bait_at + if reused: + assert bool((out[packed:] == SPLIT_SENTINEL).all()), "equal: the tail was written" + if RANK == 0: + print( + f"[mpi] q-split heights at world {WORLD}: short -> the row after the input, long -> " + f"cut, equal entries -> packed at {per}-row offsets (tail unwritten, sentinel block " + f"reused: {reused})", + flush=True, + ) + _mpi_sync() + + for name, x, sizes_ in ( + ("every input one row short", rows[begin : end - 1].clone(), sizes), + ( + f"every entry {per + 1} over {per}-row inputs", + rows[per * RANK : per * RANK + per].clone(), + equal, + ), + ): + try: + entry.allgather.is_valid(x, sizes_, GROUP, process_group=None) + except AssertionError: + refused = True + else: + refused = False + everyone = MPI_COMM.allgather(refused) + assert all(everyone), f"{name}: is_valid admitted it on ranks {everyone}" + try: + _split_gather(x, sizes_) + except AssertionError: + pass + else: + raise AssertionError(f"{name}: the entry made the call") + _mpi_sync() + + +def check_mpi_every_cell_was_driven() -> None: + """Each MPI-session cell ran through the entry's own `__call__` and `reference`.""" + for i, cell in _mpi_cells(): + calls, refs = DRIVEN.get(i, [0, 0]) + assert calls > 0 and refs > 0, ( + f"cell {cell.why!r}: the entry's __call__ / reference ran {calls} / {refs} times" + ) + + +MPI_CHECKS = ( + check_mpi_session_is_the_launchers_world, + # Before any capture: the group's first call builds the communicator. + check_mpi_eager_sweep_cells, + check_mpi_the_gate_discriminates_a_wrong_gather, + check_mpi_bits_payload_cells, + # The q-split gather's cells (tep4 and tp4 at four ranks, tep8 at eight): + # context steps, so eager only. + check_mpi_split_sizes_are_the_engines, + check_mpi_split_cells, + check_mpi_split_sweep_cells, + check_mpi_split_step_cells, + check_mpi_split_copy_back_cells, + check_mpi_split_guards_a_share_that_is_not_its_sizes_entry, + check_mpi_graph_cells, + check_mpi_round_trip_through_a_combine_in_one_graph, + check_mpi_every_cell_was_driven, +) + + +def _run_one_mpi_rank() -> int: + """Body of one mpirun rank: run every MPI check; a failing rank exits nonzero + without finalizing, and mpirun then ends its siblings and fails. A rank + still running at `MPI_DEADLINE_S` -- wedged in a collective a sibling never + joined -- dumps its stack and exits.""" + global RANK, WORLD, NODES, GROUP, MPI_COMM, entry + faulthandler.dump_traceback_later(MPI_DEADLINE_S, exit=True) + rank = int(os.environ["OMPI_COMM_WORLD_RANK"]) + world = int(os.environ["OMPI_COMM_WORLD_SIZE"]) + torch.cuda.set_device(int(os.environ["OMPI_COMM_WORLD_LOCAL_RANK"])) + + from mpi4py import MPI + + from tensorrt_llm._torch.modeling_v2.catalog.comm import allgather as module + from tensorrt_llm._utils import mpi_comm + + entry = module + MPI_COMM = mpi_comm() + RANK, WORLD = rank, world + NODES = len(set(MPI_COMM.allgather(socket.gethostname()))) + GROUP = list(range(WORLD)) + assert WORLD >= 2, f"a collective needs at least 2 ranks, got {WORLD}" + _count_entry_calls() + + for check in MPI_CHECKS: + started = time.monotonic() + try: + check() + except BaseException: + import traceback + + print(f"[mpi rank {RANK}] FAILED {check.__name__}", flush=True) + traceback.print_exc() + sys.stdout.flush() + sys.stderr.flush() + os._exit(13) + if RANK == 0: + print(f"[mpi] {check.__name__}: {time.monotonic() - started:.1f}s", flush=True) + _mpi_sync() + print( + f"[mpi rank {RANK}] {len(MPI_CHECKS)} checks passed over {len(_mpi_cells())} cells at " + f"world {WORLD} on {NODES} node(s); peak allocated " + f"{torch.cuda.max_memory_allocated() / 2**30:.2f} GiB, reserved " + f"{torch.cuda.max_memory_reserved() / 2**30:.2f} GiB", + flush=True, + ) + sys.stdout.flush() + faulthandler.cancel_dump_traceback_later() + # mpirun fails a rank that exits without finalizing. The communicators are + # left as they are, as in the `_pg` job: these processes end here. + MPI.Finalize() + os._exit(0) + + +def _session_members(sid: int) -> List[int]: + """Every process in session `sid`, read from /proc.""" + members = [] + for entry_dir in Path("/proc").iterdir(): + if not entry_dir.name.isdigit(): + continue + try: + stat = (entry_dir / "stat").read_text() + except OSError: + continue + # pid (comm) state ppid pgrp session ...; comm may hold spaces. + fields = stat[stat.rindex(")") + 2 :].split() + if int(fields[3]) == sid: + members.append(int(entry_dir.name)) + return members + + +def _end_mpi_job(job: Any) -> None: + """End mpirun and every rank it started. + + mpirun runs as a session leader; its ranks stay in that session but each in + a process group of its own, so killing mpirun's group would leave them + running. mpirun is asked first, then the session's members are killed. + """ + if job.poll() is None: + job.terminate() + try: + job.wait(timeout=30) + except subprocess.TimeoutExpired: + pass + for pid in _session_members(job.pid): + try: + os.kill(pid, signal.SIGKILL) + except ProcessLookupError: + pass + if job.poll() is None: + job.kill() + job.wait() + + +def _mpirun_ranks(world_size: int) -> None: + """Start the MPI-session job under mpirun, one rank per device, and wait for it. + + The environment loses the launching allocation's PMI/PMIx/Open MPI/Slurm + variables, so this mpirun starts its own ranks on this node instead of + joining the allocation's step (where every rank would read rank 0 of one), + and `--oversubscribe` lets it place them whatever slot count it detects. + """ + mpirun = shutil.which("mpirun") + assert mpirun is not None, "the MPI-session cells start their ranks with mpirun; none on PATH" + env = { + k: v + for k, v in os.environ.items() + if not k.startswith(("PMIX_", "PMI_", "OMPI_", "SLURM_")) and k != "TLLM_DISABLE_MPI" + } + command = [mpirun, "-n", str(world_size), "--oversubscribe", "--allow-run-as-root"] + command += [sys.executable, str(Path(__file__).resolve()), _MPI_WORKER_FLAG] + print(f"[launcher] starting {world_size} MPI-session ranks under mpirun", flush=True) + job = subprocess.Popen(command, env=env, start_new_session=True) + try: + code = job.wait(timeout=MPI_DEADLINE_S) + except subprocess.TimeoutExpired: + _end_mpi_job(job) + raise AssertionError( + f"the {world_size}-rank MPI-session run did not finish in {MPI_DEADLINE_S}s (wedged)" + ) from None + if code != 0: + _end_mpi_job(job) + raise AssertionError(f"the {world_size}-rank MPI-session run failed: mpirun exited {code}") + + def _spawn_ranks() -> None: - """Start one worker process per claimed device; no external launcher. + """Run the `_pg` job, one worker process per claimed device and no external + launcher, then the MPI-session job under mpirun. - The ranks rendezvous through torch.distributed's env:// method, so the only - thing this needs is a free port. Each child gets its own session; on the + The `_pg` ranks rendezvous through torch.distributed's env:// method, so the + only thing they need is a free port. Each child gets its own session; on the first nonzero exit or on the deadline, every session is killed, which is what keeps one wedged collective from outliving the run. """ @@ -1068,12 +2365,12 @@ def _kill_all() -> None: for p in children: p.wait() - deadline = time.monotonic() + DEADLINE_S + deadline = time.monotonic() + PG_DEADLINE_S live = set(range(world_size)) failure = None while live and failure is None: if time.monotonic() > deadline: - failure = f"did not finish in {DEADLINE_S}s (wedged)" + failure = f"did not finish in {PG_DEADLINE_S}s (wedged)" break for r in sorted(live): code = children[r].poll() @@ -1087,6 +2384,7 @@ def _kill_all() -> None: if failure is not None: _kill_all() raise AssertionError(f"the {world_size}-rank run failed: {failure}") + _mpirun_ranks(world_size) if __name__ == "__main__": @@ -1096,5 +2394,7 @@ def _kill_all() -> None: import _rank_job sys.exit(_run_one_rank(device=_rank_job.adopt_srun_env())) + if _MPI_WORKER_FLAG in sys.argv: + sys.exit(_run_one_mpi_rank()) _spawn_ranks() print("OK") diff --git a/tests/unittest/_torch/modeling_v2/comm/_reducescatter_op_matrix.py b/tests/unittest/_torch/modeling_v2/comm/_reducescatter_op_matrix.py index 6805eb9f5290..34bf21ec189d 100644 --- a/tests/unittest/_torch/modeling_v2/comm/_reducescatter_op_matrix.py +++ b/tests/unittest/_torch/modeling_v2/comm/_reducescatter_op_matrix.py @@ -3,45 +3,61 @@ """GPU certification matrix for the reducescatter catalog entry. A collective cannot be exercised in one process, so this script is its own -launcher: run it plainly and it spawns one worker process per device, with one -rank per device named in CUDA_VISIBLE_DEVICES, under a deadline the parent -enforces by killing the whole process group. A wedged collective hangs -rather than raising — and this op wedges rather than raising when the ranks -disagree about the split — so the deadline is what keeps a broken kernel -from taking the calling run down with it. +launcher: run it plainly and it starts one rank per device named in +CUDA_VISIBLE_DEVICES, under a deadline the parent enforces by killing the +ranks. A wedged collective hangs rather than raising -- and this op wedges +rather than raising when the ranks disagree about the split -- so the +deadline is what keeps a broken kernel from taking the calling run down with +it. CUDA_VISIBLE_DEVICES=0,1,2,3 python _reducescatter_op_matrix.py -That runs two jobs, in this order. The first is the matrix proper: every -`CHECKS` entry on `world_size` ranks, and it must exit 0. The second is a -four-rank sub-job of its own that deliberately mispairs this op against an -all-gather of a *different* byte count, which is the one call-order -divergence that hangs instead of returning wrong data — it cannot live in -the first job, because a job that wedges never reports. The launcher -certifies it out of band: every rank marks the file system before issuing -the pair, none marks it afterwards, and the job is ended by its own -watchdog. That sub-job doubles as the harness's positive control, since it -is a real collective deadlock this launcher has to survive. +That runs two jobs, in this order, and both must exit 0: + +- the `_pg` job: plain worker processes that rendezvous through + torch.distributed's env://, every `CHECKS` entry on `world_size` ranks. It + drives the cells that do not state `process_group`, through the + `reducescatter_pg` symbol on a boxed c10d group; +- the MPI-session job: the same number of ranks started by `mpirun`, every + `MPI_CHECKS` entry. It drives the cells that state `process_group=None` at + that world size, through `torch.ops.trtllm.reducescatter` on the + communicator the MPI session builds -- the symbol and communicator a dep4 + world launched under mpirun reaches. Its `mpirun` gets an environment with + the launching allocation's PMI/PMIx/Open MPI/Slurm variables removed, so it + starts its own ranks on this node rather than joining the allocation's step. + +The world-8 cells -- the dep8 combine, eight ranks over two nodes of four -- +are a third job, the MPI-session checks again on ranks an MPI launcher starts +across both nodes, one per device, as the dep8 engine's ranks are. One +process cannot place ranks on another node, so the launcher is the +environment's (the collected test takes it from `MODELING_V2_MPI_LAUNCH`): + + python _reducescatter_op_matrix.py --mpi-rank-worker + +Each MPI-session job drives the cells of its own world size and node count. +A rank that wedges inside a collective is ended by faulthandler at +`MPI_DEADLINE_S`, and the launcher ends its siblings. Not a pytest module, despite the `check_*` bodies. They are one fixed -sequence inside a single 4-rank job rather than independent cases: each reads -module-global rank state that only `_run_one_rank` binds, and several assert +sequence inside a single job rather than independent cases: each reads +module-global rank state that only the rank body binds, and several assert on communicator state the previous one left behind. Collected as tests they -would run at world size 1 against unbound globals — so neither this file's +would run at world size 1 against unbound globals -- so neither this file's name nor its function names match pytest's collection patterns, which is what keeps it uncollectable however pytest is pointed at this tree. -The collected entry point is -`tests/unittest/_torch/modeling_v2/comm/test_modeling_v2_reducescatter_op_matrix.py`: -it starts this job and turns its exit code into an assertion. Both halves are -started by file path: the launcher must not import `tensorrt_llm` (that calls -CUDA nor torch.distributed state of its own), and the +The collected entry point is `test_modeling_v2_reducescatter_op_matrix.py` +beside this file: it starts these jobs and turns their exit codes into +assertions. Both halves are started by file path: the launcher must not import +`tensorrt_llm` (that would give it CUDA and MPI state of its own), and the ranks reach the catalog by absolute import, so neither needs a package. """ +import faulthandler import functools import os import random +import shutil import signal import socket import subprocess @@ -59,14 +75,13 @@ assert torch.cuda.is_available(), "reducescatter requires CUDA devices" _WORKER_FLAG = "--rank-worker" -DEADLINE_S = 1800 - -# The wedge sub-job's budget. A pair that is going to return does so in -# microseconds, so a rank still inside it GRACE seconds later is wedged; the -# watchdog then ends its own process, because a rank blocked in a NCCL kernel -# that will never complete has no other way out. CAP covers the sub-job's -# import and communicator bring-up before the pair is issued (~50 s measured) -# plus the driver's teardown of four contexts holding a spinning kernel. +_MPI_WORKER_FLAG = "--mpi-rank-worker" +# Each job's own deadline; `_rank_job` reads their sum off DEADLINE_S. Each +# MPI-session rank also ends itself at MPI_DEADLINE_S, and the collected test +# allows the world-8 job's launcher that plus a grace. +PG_DEADLINE_S = 1800 +MPI_DEADLINE_S = 600 +DEADLINE_S = PG_DEADLINE_S + MPI_DEADLINE_S HIDDEN = 2560 # DeepSeek-V3-Lite hidden size, the caller that motivates this @@ -90,13 +105,17 @@ BF16_U = 2.0**-8 # Bound inside the rank body, never in the launcher: importing the entry -# pulls in tensorrt_llm, which calls MPI_Init at import, and an +# pulls in tensorrt_llm, which calls MPI_Init at import, and the launcher has +# to stay free of MPI and CUDA state. reducescatter: Any = None entry: Any = None PG_BOXED: Any = None +MPI_COMM: Any = None SUBGROUPS: Dict[Tuple[int, ...], Any] = {} RANK = 0 WORLD = 1 +#: Distinct hosts the MPI-session job's ranks run on. +NODES = 1 GROUP: List[int] = [0] @@ -422,15 +441,24 @@ def check_cuda_graph_captures_a_cold_first_call() -> None: dist.barrier() +def _on_mpi_session(cell: Any) -> bool: + """Whether `cell` drives the MPI-session symbol: it states `process_group=None`.""" + return "process_group" in cell.spec and cell.spec["process_group"] is None + + def check_certified_cells() -> None: - """Drive the entry's own cell list, inside `validating`. + """Drive the entry's `_pg` cells, inside `validating`. - The rest of this file says things no cell can -- CUDA graph behaviour, call - order, which stream the call lands on. This one says the plain thing: for - every configuration the entry claims, the slice is the sum, bitwise, and the - guard admits the input a shipped target passes. + The rest of this job says things none of these cells can -- CUDA graph + behaviour, call order, which stream the call lands on. This one says the + plain thing: for every configuration these cells claim, the slice is the + sum, bitwise, and the guard admits the input a shipped target passes. """ + driven = 0 for i, cell in enumerate(entry.reducescatter.CELLS): + if _on_mpi_session(cell): + continue # the MPI-session job drives it (`MPI_CHECKS`) + driven += 1 spec = cell.spec rows, trailing, dtype = spec["rows_per_rank"], spec["trailing"], spec["dtype"] seed = 9000 + 13 * i @@ -441,6 +469,7 @@ def check_certified_cells() -> None: assert out.shape == (rows, *trailing), (cell.why, out.shape) _assert_bitwise(out, _ref(sizes, RANK, seed, dtype, trailing), cell.why) dist.barrier() + assert driven > 0, "no cell drives the _pg symbol" def check_uniform_reduce_scatter() -> None: @@ -1409,6 +1438,692 @@ def _run_one_rank(device: Optional[int] = None) -> int: os._exit(0) +# ---- the MPI-session job ------------------------------------------------------ +# +# Everything from here to `MPI_CHECKS` drives the cells that state +# `process_group=None`, on the symbol and the communicator a dep4 or dep8 +# serving world reaches: ranks started by mpirun, `torch.ops.trtllm.reducescatter`, +# and the NCCL communicator tensorrt_llm's MPI session builds for `group` at its +# first call. These ranks have no torch.distributed group; they synchronize +# through the MPI session itself. One job drives the cells of its own world +# size: four ranks of one node (`_mpirun_ranks`), or eight over two nodes +# (started by the environment's launcher). + +#: bf16 carries 8 significant bits, so one round-to-nearest is off by at most +#: 2^-8 of the value it rounds: the format's unit roundoff. +BF16_UNIT_ROUNDOFF = 2.0**-8 + +#: The share of elements the "one rank's addend left out" variant must move +#: outside `_band`, per world size. That variant differs from the reference by +#: the left-out addend x_k alone, so it lands inside the band wherever |x_k| is +#: below (gamma_{W-1} + u) * sum_r |x_r|: a band that widens with W while x_k +#: does not. Measured at 32768 rows of standard normal payloads: at least +#: 0.9694 at world 4, and 0.8536..0.8537 on the eight ranks of world 8 (the +#: neighbouring slice, which differs in every addend, stays above 0.96 at both). +MISSING_ADDEND_FLOOR = {4: 0.9, 8: 0.8} + +#: The entry's own `__call__` and `reference`, counted (`_count_entry_calls`). +SEEN: Dict[str, int] = {"__call__": 0, "reference": 0} +#: What each MPI-session cell ran through the entry: [calls, references]. +DRIVEN: Dict[int, List[int]] = {} + + +def _mpi_cells() -> List[Tuple[int, Any]]: + """(index, cell) of every cell this MPI-session job drives: its world size's.""" + return [ + (i, c) + for i, c in enumerate(entry.reducescatter.CELLS) + if _on_mpi_session(c) and c.spec["world_size"] == WORLD + ] + + +def _mpi_sync() -> None: + """Every rank's device work done, then every rank here.""" + torch.cuda.synchronize() + MPI_COMM.Barrier() + + +def _count_entry_calls() -> None: + """What `_validating.certifying` does under pytest, for this rank body. + + The entry's own `__call__` (with `is_valid` interposed first, as + `validating` does) and its own `reference` are wrapped to count, so every + cell can assert that both ran for it rather than a local copy. + """ + cls = type(entry.reducescatter) + call, ref = cls.__dict__["__call__"], cls.__dict__["reference"] + + def counted_call(self, *args, **kwargs): + SEEN["__call__"] += 1 + self.is_valid(*args, **kwargs) + return call(self, *args, **kwargs) + + def counted_reference(self, *args, **kwargs): + SEEN["reference"] += 1 + return ref(self, *args, **kwargs) + + cls.__call__ = counted_call + cls.reference = counted_reference + + +def _account(i: int, before: Dict[str, int]) -> None: + """Add what the entry ran since `before` to cell `i`.""" + got = DRIVEN.setdefault(i, [0, 0]) + got[0] += SEEN["__call__"] - before["__call__"] + got[1] += SEEN["reference"] - before["reference"] + + +def _combine(x: torch.Tensor, spec: Dict[str, Any]) -> torch.Tensor: + """The call the dep4 and dep8 deployments make, with the arguments the cell states.""" + return entry.reducescatter(x, spec["sizes"], GROUP, process_group=spec["process_group"]) + + +def _width(spec: Dict[str, Any]) -> int: + assert len(spec["trailing"]) == 1 and spec["dtype"] is torch.bfloat16, spec + assert spec["world_size"] == WORLD and spec["sizes"] is None, (spec, WORLD) + # A cell without `nodes` is one node's, as the dep4 cells are. + assert spec.get("nodes", 1) == NODES, (spec, NODES) + return spec["trailing"][0] + + +def _payload(rank: int, shape: Tuple[int, ...], seed: int, exact: bool = True) -> torch.Tensor: + """`rank`'s input of `shape`, regenerable on every rank from the shared seed. + + `exact` draws `_block`'s value set, multiples of 1/8 with |x| <= 31/8: every + partial sum of WORLD of them is exact in bf16, so any correct reduction is + bitwise the reference. Otherwise standard normal values cast to bf16, on + which the order of the sum shows. + """ + gen = torch.Generator(device="cuda").manual_seed((seed * 977 + rank) * 131 + 7) + if exact: + raw = torch.randint(-31, 32, shape, generator=gen, device="cuda", dtype=torch.int8) + return raw.to(torch.bfloat16).div_(8.0) + return torch.randn(shape, generator=gen, device="cuda", dtype=torch.float32).to(torch.bfloat16) + + +def _slices_of_every_rank( + shape: Tuple[int, ...], seed: int, starts: Sequence[int], rows: int, exact: bool = True +) -> List[List[torch.Tensor]]: + """Rows [start, start + rows) of every rank's `_payload`, one list per start. + + Each rank's whole payload is drawn, sliced and dropped in turn, so a rank + holds one whole payload at a time rather than WORLD of them -- 3.2 GB each + at world 8 and 32768 rows per rank. A slice is bitwise those rows of the + whole payload, so what a check computes from it is unchanged. + """ + per_start: List[List[torch.Tensor]] = [[] for _ in starts] + for r in range(WORLD): + block = _payload(r, shape, seed, exact) + for got, start in zip(per_start, starts): + got.append(block[start : start + rows].clone()) + del block + return per_start + + +def _differs(a: torch.Tensor, b: torch.Tensor) -> torch.Tensor: + """Whether two bf16 tensors differ anywhere: a device bool, compared as int64 words.""" + assert a.shape == b.shape and a.dtype == b.dtype == torch.bfloat16, (a.shape, b.shape) + return torch.ne(a.reshape(-1).view(torch.int64), b.reshape(-1).view(torch.int64)).any() + + +def _sweep_rows(spec: Dict[str, Any]) -> List[int]: + """The row counts a sweep cell names, largest first.""" + lo, hi = spec["sweep_rows"] + rows = set(range(lo, hi + 1, spec["sweep_step"])) | set(spec.get("sweep_also", ())) | {hi} + assert spec["rows_per_rank"] == hi and all(lo <= r <= hi for r in rows), spec + # Largest first, so the caching allocator sizes its blocks for the top once. + return sorted(rows, reverse=True) + + +def _band(contributions: Sequence[torch.Tensor]) -> torch.Tensor: + """How far a bf16 reduction of `contributions` may sit from the entry's reference. + + Two terms, with u = 2^-8, bf16's unit roundoff. The reference accumulates + in fp32 and rounds once to bf16, so it is within u * sum_r |x_r| of the + exact sum. The op adds the same WORLD addends in bf16 in some order, + rounding each of its k = WORLD - 1 partial sums, so it is within + gamma_k * sum_r |x_r| of the exact sum, gamma_k = k u / (1 - k u): the + classical bound for recursive summation in any order. The band is their + sum. Measured on the op at 32768 rows of standard normal payloads: the + largest error over the four ranks is 0.890 of the band, and the two wrong + results `check_mpi_random_payload_cells` builds fall outside it in at least + 0.9694 of the elements. At world 8 over two nodes (k = 7) the largest error + over the eight ranks is 0.623 of the band, over three payloads; the wrong + results fall outside it in at least 0.8536 of the elements, the share + `MISSING_ADDEND_FLOOR` explains. + """ + absum = torch.zeros(contributions[0].shape, dtype=torch.float32, device="cuda") + for block in contributions: + absum += block.float().abs() + k = WORLD - 1 + gamma = k * BF16_UNIT_ROUNDOFF / (1 - k * BF16_UNIT_ROUNDOFF) + return (gamma + BF16_UNIT_ROUNDOFF) * absum + + +def check_mpi_session_is_the_launchers_world() -> None: + """The ranks are mpirun's, one device each, and the MPI session spans them all. + + `getComm` builds the communicator through the MPI session, so a session of + one process per rank -- what a launch without the ranks in place gives -- + would wedge the first call instead of failing it. Over several nodes each + node's ranks hold that node's devices 0..n-1, in rank order, as the + launcher places them. + """ + from tensorrt_llm._utils import mpi_rank, mpi_world_size + + assert os.environ.get("TLLM_DISABLE_MPI") != "1", "the MPI-session job runs with MPI" + assert mpi_world_size() == MPI_COMM.Get_size() == WORLD, (mpi_world_size(), WORLD) + assert mpi_rank() == MPI_COMM.Get_rank() == RANK, (mpi_rank(), RANK) + placed = MPI_COMM.allgather((socket.gethostname(), torch.cuda.current_device())) + hosts = list(dict.fromkeys(host for host, _ in placed)) + assert len(hosts) == NODES and WORLD % NODES == 0, placed + per_node = WORLD // NODES + assert [hosts.index(h) for h, _ in placed] == [r // per_node for r in range(WORLD)], placed + assert [d for _, d in placed] == [r % per_node for r in range(WORLD)], placed + if RANK == 0: + print( + f"[mpi] {torch.cuda.get_device_name()} sm_" + f"{''.join(map(str, torch.cuda.get_device_capability()))}, NCCL " + f"{torch.cuda.nccl.version()}; {WORLD} ranks on {NODES} node(s): {placed}", + flush=True, + ) + cells = _mpi_cells() + assert cells, "no cell states process_group=None" + for _, cell in cells: + _width(cell.spec) + _mpi_sync() + + +def check_mpi_eager_sweep_cells() -> None: + """Every sweep cell: at each row count it names, the slice is the reference's, bitwise. + + One full-height input per rank, regenerated for every rank of the world + from the shared seeds. A row count passes the leading WORLD x rows rows, + which is a contiguous input of exactly that height, so rank r's addend for + this slice is rows [RANK x rows, (RANK + 1) x rows) of rank r's buffer. + These are the group's first calls, so the first of them builds the + communicator, eagerly. A mismatch is agreed across the ranks before any + rank raises, so the call that counts it is made by all of them. + """ + for i, cell in _mpi_cells(): + spec = cell.spec + if "sweep_rows" not in spec: + continue + width = _width(spec) + rows_list = _sweep_rows(spec) + top = rows_list[0] + seed = 20000 + 13 * i + full = [_payload(r, (WORLD * top, width), seed) for r in range(WORLD)] + mine = full[RANK] + bad = torch.zeros(len(rows_list), dtype=torch.bool, device="cuda") + before = dict(SEEN) + _mpi_sync() + for k, rows in enumerate(rows_list): + out = _combine(mine[: WORLD * rows], spec) + assert out.shape == (rows, width), (cell.why, rows, out.shape) + ref = entry.reducescatter.reference( + [block[RANK * rows : (RANK + 1) * rows] for block in full] + ) + bad[k] = _differs(out, ref) + torch.cuda.synchronize() + everyone = MPI_COMM.allgather(bad.nonzero().flatten().tolist()) + failing = sorted({rows_list[k] for got in everyone for k in got}) + if failing: + rows = failing[0] + out = _combine(mine[: WORLD * rows], spec) + ref = entry.reducescatter.reference( + [block[RANK * rows : (RANK + 1) * rows] for block in full] + ) + wrong = int((out != ref).sum().item()) + raise AssertionError( + f"{cell.why}: {len(failing)} of {len(rows_list)} row counts differ from the " + f"reference on some rank, the smallest {rows} here in {wrong} of " + f"{out.numel()} elements" + ) + _account(i, before) + del full, mine + _mpi_sync() + + +def check_mpi_random_payload_cells() -> None: + """Every cell with one row count and a random payload, at that row count. + + The exact payload first: the slice is the reference's bitwise, the input is + left as it was and the output is a tensor of its own. Then standard normal + payloads, where the bf16 sum cannot be bitwise the fp32 reference: four + calls return the same bits, every element sits inside `_band` of the + reference, and the band is shown to discriminate -- the same slice with one + rank's addend left out lands outside it in at least `MISSING_ADDEND_FLOOR` + of the elements, and the neighbouring position's slice in nearly every + element. + """ + for i, cell in _mpi_cells(): + spec = cell.spec + if spec.get("payload") != "random": + continue + width, rows = _width(spec), spec["rows_per_rank"] + before = dict(SEEN) + seed = 30000 + 13 * i + mine = _payload(RANK, (WORLD * rows, width), seed) + kept = mine.clone() + _mpi_sync() + out = _combine(mine, spec) + assert out.shape == (rows, width) and out.data_ptr() != mine.data_ptr(), cell.why + (addends,) = _slices_of_every_rank((WORLD * rows, width), seed, [RANK * rows], rows) + ref = entry.reducescatter.reference(addends) + assert not bool(_differs(out, ref)), f"{cell.why}: exact payload differs" + assert not bool(_differs(mine, kept)), f"{cell.why}: the input was written" + del mine, kept, out, ref, addends + _mpi_sync() + + seed += 1 + x = _payload(RANK, (WORLD * rows, width), seed, exact=False) + outs = [_combine(x, spec) for _ in range(4)] + for k, again in enumerate(outs[1:], start=1): + assert not bool(_differs(again, outs[0])), f"{cell.why}: repeat {k} differs" + nb = (RANK + 1) % WORLD + mine_blocks, neighbour = _slices_of_every_rank( + (WORLD * rows, width), seed, [RANK * rows, nb * rows], rows, exact=False + ) + ref = entry.reducescatter.reference(mine_blocks).float() + band = _band(mine_blocks) + ratio = ((outs[0].float() - ref).abs() / band.clamp_min(1e-30)).max().item() + assert ratio <= 1.0, ( + f"{cell.why}: an element sits {ratio:.3f} of the band from the reference" + ) + missing = ref - mine_blocks[nb].float() + caught = [] + for name, wrong, floor in ( + ("one rank's addend left out", missing, MISSING_ADDEND_FLOOR[WORLD]), + ("the neighbouring slice", entry.reducescatter.reference(neighbour).float(), 0.9), + ): + outside = ((wrong - ref).abs() > band).float().mean().item() + assert outside > floor, f"{cell.why}: {name} is outside the band in only {outside:.4f}" + caught.append(outside) + ratios, caught = MPI_COMM.allgather(ratio), MPI_COMM.allgather(caught) + if RANK == 0: + print( + f"[mpi] {cell.why[:44]}: largest error {max(ratios):.3f} of the band over " + f"{WORLD} ranks; outside it, the left-out addend in >= " + f"{min(c[0] for c in caught):.4f} and the neighbouring slice in >= " + f"{min(c[1] for c in caught):.4f} of the elements", + flush=True, + ) + _account(i, before) + del x, outs, mine_blocks, neighbour, ref, band, missing + _mpi_sync() + + +def check_mpi_graph_cells() -> None: + """The graph cells together: the decode graphs a dep4 or dep8 engine holds, one pool. + + One graph per captured batch, largest first as the engine's graph runner + captures, each with `sites` combines that read their own persistent input + and copy into their own persistent output (outside the pool, whose blocks a + later capture reuses). The group is warmed eagerly at each batch before its + capture, as the engine's warmup does. Then four replay orders, each twice: + one replay at a time on fresh exact payloads checked bitwise against the + reference, then every graph's payload staged and all replayed back to back + with no synchronization. Last, random payloads: each replay bitwise equal to + the same calls made eagerly. + """ + cells = [(i, c) for i, c in _mpi_cells() if "graph_rows" in c.spec] + assert cells, "no graph cell" + spec0 = cells[0][1].spec + pool_rows, sites, width = tuple(spec0["graph_rows"]), spec0["sites"], _width(spec0) + by_rows = {c.spec["rows_per_rank"]: i for i, c in cells} + assert sorted(by_rows) == sorted(pool_rows), (sorted(by_rows), pool_rows) + for _, c in cells: + assert c.spec["sites"] == sites and tuple(c.spec["graph_rows"]) == pool_rows, c.why + + inputs: Dict[int, torch.Tensor] = {} + outputs: Dict[int, torch.Tensor] = {} + graphs: Dict[int, Any] = {} + pool = None + for rows in sorted(pool_rows, reverse=True): + spec = entry.reducescatter.CELLS[by_rows[rows]].spec + before = dict(SEEN) + x = _payload(RANK, (sites, WORLD * rows, width), 40000 + rows) + out = torch.empty(sites, rows, width, dtype=torch.bfloat16, device="cuda") + + def body(x: torch.Tensor = x, out: torch.Tensor = out, spec: Dict[str, Any] = spec) -> None: + for s in range(sites): + out[s].copy_(_combine(x[s], spec)) + + side = torch.cuda.Stream() + side.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(side): + for _ in range(2): + body() + torch.cuda.current_stream().wait_stream(side) + _mpi_sync() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, pool=pool): + body() + pool = graph.pool() if pool is None else pool + inputs[rows], outputs[rows], graphs[rows] = x, out, graph + _account(by_rows[rows], before) + _mpi_sync() + + def stage(rows: int, seed: int, exact: bool = True) -> None: + inputs[rows].copy_(_payload(RANK, (sites, WORLD * rows, width), seed, exact)) + + def verify(rows: int, seed: int, where: str) -> None: + before = dict(SEEN) + ref = entry.reducescatter.reference( + [ + _payload(r, (sites, WORLD * rows, width), seed)[:, RANK * rows : (RANK + 1) * rows] + for r in range(WORLD) + ] + ) + _account(by_rows[rows], before) + if bool(_differs(outputs[rows], ref)): + bad = [s for s in range(sites) if not torch.equal(outputs[rows][s], ref[s])] + raise AssertionError(f"{where} rows={rows}: sites {bad} differ from the reference") + + asc = sorted(pool_rows) + shuffled = list(asc) + random.Random(1234).shuffle(shuffled) + extremes = [r for pair in zip(reversed(asc), asc) for r in pair][: len(asc)] + seed = 50000 + for name, order in ( + ("ascending", asc), + ("descending", list(reversed(asc))), + ("shuffled", shuffled), + ("extremes", extremes), + ): + assert sorted(order) == asc, (name, order) + for pos, rows in enumerate(order): + seed += 1 + stage(rows, seed) + _mpi_sync() + graphs[rows].replay() + torch.cuda.synchronize() + verify(rows, seed, f"{name}/checked@{pos}") + _mpi_sync() + staged: Dict[int, int] = {} + for rows in order: + seed += 1 + staged[rows] = seed + stage(rows, seed) + _mpi_sync() + for rows in order: + graphs[rows].replay() + torch.cuda.synchronize() + for pos, rows in enumerate(order): + verify(rows, staged[rows], f"{name}/burst@{pos}") + _mpi_sync() + + # A replay that did not happen leaves the previous payload's result; the + # gate of exactly zero sees it only if two consecutive references differ. + prev = entry.reducescatter.reference( + [ + _payload(r, (WORLD * 16, width), seed - 1)[RANK * 16 : (RANK + 1) * 16] + for r in range(WORLD) + ] + ) + last = entry.reducescatter.reference( + [_payload(r, (WORLD * 16, width), seed)[RANK * 16 : (RANK + 1) * 16] for r in range(WORLD)] + ) + margin = (last.float() - prev.float()).abs().max().item() + assert margin > 8.0, f"stale-replay margin {margin}" + + for rows in shuffled: + seed += 1 + stage(rows, seed, exact=False) + _mpi_sync() + graphs[rows].replay() + torch.cuda.synchronize() + before = dict(SEEN) + spec = entry.reducescatter.CELLS[by_rows[rows]].spec + eager = torch.stack([_combine(inputs[rows][s], spec) for s in range(sites)]) + _account(by_rows[rows], before) + if bool(_differs(outputs[rows], eager)): + raise AssertionError(f"random payload, rows={rows}: a replay differs from eager") + _mpi_sync() + graphs.clear() + _mpi_sync() + + +def check_mpi_sum_is_the_ring_chain() -> None: + """What the MPI-session sum is, on payloads with no exactness property. + + The bf16 chain in ring order from the destination's successor, + `x_{pos+1} + x_{pos+2} + ... + x_pos` (indices mod WORLD), bitwise -- the + order `check_reduction_is_deterministic_and_accumulates_in_the_input_dtype` + pins on the `_pg` symbol, so the two symbols compute the same arithmetic. + NCCL's algorithm choice for this topology (every ring of the world's + communicator runs in rank order, at world 4 on one node and at world 8 over + two): if this fails, the reduction order changed and the entry's numerics + note is re-measured, not this gate loosened. + """ + spec = _mpi_cells()[0][1].spec + width = _width(spec) + for rows in (1, 16, 256, 2048, 32768): + seed = 60000 + rows + x = _payload(RANK, (WORLD * rows, width), seed, exact=False) + out = _combine(x, spec) + (mine,) = _slices_of_every_rank( + (WORLD * rows, width), seed, [RANK * rows], rows, exact=False + ) + chain = mine[(RANK + 1) % WORLD].clone() + for k in range(2, WORLD + 1): + chain = chain + mine[(RANK + k) % WORLD] + assert not bool(_differs(out, chain)), f"rows={rows}: not the ring-order chain" + del x, out, mine, chain + _mpi_sync() + + +def check_mpi_round_trip_through_a_gather_in_one_graph() -> None: + """The dep4 or dep8 decode step's collective sequence, captured: gather, window, combine. + + Per MoE layer the target gathers every rank's rows, lets this rank's expert + window act on all of them and combines the world's partials, all on the one + communicator `getComm` caches for the group. Here each of the `sites` + layers gathers with `torch.ops.trtllm.allgather` -- a fixture building the + input the way the target does, never the reference -- scales by a + per-rank factor standing in for the window, and combines; every batch's + graph in one pool. Rank i gets back its own rows times the sum of the world's + factors, computed locally; the gathered rows are checked against a local + concatenation too, so a wrong gather cannot pass as a right combine. + """ + cells = [(i, c) for i, c in _mpi_cells() if "graph_rows" in c.spec] + spec = cells[0][1].spec + sites, width, pool_rows = spec["sites"], _width(spec), spec["graph_rows"] + # Payload k/8 with |k| <= 7 and factors (r + 1) / 4: every product and every + # partial sum is exact in bf16, so the gate is bitwise. At world 8 a partial + # sum is k * s / 32 with s at most 1 + 2 + ... + 8 = 36, so |k * s| stays at + # or below 252, inside bf16's exact integers. + factor = (RANK + 1) / 4.0 + total = sum((r + 1) / 4.0 for r in range(WORLD)) + pool = None + for rows in sorted(pool_rows, reverse=True): + own = torch.empty(sites, rows, width, dtype=torch.bfloat16, device="cuda") + gathered = torch.empty(sites, WORLD * rows, width, dtype=torch.bfloat16, device="cuda") + back = torch.empty(sites, rows, width, dtype=torch.bfloat16, device="cuda") + + def body() -> None: + for s in range(sites): + g = torch.ops.trtllm.allgather(own[s], None, GROUP) + gathered[s].copy_(g) + back[s].copy_(_combine(torch.mul(g, factor), spec)) + + def payload(rank: int, seed: int) -> torch.Tensor: + gen = torch.Generator(device="cuda").manual_seed(seed * 131 + rank) + raw = torch.randint(-7, 8, (sites, rows, width), generator=gen, device="cuda") + return raw.to(torch.bfloat16).div_(8.0) + + own.copy_(payload(RANK, 70000 + rows)) + side = torch.cuda.Stream() + side.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(side): + body() + torch.cuda.current_stream().wait_stream(side) + _mpi_sync() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, pool=pool): + body() + pool = graph.pool() if pool is None else pool + for seed in (71000 + rows, 72000 + rows): + own.copy_(payload(RANK, seed)) + _mpi_sync() + graph.replay() + torch.cuda.synchronize() + everyone = torch.cat([payload(r, seed) for r in range(WORLD)], dim=1) + assert torch.equal(gathered, everyone), f"rows={rows}: the gather fixture is wrong" + expected = (own.float() * total).to(torch.bfloat16) + assert torch.equal(back, expected), f"rows={rows}: the round trip is wrong" + _mpi_sync() + del graph + _mpi_sync() + + +def check_mpi_every_cell_was_driven() -> None: + """Each MPI-session cell ran through the entry's own `__call__` and `reference`.""" + for i, cell in _mpi_cells(): + calls, refs = DRIVEN.get(i, [0, 0]) + assert calls > 0 and refs > 0, ( + f"cell {cell.why!r}: the entry's __call__ / reference ran {calls} / {refs} times" + ) + + +MPI_CHECKS = ( + check_mpi_session_is_the_launchers_world, + # Before any capture: the group's first call builds the communicator. + check_mpi_eager_sweep_cells, + check_mpi_random_payload_cells, + check_mpi_graph_cells, + check_mpi_sum_is_the_ring_chain, + check_mpi_round_trip_through_a_gather_in_one_graph, + check_mpi_every_cell_was_driven, +) + + +def _run_one_mpi_rank() -> int: + """Body of one mpirun rank: run every MPI check; a failing rank exits nonzero + without finalizing, and mpirun then ends its siblings and fails. A rank + still running at `MPI_DEADLINE_S` -- wedged in a collective a sibling never + joined -- dumps its stack and exits.""" + global RANK, WORLD, NODES, GROUP, MPI_COMM, entry + faulthandler.dump_traceback_later(MPI_DEADLINE_S, exit=True) + rank = int(os.environ["OMPI_COMM_WORLD_RANK"]) + world = int(os.environ["OMPI_COMM_WORLD_SIZE"]) + torch.cuda.set_device(int(os.environ["OMPI_COMM_WORLD_LOCAL_RANK"])) + + from mpi4py import MPI + + from tensorrt_llm._torch.modeling_v2.catalog.comm import reducescatter as module + from tensorrt_llm._utils import mpi_comm + + entry = module + MPI_COMM = mpi_comm() + RANK, WORLD = rank, world + NODES = len(set(MPI_COMM.allgather(socket.gethostname()))) + GROUP = list(range(WORLD)) + assert WORLD >= 2, f"a collective needs at least 2 ranks, got {WORLD}" + _count_entry_calls() + + for check in MPI_CHECKS: + started = time.monotonic() + try: + check() + except BaseException: + import traceback + + print(f"[mpi rank {RANK}] FAILED {check.__name__}", flush=True) + traceback.print_exc() + sys.stdout.flush() + sys.stderr.flush() + os._exit(13) + if RANK == 0: + print(f"[mpi] {check.__name__}: {time.monotonic() - started:.1f}s", flush=True) + _mpi_sync() + print( + f"[mpi rank {RANK}] {len(MPI_CHECKS)} checks passed over {len(_mpi_cells())} cells at " + f"world {WORLD} on {NODES} node(s); peak allocated " + f"{torch.cuda.max_memory_allocated() / 2**30:.2f} GiB, reserved " + f"{torch.cuda.max_memory_reserved() / 2**30:.2f} GiB", + flush=True, + ) + sys.stdout.flush() + faulthandler.cancel_dump_traceback_later() + # mpirun fails a rank that exits without finalizing. The communicators are + # left as they are, as in the `_pg` job: these processes end here. + MPI.Finalize() + os._exit(0) + + +def _session_members(sid: int) -> List[int]: + """Every process in session `sid`, read from /proc.""" + members = [] + for entry_dir in Path("/proc").iterdir(): + if not entry_dir.name.isdigit(): + continue + try: + stat = (entry_dir / "stat").read_text() + except OSError: + continue + # pid (comm) state ppid pgrp session ...; comm may hold spaces. + fields = stat[stat.rindex(")") + 2 :].split() + if int(fields[3]) == sid: + members.append(int(entry_dir.name)) + return members + + +def _end_mpi_job(job: Any) -> None: + """End mpirun and every rank it started. + + mpirun runs as a session leader; its ranks stay in that session but each in + a process group of its own, so killing mpirun's group would leave them + running. mpirun is asked first, then the session's members are killed. + """ + if job.poll() is None: + job.terminate() + try: + job.wait(timeout=30) + except subprocess.TimeoutExpired: + pass + for pid in _session_members(job.pid): + try: + os.kill(pid, signal.SIGKILL) + except ProcessLookupError: + pass + if job.poll() is None: + job.kill() + job.wait() + + +def _mpirun_ranks(world_size: int) -> None: + """Start the MPI-session job under mpirun, one rank per device, and wait for it. + + The environment loses the launching allocation's PMI/PMIx/Open MPI/Slurm + variables, so this mpirun starts its own ranks on this node instead of + joining the allocation's step (where every rank would read rank 0 of one), + and `--oversubscribe` lets it place them whatever slot count it detects. + """ + mpirun = shutil.which("mpirun") + assert mpirun is not None, "the MPI-session cells start their ranks with mpirun; none on PATH" + env = { + k: v + for k, v in os.environ.items() + if not k.startswith(("PMIX_", "PMI_", "OMPI_", "SLURM_")) and k != "TLLM_DISABLE_MPI" + } + command = [mpirun, "-n", str(world_size), "--oversubscribe", "--allow-run-as-root"] + command += [sys.executable, str(Path(__file__).resolve()), _MPI_WORKER_FLAG] + print(f"[launcher] starting {world_size} MPI-session ranks under mpirun", flush=True) + job = subprocess.Popen(command, env=env, start_new_session=True) + try: + code = job.wait(timeout=MPI_DEADLINE_S) + except subprocess.TimeoutExpired: + _end_mpi_job(job) + raise AssertionError( + f"the {world_size}-rank MPI-session run did not finish in {MPI_DEADLINE_S}s (wedged)" + ) from None + if code != 0: + _end_mpi_job(job) + raise AssertionError(f"the {world_size}-rank MPI-session run failed: mpirun exited {code}") + + def _spawn_children( world_size: int, flag: str, extra_env: Optional[Dict[str, str]] = None ) -> List[Any]: @@ -1455,7 +2170,8 @@ def _kill_children(children: List[Any]) -> None: def _spawn_ranks() -> None: - """Start one worker process per claimed device; no external launcher.""" + """Run the `_pg` job, one worker process per claimed device and no external + launcher, then the MPI-session job under mpirun.""" visible = os.environ.get("CUDA_VISIBLE_DEVICES") assert visible, ( "set CUDA_VISIBLE_DEVICES to the devices this run owns, " @@ -1466,12 +2182,12 @@ def _spawn_ranks() -> None: f"CUDA_VISIBLE_DEVICES names {world_size} device(s); a collective test needs at least 2" ) children = _spawn_children(world_size, _WORKER_FLAG) - deadline = time.monotonic() + DEADLINE_S + deadline = time.monotonic() + PG_DEADLINE_S live = set(range(world_size)) failure = None while live and failure is None: if time.monotonic() > deadline: - failure = f"did not finish in {DEADLINE_S}s (wedged)" + failure = f"did not finish in {PG_DEADLINE_S}s (wedged)" break for r in sorted(live): code = children[r].poll() @@ -1485,6 +2201,7 @@ def _spawn_ranks() -> None: if failure is not None: _kill_children(children) raise AssertionError(f"the {world_size}-rank run failed: {failure}") + _mpirun_ranks(world_size) if __name__ == "__main__": @@ -1494,5 +2211,7 @@ def _spawn_ranks() -> None: import _rank_job sys.exit(_run_one_rank(device=_rank_job.adopt_srun_env())) + if _MPI_WORKER_FLAG in sys.argv: + sys.exit(_run_one_mpi_rank()) _spawn_ranks() print("OK") diff --git a/tests/unittest/_torch/modeling_v2/comm/test_modeling_v2_allgather_op_matrix.py b/tests/unittest/_torch/modeling_v2/comm/test_modeling_v2_allgather_op_matrix.py index 3f12adb01345..6ac75287f6cf 100644 --- a/tests/unittest/_torch/modeling_v2/comm/test_modeling_v2_allgather_op_matrix.py +++ b/tests/unittest/_torch/modeling_v2/comm/test_modeling_v2_allgather_op_matrix.py @@ -9,16 +9,51 @@ are silently wrong rather than loud. The names are kept apart so review does not read one as a copy of the other. -The matrix itself is ``_allgather_op_matrix.py`` beside this file, which is -its own 4-rank launcher; see ``_rank_job`` for why that is left intact. +The matrix is ``_allgather_op_matrix.py`` beside this file. It drives the +entry's ``CELLS`` in two tests: + +- the ``_pg`` checks and the world-4 cells: the matrix is its own launcher over + 4 devices of one node -- the ``_pg`` job on plain worker processes, then the + MPI-session job (the world-4 cells that state ``process_group=None``, the + Mistral Large 4 NVFP4 dep4 gather's and the tep4 and tp4 q-split selection + gather's) on ranks it starts under mpirun. ``_rank_job`` starts it in a fresh + interpreter and turns its exit code into an assertion; see ``_rank_job`` for + why the launcher is left intact; +- the world-8 cells (the Mistral Large 4 NVFP4 dep8 gather's and the tep8 q-split + selection gather's): 8 MPI-session ranks over two nodes of 4 devices, one per + device, started by an MPI launcher as the dep8 and tep8 engines' ranks are. A + process cannot place ranks on another node by itself, so this test takes that + launcher from ``MODELING_V2_MPI_LAUNCH`` -- a command prefix that starts the + command after it once per rank over the job's two nodes, four per node -- and + skips without it, as on the one 4-GPU node pre-merge runs this file on. On a + two-node compute session whose node-0 scheduler holds both nodes for a job, + that is ``bash /scripts/mn/mn_mpirun.sh 4``. """ +import os +import shlex +import signal +import subprocess +import sys +from pathlib import Path + import _rank_job import pytest import torch assert torch.cuda.is_available(), "allgather requires CUDA devices" +_HERE = Path(__file__).resolve().parent +_MATRIX = _HERE / "_allgather_op_matrix.py" +#: Names the launcher that starts the world-8 cells' ranks. +_LAUNCH_ENV = "MODELING_V2_MPI_LAUNCH" +#: Headroom over the ranks' own deadline, so their faulthandler dump wins the race. +_LAUNCHER_GRACE_S = 300 +# The engine's ranks run with MPI on, and the launcher must not join an +# enclosing MPI job or Slurm step. +_DROPPED = ("TLLM_DISABLE_MPI",) +_DROPPED_PREFIXES = ("PMIX_", "PMI_", "OMPI_", "SLURM_", "PRTE_") + # Each case spawns its own 4 rank processes over every visible device. Under # xdist several workers would fight for the same GPUs, so this must run @@ -27,3 +62,43 @@ @pytest.mark.no_xdist def test_allgather_op_matrix() -> None: _rank_job.run("allgather") + + +@pytest.mark.no_xdist +def test_allgather_world8_mpi_ranks() -> None: + """The world-8 cells: the matrix's MPI-session job, under the environment's launcher.""" + launch = os.environ.get(_LAUNCH_ENV, "").strip() + if not launch: + pytest.skip( + f"the world-8 cells need 8 ranks over two nodes, which this process cannot " + f"start; set {_LAUNCH_ENV} to a launcher that starts them" + ) + constants = _rank_job._load_launcher_constants(_MATRIX) + # tests/unittest/_torch/modeling_v2/comm -> repo root. + repo_root = _HERE.parents[4] + assert (repo_root / "tensorrt_llm").is_dir(), f"expected the repo root at {repo_root}" + env = { + k: v + for k, v in os.environ.items() + if k not in _DROPPED and not k.startswith(_DROPPED_PREFIXES) + } + env["PYTHONPATH"] = os.pathsep.join(p for p in (str(repo_root), env.get("PYTHONPATH", "")) if p) + command = [*shlex.split(launch), sys.executable, str(_MATRIX), "--mpi-rank-worker"] + print(f"[launcher] {shlex.join(command)}", flush=True) + # The launcher's own session, so a wedged job is ended as a whole: mpirun + # takes SIGTERM down to every rank it started, on every node. + job = subprocess.Popen(command, env=env, start_new_session=True) + try: + code = job.wait(timeout=constants.MPI_DEADLINE_S + _LAUNCHER_GRACE_S) + except subprocess.TimeoutExpired: + os.killpg(job.pid, signal.SIGTERM) + try: + job.wait(timeout=60) + except subprocess.TimeoutExpired: + os.killpg(job.pid, signal.SIGKILL) + job.wait() + raise AssertionError( + f"the world-8 job did not finish in " + f"{constants.MPI_DEADLINE_S + _LAUNCHER_GRACE_S}s (wedged)" + ) from None + assert code == 0, f"the world-8 job exited {code}: {shlex.join(command)}" diff --git a/tests/unittest/_torch/modeling_v2/comm/test_modeling_v2_reducescatter_op_matrix.py b/tests/unittest/_torch/modeling_v2/comm/test_modeling_v2_reducescatter_op_matrix.py index d1e3e4d9cb37..cc541bd3ca34 100644 --- a/tests/unittest/_torch/modeling_v2/comm/test_modeling_v2_reducescatter_op_matrix.py +++ b/tests/unittest/_torch/modeling_v2/comm/test_modeling_v2_reducescatter_op_matrix.py @@ -7,19 +7,50 @@ ``AllReduce`` module, while this covers the op itself cell by cell. The names are kept apart so review does not read one as a copy of the other. -The matrix is ``_reducescatter_op_matrix.py`` beside this file, which is its -own launcher and runs two jobs: the ordered check sequence, then a separately -capped job that certifies the one call-order divergence that wedges instead of -lying (it cannot be a normal check, because the job that runs it never -reports). +The matrix is ``_reducescatter_op_matrix.py`` beside this file. It drives the +entry's ``CELLS`` in two tests: + +- the ``_pg`` checks and the world-4 cells: the matrix is its own launcher over + 4 devices of one node -- the ``_pg`` job on plain worker processes, then the + MPI-session job (the cells that state ``process_group=None``, the Mistral Large 4 + NVFP4 dep4 combine's) on ranks it starts under mpirun. ``_rank_job`` starts + it in a fresh interpreter and turns its exit code into an assertion; see + ``_rank_job`` for why the launcher is left intact; +- the world-8 cells (the Mistral Large 4 NVFP4 dep8 combine's): 8 MPI-session ranks + over two nodes of 4 devices, one per device, started by an MPI launcher as + the dep8 engine's ranks are. A process cannot place ranks on another node by + itself, so this test takes that launcher from ``MODELING_V2_MPI_LAUNCH`` -- a + command prefix that starts the command after it once per rank over the job's + two nodes, four per node -- and skips without it, as on the one 4-GPU node + pre-merge runs this file on. On a two-node compute session whose node-0 + scheduler holds both nodes for a job, that is + ``bash /scripts/mn/mn_mpirun.sh 4``. """ +import os +import shlex +import signal +import subprocess +import sys +from pathlib import Path + import _rank_job import pytest import torch assert torch.cuda.is_available(), "reducescatter requires CUDA devices" +_HERE = Path(__file__).resolve().parent +_MATRIX = _HERE / "_reducescatter_op_matrix.py" +#: Names the launcher that starts the world-8 cells' ranks. +_LAUNCH_ENV = "MODELING_V2_MPI_LAUNCH" +#: Headroom over the ranks' own deadline, so their faulthandler dump wins the race. +_LAUNCHER_GRACE_S = 300 +# The engine's ranks run with MPI on, and the launcher must not join an +# enclosing MPI job or Slurm step. +_DROPPED = ("TLLM_DISABLE_MPI",) +_DROPPED_PREFIXES = ("PMIX_", "PMI_", "OMPI_", "SLURM_", "PRTE_") + # Each case spawns its own 4 rank processes over every visible device. Under # xdist several workers would fight for the same GPUs, so this must run @@ -28,3 +59,43 @@ @pytest.mark.no_xdist def test_reducescatter_op_matrix() -> None: _rank_job.run("reducescatter") + + +@pytest.mark.no_xdist +def test_reducescatter_world8_mpi_ranks() -> None: + """The world-8 cells: the matrix's MPI-session job, under the environment's launcher.""" + launch = os.environ.get(_LAUNCH_ENV, "").strip() + if not launch: + pytest.skip( + f"the world-8 cells need 8 ranks over two nodes, which this process cannot " + f"start; set {_LAUNCH_ENV} to a launcher that starts them" + ) + constants = _rank_job._load_launcher_constants(_MATRIX) + # tests/unittest/_torch/modeling_v2/comm -> repo root. + repo_root = _HERE.parents[4] + assert (repo_root / "tensorrt_llm").is_dir(), f"expected the repo root at {repo_root}" + env = { + k: v + for k, v in os.environ.items() + if k not in _DROPPED and not k.startswith(_DROPPED_PREFIXES) + } + env["PYTHONPATH"] = os.pathsep.join(p for p in (str(repo_root), env.get("PYTHONPATH", "")) if p) + command = [*shlex.split(launch), sys.executable, str(_MATRIX), "--mpi-rank-worker"] + print(f"[launcher] {shlex.join(command)}", flush=True) + # The launcher's own session, so a wedged job is ended as a whole: mpirun + # takes SIGTERM down to every rank it started, on every node. + job = subprocess.Popen(command, env=env, start_new_session=True) + try: + code = job.wait(timeout=constants.MPI_DEADLINE_S + _LAUNCHER_GRACE_S) + except subprocess.TimeoutExpired: + os.killpg(job.pid, signal.SIGTERM) + try: + job.wait(timeout=60) + except subprocess.TimeoutExpired: + os.killpg(job.pid, signal.SIGKILL) + job.wait() + raise AssertionError( + f"the world-8 job did not finish in " + f"{constants.MPI_DEADLINE_S + _LAUNCHER_GRACE_S}s (wedged)" + ) from None + assert code == 0, f"the world-8 job exited {code}: {shlex.join(command)}" diff --git a/tests/unittest/_torch/modeling_v2/gemm/test_modeling_v2_bmm_out.py b/tests/unittest/_torch/modeling_v2/gemm/test_modeling_v2_bmm_out.py index f8984c417673..821724c7aff2 100644 --- a/tests/unittest/_torch/modeling_v2/gemm/test_modeling_v2_bmm_out.py +++ b/tests/unittest/_torch/modeling_v2/gemm/test_modeling_v2_bmm_out.py @@ -1,7 +1,29 @@ # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -"""GPU test for the bmm_out catalog entry.""" +"""GPU test for the bmm_out catalog entry. +The entry owns the contract. This file turns a cell into tensors and holds the +claims no cell carries: the guard, the refusals the op makes itself, the empty +call and the layout of `b`. The Mistral Large 4 cells add six spec keys: +`b_layout` ("nk_t": `b` is the transposed view of the V-B rows of a regrouped +`wkv_b`), `m_range` (every row count of the range, each its own call), +`m_window_edges` (`(lo, hi, width)`: every row count of the first `width`-row +window, then the first and last of each later one), `graph_m` (each row count +captured in a CUDA graph and replayed), `tail_from` (one call on the +trailing rows of full-size buffers) and `out_heads` (`out` is heads 0..batch-1 of a +`[M, out_heads, out_row]` buffer, the rest of which the op must not touch). A cell whose +spec says `large` runs in `test_large_certified_cells`, marked `large`, which pre-merge +deselects. The entry's own cells and deepseek's say `transposed_out` instead: `out` is a +transposed view of a token-major buffer, or a plain one. + +Mistral Large 4's cells are built from `_ml4` (TEP4 and TP4, 16 heads per rank), `_ML4_64` +(DEP4, all 64 heads on every rank), `_ML4_16_FQ64` (16 heads per rank into a 64-head +buffer, on the fp8_ds_mla latent KV) and `_ML4_8` (TEP8 and TP8, 8 heads per rank; the +`_ML4_8_FQ64` cells are their K-B BMM into heads 0..7 of the 64-head fused query FlashMLA +reads on the fp8_ds_mla latent KV). +""" + +import sys import zlib import pytest @@ -12,31 +34,276 @@ __extra_import_path__ = [".."] from _cells import certified_cells # noqa: E402 — needs the path declared above -from _validating import assert_within_ulp, validating # noqa: E402 — needs the path declared above +from _validating import ( # noqa: E402,F401 — path above; fixture by name + assert_within_ulp, + certifying, + validating, +) assert torch.cuda.is_available(), "bmm_out requires a CUDA device" +DEV = "cuda" +#: A bf16 NaN payload nothing here computes: what an unwritten element holds. +SENT = 0x7FC1 +#: Four sentinels in one 64-bit word, for the sweep's word-wide reset. +SENT_WORD = 0x7FC1_7FC1_7FC1_7FC1 + def _compare(out: torch.Tensor, ref: torch.Tensor) -> None: """One GEMM's worth of accumulation-order difference. - Kernel and reference consume bit-identical operands and differ only in - the order cuBLAS and torch sum over K. Over random operands a few - outputs land near zero, where that reordering shows up as a relative - difference of 10 on an absolute difference of 1.5e-4 -- which is what - torch's default band rejects, and what no bf16 result could represent - anyway: one ulp at these rows' scale is about 6e-1. - - So: 4 ulp element-wise against the row scale, 2 ulp of relative RMS. - Measured worst case across this entry's cells is 1.5e-4 absolute, or - roughly 3e-4 ulp -- four orders of margin, and a band that a genuinely - wrong product would miss by orders rather than squeak past. + Kernel and reference consume bit-identical bf16 operands and round once + to bf16; they differ only in the order cuBLAS and torch sum over K, so an + element can land on the other side of a bf16 rounding boundary. One such + flip is one bf16 spacing, at most 2^-7 of its row's largest magnitude: 2 + of `assert_within_ulp`'s 2^-8 units at the row scale. Torch's default + band cannot say this -- near-zero outputs read as relative differences of + 10 on absolute differences of 1.5e-4. + + So: 4 units element-wise against the row scale, 2 units of relative RMS. + Measured at Mistral Large 4's three shapes over 9216 rows: kernel and + reference each within 0.996 units of an fp64 product, at most 1.84 units + apart, relative RMS at most 0.013 units. The nearest wrong products measured -- + a head's weights rolled by one, `a` shifted by one token, the rope half of q in + place of the nope half, the V-B `b` reshaped instead of transposed -- land 641 + to 790 units away. Over 32768 rows: each within 0.996 units of fp64, at most + 1.97 units apart (V-B; K-B 1.87 and 1.93), relative RMS at most 0.0125 + units, and each of those wrong products, at the shapes it applies to, 682 + to 821 units away. At DEP4's 64 heads over 32768 rows: each within 0.9962 + units of fp64, at most 1.98 units apart (V-B; K-B 1.88 and 1.95), + relative RMS at most 0.0124 units, and the same wrong products 716 to 900 + units away, 362 units of relative RMS. At TEP8's and TP8's 8 heads over + 32768 rows: each within 0.996 units of fp64, at most 1.97 units apart + (V-B; K-B 1.57 and 1.87), relative RMS at most 0.0122 units, and the same + wrong products 673 to 787 units away, 362 units of relative RMS. At TEP4's + and TP4's 16 heads into the 64-head fused query (token stride 36864) over + 32768 rows: each within 0.9961 units of fp64, at most 1.87 units apart (K-B + NoPE; RoPE 1.86), relative RMS at most 0.0062 units, and the same wrong + products, with the neighbouring head's q in place of a head's own, 678.6 to + 857.0 units away (678.6 to 794.5 on TP4's draw), 361.7 to 362.1 units of + relative RMS. At TEP8's 8 heads into the 64-head fused query (token stride + 36864) over 32768 rows: each within 0.9960 units of fp64, at most 1.91 + units apart (K-B NoPE; RoPE 1.86), relative RMS at most 0.0063 units, and + the same wrong products, with the neighbouring head's q in place of a + head's own, 672.0 to 804.6 units away, 361.9 to 362.1 units of relative + RMS. Over the `transposed_out` cells the worst case measured is 1.5e-4 + absolute, roughly 3e-4 units. """ assert_within_ulp(out, ref, element_ulp=4.0, rms_ulp=2.0) -def _build(spec, seed): - """Materialize one cell. +def _gate(a: torch.Tensor, w: torch.Tensor, out: torch.Tensor) -> None: + """`out` against the entry's reference, 16 heads at a time. + + The slices keep a 64-head, 32768-row gate's fp32 temporaries to a + quarter: with the whole batch gated at once a K-B sweep peaked 36.8 GiB + above base (46.8 GiB reserved), sliced 15.1 GiB (24.1 GiB). Each slice + passing the RMS gate implies the whole batch passes it, so the slices are + no looser; at 16 heads or fewer it is one `_compare`. + """ + for h in range(0, a.shape[0], 16): + _compare(out[h : h + 16], op.reference(a[h : h + 16], w[h : h + 16], out[h : h + 16])) + + +def _bits(t: torch.Tensor) -> torch.Tensor: + """bf16 bit patterns, so `!=` is bitwise (the NaN sentinel included).""" + return t.view(torch.int16) + + +def _words(t: torch.Tensor) -> torch.Tensor: + """The same bit patterns four to a 64-bit word, for the sweep's per-call + reset and compare over whole contiguous rows.""" + return t.view(torch.int64) + + +def _weights(spec, g): + bsz, k, n, dt = spec["batch"], spec["k"], spec["n"], spec["dtype"] + if spec.get("b_layout", "kn") == "kn": + return torch.randn(bsz, k, n, generator=g, device=DEV).to(dt) + # A regrouped wkv_b: 64 K-B rows per head ahead of n V-B rows per head, + # each k wide; V-B reads its rows as [B, n, k], transposed. + kvb = torch.randn(bsz * 64 + bsz * n, k, generator=g, device=DEV).to(dt) + return torch.transpose(kvb[bsz * 64 :].reshape(bsz, n, k), 1, 2) + + +def _build(spec, seed, m=None): + """Materialize one cell at `m` rows (the spec's own by default). + + With `a_row` / `out_row` set, `a` and `out` are transposed head-major views + of token-major `[M, B, row]` buffers, as the targets pass them: `a` is the + first `k` columns of each row, `out` the first `n`. With `out_heads`, `out`'s + parent is `[M, out_heads, out_row]` and `out` its first B heads, as the tep4, + tp4 and tep8 deployments' fused query is. Both parents come back too: `a`'s so a + captured graph can be fed new rows, `out`'s because "did it land there, and + only there" is the thing worth asserting. `out`'s parent starts as the + sentinel, so an element the op did not write shows. + """ + g = torch.Generator(device=DEV).manual_seed(seed) + bsz, k, n, dt = spec["batch"], spec["k"], spec["n"], spec["dtype"] + m = spec["m"] if m is None else m + w = _weights(spec, g) + if spec["a_row"] is None: + a = a_parent = torch.randn(bsz, m, k, generator=g, device=DEV).to(dt) + else: + a_parent = torch.randn(m, bsz, spec["a_row"], generator=g, device=DEV).to(dt) + a = a_parent[..., :k].transpose(0, 1) + if spec["out_row"] is None: + out = parent = torch.empty(bsz, m, n, dtype=dt, device=DEV) + else: + heads = spec.get("out_heads", bsz) + parent = torch.empty(m, heads, spec["out_row"], dtype=dt, device=DEV) + out = parent[:, :bsz, :n].transpose(0, 1) + # Only a decode-shaped cell escapes this: a dim of extent 1 leaves a + # view torch may still call contiguous. + assert m == 1 or not out.is_contiguous() + _bits(parent).fill_(SENT) + return a, w, out, a_parent, parent + + +def _other_heads_untouched(spec, parent) -> None: + """With `out_heads`, the heads past the call's batch hold the sentinel.""" + if spec.get("out_heads", spec["batch"]) > spec["batch"]: + assert bool((_bits(parent[:, spec["batch"] :]) == SENT).all()), ( + "the op wrote a head past the call's batch" + ) + + +def _single(spec, seed) -> None: + a, w, out, _, parent = _build(spec, seed) + parent_ptr = parent.data_ptr() + # Identity, not equality: the op must have written this very buffer and + # returned it, not produced a copy that happens to match. + assert op(a, w, out) is out, "the op writes in place and returns what it wrote" + # The product has to be in the caller's buffer, not in a replacement the op + # allocated: that is the whole reason this op exists over torch.bmm. + assert parent.data_ptr() == parent_ptr + _compare(out, op.reference(a, w, out)) + # A strided `out` must not spill into the parent's other columns. + if spec["out_row"] is not None: + assert bool((_bits(parent[..., spec["n"] :]) == SENT).all()), ( + "the op wrote outside the out view" + ) + _other_heads_untouched(spec, parent) + + +def _window_edges(lo: int, hi: int, width: int) -> list[int]: + """Every m of the first `width`-row window from `lo`, then the first and + last m of each later window, up to `hi`.""" + first = range(lo, min(hi, lo + width - 1) + 1) + rest = (m for s in range(lo + width, hi + 1, width) for m in (s, min(hi, s + width - 1))) + return sorted({*first, *rest}) + + +def _sweep(spec, seed, counts) -> None: + """Every m of `counts`, each its own call, against one run at the largest. + + The run at the largest m is gated against the reference; every shorter + call must then write bitwise its leading rows -- a row's product reads + only its own row of `a`, and cuBLAS's choice of kernel, which changes with + m, was measured not to move a bit. Descending, so the check is complete at + one comparison per call: before each call the rows it owns are reset to + the sentinel, so a skipped or partial write shows; after the last call row + r holds what the smallest call above r wrote and the parent's columns past + n hold the sentinel, so one comparison of the whole parent catches a write + past any call's rows or columns. + + Each call's rows are reset and compared whole, as 64-bit words, and the + compare is reduced with `any` on the device, so the sweep synchronizes + once. Measured at 32768 rows of the fused query: an element count of bf16 + patterns is far slower per compare than `any` over words, which is what + fits a 16-head K-B sweep to 1..32768 in a cell's time. A failing m + is re-run to count its elements. + + With `out_heads`, what a call owns of each row is the batch's heads, so + only those are reset and compared; the heads past them are never reset, + so a write there by any call outlasts the sweep and fails the whole-parent + comparison, and the full run's own are checked before `full` is taken. + Measured on TEP4's 16 heads of the 64-head fused query, every m 1..32768: + 9.23 s (K-B RoPE) and 9.41 s (NoPE) resetting the batch's heads, 21.52 s + and 21.89 s resetting whole token rows (on TP4's: 9.26 s and 21.55 s). On + TEP8's 8 heads of it: 5.49 s (K-B RoPE) and 5.76 s (NoPE) resetting the + batch's heads, 20.77 s and 20.84 s resetting whole token rows. + """ + counts = sorted(counts, reverse=True) + hi = counts[0] + a, w, out, _, parent = _build(spec, seed, m=hi) + assert op(a, w, out) is out + _gate(a, w, out) + _other_heads_untouched(spec, parent) + full = parent.clone() + rows, full_rows = _words(parent), _words(full) + if spec.get("out_heads", spec["batch"]) > spec["batch"]: + rows, full_rows = rows[:, : spec["batch"]], full_rows[:, : spec["batch"]] + differs = [] + for m in counts: + rows[:m].fill_(SENT_WORD) + op(a[:, :m], w, out[:, :m]) + differs.append(torch.ne(rows[:m], full_rows[:m]).any()) + failing = torch.nonzero(torch.stack(differs)).flatten().tolist() + if failing: + m = counts[failing[-1]] + _bits(parent[:m]).fill_(SENT) + op(a[:, :m], w, out[:, :m]) + raise AssertionError( + f"{len(failing)} row counts differ from the {hi}-row run; first m={m}, " + f"re-run: {int((_bits(parent[:m]) != _bits(full[:m])).sum())} elements" + ) + stray = int((_bits(parent) != _bits(full)).sum()) + assert stray == 0, f"{stray} elements past a call's view were written" + + +def _replays(spec, seed) -> None: + """Each row count captured in its own CUDA graph, replayed on new rows. + + The replay has to equal an eager call on the same rows bit for bit -- a + graph that froze something of the capture would differ -- has to leave + the parent's other columns (and, with `out_heads`, its other heads) alone, + and is gated against the reference like any other output. Each capture is + preceded by one call on the capture stream, as an engine's warm-up is. + """ + stream = torch.cuda.Stream() + for m in spec["graph_m"]: + a, w, out, a_parent, parent = _build(spec, seed + m, m=m) + stream.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(stream): + op(a, w, out) + torch.cuda.current_stream().wait_stream(stream) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=stream): + op(a, w, out) + for replay in range(2): + fresh, _, eager, fresh_parent, _ = _build(spec, seed + 1000 * (replay + 1) + m, m=m) + a_parent.copy_(fresh_parent) + _bits(parent).fill_(SENT) + graph.replay() + op(fresh, w, eager) + assert torch.equal(_bits(out), _bits(eager)), f"m={m}: replay {replay} differs" + assert bool((_bits(parent[..., spec["n"] :]) == SENT).all()) + _other_heads_untouched(spec, parent) + _compare(out, op.reference(fresh, w, out)) + + +def _tail(spec, seed) -> None: + """The trailing rows of full-size buffers, as a mixed step's generation + rows are: bitwise the same rows of a full-size call, gated against the + reference, and nothing written ahead of them or past their columns.""" + a, w, out, _, parent = _build(spec, seed) + op(a, w, out) + full = parent.clone() + _bits(parent).fill_(SENT) + t, n = spec["tail_from"], spec["n"] + a_tail, out_tail = a[:, t:], out[:, t:] + assert out_tail.storage_offset() > 0 + assert op(a_tail, w, out_tail) is out_tail + _compare(out_tail, op.reference(a_tail, w, out_tail)) + assert torch.equal(_bits(parent[t:, :, :n]), _bits(full[t:, :, :n])) + assert bool((_bits(parent[:t]) == SENT).all()), "the op wrote rows ahead of the slice" + assert bool((_bits(parent[t:, :, n:]) == SENT).all()), "the op wrote outside the out view" + _other_heads_untouched(spec, parent) + + +def _build_transposed(spec, seed): + """Materialize one `transposed_out` cell. `transposed_out` is the part a spec cannot express as a plain `empty`: the targets hand the op a transposed view of a buffer the forward already @@ -60,6 +327,427 @@ def _build(spec, seed): return a, w, out, parent +def _transposed_single(spec, seed) -> None: + a, w, out, parent = _build_transposed(spec, seed) + parent_ptr = parent.data_ptr() + # Identity, not equality: the op must have written this very buffer and + # returned it, not produced a copy that happens to match. + assert op(a, w, out) is out, "the op writes in place and returns what it wrote" + # The product has to be in the caller's buffer, not in a replacement the op + # allocated: that is the whole reason this op exists over torch.bmm. + assert parent.data_ptr() == parent_ptr + _compare(out, op.reference(a, w, out)) + + +# Mistral Large 4's two absorbed BMMs per layer, batch = the H heads a rank holds: +# this rank's 16 at TEP4 and TP4, all 64 at DEP4 (attention data parallel), +# this rank's 8 at TEP8 and TP8. `a_row` / `out_row` are the token-major +# buffers' per-head widths. +_ML4_CALLS = { + # K-B on the 54 RoPE layers: the 64-wide nope half of q [T, H, 128] + # (nope | rope) into the first 512 columns of the fused query [T, H, 576]. + "k_b_rope": dict(k=64, n=512, a_row=128, out_row=576, b_layout="kn"), + # K-B on the 18 NoPE layers: all 128 columns of q are nope. + "k_b_nope": dict(k=128, n=512, a_row=128, out_row=576, b_layout="kn"), + # V-B: the latent attention output [T, H, 512] into the [T, H, 128] view + # of the [T, H x 128] attention output; `b` is the transposed view of the + # V-B rows of the regrouped `wkv_b`, as `weights.py` lays it out. + "v_b": dict(k=512, n=128, a_row=512, out_row=128, b_layout="nk_t"), +} + + +#: max_num_tokens of the two Mistral Large 4 targets: the FP8 target serves 9216, +#: the NVFP4 target 32768. Neither chunks prefill, so one step carries up to +#: that many rows through both BMMs. +_FP8_ROWS = 9216 +_NVFP4_ROWS = 32768 + + +def _ml4(call: str, **overrides) -> dict: + """One Mistral Large 4 call at the FP8 target's max_num_tokens 9216, unless `m` + is overridden.""" + return dict(dict(batch=16, m=_FP8_ROWS, dtype=torch.bfloat16, **_ML4_CALLS[call]), **overrides) + + +#: The NVFP4 checkpoint at DEP4: batch = all 64 heads, over the rank's own +#: tokens, up to the same max_num_tokens 32768 per rank. Each cell spreads one +#: `_ML4_CALLS` entry over this. +_ML4_64 = dict(batch=64, m=_NVFP4_ROWS, dtype=torch.bfloat16) + +#: The NVFP4 checkpoint at TEP8 and TP8 (the two shard attention alike): batch = this +#: rank's 8 of the 64 heads, over every token of the step (attention DP off), up to +#: max_num_tokens 32768. +#: Each cell spreads one `_ML4_CALLS` entry over this. +_ML4_8 = dict(batch=8, m=_NVFP4_ROWS, dtype=torch.bfloat16) + +#: The same at TEP8 and TP8 on the fp8_ds_mla latent KV: FlashMLA's sparse kernels take 64 +#: query heads, so the K-B BMM writes the rank's 8 heads into heads 0..7 of a `[T, 64, 576]` +#: fused query (`out_heads`; the token stride is 64 x 576, the head stride 576) whose heads +#: 8..63 stay zero. +_ML4_8_FQ64 = dict(_ML4_8, out_heads=64) + +#: The NVFP4 checkpoint at 16 heads per rank (TEP4, TP4) on the fp8_ds_mla latent KV: FlashMLA's +#: sparse kernels take 64 query heads, so the K-B BMM writes the rank's 16 heads into heads 0..15 +#: of a `[T, 64, 576]` fused query (`out_heads`; the token stride is 64 x 576 = 36864, the head +#: stride 576) whose heads 16..63 stay zero for the step. Chunked prefill keeps a step's rows at +#: max_num_tokens 32768 or fewer. +_ML4_16_FQ64 = dict(batch=16, m=_NVFP4_ROWS, dtype=torch.bfloat16, out_heads=64) + + +def _ml4_dspark(heads: int) -> tuple[Cell, ...]: + """DSpark's absorbed K-B and V-B at `heads` heads per rank, graph rows 1..64. + + These row counts are below the eager envelope and the integer-width + boundaries the entry's note records. + """ + return tuple( + Cell( + why=f"DSpark absorbed {call}, {heads} heads, graph rows 1..64", + spec=dict( + batch=heads, + m=64, + dtype=torch.bfloat16, + **_ML4_CALLS[call], + graph_m=tuple(range(1, 65)), + ), + ) + for call in ("k_b_rope", "k_b_nope", "v_b") + ) + + +# A `b_layout` of "nk_t" is the transposed view the targets pass for V-B; the +# other cells' `b` is contiguous, which the targets pass for K-B. +# +# Mistral Large 4 NVFP4 at TEP4 and TP4: 16 heads per rank. +_ML4_TP4_CELLS: tuple[Cell, ...] = ( + *_ml4_dspark(16), + Cell( + why="Mistral Large 4 K-B RoPE (k 64), every m 1..9216: each m's rows bitwise " + "the leading rows of the 9216-row run, which is gated (~3 s)", + spec=_ml4("k_b_rope", m_range=(1, 9216)), + ), + Cell( + why="Mistral Large 4 K-B NoPE (k 128), every m 1..9216: each m's rows bitwise " + "the leading rows of the 9216-row run, which is gated (~3 s)", + spec=_ml4("k_b_nope", m_range=(1, 9216)), + ), + Cell( + why="Mistral Large 4 V-B (transposed-view b), every m 1..9216: each m's rows " + "bitwise the leading rows of the 9216-row run, which is gated (~1.5 s)", + spec=_ml4("v_b", m_range=(1, 9216)), + ), + Cell( + why="Mistral Large 4 K-B RoPE (k 64), CUDA graphs at the captured, padded " + "decode batches 1/2/4/8/16: replays bitwise equal to eager", + spec=_ml4("k_b_rope", graph_m=(1, 2, 4, 8, 16)), + ), + Cell( + why="Mistral Large 4 K-B NoPE (k 128), CUDA graphs at the captured, padded " + "decode batches 1/2/4/8/16: replays bitwise equal to eager", + spec=_ml4("k_b_nope", graph_m=(1, 2, 4, 8, 16)), + ), + Cell( + why="Mistral Large 4 V-B (transposed-view b), CUDA graphs at the captured, " + "padded decode batches 1/2/4/8/16: replays bitwise equal to eager", + spec=_ml4("v_b", graph_m=(1, 2, 4, 8, 16)), + ), + Cell( + why="Mistral Large 4 K-B RoPE (k 64), a mixed step's 15 generation rows: " + "rows 9201..9215 of 9216-row buffers, a nonzero storage offset", + spec=_ml4("k_b_rope", tail_from=9201), + ), + Cell( + why="Mistral Large 4 K-B NoPE (k 128), a mixed step's 15 generation rows: " + "rows 9201..9215 of 9216-row buffers, a nonzero storage offset", + spec=_ml4("k_b_nope", tail_from=9201), + ), + Cell( + why="Mistral Large 4 V-B (transposed-view b), a mixed step's 15 generation " + "rows: rows 9201..9215 of 9216-row buffers, a nonzero storage offset", + spec=_ml4("v_b", tail_from=9201), + ), + # The NVFP4 target's envelope: max_num_tokens 32768 without chunked + # prefill, so one context step passes both BMMs up to 32768 rows (one + # request, or up to 16 packed), and the generation rows of a mixed step + # sit behind up to 32753 context rows (max_batch_size 16). The captured + # decode batches are 1/2/4/8/16 again, which the graph cells above hold. + Cell( + why="Mistral Large 4 32768 K-B RoPE (k 64), every m 1..32768, the NVFP4 " + "target's max_num_tokens: each m's rows bitwise the leading rows of the " + "32768-row run, which is gated (~7 s)", + spec=_ml4("k_b_rope", m=_NVFP4_ROWS, m_range=(1, _NVFP4_ROWS)), + ), + Cell( + why="Mistral Large 4 32768 K-B NoPE (k 128), every m 1..32768, the NVFP4 " + "target's max_num_tokens: each m's rows bitwise the leading rows of the " + "32768-row run, which is gated (~7 s)", + spec=_ml4("k_b_nope", m=_NVFP4_ROWS, m_range=(1, _NVFP4_ROWS)), + ), + Cell( + why="Mistral Large 4 32768 V-B (transposed-view b), every m 1..32768, the " + "NVFP4 target's max_num_tokens: each m's rows bitwise the leading rows " + "of the 32768-row run, which is gated (~4 s)", + spec=_ml4("v_b", m=_NVFP4_ROWS, m_range=(1, _NVFP4_ROWS)), + ), + Cell( + why="Mistral Large 4 32768 K-B RoPE (k 64), a mixed step's 15 generation " + "rows: rows 32753..32767 of 32768-row buffers, a nonzero storage offset", + spec=_ml4("k_b_rope", m=_NVFP4_ROWS, tail_from=_NVFP4_ROWS - 15), + ), + Cell( + why="Mistral Large 4 32768 K-B NoPE (k 128), a mixed step's 15 generation " + "rows: rows 32753..32767 of 32768-row buffers, a nonzero storage offset", + spec=_ml4("k_b_nope", m=_NVFP4_ROWS, tail_from=_NVFP4_ROWS - 15), + ), + Cell( + why="Mistral Large 4 32768 V-B (transposed-view b), a mixed step's 15 " + "generation rows: rows 32753..32767 of 32768-row buffers, a nonzero " + "storage offset", + spec=_ml4("v_b", m=_NVFP4_ROWS, tail_from=_NVFP4_ROWS - 15), + ), + # TEP4 and TP4 on the fp8_ds_mla latent KV (PrefillTarget._attention and + # DecodeTarget._attention, one call over every row of the step): the two K-B + # calls write the rank's 16 heads into heads 0..15 of the 64-head fused query, + # so the out token stride is 36864 where the `_ml4` cells' is 9216. V-B reads + # the compact `[T, 16, 512]` latent output, which the `_ml4` V-B cells above + # hold. m is a step's rows: every count 1..32768 eager (chunked prefill keeps + # a step at max_num_tokens), the padded decode batches 1/2/4/8/16 captured + # (the LLM API's default list reaches every batch 1..16), and a mixed step's + # generation rows at the tail. + Cell( + why="fused q64 K-B RoPE (k 64), 16 heads into heads 0..15 of [T, 64, 576], " + "every m 1..32768: each m's rows bitwise the leading rows of the 32768-row " + "run, which is gated; heads 16..63 never written (~9 s)", + spec=dict(_ML4_16_FQ64, **_ML4_CALLS["k_b_rope"], m_range=(1, _NVFP4_ROWS)), + ), + Cell( + why="fused q64 K-B NoPE (k 128), 16 heads into heads 0..15 of [T, 64, 576], " + "every m 1..32768: each m's rows bitwise the leading rows of the 32768-row " + "run, which is gated; heads 16..63 never written (~9 s)", + spec=dict(_ML4_16_FQ64, **_ML4_CALLS["k_b_nope"], m_range=(1, _NVFP4_ROWS)), + ), + Cell( + why="fused q64 K-B RoPE (k 64), 16 heads, CUDA graphs at every decode batch " + "1..16: replays bitwise equal to eager, heads 16..63 never written", + spec=dict(_ML4_16_FQ64, **_ML4_CALLS["k_b_rope"], graph_m=tuple(range(1, 17))), + ), + Cell( + why="fused q64 K-B NoPE (k 128), 16 heads, CUDA graphs at every decode batch " + "1..16: replays bitwise equal to eager, heads 16..63 never written", + spec=dict(_ML4_16_FQ64, **_ML4_CALLS["k_b_nope"], graph_m=tuple(range(1, 17))), + ), + Cell( + why="fused q64 K-B RoPE (k 64), 16 heads, a mixed step's 15 generation rows: " + "rows 32753..32767 of 32768-row buffers, heads 16..63 never written", + spec=dict(_ML4_16_FQ64, **_ML4_CALLS["k_b_rope"], tail_from=_NVFP4_ROWS - 15), + ), + Cell( + why="fused q64 K-B NoPE (k 128), 16 heads, a mixed step's 15 generation rows: " + "rows 32753..32767 of 32768-row buffers, heads 16..63 never written", + spec=dict(_ML4_16_FQ64, **_ML4_CALLS["k_b_nope"], tail_from=_NVFP4_ROWS - 15), + ), +) + +# The same two targets over eight ranks, TEP8 and TP8: 8 heads per rank. +_ML4_TP8_CELLS: tuple[Cell, ...] = ( + *_ml4_dspark(8), + *( + Cell( + why=f"DSpark verify {call}, 8 into 64 heads, graph rows 1..64", + spec=dict(_ML4_8_FQ64, **_ML4_CALLS[call], graph_m=tuple(range(1, 65))), + ) + for call in ("k_b_rope", "k_b_nope") + ), + # TEP8 and TP8 (the "tep8" cells are both targets' calls): the same three + # calls at batch 8, every rank holding 8 of the 64 heads and running every + # token of the step (attention DP off), so m is the step's token count: up + # to 32768 in a context step (no chunked prefill), every batch 1..16 in a + # captured decode step (the gates' padded 1/2/4/8/16 and the LLM API + # default's unpadded batches), and a mixed step's generation rows behind + # up to 32753 context rows. At half the 16-head bytes the every-m sweeps + # fit a cell's time. + Cell( + why="tep8 8 heads K-B RoPE (k 64), every m 1..32768: each m's rows " + "bitwise the leading rows of the 32768-row run, which is gated (~5 s)", + spec=dict(_ML4_8, **_ML4_CALLS["k_b_rope"], m_range=(1, _NVFP4_ROWS)), + ), + Cell( + why="tep8 8 heads K-B NoPE (k 128), every m 1..32768: each m's rows " + "bitwise the leading rows of the 32768-row run, which is gated (~5 s)", + spec=dict(_ML4_8, **_ML4_CALLS["k_b_nope"], m_range=(1, _NVFP4_ROWS)), + ), + Cell( + why="tep8 8 heads V-B (transposed-view b), every m 1..32768: each m's " + "rows bitwise the leading rows of the 32768-row run, which is gated (~4 s)", + spec=dict(_ML4_8, **_ML4_CALLS["v_b"], m_range=(1, _NVFP4_ROWS)), + ), + Cell( + why="tep8 8 heads K-B RoPE (k 64), CUDA graphs at every decode batch " + "1..16, padded and unpadded: replays bitwise equal to eager", + spec=dict(_ML4_8, **_ML4_CALLS["k_b_rope"], graph_m=tuple(range(1, 17))), + ), + Cell( + why="tep8 8 heads K-B NoPE (k 128), CUDA graphs at every decode batch " + "1..16, padded and unpadded: replays bitwise equal to eager", + spec=dict(_ML4_8, **_ML4_CALLS["k_b_nope"], graph_m=tuple(range(1, 17))), + ), + Cell( + why="tep8 8 heads V-B (transposed-view b), CUDA graphs at every decode " + "batch 1..16, padded and unpadded: replays bitwise equal to eager", + spec=dict(_ML4_8, **_ML4_CALLS["v_b"], graph_m=tuple(range(1, 17))), + ), + Cell( + why="tep8 8 heads K-B RoPE (k 64), a mixed step's 15 generation rows: " + "rows 32753..32767 of 32768-row buffers, a nonzero storage offset", + spec=dict(_ML4_8, **_ML4_CALLS["k_b_rope"], tail_from=_NVFP4_ROWS - 15), + ), + Cell( + why="tep8 8 heads K-B NoPE (k 128), a mixed step's 15 generation rows: " + "rows 32753..32767 of 32768-row buffers, a nonzero storage offset", + spec=dict(_ML4_8, **_ML4_CALLS["k_b_nope"], tail_from=_NVFP4_ROWS - 15), + ), + Cell( + why="tep8 8 heads V-B (transposed-view b), a mixed step's 15 generation " + "rows: rows 32753..32767 of 32768-row buffers, a nonzero storage offset", + spec=dict(_ML4_8, **_ML4_CALLS["v_b"], tail_from=_NVFP4_ROWS - 15), + ), + # TEP8 and TP8 on the fp8_ds_mla latent KV (PrefillTarget._attention and + # DecodeTarget._attention, one call over every row of the step): the two K-B + # calls into heads 0..7 of the 64-head fused query, so the out token stride + # is 36864 where the `_ML4_8` cells' is 4608 (V-B reads the compact latent + # output, as above). Row counts at both edges of every 16-row window (a + # 64-head row is eight times the 8-head one, so a sweep resetting whole rows + # at every m would not fit a cell's time), CUDA graphs at every decode batch + # 1..16 and a mixed step's tail: + Cell( + why="tep8 K-B RoPE (k 64) into heads 0..7 of the 64-head fused query, m 1..16 " + "and both edges of every 16-row window to 32768: each m's rows bitwise the " + "leading rows of the 32768-row run, which is gated; heads 8..63 untouched", + spec=dict(_ML4_8_FQ64, **_ML4_CALLS["k_b_rope"], m_window_edges=(1, _NVFP4_ROWS, 16)), + ), + Cell( + why="tep8 K-B NoPE (k 128) into heads 0..7 of the 64-head fused query, m 1..16 " + "and both edges of every 16-row window to 32768: each m's rows bitwise the " + "leading rows of the 32768-row run, which is gated; heads 8..63 untouched", + spec=dict(_ML4_8_FQ64, **_ML4_CALLS["k_b_nope"], m_window_edges=(1, _NVFP4_ROWS, 16)), + ), + Cell( + why="tep8 K-B RoPE (k 64) into the 64-head fused query, CUDA graphs at every " + "decode batch 1..16: replays bitwise equal to eager, heads 8..63 untouched", + spec=dict(_ML4_8_FQ64, **_ML4_CALLS["k_b_rope"], graph_m=tuple(range(1, 17))), + ), + Cell( + why="tep8 K-B NoPE (k 128) into the 64-head fused query, CUDA graphs at every " + "decode batch 1..16: replays bitwise equal to eager, heads 8..63 untouched", + spec=dict(_ML4_8_FQ64, **_ML4_CALLS["k_b_nope"], graph_m=tuple(range(1, 17))), + ), + Cell( + why="tep8 K-B RoPE (k 64) into the 64-head fused query, a mixed step's 15 " + "generation rows: rows 32753..32767 of 32768-row buffers", + spec=dict(_ML4_8_FQ64, **_ML4_CALLS["k_b_rope"], tail_from=_NVFP4_ROWS - 15), + ), + Cell( + why="tep8 K-B NoPE (k 128) into the 64-head fused query, a mixed step's 15 " + "generation rows: rows 32753..32767 of 32768-row buffers", + spec=dict(_ML4_8_FQ64, **_ML4_CALLS["k_b_nope"], tail_from=_NVFP4_ROWS - 15), + ), + # ... and every row count 1..32768: at max_seq_len 1048576 with chunked + # prefill (max_num_tokens 32768) a step is any mix of chunk rows -- a + # request's last chunk is its prompt length mod 32768 -- and up to 16 decode + # rows, so every m to max_num_tokens occurs. The sweep resets and compares + # only the call's 8 heads of each row, which keeps it within a cell's time. + Cell( + why="tep8 fused q64 K-B RoPE (k 64), heads 0..7 of [T, 64, 576], every m " + "1..32768: each m's rows bitwise the leading rows of the 32768-row run, " + "which is gated; heads 8..63 never written (~5 s)", + spec=dict(_ML4_8_FQ64, **_ML4_CALLS["k_b_rope"], m_range=(1, _NVFP4_ROWS)), + ), + Cell( + why="tep8 fused q64 K-B NoPE (k 128), heads 0..7 of [T, 64, 576], every m " + "1..32768: each m's rows bitwise the leading rows of the 32768-row run, " + "which is gated; heads 8..63 never written (~5 s)", + spec=dict(_ML4_8_FQ64, **_ML4_CALLS["k_b_nope"], m_range=(1, _NVFP4_ROWS)), + ), +) + +# Mistral Large 4 NVFP4 at DEP4 and DEP8: all 64 heads on every rank. +_ML4_DP_CELLS: tuple[Cell, ...] = ( + *_ml4_dspark(64), + # DEP4: the same three calls at batch 64, every rank holding all heads + # and running only its own requests, so m is the rank's own token count: + # up to 32768 in a context step (no chunked prefill), 1/2/4/8/16 in the + # captured decode batches, and a mixed step's generation rows behind up + # to 32753 context rows. Every m 1..32768 is four times the 16-head + # sweep's bytes and past a cell's time, so those three cells are + # `large`; pre-merge drives every m where cuBLAS's choice changes. + Cell( + why="dep4 64 heads K-B RoPE (k 64), every m 1..32768: each m's rows " + "bitwise the leading rows of the 32768-row run, which is gated " + "(large: ~27 s)", + spec=dict(_ML4_64, **_ML4_CALLS["k_b_rope"], m_range=(1, _NVFP4_ROWS), large=True), + ), + Cell( + why="dep4 64 heads K-B NoPE (k 128), every m 1..32768: each m's rows " + "bitwise the leading rows of the 32768-row run, which is gated " + "(large: ~27 s)", + spec=dict(_ML4_64, **_ML4_CALLS["k_b_nope"], m_range=(1, _NVFP4_ROWS), large=True), + ), + Cell( + why="dep4 64 heads V-B (transposed-view b), every m 1..32768: each m's " + "rows bitwise the leading rows of the 32768-row run, which is gated " + "(large: ~12 s)", + spec=dict(_ML4_64, **_ML4_CALLS["v_b"], m_range=(1, _NVFP4_ROWS), large=True), + ), + Cell( + why="dep4 64 heads K-B RoPE (k 64), edges of every 16-row window to " + "32768 and every m 1..16, the only places its kernel and grid change: " + "bitwise the leading rows of the gated 32768-row run (~4 s)", + spec=dict(_ML4_64, **_ML4_CALLS["k_b_rope"], m_window_edges=(1, _NVFP4_ROWS, 16)), + ), + Cell( + why="dep4 64 heads K-B NoPE (k 128), edges of every 16-row window to " + "32768 and every m 1..16, the only places its kernel and grid change: " + "bitwise the leading rows of the gated 32768-row run (~4 s)", + spec=dict(_ML4_64, **_ML4_CALLS["k_b_nope"], m_window_edges=(1, _NVFP4_ROWS, 16)), + ), + Cell( + why="dep4 64 heads V-B (transposed-view b), edges of every 16-row window " + "to 32768 and every m 1..16, the only places its kernel and grid change: " + "bitwise the leading rows of the gated 32768-row run (~2 s)", + spec=dict(_ML4_64, **_ML4_CALLS["v_b"], m_window_edges=(1, _NVFP4_ROWS, 16)), + ), + Cell( + why="dep4 64 heads K-B RoPE (k 64), CUDA graphs at the captured, padded " + "decode batches 1/2/4/8/16: replays bitwise equal to eager", + spec=dict(_ML4_64, **_ML4_CALLS["k_b_rope"], graph_m=(1, 2, 4, 8, 16)), + ), + Cell( + why="dep4 64 heads K-B NoPE (k 128), CUDA graphs at the captured, padded " + "decode batches 1/2/4/8/16: replays bitwise equal to eager", + spec=dict(_ML4_64, **_ML4_CALLS["k_b_nope"], graph_m=(1, 2, 4, 8, 16)), + ), + Cell( + why="dep4 64 heads V-B (transposed-view b), CUDA graphs at the captured, " + "padded decode batches 1/2/4/8/16: replays bitwise equal to eager", + spec=dict(_ML4_64, **_ML4_CALLS["v_b"], graph_m=(1, 2, 4, 8, 16)), + ), + Cell( + why="dep4 64 heads K-B RoPE (k 64), a mixed step's 15 generation rows: " + "rows 32753..32767 of 32768-row buffers, a nonzero storage offset", + spec=dict(_ML4_64, **_ML4_CALLS["k_b_rope"], tail_from=_NVFP4_ROWS - 15), + ), + Cell( + why="dep4 64 heads K-B NoPE (k 128), a mixed step's 15 generation rows: " + "rows 32753..32767 of 32768-row buffers, a nonzero storage offset", + spec=dict(_ML4_64, **_ML4_CALLS["k_b_nope"], tail_from=_NVFP4_ROWS - 15), + ), + Cell( + why="dep4 64 heads V-B (transposed-view b), a mixed step's 15 generation " + "rows: rows 32753..32767 of 32768-row buffers, a nonzero storage offset", + spec=dict(_ML4_64, **_ML4_CALLS["v_b"], tail_from=_NVFP4_ROWS - 15), + ), +) + # The shapes each shipped target calls this entry with, by target directory # under `models/`. The entry's own `CELLS` are the op's whoever calls it. TARGET_CELLS: dict[str, tuple[Cell, ...]] = { @@ -77,32 +765,105 @@ def _build(spec, seed): spec=dict(batch=64, m=2048, k=128, n=512, dtype=torch.bfloat16, transposed_out=True), ), ), + "mistral_large4/mistral_large_4_nvfp4__sm_103__tep": (*_ML4_TP4_CELLS, *_ML4_TP8_CELLS), + "mistral_large4/mistral_large_4_nvfp4__sm_103__tp": (*_ML4_TP4_CELLS, *_ML4_TP8_CELLS), + "mistral_large4/mistral_large_4_nvfp4__sm_103__dep": _ML4_DP_CELLS, } CELLS = certified_cells(op, TARGET_CELLS) +def _drive(cell) -> None: + spec = cell.spec + seed = zlib.crc32(cell.why.encode()) % 2**31 + if "transposed_out" in spec: + _transposed_single(spec, seed) + elif "m_range" in spec: + lo, hi = spec["m_range"] + _sweep(spec, seed, range(lo, hi + 1)) + elif "m_window_edges" in spec: + _sweep(spec, seed, _window_edges(*spec["m_window_edges"])) + elif "graph_m" in spec: + _replays(spec, seed) + elif "tail_from" in spec: + _tail(spec, seed) + else: + _single(spec, seed) + + @pytest.mark.parametrize("cell", CELLS, ids=[c.why[:44] for c in CELLS]) -def test_certified_cells(cell) -> None: - a, w, out, parent = _build(cell.spec, seed=zlib.crc32(cell.why.encode()) % 2**31) - parent_ptr = parent.data_ptr() - with validating(op): - # Identity, not equality: the op must have written this very buffer and - # returned it, not produced a copy that happens to match. - assert op(a, w, out) is out, "the op writes in place and returns what it wrote" - # The product has to be in the caller's buffer, not in a replacement the op - # allocated: that is the whole reason this op exists over torch.bmm. - assert parent.data_ptr() == parent_ptr - _compare(out, op.reference(a, w, out)) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — the fixture, by name + if cell.spec.get("large"): + pytest.skip("past the per-cell budget; test_large_certified_cells drives it") + certifying(op) + _drive(cell) + + +_LARGE_CELLS = tuple(c for c in CELLS if c.spec.get("large")) + + +@pytest.mark.large +@pytest.mark.parametrize("cell", _LARGE_CELLS, ids=[c.why[:44] for c in _LARGE_CELLS]) +def test_large_certified_cells(cell, certifying) -> None: # noqa: F811 — the fixture, by name + certifying(op) + _drive(cell) + + +def _ml4_spec(k: int, mode: str, batch: int = 16) -> dict: + """The Mistral Large 4 cell of K width `k` and `batch` heads whose spec carries + the key `mode`.""" + return next( + dict(c.spec) + for c in CELLS + if c.spec["batch"] == batch and c.spec["k"] == k and mode in c.spec + ) def _ok(dtype=torch.bfloat16): - a = torch.randn(4, 8, 16, dtype=dtype, device="cuda") - w = torch.randn(4, 16, 32, dtype=dtype, device="cuda") - out = torch.zeros(4, 8, 32, dtype=dtype, device="cuda") + a = torch.randn(4, 8, 16, dtype=dtype, device=DEV) + w = torch.randn(4, 16, 32, dtype=dtype, device=DEV) + out = torch.zeros(4, 8, 32, dtype=dtype, device=DEV) return a, w, out +def test_a_wrong_shaped_out_is_resized_and_the_guard_refuses_it() -> None: + """The op resizes a wrong-shaped `out` in place, with only a deprecation + warning. On Mistral Large 4's K-B view one token short, the view is re-strided + contiguous over the fused query's storage: the product is there, but not + in the buffer's layout, and it overwrites the rope halves. A standalone + `out` is re-allocated, detaching the caller's buffer.""" + spec = _ml4_spec(64, "tail_from") + assert spec["out_row"] == 576 + a, w, out, _, parent = _build(spec, seed=21, m=16) + ref = op.reference(a, w, out) + short = out[:, :15] + with pytest.warns(UserWarning, match="was resized"): + op(a, w, short) + assert short.shape == (16, 16, 512) and short.is_contiguous() + _compare(short, ref) + assert int((_bits(parent[..., 512:]) != SENT).sum()) > 0, "the rope halves were overwritten" + in_layout = int((_bits(parent[..., :512].transpose(0, 1)) == _bits(ref)).sum()) + assert in_layout < ref.numel() // 10, f"{in_layout} of {ref.numel()} in the buffer's layout" + + standalone = torch.zeros(16, 16, 511, dtype=torch.bfloat16, device=DEV) + ptr = standalone.data_ptr() + with pytest.warns(UserWarning, match="was resized"): + op(a, w, standalone) + assert standalone.shape == (16, 16, 512) and standalone.data_ptr() != ptr + + a, w, out, _, parent = _build(spec, seed=22, m=16) + for wrong, refusal in ( + (out[:, :15], r"out must be \[B, M, N\]"), + (out[..., :511], r"out must be \[B, M, N\]"), + (out[0], "a, b, out must be 3D"), + (torch.zeros_like(out)[:15], r"out must be \[B, M, N\]"), + ): + with pytest.raises(AssertionError, match=refusal): + with validating(op): + op(a, w, wrong) + assert bool((_bits(parent) == SENT).all()) + + def test_guard_refuses_a_wrong_shaped_out() -> None: """The op silently reallocates it, detaching the caller's buffer.""" a, w, _ = _ok() @@ -124,3 +885,160 @@ def test_guard_refuses_mixed_dtypes() -> None: with pytest.raises(AssertionError, match="share one dtype"): with validating(op): op(a, w.to(torch.float16), out) + + +@pytest.mark.parametrize( + "case, match", + [ + ("2d_a", "batch1 must be a 3D tensor"), + ("2d_b", "batch2 must be a 3D tensor"), + ("k_mismatch", r"Expected size for first two dimensions of batch2 tensor"), + ("batch_mismatch", r"Expected size for first two dimensions of batch2 tensor"), + ("bf16_fp16_into_bf16", r"Expected out tensor to have dtype c10::Half"), + ("bf16_fp16_into_fp16", "expected scalar type BFloat16 but found Half"), + ("bf16_into_fp32", r"Expected out tensor to have dtype c10::BFloat16, but got float"), + ], +) +def test_the_op_refuses_the_rest_itself(case, match) -> None: + """What the op refuses itself, unguarded: every one of these is refused + before anything is written. The guard restates the 3-D and single-dtype + checks (the `test_guard_refuses_*` cases).""" + a, w, out = _ok() + out.fill_(7.0) + args = { + "2d_a": (a[0], w, out), + "2d_b": (a, w[0], out), + "k_mismatch": (a, w[:, :15], out), + "batch_mismatch": (a, w[:3], out), + "bf16_fp16_into_bf16": (a, w.half(), out), + "bf16_fp16_into_fp16": (a, w.half(), out.half()), + "bf16_into_fp32": (a, w, out.float()), + }[case] + with pytest.raises(RuntimeError, match=match): + op(*args) + assert bool((args[2] == 7.0).all()) + + +def test_zero_rows_write_nothing() -> None: + spec = _ml4_spec(64, "tail_from") + a, w, out, _, parent = _build(spec, seed=23, m=4) + with validating(op): + result = op(a[:, :0], w, out[:, :0]) + assert result.shape == (16, 0, 512) + assert bool((_bits(parent) == SENT).all()) + + +@pytest.mark.parametrize("batch", [8, 16, 64]) +def test_a_transposed_b_view_matches_the_contiguous_b(batch) -> None: + """The V-B `b` as the targets hold it -- a transposed view of a weight's + row slice -- and the same values contiguous: bitwise the same product, + although cuBLAS runs a different kernel for each. At TEP8's and TP8's 8 + heads, TEP4's 16 and DEP4's 64.""" + spec = _ml4_spec(512, "tail_from", batch) + assert spec["b_layout"] == "nk_t" + for m in (1, 16, 9216, 32768): + a, w, out, _, _ = _build(spec, seed=24 + m, m=m) + assert w.stride(1) == 1 and w.storage_offset() > 0 + contiguous = torch.empty_like(out) + op(a, w, out) + op(a, w.contiguous(), contiguous) + assert torch.equal(_bits(out), _bits(contiguous)), f"m={m}" + + +@pytest.mark.parametrize("shift", [1, 48]) +def test_fused_q64_harness_sees_a_write_past_the_batch_heads(shift) -> None: + """The control for the `out_heads` assertion: the same product written + `shift` heads over in the 64-head buffer (heads 1..16, and the last 16, + 48..63) must fail it, and the product at heads 0..15 must pass it. Without + this, "heads 16..63 never written" could be a check that cannot fire.""" + spec = _ml4_spec(64, "out_heads") + assert spec["out_heads"] == 64 and spec["batch"] == 16 + a, w, out, _, parent = _build(spec, seed=25, m=16) + op(a, w, out) + _other_heads_untouched(spec, parent) + op(a, w, parent[:, shift : shift + 16, :512].transpose(0, 1)) + with pytest.raises(AssertionError, match="past the call's batch"): + _other_heads_untouched(spec, parent) + + +def test_fused_q64_sweep_sees_a_shorter_call_writing_past_the_batch_heads(monkeypatch) -> None: + """The control for the sweep's own-heads reset: heads 16..63 are never reset + or compared per call, so a write there must still fail the sweep. The op + is wrapped so that one shorter call (17 of 64 rows) also copies its head 0 + into head 16; every per-call compare of the batch's heads still passes, + and the sweep must fail on the whole-parent comparison. The same sweep + with the op itself passes.""" + spec = _ml4_spec(64, "out_heads") + assert "m_range" in spec and spec["out_heads"] == 64 and spec["batch"] == 16 + real, at, bsz = op, 17, spec["batch"] + + class _StrayHead: + def __call__(self, a, b, out): + real(a, b, out) + if out.shape[1] == at: + head0 = out[:1] + stray = torch.as_strided( + head0, head0.shape, head0.stride(), out.storage_offset() + bsz * out.stride(0) + ) + stray.copy_(head0) + return out + + def reference(self, a, b, out): + return real.reference(a, b, out) + + monkeypatch.setattr(sys.modules[__name__], "op", _StrayHead()) + with pytest.raises(AssertionError, match="past a call's view were written"): + _sweep(spec, seed=26, counts=range(1, 65)) + monkeypatch.undo() + _sweep(spec, seed=26, counts=range(1, 65)) + + +@pytest.mark.parametrize("shift", [1, 56]) +def test_fused_q64_harness_sees_a_write_past_the_batch_heads_at_8_heads(shift) -> None: + """The control for the `out_heads` assertion at TEP8's 8 heads: the same + product written `shift` heads over in the 64-head buffer (heads 1..8, and + the last 8, 56..63) must fail it, and the product at heads 0..7 must pass + it. Without this, "heads 8..63 never written" could be a check that + cannot fire.""" + spec = _ml4_spec(64, "out_heads", batch=8) + assert spec["out_heads"] == 64 and spec["batch"] == 8 + a, w, out, _, parent = _build(spec, seed=25, m=16) + op(a, w, out) + _other_heads_untouched(spec, parent) + op(a, w, parent[:, shift : shift + 8, :512].transpose(0, 1)) + with pytest.raises(AssertionError, match="past the call's batch"): + _other_heads_untouched(spec, parent) + + +def test_fused_q64_sweep_sees_a_shorter_call_writing_past_the_batch_heads_at_8_heads( + monkeypatch, +) -> None: + """The control for the sweep's own-heads reset at TEP8's 8 heads: heads + 8..63 are never reset or compared per call, so a write there must still + fail the sweep. The op is wrapped so that one shorter call (17 of 64 rows) + also copies its head 0 into head 8; every per-call compare of the batch's + heads still passes, and the sweep must fail on the whole-parent + comparison. The same sweep with the op itself passes.""" + spec = _ml4_spec(64, "out_heads", batch=8) + assert spec["out_heads"] == 64 and spec["batch"] == 8 + real, at, bsz = op, 17, spec["batch"] + + class _StrayHead: + def __call__(self, a, b, out): + real(a, b, out) + if out.shape[1] == at: + head0 = out[:1] + stray = torch.as_strided( + head0, head0.shape, head0.stride(), out.storage_offset() + bsz * out.stride(0) + ) + stray.copy_(head0) + return out + + def reference(self, a, b, out): + return real.reference(a, b, out) + + monkeypatch.setattr(sys.modules[__name__], "op", _StrayHead()) + with pytest.raises(AssertionError, match="past a call's view were written"): + _sweep(spec, seed=26, counts=range(1, 65)) + monkeypatch.undo() + _sweep(spec, seed=26, counts=range(1, 65)) diff --git a/tests/unittest/_torch/modeling_v2/gemm/test_modeling_v2_cublas_mm.py b/tests/unittest/_torch/modeling_v2/gemm/test_modeling_v2_cublas_mm.py index e5d4d9bbe225..2fab114ba661 100644 --- a/tests/unittest/_torch/modeling_v2/gemm/test_modeling_v2_cublas_mm.py +++ b/tests/unittest/_torch/modeling_v2/gemm/test_modeling_v2_cublas_mm.py @@ -12,39 +12,89 @@ __extra_import_path__ = [".."] from _cells import certified_cells # noqa: E402 — needs the path declared above -from _validating import ( # noqa: E402 — needs the path declared above +from _validating import ( # noqa: E402,F401 — needs the path declared above ULP, assert_within_ulp, + certifying, validating, ) assert torch.cuda.is_available(), "cublas_mm requires a CUDA device" +def _band(out_dtype: torch.dtype) -> tuple[float, float, float]: + """(element_ulp, rms_ulp, ulp) for an output of `out_dtype`; derived in `_compare`.""" + if out_dtype is torch.float32: + return 512.0, 256.0, ULP[torch.float32] + return 4.0, 2.0, ULP[torch.bfloat16] + + def _compare(out: torch.Tensor, ref: torch.Tensor) -> None: - """One GEMM's worth of accumulation-order difference. + """One GEMM's worth of accumulation-order difference, in the output's own ulp. Kernel and reference consume bit-identical operands and differ only in the order cuBLAS and torch sum over K. Over random operands a few outputs land near zero, where that reordering shows up as a relative difference of 10 on an absolute difference of 1.5e-4 -- which is what - torch's default band rejects, and what no bf16 result could represent - anyway: one ulp at these rows' scale is about 6e-1. + torch's default band rejects. So both gates are against the row's scale + (`assert_within_ulp`): element-wise against each row's largest magnitude, + and the relative RMS. + + bf16 out: 4 ulp element-wise, 2 ulp of relative RMS. The two fp32 sums + differ in their last bits, and where that straddles a bf16 rounding + boundary they round to neighbouring values: one bf16 spacing, which is + between 1 and 2 ulp of the row's scale as `ULP` counts it. Measured worst + case over every m in 1..9216 at k 6144, n 576 / 512 / 160: 1.98 ulp + element-wise and 0.053 RMS. A wrong product -- the weight's two K halves + swapped -- lands at 310-640 ulp element-wise and 330-370 RMS. An error + below one bf16 spacing is below what a bf16 result can show: dropping one + of the 6144 K terms lands at 1.6-15 ulp element-wise, 1.5-6.8 RMS. Over + every m in 1..32768 the worst case is 1.97 and 0.081; at a 32768-row call + the swapped halves land at 596-688 and 362, one dropped K term at 12.6-13.8 + and 3.2-3.3. + + fp32 out (the routers' logits): 512 ulp element-wise, 256 RMS, in fp32's + ulp. bf16 products are exact in fp32 and the sum is fp32, so the result + is held to fp32's resolution: scored in bf16 ulp instead, the same logits + rounded through bf16 and widened back (0.6-1.0 bf16 ulp) would pass for + them. Measured worst case over every m in 1..9216 at k 6144, n 128: 122 + ulp element-wise and 57 RMS; rounded through bf16 they land at 2.2e4-3.3e4 + and 1.4e4-1.6e4, 42x and 54x outside the gate. Over every m in 1..32768: + 121 and 57; at a 32768-row call, 124 and 57, and rounded through bf16 + 3.3e4 and 1.4e4. Over the gathered multiples of 4 in 4..131072, with zero + padding rows: 132.9 and 57.2; at a 131072-row call on other operands, + 126.5 and 57.1, where rounded through bf16 lands at 3.25e4 and 1.39e4, + one dropped K term at 5.45e5 and 1.05e5, and the weight's K halves + swapped at 2.41e7 and 1.19e7. + """ + element_ulp, rms_ulp, ulp = _band(out.dtype) + assert_within_ulp(out, ref, element_ulp=element_ulp, rms_ulp=rms_ulp, ulp=ulp) + + +def _padding_rows(t: torch.Tensor, period: int) -> torch.Tensor: + """The zero padding rows of a `zero_row_period` cell's operand or output: every + `period`-th row (i % period == period - 1), a view. + + At period 3, every tile of every kernel the heuristic picks holds some, + whatever its extent over the rows. Where they sit in a gathered step -- + the tail of each rank's T_max block -- does not matter to a GEMM, whose + rows are independent; the entry's note records the whole-block skew too. + """ + return t[period - 1 :: period] + + +def _padding_logits_not_zero(out: torch.Tensor, period: int) -> torch.Tensor: + """On the device: whether any padding row's output is anything but +0.0. - So: 4 ulp element-wise against the row scale, 2 ulp of relative RMS. - Measured worst case across this entry's cells is 1.5e-4 absolute, or - roughly 3e-4 ulp -- four orders of margin, and a band that a genuinely - wrong product would miss by orders rather than squeak past. + Read as integer bits, since -0.0 compares equal to +0.0 and only the + bits tell them apart. """ - # Scored in bf16 ulp even when the output is fp32: every cell's - # operands are bf16, so the fp32 out_dtype the router asks for is a - # wider container for a bf16-limited product, not a more precise one. - # Against fp32's own ulp the same 2.4e-6 difference reads as 5x over. - assert_within_ulp(out, ref, element_ulp=4.0, rms_ulp=2.0, ulp=ULP[torch.bfloat16]) + bits = torch.int32 if out.element_size() == 4 else torch.int16 + return _padding_rows(out, period).view(bits).any() -def _build(spec, seed): - """Materialize one cell. +def _build(spec, seed, m=None): + """Materialize one cell's operands, at `m` rows or the cell's largest. `mat_b` is built as [N, K] and transposed, because that is the one thing a spec cannot say: the op reads it column-major, and a stored weight's `.t()` @@ -52,9 +102,13 @@ def _build(spec, seed): `rows`, the weight is `[rows, K]` and `mat_b` the first N columns of its `.t()` (column stride still K); rows from `live` on are zero. """ + if m is None: + m = spec["m_range"][1] if "m_range" in spec else spec["m"] g = torch.Generator(device="cuda").manual_seed(seed) - m, k, n, dt = spec["m"], spec["k"], spec["n"], spec["dtype"] + k, n, dt = spec["k"], spec["n"], spec["dtype"] mat_a = torch.randn(m, k, generator=g, device="cuda").to(dt) + if "zero_row_period" in spec: + _padding_rows(mat_a, spec["zero_row_period"]).zero_() w = torch.randn(spec.get("rows", n), k, generator=g, device="cuda").to(dt) if "live" in spec: w[spec["live"] :] = 0 @@ -66,8 +120,395 @@ def _build(spec, seed): return mat_a, mat_b, bias, out_dtype +def _sweep(spec, seed) -> None: + """Every m of `m_range` (every `m_step`-th one when the cell names a step), each + its own call, on the leading m rows of one operand set. + + The reference is taken once, at the top of the range, and sliced: row i of + a matmul reads only row i of `mat_a`, so its leading m rows are the + reference for an m-row call. The gates are `assert_within_ulp`'s two, per + m, evaluated on the device so the sweep pays one synchronization rather + than one per m. + + Per m, the difference is formed once, in fp32, and read twice by two + reductions: each row's largest error and the error's sum of squares. That + sum accumulates in fp32, which resolves it to about 1e-6 of itself, far + inside either gate; the reference's own sum of squares is fp64. + + A `zero_row_period` cell also holds every m's padding rows to +0.0 bit + for bit, which the band cannot see: their reference is +0.0, so -0.0 + sits at zero distance from it. + """ + lo, hi = spec["m_range"] + step = spec.get("m_step", 1) + ms = range(-(-lo // step) * step, hi + 1, step) + period = spec.get("zero_row_period") + mat_a, mat_b, bias, out_dtype = _build(spec, seed) + want_dtype = out_dtype or mat_a.dtype + ref = op.reference(mat_a, mat_b, bias, out_dtype).float() + row_scale = ref.abs().amax(dim=1).clamp_min(1e-9) + ref_sq = ref.double().pow(2).sum(dim=1).cumsum(0) + element = torch.empty(len(ms), device="cuda") + diff_sq = torch.empty(len(ms), device="cuda") + not_zero = torch.zeros(len(ms), dtype=torch.bool, device="cuda") + for i, m in enumerate(ms): + out = op(mat_a[:m], mat_b, bias, out_dtype) + assert out.shape == (m, mat_b.shape[1]) and out.dtype is want_dtype + if period: + not_zero[i] = _padding_logits_not_zero(out, period) + diff = torch.sub(out, ref[:m]) + element[i] = (torch.linalg.vector_norm(diff, float("inf"), dim=1) / row_scale[:m]).amax() + diff_sq[i] = torch.linalg.vector_norm(diff).square() + rms = (diff_sq / ref_sq[ms.start - 1 : hi : step].clamp_min(1e-18)).sqrt() + element_ulp, rms_ulp, ulp = _band(want_dtype) + worst = int(element.argmax()) + assert element[worst] <= element_ulp * ulp, ( + f"m={ms[worst]}: {element[worst].item() / ulp:.3f} ulp element-wise exceeds {element_ulp}" + ) + worst = int(rms.argmax()) + assert rms[worst] <= rms_ulp * ulp, ( + f"m={ms[worst]}: relative RMS {rms[worst].item() / ulp:.3f} ulp exceeds {rms_ulp}" + ) + bad = not_zero.nonzero() + assert bad.numel() == 0, f"m={ms[int(bad[0])]}: a zero padding row's output is not +0.0" + + +def _replays(spec, seed) -> None: + """Each captured batch in its own CUDA graph, replayed on new data. + + The replay has to equal an eager call on the same data bit for bit -- a + graph that froze something of the capture would differ -- and is gated + against the reference like any other output. Each capture is preceded by + one call on the capture stream, as an engine's warm-up is: the thop + caller creates its cuBLASLt workspace on a stream's first call. A + `zero_row_period` cell's new data carries its padding rows too, and the + replay's logits for them are +0.0 bit for bit. + """ + stream = torch.cuda.Stream() + g = torch.Generator(device="cuda").manual_seed(seed) + period = spec.get("zero_row_period") + for m in spec["graph_m"]: + mat_a, mat_b, bias, out_dtype = _build(spec, seed + m, m=m) + stream.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(stream): + op(mat_a, mat_b, bias, out_dtype) + torch.cuda.current_stream().wait_stream(stream) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=stream): + out = op(mat_a, mat_b, bias, out_dtype) + fresh = torch.randn(mat_a.shape, generator=g, device="cuda").to(mat_a.dtype) + if period: + _padding_rows(fresh, period).zero_() + mat_a.copy_(fresh) + graph.replay() + eager = op(fresh, mat_b, bias, out_dtype) + assert torch.equal(out, eager), f"m={m}: graph replay differs from eager" + if period: + assert not _padding_logits_not_zero(out, period), f"m={m}: padding row not +0.0" + _compare(out, op.reference(fresh, mat_b, bias, out_dtype)) + + # The shapes each shipped target calls this entry with, by target directory # under `models/`. The entry's own `CELLS` are the op's whoever calls it. +# +# Mistral Large 4 NVFP4 at tep4 and tp4 calls it for its four unquantized +# projections, each replicated on every rank, so k and n are the checkpoint's +# own: kv_a_proj_with_mqa (576 on the RoPE layers, 512 on the NoPE layers that +# own a latent), the indexer's wk and weights_proj stacked into one 160-wide +# weight, and the router (128 experts, fp32 logits). Each call site passes one +# row per token: decode is captured at the batches the engine pads to (1, 2, 4, +# 8, 16), prefill is eager at any row count up to the step's token budget -- +# 32768 (max_num_tokens 32768 with chunked prefill off, so one context step +# carries up to 32768 rows); the 1..9216 cells are the smaller budget the same +# shapes ran at first. cuBLASLt's heuristic picks the kernel by m, at a grain +# finer than any set of edge cells could follow (see the entry's `note`), so the +# eager cells drive every m in the range, each as its own call. A 1..32768 +# sweep costs 4-5.5 s on sm_103, the 1..9216 one ~1.3 s. The dep4 deployment runs +# the same three attention projections on each rank's own tokens. +_ML4_PROJECTIONS: tuple[Cell, ...] = ( + Cell( + why="eager m 1..9216, kv_a on the RoPE layers: hidden 6144 -> [c_kv 512 | k_pe 64]; " + "prefill at every row count up to the token budget, so every heuristic bucket", + spec=dict( + m_range=(1, 9216), k=6144, n=576, dtype=torch.bfloat16, bias=False, out_dtype=None + ), + ), + Cell( + why="eager m 1..9216, kv_a on the NoPE layers that own a latent: hidden 6144 -> c_kv " + "512, no k_pe", + spec=dict( + m_range=(1, 9216), k=6144, n=512, dtype=torch.bfloat16, bias=False, out_dtype=None + ), + ), + Cell( + why="eager m 1..9216, indexer wk + weights_proj stacked: hidden 6144 -> [k 128 | head " + "weights 32]", + spec=dict( + m_range=(1, 9216), k=6144, n=160, dtype=torch.bfloat16, bias=False, out_dtype=None + ), + ), + Cell( + why="eager m 1..9216, router: hidden 6144 -> 128 experts, bf16 in, fp32 logits out", + spec=dict( + m_range=(1, 9216), + k=6144, + n=128, + dtype=torch.bfloat16, + bias=False, + out_dtype=torch.float32, + ), + ), + Cell( + why="eager m 1..32768, kv_a on the RoPE layers (576): the NVFP4 target's prefill at every " + "row count up to max_num_tokens 32768, so every heuristic bucket up to the 32768-row step", + spec=dict( + m_range=(1, 32768), k=6144, n=576, dtype=torch.bfloat16, bias=False, out_dtype=None + ), + ), + Cell( + why="eager m 1..32768, kv_a on the NoPE owners (512): every row count up to " + "max_num_tokens 32768, so every heuristic bucket up to the 32768-row step", + spec=dict( + m_range=(1, 32768), k=6144, n=512, dtype=torch.bfloat16, bias=False, out_dtype=None + ), + ), + Cell( + why="eager m 1..32768, indexer wk + weights_proj (160): every row count up to " + "max_num_tokens 32768, so every heuristic bucket up to the 32768-row step", + spec=dict( + m_range=(1, 32768), k=6144, n=160, dtype=torch.bfloat16, bias=False, out_dtype=None + ), + ), + Cell( + why="eager m 1..32768, router (128, fp32 out): every row count up to max_num_tokens " + "32768, so every heuristic bucket up to the 32768-row step", + spec=dict( + m_range=(1, 32768), + k=6144, + n=128, + dtype=torch.bfloat16, + bias=False, + out_dtype=torch.float32, + ), + ), + Cell( + why="CUDA graphs 1/2/4/8/16, kv_a RoPE layers (576): the captured decode batches, " + "replayed on new data bitwise equal to eager", + spec=dict( + graph_m=(1, 2, 4, 8, 16), + k=6144, + n=576, + dtype=torch.bfloat16, + bias=False, + out_dtype=None, + ), + ), + Cell( + why="CUDA graphs 1/2/4/8/16, kv_a NoPE layers (512): the captured decode batches, " + "replayed on new data bitwise equal to eager", + spec=dict( + graph_m=(1, 2, 4, 8, 16), + k=6144, + n=512, + dtype=torch.bfloat16, + bias=False, + out_dtype=None, + ), + ), + Cell( + why="CUDA graphs 1/2/4/8/16, indexer wk + weights_proj (160): the captured decode " + "batches, replayed on new data bitwise equal to eager", + spec=dict( + graph_m=(1, 2, 4, 8, 16), + k=6144, + n=160, + dtype=torch.bfloat16, + bias=False, + out_dtype=None, + ), + ), + Cell( + why="CUDA graphs 1/2/4/8/16, router (128, fp32 out): the captured decode batches, " + "replayed on new data bitwise equal to eager", + spec=dict( + graph_m=(1, 2, 4, 8, 16), + k=6144, + n=128, + dtype=torch.bfloat16, + bias=False, + out_dtype=torch.float32, + ), + ), +) + +# The NVFP4 checkpoint's attention-DP target (dep4) runs the router over the +# token set gathered from its four ranks: each rank zero-pads its rows to +# T_max = max(all_rank_num_tokens) <= 32768, so the router sees m = 4 * T_max -- +# every multiple of 4 up to 131072 eager, and 4/8/16/32/64 in the decode graphs +# (each rank padded to the captured batch 1/2/4/8/16). Its cells drive every one +# of those m, with zero padding rows in the operand (`zero_row_period`: every +# 3rd row), whose logits must be +0.0 bit for bit -- the exact tie the routing +# op is certified to break. The 32772..131072 sweep costs ~6.5 s and 4.6 GiB on +# sm_103, the 4..32768 one ~1.5 s. +# +# Over eight ranks (dep8) it gathers m = 8 * T_max, and it runs the router once +# per chunk of at most 32768 gathered rows (a contiguous row slice starting at a +# multiple of 32768, so m is a multiple of 8 up to 32768 eager, which the +# 1..32768 router cell and the gathered 4..32768 cell drive). Its decode graphs +# gather m = 8 x b for the captured per-rank batch b: 8/16/32/64/128 at the +# gates' 1/2/4/8/16, every 8 x 1..16 under the LLM API's default list. Its graph +# cell captures each of those. +_ML4_DP_ROUTER: tuple[Cell, ...] = ( + Cell( + why="dep4 router, eager gathered m 32772..131072: every multiple of 4 above 32768 up " + "to 4 x max_num_tokens, so every heuristic bucket a gathered step reaches past the " + "32768 cells, with zero padding rows", + spec=dict( + m_range=(32769, 131072), + m_step=4, + k=6144, + n=128, + dtype=torch.bfloat16, + bias=False, + out_dtype=torch.float32, + zero_row_period=3, + ), + ), + Cell( + why="dep4 router, eager gathered m 4..32768 with zero padding rows: the 1..32768 cell's " + "kernels at every multiple of 4, its padding rows +0.0 bit for bit", + spec=dict( + m_range=(4, 32768), + m_step=4, + k=6144, + n=128, + dtype=torch.bfloat16, + bias=False, + out_dtype=torch.float32, + zero_row_period=3, + ), + ), + Cell( + why="dep4 router, CUDA graphs 4/8/16/32/64: per-rank batches 1/2/4/8/16 gathered over " + "4 ranks, replayed on new data bitwise equal to eager, with zero padding rows", + spec=dict( + graph_m=(4, 8, 16, 32, 64), + k=6144, + n=128, + dtype=torch.bfloat16, + bias=False, + out_dtype=torch.float32, + zero_row_period=3, + ), + ), + Cell( + why="dep8 router, CUDA graphs 8..128 step 8: per-rank batches 1..16 gathered over 8 " + "ranks (8/16/32/64/128 at the captured 1/2/4/8/16), replayed on new data bitwise " + "equal to eager, with zero padding rows", + spec=dict( + graph_m=(8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128), + k=6144, + n=128, + dtype=torch.bfloat16, + bias=False, + out_dtype=torch.float32, + zero_row_period=3, + ), + ), +) + +# DSpark cells add the 30720->6144 target-feature projection through every +# prefill row count to 32768, the BF16 draft's dense and RNN projections, +# and graph rows 1..64. The RNN joint projection has a fused bias at k 7168; +# its output projection reaches n 204800. The DEP router graphs cover +# gathered multiples of four through 512 rows, including padding zeros. +# At k 30720 the prefill operand reaches 1006632960 elements / 2013265920 +# bytes: below the first 32-bit byte offset at 34953 rows, and the first +# 32-bit element offset at 69906 rows. Neither boundary is reached by this +# 32768-row envelope; these two k-30720 boundaries remain unprobed. +_ML4_DSPARK: tuple[Cell, ...] = ( + Cell( + why="DSpark target capture projection, every prefill row 1..32768", + spec=dict( + m_range=(1, 32768), + k=30720, + n=6144, + dtype=torch.bfloat16, + bias=False, + out_dtype=None, + large=True, + ), + ), + *( + Cell( + why=f"DSpark dense projection eager {k}->{n}, bias={bias}", + spec=dict(m_range=(1, 48), k=k, n=n, dtype=torch.bfloat16, bias=bias, out_dtype=None), + ) + for k, n, bias in ( + (6144, 8192, False), + (6144, 2048, False), + (6144, 1024, False), + (8192, 6144, False), + (2048, 6144, False), + (1024, 6144, False), + (6144, 24576, False), + (6144, 6144, False), + (6144, 3072, False), + (24576, 6144, False), + (3072, 6144, False), + (7168, 1536, True), + (512, 204800, False), + ) + ), + *( + Cell( + why=f"DSpark projection graph {k}->{n}, bias={bias}, fp32={fp32}", + spec=dict( + graph_m=tuple(range(1, 65)), + k=k, + n=n, + dtype=torch.bfloat16, + bias=bias, + out_dtype=torch.float32 if fp32 else None, + ), + ) + for k, n, bias, fp32 in ( + (30720, 6144, False, False), + (6144, 8192, False, False), + (6144, 2048, False, False), + (6144, 1024, False, False), + (8192, 6144, False, False), + (2048, 6144, False, False), + (1024, 6144, False, False), + (6144, 24576, False, False), + (6144, 6144, False, False), + (6144, 3072, False, False), + (24576, 6144, False, False), + (3072, 6144, False, False), + (7168, 1536, True, False), + (512, 204800, False, False), + (6144, 576, False, False), + (6144, 512, False, False), + (6144, 160, False, False), + (6144, 128, False, True), + ) + ), +) +_ML4_DSPARK_DP_ROUTER: tuple[Cell, ...] = ( + Cell( + why="DSpark DEP verification router graph, up to 8 ranks x 64 rows", + spec=dict( + graph_m=tuple(range(4, 513, 4)), + k=6144, + n=128, + dtype=torch.bfloat16, + bias=False, + out_dtype=torch.float32, + zero_row_period=3, + ), + ), +) + TARGET_CELLS: dict[str, tuple[Cell, ...]] = { "deepseek_v3/r1_0528_nvfp4__sm_10x__default": ( Cell( @@ -158,21 +599,83 @@ def _build(spec, seed): spec=dict(m=5580, k=1152, n=3456, dtype=torch.bfloat16, bias=True, out_dtype=None), ), ), + "mistral_large4/mistral_large_4_nvfp4__sm_103__tep": (*_ML4_PROJECTIONS, *_ML4_DSPARK), + "mistral_large4/mistral_large_4_nvfp4__sm_103__tp": (*_ML4_PROJECTIONS, *_ML4_DSPARK), + "mistral_large4/mistral_large_4_nvfp4__sm_103__dep": ( + *_ML4_PROJECTIONS, + *_ML4_DP_ROUTER, + *_ML4_DSPARK, + *_ML4_DSPARK_DP_ROUTER, + ), } CELLS = certified_cells(op, TARGET_CELLS) -@pytest.mark.parametrize("cell", CELLS, ids=[c.why[:44] for c in CELLS]) -def test_certified_cells(cell) -> None: - mat_a, mat_b, bias, out_dtype = _build(cell.spec, seed=zlib.crc32(cell.why.encode()) % 2**31) - with validating(op): - out = op(mat_a, mat_b, bias, out_dtype) +def _drive(cell) -> None: + seed = zlib.crc32(cell.why.encode()) % 2**31 + if "m_range" in cell.spec: + _sweep(cell.spec, seed) + return + if "graph_m" in cell.spec: + _replays(cell.spec, seed) + return + mat_a, mat_b, bias, out_dtype = _build(cell.spec, seed) + out = op(mat_a, mat_b, bias, out_dtype) assert out.shape == (mat_a.shape[0], mat_b.shape[1]) assert out.dtype is (out_dtype or mat_a.dtype) _compare(out, op.reference(mat_a, mat_b, bias, out_dtype)) +@pytest.mark.parametrize("cell", CELLS, ids=[c.why[:44] for c in CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 + if cell.spec.get("large"): + pytest.skip("past the per-cell budget; test_large_certified_cells drives it") + certifying(op) + _drive(cell) + + +_LARGE_CELLS = tuple(c for c in CELLS if c.spec.get("large")) + + +@pytest.mark.large +@pytest.mark.parametrize("cell", _LARGE_CELLS, ids=[c.why[:44] for c in _LARGE_CELLS]) +def test_large_certified_cells(cell, certifying) -> None: # noqa: F811 + certifying(op) + _drive(cell) + + +def test_the_fp32_gate_refuses_logits_rounded_through_bf16() -> None: + """The routers ask for fp32 logits; a bf16 result widened back is not one.""" + spec = dict(m=16, k=6144, n=128, dtype=torch.bfloat16, bias=False, out_dtype=torch.float32) + mat_a, mat_b, bias, out_dtype = _build(spec, seed=7) + out = op(mat_a, mat_b, bias, out_dtype) + ref = op.reference(mat_a, mat_b, bias, out_dtype) + _compare(out, ref) + with pytest.raises(AssertionError): + _compare(out.to(torch.bfloat16).to(torch.float32), ref) + + +def test_the_padding_gate_refuses_a_negative_zero() -> None: + """The padding rows' logits are held to +0.0 by their bits; -0.0 passes the band.""" + spec = dict( + m=64, + k=6144, + n=128, + dtype=torch.bfloat16, + bias=False, + out_dtype=torch.float32, + zero_row_period=3, + ) + mat_a, mat_b, bias, out_dtype = _build(spec, seed=11) + out = op(mat_a, mat_b, bias, out_dtype) + assert not _padding_logits_not_zero(out, 3) + flipped = out.clone() + _padding_rows(flipped, 3).neg_() + _compare(flipped, op.reference(mat_a, mat_b, bias, out_dtype)) + assert _padding_logits_not_zero(flipped, 3) + + def _ok(dtype=torch.bfloat16): a = torch.randn(8, 64, dtype=dtype, device="cuda") b = torch.randn(32, 64, dtype=dtype, device="cuda").t() diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/__init__.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/__init__.py new file mode 100644 index 000000000000..46c7fb1a43fa --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/__init__.py @@ -0,0 +1,7 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Package marker. + +These test basenames can repeat other targets'. Without packages, pytest's +default import mode loads both under one module name and refuses the second. +""" diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_chunk_equivalence.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_chunk_equivalence.py new file mode 100644 index 000000000000..0cbeb08449b1 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_chunk_equivalence.py @@ -0,0 +1,279 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Chunked prefill against whole-prompt prefill on Mistral-Large-4-1T-A52B-NVFP4's sm_103 dep target. + +At the checkpoint's own context the target serves prompts longer than one step: the engine +prefills them in chunks of up to max_num_tokens, and every chunk after a request's first is a +context request over a cached prefix -- its rope/append offsets past the prefix, its indexer +selecting over the cached and the new keys (the engine's chunk plan and its full-KV gather +mappings), sparse MLA attending over the pool rows the earlier chunks wrote, layer 35's latent +read by the NoPE layers 39..71 and every shared-indexer layer reusing the selection of the full +one before it. A step may carry another request's decode rows behind its context rows. + +So the same prompts run through two engines, one after the other, and greedy decoding must give +the same tokens: + +* the delivered one, `test_modeling_v2_long_context.py`'s four-rank engine: per rank max_seq_len = + max_num_tokens = 32768, chunked prefill off, so every prompt is prefilled in one step; +* the same engine with max_num_tokens 8192 and chunked prefill on. + +Both hold max_seq_len 32768, so the target runs the same paths in both: every step's context +KV fits its held workspace twice (65536 rows unchunked, twice max_num_tokens; 2**19 chunked, +where a step may attend its requests' cached prefixes), so layer 35's gather is kept for its +sharers; every indexer chunk holds at most 32768 keys and is scored whole, in tiles under the +logits budget of 2**24 both engines' max_seq_len gives; and every MoE layer makes one expert +call (`_EXPERT_CALL_TOKENS`, 131072 rows unchunked; `_EXPERT_CALL_TOKENS_CHUNKED`, 32768 = 4 x +8192 chunked). The paths past these (the decode path past the held workspace, packed indexer +chunks cut at their requests, the larger logits budget and the cut expert calls) are +test_modeling_v2_retrieval_1m.py's on the 1M engine, and test_modeling_v2_ctx_bound.py's, +test_modeling_v2_indexer_packed_chunks.py's and test_modeling_v2_routed_experts.py's on their +pieces. + +Each rank carries its own requests, pinned by `SchedulingParams.attention_dp_rank` and read back +from that rank's own iteration statistics, in two batches: + +* "mixed": rank 0 a ~32000-token prompt (four chunks, two of them 8192 tokens over a cached + prefix of 8192 and 16384), rank 1 ~20000, rank 2 ~9000 (a short chunk over an 8192-token + prefix), rank 3 ~3000 and ~12000 together (the second's last chunk shares its step with the + first's decode row); +* "far": one 26000- to 30000-token prompt per rank, so every rank prefills three full chunks + and its last over a prefix of 24576. + +Every prompt is over `index_topk` (2048), so the indexer selects, and each is a passkey prompt +(the long-context witness's) whose key lies in an earlier chunk than its question (all but rank +3's two in "mixed"), so the answer is read across chunk boundaries. + +The comparison is gated by the engine's own noise. The target's outputs are not bitwise +reproducible from run to run: the same batch run twice in the same whole-prompt engine gives +different log-probabilities at the first generated token and different greedy tokens after the +key, wherever the top two are close. So each prompt's tokens are compared over its leading steps +whose top-1 / top-2 log-probability margin is at least `NOISE` in both engines, and those steps +must hold the key: the key's own tokens carry margins of 3.4 to 12.9 in every run measured, the +token after it 0.0 to 1.1. Every step's margins in both engines are printed, and for a prompt +whose tokens differ later, both engines' top two at the first differing step. + +Runs on four GB300s with the ranks already in place: pytest under `trtllm-llmapi-launch`, with +the checkpoint under `LLM_MODELS_ROOT` as `Mistral-Large-4-1T-A52B-NVFP4` (or +`mistral-large-4-nvfp4`) in its Mistral-native format, which both engines load with +checkpoint_format "mistral". Skipped where the checkpoint is absent, below four GPUs, or +off sm_103. +""" + +from __future__ import annotations + +import collections +from types import SimpleNamespace + +import pytest +from test_common.llm_data import llm_models_root +from utils.util import skip_num_gpus_less_than + +from tensorrt_llm import LLM, SamplingParams +from tensorrt_llm._torch.modeling_v2 import MODELING_V2_VALIDATE_ENV +from tensorrt_llm._utils import get_sm_version +from tensorrt_llm.inputs import TokensPrompt +from tensorrt_llm.scheduling_params import SchedulingParams + +from . import test_modeling_v2_long_context as long_context + +pytestmark = [ + skip_num_gpus_less_than(4), + long_context._launched_as(4), + pytest.mark.skipif( + get_sm_version() != 103, + reason="the mistral_large4 modeling_v2 targets are certified on sm_103 only", + ), + pytest.mark.threadleak(enabled=False), +] + +#: dep4's four attention-DP ranks, each pinned its own prompts below. +RANKS = 4 +#: The delivered engine: whole prompts of up to 32768 tokens in one step. +UNCHUNKED = long_context._engine(RANKS) +CHUNK = 8192 +#: The same engine prefilling in steps of at most CHUNK tokens per rank. +CHUNKED = dict(UNCHUNKED, max_num_tokens=CHUNK, enable_chunked_prefill=True) +INDEX_TOPK = long_context.INDEX_TOPK + +#: Per batch, per rank, its prompts as (tokens, depth of the key). +BATCHES = { + "mixed": { + 0: ((32000, 0.05),), + 1: ((20000, 0.5),), + 2: ((9000, 0.3),), + 3: ((3000, 0.5), (12000, 0.9)), + }, + "far": {0: ((26000, 0.3),), 1: ((30000, 0.6),), 2: ((28000, 0.8),), 3: ((26500, 0.45),)}, +} +#: Greedy, with the top two log-probabilities at each step. +ANSWER = SamplingParams(max_tokens=12, temperature=0.0, logprobs=2) +#: The top-1 / top-2 log-probability margin from which a step's greedy token is compared: reruns +#: of one whole-prompt engine on the same batch flip tokens at steps whose margin is below it. +NOISE = 1.5 + + +def _context_iterations(llm: LLM) -> dict[int, list[tuple[int, int, int, int]]]: + """Since the last call, per attention-DP rank, every iteration that ran context rows there: + `(iteration, context requests, context tokens, generation requests)`, in order.""" + by_rank: dict[int, list[tuple[int, int, int, int]]] = collections.defaultdict(list) + for record in llm.get_stats(timeout=2): + ifb = record.get("inflightBatchingStats") or {} + if ifb.get("numContextRequests", 0): + rank = int(record.get("attentionDpRank", 0)) + by_rank[rank].append( + ( + int(record["iter"]), + ifb["numContextRequests"], + ifb["numCtxTokens"], + ifb.get("numGenRequests", 0), + ) + ) + return {rank: sorted(steps) for rank, steps in by_rank.items()} + + +def _engine(engine: dict) -> LLM: + return LLM( + str(_checkpoint()), + checkpoint_format="mistral", + modeling_v2="require", + env_overrides={MODELING_V2_VALIDATE_ENV: "1"}, + enable_iter_perf_stats=True, + **engine, + ) + + +def _requests(tokenizer, prompts: dict, seed: int) -> list[SimpleNamespace]: + """The prompts as token ids, BOS included, each pinned to its rank, each key distinct.""" + specs = [ + (rank, tokens, depth) for rank, prompts in prompts.items() for tokens, depth in prompts + ] + keys = long_context._keys(len(specs), seed=seed) + requests = [] + for (rank, tokens, depth), key in zip(specs, keys): + ids = long_context._passkey_ids(tokenizer, tokens, depth, key) + assert INDEX_TOPK < len(ids) <= tokens, (tokens, len(ids)) + requests.append(SimpleNamespace(rank=rank, key=key, ids=ids)) + return requests + + +def _generate(llm: LLM, requests: list) -> tuple[list, dict]: + """Every request through `llm` at once: per request (token ids, top-2 log-probs per step, + text), and each rank's context iterations.""" + _context_iterations(llm) + outputs = llm.generate( + [TokensPrompt(prompt_token_ids=r.ids) for r in requests], + ANSWER, + scheduling_params=[ + SchedulingParams(attention_dp_rank=r.rank, attention_dp_relax=False) for r in requests + ], + ) + steps = _context_iterations(llm) + results = [] + for out in outputs: + o = out.outputs[0] + top2 = [ + sorted(((lp.logprob, tok) for tok, lp in step.items()), reverse=True)[:2] + for step in o.logprobs + ] + results.append(SimpleNamespace(ids=list(o.token_ids), top2=top2, text=o.text)) + return results, steps + + +def _checkpoint(): + path = long_context._checkpoint_dir() + if path is None: + pytest.skip( + f"checkpoint {long_context.CHECKPOINTS} is not under LLM_MODELS_ROOT " + f"({llm_models_root()})" + ) + return path + + +def _margins(top2: list) -> list[float]: + """Each step's top-1 / top-2 log-probability gap.""" + return [s[0][0] - s[1][0] if len(s) > 1 else float("inf") for s in top2] + + +def _check_steps(name: str, steps: dict, per_rank: dict, chunked: bool) -> None: + """Each rank prefilled exactly its own prompts' tokens; whole prompts in one step each, or + chunks of at most CHUNK with a full one over a cached prefix.""" + assert sorted(steps) == list(range(RANKS)), (name, steps) + for rank in range(RANKS): + assert sum(t for _, _, t, _ in steps[rank]) == sum(per_rank[rank]), ( + name, + rank, + steps[rank], + per_rank[rank], + ) + if not chunked: + for rank, lengths in per_rank.items(): + if len(lengths) == 1: + assert [t for _, _, t, _ in steps[rank]] == lengths, (name, steps[rank]) + return + assert all(t <= CHUNK for s in steps.values() for _, _, t, _ in s), (name, steps) + # A rank holding one request: any of its context steps but the first is over a prefix. + over_prefix = [ + (rank, it) + for rank, lengths in per_rank.items() + if len(lengths) == 1 + for it, n, t, _ in steps[rank][1:] + if n == 1 and t == CHUNK + ] + assert over_prefix, (name, steps) + + +def test_chunked_prefill_gives_the_whole_prompt_tokens(): + _checkpoint() + with _engine(UNCHUNKED) as llm: + tokenizer = llm.tokenizer + batches = { + name: _requests(tokenizer, prompts, seed=CHUNK + i) + for i, (name, prompts) in enumerate(BATCHES.items()) + } + whole = {name: _generate(llm, reqs) for name, reqs in batches.items()} + with _engine(CHUNKED) as llm: + chunked = {name: _generate(llm, reqs) for name, reqs in batches.items()} + + problems = [] + for name, requests in batches.items(): + (w_out, w_steps), (c_out, c_steps) = whole[name], chunked[name] + per_rank = collections.defaultdict(list) + for r in requests: + per_rank[r.rank].append(len(r.ids)) + for label, steps in (("whole", w_steps), ("chunked", c_steps)): + for rank in range(RANKS): + print(f"CHUNKEQ {name} {label} rank {rank} prompts {per_rank[rank]}: " + f"(iteration, ctx requests, ctx tokens, gen requests) {steps.get(rank)}") # fmt: skip + _check_steps(f"{name} whole", w_steps, per_rank, chunked=False) + _check_steps(f"{name} chunked", c_steps, per_rank, chunked=True) + + for r, a, b in zip(requests, w_out, c_out): + ma, mb = _margins(a.top2), _margins(b.top2) + n = min(len(a.ids), len(b.ids)) + gated = next((i for i in range(n) if min(ma[i], mb[i]) < NOISE), n) + shown = tokenizer.decode(a.ids[:gated]) + print( + f"CHUNKEQ {name} rank {r.rank} tokens={len(r.ids)} key={r.key}: whole {a.text!r}, " + f"chunked {b.text!r}; {gated} steps at margin >= {NOISE} in both ({shown!r}); " + f"margins whole {[round(m, 2) for m in ma]} chunked {[round(m, 2) for m in mb]}" + ) + for label, out in (("whole", a), ("chunked", b)): + if str(r.key) not in out.text: + problems.append(f"{name} rank {r.rank}: {label} lost the key: {out.text!r}") + if a.ids[:gated] != b.ids[:gated]: + problems.append( + f"{name} rank {r.rank} tokens={len(r.ids)}: differ within the {gated} steps " + f"above the noise: {a.ids[:gated]} vs {b.ids[:gated]}" + ) + elif str(r.key) not in shown: + problems.append( + f"{name} rank {r.rank}: the key {r.key} is not within the {gated} steps " + f"above the noise ({shown!r})" + ) + k = next((i for i in range(n) if a.ids[i] != b.ids[i]), None) + if k is not None: + print( + f"CHUNKEQ {name} rank {r.rank}: first differing step {k}, whole top-2 " + f"{a.top2[k]}, chunked top-2 {b.top2[k]}" + ) + assert not problems, problems diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_ctx_bound.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_ctx_bound.py new file mode 100644 index 000000000000..ff9372ead114 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_ctx_bound.py @@ -0,0 +1,752 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The dep target's context rows at the 1M config (max_seq_len 1048576, chunked prefill, max +num_tokens 32768 and 16 requests per rank): the per-step choice between the held workspace and +the pool. + +`PrefillTarget._ctx_workspace` picks the step's path once, on this rank's host ints, from N, its +context KV (every context request's cached and new rows; the engine's `num_ctx_kv_tokens` counts +the new ones only), against the held workspace's rows, which `_derive_first_step` sizes from the +engine (`_ctx_workspace_rows`): B = 2**19 at this config, the ceiling `_CTX_WORKSPACE_ROWS`. Up to +B rows: `ds_mla_dequant_gather` writes the rows into the first N rows of one bf16 [B, 576] +workspace, allocated on the first context step and held, then `flash_mla_sparse_prefill` attends +over it; layer 35, the latent the NoPE layers 39..71 attend over, is gathered into the N rows +after the step's own when 2N fits B, which its sharers read without a gather, and otherwise +again by every sharer into the first N. Past B: `convert_req_index_to_global` turns the +selections into pool slots and `flash_mla_sparse_decode` attends with every context row a +one-row sequence, in sub-batches of at most `_CTX_DECODE_ROWS` (8192) rows with one FlashMLA +scheduler per sub-batch size. `PrefillTarget._attn_ctx` runs the path. Under attention DP a +rank's step holds its own requests only, all 64 heads of each. (The held workspace at other +configs is test_modeling_v2_ctx_workspace.py's.) + +On one GPU, over one fp8_ds_mla pool at the 63-owner layout (52000 blocks, 68.8 GB; every byte +0xFF, a NaN in every field, but the owner rows of layers 1, 35 and 36), with the engine's +[16, 32772] block table, queries at all 64 heads, and selections shaped like the indexer's +(request-local positions, every earlier position below index_topk, which is also the engine's +dense causal index on a step it skipped the indexer for, else 2048 distinct ones with the row's +own): + +* the KV-capacity estimate's step (one fresh 32768-token prompt) allocates the workspace; no + later step reallocates it, no step past B reaches the gather, and every gather writes the + held buffer, the step's first N rows or, for layer 35 on a step that fits twice, the N after; +* one packed step of four chunks at N = B (workspace path) and, with its last chunk one row + longer, at N = B + 1 (decode path, 4 x 8192): the shared rows' outputs are bitwise the same; +* a 1M prompt's last full chunk (32768 rows behind 1015808 cached keys, 4 x 8192) and the + 1048559-token prompt's last chunk (32751 rows, 8192 x 3 + 8175 on two schedulers); the + first's per-layer peak memory against the estimate's step on the workspace path; +* several requests of one rank, one of them over a 1M cached prefix, and the packed four-chunk + step at max_seq_len (cached 1032192 / 524288 / 0 / 2048): the decode path, with its per-layer + time and peak memory; +* mixed steps on the decode path and on the workspace path at N fitting B twice and once + (generation rows behind the context rows, which the context attention leaves alone), with + layer 35's sharer (39) reading row 35 after another owner (36) gathered: from the kept rows, + or gathered again after 36 overwrote the step's rows; +* a skipped-indexer (dense) step of sixteen requests within index_topk, on the workspace path + with layer 35's rows kept; +* a control: the same harness over a workspace one row short of N, and over the new tokens + alone (the engine's count), fails, though the kernel raises nothing and writes no NaN. + +Every decode-path output is checked against the same prefill kernel over an unbounded +workspace (the workspace path's computation without the bound): rows of each full 8192-row +sub-batch bitwise, and the whole within relative L2 1e-3. Sampled rows of every output are +checked against an fp32 softmax over keys read straight from the pool through the block table. +The CPU test checks the path choice on host metadata alone. +""" + +from __future__ import annotations + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch.attention.backends.interface import AttentionRuntimeFeatures +from tensorrt_llm._torch.modeling_v2.catalog._op import true_fp32_matmul +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_103__dep import ( # noqa: E501 + modeling as M, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.convert_req_index_to_global import ( # noqa: E501 + convert_req_index_to_global, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.flash_mla_sparse_prefill import ( # noqa: E501 + flash_mla_sparse_prefill, +) +from tensorrt_llm.flash_mla import get_mla_metadata + +_LAYERS = 72 +_TPB = 32 +_TOPK = 2048 +#: Attention DP: a rank runs all 64 heads, FlashMLA's own head count. +_HEADS = 64 +_LAT_DIM, _NOPE, _TILE, _ROW_BYTES = 576, 512, 128, 656 +#: Every fourth layer from 39 on attends over layer 35's latent (the checkpoint's +#: kv_source_layers); the pool holds the 63 owners' rows. +_SOURCES = [35 if i >= 39 and i % 4 == 3 else i for i in range(_LAYERS)] +_OWNERS = [i for i in range(_LAYERS) if _SOURCES[i] == i] +#: The owner layers whose rows the pool holds: a RoPE owner, the NoPE source of the sharers, +#: and the owner after it. +_WRITTEN = (1, 35, 36) +#: Pool blocks: the packed four-chunk step at max_seq_len holds 49728 of them. +_BLOCKS = 52000 +#: md.block_table at max_seq_len 1048576 on the V2 manager: [max_batch_size, 32772]. +_TABLE = (16, 32772) +_SCALE = 128**-0.5 +_SEED = 1051 +#: The 1M config's quantities on the attention metadata, which the held workspace is sized from. +_ENGINE_1M = SimpleNamespace( + max_num_tokens=32768, + max_num_requests=16, + max_seq_len=1048576, + runtime_features=AttentionRuntimeFeatures(chunked_prefill=True, cache_reuse=False), +) +_B = M._CTX_WORKSPACE_ROWS +_SUB = M._CTX_DECODE_ROWS +#: The last chunk of a 1048576-token prompt reads a cached prefix of 1015808 tokens. +_PREFIX_1M = 1048576 - 32768 +#: Four chunks (cached, new) packed to a context KV of exactly B in 32767 rows, and the same +#: step with its last chunk one row longer: B + 1 in 32768 rows. +_AT_B = [(358401, 16384), (131072, 8192), (2048, 4096), (0, 4095)] +_PAST_B = _AT_B[:-1] + [(0, 4096)] +#: The agreement between the paths, relative L2 over the step: FlashMLA's decode kernel splits +#: some rows' keys over SMs in a sub-batch shorter than 8192 (a chunk's tail), which moves those +#: rows within the survey's measured band for split sub-batches (relative L2 1.4e-04 to +#: 8.7e-04). +_AGREE = 1e-3 +#: An fp32 softmax over the selected keys against the bf16 kernels. +_FP32_BAND = 1e-2 +_SAMPLED = 128 +_GIB = 2**30 + + +def _cumsum(lens: list[int], dev=None) -> torch.Tensor: + return torch.tensor([0] + list(lens), dtype=torch.int64, device=dev).cumsum(0) + + +def _target() -> M.PrefillTarget: + """The 1M config's prefill target with only what its context and generation attention read + off the core: the held workspace's rows (`_ctx_workspace_rows` of the config, B), layer 35 + the one source kept for its sharers (`shared_source`, as every core holds it), the latent + width, the kernels' softmax scale, index_topk and the 63-owner pool's addressing.""" + target = M.PrefillTarget.__new__(M.PrefillTarget) + target.core = SimpleNamespace( + ctx_workspace_rows=M._ctx_workspace_rows(_ENGINE_1M), + shared_source=[i == 35 for i in range(_LAYERS)], + lat_dim=_LAT_DIM, + softmax_scale=_SCALE, + idx_topk=_TOPK, + _stride_factor=len(_OWNERS) * _TPB, + _pool_row=[_OWNERS.index(_SOURCES[i]) for i in range(_LAYERS)], + ) + return target + + +def _host_step(contexts, gens=(), **extra) -> dict: + """The host fields `_ctx_workspace` reads, as `PrefillTarget.step_args` builds them under + chunked prefill (`_ctx_kv_offsets` and `_ctx_host_offsets` over the engine's indptrs) for + context requests of (cached, new) tokens with generation rows behind them.""" + cached = [c for c, _ in contexts] + new = [n for _, n in contexts] + md = SimpleNamespace( + enable_context_mla_with_cached_kv=True, + ctx_cached_token_indptr=_cumsum(cached), + ctx_kv_indptr=_cumsum([c + n for c, n in contexts]), + host_ctx_uncached_token_indptr=_cumsum(new), + host_ctx_kv_indptr=_cumsum([c + n for c, n in contexts]), + max_ctx_seq_len=max(new), + num_ctx_cached_tokens=sum(cached), + ) + step = dict( + num_contexts=len(contexts), + num_ctx_tokens=sum(new), + # The engine's count: the step's new context keys only. + num_ctx_kv_tokens=sum(new), + num_tokens=sum(new) + len(gens), + skip_ctx=False, + **M._ctx_kv_offsets(md, len(contexts)), + **M._ctx_host_offsets(md, len(contexts)), + ) + step.update(extra) + return step + + +# ---- host only: the path choice ---- + + +@pytest.mark.cpu_only +def test_path_reads_cached_plus_new_on_host_ints(): + """The 1M config holds B = 2**19 rows. The path follows cached + new at B and B + 1, whatever + the engine's new-token count or the generation rows say, and layer 35's kept rows follow + 2N against B; the decode path's sub-batches tile the context rows at 8192 with one scheduler + per size; the workspace is allocated once, on the first context step.""" + assert M._ctx_workspace_rows(_ENGINE_1M) == _B == 1 << 19 + target = _target() + cpu = torch.device("cpu") + assert target._ws_buf is None + + # The KV-capacity estimate's step: one fresh 32768-token prompt per rank, kept rows beside. + estimate = _host_step([(0, 32768)]) + ctx = target._ctx_workspace(estimate, 32768, cpu) + buf = target._ws_buf + assert buf.shape == (_B, _LAT_DIM) and buf.dtype == torch.bfloat16 + assert ctx.ws_buf.shape == (32768, _LAT_DIM) and ctx.ws_buf.data_ptr() == buf.data_ptr() + assert ctx.kept.shape == (32768, _LAT_DIM) + assert ctx.kept.data_ptr() == buf.data_ptr() + 32768 * _LAT_DIM * buf.element_size() + + at_b = _host_step(_AT_B) + assert at_b["ctx_cached_tokens"] + at_b["num_ctx_tokens"] == _B + ctx = target._ctx_workspace(at_b, at_b["num_ctx_tokens"], cpu) + assert ctx.ws_buf.shape == (_B, _LAT_DIM) and ctx.ws_buf.data_ptr() == buf.data_ptr() + assert ctx.kv_req.shape == (_B,) and int(ctx.kv_pos[-1]) == 4094 and ctx.kept is None + + past_b = _host_step(_PAST_B) + assert past_b["ctx_cached_tokens"] + past_b["num_ctx_tokens"] == _B + 1 + ctx = target._ctx_workspace(past_b, past_b["num_ctx_tokens"], cpu) + assert ctx.ws_buf is None and not hasattr(ctx, "kv_req") + assert ctx.sub_batches == [(0, 8192), (8192, 16384), (16384, 24576), (24576, 32768)] + assert sorted(ctx.sched) == [8192] and not ctx.sched[8192].have_initialized + + # A 1M prompt's last full chunk and the 1048559-token prompt's last chunk (1015808 + + # 32751): their new tokens alone are far below B. + for new, subs in ((32768, [8192] * 4), (32751, [8192, 8192, 8192, 8175])): + step = _host_step([(_PREFIX_1M, new)]) + assert step["num_ctx_kv_tokens"] == new < _B + ctx = target._ctx_workspace(step, new, cpu) + assert ctx.ws_buf is None and [hi - lo for lo, hi in ctx.sub_batches] == subs + assert sorted(ctx.sched) == sorted(set(subs)) + + # Several requests of one rank, one over a 1M prefix, with generation rows behind them. + step = _host_step( + [(_PREFIX_1M, 16384), (4096, 8192), (0, 4096), (100, 4000)], gens=(524288, 3001, 17) + ) + ctx = target._ctx_workspace(step, step["num_ctx_tokens"], cpu) + assert ctx.ws_buf is None and ctx.sub_batches[-1] == (24576, 32672) + + # Dense steps (every request within index_topk) and short ones stay on the workspace, with + # layer 35's rows kept; past B / 2 a step keeps none. + for contexts in ([(0, 2048)] * 16, [(2047, 1)] * 16, [(1024, 1024)] * 16, [(5, 300)]): + step = _host_step(contexts) + ctx = target._ctx_workspace(step, step["num_ctx_tokens"], cpu) + assert ctx.ws_buf is not None and ctx.ws_buf.data_ptr() == buf.data_ptr() + assert ctx.kept is not None + for n_kv, kept in ((_B // 2, True), (_B // 2 + 1, False)): + step = _host_step([(n_kv - 640, 640)]) + ctx = target._ctx_workspace(step, 640, cpu) + assert ctx.ws_buf.shape == (n_kv, _LAT_DIM) and (ctx.kept is not None) == kept + assert target._ws_buf is buf + + +# ---- one GPU ---- + + +def _dev_rows(n: int, g: torch.Generator, nope: bool) -> torch.Tensor: + """`n` uint8 656-byte rows as the fp8_ds_mla writer stores them: e4m3 codes, power-of-two + fp32 tile scales 2^-10..2^-4, bf16 rope (zero on a NoPE owner).""" + dev = g.device + codes = ((torch.rand(n, _NOPE, generator=g, device=dev) * 2 - 1) * 448).to(torch.float8_e4m3fn) + scale = torch.exp2(torch.randint(-10, -3, (n, _NOPE // _TILE), generator=g, device=dev).float()) + if nope: + rope = torch.zeros(n, _LAT_DIM - _NOPE, device=dev) + else: + rope = torch.randn(n, _LAT_DIM - _NOPE, generator=g, device=dev) * 0.5 + return torch.cat( + [ + codes.view(torch.uint8), + scale.view(torch.uint8), + rope.to(torch.bfloat16).view(torch.uint8), + ], + dim=1, + ) + + +def _dequant(rows: torch.Tensor) -> torch.Tensor: + """fp8_ds_mla rows `[n, 656]` to bf16 `[n, 576]`: e4m3 times its tile's scale (exact in + bf16 for power-of-two scales), then the rope.""" + codes = rows[:, :_NOPE].contiguous().view(torch.float8_e4m3fn).float() + scale = rows[:, _NOPE : _NOPE + 16].contiguous().view(torch.float32) + nope = (codes.view(-1, _NOPE // _TILE, _TILE) * scale[..., None]).view(-1, _NOPE) + rope = rows[:, _NOPE + 16 :].contiguous().view(torch.bfloat16) + return torch.cat([nope.to(torch.bfloat16), rope], dim=1) + + +def _selections(pos: torch.Tensor, g: torch.Generator) -> torch.Tensor: + """int32 `[rows, 2048]` request-local selections for rows at positions `pos`: below + index_topk every earlier position in order then -1 (the engine's dense causal index), else + one draw from each of 2048 equal strata of [0, p] with p itself in the last, so distinct.""" + p = pos.long()[:, None] + col = torch.arange(_TOPK, device=pos.device)[None] + lo = col * (p + 1) // _TOPK + hi = (col + 1) * (p + 1) // _TOPK + draw = lo + (torch.rand(pos.numel(), _TOPK, generator=g, device=pos.device) * (hi - lo)).long() + draw = torch.minimum(draw, hi - 1) + draw[:, -1] = pos + return torch.where(p < _TOPK, torch.where(col <= p, col, -1), draw).to(torch.int32) + + +@pytest.fixture(scope="module") +def env(): + """The pool, the target, and the KV-capacity estimate's step already run through the + path choice; every gather the module's tests make is recorded (observe only).""" + if not torch.cuda.is_available(): + pytest.skip("needs a GPU") + from tensorrt_llm._utils import get_sm_version + + if get_sm_version() != 103: + pytest.skip("the mistral_large4 modeling_v2 targets are certified on sm_103 only") + dev = torch.device("cuda") + g = torch.Generator(device=dev).manual_seed(_SEED) + pool = torch.empty((_BLOCKS, len(_OWNERS), _TPB, _ROW_BYTES), dtype=torch.uint8, device=dev) + pool.fill_(0xFF) + rows = {layer: _OWNERS.index(layer) for layer in _WRITTEN} + for layer, row in rows.items(): + for lo in range(0, _BLOCKS, 1 << 15): + hi = min(lo + (1 << 15), _BLOCKS) + drawn = _dev_rows((hi - lo) * _TPB, g, nope=layer == 35) + pool[lo:hi, row] = drawn.view(hi - lo, _TPB, _ROW_BYTES) + del drawn + gathers = [] + real_gather = M.ds_mla_dequant_gather + + def recording_gather(pool_, row, kv_req, kv_pos, block_table, out): + gathers.append( + SimpleNamespace(rows=int(kv_req.shape[0]), out_ptr=out.data_ptr(), pool_row=row) + ) + return real_gather(pool_, row, kv_req, kv_pos, block_table, out) + + with pytest.MonkeyPatch.context() as mp: + mp.setattr(M, "ds_mla_dequant_gather", recording_gather) + e = SimpleNamespace( + dev=dev, pool=pool, kv=pool.view(-1, _TPB, 1, _ROW_BYTES), rows=rows, + target=_target(), gathers=gathers, + ) # fmt: skip + assert e.target._ws_buf is None + # The KV-capacity estimate's step at the 1M config: one fresh 32768-token prompt per + # rank. + e.estimate = _step(e, [(0, 32768)], seed=1) + e.estimate_ctx = e.target._ctx_workspace(e.estimate.step, e.estimate.tc, dev) + e.buf = e.target._ws_buf + assert e.buf.shape == (_B, _LAT_DIM) and e.estimate_ctx.ws_buf is not None + yield e + assert e.target._ws_buf is e.buf + + +def _step(env, contexts, gens=(), seed: int = 0, **extra) -> SimpleNamespace: + """A step of context requests of (cached, new) tokens with generation rows of the given KV + lengths behind them: each request's pages distinct blocks of the pool, its table row + padded to the engine's width, a query per row at all 64 heads, and its selection.""" + dev = env.dev + g = torch.Generator(device=dev).manual_seed(_SEED + seed) + kv_lens = [c + n for c, n in contexts] + list(gens) + pages = [-(-k // _TPB) for k in kv_lens] + assert sum(pages) <= _BLOCKS and max(pages) <= _TABLE[1] and len(kv_lens) <= _TABLE[0] + order = torch.randperm(_BLOCKS, generator=g, device=dev).to(torch.int32) + table = torch.zeros(_TABLE, dtype=torch.int32, device=dev) + used = 0 + for r, n in enumerate(pages): + table[r, :n] = order[used : used + n] + used += n + tc = sum(n for _, n in contexts) + req_idx = torch.cat( + [torch.full((n,), r, dtype=torch.int32, device=dev) for r, (_, n) in enumerate(contexts)] + + [torch.arange(len(contexts), len(kv_lens), dtype=torch.int32, device=dev)] + ) + pos = torch.cat( + [torch.arange(c, c + n, device=dev) for c, n in contexts] + + [torch.tensor(gens, device=dev, dtype=torch.int64) - 1] + ) + q = torch.randn(pos.numel(), _HEADS, _LAT_DIM, generator=g, device=dev).mul_(0.1) + q = q.to(torch.bfloat16) + step = _host_step(contexts, gens, **extra) + step = {k: v.to(dev) if isinstance(v, torch.Tensor) else v for k, v in step.items()} + step.update(block_table=table, tokens_per_block=_TPB, latent_pool=env.pool, latent_kv=env.kv) + return SimpleNamespace( + step=step, contexts=list(contexts), gens=list(gens), kv_lens=kv_lens, tc=tc, + rows=pos.numel(), req_idx=req_idx, pos=pos, topk=_selections(pos, g), q=q, table=table, + ) # fmt: skip + + +def _request_kv(env, s, layer: int, r: int) -> torch.Tensor: + """Request r's KV rows 0..kv_len-1 of `layer`, bf16, read through its block table.""" + n = s.kv_lens[r] + out = torch.empty(n, _LAT_DIM, dtype=torch.bfloat16, device=env.dev) + for lo in range(0, n, 1 << 19): + p = torch.arange(lo, min(lo + (1 << 19), n), device=env.dev) + out[lo : lo + p.numel()] = _dequant( + env.pool[s.table[r, p // _TPB].long(), env.rows[layer], p % _TPB] + ) + return out + + +def _unbounded(env, s, layer: int) -> torch.Tensor: + """The workspace path's computation without its bound: every context request's rows in one + bf16 buffer of N rows, the selections offset into it, the same prefill kernel.""" + ws = torch.cat([_request_kv(env, s, layer, r) for r in range(len(s.contexts))]) + off = _cumsum([c + n for c, n in s.contexts], env.dev)[:-1] + sel = s.topk[: s.tc] + idx = torch.where(sel >= 0, sel + off[s.req_idx[: s.tc].long()][:, None], sel).to(torch.int32) + out = torch.empty(s.tc, _HEADS, _NOPE, dtype=torch.bfloat16, device=env.dev) + flash_mla_sparse_prefill(s.q[: s.tc], ws, idx, _SCALE, out) + return out + + +def _fp32_check(env, s, layer: int, out: torch.Tensor, rows: torch.Tensor) -> float: + """`out`'s rows `rows` against an fp32 softmax over each row's selected keys read straight + from the pool through its request's table row; returns the relative L2 error.""" + sel = s.topk[rows].long() + req = s.req_idx[rows].long() + valid = sel >= 0 + key = sel.clamp_min(0) + blocks = s.table[req[:, None], key // _TPB].long() + k = _dequant(env.pool[blocks, env.rows[layer], key % _TPB].view(-1, _ROW_BYTES)).float() + k = k.view(rows.numel(), _TOPK, _LAT_DIM) + with true_fp32_matmul(): + logits = torch.matmul(s.q[rows].float(), k.transpose(1, 2)) * _SCALE + logits = logits.masked_fill(~valid[:, None, :], float("-inf")) + exact = torch.matmul(torch.softmax(logits, -1), k[..., :_NOPE]) + got = out[rows].float() + assert bool(torch.isfinite(got).all()), "a sampled row read an unwritten pool byte" + return float((got - exact).norm() / exact.norm()) + + +def _sample(rows: int, g_seed: int, must=()) -> torch.Tensor: + g = torch.Generator().manual_seed(g_seed) + picked = torch.randperm(rows, generator=g)[:_SAMPLED].tolist() + keep = sorted(set(picked) | {r for r in must if 0 <= r < rows}) + return torch.tensor(keep, device="cuda") + + +def _attend(env, s, layer: int, ctx=None, out=None): + """`_attn_ctx` for `layer` over the step's context rows: the step's path (made here unless + given) and the output (NaN-filled first unless given).""" + if ctx is None: + ctx = env.target._ctx_workspace(s.step, s.tc, env.dev) + if out is None: + out = torch.full((s.rows, _HEADS, _NOPE), float("nan"), dtype=torch.bfloat16, + device=env.dev) # fmt: skip + L = SimpleNamespace(kv_source=_SOURCES[layer]) + env.target._attn_ctx(s.step, L, s.q[: s.tc], out[: s.tc], s.req_idx, s.topk, ctx) + return ctx, out + + +def _agreement(got: torch.Tensor, ref: torch.Tensor) -> tuple[float, int]: + rel = float((got.float() - ref.float()).norm() / ref.float().norm()) + return rel, int((got.view(torch.int16) != ref.view(torch.int16)).sum()) + + +def _rows_differing(got: torch.Tensor, ref: torch.Tensor, bounds) -> list[int]: + return [ + int((got[lo:hi].view(torch.int16) != ref[lo:hi].view(torch.int16)).flatten(1).any(1).sum()) + for lo, hi in bounds + ] + + +def _check_decode_path(env, s, ctx, out, layer: int, label: str) -> None: + """A decode-path output: finite, within the agreement of the unbounded workspace + computation (bitwise in every full sub-batch), and the sampled rows within the fp32 band.""" + lat = out[: s.tc] + assert bool(torch.isfinite(lat).all()), f"{label}: a context row read an unwritten pool byte" + ref = _unbounded(env, s, layer) + rel, differ = _agreement(lat, ref) + rows_differ = _rows_differing(lat, ref, ctx.sub_batches) + edges = [lo for lo, _ in ctx.sub_batches] + [hi - 1 for _, hi in ctx.sub_batches] + err = _fp32_check(env, s, layer, out, _sample(s.tc, layer, edges)) + print(f"CTXBOUND {label} layer {layer}: decode path, sub-batches " + f"{[hi - lo for lo, hi in ctx.sub_batches]}, against the unbounded workspace rel L2 " + f"{rel:.3e} ({differ} of {lat.numel()} elements differ; rows differing per sub-batch " + f"{rows_differ}); fp32 rel L2 {err:.3e}") # fmt: skip + assert rel <= _AGREE, (label, rel) + for (lo, hi), n in zip(ctx.sub_batches, rows_differ): + if hi - lo == _SUB: + assert n == 0, (label, lo, hi, n) + assert err < _FP32_BAND, (label, err) + + +def _check_workspace_path(env, s, out, layer: int, label: str) -> None: + """A workspace-path output: bitwise each context request's rows through the same kernel over + that request's own gather (its own offsets, no packing), and the sampled rows within the + fp32 band.""" + lat = out[: s.tc] + assert bool(torch.isfinite(lat).all()), f"{label}: a context row read an unwritten pool byte" + start, differ = 0, 0 + for r, (_, new) in enumerate(s.contexts): + kv = _request_kv(env, s, _SOURCES[layer], r) + own = torch.empty(new, _HEADS, _NOPE, dtype=torch.bfloat16, device=env.dev) + flash_mla_sparse_prefill( + s.q[start : start + new], kv, s.topk[start : start + new], _SCALE, own + ) + differ += int((lat[start : start + new].view(torch.int16) != own.view(torch.int16)).sum()) + start += new + err = _fp32_check(env, s, _SOURCES[layer], out, _sample(s.tc, layer, (0, s.tc - 1))) + print(f"CTXBOUND {label} layer {layer}: workspace path, N {sum(s.kv_lens[: len(s.contexts)])}" + f", {differ} elements differ from each request alone; fp32 rel L2 {err:.3e}") # fmt: skip + assert differ == 0, (label, differ) + assert err < _FP32_BAND, (label, err) + + +def _measure(fn, iters: int = 5) -> tuple[float, int]: + """(ms per call, peak bytes above the allocation before the call) of a warm `fn`.""" + fn() + torch.cuda.synchronize() + base = torch.cuda.memory_allocated() + torch.cuda.reset_peak_memory_stats() + fn() + torch.cuda.synchronize() + peak = torch.cuda.max_memory_allocated() - base + start, end = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True) + start.record() + for _ in range(iters): + fn() + end.record() + torch.cuda.synchronize() + return start.elapsed_time(end) / iters, peak + + +def test_same_packed_step_at_b_and_b_plus_one(env): + """Four chunks packed to N = B take the workspace path; the same step with its last chunk + one row longer (N = B + 1, 32768 rows) takes the decode path in four sub-batches of 8192. + The 32767 shared rows' outputs are bitwise the same, as every full sub-batch's rows are + (`_check_decode_path`).""" + past = _step(env, _PAST_B, seed=2) + assert sum(past.kv_lens) == _B + 1 + ctx_past, out_past = _attend(env, past, 1) + assert ctx_past.ws_buf is None + assert ctx_past.sub_batches == [(lo, lo + _SUB) for lo in range(0, 32768, _SUB)] + _check_decode_path(env, past, ctx_past, out_past, 1, "N = B + 1") + + # The same step, its last row dropped: the same pool, table, queries and selections. + at = SimpleNamespace(**vars(past)) + at.contexts = list(_AT_B) + at.kv_lens = [c + n for c, n in at.contexts] + at.tc = at.rows = past.tc - 1 + at.step = dict(past.step, **_host_step(at.contexts)) + at.step = {k: v.to(env.dev) if isinstance(v, torch.Tensor) else v for k, v in at.step.items()} + assert sum(at.kv_lens) == _B + gathered = len(env.gathers) + ctx_at, out_at = _attend(env, at, 1) + assert ctx_at.ws_buf is not None and ctx_at.ws_buf.data_ptr() == env.buf.data_ptr() + assert [x.rows for x in env.gathers[gathered:]] == [_B] + rel, differ = _agreement(out_at[: at.tc], out_past[: at.tc]) + rows_differ = _rows_differing(out_at[: at.tc], out_past[: at.tc], ctx_past.sub_batches) + print(f"CTXBOUND N = B (workspace) against N = B + 1 (decode) on the {at.tc} shared rows: " + f"rel L2 {rel:.3e}, {differ} elements differ, rows differing per sub-batch " + f"{rows_differ}") # fmt: skip + assert rel <= _AGREE and differ == 0, (rel, differ) + err = _fp32_check(env, at, 1, out_at, _sample(at.tc, 7, (0, 16383, 16384, at.tc - 1))) + assert err < _FP32_BAND, err + # The workspace holds every request's rows, request after request. + off = 0 + for r, n in enumerate(at.kv_lens): + assert torch.equal(env.buf[off : off + n], _request_kv(env, at, 1, r)), r + off += n + + +def test_last_chunks_of_a_1m_prompt(env): + """A 1M prompt's last full chunk, 32768 rows behind 1015808 cached keys: four sub-batches of + 8192 on one scheduler the 72 layers share, its per-layer peak memory under the KV-capacity + estimate's step on the workspace path, whose peak sizes the pool. The 1048559-token prompt's + last chunk, 32751 rows behind 1015808 (1048559 KV rows): sub-batches 8192 x 3 + 8175, on two + schedulers.""" + s = _step(env, [(_PREFIX_1M, 32768)], seed=3) + gathered = len(env.gathers) + ctx, out = _attend(env, s, 1) + assert ctx.sub_batches == [(lo, lo + _SUB) for lo in range(0, 32768, _SUB)] + assert sorted(ctx.sched) == [8192] and ctx.sched[8192].config.b == 8192 + assert ctx.sched[8192].config.h_q == _HEADS + _check_decode_path(env, s, ctx, out, 1, "32768 behind 1015808") + assert len(env.gathers) == gathered + + est = env.estimate + lat_est = torch.empty(est.tc, _HEADS, _NOPE, dtype=torch.bfloat16, device=env.dev) + lat_dec = torch.empty(s.tc, _HEADS, _NOPE, dtype=torch.bfloat16, device=env.dev) + L = SimpleNamespace(kv_source=1) + ws_ms, ws_peak = _measure( + lambda: env.target._attn_ctx( + est.step, L, est.q, lat_est, est.req_idx, est.topk, env.estimate_ctx + ) + ) + dec_ms, dec_peak = _measure( + lambda: env.target._attn_ctx(s.step, L, s.q, lat_dec, s.req_idx, s.topk, ctx) + ) + print(f"CTXBOUND per layer: the estimate's step (32768 fresh rows, workspace path) " + f"{ws_ms:.3f} ms, peak {ws_peak / _GIB:.3f} GiB above the held workspace; " + f"32768 rows behind 1015808 (decode path) {dec_ms:.3f} ms, peak " + f"{dec_peak / _GIB:.3f} GiB; margin {(ws_peak - dec_peak) / _GIB:.3f} GiB") # fmt: skip + assert dec_peak <= ws_peak, (dec_peak, ws_peak) + + s = _step(env, [(_PREFIX_1M, 32751)], seed=4) + gathered = len(env.gathers) + ctx, out = _attend(env, s, 1) + assert [hi - lo for lo, hi in ctx.sub_batches] == [8192, 8192, 8192, 8175] + assert sorted(ctx.sched) == [8175, 8192] + assert all(m.have_initialized for m in ctx.sched.values()) + assert (ctx.sched[8192].config.b, ctx.sched[8175].config.b) == (8192, 8175) + _check_decode_path(env, s, ctx, out, 1, "ragged 32751 behind 1015808") + # The next layer reuses the filled schedulers. + again = _attend(env, s, 1, ctx=ctx)[1] + assert torch.equal(again[: s.tc], out[: s.tc]) + assert len(env.gathers) == gathered + + +def test_several_requests_one_over_a_1m_prefix(env): + """One rank's step of four requests (attention DP gives a rank up to 16 of its own): a chunk + over a 1015808-token prefix beside three short ones, with three generation rows behind them. + N is past B, so every context row takes the decode path, the short requests' rows too.""" + contexts = [(_PREFIX_1M, 16384), (4096, 8192), (0, 4096), (100, 4000)] + s = _step(env, contexts, gens=(524288, 3001, 17), seed=5) + gathered = len(env.gathers) + ctx, out = _attend(env, s, 36) + assert ctx.ws_buf is None and [hi - lo for lo, hi in ctx.sub_batches] == [8192] * 3 + [8096] + _check_decode_path(env, s, ctx, out, 36, "four requests, one over 1015808") + assert bool(torch.isnan(out[s.tc :].float()).all()) + assert len(env.gathers) == gathered + + +def test_packed_four_chunks_at_max_seq_len(env): + """Four chunks of 16384 / 8192 / 4096 / 4096 rows packed behind 1032192 / 524288 / 0 / 2048 + cached keys (N = 1591296): the decode path, its per-layer time and peak memory.""" + contexts = [(1032192, 16384), (524288, 8192), (0, 4096), (2048, 4096)] + s = _step(env, contexts, seed=6) + ctx, out = _attend(env, s, 1) + assert ctx.ws_buf is None and sum(s.kv_lens) == 1591296 + _check_decode_path(env, s, ctx, out, 1, "packed four chunks") + lat = torch.empty(s.tc, _HEADS, _NOPE, dtype=torch.bfloat16, device=env.dev) + L = SimpleNamespace(kv_source=1) + ms, peak = _measure(lambda: env.target._attn_ctx(s.step, L, s.q, lat, s.req_idx, s.topk, ctx)) + print(f"CTXBOUND packed four-chunk step (cached 1032192/524288/0/2048, new " + f"16384/8192/4096/4096, N {sum(s.kv_lens)}): decode path, {ms:.3f} ms per layer, " + f"peak {peak / _GIB:.3f} GiB above the held workspace") # fmt: skip + + +def test_mixed_steps_and_layer_35s_sharers(env): + """Generation rows behind the context rows on every path: the context attention writes the + context rows only, and the generation rows' own call (`_attn_gen`, one scheduler) reads the + pool. Layer 39 attends over layer 35's rows after layer 36 gathered its own: from the kept + rows where N fits B twice (5036 rows: 35 and 36 gathered once each), gathered again into the + step's rows where it fits once (300940 rows: 35, 36 and 39 each gather), and from the pool on + the decode path; each time bitwise layer 35's output.""" + target = env.target + for label, contexts, gens in ( + ("decode", [(_PREFIX_1M, 16384)], (524288, 3001, 17)), + ("kept", [(4096, 640), (0, 300)], (3001, 17)), + ("gathered again", [(300000, 640), (0, 300)], (3001, 17)), + ): + s = _step(env, contexts, gens, seed=7) + ctx = target._ctx_workspace(s.step, s.tc, env.dev) + assert (ctx.ws_buf is None) == (label == "decode") + if label != "decode": + n_kv = sum(s.kv_lens[: len(contexts)]) + assert n_kv <= _B and (ctx.kept is not None) == (label == "kept") == (2 * n_kv <= _B) + gathered = len(env.gathers) + outs = {} + for layer in (35, 36, 39): + ctx, out = _attend(env, s, layer, ctx=ctx) + assert bool(torch.isnan(out[s.tc :].float()).all()), (label, layer) + if label != "decode": + # The source's rows, request after request: in the kept rows for layer 35 on a + # step that keeps them, else in the step's own rows. + src = _SOURCES[layer] + rows = ctx.ws.get(src, ctx.ws_buf) + assert (rows is ctx.kept) == (label == "kept" and src == 35), (label, layer) + off = 0 + for r in range(len(contexts)): + n = s.kv_lens[r] + assert torch.equal(rows[off : off + n], _request_kv(env, s, src, r)) + off += n + slots = convert_req_index_to_global( + s.req_idx[s.tc :], s.table, s.topk[s.tc :], _TPB, _TOPK, + len(_OWNERS) * _TPB, target.core._pool_row[_SOURCES[layer]], + ) # fmt: skip + L = SimpleNamespace(kv_source=_SOURCES[layer]) + target._attn_gen(s.step, L, s.q[s.tc :], out[s.tc :], slots, get_mla_metadata()[0]) + outs[layer] = out + # On the workspace path one gather per owner, and the sharer's own where nothing is + # kept; none past B. + assert len(env.gathers) - gathered == {"decode": 0, "kept": 2, "gathered again": 3}[label] + assert torch.equal(outs[39], outs[35]), label + assert not torch.equal(outs[36], outs[35]), label + for layer in (35, 36): + rows = _sample(s.rows, layer, range(s.tc - 2, s.rows)) + err = _fp32_check(env, s, layer, outs[layer], rows) + print(f"CTXBOUND mixed step, {label} path, layer {layer}: {s.tc} context rows and " + f"{len(gens)} generation rows, fp32 rel L2 {err:.3e}") # fmt: skip + assert err < _FP32_BAND, (label, layer, err) + if label == "decode": + _check_decode_path(env, s, ctx, outs[35], 35, "mixed, layer 35") + else: + _check_workspace_path(env, s, outs[35], 35, "mixed, layer 35") + _check_workspace_path(env, s, outs[39], 39, "mixed, sharer 39") + + +def test_dense_step_takes_the_workspace(env): + """A step the engine skipped the indexer for: sixteen requests each within index_topk (fresh, + behind a cached prefix, and one-token tails), every row selecting its whole causal window. + N is at most 16 x 2048, so the workspace path, layer 35's rows gathered into the kept rows + after the step's own.""" + contexts = [(0, 2048)] * 4 + [(1024, 1024)] * 4 + [(2047, 1)] * 4 + [(0, 700), (300, 5)] * 2 + s = _step(env, contexts, seed=8, skip_ctx=True) + assert all(c + n <= _TOPK for c, n in contexts) and len(contexts) == 16 + ctx, out = _attend(env, s, 35) + assert ctx.ws_buf is not None and ctx.ws_buf.data_ptr() == env.buf.data_ptr() + n_kv = sum(s.kv_lens) + assert set(ctx.ws) == {35} and ctx.ws[35] is ctx.kept + assert ctx.kept.data_ptr() == env.buf.data_ptr() + n_kv * _LAT_DIM * env.buf.element_size() + _check_workspace_path(env, s, out, 35, "dense step") + + +def _short_ctx(env, s, n: int) -> SimpleNamespace: + """The workspace path's context over only the first `n` of the step's N KV rows: what an N + computed short would hand the gather and the prefill kernel.""" + cu = s.step["cu_ctx_kv"] + lens = cu[1:] - cu[:-1] + kv_req = torch.repeat_interleave(torch.arange(len(s.contexts), device=env.dev), lens) + kv_pos = torch.arange(kv_req.numel(), device=env.dev) - cu[:-1][kv_req] + return SimpleNamespace( + kv_req=kv_req[:n].to(torch.int32), kv_pos=kv_pos[:n].to(torch.int32), ws_off=cu[:-1], + ws_buf=env.buf[:n], kept=None, ws={}, + ) # fmt: skip + + +def test_short_n_fails_the_harness(env): + """The control: a workspace one row short of N, and one of the new tokens alone (the + engine's `num_ctx_kv_tokens`), drop the selections at or past its end with no error and no + NaN; the harness's checks refuse both.""" + s = _step(env, [(4096, 640), (0, 300)], seed=9) + n = sum(s.kv_lens) + for short in (n - 1, s.step["num_ctx_kv_tokens"]): + ctx = _short_ctx(env, s, short) + out = _attend(env, s, 1, ctx=ctx)[1] + lat = out[: s.tc] + sel = s.topk[: s.tc] + idx = torch.where(sel >= 0, sel + ctx.ws_off[s.req_idx[: s.tc].long()][:, None], sel) + dropped = int((idx >= short).sum()) + print(f"CTXBOUND control: N {n}, workspace of {short} rows: {dropped} selections dropped, " + f"output finite {bool(torch.isfinite(lat).all())}") # fmt: skip + assert dropped > 0 and not bool(torch.isnan(lat.float()).any()) + with pytest.raises(AssertionError): + _check_workspace_path(env, s, out, 1, f"short {short}") + + +def test_workspace_held_and_no_gather_past_b(env): + """After every step above and a sequence of its own (sixteen requests filling B exactly, + then B + 1, then a 1M chunk), the workspace is the estimate's step's allocation, every + gather wrote into it -- the step's first N rows, or layer 35's into the N after them on a + step that fits twice -- and no step past B reached the gather.""" + for contexts, path in ( + ([(30720, 2048)] * 16, "workspace"), + ([(30720, 2048)] * 15 + [(30721, 2048)], "decode"), + ([(_PREFIX_1M, 32768)], "decode"), + ([(5, 300)], "workspace"), + ): + s = _step(env, contexts, seed=10) + gathered = len(env.gathers) + ctx, out = _attend(env, s, 36) + assert (ctx.ws_buf is None) == (path == "decode"), (contexts[0], path) + assert len(env.gathers) - gathered == (path == "workspace") + assert bool(torch.isfinite(out[: s.tc]).all()) + assert env.target._ws_buf is env.buf + assert env.gathers and all(x.rows <= _B for x in env.gathers) + base, row_bytes = env.buf.data_ptr(), _LAT_DIM * env.buf.element_size() + kept = [x for x in env.gathers if x.out_ptr != base] + assert all(x.out_ptr == base + x.rows * row_bytes for x in kept) + assert all(2 * x.rows <= _B and x.pool_row == _OWNERS.index(35) for x in kept) + print(f"CTXBOUND {len(env.gathers)} gathers, at most {max(x.rows for x in env.gathers)} " + f"rows, all into the workspace allocated on the estimate's step") # fmt: skip diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_ctx_workspace.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_ctx_workspace.py new file mode 100644 index 000000000000..4903fb435b69 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_ctx_workspace.py @@ -0,0 +1,242 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The context workspace of the Mistral-Large-4-1T-A52B NVFP4 dep target: the KV rows its +dequantizing gather and prefill kernel address. Under KV block reuse a context request's KV +is its cached prefix plus its new tokens, so the workspace covers both, while the engine's +`num_ctx_kv_tokens` counts the new tokens only. Under chunked prefill every chunk after a +prompt's first is such a request, its earlier chunks the cached prefix. + +The workspace is the first rows of one held bf16 [rows, 576] buffer, allocated on the first +context step, its rows sized from the engine's quantities (`_ctx_workspace_rows`): 65536 at the +32K serving config (max_seq_len = max_num_tokens = 32768, chunked prefill and block reuse off), +2**19 at the 1M config (max_seq_len 1048576, chunked prefill). A step whose context KV fits the +held rows twice has the rows after its own (`kept`) for layer 35's gather; a step whose context +KV is past the held rows takes the decode path, which has no workspace +(test_modeling_v2_ctx_bound.py drives both on a device). No device: the step is built from +host-side metadata.""" + +import itertools +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch.attention.backends.interface import AttentionRuntimeFeatures +from tensorrt_llm._torch.attention.backends.sparse.dsa.metadata import DSAtrtllmAttentionMetadata +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_103__dep import ( # noqa: E501 + modeling as M, +) + +pytestmark = pytest.mark.cpu_only + +_LAT_DIM = 576 +# The 1M config, per rank: max_seq_len, and the most tokens one step carries. +_MAX_SEQ_LEN = 1048576 +_CHUNK = 32768 +# The engines, per rank, as (max_seq_len, chunked prefill): the 32K serving config and the 1M +# config, both at max_num_tokens 32768, 16 requests and block reuse off. +_ENGINES = {"32k": (32768, False), "1m": (_MAX_SEQ_LEN, True)} + + +def _held_rows(engine: str) -> int: + """The held workspace's rows `_derive_first_step` gives the core on `engine`: 65536 at the + 32K config (twice its max_num_tokens), 2**19 at the 1M config.""" + max_seq_len, chunked = _ENGINES[engine] + md = SimpleNamespace( + max_num_tokens=_CHUNK, + max_num_requests=16, + max_seq_len=max_seq_len, + runtime_features=AttentionRuntimeFeatures(chunked_prefill=chunked, cache_reuse=False), + ) + rows = M._ctx_workspace_rows(md) + assert rows == {"32k": 65536, "1m": 1 << 19}[engine] + return rows + + +def _target(engine: str) -> M.PrefillTarget: + """A prefill target holding just what its context workspace reads off the core on `engine`.""" + target = M.PrefillTarget.__new__(M.PrefillTarget) + target.core = SimpleNamespace(lat_dim=_LAT_DIM, ctx_workspace_rows=_held_rows(engine)) + return target + + +def _cumsum(lens): + return torch.tensor([0] + lens, dtype=torch.int64).cumsum(0) + + +def _check_decode_path(ws, tc): + """A step past the held workspace: no workspace, the context rows in sub-batches of at most + 8192, one scheduler per sub-batch size.""" + assert ws.ws_buf is None + assert ws.sub_batches == [(lo, min(lo + 8192, tc)) for lo in range(0, tc, 8192)] + assert sorted(ws.sched) == sorted({hi - lo for lo, hi in ws.sub_batches}) + + +def _metadata(cached, new): + """The fields `_ctx_kv_offsets` reads, as the engine fills them for context requests with + `cached` reused (or earlier-chunk) and `new` scheduled tokens each.""" + if any(cached): + return SimpleNamespace( + enable_context_mla_with_cached_kv=True, + ctx_cached_token_indptr=_cumsum(cached), + ctx_kv_indptr=_cumsum([c + n for c, n in zip(cached, new)]), + max_ctx_seq_len=max(new), + num_ctx_cached_tokens=sum(cached), + ) + return SimpleNamespace( + enable_context_mla_with_cached_kv=False, + mla_prepare_ctx_cu_seqlens=lambda: _cumsum(new).to(torch.int32), + seq_lens=torch.tensor(new, dtype=torch.int32), + ) + + +def _check_workspace_path(target, ws, n_kv: int) -> None: + """A step on the held workspace: its first `n_kv` rows, and the `n_kv` after them kept for + layer 35's gather when the context KV fits twice.""" + held = target._ws_buf.shape[0] + assert ws.ws_buf.shape == (n_kv, _LAT_DIM) and ws.ws_buf.dtype == torch.bfloat16 + assert ws.ws_buf.data_ptr() == target._ws_buf.data_ptr() and ws.ws == {} + if 2 * n_kv <= held: + assert ws.kept.shape == (n_kv, _LAT_DIM) + assert ws.kept.data_ptr() == ws.ws_buf.data_ptr() + n_kv * _LAT_DIM * 2 + else: + assert ws.kept is None + + +@pytest.mark.parametrize( + "engine, cached, new", + [ + pytest.param("32k", [0, 0], [7, 33], id="32k-no_reuse"), + pytest.param("32k", [64, 0, 96], [5, 40, 1], id="32k-mixed_prefixes"), + pytest.param("32k", [32], [1], id="32k-one_new_token"), + pytest.param("32k", [0], [_CHUNK], id="32k-whole_step"), + pytest.param("1m", [0, 0], [7, 33], id="1m-no_reuse"), + pytest.param("1m", [64, 0, 96], [5, 40, 1], id="1m-mixed_prefixes"), + pytest.param("1m", [32], [1], id="1m-one_new_token"), + # A 1048559-token prompt's last chunk at the 1M config (31 chunks of 32768 cached), + # beside a first chunk of another prompt on the same rank. + pytest.param("1m", [31 * _CHUNK, 0], [32751, 9], id="1m-chunk_over_1m_prefix"), + # Two 5-token last chunks over 983040-token prefixes, packed into one indexer chunk of + # 1966090 keys, past max_seq_len. + pytest.param("1m", [983040, 983040], [5, 5], id="1m-packed_tails"), + ], +) +def test_workspace_covers_the_cached_prefix_and_the_new_tokens(engine, cached, new): + """The step's workspace is its context KV, cached and new, in the held buffer of the engine's + rows; past them, the decode path.""" + step = dict( + num_contexts=len(new), + num_ctx_tokens=sum(new), + num_ctx_kv_tokens=sum(new), + **M._ctx_kv_offsets(_metadata(cached, new), len(new)), + ) + target = _target(engine) + ws = target._ctx_workspace(step, sum(new), torch.device("cpu")) + + kv_lens = [c + n for c, n in zip(cached, new)] + held = target.core.ctx_workspace_rows + assert target._ws_buf.shape == (held, _LAT_DIM) and target._ws_buf.dtype == torch.bfloat16 + if sum(kv_lens) > held: + _check_decode_path(ws, sum(new)) + return + _check_workspace_path(target, ws, sum(kv_lens)) + assert ws.kv_req.tolist() == [r for r, n in enumerate(kv_lens) for _ in range(n)] + assert ws.kv_pos.tolist() == [p for n in kv_lens for p in range(n)] + assert ws.ws_off.tolist() == _cumsum(kv_lens)[:-1].tolist() + + +def _engine_step(contexts, decode_kv_lens=()): + """One dep4 rank's step as the engine's DSA metadata prepares it under chunked prefill: + `contexts` lists each context request's (cached prefix, new tokens), `decode_kv_lens` each + generation row's KV length. `DSAtrtllmAttentionMetadata.prepare` takes the KV lengths as + cached + new (`cached_token_lens + seq_lens_kv`) and hands both to its own + `prepare_for_mla_rope_append`, which fills the cached count and the cached and total prefix + sums run here; the new tokens' host prefix sum is the base metadata's. The metadata holds + what that method reads, its buffers at this step's size.""" + num_contexts, num_generations = len(contexts), len(decode_kv_lens) + seq_lens = [new for _, new in contexts] + [1] * num_generations + cached = [prefix for prefix, _ in contexts] + [kv_len - 1 for kv_len in decode_kv_lens] + num_seqs = num_contexts + num_generations + i64 = dict(dtype=torch.int64) + md = SimpleNamespace( + num_contexts=num_contexts, + num_generations=num_generations, + num_seqs=num_seqs, + seq_lens=torch.tensor(seq_lens, dtype=torch.int32), + enable_context_mla_with_cached_kv=True, + host_ctx_uncached_token_indptr=torch.tensor( + [0, *itertools.accumulate(seq_lens[:num_contexts])], **i64 + ), + **{ + name: torch.zeros(num_seqs + 1, **i64) + for name in ( + "host_ctx_cached_token_indptr", + "ctx_cached_token_indptr", + "host_ctx_kv_indptr", + "ctx_kv_indptr", + "host_gen_cached_token_indptr", + "gen_cached_token_indptr", + "host_gen_kv_indptr", + "gen_kv_indptr", + ) + }, + ) + cached_token_lens = torch.tensor(cached, dtype=torch.int) + kv_lens = cached_token_lens + md.seq_lens + DSAtrtllmAttentionMetadata.prepare_for_mla_rope_append(md, cached_token_lens, kv_lens) + return md + + +@pytest.mark.parametrize( + "contexts, decode_kv_lens", + [ + # A 1048559-token prompt's first, second and last chunks, the last with 15 decode rows + # (up to max_seq_len keys) behind it. + ([(0, _CHUNK)], ()), + ([(_CHUNK, _CHUNK)], ()), + ([(31 * _CHUNK, 32751)], [_MAX_SEQ_LEN - 1, 2, 100000] + [4096] * 12), + # Its last chunk beside another prompt's first, and two packed 5-token tails. + ([(31 * _CHUNK, 32751), (0, 9)], ()), + ([(983040, 5), (983040, 5)], [700000]), + ], + ids=["first_chunk", "second_chunk", "last_chunk_with_decode", "last_and_first", "packed_tails"], +) +def test_n_kv_is_the_engines_cached_plus_new_total(contexts, decode_kv_lens): + """`_ctx_workspace` sizes the workspace `ctx_cached_tokens + num_ctx_tokens`: on every step + at the 1M config that is the context requests' cached + new total the engine itself sums + (`num_ctx_mla_kv_tokens`, the last entry of `ctx_kv_indptr`), whatever the generation rows + behind them hold; on every chunk after a prompt's first it is more than the step's new + tokens. The host prefix sums (`_ctx_host_offsets`) end at the same total and at the new + tokens'.""" + md = _engine_step(contexts, decode_kv_lens) + num_contexts = len(contexts) + new = [n for _, n in contexts] + step = dict( + num_contexts=num_contexts, + num_ctx_tokens=sum(new), + **M._ctx_kv_offsets(md, num_contexts), + **M._ctx_host_offsets(md, num_contexts), + ) + total = sum(prefix + n for prefix, n in contexts) + assert step["ctx_cached_tokens"] == sum(prefix for prefix, _ in contexts) + assert step["ctx_cached_tokens"] + step["num_ctx_tokens"] == total + assert total == md.num_ctx_mla_kv_tokens == int(md.host_ctx_kv_indptr[num_contexts]) + assert int(step["cu_ctx_kv"][-1]) == total + assert step["host_cu_ctx_kv"][-1] == total and step["host_cu_ctx_new"][-1] == sum(new) + if any(prefix for prefix, _ in contexts): + assert total > step["num_ctx_tokens"] + + target = _target("1m") + ws = target._ctx_workspace(step, sum(new), torch.device("cpu")) + assert target._ws_buf.shape == (M._CTX_WORKSPACE_ROWS, _LAT_DIM) + if total > M._CTX_WORKSPACE_ROWS: + _check_decode_path(ws, sum(new)) + return + _check_workspace_path(target, ws, total) + assert ws.kv_req.shape == ws.kv_pos.shape == (total,) + starts = [0, *itertools.accumulate(prefix + n for prefix, n in contexts)][:-1] + assert ws.ws_off.tolist() == starts + for r, (prefix, n) in enumerate(contexts): + lo, hi = starts[r], starts[r] + prefix + n + assert int(ws.kv_req[lo]) == int(ws.kv_req[hi - 1]) == r + assert (int(ws.kv_pos[lo]), int(ws.kv_pos[hi - 1])) == (0, prefix + n - 1) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_dep4_1m_engine_config.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_dep4_1m_engine_config.py new file mode 100644 index 000000000000..df03ce6546c7 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_dep4_1m_engine_config.py @@ -0,0 +1,1762 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Mistral-Large-4-1T-A52B-NVFP4 at dep4: the engine's configuration at the checkpoint's own +context, and the attention-DP contract the dep4 forward is written against. + +No GPU and no weights. The checkpoint goes through the engine's own configuration path with the +LLM-API options of the dep4 core unit (four ranks, attention data parallel, the routed experts +expert-parallel four ways, 16 requests and 32768 tokens per rank, the fp8_ds_mla latent KV +cache, KV block reuse off) at the 1M config: max_seq_len 1048576 (``max_position_embeddings``) +with chunked prefill on, so a context step carries at most one 32768-token chunk per rank. The +same configuration is reached three ways: from ``spec-1m-kvds-dep4/options/core.yaml`` (the +accuracy and serve gates) and from the argv the boot gate hands +``examples/llm-api/quickstart_advanced.py``, through that script's own parser and +``setup_llm``, both with kv_cache_config.dtype fp8; and with the dtype left at auto, as the CI +class passes its KvCacheConfig. All three build ``MistralLarge4CacheManager``. + +* the LLM API front end, ``TorchLlmArgs`` and ``ModelLoader._update_from_hf_quant_config`` + (``llmapi/llm_utils.py``), which settle the key the accuracy harness looks its anchor up by; +* each rank, ``_load_config_and_create_checkpoint_loader`` (``py_executor_creator``) then the + model loader's ``_load_and_validate_config``, which builds the ``ModelConfig`` and asks the + modeling_v2 router for a target. The router's ``route`` is replaced by a recorder, so what it + is handed is observed rather than judged. + +What the 1M config changes is then pinned on the engine's own host-side code: the request +admission (``BaseLLM._check_arguments`` and the worker's ``max_tokens`` deduction), the +context-MLA-with-cached-KV switch and the DSA indexer's prefill chunk plan +(``Indexer.prepare_for_chunked_prefill``) for chunks over a cached prefix up to 1M keys, the DSA +KV tables' width, and what boot runs before the KV pool is sized (the KV-capacity estimation's +dummy requests, the context-MLA workspace reserve, the sizes the target fixes on its first step +from this engine's metadata -- its held context workspace, its indexer logits budget and its +expert calls -- the warmup's shapes and the CUDA-graph capture's longest row). Under attention +DP four ranks are simulated in one process (each rank a thread, each collective a barrier). +Device-dependent values (the SM version) are fixed to sm_103 here; nothing is asserted that a +visible device would decide. + +The checkpoint is Mistral-native, as published (params.json, tekken.json, consolidated +safetensors), and every spelling loads it with checkpoint_format mistral (the options' +key, the boot gate's --checkpoint_format=mistral). The delivered 32K dep4 configuration +(unchunked) and the checkpoint at tep4 and tp4 are checked to resolve and route as they did +before the 1M config. The checkpoint is looked up under ``LLM_MODELS_ROOT`` (as +``Mistral-Large-4-1T-A52B-NVFP4`` or ``mistral-large-4-nvfp4``) by its params.json, and its tests +are skipped where it is absent. +""" + +from __future__ import annotations + +import copy +import importlib.util +import itertools +import json +import math +import sys +import threading +from pathlib import Path +from types import MethodType, SimpleNamespace + +import pytest +import torch +import yaml +from test_common.llm_data import llm_models_root + +from tensorrt_llm._torch.attention.backends import trtllm as trtllm_backend +from tensorrt_llm._torch.attention.backends.interface import ( + AttentionMetadata, + AttentionRuntimeFeatures, +) +from tensorrt_llm._torch.attention.backends.sparse.dsa import DSACacheManagerV2 +from tensorrt_llm._torch.attention.backends.sparse.dsa import indexer as dsa_indexer +from tensorrt_llm._torch.attention.backends.sparse.dsa.indexer import ( + Indexer, + _packed_logits_elem_budget, + _split_prefill_queries, +) +from tensorrt_llm._torch.attention.backends.sparse.dsa.metadata import ( + _MAX_RADIX_BLOCKS_PER_ROW, + DSAtrtllmAttentionMetadata, +) +from tensorrt_llm._torch.attention.backends.sparse.mistral_large4 import MistralLarge4CacheManager +from tensorrt_llm._torch.attention.backends.utils import get_attention_backend +from tensorrt_llm._torch.distributed.communicator import Distributed +from tensorrt_llm._torch.modeling_v2 import _router_index +from tensorrt_llm._torch.modeling_v2._router_index import NoTarget +from tensorrt_llm._torch.modeling_v2.models.mistral_large4 import routing +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_103__dep import ( # noqa: E501 + modeling as M, +) +from tensorrt_llm._torch.models.modeling_utils import DecoderModelForCausalLM +from tensorrt_llm._torch.pyexecutor import model_loader, py_executor_creator +from tensorrt_llm._torch.pyexecutor._util import ( + CacheCost, + KvCacheCreator, + _get_num_pool_groups_for_estimation, + get_attention_workspace_bytes_per_token, + get_attention_workspace_fixed_bytes, + get_attention_workspace_is_chunked_prefill_bounded, + get_kv_cache_manager_cls, + get_mla_context_workspace_kv_len_cap, + should_enable_adp_dummy_fixes, + should_enable_non_overlap_adp_forward_intent, + should_enable_scheduler_aware_adp_dummy, +) +from tensorrt_llm._torch.pyexecutor.cuda_graph_runner import CUDAGraphRunner, CUDAGraphRunnerConfig +from tensorrt_llm._torch.pyexecutor.engine.metadata import build_attention_metadata +from tensorrt_llm._torch.pyexecutor.engine.runners.common import ( + get_all_rank_num_tokens, + pad_input_tokens, +) +from tensorrt_llm._torch.pyexecutor.engine.runners.decoder.dummy_requests import DummyRequestFactory +from tensorrt_llm._torch.pyexecutor.engine.runners.decoder.warmup import ( + DecoderWarmup, + WarmupAgreement, +) +from tensorrt_llm._torch.pyexecutor.kv_cache.kv_cache_manager_v2 import ( + BASE_GENERATION_TOKEN_COUNT, + KVCacheManagerV2, + _get_generation_kv_capacity, +) +from tensorrt_llm._torch.pyexecutor.llm_request import ATTENTION_DP_DUMMY_REQUEST_ID +from tensorrt_llm._torch.pyexecutor.model_engine import ModelEngine +from tensorrt_llm._torch.pyexecutor.model_loader import ModelLoader as RankModelLoader +from tensorrt_llm._torch.pyexecutor.py_executor import PyExecutor +from tensorrt_llm._torch.pyexecutor.request_utils import derive_attention_dp_per_rank_request_cap +from tensorrt_llm._torch.pyexecutor.resource_manager import ResourceManagerType +from tensorrt_llm._torch.pyexecutor.scheduler import ScheduledRequests +from tensorrt_llm._torch.pyexecutor.scheduler.adp_router import ADPRouter, DefaultADPRouter +from tensorrt_llm._torch.pyexecutor.scheduler.rank_state import RankState +from tensorrt_llm.bindings import DataType +from tensorrt_llm.bindings.internal.batch_manager import CacheType +from tensorrt_llm.executor.base_worker import BaseWorker +from tensorrt_llm.executor.request import GenerationRequest +from tensorrt_llm.executor.utils import RequestError, is_context_length_exceeded_message +from tensorrt_llm.llmapi.llm import BaseLLM +from tensorrt_llm.llmapi.llm_args import PrefillCudaGraphBackend, TorchLlmArgs +from tensorrt_llm.llmapi.llm_utils import ModelLoader as FrontEndModelLoader +from tensorrt_llm.models.modeling_utils import QuantAlgo +from tensorrt_llm.sampling_params import SamplingParams + +pytestmark = pytest.mark.cpu_only + +_SM103 = (10, 3) +_RANKS = 4 +# max_num_tokens per rank: under chunked prefill, the most context rows one step carries. +_MAX_NUM_TOKENS = 32768 +# The checkpoint's max_position_embeddings, the 1M config's max_seq_len. +_MAX_SEQ_LEN = 1 << 20 +_TOKENS_PER_BLOCK = 32 + +# The topologies. dep4 is the core unit's; tep4 and tp4 are the other NVFP4 targets' +# (spec-tep4 / spec-tp4 options). +_DEP4 = dict(tensor_parallel_size=4, moe_expert_parallel_size=4, enable_attention_dp=True) +_TEP4 = dict(tensor_parallel_size=4, moe_expert_parallel_size=4, enable_attention_dp=False) +_TP4 = dict( + tensor_parallel_size=4, + moe_tensor_parallel_size=4, + moe_expert_parallel_size=1, + enable_attention_dp=False, +) + + +def _options( + topology: dict, + max_num_tokens: int, + max_seq_len: int | None = None, + chunked=False, + kv_dtype: str | None = None, +) -> dict: + """The LLM-API options of a Mistral Large 4 gate on ``topology``, the checkpoint in its + Mistral-native format, KV block reuse off and the latent KV cache at kv_cache_config's + default dtype (auto: the checkpoint's fp8_kv_cache) unless ``kv_dtype`` names one. Under + attention DP max_batch_size and max_num_tokens bind each rank.""" + kv_cache_config = dict(enable_block_reuse=False, free_gpu_memory_fraction=0.85) + if kv_dtype is not None: + kv_cache_config.update(dtype=kv_dtype) + return dict( + checkpoint_format="mistral", + **topology, + max_batch_size=16, + max_num_tokens=max_num_tokens, + max_seq_len=max_seq_len or max_num_tokens, + enable_chunked_prefill=chunked, + cuda_graph_config=dict(enable_padding=True, max_batch_size=16), + kv_cache_config=kv_cache_config, + ) + + +# spec-1m-kvds-dep4/options/core.yaml: the gates pass kv_cache_config.dtype fp8. +_DEP4_1M = _options(_DEP4, _MAX_NUM_TOKENS, _MAX_SEQ_LEN, chunked=True, kv_dtype="fp8") +# The CI class's (TestModelingV2MistralLarge4Nvfp4Sm103Dep): the same config, the +# dtype left at auto. +_DEP4_1M_CI = _options(_DEP4, _MAX_NUM_TOKENS, _MAX_SEQ_LEN, chunked=True) + +# The argv the boot gate hands quickstart_advanced.py (workspace +# gates/geometry/geometry.core.dep4.json), after its --model_dir. +_QUICKSTART_ARGV = ( + "--checkpoint_format=mistral", + "--max_tokens=2", + "--tp_size=4", + "--moe_ep_size=4", + "--enable_attention_dp", + "--use_cuda_graph", + "--cuda_graph_batch_sizes", + "1", + "2", + "4", + "8", + "16", + "--enable_chunked_prefill", + "--disable_kv_cache_reuse", + "--max_seq_len=1048576", + "--max_batch_size=16", + "--max_num_tokens=32768", + "--cuda_graph_padding_enabled", + "--kv_cache_dtype=fp8", +) +_QUICKSTART = Path(__file__).resolve().parents[7] / "examples/llm-api/quickstart_advanced.py" + +_NVFP4_NAMES = ("Mistral-Large-4-1T-A52B-NVFP4", "mistral-large-4-nvfp4") + +# The GSM8K reference rows, and the key the harness reads them by. +_GSM8K_REFERENCES = ( + Path(__file__).resolve().parents[6] / "integration/defs/accuracy/references/gsm8k.yaml" +) +_MODEL_NAME = "mistralai/Mistral-Large-4-1T-A52B" + +# The per-rank KV pool the delivered 32K dep4 boot gate measured, in tokens, and the bytes a +# token of that pool took: the 576-byte latent on all 72 layers and the indexer key on the 44 +# full-indexer layers. +_DELIVERED_POOL_TOKENS = 2_004_896 +_DELIVERED_BYTES_PER_TOKEN = 576 * 72 + 132 * 44 +# The per-rank pool a boot of the 1M accuracy gate (the Dep4Ctx1m class, KV fraction 0.85) +# measured on the native checkpoint, the vision tower loaded on every rank: 58,747 pages. +_BOOT_POOL_TOKENS = 58_747 * 32 + + +def _checkpoint_dir(names: tuple[str, ...]) -> Path: + """The first of ``names`` under LLM_MODELS_ROOT holding a Mistral-native params.json.""" + root = llm_models_root() + for name in names: + if root is not None and (root / name / "params.json").is_file(): + return root / name + pytest.skip(f"none of {names} is under LLM_MODELS_ROOT ({root})") + + +class _Routed: + """What the recording `route` returns: no target is built here.""" + + +def _route(pretrained_config, mapping): + """`route` for this config on ``mapping``, on sm_103 without speculative decoding.""" + model_config = SimpleNamespace( + pretrained_config=pretrained_config, mapping=mapping, spec_config=None + ) + return routing.route(model_config, None, _SM103) + + +def _resolve(checkpoint_dir: Path, options: dict) -> SimpleNamespace: + """The configuration a served engine reaches for this checkpoint, front end then rank.""" + model = str(checkpoint_dir) + llm_args = TorchLlmArgs(model=model, modeling_v2="require", **options) + prequantized = FrontEndModelLoader(llm_args)._update_from_hf_quant_config() + + routed = [] + + def route(model_config, llm_args, compute_capability): + pretrained_config = model_config.pretrained_config + routed.append( + SimpleNamespace( + pretrained_config=pretrained_config, + quantization_config=copy.deepcopy(pretrained_config.quantization_config), + mapping=model_config.mapping, + compute_capability=compute_capability, + spec_config=model_config.spec_config, + ) + ) + return _Routed + + with pytest.MonkeyPatch.context() as mp: + # The rank's config pass pins the CUDA memory fraction first; there is no device here. + mp.setattr(torch.cuda, "set_per_process_memory_fraction", lambda *args, **kwargs: None) + mp.setattr(_router_index, "_device_compute_capability", lambda: _SM103) + mp.setattr(_router_index, "registered_routes", lambda: {"MistralLarge4ForCausalLM": route}) + # One process stands in for every rank, so the rank exchange is the local resolution. + mp.setattr( + model_loader, "modeling_v2_resolve_on_all_ranks", _router_index.modeling_v2_resolve + ) + llm_args, checkpoint_loader = py_executor_creator._load_config_and_create_checkpoint_loader( + llm_args, model + ) + _, max_num_tokens, max_seq_len, max_batch_size = llm_args.get_runtime_sizes() + mapping = llm_args.parallel_config.to_mapping() + model_config = RankModelLoader( + llm_args=llm_args, + mapping=mapping, + spec_config=llm_args.speculative_config, + sparse_attention_config=llm_args.sparse_attention_config, + max_num_tokens=max_num_tokens, + max_seq_len=max_seq_len, + )._load_and_validate_config(model, checkpoint_loader) + return SimpleNamespace( + path=checkpoint_dir, + llm_args=llm_args, + prequantized=prequantized, + model_config=model_config, + mapping=mapping, + routed=routed, + max_num_tokens=max_num_tokens, + max_seq_len=max_seq_len, + max_batch_size=max_batch_size, + ) + + +def _quickstart_options(checkpoint_dir: Path) -> dict: + """The keyword arguments quickstart_advanced.py hands the LLM constructor for the boot + gate's argv: the script's own ``parse_arguments`` and ``setup_llm``, with ``LLM`` replaced + by a recorder.""" + if not _QUICKSTART.is_file(): + pytest.skip(f"{_QUICKSTART} is not in this checkout") + spec = importlib.util.spec_from_file_location("quickstart_advanced", _QUICKSTART) + quickstart = importlib.util.module_from_spec(spec) + spec.loader.exec_module(quickstart) + with pytest.MonkeyPatch.context() as mp: + mp.setattr( + sys, + "argv", + ["quickstart_advanced.py", f"--model_dir={checkpoint_dir}", *_QUICKSTART_ARGV], + ) + mp.setattr(quickstart, "LLM", lambda **kwargs: kwargs) + kwargs, _ = quickstart.setup_llm(quickstart.parse_arguments()) + assert kwargs.pop("model") == str(checkpoint_dir) + return kwargs + + +@pytest.fixture(scope="module") +def dep4(): + return _resolve(_checkpoint_dir(_NVFP4_NAMES), _DEP4_1M) + + +@pytest.fixture(scope="module") +def dep4_quickstart(): + checkpoint_dir = _checkpoint_dir(_NVFP4_NAMES) + return _resolve(checkpoint_dir, _quickstart_options(checkpoint_dir)) + + +@pytest.fixture(scope="module", params=["core_yaml", "quickstart"]) +def dep4_1m(request): + """The 1M dep4 engine, as the accuracy gate's options and as the boot gate's argv build it.""" + return request.getfixturevalue("dep4" if request.param == "core_yaml" else "dep4_quickstart") + + +@pytest.fixture(scope="module") +def dep4_ci(): + """The 1M dep4 engine as the CI class builds it: kv_cache_config.dtype left at auto.""" + return _resolve(_checkpoint_dir(_NVFP4_NAMES), _DEP4_1M_CI) + + +@pytest.fixture(scope="module") +def dep4_32k(): + """The delivered dep4 configuration: 32768 tokens per rank, unchunked.""" + return _resolve(_checkpoint_dir(_NVFP4_NAMES), _options(_DEP4, _MAX_NUM_TOKENS)) + + +@pytest.fixture(scope="module") +def tep4(): + return _resolve(_checkpoint_dir(_NVFP4_NAMES), _options(_TEP4, _MAX_NUM_TOKENS)) + + +@pytest.fixture(scope="module") +def tp4(): + return _resolve(_checkpoint_dir(_NVFP4_NAMES), _options(_TP4, _MAX_NUM_TOKENS)) + + +def _anchor(llm_args) -> float: + """The reference row `AccuracyTask.get_hypothesis_testing_params` selects for this engine.""" + quant = llm_args.quant_config + acc_specs = dict( + dtype=llm_args.dtype, + quant_algo=quant.quant_algo, + kv_cache_quant_algo=quant.kv_cache_quant_algo, + spec_dec_algo=None, + extra_acc_spec=None, + ) + for entry in yaml.safe_load(_GSM8K_REFERENCES.read_text())[_MODEL_NAME]: + if all( + entry.get(key, "auto" if key == "dtype" else None) == value + for key, value in acc_specs.items() + ): + return entry["accuracy"] + raise AssertionError(f"no {_MODEL_NAME} GSM8K reference row for {acc_specs}") + + +def _rank_mapping(mapping, rank: int): + rank_mapping = copy.deepcopy(mapping) + rank_mapping.rank = rank + return rank_mapping + + +def _kv_cost(resolved, rank: int) -> CacheCost: + """The engine's affine KV cost on ``rank``, as the KV-cache creator sizes the pool with it.""" + model_config, llm_args = resolved.model_config, resolved.llm_args + kv_cache_config = llm_args.kv_cache_config + manager = get_kv_cache_manager_cls(model_config, kv_cache_config) + return CacheCost.from_raw( + manager.get_cache_size_per_token( + model_config, + _rank_mapping(resolved.mapping, rank), + tokens_per_block=kv_cache_config.tokens_per_block, + max_seq_len=resolved.max_seq_len, + max_batch_size=resolved.max_batch_size, + max_num_tokens=resolved.max_num_tokens, + kv_cache_config=kv_cache_config, + spec_config=llm_args.speculative_config, + is_draft=False, + ) + ) + + +def _summary(resolved) -> dict: + """Everything the engine resolved for a checkpoint except its mapping.""" + pretrained = resolved.model_config.pretrained_config.to_dict() + pretrained.pop("_name_or_path", None) + llm_args = resolved.llm_args + return dict( + llm_quant=repr(llm_args.quant_config), + quant=repr(resolved.model_config.quant_config), + layers={k: repr(v) for k, v in (resolved.model_config.quant_config_dict or {}).items()}, + pretrained=pretrained, + sparse=resolved.model_config.sparse_attention_config.model_dump(), + routed=resolved.routed[0].quantization_config, + sizes=(resolved.max_num_tokens, resolved.max_seq_len, resolved.max_batch_size), + chunked_prefill=llm_args.enable_chunked_prefill, + cuda_graphs=llm_args.cuda_graph_config.model_dump(), + kv_cache=llm_args.kv_cache_config.model_dump(), + ) + + +def _runtime_features(resolved) -> AttentionRuntimeFeatures: + """The attention runtime features `_create_py_executor` builds from the LLM arguments.""" + return AttentionRuntimeFeatures( + chunked_prefill=resolved.llm_args.enable_chunked_prefill, + cache_reuse=resolved.llm_args.kv_cache_config.enable_block_reuse, + has_speculative_draft_tokens=False, + chunk_size=resolved.max_num_tokens, + ) + + +def _attention_metadata_kwargs(resolved, rank: int = 0) -> SimpleNamespace: + """What the decoder runner's ``build_attention_metadata`` hands the DSA metadata class, with + the class replaced by a recorder (no buffers are allocated).""" + return build_attention_metadata( + resolved.model_config, + max_batch_size=resolved.max_batch_size, + max_num_tokens=resolved.max_num_tokens, + max_beam_width=1, + attention_backend=SimpleNamespace(Metadata=lambda **kwargs: SimpleNamespace(**kwargs)), + attention_runtime_features=_runtime_features(resolved), + mapping=_rank_mapping(resolved.mapping, rank), + cache_indirection=None, + ) + + +def _v2_max_blocks_per_seq(max_seq_len: int) -> int: + """``KVCacheManagerV2``'s block-table width: the blocks of ``max_seq_len`` plus the generation + headroom (``_get_generation_kv_capacity``), padded to a multiple of 4 for the + copy_block_offsets kernel.""" + _, headroom = _get_generation_kv_capacity(None, is_draft=False) + blocks = -(-(max_seq_len + headroom) // _TOKENS_PER_BLOCK) + return -(-blocks // 4) * 4 + + +# ------------------------------------------------------- four ranks in one process + + +class _TpGroup: + """The four ranks of the attention-DP group in one process: rank r is a thread, and every + exchange is a barrier, so each rank's call sees what every rank sent, as over MPI.""" + + def __init__(self, size: int = _RANKS): + self.size = size + self._barrier = threading.Barrier(size, timeout=60) + self._slots = [None] * size + + def exchange(self, rank: int, obj): + self._slots[rank] = obj + self._barrier.wait() + gathered = list(self._slots) + self._barrier.wait() + return gathered + + def run(self, step): + """``step(rank)`` on every rank at once; the per-rank results in rank order.""" + results, errors = [None] * self.size, [None] * self.size + + def body(rank): + try: + results[rank] = step(rank) + except BaseException as error: # noqa: BLE001 - re-raised below, after every rank + errors[rank] = error + self._barrier.abort() + + threads = [threading.Thread(target=body, args=(rank,)) for rank in range(self.size)] + for thread in threads: + thread.start() + for thread in threads: + thread.join() + # The rank that failed first, not a peer it left waiting at the barrier. + raised = [error for error in errors if error is not None] + for error in raised: + if not isinstance(error, threading.BrokenBarrierError): + raise error + if raised: + raise raised[0] + return results + + def dist(self, rank: int) -> "_RankDist": + return _RankDist(self, rank) + + +class _RankDist: + """One rank's communicator: the gathers the engine's host code issues under attention DP.""" + + # The engine's own int64 gather, over the `tp_allgather` below. + tp_allgather_int64 = Distributed.tp_allgather_int64 + + def __init__(self, group: _TpGroup, rank: int): + self.group = group + self.rank = self.tp_rank = rank + self.world_size = self.tp_size = group.size + self.has_cp_helix = False + + def tp_allgather(self, obj, *, small_payload: bool = False): + return self.group.exchange(self.rank, obj) + + def allgather(self, obj): + return self.group.exchange(self.rank, obj) + + +_request_ids = itertools.count(1) + + +def _decode_row(kv_len: int, *, dummy: bool = False) -> SimpleNamespace: + """A generation request whose step attends ``kv_len`` keys (its own token included).""" + return SimpleNamespace( + py_request_id=next(_request_ids), + py_draft_tokens=[], + max_beam_num_tokens=kv_len, + is_dummy=dummy, + is_cuda_graph_dummy=False, + py_batch_idx=None, + ) + + +def _context_row(tokens: int) -> SimpleNamespace: + return SimpleNamespace( + py_request_id=next(_request_ids), + is_last_context_chunk=True, + context_chunk_size=tokens, + encoder_output_len=0, + py_skip_cross_kv_projection=False, + ) + + +def _batch(*, decode=(), context=()) -> ScheduledRequests: + batch = ScheduledRequests() + batch.reset_context_requests([_context_row(n) for n in context]) + batch.generation_requests = [ + row if isinstance(row, SimpleNamespace) else _decode_row(row) for row in decode + ] + return batch + + +class _KvPool: + """The KV manager surface the warmup's dummy requests read: ``tokens`` free tokens in pages of + 32, and dummies recorded as the token counts they were asked for (a generation dummy with no + count holds one token, as the engine's managers default).""" + + tokens_per_block = _TOKENS_PER_BLOCK + num_extra_kv_tokens = 0 + + def __init__(self, tokens: int, max_seq_len: int): + self.tokens = tokens + self.max_seq_len = max_seq_len + self.calls = [] + + def get_num_available_tokens(self, *, token_num_upper_bound, max_num_draft_tokens=0, **_): + return min(token_num_upper_bound, self.tokens) + + def get_num_free_blocks(self) -> int: + return self.tokens // self.tokens_per_block + + def add_dummy_requests(self, request_ids, token_nums=None, is_gen=False, **kwargs): + self.calls.append( + dict(request_ids=request_ids, token_nums=token_nums, is_gen=is_gen, **kwargs) + ) + if token_nums is None: + token_nums = [1 + kwargs.get("max_num_draft_tokens", 0)] * len(request_ids) + return [ + SimpleNamespace( + py_request_id=request_id, + token_num=token_num, + is_gen=is_gen, + is_last_context_chunk=True, + is_attention_dp_dummy=False, + py_draft_tokens=[], + py_needs_onehot_draft_probs=False, + ) + for request_id, token_num in zip(request_ids, token_nums) + ] + + def free_resources(self, request) -> None: + pass + + +class _Resources: + def __init__(self, kv: _KvPool): + self.kv = kv + + def get_resource_manager(self, key): + return self.kv if key == ResourceManagerType.KV_CACHE_MANAGER else None + + +def _runner_config(resolved) -> SimpleNamespace: + """The decoder runner's sizes for this engine. No speculative decoding: every draft width is + zero, and beams are one wide.""" + graphs = resolved.llm_args.cuda_graph_config + return SimpleNamespace( + max_num_tokens=resolved.max_num_tokens, + max_seq_len=resolved.max_seq_len, + max_batch_size=resolved.max_batch_size, + cuda_graph_batch_sizes=graphs.batch_sizes, + max_beam_width=1, + spec_config=None, + use_mrope=False, + max_draft_len=0, + original_max_draft_len=0, + max_total_draft_tokens=0, + max_draft_loop_tokens=0, + ) + + +def _estimation_creator(resolved) -> KvCacheCreator: + """The KV-cache creator's state its estimation workload reads, for this engine: the model + engine's sizes after ``_init_model_capacity``, the V2 manager, no speculation.""" + creator = KvCacheCreator.__new__(KvCacheCreator) + creator._model_engine = SimpleNamespace( + model=SimpleNamespace(model_config=resolved.model_config), + batch_size=resolved.max_batch_size, + max_cuda_graph_batch_size=resolved.llm_args.cuda_graph_config.max_batch_size, + max_seq_len=resolved.max_seq_len, + ) + creator._max_num_tokens = resolved.max_num_tokens + creator._max_beam_width = 1 + creator._mapping = resolved.mapping + creator._net_max_seq_len = resolved.max_seq_len + creator._dummy_reqs = None + creator._speculative_config = None + creator._llm_args = resolved.llm_args + creator._tokens_per_block = resolved.llm_args.kv_cache_config.tokens_per_block + creator._kv_cache_config = resolved.llm_args.kv_cache_config + creator._is_kv_cache_manager_v2 = issubclass( + get_kv_cache_manager_cls(resolved.model_config, creator._kv_cache_config), KVCacheManagerV2 + ) + return creator + + +# The estimation's KV quota at the 1M config, per rank: the CUDA-graph capture's longest row +# (ceil((1048576 + 1) / 32) blocks) beside the 15 other rows of the largest captured batch. +_ESTIMATION_BLOCKS = -(-(_MAX_SEQ_LEN + 1) // _TOKENS_PER_BLOCK) + 16 - 1 +_ESTIMATION_TOKENS = _ESTIMATION_BLOCKS * _TOKENS_PER_BLOCK + + +# ------------------------------------------------------- (1) topology, limits, quantization + + +def test_dep4_topology_and_per_rank_limits(dep4_1m): + """Four ranks, tensor_parallel_size 4 spent on attention data parallelism (each rank holds + every head; attention TP is 1), the routed experts expert-parallel four ways and not split + along their width, one pipeline stage. lm_head is not vocab-parallel under attention DP. + max_batch_size and max_num_tokens are each rank's own: 16 requests and 32768 tokens per rank, + 64 requests in flight over the four. max_seq_len is the checkpoint's 1048576, chunked prefill + is on (a step carries at most one 32768-token chunk per rank) and KV block reuse is off. The + accuracy gate's options and the boot gate's quickstart argv agree on all of it.""" + resolved = dep4_1m + llm_args, mapping = resolved.llm_args, resolved.mapping + parallel = llm_args.parallel_config + assert (parallel.tp_size, parallel.moe_ep_size, parallel.pp_size) == (4, 4, 1) + assert llm_args.enable_attention_dp is True and parallel.enable_attention_dp is True + + assert (mapping.world_size, mapping.tp_size, mapping.pp_size, mapping.cp_size) == (4, 4, 1, 1) + assert (mapping.moe_ep_size, mapping.moe_tp_size) == (4, 1) + assert mapping.enable_attention_dp is True and mapping.dp_size == 4 + assert mapping.tp_size // mapping.dp_size == 1 + # The flag, not attn_tp_size, says attention is replicated: attn_tp_size still reads 4. + assert mapping.attn_tp_size == 4 + assert mapping.enable_lm_head_tp_in_adp is False + # Rank r holds the expert window [32 r, 32 r + 32). + for rank in range(_RANKS): + rank_mapping = _rank_mapping(mapping, rank) + assert (rank_mapping.tp_rank, rank_mapping.moe_ep_rank, rank_mapping.moe_tp_rank) == ( + rank, + rank, + 0, + ) + assert resolved.model_config.pretrained_config.n_routed_experts // mapping.moe_ep_size == 32 + + assert (resolved.max_batch_size, resolved.max_num_tokens, resolved.max_seq_len) == ( + 16, + 32768, + 1048576, + ) + assert (resolved.model_config.max_num_tokens, resolved.model_config.max_seq_len) == ( + 32768, + 1048576, + ) + assert llm_args.enable_chunked_prefill is True + assert llm_args.kv_cache_config.enable_block_reuse is False + assert llm_args.kv_cache_config.dtype == "fp8" + assert llm_args.kv_cache_config.use_kv_cache_manager_v2 is True + assert llm_args.disable_overlap_scheduler is False + # PyExecutor's per-rank cap on active requests under attention DP. + cap = derive_attention_dp_per_rank_request_cap( + base_cap=mapping.pp_size * resolved.max_batch_size, + max_num_tokens=resolved.max_num_tokens, + max_total_draft_tokens=0, + ) + assert cap == 16 and cap * mapping.dp_size == 64 + # No attention-DP balancing or KV-aware routing: the default router places requests (the + # quickstart passes an AttentionDpConfig with balancing off, core.yaml none). + router = ADPRouter.create( + dist=None, has_seq_slot_headroom=False, attention_dp_config=llm_args.attention_dp_config + ) + assert type(router) is DefaultADPRouter + + +def test_max_seq_len_is_not_capped(dep4): + """Nothing lowers max_seq_len below 1048576. The checkpoint's max_position_embeddings is + 1048576 and its rope is YaRN (factor 2048 over 512 original positions), which the engine's + inference does not scale, so the model engine's ``_init_model_capacity``, reading the + modeling_v2 shell's ``infer_max_seq_len``, keeps 1048576 and max_num_tokens 32768. The target + builds its rope tables at ``model_config.max_seq_len`` rows in ``post_load_weights``: 1048576 + rows, before the KV pool is sized.""" + cfg = dep4.model_config.pretrained_config + assert cfg.max_position_embeddings == _MAX_SEQ_LEN + assert (cfg.rope_scaling["rope_type"], cfg.rope_scaling["factor"]) == ("yarn", 2048.0) + assert cfg.rope_scaling["original_max_position_embeddings"] == 512 + + shell = SimpleNamespace(config=cfg) + shell.infer_max_seq_len = MethodType(DecoderModelForCausalLM.infer_max_seq_len, shell) + engine = SimpleNamespace( + model=shell, + max_seq_len=dep4.max_seq_len, + max_num_tokens=dep4.max_num_tokens, + batch_size=dep4.max_batch_size, + ) + for name in ("_init_max_seq_len", "_init_max_num_tokens", "_init_model_capacity"): + setattr(engine, name, MethodType(getattr(ModelEngine, name), engine)) + assert shell.infer_max_seq_len() == _MAX_SEQ_LEN + engine._init_model_capacity() + assert (engine.max_seq_len, engine.max_num_tokens) == (_MAX_SEQ_LEN, _MAX_NUM_TOKENS) + assert dep4.model_config.max_seq_len == _MAX_SEQ_LEN + + +def test_quantization_is_the_anchor_key(dep4_1m): + """At 1M as at 32K and at tep4: MIXED_PRECISION with the fp8 KV cache, the GSM8K anchor row's + key, and the router handed params.json's quantization_config with its 475 quantized_layers + intact plus the FP8 KV-cache algorithm the native loader records from fp8_kv_cache.""" + resolved = dep4_1m + assert resolved.prequantized is True + for quant in (resolved.llm_args.quant_config, resolved.model_config.quant_config): + assert quant.quant_algo == QuantAlgo.MIXED_PRECISION + assert quant.kv_cache_quant_algo == QuantAlgo.FP8 + assert quant.group_size == 16 + assert resolved.llm_args.kv_cache_config.dtype == "fp8" + assert _anchor(resolved.llm_args) == 91.926 + + on_disk = json.loads((resolved.path / "params.json").read_text())["quantization_config"] + assert len(on_disk["quantized_layers"]) == 475 + routed = dict(on_disk, kv_cache_quant_algo="FP8") + assert len(resolved.routed) == 1 + seen = resolved.routed[0] + assert seen.quantization_config == routed + assert resolved.model_config.pretrained_config.quantization_config == routed + assert seen.pretrained_config is resolved.model_config.pretrained_config + assert resolved.model_config.modeling_v2_target is _Routed + assert seen.compute_capability == _SM103 and seen.spec_config is None + assert seen.mapping.enable_attention_dp is True + assert (seen.mapping.world_size, seen.mapping.tp_size, seen.mapping.moe_ep_size) == (4, 4, 4) + + +def test_core_yaml_and_quickstart_build_the_same_engine(dep4, dep4_quickstart): + """The boot gate's quickstart argv and the accuracy gate's core.yaml give the same mapping, + model config, limits and CUDA graphs. They differ in two settings that do not reach the + forward: the KV cache's free_gpu_memory_fraction (quickstart's default 0.9 against 0.85) and + the attention-DP config (quickstart passes one with balancing off, core.yaml none); the + quickstart's mamba cache dtype default is inert on this model.""" + assert dep4.mapping.to_dict() == dep4_quickstart.mapping.to_dict() + core, quick = _summary(dep4), _summary(dep4_quickstart) + assert {key for key in core if core[key] != quick[key]} == {"kv_cache"} + kv_core, kv_quick = core["kv_cache"], quick["kv_cache"] + assert {key for key in kv_core if kv_core[key] != kv_quick[key]} == { + "free_gpu_memory_fraction", + "mamba_ssm_cache_dtype", + } + assert (kv_core["free_gpu_memory_fraction"], kv_quick["free_gpu_memory_fraction"]) == ( + 0.85, + 0.9, + ) + assert dep4.llm_args.attention_dp_config is None + quick_adp = dep4_quickstart.llm_args.attention_dp_config + assert quick_adp.enable_balance is False + assert quick_adp.enable_kv_cache_aware_routing is False + for resolved in (dep4, dep4_quickstart): + kv_cache_config = resolved.llm_args.kv_cache_config + manager = get_kv_cache_manager_cls(resolved.model_config, kv_cache_config) + assert manager is MistralLarge4CacheManager + + +def _manager_arguments(resolved, dtype) -> dict: + """What ``MistralLarge4CacheManager.__init__`` (attention/backends/sparse/mistral_large4/ + cache_manager.py) hands the DSA V2 manager it builds on, given what ``_create_kv_cache_manager`` + (pyexecutor/_util.py) passes it for rank 0 of this engine: the LLM API's kv_cache_config, one + 576-wide MLA head on each of the 72 layers, the latent ``dtype``. The base is not built (it + allocates the pool on the device): its constructor records its arguments and stops.""" + + class _Recorded(Exception): + pass + + seen = {} + + def record(self, kv_cache_config, kv_cache_type, **kwargs): + seen.update(kwargs, kv_cache_config=kv_cache_config, kv_cache_type=kv_cache_type) + raise _Recorded + + model_config = resolved.model_config + with pytest.MonkeyPatch.context() as mp: + mp.setattr(DSACacheManagerV2, "__init__", record) + with pytest.raises(_Recorded): + MistralLarge4CacheManager( + resolved.llm_args.kv_cache_config, + CacheType.SELFKONLY, + num_layers=model_config.pretrained_config.num_hidden_layers, + num_kv_heads=1, + head_dim=576, + tokens_per_block=_TOKENS_PER_BLOCK, + max_seq_len=resolved.max_seq_len, + max_batch_size=resolved.max_batch_size, + mapping=_rank_mapping(resolved.mapping, 0), + dtype=dtype, + spec_config=None, + layer_mask=None, + max_num_tokens=resolved.max_num_tokens, + max_beam_width=1, + sparse_attention_config=model_config.sparse_attention_config, + pretrained_config=model_config.pretrained_config, + ) + return seen + + +def test_kv_cache_dtype_auto_and_fp8_build_the_fp8_ds_mla_pool(dep4, dep4_quickstart, dep4_ci): + """kv_cache_config.dtype fp8 (the gates: core.yaml's and the quickstart's --kv_cache_dtype=fp8) + and auto (the CI class: the checkpoint's fp8_kv_cache) build the same cache. Each resolves the + latent to FP8 (``kv_cache_quant_algo``, so ``_create_kv_cache_manager`` hands the manager + ``DataType.FP8``, ``quant_mode.has_fp8_kv_cache()``) on the same GSM8K anchor row; the sparse + registry picks ``MistralLarge4CacheManager``, whose ``latent_layout`` is fp8_ds_mla, at 47136 + bytes a token on every rank; and the manager builds its base over all 72 layers, a 656-byte + head on the 63 KV owners only, as FP8, with the dtype it was asked for. A latent dtype the + layout cannot hold, bf16, is refused at construction. The CI class's engine is the gates' + in everything else but how the FP8 KV algo is recorded: fp8 writes the requested dtype onto + the global and each of the 475 per-layer quant configs, auto takes both from the checkpoint's + quantization_config, where the native loader records fp8_kv_cache. Every layer reads FP8 + either way, and the latent's dtype is read from the global one.""" + for resolved, requested in ((dep4, "fp8"), (dep4_quickstart, "fp8"), (dep4_ci, "auto")): + model_config, llm_args = resolved.model_config, resolved.llm_args + assert llm_args.kv_cache_config.dtype == requested + for quant in (llm_args.quant_config, model_config.quant_config): + assert quant.kv_cache_quant_algo == QuantAlgo.FP8 + assert model_config.quant_config.quant_mode.has_fp8_kv_cache() + assert _anchor(llm_args) == 91.926 + manager = get_kv_cache_manager_cls(model_config, llm_args.kv_cache_config) + assert manager is MistralLarge4CacheManager and manager.latent_layout == "fp8_ds_mla" + assert issubclass(manager, KVCacheManagerV2) + for rank in range(_RANKS): + cost = _kv_cost(resolved, rank) + assert (cost.slope, cost.intercept) == (47136, 0) + + sources = model_config.pretrained_config.kv_source_layers + base = _manager_arguments(resolved, DataType.FP8) + assert (base["num_layers"], base["head_dim"]) == (72, 656) + assert base["num_kv_heads"] == [1 if s == i else None for i, s in enumerate(sources)] + assert base["num_kv_heads"].count(1) == 63 + assert base["dtype"] == DataType.FP8 + assert base["layer_mask"] is None + assert base["kv_cache_config"].dtype == requested + assert (base["tokens_per_block"], base["max_seq_len"]) == (_TOKENS_PER_BLOCK, _MAX_SEQ_LEN) + with pytest.raises(ValueError, match="fp8_ds_mla"): + _manager_arguments(resolved, DataType.BF16) + + gate, ci = _summary(dep4), _summary(dep4_ci) + assert {key for key in gate if gate[key] != ci[key]} == {"kv_cache", "quant", "layers"} + for resolved in (dep4, dep4_ci): + layers = resolved.model_config.quant_config_dict + assert len(layers) == 475 + assert all(quant.kv_cache_quant_algo == QuantAlgo.FP8 for quant in layers.values()) + assert {key for key in gate["kv_cache"] if gate["kv_cache"][key] != ci["kv_cache"][key]} == { + "dtype" + } + assert dep4.mapping.to_dict() == dep4_ci.mapping.to_dict() + + +# ------------------------------------------------------- (2) admitting a 1M prompt + + +def _check_arguments(llm_args, prompt_len: int) -> None: + """``BaseLLM._check_arguments`` on an LLM holding ``llm_args``, for a plain request.""" + llm = SimpleNamespace(args=llm_args) + llm._check_one_model_speculative_sampling = MethodType( + BaseLLM._check_one_model_speculative_sampling, llm + ) + BaseLLM._check_arguments(llm, prompt_len, SamplingParams(max_tokens=8), is_gen_only=False) + + +def _deduced_max_tokens(resolved, prompt_len: int, max_tokens: int | None) -> int: + """The ``max_tokens`` the worker's ``_enqueue_request`` gives the executor request, for a + ``prompt_len``-token prompt asking ``max_tokens`` (None: as many as fit).""" + enqueued = [] + worker = SimpleNamespace( + _lora_manager=None, + mapping=_rank_mapping(resolved.mapping, 0), + max_seq_len=resolved.max_seq_len, + engine=SimpleNamespace(enqueue_request=lambda request, **_: enqueued.append(request) or 1), + ) + request = GenerationRequest( + [7] * prompt_len, sampling_params=SamplingParams(max_tokens=max_tokens, end_id=2, pad_id=2) + ) + request.id = 1 + BaseWorker._enqueue_request(worker, request) + (executor_request,) = enqueued + return executor_request.max_tokens + + +def test_a_1m_prompt_passes_the_front_end(dep4, dep4_32k): + """``_check_arguments`` refuses a prompt longer than max_num_tokens only without chunked + prefill: at the 1M config a 1,040,000-token prompt passes (no length is refused there), + while the delivered unchunked engine refuses one token past 32768.""" + for prompt_len in (32769, 1_040_000, _MAX_SEQ_LEN - 1, _MAX_SEQ_LEN): + _check_arguments(dep4.llm_args, prompt_len) + + _check_arguments(dep4_32k.llm_args, _MAX_NUM_TOKENS) + with pytest.raises(RequestError, match=r"should not exceed max_num_tokens \(32768\)"): + _check_arguments(dep4_32k.llm_args, _MAX_NUM_TOKENS + 1) + + +def test_the_worker_refuses_a_prompt_of_max_seq_len_tokens(dep4): + """The length the engine refuses is max_seq_len against the prompt alone: the worker deduces + ``max_tokens`` as ``max_seq_len - prompt_len`` and refuses the request only when that is not + positive, so a 1048576-token prompt is refused with the context-length message and a + 1048575-token prompt generates one token. A prompt plus max_tokens past max_seq_len is not + refused: max_tokens is cut to what fits (a 1,040,000-token prompt gets at most 8576).""" + assert _deduced_max_tokens(dep4, 1_040_000, None) == 8576 + assert _deduced_max_tokens(dep4, 1_040_000, 64) == 64 + assert _deduced_max_tokens(dep4, 1_040_000, 10_000) == 8576 + assert _deduced_max_tokens(dep4, _MAX_SEQ_LEN - 1, None) == 1 + with pytest.raises(RequestError) as refused: + _deduced_max_tokens(dep4, _MAX_SEQ_LEN, 1) + assert is_context_length_exceeded_message(str(refused.value)) + assert "maximum context length is 1048576 tokens" in str(refused.value) + assert "resulted in 1048576 tokens" in str(refused.value) + + +# ------------------------------------------------------- (3) chunked prefill under MLA and DSA + + +def test_chunked_prefill_turns_on_context_mla_with_cached_kv(dep4, dep4_32k): + """``build_attention_metadata`` turns ``enable_context_mla_with_cached_kv`` on for MLA when + the runtime features carry chunked prefill (or KV reuse): on at the 1M config, off at the + delivered 32K one. The DSA metadata is sized per rank (16 sequences, 32768 tokens) and its + params keep the indexer's own chunk size 32768, the TP query-split threshold 8192, top-k + 2048, the short-sequence indexer skip and the shared-indexer layers.""" + for resolved, cached_kv in ((dep4, True), (dep4_32k, False)): + features = _runtime_features(resolved) + assert (features.chunked_prefill, features.cache_reuse) == (cached_kv, False) + assert features.chunk_size == _MAX_NUM_TOKENS + kwargs = _attention_metadata_kwargs(resolved) + assert kwargs.enable_context_mla_with_cached_kv is cached_kv + assert (kwargs.max_num_requests, kwargs.max_num_sequences) == (16, 16) + assert kwargs.max_num_tokens == _MAX_NUM_TOKENS + params = kwargs.sparse_metadata_params + assert (params.indexer_max_chunk_size, params.q_split_threshold) == (32768, 8192) + assert (params.max_sparse_topk, params.index_head_dim) == (2048, 128) + assert params.enable_indexer_skip is True and params.has_shared_indexer_layers is True + + sparse_params = dep4.model_config.sparse_attention_config.to_sparse_params( + pretrained_config=dep4.model_config.pretrained_config + ) + backend = get_attention_backend(dep4.llm_args.attn_backend, sparse_params=sparse_params) + assert backend.Metadata is DSAtrtllmAttentionMetadata + + +def test_attention_dp_leaves_the_indexer_queries_unsplit(dep4, tep4): + """``_split_prefill_queries`` splits a step's context queries across TP ranks from 8192 of + them on, but never under attention DP: every dep4 rank computes all 32768 of its own, while a + tep4 rank takes a quarter.""" + for rank in range(_RANKS): + assert _split_prefill_queries(_MAX_NUM_TOKENS, _rank_mapping(dep4.mapping, rank), 8192) == ( + 0, + _MAX_NUM_TOKENS, + None, + ) + assert _split_prefill_queries(_MAX_NUM_TOKENS, _rank_mapping(tep4.mapping, rank), 8192) == ( + 8192 * rank, + 8192 * (rank + 1), + [8192] * _RANKS, + ) + + +def _indexer_prefill_step(resolved, contexts, decode_kv_lens=(), rank: int = 0): + """``Indexer.prepare_for_chunked_prefill`` on one dep4 rank's step: ``contexts`` lists each + context request's (cached prefix, new tokens) and ``decode_kv_lens`` each generation row's KV + length. The metadata holds what that method and its gather mappings read; its block table + gives request i the pages i * width + j. Pinned host memory and the device copy of the gather + mappings are skipped: there is no device here.""" + width = _v2_max_blocks_per_seq(resolved.max_seq_len) + num_contexts, num_generations = len(contexts), len(decode_kv_lens) + seq_lens = [new for _, new in contexts] + [1] * num_generations + cached = [prefix for prefix, _ in contexts] + [kv_len - 1 for kv_len in decode_kv_lens] + cached_indptr = [0, *itertools.accumulate(prefix for prefix, _ in contexts)] + kv_indptr = [0, *itertools.accumulate(prefix + new for prefix, new in contexts)] + num_seqs = num_contexts + num_generations + md = SimpleNamespace( + mapping=_rank_mapping(resolved.mapping, rank), + enable_context_mla_with_cached_kv=True, + runtime_features=_runtime_features(resolved), + sparse_metadata_params=_attention_metadata_kwargs(resolved, rank).sparse_metadata_params, + compress_ratios=[1], + max_num_tokens=resolved.max_num_tokens, + kv_cache_manager=SimpleNamespace( + max_seq_len=resolved.max_seq_len, index_head_dim=128, tokens_per_block=32 + ), + indexer_head_dim=128, + indexer_quant_block_size=128, + _tokens_per_block=_TOKENS_PER_BLOCK, + num_contexts=num_contexts, + num_generations=num_generations, + num_ctx_tokens=sum(new for _, new in contexts), + request_ids=list(range(num_seqs)), + kv_cache_params=SimpleNamespace(num_cached_tokens_per_seq=cached), + seq_lens=torch.tensor(seq_lens, dtype=torch.int32), + host_ctx_cached_token_indptr=torch.tensor(cached_indptr, dtype=torch.int64), + host_ctx_kv_indptr=torch.tensor(kv_indptr, dtype=torch.int64), + cu_seqlen_ks=torch.empty(resolved.max_num_tokens, dtype=torch.int32), + skip_indexer_for_ctx_reqs=False, + slot_mapping_fp8=torch.empty(resolved.max_num_tokens, dtype=torch.int64), + slot_mapping_scale=torch.empty(resolved.max_num_tokens, dtype=torch.int64), + host_indexer_k_cache_block_offsets=torch.arange( + resolved.max_batch_size * width, dtype=torch.int32 + ).reshape(resolved.max_batch_size, width), + indexer_prefill_q_split="unset", + indexer_prefill_chunks=None, + ) + with pytest.MonkeyPatch.context() as mp: + mp.setattr(dsa_indexer, "prefer_pinned", lambda: False) + mp.setattr(dsa_indexer, "maybe_pin_memory", lambda tensor: tensor) + mp.setattr(torch.Tensor, "cuda", lambda tensor, *args, **kwargs: tensor) + Indexer.prepare_for_chunked_prefill(md, Indexer.build_indexer_params(md)) + return md + + +def _chunk_ranges(md) -> list[tuple[int, int, int, int]]: + return [ + (chunk.token_start, chunk.token_end, chunk.k_token_start, chunk.k_token_end) + for chunk in md.indexer_prefill_chunks + ] + + +def _indexer_slot(md, request: int, position: int) -> tuple[int, int]: + """The indexer key pool's byte offsets of ``position`` in ``request``: 128 fp8 codes then one + fp32 scale per token, pages of 32 tokens, page stride 32 * 132 bytes.""" + page = int(md.host_indexer_k_cache_block_offsets[request, position // _TOKENS_PER_BLOCK]) + base = page * _TOKENS_PER_BLOCK * (128 + 4) + offset = position % _TOKENS_PER_BLOCK + return base + offset * 128, base + _TOKENS_PER_BLOCK * 128 + offset * 4 + + +def test_indexer_prefill_chunks_over_a_1m_prefix(dep4): + """What ``Indexer.prepare_for_chunked_prefill`` plans on a dep4 rank under chunked prefill, + as (token_start, token_end, k_token_start, k_token_end) per chunk. The planner is + ``plan_mla_chunked_prefill_chunks`` over each request's whole KV (cached + new), packing + requests while their (new tokens) x (KV) stays within ``_packed_logits_elem_budget``: 1 << 28 + here, the packed cap, below the 1 << 31 single-request tile and 32768 x 1048576. The query + split stays None on every step. The target re-tiles each chunk's queries at the logits budget + it takes from this engine (``_prefill_logits_budget(32768, 1048576)``, the ceiling 2**26, + ``test_target_sizes_from_the_engines_metadata``) against the chunk's n_k = k_token_end - + k_token_start gathered keys, or, for a chunk past max_seq_len keys (d), against each packed + request's own (``_prefill_blocks``). + + (a) a 1M request's last chunk, 32768 new tokens over 1048576 - 32768 cached: one chunk, + n_k 1048576; its rows attend keys [0, 1015809) to [0, 1048576). + (b) its first chunk, 32768 over no prefix: one chunk, n_k 32768. + (c) a mixed step, a 16384-token chunk over a 500000-token prefix with 15 decode rows behind + it: one chunk of the context rows, n_k 516384; the decode rows are not planned here. + (d) two requests' 5-token last chunks over 983040-token prefixes on one rank: packed into one + chunk (10 x 1966090 is under the budget), so that indexer call sees n_k 1966090, more than + max_seq_len; a packed chunk's n_k is bounded only by the budget over its new tokens and by + the rank's KV pool. + + The key gather mappings (``slot_mapping_fp8_fullkv`` / ``slot_mapping_scale_fullkv``) are + int64 over every context key of the step, cached and new, in request order: 1048576 long in + (a). With no cached prefix (b) they are the step's own slot mappings.""" + budget_md = SimpleNamespace( + max_num_tokens=dep4.max_num_tokens, + kv_cache_manager=SimpleNamespace(max_seq_len=dep4.max_seq_len), + ) + assert dsa_indexer._INDEXER_PACKED_LOGITS_ELEM_BUDGET == 1 << 28 + assert dsa_indexer._INDEXER_MQA_LOGITS_ELEM_BUDGET == 1 << 31 + assert _packed_logits_elem_budget(budget_md) == 1 << 28 + + prefix = _MAX_SEQ_LEN - _MAX_NUM_TOKENS + last = _indexer_prefill_step(dep4, [(prefix, _MAX_NUM_TOKENS)]) + assert last.indexer_prefill_q_split is None + assert _chunk_ranges(last) == [(0, _MAX_NUM_TOKENS, 0, _MAX_SEQ_LEN)] + (chunk,) = last.indexer_prefill_chunks + assert torch.equal(chunk.cu_seqlen_ks, torch.zeros(_MAX_NUM_TOKENS, dtype=torch.int32)) + assert torch.equal( + chunk.cu_seqlen_ke, torch.arange(prefix + 1, _MAX_SEQ_LEN + 1, dtype=torch.int32) + ) + for mapping in (last.slot_mapping_fp8_fullkv, last.slot_mapping_scale_fullkv): + assert mapping.dtype == torch.int64 and tuple(mapping.shape) == (_MAX_SEQ_LEN,) + for position in (0, 31, 32, prefix, _MAX_SEQ_LEN - 1): + assert ( + int(last.slot_mapping_fp8_fullkv[position]), + int(last.slot_mapping_scale_fullkv[position]), + ) == _indexer_slot(last, 0, position) + + first = _indexer_prefill_step(dep4, [(0, _MAX_NUM_TOKENS)]) + assert first.indexer_prefill_q_split is None + assert _chunk_ranges(first) == [(0, _MAX_NUM_TOKENS, 0, _MAX_NUM_TOKENS)] + assert first.slot_mapping_fp8_fullkv is first.slot_mapping_fp8 + assert first.slot_mapping_scale_fullkv is first.slot_mapping_scale + + mixed = _indexer_prefill_step(dep4, [(500_000, 16384)], decode_kv_lens=[100_000] * 15) + assert mixed.indexer_prefill_q_split is None + assert _chunk_ranges(mixed) == [(0, 16384, 0, 516_384)] + assert tuple(mixed.slot_mapping_fp8_fullkv.shape) == (516_384,) + assert 16384 + 15 <= dep4.max_num_tokens + + packed = _indexer_prefill_step(dep4, [(983_040, 5), (983_040, 5)]) + assert packed.indexer_prefill_q_split is None + assert _chunk_ranges(packed) == [(0, 10, 0, 1_966_090)] + assert 10 * 1_966_090 <= 1 << 28 + assert tuple(packed.slot_mapping_fp8_fullkv.shape) == (1_966_090,) + assert int(packed.slot_mapping_fp8_fullkv[983_045]) == _indexer_slot(packed, 1, 0)[0] + # The delivered per-rank pool (measured at 47280 bytes a token, the language model only) held + # two such requests; the 1M boot's, with the vision tower on every rank, holds one. + two = 2 * -(-(983_045 + 1) // _TOKENS_PER_BLOCK) * _TOKENS_PER_BLOCK + assert _BOOT_POOL_TOKENS < two <= _DELIVERED_POOL_TOKENS + + +# ------------------------------------------------------- (4) the KV pool and its tables + + +def test_dsa_kv_tables_at_1m(dep4, dep4_32k): + """The engine builds ``MistralLarge4CacheManager``, a V2 manager (``use_kv_cache_manager_v2`` + defaults on). Its block-table width is ceil((max_seq_len + 1) / 32), the one-token + generation headroom included, padded to a multiple of 4: 32772 at 1048576 (1028 at 32768). + The DSA metadata sizes ``block_table`` and ``indexer_k_cache_block_offsets`` [16, width] and + ``kv_cache_block_offsets`` [pools, 16, 2, width] from it. The radix top-k's aux buffers are + [16, 10, 2048] (one row per sequence without speculation) and ``topk_indices_buffer``, kept + for the short-sequence indexer skip, is [32768, 2048]: neither grows with max_seq_len.""" + for resolved in (dep4, dep4_32k): + kv_cache_config = resolved.llm_args.kv_cache_config + manager = get_kv_cache_manager_cls(resolved.model_config, kv_cache_config) + assert manager is MistralLarge4CacheManager and issubclass(manager, KVCacheManagerV2) + assert kv_cache_config.tokens_per_block == _TOKENS_PER_BLOCK + assert _get_generation_kv_capacity(None, is_draft=False) == (0, BASE_GENERATION_TOKEN_COUNT) + assert BASE_GENERATION_TOKEN_COUNT == 1 + assert _v2_max_blocks_per_seq(dep4.max_seq_len) == 32772 + assert _v2_max_blocks_per_seq(dep4_32k.max_seq_len) == 1028 + + kwargs = _attention_metadata_kwargs(dep4) + params = kwargs.sparse_metadata_params + # One radix row per sequence: `_radix_rows_per_sequence` is 1 without speculation. + radix = (kwargs.max_num_sequences * 1, _MAX_RADIX_BLOCKS_PER_ROW, params.max_sparse_topk) + assert radix == (16, 10, 2048) + assert params.enable_indexer_skip is True + assert (kwargs.max_num_tokens, params.max_sparse_topk) == (32768, 2048) + + +def test_kv_pool_has_no_rank_factor(dep4, dep4_32k, tep4): + """Each rank holds the latent and indexer-key caches of its own requests only, and the cost of + a token is the same on every rank, at 1M as at 32K and at tep4: an fp8_ds_mla latent of + 656 bytes (512 e4m3, four fp32 tile scales, 64 bf16 rope) on each of the 63 KV owners plus a + 128-byte fp8 indexer key and its fp32 scale on each of the 44 full-indexer layers, in pages + of 32, with no per-request intercept. One 1,048,576-token request is 49,425,678,336 B + (46.03 GiB) of a rank's pool, against 49,576,673,280 B (46.17 GiB) at the 576-byte latent + of all 72 layers. The bytes of the delivered 32K boot's pool (2,004,896 tokens at 47280 + bytes) hold one such request at either token, not two, as does the 1M boot's pool on the + native checkpoint (1,879,904 tokens). The configured batch is another + figure at this config: 16 requests of max_seq_len would be 736.5 GiB of a rank's pool, + which no rank holds, so the pool and not max_batch_size bounds how many 1M requests a rank + holds at once; at 32K the same 16 requests of max_seq_len are 23.016 GiB.""" + model_config, kv_cache_config = dep4.model_config, dep4.llm_args.kv_cache_config + manager = get_kv_cache_manager_cls(model_config, kv_cache_config) + assert manager is MistralLarge4CacheManager + assert manager is get_kv_cache_manager_cls(tep4.model_config, tep4.llm_args.kv_cache_config) + assert kv_cache_config.tokens_per_block == _TOKENS_PER_BLOCK + + cfg, sparse = model_config.pretrained_config, model_config.sparse_attention_config + latent = 656 # fp8_ds_mla: 512 e4m3 + 4 fp32 tile scales + 64 bf16 rope + owners = sum(1 for i, s in enumerate(cfg.kv_source_layers) if s == i) + indexer_key = sparse.index_head_dim + sparse.index_head_dim // 128 * 4 + full_indexer_layers = cfg.index_topk_pattern.count("F") + per_token = latent * owners + indexer_key * full_indexer_layers + assert (latent, owners, indexer_key, full_indexer_layers) == (656, 63, 132, 44) + assert per_token == 47136 + + for rank in range(_RANKS): + cost = _kv_cost(dep4, rank) + assert (cost.slope, cost.intercept) == (per_token, 0) + assert cost == _kv_cost(dep4_32k, rank) == _kv_cost(tep4, rank) + + request = _kv_cost(dep4, 0).bytes_for_tokens(_MAX_SEQ_LEN) + assert request == _MAX_SEQ_LEN * per_token == 49_425_678_336 + assert math.isclose(request / 2**30, 46.03, abs_tol=5e-3) + assert _MAX_SEQ_LEN * _DELIVERED_BYTES_PER_TOKEN == 49_576_673_280 + # The measured pool, read through its bytes: one 1M request at either token, not two. + delivered = _DELIVERED_POOL_TOKENS * _DELIVERED_BYTES_PER_TOKEN + assert delivered // request == delivered // (_MAX_SEQ_LEN * _DELIVERED_BYTES_PER_TOKEN) == 1 + assert _BOOT_POOL_TOKENS * per_token // request == 1 + + # The per-rank cost of the configured batch, 16 requests of max_seq_len: at 1M past any + # rank's pool, at 32K the 23.016 GiB the 32K manager test pins. + batch = _kv_cost(dep4, 0).bytes_for_tokens(dep4.max_batch_size * dep4.max_seq_len) + assert batch == 16 * request and math.isclose(batch / 2**30, 736.5, abs_tol=5e-2) + assert batch > delivered + tokens_32k = dep4_32k.max_batch_size * dep4_32k.max_seq_len + batch_32k = _kv_cost(dep4_32k, 0).bytes_for_tokens(tokens_32k) + assert batch_32k == tokens_32k * per_token + assert math.isclose(batch_32k / 2**30, 23.016, abs_tol=5e-4) + + +# ------------------------------------------------------- (5) boot, before the pool is sized + + +def test_kv_estimation_profiles_one_fresh_chunk_per_rank(dep4): + """Before the real pool exists the engine runs a profiling pass. Its dummies are as long as a + request may be (net max_seq_len - 1 = 1048575), cut to max_num_tokens: one 32768-token + prompt, one copy per rank under attention DP, and the default router hands every rank one. + MLA chunk-aware profiling does not apply to DSA, so each rank's profiled context step is that + one fresh 32768-token prompt in a single chunk: no cached prefix and no KV length above + 32768. The estimation's own V2 quota is sized for the CUDA-graph capture's longest row + instead: ceil((1048576 + 1) / 32) + 16 - 1 = 32784 blocks, 1,049,088 tokens per rank.""" + creator = _estimation_creator(dep4) + assert creator._is_kv_cache_manager_v2 is True + assert ( + _get_num_pool_groups_for_estimation( + dep4.model_config, _MAX_SEQ_LEN, creator._kv_cache_config + ) + == 1 + ) + assert creator._get_token_num_for_estimation() == _ESTIMATION_TOKENS == 1_049_088 + assert creator._mla_chunked_profile_length is None + lengths = [len(request.input_token_ids) for request in creator._dummy_reqs] + assert lengths == [_MAX_NUM_TOKENS] * _RANKS + + router = ADPRouter.create( + dist=None, + has_seq_slot_headroom=False, + attention_dp_config=dep4.llm_args.attention_dp_config, + ) + assigned, expected = router.route_requests( + [RankState(rank=rank) for rank in range(_RANKS)], + [ + SimpleNamespace(request=SimpleNamespace(num_input_tokens=n, py_scheduling_params=None)) + for n in lengths + ], + max_num_active_requests=dep4.max_batch_size, + ) + assert expected == 1 + for rank in range(_RANKS): + assert [item.request.num_input_tokens for item in assigned[rank]] == [_MAX_NUM_TOKENS] + + +def test_no_context_mla_workspace_is_reserved(dep4, monkeypatch): + """At sm_103 the fp8-KV context MLA of a DSA model takes the sparse path, which stages no + K/V: the attention backend declares 0 bytes per attended token and no capacity-sized + workspace, so the KV-capacity estimation subtracts no context-MLA reserve from the pool and + the scheduler gets no attended-KV admission cap, chunked prefill or not. (The DeepSeek-V4 + logits reserve is for that algorithm only; this one is "dsa".) The target's own fp8_ds_mla + context workspace (``PrefillTarget._ctx_workspace``) is not the backend's and is not reserved + here either.""" + monkeypatch.setattr(trtllm_backend, "get_sm_version", lambda: 103) + monkeypatch.delenv("TRTLLM_MLA_SHORT_SEQ_MHA_THRESHOLD", raising=False) + model_config = dep4.model_config + assert model_config.sparse_attention_config.algorithm == "dsa" + assert model_config.attn_backend == "TRTLLM" + quant_mode = model_config.quant_config.quant_mode + assert quant_mode.has_fp8_kv_cache() and not quant_mode.has_fp4_kv_cache() + mapping = _rank_mapping(dep4.mapping, 0) + bytes_per_token = get_attention_workspace_bytes_per_token(model_config, mapping) + assert bytes_per_token == 0 + assert get_attention_workspace_is_chunked_prefill_bounded(model_config) is True + assert ( + get_attention_workspace_fixed_bytes( + model_config, + mapping, + max_batch_size=dep4.max_batch_size, + max_num_tokens=dep4.max_num_tokens, + max_seq_len=dep4.max_seq_len, + enable_cuda_graph=True, + ) + == 0 + ) + assert ( + get_mla_context_workspace_kv_len_cap( + dep4.llm_args.kv_cache_config, + dep4.max_batch_size, + dep4.max_num_tokens, + dep4.max_seq_len, + dep4.llm_args.enable_chunked_prefill, + workspace_is_chunked_prefill_bounded=True, + # DSA takes no chunk-aware profiling; the requirement holds only for w > 0. + chunked_workspace_profiled=False, + require_chunked_workspace_profile=bytes_per_token > 0, + ) + is None + ) + + +def test_target_sizes_from_the_engines_metadata(dep4, dep4_32k, monkeypatch): + """What the dep target fixes on the first step (``_derive_first_step``) from the metadata + this engine builds -- ``build_attention_metadata``'s max_num_requests, max_num_tokens and + runtime features, which ``_create_py_executor`` builds from the LLM arguments (chunked + prefill from enable_chunked_prefill, cache reuse from the KvCacheConfig's + enable_block_reuse, off at both configs), and the KV cache manager's max_seq_len. At the 1M + config, chunked prefill lets a step attend min(16, 32768) x 1048576 = 2**24 rows of context + KV, so: the held context workspace at its ceiling, 2**19 rows (bf16 [2**19, 576], 576 MiB, + allocated on the first context step -- the KV-capacity estimation's profiling forward -- and + held, so the pool is sized with it); the indexer's logits budget at its ceiling, 2**26 (64 + query rows of 1048576 keys); and, under chunked prefill, expert calls of 32768 of the up to + 131072 gathered rows, the autotuner's bucket 32768. At the delivered 32K configuration + (neither chunked prefill nor block reuse) a step attends at most its 32768 new rows: 65536 + rows, 2**24 and one call of 131072. Built on the meta device, with stand-ins for its kernel + builds.""" + monkeypatch.delenv("TLLM_INDEXER_MQA_LOGITS_ELEM_BUDGET", raising=False) + monkeypatch.setattr( + M, "_rope_tables", lambda cfg, alpha, rows, dtype, device: SimpleNamespace(rows=rows) + ) + monkeypatch.setattr(M, "build_ix_q_side", lambda dev, is_neox: None) + monkeypatch.setattr(M, "build_ix_k_cache", lambda dev, is_neox, cache_contiguous: None) + monkeypatch.setattr( + M, "build_ix_topk_decode", lambda dev, next_n: SimpleNamespace(next_n=next_n) + ) + monkeypatch.setattr(torch.cuda, "current_device", lambda: 0) + sources = list(dep4.model_config.pretrained_config.kv_source_layers) + owners = [i for i, s in enumerate(sources) if s == i] + assert (1 << 19) * 576 * 2 == 576 * 2**20 + for resolved, most, sizes in ( + (dep4, 1 << 24, (1 << 19, 1 << 26, 32768, 32768)), + (dep4_32k, 32768, (65536, 1 << 24, 131072, 131072)), + ): + llm_args = resolved.llm_args + kwargs = _attention_metadata_kwargs(resolved) + features = kwargs.runtime_features + assert features == _runtime_features(resolved) + assert features.chunked_prefill is llm_args.enable_chunked_prefill is (resolved is dep4) + assert features.cache_reuse is llm_args.kv_cache_config.enable_block_reuse is False + md = SimpleNamespace( + max_num_requests=kwargs.max_num_requests, + max_num_tokens=kwargs.max_num_tokens, + runtime_features=features, + max_seq_len=resolved.max_seq_len, + tokens_per_block=_TOKENS_PER_BLOCK, + kv_cache_manager=SimpleNamespace( + latent_layout=MistralLarge4CacheManager.latent_layout, + kv_source_layers=sources, + latent_pool_row=lambda i: owners.index(sources[i]), + kv_owner_layers=owners, + ), + ) + assert M._most_ctx_kv(md) == most + with torch.device("meta"): + core = M.MistralLarge4DepModelingV2(resolved.model_config) + core.build_kernels(0) + core._derive_first_step(md) + got = ( + core.ctx_workspace_rows, + core.prefill_logits_budget, + core.expert_call_tokens, + core.expert_tune_tokens, + ) + assert got == sizes, (resolved.max_seq_len, got) + + +def test_warmup_runs_one_chunk_per_rank_and_captures_a_1m_decode_row(dep4): + """The warmup's shapes under attention DP, on the estimation's 1,049,088-token pool (the + warmup and the capture run in the profiling executor first, then again in the final one). + Each rank proposes its shapes from its own free KV, the ranks keep the ones all proposed, and + every rank enters each shape's forward. The max context shape is max_num_tokens of context + on every rank at once, now one fresh 32768-token prompt (a prompt may be up to + max_seq_len - 1 long), so the routed experts see 131072 gathered rows. Each decode graph is + captured on a batch whose first row is max_seq_len - 1 = 1048575 tokens long (long family; + its step attends 1048576 keys) or threshold - 2 = 2046 (short family), the other rows one + token. So the 1M decode row runs in the profiling executor's capture, before the pool is + sized, and the decode scorer's [16, 1048576] fp32 arena -- sized by max_seq_len, whatever + the KV lengths -- is reserved there.""" + config = _runner_config(dep4) + sparse = copy.deepcopy(dep4.model_config.sparse_attention_config) + assert sparse.needs_separate_short_long_cuda_graphs() is True + model = SimpleNamespace(model_config=dep4.model_config) + group = _TpGroup() + + def rank_warmup(rank): + resources = _Resources(_KvPool(_ESTIMATION_TOKENS, config.max_seq_len)) + warmup = DecoderWarmup.__new__(DecoderWarmup) + warmup._config = config + agreement = WarmupAgreement(group.dist(rank), _rank_mapping(dep4.mapping, rank)) + shapes = agreement.agree_shapes(warmup._get_full_general_warmup_requests(resources)) + factory = DummyRequestFactory( + config=config, + model=model, + get_runtime_tokens_per_gen_step=lambda draft_len: 1 + draft_len, + release_request_state=None, + ) + max_context = max(tokens for tokens, gen in shapes if gen == 0) + batch = factory.create_warmup_request(resources, max_context, 0) + enters = agreement.should_run_warmup_batch(batch, max_context, "max context") + prompts = [request.token_num for request in batch.context_requests] + captures = {} + for family, max_seq_len in ( + ("long", config.max_seq_len), + ("short", sparse.seq_len_threshold - 1), + ): + capture = factory.create_cuda_graph_warmup_request( + resources, config.cuda_graph_batch_sizes[-1], 0, max_seq_len + ) + captures[family] = [request.token_num for request in capture.generation_requests] + return shapes, enters, prompts, captures + + results = group.run(rank_warmup) + for shapes, enters, prompts, captures in results: + assert shapes == [(1, 0), (1, 1), (32768, 0), (16, 16), (2, 0)] + assert enters is True + assert prompts == [32768] + assert captures == {"long": [1048575] + [1] * 15, "short": [2046] + [1] * 15} + assert sum(sum(prompts) for _, _, prompts, _ in results) == 131072 + # The capture's batch fits the estimation quota: the longest row's pages and one per other row. + assert -(-(1048575 + 1) // _TOKENS_PER_BLOCK) + 15 <= _ESTIMATION_BLOCKS + + +# ------------------------------------------------------- what the forward receives + + +def test_every_rank_is_handed_every_rank_token_count(dep4): + """``attn_metadata.all_rank_num_tokens`` is set on every step of every rank, by the gather in + the decoder's input preparation (eager, warmup and capture steps alike): each rank's own + token count, in rank order, identical on every rank and NOT padded (no prefill CUDA graph + backend is configured). An idle rank counts the one token of its generation dummy; a mixed + step counts its chunk and its decode rows. Padding to the largest count before the experts' + collectives is the forward's own job.""" + llm_args, model_config = dep4.llm_args, dep4.model_config + sparse_params = model_config.sparse_attention_config.to_sparse_params( + pretrained_config=model_config.pretrained_config + ) + metadata_cls = get_attention_backend( + llm_args.attn_backend, sparse_params=sparse_params + ).Metadata + assert metadata_cls is DSAtrtllmAttentionMetadata + # DSA exchanges the one plain count; it has no per-stage counts of its own. + assert metadata_cls.get_adp_token_counts is AttentionMetadata.get_adp_token_counts + assert metadata_cls.set_adp_token_counts is AttentionMetadata.set_adp_token_counts + assert llm_args.prefill_cuda_graph_backend == PrefillCudaGraphBackend.DISABLED + + group = _TpGroup() + steps = { + # every rank runs a whole 32768-token chunk, as the warmup's max shape does + "four chunks": ([32768] * 4, [1] * 4), + # a chunk with 15 decode rows behind it, an idle rank's dummy, a chunk, sixteen decode rows + "unequal": ([16384 + 15, 1, 32768, 16], [1, 0, 1, 0]), + } + for counts, contexts in steps.values(): + + def rank_step(rank, counts=counts, contexts=contexts): + md = SimpleNamespace( + num_tokens=counts[rank], padded_num_tokens=None, all_rank_num_tokens=None + ) + md.get_adp_token_counts = MethodType(metadata_cls.get_adp_token_counts, md) + md.set_adp_token_counts = MethodType(metadata_cls.set_adp_token_counts, md) + dist = group.dist(rank) + gathered = get_all_rank_num_tokens( + md, + enable_attention_dp=True, + mapping=_rank_mapping(dep4.mapping, rank), + dist=dist, + ) + num_tokens = pad_input_tokens( + md, + counts[rank], + contexts[rank], + gathered, + input_ids=torch.zeros(counts[rank], dtype=torch.int32), + position_ids=torch.zeros(counts[rank], dtype=torch.int32), + dist=dist, + enable_attention_dp=True, + prefill_cuda_graph_backend=llm_args.prefill_cuda_graph_backend, + prefill_cuda_graph_num_tokens=[], + ) + return num_tokens, md.all_rank_num_tokens, md.padded_num_tokens + + for rank, (num_tokens, all_rank, padded) in enumerate(group.run(rank_step)): + assert num_tokens == counts[rank] and padded is None + assert all_rank == counts + + # Four 32768-token chunks in one step: the experts see 131072 gathered rows, 524288 after + # top-4, as at 32K. + rows = sum(steps["four chunks"][0]) + assert rows == _RANKS * dep4.max_num_tokens == 131072 + assert rows * model_config.pretrained_config.num_experts_per_tok == 524288 + + +def test_an_idle_rank_steps_with_a_one_token_generation_dummy(dep4): + """When some rank has work and this one has none, the executor gives it one dummy before + scheduling: request id ATTENTION_DP_DUMMY_REQUEST_ID, a generation request with the KV + manager's default of one token (no draft tokens), its KV prepared like a real request's. + Aggregated serving with the overlap scheduler takes this plain path: no KV-cache transceiver, + and neither the scheduler-aware nor the forward-intent variant. The rank's step is then + generation-only with one token, so it runs the decode target while its peers may prefill + (one 1M prompt runs its whole prefill on one rank, the others stepping beside it). When no + rank has work, no rank gets a dummy and none steps.""" + llm_args, mapping = dep4.llm_args, dep4.mapping + assert llm_args.disable_overlap_scheduler is False + assert should_enable_adp_dummy_fixes(mapping) is True + model_type = dep4.model_config.pretrained_config.model_type + assert not should_enable_scheduler_aware_adp_dummy( + model_type, mapping, llm_args.disable_overlap_scheduler + ) + assert not should_enable_non_overlap_adp_forward_intent( + mapping, llm_args.disable_overlap_scheduler + ) + + def executor(expected_num_active_requests, active_requests): + """The executor state `_pad_attention_dp_dummy_request` reads, with its own helpers.""" + executor = SimpleNamespace( + enable_attention_dp=mapping.enable_attention_dp, + # The attention-DP group: a single-rank group skips dummy padding. + dist=SimpleNamespace(tp_size=mapping.tp_size), + _enable_remote_tail_adp=False, + # No speculative decoding: no draft KV cache joins the target's reuse. + enable_joint_kv_cache_reuse=False, + _enable_adp_dummy_fixes=should_enable_adp_dummy_fixes(mapping), + _enable_scheduler_aware_adp_dummy=False, + _enable_non_overlap_adp_forward_intent=False, + _benchmark_fill_phase_active=False, + is_warmup=False, + kv_cache_transceiver=None, + # PyExecutor's initial dummy role; only disaggregated serving changes it. + _adp_dummy_is_gen=True, + max_total_draft_tokens=0, + expected_num_active_requests=expected_num_active_requests, + active_requests=list(active_requests), + adp_router=SimpleNamespace(exclude_retiring_requests=False), + kv_cache_manager=_KvPool(_DELIVERED_POOL_TOKENS, dep4.max_seq_len), + resource_manager=_Resources(None), + ) + for name in ( + "_count_schedulable_active_requests", + "_should_skip_dummy_for_benchmark_disagg", + "_has_adp_dummy_kv_capacity", + ): + setattr(executor, name, MethodType(getattr(PyExecutor, name), executor)) + PyExecutor._pad_attention_dp_dummy_request(executor) + return executor + + idle = executor(expected_num_active_requests=1, active_requests=[]) + assert idle.kv_cache_manager.calls == [ + dict( + request_ids=[ATTENTION_DP_DUMMY_REQUEST_ID], + token_nums=None, + is_gen=True, + max_num_draft_tokens=0, + draft_kv_cache_manager=None, + ) + ] + (dummy,) = idle.active_requests + assert dummy.is_attention_dp_dummy is True and dummy.is_gen is True + assert (dummy.py_request_id, dummy.token_num) == (ATTENTION_DP_DUMMY_REQUEST_ID, 1) + + busy = executor(expected_num_active_requests=1, active_requests=[_decode_row(100)]) + assert busy.kv_cache_manager.calls == [] and len(busy.active_requests) == 1 + fleet_idle = executor(expected_num_active_requests=0, active_requests=[]) + assert fleet_idle.kv_cache_manager.calls == [] and fleet_idle.active_requests == [] + + +def test_decode_graphs_replay_only_when_every_rank_decodes(dep4, monkeypatch): + """The CUDA-graph rule under attention DP, on the engine's own runner: a step replays only + when every rank's batch is generation-only. Then each rank pads to the largest batch across + the ranks, rounded up to a captured size, so all replay the same batch size; an idle rank's + one-row dummy is padded like any batch. When any rank has a context request (a chunk), no + rank replays; a decode rank still pads its own batch to its own captured size and runs it + eagerly. The ranks agree on eligibility and batch size only: DSA captures a short- and a + long-sequence family at each size, and each rank picks its family from its own sequences, so + ranks can replay different families in one step (a 1M sequence takes the long one).""" + llm_args, model_config = dep4.llm_args, dep4.model_config + graphs = llm_args.cuda_graph_config + assert graphs.batch_sizes == [1, 2, 4, 8, 16] + assert graphs.enable_padding is True and graphs.max_batch_size == 16 + # Asking for the families writes the threshold onto the config; ask a copy. + sparse = copy.deepcopy(model_config.sparse_attention_config) + assert sparse.needs_separate_short_long_cuda_graphs() is True + threshold = sparse.seq_len_threshold + assert threshold == sparse.index_topk == 2048 + + # No device here: the static input buffers are not built. + monkeypatch.setattr(CUDAGraphRunner, "_create_shared_static_tensors", lambda self: None) + group = _TpGroup() + captured = SimpleNamespace(is_cuda_graph=True) + runners = [] + for rank in range(_RANKS): + runner = CUDAGraphRunner( + CUDAGraphRunnerConfig( + use_cuda_graph=True, + cuda_graph_padding_enabled=graphs.enable_padding, + cuda_graph_batch_sizes=graphs.batch_sizes, + max_cuda_graph_batch_size=graphs.max_batch_size, + max_beam_width=1, + max_num_tokens=dep4.max_num_tokens, + spec_config=None, + use_mrope=False, + original_max_total_draft_tokens=0, + enable_attention_dp=dep4.mapping.enable_attention_dp, + is_encoder_decoder=False, + batch_size=dep4.max_batch_size, + mapping=_rank_mapping(dep4.mapping, rank), + dist=group.dist(rank), + sparse_attention_config=sparse, + short_long_seq_len_threshold=threshold, + ) + ) + # What warmup leaves behind: the padding dummy (one generation token) and a graph per + # captured size in each family, captured with a longest row of max_seq_len - 1 (long) + # and threshold - 1 - 1 (short). + runner.padding_dummy_requests[0] = _decode_row(1, dummy=True) + for size in graphs.batch_sizes: + for longest in (dep4.max_seq_len - 1, threshold - 2): + key = runner.get_graph_key(_batch(decode=[longest] + [1] * (size - 1))) + runner.graph_metadata[key] = {"attn_metadata": captured, "spec_metadata": None} + runners.append(runner) + assert len(runners[0].graph_metadata) == 2 * len(graphs.batch_sizes) + + def step(batches): + def rank_step(rank): + runner = runners[rank] + with runner.pad_batch(batches[rank], None, 0) as padded: + attn_metadata, _, key = runner.maybe_get_cuda_graph( + padded, enable_spec_decode=False, attn_metadata=None + ) + return padded.batch_size, key, attn_metadata + + return group.run(rank_step) + + def idle(): + return _batch(decode=[_decode_row(1, dummy=True)]) + + # Every rank decodes, short sequences, unequal batches: all pad to 16 and replay it. + for sizes in ([3, None, 9, 16], [3, None, 9, 5]): + batches = [idle() if n is None else _batch(decode=[100] * n) for n in sizes] + for size, key, attn_metadata in step(batches): + assert size == 16 and attn_metadata is captured + assert (key.batch_size, key.draft_len, key.short_seq_len_mode) == (16, 0, True) + + # Small batches pad to the smallest captured size that holds the largest. + batches = [_batch(decode=[100, 100]), idle(), _batch(decode=[100, 100]), idle()] + assert [(size, key.batch_size) for size, key, _ in step(batches)] == [(2, 2)] * 4 + + # Equal sizes, different families: the ranks holding a 30000- and a 1048575-token sequence + # replay the long graph while their peers replay the short one. + long_rows = {1: 30000, 2: dep4.max_seq_len - 1} + batches = [_batch(decode=[long_rows.get(rank, 100)] * 16) for rank in range(_RANKS)] + modes = [key.short_seq_len_mode for _, key, _ in step(batches)] + assert modes == [True, False, False, True] + + # One rank runs a chunk: nobody replays, and a decode rank runs its own padded batch eagerly. + batches = [_batch(context=[32768]), idle(), _batch(decode=[100] * 9), _batch(decode=[100] * 16)] + results = step(batches) + assert [size for size, _, _ in results] == [1, 1, 16, 16] + assert all(key is None and attn_metadata is None for _, key, attn_metadata in results) + + +# ------------------------------------------------------- (6) the other configurations as before + + +def test_the_1m_config_moves_max_seq_len_and_chunked_prefill_only(dep4_ci, dep4_32k): + """Against the delivered 32K dep4 configuration the 1M config changes exactly two LLM-API + settings, max_seq_len (32768 -> 1048576) and enable_chunked_prefill (False -> True), and + nothing else the engine resolves: same mapping, quantization, model config, DSA config, + CUDA graphs and KV-cache config. Both as the CI classes pass them, the KV dtype at auto; the + gates' fp8 differs from it in that field alone + (``test_kv_cache_dtype_auto_and_fp8_build_the_fp8_ds_mla_pool``).""" + fields_1m, fields_32k = ( + {name: repr(getattr(resolved.llm_args, name)) for name in TorchLlmArgs.model_fields} + for resolved in (dep4_ci, dep4_32k) + ) + assert {name for name in fields_1m if fields_1m[name] != fields_32k[name]} == { + "max_seq_len", + "enable_chunked_prefill", + } + summary_1m, summary_32k = _summary(dep4_ci), _summary(dep4_32k) + assert {key for key in summary_1m if summary_1m[key] != summary_32k[key]} == { + "sizes", + "chunked_prefill", + } + assert (summary_1m["sizes"], summary_32k["sizes"]) == ((32768, 1048576, 16), (32768, 32768, 16)) + assert dep4_ci.mapping.to_dict() == dep4_32k.mapping.to_dict() + + +def test_tep4_and_tp4_route_as_before(tep4, tp4): + """The NVFP4 checkpoint at tep4 and at tp4 resolves and routes as at base: attention DP off, + unchunked at 32768, each claimed by its own target, each on the checkpoint's anchor and on + the fp8_ds_mla pool (``MistralLarge4CacheManager``), the same 47136-byte token.""" + cases = ( + (tep4, (4, 1), "ModelingV2MistralLarge4Nvfp4Sm103Tep"), + (tp4, (1, 4), "ModelingV2MistralLarge4Nvfp4Sm103Tp"), + ) + for resolved, moe_split, target in cases: + seen = resolved.routed[0] + mapping = seen.mapping + assert (mapping.world_size, mapping.tp_size, mapping.pp_size) == (4, 4, 1) + assert (mapping.moe_ep_size, mapping.moe_tp_size) == moe_split + assert mapping.enable_attention_dp is False + assert _route(seen.pretrained_config, mapping).__name__ == target + assert resolved.llm_args.quant_config.quant_algo == QuantAlgo.MIXED_PRECISION + assert resolved.llm_args.quant_config.kv_cache_quant_algo == QuantAlgo.FP8 + assert _anchor(resolved.llm_args) == 91.926 + manager = get_kv_cache_manager_cls(resolved.model_config, resolved.llm_args.kv_cache_config) + assert manager is MistralLarge4CacheManager + assert (_kv_cost(resolved, 0).slope, _kv_cost(resolved, 0).intercept) == (47136, 0) + assert resolved.llm_args.enable_chunked_prefill is False + assert (resolved.max_num_tokens, resolved.max_seq_len, resolved.max_batch_size) == ( + 32768, + 32768, + 16, + ) + + +def test_dep4_differs_from_tep4_in_attention_dp_only(dep4, dep4_32k, tep4): + """The same checkpoint at the delivered dep4 configuration and at tep4 resolves to the same + configuration -- quantization, per-layer quantization, model config, DSA config, limits, what + the router reads -- except the mapping's attention-DP bit. Routing reads neither max_seq_len + nor chunked prefill: the dep4 branch claims dep4 at 32K and at 1M alike, and the tep4 branch + claims neither.""" + assert _summary(dep4_32k) == _summary(tep4) + dep4_mapping, tep4_mapping = dep4_32k.mapping.to_dict(), tep4.mapping.to_dict() + assert {k for k in dep4_mapping if dep4_mapping[k] != tep4_mapping[k]} == { + "enable_attention_dp" + } + + for resolved in (dep4_32k, dep4): + seen = resolved.routed[0] + try: + target = _route(seen.pretrained_config, seen.mapping) + except NoTarget: + target = None + assert target is not None + assert target.__name__ == "ModelingV2MistralLarge4Nvfp4Sm103Dep" diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_dep4_engine_config.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_dep4_engine_config.py new file mode 100644 index 000000000000..dac2255988cb --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_dep4_engine_config.py @@ -0,0 +1,1589 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Mistral-Large-4-1T-A52B-NVFP4 at dep4: the engine's configuration, and the attention-DP contract +the dep4 forward is written against, at both worlds the target serves: four ranks on one GB300 +node, and eight over two nodes of four (dep8). + +No GPU and no weights. The checkpoint goes through the engine's own configuration path with the +LLM-API options of the core unit (four or eight ranks, attention data parallel, the routed +experts expert-parallel over every rank, 32768 tokens and 16 requests per rank, the latent KV +cache at its default dtype, the checkpoint's fp8_kv_cache: the fp8_ds_mla pool), in the +spellings the gates use: + +* the accuracy and serve gates' LLM-API options (spec-dep4/options/core.yaml and + spec-dep8/options/core.yaml), handed to ``TorchLlmArgs``; +* at eight ranks also the boot gate's command, parsed and turned into ``LLM`` keyword arguments + by the tree's own ``examples/llm-api/quickstart_advanced.py`` (``parse_arguments`` and + ``setup_llm``), as the gate harness builds its engine, plus the ``modeling_v2`` argument the + harness adds. + +Each then runs the LLM API front end (``ModelLoader._update_from_hf_quant_config``, +``llmapi/llm_utils.py``), which settles the key the accuracy harness looks its anchor up by, and +each rank's configuration pass (``_load_config_and_create_checkpoint_loader`` in +``py_executor_creator``, then the model loader's ``_load_and_validate_config``), which builds the +``ModelConfig`` and asks the modeling_v2 router for a target. The router's ``route`` is replaced +by a recorder, so what it is handed is observed rather than judged. + +No spelling states ``gpus_per_node``: ``TorchLlmArgs`` takes it from +``torch.cuda.device_count()`` on rank 0, which on a GB300 node of this cluster is 4. The count is +set to 4 where the arguments are built rather than read from the test host, which may see no +device or a whole node. What the forward receives under attention DP -- the per-rank token counts +in ``attn_metadata.all_rank_num_tokens``, an idle rank's dummy request, the CUDA-graph padding and +replay rule, the warmup's shapes, the KV-cache estimation's context step from the request queue +to the ranks -- is pinned on the engine's own host code at each world, its ranks simulated in +one process where they exchange anything (each rank a thread, each collective a barrier); so is +the expert path's call size, which the core fixes on the first step from the world and the +engine's runtime features, and at eight ranks where the engine reads the node layout on its way +to a modeling_v2 target. + +The checkpoint is Mistral-native, as published (params.json, tekken.json, consolidated +safetensors), and is loaded with checkpoint_format mistral: the gates' LLM-API options +state it and the boot gate's command passes --checkpoint_format=mistral. The checkpoint +at tep4 is checked to resolve and route as it did before this target, and at dep4 and at dep8 +to route to it. The checkpoint is looked up under ``LLM_MODELS_ROOT`` (as +``Mistral-Large-4-1T-A52B-NVFP4`` or ``mistral-large-4-nvfp4``) by its params.json, and its +tests are skipped where it is absent. +""" + +from __future__ import annotations + +import copy +import dataclasses +import importlib.util +import itertools +import json +import math +import queue +import sys +import threading +from pathlib import Path +from types import MethodType, SimpleNamespace + +import pytest +import torch +import yaml +from test_common.llm_data import llm_models_root + +from tensorrt_llm._ipc_utils import IpcMemory +from tensorrt_llm._torch.attention.backends.interface import ( + AttentionMetadata, + AttentionRuntimeFeatures, +) +from tensorrt_llm._torch.attention.backends.sparse.dsa.metadata import DSAtrtllmAttentionMetadata +from tensorrt_llm._torch.attention.backends.sparse.mistral_large4 import MistralLarge4CacheManager +from tensorrt_llm._torch.attention.backends.utils import get_attention_backend +from tensorrt_llm._torch.autotuner import AutoTuner, DistributedTuningStrategy +from tensorrt_llm._torch.custom_ops.trtllm_gen_custom_ops import FP4BlockScaleMoERunner +from tensorrt_llm._torch.distributed.communicator import Distributed, MPIDist +from tensorrt_llm._torch.modeling_v2 import _router_index +from tensorrt_llm._torch.modeling_v2._router_index import NoTarget +from tensorrt_llm._torch.modeling_v2.catalog.comm.allgather import allgather +from tensorrt_llm._torch.modeling_v2.catalog.comm.reducescatter import reducescatter +from tensorrt_llm._torch.modeling_v2.models.mistral_large4 import routing +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_103__dep import ( # noqa: E501 + modeling as M, +) +from tensorrt_llm._torch.models.modeling_utils import DecoderModel, DecoderModelForCausalLM +from tensorrt_llm._torch.moe.fused_moe.moe_load_balancer import maybe_create_moe_load_balancer +from tensorrt_llm._torch.pyexecutor import model_loader, py_executor_creator +from tensorrt_llm._torch.pyexecutor._util import ( + CacheCost, + KvCacheCreator, + get_kv_cache_manager_cls, + should_enable_adp_dummy_fixes, + should_enable_non_overlap_adp_forward_intent, + should_enable_scheduler_aware_adp_dummy, +) +from tensorrt_llm._torch.pyexecutor.cuda_graph_runner import CUDAGraphRunner, CUDAGraphRunnerConfig +from tensorrt_llm._torch.pyexecutor.engine.runners.common import ( + get_all_rank_num_tokens, + pad_input_tokens, +) +from tensorrt_llm._torch.pyexecutor.engine.runners.decoder.dummy_requests import DummyRequestFactory +from tensorrt_llm._torch.pyexecutor.engine.runners.decoder.warmup import ( + DecoderWarmup, + WarmupAgreement, +) +from tensorrt_llm._torch.pyexecutor.executor_request_queue import ( + ExecutorRequestQueue, + RequestQueueItem, +) +from tensorrt_llm._torch.pyexecutor.kv_cache.kv_cache_manager_v2 import ( + _compute_auto_host_tier_quota, +) +from tensorrt_llm._torch.pyexecutor.llm_request import ATTENTION_DP_DUMMY_REQUEST_ID +from tensorrt_llm._torch.pyexecutor.model_loader import ModelLoader as RankModelLoader +from tensorrt_llm._torch.pyexecutor.py_executor import PyExecutor +from tensorrt_llm._torch.pyexecutor.request_utils import ( + derive_attention_dp_per_rank_request_cap, + get_from_waiting_queue, +) +from tensorrt_llm._torch.pyexecutor.resource_manager import ResourceManagerType +from tensorrt_llm._torch.pyexecutor.scheduler import ScheduledRequests +from tensorrt_llm._torch.pyexecutor.scheduler.adp_router import ADPRouter, DefaultADPRouter +from tensorrt_llm._torch.pyexecutor.scheduler.rank_state import RankState +from tensorrt_llm._torch.pyexecutor.scheduler.waiting_queue import FCFSWaitingQueue +from tensorrt_llm.executor import base_worker +from tensorrt_llm.executor.base_worker import BaseWorker +from tensorrt_llm.llmapi.llm_args import PrefillCudaGraphBackend, TorchLlmArgs +from tensorrt_llm.llmapi.llm_utils import ModelLoader as FrontEndModelLoader +from tensorrt_llm.mapping import Mapping, MpiTopology +from tensorrt_llm.models.modeling_utils import QuantAlgo + +pytestmark = pytest.mark.cpu_only + +_SM103 = (10, 3) +# The attention-DP worlds the target serves: four ranks on one node, eight over two. +_WORLDS = (4, 8) +# dep8's ranks: two nodes of four. +_DEP8_RANKS = 8 +# Devices per GB300 node on this cluster: one node holds four ranks, two hold eight. +_GPUS_PER_NODE = 4 +# max_num_tokens = max_seq_len per rank: the 32K serving config. +_TOKENS = 32768 + +# The topologies. dep4 and dep8 are the core units' (spec-dep4/options/core.yaml, +# spec-dep8/options/core.yaml); tep4 is where the checkpoint already serves. +_DEP4 = dict(tensor_parallel_size=4, moe_expert_parallel_size=4, enable_attention_dp=True) +_DEP8 = dict(tensor_parallel_size=8, moe_expert_parallel_size=8, enable_attention_dp=True) +_TEP4 = dict(tensor_parallel_size=4, moe_expert_parallel_size=4, enable_attention_dp=False) + +# Each world, and the fixtures of its engine's spellings: core.yaml's, and at eight ranks the +# boot gate's command. +_each_world = pytest.mark.parametrize("world", _WORLDS, ids=lambda w: f"world{w}") +_ENGINES = {4: ("dep4",), 8: ("dep8", "dep8_gate")} + +# The dep8 boot gate's quickstart command after --model_dir: the checkpoint's Mistral-native +# format, the dep8 topology flags, the core unit's CUDA-graph capture list, and task.yaml's +# runtime flags. +_GATE_ARGV = ( + "--checkpoint_format=mistral", + "--max_tokens=64", + "--tp_size=8", + "--moe_ep_size=8", + "--enable_attention_dp", + "--use_cuda_graph", + "--cuda_graph_batch_sizes", + "1", + "2", + "4", + "8", + "16", + "--disable_kv_cache_reuse", + "--max_seq_len=32768", + "--max_batch_size=16", + "--max_num_tokens=32768", + "--cuda_graph_padding_enabled", +) +_QUICKSTART = Path(__file__).resolve().parents[7] / "examples/llm-api/quickstart_advanced.py" + + +def _options(topology: dict, tokens: int) -> dict: + """The LLM-API options of a Mistral Large 4 accuracy gate on ``topology``, the checkpoint in its + Mistral-native format, with the latent KV cache at its default (the checkpoint's + fp8_kv_cache: the fp8_ds_mla pool). Under attention DP max_batch_size and max_num_tokens bind + each rank.""" + return dict( + checkpoint_format="mistral", + **topology, + max_batch_size=16, + max_num_tokens=tokens, + max_seq_len=tokens, + enable_chunked_prefill=False, + cuda_graph_config=dict(enable_padding=True, max_batch_size=16), + kv_cache_config=dict(enable_block_reuse=False, free_gpu_memory_fraction=0.85), + ) + + +_NVFP4_NAMES = ("Mistral-Large-4-1T-A52B-NVFP4", "mistral-large-4-nvfp4") + +# The GSM8K reference rows, and the key the harness reads them by. +_GSM8K_REFERENCES = ( + Path(__file__).resolve().parents[6] / "integration/defs/accuracy/references/gsm8k.yaml" +) +_MODEL_NAME = "mistralai/Mistral-Large-4-1T-A52B" + +_DEP4_TARGET = "ModelingV2MistralLarge4Nvfp4Sm103Dep" + + +def _checkpoint_dir(names: tuple[str, ...]) -> Path: + """The first of ``names`` under LLM_MODELS_ROOT holding a Mistral-native params.json.""" + root = llm_models_root() + for name in names: + if root is not None and (root / name / "params.json").is_file(): + return root / name + pytest.skip(f"none of {names} is under LLM_MODELS_ROOT ({root})") + + +def _llm_args(devices: int = _GPUS_PER_NODE, **kwargs) -> TorchLlmArgs: + """``TorchLlmArgs`` as the LLM constructor builds it on rank 0 of a node with ``devices`` + visible devices (outside Ray).""" + with pytest.MonkeyPatch.context() as mp: + mp.delenv("RAY_LOCAL_WORLD_SIZE", raising=False) + mp.setattr(torch.cuda, "device_count", lambda: devices) + return TorchLlmArgs(**kwargs) + + +def _gate_kwargs(checkpoint_dir: Path, argv: tuple[str, ...]) -> dict: + """The ``LLM`` keyword arguments of a quickstart gate command: the tree's own + quickstart_advanced.py parses ``argv`` and builds them in ``setup_llm``, and the gate harness + adds ``modeling_v2``. Every KV-cache key of core.yaml has a quickstart flag, so the harness + hands over no other.""" + if not _QUICKSTART.is_file(): + pytest.skip(f"{_QUICKSTART} is not in this checkout") + spec = importlib.util.spec_from_file_location("quickstart_advanced", _QUICKSTART) + quickstart = importlib.util.module_from_spec(spec) + spec.loader.exec_module(quickstart) + built = {} + + def llm(**kwargs): + built.update(kwargs) + return SimpleNamespace() + + with pytest.MonkeyPatch.context() as mp: + mp.setattr(sys, "argv", ["quickstart_advanced.py", f"--model_dir={checkpoint_dir}", *argv]) + args = quickstart.parse_arguments() + mp.setattr(quickstart, "LLM", llm) + quickstart.setup_llm(args) + return dict(built, modeling_v2="require") + + +class _Routed: + """What the recording `route` returns: no target is built here.""" + + +def _route(pretrained_config, mapping): + """`route` for this config on ``mapping``, on sm_103 without speculative decoding.""" + model_config = SimpleNamespace( + pretrained_config=pretrained_config, mapping=mapping, spec_config=None + ) + return routing.route(model_config, None, _SM103) + + +def _resolve(checkpoint_dir: Path, llm_args: TorchLlmArgs) -> SimpleNamespace: + """The configuration a served engine reaches for this checkpoint, front end then rank.""" + model = str(checkpoint_dir) + prequantized = FrontEndModelLoader(llm_args)._update_from_hf_quant_config() + + routed = [] + + def route(model_config, llm_args, compute_capability): + pretrained_config = model_config.pretrained_config + routed.append( + SimpleNamespace( + pretrained_config=pretrained_config, + quantization_config=copy.deepcopy(pretrained_config.quantization_config), + mapping=model_config.mapping, + compute_capability=compute_capability, + spec_config=model_config.spec_config, + ) + ) + return _Routed + + with pytest.MonkeyPatch.context() as mp: + # The rank's config pass pins the CUDA memory fraction first; there is no device here. + mp.setattr(torch.cuda, "set_per_process_memory_fraction", lambda *args, **kwargs: None) + mp.setattr(_router_index, "_device_compute_capability", lambda: _SM103) + mp.setattr(_router_index, "registered_routes", lambda: {"MistralLarge4ForCausalLM": route}) + # One process stands in for every rank, so the rank exchange is the local resolution. + mp.setattr( + model_loader, "modeling_v2_resolve_on_all_ranks", _router_index.modeling_v2_resolve + ) + llm_args, checkpoint_loader = py_executor_creator._load_config_and_create_checkpoint_loader( + llm_args, model + ) + _, max_num_tokens, max_seq_len, max_batch_size = llm_args.get_runtime_sizes() + mapping = llm_args.parallel_config.to_mapping() + model_config = RankModelLoader( + llm_args=llm_args, + mapping=mapping, + spec_config=llm_args.speculative_config, + sparse_attention_config=llm_args.sparse_attention_config, + max_num_tokens=max_num_tokens, + max_seq_len=max_seq_len, + )._load_and_validate_config(model, checkpoint_loader) + return SimpleNamespace( + path=checkpoint_dir, + llm_args=llm_args, + prequantized=prequantized, + model_config=model_config, + mapping=mapping, + routed=routed, + max_num_tokens=max_num_tokens, + max_seq_len=max_seq_len, + max_batch_size=max_batch_size, + ) + + +def _resolve_options(names: tuple[str, ...], topology: dict, tokens: int) -> SimpleNamespace: + path = _checkpoint_dir(names) + options = _options(topology, tokens) + return _resolve(path, _llm_args(model=str(path), modeling_v2="require", **options)) + + +@pytest.fixture(scope="module") +def dep4(): + """The four-rank accuracy and serve gates' engine: spec-dep4/options/core.yaml.""" + return _resolve_options(_NVFP4_NAMES, _DEP4, _TOKENS) + + +@pytest.fixture(scope="module") +def dep8(): + """The eight-rank accuracy and serve gates' engine: spec-dep8/options/core.yaml.""" + return _resolve_options(_NVFP4_NAMES, _DEP8, _TOKENS) + + +@pytest.fixture(scope="module") +def dep8_gate(): + """The eight-rank boot gate's engine: its quickstart command.""" + path = _checkpoint_dir(_NVFP4_NAMES) + return _resolve(path, _llm_args(**_gate_kwargs(path, _GATE_ARGV))) + + +@pytest.fixture(scope="module") +def tep4(): + return _resolve_options(_NVFP4_NAMES, _TEP4, _TOKENS) + + +@pytest.fixture +def dep(request, world): + """The attention-DP engine at ``world`` ranks, from its core.yaml.""" + return request.getfixturevalue(_ENGINES[world][0]) + + +@pytest.fixture +def engines(request, world): + """Every spelling of the attention-DP engine at ``world`` ranks, core.yaml's first.""" + return [request.getfixturevalue(name) for name in _ENGINES[world]] + + +def _anchor(llm_args) -> float: + """The reference row `AccuracyTask.get_hypothesis_testing_params` selects for this engine.""" + quant = llm_args.quant_config + acc_specs = dict( + dtype=llm_args.dtype, + quant_algo=quant.quant_algo, + kv_cache_quant_algo=quant.kv_cache_quant_algo, + spec_dec_algo=None, + extra_acc_spec=None, + ) + for entry in yaml.safe_load(_GSM8K_REFERENCES.read_text())[_MODEL_NAME]: + if all( + entry.get(key, "auto" if key == "dtype" else None) == value + for key, value in acc_specs.items() + ): + return entry["accuracy"] + raise AssertionError(f"no {_MODEL_NAME} GSM8K reference row for {acc_specs}") + + +def _rank_mapping(mapping, rank: int): + rank_mapping = copy.deepcopy(mapping) + rank_mapping.rank = rank + return rank_mapping + + +def _kv_cost(resolved, rank: int) -> CacheCost: + """The engine's affine KV cost on ``rank``, as the KV-cache creator sizes the pool with it.""" + model_config, llm_args = resolved.model_config, resolved.llm_args + kv_cache_config = llm_args.kv_cache_config + manager = get_kv_cache_manager_cls(model_config, kv_cache_config) + return CacheCost.from_raw( + manager.get_cache_size_per_token( + model_config, + _rank_mapping(resolved.mapping, rank), + tokens_per_block=kv_cache_config.tokens_per_block, + max_seq_len=resolved.max_seq_len, + max_batch_size=resolved.max_batch_size, + max_num_tokens=resolved.max_num_tokens, + kv_cache_config=kv_cache_config, + spec_config=llm_args.speculative_config, + is_draft=False, + ) + ) + + +def _summary(resolved) -> dict: + """Everything the engine resolved for a checkpoint except its mapping and the MoE token + budget, which reads the mapping's attention-DP width.""" + pretrained = resolved.model_config.pretrained_config.to_dict() + pretrained.pop("_name_or_path", None) + llm_args, model_config = resolved.llm_args, resolved.model_config + return dict( + llm_quant=repr(llm_args.quant_config), + quant=repr(model_config.quant_config), + layers={k: repr(v) for k, v in (model_config.quant_config_dict or {}).items()}, + pretrained=pretrained, + sparse=model_config.sparse_attention_config.model_dump(), + routed=resolved.routed[0].quantization_config, + sizes=(resolved.max_num_tokens, resolved.max_seq_len, resolved.max_batch_size), + cuda_graphs=llm_args.cuda_graph_config.model_dump(), + kv_cache=llm_args.kv_cache_config.model_dump(), + moe=llm_args.moe_config.model_dump(), + moe_backend=model_config.moe_backend, + ) + + +def _route_or_none(seen): + try: + return _route(seen.pretrained_config, seen.mapping) + except NoTarget: + return None + + +# ------------------------------------------------------- the world's ranks in one process + + +class _Group: + """The ranks of the attention-DP group in one process: rank r is a thread, and every exchange + is a barrier, so each rank's call sees what every rank sent, as over MPI.""" + + def __init__(self, size: int): + self.size = size + self._barrier = threading.Barrier(size, timeout=60) + self._slots = [None] * size + + def exchange(self, rank: int, obj): + self._slots[rank] = obj + self._barrier.wait() + gathered = list(self._slots) + self._barrier.wait() + return gathered + + def run(self, step): + """``step(rank)`` on every rank at once; the per-rank results in rank order.""" + results, errors = [None] * self.size, [None] * self.size + + def body(rank): + try: + results[rank] = step(rank) + except BaseException as error: # noqa: BLE001 - re-raised below, after every rank + errors[rank] = error + self._barrier.abort() + + threads = [threading.Thread(target=body, args=(rank,)) for rank in range(self.size)] + for thread in threads: + thread.start() + for thread in threads: + thread.join() + # The rank that failed first, not a peer it left waiting at the barrier. + raised = [error for error in errors if error is not None] + for error in raised: + if not isinstance(error, threading.BrokenBarrierError): + raise error + if raised: + raise raised[0] + return results + + def dist(self, rank: int) -> "_RankDist": + return _RankDist(self, rank) + + +class _RankDist: + """One rank's communicator: the gathers the engine's host code issues under attention DP.""" + + # The engine's own int64 gather, over the `tp_allgather` below. + tp_allgather_int64 = Distributed.tp_allgather_int64 + + def __init__(self, group: _Group, rank: int): + self.group = group + self.rank = self.tp_rank = rank + self.world_size = self.tp_size = group.size + self.has_cp_helix = False + + def tp_allgather(self, obj, *, small_payload: bool = False): + return self.group.exchange(self.rank, obj) + + def allgather(self, obj): + return self.group.exchange(self.rank, obj) + + +_request_ids = itertools.count(1) + + +def _decode_row(kv_len: int, *, dummy: bool = False) -> SimpleNamespace: + """A generation request whose step attends ``kv_len`` keys (its own token included).""" + return SimpleNamespace( + py_request_id=next(_request_ids), + py_draft_tokens=[], + max_beam_num_tokens=kv_len, + is_dummy=dummy, + is_cuda_graph_dummy=False, + py_batch_idx=None, + ) + + +def _context_row(tokens: int) -> SimpleNamespace: + return SimpleNamespace( + py_request_id=next(_request_ids), + is_last_context_chunk=True, + context_chunk_size=tokens, + encoder_output_len=0, + py_skip_cross_kv_projection=False, + ) + + +def _batch(*, decode=(), context=()) -> ScheduledRequests: + batch = ScheduledRequests() + batch.reset_context_requests([_context_row(n) for n in context]) + batch.generation_requests = [ + row if isinstance(row, SimpleNamespace) else _decode_row(row) for row in decode + ] + return batch + + +class _KvPool: + """The KV manager surface the warmup's dummy requests read: ``tokens`` free tokens in pages of + 32, and dummies recorded as the token counts they were asked for (a generation dummy with no + count holds one token, as the engine's managers default).""" + + tokens_per_block = 32 + num_extra_kv_tokens = 0 + + def __init__(self, tokens: int, max_seq_len: int): + self.tokens = tokens + self.max_seq_len = max_seq_len + self.calls = [] + + def get_num_available_tokens(self, *, token_num_upper_bound, max_num_draft_tokens=0, **_): + return min(token_num_upper_bound, self.tokens) + + def get_num_free_blocks(self) -> int: + return self.tokens // self.tokens_per_block + + def add_dummy_requests(self, request_ids, token_nums=None, is_gen=False, **kwargs): + self.calls.append( + dict(request_ids=request_ids, token_nums=token_nums, is_gen=is_gen, **kwargs) + ) + if token_nums is None: + token_nums = [1 + kwargs.get("max_num_draft_tokens", 0)] * len(request_ids) + return [ + SimpleNamespace( + py_request_id=request_id, + token_num=token_num, + is_gen=is_gen, + is_last_context_chunk=True, + is_attention_dp_dummy=False, + py_draft_tokens=[], + py_needs_onehot_draft_probs=False, + ) + for request_id, token_num in zip(request_ids, token_nums) + ] + + def free_resources(self, request) -> None: + pass + + +class _Resources: + def __init__(self, kv: _KvPool): + self.kv = kv + + def get_resource_manager(self, key): + return self.kv if key == ResourceManagerType.KV_CACHE_MANAGER else None + + +def _runner_config(resolved) -> SimpleNamespace: + """The decoder runner's sizes for this engine. No speculative decoding: every draft width is + zero, and beams are one wide.""" + graphs = resolved.llm_args.cuda_graph_config + return SimpleNamespace( + max_num_tokens=resolved.max_num_tokens, + max_seq_len=resolved.max_seq_len, + max_batch_size=resolved.max_batch_size, + cuda_graph_batch_sizes=graphs.batch_sizes, + max_beam_width=1, + spec_config=None, + use_mrope=False, + max_draft_len=0, + original_max_draft_len=0, + max_total_draft_tokens=0, + max_draft_loop_tokens=0, + ) + + +# ------------------------------------------------------- (1) topology, node layout, limits + + +@_each_world +def test_dep4_topology_and_per_rank_limits(world, engines): + """``world`` ranks -- four on one node, or eight over two nodes of four (dep8) -- + tensor_parallel_size ``world`` spent on attention data parallelism (each rank holds every head + and schedules its own requests; attention TP is 1), the routed experts expert-parallel over + every rank and not split along their width, one pipeline stage -- at eight ranks from + core.yaml and from the boot gate's command alike. lm_head is not vocab-parallel under + attention DP. Rank r is TP rank r and MoE EP rank r, MoE TP rank 0: its expert window is + [E r, E r + E), E = 128 / world (32 at four ranks, 16 at eight). mapping.tp_size, + attn_tp_size and dp_size all read the world, so no width outside the expert window may be + divided by them. max_batch_size and max_num_tokens are each rank's own: 16 requests and 32768 + tokens per rank, 16 x world requests in flight (64 over four ranks, 128 over eight), + unchunked.""" + local = {4: 32, 8: 16}[world] + for resolved in engines: + llm_args, mapping = resolved.llm_args, resolved.mapping + parallel = llm_args.parallel_config + assert (parallel.tp_size, parallel.pp_size, parallel.cp_size) == (world, 1, 1) + # Every spelling states the EP split only; the MoE TP size is left to Mapping. + assert (parallel.moe_ep_size, parallel.moe_tp_size) == (world, -1) + assert llm_args.enable_attention_dp is True and parallel.enable_attention_dp is True + + assert (mapping.world_size, mapping.tp_size, mapping.pp_size, mapping.cp_size) == ( + world, + world, + 1, + 1, + ) + assert (mapping.moe_ep_size, mapping.moe_tp_size, mapping.moe_cluster_size) == (world, 1, 1) + assert mapping.enable_attention_dp is True + # The flag, not a width, says attention is replicated: all three read the world. + assert (mapping.tp_size, mapping.attn_tp_size, mapping.dp_size) == (world, world, world) + assert mapping.tp_size // mapping.dp_size == 1 + assert mapping.enable_lm_head_tp_in_adp is False + assert mapping.has_moe_ep() and not mapping.has_moe_tp() + assert resolved.model_config.mapping is mapping + + cfg = resolved.model_config.pretrained_config + assert (cfg.num_attention_heads, cfg.n_routed_experts) == (64, 128) + assert cfg.n_routed_experts // mapping.moe_ep_size == local + for rank in range(world): + rank_mapping = _rank_mapping(mapping, rank) + assert ( + rank_mapping.tp_rank, + rank_mapping.moe_ep_rank, + rank_mapping.moe_tp_rank, + rank_mapping.moe_cluster_rank, + ) == (rank, rank, 0, 0) + assert rank_mapping.ep_experts(cfg.n_routed_experts) == list( + range(local * rank, local * rank + local) + ) + assert rank_mapping.tp_group == list(range(world)) + + assert (resolved.max_batch_size, resolved.max_num_tokens, resolved.max_seq_len) == ( + 16, + 32768, + 32768, + ) + model_config = resolved.model_config + assert (model_config.max_num_tokens, model_config.max_seq_len) == (32768, 32768) + assert llm_args.enable_chunked_prefill is False + # PyExecutor's per-rank cap on active requests under attention DP. + cap = derive_attention_dp_per_rank_request_cap( + base_cap=mapping.pp_size * resolved.max_batch_size, + max_num_tokens=resolved.max_num_tokens, + max_total_draft_tokens=0, + ) + assert cap == 16 and cap * mapping.dp_size == {4: 64, 8: 128}[world] + # No attention-DP balancing or KV-aware routing: the default router places requests. + # core.yaml states no attention_dp_config; the quickstart builds one at its defaults. + adp = llm_args.attention_dp_config + assert (adp is None) == (resolved is engines[0]) + assert adp is None or (adp.enable_balance, adp.enable_kv_cache_aware_routing) == ( + False, + False, + ) + router = ADPRouter.create(dist=None, has_seq_slot_headroom=False, attention_dp_config=adp) + assert type(router) is DefaultADPRouter + + +def test_gpus_per_node_is_the_visible_device_count(dep8, dep8_gate): + """At eight ranks over two nodes of four (dep8), neither gate states gpus_per_node. + ``TorchLlmArgs.gpus_per_node`` defaults to None and its validator (outside Ray) replaces None + with ``torch.cuda.device_count()`` of the process that builds the arguments, rank 0; each node's + ranks see that node's four held devices, so it is 4. The mapping every rank builds from + ``parallel_config`` carries it: rank r is local rank r % 4 on node r // 4, and the deployment is + multi-node. The LLM constructor's device check reads it too: world_size_per_node is 4, which + four visible devices satisfy. + + The defaults of ``_ParallelConfig`` and ``Mapping`` are 8, not the device count: an + eight-rank mapping built without it reads one node of eight.""" + field = TorchLlmArgs.model_fields["gpus_per_node"] + assert field.default is None and field.validate_default is True + assert "gpus_per_node" not in _options(_DEP8, _TOKENS) + assert "gpus_per_node" not in _gate_kwargs(dep8_gate.path, _GATE_ARGV) + + for resolved in (dep8, dep8_gate): + llm_args, mapping = resolved.llm_args, resolved.mapping + assert llm_args.gpus_per_node == _GPUS_PER_NODE == 4 + assert llm_args.parallel_config.gpus_per_node == 4 + assert mapping.gpus_per_node == 4 and resolved.model_config.mapping.gpus_per_node == 4 + assert llm_args.parallel_config.world_size_per_node == 4 + for rank in range(_DEP8_RANKS): + rank_mapping = _rank_mapping(mapping, rank) + assert (rank_mapping.local_rank, rank_mapping.node_rank) == (rank % 4, rank // 4) + assert rank_mapping.get_node_rank(rank) == rank // 4 + assert rank_mapping.is_multi_node() is True + + # The count is the visible devices' count, read when the arguments are built. + path = str(dep8.path) + for devices in (4, 8): + llm_args = _llm_args(devices, model=path, **_options(_DEP8, _TOKENS)) + assert llm_args.gpus_per_node == devices + assert llm_args.parallel_config.world_size_per_node == devices + stated = _llm_args(1, model=path, gpus_per_node=4, **_options(_DEP8, _TOKENS)) + assert stated.gpus_per_node == 4 + + +@_each_world +def test_each_rank_selects_its_own_nodes_device(world, dep): + """The device each worker sets before building its executor (``BaseWorker._configure_device``, + executor/base_worker.py): its global MPI rank modulo the devices its node shows, so rank r + drives device r % 4 of its node -- the mapping's local rank, at four ranks on the one node and + at eight on either. Each rank checks that its device's architecture is supported and pins its + CPU affinity to it.""" + group = _Group(world) + local = threading.local() + with pytest.MonkeyPatch.context() as mp: + mp.setattr(torch.cuda, "device_count", lambda: _GPUS_PER_NODE) + mp.setattr(torch.cuda, "set_device", lambda device: setattr(local, "device", device)) + mp.setattr( + base_worker, + "check_cuda_architecture_supported", + lambda device: setattr(local, "checked", device), + ) + mp.setattr( + base_worker, "configure_cpu_affinity", lambda device: setattr(local, "cpus", device) + ) + + def step(rank): + BaseWorker._configure_device(SimpleNamespace(global_rank=rank)) + return local.checked, local.device, local.cpus + + results = group.run(step) + for rank, (checked, device, cpus) in enumerate(results): + assert checked == device == cpus == rank % 4 + assert device == _rank_mapping(dep.mapping, rank).local_rank + + +def test_boot_command_and_core_yaml_build_one_deployment(dep8, dep8_gate): + """At eight ranks, the boot gate's quickstart command and core.yaml resolve to the same mapping + on every rank and the same model, quantization, DSA, limits, CUDA graphs, KV dtype, page size, + block-reuse setting, KV manager and MoE token budget; the router is handed equal mappings. The + KV fraction is not shared: the quickstart's --kv_cache_fraction default, 0.9, against + core.yaml's 0.85, so the boot gate's pool is sized at 0.9 of the free memory and the accuracy + gate's at 0.85.""" + assert dep8_gate.mapping.to_dict() == dep8.mapping.to_dict() + assert dep8_gate.routed[0].mapping.to_dict() == dep8.routed[0].mapping.to_dict() + gate, core = _summary(dep8_gate), _summary(dep8) + for key in ("llm_quant", "quant", "layers", "pretrained", "sparse", "routed", "sizes"): + assert gate[key] == core[key], key + assert dep8_gate.model_config.moe_max_num_tokens == dep8.model_config.moe_max_num_tokens + for resolved in (dep8, dep8_gate): + graphs = resolved.llm_args.cuda_graph_config + assert graphs.batch_sizes == [1, 2, 4, 8, 16] and graphs.enable_padding is True + kv = resolved.llm_args.kv_cache_config + assert (kv.dtype, kv.enable_block_reuse, kv.tokens_per_block) == ("auto", False, 32) + assert kv.use_kv_cache_manager_v2 is True + assert resolved.llm_args.disable_overlap_scheduler is False + assert dep8.llm_args.kv_cache_config.free_gpu_memory_fraction == 0.85 + assert dep8_gate.llm_args.kv_cache_config.free_gpu_memory_fraction == 0.9 + + +# ------------------------------------------------------- (1) quantization, the anchor's key + + +@_each_world +def test_quantization_is_the_anchor_key(world, engines): + """At dep4 and dep8 as at tep4: MIXED_PRECISION with the FP8 KV cache, the GSM8K anchor row's + key, and the router handed params.json's quantization_config with its 475 quantized_layers + intact plus the FP8 KV-cache algorithm the native loader records from fp8_kv_cache, on the + world's mapping -- at eight ranks from core.yaml and from the boot gate's command alike.""" + on_disk = json.loads((engines[0].path / "params.json").read_text())["quantization_config"] + assert len(on_disk["quantized_layers"]) == 475 + routed = dict(on_disk, kv_cache_quant_algo="FP8") + for resolved in engines: + assert resolved.prequantized is True + for quant in (resolved.llm_args.quant_config, resolved.model_config.quant_config): + assert quant.quant_algo == QuantAlgo.MIXED_PRECISION + assert quant.kv_cache_quant_algo == QuantAlgo.FP8 + assert quant.group_size == 16 + assert resolved.llm_args.kv_cache_config.dtype == "auto" + assert _anchor(resolved.llm_args) == 91.926 + + assert len(resolved.routed) == 1 + seen = resolved.routed[0] + assert seen.quantization_config == routed + assert resolved.model_config.pretrained_config.quantization_config == routed + assert seen.pretrained_config is resolved.model_config.pretrained_config + assert resolved.model_config.modeling_v2_target is _Routed + assert seen.compute_capability == _SM103 and seen.spec_config is None + assert seen.mapping is resolved.model_config.mapping + mapping = seen.mapping + assert (mapping.world_size, mapping.tp_size, mapping.pp_size) == (world, world, 1) + assert (mapping.moe_ep_size, mapping.moe_tp_size) == (world, 1) + assert mapping.enable_attention_dp is True + assert mapping.enable_lm_head_tp_in_adp is False + + +# ------------------------------------------------------- (2) the KV pool, per rank + + +@_each_world +def test_kv_pool_has_no_rank_factor(world, engines, dep4, tep4): + """Each rank holds the latent and indexer-key caches of its own requests only, and the cost of + a token is the same on each of the world's ranks, at four and at eight, as on each tep4 rank: + an fp8_ds_mla latent of 656 bytes (512 e4m3, four fp32 tile scales, 64 bf16 rope) on each of + the 63 KV owners plus a 128-byte fp8 indexer key and its fp32 scale on each of the 44 + full-indexer layers, in pages of 32, in the MistralLarge4CacheManager the engine builds for + this checkpoint; the nine KV sharers keep no latent. A rank's own 16 requests at 32768 tokens + need that times their tokens.""" + for resolved in (*engines, dep4, tep4): + model_config, kv_cache_config = resolved.model_config, resolved.llm_args.kv_cache_config + assert get_kv_cache_manager_cls(model_config, kv_cache_config) is MistralLarge4CacheManager + assert kv_cache_config.tokens_per_block == 32 + + dep = engines[0] + cfg, sparse = dep.model_config.pretrained_config, dep.model_config.sparse_attention_config + latent = 656 # fp8_ds_mla: 512 e4m3 + 4 fp32 tile scales + 64 bf16 rope + owners = sum(1 for i, s in enumerate(cfg.kv_source_layers) if s == i) + indexer_key = sparse.index_head_dim + sparse.index_head_dim // 128 * 4 + full_indexer_layers = cfg.index_topk_pattern.count("F") + per_token = latent * owners + indexer_key * full_indexer_layers + assert (latent, owners, indexer_key, full_indexer_layers) == (656, 63, 132, 44) + assert per_token == 47136 + + for rank in range(world): + cost = _kv_cost(dep, rank) + assert (cost.slope, cost.intercept) == (per_token, 0) + assert all(cost == _kv_cost(other, rank) for other in engines[1:]) + assert cost == _kv_cost(dep4, rank % 4) == _kv_cost(tep4, rank % 4) + assert cost.bytes_for_tokens(32) == 32 * per_token + + tokens = dep.max_batch_size * dep.max_seq_len + pool = _kv_cost(dep, 0).bytes_for_tokens(tokens) + assert pool == tokens * per_token + assert math.isclose(pool / 2**30, 23.016, abs_tol=5e-4) + + +@_each_world +def test_indexer_chunk_holds_a_whole_prompt(world, dep): + """The indexer's [T, N] logits are cut at indexer_max_chunk_size: a rank's whole 32K prompt, + its largest unchunked step, is one chunk.""" + model_config = dep.model_config + params = model_config.sparse_attention_config.to_sparse_metadata_params( + pretrained_config=model_config.pretrained_config + ) + assert params.indexer_max_chunk_size >= 32768 + assert params.indexer_max_chunk_size >= dep.max_num_tokens + + +# ------------------------------------------------------- (3) what the forward receives + + +@_each_world +def test_every_rank_is_handed_every_rank_token_count(world, dep): + """``attn_metadata.all_rank_num_tokens`` is set on every step of every rank, by the gather in + the decoder's input preparation (eager, warmup and capture steps alike): each rank's own + token count, one entry per rank in rank order, identical on every rank and NOT padded (no + prefill CUDA graph backend is configured). An idle rank counts the one token of its + generation dummy. Padding to the largest count before the experts' collectives is the + forward's own job. When every rank prefills a whole step at once, the experts see world x + 32768 gathered rows: 131072 at four ranks, 524288 after top-4; 262144 at eight, 1048576 + after top-4.""" + llm_args, model_config = dep.llm_args, dep.model_config + sparse_params = model_config.sparse_attention_config.to_sparse_params( + pretrained_config=model_config.pretrained_config + ) + metadata_cls = get_attention_backend( + llm_args.attn_backend, sparse_params=sparse_params + ).Metadata + assert metadata_cls is DSAtrtllmAttentionMetadata + # DSA exchanges the one plain count; it has no per-stage counts of its own. + assert metadata_cls.get_adp_token_counts is AttentionMetadata.get_adp_token_counts + assert metadata_cls.set_adp_token_counts is AttentionMetadata.set_adp_token_counts + assert llm_args.prefill_cuda_graph_backend == PrefillCudaGraphBackend.DISABLED + + group = _Group(world) + unequal = { + # a 32K prefill, an idle rank's dummy, a mixed step, sixteen decode rows + 4: ([32768, 1, 517, 16], [1, 0, 1, 0]), + # 32K prefills on both nodes, idle ranks' dummies, a mixed step, sixteen decode rows + 8: ([32768, 1, 517, 16, 1, 32768, 3, 1], [1, 0, 1, 0, 0, 1, 0, 0]), + } + steps = { + # every rank prefills a whole 32K step, as the warmup's max shape does + "prefills": ([32768] * world, [2] * world), + "unequal": unequal[world], + # a decode graph capture: every rank at the same captured size + "capture": ([16] * world, [0] * world), + } + for counts, contexts in steps.values(): + + def rank_step(rank, counts=counts, contexts=contexts): + md = SimpleNamespace( + num_tokens=counts[rank], padded_num_tokens=None, all_rank_num_tokens=None + ) + md.get_adp_token_counts = MethodType(metadata_cls.get_adp_token_counts, md) + md.set_adp_token_counts = MethodType(metadata_cls.set_adp_token_counts, md) + dist = group.dist(rank) + gathered = get_all_rank_num_tokens( + md, + enable_attention_dp=True, + mapping=_rank_mapping(dep.mapping, rank), + dist=dist, + ) + num_tokens = pad_input_tokens( + md, + counts[rank], + contexts[rank], + gathered, + input_ids=torch.zeros(counts[rank], dtype=torch.int32), + position_ids=torch.zeros(counts[rank], dtype=torch.int32), + dist=dist, + enable_attention_dp=True, + prefill_cuda_graph_backend=llm_args.prefill_cuda_graph_backend, + prefill_cuda_graph_num_tokens=[], + ) + return num_tokens, md.all_rank_num_tokens, md.padded_num_tokens + + for rank, (num_tokens, all_rank, padded) in enumerate(group.run(rank_step)): + assert num_tokens == counts[rank] and padded is None + assert all_rank == counts and len(all_rank) == world + + rows = sum(steps["prefills"][0]) + assert rows == world * dep.max_num_tokens == {4: 131072, 8: 262144}[world] + top4 = rows * model_config.pretrained_config.num_experts_per_tok + assert top4 == {4: 524288, 8: 1048576}[world] + + +@_each_world +def test_an_idle_rank_steps_with_a_one_token_generation_dummy(world, dep): + """When some rank has work and this one has none, the executor gives it one dummy before + scheduling: request id ATTENTION_DP_DUMMY_REQUEST_ID, a generation request with the KV + manager's default of one token (no draft tokens), its KV prepared like a real request's. + Aggregated serving with the overlap scheduler takes this plain path: no KV-cache transceiver, + and neither the scheduler-aware nor the forward-intent variant. The rank's step is then + generation-only with one token, so it runs the decode target while its peers may prefill. + When no rank has work, no rank gets a dummy and none steps.""" + llm_args, mapping = dep.llm_args, dep.mapping + assert llm_args.disable_overlap_scheduler is False + assert should_enable_adp_dummy_fixes(mapping) is True + model_type = dep.model_config.pretrained_config.model_type + assert not should_enable_scheduler_aware_adp_dummy( + model_type, mapping, llm_args.disable_overlap_scheduler + ) + assert not should_enable_non_overlap_adp_forward_intent( + mapping, llm_args.disable_overlap_scheduler + ) + + def executor(expected_num_active_requests, active_requests): + """The executor state `_pad_attention_dp_dummy_request` reads, with its own helpers.""" + executor = SimpleNamespace( + enable_attention_dp=mapping.enable_attention_dp, + # The attention-DP group: a single-rank group skips dummy padding. + dist=SimpleNamespace(tp_size=mapping.tp_size), + _enable_remote_tail_adp=False, + # No speculative decoding: no draft KV cache joins the target's reuse. + enable_joint_kv_cache_reuse=False, + _enable_adp_dummy_fixes=should_enable_adp_dummy_fixes(mapping), + _enable_scheduler_aware_adp_dummy=False, + _enable_non_overlap_adp_forward_intent=False, + _benchmark_fill_phase_active=False, + is_warmup=False, + kv_cache_transceiver=None, + # PyExecutor's initial dummy role; only disaggregated serving changes it. + _adp_dummy_is_gen=True, + max_total_draft_tokens=0, + expected_num_active_requests=expected_num_active_requests, + active_requests=list(active_requests), + adp_router=SimpleNamespace(exclude_retiring_requests=False), + kv_cache_manager=_KvPool(dep.max_batch_size * dep.max_seq_len, dep.max_seq_len), + resource_manager=_Resources(None), + ) + for name in ( + "_count_schedulable_active_requests", + "_should_skip_dummy_for_benchmark_disagg", + "_has_adp_dummy_kv_capacity", + ): + setattr(executor, name, MethodType(getattr(PyExecutor, name), executor)) + PyExecutor._pad_attention_dp_dummy_request(executor) + return executor + + idle = executor(expected_num_active_requests=1, active_requests=[]) + assert idle.kv_cache_manager.calls == [ + dict( + request_ids=[ATTENTION_DP_DUMMY_REQUEST_ID], + token_nums=None, + is_gen=True, + max_num_draft_tokens=0, + draft_kv_cache_manager=None, + ) + ] + (dummy,) = idle.active_requests + assert dummy.is_attention_dp_dummy is True and dummy.is_gen is True + assert (dummy.py_request_id, dummy.token_num) == (ATTENTION_DP_DUMMY_REQUEST_ID, 1) + + busy = executor(expected_num_active_requests=1, active_requests=[_decode_row(100)]) + assert busy.kv_cache_manager.calls == [] and len(busy.active_requests) == 1 + fleet_idle = executor(expected_num_active_requests=0, active_requests=[]) + assert fleet_idle.kv_cache_manager.calls == [] and fleet_idle.active_requests == [] + + +@_each_world +def test_decode_graphs_replay_only_when_every_rank_decodes(world, dep, monkeypatch): + """The CUDA-graph rule under attention DP, on the engine's own runner: a step replays only + when every rank's batch, over the four or the eight, is generation-only. Then each rank pads + to the largest batch across the ranks, rounded up to a captured size, so all replay the same + batch size; an idle rank's one-row dummy is padded like any batch. When any rank has a + context request, no rank replays; a decode rank still pads its own batch to its own captured + size and runs it eagerly. The ranks agree on eligibility and batch size only: DSA captures a + short- and a long-sequence family at each size, and each rank picks its family from its own + sequences, so ranks can replay different families in one step.""" + llm_args, model_config = dep.llm_args, dep.model_config + graphs = llm_args.cuda_graph_config + assert graphs.batch_sizes == [1, 2, 4, 8, 16] + assert graphs.enable_padding is True and graphs.max_batch_size == 16 + # Asking for the families writes the threshold onto the config; ask a copy. + sparse = copy.deepcopy(model_config.sparse_attention_config) + assert sparse.needs_separate_short_long_cuda_graphs() is True + threshold = sparse.seq_len_threshold + assert threshold == sparse.index_topk == 2048 + + # No device here: the static input buffers are not built. + monkeypatch.setattr(CUDAGraphRunner, "_create_shared_static_tensors", lambda self: None) + group = _Group(world) + captured = SimpleNamespace(is_cuda_graph=True) + runners = [] + for rank in range(world): + runner = CUDAGraphRunner( + CUDAGraphRunnerConfig( + use_cuda_graph=True, + cuda_graph_padding_enabled=graphs.enable_padding, + cuda_graph_batch_sizes=graphs.batch_sizes, + max_cuda_graph_batch_size=graphs.max_batch_size, + max_beam_width=1, + max_num_tokens=dep.max_num_tokens, + spec_config=None, + use_mrope=False, + original_max_total_draft_tokens=0, + enable_attention_dp=dep.mapping.enable_attention_dp, + is_encoder_decoder=False, + batch_size=dep.max_batch_size, + mapping=_rank_mapping(dep.mapping, rank), + dist=group.dist(rank), + sparse_attention_config=sparse, + short_long_seq_len_threshold=threshold, + ) + ) + # What warmup leaves behind: the padding dummy (one generation token) and a graph per + # captured size in each family, captured with a longest row of max_seq_len - 1 (long) + # and threshold - 1 - 1 (short). + runner.padding_dummy_requests[0] = _decode_row(1, dummy=True) + for size in graphs.batch_sizes: + for longest in (dep.max_seq_len - 1, threshold - 2): + key = runner.get_graph_key(_batch(decode=[longest] + [1] * (size - 1))) + runner.graph_metadata[key] = {"attn_metadata": captured, "spec_metadata": None} + runners.append(runner) + assert len(runners[0].graph_metadata) == 2 * len(graphs.batch_sizes) + + def step(batches): + def rank_step(rank): + runner = runners[rank] + with runner.pad_batch(batches[rank], None, 0) as padded: + attn_metadata, _, key = runner.maybe_get_cuda_graph( + padded, enable_spec_decode=False, attn_metadata=None + ) + return padded.batch_size, key, attn_metadata + + return group.run(rank_step) + + def idle(): + return _batch(decode=[_decode_row(1, dummy=True)]) + + # Every rank decodes, short sequences, unequal batches (on both nodes at eight ranks): all + # pad to 16 and replay it. + unequal = { + 4: ([3, None, 9, 16], [3, None, 9, 5]), + 8: ([3, None, 9, 16, 1, None, 5, 2], [3, None, 9, 5, None, None, 1, 12]), + } + for sizes in unequal[world]: + batches = [idle() if n is None else _batch(decode=[100] * n) for n in sizes] + for size, key, attn_metadata in step(batches): + assert size == 16 and attn_metadata is captured + assert (key.batch_size, key.draft_len, key.short_seq_len_mode) == (16, 0, True) + + # Small batches pad to the smallest captured size that holds the largest. + batches = [_batch(decode=[100, 100]) if rank % 2 == 0 else idle() for rank in range(world)] + assert [(size, key.batch_size) for size, key, _ in step(batches)] == [(2, 2)] * world + + # Equal sizes, different families: the rank holding a 30000-token sequence (rank 1 at four + # ranks, rank 5 on the second node at eight) replays the long graph while its peers replay + # the short one. + long_rank = {4: 1, 8: 5}[world] + batches = [_batch(decode=[30000 if rank == long_rank else 100] * 16) for rank in range(world)] + modes = [key.short_seq_len_mode for _, key, _ in step(batches)] + assert modes == {4: [True, False, True, True], 8: [True] * 5 + [False] + [True] * 2}[world] + + # One rank prefills: nobody replays, and a decode rank runs its own padded batch eagerly. + prefill_step = { + 4: ((None, 9, 16), [1, 1, 16, 16]), + 8: ((None, 9, 16, None, None, 3, 1), [1, 1, 16, 16, 1, 1, 4, 1]), + } + others, sizes = prefill_step[world] + batches = [_batch(context=[32768])] + [ + idle() if n is None else _batch(decode=[100] * n) for n in others + ] + results = step(batches) + assert [size for size, _, _ in results] == sizes + assert all(key is None and attn_metadata is None for _, key, attn_metadata in results) + + +@_each_world +def test_warmup_runs_the_max_context_step_on_every_rank_at_once(world, dep): + """The warmup's shapes under attention DP. Each rank proposes its shapes from its own free + KV, the ranks keep the ones all proposed, and every rank enters each shape's forward. The max + context shape is max_num_tokens of context on every rank in the same step -- a 32767-token + prompt and a 1-token prompt, since a prompt leaves room for one decode token -- so the routed + experts see a 32K prefill on every rank at boot: 131072 gathered rows at four ranks, 262144 at + eight. Each decode graph is captured on a batch whose first row attends max_seq_len - 1 keys + (long family) or threshold - 2 (short family) and whose other rows attend one.""" + config = _runner_config(dep) + sparse = copy.deepcopy(dep.model_config.sparse_attention_config) + assert sparse.needs_separate_short_long_cuda_graphs() is True + model = SimpleNamespace(model_config=dep.model_config) + group = _Group(world) + + def rank_warmup(rank): + # A rank's own pool holds at least its 16 requests of max_seq_len. + resources = _Resources( + _KvPool(config.max_batch_size * config.max_seq_len, config.max_seq_len) + ) + warmup = DecoderWarmup.__new__(DecoderWarmup) + warmup._config = config + agreement = WarmupAgreement(group.dist(rank), _rank_mapping(dep.mapping, rank)) + assert agreement.is_distributed_forward() + shapes = agreement.agree_shapes(warmup._get_full_general_warmup_requests(resources)) + factory = DummyRequestFactory( + config=config, + model=model, + get_runtime_tokens_per_gen_step=lambda draft_len: 1 + draft_len, + release_request_state=None, + ) + max_context = max(tokens for tokens, gen in shapes if gen == 0) + batch = factory.create_warmup_request(resources, max_context, 0) + enters = agreement.should_run_warmup_batch(batch, max_context, "max context") + prompts = [request.token_num for request in batch.context_requests] + captures = {} + for family, max_seq_len in ( + ("long", config.max_seq_len), + ("short", sparse.seq_len_threshold - 1), + ): + capture = factory.create_cuda_graph_warmup_request( + resources, config.cuda_graph_batch_sizes[-1], 0, max_seq_len + ) + captures[family] = [request.token_num for request in capture.generation_requests] + return shapes, enters, prompts, captures + + results = group.run(rank_warmup) + for shapes, enters, prompts, captures in results: + assert shapes == [(1, 0), (1, 1), (32768, 0), (16, 16), (2, 0)] + assert enters is True + assert prompts == [32767, 1] + assert captures == {"long": [32767] + [1] * 15, "short": [2046] + [1] * 15} + total = sum(sum(prompts) for _, _, prompts, _ in results) + assert total == world * dep.max_num_tokens == {4: 131072, 8: 262144}[world] + + +@_each_world +def test_kv_estimation_puts_a_whole_context_step_on_every_rank(world, dep): + """Before the real pool exists the engine runs a profiling pass: under attention DP it builds + one copy of its max_num_tokens of dummy context per rank (a 32767-token and a 1-token prompt + each, 2 x world requests) and rank 0 enqueues all of them before the executor loop starts. + The loop's first fetch finds the queue non-empty and drains it whole (no batch wait is + configured); the waiting queue hands all of them on, within the world x 16 admission capacity + (64 at four ranks, 128 at eight); and the default router gives every rank one of each. So the + estimation's first step is a 32768-token context step on every rank at once -- 131072 + gathered rows at every MoE layer at four ranks, 262144 at eight -- and each rank's scheduler + takes its two prompts together (32767 + 1 tokens within max_num_tokens, unchunked).""" + llm_args = dep.llm_args + creator = KvCacheCreator.__new__(KvCacheCreator) + creator._model_engine = SimpleNamespace(model=SimpleNamespace(model_config=dep.model_config)) + creator._max_num_tokens = dep.max_num_tokens + creator._max_beam_width = 1 + creator._mapping = dep.mapping + # The estimation's dummies are as long as a request may be: net max_seq_len - 1. + requests = creator._create_dummy_context_requests(max(1, dep.max_seq_len - 1)) + assert creator._mla_chunked_profile_length is None + lengths = [len(request.input_token_ids) for request in requests] + assert lengths == [32767, 1] * world + + # Rank 0's request queue, filled before the loop: the first fetch takes every item. + assert llm_args.batch_wait_timeout_ms == 0 + request_queue = queue.Queue() + for request_id, request in enumerate(requests, start=1): + request_queue.put(RequestQueueItem(request_id, request)) + fetched = ExecutorRequestQueue.get_from_request_queue( + SimpleNamespace( + request_queue=request_queue, + batch_wait_timeout_ms=llm_args.batch_wait_timeout_ms, + max_batch_size=dep.max_batch_size, + ), + None, + ) + assert len(fetched) == 2 * world and request_queue.empty() + + # Every rank's copy of the waiting queue gives them all up at once. + waiting = FCFSWaitingQueue() + waiting.add_requests(fetched) + capacity = PyExecutor._get_request_admission_capacity( + SimpleNamespace( + enable_attention_dp=True, + dist=SimpleNamespace(tp_size=dep.mapping.tp_size), + max_num_active_requests=dep.max_batch_size, + ) + ) + assert capacity == 16 * world == {4: 64, 8: 128}[world] + popped = get_from_waiting_queue( + waiting, + capacity, + enable_attention_dp=True, + max_num_active_requests=dep.max_batch_size, + all_ranks_num_active_requests=[0] * world, + ) + assert len(popped) == 2 * world and not waiting + + router = ADPRouter.create( + dist=None, + has_seq_slot_headroom=False, + attention_dp_config=llm_args.attention_dp_config, + ) + assigned, expected = router.route_requests( + [RankState(rank=rank) for rank in range(world)], popped, dep.max_batch_size + ) + assert expected == 2 + for rank in range(world): + prompts = sorted((len(item.request.input_token_ids) for item in assigned[rank])) + assert prompts == [1, 32767] and sum(prompts) == dep.max_num_tokens + + +# ------------------------------------------------------- (4) the engine's reads of the node layout + + +@_each_world +def test_shell_lm_head_is_whole_under_attention_dp(world, dep): + """The causal-LM shell every target subclasses builds, under attention DP with lm_head TP + off, a replicated lm_head: the full 204800-token vocabulary of 6144 on every rank, at four + ranks and at eight, on a one-rank mapping with no tensor-parallel mode, so no gather and no + reduction. The node layout does not enter. Built on the meta device.""" + for rank in range(world): + config = dataclasses.replace(dep.model_config, mapping=_rank_mapping(dep.mapping, rank)) + cfg = config.pretrained_config + with torch.device("meta"): + shell = DecoderModelForCausalLM( + DecoderModel(config), + config=config, + hidden_size=cfg.hidden_size, + vocab_size=cfg.vocab_size, + ) + head = shell.lm_head + assert tuple(head.weight.shape) == (204800, 6144) + assert (head.tp_size, head.tp_mode, head.gather_output) == (1, None, False) + assert head.mapping.world_size == 1 + assert (shell.pp_rank, shell.pp_size) == (0, 1) + + +def test_collective_paths_read_the_node_layout(dep8, monkeypatch): + """Where the engine's host exchanges and the target's collectives meet the node layout at + dep8, on each rank's mapping: + + * the rank's mapping is an ``MpiTopology`` and its communicator an ``MPIDist`` (the MPI + launch leaves TLLM_DISABLE_MPI unset); every host exchange of attention DP -- the + token-count gather, the CUDA-graph batch agreement, the warmup's agreement -- runs over + MPI on ``MPIDist.tp_comm``, the group of ``mapping.tp_group``, all eight ranks over both + nodes; + * the catalog's comm entries with ``process_group=None`` (the flavor a target picks under + MPI) call the MPI-session symbols ``torch.ops.trtllm.allgather`` / ``reducescatter`` with + the group they are handed, whose NCCL communicator ``getComm`` builds for that rank set + through the MPI session: it reads no node layout itself; + * the mapping says multi-node, and the one-node IPC workspace an engine AllReduce would + open stays closed past one node's devices; no engine AllReduce is built on this path.""" + assert dep8.llm_args.allreduce_strategy == "AUTO" + monkeypatch.delenv("TLLM_DISABLE_MPI", raising=False) + for rank in range(_DEP8_RANKS): + rank_mapping = _rank_mapping(dep8.mapping, rank) + assert type(rank_mapping) is MpiTopology + assert type(Distributed.get(rank_mapping)) is MPIDist + assert rank_mapping.tp_group == list(range(_DEP8_RANKS)) + assert rank_mapping.is_multi_node() is True + assert IpcMemory(rank_mapping, 1024).open_ipc is False + + calls = [] + for name in ("allgather", "reducescatter", "allgather_pg", "reducescatter_pg"): + monkeypatch.setattr( + torch.ops.trtllm, + name, + lambda *args, name=name: calls.append((name, args[2], args[3:])) or args[0], + ) + rows = torch.zeros(4, 6144, dtype=torch.bfloat16) + group = list(range(_DEP8_RANKS)) + allgather(rows, None, group, None) + reducescatter(rows, None, group, None) + allgather(rows, None, group, "pg") + assert calls == [ + ("allgather", group, ()), + ("reducescatter", group, ()), + ("allgather_pg", group, ("pg",)), + ] + + +@pytest.mark.parametrize("chunk", [32768, 65536]) +def test_moe_autotuner_at_sixteen_local_experts(dep8, dep4, chunk): + """At eight ranks (dep8), the autotuner's tuning group is the TP group, ``(tp_size, tp_rank)``: + all eight ranks over both nodes. The NVFP4 MoE runner tunes INDEPENDENT, each rank alone, so + tuning sends nothing between the nodes. Its expert-parallel size, num_experts // + local_num_experts, is 8; with ``use_dp`` False, as the dep target passes it, that moves no + bucket: an expert call's rows map to the buckets 1..8192 and the ``tune_max_num_tokens`` bucket + exactly as at dep4, so a chunk of ``chunk`` gathered rows tunes and runs at ``chunk``. + ``use_dp`` True would deflate the rows by 8 first. The engine's own MoE token budget, + ModelConfig.moe_max_num_tokens, reads the attention-DP width (8 x 32768), and no MoE load + balancer is built.""" + for rank in range(_DEP8_RANKS): + tuner = SimpleNamespace(mapping=_rank_mapping(dep8.mapping, rank)) + assert AutoTuner._tuning_group(tuner) == (8, rank) + + cfg = dep8.model_config.pretrained_config + local_experts = cfg.n_routed_experts // dep8.mapping.moe_ep_size + ep_size = cfg.n_routed_experts // local_experts + assert (local_experts, ep_size) == (16, 8) + dep8_tuning = FP4BlockScaleMoERunner.get_tuning_config( + ep_size, tune_max_num_tokens=chunk, use_dp=False + ) + dep4_tuning = FP4BlockScaleMoERunner.get_tuning_config( + 4, tune_max_num_tokens=chunk, use_dp=False + ) + assert dep8_tuning.distributed_tuning_strategy == DistributedTuningStrategy.INDEPENDENT + (dep8_spec,) = dep8_tuning.dynamic_tensor_specs + (dep4_spec,) = dep4_tuning.dynamic_tensor_specs + assert dep8_spec.gen_tuning_buckets == dep4_spec.gen_tuning_buckets + assert max(dep8_spec.gen_tuning_buckets) == chunk + for tokens in (1, 3, 16, 17, 4097, 8192, 8193, 16384, chunk - 1, chunk): + assert dep8_spec.map_to_tuning_buckets(tokens) == dep4_spec.map_to_tuning_buckets(tokens) + assert dep8_spec.map_to_tuning_buckets(chunk) == chunk + (deflated,) = FP4BlockScaleMoERunner.get_tuning_config( + ep_size, tune_max_num_tokens=chunk, use_dp=True + ).dynamic_tensor_specs + assert deflated.map_to_tuning_buckets(chunk) == min(chunk // ep_size, 8192) + + model_config = dep8.model_config + assert model_config.moe_max_num_tokens == dep8.max_num_tokens * dep8.mapping.dp_size == 262144 + assert dep4.model_config.moe_max_num_tokens == 131072 + assert dep8.llm_args.moe_config.load_balancer is None and model_config.moe_load_balancer is None + for rank in range(_DEP8_RANKS): + with maybe_create_moe_load_balancer(model_config, _rank_mapping(dep8.mapping, rank)) as lb: + assert lb is None + + +def _executor_runtime_features(llm_args, max_num_tokens: int) -> AttentionRuntimeFeatures: + """The attention runtime features ``_create_py_executor`` builds once for the service from + the LLM arguments: chunked prefill from enable_chunked_prefill, cache reuse from the + KvCacheConfig's enable_block_reuse, no speculative drafter, chunks of max_num_tokens.""" + return AttentionRuntimeFeatures( + chunked_prefill=llm_args.enable_chunked_prefill, + cache_reuse=llm_args.kv_cache_config.enable_block_reuse, + has_speculative_draft_tokens=False, + chunk_size=max_num_tokens, + ) + + +def _first_step_md( + resolved, + max_seq_len: int | None = None, + enable_chunked_prefill: bool | None = None, + enable_block_reuse: bool | None = None, +) -> SimpleNamespace: + """The attention metadata of the first step ``resolved``'s engine hands the forward, as + ``_derive_first_step`` reads it: max_num_tokens, max_batch_size (``max_num_requests``), + max_seq_len, the runtime features the executor builds from the LLM arguments + (``_executor_runtime_features``), the pool's pages and a fresh fp8_ds_mla KV cache manager + over the checkpoint's kv_source_layers. ``max_seq_len`` overrides the engine's; + ``enable_chunked_prefill`` and ``enable_block_reuse`` override the LLM arguments' (the + latter their KvCacheConfig's) before the runtime features are built from them.""" + llm_args = resolved.llm_args + kv_cache_config = llm_args.kv_cache_config + if enable_block_reuse is not None: + kv_cache_config = kv_cache_config.model_copy( + update=dict(enable_block_reuse=enable_block_reuse) + ) + update = dict(kv_cache_config=kv_cache_config) + if enable_chunked_prefill is not None: + update.update(enable_chunked_prefill=enable_chunked_prefill) + llm_args = llm_args.model_copy(update=update) + sources = list(resolved.model_config.pretrained_config.kv_source_layers) + owners = [i for i, s in enumerate(sources) if s == i] + return SimpleNamespace( + max_num_tokens=resolved.max_num_tokens, + max_num_requests=resolved.max_batch_size, + max_seq_len=resolved.max_seq_len if max_seq_len is None else max_seq_len, + runtime_features=_executor_runtime_features(llm_args, resolved.max_num_tokens), + tokens_per_block=llm_args.kv_cache_config.tokens_per_block, + kv_cache_manager=SimpleNamespace( + latent_layout=MistralLarge4CacheManager.latent_layout, + kv_source_layers=sources, + latent_pool_row=lambda i: owners.index(sources[i]), + kv_owner_layers=owners, + ), + ) + + +@_each_world +def test_expert_call_size_is_the_worlds(world, dep, monkeypatch): + """The expert path's call -- the most gathered rows one router / routing / NVFP4-quantize / + expert call takes, and the autotuner's largest bucket -- is fixed on the first step the + engine hands over (``_derive_first_step``), from the world and the engine's chunked prefill + flag alone, as the runtime features the executor builds from the LLM arguments carry it: on + this engine (32768 tokens per rank, neither chunked prefill nor block reuse) at four ranks + all 131072 gathered rows of a 32K step on every rank, one call per MoE layer; at eight ranks + 32768 of the up to 262144, in calls of 32768, the dep8 autotuner's 32768 chunk. The same + engine with the KvCacheConfig's enable_block_reuse (the runtime features' cache_reuse) calls + the same, though a step may then attend more context KV than max_num_tokens (its held + context workspace grows from 65536 rows to 2**19). With enable_chunked_prefill, at + max_seq_len 1048576 (the 1M config) or at 32768, block reuse or not, it calls 32768 at both: + at four ranks the up to 131072 gathered rows run in four calls, one call's workspace over all + of them costing the engine about 5.6 GiB of free memory, more than the 1M config's margin + spares. The core's expert window and attention-DP group are the world's. Built on the meta + device, with stand-ins for its kernel builds.""" + assert M._EXPERT_CALL_TOKENS == {4: 131072, 8: 32768} + assert M._EXPERT_CALL_TOKENS_CHUNKED == {4: 32768, 8: 32768} + assert (dep.max_num_tokens, dep.max_seq_len, dep.max_batch_size) == (32768, 32768, 16) + assert not dep.llm_args.enable_chunked_prefill + assert not dep.llm_args.kv_cache_config.enable_block_reuse + monkeypatch.setattr( + M, "_rope_tables", lambda cfg, alpha, rows, dtype, device: SimpleNamespace(rows=rows) + ) + monkeypatch.setattr(M, "build_ix_q_side", lambda dev, is_neox: None) + monkeypatch.setattr(M, "build_ix_k_cache", lambda dev, is_neox, cache_contiguous: None) + monkeypatch.setattr( + M, "build_ix_topk_decode", lambda dev, next_n: SimpleNamespace(next_n=next_n) + ) + monkeypatch.setattr(torch.cuda, "current_device", lambda: 0) + with torch.device("meta"): + core = M.MistralLarge4DepModelingV2(dep.model_config) + assert (core.expert_call_tokens, core.expert_tune_tokens) == (0, 0) + core.build_kernels(0) + # (overrides of the engine's max_seq_len and LLM arguments, the expert call, the held + # context workspace's rows) + for engine, call, rows in ( + (dict(), {4: 131072, 8: 32768}[world], 65536), + (dict(enable_block_reuse=True), {4: 131072, 8: 32768}[world], 1 << 19), + (dict(max_seq_len=1048576, enable_chunked_prefill=True), 32768, 1 << 19), + (dict(enable_chunked_prefill=True), 32768, 1 << 19), + (dict(enable_chunked_prefill=True, enable_block_reuse=True), 32768, 1 << 19), + ): + # Each with a KV cache manager of its own: the core derives its sizes once per manager. + md = _first_step_md(dep, **engine) + features = md.runtime_features + assert features.chunked_prefill is engine.get("enable_chunked_prefill", False), engine + assert features.cache_reuse is engine.get("enable_block_reuse", False), engine + core._derive_first_step(md) + assert core.expert_call_tokens == call, engine + assert core.ctx_workspace_rows == rows, engine + assert core.expert_tune_tokens == min(call, world * dep.max_num_tokens) == call + assert (core.dp_group, core.dp_rank) == (list(range(world)), 0) + assert (core.local_experts, core.expert_offset) == (128 // world, 0) + + +@pytest.mark.parametrize( + "world_size, tp_size, pp_size", + [(2, 2, 1), (16, 16, 1), (8, 4, 2)], + ids=["world2", "world16", "world8-pp2"], +) +def test_expert_path_refuses_another_world(dep4, world_size, tp_size, pp_size): + """The core serves four ranks on one node or eight over two nodes of four, every rank in the + one attention-DP group: another world size has no expert call size, and a group short of the + world (two pipeline stages of four) is not the expert path's communicator. Both are refused + before anything is built.""" + mapping = Mapping( + world_size=world_size, + rank=0, + gpus_per_node=_GPUS_PER_NODE, + tp_size=tp_size, + pp_size=pp_size, + moe_tp_size=1, + moe_ep_size=tp_size, + enable_attention_dp=True, + ) + assert len(mapping.tp_group) == tp_size + config = dataclasses.replace(dep4.model_config, mapping=mapping) + with torch.device("meta"), pytest.raises(ValueError, match="serves 4 ranks on one node or 8"): + M.MistralLarge4DepModelingV2(config) + + +def test_kv_host_tier_is_shared_by_the_node_four_ways(): + """At eight ranks over two nodes of four (dep8), the V2 KV manager provisions a host tier per + rank (no host_cache_size is set): its device quota, capped at half the node's free physical + pages divided by the ranks on the node -- ``Distributed.local_world_size``, the size of the MPI + communicator split by host, four at dep8 -- then the minimum over all eight ranks. So a dep8 + node's four tiers together hold at most half the host memory free when the pool is built.""" + gib = 2**30 + quota = 160 * gib + assert _compute_auto_host_tier_quota(quota, 4, 800 * gib, float("inf")) == 100 * gib + assert _compute_auto_host_tier_quota(quota, 4, 2000 * gib, float("inf")) == quota + assert _compute_auto_host_tier_quota(quota, 4, float("inf"), float("inf")) == quota + + +# ------------------------------------------------------- (5) the four-rank engines, and dep8 + + +def test_tep4_and_dep4_route_as_before(tep4, dep4): + """The NVFP4 checkpoint at tep4 and at dep4 resolves and routes: tep4 as before this target, + dep4 to it, each claimed by its own target, on one node of four, the routed experts + expert-parallel four ways and not split along their width, each on the checkpoint's anchor + and the same 47136-byte token.""" + cases = ( + (tep4, False, "ModelingV2MistralLarge4Nvfp4Sm103Tep"), + (dep4, True, _DEP4_TARGET), + ) + for resolved, attention_dp, target in cases: + seen = resolved.routed[0] + mapping = seen.mapping + assert (mapping.world_size, mapping.tp_size, mapping.pp_size) == (4, 4, 1) + assert (mapping.moe_ep_size, mapping.moe_tp_size, mapping.enable_attention_dp) == ( + 4, + 1, + attention_dp, + ) + assert mapping.gpus_per_node == 4 and mapping.is_multi_node() is False + for rank in range(4): + rank_mapping = _rank_mapping(mapping, rank) + assert (rank_mapping.moe_ep_rank, rank_mapping.moe_tp_rank) == (rank, 0) + assert (rank_mapping.local_rank, rank_mapping.node_rank) == (rank, 0) + assert _route(seen.pretrained_config, mapping).__name__ == target + assert resolved.llm_args.quant_config.quant_algo == QuantAlgo.MIXED_PRECISION + assert resolved.llm_args.quant_config.kv_cache_quant_algo == QuantAlgo.FP8 + assert _anchor(resolved.llm_args) == 91.926 + assert (_kv_cost(resolved, 0).slope, _kv_cost(resolved, 0).intercept) == (47136, 0) + assert (tep4.max_num_tokens, tep4.max_seq_len, tep4.max_batch_size) == (32768, 32768, 16) + assert (dep4.max_num_tokens, dep4.max_seq_len, dep4.max_batch_size) == (32768, 32768, 16) + + +def test_dep4_differs_from_tep4_in_attention_dp_only(dep4, tep4): + """The same checkpoint at dep4 and at tep4 resolves to the same configuration -- quantization, + per-layer quantization, model config, DSA config, limits, CUDA graphs, KV config, the MoE + config, what the router reads -- except the mapping's attention-DP bit, and the tep4 branch + of the routing does not claim dep4.""" + assert _summary(dep4) == _summary(tep4) + dep4_mapping, tep4_mapping = dep4.mapping.to_dict(), tep4.mapping.to_dict() + assert {k for k in dep4_mapping if dep4_mapping[k] != tep4_mapping[k]} == { + "enable_attention_dp" + } + + target = _route_or_none(dep4.routed[0]) + assert target is None or target.__name__ != "ModelingV2MistralLarge4Nvfp4Sm103Tep" + + +def test_dep8_differs_from_dep4_in_the_world_only(dep8, dep8_gate, dep4): + """The same checkpoint at dep8 and at dep4 resolves to the same configuration -- + quantization, per-layer quantization, model config, DSA config, limits, CUDA graphs, KV + config, the MoE config, what the router reads -- except the mapping's world: its size, the TP + and attention-TP widths and the EP width (and the MoE token budget that reads them). The tep4 + branch of the routing does not claim dep8: the dep target does, from core.yaml and from the + boot gate's command alike, the same class as at four ranks.""" + assert _summary(dep8) == _summary(dep4) + dep8_mapping, dep4_mapping = dep8.mapping.to_dict(), dep4.mapping.to_dict() + assert {k for k in dep8_mapping if dep8_mapping[k] != dep4_mapping[k]} == { + "world_size", + "tp_size", + "attn_tp_size", + "moe_ep_size", + } + for resolved in (dep8, dep8_gate): + seen = resolved.routed[0] + target = _route(seen.pretrained_config, seen.mapping) + assert target.__name__ == _DEP4_TARGET diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_indexer_packed_chunks.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_indexer_packed_chunks.py new file mode 100644 index 000000000000..789cc7c4be28 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_indexer_packed_chunks.py @@ -0,0 +1,348 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The dep target's indexer selection over packed prefill chunks past max_seq_len. + +Under chunked prefill the engine plans the indexer's prefill chunks with +`plan_mla_chunked_prefill_chunks` over each context request's whole KV (cached + new), packing +requests into one chunk while (summed new tokens) x (summed KV) stays within its packed-logits +budget (1 << 28). A few short last chunks over long prefixes on one rank therefore form one +chunk with more keys than one request can hold: two 5-token chunks over 983040 cached keys each +plan as (token_start, token_end, k_token_start, k_token_end) = (0, 10, 0, 1966090). +`PrefillTarget._select_ctx` cuts such a chunk at its request boundaries (`_prefill_blocks`), at +any engine: each request's rows are scored against its own keys only, gathered at that +request's `k_token_start` in the step's full-KV mappings, its rows' windows rebased to it, in +query tiles under the engine's logits budget (`_prefill_logits_budget(max_num_tokens, +max_seq_len)`, the ceiling 2**26 at max_num_tokens 32768 and max_seq_len 65536 or 1048576). A +chunk of at most max_seq_len keys is scored whole. + +Each case builds its step as the engine does: `Indexer.prepare_for_update_k_cache` and +`Indexer.prepare_for_chunked_prefill` on DSA metadata holding what they read (each context +request's cached and new tokens, its pages in an indexer key pool of random e4m3 keys and fp32 +scales), and the target's own `_ctx_kv_offsets` and `_ctx_host_offsets` projections of it. The +target's `_select_ctx` then runs on it, with every gather, logits and top-k call recorded, and +on each request's own step, that request alone. Every call must stay at or below max_seq_len +keys, and each row's selection must be the one it gets alone, compared as sets: keys tied at +the 2048th score may be kept differently by two calls, so a key in one selection and not the +other must score exactly the row's 2048th. + +The cases: case (d) itself; a ~1M request's 100-token last chunk packed with a fresh 155-token +prompt (1048655 keys); a scaled analogue, four requests whose summed 110330 keys pass a +max_seq_len of 65536; and, not cut, sixteen tails packed to exactly 1048576 keys. + +One GB300 (sm_103, where the indexer kernels are certified); `_prefill_blocks` alone on CPU. +""" + +from __future__ import annotations + +import itertools +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch.attention.backends.sparse.dsa.indexer import ( + Indexer, + _packed_logits_elem_budget, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_103__dep import ( # noqa: E501 + modeling as M, +) +from tensorrt_llm._utils import get_sm_version + +MAX_SEQ_LEN = 1048576 +MAX_NUM_TOKENS = 32768 +TOP_K = 2048 +HEADS = 32 +HEAD_DIM = 128 +TOKENS_PER_BLOCK = 32 +# One indexer key page: 32 keys of 128 e4m3 codes, then their 32 fp32 scales. +PAGE_BYTES = TOKENS_PER_BLOCK * (HEAD_DIM + 4) +# A selection slot no call wrote. +UNWRITTEN = -7 + +gpu = pytest.mark.skipif( + not torch.cuda.is_available() or get_sm_version() != 103, + reason="the mistral_large4 modeling_v2 targets are certified on sm_103 only", +) + +#: name -> (max_seq_len, [(cached, new)] per context request, the engine's chunks, the blocks +#: `_select_ctx` scores, as (token_start, token_end, k_token_start, n_k)). +CASES = { + "case_d": ( + MAX_SEQ_LEN, + [(983_040, 5), (983_040, 5)], + [(0, 10, 0, 1_966_090)], + [(0, 5, 0, 983_045), (5, 10, 983_045, 983_045)], + ), + "tail_and_fresh": ( + MAX_SEQ_LEN, + [(1_048_400, 100), (0, 155)], + [(0, 255, 0, 1_048_655)], + [(0, 100, 0, 1_048_500), (100, 255, 1_048_500, 155)], + ), + "scaled": ( + 65536, + [(40_000, 7), (50_000, 3), (0, 300), (20_000, 20)], + [(0, 330, 0, 110_330)], + [ + (0, 7, 0, 40_007), + (7, 10, 40_007, 50_003), + (10, 310, 90_010, 300), + (310, 330, 90_310, 20_020), + ], + ), # fmt: skip + "sixteen_tails_at_max": ( + MAX_SEQ_LEN, + [(65_520, 16)] * 16, + [(0, 256, 0, MAX_SEQ_LEN)], + [(0, 256, 0, MAX_SEQ_LEN)], + ), +} + + +def _pool(pages: int, gen: torch.Generator) -> torch.Tensor: + """An indexer key pool `[pages, 32, 1, 132]` (the V2 view's layout) of random keys.""" + pool = torch.empty(pages, PAGE_BYTES, dtype=torch.uint8, device="cuda") + codes_end = TOKENS_PER_BLOCK * HEAD_DIM + for lo in range(0, pages, 8192): + hi = min(lo + 8192, pages) + codes = torch.randn(hi - lo, codes_end, generator=gen, device="cuda") + pool[lo:hi, :codes_end] = codes.to(torch.float8_e4m3fn).view(torch.uint8) + scales = torch.rand(hi - lo, TOKENS_PER_BLOCK, generator=gen, device="cuda") + 0.5 + pool[lo:hi, codes_end:] = scales.view(torch.uint8) + return pool.view(pages, TOKENS_PER_BLOCK, 1, HEAD_DIM + 4) + + +def _step(contexts, table, max_seq_len: int) -> dict: + """`Indexer.prepare_for_update_k_cache` and `prepare_for_chunked_prefill` on one dep4 rank's + step of context requests `contexts` ((cached, new) each, its pages on its row of `table`), + then the target's step state over it, as `PrefillTarget.step_args` hands it over: the + engine's chunks and full-KV mappings (the step's own slot mappings when no request has a + cached prefix), `_ctx_kv_offsets` and `_ctx_host_offsets`.""" + num_contexts = len(contexts) + new = [n for _, n in contexts] + cached = [c for c, _ in contexts] + cu_new = [0, *itertools.accumulate(new)] + cu_cached = [0, *itertools.accumulate(cached)] + cu_kv = [0, *itertools.accumulate(c + n for c, n in contexts)] + i64 = dict(dtype=torch.int64) + md = SimpleNamespace( + mapping=SimpleNamespace(enable_attention_dp=True, tp_size=4, tp_rank=0), + enable_context_mla_with_cached_kv=True, + runtime_features=SimpleNamespace(chunked_prefill=True), + sparse_metadata_params=SimpleNamespace(q_split_threshold=8192), + compress_ratios=[1], + max_num_tokens=MAX_NUM_TOKENS, + kv_cache_manager=SimpleNamespace( + max_seq_len=max_seq_len, index_head_dim=HEAD_DIM, tokens_per_block=TOKENS_PER_BLOCK + ), + indexer_head_dim=HEAD_DIM, + indexer_quant_block_size=HEAD_DIM, + _tokens_per_block=TOKENS_PER_BLOCK, + num_contexts=num_contexts, + num_generations=0, + num_ctx_tokens=sum(new), + request_ids=list(range(num_contexts)), + kv_cache_params=SimpleNamespace(num_cached_tokens_per_seq=cached), + seq_lens=torch.tensor(new, dtype=torch.int32), + host_ctx_cached_token_indptr=torch.tensor(cu_cached, **i64), + host_ctx_uncached_token_indptr=torch.tensor(cu_new, **i64), + host_ctx_kv_indptr=torch.tensor(cu_kv, **i64), + ctx_cached_token_indptr=torch.tensor(cu_cached, **i64, device="cuda"), + ctx_kv_indptr=torch.tensor(cu_kv, **i64, device="cuda"), + max_ctx_seq_len=max(new), + num_ctx_cached_tokens=sum(cached), + cu_seqlen_ks=torch.empty(MAX_NUM_TOKENS, dtype=torch.int32, device="cuda"), + skip_indexer_for_ctx_reqs=False, + slot_mapping_fp8=torch.empty(MAX_NUM_TOKENS, **i64, device="cuda"), + slot_mapping_scale=torch.empty(MAX_NUM_TOKENS, **i64, device="cuda"), + host_slot_mapping_fp8=torch.empty(MAX_NUM_TOKENS, **i64), + host_slot_mapping_scale=torch.empty(MAX_NUM_TOKENS, **i64), + host_indexer_k_cache_block_offsets=table, + indexer_prefill_q_split="unset", + indexer_prefill_chunks=None, + ) + params = Indexer.build_indexer_params(md) + Indexer.prepare_for_update_k_cache(md, params) + Indexer.prepare_for_chunked_prefill(md, params) + assert md.indexer_prefill_q_split is None + assert _packed_logits_elem_budget(md) == 1 << 28 + return dict( + indexer_prefill_chunks=md.indexer_prefill_chunks, + slot_mapping_fp8_fullkv=md.slot_mapping_fp8_fullkv, + slot_mapping_scale_fullkv=md.slot_mapping_scale_fullkv, + num_ctx_kv_tokens=cu_kv[-1], + cu_seqlen_ks=None, + cu_seqlen_ke=None, + **M._ctx_kv_offsets(md, num_contexts), + **M._ctx_host_offsets(md, num_contexts), + ) + + +def _budget(max_seq_len: int) -> int: + """The logits budget `_derive_first_step` gives the core at max_num_tokens 32768 and + `max_seq_len`.""" + return M._prefill_logits_budget(MAX_NUM_TOKENS, max_seq_len) + + +def _select(step, max_seq_len: int, pool, q, weights) -> torch.Tensor: + """The target's `_select_ctx` over every context row of `step`, on a prefill target holding + just what it reads off its core at max_num_tokens 32768 and `max_seq_len`.""" + target = M.PrefillTarget.__new__(M.PrefillTarget) + target.core = SimpleNamespace( + idx_dim=HEAD_DIM, + idx_topk=TOP_K, + _max_ctx_len=max_seq_len, + prefill_logits_budget=_budget(max_seq_len), + ) + rows = q.shape[0] + out = torch.full([rows, TOP_K], UNWRITTEN, dtype=torch.int32, device="cuda") + target._select_ctx(step, pool, q, weights, out, rows) + return out + + +def _recorded(monkeypatch) -> list[tuple]: + """Record every gather, logits and top-k call the target makes, then make it.""" + calls: list[tuple] = [] + gather = M.indexer_k_cache_gather_op + logits = M.fp8_mqa_logits + topk = M.cute_dsl_gvr_topk_prefill + + def gather_rec(k_pool, fp8, scale, k_token_start, num_tokens, head_dim): + calls.append(("gather", k_token_start, num_tokens)) + return gather(k_pool, fp8, scale, k_token_start, num_tokens, head_dim) + + def logits_rec(q, kv, weights, ks, ke, schedule_meta=None): + assert schedule_meta is not None, "a tile scored without its split-KV schedule" + calls.append(("logits", q.shape[0], kv[0].shape[0])) + return logits(q, kv, weights, ks, ke, schedule_meta=schedule_meta) + + def topk_rec(values, ks, ke, out, top_k, workspace=None): + calls.append(("topk", values.shape[0], values.shape[1])) + return topk(values, ks, ke, out, top_k, workspace) + + monkeypatch.setattr(M, "indexer_k_cache_gather_op", gather_rec) + monkeypatch.setattr(M, "fp8_mqa_logits", logits_rec) + monkeypatch.setattr(M, "cute_dsl_gvr_topk_prefill", topk_rec) + return calls + + +def _alone(context, row, max_seq_len: int, pool, q, weights): + """One request scored in a step of its own: its selection, and its rows' logits and + windows to judge a tie by.""" + step = _step([context], row, max_seq_len) + sel = _select(step, max_seq_len, pool, q, weights) + (chunk,) = step["indexer_prefill_chunks"] + keys, scales = M.indexer_k_cache_gather_op( + pool, step["slot_mapping_fp8_fullkv"], step["slot_mapping_scale_fullkv"], 0, + chunk.k_token_end, HEAD_DIM, + ) # fmt: skip + ks, ke = chunk.cu_seqlen_ks, chunk.cu_seqlen_ke + scores = M.fp8_mqa_logits(q, (keys, torch.reshape(scales, [-1])), weights, ks, ke) + return sel, scores, ks.tolist(), ke.tolist() + + +def _compare(where, got, sel, scores, ks, ke) -> tuple[int, int]: + """A request's rows of the packed step's selection `got` against its own `sel`, as sets; a + key in one and not the other must score exactly the row's 2048th. Returns (rows the same + set, rows that differ on such a tie).""" + same = tied = 0 + for i in range(got.shape[0]): + a, b = set(got[i].tolist()), set(sel[i].tolist()) + assert UNWRITTEN not in a and UNWRITTEN not in b, (where, i, "a row was not written") + if a == b: + same += 1 + continue + width = ke[i] - ks[i] + assert width > TOP_K, (where, i, width, sorted(a ^ b)[:8]) + row = scores[i, ks[i] : ke[i]] + kth = torch.topk(row, TOP_K).values[-1] + off = [p for p in a ^ b if p < 0 or p >= width or row[p] != kth] + assert not off, (where, i, "differs off the 2048th score", off[:8], float(kth)) + tied += 1 + return same, tied + + +def test_prefill_blocks_cut_only_past_max_seq_len(): + """`_prefill_blocks` on host values: a chunk within max_seq_len is one block, itself; past + it, one block per request, each its own rows over its own keys.""" + packed = SimpleNamespace(token_start=0, token_end=10, k_token_start=0, k_token_end=1_966_090) + cu_new, cu_kv = [0, 5, 10], [0, 983_045, 1_966_090] + assert M._prefill_blocks(packed, cu_new, cu_kv, MAX_SEQ_LEN) == [ + (0, 5, 0, 983_045), + (5, 10, 983_045, 983_045), + ] + assert M._prefill_blocks(packed, cu_new, cu_kv, 1_966_090) == [(0, 10, 0, 1_966_090)] + # A packed chunk behind an earlier context request: its rows and keys start past that one's. + behind = SimpleNamespace( + token_start=32_000, token_end=32_010, k_token_start=500_000, k_token_end=2_466_090 + ) + cu_new, cu_kv = [0, 32_000, 32_005, 32_010], [0, 500_000, 1_483_045, 2_466_090] + assert M._prefill_blocks(behind, cu_new, cu_kv, MAX_SEQ_LEN) == [ + (32_000, 32_005, 500_000, 983_045), + (32_005, 32_010, 1_483_045, 983_045), + ] + # One request's 32768-row last chunk over its whole 1M KV is never cut. + last = SimpleNamespace(token_start=0, token_end=32768, k_token_start=0, k_token_end=MAX_SEQ_LEN) + assert M._prefill_blocks(last, [0, 32768], [0, MAX_SEQ_LEN], MAX_SEQ_LEN) == [ + (0, 32768, 0, MAX_SEQ_LEN) + ] + + +@gpu +@pytest.mark.parametrize("name", list(CASES)) +def test_packed_chunk_selections_are_each_requests_own(name, monkeypatch): + """The engine's chunk plan of the case, the blocks the target scores, every call at or below + max_seq_len keys, and every request's rows selecting as that request alone does.""" + max_seq_len, contexts, chunks, blocks = CASES[name] + gen = torch.Generator(device="cuda").manual_seed(len(name)) + kv = [c + n for c, n in contexts] + pages = [-(-k // TOKENS_PER_BLOCK) for k in kv] + # Each request's pages, scattered over the pool. + order = torch.randperm(sum(pages), generator=torch.Generator().manual_seed(7)) + table = torch.zeros(len(contexts), max(pages), dtype=torch.int32) + for r, (lo, n) in enumerate(zip(itertools.accumulate([0, *pages]), pages)): + table[r, :n] = order[lo : lo + n].to(torch.int32) + pool = _pool(sum(pages), gen) + rows = sum(n for _, n in contexts) + q = torch.randn(rows, HEADS, HEAD_DIM, generator=gen, device="cuda") * 0.5 + q = q.to(torch.float8_e4m3fn) + weights = torch.randn(rows, HEADS, generator=gen, device="cuda") + + step = _step(contexts, table, max_seq_len) + planned = [ + (c.token_start, c.token_end, c.k_token_start, c.k_token_end) + for c in step["indexer_prefill_chunks"] + ] + assert planned == chunks, planned + assert step["host_cu_ctx_kv"][-1] == step["slot_mapping_fp8_fullkv"].shape[0] == sum(kv) + + calls = _recorded(monkeypatch) + got = _select(step, max_seq_len, pool, q, weights) + torch.cuda.synchronize() + monkeypatch.undo() + print(f"PACKED {name}: max_seq_len {max_seq_len}, contexts {contexts}, chunks {planned}") + print(f"PACKED {name}: calls {calls}") + + # One gather per block at its own keys; then its rows in tiles of the engine's logits + # budget. + expected = [] + for lo, hi, k_start, n_k in blocks: + expected.append(("gather", k_start, n_k)) + tile = max(1, _budget(max_seq_len) // n_k) + for t in range(lo, hi, tile): + expected += [("logits", min(tile, hi - t), n_k), ("topk", min(tile, hi - t), n_k)] + assert calls == expected, calls + assert all(n_k <= max_seq_len for _, _, n_k in calls), calls + + cu_new = step["host_cu_ctx_new"] + for r, context in enumerate(contexts): + mine = slice(cu_new[r], cu_new[r + 1]) + sel, scores, ks, ke = _alone(context, table[r : r + 1], max_seq_len, pool, q[mine], + weights[mine]) # fmt: skip + same, tied = _compare((name, r), got[mine], sel, scores, ks, ke) + print( + f"PACKED {name} request {r} {context}: {same} of {len(ks)} rows the same set as " + f"alone, {tied} differ on a tie at the 2048th score; bitwise equal " + f"{torch.equal(got[mine], sel)}" + ) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_kv_b_regroup.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_kv_b_regroup.py new file mode 100644 index 000000000000..6680e9a443b5 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_kv_b_regroup.py @@ -0,0 +1,136 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The 64-head `wkv_b` regroup of the dep4 weight table, on the checkpoint's own tensors. + +Under attention DP every rank holds `wkv_b` whole, all 64 heads, and the table regroups it +from the checkpoint's per-head `[k_nope | v]` rows into `[all heads' k_nope ; all heads' v]`, +the K-B and V-B operands of the absorbed attention. A regroup at the tep4 count of 16 heads, +applied to the whole tensor, has exactly the same shape -- 12288 rows on a RoPE layer, 16384 +on a NoPE layer -- and puts most rows in the wrong place, so no shape check sees it. + +On one RoPE layer (0) and one NoPE layer (3) of Mistral-Large-4-1T-A52B-NVFP4, the table's +own manifest row (its source key, destination and transform) is applied on every rank -- of +four ranks on one node, and of eight over two nodes of four (dep8) -- to the stored tensor and +compared, bit for bit, with an independent per-head split of the same tensor; and the 16-head +regroup is shown to differ from it. No GPU. Skipped where the checkpoint is absent (looked up +as `mistral-large-4-nvfp4` or `Mistral-Large-4-1T-A52B-NVFP4` under `LLM_MODELS_ROOT`). +""" + +from __future__ import annotations + +import json +from pathlib import Path +from types import SimpleNamespace + +import pytest +import torch +from safetensors import safe_open +from test_common.llm_data import llm_models_root + +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_103__dep import ( # noqa: E501 + weights as W, +) +from tensorrt_llm._torch.models.checkpoints.mistral.config_loader import MistralConfigLoader +from tensorrt_llm.mapping import Mapping + +pytestmark = pytest.mark.cpu_only + +#: The attention-DP worlds the target serves: four ranks on one node, eight over two. +WORLDS = (4, 8) +#: GB300 nodes here hold four devices. +GPUS_PER_NODE = 4 +#: The tep target's ranks: its 16 heads per rank are the regroup shown wrong here. +TEP4_RANKS = 4 +CHECKPOINTS = ("mistral-large-4-nvfp4", "Mistral-Large-4-1T-A52B-NVFP4") +#: (layer, k_nope width): a RoPE layer and a NoPE layer, whose nope spends the rope dims too. +LAYERS = ((0, 64), (3, 128)) + + +@pytest.fixture(scope="module") +def checkpoint(): + root = llm_models_root() + for name in CHECKPOINTS: + path = None if root is None else Path(root) / name + if path is not None and all((path / f).is_file() for f in ("params.json", W.SHARD_INDEX)): + # The configuration the engine adapts from the checkpoint's params.json. + config = MistralConfigLoader().load_pretrained_config(str(path)) + weight_map = json.loads((path / W.SHARD_INDEX).read_text())["weight_map"] + return SimpleNamespace(path=path, config=config, weight_map=weight_map) + pytest.skip(f"checkpoint {CHECKPOINTS} is not under LLM_MODELS_ROOT ({root})") + + +def _core(config, rank: int, world: int) -> SimpleNamespace: + mapping = Mapping( + world_size=world, + rank=rank, + gpus_per_node=GPUS_PER_NODE, + tp_size=world, + moe_tp_size=1, + moe_ep_size=world, + enable_attention_dp=True, + ) + assert (mapping.tp_rank, mapping.moe_ep_rank, mapping.moe_tp_rank) == (rank, rank, 0) + return SimpleNamespace( + model_config=SimpleNamespace( + pretrained_config=config, + mapping=mapping, + torch_dtype=torch.bfloat16, + ) + ) + + +def _stored(checkpoint, key: str) -> torch.Tensor: + with safe_open( + str(checkpoint.path / checkpoint.weight_map[key]), framework="pt", device="cpu" + ) as handle: + return handle.get_tensor(key) + + +def _per_head_split(t: torch.Tensor, heads: int, nope: int, v: int) -> torch.Tensor: + """Head by head, off the stored rows: head h owns rows `h * (nope + v)` to + `(h + 1) * (nope + v)`, k_nope first. All heads' k_nope, then all heads' v.""" + k_rows, v_rows = [], [] + for h in range(heads): + first = h * (nope + v) + k_rows.append(t[first : first + nope]) + v_rows.append(t[first + nope : first + nope + v]) + return torch.cat(k_rows + v_rows, dim=0) + + +@pytest.mark.parametrize("world", WORLDS, ids=lambda w: f"world{w}") +@pytest.mark.parametrize("layer, nope", LAYERS, ids=["rope-layer0", "nope-layer3"]) +def test_wkv_b_regroup_is_the_per_head_split_of_all_64_heads(checkpoint, layer, nope, world): + cfg = checkpoint.config + heads, v = cfg.num_attention_heads, cfg.v_head_dim + assert heads == 64 and (cfg.qk_nope_head_dim if layer % 4 != 3 else 128) == nope + key = f"layers.{layer}.attention.wkv_b.weight" + stored = _stored(checkpoint, key) + assert tuple(stored.shape) == (heads * (nope + v), cfg.kv_lora_rank), stored.shape + expected = _per_head_split(stored, heads, nope, v) + + for rank in range(world): + core = _core(cfg, rank, world) + d = W.MODEL_WEIGHTS.dims(core) + rows = W.MODEL_WEIGHTS.manifest(core)[f"l{layer}_wkv_b"] + assert len(rows) == 1, rows + ckpt, index, transform = rows[0] + assert (ckpt, index) == (key, None), (ckpt, index) + got = transform(stored.contiguous(), core) + # Two rows share the name over disjoint layer sets: the RoPE and the NoPE layers'. + declared = next( + r for r in W.MODEL_WEIGHTS.WEIGHTS if r.name == "wkv_b" and layer in r.layer_indices(d) + ).shape(d) + assert tuple(got.shape) == tuple(declared) == tuple(expected.shape) + assert torch.equal(got, expected), f"rank {rank}: the regroup is not the per-head split" + + # The tep4 head count on the whole tensor: each "head" is then four real heads' rows, split + # into a 4 * (nope + v) - v block and a v block. The same shape, other rows. + t3 = stored.reshape(heads // TEP4_RANKS, -1, stored.shape[1]) + nope16 = t3.shape[1] - v + tep4 = torch.cat( + [t3[:, :nope16].reshape(-1, t3.shape[2]), t3[:, nope16:].reshape(-1, t3.shape[2])] + ) + assert tep4.shape == expected.shape + moved = (tep4 != expected).any(dim=1).float().mean().item() + print(f"layer {layer}: a 16-head regroup moves {moved:.1%} of the {len(expected)} rows") + assert moved > 0.5, moved diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_long_context.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_long_context.py new file mode 100644 index 000000000000..5a26763f2648 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_long_context.py @@ -0,0 +1,270 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The long-context witness of Mistral-Large-4-1T-A52B-NVFP4 on its sm_103 dep target. + +GSM8K's prompts never pass the indexer's `index_topk` (2048 positions). Below it the engine +skips the indexer and sparse MLA attends over the causal window, so the accuracy gate never +runs the indexer's selection, nor SSMax, LeRoPE and YaRN at long positions. Here the whole +model runs under its serving config -- per rank, max_seq_len = max_num_tokens = 32768, +chunked prefill off, the fp8_ds_mla latent KV cache at kv_cache_config's default dtype, CUDA +graphs over batches 1, 2, 4, 8 and 16, padded -- attention data parallel over the ranks of the +world, four on one node or eight on two nodes of four (dep8), and greedy decoding must recall a +passkey hidden in plain filler text: + +* in prompts of about 4K, 16K and 31K tokens, one prompt at a time: one rank prefills it in one + context step while the other ranks step beside it on the engine's one-token dummy, so the + experts gather the world size times the prompt's rows -- up to 4 x 31K in one call per layer + at four ranks, up to 8 x 31K in calls of 32768 rows at eight; +* in one ~31K-token prompt per rank admitted together (read back from each rank's own + iteration statistics), so every rank prefills a ~31K prompt in the same step and the experts + see all of those prefills at once -- about 127K gathered rows at four ranks, about 254K + through every expert call of every MoE layer at eight -- then every rank decodes. + +Text only: no image, no chat template. Two engine options beyond the serving ones make the +steps observable and do not change what a step computes: iteration statistics, which under +attention DP carry one record per rank, and a request queue that waits up to `BATCH_WAIT_MS` +for a full batch so the prompts are admitted in one step rather than split by their arrival. + +Runs with the ranks already in place: pytest under `trtllm-llmapi-launch`, as the whole-model +accuracy class runs, with the checkpoint under `LLM_MODELS_ROOT` as +`Mistral-Large-4-1T-A52B-NVFP4` (or `mistral-large-4-nvfp4`) as published, +Mistral-native (params.json, tekken.json and the consolidated safetensors, which the engine loads +with checkpoint_format "mistral") -- on four GB300s for the four-rank engine, on eight +over two nodes of four for the eight-rank one. Each world runs only in a launch of its own size +(or unlaunched): the four-rank engine where four GPUs are visible, the eight-rank one where eight +ranks are, counted on the launched MPI world rather than the devices a node shows (each node sees +four). Skipped where the checkpoint is absent, below the world's ranks, or off sm_103. +""" + +from __future__ import annotations + +import collections +import random +from pathlib import Path + +import pytest +import torch +from test_common.llm_data import llm_models_root +from utils.util import skip_num_gpus_less_than + +from tensorrt_llm import LLM, SamplingParams +from tensorrt_llm._torch.modeling_v2 import MODELING_V2_VALIDATE_ENV +from tensorrt_llm._utils import get_sm_version +from tensorrt_llm.inputs import TokensPrompt +from tensorrt_llm.llmapi import CudaGraphConfig, KvCacheConfig +from tensorrt_llm.llmapi.mpi_session import get_mpi_world_size + + +def _ranks() -> int: + """The ranks this process can serve: the MPI world a launcher started (eight over two nodes + of four, where each node sees four devices), or, unlaunched, the devices the LLM API spawns + its workers on -- as the integration conftest's `skip_less_mpi_world_size` counts.""" + world = get_mpi_world_size() + return torch.cuda.device_count() if world == 1 else world + + +def _launched_as(world: int): + """An engine of `world` ranks starts in no launch of another size.""" + launched = get_mpi_world_size() + return pytest.mark.skipif( + launched not in (1, world), + reason=f"the {world}-rank engine does not start in a launch of {launched} ranks", + ) + + +#: The worlds the target serves, each gated on its own ranks: four ranks on one node, eight +#: over two nodes of four. +WORLDS = [ + pytest.param(4, marks=[skip_num_gpus_less_than(4), _launched_as(4)], id="world4"), + pytest.param( + 8, + marks=[ + pytest.mark.skipif(_ranks() < 8, reason="the dep8 configuration needs eight ranks"), + _launched_as(8), + ], + id="world8", + ), +] + +pytestmark = [ + pytest.mark.skipif( + get_sm_version() != 103, + reason="the mistral_large4 modeling_v2 targets are certified on sm_103 only", + ), + # One engine serves each world: the LLM API starts its result-dispatch thread on the first + # request, inside the first test, and stops it when the fixture shuts the engine down. + pytest.mark.threadleak(enabled=False), +] + +CHECKPOINTS = ("Mistral-Large-4-1T-A52B-NVFP4", "mistral-large-4-nvfp4") + +# The serving config: dep4 (dep8 at eight ranks), sixteen sequences and unchunked steps of up +# to 32768 tokens per rank. +MAX_NUM_TOKENS = 32768 + + +def _engine(world: int) -> dict: + return dict( + tensor_parallel_size=world, + moe_expert_parallel_size=world, + enable_attention_dp=True, + max_batch_size=16, + max_num_tokens=MAX_NUM_TOKENS, + max_seq_len=MAX_NUM_TOKENS, + enable_chunked_prefill=False, + cuda_graph_config=CudaGraphConfig(max_batch_size=16, enable_padding=True), + kv_cache_config=KvCacheConfig(enable_block_reuse=False, free_gpu_memory_fraction=0.85), + ) + + +# How long the request queue holds an iteration for more requests while fewer than a full +# batch are waiting; every iteration pays it, decode steps included. The prompts admitted +# together are submitted as token ids and arrive within milliseconds of each other. +BATCH_WAIT_MS = 1000 + +INDEX_TOPK = 2048 +# Prompts of about 4K, 16K and 31K tokens, a passkey in the middle of each. +LENGTHS = (4096, 16384, 31744) +# One ~31K prompt per rank, the needle at a different depth in each. +TOGETHER = 31744 +TOGETHER_DEPTHS = { + 4: (0.1, 0.4, 0.6, 0.9), + 8: (0.05, 0.15, 0.3, 0.4, 0.55, 0.7, 0.8, 0.95), +} +# A prompt "about 31K" when it lands within 3% of TOGETHER. +TOGETHER_FILL = 0.97 + +HEAD = ( + "There is an important piece of information hidden inside a lot of irrelevant text. " + "Find it and memorize it. I will quiz you about it.\n\n" +) +FILLER = ( + "The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again. " +) +QUESTION = "\nWhat is the pass key? The pass key is" +# A five-digit key and its closing punctuation fit with room to spare. +ANSWER = SamplingParams(max_tokens=12, temperature=0.0) + + +def _checkpoint_dir() -> Path | None: + """The checkpoint's directory, found by its Mistral-native config, params.json.""" + root = llm_models_root() + if root is None: + return None + for name in CHECKPOINTS: + if (Path(root) / name / "params.json").is_file(): + return Path(root) / name + return None + + +def _prompt(reps: int, depth: float, key: int) -> str: + before = round(reps * depth) + needle = f"\nThe pass key is {key}. Remember it. {key} is the pass key.\n" + return HEAD + FILLER * before + needle + FILLER * (reps - before) + QUESTION + + +def _passkey_ids(tokenizer, tokens: int, depth: float, key: int) -> list[int]: + """The prompt's token ids, BOS included as the LLM API's own tokenization adds it, at most + `tokens` long and within one filler sentence of it.""" + unit = len(tokenizer.encode(FILLER * 64)) / 64 + bare = len(tokenizer.encode(_prompt(0, depth, key))) + reps = max(0, int((tokens - bare) / unit)) + ids = tokenizer.encode(_prompt(reps, depth, key)) + while len(ids) > tokens and reps > 0: + reps -= 1 + ids = tokenizer.encode(_prompt(reps, depth, key)) + return ids + + +def _keys(count: int, seed: int) -> list[int]: + return random.Random(seed).sample(range(10000, 100000), count) + + +def _context_steps(llm: LLM) -> dict[int, dict[int, tuple[int, int]]]: + """Since the last call, every iteration that ran context rows on some rank: + `{iteration: {attention-DP rank: (context requests, context tokens)}}`, one entry per rank + whose own record that iteration carries context rows.""" + steps: dict[int, dict[int, tuple[int, int]]] = collections.defaultdict(dict) + for record in llm.get_stats(timeout=2): + ifb = record.get("inflightBatchingStats") or {} + if ifb.get("numContextRequests", 0): + rank = int(record.get("attentionDpRank", 0)) + steps[int(record["iter"])][rank] = (ifb["numContextRequests"], ifb["numCtxTokens"]) + return dict(steps) + + +@pytest.fixture(scope="module", params=WORLDS) +def world(request) -> int: + return request.param + + +@pytest.fixture(scope="module") +def llm(world): + path = _checkpoint_dir() + if path is None: + pytest.skip(f"checkpoint {CHECKPOINTS} is not under LLM_MODELS_ROOT ({llm_models_root()})") + with LLM( + str(path), + checkpoint_format="mistral", + modeling_v2="require", + env_overrides={MODELING_V2_VALIDATE_ENV: "1"}, + enable_iter_perf_stats=True, + batch_wait_timeout_ms=BATCH_WAIT_MS, + **_engine(world), + ) as engine: + capacity = engine.get_kv_cache_capacity() + print(f"LONGCTX world {world} kv_cache_capacity {capacity}", flush=True) + _context_steps(engine) + yield engine + + +@pytest.mark.parametrize("tokens", LENGTHS) +def test_passkey_alone_in_one_context_step(llm, tokens): + """A passkey in the middle of a prompt of about `tokens` tokens, prefilled by one rank in + one context step while the others step on their dummies, recalled by greedy decoding.""" + (key,) = _keys(1, seed=tokens) + ids = _passkey_ids(llm.tokenizer, tokens, 0.5, key) + assert INDEX_TOPK < len(ids) <= tokens, (tokens, len(ids)) + answer = llm.generate(TokensPrompt(prompt_token_ids=ids), ANSWER).outputs[0].text + steps = _context_steps(llm) + print( + f"LONGCTX alone tokens={len(ids)} key={key} answer={answer!r} context_steps={steps}", + flush=True, + ) + assert [list(by_rank.values()) for by_rank in steps.values()] == [[(1, len(ids))]], steps + assert str(key) in answer, (len(ids), key, answer) + + +def test_31k_prompts_one_per_rank_in_one_step(llm, world): + """One ~31K-token passkey prompt per rank -- four at four ranks, eight at eight -- admitted + together: each rank prefills one of them in the same context step, so the experts see every + rank's ~31K prefill at once, and every one recalls its key.""" + keys = _keys(world, seed=TOGETHER) + prompts = [ + _passkey_ids(llm.tokenizer, TOGETHER, depth, key) + for depth, key in zip(TOGETHER_DEPTHS[world], keys) + ] + lengths = [len(ids) for ids in prompts] + assert len(lengths) == world + assert all(TOGETHER_FILL * TOGETHER <= n <= TOGETHER for n in lengths), lengths + + outputs = llm.generate([TokensPrompt(prompt_token_ids=ids) for ids in prompts], ANSWER) + steps = _context_steps(llm) + answers = [out.outputs[0].text for out in outputs] + for n, key, answer in zip(lengths, keys, answers): + print(f"LONGCTX together tokens={n} key={key} answer={answer!r}", flush=True) + print(f"LONGCTX together context_steps={steps}", flush=True) + + # One iteration, every rank one ~31K prompt: the per-rank token counts of that step. + assert len(steps) == 1, steps + ((iteration, by_rank),) = steps.items() + assert sorted(by_rank) == list(range(world)), by_rank + assert all(requests == 1 for requests, _ in by_rank.values()), by_rank + assert sorted(tokens for _, tokens in by_rank.values()) == sorted(lengths), (by_rank, lengths) + print( + f"LONGCTX together iteration {iteration}: per-rank context tokens " + f"{[by_rank[r][1] for r in range(world)]}", + flush=True, + ) + misses = [(n, key, a) for n, key, a in zip(lengths, keys, answers) if str(key) not in a] + assert not misses, misses diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_moe_round_trip.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_moe_round_trip.py new file mode 100644 index 000000000000..10de317a3458 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_moe_round_trip.py @@ -0,0 +1,842 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""One real MoE layer's attention-DP round trip: the dep target's `_moe`, on four GPUs of one +node and on eight ranks over two nodes of four (dep8). + +Under attention DP the ranks hold different tokens, in different numbers, and may run +different targets in the same step -- one prefilling, another decoding, another holding only +the engine's one-token dummy. Each MoE layer pads its rows to the largest rank's count, gathers +every rank's rows before the router, runs this rank's expert window over them (32 experts at +four ranks, 16 at eight) in calls of at most `_EXPERT_CALL_TOKENS[world]` rows (131072 at four, +32768 at eight; on an engine with chunked prefill, as the 1M config at max_seq_len 1048576 runs, +`_EXPERT_CALL_TOKENS_CHUNKED[world]`, 32768 at both), and reduce-scatters the partial sums +back. A padding, chunking or ordering defect there raises nothing: the collectives pair by +position and move equal byte counts, and every rank simply gets other rows. + +So, on layer 1 of Mistral-Large-4-1T-A52B-NVFP4 itself (its router, its shared expert, each +rank's own window of its routed experts, laid out by the dep4 weight table's own manifest rows), +every rank runs the target's own `PrefillTarget._moe` or `DecodeTarget._moe` on its own rows, +and every rank's result is compared with the same layer computed on one rank over all ranks' +tokens concatenated: the shared expert plus every window's partial sum, added in fp32. + +At four ranks, in each table's expert calls (the 32K config's and the 1M config's): + +* unequal ranks: 1003 / 1 / 16 / 257 rows, ranks 0 and 3 prefilling, rank 1 decoding the + one-row dummy and rank 2 sixteen rows, both padded to 1003 beside the prefilling peer; +* the gathered maximum: 32768 / 1 / 16 / 9001 rows, 131072 gathered rows in one expert call + (in four of 32768 at the 1M config), the decoding ranks' few rows among them. + +At eight ranks, where both tables call 32768 rows: + +* unequal ranks: 1003 / 1 / 16 / 257 / 1 / 40 / 700 / 3 rows, ranks 0, 3, 5 and 6 prefilling + (rank 5's 40 rows padded to 1003), ranks 1 and 4 decoding the one-row dummy and ranks 2 and + 7 decoding beside them; 8024 gathered rows, one expert call; +* past one call: 5000 / 1 / 16 / 4100 / 1 / 33 / 2048 / 7 rows, 40000 gathered rows in two + expert calls (32768 + 7232), the decoding ranks' few rows among them; +* the gathered maximum: 32768 / 1 / 16 / 9001 / 1 / 2 / 300 / 32768 rows, 262144 gathered + rows in eight expert calls. + +Each rank marks the gathered padding rows as the targets' forwards do (`_padding_rows`): they +take expert id -1 after the routing and reach no rank's experts. + +At both, CUDA graphs: every rank decoding the same captured batch, 1 and 16 rows (4 and 64 +gathered at four ranks, 8 and 128 at eight), captured once and replayed on new tokens, as the +engine replays a decode graph only when every rank is generation-only at the same padded batch +(every count equal, so no padding rows to mark). + +The gate is the runner entry's, against the row's scale: 16 bf16 ulp per element, 2 ulp +relative RMS (the reduce-scatter sums bf16 partials, the reference sums them in fp32). Tokens +whose 4th and 5th router logits are within `TIE` of each other are not compared: the router +GEMM's rounding depends on its row count, and the concatenated and gathered calls differ in it. +Two defects are shown to land far outside the gate: padding placed ahead of the rows instead of +behind them, and the reduce-scatter handing each rank its neighbour's rows. Each case's expert +calls are counted against its gathered rows cut at the call size. + +The four ranks are processes this file starts itself, once per table, on the first four +visible devices, rendezvousing through torch.distributed with MPI disabled; the target's +collectives then run on the process group's communicator, the flavor the engine's own +collectives take without MPI. Rank 0 computes the reference and hands it to the others. + +The eight ranks are already in place: pytest on each of eight MPI ranks started by the +session's launcher over two nodes of four (`mn_mpirun.sh 4`), each rank on its node-local +device, running the catalog's allgather and reduce-scatter on the MPI-session communicator over +both nodes (process_group None: the symbol a dep8 engine launched under MPI calls); every rank +computes the reference itself. It is gated on the launched MPI world (`get_mpi_world_size()`), +not on the devices a node shows: each node sees four. + +Each world runs only in a launch of its own size: the four-rank test unlaunched (or in a launch +of four), the eight-rank test in a launch of eight. Skipped where the checkpoint is absent +(looked up as `Mistral-Large-4-1T-A52B-NVFP4` or `mistral-large-4-nvfp4` under +`LLM_MODELS_ROOT`), below four GPUs or eight launched ranks, or off sm_103. +""" + +from __future__ import annotations + +import collections +import faulthandler +import json +import os +import signal +import socket +import subprocess +import sys +import time +from pathlib import Path +from types import SimpleNamespace + +import pytest +import torch + +__extra_import_path__ = ["../../.."] +from _precision import true_fp32_matmul # noqa: E402 + +#: Four ranks on one node: processes this file starts, over torch.distributed. +#: The modeling_v2 test root, where `_precision` lives. +_TESTS_ROOT = Path(__file__).resolve().parents[3] +SPAWNED_WORLD = 4 +#: Eight ranks over two nodes of four: MPI ranks a launcher starts. +MPI_WORLD = 8 +#: The whole job's budget; a broken collective hangs rather than raising. A rank wedged in a +#: collective a sibling never joined dumps its stack and exits by then. +DEADLINE_S = 2400 +_WORKER_FLAG = "--rank-worker" +CHECKPOINTS = ("Mistral-Large-4-1T-A52B-NVFP4", "mistral-large-4-nvfp4") +SHARD_INDEX = "consolidated.safetensors.index.json" + +#: The MoE layer: the checkpoint's largest FC2 global scale. +LAYER = 1 +#: Per world: (name, rows per rank, target per rank). +CASES = { + 4: ( + ("unequal", (1003, 1, 16, 257), ("prefill", "decode", "decode", "prefill")), + ("gathered maximum", (32768, 1, 16, 9001), ("prefill", "decode", "decode", "prefill")), + ), + 8: ( + ( + "unequal", + (1003, 1, 16, 257, 1, 40, 700, 3), + ("prefill", "decode", "decode", "prefill", "decode", "prefill", "prefill", "decode"), + ), + ( + "past one call", + (5000, 1, 16, 4100, 1, 33, 2048, 7), + ("prefill", "decode", "decode", "prefill", "decode", "prefill", "prefill", "decode"), + ), + ( + "gathered maximum", + (32768, 1, 16, 9001, 1, 2, 300, 32768), + ("prefill", "decode", "decode", "prefill", "decode", "prefill", "prefill", "prefill"), + ), + ), +} +#: The four-rank round trip's expert calls by engine, the modeling's table `_derive_first_step` +#: picks there by the engine's chunked prefill flag alone (block reuse does not change it): +#: `_EXPERT_CALL_TOKENS` without it (the 32K serving config: max_seq_len = max_num_tokens = +#: 32768), `_EXPERT_CALL_TOKENS_CHUNKED` with it (the 1M config: max_seq_len 1048576). +EXPERT_CALLS = {"32k": "_EXPERT_CALL_TOKENS", "1m": "_EXPERT_CALL_TOKENS_CHUNKED"} +#: Decode batches captured on every rank. +GRAPH_BATCHES = (1, 16) +#: One bf16 ulp, relative: 2^-8 of the binade top. +ULP = 2.0**-8 +GATE_ELT, GATE_RMS = 16.0, 2.0 +#: A routing the row count's rounding may flip: the 4th and 5th logits closer than this +#: fraction of the row's largest logit (the router GEMM moves a logit by up to 133 fp32 ulp of +#: the row's largest, about 1.6e-5 of it). +TIE = 1e-4 +#: The largest e2m1 magnitude and the largest finite e4m3 one. +E2M1_MAX, E4M3_MAX = 6.0, 448.0 +#: The layer's parameters, in the weight table's names. +PARAMS = ( + "router", + "mlp_gu", + "mlp_gu_s", + "mlp_dn", + "mlp_dn_s", + "fc1_w", + "fc1_s", + "fc2_w", + "fc2_s", + "isc1", + "isc2", + "ws2_13", + "ws2_2", +) +WINDOW_PARAMS = ("fc1_w", "fc1_s", "fc2_w", "fc2_s", "ws2_13", "ws2_2") + + +def _checkpoint_dir() -> Path | None: + from test_common.llm_data import llm_models_root + + root = llm_models_root() + if root is None: + return None + for name in CHECKPOINTS: + path = Path(root) / name + if (path / "params.json").is_file() and (path / SHARD_INDEX).is_file(): + return path + return None + + +def _modules(): + """The dep target's modeling and weight table. Imported where a rank runs, after its + communicator is up.""" + from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_103__dep import ( # noqa: E501 + modeling as M, + ) + from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_103__dep import ( # noqa: E501 + weights as W, + ) + + return M, W + + +def _config(ck: Path): + """The configuration the engine adapts from the checkpoint's params.json.""" + from tensorrt_llm._torch.models.checkpoints.mistral.config_loader import MistralConfigLoader + + return MistralConfigLoader().load_pretrained_config(str(ck)) + + +# ---------------------------------------------------------------------------- one rank's body + + +def _core(config, rank: int, world: int) -> SimpleNamespace: + """The surface the weight table reads off the core, for rank `rank` of `world`. The mapping + is stated rather than built: with MPI disabled a `Mapping` takes the calling process's own + rank from torch.distributed whatever `rank` it is given, and one rank loads the other ranks' + windows for the reference.""" + mapping = SimpleNamespace( + world_size=world, + rank=rank, + tp_size=world, + tp_rank=rank, + moe_tp_size=1, + moe_ep_size=world, + moe_ep_rank=rank, + enable_attention_dp=True, + ) + return SimpleNamespace( + model_config=SimpleNamespace( + pretrained_config=config, + mapping=mapping, + torch_dtype=torch.bfloat16, + ) + ) + + +def _load(W, ck: Path, config, weight_map: dict, rank: int, world: int, names) -> tuple: + """Layer `LAYER`'s parameters `names` as rank `rank` of `world`'s `load` leaves them: every + manifest source read off the shard files and copied to its index through its transform, + then each row's `layout` on every expert slot. Also the rank's dimension bundle.""" + from safetensors import safe_open + + core = _core(config, rank, world) + d = W.MODEL_WEIGHTS.dims(core) + manifest = W.MODEL_WEIGHTS.manifest(core) + rows = { + r.name: r + for r in W.MODEL_WEIGHTS.WEIGHTS + if r.name in names and LAYER in (r.layer_indices(d) or []) + } + keys = {k for n in names for ckpt, *_ in manifest[f"l{LAYER}_{n}"] for k in W._keys(ckpt)} + by_file = collections.defaultdict(list) + for key in keys: + by_file[weight_map[key]].append(key) + stored = {} + for name, file_keys in by_file.items(): + with safe_open(str(ck / name), framework="pt", device="cpu") as handle: + for key in file_keys: + stored[key] = handle.get_tensor(key) + out = {} + for n in names: + row = rows[n] + param = torch.empty(row.shape(d), dtype=row.dtype or d.dtype, device="cuda") + for ckpt, index, transform in manifest[f"l{LAYER}_{n}"]: + dst = param if index is None else param[index] + if isinstance(ckpt, tuple): + src = transform(tuple(stored[k].cuda() for k in ckpt), core) + else: + src = stored[ckpt].cuda() + if transform is not None: + src = transform(src.contiguous(), core) + dst.copy_(src) + if row.layout is not None: + for slot in range(param.shape[0]): + param[slot].copy_(row.layout(param[slot])) + out[n] = param + return out, d + + +def _layer(M, p: dict, window: dict) -> SimpleNamespace: + """The `L` `_moe` reads, as `_layer_views` takes it: GEMM operands and FP8 scale storage + transposed, the window's stacks and the runner's call scalars.""" + return SimpleNamespace( + mlp_gu=p["mlp_gu"], + mlp_gu_s=p["mlp_gu_s"].t(), + mlp_dn=p["mlp_dn"], + mlp_dn_s=p["mlp_dn_s"].t(), + moe=SimpleNamespace( + router=p["router"].t(), + fc1_w=window["fc1_w"], + fc1_s=window["fc1_s"], + fc2_w=window["fc2_w"], + fc2_s=window["fc2_s"], + **vars(M._moe_scalars(p["isc1"], p["isc2"], window["ws2_13"], window["ws2_2"])), + ), + ) + + +def _target(M, kind: str, d, rank: int, world: int, pg, side, call: int) -> object: + """The target class `kind` runs over the surface `_moe` reads off the core: the four ranks + on the process group's communicator, or the eight on the MPI-session communicator (`pg` + None, as the core holds it under MPI), in expert calls of `call` rows.""" + target = object.__new__(M.PrefillTarget if kind == "prefill" else M.DecodeTarget) + # The core's values at max_num_tokens 32768, both `call`: one call's tokens and the + # autotuner's largest bucket -- all 131072 gathered at four ranks, 32768 of up to 262144 + # gathered at eight; with chunked prefill 32768 at both. + target.core = SimpleNamespace( + topk=4, + num_experts=d.num_experts, + moe_inter=d.moe_inter, + expert_offset=d.expert_offset, + local_experts=d.local_experts, + expert_call_tokens=call, + expert_tune_tokens=call, + dp_group=list(range(world)), + dp_rank=rank, + dp_pg=pg, + _side_stream=side, + ) + return target + + +def _tokens(total: int, seed: int, scale: float) -> torch.Tensor: + """Every rank's tokens concatenated in rank order, the same on every rank.""" + g = torch.Generator(device="cuda").manual_seed(seed) + return (torch.randn(total, 6144, generator=g, device="cuda") * scale).to(torch.bfloat16) + + +def _reference(M, x, L, windows: dict, offsets: dict, shared_of, call: int) -> tuple: + """The layer on one rank over `x`, the concatenated tokens: the shared expert plus every + window's partials, summed in fp32, in expert calls of `call` rows, the target's size. Also + each token's router near-tie flag.""" + acc = torch.zeros(x.shape, dtype=torch.float32, device=x.device) + tie = torch.zeros(x.shape[0], dtype=torch.bool, device=x.device) + for lo in range(0, x.shape[0], call): + hi = min(lo + call, x.shape[0]) + chunk = x[lo:hi] + logits = M.cublas_mm(chunk, L.moe.router, None, torch.float32) + top = logits.topk(5, dim=1).values + tie[lo:hi] = (top[:, 3] - top[:, 4]) < TIE * top.abs().amax(dim=1) + ids, weights = M.renorm_moe_routing_op(logits, 4, torch.bfloat16) + xq, xs = M.fp4_quantize(chunk, L.moe.gscale, 16, False, False) + for w, m in windows.items(): + part = M.fp4_block_scale_moe_runner( + None, None, xq, xs.view(torch.float8_e4m3fn), m.fc1_w, m.fc1_s, None, None, + None, None, m.fc2_w, m.fc2_s, None, m.out1, m.out1_gate, m.out2, 128, 4, None, + None, 6144, offsets[w], m.fc1_w.shape[0], None, 1, True, 0, + topk_weights=weights, topk_ids=ids, tune_max_num_tokens=call, + )[0] # fmt: skip + acc[lo:hi] += part.float() + shared = shared_of(x) + rms = lambda t: t.float().pow(2).mean().sqrt().item() # noqa: E731 + print( + f"[reference] {x.shape[0]} tokens: routed rms {rms(acc):.4g}, shared rms " + f"{rms(shared):.4g}, input rms {rms(x):.4g}", + flush=True, + ) + return (acc.to(torch.bfloat16).float() + shared.float()).to(torch.bfloat16), tie + + +def _deviation(out: torch.Tensor, ref: torch.Tensor) -> tuple[float, float]: + """(worst element deviation against its row's scale, relative RMS deviation), in ulp.""" + o, r = out.float(), ref.float() + row = r.abs().amax(dim=1, keepdim=True).clamp_min(1e-30) + elt = ((o - r).abs() / row).max().item() / ULP + rms = ((o - r).pow(2).mean().sqrt() / r.pow(2).mean().sqrt().clamp_min(1e-30)).item() / ULP + return elt, rms + + +# ------------------------------------------------------- four ranks: spawned, torch.distributed + + +def _run_spawned_rank() -> None: + import torch.distributed as dist + + world = SPAWNED_WORLD + rank = int(os.environ["RANK"]) + torch.cuda.set_device(rank) + dist.init_process_group("nccl", init_method="env://", device_id=torch.device("cuda", rank)) + pg = dist.distributed_c10d._get_default_group().boxed() + + M, W = _modules() + # The expert call table these four ranks run (`_spawn_ranks`), by its engine. + engine = os.environ["ML4_ENGINE"] + call = getattr(M, EXPERT_CALLS[engine])[world] + print(f"[rank {rank}] {engine}: expert calls of {call} rows", flush=True) + + ck = Path(os.environ["ML4_CHECKPOINT"]) + config = _config(ck) + weight_map = json.loads((ck / SHARD_INDEX).read_text())["weight_map"] + + # Every rank: the replicated part and its own window. Rank 0, for the reference: all four. + p, d = _load(W, ck, config, weight_map, rank, world, PARAMS) + windows, offsets = {rank: p}, {rank: d.expert_offset} + if rank == 0: + for w in range(1, world): + windows[w], dw = _load(W, ck, config, weight_map, w, world, WINDOW_PARAMS) + offsets[w] = dw.expert_offset + assert offsets == {w: w * d.local_experts for w in windows}, offsets + L = _layer(M, p, p) + ref_windows = {w: _layer(M, p, windows[w]).moe for w in windows} + side = torch.cuda.Stream() + shared_of = _target(M, "prefill", d, rank, world, pg, side, call)._dense + # Activations at the amax the layer's static input scale was calibrated for. + scale = E4M3_MAX * E2M1_MAX * float(p["isc1"]) / 5 + failures = [] + + def check(name: str, counts, out: torch.Tensor, x: torch.Tensor, gate: bool = True): + """Rank 0 computes the layer over the concatenated tokens and hands every rank the + reference; each rank compares its own rows. Returns (elt, rms, compared rows).""" + if rank == 0: + with true_fp32_matmul(): + ref, tie = _reference( + M, x, L, ref_windows, offsets, lambda t: shared_of(t, L), call + ) + else: + ref = torch.empty(x.shape, dtype=torch.bfloat16, device="cuda") + tie = torch.empty(x.shape[0], dtype=torch.bool, device="cuda") + dist.broadcast(ref, 0) + dist.broadcast(tie, 0) + lo = sum(counts[:rank]) + mine = slice(lo, lo + counts[rank]) + keep = ~tie[mine] + elt, rms = _deviation(out[keep], ref[mine][keep]) + n = int(keep.sum()) + print( + f"[rank {rank}] {name}: rows {counts} -> {n}/{counts[rank]} compared, " + f"{counts[rank] - n} near-tie; {elt:.2f} elt / {rms:.3f} rms ulp", + flush=True, + ) + if gate and not (n > 0 and elt <= GATE_ELT and rms <= GATE_RMS): + failures.append(f"{name}: {elt:.2f} elt / {rms:.3f} rms ulp over {n} rows") + return elt, rms, n + + def round_trip(counts, kinds, x): + """This rank's own rows through its target's `_moe`; also the expert calls it made.""" + lo = sum(counts[:rank]) + own = x[lo : lo + counts[rank]].contiguous() + target = _target(M, kinds[rank], d, rank, world, pg, side, call) + padded = M._padded_rows(target.core, {"all_rank_num_tokens": list(counts)}, counts[rank]) + runner = M.fp4_block_scale_moe_runner + calls = [] + + def counted(*args, **kwargs): + calls.append(args[2].shape[0]) + return runner(*args, **kwargs) + + M.fp4_block_scale_moe_runner = counted + try: + out = target._moe(own, L, padded, M._padding_rows(list(counts), padded, own.device)) + finally: + M.fp4_block_scale_moe_runner = runner + torch.cuda.synchronize() + return out, calls + + for seed, (name, counts, kinds) in enumerate(CASES[world]): + x = _tokens(sum(counts), seed, scale) + out, calls = round_trip(counts, kinds, x) + gathered = world * max(counts) + expected = [min(call, gathered - lo) for lo in range(0, gathered, call)] + print(f"[rank {rank}] {name}: {gathered} gathered rows in expert calls {calls}", flush=True) + if calls != expected: + failures.append(f"{name}: expert calls {calls}, expected {expected}") + check(name, counts, out, x) + del x, out + + # Two defects the comparison must see: the padding ahead of the rows, and each rank handed + # its neighbour's rows by the reduce-scatter. + name, counts, kinds = CASES[world][0] + x = _tokens(sum(counts), 0, scale) + real_pad = torch.nn.functional.pad + + def front_pad(t, pad, *args, **kwargs): + if len(pad) == 4: + pad = [pad[0], pad[1], pad[3], pad[2]] + return real_pad(t, pad, *args, **kwargs) + + torch.nn.functional.pad = front_pad + try: + out, _ = round_trip(counts, kinds, x) + finally: + torch.nn.functional.pad = real_pad + elt, rms, _ = check("defect: padding ahead of the rows", counts, out, x, gate=False) + if counts[rank] < max(counts) and rms <= 5 * GATE_RMS: + failures.append(f"a front-padded rank passes: {rms:.3f} rms ulp") + real_rs = M.reducescatter + + def neighbours_rows(t, sizes, group, process_group=None): + return real_rs(torch.roll(t, -(t.shape[0] // world), 0), sizes, group, process_group) + + M.reducescatter = neighbours_rows + try: + out, _ = round_trip(counts, kinds, x) + finally: + M.reducescatter = real_rs + elt, rms, _ = check("defect: the neighbour's rows", counts, out, x, gate=False) + if rms <= 5 * GATE_RMS: + failures.append(f"the neighbour's rows pass: {rms:.3f} rms ulp") + + # CUDA graphs: every rank decoding the same batch, captured once, replayed on new tokens. + for b in GRAPH_BATCHES: + counts = (b,) * world + target = _target(M, "decode", d, rank, world, pg, side, call) + static = torch.zeros(b, 6144, dtype=torch.bfloat16, device="cuda") + warm = torch.cuda.Stream() + warm.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(warm): + for _ in range(2): + target._moe(static, L, b) + torch.cuda.current_stream().wait_stream(warm) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + out = target._moe(static, L, b) + for replay in range(2): + x = _tokens(world * b, 100 + 10 * b + replay, scale) + static.copy_(x[rank * b : (rank + 1) * b]) + graph.replay() + torch.cuda.synchronize() + check(f"graph {b} rows, replay {replay}", counts, out.clone(), x) + + dist.barrier() + if failures: + raise AssertionError(f"rank {rank}: " + "; ".join(failures)) + print(f"[rank {rank}] all checks passed", flush=True) + + +def _worker() -> None: + try: + _run_spawned_rank() + except BaseException: + import traceback + + traceback.print_exc() + sys.stdout.flush() + sys.stderr.flush() + # A rank that leaves a collective early wedges its siblings; the launcher kills them on + # this exit. + os._exit(13) + sys.stdout.flush() + # No process-group teardown: a communicator that captured collectives can wedge NCCL's + # watchdog at destruction, and the launcher owns these processes. + os._exit(0) + + +def _spawn_ranks(checkpoint: Path, engine: str) -> None: + """One fresh process per device, with its own session, each running `engine`'s expert + calls; on the first nonzero exit or on the deadline every session is killed, so a wedged + collective cannot outlive the run.""" + world = SPAWNED_WORLD + visible = os.environ.get("CUDA_VISIBLE_DEVICES") + devices = ( + [v for v in visible.split(",") if v.strip()] + if visible + else [str(i) for i in range(torch.cuda.device_count())] + ) + repo_root = Path(__file__).resolve().parents[7] + assert (repo_root / "tensorrt_llm").is_dir(), repo_root + with socket.socket() as s: + s.bind(("127.0.0.1", 0)) + port = s.getsockname()[1] + # The launching step's PMIx/PMI/OMPI/SLURM namespace stays out of the ranks: they + # rendezvous through torch.distributed, and a hard exit registered with that step would + # take it down whole. + env = { + k: v + for k, v in os.environ.items() + if not k.startswith(("PMIX_", "PMI_", "OMPI_", "SLURM_")) + } + env.update( + CUDA_VISIBLE_DEVICES=",".join(devices[:world]), + MASTER_ADDR="127.0.0.1", + MASTER_PORT=str(port), + WORLD_SIZE=str(world), + TLLM_DISABLE_MPI="1", + ML4_CHECKPOINT=str(checkpoint), + ML4_ENGINE=engine, + # The worker runs this file as a script, outside pytest's import hook, so the + # directory of `_precision` is on its path the way `__extra_import_path__` puts it. + PYTHONPATH=os.pathsep.join( + p for p in (str(repo_root), str(_TESTS_ROOT), env.get("PYTHONPATH", "")) if p + ), + ) + children = [ + subprocess.Popen( + [sys.executable, str(Path(__file__).resolve()), _WORKER_FLAG], + env=dict(env, RANK=str(r)), + start_new_session=True, + ) + for r in range(world) + ] + + def kill_all() -> None: + for child in children: + try: + os.killpg(child.pid, signal.SIGKILL) + except ProcessLookupError: + pass + for child in children: + child.wait() + + deadline = time.monotonic() + DEADLINE_S + live = set(range(world)) + failure = None + while live and failure is None: + if time.monotonic() > deadline: + failure = f"did not finish in {DEADLINE_S}s (wedged)" + break + for r in sorted(live): + code = children[r].poll() + if code is None: + continue + live.discard(r) + if code != 0: + failure = f"rank {r} exited {code}" + break + time.sleep(0.5) + if failure is not None: + kill_all() + raise AssertionError(f"the {world}-rank MoE round trip ({engine}) failed: {failure}") + + +# ------------------------------------------------------- eight ranks: launched, MPI + + +def _run_mpi_rank(ck: Path) -> list[str]: + """One rank's body. Returns the rank's gate failures; raises on anything else.""" + from tensorrt_llm._utils import mpi_comm, mpi_rank + + world = MPI_WORLD + M, W = _modules() + # The expert call at eight ranks, with or without chunked prefill. + call = M._EXPERT_CALL_TOKENS[world] + assert call == M._EXPERT_CALL_TOKENS_CHUNKED[world] == 32768, call + rank = mpi_rank() + comm = mpi_comm() + hosts = comm.allgather(socket.gethostname()) + print( + f"[rank {rank}] host {hosts[rank]}, device {torch.cuda.current_device()} of " + f"{torch.cuda.device_count()} visible; {len(set(hosts))} node(s)", + flush=True, + ) + config = _config(ck) + weight_map = json.loads((ck / SHARD_INDEX).read_text())["weight_map"] + + # Every rank: the replicated part and its own window, then the seven other windows for the + # reference it computes itself. + p, d = _load(W, ck, config, weight_map, rank, world, PARAMS) + windows, offsets = {rank: p}, {rank: d.expert_offset} + for w in range(world): + if w != rank: + windows[w], dw = _load(W, ck, config, weight_map, w, world, WINDOW_PARAMS) + offsets[w] = dw.expert_offset + assert offsets == {w: 16 * w for w in range(world)}, offsets + assert d.local_experts == 16, d.local_experts + L = _layer(M, p, p) + ref_windows = {w: _layer(M, p, windows[w]).moe for w in sorted(windows)} + side = torch.cuda.Stream() + shared_of = _target(M, "prefill", d, rank, world, None, side, call)._dense + # Activations at the amax the layer's static input scale was calibrated for. + scale = E4M3_MAX * E2M1_MAX * float(p["isc1"]) / 5 + failures = [] + + def check(name: str, counts, out: torch.Tensor, x: torch.Tensor, gate: bool = True): + """Every rank computes the layer over the concatenated tokens and compares its own + rows. Returns (elt, rms, compared rows).""" + with true_fp32_matmul(): + ref, tie = _reference(M, x, L, ref_windows, offsets, lambda t: shared_of(t, L), call) + lo = sum(counts[:rank]) + mine = slice(lo, lo + counts[rank]) + keep = ~tie[mine] + elt, rms = _deviation(out[keep], ref[mine][keep]) + n = int(keep.sum()) + print( + f"[rank {rank}] {name}: rows {counts} -> {n}/{counts[rank]} compared, " + f"{counts[rank] - n} near-tie; {elt:.2f} elt / {rms:.3f} rms ulp", + flush=True, + ) + if gate and not (n > 0 and elt <= GATE_ELT and rms <= GATE_RMS): + failures.append(f"{name}: {elt:.2f} elt / {rms:.3f} rms ulp over {n} rows") + return elt, rms, n + + def round_trip(counts, kinds, x): + """This rank's own rows through its target's `_moe`; also the expert calls it made.""" + lo = sum(counts[:rank]) + own = x[lo : lo + counts[rank]].contiguous() + target = _target(M, kinds[rank], d, rank, world, None, side, call) + padded = M._padded_rows(target.core, {"all_rank_num_tokens": list(counts)}, counts[rank]) + runner = M.fp4_block_scale_moe_runner + calls = [] + + def counted(*args, **kwargs): + calls.append(args[2].shape[0]) + return runner(*args, **kwargs) + + M.fp4_block_scale_moe_runner = counted + try: + out = target._moe(own, L, padded, M._padding_rows(list(counts), padded, own.device)) + finally: + M.fp4_block_scale_moe_runner = runner + torch.cuda.synchronize() + return out, calls + + for seed, (name, counts, kinds) in enumerate(CASES[world]): + x = _tokens(sum(counts), seed, scale) + out, calls = round_trip(counts, kinds, x) + gathered = world * max(counts) + expected = [min(call, gathered - lo) for lo in range(0, gathered, call)] + print(f"[rank {rank}] {name}: {gathered} gathered rows in expert calls {calls}", flush=True) + if calls != expected: + failures.append(f"{name}: expert calls {calls}, expected {expected}") + check(name, counts, out, x) + del x, out + + # Two defects the comparison must see: the padding ahead of the rows, and each rank handed + # its neighbour's rows by the reduce-scatter. + name, counts, kinds = CASES[world][0] + x = _tokens(sum(counts), 0, scale) + real_pad = torch.nn.functional.pad + + def front_pad(t, pad, *args, **kwargs): + if len(pad) == 4: + pad = [pad[0], pad[1], pad[3], pad[2]] + return real_pad(t, pad, *args, **kwargs) + + torch.nn.functional.pad = front_pad + try: + out, _ = round_trip(counts, kinds, x) + finally: + torch.nn.functional.pad = real_pad + elt, rms, _ = check("defect: padding ahead of the rows", counts, out, x, gate=False) + if counts[rank] < max(counts) and rms <= 5 * GATE_RMS: + failures.append(f"a front-padded rank passes: {rms:.3f} rms ulp") + real_rs = M.reducescatter + + def neighbours_rows(t, sizes, group, process_group=None): + return real_rs(torch.roll(t, -(t.shape[0] // world), 0), sizes, group, process_group) + + M.reducescatter = neighbours_rows + try: + out, _ = round_trip(counts, kinds, x) + finally: + M.reducescatter = real_rs + elt, rms, _ = check("defect: the neighbour's rows", counts, out, x, gate=False) + if rms <= 5 * GATE_RMS: + failures.append(f"the neighbour's rows pass: {rms:.3f} rms ulp") + + # CUDA graphs: every rank decoding the same batch, captured once, replayed on new tokens. + for b in GRAPH_BATCHES: + counts = (b,) * world + target = _target(M, "decode", d, rank, world, None, side, call) + static = torch.zeros(b, 6144, dtype=torch.bfloat16, device="cuda") + warm = torch.cuda.Stream() + warm.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(warm): + for _ in range(2): + target._moe(static, L, b) + torch.cuda.current_stream().wait_stream(warm) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + out = target._moe(static, L, b) + for replay in range(2): + x = _tokens(world * b, 100 + 10 * b + replay, scale) + static.copy_(x[rank * b : (rank + 1) * b]) + graph.replay() + torch.cuda.synchronize() + check(f"graph {b} rows, replay {replay}", counts, out.clone(), x) + del graph + + print( + f"[rank {rank}] peak allocated {torch.cuda.max_memory_allocated() / 2**30:.2f} GiB", + flush=True, + ) + return failures + + +def _launched_world() -> int: + """The MPI world this process is a rank of, when a launcher started it as one; 1 otherwise + (including rank 0 of trtllm-llmapi-launch, whose other ranks serve the LLM API, not this + body).""" + from tensorrt_llm._utils import mpi_world_size + from tensorrt_llm.llmapi.mpi_session import get_mpi_world_size + + world = get_mpi_world_size() + return world if world == mpi_world_size() else 1 + + +# ---------------------------------------------------------------------------- the tests + + +@pytest.mark.no_xdist +@pytest.mark.parametrize("engine", list(EXPERT_CALLS)) +def test_moe_round_trip_on_four_ranks(engine): + checkpoint = _checkpoint_dir() + if checkpoint is None: + pytest.skip(f"checkpoint {CHECKPOINTS} is not under LLM_MODELS_ROOT") + if torch.cuda.device_count() < SPAWNED_WORLD: + pytest.skip(f"needs {SPAWNED_WORLD} GPUs") + launched = _launched_world() + if launched not in (1, SPAWNED_WORLD): + pytest.skip(f"the {SPAWNED_WORLD}-rank round trip does not start in {launched} MPI ranks") + if torch.cuda.get_device_capability(0) != (10, 3): + pytest.skip("the mistral_large4 modeling_v2 targets are certified on sm_103 only") + _spawn_ranks(checkpoint, engine) + + +@pytest.mark.no_xdist +@pytest.mark.threadleak(enabled=False) +def test_moe_round_trip_on_eight_ranks(): + from tensorrt_llm._utils import ( + get_sm_version, + local_mpi_rank, + mpi_comm, + mpi_rank, + mpi_world_size, + ) + from tensorrt_llm.llmapi.mpi_session import get_mpi_world_size + + checkpoint = _checkpoint_dir() + if checkpoint is None: + pytest.skip(f"checkpoint {CHECKPOINTS} is not under LLM_MODELS_ROOT") + if _launched_world() != MPI_WORLD: + pytest.skip( + f"needs {MPI_WORLD} launched MPI ranks (this world: get_mpi_world_size() " + f"{get_mpi_world_size()}, MPI world {mpi_world_size()})" + ) + if get_sm_version() != 103: + pytest.skip("the mistral_large4 modeling_v2 targets are certified on sm_103 only") + torch.cuda.set_device(local_mpi_rank()) + faulthandler.dump_traceback_later(DEADLINE_S, exit=True) + rank = mpi_rank() + try: + failures = _run_mpi_rank(checkpoint) + except BaseException: + import traceback + + print(f"[rank {rank}] raised", flush=True) + traceback.print_exc() + sys.stdout.flush() + sys.stderr.flush() + # A rank that leaves the collective sequence wedges its siblings; exiting without + # finalizing makes the launcher end them. + os._exit(13) + # Every rank sees every rank's failures, so all eight pass or fail together. + every = mpi_comm().allgather(failures) + faulthandler.cancel_dump_traceback_later() + bad = [f"rank {r}: {f}" for r, fs in enumerate(every) for f in fs] + assert not bad, "; ".join(bad) + print(f"[rank {rank}] all checks passed on every rank", flush=True) + + +if __name__ == "__main__" and _WORKER_FLAG in sys.argv: + _worker() diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_retrieval_1m.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_retrieval_1m.py new file mode 100644 index 000000000000..6ec8c59f82e4 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_retrieval_1m.py @@ -0,0 +1,168 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Retrieval at length on Mistral-Large-4-1T-A52B-NVFP4's sm_103 dep target, at the checkpoint's +own context. + +The engine is the one the core unit serves: per rank max_seq_len 1048576, prefill in chunks of +max_num_tokens 32768 (chunked prefill on), sixteen sequences, the fp8_ds_mla latent KV cache +at kv_cache_config's default dtype, block reuse off, 0.85 of the free memory to the KV cache, +CUDA graphs over batches 1, 2, 4, 8 and 16, padded; attention data parallel over four ranks, +the routed experts expert parallel. A passkey needle +is hidden in plain filler text (`test_modeling_v2_long_context.py`'s prompt) at an early, a +middle and a late depth of prompts of about 32768, 131072, 524288 and 1040000 tokens, and +greedy decoding must recall every key. Text only: token ids through the LLM API, no chat +template. + +One request at a time: one rank holds it and prefills it chunk by chunk, every chunk after its +first over the cached prefix the earlier ones wrote, while the other three ranks step beside it +on the engine's dummy rows (each step's experts gather every rank's rows). The rank is read back +from the iteration statistics, which under attention DP carry one record per rank. The time to +the first token, as the client sees it streaming, is printed per request with the rank and its +context steps. + +Runs on four GB300s with the ranks already in place: pytest under `trtllm-llmapi-launch`, as +the whole-model accuracy class runs, with the checkpoint under `LLM_MODELS_ROOT` as +`Mistral-Large-4-1T-A52B-NVFP4` (or `mistral-large-4-nvfp4`) in its Mistral-native +format, which the engine loads with checkpoint_format "mistral". Skipped where the +checkpoint is absent, below four GPUs, or off sm_103. +""" + +from __future__ import annotations + +import collections +import time + +import pytest +from test_common.llm_data import llm_models_root +from utils.util import skip_num_gpus_less_than + +from tensorrt_llm import LLM, SamplingParams +from tensorrt_llm._torch.modeling_v2 import MODELING_V2_VALIDATE_ENV +from tensorrt_llm._utils import get_sm_version +from tensorrt_llm.inputs import TokensPrompt +from tensorrt_llm.llmapi import CudaGraphConfig, KvCacheConfig + +from . import test_modeling_v2_long_context as long_context + +pytestmark = [ + skip_num_gpus_less_than(4), + pytest.mark.skipif( + get_sm_version() != 103, + reason="the mistral_large4 modeling_v2 targets are certified on sm_103 only", + ), + # One engine serves the module: the LLM API starts its result-dispatch thread on the first + # request, inside the first test, and stops it when the fixture shuts the engine down. + pytest.mark.threadleak(enabled=False), +] + +#: dep4's four attention-DP ranks. +RANKS = 4 +MAX_SEQ_LEN = 1048576 +MAX_NUM_TOKENS = 32768 +# The core unit's serving engine (its LLM-API options), per rank. +ENGINE = dict( + tensor_parallel_size=RANKS, + moe_expert_parallel_size=RANKS, + enable_attention_dp=True, + max_batch_size=16, + max_num_tokens=MAX_NUM_TOKENS, + max_seq_len=MAX_SEQ_LEN, + enable_chunked_prefill=True, + cuda_graph_config=CudaGraphConfig(max_batch_size=16, enable_padding=True), + kv_cache_config=KvCacheConfig(enable_block_reuse=False, free_gpu_memory_fraction=0.85), +) +# Prompt lengths, and the needle's depth in each, early to late. +LENGTHS = (32768, 131072, 524288, 1040000) +DEPTHS = (0.1, 0.5, 0.9) +# A prompt "about" its length when it lands within one percent of it. +FILL = 0.99 +ANSWER = SamplingParams(max_tokens=12, temperature=0.0) +#: Every request's key, distinct, fixed by its length and depth. +KEYS = dict( + zip( + [(n, d) for n in LENGTHS for d in DEPTHS], + long_context._keys(len(LENGTHS) * len(DEPTHS), seed=MAX_SEQ_LEN), + ) +) + + +def _rank_steps(llm: LLM) -> dict[int, list[tuple[int, int, int, int]]]: + """Since the last call, per attention-DP rank, every iteration it recorded: + `(iteration, context requests, context tokens, generation requests)`, in order.""" + by_rank: dict[int, list[tuple[int, int, int, int]]] = collections.defaultdict(list) + for record in llm.get_stats(timeout=2): + ifb = record.get("inflightBatchingStats") or {} + by_rank[int(record.get("attentionDpRank", 0))].append( + ( + int(record["iter"]), + ifb.get("numContextRequests", 0), + ifb.get("numCtxTokens", 0), + ifb.get("numGenRequests", 0), + ) + ) + return {rank: sorted(steps) for rank, steps in by_rank.items()} + + +@pytest.fixture(scope="module") +def llm(): + path = long_context._checkpoint_dir() + if path is None: + pytest.skip( + f"checkpoint {long_context.CHECKPOINTS} is not under LLM_MODELS_ROOT " + f"({llm_models_root()})" + ) + with LLM( + str(path), + checkpoint_format="mistral", + modeling_v2="require", + env_overrides={MODELING_V2_VALIDATE_ENV: "1"}, + enable_iter_perf_stats=True, + **ENGINE, + ) as engine: + print(f"RETRIEVAL kv_cache_capacity {engine.get_kv_cache_capacity()}", flush=True) + _rank_steps(engine) + yield engine + + +@pytest.mark.parametrize("depth", DEPTHS) +@pytest.mark.parametrize("tokens", LENGTHS) +def test_passkey_at_length(llm, tokens, depth): + """A passkey at `depth` of a prompt of about `tokens` tokens, prefilled in 32768-token + chunks by one rank while the others step beside it, recalled by greedy decoding.""" + key = KEYS[(tokens, depth)] + ids = long_context._passkey_ids(llm.tokenizer, tokens, depth, key) + assert FILL * tokens <= len(ids) <= tokens, (tokens, len(ids)) + + start = time.monotonic() + ttft = None + for out in llm.generate_async(TokensPrompt(prompt_token_ids=ids), ANSWER, streaming=True): + if ttft is None and out.outputs[0].token_ids: + ttft = time.monotonic() - start + total = time.monotonic() - start + answer = out.outputs[0].text + steps = _rank_steps(llm) + + held = { + rank: [(it, t) for it, n, t, _ in s if n] + for rank, s in steps.items() + if any(n for _, n, _, _ in s) + } + print( + f"RETRIEVAL tokens={len(ids)} depth={depth} key={key} answer={answer!r} " + f"ttft_s={ttft:.1f} total_s={total:.1f} rank={list(held)}", + flush=True, + ) + for rank in sorted(steps): + ctx = [(it, t) for it, n, t, _ in steps[rank] if n] + print( + f"RETRIEVAL tokens={len(ids)} depth={depth} rank {rank}: {len(steps[rank])} " + f"iterations recorded (iterations {steps[rank][0][0]}..{steps[rank][-1][0]}), " + f"{len(ctx)} with context rows {ctx}", + flush=True, + ) + # One rank prefilled the whole prompt, in chunks of at most max_num_tokens. + assert len(held) == 1, held + ((rank, chunks),) = held.items() + assert sum(t for _, t in chunks) == len(ids), (rank, chunks) + assert all(t <= MAX_NUM_TOKENS for _, t in chunks), (rank, chunks) + assert str(key) in answer, (len(ids), depth, key, answer) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_routed_experts.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_routed_experts.py new file mode 100644 index 000000000000..acc7932641cb --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_routed_experts.py @@ -0,0 +1,774 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Real layers' routed experts through the prepared layout, on one GPU: the dep target's expert +windows at four ranks and at eight (dep8). + +The NVFP4 MoE runner reads its expert stacks in a layout no checkpoint stores -- FC1's halves +interleaved, every stack's rows block-shuffled, the block scales 128x4-swizzled -- and nothing +checks it: a wrong layout costs hundreds of bf16 ulp and raises nothing. Rank r holds the whole +experts `[Er, Er + E)`, E = 128 / world (32 at four ranks, 16 at eight over two nodes of four), +runs them over every rank's gathered tokens in calls of at most `_EXPERT_CALL_TOKENS[world]` +rows (131072 at four ranks, 32768 at eight), and the reduce-scatter sums the windows' partials. +On an engine with chunked prefill (as the 1M config at max_seq_len 1048576 runs), the calls are +of at most `_EXPERT_CALL_TOKENS_CHUNKED[world]` rows, 32768 at both. + +At four ranks, on one MoE layer and one rank's 32-expert window of Mistral-Large-4-1T-A52B-NVFP4 +itself (layer 1 rank 0 and layer 45 rank 3): + +* the window's stacks are filled from the shard files by the weight table's own manifest rows + (sources, destination indices, transforms) and laid out by the rows' own `layout`, the steps + the table's `load` takes (the CPU manifest test checks `load` itself against an independent + construction of the layout); that layout is byte for byte the engine's own TRTLLM-Gen + preparation of the same stored bytes (`trtllmgen_maybe_get_cached_w3_w1_permute_indices`, + `shuffle_matrix`, `block_scale_interleave`); +* each target's `_moe` -- the layer's router through `renorm_moe_routing_op`, `fp4_quantize` + under the layer's static scale, the pre-routed runner with `_moe_scalars`, in calls of at + most 131072 tokens, or of at most 32768 as the 1M config runs it -- is run on + bf16 activations at the magnitudes the layer's input scale was calibrated for, with the + shared expert zeroed so the routed half is all that is measured, and compared with an fp32 + reference on the window's dequantized checkpoint experts (`e2m1 * weight_scale * + weight_scale_2`, activations `e2m1 * scale * input_scale`, the SwiGLU output requantized to + NVFP4 under `1 / input_scale(w2)` as the reference implementation does), at the runner + entry's gate: 16 ulp per element of the row's scale, 2 ulp relative RMS. One rank alone + here, so the expert path's gather and reduce-scatter are the identity (the multi-rank round + trip is `test_modeling_v2_moe_round_trip.py`'s); 138304 rows run it in two calls, as 40000 + do in the 1M config's; +* the runner on the stored bytes without the layout, and on FC1 with its halves swapped, + lands far outside that gate (measured 260.8 and 22.5 rms ulp at layer 1, 376.9 and 289.2 at + layer 45, where the right layout lands at 0.70-0.76): the comparison sees a wrong layout; +* the cut into `_EXPERT_CALL_TOKENS_CHUNKED` calls changes no bit: each target's `_moe` over a + full step's 131072 gathered rows (four ranks' 32768, three of them mostly the zero padding + of ranks holding few rows) in calls of 32768 equals, bit for bit, one router / routing / + NVFP4-quantize / runner call over all of them tuned to 131072, the one call per layer of an + engine without chunked prefill (the 32K serving config); so do the router logits, the + routing and the quantized activations of the calls against the one call's; +* the padding rows reach no expert: over a step of 8192 / 16 / 3 / 64 rows, each zero-padded + to 8192, with the gathered padding rows marked by `_padding_rows` (as the targets' forwards + mark them), each target's `_moe` hands the runner expert id -1 in every padding row's slots + and the routing's own ids elsewhere, and returns every real row bit for bit as unmarked (the + padding then routed to experts 0..3, an exact tie at +0.0) and +0.0 in every padding row. + +At eight ranks (dep8), where both tables call 32768 rows, on an early, a middle and a late +MoE layer (1, 35 and 71), for every rank r in 0..7 and its 16-expert window: + +* the window's stacks are filled and laid out as above, byte for byte the engine's own + preparation; +* each target's `_moe`, in calls of at most 32768 rows, is run as the one rank of a one-rank + group: 16 rows through `DecodeTarget`, 1031 and one whole call of 32768 through + `PrefillTarget`, and 32768 + 7232 through `DecodeTarget` (two calls, each writing its rows of + one output), each compared at the same gate with the same fp32 reference on the window's + dequantized checkpoint experts; +* the eight partials, summed in fp32 in rank order, are compared at the same gate with the + same reference over all 128 experts, built from the checkpoint's expert keys 0..127 directly + rather than from the table's windows, so a window the table misplaces, doubles or drops is + seen; leaving any one partial out lands far outside; +* with every token's top-4 skewed into one window (the router's column 0 biased for that + window's experts, the activations' element 0 held positive), on one call of the chunk's + size -- 32768 tokens, so 131072 routed rows on one rank -- that window's partial alone meets + the 128-expert reference and every other window's is exactly zero; +* the runner on the stored bytes without the layout, on FC1 with its halves swapped, and on a + neighbouring rank's laid-out window under this rank's offset lands far outside the gate. + +Skipped where the checkpoint is absent (looked up as `mistral-large-4-nvfp4` or +`Mistral-Large-4-1T-A52B-NVFP4` under `LLM_MODELS_ROOT`). +""" + +from __future__ import annotations + +import collections +import json +from pathlib import Path +from types import SimpleNamespace + +import pytest +import torch +from safetensors import safe_open +from test_common.llm_data import llm_models_root + +from tensorrt_llm._torch.modeling_v2.catalog.gemm.cublas_mm import cublas_mm +from tensorrt_llm._torch.modeling_v2.catalog.moe.fp4_block_scale_moe_runner import ( # noqa: E501 + fp4_block_scale_moe_runner, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_103__dep import ( # noqa: E501 + modeling as M, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_103__dep import ( # noqa: E501 + weights as W, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.moe.renorm_moe_routing_op import ( + renorm_moe_routing_op, +) +from tensorrt_llm._torch.models.checkpoints.mistral.config_loader import MistralConfigLoader +from tensorrt_llm._torch.moe.fused_moe.quantization import ( + trtllmgen_maybe_get_cached_w2_permute_indices, + trtllmgen_maybe_get_cached_w3_w1_permute_indices, +) +from tensorrt_llm.mapping import Mapping + +__extra_import_path__ = ["../../.."] +from _precision import true_fp32_matmul # noqa: E402 + +DEV = "cuda" +#: GB300 nodes here hold four devices: four ranks fill one, eight span two. +GPUS_PER_NODE = 4 +CHECKPOINTS = ("mistral-large-4-nvfp4", "Mistral-Large-4-1T-A52B-NVFP4") +#: The expert parameters of one MoE layer, in the table's names. +EXPERT_PARAMS = ("fc1_w", "fc1_s", "fc2_w", "fc2_s", "isc1", "isc2", "ws2_13", "ws2_2") +#: The runner's epilogue tile, which fixes its 32-row shuffle (the engine's own constant). +EPILOGUE_TILE_M = 128 +#: One bf16 ulp, relative: 2^-8 of the binade top. +ULP = 2.0**-8 +#: The largest e2m1 magnitude and the largest finite e4m3 one. +E2M1_MAX, E4M3_MAX = 6.0, 448.0 +#: The token counts each window is driven at, at eight ranks: a decode step, a ragged prefill, +#: one whole expert call, and two calls (the second ragged). +STEPS = ( + ("DecodeTarget", 16), + ("PrefillTarget", 1031), + ("PrefillTarget", M._EXPERT_CALL_TOKENS[8]), + ("DecodeTarget", M._EXPERT_CALL_TOKENS[8] + 7232), +) +#: An early, a middle and a late MoE layer, every window of each at eight ranks. +LAYERS = (1, 35, 71) + + +def _checkpoint_dir() -> Path | None: + root = llm_models_root() + if root is None: + return None + for name in CHECKPOINTS: + path = root / name + if (path / "params.json").is_file() and (path / W.SHARD_INDEX).is_file(): + return path + return None + + +@pytest.fixture(scope="module") +def checkpoint(): + path = _checkpoint_dir() + if path is None: + pytest.skip(f"checkpoint {CHECKPOINTS} is not under LLM_MODELS_ROOT ({llm_models_root()})") + assert torch.cuda.is_available(), "the routed-expert check needs a GPU" + weight_map = json.loads((path / W.SHARD_INDEX).read_text())["weight_map"] + # The configuration the engine adapts from the checkpoint's params.json. + config = MistralConfigLoader().load_pretrained_config(str(path)) + return SimpleNamespace(path=path, weight_map=weight_map, config=config) + + +def _core(config, rank: int, world: int) -> SimpleNamespace: + """The surface the weight table reads off the core, for rank `rank` of `world`.""" + mapping = Mapping( + world_size=world, + rank=rank, + gpus_per_node=GPUS_PER_NODE, + tp_size=world, + moe_tp_size=1, + moe_ep_size=world, + enable_attention_dp=True, + ) + assert (mapping.tp_rank, mapping.moe_ep_rank, mapping.moe_tp_rank) == (rank, rank, 0) + return SimpleNamespace( + model_config=SimpleNamespace( + pretrained_config=config, + mapping=mapping, + torch_dtype=torch.bfloat16, + ) + ) + + +def _read(checkpoint, keys) -> dict[str, torch.Tensor]: + """The given checkpoint tensors, off the shard files, one file at a time.""" + by_file = collections.defaultdict(list) + for key in keys: + by_file[checkpoint.weight_map[key]].append(key) + out = {} + for name, names in by_file.items(): + with safe_open(str(checkpoint.path / name), framework="pt", device="cpu") as handle: + for key in names: + out[key] = handle.get_tensor(key) + return out + + +def _window(checkpoint, layer: int, rank: int, world: int): + """Layer `layer`'s expert parameters on rank `rank` of `world`, as the table's `load` leaves + them: every manifest source copied to its index through its transform, then each row's + `layout` on every expert. Also returns the stored tensors it read and the router.""" + core = _core(checkpoint.config, rank, world) + d = W.MODEL_WEIGHTS.dims(core) + manifest = W.MODEL_WEIGHTS.manifest(core) + rows = {r.name: r for r in W.MODEL_WEIGHTS.WEIGHTS if r.name in EXPERT_PARAMS} + keys = { + k + for name in EXPERT_PARAMS + for ckpt, *_ in manifest[f"l{layer}_{name}"] + for k in W._keys(ckpt) + } + router_key = f"layers.{layer}.gate.weight" + stored = _read(checkpoint, sorted(keys | {router_key})) + params = {} + for name in EXPERT_PARAMS: + row = rows[name] + param = torch.empty(row.shape(d), dtype=row.dtype or d.dtype, device=DEV) + for ckpt, index, transform in manifest[f"l{layer}_{name}"]: + dst = param if index is None else param[index] + if isinstance(ckpt, tuple): + src = transform(tuple(W._materialize(stored[k]).to(DEV) for k in ckpt), core) + else: + src = W._materialize(stored[ckpt]).to(DEV) + if transform is not None: + src = transform(src.contiguous(), core) + assert tuple(dst.shape) == tuple(src.shape) and dst.dtype == src.dtype, (name, ckpt) + dst.copy_(src) + if row.layout is not None: + for slot in range(param.shape[0]): + param[slot].copy_(row.layout(param[slot])) + params[name] = param + return d, params, stored, stored[router_key].to(DEV) + + +def _stored_experts(stored: dict, layer: int, experts) -> dict[str, list[torch.Tensor]]: + """The stored bytes of the given global experts: `[w3 ; w1]` codes and block scales, w2's, + and each expert's `weight_scale_2` pair.""" + out = collections.defaultdict(list) + for expert in experts: + e = f"layers.{layer}.experts.{expert}" + out["fc1_w"].append(torch.cat([stored[f"{e}.w3.weight"], stored[f"{e}.w1.weight"]])) + out["fc1_s"].append( + torch.cat([stored[f"{e}.w3.weight_scale"], stored[f"{e}.w1.weight_scale"]]) + ) + out["fc2_w"].append(stored[f"{e}.w2.weight"]) + out["fc2_s"].append(stored[f"{e}.w2.weight_scale"]) + out["ws2_13"].append(stored[f"{e}.w1.weight_scale_2"]) + out["ws2_2"].append(stored[f"{e}.w2.weight_scale_2"]) + return out + + +def _engine_layout(stored: dict[str, list[torch.Tensor]]) -> dict[str, torch.Tensor]: + """The engine's own TRTLLM-Gen NVFP4 MoE preparation of the stored bytes, expert by expert: + its gated-activation row reorder chained with its row shuffle, `shuffle_matrix`, and + `block_scale_interleave` on the scales.""" + cache: dict = {} + sf = torch.uint8 + out = collections.defaultdict(list) + for w31, s31, w2, s2 in zip(stored["fc1_w"], stored["fc1_s"], stored["fc2_w"], stored["fc2_s"]): + w31, s31, w2, s2 = (t.to(DEV) for t in (w31, s31, w2, s2)) + perm = trtllmgen_maybe_get_cached_w3_w1_permute_indices(w31, cache, EPILOGUE_TILE_M) + out["fc1_w"].append(torch.ops.trtllm.shuffle_matrix(w31, perm)) + perm = trtllmgen_maybe_get_cached_w3_w1_permute_indices( + s31.view(sf), cache, EPILOGUE_TILE_M, num_elts_per_sf=16 + ) + shuffled = torch.ops.trtllm.shuffle_matrix(s31.view(sf), perm) + out["fc1_s"].append( + torch.ops.trtllm.block_scale_interleave(shuffled.reshape(s31.shape)).reshape(s31.shape) + ) + perm = trtllmgen_maybe_get_cached_w2_permute_indices(w2, cache, EPILOGUE_TILE_M) + out["fc2_w"].append(torch.ops.trtllm.shuffle_matrix(w2, perm)) + perm = trtllmgen_maybe_get_cached_w2_permute_indices( + s2.view(sf), cache, EPILOGUE_TILE_M, num_elts_per_sf=16 + ) + shuffled = torch.ops.trtllm.shuffle_matrix(s2.view(sf), perm) + out["fc2_s"].append( + torch.ops.trtllm.block_scale_interleave(shuffled.reshape(s2.shape)).reshape(s2.shape) + ) + return {name: torch.stack(t) for name, t in out.items()} + + +def _unpack(codes: torch.Tensor) -> torch.Tensor: + """`[..., K/2]` packed e2m1 -> `[..., K]` codes, the even element in the low nibble.""" + return torch.stack([codes & 0x0F, codes >> 4], dim=-1).reshape(*codes.shape[:-1], -1) + + +def _reference(x_q, x_s, isc1, isc2, ids, weights, stored, first: int, inter: int): + """The routed half in fp32 over the given experts' dequantized checkpoint bytes, global + experts `[first, first + len(stored))`: the runner entry's `reference`, handed the stored + codes (unpacked) and their block scales with each expert's stored `weight_scale_2` + multiplied in, and the activation's exact NVFP4 value with its global scale undone + (`code * scale * input_scale`). fp32; the caller rounds.""" + rows = x_q.shape[0] + act = _unpack(x_q).long() + act_scale = x_s.view(torch.float8_e4m3fn).float().reshape(rows, -1) + e2m1 = torch.tensor( + [0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0, -0.0, -0.5, -1.0, -1.5, -2.0, -3.0, -4.0, -6.0], + device=DEV, + ) + hidden = e2m1[act] * act_scale.repeat_interleave(16, dim=1) * isc1.float() + count = len(stored["fc1_w"]) + + def operand(name, half, global_scales): + codes = torch.stack([_unpack(t.to(DEV))[half] for t in stored[name + "_w"]]) + scales = torch.stack([t.to(DEV).float()[half] for t in stored[name + "_s"]]) + g = torch.stack([s.to(DEV).float() for s in global_scales]) + return codes, scales * g.view(-1, 1, 1) + + up = operand("fc1", slice(0, inter), stored["ws2_13"]) + gate = operand("fc1", slice(inter, 2 * inter), stored["ws2_13"]) + down = operand("fc2", slice(None), stored["ws2_2"]) + with true_fp32_matmul(): + return fp4_block_scale_moe_runner.reference( + hidden, + ids, + weights, + up, + gate, + down, + fc2_global_scale=1.0 / float(isc2), + local_expert_offset=first, + local_num_experts=count, + ) + + +def _deviation(out: torch.Tensor, ref: torch.Tensor) -> tuple[float, float]: + """(worst element deviation against its row's scale, relative RMS deviation), in ulp.""" + o, r = out.float(), ref.float() + row = r.abs().amax(dim=1, keepdim=True).clamp_min(1e-30) + elt = ((o - r).abs() / row).max().item() / ULP + rms = ((o - r).pow(2).mean().sqrt() / r.pow(2).mean().sqrt().clamp_min(1e-30)).item() / ULP + return elt, rms + + +def _gate(out: torch.Tensor, ref: torch.Tensor, what: str) -> tuple[float, float]: + """The runner entry's gate on the rows `ref` is nonzero in: 16 ulp per element of the row's + scale, 2 ulp relative RMS.""" + live = (ref != 0).any(dim=1) + assert bool(live.any()), f"{what}: no token routed here, nothing is measured" + elt, rms = _deviation(out[live], ref[live].to(torch.bfloat16)) + print(f"{what}: {int(live.sum())} tokens, {elt:.2f} elt / {rms:.3f} rms ulp") + assert rms <= 2.0, f"{what}: relative RMS {rms:.2f} ulp > 2" + assert elt <= 16.0, f"{what}: element deviation {elt:.2f} ulp > 16" + return elt, rms + + +def _layer(d, params, router) -> SimpleNamespace: + """The `L` a target's `_moe` reads: a shared expert whose down projection is zero (its + partial is exactly 0, so the result is the routed half alone), the router's operand, the + laid-out window and the call scalars from `_moe_scalars`.""" + shared = d.shared_inter + unit_sf = lambda n, k: W._deepgemm_sf( # noqa: E731 + torch.ones(n // W.BLOCK, k // W.BLOCK, device=DEV), None + ).t() + return SimpleNamespace( + mlp_gu=(torch.randn(2 * shared, d.hidden, device=DEV) * 0.1).to(torch.float8_e4m3fn), + mlp_gu_s=unit_sf(2 * shared, d.hidden), + mlp_dn=torch.zeros(d.hidden, shared, device=DEV).to(torch.float8_e4m3fn), + mlp_dn_s=unit_sf(d.hidden, shared), + moe=SimpleNamespace( + router=router.t(), + fc1_w=params["fc1_w"], + fc1_s=params["fc1_s"], + fc2_w=params["fc2_w"], + fc2_s=params["fc2_s"], + **vars( + M._moe_scalars(params["isc1"], params["isc2"], params["ws2_13"], params["ws2_2"]) + ), + ), + ) + + +def _target(cls, d, side, call: int): + """A `PrefillTarget` or `DecodeTarget` over the surface `_moe` reads off the core, as the one + rank of a one-rank group, in calls of, and tuning to, `call` tokens: one call's, as the core + takes it at its world size and chunked prefill flag (`_derive_first_step`).""" + target = object.__new__(cls) + target.core = SimpleNamespace( + topk=4, + num_experts=d.num_experts, + moe_inter=d.moe_inter, + expert_offset=d.expert_offset, + local_experts=d.local_experts, + expert_call_tokens=call, + expert_tune_tokens=call, + dp_group=[0], + dp_rank=0, + dp_pg=None, + _side_stream=side, + ) + return target + + +@pytest.fixture +def one_rank(monkeypatch): + """The expert path's collectives over a one-rank group: the gather is the rank's own rows, + the reduce-scatter its own partial sum.""" + monkeypatch.setattr(M, "allgather", lambda x, sizes, group, pg=None: x.clone()) + monkeypatch.setattr(M, "reducescatter", lambda x, sizes, group, pg=None: x.clone()) + + +def _routing(o, router, call: int): + """Top-4 ids and weights of every row, routed in the target's expert calls of at most `call` + rows: the router GEMM's rounding depends on its row count.""" + ids, weights = [], [] + for x in torch.split(o, call): + logits = cublas_mm(x, router.t(), None, torch.float32) + i, w = renorm_moe_routing_op(logits, 4, torch.bfloat16) + ids.append(i) + weights.append(w) + return torch.cat(ids), torch.cat(weights) + + +#: (layer, rank) at four ranks: the layers holding the checkpoint's largest and smallest FC2 +#: global scale (1 / input_scale(w2): 6898.53 at layer 1, 1.36898 at layer 45), on the first and +#: the last expert window. +CASES = [(1, 0), (45, 3)] +#: The expert call at four ranks: without chunked prefill (the 32K config's), every gathered +#: row of a full step in one call, and with it (the 1M config's). +CALLS_AT_FOUR = [ + pytest.param(M._EXPERT_CALL_TOKENS[4], id="32k"), + pytest.param(M._EXPERT_CALL_TOKENS_CHUNKED[4], id="1m"), +] + + +@pytest.mark.parametrize("call", CALLS_AT_FOUR) +@pytest.mark.parametrize("layer, rank", CASES, ids=[f"layer{n}-rank{r}" for n, r in CASES]) +def test_routed_experts_of_one_real_window(checkpoint, one_rank, layer, rank, call): + """At four ranks: one 32-expert window of the layer, in the config's expert calls.""" + world = 4 + torch.manual_seed(1000 * layer + rank) + side = torch.cuda.Stream() + d, params, stored_tensors, router = _window(checkpoint, layer, rank, world) + window = range(d.expert_offset, d.expert_offset + d.local_experts) + stored = _stored_experts(stored_tensors, layer, window) + + # The table's layout is the engine's own preparation of the same bytes. + engine = _engine_layout(stored) + for name in ("fc1_w", "fc1_s", "fc2_w", "fc2_s"): + mine = params[name].view(torch.uint8) + assert torch.equal(mine, engine[name].view(torch.uint8)), name + + isc1, isc2 = params["isc1"], params["isc2"] + # Activations at the amax the layer's static input scale was calibrated for (448 * 6 * + # input_scale): a gaussian whose largest elements reach about it. + calibrated = E4M3_MAX * E2M1_MAX * float(isc1) + L = _layer(d, params, router) + cases = ( + (M.DecodeTarget, 16), + (M.PrefillTarget, 1031), + (M.PrefillTarget, 32768), + # Past one expert call: 131072 + 7232 rows (32768 + 7232 at the 1M config), each call + # writing its rows of the output. + (M.DecodeTarget, call + 7232), + ) + for cls, rows in cases: + o = (torch.randn(rows, d.hidden, device=DEV) * (calibrated / 5)).to(torch.bfloat16) + out = _target(cls, d, side, call)._moe(o, L, rows) + + # The routing the target's calls see: the router GEMM's rounding depends on its row + # count, so the reference routes in the same calls. + ids, weights = _routing(o, router, call) + local = ids.long() - d.expert_offset + live = ((local >= 0) & (local < d.local_experts)).any(dim=1) + assert bool(live.any()), "no token routed into the window: nothing is measured" + x_q, x_s = M.fp4_quantize(o, L.moe.gscale, 16, False, False) + ref = _reference(x_q, x_s, isc1, isc2, ids, weights, stored, d.expert_offset, d.moe_inter) + ref = ref.to(torch.bfloat16) + assert bool((out[~live] == 0).all()), "a token with no slot in the window must be zero" + elt, rms = _deviation(out[live], ref[live]) + print( + f"layer {layer} rank {rank} {cls.__name__} T={rows}: {int(live.sum())} tokens in " + f"the window, {elt:.2f} elt / {rms:.3f} rms ulp" + ) + assert rms <= 2.0, f"relative RMS {rms:.2f} ulp > 2" + assert elt <= 16.0, f"element deviation {elt:.2f} ulp > 16" + + # A wrong layout runs without complaint and lands far away. + rows = 1031 + o = (torch.randn(rows, d.hidden, device=DEV) * (calibrated / 5)).to(torch.bfloat16) + logits = cublas_mm(o, L.moe.router, None, torch.float32) + ids, weights = renorm_moe_routing_op(logits, 4, torch.bfloat16) + x_q, x_s = M.fp4_quantize(o, L.moe.gscale, 16, False, False) + ref = _reference(x_q, x_s, isc1, isc2, ids, weights, stored, d.expert_offset, d.moe_inter) + ref = ref.to(torch.bfloat16) + live = ((ids.long() - d.expert_offset >= 0) & (ids.long() - d.expert_offset < 32)).any(dim=1) + as_stored = {n: torch.stack(stored[n]).to(DEV) for n in ("fc1_w", "fc1_s", "fc2_w", "fc2_s")} + inter = d.moe_inter + swapped = torch.cat([as_stored["fc1_w"][:, inter:], as_stored["fc1_w"][:, :inter]], dim=1) + swapped_layout = torch.stack([W._fc1_codes(t) for t in swapped]) + wrong = { + "no layout": as_stored, + "halves swapped": dict(params, fc1_w=swapped_layout), + } + for what, ops in wrong.items(): + s = L.moe + out = fp4_block_scale_moe_runner( + None, None, x_q, x_s.view(torch.float8_e4m3fn), ops["fc1_w"].contiguous(), + ops["fc1_s"].contiguous(), None, None, None, None, ops["fc2_w"].contiguous(), + ops["fc2_s"].contiguous(), None, s.out1, s.out1_gate, s.out2, d.num_experts, 4, None, + None, d.moe_inter, d.expert_offset, d.local_experts, None, 1, True, 0, + topk_weights=weights, topk_ids=ids, tune_max_num_tokens=32768, + )[0] # fmt: skip + _, rms = _deviation(out[live], ref[live]) + print(f"layer {layer} rank {rank} {what}: {rms:.1f} rms ulp") + assert rms > 5 * 2.0, f"{what}: {rms:.2f} rms ulp is not told apart from the right layout" + + +@pytest.mark.parametrize("layer", LAYERS, ids=[f"layer{n}" for n in LAYERS]) +def test_routed_experts_of_every_window(checkpoint, one_rank, layer): + """At eight ranks: every 16-expert window of the layer, alone and summed, in the expert + calls every engine takes there.""" + world, call = 8, M._EXPERT_CALL_TOKENS[8] + assert call == M._EXPERT_CALL_TOKENS_CHUNKED[8] == 32768 + torch.manual_seed(1000 * layer) + side = torch.cuda.Stream() + windows = [_window(checkpoint, layer, rank, world) for rank in range(world)] + d0, params0, _, router = windows[0] + inter = d0.moe_inter + # Every window is whole experts at 16 r, and the eight tile the 128 routed experts. + assert [w[0].expert_offset for w in windows] == [16 * r for r in range(world)] + assert {w[0].local_experts for w in windows} == {16} + + # The table's layout is the engine's own preparation of the same bytes, on every window. + stored_all: dict[str, torch.Tensor] = {} + for rank, (d, params, stored_tensors, _) in enumerate(windows): + stored = _stored_experts(stored_tensors, layer, range(16 * rank, 16 * rank + 16)) + engine = _engine_layout(stored) + for name in ("fc1_w", "fc1_s", "fc2_w", "fc2_s"): + assert torch.equal(params[name].view(torch.uint8), engine[name].view(torch.uint8)), ( + rank, + name, + ) + stored_all.update(stored_tensors) + # Every expert of the layer, by its checkpoint keys: what the 128-expert reference reads. + full = _stored_experts(stored_all, layer, range(d0.num_experts)) + isc1, isc2 = params0["isc1"], params0["isc2"] + for _, params, _, _ in windows: + assert torch.equal(params["isc1"], isc1) and torch.equal(params["isc2"], isc2) + + # Activations at the amax the layer's static input scale was calibrated for (448 * 6 * + # input_scale): a gaussian whose largest elements reach about it. + calibrated = E4M3_MAX * E2M1_MAX * float(isc1) + layers = [_layer(d, params, router) for d, params, _, _ in windows] + for cls_name, rows in STEPS: + cls = getattr(M, cls_name) + o = (torch.randn(rows, d0.hidden, device=DEV) * (calibrated / 5)).to(torch.bfloat16) + ids, weights = _routing(o, router, call) + # NVFP4 quantization is per row under the layer's static scale: one call or several + # quantize every row alike. + x_q, x_s = M.fp4_quantize(o, layers[0].moe.gscale, 16, False, False) + parts = [] + for rank, ((d, params, stored_tensors, _), L) in enumerate(zip(windows, layers)): + out = _target(cls, d, side, call)._moe(o, L, rows) + stored = _stored_experts(stored_tensors, layer, range(16 * rank, 16 * rank + 16)) + ref = _reference(x_q, x_s, isc1, isc2, ids, weights, stored, 16 * rank, inter) + local = ids.long() - 16 * rank + outside = ~((local >= 0) & (local < 16)).any(dim=1) + assert bool((out[outside] == 0).all()), "a token with no slot in the window is zero" + _gate(out, ref, f"layer {layer} rank {rank} {cls_name} T={rows}") + parts.append(out) + # The windows' sum: fp32 in rank order, rounded once. + summed = sum(p.float() for p in parts).to(torch.bfloat16) + ref_full = _reference(x_q, x_s, isc1, isc2, ids, weights, full, 0, inter) + assert bool((ref_full != 0).any(dim=1).all()), "every token routes somewhere" + _gate(summed, ref_full, f"layer {layer} eight windows summed {cls_name} T={rows}") + if rows >= 1031: + for drop in range(world): + partial = sum(p.float() for i, p in enumerate(parts) if i != drop) + _, rms = _deviation(partial.to(torch.bfloat16), ref_full.to(torch.bfloat16)) + assert rms > 25.0, f"T={rows} without window {drop}: only {rms:.1f} rms ulp" + del parts, summed, ref_full, ref + + # Skew: every token's top-4 in one window, on one call of the chunk's size, so that window's + # rank runs 4 x 32768 routed rows. Column 0 of the router is biased up for that window's + # experts and down for the rest, and every token's element 0 is positive, so the bias + # dominates the logits; the ranking inside the window is the router's own. + rows = call + for skew in (0, world - 1, 3): + o = (torch.randn(rows, d0.hidden, device=DEV) * (calibrated / 5)).to(torch.bfloat16) + o[:, 0] = o[:, 0].abs() + calibrated / 10 + base = cublas_mm(o, router.t(), None, torch.float32) + bias = 64.0 * float(base.abs().max()) / float(o[:, 0].float().min()) + skewed = router.clone() + inside = torch.zeros(d0.num_experts, dtype=torch.bool, device=DEV) + inside[16 * skew : 16 * skew + 16] = True + skewed[inside, 0] += bias + skewed[~inside, 0] -= bias + ids, weights = _routing(o, skewed, call) + assert bool(((ids >= 16 * skew) & (ids < 16 * skew + 16)).all()), "skew missed" + x_q, x_s = M.fp4_quantize(o, layers[0].moe.gscale, 16, False, False) + ref_full = _reference(x_q, x_s, isc1, isc2, ids, weights, full, 0, inter) + parts = [] + for rank, (d, params, _, _) in enumerate(windows): + L = _layer(d, params, skewed) + out = _target(M.PrefillTarget, d, side, call)._moe(o, L, rows) + if rank != skew: + assert bool((out == 0).all()), f"skew {skew}: window {rank} wrote" + parts.append(out) + _gate(parts[skew], ref_full, f"layer {layer} skew into window {skew} T={rows}") + summed = sum(p.float() for p in parts).to(torch.bfloat16) + assert torch.equal(summed, parts[skew]), "the other windows' zeros change the sum" + del parts, summed, ref_full + + # A wrong layout, or a neighbour's window under this rank's offset, runs without complaint + # and lands far away. + rows = 1031 + o = (torch.randn(rows, d0.hidden, device=DEV) * (calibrated / 5)).to(torch.bfloat16) + ids, weights = _routing(o, router, call) + x_q, x_s = M.fp4_quantize(o, layers[0].moe.gscale, 16, False, False) + for rank in (0, world - 1): + d, params, stored_tensors, _ = windows[rank] + stored = _stored_experts(stored_tensors, layer, range(16 * rank, 16 * rank + 16)) + ref = _reference(x_q, x_s, isc1, isc2, ids, weights, stored, 16 * rank, inter) + live = (ref != 0).any(dim=1) + as_stored = {name: torch.stack(stored[name]).to(DEV) for name in ("fc1_w", "fc1_s")} + as_stored.update({name: torch.stack(stored[name]).to(DEV) for name in ("fc2_w", "fc2_s")}) + swapped = torch.cat([as_stored["fc1_w"][:, inter:], as_stored["fc1_w"][:, :inter]], dim=1) + swapped_layout = torch.stack([W._fc1_codes(t) for t in swapped]) + neighbour = windows[(rank + 1) % world][1] + wrong = { + "no layout": dict(params, **as_stored), + "halves swapped": dict(params, fc1_w=swapped_layout), + f"window {(rank + 1) % world}'s experts": dict(neighbour), + } + s = layers[rank].moe + for what, ops in wrong.items(): + out = fp4_block_scale_moe_runner( + None, None, x_q, x_s.view(torch.float8_e4m3fn), ops["fc1_w"].contiguous(), + ops["fc1_s"].contiguous(), None, None, None, None, ops["fc2_w"].contiguous(), + ops["fc2_s"].contiguous(), None, s.out1, s.out1_gate, s.out2, d.num_experts, 4, + None, None, inter, d.expert_offset, d.local_experts, None, 1, True, 0, + topk_weights=weights, topk_ids=ids, tune_max_num_tokens=call, + )[0] # fmt: skip + _, rms = _deviation(out[live], ref[live].to(torch.bfloat16)) + print(f"layer {layer} rank {rank} {what}: {rms:.1f} rms ulp") + assert rms > 5 * 2.0, f"{what}: {rms:.2f} rms ulp is not told apart from the right one" + + +#: A full step's gathered rows at four ranks, every rank's max_num_tokens: one expert call +#: without chunked prefill (the 32K config). +GATHERED = 4 * 32768 +#: Each rank's live rows in that step, the round trip's skew: rank 0 prefills a full chunk, +#: the others hold a decode dummy, sixteen sequences and a part chunk, zero-padded to 32768. +SKEW = (32768, 1, 16, 9001) + + +def _one_call(o: torch.Tensor, L, d) -> SimpleNamespace: + """One expert-path call over every row of `o`: router, routing, NVFP4 quantization and the + runner each once, tuned to all of them. Over a full step's gathered rows at four ranks, the + 32K config's one call per layer.""" + m = L.moe + logits = cublas_mm(o, m.router, None, torch.float32) + ids, weights = renorm_moe_routing_op(logits, 4, torch.bfloat16) + xq, xs = M.fp4_quantize(o, m.gscale, 16, False, False) + out = fp4_block_scale_moe_runner( + None, None, xq, xs.view(torch.float8_e4m3fn), m.fc1_w, m.fc1_s, None, None, None, None, + m.fc2_w, m.fc2_s, None, m.out1, m.out1_gate, m.out2, d.num_experts, 4, None, None, + d.moe_inter, d.expert_offset, d.local_experts, None, 1, True, 0, + topk_weights=weights, topk_ids=ids, tune_max_num_tokens=o.shape[0], + )[0] # fmt: skip + return SimpleNamespace(logits=logits, ids=ids, weights=weights, xq=xq, xs=xs, out=out) + + +def _rows_differing(a: torch.Tensor, b: torch.Tensor) -> int: + """How many rows of two equally shaped tensors differ in any bit.""" + a, b = a.reshape(a.shape[0], -1), b.reshape(b.shape[0], -1) + if a.dtype.is_floating_point: + a, b = a.view(torch.uint8), b.view(torch.uint8) + return int((a != b).any(dim=1).sum()) + + +@pytest.mark.parametrize("layer, rank", CASES, ids=[f"layer{n}-rank{r}" for n, r in CASES]) +def test_expert_calls_give_one_calls_bits(checkpoint, one_rank, layer, rank): + """At four ranks: cutting a step's gathered rows into the 1M config's + `_EXPERT_CALL_TOKENS_CHUNKED[4]`-row calls changes no bit of the router logits, the routing, + the quantized activations or each target's `_moe` output against one call over all of them, + the 32K config's (`_EXPERT_CALL_TOKENS[4]` rows).""" + world, call = 4, M._EXPERT_CALL_TOKENS_CHUNKED[4] + assert GATHERED == M._EXPERT_CALL_TOKENS[4] + assert GATHERED % call == 0 and GATHERED > call + torch.manual_seed(7000 + 1000 * layer + rank) + side = torch.cuda.Stream() + d, params, _, router = _window(checkpoint, layer, rank, world) + L = _layer(d, params, router) + calibrated = E4M3_MAX * E2M1_MAX * float(params["isc1"]) + o = torch.zeros(GATHERED, d.hidden, dtype=torch.bfloat16, device=DEV) + block = GATHERED // len(SKEW) + for r, live in enumerate(SKEW): + rows = torch.randn(live, d.hidden, device=DEV) * (calibrated / 5) + o[r * block : r * block + live] = rows.to(torch.bfloat16) + + one = _one_call(o, L, d) + calls = [_one_call(x, L, d) for x in torch.split(o, call)] + pieces = { + "router logits": ("logits", one.logits), + "top-4 ids": ("ids", one.ids), + "top-4 weights": ("weights", one.weights), + "NVFP4 codes": ("xq", one.xq), + "NVFP4 scales": ("xs", one.xs), + } + differ = {} + for what, (field, whole) in pieces.items(): + cut = torch.cat([getattr(c, field) for c in calls]) + differ[what] = _rows_differing(cut, whole) + for cls in (M.PrefillTarget, M.DecodeTarget): + target = _target(cls, d, side, call) + out = target._moe(o, L, GATHERED) + # The shared expert's down projection is zero, so its partial adds +0.0 to each value. + shared = target._dense(o, L) + assert int(shared.ne(0).sum()) == 0 + differ[f"{cls.__name__}._moe"] = _rows_differing(out, torch.add(one.out, shared)) + local = one.ids.long() - d.expert_offset + live = int(((local >= 0) & (local < d.local_experts)).any(dim=1).sum()) + print( + f"layer {layer} rank {rank}: {GATHERED} gathered rows in " + f"{GATHERED // call} calls of {call} against one call, " + f"{live} rows routed into the window; rows differing {differ}" + ) + assert live > 0, "no token routed into the window: the runner's bits are not measured" + assert all(n == 0 for n in differ.values()), differ + + +#: Each rank's own rows in a step whose gathered rows are mostly padding, at four ranks: rank 0 +#: prefills, the others decode beside it, each zero-padded to 8192. +UNEVEN = (8192, 16, 3, 64) + + +@pytest.mark.parametrize("call", CALLS_AT_FOUR) +@pytest.mark.parametrize("layer, rank", CASES, ids=[f"layer{n}-rank{r}" for n, r in CASES]) +def test_padding_rows_reach_no_expert(checkpoint, one_rank, monkeypatch, layer, rank, call): + """At four ranks: with the gathered padding rows marked by `_padding_rows`, as the targets' + forwards mark them, each target's `_moe` hands the runner expert id -1 in every slot of + every padding row and the routing's own ids in every other row, and returns every real row + bit for bit as without the mark (the padding then routed, an exact tie at +0.0, to experts + 0..3 of rank 0's window), and +0.0 in every padding row.""" + world = 4 + torch.manual_seed(9000 + 1000 * layer + rank) + side = torch.cuda.Stream() + d, params, _, router = _window(checkpoint, layer, rank, world) + L = _layer(d, params, router) + calibrated = E4M3_MAX * E2M1_MAX * float(params["isc1"]) + seg = max(UNEVEN) + o = torch.zeros(world * seg, d.hidden, dtype=torch.bfloat16, device=DEV) + for r, live in enumerate(UNEVEN): + rows = torch.randn(live, d.hidden, device=DEV) * (calibrated / 5) + o[r * seg : r * seg + live] = rows.to(torch.bfloat16) + pad = M._padding_rows(list(UNEVEN), seg, torch.device(DEV)) + assert M._padding_rows([seg] * world, seg, torch.device(DEV)) is None + assert pad.shape == (world * seg,) and int(pad.sum()) == world * seg - sum(UNEVEN) + for r, live in enumerate(UNEVEN): + assert not pad[r * seg : r * seg + live].any() and pad[r * seg + live : (r + 1) * seg].all() + + runner = M.fp4_block_scale_moe_runner + for cls in (M.PrefillTarget, M.DecodeTarget): + target = _target(cls, d, side, call) + out, ids = {}, {} + for marked in (False, True): + seen = [] + + def recorded(*args, _seen=seen, **kwargs): + _seen.append(kwargs["topk_ids"].clone()) + return runner(*args, **kwargs) + + monkeypatch.setattr(M, "fp4_block_scale_moe_runner", recorded) + out[marked] = target._moe(o, L, o.shape[0], pad if marked else None) + ids[marked] = torch.cat(seen) + torch.cuda.synchronize() + real = ~pad + differ = _rows_differing(out[True][real], out[False][real]) + print( + f"layer {layer} rank {rank} {cls.__name__}, calls of {call}: {int(pad.sum())} padding " + f"rows of {o.shape[0]}, unmarked routed to experts {ids[False][pad].unique().tolist()}; " + f"real rows differing {differ}" + ) + assert sorted(ids[False][pad].unique().tolist()) == [0, 1, 2, 3] + assert bool((ids[True][pad] == -1).all()) + assert torch.equal(ids[True][real], ids[False][real]) + assert differ == 0 + assert int(out[True][pad].ne(0).sum()) == 0 diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_step_paths.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_step_paths.py new file mode 100644 index 000000000000..0f3ba743bca1 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_step_paths.py @@ -0,0 +1,999 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""What the Mistral-Large-4-1T-A52B NVFP4 dep target chooses per step, and the sizes it reads off +the engine to choose by. + +No path is selected by the serving configuration. The core fixes its sizes on the first step the +engine hands over (`_derive_first_step`, again for a replaced KV cache manager) from the attention +metadata's max_num_tokens, max_num_requests, max_seq_len and runtime_features (chunked prefill, +block reuse: the executor builds them once from enable_chunked_prefill and the KvCacheConfig's +enable_block_reuse); every step then takes its paths from its own sizes: + +* the held context workspace, bf16 `[core.ctx_workspace_rows, 576]`, allocated on the first + context step whichever path it takes and held: `_ctx_workspace_rows` is twice the most + context KV one step can attend (`_most_ctx_kv`: max_num_tokens with neither chunked prefill + nor block reuse, else mainline's min(max_num_requests, max_num_tokens) x max_seq_len), up to + `_CTX_WORKSPACE_ROWS` (2**19): 65536 rows at the 32K serving config (max_seq_len = + max_num_tokens = 32768, both off), 2**19 with block reuse at max_batch_size 16 and at the 1M + config (max_seq_len 1048576, chunked prefill); +* the context path, by the step's context KV n_kv (every context request's cached and new rows): + the sparse prefill over the held workspace's first n_kv rows while n_kv fits the held rows, + past them the sparse fp8 decode over the pool, every context row a one-row sequence, in + sub-batches of at most `_CTX_DECODE_ROWS` (8192) with one scheduler per sub-batch size; +* layer 35's latent, which the NoPE layers 39, 43, ..., 71 attend over (`shared_source`, layer 35 + only, at every max_seq_len): when 2 n_kv fits the held rows it is gathered once into the n_kv + rows after the step's own (`kept`) and 39 and 43 read it there without a gather; otherwise + every layer gathers its source into the step's rows, 35's again for 39 and for 43; +* the indexer's logits budget, `_prefill_logits_budget(max_num_tokens, max_seq_len)`: mainline's + reachable MQA-logits tile (TLLM_INDEXER_MQA_LOGITS_ELEM_BUDGET lowers it), within + `_PREFILL_LOGITS_CEILING` (2**26 at dep4) and max_seq_len x max_seq_len / 64 -- 2**24 at + max_seq_len 32768 (512-row tiles at 32768 keys), the ceiling at 131072 and at 1048576 -- with + each tile's logits freed before the next tile's are allocated; +* the indexer's scoring blocks: an engine chunk of more than max_seq_len keys is cut at its + requests over the host prefix sums `step_args` carries on every context step; a chunk within + max_seq_len is one block, several requests or not; a block with no key fills its rows' slots + with -1; +* the expert path's calls, keyed on chunked prefill alone: `_EXPERT_CALL_TOKENS[world]` (131072 + at four ranks, 32768 at eight) without it, block reuse or not, and + `_EXPERT_CALL_TOKENS_CHUNKED[world]` (32768 at both) with it, at any max_seq_len; the MoE + autotuner's largest bucket is the call, capped at every rank's max_num_tokens gathered; +* the rope tables: `rows`, `indexer` and `identity`, and no MLA table. + +No device: the core is built on the meta device from the checkpoint's native config, params.json +(skipped where it is absent under LLM_MODELS_ROOT); everything else runs on host values, with +recording stand-ins for the kernels the paths call. +""" + +from __future__ import annotations + +import weakref +from types import SimpleNamespace + +import pytest +import torch +from test_common.llm_data import llm_models_root + +from tensorrt_llm._torch.attention.backends.interface import AttentionRuntimeFeatures +from tensorrt_llm._torch.attention.backends.sparse.params import ( + get_indexer_mqa_logits_workspace_bytes, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_103__dep import ( # noqa: E501 + modeling as M, +) + +pytestmark = pytest.mark.cpu_only + +CHECKPOINTS = ("Mistral-Large-4-1T-A52B-NVFP4", "mistral-large-4-nvfp4") +#: The checkpoint's layers, the owner its NoPE layers 39, 43, ..., 71 attend over, each layer's +#: latent source, the 63 owners, and the latent row's width. +LAYERS = 72 +SHARED_SOURCE = 35 +KV_SOURCE = [SHARED_SOURCE if i > SHARED_SOURCE and i % 4 == 3 else i for i in range(LAYERS)] +OWNERS = [i for i in range(LAYERS) if KV_SOURCE[i] == i] +LAT_DIM = 576 +#: The checkpoint's rope surface (its native config, adapted from params.json), what +#: `_rope_tables` reads of it. +ROPE_CFG = SimpleNamespace( + rope_parameters=dict( + rope_type="yarn", + rope_theta=10000.0, + factor=2048.0, + original_max_position_embeddings=512, + beta_fast=32.0, + beta_slow=1.0, + mscale_all_dim=1, + attention_factor=1.0, + ), + qk_rope_head_dim=64, +) +_BUDGET_ENV = "TLLM_INDEXER_MQA_LOGITS_ELEM_BUDGET" + +#: Engines by name: (max_num_tokens, max_num_requests, max_seq_len, chunked prefill, block reuse). +#: The 32K serving config, it with block reuse, with chunked prefill and block reuse, and with +#: chunked prefill at one request per step, the Pareto config, the 1M config, and max_seq_len +#: 131072 with chunked prefill and with neither. Both flags are the attention metadata's runtime +#: features, as the executor builds them from enable_chunked_prefill and the KvCacheConfig's +#: enable_block_reuse. +ENGINES = { + "32k": (32768, 16, 32768, False, False), + "32k_block_reuse": (32768, 16, 32768, False, True), + "32k_chunked_block_reuse": (32768, 16, 32768, True, True), + "32k_chunked_one_request": (32768, 1, 32768, True, False), + "pareto": (16384, 16, 32768, False, False), + "1m": (32768, 16, 1048576, True, False), + "131072_chunked": (32768, 16, 131072, True, False), + "131072": (32768, 16, 131072, False, False), +} +#: What each engine sizes: (the most context KV one step attends, the held workspace's rows, the +#: indexer's logits budget). The expert path's calls follow the engine's chunked prefill flag +#: (`CALLS`), not the context KV: block reuse without it keeps `_EXPERT_CALL_TOKENS`, one request +#: per step under it, which attends at most max_seq_len = max_num_tokens, takes +#: `_EXPERT_CALL_TOKENS_CHUNKED`. +SIZES = { + "32k": (32768, 65536, 1 << 24), + "32k_block_reuse": (1 << 19, 1 << 19, 1 << 24), + "32k_chunked_block_reuse": (1 << 19, 1 << 19, 1 << 24), + "32k_chunked_one_request": (32768, 65536, 1 << 24), + "pareto": (16384, 32768, 1 << 24), + "1m": (1 << 24, 1 << 19, 1 << 26), + "131072_chunked": (1 << 21, 1 << 19, 1 << 26), + "131072": (32768, 65536, 1 << 26), +} +#: The expert path's call by (chunked prefill, world). +CALLS = {(False, 4): 131072, (False, 8): 32768, (True, 4): 32768, (True, 8): 32768} + + +@pytest.fixture(autouse=True) +def _default_logits_budget(monkeypatch): + """Mainline's logits budget at its default: a TLLM_INDEXER_MQA_LOGITS_ELEM_BUDGET set in the + environment would lower the budgets pinned here.""" + monkeypatch.delenv(_BUDGET_ENV, raising=False) + + +def _md(name: str, **extra) -> SimpleNamespace: + """The attention metadata's engine quantities for engine `name`, as `_most_ctx_kv` and + `_derive_first_step` read them, plus `extra`.""" + max_num_tokens, max_num_requests, max_seq_len, chunked, reuse = ENGINES[name] + return SimpleNamespace( + max_num_tokens=max_num_tokens, + max_num_requests=max_num_requests, + max_seq_len=max_seq_len, + runtime_features=AttentionRuntimeFeatures( + chunked_prefill=chunked, cache_reuse=reuse, chunk_size=max_num_tokens + ), + **extra, + ) + + +def _manager() -> SimpleNamespace: + """A KV cache manager with what `_derive_first_step` reads of it: the fp8_ds_mla layout, the + checkpoint's kv_source_layers and its 63 owners' pool rows. A new one each call: the core + derives again for a replaced manager.""" + return SimpleNamespace( + latent_layout=M._LATENT_LAYOUT, + kv_source_layers=KV_SOURCE, + latent_pool_row=lambda i: OWNERS.index(KV_SOURCE[i]), + kv_owner_layers=OWNERS, + ) + + +def _first_step_md(name: str) -> SimpleNamespace: + """A first step's metadata on engine `name`: its quantities, a fresh KV cache manager and the + pool's 32-token pages.""" + return _md(name, kv_cache_manager=_manager(), tokens_per_block=32) + + +def _stub_builds(monkeypatch) -> list: + """Stand-ins for the kernel builds and the rope tables, and a device index: `_rope_tables` + takes exactly `(cfg, alpha, positions, dtype, device)` and records each table's rows.""" + built = [] + + def tables(cfg, alpha, positions, dtype, device): + built.append(positions) + return SimpleNamespace(rows=positions) + + monkeypatch.setattr(M, "_rope_tables", tables) + monkeypatch.setattr(M, "build_ix_q_side", lambda dev, is_neox: None) + monkeypatch.setattr(M, "build_ix_k_cache", lambda dev, is_neox, cache_contiguous: None) + monkeypatch.setattr( + M, "build_ix_topk_decode", lambda dev, next_n: SimpleNamespace(next_n=next_n) + ) + monkeypatch.setattr(torch.cuda, "current_device", lambda: 0) + return built + + +def _core(max_seq_len: int | None = None, world: int = 4): + """The core of rank 0 at `world` attention-DP ranks, on the meta device, built from the + checkpoint's params.json, as the native format reads it, with the engine's max_seq_len.""" + from tensorrt_llm._torch.model_config import ModelConfig + from tensorrt_llm._torch.models.checkpoints.mistral.config_loader import MistralConfigLoader + from tensorrt_llm.mapping import Mapping + + root = llm_models_root() + paths = [root / name for name in CHECKPOINTS] if root is not None else [] + path = next((p for p in paths if (p / "params.json").is_file()), None) + if path is None: + pytest.skip(f"none of {CHECKPOINTS} is under LLM_MODELS_ROOT ({root})") + mapping = Mapping( + world_size=world, + rank=0, + gpus_per_node=4, + tp_size=world, + pp_size=1, + moe_ep_size=world, + moe_tp_size=1, + enable_attention_dp=True, + ) + model_config = ModelConfig( + pretrained_config=MistralConfigLoader().load_pretrained_config(str(path)), + mapping=mapping, + max_num_tokens=32768, + max_seq_len=max_seq_len, + ) + with torch.device("meta"): + return M.MistralLarge4DepModelingV2(model_config) + + +def _check_sizes(core, name: str, world: int) -> None: + """The core's four engine sizes after `_derive_first_step` on engine `name`.""" + max_num_tokens, chunked = ENGINES[name][0], ENGINES[name][3] + _, rows, budget = SIZES[name] + call = CALLS[(chunked, world)] + assert core.prefill_logits_budget == budget + assert core.ctx_workspace_rows == rows + assert core.expert_call_tokens == call + assert core.expert_tune_tokens == min(call, world * max_num_tokens) + + +# ---- the engine's sizes ---- + + +@pytest.mark.parametrize("name", list(ENGINES)) +def test_workspace_rows_from_the_runtime_features(name): + """`_most_ctx_kv` and `_ctx_workspace_rows` on each engine's metadata: with neither chunked + prefill nor block reuse a step attends at most max_num_tokens of context KV (32768 at the 32K + config and at 131072 without either, 16384 at the Pareto config), so the held workspace is + twice that; with either, mainline's never-stall worst case min(max_num_requests, + max_num_tokens) x max_seq_len (2**19 at 32K with block reuse at max_batch_size 16, chunked + prefill or not, 2**24 at the 1M config, 2**21 at 131072 chunked, 32768 at one request per + step), the workspace capped at 2**19 rows.""" + assert M._CTX_WORKSPACE_ROWS == 1 << 19 + most, rows, _ = SIZES[name] + md = _md(name) + assert M._most_ctx_kv(md) == most + assert M._ctx_workspace_rows(md) == rows == min(2 * most, M._CTX_WORKSPACE_ROWS) + + +def test_block_reuse_is_read_from_the_runtime_features(): + """Block reuse is the attention metadata's runtime feature, `cache_reuse` (the executor builds + it from the KvCacheConfig's enable_block_reuse): at the 32K config, cache_reuse on gives + mainline's never-stall worst case, min(16, 32768) x 32768 = 2**19 rows of context KV and a + held workspace of 2**19 rows, off max_num_tokens and 65536 rows; with chunked prefill the + worst case either way. Nothing else is read: the metadata carries no KV cache manager.""" + for chunked, reuse, most in ( + (False, False, 32768), + (False, True, 1 << 19), + (True, False, 1 << 19), + (True, True, 1 << 19), + ): + md = _md("32k") + md.runtime_features = AttentionRuntimeFeatures( + chunked_prefill=chunked, cache_reuse=reuse, chunk_size=32768 + ) + assert not hasattr(md, "kv_cache_manager") + assert M._most_ctx_kv(md) == most, (chunked, reuse) + assert M._ctx_workspace_rows(md) == min(2 * most, M._CTX_WORKSPACE_ROWS), (chunked, reuse) + + +@pytest.mark.parametrize( + "max_num_tokens, max_seq_len, budget, tile", + [ + (32768, 32768, 1 << 24, 512), + (16384, 32768, 1 << 24, 512), + (32768, 65536, 1 << 26, 1024), + (32768, 131072, 1 << 26, 512), + (32768, 1048576, 1 << 26, 64), + (64, 32768, 1 << 21, 64), + ], + ids=["32k", "pareto", "65536", "131072", "1m", "64_tokens"], +) +def test_prefill_logits_budget(max_num_tokens, max_seq_len, budget, tile): + """The budget is the least of mainline's reachable tile (its 2**31 default within max_num_tokens + x max_seq_len), the ceiling 2**26 and max_seq_len x max_seq_len / 64: 2**24 at max_seq_len + 32768, so 512-row tiles at 32768 keys; the ceiling from 65536 on, 512-row tiles at 131072 keys + and 64 at 1048576; the reachable tile where max_num_tokens binds it (64 rows at 64 tokens).""" + assert M._PREFILL_LOGITS_CEILING == 1 << 26 + reachable = get_indexer_mqa_logits_workspace_bytes(max_num_tokens, max_seq_len) // 4 + got = M._prefill_logits_budget(max_num_tokens, max_seq_len) + assert got == budget == min(reachable, 1 << 26, max_seq_len * max_seq_len // 64) + assert got // max_seq_len == tile + + +def test_logits_budget_env_override_only_lowers(monkeypatch): + """TLLM_INDEXER_MQA_LOGITS_ELEM_BUDGET lowers mainline's reachable tile and so the budget at + every engine; set past the ceiling it raises nothing.""" + monkeypatch.setenv(_BUDGET_ENV, str(1 << 20)) + assert M._prefill_logits_budget(32768, 32768) == 1 << 20 + assert M._prefill_logits_budget(32768, 1048576) == 1 << 20 + monkeypatch.setenv(_BUDGET_ENV, str(1 << 40)) + assert M._prefill_logits_budget(32768, 32768) == 1 << 24 + assert M._prefill_logits_budget(32768, 1048576) == M._PREFILL_LOGITS_CEILING + + +@pytest.mark.parametrize("world", [4, 8], ids=["dep4", "dep8"]) +@pytest.mark.parametrize("max_seq_len", [8192, 32768, 32769, 1048576, None]) +def test_core_holds_no_engine_size_until_the_first_step(max_seq_len, world): + """Built at any engine max_seq_len (None: the engine infers one), the core chooses nothing by + it: its four engine sizes are 0 until `_derive_first_step`, and `shared_source` is + structural, layer 35 alone, the owner the NoPE layers attend over.""" + core = _core(max_seq_len, world) + assert not hasattr(core, "long_context") + # The fused attention glue at every max_seq_len, unless `_UNFUSED_GLUE`. + assert core.unfused_glue is M._UNFUSED_GLUE + assert core.kv_source == KV_SOURCE + assert [i for i, keep in enumerate(core.shared_source) if keep] == [SHARED_SOURCE] + sizes = (core.prefill_logits_budget, core.ctx_workspace_rows) + assert sizes + (core.expert_call_tokens, core.expert_tune_tokens) == (0, 0, 0, 0) + assert M._EXPERT_CALL_TOKENS == {4: 131072, 8: 32768} + assert M._EXPERT_CALL_TOKENS_CHUNKED == {4: 32768, 8: 32768} + assert not hasattr(M, "_EXPERT_CALL_TOKENS_CACHED") + + +@pytest.mark.parametrize("world", [4, 8], ids=["dep4", "dep8"]) +@pytest.mark.parametrize("name", list(ENGINES)) +def test_first_step_sizes_the_core(monkeypatch, name, world): + """`_derive_first_step` on the real core (meta device) sets the logits budget, the held + workspace's rows and the expert path's call and autotuner bucket from the first step's + metadata: at four ranks one call of 131072 without chunked prefill, block reuse or not (the + autotuner's bucket every rank's max_num_tokens gathered: 65536 at the Pareto config), 32768 + with it; at eight ranks 32768 either way.""" + _stub_builds(monkeypatch) + core = _core(world=world) + core.build_kernels(0) + core._derive_first_step(_first_step_md(name)) + _check_sizes(core, name, world) + assert core._stride_factor == len(OWNERS) * 32 and core._max_ctx_len == ENGINES[name][2] + + +@pytest.mark.parametrize("world", [4, 8], ids=["dep4", "dep8"]) +def test_expert_calls_keyed_on_chunked_prefill_alone(monkeypatch, world): + """The expert path's call follows the engine's chunked prefill flag alone: without it + `_EXPERT_CALL_TOKENS[world]` (131072 at four ranks, 32768 at eight), block reuse or not -- a + 32K step under block reuse, which may attend 2**19 rows of context KV (a held workspace of + 2**19 rows), still makes one call of every rank's 32768 rows at four -- and with it + `_EXPERT_CALL_TOKENS_CHUNKED[world]` (32768 at both), at max_seq_len 32768, 131072 and + 1048576, block reuse or not, and at one request per step, whose context KV is + max_num_tokens. The autotuner's bucket is the call within every rank's max_num_tokens + gathered (65536 at the Pareto config's 16384 at four ranks). On the real core (meta device), + each engine with a KV cache manager of its own.""" + _stub_builds(monkeypatch) + core = _core(world=world) + core.build_kernels(0) + unchunked, chunked = M._EXPERT_CALL_TOKENS[world], M._EXPERT_CALL_TOKENS_CHUNKED[world] + assert (unchunked, chunked) == ({4: 131072, 8: 32768}[world], 32768) + # (engine, its runtime features' chunked prefill and block reuse) + for name, chunked_prefill, block_reuse in ( + ("32k", False, False), + ("32k_block_reuse", False, True), + ("pareto", False, False), + ("131072", False, False), + ("32k_chunked_one_request", True, False), + ("32k_chunked_block_reuse", True, True), + ("131072_chunked", True, False), + ("1m", True, False), + ): + md = _first_step_md(name) + assert md.runtime_features.chunked_prefill is chunked_prefill + assert md.runtime_features.cache_reuse is block_reuse + call = chunked if chunked_prefill else unchunked + core._derive_first_step(md) + assert core.expert_call_tokens == call, name + assert core.expert_tune_tokens == min(call, world * md.max_num_tokens), name + assert core.ctx_workspace_rows == SIZES[name][1], name + + +def test_a_replaced_kv_cache_manager_sizes_the_core_again(monkeypatch): + """The sizes are fixed once per KV cache manager: a later step on the same manager changes + none of them, a replaced manager derives them again from its step's metadata.""" + _stub_builds(monkeypatch) + core = _core() + core.build_kernels(0) + first = _first_step_md("32k") + core._derive_first_step(first) + _check_sizes(core, "32k", 4) + core._derive_first_step(_md("1m", kv_cache_manager=first.kv_cache_manager, tokens_per_block=32)) + _check_sizes(core, "32k", 4) + core._derive_first_step(_first_step_md("1m")) + _check_sizes(core, "1m", 4) + + +def test_another_world_is_refused(): + """The expert call tables hold four and eight ranks only: two ranks are refused before + anything is built, whatever the engine's max_seq_len.""" + for max_seq_len in (32768, 1048576): + with pytest.raises(ValueError, match="serves 4 ranks on one node or 8"): + _core(max_seq_len, 2) + + +# ---- the rope tables ---- + + +def test_rope_tables_hold_no_mla_table(): + """`_rope_tables` builds `rows`, the indexer's `[cos | sin]` (which `ds_mla_rope_append` + reads too) and the NoPE identity table, and takes no `with_mla`.""" + alpha = torch.linspace(-0.5, 0.5, 32) + tables = M._rope_tables(ROPE_CFG, alpha, 300, torch.bfloat16, "cpu") + assert set(vars(tables)) == {"rows", "indexer", "identity"} + assert tables.rows == 300 and tables.indexer.shape == tables.identity.shape == (300, 64) + with pytest.raises(TypeError): + M._rope_tables(ROPE_CFG, alpha, 300, torch.bfloat16, "cpu", with_mla=True) + + +def test_build_kernels_and_first_step_build_the_rope_tables(monkeypatch): + """`build_kernels` builds the tables at the core's first size and `_derive_first_step` grows + them to the engine's max_seq_len, both through `(cfg, alpha, positions, dtype, device)` + alone; a replaced manager at a shorter max_seq_len keeps them. On a core holding just what + the two read, sized like any core.""" + built = _stub_builds(monkeypatch) + core = SimpleNamespace( + model_config=SimpleNamespace(pretrained_config=ROPE_CFG), + w={"lerope_alpha": None, "final_norm": torch.empty(0, dtype=torch.bfloat16)}, + rope_rows=16, + kv_source=KV_SOURCE, + num_layers=LAYERS, + dp_group=[0, 1, 2, 3], + _derived_for=None, + _ix_k=None, + unfused_glue=False, + ) + M.MistralLarge4DepModelingV2.build_kernels(core, 0) + md = _first_step_md("1m") + M.MistralLarge4DepModelingV2._derive_first_step(core, md) + assert built == [16, 1048576] and core._rope.rows == core.rope_rows == 1048576 + _check_sizes(core, "1m", 4) + M.MistralLarge4DepModelingV2._derive_first_step(core, _first_step_md("32k")) + assert built == [16, 1048576] and core._rope.rows == 1048576 + _check_sizes(core, "32k", 4) + + +# ---- the decode top-k's selector + + +#: Engine max_seq_lens, the decode scorer's column count, and whether cute_dsl_radix_topk_decode +#: serves it (`_TOPK_DECODE_COLUMNS`, 4096..65536): either side of both ends, the 32K and the 1M +#: configs and 131072. +TOPK_WIDTHS = [ + (2048, False), + (4095, False), + (4096, True), + (8192, True), + (32768, True), + (65536, True), + (65537, False), + (131072, False), + (1048576, False), +] + + +def _topk_md(max_seq_len: int, chunked: bool, reuse: bool) -> SimpleNamespace: + """A first step's metadata at `max_seq_len` (32768 tokens, 16 requests) on a fresh KV cache + manager, with chunked prefill and block reuse as given.""" + return SimpleNamespace( + max_num_tokens=32768, + max_num_requests=16, + max_seq_len=max_seq_len, + runtime_features=AttentionRuntimeFeatures( + chunked_prefill=chunked, cache_reuse=reuse, chunk_size=32768 + ), + kv_cache_manager=SimpleNamespace( + latent_layout=M._LATENT_LAYOUT, + kv_source_layers=KV_SOURCE, + latent_pool_row=lambda i: OWNERS.index(KV_SOURCE[i]), + kv_owner_layers=OWNERS, + ), + tokens_per_block=32, + ) + + +@pytest.mark.parametrize("max_seq_len, serves", TOPK_WIDTHS, ids=[str(w) for w, _ in TOPK_WIDTHS]) +def test_decode_topk_selector_by_the_scorer_width(monkeypatch, max_seq_len, serves): + """`_derive_first_step` holds cute_dsl_radix_topk_decode's handle exactly when the decode + scorer's column count, the engine's max_seq_len, is one the selector serves (4096..65536); + otherwise none, and both decode call sites run the trtllm indexer_topk_decode op. Nothing + else decides it: chunked prefill and block reuse, on or off, keep the one handle, built + once. A replaced KV cache manager at a width on the other side of the range flips it, and + `_TOPK_DECODE_CUTE_DSL` False builds none at any width.""" + assert M._TOPK_DECODE_CUTE_DSL is True + assert M._TOPK_DECODE_COLUMNS == (4096, 65536) + calls = [] + monkeypatch.setattr(M, "_rope_tables", lambda *a: SimpleNamespace(rows=a[2])) + monkeypatch.setattr(M, "build_ix_k_cache", lambda dev, is_neox, cache_contiguous: None) + monkeypatch.setattr( + M, "build_ix_topk_decode", lambda dev, next_n: calls.append(next_n) or object() + ) + monkeypatch.setattr(torch.cuda, "current_device", lambda: 0) + core = SimpleNamespace( + model_config=SimpleNamespace(pretrained_config=ROPE_CFG), + w={"lerope_alpha": None, "final_norm": torch.empty(0, dtype=torch.bfloat16)}, + rope_rows=16, + kv_source=KV_SOURCE, + num_layers=LAYERS, + dp_group=[0, 1, 2, 3], + _derived_for=None, + _ix_k=None, + _ix_topk=None, + _rope=SimpleNamespace(rows=16), + ) + derive = M.MistralLarge4DepModelingV2._derive_first_step + handles = [] + for chunked, reuse in ((False, False), (True, False), (False, True), (True, True)): + derive(core, _topk_md(max_seq_len, chunked, reuse)) + handles.append(core._ix_topk) + if serves: + assert calls == [1] and handles[0] is not None + assert all(h is handles[0] for h in handles) + else: + assert calls == [] and handles == [None] * 4 + derive(core, _topk_md(1048576 if serves else 32768, False, False)) + assert (core._ix_topk is None) is serves + monkeypatch.setattr(M, "_TOPK_DECODE_CUTE_DSL", False) + derive(core, _topk_md(32768, False, False)) + assert core._ix_topk is None + + +# ---- the indexer's context selection ---- + + +def _packed(): + """Two context requests of 40 cached + 8 new and 30 cached + 6 new tokens, packed by the + engine into one prefill chunk of 14 rows over 84 keys: the rows' windows `[ks, ke)` in the + chunk's keys, and the metadata's host prefix sums.""" + ks = torch.tensor([0] * 8 + [48] * 6, dtype=torch.int32) + ke = torch.tensor([*range(41, 49), *range(79, 85)], dtype=torch.int32) + chunk = SimpleNamespace( + token_start=0, + token_end=14, + k_token_start=0, + k_token_end=84, + cu_seqlen_ks=ks, + cu_seqlen_ke=ke, + ) + md = SimpleNamespace( + enable_context_mla_with_cached_kv=True, + host_ctx_uncached_token_indptr=torch.tensor([0, 8, 14]), + host_ctx_kv_indptr=torch.tensor([0, 48, 84]), + ) + return chunk, md + + +def test_ctx_offsets_device_and_host(): + """`_ctx_kv_offsets` hands over the device offsets the context workspace is laid out by; + `_ctx_host_offsets` the host prefix sums the indexer's blocks are cut by: the engine's host + indptrs under context MLA with cached KV, the prefix sum of `seq_lens` without it.""" + md = SimpleNamespace( + enable_context_mla_with_cached_kv=False, + mla_prepare_ctx_cu_seqlens=lambda: torch.tensor([0, 8, 14], dtype=torch.int32), + seq_lens=torch.tensor([8, 6, 1], dtype=torch.int32), + ) + assert set(M._ctx_kv_offsets(md, 2)) == { + "cu_ctx_cached", + "cu_ctx_kv", + "max_ctx_new", + "ctx_cached_tokens", + } + assert M._ctx_host_offsets(md, 2) == dict(host_cu_ctx_new=[0, 8, 14], host_cu_ctx_kv=[0, 8, 14]) + _, md = _packed() + assert M._ctx_host_offsets(md, 2) == dict( + host_cu_ctx_new=[0, 8, 14], host_cu_ctx_kv=[0, 48, 84] + ) + + +def _recorded_kernels(monkeypatch) -> list: + """Stand-ins for the context selection's kernels, recording each key gather's + `(k_token_start, n_k)`, each scored tile's rows and keys and whether the previous tile's + logits are still alive when it is scored, and each top-k call's windows. None writes the + selection. Each tile's split-KV schedule is a marker of the windows and keys it was built + for, which its logits call must be handed (checked here), and the top-k's scratch a + marker the target must hand every top-k call.""" + calls = [] + previous = [] + + def gather(pool, fp8, scale, k_token_start, n_k, dim): + calls.append(("gather", k_token_start, n_k)) + return torch.empty(n_k, dim), torch.empty(n_k, 1) + + def schedule(ks, ke, n_k, heads): + return ("schedule", ks.tolist(), ke.tolist(), n_k, heads) + + def logits(q, kv, weights, ks, ke, schedule_meta=None): + assert schedule_meta == ("schedule", ks.tolist(), ke.tolist(), kv[0].shape[0], q.shape[1]) + alive = previous[-1]() is not None if previous else None + calls.append(("logits", q.shape[0], kv[0].shape[0], alive)) + out = torch.empty(1) + previous.append(weakref.ref(out)) + return out + + def topk(scores, ks, ke, out, k, workspace=None): + assert workspace == "gvr-scratch" + calls.append(("topk", ks.tolist(), ke.tolist())) + + monkeypatch.setattr(M, "indexer_k_cache_gather_op", gather) + monkeypatch.setattr(M, "mqa_logits_schedule", schedule) + monkeypatch.setattr(M, "fp8_mqa_logits", logits) + monkeypatch.setattr(M, "cute_dsl_gvr_topk_prefill", topk) + monkeypatch.setattr(M, "gvr_workspace", lambda device: "gvr-scratch") + monkeypatch.setattr(M, "gvr_warmup", lambda top_k, cols, elems, device, ws: 0) + return calls + + +def _prefill_target(**core) -> M.PrefillTarget: + target = M.PrefillTarget.__new__(M.PrefillTarget) + target.core = SimpleNamespace(**core) + return target + + +def _select_step(chunks) -> dict: + """What `_select_ctx` reads of a step over `_packed`'s two context requests (no decode + rows), the metadata's host prefix sums included, as `step_args` always carries them.""" + chunk, md = _packed() + return dict( + indexer_prefill_chunks=chunks, + num_ctx_kv_tokens=84, + cu_seqlen_ks=chunk.cu_seqlen_ks, + cu_seqlen_ke=chunk.cu_seqlen_ke, + slot_mapping_fp8_fullkv=None, + slot_mapping_scale_fullkv=None, + **M._ctx_host_offsets(md, 2), + ) + + +def _select_target(max_seq_len: int, budget: int = 1 << 24) -> M.PrefillTarget: + return _prefill_target( + idx_dim=128, idx_topk=4, prefill_logits_budget=budget, _max_ctx_len=max_seq_len + ) + + +@pytest.mark.parametrize("max_seq_len", [64, 84, 128], ids=["past", "at", "within"]) +def test_select_ctx_cuts_a_chunk_only_past_max_seq_len(monkeypatch, max_seq_len): + """The engine's chunk of two requests over 84 keys: past a max_seq_len of 64 it is cut at its + requests, each request's keys gathered alone and its windows rebased to its first key; at + max_seq_len 84 or 128 it is scored whole, one gather over its 84 keys and the chunk's own + windows.""" + calls = _recorded_kernels(monkeypatch) + chunk, _ = _packed() + out = torch.empty(14, 4, dtype=torch.int32) + _select_target(max_seq_len)._select_ctx( + _select_step([chunk]), None, torch.empty(14, 1), torch.empty(14, 1), out, 14 + ) + gathers = [c[1:] for c in calls if c[0] == "gather"] + windows = [c[1:] for c in calls if c[0] == "topk"] + if max_seq_len < 84: + assert gathers == [(0, 48), (48, 36)] + assert windows == [([0] * 8, list(range(41, 49))), ([0] * 6, list(range(31, 37)))] + return + assert gathers == [(0, 84)] + assert windows == [(chunk.cu_seqlen_ks.tolist(), chunk.cu_seqlen_ke.tolist())] + + +def test_select_ctx_fills_a_block_with_no_key(monkeypatch): + """A chunk whose rows have no key leaves every slot of those rows -1, gathering and scoring + nothing for them; the next chunk's rows are selected as usual.""" + calls = _recorded_kernels(monkeypatch) + empty = SimpleNamespace(token_start=0, token_end=3, k_token_start=0, k_token_end=0) + keyed = SimpleNamespace( + token_start=3, + token_end=6, + k_token_start=0, + k_token_end=10, + cu_seqlen_ks=torch.zeros(3, dtype=torch.int32), + cu_seqlen_ke=torch.tensor([8, 9, 10], dtype=torch.int32), + ) + step = dict( + indexer_prefill_chunks=[empty, keyed], + slot_mapping_fp8_fullkv=None, + slot_mapping_scale_fullkv=None, + host_cu_ctx_new=[0, 6], + host_cu_ctx_kv=[0, 10], + ) + out = torch.full((6, 4), 7, dtype=torch.int32) + _select_target(64)._select_ctx(step, None, torch.empty(6, 1), torch.empty(6, 1), out, 6) + assert out[:3].eq(-1).all() and out[3:].eq(7).all() + assert [c[:3] for c in calls if c[0] != "topk"] == [("gather", 0, 10), ("logits", 3, 10)] + + +@pytest.mark.parametrize("engine", ["32k", "1m", "env_2**20"]) +def test_select_ctx_tiles_under_the_engines_budget(monkeypatch, engine): + """2049 context rows over 32768 keys, one unplanned chunk, in tiles of the budget over the + keys: 512 rows at the 32K config's 2**24, 2048 at the 1M config's 2**26, 32 under + TLLM_INDEXER_MQA_LOGITS_ELEM_BUDGET 2**20; each tile's logits freed before the next tile is + scored.""" + if engine == "env_2**20": + monkeypatch.setenv(_BUDGET_ENV, str(1 << 20)) + max_seq_len = {"32k": 32768, "1m": 1048576, "env_2**20": 32768}[engine] + budget = M._prefill_logits_budget(32768, max_seq_len) + tile = {"32k": 512, "1m": 2048, "env_2**20": 32}[engine] + calls = _recorded_kernels(monkeypatch) + rows, keys = 2049, 32768 + step = dict( + indexer_prefill_chunks=None, + num_ctx_kv_tokens=keys, + cu_seqlen_ks=torch.zeros(rows, dtype=torch.int32), + cu_seqlen_ke=torch.arange(keys - rows + 1, keys + 1, dtype=torch.int32), + slot_mapping_fp8_fullkv=None, + slot_mapping_scale_fullkv=None, + host_cu_ctx_new=[0, rows], + host_cu_ctx_kv=[0, keys], + ) + target = _select_target(max_seq_len, budget) + target._select_ctx( + step, None, torch.empty(rows, 1), torch.empty(rows, 1), torch.empty(rows, 1), rows + ) + tiles = [c[1:] for c in calls if c[0] == "logits"] + assert budget // keys == tile and calls[0] == ("gather", 0, keys) + full, last = divmod(rows, tile) + expected = [(tile, keys, False)] * full + [(last, keys, False)] * bool(last) + assert tiles == [(tile, keys, None)] + expected[1:] + + +def _step_md(cached_kv: bool) -> SimpleNamespace: + """A step's metadata of two context requests, as `PrefillTarget.step_args` reads it: under + context MLA with cached KV `_packed`'s two (40 + 8 and 30 + 6 tokens, 84 keys), the engine's + host indptrs beside its device ones; without it two fresh prompts of 8 and 6 tokens, whose + host prefix sums are those of `seq_lens`.""" + i64 = dict(dtype=torch.int64) + if cached_kv: + chunk, host = _packed() + phase = dict( + num_ctx_kv_tokens=14, + enable_context_mla_with_cached_kv=True, + ctx_cached_token_indptr=torch.tensor([0, 40, 70], **i64), + ctx_kv_indptr=torch.tensor([0, 48, 84], **i64), + max_ctx_seq_len=8, + num_ctx_cached_tokens=70, + host_ctx_uncached_token_indptr=host.host_ctx_uncached_token_indptr, + host_ctx_kv_indptr=host.host_ctx_kv_indptr, + ) + else: + chunk = SimpleNamespace( + token_start=0, + token_end=14, + k_token_start=0, + k_token_end=14, + cu_seqlen_ks=torch.tensor([0] * 8 + [8] * 6, dtype=torch.int32), + cu_seqlen_ke=torch.tensor([*range(1, 9), *range(9, 15)], dtype=torch.int32), + ) + phase = dict( + num_ctx_kv_tokens=14, + enable_context_mla_with_cached_kv=False, + mla_prepare_ctx_cu_seqlens=lambda: torch.tensor([0, 8, 14], dtype=torch.int32), + ) + return SimpleNamespace( + kv_cache_manager=SimpleNamespace( + get_latent_pool=lambda: torch.zeros(2, 3, 32, 656, dtype=torch.uint8), + get_indexer_k_cache_buffers=lambda i: None, + ), + num_seqs=2, + num_tokens=14, + all_rank_num_tokens=[14, 1, 1, 1], + kv_lens_cuda_runtime=None, + kv_lens_runtime=None, + prompt_lens_cpu_runtime=None, + kv_cache_block_offsets=None, + host_kv_cache_pool_pointers=None, + host_kv_cache_pool_mapping=None, + tokens_per_block=32, + max_seq_len=64, + block_table=None, + req_idx_per_token=None, + slot_mapping_fp8=None, + slot_mapping_scale=None, + skip_indexer_for_gen_reqs=False, + enable_indexer_skip=False, + indexer_k_cache_block_offsets=None, + scheduler_metadata_buffer=None, + radix_aux_indices=None, + radix_aux_logits=None, + num_contexts=2, + num_ctx_tokens=14, + seq_lens=torch.tensor([8, 6], dtype=torch.int32), + skip_indexer_for_ctx_reqs=False, + cu_seqlen_ks=chunk.cu_seqlen_ks, + cu_seqlen_ke=chunk.cu_seqlen_ke, + indexer_prefill_q_split=None, + indexer_prefill_chunks=[chunk], + slot_mapping_fp8_fullkv=None, + slot_mapping_scale_fullkv=None, + **phase, + ) + + +@pytest.mark.parametrize("cached_kv", [True, False], ids=["cached_kv", "fresh"]) +def test_step_args_always_carries_the_host_prefix_sums(monkeypatch, cached_kv): + """`step_args` hands over the host prefix sums on every context step, from the engine's host + indptrs under context MLA with cached KV and from `seq_lens` without it, and the selection + cuts the step's chunk at its requests by them once it holds more keys than max_seq_len: + 84 keys past 64, and 14 fresh keys past a max_seq_len of 10.""" + md = _step_md(cached_kv) + target = _select_target(64 if cached_kv else 10) + target.core.indexer_layers = [1] + step = target.step_args(md) + assert step["indexer_prefill_chunks"] is md.indexer_prefill_chunks + if cached_kv: + assert step["ctx_cached_tokens"] == 70 and int(step["cu_ctx_kv"][-1]) == 84 + assert (step["host_cu_ctx_new"], step["host_cu_ctx_kv"]) == ([0, 8, 14], [0, 48, 84]) + else: + assert step["ctx_cached_tokens"] == 0 and int(step["cu_ctx_kv"][-1]) == 14 + assert (step["host_cu_ctx_new"], step["host_cu_ctx_kv"]) == ([0, 8, 14], [0, 8, 14]) + calls = _recorded_kernels(monkeypatch) + out = torch.empty(14, 4, dtype=torch.int32) + target._select_ctx(step, None, torch.empty(14, 1), torch.empty(14, 1), out, 14) + gathers = [c[1:] for c in calls if c[0] == "gather"] + windows = [c[1:] for c in calls if c[0] == "topk"] + assert gathers == ([(0, 48), (48, 36)] if cached_kv else [(0, 8), (8, 6)]) + if not cached_kv: + assert windows == [([0] * 8, list(range(1, 9))), ([0] * 6, list(range(1, 7)))] + + +# ---- the context workspace and the context attention ---- + + +def _attn_target(held: int) -> M.PrefillTarget: + """A prefill target over what its context workspace and attention read off the core: a held + workspace of `held` rows (`ctx_workspace_rows`, small here to reach each path cheaply), and + `shared_source` as every core holds it (layer 35 alone, + `test_core_holds_no_engine_size_until_the_first_step`).""" + return _prefill_target( + lat_dim=LAT_DIM, + ctx_workspace_rows=held, + shared_source=[i == SHARED_SOURCE for i in range(LAYERS)], + softmax_scale=1.0, + idx_topk=4, + _pool_row=list(range(LAYERS)), + _stride_factor=1, + ) + + +def _ctx_step(contexts) -> dict: + """What `_ctx_workspace` and `_attn_ctx` read of a step of context requests of (cached, new) + tokens each.""" + kv = [c + n for c, n in contexts] + return dict( + num_contexts=len(contexts), + num_ctx_tokens=sum(n for _, n in contexts), + ctx_cached_tokens=sum(c for c, _ in contexts), + cu_ctx_kv=torch.tensor([0, *kv]).cumsum(0), + latent_pool=None, + latent_kv=None, + block_table=None, + tokens_per_block=32, + ) + + +def _rows_of(view: torch.Tensor, buf: torch.Tensor) -> tuple[int, int]: + """The held buffer's rows `[lo, hi)` a view of it covers.""" + assert view.untyped_storage().data_ptr() == buf.untyped_storage().data_ptr() + lo = (view.data_ptr() - buf.data_ptr()) // (LAT_DIM * buf.element_size()) + return lo, lo + view.shape[0] + + +@pytest.mark.parametrize( + "contexts, path", + [ + ([(7, 3), (0, 6)], "kept"), + ([(14, 6)], "kept"), + ([(15, 6)], "workspace"), + ([(30, 10)], "workspace"), + ([(31, 10)], "decode"), + ], + ids=["n_kv16", "n_kv20", "n_kv21", "n_kv40", "n_kv41"], +) +def test_context_path_by_n_kv(contexts, path): + """Over a held workspace of 40 rows: a step's context KV of 16 or 20 rows fits it twice (the + step's rows first, then as many kept), 21 or 40 once (no kept rows), 41 not at all (the + decode path, every context row a one-row sequence, no workspace). The held workspace is the + core's `ctx_workspace_rows` x 576 bf16, allocated on the step.""" + target = _attn_target(40) + step = _ctx_step(contexts) + tc = step["num_ctx_tokens"] + n_kv = step["ctx_cached_tokens"] + tc + ctx = target._ctx_workspace(step, tc, torch.device("cpu")) + buf = target._ws_buf + assert buf.shape == (40, LAT_DIM) and buf.dtype == torch.bfloat16 + if path == "decode": + assert ctx.ws_buf is None and not hasattr(ctx, "kv_req") + assert ctx.sub_batches == [(0, tc)] and sorted(ctx.sched) == [tc] + return + assert _rows_of(ctx.ws_buf, buf) == (0, n_kv) and ctx.ws == {} + if path == "kept": + assert _rows_of(ctx.kept, buf) == (n_kv, 2 * n_kv) + else: + assert ctx.kept is None + kv = [c + n for c, n in contexts] + assert ctx.kv_req.tolist() == [r for r, n in enumerate(kv) for _ in range(n)] + assert ctx.kv_pos.tolist() == [p for n in kv for p in range(n)] + assert ctx.ws_off.tolist() == [0, *torch.tensor(kv).cumsum(0).tolist()][:-1] + + +def test_decode_path_in_sub_batches(monkeypatch): + """20000 context rows past the held rows: the selections become pool slots once, then the + sparse decode runs in sub-batches of 8192, 8192 and 3616 rows on their sizes' schedulers, and + nothing is gathered.""" + calls = [] + monkeypatch.setattr( + M, "ds_mla_dequant_gather", lambda *args: pytest.fail("the decode path gathered") + ) + monkeypatch.setattr( + M, + "convert_req_index_to_global", + lambda req, table, sel, tpb, topk, stride, row: calls.append(("slots", row)) or sel, + ) + monkeypatch.setattr( + M, + "flash_mla_sparse_decode", + lambda q, kv, slots, sched, scale, out: calls.append(("decode", q.shape[0], sched)), + ) + target = _attn_target(40) + step = _ctx_step([(100, 20000)]) + ctx = target._ctx_workspace(step, 20000, torch.device("cpu")) + assert ctx.ws_buf is None + assert ctx.sub_batches == [(0, 8192), (8192, 16384), (16384, 20000)] + assert sorted(ctx.sched) == [3616, 8192] + L = SimpleNamespace(kv_source=KV_SOURCE[39]) + sel = torch.zeros(20000, 4, dtype=torch.int32) + target._attn_ctx( + step, L, torch.empty(20000, 1), torch.empty(20000, 1), torch.zeros(20000), sel, ctx + ) + assert calls == [ + ("slots", SHARED_SOURCE), + ("decode", 8192, ctx.sched[8192]), + ("decode", 8192, ctx.sched[8192]), + ("decode", 3616, ctx.sched[3616]), + ] + + +@pytest.mark.parametrize("held, kept", [(40, True), (24, False)], ids=["fits_twice", "fits_once"]) +def test_layer_35_kept_or_gathered_again_by_fit(monkeypatch, held, kept): + """Layers 35, 39 (attending over 35's latent), 40 and 43 (over 35's) on one step of two + context requests of 10 and 6 KV rows. Over 40 held rows the step fits twice: layer 35 is + gathered once into the kept rows, which 39 and 43 read without a gather, and 40 into the + step's own rows. Over 24 it fits once: every layer gathers its source's rows into the step's + rows, 35's again for 39 and for 43, and nothing is kept.""" + gathers, prefills = [], [] + + def gather(pool, row, kv_req, kv_pos, block_table, out): + gathers.append((row, out)) + return out + + monkeypatch.setattr(M, "ds_mla_dequant_gather", gather) + monkeypatch.setattr( + M, "flash_mla_sparse_prefill", lambda q, ws, idx, scale, out: prefills.append(ws) + ) + target = _attn_target(held) + step = _ctx_step([(7, 3), (0, 6)]) + ctx = target._ctx_workspace(step, 9, torch.device("cpu")) + buf = target._ws_buf + assert _rows_of(ctx.ws_buf, buf) == (0, 16) and ctx.ws == {} + req_idx = torch.tensor([0] * 3 + [1] * 6) + sel = torch.zeros(9, 4, dtype=torch.int32) + for layer in (SHARED_SOURCE, 39, 40, 43): + L = SimpleNamespace(kv_source=KV_SOURCE[layer]) + target._attn_ctx(step, L, torch.empty(9, 1), torch.empty(9, 1), req_idx, sel, ctx) + + if not kept: + assert ctx.kept is None + assert [row for row, _ in gathers] == [35, 35, 40, 35] + assert all(out is ctx.ws_buf for _, out in gathers) + assert all(ws is ctx.ws_buf for ws in prefills) and ctx.ws == {} + return + assert _rows_of(ctx.kept, buf) == (16, 32) + assert [row for row, _ in gathers] == [35, 40] + assert gathers[0][1] is ctx.kept and gathers[1][1] is ctx.ws_buf + assert ctx.ws == {SHARED_SOURCE: ctx.kept} + assert [ws is ctx.kept for ws in prefills] == [True, True, False, True] + assert prefills[2] is ctx.ws_buf + + +def test_workspace_is_allocated_once_and_held(): + """The held workspace is allocated on the first context step, here one on the decode path, + and every later step's rows, kept rows included, are that buffer's: 16 KV rows (kept), 30 + (fits once), 41 (decode path) and 5 (kept) over 40 held rows.""" + target = _attn_target(40) + assert target._ws_buf is None + cpu = torch.device("cpu") + ctx = target._ctx_workspace(_ctx_step([(35, 6)]), 6, cpu) + buf = target._ws_buf + assert ctx.ws_buf is None and buf.shape == (40, LAT_DIM) + for contexts, rows, kept in ( + ([(10, 6)], (0, 16), (16, 32)), + ([(20, 10)], (0, 30), None), + ([(1, 40)], None, None), + ([(0, 2), (0, 3)], (0, 5), (5, 10)), + ): + step = _ctx_step(contexts) + ctx = target._ctx_workspace(step, step["num_ctx_tokens"], cpu) + assert target._ws_buf is buf + if rows is None: + assert ctx.ws_buf is None + continue + assert _rows_of(ctx.ws_buf, buf) == rows + assert (ctx.kept if kept is None else _rows_of(ctx.kept, buf)) == kept diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_weight_manifest.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_weight_manifest.py new file mode 100644 index 000000000000..84c2ebf67a6e --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__dep/test_modeling_v2_weight_manifest.py @@ -0,0 +1,1398 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The Mistral Large 4 NVFP4 dep4 weight table against the checkpoint it reads, at both of the worlds +it serves: four ranks on one node, and eight on two nodes of four (dep8; rank r is local rank +r % 4 on node r // 4); attention data parallel, the routed experts expert-parallel over every +rank. + +No GPU. Three kinds of evidence: + +* the table at the checkpoint's own language-model configuration (written out below, and + compared with the configuration adapted from the checkpoint's `params.json` when it is + present): which layers carry which weight at what per-rank shape -- every head and the full + MLP widths on every rank -- and that each rank's manifest reads only its own routed experts + (32 at four ranks, 16 at eight) and the same replicated rest, the windows tiling the 128 + experts exactly once, at eight ranks the first four on node 0 and the last four on node 1; +* the checkpoint itself, through its index and safetensors headers, as meta tensors, plus the + values of its small scale tensors: what each key stores; the union of the ranks' sources is + exactly the index's keys but the DSpark draft's, which every rank declares unconsumed; each + rank consumes all 1,721 language-model keys and all 276 vision-tower keys whole and its + window's expert keys, 27,264 and 159.86 GiB in all at four ranks, 13,632 and 92.46 GiB at + eight, 3.17 GiB of it the vision tower; the production `load` -- every transform, shape and + dtype assert, the expert-scalar asserts on the checkpoint's own scalars and the two-way + coverage assert -- passes on each rank, reading each key once and whole, beyond the manifest + only the other ranks' expert scalars, for the asserts (163,584 B at four ranks, 190,848 B at + eight); at eight ranks each node's four ranks read 294.65 GiB of distinct bytes; and every + FP8 block scale and every expert scalar satisfies the invariants `load` asserts. Skipped + where the checkpoint is absent (looked up as `mistral-large-4-nvfp4` or + `Mistral-Large-4-1T-A52B-NVFP4` under `LLM_MODELS_ROOT`); +* a scaled-down configuration with real CPU tensors, loaded on every rank and compared + element for element with an independent construction of what each parameter must hold: + the replicated weights whole and identical on every rank, the `wkv_b` regroup, both half + orders, the packed scale operand, the expert window in the MoE runner's layout and its + scalars, the whole `lm_head`, and the DSpark draft tensors left alone; plus what the + streaming path reads off the shard files, and each assert's refusal. +""" + +from __future__ import annotations + +import json +import re +import struct +from pathlib import Path +from types import SimpleNamespace + +import pytest +import torch +import torch.nn as nn +from safetensors import safe_open +from test_common.llm_data import llm_models_root + +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_103__dep import ( # noqa: E501 + weights as W, +) +from tensorrt_llm._torch.models.checkpoints.mistral.config_loader import MistralConfigLoader +from tensorrt_llm.mapping import Mapping +from tensorrt_llm.quantization.utils.fp8_utils import ( + check_sf_layout, + inverse_transform_sf, + transform_sf_into_required_layout, +) + +pytestmark = pytest.mark.cpu_only + +#: The worlds the table serves: four ranks on one node, eight over two nodes of four (dep8). +WORLDS = (4, 8) +#: Devices per GB300 node on this cluster: one node holds four ranks, two hold eight. +GPUS_PER_NODE = 4 +CHECKPOINTS = ("mistral-large-4-nvfp4", "Mistral-Large-4-1T-A52B-NVFP4") + +_LAYERS = 72 +_EXPERTS = 128 + +#: Each world, and each rank of each world. +_each_world = pytest.mark.parametrize("world", WORLDS, ids=lambda w: f"world{w}") +_each_rank = pytest.mark.parametrize( + "world, rank", + [(w, r) for w in WORLDS for r in range(w)], + ids=[f"world{w}-rank{r}" for w in WORLDS for r in range(w)], +) + +#: Per world, a rank's declared routed experts and all its language-model parameters with the +#: whole lm_head (GiB), as the table declares them at `_CONFIG`. +_DECLARED_GIB = {4: (134.79, 156.80), 8: (67.39, 89.41)} +#: The BF16 vision tower -- encoder, pre-projector norm, patch merger and language adapter -- +#: which every rank reads whole: its keys and bytes in the checkpoint (3.17 GiB). +_VISION_KEYS, _VISION_BYTES = 276, 3_408_437_248 +#: The DSpark draft's keys, all in the checkpoint's last shard, which no rank reads. +_DRAFT_KEYS = 38 +#: Per world, what one rank reads off the checkpoint: its window of routed experts (GiB), the +#: window with the replicated rest, the vision tower among it (bytes and GiB), and what the +#: production `load` reads in all -- that and the other ranks' expert scalars, for its asserts +#: -- in keys and bytes. +_READS = { + 4: dict(window_gib=134.79, total=171_644_394_304, total_gib=159.86, keys=70157), + 8: dict(window_gib=67.39, total=99_280_040_128, total_gib=92.46, keys=63341), +} +_LOAD_READ = {4: 171_644_557_888, 8: 99_280_230_976} + +#: The language-model configuration surface the table reads, at the checkpoint's values. The +#: checkpoint's configuration adds its vision encoder, whose rows are checked against the +#: checkpoint below. +_CONFIG = dict( + num_hidden_layers=_LAYERS, + hidden_size=6144, + num_attention_heads=64, + qk_nope_head_dim=64, + qk_rope_head_dim=64, + v_head_dim=128, + kv_lora_rank=512, + nope_layer_flags=[i % 4 == 3 for i in range(_LAYERS)], + kv_source_layers=[35 if i >= 39 and i % 4 == 3 else i for i in range(_LAYERS)], + index_sharing=["shared" if i >= 17 and (i - 17) % 4 < 2 else "full" for i in range(_LAYERS)], + index_n_heads=32, + index_head_dim=128, + first_k_dense_replace=1, + n_routed_experts=_EXPERTS, + moe_intermediate_size=6144, + n_shared_experts=1, + intermediate_size=24576, + vocab_size=204800, +) + +_SAFETENSORS_DTYPES = { + "BF16": torch.bfloat16, + "F32": torch.float32, + "F8_E4M3": torch.float8_e4m3fn, + "U8": torch.uint8, +} + +_EXPERT_PARAMS = ("fc1_w", "fc1_s", "fc2_w", "fc2_s", "isc1", "isc2", "ws2_13", "ws2_2") +_EXPERT_SUFFIXES = { + f"{proj}.{s}" + for proj in ("w1", "w2", "w3") + for s in ("weight", "weight_scale", "weight_scale_2", "input_scale") +} + + +@pytest.fixture(autouse=True) +def _no_device_memory(): + """Nothing here allocates on a GPU. The context alone proves nothing: `import tensorrt_llm` + creates one wherever a device is visible.""" + yield + if torch.cuda.is_initialized(): + assert torch.cuda.memory_reserved() == 0, "the weight table test allocated on a GPU" + + +def _mapping(rank: int, world: int) -> Mapping: + """Rank `rank` of the `world`-rank deployment, as the engine builds it. A topology that + ignored `rank=` (with MPI disabled, `Mapping` takes the process's own rank) would hand every + rank rank 0's window, so the rank is checked.""" + mapping = Mapping( + world_size=world, + rank=rank, + gpus_per_node=GPUS_PER_NODE, + tp_size=world, + moe_tp_size=1, + moe_ep_size=world, + enable_attention_dp=True, + ) + assert (mapping.tp_rank, mapping.moe_ep_rank, mapping.moe_tp_rank) == (rank, rank, 0) + return mapping + + +def _core(config: dict, rank: int, world: int) -> SimpleNamespace: + return SimpleNamespace( + model_config=SimpleNamespace( + pretrained_config=SimpleNamespace(**config), + mapping=_mapping(rank, world), + torch_dtype=torch.bfloat16, + ) + ) + + +class _Core(nn.Module): + """The surface `load` reads off the core: its config and its declared weights.""" + + def __init__(self, config: dict, rank: int, world: int, device: str): + super().__init__() + self.model_config = _core(config, rank, world).model_config + with torch.device(device): + self.w = W.MODEL_WEIGHTS.declare(W.MODEL_WEIGHTS.dims(self)) + + +class _Shell(nn.Module): + """The causal-LM shell: the core as `model`, and the `lm_head` it builds under attention + DP, the full vocabulary (`lm_rows` overrides that, to build the tensor-parallel shape).""" + + def __init__( + self, config: dict, rank: int, world: int, device: str, lm_rows: int | None = None + ): + super().__init__() + self.model = _Core(config, rank, world, device) + self.lm_head = nn.Module() + rows = config["vocab_size"] if lm_rows is None else lm_rows + self.lm_head.weight = nn.Parameter( + torch.empty(rows, config["hidden_size"], dtype=torch.bfloat16, device=device), + requires_grad=False, + ) + + +def _source_keys(manifest: dict) -> set: + """Every checkpoint key the manifest reads, key tuples flattened.""" + return {k for sources in manifest.values() for ckpt, *_ in sources for k in W._keys(ckpt)} + + +def _is_vision(key: str) -> bool: + return key.startswith(W.VISION_PREFIXES) + + +def _is_draft(key: str) -> bool: + """A DSpark draft key: the speculative-decoding head, which these targets do not run.""" + return key.startswith("mtp_dspark.") + + +def _is_expert(key: str) -> bool: + return ".experts." in key + + +def _is_expert_scalar(key: str) -> bool: + return _is_expert(key) and key.endswith((".weight_scale_2", ".input_scale")) + + +def _role(key: str) -> str: + """A parameter key's weight role: `l{i}_` per layer, the key itself otherwise.""" + head, _, rest = key.partition("_") + return rest if head[:1] == "l" and head[1:].isdigit() else key + + +def _expert_of(key: str) -> tuple[int, int, str]: + """`layers.N.experts.E.` -> (N, E, suffix).""" + parts = key.split(".") + assert parts[0] == "layers" and parts[2] == "experts", key + return int(parts[1]), int(parts[3]), ".".join(parts[4:]) + + +# --------------------------------------------------------------------- the configuration + + +def _layers_of(declared, name: str) -> dict[int, tuple]: + """Layer index -> declared shape, for every `l{i}_`.""" + out = {} + for key, param in declared.items(): + head, _, rest = key.partition("_") + if head.startswith("l") and head[1:].isdigit() and rest == name: + out[int(head[1:])] = tuple(param.shape) + return out + + +@pytest.mark.parametrize( + "world, rank", [(4, 0), (8, 0), (8, 7)], ids=["world4-rank0", "world8-rank0", "world8-rank7"] +) +def test_layer_sets_and_per_rank_shapes(world, rank): + """Which layers carry which weight, at what per-rank shape: all 64 heads of `wq`, `wkv_b`, + `wo` and SSMax, and the dense MLP (I 24576) and shared expert (I 6144) at full width, on + every rank; 32 routed experts per rank at four ranks, 16 at eight. The NoPE layers are 3, 7, + ..., 71; the KV sharers 39, 43, ..., 71; the 'shared' indexer layers 17, 18, 21, 22, ..., 69, + 70.""" + core = _core(_CONFIG, rank, world) + local = _EXPERTS // world + with torch.device("meta"): + declared = W.MODEL_WEIGHTS.declare(W.MODEL_WEIGHTS.dims(core)) + nope = {i for i in range(_LAYERS) if i % 4 == 3} + sharers = {i for i in range(39, _LAYERS, 4)} + full = {i for i in range(_LAYERS) if i < 17 or (i - 17) % 4 >= 2} + moe = range(1, _LAYERS) + assert len(nope) == 18 and len(sharers) == 9 and len(full) == 44 + + every = {i: None for i in range(_LAYERS)} + expect = { + "attn_norm": {i: (6144,) for i in every}, + "wq": {i: (8192, 6144) for i in every}, + "wq_s": {i: (12, 8192) for i in every}, + "ssmax_s": {i: (64,) for i in every}, + "ssmax_b": {i: (64,) for i in every}, + "wkv_a": {i: (512 if i in nope else 576, 6144) for i in range(_LAYERS) if i not in sharers}, + "kv_a_norm": {i: (512,) for i in range(_LAYERS) if i not in sharers}, + "wkv_b": {i: (16384 if i in nope else 12288, 512) for i in every}, + "wo": {i: (6144, 8192) for i in every}, + "wo_s": {i: (16, 6144) for i in every}, + "ffn_norm": {i: (6144,) for i in every}, + "ix_q_norm_w": {i: (6144,) for i in full}, + "ix_q_norm_b": {i: (6144,) for i in full}, + "ix_wq_b": {i: (4096, 6144) for i in full}, + "ix_wq_b_s": {i: (12, 4096) for i in full}, + "ix_wk_wp": {i: (160, 6144) for i in full}, + "ix_k_norm_w": {i: (128,) for i in full}, + "ix_k_norm_b": {i: (128,) for i in full}, + "ix_ssmax_s": {i: (32,) for i in full}, + "ix_ssmax_b": {i: (32,) for i in full}, + "mlp_gu": {i: (49152 if i == 0 else 12288, 6144) for i in every}, + "mlp_gu_s": {i: (12, 49152 if i == 0 else 12288) for i in every}, + "mlp_dn": {i: (6144, 24576 if i == 0 else 6144) for i in every}, + "mlp_dn_s": {i: (48 if i == 0 else 12, 6144) for i in every}, + "router": {i: (128, 6144) for i in moe}, + "fc1_w": {i: (local, 12288, 3072) for i in moe}, + "fc1_s": {i: (local, 12288, 384) for i in moe}, + "fc2_w": {i: (local, 6144, 3072) for i in moe}, + "fc2_s": {i: (local, 6144, 384) for i in moe}, + "isc1": {i: (1,) for i in moe}, + "isc2": {i: (1,) for i in moe}, + "ws2_13": {i: (local,) for i in moe}, + "ws2_2": {i: (local,) for i in moe}, + } + for name, want in expect.items(): + assert _layers_of(declared, name) == want, name + per_layer = {k for k in declared if k.startswith("l") and k.split("_")[0][1:].isdigit()} + assert len(per_layer) == sum(len(v) for v in expect.values()), "an undeclared-for role" + assert {k: tuple(v.shape) for k, v in declared.items() if k not in per_layer} == { + "final_norm": (6144,), + "embed": (204800, 6144), + "lerope_alpha": (32,), + } + dtypes = {_role(k): v.dtype for k, v in declared.items()} + assert dtypes["wq"] == dtypes["mlp_dn"] == dtypes["ix_wq_b"] == torch.float8_e4m3fn + assert dtypes["wq_s"] == dtypes["mlp_gu_s"] == dtypes["ix_wq_b_s"] == torch.int32 + assert dtypes["fc1_w"] == dtypes["fc2_w"] == torch.uint8 + assert dtypes["fc1_s"] == dtypes["fc2_s"] == torch.float8_e4m3fn + assert dtypes["isc1"] == dtypes["isc2"] == dtypes["ws2_13"] == dtypes["ws2_2"] == torch.float32 + assert dtypes["ix_k_norm_w"] == dtypes["lerope_alpha"] == torch.float32 + assert dtypes["wkv_b"] == dtypes["router"] == dtypes["embed"] == torch.bfloat16 + + # Per rank, with the shell's whole lm_head. The routed experts are exactly their checkpoint + # bytes; the FP8 scale operands, one int32 word per four 128-blocks per output row, are + # 0.12 GiB more than the fp32 grids they are built from. + nbytes = {k: p.numel() * p.element_size() for k, p in declared.items()} + experts = sum(n for k, n in nbytes.items() if _role(k) in _EXPERT_PARAMS) + total = sum(nbytes.values()) + 204800 * 6144 * 2 + assert (round(experts / 2**30, 2), round(total / 2**30, 2)) == _DECLARED_GIB[world], ( + experts / 2**30, + total / 2**30, + ) + + +@_each_world +def test_each_rank_reads_only_its_expert_window(world): + """Rank r's manifest names routed experts [Er, Er + E), E = 128 / world (32 at four ranks, + 16 at eight), and nothing else, on every MoE layer, local slot s <- global expert Er + s, all + twelve tensors of each; the one-per-layer scalars read the window's copies. The windows are + disjoint and cover experts 0..127, node by node: at four ranks the one node holds them all; + at eight node 0 (ranks 0-3, local ranks 0-3) holds experts 0..63, node 1 (ranks 4-7) + 64..127. The known-not-loaded set is the other 128 - E experts: their scalars predicted, + their codes and block scales only where found.""" + local = _EXPERTS // world + nodes = world // GPUS_PER_NODE + seen = set() + windows = {} + by_node: dict[int, set] = {node: set() for node in range(nodes)} + for rank in range(world): + core = _core(_CONFIG, rank, world) + mapping = core.model_config.mapping + assert (mapping.local_rank, mapping.node_rank) == ( + rank % GPUS_PER_NODE, + rank // GPUS_PER_NODE, + ) + manifest = W.MODEL_WEIGHTS.manifest(core) + lo, hi = local * rank, local * rank + local + d = W.MODEL_WEIGHTS.dims(core) + assert (d.local_experts, d.expert_offset) == (local, lo) + experts: dict = {} + for key, sources in manifest.items(): + for ckpt, index, *_ in sources: + keys = W._keys(ckpt) + if not any(_is_expert(k) for k in keys): + continue + assert all(_is_expert(k) for k in keys), (key, ckpt) + param = _role(key) + assert param in _EXPERT_PARAMS, key + for k in keys: + layer, e, suffix = _expert_of(k) + assert lo <= e < hi, (rank, key, k) + assert key == f"l{layer}_{param}", (key, k) + if param in ("isc1", "isc2"): + assert index == (0,), (key, index) + else: + assert index[0] == e - lo, (key, k, index) + experts.setdefault((layer, e), set()).add(suffix) + assert set(experts) == {(i, e) for i in range(1, _LAYERS) for e in range(lo, hi)} + assert all(v == _EXPERT_SUFFIXES for v in experts.values()) + seen |= set(experts) + windows[rank] = {e for _, e in experts} + by_node[mapping.node_rank] |= windows[rank] + + predicted = W.MODEL_WEIGHTS.expected_unconsumed(core, {}) + assert len(predicted) == 71 * (_EXPERTS - local) * 6 + assert all(_is_expert_scalar(k) for k in predicted) + assert all(not lo <= _expert_of(k)[1] < hi for k in predicted) + assert seen == {(i, e) for i in range(1, _LAYERS) for e in range(_EXPERTS)} + assert sum(len(w) for w in windows.values()) == _EXPERTS + assert set().union(*windows.values()) == set(range(_EXPERTS)) + per_node = _EXPERTS // nodes + assert by_node == {n: set(range(per_node * n, per_node * n + per_node)) for n in range(nodes)} + if world == 8: + assert by_node == {0: set(range(64)), 1: set(range(64, 128))} + + +@_each_world +def test_every_rank_reads_the_same_replicated_sources(world): + """Outside the routed experts every rank's manifest is the same: the same parameters from + the same checkpoint keys at the same indices, with no per-rank slice anywhere.""" + + def replicated(rank): + manifest = W.MODEL_WEIGHTS.manifest(_core(_CONFIG, rank, world)) + return { + key: [(ckpt, index) for ckpt, index, _ in sources] + for key, sources in manifest.items() + if _role(key) not in _EXPERT_PARAMS + } + + first = replicated(0) + assert len(first) == len(W.MODEL_WEIGHTS.manifest(_core(_CONFIG, 0, world))) - 71 * 8 + for rank in range(1, world): + assert replicated(rank) == first, rank + for key, sources in first.items(): + for ckpt, index in sources: + assert not any(_is_expert(k) for k in W._keys(ckpt)), (key, ckpt) + + +@_each_world +def test_manifest_and_declaration_agree(world): + """A declared weight with no source allocates memory nothing fills.""" + for rank in range(world): + core = _core(_CONFIG, rank, world) + manifest = W.MODEL_WEIGHTS.manifest(core) + with torch.device("meta"): + declared = W.MODEL_WEIGHTS.declare(W.MODEL_WEIGHTS.dims(core)) + assert set(manifest) == set(declared) + assert all(manifest.values()) + + +# --------------------------------------------------------------------- the checkpoint + + +def _checkpoint_dir() -> Path | None: + root = llm_models_root() + if root is None: + return None + for name in CHECKPOINTS: + path = root / name + if (path / "params.json").is_file() and (path / W.SHARD_INDEX).is_file(): + return path + return None + + +@pytest.fixture(scope="module") +def checkpoint(): + path = _checkpoint_dir() + if path is None: + pytest.skip(f"checkpoint {CHECKPOINTS} is not under LLM_MODELS_ROOT ({llm_models_root()})") + index = json.loads((path / W.SHARD_INDEX).read_text())["weight_map"] + headers = {} + for name in sorted(set(index.values())): + with open(path / name, "rb") as f: + (size,) = struct.unpack(" dict: + return { + key: torch.empty(shape, dtype=dtype, device="meta") + for key, (dtype, shape) in checkpoint.headers.items() + } + + +def _nbytes(checkpoint, key: str) -> int: + dtype, shape = checkpoint.headers[key] + n = torch.empty((), dtype=dtype).element_size() + for s in shape: + n *= s + return n + + +def test_checkpoint_config_is_the_one_tested(checkpoint): + """The configuration above is the checkpoint's, on every key the table reads.""" + assert {k: checkpoint.config[k] for k in _CONFIG} == _CONFIG + + +#: The FP8 128x128-block linears (`FP8_PB_WO`): e4m3 codes, fp32 `[N/128, 1, K/128, 1]` scales. +_FP8_LINEAR = re.compile( + r"layers\.\d+\.(attention\.(wq|wo|indexer\.wq_b)|feed_forward\.w[123]|shared_experts\.w[123])" +) + + +def test_checkpoint_dtypes_by_role(checkpoint): + """What each key stores, which is what the table's rows read: the FP8 linears e4m3 with + fp32 block scales, the routed experts NVFP4 (packed codes, e4m3 block scales, fp32 + per-tensor scalars), every other language-model tensor and the vision tower bf16.""" + for key, (dtype, shape) in checkpoint.headers.items(): + if _is_draft(key): + continue + module, _, leaf = key.rpartition(".") + if _is_expert(key): + want = { + "weight": torch.uint8, + "weight_scale": torch.float8_e4m3fn, + "weight_scale_2": torch.float32, + "input_scale": torch.float32, + }[leaf] + assert dtype == want, key + assert (shape == ()) == _is_expert_scalar(key), (key, shape) + elif _FP8_LINEAR.fullmatch(module): + want = {"weight": torch.float8_e4m3fn, "weight_scale": torch.float32}[leaf] + assert dtype == want, key + else: + assert dtype == torch.bfloat16, key + + +@_each_world +def test_union_over_ranks_is_every_key_but_the_draft(checkpoint, world): + """Every key of the index but the DSpark draft's 38 is read by some rank (lm_head's + `output.weight` by the shell); per rank, read and known-not-loaded split the index, the + known-not-loaded the draft and the other ranks' experts, and a rank reads every one of the + 1,721 language-model keys, the 276 vision-tower keys and its window's expert keys: 71 layers + x 32 experts x 3 projections x 4 tensors = 27,264 at four ranks, 71 x 16 x 3 x 4 = 13,632 at + eight.""" + local = _EXPERTS // world + keys = set(checkpoint.index) + draft = {k for k in keys if _is_draft(k)} + vision = {k for k in keys if _is_vision(k)} + lm = {k for k in keys - draft - vision if not _is_expert(k)} + assert len(keys) == 111091 and len(draft) == _DRAFT_KEYS and len(vision) == _VISION_KEYS + assert len(lm) == 1721 + union = set() + for rank in range(world): + core = _core(checkpoint.config, rank, world) + read = _source_keys(W.MODEL_WEIGHTS.manifest(core)) | {"output.weight"} + skipped = W.MODEL_WEIGHTS.expected_unconsumed(core, checkpoint.index) + assert not read & set(skipped) + assert read | set(skipped) == keys + assert {k for k in read if not _is_expert(k)} == lm | vision + own = len({k for k in read if _is_expert(k)}) + assert own == {4: 27264, 8: 13632}[world] == 71 * local * 3 * 4 + assert {k for k in skipped if not _is_expert(k)} == draft + assert {skipped[k] for k in draft} == {"DSpark draft disabled"} + assert len(skipped) == _DRAFT_KEYS + 71 * (_EXPERTS - local) * 12 + union |= read + assert union == keys - draft + + +def _category(key: str) -> str: + if _is_vision(key): + return "vision" + if _is_expert(key): + return "routed" + if ".shared_experts." in key: + return "shared" + if ".feed_forward." in key: + return "dense" + if key in ("tok_embeddings.weight", "output.weight"): + return "embed+output" + if ".attention." in key: + return "attention+indexer" + if key.endswith(".gate.weight"): + return "router" + return "norms+lerope" + + +@_each_world +def test_per_rank_bytes_from_the_headers(checkpoint, world): + """What each rank moves, from the safetensors headers through the production manifest: + every language-model tensor and the vision tower whole and this rank's routed-expert window, + the same on every rank -- 171,644,394,304 B (159.86 GiB) at four ranks, 134.79 GiB of it the + window, and 99,280,040,128 B (92.46 GiB) at eight, 67.39 GiB of it the window -- 21.89 GiB + the replicated language-model rest: attention and indexer 9.19, shared experts 7.49, + embedding and output 4.6875, the dense layer-0 MLP 0.42, the router 0.10 -- and the + vision tower's 3,408,437,248 B (3.17 GiB) (each source is read whole, so it is the headers' + own byte count).""" + local, reads = _EXPERTS // world, _READS[world] + lm = sum( + _nbytes(checkpoint, k) + for k in checkpoint.index + if not _is_vision(k) and not _is_draft(k) and not _is_expert(k) + ) + assert round(lm / 2**30, 2) == 21.89, lm / 2**30 + vision = sum(_nbytes(checkpoint, k) for k in checkpoint.index if _is_vision(k)) + assert vision == _VISION_BYTES, vision + for rank in range(world): + core = _core(checkpoint.config, rank, world) + lo, hi = local * rank, local * rank + local + window = sum( + _nbytes(checkpoint, k) + for k in checkpoint.index + if _is_expert(k) and lo <= _expert_of(k)[1] < hi + ) + meta = {} + + def entry(key): + if key not in meta: + dtype, shape = checkpoint.headers[key] + meta[key] = torch.empty(shape, dtype=dtype, device="meta") + return meta[key] + + by_category: dict[str, int] = {} + for sources in W.MODEL_WEIGHTS.manifest(core).values(): + for ckpt, _, _ in sources: + for key in W._keys(ckpt): + t = W._materialize(entry(key)) + n = t.numel() * t.element_size() + by_category[_category(key)] = by_category.get(_category(key), 0) + n + by_category["embed+output"] += _nbytes(checkpoint, "output.weight") + total = sum(by_category.values()) + assert by_category["routed"] == window, (rank, by_category["routed"], window) + assert by_category["vision"] == vision, (rank, by_category["vision"], vision) + assert total == lm + vision + window == reads["total"], (rank, total, lm + vision + window) + assert round(total / 2**30, 2) == reads["total_gib"], (rank, total / 2**30) + # The embedding and the whole lm_head: 2 x 2.34375 GiB, exactly. + assert by_category.pop("embed+output") == 2 * 204800 * 6144 * 2 + gib = {c: round(n / 2**30, 2) for c, n in by_category.items()} + assert gib == { + "routed": reads["window_gib"], + "attention+indexer": 9.19, + "shared": 7.49, + "dense": 0.42, + "router": 0.10, + "norms+lerope": 0.0, + "vision": 3.17, + }, (rank, gib) + + +def test_per_node_bytes_from_the_headers(checkpoint): + """At eight ranks over two nodes of four (dep8), what each node's page cache serves: the + distinct keys its four ranks read. Both nodes read the whole replicated set -- the language + model's 21.89 GiB and the vision tower's 3.17 GiB -- and each one half of the expert stack, + node 0 experts 0..63 and node 1 experts 64..127 (269.58 GiB): 316,373,102,656 B + (294.65 GiB) per node, against the four ranks' 4 x 92.46 GiB of reads.""" + world = 8 + nodes: dict[int, set] = {0: set(), 1: set()} + for rank in range(world): + core = _core(checkpoint.config, rank, world) + read = _source_keys(W.MODEL_WEIGHTS.manifest(core)) | {"output.weight"} + nodes[core.model_config.mapping.node_rank] |= read + expert_halves = [] + for node, keys in nodes.items(): + experts = {_expert_of(k)[1] for k in keys if _is_expert(k)} + assert experts == set(range(64 * node, 64 * node + 64)), node + routed = sum(_nbytes(checkpoint, k) for k in keys if _is_expert(k)) + vision = sum(_nbytes(checkpoint, k) for k in keys if _is_vision(k)) + rest = sum(_nbytes(checkpoint, k) for k in keys if not _is_expert(k) and not _is_vision(k)) + assert round(routed / 2**30, 2) == 269.58 and round(rest / 2**30, 2) == 21.89, node + assert vision == _VISION_BYTES, (node, vision) + assert routed + rest + vision == 316_373_102_656, (node, routed + rest + vision) + assert round((routed + rest + vision) / 2**30, 2) == 294.65, node + expert_halves.append({k for k in keys if _is_expert(k)}) + assert not expert_halves[0] & expert_halves[1] + assert nodes[0] - expert_halves[0] == nodes[1] - expert_halves[1] + + +class _Recorded: + """A checkpoint entry that records each read `load` makes of it -- its key and the bytes it + takes -- and hands back the entry so indexed.""" + + def __init__(self, key: str, entry: torch.Tensor, log: list): + self.key, self.entry, self.log = key, entry, log + + def __getitem__(self, index): + t = self.entry[index] + self.log.append((self.key, t.numel() * t.element_size())) + return t + + +@_each_rank +def test_load_on_meta_matches_the_headers(checkpoint, expert_scalars, world, rank): + """The production `load` over meta tensors shaped like the safetensors headers, with the + checkpoint's own expert scalars: the scalar asserts pass on every layer, every source is + transformed to its declared per-rank shape and dtype, every declared parameter is filled, + and every key is consumed or known-not-loaded. What it reads, recorded key by key, is the + loader's own per-rank total: each key once and whole, every vision key and no DSpark draft + key, the expert codes and block scales of this rank's window only -- the manifest's bytes + and the other experts' scalars, read for the asserts: 70,157 keys and 171,644,557,888 B at + four ranks (the other 96 experts' 40,896 scalars, 163,584 B), 63,341 keys and + 99,280,230,976 B at eight (the other 112 experts' 47,712 scalars, 190,848 B).""" + local, reads = _EXPERTS // world, _READS[world] + log: list = [] + weights = {key: _Recorded(key, t, log) for key, t in _meta(checkpoint).items()} + weights.update({key: _Recorded(key, t, log) for key, t in expert_scalars.items()}) + model = _Shell(checkpoint.config, rank, world, "meta") + W.MODEL_WEIGHTS.load(model, weights) + + keys = [key for key, _ in log] + assert len(keys) == len(set(keys)) == reads["keys"], "a key was read twice" + assert not any(_is_draft(k) for k in keys) + assert {k for k in keys if _is_vision(k)} == {k for k in checkpoint.index if _is_vision(k)} + for key in keys: + if _is_expert(key) and not _is_expert_scalar(key): + assert _expert_of(key)[1] // local == rank, key + read = sum(n for _, n in log) + off_window = {k for k in keys if _is_expert(k) and _expert_of(k)[1] // local != rank} + off_scalars = 71 * (_EXPERTS - local) * 6 + assert len(off_window) == off_scalars and all(_is_expert_scalar(k) for k in off_window) + assert read == reads["total"] + off_scalars * 4 == _LOAD_READ[world], read + assert all(n == _nbytes(checkpoint, k) for k, n in log), "a key was read in part" + + +@_each_world +def test_scale_invariants_hold_on_the_checkpoint(checkpoint, expert_scalars, world): + """Every FP8 block scale goes through the production transform (squeezed from + `[N/128, 1, K/128, 1]`, asserted UE8M0, packed), and every expert scalar passes the + production check: per layer, w1 == w3 on both scalars of every expert, and one input_scale + per projection over all 128 experts.""" + fp8 = sorted( + k + for k, (dtype, _) in checkpoint.headers.items() + if k.endswith(".weight_scale") and dtype == torch.float32 + ) + assert len(fp8) == 404 == 72 + 72 + 44 + 71 * 3 + 3 + by_file: dict = {} + for key in fp8: + by_file.setdefault(checkpoint.index[key], []).append(key) + for name, keys in sorted(by_file.items()): + with safe_open(str(checkpoint.path / name), framework="pt", device="cpu") as handle: + for key in keys: + s = handle.get_tensor(key) + n_blk, _, k_blk, _ = s.shape + packed = W._block_sf(s, None) + assert packed.shape == (W._sf_words(k_blk * 128), n_blk * 128), key + + d = W.MODEL_WEIGHTS.dims(_core(checkpoint.config, 0, world)) + assert len(expert_scalars) == 71 * 128 * 6 + W._check_expert_scalars(expert_scalars, d) + for i in range(1, _LAYERS): + isc1 = expert_scalars[f"layers.{i}.experts.0.w1.input_scale"].item() + isc2 = expert_scalars[f"layers.{i}.experts.0.w2.input_scale"].item() + assert 0 < isc1 < 1 and 0 < isc2 < 1, (i, isc1, isc2) + + +# --------------------------------------------------------------------- values, scaled down + +_TINY_LAYERS = 8 +#: Every structural feature at a size a CPU fills in a second: a dense layer, RoPE and NoPE +#: layers (3, 7), a KV sharer (7, reading 3), 'shared' indexer layers (2, 5, 6), two experts +#: per rank (`_tiny`), several 128-blocks on every FP8 axis. +_TINY = dict( + num_hidden_layers=_TINY_LAYERS, + hidden_size=256, + num_attention_heads=8, + qk_nope_head_dim=64, + qk_rope_head_dim=64, + v_head_dim=128, + kv_lora_rank=128, + nope_layer_flags=[i % 4 == 3 for i in range(_TINY_LAYERS)], + kv_source_layers=[0, 1, 2, 3, 4, 5, 6, 3], + index_sharing=["full", "full", "shared", "full", "full", "shared", "shared", "full"], + index_n_heads=4, + index_head_dim=128, + first_k_dense_replace=1, + moe_intermediate_size=512, + n_shared_experts=1, + intermediate_size=1024, + vocab_size=512, +) +_TINY_LOCAL = 2 + + +def _tiny(world: int) -> dict: + """`_TINY` at `world` ranks: two routed experts per rank, 8 at four ranks and 16 at eight.""" + return dict(_TINY, n_routed_experts=_TINY_LOCAL * world) + + +def _tiny_checkpoint(world: int, seed: int = 0) -> dict: + """A Mistral-named checkpoint at `_tiny(world)`: e4m3 weights with power-of-two fp32 block scales + stored `[N/128, 1, K/128, 1]` where the real one has them, NVFP4 routed experts -- packed + codes, e4m3 block scales, a weight_scale_2 per expert that w1 and w3 share, one input_scale + per projection and layer -- bf16 elsewhere, and two DSpark draft tensors.""" + c = SimpleNamespace(**_tiny(world)) + g = torch.Generator().manual_seed(seed) + bf16 = lambda *s: torch.randn(*s, generator=g).to(torch.bfloat16) # noqa: E731 + scalar = lambda: torch.rand((), generator=g) * 1e-3 + 1e-5 # noqa: E731 + + def fp8(name, n, k): + codes = (torch.rand(n, k, generator=g) * 800 - 400).to(torch.float8_e4m3fn) + exps = torch.randint(-15, -7, (n // 128, 1, k // 128, 1), generator=g).float() + return {f"{name}.weight": codes, f"{name}.weight_scale": torch.exp2(exps)} + + def nvfp4(name, n, k, ws2, isc): + codes = torch.randint(0, 256, (n, k // 2), dtype=torch.uint8, generator=g) + sf = (torch.rand(n, k // 16, generator=g) * 400 + 1).to(torch.float8_e4m3fn) + return { + f"{name}.weight": codes, + f"{name}.weight_scale": sf, + f"{name}.weight_scale_2": ws2.clone(), + f"{name}.input_scale": isc.clone(), + } + + h, qk = c.hidden_size, c.qk_nope_head_dim + c.qk_rope_head_dim + ck = {"tok_embeddings.weight": bf16(c.vocab_size, h), "norm.weight": bf16(h)} + ck["output.weight"] = bf16(c.vocab_size, h) + ck["lerope.alpha"] = bf16(c.qk_rope_head_dim // 2) + ck["mtp_dspark.backbone.final_norm.weight"] = bf16(32) + ck["mtp_dspark.backbone.layers.0.attention.wq.weight"] = bf16(32, 64) + for i in range(c.num_hidden_layers): + p = f"layers.{i}" + nope = c.nope_layer_flags[i] + ck[f"{p}.attention_norm.weight"] = bf16(h) + ck[f"{p}.ffn_norm.weight"] = bf16(h) + ck.update(fp8(f"{p}.attention.wq", c.num_attention_heads * qk, h)) + ck.update(fp8(f"{p}.attention.wo", h, c.num_attention_heads * c.v_head_dim)) + ck[f"{p}.attention.ssmax_scaling.s"] = bf16(c.num_attention_heads) + ck[f"{p}.attention.ssmax_scaling.b"] = bf16(c.num_attention_heads) + per_head = (qk if nope else c.qk_nope_head_dim) + c.v_head_dim + ck[f"{p}.attention.wkv_b.weight"] = bf16(c.num_attention_heads * per_head, c.kv_lora_rank) + if c.kv_source_layers[i] == i: + lat = c.kv_lora_rank + (0 if nope else c.qk_rope_head_dim) + ck[f"{p}.attention.wkv_a_with_mqa.weight"] = bf16(lat, h) + ck[f"{p}.attention.kv_a_norm.weight"] = bf16(c.kv_lora_rank) + if c.index_sharing[i] == "full": + x = f"{p}.attention.indexer" + ck.update(fp8(f"{x}.wq_b", c.index_n_heads * c.index_head_dim, h)) + ck[f"{x}.wk.weight"] = bf16(c.index_head_dim, h) + ck[f"{x}.weights_proj.weight"] = bf16(c.index_n_heads, h) + for norm, n in (("q_input_norm", h), ("k_norm", c.index_head_dim)): + ck[f"{x}.{norm}.weight"] = bf16(n) + ck[f"{x}.{norm}.bias"] = bf16(n) + ck[f"{x}.ssmax_scaling.s"] = bf16(c.index_n_heads) + ck[f"{x}.ssmax_scaling.b"] = bf16(c.index_n_heads) + if i < c.first_k_dense_replace: + for w, (n, k) in (("w1", (c.intermediate_size, h)), ("w3", (c.intermediate_size, h))): + ck.update(fp8(f"{p}.feed_forward.{w}", n, k)) + ck.update(fp8(f"{p}.feed_forward.w2", h, c.intermediate_size)) + continue + ck[f"{p}.gate.weight"] = bf16(c.n_routed_experts, h) + inter = c.moe_intermediate_size + m = f"{p}.shared_experts" + ck.update(fp8(f"{m}.w1", inter, h)) + ck.update(fp8(f"{m}.w3", inter, h)) + ck.update(fp8(f"{m}.w2", h, inter)) + isc1, isc2 = scalar(), scalar() + for e in range(c.n_routed_experts): + m = f"{p}.experts.{e}" + ws2_13 = scalar() + ck.update(nvfp4(f"{m}.w1", inter, h, ws2_13, isc1)) + ck.update(nvfp4(f"{m}.w3", inter, h, ws2_13, isc1)) + ck.update(nvfp4(f"{m}.w2", h, inter, scalar(), isc2)) + return ck + + +def _bytes(t: torch.Tensor) -> torch.Tensor: + """Compare by bit pattern: e4m3 has no `==`, and it is exactness that is asserted.""" + return t.contiguous().view(torch.uint8) if t.element_size() == 1 else t + + +def _grid(s: torch.Tensor) -> torch.Tensor: + """An FP8 block scale as stored, `[N/128, 1, K/128, 1]`, as its `[N/128, K/128]` grid.""" + return s.reshape(s.shape[0], s.shape[2]) + + +def _decode_sf(storage: torch.Tensor, n: int, k_blocks: int) -> torch.Tensor: + """Independent reading of the packed scale storage `[ceil(k_blocks/4), n]` int32: word + `[w, row]` holds the UE8M0 bytes of K blocks 4w..4w+3 of `row`, low byte first.""" + words = storage.t().contiguous() # [n, words] + out = torch.empty(n, k_blocks, dtype=torch.float32) + for kb in range(k_blocks): + byte = (words[:, kb // 4] >> (8 * (kb % 4))) & 0xFF + out[:, kb] = torch.exp2(byte.float() - 127) + return out + + +def _runner_rows(n: int, interleave: bool) -> torch.Tensor: + """The stored row each row of an expert's runner operand holds, one row at a time: row + `32b + 8v + u` takes row `32b + 4u + v` of the interleaved stack, whose row `2i` is up row + `i` (stored row `i`) and row `2i + 1` gate row `i` (stored row `n/2 + i`).""" + rows = [] + for r in range(n): + b, within = divmod(r, 32) + v, u = divmod(within, 8) + j = 32 * b + 4 * u + v + rows.append((j // 2 if j % 2 == 0 else n // 2 + j // 2) if interleave else j) + return torch.tensor(rows) + + +def _swizzle_offsets(m: int, c: int) -> torch.Tensor: + """`[m, c]`: where scale byte `(row, col)` lands in the 128x4-swizzled buffer.""" + r = torch.arange(m).view(-1, 1) + k = torch.arange(c).view(1, -1) + return (r // 128) * 128 * c + (k // 4) * 512 + (r % 32) * 16 + ((r % 128) // 32) * 4 + k % 4 + + +def _swizzled(s: torch.Tensor) -> torch.Tensor: + """`[m, c]` scale bytes scattered to their swizzled offsets, viewed `[m, c]` again.""" + m, c = s.shape + out = torch.empty(m * c, dtype=torch.uint8) + out[_swizzle_offsets(m, c).reshape(-1)] = s.reshape(-1) + return out.view(m, c) + + +def _unswizzled(s: torch.Tensor) -> torch.Tensor: + m, c = s.shape + return s.reshape(-1)[_swizzle_offsets(m, c)] + + +def _expected(ck: dict, rank: int, world: int) -> dict: + """What each parameter of `rank` must hold, built from the checkpoint whole -- only the + routed experts depend on the rank -- with fp32 block-scale grids standing for the packed + scale storage (`*_s` int32).""" + c = SimpleNamespace(**_tiny(world)) + heads = c.num_attention_heads + qk, v = c.qk_nope_head_dim + c.qk_rope_head_dim, c.v_head_dim + local = c.n_routed_experts // world + out = {} + + for i in range(c.num_hidden_layers): + p, q = f"layers.{i}", f"l{i}_" + nope = qk if c.nope_layer_flags[i] else c.qk_nope_head_dim + out[q + "attn_norm"] = ck[f"{p}.attention_norm.weight"] + out[q + "ffn_norm"] = ck[f"{p}.ffn_norm.weight"] + out[q + "wq"] = ck[f"{p}.attention.wq.weight"] + out[q + "wq_s"] = _grid(ck[f"{p}.attention.wq.weight_scale"]) + out[q + "wo"] = ck[f"{p}.attention.wo.weight"] + out[q + "wo_s"] = _grid(ck[f"{p}.attention.wo.weight_scale"]) + for s in ("s", "b"): + out[q + f"ssmax_{s}"] = ck[f"{p}.attention.ssmax_scaling.{s}"] + kvb = ck[f"{p}.attention.wkv_b.weight"] + per_head = [kvb[h * (nope + v) : (h + 1) * (nope + v)] for h in range(heads)] + out[q + "wkv_b"] = torch.cat([t[:nope] for t in per_head] + [t[nope:] for t in per_head]) + if c.kv_source_layers[i] == i: + out[q + "wkv_a"] = ck[f"{p}.attention.wkv_a_with_mqa.weight"] + out[q + "kv_a_norm"] = ck[f"{p}.attention.kv_a_norm.weight"] + if c.index_sharing[i] == "full": + x = f"{p}.attention.indexer" + out[q + "ix_q_norm_w"] = ck[f"{x}.q_input_norm.weight"] + out[q + "ix_q_norm_b"] = ck[f"{x}.q_input_norm.bias"] + out[q + "ix_wq_b"] = ck[f"{x}.wq_b.weight"] + out[q + "ix_wq_b_s"] = _grid(ck[f"{x}.wq_b.weight_scale"]) + out[q + "ix_wk_wp"] = torch.cat([ck[f"{x}.wk.weight"], ck[f"{x}.weights_proj.weight"]]) + out[q + "ix_k_norm_w"] = ck[f"{x}.k_norm.weight"].float() + out[q + "ix_k_norm_b"] = ck[f"{x}.k_norm.bias"].float() + out[q + "ix_ssmax_s"] = ck[f"{x}.ssmax_scaling.s"] + out[q + "ix_ssmax_b"] = ck[f"{x}.ssmax_scaling.b"] + m = f"{p}.feed_forward" if i < c.first_k_dense_replace else f"{p}.shared_experts" + # Gate rows first. + out[q + "mlp_gu"] = torch.cat([_bytes(ck[f"{m}.{w}.weight"]) for w in ("w1", "w3")]) + out[q + "mlp_gu_s"] = torch.cat([_grid(ck[f"{m}.{w}.weight_scale"]) for w in ("w1", "w3")]) + out[q + "mlp_dn"] = ck[f"{m}.w2.weight"] + out[q + "mlp_dn_s"] = _grid(ck[f"{m}.w2.weight_scale"]) + if i < c.first_k_dense_replace: + continue + out[q + "router"] = ck[f"{p}.gate.weight"] + window = [f"{p}.experts.{rank * local + s}" for s in range(local)] + # Up rows first, interleaved and block-shuffled; the block scales also swizzled. + fc1 = _runner_rows(2 * c.moe_intermediate_size, True) + fc2 = _runner_rows(c.hidden_size, False) + + def up_gate(e, suffix): + return torch.cat([_bytes(ck[f"{e}.{w}.{suffix}"]) for w in ("w3", "w1")])[fc1] + + out[q + "fc1_w"] = torch.stack([up_gate(e, "weight") for e in window]) + out[q + "fc1_s"] = torch.stack([_swizzled(up_gate(e, "weight_scale")) for e in window]) + out[q + "fc2_w"] = torch.stack([ck[f"{e}.w2.weight"][fc2] for e in window]) + out[q + "fc2_s"] = torch.stack( + [_swizzled(_bytes(ck[f"{e}.w2.weight_scale"])[fc2]) for e in window] + ) + out[q + "isc1"] = ck[f"{window[0]}.w1.input_scale"].reshape(1) + out[q + "isc2"] = ck[f"{window[0]}.w2.input_scale"].reshape(1) + out[q + "ws2_13"] = torch.stack([ck[f"{e}.w1.weight_scale_2"] for e in window]) + out[q + "ws2_2"] = torch.stack([ck[f"{e}.w2.weight_scale_2"] for e in window]) + out["final_norm"] = ck["norm.weight"] + out["embed"] = ck["tok_embeddings.weight"] + out["lerope_alpha"] = ck["lerope.alpha"].float() + return out + + +@_each_rank +def test_tiny_load_values(world, rank): + """Every parameter of every rank, bit for bit against `_expected`.""" + ck = _tiny_checkpoint(world) + model = _Shell(_tiny(world), rank, world, "cpu") + W.MODEL_WEIGHTS.load(model, dict(ck)) + got = dict(model.model.w) + want = _expected(ck, rank, world) + assert set(got) == set(want) + for key, ref in want.items(): + param = got[key].data + if key.endswith("_s") and param.dtype == torch.int32: + # DeepGEMM's operand is the transpose of the storage: MN-major, row stride N. + n = param.shape[1] + k = ref.shape[1] * 128 + operand = param.t() + check_sf_layout( + operand, + mn=n, + k=k, + gran=(1, 128), + num_groups=None, + tma_stride_check=True, + type_check=torch.int, + ) + per_row = ref.repeat_interleave(128, dim=0) + assert torch.equal(_decode_sf(param, n, ref.shape[1]), per_row), key + assert torch.equal(inverse_transform_sf(operand, mn=n, k=k), ref), key + if key.endswith(("wq_s", "wo_s", "ix_wq_b_s", "mlp_dn_s")): + engine = transform_sf_into_required_layout( + ref.contiguous(), mn=n, k=k, recipe=(1, 128, 128), is_sfa=False + ) + assert engine.shape == operand.shape and engine.stride() == operand.stride() + assert torch.equal(engine, operand), key + continue + if ref.dtype == torch.uint8 and param.dtype == torch.float8_e4m3fn: + assert key.endswith(("mlp_gu", "fc1_s", "fc2_s")), key + else: + assert param.dtype == ref.dtype, (key, param.dtype, ref.dtype) + assert torch.equal(_bytes(param), _bytes(ref)), key + assert torch.equal(model.lm_head.weight.data, ck["output.weight"]) + + +@_each_world +def test_tiny_replicated_weights_are_identical_on_every_rank(world): + """Under attention DP every rank holds the same bytes outside its expert window, the whole + `lm_head` included, and disjoint windows of the routed experts, one per rank.""" + ck = _tiny_checkpoint(world) + models = [] + for rank in range(world): + model = _Shell(_tiny(world), rank, world, "cpu") + W.MODEL_WEIGHTS.load(model, dict(ck)) + models.append(dict(model.named_parameters())) + first = models[0] + for rank, params in enumerate(models[1:], start=1): + assert set(params) == set(first) + for key, param in params.items(): + same = torch.equal(_bytes(param.data), _bytes(first[key].data)) + routed = _role(key.removeprefix("model.w.")) in _EXPERT_PARAMS + # isc1/isc2 are the layer's one value, the same in every window. + assert same == (not routed or key.endswith(("_isc1", "_isc2"))), (rank, key) + + +@_each_world +def test_tiny_expert_halves_are_up_then_gate(world): + """Undo the block shuffle (and the scales' swizzle) of FC1: its even rows are w3 (up), its + odd rows w1 (gate), codes and block scales alike -- the order the NVFP4 runner's FC1 reads, + up first, the opposite of `mlp_gu`'s gate first.""" + ck = _tiny_checkpoint(world) + model = _Shell(_tiny(world), 1, world, "cpu") + W.MODEL_WEIGHTS.load(model, dict(ck)) + inter, e = _TINY["moe_intermediate_size"], _TINY_LOCAL # rank 1's local slot 0 + w = model.model.w + shuffled = _runner_rows(2 * inter, False) + unshuffle = torch.empty_like(shuffled) + unshuffle[shuffled] = torch.arange(2 * inter) + codes = w["l4_fc1_w"][0][unshuffle] + assert torch.equal(codes[0::2], ck[f"layers.4.experts.{e}.w3.weight"]) + assert torch.equal(codes[1::2], ck[f"layers.4.experts.{e}.w1.weight"]) + scales = _unswizzled(_bytes(w["l4_fc1_s"][0]))[unshuffle] + assert torch.equal(scales[0::2], _bytes(ck[f"layers.4.experts.{e}.w3.weight_scale"])) + assert torch.equal(scales[1::2], _bytes(ck[f"layers.4.experts.{e}.w1.weight_scale"])) + + +@_each_world +def test_tiny_layout_is_applied_once_per_load(world): + """A second `load` into the same parameters starts again from the stored bytes, so it lands + on the same layout rather than shuffling the first one's result again.""" + ck = _tiny_checkpoint(world) + model = _Shell(_tiny(world), 3, world, "cpu") + W.MODEL_WEIGHTS.load(model, dict(ck)) + first = {k: _bytes(v.data).clone() for k, v in model.model.w.items() if k.startswith("l5_fc")} + W.MODEL_WEIGHTS.load(model, dict(ck)) + for key, value in first.items(): + assert torch.equal(_bytes(model.model.w[key].data), value), key + + +def _write_shards(ck: dict, directory: Path, files: int = 3) -> None: + """`ck` as Mistral-named shard files with their index, keys dealt round-robin, so one + expert's tensors -- and w1's and w3's scalars -- land in different files.""" + from safetensors.torch import save_file + + names = [f"consolidated-{i + 1:05d}-of-{files:05d}.safetensors" for i in range(files)] + weight_map = {key: names[i % files] for i, key in enumerate(sorted(ck))} + for name in names: + part = {k: v.contiguous() for k, v in ck.items() if weight_map[k] == name} + save_file(part, str(directory / name)) + (directory / W.SHARD_INDEX).write_text(json.dumps({"weight_map": weight_map})) + + +class _ReadLog: + """`safe_open` that records every key read through it.""" + + def __init__(self, real): + self.real, self.keys = real, [] + + def __call__(self, *args, **kwargs): + log = self + + class Handle: + def __enter__(self): + self.h = log.real(*args, **kwargs).__enter__() + return self + + def __exit__(self, *exc): + return self.h.__exit__(*exc) + + def get_slice(self, key): + log.keys.append(key) + return self.h.get_slice(key) + + return Handle() + + +@_each_rank +def test_tiny_load_streams_from_the_shard_files(world, rank, tmp_path, monkeypatch): + """With the checkpoint's shard files found through the directory the engine records on + the config, every source is read from them -- the given dict is never read: here it + holds meta tensors, which cannot be copied out -- and the result is bit for bit the + dict path's. Each key is read once; of the other ranks' experts only the scalars are read, + and nothing of the DSpark draft. Each shard file's cached pages are dropped once, after + its sources are read.""" + ck = _tiny_checkpoint(world) + _write_shards(ck, tmp_path) + log = _ReadLog(W.safe_open) + dropped = [] + monkeypatch.setattr(W, "safe_open", log) + monkeypatch.setattr(W, "_drop_page_cache", lambda path: dropped.append(path.name)) + streamed = _Shell(_tiny(world), rank, world, "cpu") + streamed.model.model_config.pretrained_config._name_or_path = str(tmp_path) + W.MODEL_WEIGHTS.load(streamed, {k: torch.empty_like(v, device="meta") for k, v in ck.items()}) + monkeypatch.undo() + direct = _Shell(_tiny(world), rank, world, "cpu") + W.MODEL_WEIGHTS.load(direct, dict(ck)) + got, want = dict(streamed.named_parameters()), dict(direct.named_parameters()) + assert set(got) == set(want) + for key in want: + assert torch.equal(_bytes(got[key].data), _bytes(want[key].data)), key + + read = set(log.keys) + assert len(read) == len(log.keys), "a key was read twice" + lo, hi = _TINY_LOCAL * rank, _TINY_LOCAL * rank + _TINY_LOCAL + off = {k for k in ck if _is_expert(k) and not lo <= _expert_of(k)[1] < hi} + own = {k for k in ck if not _is_draft(k)} - off + assert read == own | {k for k in off if _is_expert_scalar(k)} + assert sorted(dropped) == sorted(p.name for p in tmp_path.glob("*.safetensors")) + + +@_each_world +@pytest.mark.parametrize("with_index", [True, False]) +def test_tiny_load_checkpoint_reads_the_directory(world, with_index, tmp_path): + """Given the checkpoint directory alone, as the engine gives it, the result is bit for bit + the dict path's: the files come from the directory's index, or without one from the shard + files' headers.""" + ck = _tiny_checkpoint(world) + _write_shards(ck, tmp_path) + if not with_index: + (tmp_path / W.SHARD_INDEX).unlink() + loaded = _Shell(_tiny(world), 2, world, "cpu") + W.MODEL_WEIGHTS.load_checkpoint(loaded, tmp_path) + direct = _Shell(_tiny(world), 2, world, "cpu") + W.MODEL_WEIGHTS.load(direct, dict(ck)) + got, want = dict(loaded.named_parameters()), dict(direct.named_parameters()) + assert set(got) == set(want) + for key in want: + assert torch.equal(_bytes(got[key].data), _bytes(want[key].data)), key + + +@_each_world +def test_tiny_load_falls_back_to_the_dict_without_an_index(world, tmp_path): + """A recorded directory without the shard index is not the checkpoint: the dict is read.""" + ck = _tiny_checkpoint(world) + model = _Shell(_tiny(world), 2, world, "cpu") + model.model.model_config.pretrained_config._name_or_path = str(tmp_path) + W.MODEL_WEIGHTS.load(model, dict(ck)) + + +@_each_world +def test_tiny_load_without_draft_tensors(world): + """The DSpark draft is never asserted on: a checkpoint without it loads too.""" + ck = {k: v for k, v in _tiny_checkpoint(world).items() if not _is_draft(k)} + W.MODEL_WEIGHTS.load(_Shell(_tiny(world), 1, world, "cpu"), ck) + + +@_each_world +def test_tiny_load_without_off_window_codes(world): + """The other ranks' expert codes and block scales are never read nor asserted present.""" + ck = _tiny_checkpoint(world) + for e in range(_TINY_LOCAL, _TINY_LOCAL * world): + for proj in ("w1", "w2", "w3"): + del ck[f"layers.3.experts.{e}.{proj}.weight"] + del ck[f"layers.3.experts.{e}.{proj}.weight_scale"] + W.MODEL_WEIGHTS.load(_Shell(_tiny(world), 0, world, "cpu"), ck) + + +def _load_raises(ck: dict, match: str, world: int, rank: int = 0) -> None: + with pytest.raises(AssertionError, match=match): + W.MODEL_WEIGHTS.load(_Shell(_tiny(world), rank, world, "cpu"), ck) + + +@_each_world +def test_tiny_load_refuses_a_vocab_parallel_lm_head(world): + """The whole `output.weight` does not fit a tensor-parallel `[vocab/world, hidden]` lm_head, + and nothing slices it to fit.""" + model = _Shell(_tiny(world), 1, world, "cpu", lm_rows=_TINY["vocab_size"] // world) + with pytest.raises(AssertionError, match="output.weight"): + W.MODEL_WEIGHTS.load(model, _tiny_checkpoint(world)) + + +@pytest.mark.parametrize( + "key", + [ + "layers.3.experts.1.w3.weight", + "layers.3.experts.1.w1.weight_scale", + "layers.3.experts.0.w3.weight_scale_2", + "layers.3.experts.1.w2.input_scale", + ], +) +@_each_world +def test_tiny_load_refuses_a_missing_in_window_tensor(key, world): + ck = _tiny_checkpoint(world) + del ck[key] + _load_raises(ck, f"checkpoint key missing: {key}", world) + + +@pytest.mark.parametrize("world, expert", [(4, 5), (8, 13)], ids=["world4", "world8"]) +def test_tiny_load_refuses_a_missing_off_window_scalar(world, expert): + """The other ranks' scalars are read for the per-layer asserts, so they must be there: here + those of an expert of rank 2 at four ranks, of rank 6 at eight.""" + ck = _tiny_checkpoint(world) + del ck[f"layers.3.experts.{expert}.w2.input_scale"] + _load_raises(ck, f"checkpoint key missing: layers.3.experts.{expert}.w2.input_scale", world) + + +@_each_world +def test_tiny_load_refuses_an_unknown_tensor(world): + ck = _tiny_checkpoint(world) + ck["layers.2.attention.wq_extra.weight"] = torch.zeros(4) + _load_raises(ck, "checkpoint coverage", world) + + +@pytest.mark.parametrize( + "key", + [ + "layers.0.attention.wq.weight_scale", + "layers.0.feed_forward.w3.weight_scale", + "layers.4.shared_experts.w2.weight_scale", + "layers.4.attention.indexer.wq_b.weight_scale", + "layers.5.attention.wo.weight_scale", + ], +) +@_each_world +def test_tiny_load_refuses_a_scale_that_is_not_ue8m0(key, world): + """Rank 0 checks the whole grid: the bad block is the last one, which a tensor-parallel + rank 0 would never read.""" + ck = _tiny_checkpoint(world) + ck[key] = ck[key].clone() + ck[key].view(-1)[-1] *= 1.5 + _load_raises(ck, "not a positive normal power of two", world) + + +@_each_world +def test_tiny_load_refuses_a_two_dimensional_fp8_scale(world): + """A `[N/128, K/128]` block-scale layout is not this one's.""" + ck = _tiny_checkpoint(world) + key = "layers.4.attention.indexer.wq_b.weight_scale" + ck[key] = _grid(ck[key]).contiguous() + _load_raises(ck, r"not \[N/128, 1, K/128, 1\]", world) + + +@pytest.mark.parametrize( + "world, expert", + [(4, 0), (4, 6), (8, 0), (8, 14)], + ids=["world4-0", "world4-6", "world8-0", "world8-14"], +) +@pytest.mark.parametrize("scalar", ["weight_scale_2", "input_scale"]) +def test_tiny_load_refuses_w1_and_w3_scalars_that_differ(world, expert, scalar): + """On this rank's experts and on the others' alike.""" + ck = _tiny_checkpoint(world) + key = f"layers.5.experts.{expert}.w3.{scalar}" + ck[key] = ck[key] * 2 + count = _TINY_LOCAL * world + want = f"layers.5: w1 and w3 {scalar} differ on 1 of {count} experts, first expert {expert}:" + _load_raises(ck, want, world) + + +@_each_world +@pytest.mark.parametrize("proj", ["w1", "w2"]) +def test_tiny_load_refuses_an_input_scale_that_varies_over_the_experts(proj, world): + """Rank 0 sees the other ranks' experts' input_scale: one off-window expert disagreeing -- + here the last rank's, at eight ranks on the other node -- is a quantization rank 0's window + would not share.""" + ck = _tiny_checkpoint(world) + last = _TINY_LOCAL * world - 1 + for p in {proj, "w3"} if proj == "w1" else {proj}: + key = f"layers.6.experts.{last}.{p}.input_scale" + ck[key] = ck[key] * 2 + want = f"layers.6: {proj}.input_scale is not one value over the experts: expert {last}" + _load_raises(ck, want, world) + + +@pytest.mark.parametrize("value", [0.0, -1e-4, float("inf"), float("nan")]) +@_each_world +def test_tiny_load_refuses_a_scalar_that_is_not_a_positive_finite_scale(value, world): + ck = _tiny_checkpoint(world) + ck["layers.2.experts.3.w2.weight_scale_2"] = torch.tensor(value) + want = "layers.2: w2.weight_scale_2 of expert 3 is .*not a positive finite scale" + _load_raises(ck, want, world) + + +def test_shared_scalar_refuses_copies_that_differ(): + """The row transform asserts on its own copies too, whatever ran before it.""" + one = torch.tensor(0.5) + assert W._shared_scalar((one, one.clone(), one.clone()), None) is one + with pytest.raises(AssertionError, match="1 of 3 copies"): + W._shared_scalar((one, one.clone(), torch.tensor(0.25)), None) + + +@pytest.mark.parametrize( + "value, exact", + [ + (1.0, True), + (2.0**-126, True), + (2.0**127, True), + (2.0**-15, True), + (0.0, False), + (-(2.0**-12), False), + (3.0 * 2.0**-12, False), + (2.0**-130, False), # subnormal + (float("inf"), False), + (float("nan"), False), + ], +) +def test_ue8m0_assert(value, exact): + s = torch.full((2, 3), 2.0**-10) + s[1, 2] = value + if exact: + W._assert_ue8m0(s) + else: + with pytest.raises(AssertionError, match="1 of 6"): + W._assert_ue8m0(s) + + +@pytest.mark.parametrize("world", [4, 8]) +def test_native_dspark_manifest_shards_and_off_coverage(world): + native = { + "n_layers": 3, + "depth": 3, + "target_activation_layers": "1,18,36,54,71", + "dspark": {"markov_rank": 512}, + } + # Tensor dimensions from the checkpoint's safetensors header, independently + # of the table's shape lambdas. Values encode the sliced axis so a row / + # column or head-boundary error is visible without allocating the weights. + projections = { + "wq": (8192, 6144, 0), + "wkv_b": (12288, 512, 0), + "wo": (6144, 8192, 1), + "w1": (24576, 6144, 0), + "w3": (24576, 6144, 0), + "w2": (6144, 24576, 1), + } + shards = {key: [] for key in projections} + for rank in range(world): + core = _core(_CONFIG, world=world, rank=rank) + off = W.MODEL_WEIGHTS.manifest(core) + core.model_config.spec_config = SimpleNamespace( + _draft_model_config=native, target_layer_ids=[1, 18, 36, 54, 71] + ) + manifest = W.MODEL_WEIGHTS.manifest(core) + added = set(manifest) - set(off) + assert len(added) == 38 + keys = {manifest[name][0][0] for name in added} + assert len(keys) == 38 and all(key.startswith("mtp_dspark.") for key in keys) + assert not keys.intersection(W.MODEL_WEIGHTS.expected_unconsumed(core, dict.fromkeys(keys))) + with torch.device("meta"): + declared = W.MODEL_WEIGHTS.declare(W.MODEL_WEIGHTS.dims(core)) + assert tuple(declared["draft_target_projection"].shape) == (6144, 30720) + assert tuple(declared["draft_joint"].shape) == (1536, 7168) + assert tuple(declared["draft_joint_bias"].shape) == (1536,) + assert tuple(declared["draft_mask"].shape) == (2, 6144) + assert all(declared[name].dtype == torch.bfloat16 for name in added) + for role, (rows, cols, axis) in projections.items(): + ids = torch.arange((rows, cols)[axis]) + source = ( + ids[:, None].expand(rows, cols) if axis == 0 else ids[None, :].expand(rows, cols) + ) + key, index, transform = manifest[f"l0_draft_{role}"][0] + assert index is None and transform is None + sliced = source + assert tuple(sliced.shape) == tuple(declared[f"l0_draft_{role}"].shape) + shards[role].append(sliced[:, 0] if axis == 0 else sliced[0]) + core.model_config.spec_config = None + assert keys.issubset(W.MODEL_WEIGHTS.expected_unconsumed(core, dict.fromkeys(keys))) + for role, parts in shards.items(): + rows, cols, axis = projections[role] + expected = torch.arange((rows, cols)[axis]) + for part in parts: + torch.testing.assert_close(part, expected) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/__init__.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/__init__.py new file mode 100644 index 000000000000..46c7fb1a43fa --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/__init__.py @@ -0,0 +1,7 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Package marker. + +These test basenames can repeat other targets'. Without packages, pytest's +default import mode loads both under one module name and refuses the second. +""" diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/_chunk_equivalence_ranks.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/_chunk_equivalence_ranks.py new file mode 100644 index 000000000000..d0057470bfe4 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/_chunk_equivalence_ranks.py @@ -0,0 +1,124 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The rank side of `test_modeling_v2_chunk_equivalence.py`. + +`rank_probe` runs on every rank of the launcher's MPI world (four ranks at tep4, eight over two +nodes of four at tep8), through the session the engines use, while no engine is up (the session +runs one task at a time). It reports the rank's device free memory and the context steps the +target ran since the last probe, and installs or removes an observe-only record of those steps: +`PrefillTarget.forward` and the module's `allgather` wrapped so each context step appends what +it received -- host values the step already holds -- and how many selection gathers it made. +Both wrappers call the original unchanged. + +Nothing here touches a device at import: the module is imported on every rank, and a CUDA call +there would open a context on the wrong device. +""" + +from __future__ import annotations + +import gc +import importlib +import socket +from types import SimpleNamespace + +import torch + +from tensorrt_llm._utils import global_mpi_rank + +TARGET = ( + "tensorrt_llm._torch.modeling_v2.models.mistral_large4." + "mistral_large_4_nvfp4__sm_103__tep.modeling" +) + +# This rank's context-step records, and the originals they wrap while the record is installed. +_RECORD = SimpleNamespace(steps=[], originals=None) + + +class _CountedGather: + """The target's selection gather, counted: the same call, the same result.""" + + def __init__(self, gather): + self.gather = gather + self.calls = 0 + + def __call__(self, input, sizes, group, process_group=None): + self.calls += 1 + return self.gather(input, sizes, group, process_group) + + def __getattr__(self, name): + return getattr(self.gather, name) + + +def _install(modeling) -> None: + if _RECORD.originals is not None: + return + forward, gather = modeling.PrefillTarget.forward, modeling.allgather + counted = _CountedGather(gather) + + def recorded(self, attn_metadata, *args, **kwargs): + md = attn_metadata + ctx = int(md.num_contexts) + q_split = md.indexer_prefill_q_split + if q_split is not None: + q_split = (int(q_split[0]), int(q_split[1]), [int(s) for s in q_split[2]]) + counted.calls = 0 + out = forward(self, attn_metadata, *args, **kwargs) + _RECORD.steps.append( + dict( + rank=self.core.tp_rank, + contexts=ctx, + cached=[int(c) for c in md.kv_cache_params.num_cached_tokens_per_seq[:ctx]], + new=[int(n) for n in md.seq_lens[:ctx].tolist()], + tc=int(md.num_ctx_tokens), + gen=int(md.num_tokens) - int(md.num_ctx_tokens), + skip_ctx=bool(md.skip_indexer_for_ctx_reqs), + cached_kv=bool(md.enable_context_mla_with_cached_kv), + q_split=q_split, + chunks=[ + (c.token_start, c.token_end, c.k_token_start, c.k_token_end) + for c in (md.indexer_prefill_chunks or []) + ], + gathers=counted.calls, + ) + ) + return out + + _RECORD.originals = (forward, gather) + modeling.PrefillTarget.forward = recorded + modeling.allgather = counted + + +def _remove(modeling) -> None: + if _RECORD.originals is not None: + modeling.PrefillTarget.forward, modeling.allgather = _RECORD.originals + _RECORD.originals = None + + +def rank_probe(action: str) -> dict: + """This rank's device free memory after a garbage collection and a cache release, what of + the device this process's torch allocator still holds, and the context steps recorded since + the last probe. `action` is "install" (start recording), "collect" (keep recording) or + "remove" (restore the target's own methods).""" + rank = global_mpi_rank() + # The engine's device for this rank: its local rank among the node's devices. + device = rank % torch.cuda.device_count() + torch.cuda.set_device(device) + gc.collect() + torch.cuda.empty_cache() + free, total = torch.cuda.mem_get_info(device) + modeling = importlib.import_module(TARGET) + steps, _RECORD.steps = _RECORD.steps, [] + if action == "install": + _install(modeling) + elif action == "remove": + _remove(modeling) + return dict( + rank=rank, + host=socket.gethostname(), + device=device, + free=free, + total=total, + allocated=torch.cuda.memory_allocated(device), + reserved=torch.cuda.memory_reserved(device), + steps=steps, + ) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/_retrieval_at_length_ranks.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/_retrieval_at_length_ranks.py new file mode 100644 index 000000000000..01225bb3d33a --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/_retrieval_at_length_ranks.py @@ -0,0 +1,228 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The rank side of `test_modeling_v2_retrieval_at_length.py`. + +`rank_probe` runs on every rank of the launcher's MPI world (four ranks at tep4, eight over two +nodes of four at tep8), through the session the engines use, while no engine is up (the session +runs one task at a time). "install" starts two observe-only records on the rank, "collect" hands +over and clears what they recorded, "remove" stops both and restores the target's own methods. + +* Device memory: a thread reads the rank's device through NVML every `POLL_S` seconds and keeps + each change of its used memory with the wall-clock time. Device-wide, so the KV pool and every + workspace outside torch's allocator count. NVML, not `torch.cuda.mem_get_info`: the engine + captures its decode graphs in CUDA's global capture mode, which refuses CUDA calls from any + other thread of the capturing process. +* Context steps: `PrefillTarget.forward`, the module's selection gather and its indexer prefill + entries (`indexer_k_cache_gather_op`, `fp8_mqa_logits`, `cute_dsl_gvr_topk_prefill`) are + wrapped, so each context step appends the host values it received (cached prefixes, new + lengths, the query split, the planned chunks), how many selection gathers it made and the + most keys any call of each entry received. Every wrapper calls the original unchanged. + +Nothing here touches a device at import: the module is imported on every rank, and a CUDA call +there would open a context on the wrong device. +""" + +from __future__ import annotations + +import gc +import importlib +import socket +import threading +import time +from types import SimpleNamespace + +import torch + +from tensorrt_llm._utils import global_mpi_rank + +TARGET = ( + "tensorrt_llm._torch.modeling_v2.models.mistral_large4." + "mistral_large_4_nvfp4__sm_103__tep.modeling" +) +POLL_S = 0.05 + +# The key count each observed entry's call states: the gather's `num_tokens`, the scorer's +# key rows, the top-k's columns. +_KEYS = { + "indexer_k_cache_gather_op": lambda *a, **k: int(a[4]), + "fp8_mqa_logits": lambda *a, **k: int(a[1][0].shape[0]), + "cute_dsl_gvr_topk_prefill": lambda *a, **k: int(a[0].shape[-1]), + "allgather": lambda *a, **k: 0, +} + +_STATE = SimpleNamespace(steps=[], originals=None, poller=None, poller_error=None) + + +class _Observed: + """One of the target's module-level entries, observed: the same call, the same result; how + many calls and the most keys any of them received since the last `take`.""" + + def __init__(self, entry, keys_of): + self.entry = entry + self.keys_of = keys_of + self.calls = 0 + self.max_keys = 0 + + def __call__(self, *args, **kwargs): + self.calls += 1 + self.max_keys = max(self.max_keys, self.keys_of(*args, **kwargs)) + return self.entry(*args, **kwargs) + + def __getattr__(self, name): + return getattr(self.entry, name) + + def take(self) -> tuple[int, int]: + out = (self.calls, self.max_keys) + self.calls = self.max_keys = 0 + return out + + +class _Poller: + """The rank's device's used memory through NVML, every change kept with its time.""" + + def __init__(self, device: int): + import pynvml + + uuid = str(torch.cuda.get_device_properties(device).uuid).lower().removeprefix("gpu-") + self.handle = None + try: + pynvml.nvmlInit() + for index in range(pynvml.nvmlDeviceGetCount()): + handle = pynvml.nvmlDeviceGetHandleByIndex(index) + other = pynvml.nvmlDeviceGetUUID(handle) + other = (other.decode() if isinstance(other, bytes) else other).lower() + if other.removeprefix("gpu-") == uuid: + self.handle = handle + break + if self.handle is None: + raise RuntimeError(f"no NVML device has the UUID of CUDA device {device} ({uuid})") + info = pynvml.nvmlDeviceGetMemoryInfo(self.handle) + except pynvml.NVMLError as exc: + raise RuntimeError(f"NVML: {exc}") from exc + self.nvml = pynvml + self.total = int(info.total) + self.lock = threading.Lock() + self.changes = [(time.time(), int(info.used))] + self.stopped = threading.Event() + self.thread = threading.Thread(target=self._run, daemon=True, name="retrieval-memory") + self.thread.start() + + def _run(self) -> None: + last = self.changes[-1][1] + while not self.stopped.wait(POLL_S): + used = int(self.nvml.nvmlDeviceGetMemoryInfo(self.handle).used) + if used != last: + with self.lock: + self.changes.append((time.time(), used)) + last = used + + def take(self) -> list[tuple[float, int]]: + """Every change since the last take, the value then in effect first.""" + with self.lock: + changes, self.changes = self.changes, self.changes[-1:] + return changes + + def stop(self) -> None: + self.stopped.set() + self.thread.join() + self.nvml.nvmlShutdown() + + +def _install(modeling) -> None: + if _STATE.originals is not None: + return + forward = modeling.PrefillTarget.forward + originals = {name: getattr(modeling, name) for name in _KEYS} + observed = {name: _Observed(entry, _KEYS[name]) for name, entry in originals.items()} + + def recorded(self, attn_metadata, *args, **kwargs): + md = attn_metadata + ctx = int(md.num_contexts) + cached = [int(c) for c in md.kv_cache_params.num_cached_tokens_per_seq[:ctx]] + new = [int(n) for n in md.seq_lens[:ctx].tolist()] + q_split = md.indexer_prefill_q_split + if q_split is not None: + q_split = (int(q_split[0]), int(q_split[1]), [int(s) for s in q_split[2]]) + chunks = [ + (c.token_start, c.token_end, c.k_token_start, c.k_token_end) + for c in (md.indexer_prefill_chunks or []) + ] + for entry in observed.values(): + entry.take() + out = forward(self, attn_metadata, *args, **kwargs) + calls = {name: entry.take() for name, entry in observed.items()} + _STATE.steps.append( + dict( + rank=self.core.tp_rank, + time=time.time(), + contexts=ctx, + cached=cached, + new=new, + tc=int(md.num_ctx_tokens), + gen=int(md.num_tokens) - int(md.num_ctx_tokens), + skip_ctx=bool(md.skip_indexer_for_ctx_reqs), + cached_kv=bool(md.enable_context_mla_with_cached_kv), + last_position=max((c + n for c, n in zip(cached, new)), default=0) - 1, + q_split=q_split, + chunks=chunks, + gathers=calls["allgather"][0], + calls={name: calls[name][0] for name in _KEYS if name != "allgather"}, + keys={name: calls[name][1] for name in _KEYS if name != "allgather"}, + ) + ) + return out + + _STATE.originals = (forward, originals) + modeling.PrefillTarget.forward = recorded + for name, entry in observed.items(): + setattr(modeling, name, entry) + + +def _remove(modeling) -> None: + if _STATE.originals is not None: + forward, originals = _STATE.originals + modeling.PrefillTarget.forward = forward + for name, entry in originals.items(): + setattr(modeling, name, entry) + _STATE.originals = None + + +def rank_probe(action: str) -> dict: + """This rank's device free memory now (`mem_get_info`, after a garbage collection and a + cache release), the device memory changes and context steps recorded since the last probe. + `action` is "install" (start recording), "collect" (keep recording) or "remove" (stop).""" + rank = global_mpi_rank() + # The engine's device for this rank: its local rank among the node's devices. + device = rank % torch.cuda.device_count() + torch.cuda.set_device(device) + gc.collect() + torch.cuda.empty_cache() + free, total = torch.cuda.mem_get_info(device) + modeling = importlib.import_module(TARGET) + if action == "install": + _install(modeling) + if _STATE.poller is None and _STATE.poller_error is None: + try: + _STATE.poller = _Poller(device) + except (ImportError, OSError, RuntimeError) as exc: + _STATE.poller_error = repr(exc) + poller = _STATE.poller + memory = poller.take() if poller is not None else [] + nvml_total = poller.total if poller is not None else None + steps, _STATE.steps = _STATE.steps, [] + if action == "remove": + _remove(modeling) + if poller is not None: + poller.stop() + _STATE.poller = None + return dict( + rank=rank, + host=socket.gethostname(), + device=device, + free=free, + total=total, + nvml_total=nvml_total, + nvml_error=_STATE.poller_error, + memory=memory, + steps=steps, + ) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_chunk_equivalence.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_chunk_equivalence.py new file mode 100644 index 000000000000..6ffbb7d8b2d2 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_chunk_equivalence.py @@ -0,0 +1,503 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Chunk equivalence of Mistral-Large-4-1T-A52B-NVFP4 on its sm_103 tep target, at tep4 (four +ranks) and at tep8 (eight ranks over two nodes of four). + +Under chunked prefill the engine prefills a prompt in steps of up to `max_num_tokens`. +Every chunk after a request's first is a context request over a cached prefix -- the rope +append at the prefix's positions, the indexer's chunk plan and key gather over the whole +prefix, sparse MLA over selections that reach into it -- and from `q_split_threshold` (8192) +context rows on, the engine splits a step's queries over every TP rank (the four, or the eight +of both nodes) and the target all-gathers their selections. The same text prompts of up to +32768 tokens go through two engines, one at a time, and greedy decoding must give the same +tokens: + +* chunked: the 1M config (max_seq_len 1048576, chunked prefill, sixteen sequences, the + fp8_ds_mla latent KV cache at its default dtype, CUDA graphs over batches 1, 2, 4, 8 and 16, + padded) at max_num_tokens 8192, so each prompt is prefilled in steps of 8192 tokens and a + last shorter one, and the engine splits each full step 2048 rows per rank at tep4, 1024 at + tep8; +* delivered: the engine the target was delivered at (max_seq_len = max_num_tokens = 32768, + chunked prefill off), where each prompt is one context step with no cached prefix and no + split. + +The target runs the same code in both: each step picks its paths from its own sizes, and only +what the engines' quantities size differs -- the indexer's logits tiles (`_prefill_logits_budget`: +2**28 logits in the chunked engine, 2**24 delivered) and the held context workspace's rows +(`_ctx_workspace_rows`: 2**19 chunked, 65536 delivered). Every step of either engine attends its +context KV, at most 32768 rows, through the held workspace with layer 35's gather kept for the +NoPE sharers. + +Every prompt passes `index_topk` (2048), so the indexer scores and selects in both engines and +every layer runs, layer 35's NoPE sharers and the shared-indexer layers included. The step +composition is read back from the engines' iteration statistics, and what each rank's context +steps received from an observe-only record on every rank: the query split, the indexer chunk +plan, the cached prefixes and how many selection gathers the step made. + +Same tokens, not the same bits: the row-bound GEMMs run at another M (8192 against up to 32768 +rows), cuBLAS may pick another algorithm for them, and row-count rounding through the FP8 +latent moves top-1/top-2 margins. Neither engine repeats its own near ties either: what the +model writes after a passkey changes between two generations of one prompt in one engine. So +the compared continuations are ones that win by a wide margin -- a passkey's digits, a counting +sequence, a cycle of month names -- `max_tokens` stays at 32 or fewer, and a mismatch reports +the first divergence and every step's top-1/top-2 logprob margin from both engines. + +Text only: no image, no chat template. Runs with the ranks already in place, on four GB300s at +tep4 or on eight over two nodes of four at tep8: pytest under `trtllm-llmapi-launch`, as the +whole-model accuracy class runs, with the checkpoint under `LLM_MODELS_ROOT` as +`Mistral-Large-4-1T-A52B-NVFP4` (or `mistral-large-4-nvfp4`) in its Mistral-native format, +which both engines load with checkpoint_format "mistral". The ranks' devices are not +there for two engines at once, so one process holds them in turn. The second engine sizes its +KV pool from the memory the first left free, so before the first boots and after each shuts +down every rank reports its device's free memory, through the launcher's MPI session (the +engines' own channel, idle between them): the first engine's shutdown must give back what it +took. The same probe installs and collects the per-step record (`_chunk_equivalence_ranks.py`); +outside the launcher there is no such session, and the checks on what the probe reports are +skipped. Skipped where the checkpoint is absent or off sm_103; each world size is skipped below +its own ranks (four GPUs at tep4, eight ranks at tep8) and inside a launch of another world size. +""" + +from __future__ import annotations + +import random +import site +from pathlib import Path +from types import SimpleNamespace + +import pytest +import torch +from test_common.llm_data import llm_models_root +from utils.util import skip_num_gpus_less_than + +from tensorrt_llm import LLM, SamplingParams +from tensorrt_llm._torch.modeling_v2 import MODELING_V2_VALIDATE_ENV +from tensorrt_llm._torch.models.checkpoints.mistral.config_loader import MistralConfigLoader +from tensorrt_llm._utils import get_sm_version +from tensorrt_llm.executor.utils import create_mpi_comm_session, get_spawn_proxy_process_env +from tensorrt_llm.inputs import TokensPrompt +from tensorrt_llm.llmapi import CudaGraphConfig, KvCacheConfig +from tensorrt_llm.llmapi.mpi_session import get_mpi_world_size + +from . import _chunk_equivalence_ranks as rank_side + + +def _ranks() -> int: + """The ranks this process can serve: the MPI world a launcher started (eight over two nodes + of four, where each node sees four devices), or, unlaunched, the devices the LLM API spawns + its workers on -- as the integration conftest's `skip_less_mpi_world_size` counts.""" + world = get_mpi_world_size() + return torch.cuda.device_count() if world == 1 else world + + +def _launched_elsewhere(world: int) -> bool: + """A launcher started an MPI world of another size: an engine of `world` ranks does not + start inside it.""" + return get_mpi_world_size() not in (1, world) + + +#: The engines' world sizes, each gated on its own ranks: tep4 on four GPUs, tep8 on eight +#: ranks over two nodes of four. +WORLDS = [ + pytest.param( + 4, + marks=[ + skip_num_gpus_less_than(4), + pytest.mark.skipif( + _launched_elsewhere(4), reason="tep4 needs a launch of four ranks or none" + ), + ], + id="tep4", + ), + pytest.param( + 8, + marks=[ + pytest.mark.skipif(_ranks() < 8, reason="tep8 needs eight ranks"), + pytest.mark.skipif( + _launched_elsewhere(8), reason="tep8 needs a launch of eight ranks or none" + ), + ], + id="tep8", + ), +] + +pytestmark = [ + pytest.mark.skipif( + get_sm_version() != 103, + reason="the mistral_large4 modeling_v2 targets are certified on sm_103 only", + ), + # The module's engines start the LLM API's result-dispatch thread on their first request + # and stop it when the fixture shuts them down. + pytest.mark.threadleak(enabled=False), +] + +CHECKPOINTS = ("Mistral-Large-4-1T-A52B-NVFP4", "mistral-large-4-nvfp4") + +INDEX_TOPK = 2048 +# sparse_attention_config.q_split_threshold's default (llmapi/llm_args.py): a step of at least +# this many context rows is split over the TP ranks. +Q_SPLIT_THRESHOLD = 8192 +CHUNK = 8192 +DELIVERED_LEN = 32768 +# What one engine may keep of a rank's device once it is shut down: its free memory must come +# back to within this of what the rank had before the first engine booted, or the second +# engine would size its pool from a device the first still holds. The engine itself is far +# above it (about 75 GiB of weights and a KV pool of about 150 GiB per rank at tep8, twice the +# weights at tep4); what a process keeps of an engine it has shut down was 1.82 to 2.56 GiB per +# rank on two GB300 nodes at tep8, a few hundred MiB of it still allocated in torch's caching +# allocator. +RELEASE_SLACK = 4 << 30 + +COMMON = dict( + enable_attention_dp=False, + max_batch_size=16, + cuda_graph_config=CudaGraphConfig(max_batch_size=16, enable_padding=True), + kv_cache_config=KvCacheConfig(enable_block_reuse=False, free_gpu_memory_fraction=0.85), +) +ENGINES = { + "chunked": dict(max_seq_len=1048576, max_num_tokens=CHUNK, enable_chunked_prefill=True), + "delivered": dict( + max_seq_len=DELIVERED_LEN, max_num_tokens=DELIVERED_LEN, enable_chunked_prefill=False + ), +} + +HEAD = ( + "There is an important piece of information hidden inside a lot of irrelevant text. " + "Find it and memorize it. I will quiz you about it.\n\n" +) +FILLER = ( + "The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again. " +) +QUESTION = "\nWhat is the pass key? The pass key is" +MONTHS = ( + "January, February, March, April, May, June, July, August, September, October, " + "November, December, " +) + +# (name, kind, prompt tokens at most, passkey depth): one prompt within one chunk, the others +# two to four chunks with the passkey in their first chunk or a middle one, so the question in +# the last chunk reads it through the cached prefix. +PROMPTS = ( + ("passkey-6k-mid", "passkey", 6000, 0.5), + ("passkey-12k-early", "passkey", 12000, 0.25), + ("passkey-20k-late", "passkey", 20000, 0.75), + ("passkey-31k-early", "passkey", 31000, 0.1), + ("counting-16k", "counting", 16000, None), + ("months-28k", "months", 28000, None), +) +# A passkey's answer is the key's three tokens (the space, then its five digits in two +# pieces). What the model writes after the key is a near tie (top-1/top-2 margins well under a +# nat) that differs between two boots of one and the same engine, so it is not generated. A +# sequence: its next 32 tokens. +MAX_TOKENS = {"passkey": 3, "counting": 32, "months": 32} + + +# ------------------------------------------------------------------- every rank, between engines + + +def _reach_ranks(session) -> None: + """Make the rank side importable on every rank: a task's function is unpickled by module + path before the session hands the rank this process's `sys.path`, so first add the + directory that holds the rank side's top-level package.""" + base = Path(rank_side.__file__).resolve().parents[rank_side.__name__.count(".")] + session.submit_sync(site.addsitedir, str(base)) + + +def _probe(session, action: str, world: int) -> dict[int, dict] | None: + """`rank_probe(action)` on every rank, by rank; None without the launcher's session.""" + if session is None: + return None + reports = {r["rank"]: r for r in session.submit_sync(rank_side.rank_probe, action)} + assert sorted(reports) == list(range(world)), sorted(reports) + return reports + + +# ------------------------------------------------------------------------------- the prompts + + +def _checkpoint_dir() -> Path | None: + """The checkpoint's directory, found by its Mistral-native config, params.json.""" + root = llm_models_root() + if root is None: + return None + for name in CHECKPOINTS: + if (Path(root) / name / "params.json").is_file(): + return Path(root) / name + return None + + +def _fit(tokenizer, text_of, budget: int) -> tuple[list[int], int]: + """The longest `text_of(n)` whose ids (BOS included, as the LLM API's own tokenization adds + it) fit in `budget`, by bisection over n.""" + lo, hi = 0, 1 + while len(tokenizer.encode(text_of(hi))) <= budget: + lo, hi = hi, hi * 2 + while hi - lo > 1: + mid = (lo + hi) // 2 + if len(tokenizer.encode(text_of(mid))) <= budget: + lo = mid + else: + hi = mid + return tokenizer.encode(text_of(lo)), lo + + +def _build(tokenizer, name: str, kind: str, budget: int, depth: float | None) -> SimpleNamespace: + if kind == "passkey": + key = random.Random(budget).randrange(10000, 100000) + + def text_of(reps: int) -> str: + before = round(reps * depth) + needle = f"\nThe pass key is {key}. Remember it. {key} is the pass key.\n" + return HEAD + FILLER * before + needle + FILLER * (reps - before) + QUESTION + + ids, _ = _fit(tokenizer, text_of, budget) + expect = str(key) + elif kind == "counting": + + def text_of(n: int) -> str: + return "Count up by one:\n" + "".join(f"{i}, " for i in range(1, n + 1)) + + _, n = _fit(tokenizer, text_of, budget) + expect = f"{n + 1}, {n + 2}" + # The prompt ends on the comma: the continuation is " n+1, n+2, ...". + ids = tokenizer.encode(text_of(n).rstrip()) + else: + + def text_of(n: int) -> str: + return "The months, over and over:\n" + MONTHS * n + + _, n = _fit(tokenizer, text_of, budget) + expect = "January, February" + ids = tokenizer.encode(text_of(n).rstrip()) + # Every key count stays within the delivered 32768, the widths the prefill rows' catalog + # cells hold. + assert INDEX_TOPK < len(ids) <= min(budget, DELIVERED_LEN), (name, len(ids)) + return SimpleNamespace( + name=name, + kind=kind, + ids=ids, + expect=expect, + params=SamplingParams(max_tokens=MAX_TOKENS[kind], temperature=0.0, logprobs=2), + ) + + +def _context_steps(llm: LLM) -> list[tuple[int, int]]: + """(context requests, context tokens) of each iteration since the last call that ran any + context rows.""" + steps = [] + for record in llm.get_stats(timeout=2): + ifb = record.get("inflightBatchingStats") or {} + if ifb.get("numContextRequests", 0): + steps.append((ifb["numContextRequests"], ifb["numCtxTokens"])) + return steps + + +def _margins(output) -> list[tuple[int, float, int | None, float]]: + """Each generated step's (top-1 token, its logprob, top-2 token, its logprob).""" + rows = [] + for step in output.logprobs: + ranked = [(t, lp.logprob) for t, lp in sorted(step.items(), key=lambda kv: -kv[1].logprob)] + ranked.append((None, float("-inf"))) + (t1, l1), (t2, l2) = ranked[:2] + rows.append((t1, l1, t2, l2)) + return rows + + +def _run(path: Path, engine: str, prompts, world: int) -> tuple[list, dict]: + """Every prompt through one engine of `world` ranks, one at a time. Without `prompts`, they + are built with this engine's tokenizer (the checkpoint's tekken.json), so both engines read + the same ids.""" + results = {} + with LLM( + str(path), + checkpoint_format="mistral", + modeling_v2="require", + env_overrides={MODELING_V2_VALIDATE_ENV: "1"}, + enable_iter_perf_stats=True, + tensor_parallel_size=world, + moe_expert_parallel_size=world, + **COMMON, + **ENGINES[engine], + ) as llm: + print(f"CHUNKEQ {engine} kv_cache_capacity {llm.get_kv_cache_capacity()}", flush=True) + if prompts is None: + prompts = [_build(llm.tokenizer, *spec) for spec in PROMPTS] + _context_steps(llm) + for p in prompts: + out = llm.generate(TokensPrompt(prompt_token_ids=p.ids), p.params).outputs[0] + steps = _context_steps(llm) + margins = _margins(out) + results[p.name] = SimpleNamespace( + tokens=list(out.token_ids), text=out.text, steps=steps, margins=margins + ) + print( + f"CHUNKEQ {engine} {p.name} tokens={len(p.ids)} context_steps={steps} " + f"text={out.text!r} ids={list(out.token_ids)} " + f"margins={[round(m[1] - m[3], 3) for m in margins]}", + flush=True, + ) + return prompts, results + + +def _print_probe( + label: str, reports: dict[int, dict] | None, baseline: dict[int, dict] | None +) -> None: + if reports is None: + return + for rank, report in sorted(reports.items()): + delta = "" + if baseline is not None: + delta = f" vs baseline {report['free'] - baseline[rank]['free']:+d} B" + print( + f"CHUNKEQ probe {label} rank {rank} {report['host']}:{report['device']} free " + f"{report['free'] / 2**30:.3f} of {report['total'] / 2**30:.3f} GiB{delta}, torch " + f"allocated {report['allocated']} B reserved {report['reserved']} B, " + f"{len(report['steps'])} context steps", + flush=True, + ) + for step in reports[0]["steps"]: + print(f"CHUNKEQ step {label} {step}", flush=True) + + +@pytest.fixture(scope="module", params=WORLDS) +def runs(request): + world = request.param + path = _checkpoint_dir() + if path is None: + pytest.skip(f"checkpoint {CHECKPOINTS} is not under LLM_MODELS_ROOT ({llm_models_root()})") + # The checkpoint's config as the engines read it from params.json. + config = MistralConfigLoader().load_pretrained_config(str(path)) + # The engines' own session under trtllm-llmapi-launch: a singleton every LLM in this + # process reuses. Unlaunched, each engine spawns its own workers and no probe runs. + session = None + if get_spawn_proxy_process_env(): + session = create_mpi_comm_session(world) + _reach_ranks(session) + baseline = _probe(session, "install", world) + _print_probe("before-chunked", baseline, None) + prompts, chunked = _run(path, "chunked", None, world) + after_chunked = _probe(session, "collect", world) + _print_probe("after-chunked", after_chunked, baseline) + _, delivered = _run(path, "delivered", prompts, world) + after_delivered = _probe(session, "remove", world) + _print_probe("after-delivered", after_delivered, baseline) + return SimpleNamespace( + world=world, + prompts={p.name: p for p in prompts}, + chunked=chunked, + delivered=delivered, + selecting_layers=sum(role == "full" for role in config.index_sharing), + baseline=baseline, + after_chunked=after_chunked, + after_delivered=after_delivered, + ) + + +# ------------------------------------------------------------------------------------ tests + + +def _chunks(n: int) -> list[tuple[int, int]]: + return [(1, min(CHUNK, n - lo)) for lo in range(0, n, CHUNK)] + + +def _probed(runs) -> None: + if runs.baseline is None: + pytest.skip("the per-rank probe runs through trtllm-llmapi-launch's MPI session") + + +def test_the_first_engine_gives_its_devices_back(runs): + """After the chunked engine shuts down, every rank's device is back within RELEASE_SLACK of + its free memory before that engine booted, so the delivered engine sizes its pool from the + same devices.""" + _probed(runs) + for rank, before in runs.baseline.items(): + after = runs.after_chunked[rank] + assert (after["host"], after["device"]) == (before["host"], before["device"]) + assert before["free"] - after["free"] <= RELEASE_SLACK, ( + rank, + before["free"], + after["free"], + ) + + +def test_the_chunked_engine_splits_its_chunks(runs): + """Each prompt is prefilled in steps of 8192 tokens and a last shorter one, one request per + step. A step of q_split_threshold context rows or more is one the engine splits over every + rank (`_split_prefill_queries` with attention DP off at tp 4 or tp 8, pinned on CPU by the + target's engine-configuration test); the prompt set must hold at least one.""" + split_steps = 0 + for name, p in runs.prompts.items(): + steps = runs.chunked[name].steps + assert steps == _chunks(len(p.ids)), (name, len(p.ids), steps) + split_steps += sum(tokens >= Q_SPLIT_THRESHOLD for _, tokens in steps) + assert split_steps >= 1 + + +def _shares(tc: int, rank: int, world: int) -> tuple[int, int, list[int]]: + """`_split_prefill_queries` at tp `world`, attention DP off: rank r's + [tc*r//world, tc*(r+1)//world).""" + bounds = [tc * r // world for r in range(world + 1)] + return bounds[rank], bounds[rank + 1], [bounds[r + 1] - bounds[r] for r in range(world)] + + +def test_the_chunked_engine_splits_its_full_chunks(runs): + """On every rank, each context step of q_split_threshold rows or more carries the engine's + split, this rank's share of it, and one selection gather per selecting indexer layer (none + where the engine skipped the context indexer); a shorter step carries no split and gathers + nothing. Every rank sees the same steps. Among them are full 8192-row chunks over a cached + prefix.""" + _probed(runs) + per_rank = {r: report["steps"] for r, report in runs.after_chunked.items()} + shape = [(s["tc"], s["q_split"] and s["q_split"][2], s["gathers"]) for s in per_rank[0]] + over_prefix = 0 + for rank, steps in per_rank.items(): + assert [(s["tc"], s["q_split"] and s["q_split"][2], s["gathers"]) for s in steps] == shape + for s in steps: + assert s["rank"] == rank and s["cached_kv"], s + if s["tc"] >= Q_SPLIT_THRESHOLD: + assert s["q_split"] == _shares(s["tc"], rank, runs.world), (rank, s) + assert s["gathers"] == (0 if s["skip_ctx"] else runs.selecting_layers), (rank, s) + over_prefix += rank == 0 and not s["skip_ctx"] and sum(s["cached"]) > 0 + else: + assert s["q_split"] is None and s["gathers"] == 0, (rank, s) + assert over_prefix >= 1, shape + + +def test_the_delivered_engine_prefills_each_prompt_whole(runs): + """Each prompt is one context step of one request: no cached prefix and no split; on every + rank the probe reached, no context step over cached KV, no split and no gather.""" + for name, p in runs.prompts.items(): + assert runs.delivered[name].steps == [(1, len(p.ids))], (name, runs.delivered[name].steps) + for rank, report in (runs.after_delivered or {}).items(): + for s in report["steps"]: + assert not s["cached_kv"] and s["q_split"] is None and s["gathers"] == 0, (rank, s) + + +def _report(name: str, chunked, delivered) -> str: + first = next( + (i for i, (a, b) in enumerate(zip(chunked.tokens, delivered.tokens)) if a != b), + min(len(chunked.tokens), len(delivered.tokens)), + ) + lines = [f"{name}: first divergence at generated step {first}"] + for label, run in (("chunked", chunked), ("delivered", delivered)): + lines.append(f" {label}: text={run.text!r} ids={run.tokens}") + lines.append( + f" {label} (top-1, top-2, margin): " + + ", ".join(f"({t1}, {t2}, {l1 - l2:.3f})" for t1, l1, t2, l2 in run.margins) + ) + return "\n".join(lines) + + +@pytest.mark.parametrize("name", [spec[0] for spec in PROMPTS]) +def test_same_greedy_tokens(runs, name): + """Both engines decode the same greedy tokens from the same prompt.""" + chunked, delivered = runs.chunked[name], runs.delivered[name] + assert chunked.tokens == delivered.tokens, _report(name, chunked, delivered) + + +@pytest.mark.parametrize("name", [spec[0] for spec in PROMPTS]) +def test_the_answer_is_in_both(runs, name): + """The passkey, or the sequence's next items, is in both engines' text: the prompts ask + something both engines answer.""" + expect = runs.prompts[name].expect + assert expect in runs.chunked[name].text, (name, expect, runs.chunked[name].text) + assert expect in runs.delivered[name].text, (name, expect, runs.delivered[name].text) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_ctx_bound.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_ctx_bound.py new file mode 100644 index 000000000000..6dd70e59fa62 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_ctx_bound.py @@ -0,0 +1,623 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The bound on the tep target's context-row attention at the 1M config (max_seq_len 1048576, +chunked prefill: a held context workspace of 2**19 rows, `_ctx_workspace_rows`), on one GPU, at a +rank's heads at tep4 (four ranks: 16 of the 64 MLA heads) and at tep8 (eight ranks over two nodes +of four: 8 of them). + +`PrefillTarget._ctx_workspace` picks each step's path on host ints from the step's context KV N +(every context request's cached and new rows): while N fits the held rows the workspace path (the +rows dequantized into the first N rows of the held bf16 [2**19, 576] buffer, then FlashMLA's sparse +prefill), past them the decode path (FlashMLA's sparse fp8 decode straight off the pool, each +context row a one-row sequence, in sub-batches of at most 8192 rows). `PrefillTarget._attn_ctx` runs +it. Both are driven here as the forward drives them, at the rank's live heads (16 at tep4, 8 at +tep8) first in the kernels' 64-head query (the others zero), over an fp8_ds_mla latent pool of the +63-owner layout built on the device: every byte is 0xFF (NaN as e4m3, as a tile's fp32 scale and as +bf16 rope) but the owner rows of the driven layers on the requests' pages, so a row read from +anywhere else turns its output NaN. Each step runs layer 35 (the NoPE owner), then layer 1 (a RoPE +owner, whose gather overwrites the workspace's first N rows), then layer 39, which attends over +layer 35's latent: on a workspace-path step whose N fits the held rows twice it reads layer 35's +rows where layer 35's gather kept them, beside the workspace's first N, and otherwise gathers them +again; generation rows behind the context rows run as `_attn_gen` runs them, on +cute_dsl_ds_mla_sparse_decode at the rank's heads (FlashMLA's arithmetic, unpadded) beside the +context rows' FlashMLA schedulers. + +Every step is checked three ways: every output row finite; each layer's output against the other +path on the same step (a second target whose held workspace is sized so that it takes it); and +up to 512 rows, the sub-batch edges among them, against the decode entry's fp32 reference over +slots the test computes itself (block * 63 + owner row) * 32 + offset. The two paths read the +same pool bytes through the same bf16 arithmetic and are bitwise equal on every row FlashMLA's +decode runs in one split. Its scheduler splits some rows over their keys (its `num_splits` +says which: none of an 8192-row sub-batch here, some of a ragged one, every row of a 16-row +tail) and its combine kernel merges the parts in fp32, which rounds otherwise than the prefill +kernel's one pass: those rows are held to the entry's band instead (relative L2 per row and head +under 1e-2, the rows under 4e-3). The generation rows run the same code on both and are bitwise +equal. + +The steps: the KV-capacity estimate's (one fresh 32768-token prompt, split over the ranks: the +workspace path, which allocates the buffer, layer 35's rows kept); the same 32768-row chunk at +context KV 2**19 (workspace, nothing kept) and 2**19 + 1 (decode, 4 x 8192); the 1M gate prompt's +last chunk, 32751 rows over 1015808 cached keys (8192 x 3 and 8175) with three generation rows +behind; a packed step of chunks over 1032192, 524288, 0 and 2048 cached keys with twelve generation +rows; a 16-row tail over 1048544 cached keys; and nine 1000-token prompts on the engine's dense +causal index (the indexer skipped, layer 35's rows kept). The buffer is never reallocated after the +estimate's step, and the context attention's peak on a decode-path step stays under the estimate +step's. + +The prefill kernel skips an index past its KV rows as it skips -1, with no error and no NaN, so +a workspace shorter than the step's context KV drops selections silently: every workspace-path +run is checked to hold exactly the step's context KV rows with every selection inside them, and +controls run the target's prefill over the same step with the workspace one row short and with +it cut to the step's new tokens (the engine's `num_ctx_kv_tokens`), which the checks refuse. +The query's heads past the rank's (16..63 at tep4, 8..63 at tep8) hold no head of this rank and +stay zero: NaN there leaves every output row bitwise unchanged on both paths, so nothing the +kernels compute on them reaches the rank's heads. + +Every step runs at both head counts (`-k tep4`, `-k tep8`); the steps, the pool and the paths +do not depend on them. +""" + +from __future__ import annotations + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_103__tep import ( # noqa: E501 + modeling as M, +) + + +def _skip_reason() -> str | None: + if not torch.cuda.is_available(): + return "needs a GPU" + from tensorrt_llm._utils import get_sm_version + + if get_sm_version() != 103: + return "the mistral_large4 modeling_v2 targets are certified on sm_103 only" + return None + + +pytestmark = pytest.mark.skipif(_skip_reason() is not None, reason=str(_skip_reason())) + +_LAYERS = 72 +#: Every fourth layer from 39 on attends over layer 35's latent (the checkpoint's +#: kv_source_layers); the pool holds the 63 owners' rows. +_SOURCES = [35 if i >= 39 and i % 4 == 3 else i for i in range(_LAYERS)] +_OWNERS = [i for i in range(_LAYERS) if _SOURCES[i] == i] +#: Driven in this order: the NoPE source, a RoPE owner, a sharer of the source. +_DRIVEN = (35, 1, 39) +_TPB = 32 +_TOPK = 2048 +#: The rank's live heads of the 64 MLA heads: 16 at tep4 (four ranks), 8 at tep8 (eight ranks +#: over two nodes of four). +HEADS = [pytest.param(16, id="tep4"), pytest.param(8, id="tep8")] +_KERNEL_HEADS = 64 +_LAT, _NOPE, _ROW_BYTES = 576, 512, 656 +#: qk_nope_head_dim + qk_rope_head_dim = 192, every layer's softmax scale. +_SCALE = 192**-0.5 +#: md.block_table at the 1M config: 16 requests of ceil(1048576 / 32) + 4 columns. +_TABLE = (16, 32772) +_ROWS_B = 2**19 +_SUB = 8192 +_FILL = 0xFF +_DRAW = 1 << 18 +_GATED = 512 +_MIB = 2**20 + + +def test_constants_are_the_rows_certifications(): + """The held rows at the 1M config (16 requests of 32768 tokens, max_seq_len 1048576, chunked + prefill) are the workspace's 2**19 ceiling; the decode path's sub-batch is 8192 rows.""" + assert (M._CTX_WORKSPACE_ROWS, M._CTX_DECODE_ROWS) == (_ROWS_B, _SUB) + features = SimpleNamespace(chunked_prefill=True, cache_reuse=False) + md = SimpleNamespace( + max_num_tokens=32768, max_num_requests=16, max_seq_len=1048576, runtime_features=features + ) + assert M._ctx_workspace_rows(md) == _ROWS_B + assert len(_OWNERS) == 63 and _OWNERS.index(35) == 35 and _SOURCES[39] == 35 + + +# ---------------------------------------------------------------- the step, built on the device + + +def _draw_rows(n: int, g: torch.Generator, nope: bool) -> torch.Tensor: + """`n` fp8_ds_mla rows [656] as the writer leaves them: e4m3 codes, power-of-two fp32 tile + scales 2^-13..2^0, the bf16 rope key (zero on a NoPE layer).""" + dev = g.device + codes = ((torch.rand(n, _NOPE, generator=g, device=dev) * 2 - 1) * 448).to(torch.float8_e4m3fn) + scale = torch.exp2(torch.randint(-13, 1, (n, 4), generator=g, device=dev).float()) + rope = torch.zeros(n, 64, device=dev) + if not nope: + rope = torch.randn(n, 64, generator=g, device=dev) * 3 + rope = rope.to(torch.bfloat16) + return torch.cat([codes.view(torch.uint8), scale.view(torch.uint8), rope.view(torch.uint8)], 1) + + +def _pool(kv_lens: list[int], g: torch.Generator): + """The pool [blocks, 63, 32, 656] and the block table: each request on distinct pages + drawn over the whole pool, the driven owners' rows written on them.""" + dev = g.device + counts = [-(-kv // _TPB) for kv in kv_lens] + blocks = sum(counts) + 1 + shape = (blocks, len(_OWNERS), _TPB, _ROW_BYTES) + pool = torch.full(shape, _FILL, dtype=torch.uint8, device=dev) + pages = torch.randperm(blocks, generator=g, device=dev)[: sum(counts)] + table = torch.zeros(_TABLE, dtype=torch.int32, device=dev) + first = 0 + for r, n in enumerate(counts): + table[r, :n] = pages[first : first + n].int() + first += n + for layer in (1, 35): + row = _OWNERS.index(layer) + for lo in range(0, pages.numel(), _DRAW // _TPB): + part = pages[lo : lo + _DRAW // _TPB] + pool[part, row] = _draw_rows(part.numel() * _TPB, g, layer == 35).view(-1, _TPB, 656) + return pool, table + + +def _selections(pos: torch.Tensor, g: torch.Generator, dense: bool) -> torch.Tensor: + """int32 [rows, 2048] request-local positions for rows at positions `pos`, as the indexer + hands them over: a row that sees at most 2048 keys has all of them, then -1; else 2048 + distinct keys, one in each of 2048 equal strata of [0, p], p itself the last, in a random + order. `dense`: the engine's causal index of a step it skipped the indexer (0..p, -1 after).""" + dev = g.device + n_keys = pos[:, None].long() + 1 + col = torch.arange(_TOPK + 1, device=dev)[None] + short = torch.where(col[:, :-1] < n_keys, col[:, :-1], -1) + if dense: + return short.int() + edge = col * n_keys // _TOPK + draw = ( + edge[:, :-1] + + ( + torch.rand(pos.numel(), _TOPK, generator=g, device=dev) * (edge[:, 1:] - edge[:, :-1]) + ).long() + ) + draw[:, -1] = pos + del edge + sel = torch.where(n_keys > _TOPK, draw, short) + del draw, short + order = (torch.rand(sel.shape, generator=g, device=dev) + (sel < 0)).argsort(1) + return sel.gather(1, order).int() + + +def _cumsum(lens: list[int]) -> torch.Tensor: + return torch.tensor([0, *lens], dtype=torch.int64).cumsum(0) + + +class _Step(SimpleNamespace): + pass + + +def _offsets(ctx) -> dict: + """The context step entries `_ctx_workspace` reads, as `_ctx_kv_offsets` takes them off the + engine's metadata under chunked prefill.""" + cached, new = [c for c, _ in ctx], [n for _, n in ctx] + md = SimpleNamespace( + enable_context_mla_with_cached_kv=True, + ctx_cached_token_indptr=_cumsum(cached), + ctx_kv_indptr=_cumsum([c + n for c, n in ctx]), + max_ctx_seq_len=max(new), + num_ctx_cached_tokens=sum(cached), + ) + offsets = { + k: v.cuda() if isinstance(v, torch.Tensor) else v + for k, v in M._ctx_kv_offsets(md, len(ctx)).items() + } + # num_ctx_kv_tokens is the engine's count: the step's new context keys only. + return dict( + num_contexts=len(ctx), num_ctx_tokens=sum(new), num_ctx_kv_tokens=sum(new), **offsets + ) + + +def _step(ctx, gen=(), *, heads: int, seed: int, dense: bool = False, pool_kv=None) -> _Step: + """A step of context requests `ctx` ((cached, new) each) and generation rows `gen` (the + KV length of each, one query at its last position) behind them, with the state the + target reads: the context offsets, the pool, md.block_table, the selections and one query + per layer kind, whose first `heads` heads are the rank's (the rest zero).""" + dev = torch.device("cuda") + g = torch.Generator(device=dev).manual_seed(seed) + kv_lens = [c + n for c, n in ctx] + list(gen) + pool, table = _pool(pool_kv or kv_lens, g) + tc = sum(n for _, n in ctx) + rows = tc + len(gen) + pos = torch.cat( + [torch.arange(c, c + n, device=dev) for c, n in ctx] + + [torch.tensor([kv - 1 for kv in gen], device=dev, dtype=torch.long)] + ) + req = torch.cat( + [torch.full((n,), r, dtype=torch.int32, device=dev) for r, (_, n) in enumerate(ctx)] + + [torch.arange(len(ctx), len(kv_lens), dtype=torch.int32, device=dev)] + ) + sel = _selections(pos, g, dense) + q = torch.zeros(rows, _KERNEL_HEADS, _LAT, dtype=torch.bfloat16, device=dev) + q[:, :heads] = (torch.randn(rows, heads, _LAT, generator=g, device=dev) * 0.5).to(q.dtype) + q_nope = q.clone() + q_nope[..., _NOPE:] = 0 + step = dict( + latent_pool=pool, + latent_kv=pool.view(-1, _TPB, 1, _ROW_BYTES), + block_table=table, + tokens_per_block=_TPB, + **_offsets(ctx), + ) + return _Step( + step=step, tc=tc, rows=rows, n_kv=sum(c + n for c, n in ctx), req=req, pos=pos, sel=sel, + q={35: q_nope, 1: q, 39: q_nope}, g=g, heads=heads, + ) # fmt: skip + + +#: cute_dsl_ds_mla_sparse_decode's handles by head count, built once. +_NATIVE = {} + + +def _native_decode(heads: int): + """The generation rows' kernel handle at the rank's `heads`, as `build_kernels` builds it.""" + if heads not in _NATIVE: + _NATIVE[heads] = M.build_ds_mla_decode(torch.cuda.current_device(), num_heads=heads) + return _NATIVE[heads] + + +def _target(heads: int, rows: int = _ROWS_B) -> M.PrefillTarget: + """A prefill target with only what its context and generation attention read off the core: + the held workspace's rows as `_derive_first_step` sets them (the 1M config's 2**19 unless + `rows` says otherwise: 0 sends every step down the decode path, a step's own context KV + sends it down the workspace path), layer 35 the one shared source, as the core builds it, + and the generation rows' native decode at the rank's `heads`.""" + target = M.PrefillTarget.__new__(M.PrefillTarget) + target.core = SimpleNamespace( + ctx_workspace_rows=rows, + # The generation rows on cute_dsl_ds_mla_sparse_decode at the rank's heads, as shipped. + gen_attn_flash_mla=False, + _mla_decode=_native_decode(heads), + lat_dim=_LAT, + idx_topk=_TOPK, + softmax_scale=_SCALE, + _stride_factor=len(_OWNERS) * _TPB, + _pool_row=[_OWNERS.index(_SOURCES[i]) for i in range(_LAYERS)], + shared_source=[i == 35 for i in range(_LAYERS)], + ) + return target + + +def _run(target, s: _Step, *, cut=None) -> SimpleNamespace: + """The step's latent attention on every driven layer, as `PrefillTarget.forward` runs it: + the context path chosen once (`_ctx_workspace`, on the rows the target's held workspace was + allocated with on its first context step), then per layer the context rows (`_attn_ctx`) + and the generation rows behind them (the selections converted to slots, then `_attn_gen`, + the native decode at the rank's heads), each layer's output kept. On a workspace-path step layer 35's gather + is kept for layer 39 exactly when the step's context KV fits the held rows twice. With `cut` + (a control), the workspace path keeps only the first `cut` rows of the step's context KV, + and every layer gathers into them.""" + ctx = target._ctx_workspace(s.step, s.tc, torch.device("cuda")) + if ctx.ws_buf is not None: + held = target._ws_buf.shape[0] + assert (ctx.kept is not None) == (2 * s.n_kv <= held), (s.n_kv, held) + if cut is not None: + assert ctx.ws_buf is not None and cut < ctx.ws_buf.shape[0] + ctx = SimpleNamespace( + kv_req=ctx.kv_req[:cut], kv_pos=ctx.kv_pos[:cut], ws_off=ctx.ws_off, + ws_buf=ctx.ws_buf[:cut], kept=None, ws={}, + ) # fmt: skip + sched = M.get_mla_metadata()[0] + outs = {} + for layer in _DRIVEN: + L = SimpleNamespace(index=layer, kv_source=_SOURCES[layer]) + lat_out = torch.empty(s.rows, s.heads, _NOPE, dtype=torch.bfloat16, device="cuda") + lat_out.view(torch.int16).fill_(-1) + target._attn_ctx(s.step, L, s.q[layer][: s.tc], lat_out[: s.tc], s.req, s.sel, ctx) + if s.rows > s.tc: + # The context attention leaves the generation rows alone. + assert bool((lat_out[s.tc :].view(torch.int16) == -1).all()) + slots = M.convert_req_index_to_global( + s.req[s.tc :], s.step["block_table"], s.sel[s.tc :], _TPB, _TOPK, + target.core._stride_factor, target.core._pool_row[L.kv_source], + ) # fmt: skip + target._attn_gen(s.step, L, s.q[layer][s.tc :], lat_out[s.tc :], slots, sched) + outs[layer] = lat_out + if ctx.ws_buf is not None: + # Layer 35's gather is remembered for its sharer exactly when it went to the kept rows. + assert (35 in ctx.ws) == (ctx.kept is not None) + torch.cuda.synchronize() + return SimpleNamespace(ctx=ctx, outs=outs) + + +def _reference_rows(s: _Step) -> torch.Tensor: + """The rows gated against the fp32 reference: each sub-batch's first and last row, every + generation row, the rest drawn.""" + edges = {0, s.tc - 1} + for lo in range(0, s.tc, _SUB): + edges |= {lo, min(lo + _SUB, s.tc) - 1} + edges |= set(range(s.tc, s.rows)) + edges = sorted(e for e in edges if 0 <= e < s.rows) + rest = _GATED - len(edges) + if rest > 0 and s.tc > len(edges): + drawn = torch.randperm(s.tc, generator=s.g, device=s.g.device)[:rest].tolist() + edges = sorted(set(edges) | set(drawn)) + return torch.tensor(edges, device="cuda") + + +def _split_rows(ctx, tc: int) -> torch.Tensor: + """The context rows FlashMLA's decode scheduler split over their keys on a decode-path run + (more than one split in its `num_splits` prefix sums), whose parts its combine kernel merges + in fp32; none on a workspace-path run.""" + split = torch.zeros(tc, dtype=torch.bool, device="cuda") + if ctx.ws_buf is None: + for lo, hi in ctx.sub_batches: + starts = ctx.sched[hi - lo].num_splits + split[lo:hi] = (starts[1:] - starts[:-1]) > 1 + return split + + +def _check(s: _Step, run: SimpleNamespace, other: SimpleNamespace) -> dict: + """Every row finite; the sharer equal to its source; each layer's context rows bitwise the + other path's on every row neither run split over its keys, a split row within the band; + the generation rows bitwise; the gated rows against the fp32 reference.""" + from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.flash_mla_sparse_decode import ( # noqa: E501 + flash_mla_sparse_decode, + ) + + for r in (run, other): + if r.ctx.ws_buf is not None: + # The prefill kernel skips a selection past the workspace's rows as it skips -1. + held_rows = r.ctx.ws_buf.shape[0] + assert held_rows == r.ctx.kv_req.shape[0] == s.n_kv, ( + f"the workspace holds {held_rows} rows of the step's {s.n_kv}" + ) + assert r.ctx.kept is None or r.ctx.kept.shape[0] == s.n_kv + sel = s.sel[: s.tc] + idx = torch.where(sel >= 0, sel + r.ctx.ws_off[s.req[: s.tc]][:, None], -1) + assert int(idx.max()) < held_rows + rows = _reference_rows(s) + split = _split_rows(run.ctx, s.tc) | _split_rows(other.ctx, s.tc) + report = dict(split_rows=int(split.sum())) + for layer in _DRIVEN: + out, alt = run.outs[layer], other.outs[layer] + bad = ~torch.isfinite(out.float()).flatten(1).all(1) + assert not bool(bad.any()), f"layer {layer}: {int(bad.sum())} rows read an unheld byte" + differs = (out.view(torch.int16) != alt.view(torch.int16)).flatten(1).any(1) + rel = float((out.float() - alt.float()).norm() / alt.float().norm()) + unsplit = int((differs[: s.tc] & ~split).sum()) + assert unsplit == 0, f"layer {layer}: {unsplit} unsplit rows differ between paths" + assert not bool(differs[s.tc :].any()), f"layer {layer}: generation rows differ" + if bool(split.any()): + _gate(out[: s.tc][split], alt[: s.tc][split]) + src = _OWNERS.index(_SOURCES[layer]) + sel = s.sel[rows] + req = s.req[rows].long() + key = sel.clamp_min(0).long() + page = s.step["block_table"][req[:, None], key // _TPB].long() + slots = torch.where(sel >= 0, (page * len(_OWNERS) + src) * _TPB + key % _TPB, -1) + ref = flash_mla_sparse_decode.reference( + s.q[layer][rows], s.step["latent_kv"], slots.int(), None, _SCALE, out[rows] + ) + _gate(out[rows], ref) + report[layer] = dict(rows_differ=int(differs.sum()), rel_l2_paths=f"{rel:.2e}") + assert torch.equal(run.outs[39], run.outs[35]), "the sharer differs from its source" + assert not torch.equal(run.outs[1], run.outs[35]) + return report + + +def _gate(out: torch.Tensor, ref: torch.Tensor) -> None: + """The decode entry's band: relative L2 per (row, head) under 1e-2, the whole under 4e-3.""" + o, r = out.float(), ref.float() + per = (o - r).norm(dim=-1) / r.norm(dim=-1).clamp_min(1e-30) + whole = float((o - r).norm() / r.norm()) + assert float(per.max()) < 1e-2 and whole < 4e-3, (float(per.max()), whole) + + +def _path(run) -> str: + return "workspace" if run.ctx.ws_buf is not None else "decode" + + +# ----------------------------------------------------------------------------- the steps + + +@pytest.fixture(scope="module", params=HEADS) +def held(request): + """The production target's first context step, the KV-capacity estimate's: one fresh + 32768-token prompt (N = 32768, a query-split step with the indexer running), at the rank's + heads of the world (16 at tep4, 8 at tep8). It takes the workspace path, allocates the held + [2**19, 576] buffer and keeps layer 35's rows in it for layer 39 (N fits twice); the context + attention's peak allocation over the step's three layers is recorded.""" + heads = request.param + target = _target(heads) + s = _step([(0, 32768)], heads=heads, seed=1) + torch.cuda.synchronize() + torch.cuda.reset_peak_memory_stats() + base = torch.cuda.memory_allocated() + run = _run(target, s) + peak = torch.cuda.max_memory_allocated() - base + assert _path(run) == "workspace" + buf = target._ws_buf + assert buf.shape == (_ROWS_B, _LAT) and buf.dtype == torch.bfloat16 + assert run.ctx.ws_buf.data_ptr() == buf.data_ptr() and run.ctx.ws_buf.shape == (32768, _LAT) + assert run.ctx.kept is not None and run.ctx.kept.shape == (32768, _LAT) + other = _run(_target(heads, 0), s) + assert _path(other) == "decode" and other.ctx.sub_batches == [ + (lo, lo + _SUB) for lo in range(0, 32768, _SUB) + ] + report = _check(s, run, other) + assert report["split_rows"] == 0 + print( + f"\n[estimate] {heads} heads, N 32768 workspace; peak {peak / _MIB:.0f} MiB (buffer " + f"included); {report}" + ) + del s, run, other + torch.cuda.empty_cache() + return SimpleNamespace(target=target, ptr=buf.data_ptr(), peak=peak, heads=heads) + + +def test_estimate_step_allocates_the_workspace(held): + assert held.target._ws_buf.data_ptr() == held.ptr + assert held.peak >= _ROWS_B * _LAT * 2 + + +def test_switch_at_the_workspace_rows(held): + """The same 32768-row chunk -- the same queries, selections and pool bytes -- once over + 491520 cached keys (N = 2**19: the workspace path, fitting the held rows once, so layer 39 + gathers layer 35's rows again) and once over 491521 (N = 2**19 + 1: the decode path in four + 8192-row sub-batches, none split): equal outputs, bitwise. + + Controls: the workspace path over the first 2**19 - 1 rows of the step's KV (the last row's + own key dropped) and over its first 32768 (the step's new tokens, `num_ctx_kv_tokens`), with + the held buffer filled with 0xFF first so that no read past the cut finds a row an earlier + gather left there, runs without an error; the checks refuse both. One dropped key of 2048 can + carry too little weight to move a bf16 output, so the workspace's rows are checked against + the step's context KV as well as its output; the cut to the new tokens moves the output of + every layer.""" + new = 32768 + cached = _ROWS_B - new + at_b = _step([(cached, new)], heads=held.heads, seed=2, pool_kv=[_ROWS_B + 1]) + past_b = _Step(**vars(at_b)) + past_b.step = dict(at_b.step, **_offsets([(cached + 1, new)])) + past_b.n_kv = _ROWS_B + 1 + assert at_b.n_kv == _ROWS_B + ws = _run(held.target, at_b) + dec = _run(held.target, past_b) + assert (_path(ws), _path(dec)) == ("workspace", "decode") + assert ws.ctx.ws_buf.shape == (_ROWS_B, _LAT) and ws.ctx.kept is None + assert dec.ctx.sub_batches == [(lo, lo + _SUB) for lo in range(0, new, _SUB)] + assert sorted(dec.ctx.sched) == [_SUB] + report = _check(at_b, ws, dec) + assert report["split_rows"] == 0 + print(f"\n[switch] N 2**19 workspace vs 2**19 + 1 decode: {report}") + assert at_b.step["num_ctx_kv_tokens"] == new + for cut in (_ROWS_B - 1, new): + held.target._ws_buf.view(torch.int16).fill_(-1) + short = _run(held.target, at_b, cut=cut) + differ, nan = {}, {} + for layer in _DRIVEN: + out, full = short.outs[layer], ws.outs[layer] + moved = (out.view(torch.int16) != full.view(torch.int16)).flatten(1).any(1) + differ[layer] = int(moved.sum()) + nan[layer] = int((~torch.isfinite(out.float())).flatten(1).any(1).sum()) + print(f"[switch] control: workspace cut to {cut} rows; rows differing {differ}, NaN {nan}") + if cut == new: + assert all(differ.values()) + with pytest.raises(AssertionError, match="the workspace holds"): + _check(at_b, short, dec) + assert held.target._ws_buf.data_ptr() == held.ptr + + +def test_last_chunk_of_the_envelope_prompt(held): + """The 1M gate's 1048559-token prompt's last chunk, 32751 rows over 1015808 cached keys, + with three generation rows behind it: the decode path in sub-batches of 8192 x 3 and 8175 on + two schedulers, the generation rows on the native decode. Against the workspace path over a buffer of + all 1048559 rows. The context attention's peak stays under the estimate step's.""" + s = _step([(1015808, 32751)], gen=[300001, 2049, 17], heads=held.heads, seed=3) + torch.cuda.synchronize() + torch.cuda.reset_peak_memory_stats() + base = torch.cuda.memory_allocated() + run = _run(held.target, s) + peak = torch.cuda.max_memory_allocated() - base + assert _path(run) == "decode" and s.n_kv == 1048559 + assert run.ctx.sub_batches == [(0, 8192), (8192, 16384), (16384, 24576), (24576, 32751)] + assert sorted(run.ctx.sched) == [8175, 8192] + other = _run(_target(held.heads, s.n_kv), s) + assert _path(other) == "workspace" + report = _check(s, run, other) + print( + f"\n[last chunk] N {s.n_kv} decode; peak {peak / _MIB:.0f} MiB against the estimate " + f"step's {held.peak / _MIB:.0f}; {report}" + ) + assert peak < held.peak + assert held.target._ws_buf.data_ptr() == held.ptr + + +def test_packed_chunks_over_long_prefixes(held): + """Four context requests packed in one step, chunks of 16384, 8192, 4096 and 4080 rows over + 1032192, 524288, 0 and 2048 cached keys (N = 1591280), twelve generation rows behind them: + the decode path over the 32752 context rows, each request's rows through its own pages.""" + ctx = [(1032192, 16384), (524288, 8192), (0, 4096), (2048, 4080)] + gen = [1048000, 262145, 131073, 65537, 16385, 9217, 4097, 2049, 2048, 1025, 33, 1] + s = _step(ctx, gen=gen, heads=held.heads, seed=4) + run = _run(held.target, s) + assert _path(run) == "decode" and s.n_kv == 1591280 and s.tc == 32752 + assert run.ctx.sub_batches == [(0, 8192), (8192, 16384), (16384, 24576), (24576, 32752)] + other = _run(_target(held.heads, s.n_kv), s) + report = _check(s, run, other) + print(f"\n[packed] N {s.n_kv} decode, {len(gen)} generation rows: {report}") + assert held.target._ws_buf.data_ptr() == held.ptr + + +def test_short_tail_over_a_1m_prefix(held): + """A 16-row tail over 1048544 cached keys: the decode path in one 16-row call. FlashMLA's + decode splits each of so few rows over the keys and merges the parts in fp32, so it rounds + otherwise than the prefill kernel's one pass: held to the band against the workspace path, + and against the fp32 reference.""" + s = _step([(1048544, 16)], heads=held.heads, seed=5) + run = _run(held.target, s) + assert _path(run) == "decode" and run.ctx.sub_batches == [(0, 16)] + other = _run(_target(held.heads, s.n_kv), s) + report = _check(s, run, other) + assert report["split_rows"] > 0 + print(f"\n[tail] N {s.n_kv} decode, 16 rows: {report}") + assert held.target._ws_buf.data_ptr() == held.ptr + + +def test_dense_step_takes_the_workspace(held): + """Nine 1000-token prompts, every one within index_topk, so the engine skips the context + indexer and hands its causal index (0..p, -1 after): N = 9000, the workspace path, layer + 35's rows kept. Against the decode path: bitwise on its unsplit rows, the band on the rows + its 808-row second sub-batch splits.""" + s = _step([(0, 1000)] * 9, heads=held.heads, seed=6, dense=True) + run = _run(held.target, s) + assert _path(run) == "workspace" and run.ctx.ws_buf.shape == (9000, _LAT) + assert run.ctx.kept is not None + assert run.ctx.ws_off.tolist() == [1000 * r for r in range(9)] + other = _run(_target(held.heads, 0), s) + assert other.ctx.sub_batches == [(0, 8192), (8192, 9000)] + report = _check(s, run, other) + print(f"\n[dense] N {s.n_kv} workspace: {report}") + assert held.target._ws_buf.data_ptr() == held.ptr + + +def test_heads_past_the_rank_are_never_read(held): + """Two context requests (4096 rows over 4096 cached keys, a fresh 2048-row prompt) with two + generation rows behind them, on the workspace path and on the decode path. `lat_out` is the + rank's heads (16 at tep4, 8 at tep8), every element of it written (the sentinel `_run` fills + it with is gone), its gated rows within the band of the fp32 reference over the live heads + alone (`_check`). The query's heads past them are zero, as the forward leaves them; run + again with them NaN, every output row of every driven layer is bitwise the same: nothing the + kernels compute on those heads reaches the rank's. With them N(0, 0.5) instead, FlashMLA + rounds some live rows otherwise (printed; held to the band of the zero-head output): the + kernels' arithmetic on the live heads sees the other heads' values, so the target keeps them + zero, as every certified cell runs them.""" + heads = held.heads + s = _step([(4096, 4096), (0, 2048)], gen=[300001, 17], heads=heads, seed=7) + g = torch.Generator(device="cuda").manual_seed(8) + fills = { + "nan": torch.full((s.rows, _KERNEL_HEADS - heads, _LAT), float("nan"), device="cuda"), + "randn": torch.randn(s.rows, _KERNEL_HEADS - heads, _LAT, generator=g, device="cuda") / 2, + } + for rows_b in (None, 0): + target = held.target if rows_b is None else _target(heads, 0) + base = _run(target, s) + assert _path(base) == ("workspace" if rows_b is None else "decode") + other = _run(_target(heads, s.n_kv if rows_b == 0 else 0), s) + _check(s, base, other) + for layer in _DRIVEN: + assert not bool((s.q[layer][:, heads:] != 0).any()) + assert base.outs[layer].shape == (s.rows, heads, _NOPE) + moved = {} + for name, fill in fills.items(): + filled = _Step(**vars(s)) + filled.q = {} + for layer, q in s.q.items(): + filled.q[layer] = q.clone() + filled.q[layer][:, heads:] = fill.to(q.dtype) + run = _run(target, filled) + moved[name] = {} + for layer in _DRIVEN: + out, zero = run.outs[layer], base.outs[layer] + assert bool(torch.isfinite(out.float()).all()), (name, layer) + rows = (out.view(torch.int16) != zero.view(torch.int16)).flatten(1).any(1) + moved[name][layer] = int(rows.sum()) + if name == "nan": + assert moved[name][layer] == 0, layer + else: + _gate(out, zero) + print(f"\n[heads] {_path(base)}: live rows moved by heads {heads}..63 of the query {moved}") diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_ctx_workspace.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_ctx_workspace.py new file mode 100644 index 000000000000..df991e7e8da9 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_ctx_workspace.py @@ -0,0 +1,129 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The context workspace of the Mistral-Large-4-1T-A52B NVFP4 tep target: the KV rows its +dequantizing gather and prefill kernel address. Under KV block reuse a context request's KV +is its cached prefix plus its new tokens, so the workspace covers both, while the engine's +`num_ctx_kv_tokens` counts the new tokens only. Under chunked prefill every chunk after a +prompt's first is such a request, its earlier chunks the cached prefix. + +The workspace is the first KV rows of one held bf16 [rows, 576] buffer, allocated on the first +context step at the rows `_ctx_workspace_rows` takes from the engine's attention metadata (chunked +prefill and block reuse from its runtime features): 65536 at the 32768 serving config +(max_seq_len = max_num_tokens = 32768, chunked prefill and block reuse off), 2**19 at that config +with block reuse and at the 1M config (max_seq_len 1048576, chunked prefill). A step whose KV +fits the buffer twice also keeps the next KV rows for layer 35's gather; a step whose KV is past +the buffer takes the decode path, in sub-batches of at most 8192 context rows. The latent is +replicated, so every rank builds the same workspace at tep4 (four ranks) and at tep8 (eight +ranks over two nodes of four). No device: the step is built from host-side metadata.""" + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch.attention.backends.interface import AttentionRuntimeFeatures +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_103__tep import ( # noqa: E501 + modeling as M, +) + +pytestmark = pytest.mark.cpu_only + +_LAT_DIM = 576 + + +def _cumsum(lens): + return torch.tensor([0] + lens, dtype=torch.int64).cumsum(0) + + +def _metadata(cached, new): + """The fields `_ctx_kv_offsets` reads, as the engine fills them for context requests with + `cached` reused (or earlier-chunk) and `new` scheduled tokens each.""" + if any(cached): + return SimpleNamespace( + enable_context_mla_with_cached_kv=True, + ctx_cached_token_indptr=_cumsum(cached), + ctx_kv_indptr=_cumsum([c + n for c, n in zip(cached, new)]), + max_ctx_seq_len=max(new), + num_ctx_cached_tokens=sum(cached), + ) + return SimpleNamespace( + enable_context_mla_with_cached_kv=False, + mla_prepare_ctx_cu_seqlens=lambda: _cumsum(new).to(torch.int32), + seq_lens=torch.tensor(new, dtype=torch.int32), + ) + + +#: The held workspace's rows, from the engine's metadata (`_ctx_workspace_rows`, chunked prefill +#: and block reuse from its runtime features): the 32768 config's (max_num_tokens 32768, no +#: chunked prefill or block reuse), the same with block reuse at max_batch_size 16, and the 1M +#: config's (max_seq_len 1048576, chunked prefill). +_HELD = { + "32k": ((32768, 32768, False, False), 1 << 16), + "32k_reuse": ((32768, 32768, False, True), 1 << 19), + "1m": ((32768, 1048576, True, False), 1 << 19), +} + + +def _held_rows(config: str) -> int: + (max_num_tokens, max_seq_len, chunked, reuse), rows = _HELD[config] + md = SimpleNamespace( + max_num_tokens=max_num_tokens, + max_num_requests=16, + max_seq_len=max_seq_len, + runtime_features=AttentionRuntimeFeatures( + chunked_prefill=chunked, cache_reuse=reuse, chunk_size=max_num_tokens + ), + ) + assert M._ctx_workspace_rows(md) == rows + return rows + + +@pytest.mark.parametrize( + "config, cached, new", + [ + pytest.param("32k", [0, 0], [7, 33], id="32k-no_reuse"), + # A full 32768-token step: its KV fits the 65536 held rows twice, exactly. + pytest.param("32k", [0], [32768], id="32k-full_step"), + pytest.param("32k_reuse", [64, 0, 96], [5, 40, 1], id="32k_reuse-mixed_prefixes"), + pytest.param("32k_reuse", [32], [1], id="32k_reuse-one_new_token"), + pytest.param("1m", [0, 0], [7, 33], id="1m-no_reuse"), + pytest.param("1m", [64, 0, 96], [5, 40, 1], id="1m-mixed_prefixes"), + pytest.param("1m", [32], [1], id="1m-one_new_token"), + # The 16th chunk of a 1M prompt: KV 2**19, the held rows once, nothing kept. + pytest.param("1m", [15 * 32768], [32768], id="1m-chunk_at_the_held_rows"), + # A 1M prompt's last chunk (31 chunks of 32768 cached), beside a first chunk of another + # prompt: past the held rows. + pytest.param("1m", [31 * 32768, 0], [32751, 9], id="1m-chunk_over_1m_prefix"), + ], +) +def test_workspace_covers_the_cached_prefix_and_the_new_tokens(config, cached, new): + step = dict( + num_contexts=len(new), + num_ctx_tokens=sum(new), + num_ctx_kv_tokens=sum(new), + **M._ctx_kv_offsets(_metadata(cached, new), len(new)), + ) + held = _held_rows(config) + target = M.PrefillTarget.__new__(M.PrefillTarget) + target.core = SimpleNamespace(lat_dim=_LAT_DIM, ctx_workspace_rows=held) + ws = target._ctx_workspace(step, sum(new), torch.device("cpu")) + + kv_lens = [c + n for c, n in zip(cached, new)] + n_kv = sum(kv_lens) + assert target._ws_buf.shape == (held, _LAT_DIM) and target._ws_buf.dtype == torch.bfloat16 + if n_kv > held: + assert ws.ws_buf is None + assert ws.sub_batches == [(0, 8192), (8192, 16384), (16384, 24576), (24576, sum(new))] + return + buf = target._ws_buf + assert ws.ws_buf.data_ptr() == buf.data_ptr() + assert ws.ws_buf.shape == (n_kv, _LAT_DIM) + if 2 * n_kv <= held: + assert ws.kept.shape == (n_kv, _LAT_DIM) + assert ws.kept.data_ptr() == buf.data_ptr() + n_kv * _LAT_DIM * buf.element_size() + else: + assert ws.kept is None + assert ws.ws == {} + assert ws.kv_req.tolist() == [r for r, n in enumerate(kv_lens) for _ in range(n)] + assert ws.kv_pos.tolist() == [p for n in kv_lens for p in range(n)] + assert ws.ws_off.tolist() == _cumsum(kv_lens)[:-1].tolist() diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_engine_config.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_engine_config.py new file mode 100644 index 000000000000..b66de5e7c288 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_engine_config.py @@ -0,0 +1,3760 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The engine's configuration of Mistral-Large-4-1T-A52B-NVFP4 at tep4, as a served engine reads it. + +No GPU and no weights. The target serves the checkpoint's own context: max_seq_len 1048576 (its +max_position_embeddings), chunked prefill on in steps of max_num_tokens 32768, 16 requests, CUDA +graphs over batches 1, 2, 4, 8 and 16 padded, the fp8_ds_mla latent KV cache +(`MistralLarge4CacheManager`, 656 bytes a token on each KV-owning layer) at kv_cache_config's +default dtype as the CI class runs it and at the gates' fp8, block reuse off. The delivered +config (max_seq_len = max_num_tokens = 32768, chunked prefill off) stays pinned beside it. +At each config the checkpoint goes through the engine's own configuration path, with the LLM-API +options of its accuracy gate: + +* the LLM API front end: ``TorchLlmArgs`` and ``ModelLoader._update_from_hf_quant_config`` + (``llmapi/llm_utils.py``), which settle ``llm.args.quant_config``, the key the accuracy + harness looks its reference row up by; +* each rank: ``_load_config_and_create_checkpoint_loader`` (``py_executor_creator``), then the + model loader's ``_load_and_validate_config``, which builds the ``ModelConfig`` through + ``ModelConfig.from_pretrained`` and asks the modeling_v2 router for a target. The router's + ``route`` is replaced by a recorder, so what it is handed is observed rather than judged. + +What the engine then does with that configuration -- the sizes the target takes from it on its +first step (the indexer's logits budget and the held context workspace's rows), the context path +a chunk of a long prompt takes, the query split across the four TP ranks, the scheduler's chunks, +the buffers sized by max_seq_len, and what warmup, the KV-capacity estimation and CUDA-graph +capture send -- is pinned on the engine's own host code. Where that code allocates on the device, +the allocation is taken on PyTorch's meta device; where it copies to the device, the copy is the +identity. Nothing asserted depends on a visible device or on its SM count. + +The checkpoint is Mistral-native, as published (params.json, tekken.json, consolidated +safetensors), and is loaded with checkpoint_format mistral: the gates' LLM-API options +state it and the boot gate's command passes --checkpoint_format=mistral. It is the +ModelOpt MIXED_PRECISION export, whose params.json quantization_config names Mistral modules +(NVFP4 routed experts, FP8_PB_WO block-FP8 linears, group_size 16). The checkpoint at dep4 and +at tp4 is checked to resolve and route as it did before the 1M config was added. + +The same target serves the NVFP4 checkpoint at tep8, eight ranks over two GB300 nodes of four +(attention tensor parallel over all eight, attention DP off, the routed experts expert-parallel +eight ways, 16 requests over the whole engine), and tep8 is checked at the 1M config and at +the delivered 32768 config. At 1M the gates spell the latent KV cache fp8 and the CI class +leaves it at its default, auto (the checkpoint's fp8_kv_cache); the delivered gates leave it at +its default; every spelling builds the fp8_ds_mla pool of MistralLarge4CacheManager. At each +config the checkpoint goes through the same path with the two spellings the gates use for the +tep8 core unit: + +* the accuracy gate's LLM-API options (spec-1m-tep8/options/core.yaml at 1M, + spec-tep8/options/core.yaml delivered), handed to ``TorchLlmArgs``; +* the boot gate's command, parsed and turned into ``LLM`` keyword arguments by the tree's own + ``examples/llm-api/quickstart_advanced.py`` (``parse_arguments`` and ``setup_llm``), as the + gate harness builds its engine, plus the ``modeling_v2`` argument the harness adds. + +What the engine does with the 1M configuration at eight ranks -- the prompts admitted, the +context path, the query split across the eight TP ranks and each rank's indexer chunk plan, the +scheduler's chunks, the buffers sized by max_seq_len, warmup, the KV-capacity estimation, CUDA +graph capture and the load the measured tep8 pools hold -- is pinned as at tep4. + +Neither gate states ``gpus_per_node``: ``TorchLlmArgs`` takes it from +``torch.cuda.device_count()`` on rank 0, which on a GB300 node of this cluster is 4. The count is +set to 4 where the tep8 arguments are built rather than read from the test host, which may see +no device or a whole node. Where the engine reads the node layout on its way to a modeling_v2 +target -- the device each rank selects, the communicators, the causal-LM shell's lm_head, the +MNNVL communicator split, the AllReduce strategy's node test, the MoE autotuner, the KV host +tier -- is pinned on the engine's own host code, eight ranks simulated in one process where they +exchange anything. tep8 is checked against the NVFP4 checkpoint at tep4 (on the options of +either config) and at dep4. + +The checkpoint is looked up under ``LLM_MODELS_ROOT`` (as ``Mistral-Large-4-1T-A52B-NVFP4`` or +``mistral-large-4-nvfp4``) by its params.json, and its tests are skipped where it is absent. +""" + +from __future__ import annotations + +import copy +import dataclasses +import importlib.util +import json +import math +import sys +import threading +from collections import Counter +from dataclasses import dataclass +from pathlib import Path +from types import MethodType, SimpleNamespace +from unittest.mock import Mock, patch + +import pytest +import torch +import yaml +from test_common.llm_data import llm_models_root + +import tensorrt_llm.bindings +from tensorrt_llm._ipc_utils import IpcMemory +from tensorrt_llm._torch.attention.backends import trtllm as trtllm_backend +from tensorrt_llm._torch.attention.backends.interface import ( + AttentionMetadata, + AttentionRuntimeFeatures, +) +from tensorrt_llm._torch.attention.backends.sparse.dsa import DSACacheManagerV2 +from tensorrt_llm._torch.attention.backends.sparse.dsa import indexer as dsa_indexer +from tensorrt_llm._torch.attention.backends.sparse.dsa import metadata as dsa_metadata +from tensorrt_llm._torch.attention.backends.sparse.dsa.indexer import ( + Indexer, + plan_mla_chunked_prefill_chunks, + select_indexer_chunk_size, + split_prefill_chunks, +) +from tensorrt_llm._torch.attention.backends.sparse.dsa.metadata import DSAtrtllmAttentionMetadata +from tensorrt_llm._torch.attention.backends.sparse.mistral_large4 import MistralLarge4CacheManager +from tensorrt_llm._torch.autotuner import AutoTuner, DistributedTuningStrategy +from tensorrt_llm._torch.custom_ops.trtllm_gen_custom_ops import FP4BlockScaleMoERunner +from tensorrt_llm._torch.distributed import ops as distributed_ops +from tensorrt_llm._torch.distributed.communicator import Distributed, MPIDist +from tensorrt_llm._torch.distributed.mnnvl_memory import MnnvlMemory +from tensorrt_llm._torch.modeling_v2 import _router_index +from tensorrt_llm._torch.modeling_v2._router_index import NoTarget +from tensorrt_llm._torch.modeling_v2.models.mistral_large4 import routing +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_103__tep import ( # noqa: E501 + modeling as tep4_modeling, +) +from tensorrt_llm._torch.models.checkpoints.mistral.config_loader import MistralConfigLoader +from tensorrt_llm._torch.models.modeling_utils import DecoderModel, DecoderModelForCausalLM +from tensorrt_llm._torch.modules.linear import TensorParallelMode +from tensorrt_llm._torch.moe.fused_moe.moe_load_balancer import maybe_create_moe_load_balancer +from tensorrt_llm._torch.pyexecutor import model_loader, py_executor_creator +from tensorrt_llm._torch.pyexecutor._util import ( + CacheCost, + KvCacheCreator, + get_attention_workspace_bytes_per_token, + get_attention_workspace_fixed_bytes, + get_attention_workspace_is_chunked_prefill_bounded, + get_kv_cache_manager_cls, + get_mla_context_workspace_kv_len_cap, +) +from tensorrt_llm._torch.pyexecutor.engine.cuda_graph import ( + filter_cuda_graph_batch_sizes, + filter_prefill_capture_num_tokens, +) +from tensorrt_llm._torch.pyexecutor.engine.metadata import build_attention_metadata +from tensorrt_llm._torch.pyexecutor.engine.runners.decoder.dummy_requests import DummyRequestFactory +from tensorrt_llm._torch.pyexecutor.engine.runners.decoder.warmup import ( + DecoderWarmup, + WarmupAgreement, +) +from tensorrt_llm._torch.pyexecutor.kv_cache.kv_cache_manager_v2 import ( + BlockReusePolicy, + KVCacheManagerV2, + _compute_auto_host_tier_quota, + _get_generation_kv_capacity, +) +from tensorrt_llm._torch.pyexecutor.llm_request import LlmRequestState +from tensorrt_llm._torch.pyexecutor.model_engine import ModelEngine +from tensorrt_llm._torch.pyexecutor.model_loader import ModelLoader as RankModelLoader +from tensorrt_llm._torch.pyexecutor.resource_manager import ResourceManagerType +from tensorrt_llm._torch.pyexecutor.scheduler.scheduler_v2 import KVCacheV2Scheduler +from tensorrt_llm.bindings.internal.batch_manager import CacheType +from tensorrt_llm.executor import base_worker +from tensorrt_llm.executor.base_worker import BaseWorker +from tensorrt_llm.executor.request import GenerationRequest +from tensorrt_llm.executor.utils import RequestError +from tensorrt_llm.llmapi.llm import BaseLLM +from tensorrt_llm.llmapi.llm_args import ( + CapacitySchedulerPolicy, + ContextChunkingPolicy, + DeepSeekSparseAttentionConfig, + TorchLlmArgs, + _ParallelConfig, +) +from tensorrt_llm.llmapi.llm_utils import ModelLoader as FrontEndModelLoader +from tensorrt_llm.mapping import Mapping, MpiTopology +from tensorrt_llm.models.modeling_utils import QuantAlgo +from tensorrt_llm.sampling_params import SamplingParams + +pytestmark = pytest.mark.cpu_only + +_LAYERS = 72 +_SM103 = (10, 3) +_RANKS = 4 + +# The serving config (spec-1m-kvds-tep4/options/core.yaml with task.yaml's runtime flags). +_MAX_SEQ_LEN = 1048576 +_MAX_NUM_TOKENS = 32768 +_MAX_BATCH_SIZE = 16 +_TOKENS_PER_BLOCK = 32 +_INDEX_TOPK = 2048 +# The 1M gate's long prompt: max_seq_len less the 16 tokens it generates, less one. +_ENVELOPE_PROMPT = _MAX_SEQ_LEN - 16 - 1 +# The pool the delivered tep4 boot (32768 unchunked, the 576-byte latent pool on every layer) +# gave each rank, in tokens (NOTES.md), and that pool's bytes a token. +_DELIVERED_POOL_TOKENS = 2_373_024 +_DELIVERED_BYTES_PER_TOKEN = 576 * 72 + 132 * 44 +# One token of the DSA pool per rank: the fp8_ds_mla latent (512 e4m3, four fp32 tile scales, +# 64 bf16 rope: 656 bytes) on the 63 KV-owning layers, the fp8 indexer key (128) and its fp32 +# scale on the 44 full-indexer layers. +_KV_BYTES_PER_TOKEN = 656 * 63 + 132 * 44 +# The vision tower's bf16 weights, which every rank of the NVFP4 targets loads whole. +_VISION_TOWER_BYTES = 3_408_437_248 +# The pool a boot of the 1M accuracy gate (the Tep4Ctx1m class, KV fraction 0.85) gave each +# rank on the native checkpoint, the vision tower loaded, in tokens: 66,734 pages. +_TEP4_1M_POOL_TOKENS = 66_734 * 32 + +# The tep4 topology, and the other members of this checkpoint at their own base options +# (spec-dep4/options/core.yaml, spec-tp4/options/core.yaml), with the latent KV cache at its +# default: the checkpoint's fp8_kv_cache, the fp8_ds_mla pool. +_TEP4 = dict(tensor_parallel_size=4, moe_expert_parallel_size=4, enable_attention_dp=False) +_DEP4 = dict(tensor_parallel_size=4, moe_expert_parallel_size=4, enable_attention_dp=True) +_TP4 = dict( + tensor_parallel_size=4, + moe_tensor_parallel_size=4, + moe_expert_parallel_size=1, + enable_attention_dp=False, +) + + +def _options( + max_seq_len: int, + max_num_tokens: int, + chunked: bool, + topology: dict = _TEP4, + kv_dtype: str | None = None, +) -> dict: + """The LLM-API options of a Mistral Large 4 accuracy gate, the checkpoint in its Mistral-native + format, the latent KV cache at its default dtype (the checkpoint's fp8_kv_cache) unless + ``kv_dtype`` names one.""" + kv = dict(enable_block_reuse=False, free_gpu_memory_fraction=0.85) + if kv_dtype is not None: + kv.update(dtype=kv_dtype) + return dict( + checkpoint_format="mistral", + **topology, + max_batch_size=_MAX_BATCH_SIZE, + max_num_tokens=max_num_tokens, + max_seq_len=max_seq_len, + enable_chunked_prefill=chunked, + cuda_graph_config=dict(enable_padding=True, max_batch_size=16), + kv_cache_config=kv, + ) + + +@dataclass(frozen=True) +class _Case: + """One config of the checkpoint, the bounds its accuracy gate runs with, and what it must + resolve to.""" + + names: tuple[str, ...] + max_seq_len: int + max_num_tokens: int + chunked: bool + quant_algo: QuantAlgo + group_size: int + # exclude_modules as llm.args reports it, and as the ModelConfig the ranks build holds it + llm_exclude_modules: list + engine_exclude_modules: list + # (quant_algo, group_size) -> number of per-layer configs + layer_algos: dict + rope_factor: float + max_position_embeddings: int + anchor: float + target: str + # kv_cache_config.dtype as the gate passes it; None leaves the default (auto) + kv_dtype: str | None = None + + @property + def options(self) -> dict: + return _options(self.max_seq_len, self.max_num_tokens, self.chunked, kv_dtype=self.kv_dtype) + + +_NVFP4_NAMES = ("Mistral-Large-4-1T-A52B-NVFP4", "mistral-large-4-nvfp4") + +# The serving config. +_NVFP4 = _Case( + names=_NVFP4_NAMES, + max_seq_len=_MAX_SEQ_LEN, + max_num_tokens=_MAX_NUM_TOKENS, + chunked=True, + quant_algo=QuantAlgo.MIXED_PRECISION, + group_size=16, + llm_exclude_modules=[], + engine_exclude_modules=[], + layer_algos={(QuantAlgo.NVFP4, 16): 71, (QuantAlgo.FP8_BLOCK_SCALES, None): 404}, + rope_factor=2048.0, + max_position_embeddings=1048576, + anchor=91.926, + target="ModelingV2MistralLarge4Nvfp4Sm103Tep", +) +# The serving config as the gates run it: spec-1m-kvds-tep4/options/core.yaml and the +# runtime's --kv_cache_dtype=fp8. +_NVFP4_GATE = dataclasses.replace(_NVFP4, kv_dtype="fp8") +# The delivered config: 32768 tokens, one step per prompt. +_NVFP4_32K = dataclasses.replace(_NVFP4, max_seq_len=32768, chunked=False) + +# The GSM8K reference rows, and the key the harness reads them by. +_GSM8K_REFERENCES = ( + Path(__file__).resolve().parents[6] / "integration/defs/accuracy/references/gsm8k.yaml" +) +_MODEL_NAME = "mistralai/Mistral-Large-4-1T-A52B" + + +def _checkpoint_dir(names: tuple[str, ...]) -> Path: + """The first of ``names`` under LLM_MODELS_ROOT holding a Mistral-native params.json.""" + root = llm_models_root() + for name in names: + if root is not None and (root / name / "params.json").is_file(): + return root / name + pytest.skip(f"none of {names} is under LLM_MODELS_ROOT ({root})") + + +class _Routed: + """What the recording `route` returns: no target is built here.""" + + +def _route(pretrained_config, mapping): + """`route` for this config on ``mapping``, on sm_103 without speculative decoding.""" + model_config = SimpleNamespace( + pretrained_config=pretrained_config, mapping=mapping, spec_config=None + ) + return routing.route(model_config, None, _SM103) + + +def _resolve(checkpoint_dir: Path, options: dict) -> SimpleNamespace: + """The configuration a served engine reaches for this checkpoint under ``options``.""" + llm_args = TorchLlmArgs(model=str(checkpoint_dir), modeling_v2="require", **options) + return _resolve_args(checkpoint_dir, llm_args) + + +def _resolve_args(checkpoint_dir: Path, llm_args: TorchLlmArgs) -> SimpleNamespace: + """The configuration a served engine reaches for this checkpoint, front end then rank.""" + model = str(checkpoint_dir) + prequantized = FrontEndModelLoader(llm_args)._update_from_hf_quant_config() + + routed = [] + + def route(model_config, llm_args, compute_capability): + pretrained_config = model_config.pretrained_config + routed.append( + SimpleNamespace( + pretrained_config=pretrained_config, + quantization_config=copy.deepcopy(pretrained_config.quantization_config), + mapping=model_config.mapping, + compute_capability=compute_capability, + spec_config=model_config.spec_config, + ) + ) + return _Routed + + with pytest.MonkeyPatch.context() as mp: + # The rank's config pass pins the CUDA memory fraction first; there is no device here. + mp.setattr(torch.cuda, "set_per_process_memory_fraction", lambda *args, **kwargs: None) + mp.setattr(_router_index, "_device_compute_capability", lambda: _SM103) + mp.setattr(_router_index, "registered_routes", lambda: {"MistralLarge4ForCausalLM": route}) + # One process stands in for every rank, so the rank exchange is the local resolution. + mp.setattr( + model_loader, "modeling_v2_resolve_on_all_ranks", _router_index.modeling_v2_resolve + ) + llm_args, checkpoint_loader = py_executor_creator._load_config_and_create_checkpoint_loader( + llm_args, model + ) + _, max_num_tokens, max_seq_len, max_batch_size = llm_args.get_runtime_sizes() + mapping = llm_args.parallel_config.to_mapping() + model_config = RankModelLoader( + llm_args=llm_args, + mapping=mapping, + spec_config=llm_args.speculative_config, + sparse_attention_config=llm_args.sparse_attention_config, + max_num_tokens=max_num_tokens, + max_seq_len=max_seq_len, + )._load_and_validate_config(model, checkpoint_loader) + return SimpleNamespace( + path=checkpoint_dir, + llm_args=llm_args, + prequantized=prequantized, + model_config=model_config, + mapping=mapping, + routed=routed, + max_num_tokens=max_num_tokens, + max_seq_len=max_seq_len, + max_batch_size=max_batch_size, + ) + + +def _resolve_case(case: _Case) -> SimpleNamespace: + path = _checkpoint_dir(case.names) + return SimpleNamespace( + case=case, + # params.json as the native loader adapts it (models/checkpoints/mistral/config_loader.py) + native=MistralConfigLoader().load_pretrained_config(str(path)), + **vars(_resolve(path, case.options)), + ) + + +@pytest.fixture(scope="module") +def envelope(): + return _resolve_case(_NVFP4) + + +@pytest.fixture(scope="module") +def gate(): + return _resolve_case(_NVFP4_GATE) + + +@pytest.fixture(scope="module") +def delivered(): + return _resolve_case(_NVFP4_32K) + + +@pytest.fixture(scope="module", params=["envelope", "gate", "delivered"]) +def resolved(request): + return request.getfixturevalue(request.param) + + +def _anchor(llm_args) -> float: + """The reference row `AccuracyTask.get_hypothesis_testing_params` selects for this engine.""" + quant = llm_args.quant_config + acc_specs = dict( + dtype=llm_args.dtype, + quant_algo=quant.quant_algo, + kv_cache_quant_algo=quant.kv_cache_quant_algo, + spec_dec_algo=None, + extra_acc_spec=None, + ) + for entry in yaml.safe_load(_GSM8K_REFERENCES.read_text())[_MODEL_NAME]: + if all( + entry.get(key, "auto" if key == "dtype" else None) == value + for key, value in acc_specs.items() + ): + return entry["accuracy"] + raise AssertionError(f"no {_MODEL_NAME} GSM8K reference row for {acc_specs}") + + +def _rank_mapping(mapping, rank: int): + rank_mapping = copy.deepcopy(mapping) + rank_mapping.rank = rank + return rank_mapping + + +def _kv_cost(resolved, rank: int = 0) -> CacheCost: + """The engine's affine KV cost on ``rank``, as the KV-cache creator sizes the pool with it.""" + model_config, llm_args = resolved.model_config, resolved.llm_args + kv_cache_config = llm_args.kv_cache_config + manager = get_kv_cache_manager_cls(model_config, kv_cache_config) + return CacheCost.from_raw( + manager.get_cache_size_per_token( + model_config, + _rank_mapping(resolved.mapping, rank), + tokens_per_block=kv_cache_config.tokens_per_block, + max_seq_len=resolved.max_seq_len, + max_batch_size=resolved.max_batch_size, + max_num_tokens=resolved.max_num_tokens, + kv_cache_config=kv_cache_config, + spec_config=llm_args.speculative_config, + is_draft=False, + ) + ) + + +def _runtime_features(resolved) -> AttentionRuntimeFeatures: + """The attention runtime features `_create_py_executor` (py_executor_creator.py) builds: + chunked prefill from the LLM API's enable_chunked_prefill, cache reuse from its + kv_cache_config.enable_block_reuse, the chunk size max_num_tokens. They are what the target + reads chunked prefill and block reuse off on its first step.""" + llm_args = resolved.llm_args + return AttentionRuntimeFeatures( + chunked_prefill=llm_args.enable_chunked_prefill, + cache_reuse=llm_args.kv_cache_config.enable_block_reuse, + has_speculative_draft_tokens=False, + chunk_size=resolved.max_num_tokens, + ) + + +def _target_sizing(resolved) -> SimpleNamespace: + """What the target sizes its indexer logits budget and its held context workspace from on + its first step (`_derive_first_step`): the attention metadata `build_attention_metadata` + (pyexecutor/engine/metadata.py) makes for this engine -- max_num_requests (max_batch_size), + max_num_tokens and the runtime features (`_runtime_features`, chunked prefill and block + reuse) -- whose max_seq_len is the KV cache manager's (`TrtllmAttentionMetadata.max_seq_len`), + the one thing the target's sizing reads through that manager.""" + + def metadata(**kwargs): + return SimpleNamespace(**kwargs, max_seq_len=kwargs["kv_cache_manager"].max_seq_len) + + return build_attention_metadata( + resolved.model_config, + max_batch_size=resolved.max_batch_size, + max_num_tokens=resolved.max_num_tokens, + max_beam_width=1, + attention_backend=SimpleNamespace(Metadata=metadata), + attention_runtime_features=_runtime_features(resolved), + mapping=resolved.mapping, + cache_indirection=None, + kv_cache_manager=SimpleNamespace(max_seq_len=resolved.max_seq_len), + ) + + +def _metadata_params(resolved): + model_config = resolved.model_config + return model_config.sparse_attention_config.to_sparse_metadata_params( + pretrained_config=model_config.pretrained_config + ) + + +def _max_blocks_per_seq(max_seq_len: int) -> int: + """`KVCacheManagerV2.__init__`'s block-table width: max_seq_len plus the generation headroom + `_get_generation_kv_capacity` gives a target without speculation, in pages, padded to a + multiple of 4 for the copy_block_offsets kernel.""" + _, headroom = _get_generation_kv_capacity(None, is_draft=False) + blocks = (max_seq_len + headroom + _TOKENS_PER_BLOCK - 1) // _TOKENS_PER_BLOCK + return (blocks + 3) // 4 * 4 + + +# ------------------------------------------------------- (1) LLM-API resolution, the anchor + + +def test_llm_args_quantization_is_the_anchor_key(resolved): + """`llm.args.quant_config`, as `ModelLoader._update_from_hf_quant_config` (llmapi/ + llm_utils.py) settles it, is what the accuracy harness keys the reference row on + (`AccuracyTask.get_hypothesis_testing_params`): the same row at every config.""" + case, quant = resolved.case, resolved.llm_args.quant_config + assert resolved.prequantized is True + assert quant.quant_algo == case.quant_algo + assert quant.kv_cache_quant_algo == QuantAlgo.FP8 + assert quant.group_size == case.group_size + assert quant.exclude_modules == case.llm_exclude_modules + assert _anchor(resolved.llm_args) == case.anchor + + +def test_engine_quantization(resolved): + """The ranks' ModelConfig (`ModelConfig.from_pretrained`, _torch/model_config.py, through + `ModelLoader._load_and_validate_config`): the same global algorithm, the fp8 latent cache, + and one per-layer config for every Mistral-named quantized layer, the ModelOpt spellings + canonicalised (FP8_PB_WO is block-128 FP8) and group_size 16 accepted for the NVFP4 + experts.""" + case, model_config = resolved.case, resolved.model_config + quant = model_config.quant_config + assert quant.quant_algo == case.quant_algo + assert quant.kv_cache_quant_algo == QuantAlgo.FP8 + assert quant.group_size == case.group_size + assert quant.exclude_modules == case.engine_exclude_modules + + layers = model_config.quant_config_dict or {} + assert Counter((q.quant_algo, q.group_size) for q in layers.values()) == case.layer_algos + # The latent cache's algorithm is the engine-wide one above (the checkpoint's fp8_kv_cache + # under the default kv_cache_config.dtype); a per-layer config sets none of its own. + assert {q.kv_cache_quant_algo for q in layers.values()} <= {QuantAlgo.FP8, None} + quantized_layers = resolved.native.quantization_config.get("quantized_layers", {}) + assert set(layers) == set(quantized_layers) + assert {name for name, q in layers.items() if q.quant_algo == QuantAlgo.NVFP4} == { + f"layers.{layer}.experts" for layer in range(1, _LAYERS) + } + + +def test_envelope_resolves_as_configured(envelope, delivered): + """`TorchLlmArgs` (llmapi/llm_args.py) and `get_runtime_sizes` keep the 1M config as given: tp + 4, moe_ep 4, attention DP off; max_seq_len 1048576, max_num_tokens 32768 (the chunk), + max_batch_size 16; chunked prefill accepted with the DSA sparse attention the checkpoint + configures, the latent KV cache at its default dtype and block reuse off, on the V2 KV manager + the sparse registry picks for a MistralLarge4Config, `MistralLarge4CacheManager` + (attention/backends/sparse/registry.py, the fp8_ds_mla pool). The DSA config is the + checkpoint's, with its q_split_threshold default. `_create_py_executor` (py_executor_creator.py) + keeps chunked prefill for MLA on sm_103 (`_MLA_CHUNKED_PREFILL_SUPPORTED_SM_VERSIONS`). Beside + it the delivered config: 32768 tokens, one step per prompt.""" + for resolved, max_seq_len, chunked in ( + (envelope, _MAX_SEQ_LEN, True), + (delivered, 32768, False), + ): + llm_args, mapping = resolved.llm_args, resolved.mapping + assert (llm_args.tensor_parallel_size, llm_args.moe_expert_parallel_size) == (4, 4) + assert llm_args.enable_attention_dp is False + assert (mapping.world_size, mapping.tp_size, mapping.pp_size, mapping.cp_size) == ( + 4, + 4, + 1, + 1, + ) + assert (mapping.moe_ep_size, mapping.moe_tp_size, mapping.enable_attention_dp) == ( + 4, + 1, + False, + ) + assert (resolved.max_num_tokens, resolved.max_seq_len, resolved.max_batch_size) == ( + _MAX_NUM_TOKENS, + max_seq_len, + _MAX_BATCH_SIZE, + ) + assert llm_args.get_runtime_sizes() == (1, _MAX_NUM_TOKENS, max_seq_len, _MAX_BATCH_SIZE) + assert (llm_args.max_seq_len, llm_args.max_num_tokens) == (max_seq_len, _MAX_NUM_TOKENS) + assert llm_args.enable_chunked_prefill is chunked + assert llm_args.disable_overlap_scheduler is False + + kv_cache_config = llm_args.kv_cache_config + assert kv_cache_config.dtype == "auto" and kv_cache_config.enable_block_reuse is False + assert kv_cache_config.tokens_per_block == _TOKENS_PER_BLOCK + assert kv_cache_config.use_kv_cache_manager_v2 is True + assert kv_cache_config.max_attention_window is None + manager = get_kv_cache_manager_cls(resolved.model_config, kv_cache_config) + assert manager is MistralLarge4CacheManager and issubclass(manager, KVCacheManagerV2) + + graphs = llm_args.cuda_graph_config + assert (graphs.batch_sizes, graphs.enable_padding) == ([1, 2, 4, 8, 16], True) + assert llm_args.prefill_capture_num_tokens is None + assert llm_args.torch_compile_config is None + + sparse = resolved.model_config.sparse_attention_config + assert type(sparse) is DeepSeekSparseAttentionConfig and sparse.algorithm == "dsa" + assert sparse.q_split_threshold == 8192 + assert sparse.skip_indexer_for_short_seqs is True + assert sparse.use_cute_dsl_topk is False and sparse.enable_heuristic_topk is False + assert resolved.model_config.attn_backend == "TRTLLM" + + assert 103 in py_executor_creator._MLA_CHUNKED_PREFILL_SUPPORTED_SM_VERSIONS + + +def _manager_arguments(resolved, dtype) -> dict: + """What `MistralLarge4CacheManager.__init__` (attention/backends/sparse/mistral_large4/ + cache_manager.py) hands the DSA V2 manager it builds on, given what `_create_kv_cache_manager` + (pyexecutor/_util.py) passes it for this engine: the LLM API's kv_cache_config, one 576-wide + MLA head on each of the 72 layers, the latent ``dtype``. The base is not built (it allocates + the pool on the device): its constructor records its arguments and stops.""" + + class _Recorded(Exception): + pass + + seen = {} + + def record(self, kv_cache_config, kv_cache_type, **kwargs): + seen.update(kwargs, kv_cache_config=kv_cache_config, kv_cache_type=kv_cache_type) + raise _Recorded + + model_config = resolved.model_config + with pytest.MonkeyPatch.context() as mp: + mp.setattr(DSACacheManagerV2, "__init__", record) + with pytest.raises(_Recorded): + MistralLarge4CacheManager( + resolved.llm_args.kv_cache_config, + CacheType.SELFKONLY, + num_layers=_LAYERS, + num_kv_heads=1, + head_dim=576, + tokens_per_block=_TOKENS_PER_BLOCK, + max_seq_len=resolved.max_seq_len, + max_batch_size=resolved.max_batch_size, + mapping=resolved.mapping, + dtype=dtype, + spec_config=None, + layer_mask=None, + max_num_tokens=resolved.max_num_tokens, + max_beam_width=1, + sparse_attention_config=model_config.sparse_attention_config, + pretrained_config=model_config.pretrained_config, + ) + return seen + + +def test_kv_cache_dtype_auto_and_fp8_build_the_fp8_ds_mla_pool(envelope, gate): + """kv_cache_config.dtype auto (the CI class: the checkpoint's fp8_kv_cache) and fp8 (the + gates: spec-1m-kvds-tep4/options/core.yaml, the runtime's --kv_cache_dtype=fp8) build the + same cache. Both resolve the latent to FP8 (`kv_cache_quant_algo`, so `_create_kv_cache_ + manager` hands the manager `DataType.FP8`, `quant_mode.has_fp8_kv_cache()`); the sparse + registry picks `MistralLarge4CacheManager`, whose `latent_layout` is fp8_ds_mla, at 47136 + bytes a token; and the manager builds its base over all 72 layers, one 656-byte head on each + of the 63 KV owners and none on the nine sharers, as FP8, with the dtype it was asked for. A + latent dtype the layout cannot hold, bf16, is refused at construction.""" + for resolved, requested in ((envelope, "auto"), (gate, "fp8")): + model_config, llm_args = resolved.model_config, resolved.llm_args + assert llm_args.kv_cache_config.dtype == requested + assert llm_args.quant_config.kv_cache_quant_algo == QuantAlgo.FP8 + assert model_config.quant_config.quant_mode.has_fp8_kv_cache() + manager = get_kv_cache_manager_cls(model_config, llm_args.kv_cache_config) + assert manager is MistralLarge4CacheManager and manager.latent_layout == "fp8_ds_mla" + assert (_kv_cost(resolved).slope, _kv_cost(resolved).intercept) == (47136, 0) + + sources = model_config.pretrained_config.kv_source_layers + base = _manager_arguments(resolved, tensorrt_llm.bindings.DataType.FP8) + assert (base["num_layers"], base["head_dim"]) == (_LAYERS, 656) + assert base["num_kv_heads"] == [1 if s == i else None for i, s in enumerate(sources)] + assert base["num_kv_heads"].count(1) == 63 + assert base["dtype"] == tensorrt_llm.bindings.DataType.FP8 + assert base["layer_mask"] is None + assert base["kv_cache_config"].dtype == requested + assert (base["tokens_per_block"], base["max_seq_len"]) == (32, _MAX_SEQ_LEN) + with pytest.raises(ValueError, match="fp8_ds_mla"): + _manager_arguments(resolved, tensorrt_llm.bindings.DataType.BF16) + + +def test_model_engine_keeps_max_seq_len(envelope, delivered): + """`ModelEngine._init_model_capacity` (pyexecutor/model_engine.py) lowers a + max_seq_len above the model's inferred limit and a max_num_tokens above max_batch_size x + max_seq_len. The target's shell infers the limit with `DecoderModelForCausalLM + .infer_max_seq_len`: YaRN does not scale it, so it is max_position_embeddings, 1048576, and + the 1M config's max_seq_len is kept whole, as the delivered one is.""" + for resolved in (envelope, delivered): + cfg = resolved.model_config.pretrained_config + inferred = DecoderModelForCausalLM.infer_max_seq_len(SimpleNamespace(config=cfg)) + assert inferred == cfg.max_position_embeddings == _MAX_SEQ_LEN + + engine = SimpleNamespace( + model=SimpleNamespace(infer_max_seq_len=lambda inferred=inferred: inferred), + max_seq_len=resolved.max_seq_len, + max_num_tokens=resolved.max_num_tokens, + batch_size=resolved.max_batch_size, + ) + for name in ("_init_max_seq_len", "_init_max_num_tokens", "_infer_max_seq_len_from_config"): + setattr(engine, name, MethodType(getattr(ModelEngine, name), engine)) + with pytest.MonkeyPatch.context() as mp: + mp.delenv("TLLM_ALLOW_LONG_MAX_MODEL_LEN", raising=False) + ModelEngine._init_model_capacity(engine) + assert (engine.max_seq_len, engine.max_num_tokens) == ( + resolved.max_seq_len, + resolved.max_num_tokens, + ) + assert envelope.model_config.max_seq_len == _MAX_SEQ_LEN + assert envelope.model_config.max_num_tokens == _MAX_NUM_TOKENS + + +def _check_arguments(resolved, prompt_len: int) -> None: + llm = SimpleNamespace(args=resolved.llm_args) + llm._check_one_model_speculative_sampling = MethodType( + BaseLLM._check_one_model_speculative_sampling, llm + ) + BaseLLM._check_arguments(llm, prompt_len, SamplingParams(max_tokens=16), is_gen_only=False) + + +def _enqueue(resolved, prompt_len: int, max_tokens: int): + """The executor request `BaseWorker._enqueue_request` hands the engine for a prompt of + ``prompt_len`` tokens.""" + enqueued = [] + worker = SimpleNamespace( + _lora_manager=None, + max_seq_len=resolved.max_seq_len, + mapping=resolved.mapping, + engine=SimpleNamespace(enqueue_request=lambda request, **_: enqueued.append(request) or 1), + ) + request = GenerationRequest( + [1] * prompt_len, sampling_params=SamplingParams(max_tokens=max_tokens, end_id=2) + ) + request.set_id(1) + BaseWorker._enqueue_request(worker, request) + return enqueued[0] + + +def test_llm_api_admits_the_envelope_prompt(envelope, delivered): + """`BaseLLM._check_arguments` (llmapi/llm.py) refuses a prompt longer than max_num_tokens only + without chunked prefill: the delivered engine refuses 32769 tokens, the 1M engine admits the + 1M gate's 1048559. The bound on max_seq_len is the worker's: `BaseWorker + ._enqueue_request` (executor/base_worker.py) gives a request at most max_seq_len less its + prompt to generate, and refuses a prompt that leaves no room for one token.""" + _check_arguments(envelope, _ENVELOPE_PROMPT) + _check_arguments(delivered, _MAX_NUM_TOKENS) + with pytest.raises(RequestError, match=r"should not exceed max_num_tokens \(32768\)"): + _check_arguments(delivered, _MAX_NUM_TOKENS + 1) + + assert _enqueue(envelope, _ENVELOPE_PROMPT, 16).max_tokens == 16 + assert _enqueue(envelope, _MAX_SEQ_LEN - 1, 16).max_tokens == 1 + for prompt_len in (_MAX_SEQ_LEN, _MAX_SEQ_LEN + 1): + with pytest.raises(RequestError, match="maximum context length is 1048576 tokens"): + _enqueue(envelope, prompt_len, 16) + + +# ------------------------------------------------------- (2) the context path of a long prompt + + +def test_chunked_prefill_hands_the_target_the_cached_kv_context_path(envelope, delivered): + """`build_attention_metadata` (pyexecutor/engine/metadata.py) turns on + `enable_context_mla_with_cached_kv` for MLA under cache reuse or chunked prefill: on at the + 1M config, off at the delivered engine. With it and `runtime_features.chunked_prefill`, + `Indexer.prepare_for_chunked_prefill` (attention/backends/sparse/dsa/indexer.py) takes its + MLA-chunked-prefill branch. The metadata is sized by the engine bounds, not max_seq_len.""" + for resolved, cached_kv in ((envelope, True), (delivered, False)): + made = build_attention_metadata( + resolved.model_config, + max_batch_size=resolved.max_batch_size, + max_num_tokens=resolved.max_num_tokens, + max_beam_width=1, + attention_backend=SimpleNamespace(Metadata=lambda **kwargs: SimpleNamespace(**kwargs)), + attention_runtime_features=_runtime_features(resolved), + mapping=resolved.mapping, + cache_indirection=None, + ) + assert made.enable_context_mla_with_cached_kv is cached_kv + assert made.runtime_features.chunked_prefill is cached_kv + assert made.runtime_features.cache_reuse is False + assert made.runtime_features.chunk_size == _MAX_NUM_TOKENS + assert (made.max_num_requests, made.max_num_tokens, made.max_num_sequences) == ( + 16, + 32768, + 16, + ) + assert made.sparse_metadata_params == _metadata_params(resolved) + + +def test_target_sizes_from_the_engine(monkeypatch, envelope, gate, delivered): + """What the target takes from the engine on its first step, read off the metadata this engine + builds (`_target_sizing`), whose runtime features are the executor's: chunked prefill as + enable_chunked_prefill, block reuse as kv_cache_config.enable_block_reuse, off in every + config here. The 1M config (max_seq_len 1048576, chunked prefill, 16 requests, in either KV + dtype spelling): the indexer's logits budget at its 2**28 ceiling (`_prefill_logits_budget`, + 256 query rows at 1048576 keys), and a step may attend min(16, 32768) x 1048576 rows of + context KV (`_most_ctx_kv`, chunked prefill bounds no gather), so the held context workspace + takes its 2**19 ceiling (`_ctx_workspace_rows`). The delivered config (32768 unchunked, block + reuse off): 2**24 logits (512 rows at 32768 keys), at most max_num_tokens of context KV a + step, all of it new, and 65536 rows held, twice that.""" + monkeypatch.delenv("TLLM_INDEXER_MQA_LOGITS_ELEM_BUDGET", raising=False) + for resolved, most, rows, budget in ( + (envelope, 16 * _MAX_SEQ_LEN, 2**19, 2**28), + (gate, 16 * _MAX_SEQ_LEN, 2**19, 2**28), + (delivered, _MAX_NUM_TOKENS, 2**16, 2**24), + ): + md = _target_sizing(resolved) + assert (md.max_num_requests, md.max_num_tokens, md.max_seq_len) == ( + 16, + 32768, + resolved.max_seq_len, + ) + features, llm_args = md.runtime_features, resolved.llm_args + assert features.chunked_prefill is llm_args.enable_chunked_prefill is resolved.case.chunked + assert features.cache_reuse is llm_args.kv_cache_config.enable_block_reuse is False + assert tep4_modeling._most_ctx_kv(md) == most + assert tep4_modeling._ctx_workspace_rows(md) == rows + assert tep4_modeling._prefill_logits_budget(md.max_num_tokens, md.max_seq_len) == budget + assert tep4_modeling._CTX_WORKSPACE_ROWS == 2**19 + assert tep4_modeling._PREFILL_LOGITS_CEILING == 2**28 + assert (2**28 // _MAX_SEQ_LEN, 2**24 // 32768) == (256, 512) + + +def test_target_sizes_block_reuse_from_the_executors_runtime_features( + monkeypatch, envelope, delivered +): + """Block reuse reaches the target through the attention runtime features the executor builds + from the LLM API's kv_cache_config (`_runtime_features`: cache_reuse is + kv_cache_config.enable_block_reuse). The delivered and the 1M options with block reuse on, + resolved as the engine resolves them: the runtime features carry it, the metadata takes the + cached-KV MLA path at both, and a step may attend min(16, 32768) x max_seq_len rows of + context KV, cached prefixes included (`_most_ctx_kv`), so the held context workspace takes + its 2**19 ceiling at both: the delivered config's 65536 rows grow to it, the 1M config's stay. + The logits budget, from max_num_tokens and max_seq_len alone, is 2**24 and 2**28 as without + block reuse.""" + monkeypatch.delenv("TLLM_INDEXER_MQA_LOGITS_ELEM_BUDGET", raising=False) + for base, most, budget in ( + (delivered, 16 * _MAX_NUM_TOKENS, 2**24), + (envelope, 16 * _MAX_SEQ_LEN, 2**28), + ): + options = base.case.options + options["kv_cache_config"] = dict(options["kv_cache_config"], enable_block_reuse=True) + resolved = _resolve(base.path, options) + assert resolved.llm_args.kv_cache_config.enable_block_reuse is True + assert (resolved.max_num_tokens, resolved.max_seq_len) == ( + base.max_num_tokens, + base.max_seq_len, + ) + md = _target_sizing(resolved) + assert md.runtime_features.cache_reuse is True + assert md.runtime_features.chunked_prefill is base.case.chunked + assert md.enable_context_mla_with_cached_kv is True + assert tep4_modeling._most_ctx_kv(md) == most + assert tep4_modeling._ctx_workspace_rows(md) == 2**19 + assert tep4_modeling._prefill_logits_budget(md.max_num_tokens, md.max_seq_len) == budget + + +def _block_table(num_seqs: int, max_blocks: int) -> torch.Tensor: + """A page table whose every entry is a distinct page id, none in its own row position.""" + pages = torch.arange(num_seqs * max_blocks, dtype=torch.int32).flip(0) * 3 + 7 + return pages.view(num_seqs, max_blocks).contiguous() + + +def _indexer_step(resolved, rank: int, ctx, gen=()): + """One step's DSA metadata, prepared on the host by the engine's own code. + + ``ctx`` holds (cached, new) tokens per context request, ``gen`` the cached tokens of each + decode row behind them. `DSAtrtllmAttentionMetadata.prepare_for_mla_rope_append`, + `.prepare_for_skip_indexer` and `Indexer.prepare` run as `DSAtrtllmAttentionMetadata.prepare` + calls them, on rank ``rank``'s mapping; DeepGEMM's decode schedule is recorded, not built. + """ + params = _metadata_params(resolved) + num_contexts, num_generations = len(ctx), len(gen) + num_seqs = num_contexts + num_generations + seq_lens = torch.tensor([new for _, new in ctx] + [1] * num_generations, dtype=torch.int32) + cached = torch.tensor([c for c, _ in ctx] + list(gen), dtype=torch.int32) + num_tokens, num_ctx_tokens = int(seq_lens.sum()), sum(new for _, new in ctx) + topk = params.max_sparse_topk + + def buffer(*shape, dtype=torch.int64): + return torch.zeros(*shape, dtype=dtype) + + md = SimpleNamespace( + kv_cache_manager=SimpleNamespace(index_head_dim=128, max_seq_len=resolved.max_seq_len), + sparse_metadata_params=params, + indexer_max_chunk_size=params.indexer_max_chunk_size, + num_sparse_topk=topk, + enable_indexer_skip=params.enable_indexer_skip, + indexer_head_dim=params.index_head_dim, + indexer_quant_block_size=128, + compress_ratios=[1], + _tokens_per_block=_TOKENS_PER_BLOCK, + max_num_tokens=resolved.max_num_tokens, + mapping=_rank_mapping(resolved.mapping, rank), + enable_context_mla_with_cached_kv=resolved.llm_args.enable_chunked_prefill, + runtime_features=_runtime_features(resolved), + num_contexts=num_contexts, + num_generations=num_generations, + num_seqs=num_seqs, + num_ctx_tokens=num_ctx_tokens, + num_tokens=num_tokens, + seq_lens=seq_lens, + request_ids=list(range(num_seqs)), + kv_cache_params=SimpleNamespace( + num_cached_tokens_per_seq=cached.tolist(), num_extra_kv_tokens=0 + ), + host_indexer_k_cache_block_offsets=_block_table( + num_seqs, _max_blocks_per_seq(resolved.max_seq_len) + ), + host_slot_mapping_fp8=buffer(num_tokens), + host_slot_mapping_scale=buffer(num_tokens), + slot_mapping_fp8=buffer(num_tokens), + slot_mapping_scale=buffer(num_tokens), + cu_seqlen_ks=buffer(max(num_ctx_tokens, 1), dtype=torch.int32), + cu_seqlen_ke=buffer(max(num_ctx_tokens, 1), dtype=torch.int32), + host_ctx_cached_token_indptr=buffer(num_contexts + 1), + ctx_cached_token_indptr=buffer(num_contexts + 1), + host_ctx_kv_indptr=buffer(num_contexts + 1), + ctx_kv_indptr=buffer(num_contexts + 1), + host_gen_cached_token_indptr=buffer(num_generations + 1), + gen_cached_token_indptr=buffer(num_generations + 1), + host_gen_kv_indptr=buffer(num_generations + 1), + gen_kv_indptr=buffer(num_generations + 1), + host_topk_indices_buffer=buffer(num_tokens, topk, dtype=torch.int32), + topk_indices_buffer=buffer(num_tokens, topk, dtype=torch.int32), + indexer_prefill_chunks=None, + indexer_prefill_q_split=None, + decode_schedules=[], + ) + md.prepare_dense_topk_indices = MethodType( + DSAtrtllmAttentionMetadata.prepare_dense_topk_indices, md + ) + md._get_dense_topk_indices = MethodType(DSAtrtllmAttentionMetadata._get_dense_topk_indices, md) + kv_lens = cached + seq_lens + with pytest.MonkeyPatch.context() as mp: + mp.setattr(dsa_indexer, "maybe_pin_memory", lambda tensor: tensor) + mp.setattr(dsa_indexer, "prefer_pinned", lambda: False) + mp.setattr(torch.Tensor, "cuda", lambda tensor, *args, **kwargs: tensor) + mp.setattr( + Indexer, + "prepare_scheduler_metadata", + staticmethod( + lambda metadata: metadata.decode_schedules.append(metadata.num_generations) + ), + ) + DSAtrtllmAttentionMetadata.prepare_for_mla_rope_append(md, cached, kv_lens) + DSAtrtllmAttentionMetadata.prepare_for_skip_indexer(md, kv_lens) + Indexer.prepare(md) + return md + + +def _key_slots(block_table: torch.Tensor, row: int, positions: torch.Tensor): + """Where an fp8 indexer key and its fp32 scale live: pages of 32 tokens of 128 key bytes + followed by 32 scales of 4 bytes, 4224 bytes a page.""" + pages = block_table[row, positions // _TOKENS_PER_BLOCK].to(torch.int64) + offset = positions % _TOKENS_PER_BLOCK + page_bytes = _TOKENS_PER_BLOCK * (128 + 4) + assert page_bytes == 4224 + return ( + pages * page_bytes + offset * 128, + pages * page_bytes + _TOKENS_PER_BLOCK * 128 + offset * 4, + ) + + +# The chunks of the 1M gate's prompt that the scheduler admits alone: 31 of 32768 tokens +# and a last of 32751. +_CHUNKS = [_MAX_NUM_TOKENS] * 31 + [_ENVELOPE_PROMPT - 31 * _MAX_NUM_TOKENS] + + +@pytest.mark.parametrize("chunk", [0, 1, 2, 31], ids=["first", "second", "third", "last"]) +def test_later_chunks_carry_their_cached_prefix(envelope, chunk): + """Every chunk of one request after its first is a context request with a cached prefix. + `DSAtrtllmAttentionMetadata.prepare_for_mla_rope_append` (attention/backends/sparse/dsa/ + metadata.py) hands the target its cached and total prefix sums, `ctx_cached_token_indptr` + [0, cached] and `ctx_kv_indptr` [0, cached + new], `num_ctx_cached_tokens` = cached and + `max_ctx_seq_len` = new; `Indexer.prepare` counts `num_ctx_kv_tokens` = new keys written this + step, and `Indexer.recompute_context_kv_gather_mappings` builds `slot_mapping_fp8_fullkv` / + `slot_mapping_scale_fullkv` over the cached and the new keys, position p at page p // 32 of + the request's indexer page table. On a first chunk there is nothing cached and the full + mappings are the step's own.""" + cached, new = sum(_CHUNKS[:chunk]), _CHUNKS[chunk] + md = _indexer_step(envelope, 0, ctx=[(cached, new)]) + total = cached + new + assert md.host_ctx_cached_token_indptr.tolist() == [0, cached] + assert md.ctx_cached_token_indptr.tolist() == [0, cached] + assert md.host_ctx_kv_indptr.tolist() == [0, total] + assert md.ctx_kv_indptr.tolist() == [0, total] + assert (md.num_ctx_cached_tokens, md.max_ctx_seq_len, md.max_ctx_kv_len) == (cached, new, total) + assert md.num_ctx_mla_kv_tokens == total + assert md.num_ctx_kv_tokens == new + # The target's context workspace holds the request's whole KV, the step's cached count plus + # its context tokens (`PrefillTarget._ctx_workspace`), placed by the total prefix sums. + assert md.num_ctx_cached_tokens + md.num_ctx_tokens == int(md.ctx_kv_indptr[-1]) == total + assert md.skip_indexer_for_ctx_reqs is False + + table = md.host_indexer_k_cache_block_offsets + fp8, scale = _key_slots(table, 0, torch.arange(cached, total)) + assert torch.equal(md.slot_mapping_fp8, fp8) and torch.equal(md.slot_mapping_scale, scale) + if chunk == 0: + assert md.slot_mapping_fp8_fullkv is md.slot_mapping_fp8 + assert md.slot_mapping_scale_fullkv is md.slot_mapping_scale + else: + fp8, scale = _key_slots(table, 0, torch.arange(total)) + assert md.slot_mapping_fp8_fullkv.shape == (total,) + assert torch.equal(md.slot_mapping_fp8_fullkv, fp8) + assert torch.equal(md.slot_mapping_scale_fullkv, scale) + assert torch.equal(md.slot_mapping_fp8_fullkv[cached:], md.slot_mapping_fp8) + # The last chunk ends one token short of max_seq_len less the tokens it generates. + assert total <= _ENVELOPE_PROMPT and (chunk != 31 or total == _ENVELOPE_PROMPT) + + +# ------------------------------------------------------- (3) the query split across TP ranks + + +def test_q_split_threshold_is_8192(envelope): + """`DeepSeekSparseAttentionConfig.q_split_threshold` (llmapi/llm_args.py) defaults to 8192 and + reaches `DSAMetadataParams`, which `_prefill_q_split_threshold` (dsa/indexer.py) reads; the + split is on, not disabled by a negative value.""" + params = _metadata_params(envelope) + assert envelope.model_config.sparse_attention_config.q_split_threshold == 8192 + assert params.q_split_threshold == 8192 + assert ( + dsa_indexer._prefill_q_split_threshold(SimpleNamespace(sparse_metadata_params=params)) + == 8192 + ) + + +# context tokens in a step -> per-rank sizes of the split (None: no split) +_SPLITS = { + 8191: None, + 8192: [2048, 2048, 2048, 2048], + 8193: [2048, 2048, 2048, 2049], + 32767: [8191, 8192, 8192, 8192], + 32768: [8192, 8192, 8192, 8192], + # several short prompts, 3000 + 3000 + 3000 + 500 tokens + 9500: [2375, 2375, 2375, 2375], +} + + +@pytest.mark.parametrize("count", list(_SPLITS)) +def test_split_prefill_queries_per_rank(envelope, count): + """`_split_prefill_queries` (dsa/indexer.py) on each TP rank's mapping: below 8192 context + tokens every rank takes them all; from 8192 on, rank r takes [count r / 4, count (r + 1) / 4) + and every rank knows every share's size. Attention DP (dep4's mapping) never splits.""" + sizes = _SPLITS[count] + begins = [0] if sizes is None else [sum(sizes[:r]) for r in range(_RANKS + 1)] + for rank in range(_RANKS): + mapping = _rank_mapping(envelope.mapping, rank) + assert mapping.tp_rank == rank + split = dsa_indexer._split_prefill_queries(count, mapping, 8192) + if sizes is None: + assert split == (0, count, None) + else: + assert split == (begins[rank], begins[rank + 1], sizes) + adp = copy.deepcopy(envelope.mapping) + adp.enable_attention_dp = True + assert dsa_indexer._split_prefill_queries(count, adp, 8192) == (0, count, None) + + +def _covered(chunks, q_lens): + q_starts = [sum(q_lens[:i]) for i in range(len(q_lens))] + spans = [] + for chunk in chunks: + for req_idx, start, end, req_start in chunk: + assert req_start == q_starts[req_idx] + spans.append((req_start + start, req_start + end)) + return spans + + +@pytest.mark.parametrize( + "q_lens,kv_lens", + [ + pytest.param([32768], [3 * 32768], id="one-chunk-over-a-cached-prefix"), + pytest.param([32751], [_ENVELOPE_PROMPT], id="last-chunk"), + pytest.param([3000, 3000, 3000, 500], [3000, 3000, 3000, 500], id="short-prompts"), + pytest.param([2048] * 16, [2048] * 16, id="sixteen-prompts"), + pytest.param([600] * 15 + [23768], [600] * 15 + [23768 + 5 * 32768], id="mixed-lengths"), + ], +) +def test_chunk_plan_covers_each_rank_share(envelope, delivered, q_lens, kv_lens): + """`_packed_logits_elem_budget` (dsa/indexer.py) is 1 << 28 at both configs (the packed cap; + `_INDEXER_MQA_LOGITS_ELEM_BUDGET` x max_num_tokens x max_seq_len do not bind). Under it + `plan_mla_chunked_prefill_chunks` covers exactly the rank's share [q_begin, q_end) of the + context tokens, in order, once each; a request the share boundary cuts is a chunk of its + own, and whole requests share a chunk only within the budget.""" + for resolved in (envelope, delivered): + budget = dsa_indexer._packed_logits_elem_budget( + SimpleNamespace( + max_num_tokens=resolved.max_num_tokens, + kv_cache_manager=SimpleNamespace(max_seq_len=resolved.max_seq_len), + ) + ) + assert budget == dsa_indexer._INDEXER_PACKED_LOGITS_ELEM_BUDGET == 1 << 28 + assert dsa_indexer._INDEXER_MQA_LOGITS_ELEM_BUDGET == 1 << 31 + + total = sum(q_lens) + covered = [] + for rank in range(_RANKS): + q_begin, q_end, sizes = dsa_indexer._split_prefill_queries( + total, _rank_mapping(envelope.mapping, rank), 8192 + ) + assert sizes is not None + chunks = plan_mla_chunked_prefill_chunks(q_lens, kv_lens, 1 << 28, q_begin, q_end) + spans = _covered(chunks, q_lens) + assert spans[0][0] == q_begin and spans[-1][1] == q_end + assert all(prev[1] == cur[0] for prev, cur in zip(spans, spans[1:])) + for chunk in chunks: + whole = all(start == 0 and end == q_lens[i] for i, start, end, _ in chunk) + if len(chunk) > 1: + assert whole + assert sum(q_lens[i] for i, *_ in chunk) * sum(kv_lens[i] for i, *_ in chunk) <= ( + 1 << 28 + ) + elif not whole: + req_idx, start, end, req_start = chunk[0] + assert req_start + start == q_begin or req_start + end == q_end + covered += spans + assert covered[0][0] == 0 and covered[-1][1] == total + assert all(prev[1] == cur[0] for prev, cur in zip(covered, covered[1:])) + + +@pytest.mark.parametrize("chunk", [1, 31], ids=["second", "last"]) +def test_each_rank_plans_its_share_of_a_chunk(envelope, chunk): + """Through `Indexer.prepare` on each TP rank: a chunk of the long prompt (32768 or 32751 + tokens over its cached prefix) sets `indexer_prefill_q_split` = (q_begin, q_end, sizes) and + one `indexer_prefill_chunks` entry for the rank's share, queries [q_begin, q_end) of the + chunk against keys [0, cached + q_end) of the request, each query's window ending at its own + position. The four shares tile the chunk.""" + cached, new = sum(_CHUNKS[:chunk]), _CHUNKS[chunk] + sizes = _SPLITS[32768] if new == 32768 else [8187, 8188, 8188, 8188] + assert sum(sizes) == new + for rank in range(_RANKS): + md = _indexer_step(envelope, rank, ctx=[(cached, new)]) + q_begin, q_end = sum(sizes[:rank]), sum(sizes[: rank + 1]) + assert md.indexer_prefill_q_split == (q_begin, q_end, sizes) + assert len(md.indexer_prefill_chunks) == 1 + plan = md.indexer_prefill_chunks[0] + assert (plan.token_start, plan.token_end) == (q_begin, q_end) + assert (plan.k_token_start, plan.k_token_end) == (0, cached + q_end) + assert torch.equal(plan.cu_seqlen_ks, torch.zeros(q_end - q_begin, dtype=torch.int32)) + assert torch.equal( + plan.cu_seqlen_ke, + torch.arange(cached + q_begin + 1, cached + q_end + 1, dtype=torch.int32), + ) + assert md.decode_schedules == [] + + +def test_q_split_on_skipped_and_mixed_steps(envelope): + """What `Indexer.prepare_for_chunked_prefill` does on the two steps the target must not + misread. (a) A step whose every context is within index_topk 2048: `prepare_for_skip_indexer` + sets `skip_indexer_for_ctx_reqs` and fills the dense top-k rows, and the engine STILL sets + `indexer_prefill_q_split` once the context tokens reach 8192 (and plans the share's chunks); + only the full-KV gather mappings are skipped. A target that gathers on the split must not do + so on a skipped step. (b) A mixed step, one chunk with 15 decode rows behind it: the split + counts the context tokens only and covers the context rows; the decode rows keep their own + schedule.""" + skipped = _indexer_step(envelope, 1, ctx=[(0, 1000)] * 9) + assert skipped.skip_indexer_for_ctx_reqs is True + assert skipped.indexer_prefill_q_split == (2250, 4500, [2250] * 4) + # Rank 1's share cuts the third prompt after 250 tokens and the fifth after 500. + assert [(c.token_start, c.token_end) for c in skipped.indexer_prefill_chunks] == [ + (2250, 3000), + (3000, 4000), + (4000, 4500), + ] + assert skipped.slot_mapping_fp8_fullkv is skipped.slot_mapping_fp8 + dense = skipped.host_topk_indices_buffer[:9000] + assert torch.equal(dense[999, :1000], torch.arange(1000, dtype=torch.int32)) + assert bool((dense[999, 1000:] == -1).all()) + + cached, new = 2 * 32768, 32768 - 32 + decode = [1048000 - 64 * i for i in range(15)] + mixed = _indexer_step(envelope, 3, ctx=[(cached, new)], gen=decode) + assert (mixed.num_contexts, mixed.num_generations, mixed.num_tokens) == (1, 15, 32751) + assert mixed.indexer_prefill_q_split == (3 * 8184, 32736, [8184] * 4) + (plan,) = mixed.indexer_prefill_chunks + assert (plan.token_start, plan.token_end, plan.k_token_end) == (24552, 32736, cached + 32736) + assert mixed.decode_schedules == [15] + assert mixed.max_gen_seq_len == 1 + assert mixed.host_gen_kv_indptr[-1].item() == sum(decode) + 15 + + +def test_context_rows_take_the_path_of_their_kv(envelope): + """The target's context-row path (`PrefillTarget._ctx_workspace`) on the engine's own host + metadata at the 1M config, rank by rank, over the held workspace's rows the target takes from + this engine (`_ctx_workspace_rows`: 2**19, chunked prefill letting a step attend cached + prefixes). It reads a step's context KV as cached + new over its context requests + (`_ctx_kv_offsets`, the host `ctx_kv_indptr[num_contexts]`), never `num_ctx_kv_tokens`, which + counts the new tokens only: + + * the KV-capacity estimate's step, one fresh 32768-token prompt split 8192 a rank, takes the + workspace path, and that first context step allocates the held bf16 [2**19, 576] workspace; + * the 1M gate prompt's chunks, each a query-split step with the indexer running: the first + 16 (context KV up to 2**19) on the workspace path, from the 17th on the decode path in + sub-batches of 8192, the last (32751 rows behind 1015808 cached) 8192 x 3 and 8175; + * one row of context KV either side of 2**19, with a decode row behind; + * a packed step of four chunks over 1032192, 524288, 0 and 2048 cached keys with twelve + decode rows behind them: the decode path over the 32752 context rows only; + * nine prompts within index_topk (the context indexer skipped, the split still planned): the + workspace path. + + A workspace-path step keeps layer 35's gather beside its rows when its context KV fits the + held rows twice (the first 8 chunks), and gathers it again for each sharer otherwise. The + workspace is never reallocated after the estimate's step.""" + rows_b = tep4_modeling._ctx_workspace_rows(_target_sizing(envelope)) + assert (rows_b, tep4_modeling._CTX_DECODE_ROWS) == (2**19, 8192) + target = tep4_modeling.PrefillTarget.__new__(tep4_modeling.PrefillTarget) + target.core = SimpleNamespace(lat_dim=576, ctx_workspace_rows=rows_b) + + def path(rank, ctx, gen=()): + md = _indexer_step(envelope, rank, ctx=ctx, gen=gen) + step = dict( + num_contexts=md.num_contexts, + num_ctx_tokens=md.num_ctx_tokens, + **tep4_modeling._ctx_kv_offsets(md, md.num_contexts), + ) + n_kv = sum(c + n for c, n in ctx) + assert int(md.host_ctx_kv_indptr[md.num_contexts]) == n_kv + assert not any(c for c, _ in ctx) or md.num_ctx_kv_tokens == md.num_ctx_tokens < n_kv + ws = target._ctx_workspace(step, md.num_ctx_tokens, torch.device("cpu")) + split = md.indexer_prefill_q_split + if ws.ws_buf is not None: + assert n_kv <= rows_b and ws.ws_buf.shape == (n_kv, 576) + assert ws.ws_buf.data_ptr() == target._ws_buf.data_ptr() + # Layer 35's gather is kept beside the step's rows where they fit twice. + assert (ws.kept is not None) == (2 * n_kv <= rows_b) + return "workspace", split, None + assert n_kv > rows_b and sorted(ws.sched) == sorted({hi - lo for lo, hi in ws.sub_batches}) + return "decode", split, ws.sub_batches + + full = [(0, 8192), (8192, 16384), (16384, 24576), (24576, 32768)] + assert path(1, [(0, _MAX_NUM_TOKENS)]) == ("workspace", (8192, 16384, [8192] * 4), None) + held = target._ws_buf + assert held.shape == (rows_b, 576) and held.dtype == torch.bfloat16 + for k, new in enumerate(_CHUNKS): + got = path(k % _RANKS, [(sum(_CHUNKS[:k]), new)]) + q_begin, q_end, sizes = got[1] + assert sum(sizes) == new and q_end - q_begin == sizes[k % _RANKS] + if k < 16: + assert got[0] == "workspace", k + elif k < 31: + assert got[0::2] == ("decode", full), k + else: + assert got[0::2] == ("decode", full[:3] + [(24576, 32751)]) + edge = rows_b - 32767 + assert path(0, [(edge, 32767)], gen=[4096])[0] == "workspace" + assert path(0, [(edge + 1, 32767)], gen=[4096])[0::2] == ("decode", full[:3] + [(24576, 32767)]) + packed = [(1032192, 16384), (524288, 8192), (0, 4096), (2048, 4080)] + got = path(2, packed, gen=[1048000 - 64 * i for i in range(12)]) + assert got[0::2] == ("decode", full[:3] + [(24576, 32752)]) + assert got[1] is not None + skipped = path(3, [(0, 1000)] * 9) + assert skipped == ("workspace", (6750, 9000, [2250] * 4), None) + assert target._ws_buf is held + + +# ------------------------------------------------------- (4) the scheduler's chunks + + +def _scheduler_pool() -> Mock: + """The surface of a KVCacheManagerV2 the V2 scheduler reads, with room for everything.""" + pool = Mock() + pool.kv_connector_manager = None + pool.tokens_per_block = _TOKENS_PER_BLOCK + pool.block_reuse_policy = BlockReusePolicy.PER_REQUEST + pool.enable_partial_reuse = True + pool.enable_joint_kv_cache_reuse = False + pool.enable_block_reuse = False + pool.num_extra_kv_tokens = 0 + pool._spec_recompute_tail = 0 + pool.is_draft = False + pool.context_replay_tokens.return_value = None + pool.can_evict = False + pool._has_cp_helix = False + pool.is_vswa = False + pool.probe_first_new_block_key.return_value = None + pool.kv_cache_map = {} + pool.prepare_context.return_value = True + pool.resize_context.return_value = True + pool.try_allocate_generation.return_value = True + pool.has_cache_tier_below_gpu = True + pool.has_pending_preemption.return_value = False + pool.can_share_inflight_prefix = False + pool.inforward_snapshot_rows = 0 + return pool + + +def _request(request_id: int, state: LlmRequestState, **fields) -> Mock: + request = Mock() + request.request_id = request.py_request_id = request_id + request.state_value = state.value + request.py_csa2_remote_tail_mode = None + request.py_encoder_output_ready_event = None + request.py_multimodal_data = None + request.py_ced_replay = None + request.py_uncached_prefill_tokens = None + request.py_ctx_sched_since_iter = None + request.cache_salt = None + request.lora_task_id = None + request.num_draft_tokens = 0 + request.has_draft_tokens = False + request.py_draft_tokens = [] + request.get_beam_width_by_iter.return_value = 1 + request.is_context_init_state = state == LlmRequestState.CONTEXT_INIT + request.is_generation_in_progress_state = state == LlmRequestState.GENERATION_IN_PROGRESS + request.is_generation_to_complete_state = False + for name, value in fields.items(): + setattr(request, name, value) + return request + + +def _schedule_prompt(envelope, prompt_len: int, decode_rows: int) -> list[tuple[int, int]]: + """Steps of the scheduler `create_py_executor_instance` (pyexecutor/_util.py) builds for a V2 + KV manager, admitting one prompt of ``prompt_len`` tokens while ``decode_rows`` requests + decode beside it: (context tokens, generation rows) per step.""" + llm_args = envelope.llm_args + pool = _scheduler_pool() + with patch( + "tensorrt_llm._torch.pyexecutor.kv_cache.kv_cache_manager_v2.KVCacheManagerV2", + new=type(pool), + ): + scheduler = KVCacheV2Scheduler( + max_batch_size=envelope.max_batch_size, + max_num_tokens=envelope.max_num_tokens, + kv_cache_manager=pool, + scheduler_policy=llm_args.scheduler_config.capacity_scheduler_policy, + # `_create_py_executor` (py_executor_creator.py): first come first served, in units of + # tokens_per_block when no attention window is shorter than max_seq_len. + ctx_chunk_config=(ContextChunkingPolicy.FIRST_COME_FIRST_SERVED, _TOKENS_PER_BLOCK), + scheduler_capacity=envelope.max_batch_size, + ) + assert scheduler.policy == CapacitySchedulerPolicy.MAX_UTILIZATION + assert (scheduler.chunking_enabled, scheduler.chunk_unit_size) == (True, _TOKENS_PER_BLOCK) + assert scheduler.max_context_length == _MAX_NUM_TOKENS + + prompt = _request( + 0, + LlmRequestState.CONTEXT_INIT, + prompt_len=prompt_len, + context_remaining_length=prompt_len, + context_current_position=0, + context_chunk_size=0, + py_served_prefix_end=0, + expect_snapshot_points=[], + is_first_context_chunk=True, + is_last_context_chunk=False, + encoder_output_len=None, + py_skip_cross_kv_projection=False, + ) + decoding = [ + _request(1 + i, LlmRequestState.GENERATION_IN_PROGRESS, is_first_context_chunk=False) + for i in range(decode_rows) + ] + steps = [] + while prompt.context_remaining_length > 0: + out = scheduler.schedule_request([prompt] + decoding, set()) + assert out.context_requests == [prompt] + chunk = prompt.context_chunk_size + steps.append((chunk, len(out.generation_requests))) + prompt.context_remaining_length -= chunk + prompt.context_current_position += chunk + prompt.is_first_context_chunk = False + return steps + + +def test_scheduler_admits_the_envelope_prompt_in_chunks(envelope): + """`KVCacheV2Scheduler` (pyexecutor/scheduler/scheduler_v2.py), the scheduler of a V2 KV + manager, with the chunking `_create_py_executor` configures. It schedules the decode rows + first, then cuts the context chunk from what the step's 32768 tokens leave, rounded down to + the 32-token unit unless it is the prompt's last. The configured GUARANTEED_NO_EVICT is + served as MAX_UTILIZATION. Alone, the 1048559-token prompt takes 32 steps, 31 of 32768 and a + last of 32751; beside 15 decode rows each chunk is 32736, context and decode rows 32751 of + the 32768, and the prompt takes 33 steps, the last 1007.""" + assert envelope.llm_args.scheduler_config.capacity_scheduler_policy == ( + CapacitySchedulerPolicy.GUARANTEED_NO_EVICT + ) + alone = _schedule_prompt(envelope, _ENVELOPE_PROMPT, 0) + assert alone == [(c, 0) for c in _CHUNKS] + assert len(alone) == 32 and alone[-1] == (32751, 0) + + beside = _schedule_prompt(envelope, _ENVELOPE_PROMPT, 15) + assert beside == [(32736, 15)] * 32 + [(1007, 15)] + assert (32768 - 15) // 32 * 32 == 32736 + assert all(chunk + rows <= _MAX_NUM_TOKENS for chunk, rows in beside) + assert sum(chunk for chunk, _ in beside) == _ENVELOPE_PROMPT + + +# ------------------------------------------------------- (5) what max_seq_len sizes + + +def test_kv_cache_sizing(resolved): + """The DSA pool per rank (`MistralLarge4CacheManager.get_cache_size_per_token`, attention/ + backends/sparse/mistral_large4/cache_manager.py): the 63 KV owners keep an fp8_ds_mla latent + of 656 bytes (512 e4m3, four fp32 tile scales, 64 bf16 rope), the 44 full-indexer layers an + indexer key of 128 fp8 bytes and one fp32 scale, pages of 32; the nine sharers keep no + latent. 47136 bytes a token on every rank, whatever max_seq_len; 16 requests at the + engine's sequence length need that per token times their tokens.""" + model_config, llm_args = resolved.model_config, resolved.llm_args + kv_cache_config = llm_args.kv_cache_config + manager = get_kv_cache_manager_cls(model_config, kv_cache_config) + assert manager is MistralLarge4CacheManager + assert kv_cache_config.tokens_per_block == _TOKENS_PER_BLOCK + + cfg = model_config.pretrained_config + assert cfg.kv_lora_rank + cfg.qk_rope_head_dim == 576 + assert sum(1 for i, s in enumerate(cfg.kv_source_layers) if s == i) == 63 + assert cfg.index_topk_pattern.count("F") == 44 + for rank in range(_RANKS): + cost = _kv_cost(resolved, rank) + assert (cost.slope, cost.intercept) == (656 * 63 + 132 * 44, 0) == (47136, 0) + tokens = resolved.max_batch_size * resolved.max_seq_len + pool = cost.bytes_for_tokens(tokens) + assert pool == tokens * 47136 + if resolved.max_seq_len == _MAX_SEQ_LEN: + assert tokens == 16 * _MAX_SEQ_LEN == 16_777_216 + assert pool == 16_777_216 * 47136 == 790_810_853_376 + assert pool / 2**30 == 736.5 + if resolved.case is _NVFP4_32K: + assert tokens == 524288 + assert pool == 524288 * 47136 == 24_712_839_168 + assert math.isclose(pool / 2**30, 23.016, abs_tol=5e-4) + + +class _Pool(MistralLarge4CacheManager): + """The fp8_ds_mla KV manager as the metadata reads it at construction: its page geometry + only, its local layers the 63 KV owners.""" + + tokens_per_block = _TOKENS_PER_BLOCK + num_pools = 1 + num_attention_op_pools = 1 + num_local_layers = 63 + host_kv_cache_block_offsets = None + dtype = tensorrt_llm.bindings.DataType.FP8 + get_fp4_mla_page_table_spec = None + use_fp8_ds_mla = False + + def __init__(self, max_seq_len: int): + self.max_seq_len = max_seq_len + self.max_blocks_per_seq = _max_blocks_per_seq(max_seq_len) + + +def _metadata_buffers(resolved) -> dict: + """Every buffer `DSAtrtllmAttentionMetadata.__post_init__` (dsa/metadata.py), with + `TrtllmAttentionMetadata._post_init_with_buffers` (attention/backends/trtllm.py), allocates + for this engine: cache name -> (shape, dtype). Allocated on the meta device.""" + allocated = {} + + def get_empty(buffers, tensor_shape, dtype, cache_name, capture_graph=False): + allocated[cache_name] = (tuple(tensor_shape), dtype) + return torch.empty(tuple(tensor_shape), dtype=dtype, device="meta") + + md = object.__new__(DSAtrtllmAttentionMetadata) + md.__dict__.update( + max_num_requests=resolved.max_batch_size, + max_num_sequences=resolved.max_batch_size, + max_num_tokens=resolved.max_num_tokens, + kv_cache_manager=object.__new__(_Pool), + draft_kv_cache_manager=None, + mapping=resolved.mapping, + runtime_features=_runtime_features(resolved), + enable_context_mla_with_cached_kv=resolved.llm_args.enable_chunked_prefill, + enable_flash_mla=False, + sparse_metadata_params=_metadata_params(resolved), + is_cross=False, + cross=None, + is_cuda_graph=False, + cuda_graph_buffers=None, + workspace=torch.empty(0, device="meta"), + cuda_graph_workspace=torch.empty(0, device="meta"), + max_draft_tokens=0, + num_sms=148, + enable_ragged_verification=False, + ragged_verify_lens=None, + _radix_rows_per_sequence=1, + ) + _Pool.__init__(md.kv_cache_manager, resolved.max_seq_len) + with pytest.MonkeyPatch.context() as mp: + mp.setattr(AttentionMetadata, "get_empty", staticmethod(get_empty)) + mp.setattr(dsa_metadata, "prefer_pinned", lambda: False) + mp.setattr(trtllm_backend, "prefer_pinned", lambda: False) + mp.setattr(dsa_metadata, "get_sm_version", lambda: 103) + md.__post_init__() + assert md.use_cute_dsl_topk is False and md.needs_gvr_prior is False + return allocated + + +def test_buffers_sized_by_max_seq_len(envelope, delivered): + """The KV manager's page tables are max_seq_len wide: `KVCacheManagerV2.__init__` + (pyexecutor/kv_cache/kv_cache_manager_v2.py) takes max_seq_len plus one token of generation + headroom (`_get_generation_kv_capacity`) in pages of 32 and pads to a multiple of 4: 32772 + columns at 1048576 (1028 at 32768). The attention metadata the target reads (`kv_cache_block_ + offsets`, the DSA `block_table`, `indexer_k_cache_block_offsets`, `block_table_expanded`) is + 16 sequences by that width. Every other buffer -- `topk_indices_buffer` [32768, 2048], the + radix top-k's `radix_aux_indices` / `radix_aux_logits` [16, 10, 2048], `scheduler_metadata_ + buffer` [num_sms + 1, 2], the slot mappings and query bounds [32768] -- is the same at both + configs; none is max_batch_size x max_seq_len.""" + assert _get_generation_kv_capacity(None, is_draft=False) == (0, 1) + assert _max_blocks_per_seq(_MAX_SEQ_LEN) == 32772 and _max_blocks_per_seq(32768) == 1028 + wide, narrow = _metadata_buffers(envelope), _metadata_buffers(delivered) + + tables = { + "kv_cache_block_offsets": ((1, 16, 2), torch.int32), + "block_table": ((16,), torch.int32), + "indexer_k_cache_block_offsets": ((16,), torch.int32), + "block_table_expanded": ((16,), torch.int32), + } + for name, (rows, dtype) in tables.items(): + assert wide[name] == (rows + (32772,), dtype) + assert narrow[name] == (rows + (1028,), dtype) + assert wide["topk_indices_buffer"] == ((32768, 2048), torch.int32) + assert wide["radix_aux_indices"] == ( + (16, dsa_metadata._MAX_RADIX_BLOCKS_PER_ROW, 2048), + torch.int32, + ) + assert wide["radix_aux_logits"] == ((16, 10, 2048), torch.float32) + assert wide["scheduler_metadata_buffer"] == ((149, 2), torch.int32) + assert wide["slot_mapping_fp8"] == wide["slot_mapping_scale"] == ((32768,), torch.int64) + assert wide["cu_seqlen_ks"] == wide["cu_seqlen_ke"] == ((32768,), torch.int32) + assert wide["shared_topk_indices"] == ((32768, 2048), torch.int32) + assert wide["ctx_cached_token_indptr"] == wide["ctx_kv_indptr"] == ((17,), torch.int64) + + # Under cached-KV MLA the base metadata also keeps the uncached prefix sums. + assert set(wide) - set(narrow) == {"ctx_uncached_token_indptr"} + assert set(narrow) <= set(wide) + for name, (shape, dtype) in wide.items(): + assert _MAX_SEQ_LEN not in shape, name + if name not in tables and name in narrow: + assert (shape, dtype) == narrow[name], name + + +class _KvPool: + """The KV manager surface warmup and capture read: ``tokens`` free tokens in pages of 32, + dummies recorded as the token counts they were asked for.""" + + tokens_per_block = _TOKENS_PER_BLOCK + num_extra_kv_tokens = 0 + + def __init__(self, tokens: int, max_seq_len: int): + self.tokens = tokens + self.max_seq_len = max_seq_len + self.added = [] + + def get_num_available_tokens(self, *, token_num_upper_bound, batch_size=1, **_): + return min(token_num_upper_bound, self.tokens) + + def get_num_free_blocks(self) -> int: + return self.tokens // self.tokens_per_block + + def add_dummy_requests(self, request_ids, token_nums=None, is_gen=False, **kwargs): + if token_nums is None: + token_nums = [1 + kwargs.get("max_num_draft_tokens", 0)] * len(request_ids) + self.added.append((list(token_nums), is_gen)) + return [ + SimpleNamespace( + py_request_id=request_id, + token_num=token_num, + is_gen=is_gen, + is_last_context_chunk=True, + is_attention_dp_dummy=False, + py_draft_tokens=[], + py_needs_onehot_draft_probs=False, + ) + for request_id, token_num in zip(request_ids, token_nums) + ] + + def free_resources(self, request) -> None: + pass + + +class _Resources: + def __init__(self, kv: _KvPool): + self.kv = kv + + def get_resource_manager(self, key): + return self.kv if key == ResourceManagerType.KV_CACHE_MANAGER else None + + +def _runner_config(resolved) -> SimpleNamespace: + """The decoder runner's sizes for this engine. No speculative decoding: every draft width is + zero, and beams are one wide.""" + graphs = resolved.llm_args.cuda_graph_config + return SimpleNamespace( + max_num_tokens=resolved.max_num_tokens, + max_seq_len=resolved.max_seq_len, + max_batch_size=resolved.max_batch_size, + cuda_graph_batch_sizes=filter_cuda_graph_batch_sizes( + graphs.batch_sizes, + resolved.max_batch_size, + resolved.max_num_tokens, + 1, + graphs.enable_padding, + ), + max_beam_width=1, + spec_config=None, + use_mrope=False, + max_draft_len=0, + original_max_draft_len=0, + max_total_draft_tokens=0, + max_draft_loop_tokens=0, + ) + + +def _dummy_factory(resolved) -> DummyRequestFactory: + return DummyRequestFactory( + config=_runner_config(resolved), + model=SimpleNamespace(model_config=resolved.model_config), + get_runtime_tokens_per_gen_step=lambda draft_len: 1 + draft_len, + release_request_state=None, + ) + + +def _kv_creator(resolved) -> KvCacheCreator: + creator = KvCacheCreator.__new__(KvCacheCreator) + config = _runner_config(resolved) + creator._model_engine = SimpleNamespace( + model=SimpleNamespace(model_config=resolved.model_config), + batch_size=resolved.max_batch_size, + max_seq_len=resolved.max_seq_len, + max_cuda_graph_batch_size=config.cuda_graph_batch_sizes[-1], + ) + creator._max_num_tokens = resolved.max_num_tokens + creator._max_beam_width = 1 + creator._mapping = resolved.mapping + creator._speculative_config = None + creator._llm_args = resolved.llm_args + creator._tokens_per_block = _TOKENS_PER_BLOCK + creator._is_kv_cache_manager_v2 = True + creator._kv_cache_config = resolved.llm_args.kv_cache_config + creator._net_max_seq_len = resolved.max_seq_len + creator._dummy_reqs = None + return creator + + +def test_kv_estimation_profiles_one_full_step(envelope, delivered): + """Before the pool exists `KvCacheCreator` (pyexecutor/_util.py) profiles dummy context + requests as long as a request may be, net max_seq_len - 1, but no more than max_num_tokens in + all: at the 1M config one 32768-token prompt (the delivered engine: a 32767- and a 1-token + prompt). The DSA config keeps it off the chunked-MLA profile + (`_get_mla_chunked_profile_length`), so the profiled step is one unchunked 32768-token + context. `_get_token_num_for_estimation` sizes the temporary pool for the larger of those + prompts and CUDA-graph capture's longest decode row: ceil((max_seq_len + 1) / 32) + 16 - 1 + pages, 32784 at the 1M config -- 1049088 tokens, 46.05 GiB at the fp8_ds_mla pool's 47136 + bytes a token (46.19 GiB at the 576-byte pool's). It holds a max_seq_len request, so the + estimation KV manager does not lower max_seq_len + (`KVCacheManagerV2.__init__` caps it by `get_num_available_tokens`, which a V2 estimate caps + by these max_tokens, and `KvCacheCreator._create_kv_cache_manager` would carry a lowered + value to every later manager).""" + expected = { + id(envelope): ([32768], 32784), + id(delivered): ([32767, 1], 1040), + } + for resolved in (envelope, delivered): + creator = _kv_creator(resolved) + requests = creator._create_dummy_context_requests(max(1, resolved.max_seq_len - 1)) + assert creator._mla_chunked_profile_length is None + lengths = [len(request.input_token_ids) for request in requests] + prompts, pages = expected[id(resolved)] + assert lengths == prompts and sum(lengths) == resolved.max_num_tokens + creator._dummy_reqs = requests + tokens = creator._get_token_num_for_estimation() + assert tokens == pages * _TOKENS_PER_BLOCK + assert tokens >= resolved.max_seq_len + 1 + assert 32784 * _TOKENS_PER_BLOCK == 1_049_088 + assert _KV_BYTES_PER_TOKEN == 47136 + assert 1_049_088 * _KV_BYTES_PER_TOKEN == 49_449_811_968 + assert math.isclose(1_049_088 * _KV_BYTES_PER_TOKEN / 2**30, 46.05, abs_tol=5e-3) + assert 32784 >= math.ceil((_MAX_SEQ_LEN + 1) / _TOKENS_PER_BLOCK) == 32769 + + +def test_kv_estimation_reserves_no_workspace(envelope): + """What `KvCacheCreator.configure_kv_cache_capacity` (pyexecutor/_util.py) takes out of the KV + budget beside the profiled peak, at the 1M config on sm_103: the indexer MQA-logits tile + (`_INDEXER_MQA_LOGITS_ELEM_BUDGET`) is reserved only for the deepseek_v4 algorithm, not DSA; + the TRTLLM backend declares no context-MLA workspace for fp8 DSA on sm_103 + (`TrtllmAttention.runtime_workspace_bytes_per_token`, attention/backends/trtllm.py: sparse + MLA stages no buffer), so `get_mla_context_workspace_kv_len_cap` installs no attended-KV cap + though chunked prefill is on; and no fixed capacity-sized reserve. The pool is the profiled + budget whole.""" + model_config, mapping = envelope.model_config, envelope.mapping + assert model_config.sparse_attention_config.algorithm == "dsa" != "deepseek_v4" + with pytest.MonkeyPatch.context() as mp: + mp.setattr(trtllm_backend, "get_sm_version", lambda: 103) + w = get_attention_workspace_bytes_per_token(model_config, mapping) + bounded = get_attention_workspace_is_chunked_prefill_bounded(model_config) + fixed = get_attention_workspace_fixed_bytes( + model_config, + mapping, + max_batch_size=_MAX_BATCH_SIZE, + max_num_tokens=_MAX_NUM_TOKENS, + max_seq_len=_MAX_SEQ_LEN, + enable_cuda_graph=True, + ) + assert (w, bounded, fixed) == (0, True, 0) + cap = get_mla_context_workspace_kv_len_cap( + envelope.llm_args.kv_cache_config, + _MAX_BATCH_SIZE, + _MAX_NUM_TOKENS, + _MAX_SEQ_LEN, + True, + bounded, + chunked_workspace_profiled=False, + require_chunked_workspace_profile=w > 0, + ) + assert cap is None + + +def test_warmup_runs_one_full_step(envelope, delivered): + """`DecoderWarmup._get_full_general_warmup_requests` (pyexecutor/engine/runners/decoder/ + warmup.py) bounds its largest shape by min(max_num_tokens, max_batch_size x (max_seq_len - 1)) = + 32768; `DummyRequestFactory.create_warmup_request` (decoder/dummy_requests.py) builds it as + prompts of at most max_seq_len - 1: one 32768-token prompt at the 1M config (a 32767- and a + 1-token prompt delivered), never a 1M prompt -- warmup bypasses the scheduler and runs each + shape as one step. No prefill CUDA graph is captured (`filter_prefill_capture_num_tokens`, + pyexecutor/engine/cuda_graph.py: its 16 x 1048575 ceiling caps only capture candidates).""" + for resolved, prompts in ((envelope, [32768]), (delivered, [32767, 1])): + config = _runner_config(resolved) + resources = _Resources(_KvPool(2 * _MAX_SEQ_LEN, resolved.max_seq_len)) + warmup = DecoderWarmup.__new__(DecoderWarmup) + warmup._config = config + shapes = warmup._get_full_general_warmup_requests(resources) + assert shapes == [(1, 0), (1, 1), (32768, 0), (16, 16), (2, 0)] + batch = _dummy_factory(resolved).create_warmup_request(resources, 32768, 0) + assert [request.token_num for request in batch.context_requests] == prompts + + kept, unrecordable = filter_prefill_capture_num_tokens( + config.cuda_graph_batch_sizes, + resolved.max_num_tokens, + _MAX_BATCH_SIZE, + resolved.max_seq_len, + ) + assert (kept, unrecordable) == ([1, 2, 4, 8, 16], []) + assert resolved.llm_args.prefill_cuda_graph_backend == "disabled" + assert _MAX_BATCH_SIZE * (_MAX_SEQ_LEN - 1) == 16_777_200 + + +def test_cuda_graph_capture_decodes_at_max_seq_len(envelope): + """`DecoderCudaGraphs.warmup_and_capture` (decoder/graphs.py) captures every graph batch (1, + 2, 4, 8, 16) twice for DSA, whose config asks for separate short and long graphs + (`needs_separate_short_long_cuda_graphs`, its threshold index_topk 2048): once at max_seq_len + and once at 2048 less the one token a decode step adds. `create_cuda_graph_warmup_request` + fills each batch with one-token decode rows and one row of the family's longest history + (`add_longest_dummy_request`): 1048575 tokens -- the decode kernels are captured at KV length + 1048576 -- and 2046. At batch 16 the long family needs 32768 + 15 pages, within the 32784 + the estimation pool holds.""" + sparse = copy.deepcopy(envelope.model_config.sparse_attention_config) + assert sparse.needs_separate_short_long_cuda_graphs() is True + families = [_MAX_SEQ_LEN, sparse.seq_len_threshold - 1] + assert families == [1048576, 2047] + + config = _runner_config(envelope) + assert config.cuda_graph_batch_sizes == [1, 2, 4, 8, 16] + for family, longest in zip(families, (1048575, 2046)): + for batch_size in config.cuda_graph_batch_sizes: + pool = _KvPool(32784 * _TOKENS_PER_BLOCK, envelope.max_seq_len) + batch = _dummy_factory(envelope).create_cuda_graph_warmup_request( + _Resources(pool), batch_size, 0, family + ) + rows = [request.token_num for request in batch.generation_requests] + assert rows == [longest] + [1] * (batch_size - 1) + assert all(request.is_gen for request in batch.generation_requests) + pages = math.ceil((longest + 1) / _TOKENS_PER_BLOCK) + (batch_size - 1) + assert pages <= 32784 + assert math.ceil(1048576 / _TOKENS_PER_BLOCK) + 15 == 32783 + + +def test_pool_holds_the_envelope_load(): + """Arithmetic on the delivered pool, 2,373,024 tokens a rank of the 576-byte pool (47280 bytes + a token), re-read at the fp8_ds_mla pool's 47136 (`MistralLarge4CacheManager.get_cache_size_ + per_token`; the V2 manager gives every request ceil((tokens + 1) / 32) pages, + `KVCacheManagerV2.__init__`'s generation headroom): the same bytes hold 2,380,273 tokens. + Charged to them whole: the width-sized allocations the measured 32768 engine did not have -- + the target's rope tables (indexer 64 x fp32 and identity 64 x fp32 at 1048576 rows, the MLA + rope reading the indexer's: 512 MiB, against the 32 MiB that engine held at 32768 rows, the + MLA rope ops' duplicated 2 x 64 x fp32 layout among them, which the target no longer + builds), one gathered key block of 1048576 x 132 bytes, the indexer logits tile at the 1M + config's 1 << 28 budget (1 GiB against 64 MiB at the 32768 config's 1 << 24, + `_prefill_logits_budget`) and the decode scorer's [16, 1048576] fp32 arena --; the + fp8_ds_mla attention's larger scratch in the profiled 32768-token step (the 64-head bf16 + query FlashMLA takes, `PrefillTarget._scratch`, against the 576-byte path's 16-head bf16 and + e4m3 queries and zero rope tail); and the held context workspace (the 1M config's 2**19 rows + of bf16 576, `_ctx_workspace_rows`, `PrefillTarget._ctx_workspace`; a step with more context + KV attends its context rows straight off the pool), charged whole though a 32768 engine now + holds 65536 rows of its own; and the vision tower every rank now loads whole (3,408,437,248 + bytes of bf16), which the delivered engine, the language model only, did not hold. + Together under 160,000 tokens; what is left holds the 1M gate's load -- the + 1048559-token prompt generating 16 tokens beside 15 requests of at most a full step each -- + and two requests of max_seq_len, as does the pool a boot of the 1M accuracy gate measured on + the native checkpoint (2,135,488 tokens a rank at its KV fraction 0.85).""" + mib = 2**20 + rows, heads, lat = _MAX_NUM_TOKENS, 16, 576 + grown = { + "rope tables": 1048576 * (64 + 64) * 4 - 32768 * (2 * 64 + 64 + 64) * 4, + "gathered keys": (1048576 - 32768) * 132, + "logits tile": ((1 << 28) - (1 << 24)) * 4, + "decode arena": 16 * (1048576 - 32768) * 4, + "64-head query": rows * 64 * lat * 2 - rows * heads * (lat * 2 + lat + 64 * 2), + "context workspace": 2**19 * lat * 2, + "vision tower": _VISION_TOWER_BYTES, + } + assert 1048576 * (64 + 64) * 4 == 512 * mib + assert grown["64-head query"] == 1_442_840_576 + assert grown["context workspace"] == 576 * mib + assert sum(grown.values()) == 7_164_305_408 < 6.7 * 2**30 + budget = _DELIVERED_POOL_TOKENS * _DELIVERED_BYTES_PER_TOKEN + assert _DELIVERED_BYTES_PER_TOKEN == 47280 and budget == 112_196_574_720 + assert budget // _KV_BYTES_PER_TOKEN == 2_380_273 + lost = math.ceil(sum(grown.values()) / _KV_BYTES_PER_TOKEN) + assert lost == 151_993 < 160_000 + pool = budget // _KV_BYTES_PER_TOKEN - lost + + def pages(tokens): + return math.ceil((tokens + 1) / _TOKENS_PER_BLOCK) + + gate = pages(_ENVELOPE_PROMPT + 16) + 15 * pages(_MAX_NUM_TOKENS + 16) + assert gate * _TOKENS_PER_BLOCK == 1_540_576 <= pool + two = 2 * pages(_MAX_SEQ_LEN) + assert two * _TOKENS_PER_BLOCK == 2_097_216 <= pool + assert pool - two * _TOKENS_PER_BLOCK > 130_000 + # The measured 1M pool holds the same load. + assert gate * _TOKENS_PER_BLOCK <= two * _TOKENS_PER_BLOCK <= _TEP4_1M_POOL_TOKENS + + +# ------------------------------------------------------- (6) what the router reads, the members + + +def test_router_reads_the_quantization_config_intact(resolved): + """`route`, which `ModelLoader._load_and_validate_config` (pyexecutor/model_loader.py) asks + through `_router_index` for a target, is handed the checkpoint's `quantization_config` + exactly as the native loader reads it from params.json -- params.json's own, the whole + `quantized_layers` map in its ModelOpt spelling, plus the FP8 KV-cache algorithm the loader + records from `fp8_kv_cache` -- and it stays so on the ModelConfig the target is built from, + on the tep4 mapping. The copies the LLM API and ModelConfig normalise (and pop + `quantized_layers` from) are their own.""" + assert len(resolved.routed) == 1 + seen = resolved.routed[0] + native = resolved.native.quantization_config + params = json.loads((resolved.path / "params.json").read_text()) + assert native == dict(params["quantization_config"], kv_cache_quant_algo="FP8") + assert seen.quantization_config == native + assert resolved.model_config.pretrained_config.quantization_config == native + assert seen.pretrained_config is resolved.model_config.pretrained_config + assert resolved.model_config.modeling_v2_target is _Routed + assert seen.compute_capability == _SM103 and seen.spec_config is None + mapping = seen.mapping + assert (mapping.world_size, mapping.tp_size, mapping.pp_size) == (4, 4, 1) + assert (mapping.moe_ep_size, mapping.moe_tp_size, mapping.enable_attention_dp) == (4, 1, False) + + +def test_router_quantization_map_of_the_nvfp4_export(): + """The map the NVFP4 branch (`_nvfp4_mixed_precision`, models/mistral_large4/routing.py) + compares as data: 475 Mistral module names, NVFP4 group 16 for the routed experts of layers + 1..71, FP8_PB_WO for wq, wo, the 44 full indexers' wq_b, the shared experts and layer 0's + dense MLP, nothing else.""" + quant = json.loads((_checkpoint_dir(_NVFP4_NAMES) / "params.json").read_text())[ + "quantization_config" + ] + full = [i < 17 or i % 4 in (0, 3) for i in range(_LAYERS)] + fp8 = {"quant_algo": "FP8_PB_WO"} + expected = { + f"layers.{i}.experts": {"quant_algo": "NVFP4", "group_size": 16} for i in range(1, _LAYERS) + } + for i in range(_LAYERS): + expected[f"layers.{i}.attention.wq"] = fp8 + expected[f"layers.{i}.attention.wo"] = fp8 + if full[i]: + expected[f"layers.{i}.attention.indexer.wq_b"] = fp8 + mlp = "feed_forward" if i == 0 else "shared_experts" + for w in ("w1", "w2", "w3"): + expected[f"layers.{i}.{mlp}.{w}"] = fp8 + assert {k: v for k, v in quant.items() if k != "quantized_layers"} == { + "quant_method": "modelopt", + "quant_algo": "MIXED_PRECISION", + "group_size": 16, + "ignore": [], + } + assert quant["quantized_layers"] == expected + assert len(expected) == 475 + + +def test_tep4_branch_claims_the_checkpoint_at_both_envelopes(envelope, delivered): + """The routing (`route`, models/mistral_large4/routing.py), on what each engine hands it: the + NVFP4 tep4 branch claims the NVFP4 export, at the 1M config as delivered -- no criterion + reads max_seq_len or chunked prefill. The quantization map, read off the `pretrained_config` + the router is handed, where it survives intact, is the one the branch compares.""" + for resolved in (envelope, delivered): + seen = resolved.routed[0] + target = _route(seen.pretrained_config, seen.mapping) + assert target.__name__ == _NVFP4.target + assert routing._nvfp4_mixed_precision(seen.pretrained_config) + + +def test_other_members_resolve_and_route_as_at_base(): + """The members the 1M config does not move, at their own base options: the NVFP4 checkpoint + at dep4 (attention DP) and at tp4 (the experts cut along their width), each 32768 tokens with + chunked prefill off. Through the same engine path (`_load_config_and_create_checkpoint_ + loader`, `ModelLoader._load_and_validate_config`) each is claimed by its own target (`route`, + models/mistral_large4/routing.py), keyed on the checkpoint's anchor, on the same fp8_ds_mla + pool, 47136 bytes a token (`MistralLarge4CacheManager.get_cache_size_per_token`), the DSA + config unchanged.""" + nvfp4 = _checkpoint_dir(_NVFP4_NAMES) + dep4 = _resolve(nvfp4, _options(32768, 32768, False, _DEP4)) + tp4 = _resolve(nvfp4, _options(32768, 32768, False, _TP4)) + cases = ( + (dep4, "ModelingV2MistralLarge4Nvfp4Sm103Dep", (4, 1, True)), + (tp4, "ModelingV2MistralLarge4Nvfp4Sm103Tp", (1, 4, False)), + ) + for resolved, target, (moe_ep, moe_tp, adp) in cases: + seen = resolved.routed[0] + mapping = seen.mapping + assert (mapping.world_size, mapping.tp_size, mapping.pp_size) == (4, 4, 1) + assert (mapping.moe_ep_size, mapping.moe_tp_size, mapping.enable_attention_dp) == ( + moe_ep, + moe_tp, + adp, + ) + assert _route(seen.pretrained_config, mapping).__name__ == target + assert _anchor(resolved.llm_args) == 91.926 + assert (resolved.max_num_tokens, resolved.max_seq_len, resolved.max_batch_size) == ( + 32768, + 32768, + 16, + ) + assert resolved.llm_args.enable_chunked_prefill is False + manager = get_kv_cache_manager_cls(resolved.model_config, resolved.llm_args.kv_cache_config) + assert manager is MistralLarge4CacheManager + assert (_kv_cost(resolved).slope, _kv_cost(resolved).intercept) == (47136, 0) + sparse = resolved.model_config.sparse_attention_config + assert (sparse.algorithm, sparse.q_split_threshold, sparse.index_topk) == ( + "dsa", + 8192, + 2048, + ) + + +# ------------------------------------------------------- DSA, rope, positions, params.json + + +def test_dsa_rope_and_positions_under_the_envelope(resolved): + """The native config's normalised DSA keys, its YaRN parameters and its maximum position, + adapted from params.json, are read without error under each gate's bounds; the engine keeps + max_seq_len and max_num_tokens as configured, chunked prefill on at the 1M config and off at + the delivered one, and max_seq_len is within the inferred limit, the + checkpoint's maximum position (`DecoderModelForCausalLM.infer_max_seq_len`, + _torch/models/modeling_utils.py: YaRN does not scale it). At the 1M config max_seq_len is + that limit.""" + case, model_config, llm_args = resolved.case, resolved.model_config, resolved.llm_args + cfg = model_config.pretrained_config + assert (cfg.index_topk, cfg.index_n_heads, cfg.index_head_dim) == (2048, 32, 128) + assert cfg.indexer_norm_eps == 1e-06 and cfg.indexer_ssmax_scaling is True + assert ( + cfg.indexer_nope_layer_flags == cfg.nope_layer_flags == [i % 4 == 3 for i in range(_LAYERS)] + ) + assert cfg.index_topk_pattern == "".join( + "F" if i < 17 or i % 4 in (0, 3) else "S" for i in range(_LAYERS) + ) + assert cfg.rope_parameters == { + "rope_type": "yarn", + "factor": case.rope_factor, + "original_max_position_embeddings": 512, + "beta_fast": 32.0, + "beta_slow": 1.0, + "attention_factor": 1.0, + "mscale_all_dim": 1, + "rope_theta": 10000.0, + } + assert cfg.max_position_embeddings == case.max_position_embeddings + + sparse = model_config.sparse_attention_config + assert type(sparse) is DeepSeekSparseAttentionConfig + assert (sparse.index_n_heads, sparse.index_head_dim, sparse.index_topk) == (32, 128, 2048) + assert sparse.indexer_rope_interleave is True and sparse.indexer_k_dtype == "fp8" + + assert llm_args.enable_chunked_prefill is case.chunked + assert (resolved.max_num_tokens, resolved.max_seq_len, resolved.max_batch_size) == ( + case.max_num_tokens, + case.max_seq_len, + 16, + ) + assert (model_config.max_num_tokens, model_config.max_seq_len) == ( + case.max_num_tokens, + case.max_seq_len, + ) + inferred = DecoderModelForCausalLM.infer_max_seq_len(SimpleNamespace(config=cfg)) + assert inferred == case.max_position_embeddings >= resolved.max_seq_len + if case is _NVFP4: + assert resolved.max_seq_len == inferred == 1048576 + + +def test_indexer_chunking_under_each_envelope(envelope, delivered): + """`Indexer.prepare_for_chunked_prefill` (dsa/indexer.py) on the largest step of each engine. + Delivered (32768 unchunked, no cached KV) it takes the indexer's own chunking: + `indexer_max_chunk_size` resolves to its 32768 default, `select_indexer_chunk_size` keeps it + below 256K keys, and a whole 32768-token prompt is one chunk, so no chunk list is built. At + the 1M config the MLA-chunked-prefill branch plans the chunks instead (by request and packed + logits, see the q-split tests); `indexer_max_chunk_size` is not read.""" + params = _metadata_params(delivered) + assert params.indexer_max_chunk_size == 32768 >= delivered.max_num_tokens + assert select_indexer_chunk_size(params.indexer_max_chunk_size, 32768) == 32768 + groups = split_prefill_chunks(torch.tensor([32768], dtype=torch.int32), 32768) + assert groups == [[(0, 0, 32768, 0)]] + step = _indexer_step(delivered, 0, ctx=[(0, 32768)]) + assert step.indexer_prefill_chunks is None and step.indexer_prefill_q_split is None + assert step.slot_mapping_fp8_fullkv is step.slot_mapping_fp8 + + step = _indexer_step(envelope, 0, ctx=[(0, 32768)]) + assert step.indexer_prefill_q_split == (0, 8192, [8192] * 4) + assert [(c.token_start, c.token_end) for c in step.indexer_prefill_chunks] == [(0, 8192)] + + +def test_config_resolution_reads_params_json_only(resolved, tmp_path): + """Nothing on this path (the front end's `_update_from_hf_quant_config`, then + `_load_config_and_create_checkpoint_loader`, py_executor_creator.py, and + `ModelLoader._load_and_validate_config`) reads any file of the checkpoint beside params.json: + a directory holding params.json alone -- no tekken.json, no weights or their index -- + resolves to the same configuration, at each config. Its `dsa` block is parsed by the native + loader alone (`_adapt_mistral_large4_config`).""" + (tmp_path / "params.json").symlink_to(resolved.path / "params.json") + alone = _resolve(tmp_path, resolved.case.options) + + def summary(r): + cfg = r.model_config.pretrained_config.to_dict() + cfg.pop("_name_or_path", None) + return dict( + llm_quant=repr(r.llm_args.quant_config), + quant=repr(r.model_config.quant_config), + layers={k: repr(v) for k, v in (r.model_config.quant_config_dict or {}).items()}, + pretrained=cfg, + sparse=r.model_config.sparse_attention_config.model_dump(), + routed=r.routed[0].quantization_config, + sizes=(r.max_num_tokens, r.max_seq_len, r.max_batch_size), + chunked=r.llm_args.enable_chunked_prefill, + ) + + assert summary(alone) == summary(resolved) + + +def test_params_json_dsa_block_is_the_engines_config(resolved): + """params.json's `dsa` block is what the engine's config carries normalised: the native + loader (`_adapt_mistral_large4_config`, models/checkpoints/mistral/config_loader.py) reads it + into the keys the indexer, its sharing pattern and the latent cache are built from, and the + DSA sparse config the forward reads takes the indexer's from them.""" + params = json.loads((resolved.path / "params.json").read_text()) + dsa, cfg = params["dsa"], resolved.model_config.pretrained_config + assert dsa["topk"] == cfg.index_topk + assert dsa["n_heads"] == cfg.index_n_heads + assert dsa["head_dim"] == cfg.index_head_dim + assert dsa["rope_head_dim"] == cfg.qk_rope_head_dim + assert dsa["norm_eps"] == cfg.indexer_norm_eps + assert dsa["ssmax_scaling"] == cfg.indexer_ssmax_scaling + assert dsa["match_nope"] is True + assert cfg.indexer_nope_layer_flags == cfg.nope_layer_flags + assert dsa["sharing"].split(",") == cfg.index_sharing + assert dsa["fp8_kv_cache"] is cfg.fp8_kv_cache is True + sparse = resolved.model_config.sparse_attention_config + assert (sparse.index_topk, sparse.index_n_heads, sparse.index_head_dim) == ( + dsa["topk"], + dsa["n_heads"], + dsa["head_dim"], + ) + + +# ------------------------------------------------------- tep8: eight ranks, two nodes of four + +_TEP8_RANKS = 8 +# Devices per GB300 node on this cluster: two nodes hold the eight ranks. +_GPUS_PER_NODE = 4 +# The delivered config's max_seq_len (32768 tokens, one step per prompt). +_DELIVERED_MAX_SEQ_LEN = 32768 +# The KV owners: the layers whose kv_source_layers entry is themselves, every layer but the +# nine NoPE sharers 39, 43, ..., 71 (layer 35's latent is theirs). +_KV_OWNERS = 63 +# The per-rank KV pools measured at tep8 on the fp8_ds_mla cache, with the 64-head FlashMLA +# query in the profiled step and the native checkpoint's vision tower on every rank: at 32768 +# at the accuracy gate's KV fraction (core.yaml's, 0.85), 106,325 pages; at the 1M config at +# the boot gate's (the quickstart's, 0.9), 111,923 pages, its load-time rope tables and the +# held context workspace already out of it. +_TEP8_ACCURACY_POOL_TOKENS = 106_325 * 32 +_TEP8_BOOT_POOL_TOKENS = 111_923 * 32 + +# The topologies beside tep4 and dep4 (above). tep8 is the core unit's (spec-1m-tep8/options/ +# core.yaml at 1M, spec-tep8/options/core.yaml delivered); tensor_parallel_size 8 alone is what +# a command stating no MoE split builds. +_TEP8 = dict(tensor_parallel_size=8, moe_expert_parallel_size=8, enable_attention_dp=False) +_TP8_ALONE = dict(tensor_parallel_size=8) + +# spec-1m-tep8/options/core.yaml: the 1M config, the latent KV cache spelled fp8. +_CORE = _options(_MAX_SEQ_LEN, _MAX_NUM_TOKENS, True, _TEP8, kv_dtype="fp8") +# The CI class's (TestModelingV2MistralLarge4Nvfp4Sm103Tep8): the same config, its +# KvCacheConfig without a dtype. +_CORE_CI = _options(_MAX_SEQ_LEN, _MAX_NUM_TOKENS, True, _TEP8) +# spec-tep8/options/core.yaml: the delivered config, 32768 unchunked, the latent KV cache at +# its default. +_CORE_32K = _options(_DELIVERED_MAX_SEQ_LEN, _MAX_NUM_TOKENS, False, _TEP8) + +# The boot gate's quickstart command after --model_dir: the checkpoint's Mistral-native format, +# the tep8 topology flags, the core unit's CUDA-graph capture list, chunked prefill on and block +# reuse off (task.yaml's runtime block), then task.yaml's extra flags. +_GATE_ARGV = ( + "--checkpoint_format=mistral", + "--max_tokens=4", + "--tp_size=8", + "--moe_ep_size=8", + "--use_cuda_graph", + "--cuda_graph_batch_sizes", + "1", + "2", + "4", + "8", + "16", + "--enable_chunked_prefill", + "--disable_kv_cache_reuse", + "--max_seq_len=1048576", + "--max_batch_size=16", + "--max_num_tokens=32768", + "--cuda_graph_padding_enabled", + "--kv_cache_dtype=fp8", +) +# The delivered boot gate's: 32768 tokens, chunked prefill off. No --kv_cache_dtype: the latent +# cache is the checkpoint's default (quickstart's "auto"), the fp8_ds_mla pool. +_GATE_ARGV_32K = ( + "--checkpoint_format=mistral", + "--max_tokens=4", + "--tp_size=8", + "--moe_ep_size=8", + "--use_cuda_graph", + "--cuda_graph_batch_sizes", + "1", + "2", + "4", + "8", + "16", + "--disable_kv_cache_reuse", + "--max_seq_len=32768", + "--max_batch_size=16", + "--max_num_tokens=32768", + "--cuda_graph_padding_enabled", +) +_QUICKSTART = Path(__file__).resolve().parents[7] / "examples/llm-api/quickstart_advanced.py" + +# The NVFP4 target that serves tep4 and tep8 alike. +_NVFP4_TEP4_TARGET = "ModelingV2MistralLarge4Nvfp4Sm103Tep" + + +def _llm_args(devices: int = _GPUS_PER_NODE, **kwargs) -> TorchLlmArgs: + """``TorchLlmArgs`` as the LLM constructor builds it on rank 0 of a node with ``devices`` + visible devices (outside Ray).""" + with pytest.MonkeyPatch.context() as mp: + mp.delenv("RAY_LOCAL_WORLD_SIZE", raising=False) + mp.setattr(torch.cuda, "device_count", lambda: devices) + return TorchLlmArgs(**kwargs) + + +def _gate_kwargs(checkpoint_dir: Path, argv: tuple[str, ...]) -> dict: + """The ``LLM`` keyword arguments of a quickstart gate command: the tree's own + quickstart_advanced.py parses ``argv`` and builds them in ``setup_llm``, and the gate harness + adds ``modeling_v2``. Every KV-cache key of core.yaml has a quickstart flag, so the harness + hands over no other.""" + if not _QUICKSTART.is_file(): + pytest.skip(f"{_QUICKSTART} is not in this checkout") + spec = importlib.util.spec_from_file_location("quickstart_advanced", _QUICKSTART) + quickstart = importlib.util.module_from_spec(spec) + spec.loader.exec_module(quickstart) + built = {} + + def llm(**kwargs): + built.update(kwargs) + return SimpleNamespace() + + with pytest.MonkeyPatch.context() as mp: + mp.setattr(sys, "argv", ["quickstart_advanced.py", f"--model_dir={checkpoint_dir}", *argv]) + args = quickstart.parse_arguments() + mp.setattr(quickstart, "LLM", llm) + quickstart.setup_llm(args) + return dict(built, modeling_v2="require") + + +def _resolve_options(names: tuple[str, ...], options: dict) -> SimpleNamespace: + """The configuration a served engine reaches for the checkpoint under ``names`` from the + LLM-API ``options``, its arguments built as on rank 0 of a node of four.""" + path = _checkpoint_dir(names) + return _resolve_args(path, _llm_args(model=str(path), modeling_v2="require", **options)) + + +@pytest.fixture(scope="module") +def tep8(): + """The accuracy gate's engine at the 1M config: spec-1m-tep8/options/core.yaml.""" + return _resolve_options(_NVFP4_NAMES, _CORE) + + +@pytest.fixture(scope="module") +def tep8_gate(): + """The boot gate's engine at the 1M config: its quickstart command.""" + path = _checkpoint_dir(_NVFP4_NAMES) + return _resolve_args(path, _llm_args(**_gate_kwargs(path, _GATE_ARGV))) + + +@pytest.fixture(scope="module") +def tep8_ci(): + """The CI class's engine at the 1M config: core.yaml's options with the KV dtype left at its + default.""" + return _resolve_options(_NVFP4_NAMES, _CORE_CI) + + +@pytest.fixture(scope="module") +def tep8_32k(): + """The delivered accuracy gate's engine: spec-tep8/options/core.yaml, 32768 unchunked.""" + return _resolve_options(_NVFP4_NAMES, _CORE_32K) + + +@pytest.fixture(scope="module") +def tep8_32k_gate(): + """The delivered boot gate's engine: its quickstart command at 32768, unchunked.""" + path = _checkpoint_dir(_NVFP4_NAMES) + return _resolve_args(path, _llm_args(**_gate_kwargs(path, _GATE_ARGV_32K))) + + +# Each tep8 engine: the fixture, its max_seq_len, whether chunked prefill is on, and the latent +# KV dtype it states (fp8 at 1M; the delivered gates leave it at its default). +_DEPLOYMENTS = { + "1m-core-yaml": ("tep8", _MAX_SEQ_LEN, True, "fp8"), + "1m-quickstart": ("tep8_gate", _MAX_SEQ_LEN, True, "fp8"), + "32k-core-yaml": ("tep8_32k", _DELIVERED_MAX_SEQ_LEN, False, "auto"), + "32k-quickstart": ("tep8_32k_gate", _DELIVERED_MAX_SEQ_LEN, False, "auto"), +} + + +@pytest.fixture(scope="module", params=list(_DEPLOYMENTS)) +def deployment(request): + """One tep8 engine -- the 1M config or the delivered one, from core.yaml or from the boot + gate's command -- with the max_seq_len, the chunked prefill and the KV dtype it is configured + with.""" + name, max_seq_len, chunked, kv_dtype = _DEPLOYMENTS[request.param] + return request.getfixturevalue(name), max_seq_len, chunked, kv_dtype + + +@pytest.fixture( + scope="module", params=["1m-options", "1m-quickstart", "32k-options", "32k-quickstart"] +) +def tp8_alone(request): + """tensor_parallel_size 8 and no MoE split, as LLM-API options and as ``--tp_size=8``, at + each config; with the tep8 engine of the same config.""" + path = _checkpoint_dir(_NVFP4_NAMES) + size, spelling = request.param.split("-") + wide = size == "1m" + reference = request.getfixturevalue("tep8" if wide else "tep8_32k") + if spelling == "options": + if wide: + options = _options(_MAX_SEQ_LEN, _MAX_NUM_TOKENS, True, _TP8_ALONE, kv_dtype="fp8") + else: + options = _options(_DELIVERED_MAX_SEQ_LEN, _MAX_NUM_TOKENS, False, _TP8_ALONE) + return _resolve_options(_NVFP4_NAMES, options), reference + argv = _GATE_ARGV if wide else _GATE_ARGV_32K + argv = tuple(word for word in argv if word != "--moe_ep_size=8") + return _resolve_args(path, _llm_args(**_gate_kwargs(path, argv))), reference + + +@pytest.fixture(scope="module") +def tep4(): + """The NVFP4 checkpoint at tep4 on the delivered config, built as the tep8 arguments are.""" + return _resolve_options(_NVFP4_NAMES, _options(_DELIVERED_MAX_SEQ_LEN, _MAX_NUM_TOKENS, False)) + + +@pytest.fixture(scope="module") +def tep4_1m(): + """The NVFP4 checkpoint at tep4 on the 1M config's gate options (spec-1m-kvds-tep4/options/ + core.yaml, the latent KV cache spelled fp8), built as the tep8 arguments are: the + configuration tep8's is compared against.""" + return _resolve_options( + _NVFP4_NAMES, _options(_MAX_SEQ_LEN, _MAX_NUM_TOKENS, True, kv_dtype="fp8") + ) + + +@pytest.fixture(scope="module") +def dep4(): + return _resolve_options( + _NVFP4_NAMES, _options(_DELIVERED_MAX_SEQ_LEN, _MAX_NUM_TOKENS, False, _DEP4) + ) + + +def _summary(resolved) -> dict: + """Everything the engine resolved for a checkpoint except its mapping.""" + pretrained = resolved.model_config.pretrained_config.to_dict() + pretrained.pop("_name_or_path", None) + llm_args, model_config = resolved.llm_args, resolved.model_config + return dict( + llm_quant=repr(llm_args.quant_config), + quant=repr(model_config.quant_config), + layers={k: repr(v) for k, v in (model_config.quant_config_dict or {}).items()}, + pretrained=pretrained, + sparse=model_config.sparse_attention_config.model_dump(), + routed=resolved.routed[0].quantization_config, + sizes=(resolved.max_num_tokens, resolved.max_seq_len, resolved.max_batch_size), + chunked=llm_args.enable_chunked_prefill, + cuda_graphs=llm_args.cuda_graph_config.model_dump(), + kv_cache=llm_args.kv_cache_config.model_dump(), + moe=llm_args.moe_config.model_dump(), + moe_backend=model_config.moe_backend, + moe_max_num_tokens=model_config.moe_max_num_tokens, + ) + + +def _route_or_none(seen): + try: + return _route(seen.pretrained_config, seen.mapping) + except NoTarget: + return None + + +def _pages(tokens: int) -> int: + """The pages the V2 manager holds for a request of ``tokens`` tokens: its one token of + generation headroom included (`KVCacheManagerV2.__init__`).""" + return math.ceil((tokens + 1) / _TOKENS_PER_BLOCK) + + +# ------------------------------------------------------- eight ranks in one process + + +class _Group: + """The eight ranks in one process: rank r is a thread, and every exchange is a barrier, so + each rank's call sees what every rank sent, as over MPI.""" + + def __init__(self, size: int = _TEP8_RANKS): + self.size = size + self._barrier = threading.Barrier(size, timeout=60) + self._slots = [None] * size + + def exchange(self, rank: int, obj): + self._slots[rank] = obj + self._barrier.wait() + gathered = list(self._slots) + self._barrier.wait() + return gathered + + def run(self, step): + """``step(rank)`` on every rank at once; the per-rank results in rank order.""" + results, errors = [None] * self.size, [None] * self.size + + def body(rank): + try: + results[rank] = step(rank) + except BaseException as error: # noqa: BLE001 - re-raised below, after every rank + errors[rank] = error + self._barrier.abort() + + threads = [threading.Thread(target=body, args=(rank,)) for rank in range(self.size)] + for thread in threads: + thread.start() + for thread in threads: + thread.join() + # The rank that failed first, not a peer it left waiting at the barrier. + raised = [error for error in errors if error is not None] + for error in raised: + if not isinstance(error, threading.BrokenBarrierError): + raise error + if raised: + raise raised[0] + return results + + +class _RankDist: + """One rank's communicator: the world gather the warmup agrees its shapes over.""" + + def __init__(self, group: _Group, rank: int): + self.group = group + self.rank = self.tp_rank = rank + self.world_size = self.tp_size = group.size + + def allgather(self, obj): + return self.group.exchange(self.rank, obj) + + +# ------------------------------------------------------- tep8 (1): topology, node layout, limits + + +def test_tep8_topology_and_limits(deployment): + """Eight ranks, tensor parallel attention over all eight with no attention DP (8 of the 64 + heads per rank), the routed experts expert-parallel eight ways and not split along their + width, one pipeline stage -- from core.yaml and from the boot gate's command alike, at the + 1M config and at the delivered one. Rank r is TP rank r and MoE EP rank r, MoE TP rank 0: + its expert window is [16 r, 16 r + 16). One scheduler drives the eight ranks: 16 requests and + 32768 tokens per step in all (`TorchLlmArgs.get_runtime_sizes`, llmapi/llm_args.py), up to + max_seq_len tokens a request: 1048576 in chunks, or 32768 unchunked as delivered.""" + resolved, max_seq_len, chunked, _ = deployment + llm_args, mapping = resolved.llm_args, resolved.mapping + parallel = llm_args.parallel_config + assert (parallel.tp_size, parallel.pp_size, parallel.cp_size) == (8, 1, 1) + # core.yaml and the command state the EP split only; the MoE TP size is left to Mapping. + moe_split = (parallel.moe_ep_size, parallel.moe_tp_size, parallel.moe_cluster_size) + assert moe_split == (8, -1, -1) + assert llm_args.enable_attention_dp is False and parallel.enable_attention_dp is False + + assert (mapping.world_size, mapping.tp_size, mapping.pp_size) == (8, 8, 1) + assert mapping.cp_size == 1 + assert (mapping.moe_ep_size, mapping.moe_tp_size, mapping.moe_cluster_size) == (8, 1, 1) + assert mapping.enable_attention_dp is False and mapping.dp_size == 1 + assert mapping.attn_tp_size == 8 and mapping.enable_lm_head_tp_in_adp is False + assert mapping.has_moe_ep() and not mapping.has_moe_tp() + assert resolved.model_config.mapping is mapping + + cfg = resolved.model_config.pretrained_config + heads = cfg.num_attention_heads // mapping.tp_size + local_experts = cfg.n_routed_experts // mapping.moe_ep_size + assert (cfg.num_attention_heads, cfg.n_routed_experts) == (64, 128) + assert (heads, local_experts) == (8, 16) + for rank in range(_TEP8_RANKS): + rank_mapping = _rank_mapping(mapping, rank) + assert ( + rank_mapping.tp_rank, + rank_mapping.moe_ep_rank, + rank_mapping.moe_tp_rank, + rank_mapping.moe_cluster_rank, + ) == (rank, rank, 0, 0) + assert rank_mapping.ep_experts(cfg.n_routed_experts) == list( + range(16 * rank, 16 * rank + 16) + ) + assert local_experts * rank_mapping.moe_ep_rank == 16 * rank + assert rank_mapping.tp_group == list(range(_TEP8_RANKS)) + + assert (resolved.max_batch_size, resolved.max_num_tokens, resolved.max_seq_len) == ( + 16, + 32768, + max_seq_len, + ) + assert llm_args.get_runtime_sizes() == (1, 32768, max_seq_len, 16) + model_config = resolved.model_config + assert (model_config.max_num_tokens, model_config.max_seq_len) == (32768, max_seq_len) + assert llm_args.enable_chunked_prefill is chunked + + +def test_gpus_per_node_is_the_visible_device_count(deployment): + """Neither gate states gpus_per_node. ``TorchLlmArgs.gpus_per_node`` defaults to None and its + validator (outside Ray) replaces None with ``torch.cuda.device_count()`` of the process that + builds the arguments, rank 0; each node's ranks see that node's four held devices, so it is 4. + The mapping every rank builds from ``parallel_config`` carries it: rank r is local rank r % 4 + on node r // 4, and the deployment is multi-node. The LLM constructor's device check reads it + too: world_size_per_node is 4, which four visible devices satisfy. + + The defaults of ``_ParallelConfig`` and ``Mapping`` are 8, not the device count: an + eight-rank mapping built without it reads one node of eight -- local rank r, never multi-node + -- and would demand eight devices on rank 0's node.""" + resolved = deployment[0] + field = TorchLlmArgs.model_fields["gpus_per_node"] + assert field.default is None and field.validate_default is True + for options in (_CORE, _CORE_32K): + assert "gpus_per_node" not in options + for argv in (_GATE_ARGV, _GATE_ARGV_32K): + assert "gpus_per_node" not in _gate_kwargs(resolved.path, argv) + + llm_args, mapping = resolved.llm_args, resolved.mapping + assert llm_args.gpus_per_node == _GPUS_PER_NODE == 4 + assert llm_args.parallel_config.gpus_per_node == 4 + assert mapping.gpus_per_node == 4 and resolved.model_config.mapping.gpus_per_node == 4 + assert llm_args.parallel_config.world_size_per_node == 4 + for rank in range(_TEP8_RANKS): + rank_mapping = _rank_mapping(mapping, rank) + assert (rank_mapping.local_rank, rank_mapping.node_rank) == (rank % 4, rank // 4) + assert rank_mapping.get_node_rank(rank) == rank // 4 + assert rank_mapping.is_multi_node() is True + + # The count is the visible devices' count, read when the arguments are built. + path = str(resolved.path) + for devices in (4, 8): + built = _llm_args(devices, model=path, **_CORE) + assert built.gpus_per_node == devices + assert built.parallel_config.world_size_per_node == devices + stated = _llm_args(1, model=path, gpus_per_node=4, **_CORE) + assert stated.gpus_per_node == 4 + + assert _ParallelConfig.model_fields["gpus_per_node"].default == 8 + assert _ParallelConfig(tp_size=8, moe_ep_size=8).world_size_per_node == 8 + unstated = Mapping(world_size=8, rank=5, tp_size=8, moe_ep_size=8) + assert unstated.gpus_per_node == 8 + assert (unstated.local_rank, unstated.node_rank, unstated.is_multi_node()) == (5, 0, False) + + +def test_each_rank_selects_its_own_nodes_device(tep8): + """The device each worker sets before building its executor (``BaseWorker._configure_device``, + executor/base_worker.py): its global MPI rank modulo the devices its node shows, so rank r + drives device r % 4 of its node -- the mapping's local rank. Each rank checks that its device's + architecture is supported and pins its CPU affinity to it.""" + group = _Group() + local = threading.local() + with pytest.MonkeyPatch.context() as mp: + mp.setattr(torch.cuda, "device_count", lambda: _GPUS_PER_NODE) + mp.setattr(torch.cuda, "set_device", lambda device: setattr(local, "device", device)) + mp.setattr( + base_worker, + "check_cuda_architecture_supported", + lambda device: setattr(local, "checked", device), + ) + mp.setattr( + base_worker, "configure_cpu_affinity", lambda device: setattr(local, "cpus", device) + ) + + def step(rank): + BaseWorker._configure_device(SimpleNamespace(global_rank=rank)) + return local.checked, local.device, local.cpus + + results = group.run(step) + for rank, (checked, device, cpus) in enumerate(results): + assert checked == device == cpus == rank % 4 + assert device == _rank_mapping(tep8.mapping, rank).local_rank + + +def test_tp_size_8_alone_is_not_this_deployment(tp8_alone): + """tensor_parallel_size 8 with no MoE split -- as LLM-API options or as ``--tp_size=8``, + whose MoE sizes the quickstart passes as -1 -- gives the MoE the whole TP width: the routed + experts tensor parallel eight ways (MoE TP rank r) and not expert-parallel, every rank's + window all 128 experts. Everything else resolves as at tep8, at either config. Neither the + tep4 branch, at either size, nor the dep4 branch of the routing claims it.""" + alone, reference = tp8_alone + mapping = alone.mapping + assert (mapping.world_size, mapping.tp_size, mapping.moe_tp_size, mapping.moe_ep_size) == ( + 8, + 8, + 8, + 1, + ) + assert mapping.enable_attention_dp is False and mapping.has_moe_tp() + for rank in range(_TEP8_RANKS): + rank_mapping = _rank_mapping(mapping, rank) + assert (rank_mapping.moe_tp_rank, rank_mapping.moe_ep_rank) == (rank, 0) + assert rank_mapping.ep_experts(128) == list(range(128)) + differs = {k for k, v in mapping.to_dict().items() if v != reference.mapping.to_dict()[k]} + assert differs == {"moe_tp_size", "moe_ep_size"} + assert _summary(alone)["sizes"] == _summary(reference)["sizes"] + assert _summary(alone)["chunked"] == _summary(reference)["chunked"] + assert len(alone.routed) == 1 + target = _route_or_none(alone.routed[0]) + assert target is None or target.__name__ not in ( + _NVFP4_TEP4_TARGET, + "ModelingV2MistralLarge4Nvfp4Sm103Dep", + ) + + +@pytest.mark.parametrize("tep8_envelope", ["1m", "32k"]) +def test_boot_command_and_core_yaml_build_one_deployment(request, tep8_envelope): + """The boot gate's quickstart command and core.yaml resolve to the same mapping on every rank + and the same model, quantization, DSA, limits, chunked prefill, CUDA graphs, KV dtype (fp8 at + the 1M config, the default auto as delivered), page size, block-reuse setting and KV + manager; the router is handed equal mappings -- at the 1M config as at the delivered one. + The KV fraction is not shared: the quickstart's --kv_cache_fraction default, 0.9, against + core.yaml's 0.85, so the boot gate's pool is sized at 0.9 of the free memory and the + accuracy gate's at 0.85.""" + wide = tep8_envelope == "1m" + suffix = "" if wide else "_32k" + core = request.getfixturevalue(f"tep8{suffix}") + gate = request.getfixturevalue(f"tep8{suffix}_gate") + assert gate.mapping.to_dict() == core.mapping.to_dict() + assert gate.routed[0].mapping.to_dict() == core.routed[0].mapping.to_dict() + gate_summary, core_summary = _summary(gate), _summary(core) + shared = ("llm_quant", "quant", "layers", "pretrained", "sparse", "routed", "sizes", "chunked") + for key in shared: + assert gate_summary[key] == core_summary[key], key + assert gate_summary["moe_max_num_tokens"] == core_summary["moe_max_num_tokens"] == 32768 + kv_dtype = "fp8" if wide else "auto" + for resolved in (core, gate): + graphs = resolved.llm_args.cuda_graph_config + assert graphs.batch_sizes == [1, 2, 4, 8, 16] and graphs.enable_padding is True + kv = resolved.llm_args.kv_cache_config + assert (kv.dtype, kv.enable_block_reuse, kv.tokens_per_block) == (kv_dtype, False, 32) + assert kv.use_kv_cache_manager_v2 is True + assert resolved.llm_args.disable_overlap_scheduler is False + assert core.llm_args.kv_cache_config.free_gpu_memory_fraction == 0.85 + assert gate.llm_args.kv_cache_config.free_gpu_memory_fraction == 0.9 + + +def test_tep8_envelope_resolves_as_configured(tep8, tep8_gate, tep8_32k): + """`TorchLlmArgs` (llmapi/llm_args.py) keeps the 1M config as given at tep8: max_seq_len + 1048576, max_num_tokens 32768 (the chunk), max_batch_size 16; chunked prefill accepted with the + DSA sparse attention the checkpoint configures, the latent KV cache spelled fp8 and block reuse + off, on MistralLarge4CacheManager, the V2 KV manager the sparse registry picks for a + MistralLarge4Config (`get_kv_cache_manager_cls`, pyexecutor/_util.py). The DSA config is the + checkpoint's, with its `q_split_threshold` default. `_create_py_executor` + (py_executor_creator.py) keeps chunked prefill for MLA on sm_103 + (`_MLA_CHUNKED_PREFILL_SUPPORTED_SM_VERSIONS`). Beside it the delivered config: 32768 tokens, + one step per prompt, the KV dtype at its default.""" + for resolved, max_seq_len, chunked, kv_dtype in ( + (tep8, _MAX_SEQ_LEN, True, "fp8"), + (tep8_gate, _MAX_SEQ_LEN, True, "fp8"), + (tep8_32k, _DELIVERED_MAX_SEQ_LEN, False, "auto"), + ): + llm_args = resolved.llm_args + assert (llm_args.max_seq_len, llm_args.max_num_tokens) == (max_seq_len, _MAX_NUM_TOKENS) + assert llm_args.enable_chunked_prefill is chunked + assert llm_args.disable_overlap_scheduler is False + + kv_cache_config = llm_args.kv_cache_config + assert kv_cache_config.dtype == kv_dtype and kv_cache_config.enable_block_reuse is False + assert kv_cache_config.tokens_per_block == _TOKENS_PER_BLOCK + assert kv_cache_config.use_kv_cache_manager_v2 is True + assert kv_cache_config.max_attention_window is None + manager = get_kv_cache_manager_cls(resolved.model_config, kv_cache_config) + assert manager is MistralLarge4CacheManager and issubclass(manager, KVCacheManagerV2) + + assert llm_args.prefill_capture_num_tokens is None + assert llm_args.torch_compile_config is None + + sparse = resolved.model_config.sparse_attention_config + assert type(sparse) is DeepSeekSparseAttentionConfig and sparse.algorithm == "dsa" + assert sparse.q_split_threshold == 8192 + assert sparse.skip_indexer_for_short_seqs is True + assert sparse.use_cute_dsl_topk is False and sparse.enable_heuristic_topk is False + assert resolved.model_config.attn_backend == "TRTLLM" + + assert 103 in py_executor_creator._MLA_CHUNKED_PREFILL_SUPPORTED_SM_VERSIONS + + +def test_tep8_model_engine_keeps_max_seq_len(tep8, tep8_32k): + """`ModelEngine._init_model_capacity` (pyexecutor/model_engine.py) lowers a + max_seq_len above the model's inferred limit and a max_num_tokens above max_batch_size x + max_seq_len. The target's shell infers the limit with + `DecoderModelForCausalLM.infer_max_seq_len` (_torch/models/modeling_utils.py): YaRN + (512 x 2048) does not scale it, so it is max_position_embeddings, 1048576, and the 1M + config's max_seq_len is kept whole at tep8, as the delivered one is.""" + for resolved in (tep8, tep8_32k): + cfg = resolved.model_config.pretrained_config + assert cfg.rope_parameters["rope_type"] == "yarn" + assert cfg.rope_parameters["factor"] == 2048.0 + assert cfg.rope_parameters["original_max_position_embeddings"] == 512 + inferred = DecoderModelForCausalLM.infer_max_seq_len(SimpleNamespace(config=cfg)) + assert inferred == cfg.max_position_embeddings == _MAX_SEQ_LEN + + engine = SimpleNamespace( + model=SimpleNamespace(infer_max_seq_len=lambda inferred=inferred: inferred), + max_seq_len=resolved.max_seq_len, + max_num_tokens=resolved.max_num_tokens, + batch_size=resolved.max_batch_size, + ) + for name in ("_init_max_seq_len", "_init_max_num_tokens", "_infer_max_seq_len_from_config"): + setattr(engine, name, MethodType(getattr(ModelEngine, name), engine)) + with pytest.MonkeyPatch.context() as mp: + mp.delenv("TLLM_ALLOW_LONG_MAX_MODEL_LEN", raising=False) + ModelEngine._init_model_capacity(engine) + assert (engine.max_seq_len, engine.max_num_tokens) == ( + resolved.max_seq_len, + resolved.max_num_tokens, + ) + assert tep8.model_config.max_seq_len == _MAX_SEQ_LEN + assert tep8.model_config.max_num_tokens == _MAX_NUM_TOKENS + + +def test_tep8_llm_api_admits_the_envelope_prompt(tep8, tep8_gate, tep8_32k): + """`BaseLLM._check_arguments` (llmapi/llm.py) refuses a prompt longer than max_num_tokens only + without chunked prefill: the delivered tep8 engine refuses 32769 tokens, the 1M engine admits + the 1M gate's 1048559. The bound on max_seq_len is the worker's: + `BaseWorker._enqueue_request` (executor/base_worker.py) gives a request at most max_seq_len + less its prompt to generate, and refuses a prompt that leaves no room for one token.""" + for resolved in (tep8, tep8_gate): + _check_arguments(resolved, _ENVELOPE_PROMPT) + assert _enqueue(resolved, _ENVELOPE_PROMPT, 16).max_tokens == 16 + assert _enqueue(resolved, _MAX_SEQ_LEN - 1, 16).max_tokens == 1 + for prompt_len in (_MAX_SEQ_LEN, _MAX_SEQ_LEN + 1): + with pytest.raises(RequestError, match="maximum context length is 1048576 tokens"): + _enqueue(resolved, prompt_len, 16) + _check_arguments(tep8_32k, _MAX_NUM_TOKENS) + with pytest.raises(RequestError, match=r"should not exceed max_num_tokens \(32768\)"): + _check_arguments(tep8_32k, _MAX_NUM_TOKENS + 1) + + +# ------------------------------------------------------- tep8 (2): quantization, the anchor's key + + +def test_tep8_quantization_is_the_anchor_key(deployment): + """At tep8 as at tep4, at the 1M config as delivered: MIXED_PRECISION with the FP8 KV + cache, the GSM8K anchor row's key, and the router handed params.json's quantization_config + with its 475 quantized_layers intact plus the FP8 KV-cache algorithm the native loader + records from fp8_kv_cache, on the tep8 mapping -- from core.yaml and from the boot gate's + command alike, whether they spell the latent fp8 (1M) or leave it at auto (delivered).""" + resolved, _, _, kv_dtype = deployment + on_disk = json.loads((resolved.path / "params.json").read_text())["quantization_config"] + assert len(on_disk["quantized_layers"]) == 475 + routed = dict(on_disk, kv_cache_quant_algo="FP8") + assert resolved.prequantized is True + for quant in (resolved.llm_args.quant_config, resolved.model_config.quant_config): + assert quant.quant_algo == QuantAlgo.MIXED_PRECISION + assert quant.kv_cache_quant_algo == QuantAlgo.FP8 + assert quant.group_size == 16 + assert resolved.llm_args.kv_cache_config.dtype == kv_dtype + assert _anchor(resolved.llm_args) == 91.926 + + assert len(resolved.routed) == 1 + seen = resolved.routed[0] + assert seen.quantization_config == routed + assert resolved.model_config.pretrained_config.quantization_config == routed + assert seen.pretrained_config is resolved.model_config.pretrained_config + assert resolved.model_config.modeling_v2_target is _Routed + assert seen.compute_capability == _SM103 and seen.spec_config is None + assert seen.mapping is resolved.model_config.mapping + assert (seen.mapping.world_size, seen.mapping.tp_size, seen.mapping.pp_size) == (8, 8, 1) + assert (seen.mapping.moe_ep_size, seen.mapping.moe_tp_size) == (8, 1) + assert seen.mapping.enable_attention_dp is False + + +def test_tep8_kv_dtype_fp8_and_auto_build_the_fp8_ds_mla_pool(tep8, tep8_gate, tep8_ci): + """The gates spell the latent KV cache fp8 (core.yaml's kv_cache_config.dtype, the boot gate's + --kv_cache_dtype=fp8); the CI class leaves it at its default, auto. Both reach the same pool: + `validate_and_set_kv_cache_quant` (pyexecutor/model_loader.py) sets the KV quant algo FP8 under + fp8 and keeps the checkpoint's, FP8, under auto, so `_create_kv_cache_manager` hands the + manager the FP8 element type; the sparse registry picks MistralLarge4CacheManager for the + MistralLarge4Config whatever the dtype (`get_kv_cache_manager_cls`), and it accepts either + spelling over FP8 and builds the fp8_ds_mla layout -- its base over all 72 layers, one + 656-byte head on each of the 63 KV owners and none on the nine sharers, as FP8 -- 47136 bytes + a token on every rank, the same anchor row. Any other element type is refused at + construction.""" + assert tep8_ci.llm_args.kv_cache_config.dtype == "auto" + assert tep8.llm_args.kv_cache_config.dtype == tep8_gate.llm_args.kv_cache_config.dtype == "fp8" + cfg = tep8.model_config.pretrained_config + owners = [s == i for i, s in enumerate(cfg.kv_source_layers)] + assert sum(owners) == _KV_OWNERS + for resolved in (tep8, tep8_gate, tep8_ci): + model_config, kv_cache_config = resolved.model_config, resolved.llm_args.kv_cache_config + assert model_config.quant_config.kv_cache_quant_algo == QuantAlgo.FP8 + assert model_config.quant_config.quant_mode.has_fp8_kv_cache() + manager = get_kv_cache_manager_cls(model_config, kv_cache_config) + assert manager is MistralLarge4CacheManager and manager.latent_layout == "fp8_ds_mla" + base = _manager_arguments(resolved, tensorrt_llm.bindings.DataType.FP8) + assert base["kv_cache_config"].dtype == kv_cache_config.dtype + assert base["dtype"] == tensorrt_llm.bindings.DataType.FP8 + assert (base["num_layers"], base["head_dim"]) == (_LAYERS, 656) + assert base["num_kv_heads"] == [1 if owner else None for owner in owners] + assert base["layer_mask"] is None + for rank in range(_TEP8_RANKS): + assert _kv_cost(resolved, rank) == _kv_cost(tep8, 0) + assert (_kv_cost(resolved, 0).slope, _kv_cost(resolved, 0).intercept) == ( + _KV_BYTES_PER_TOKEN, + 0, + ) + assert _anchor(resolved.llm_args) == 91.926 + with pytest.raises(ValueError, match="fp8_ds_mla"): + _manager_arguments(resolved, tensorrt_llm.bindings.DataType.BF16) + assert _KV_BYTES_PER_TOKEN == 47136 + + +# ------------------------------------------------------- tep8 (3): a long prompt's context path + + +def test_tep8_chunked_prefill_hands_the_target_the_cached_kv_context_path(tep8, tep8_32k): + """`build_attention_metadata` (pyexecutor/engine/metadata.py) turns on + `enable_context_mla_with_cached_kv` for MLA under cache reuse or chunked prefill: on at the + 1M config, off at the delivered engine. With it and `runtime_features.chunked_prefill`, + `Indexer.prepare_for_chunked_prefill` (attention/backends/sparse/dsa/indexer.py) takes its + MLA-chunked-prefill branch. The metadata is sized by the engine bounds, not max_seq_len.""" + for resolved, cached_kv in ((tep8, True), (tep8_32k, False)): + made = build_attention_metadata( + resolved.model_config, + max_batch_size=resolved.max_batch_size, + max_num_tokens=resolved.max_num_tokens, + max_beam_width=1, + attention_backend=SimpleNamespace(Metadata=lambda **kwargs: SimpleNamespace(**kwargs)), + attention_runtime_features=_runtime_features(resolved), + mapping=resolved.mapping, + cache_indirection=None, + ) + assert made.enable_context_mla_with_cached_kv is cached_kv + assert made.runtime_features.chunked_prefill is cached_kv + assert made.runtime_features.cache_reuse is False + assert made.runtime_features.chunk_size == _MAX_NUM_TOKENS + assert (made.max_num_requests, made.max_num_tokens, made.max_num_sequences) == ( + 16, + 32768, + 16, + ) + assert made.sparse_metadata_params == _metadata_params(resolved) + + +def test_tep8_target_sizes_from_the_engine(monkeypatch, tep8, tep8_gate, tep8_32k): + """What the target takes from the engine on its first step at tep8, as at tep4: one + scheduler holds the 16 requests and the 32768 tokens of a step for all eight ranks, so every + rank reads the same metadata (`_target_sizing`), its runtime features the executor's: chunked + prefill as enable_chunked_prefill, block reuse as kv_cache_config.enable_block_reuse (off in + core.yaml, and by the boot gate's --disable_kv_cache_reuse). At the 1M config the logits + budget at its 2**28 ceiling and the held context workspace at its 2**19; delivered 2**24 and + 65536.""" + monkeypatch.delenv("TLLM_INDEXER_MQA_LOGITS_ELEM_BUDGET", raising=False) + for resolved, chunked, rows, budget in ( + (tep8, True, 2**19, 2**28), + (tep8_gate, True, 2**19, 2**28), + (tep8_32k, False, 2**16, 2**24), + ): + md = _target_sizing(resolved) + assert (md.max_num_requests, md.max_num_tokens) == (16, 32768) + assert md.max_seq_len == resolved.max_seq_len + features, llm_args = md.runtime_features, resolved.llm_args + assert features.chunked_prefill is llm_args.enable_chunked_prefill is chunked + assert features.cache_reuse is llm_args.kv_cache_config.enable_block_reuse is False + assert tep4_modeling._ctx_workspace_rows(md) == rows + assert tep4_modeling._prefill_logits_budget(md.max_num_tokens, md.max_seq_len) == budget + + +def _chunk_ranges(md) -> list[tuple[int, int, int, int]]: + """Each planned indexer chunk as (token_start, token_end, k_token_start, k_token_end).""" + return [ + (chunk.token_start, chunk.token_end, chunk.k_token_start, chunk.k_token_end) + for chunk in md.indexer_prefill_chunks + ] + + +@pytest.mark.parametrize("chunk", [0, 1, 2, 31], ids=["first", "second", "third", "last"]) +def test_tep8_later_chunks_carry_their_cached_prefix(tep8, chunk): + """Every chunk of one request after its first is a context request with a cached prefix. + `DSAtrtllmAttentionMetadata.prepare_for_mla_rope_append` (attention/backends/sparse/dsa/ + metadata.py) hands the target its cached and total prefix sums, `ctx_cached_token_indptr` + [0, cached] and `ctx_kv_indptr` [0, cached + new], `num_ctx_cached_tokens` = cached and + `max_ctx_seq_len` = new; `Indexer.prepare` counts `num_ctx_kv_tokens` = new keys written this + step, and `Indexer.recompute_context_kv_gather_mappings` builds `slot_mapping_fp8_fullkv` / + `slot_mapping_scale_fullkv` over the cached and the new keys, position p at page p // 32 of + the request's indexer page table. On a first chunk there is nothing cached and the full + mappings are the step's own. Every TP rank gets the same: the query split (below) cuts only + the indexer's queries, not the keys they are scored against. The target's context workspace + (`PrefillTarget._ctx_workspace`, from `_ctx_kv_offsets`, held at the 2**19 rows it takes from + this engine) holds the request's whole KV on each chunk whose KV fits those rows, its cached + prefix and the chunk, each row at its own position: `num_ctx_kv_tokens` (the new keys) would + leave the prefix out. The last chunk's KV is past them, so its rows attend off the pool in + sub-batches of 8192 instead.""" + cached, new = sum(_CHUNKS[:chunk]), _CHUNKS[chunk] + total = cached + new + rows = tep4_modeling._ctx_workspace_rows(_target_sizing(tep8)) + assert rows == 2**19 + for rank in (0, _TEP8_RANKS - 1): + md = _indexer_step(tep8, rank, ctx=[(cached, new)]) + assert md.host_ctx_cached_token_indptr.tolist() == [0, cached] + assert md.ctx_cached_token_indptr.tolist() == [0, cached] + assert md.host_ctx_kv_indptr.tolist() == [0, total] + assert md.ctx_kv_indptr.tolist() == [0, total] + assert (md.num_ctx_cached_tokens, md.max_ctx_seq_len, md.max_ctx_kv_len) == ( + cached, + new, + total, + ) + assert md.num_ctx_mla_kv_tokens == total + assert md.num_ctx_kv_tokens == new + assert md.skip_indexer_for_ctx_reqs is False + + offsets = tep4_modeling._ctx_kv_offsets(md, md.num_contexts) + assert offsets["ctx_cached_tokens"] == cached + assert offsets["cu_ctx_kv"].tolist() == [0, total] + prefill = object.__new__(tep4_modeling.PrefillTarget) + prefill.core = SimpleNamespace(lat_dim=1, ctx_workspace_rows=rows) + step = dict(num_contexts=1, num_ctx_tokens=md.num_ctx_tokens, **offsets) + ws = prefill._ctx_workspace(step, new, "cpu") + if total > rows: + assert chunk == 31 and ws.ws_buf is None + assert ws.sub_batches == [(0, 8192), (8192, 16384), (16384, 24576), (24576, new)] + else: + assert ws.ws_buf.shape[0] == ws.kv_req.shape[0] == ws.kv_pos.shape[0] == total + assert int(ws.kv_pos[0]) == 0 and int(ws.kv_pos[-1]) == total - 1 + assert ws.ws_off.tolist() == [0] and int(ws.kv_req.max()) == 0 + + table = md.host_indexer_k_cache_block_offsets + fp8, scale = _key_slots(table, 0, torch.arange(cached, total)) + assert torch.equal(md.slot_mapping_fp8, fp8) and torch.equal(md.slot_mapping_scale, scale) + if chunk == 0: + assert md.slot_mapping_fp8_fullkv is md.slot_mapping_fp8 + assert md.slot_mapping_scale_fullkv is md.slot_mapping_scale + else: + fp8, scale = _key_slots(table, 0, torch.arange(total)) + assert md.slot_mapping_fp8_fullkv.shape == (total,) + assert torch.equal(md.slot_mapping_fp8_fullkv, fp8) + assert torch.equal(md.slot_mapping_scale_fullkv, scale) + assert torch.equal(md.slot_mapping_fp8_fullkv[cached:], md.slot_mapping_fp8) + # The last chunk ends one token short of max_seq_len less the tokens it generates. + assert total <= _ENVELOPE_PROMPT and (chunk != 31 or total == _ENVELOPE_PROMPT) + + +# ------------------------------------------------------- tep8 (4): the query split over eight ranks + + +def test_tep8_q_split_threshold_is_8192(tep8, tep8_gate): + """`DeepSeekSparseAttentionConfig.q_split_threshold` (llmapi/llm_args.py) defaults to 8192 and + reaches `DSAMetadataParams`, which `_prefill_q_split_threshold` (dsa/indexer.py) reads; the + split is on, not disabled by a negative value, from core.yaml and the boot gate's command + alike.""" + for resolved in (tep8, tep8_gate): + params = _metadata_params(resolved) + assert resolved.model_config.sparse_attention_config.q_split_threshold == 8192 + assert params.q_split_threshold == 8192 + assert ( + dsa_indexer._prefill_q_split_threshold(SimpleNamespace(sparse_metadata_params=params)) + == 8192 + ) + + +# context tokens in a step -> per-rank sizes of the split at tp 8 (None: no split) +_TEP8_SPLITS = { + 8191: None, + 8192: [1024] * 8, + 8193: [1024] * 7 + [1025], + 32767: [4095] + [4096] * 7, + # a whole chunk, the warmup's and the KV estimation's step + 32768: [4096] * 8, + # the 1M gate prompt's last chunk + 32751: [4093] + [4094] * 7, + # a chunk beside 15 decode rows + 32736: [4092] * 8, + # several short prompts, 3000 + 3000 + 3000 + 500 tokens + 9500: [1187, 1188] * 4, +} + + +@pytest.mark.parametrize("count", list(_TEP8_SPLITS)) +def test_tep8_split_prefill_queries_per_rank(tep8, count): + """`_split_prefill_queries` (dsa/indexer.py) on each of the eight TP ranks' mappings: below + 8192 context tokens every rank takes them all; from 8192 on, rank r takes + [count r // 8, count (r + 1) // 8) and every rank knows every share's size (4096 rows a rank + of a 32768-token step). Attention DP (dep4's mapping) never splits.""" + sizes = _TEP8_SPLITS[count] + assert sizes is None or sum(sizes) == count + begins = [0] if sizes is None else [sum(sizes[:r]) for r in range(_TEP8_RANKS + 1)] + for rank in range(_TEP8_RANKS): + mapping = _rank_mapping(tep8.mapping, rank) + assert mapping.tp_rank == rank + split = dsa_indexer._split_prefill_queries(count, mapping, 8192) + if sizes is None: + assert split == (0, count, None) + else: + assert split == (begins[rank], begins[rank + 1], sizes) + adp = copy.deepcopy(tep8.mapping) + adp.enable_attention_dp = True + assert dsa_indexer._split_prefill_queries(count, adp, 8192) == (0, count, None) + + +@pytest.mark.parametrize( + "q_lens,kv_lens", + [ + pytest.param([32768], [3 * 32768], id="one-chunk-over-a-cached-prefix"), + pytest.param([32751], [_ENVELOPE_PROMPT], id="last-chunk"), + pytest.param([3000, 3000, 3000, 500], [3000, 3000, 3000, 500], id="short-prompts"), + pytest.param([2048] * 16, [2048] * 16, id="sixteen-prompts"), + pytest.param([600] * 15 + [23768], [600] * 15 + [23768 + 5 * 32768], id="mixed-lengths"), + pytest.param([5, 5, 8182], [983045, 983045, 8182], id="packed-tails-beside-a-prompt"), + ], +) +def test_tep8_chunk_plan_covers_each_rank_share(tep8, tep8_32k, q_lens, kv_lens): + """`_packed_logits_elem_budget` (dsa/indexer.py) is 1 << 28 at both configs (the packed cap; + `_INDEXER_MQA_LOGITS_ELEM_BUDGET` x max_num_tokens x max_seq_len do not bind). Under it + `plan_mla_chunked_prefill_chunks` covers exactly each of the eight ranks' shares + [q_begin, q_end) of the context tokens, in order, once each; a request the share boundary + cuts is a chunk of its own, and whole requests share a chunk only within the budget.""" + for resolved in (tep8, tep8_32k): + budget = dsa_indexer._packed_logits_elem_budget( + SimpleNamespace( + max_num_tokens=resolved.max_num_tokens, + kv_cache_manager=SimpleNamespace(max_seq_len=resolved.max_seq_len), + ) + ) + assert budget == dsa_indexer._INDEXER_PACKED_LOGITS_ELEM_BUDGET == 1 << 28 + assert dsa_indexer._INDEXER_MQA_LOGITS_ELEM_BUDGET == 1 << 31 + + total = sum(q_lens) + covered = [] + for rank in range(_TEP8_RANKS): + q_begin, q_end, sizes = dsa_indexer._split_prefill_queries( + total, _rank_mapping(tep8.mapping, rank), 8192 + ) + assert sizes is not None and len(sizes) == _TEP8_RANKS + chunks = plan_mla_chunked_prefill_chunks(q_lens, kv_lens, 1 << 28, q_begin, q_end) + spans = _covered(chunks, q_lens) + assert spans[0][0] == q_begin and spans[-1][1] == q_end + assert all(prev[1] == cur[0] for prev, cur in zip(spans, spans[1:])) + for chunk in chunks: + whole = all(start == 0 and end == q_lens[i] for i, start, end, _ in chunk) + if len(chunk) > 1: + assert whole + assert sum(q_lens[i] for i, *_ in chunk) * sum(kv_lens[i] for i, *_ in chunk) <= ( + 1 << 28 + ) + elif not whole: + req_idx, start, end, req_start = chunk[0] + assert req_start + start == q_begin or req_start + end == q_end + covered += spans + assert covered[0][0] == 0 and covered[-1][1] == total + assert all(prev[1] == cur[0] for prev, cur in zip(covered, covered[1:])) + + +@pytest.mark.parametrize("chunk", [1, 31], ids=["second", "last"]) +def test_tep8_each_rank_plans_its_share_of_a_chunk(tep8, chunk): + """Through `Indexer.prepare` on each of the eight TP ranks: a chunk of the long prompt (32768 + or 32751 tokens over its cached prefix) sets `indexer_prefill_q_split` = (q_begin, q_end, + sizes) and one `indexer_prefill_chunks` entry for the rank's share, queries [q_begin, q_end) + of the chunk against keys [0, cached + q_end) of the request, each query's window ending at + its own position. The eight shares tile the chunk.""" + cached, new = sum(_CHUNKS[:chunk]), _CHUNKS[chunk] + sizes = _TEP8_SPLITS[new] + assert sum(sizes) == new + for rank in range(_TEP8_RANKS): + md = _indexer_step(tep8, rank, ctx=[(cached, new)]) + q_begin, q_end = sum(sizes[:rank]), sum(sizes[: rank + 1]) + assert md.indexer_prefill_q_split == (q_begin, q_end, sizes) + assert _chunk_ranges(md) == [(q_begin, q_end, 0, cached + q_end)] + (plan,) = md.indexer_prefill_chunks + assert torch.equal(plan.cu_seqlen_ks, torch.zeros(q_end - q_begin, dtype=torch.int32)) + assert torch.equal( + plan.cu_seqlen_ke, + torch.arange(cached + q_begin + 1, cached + q_end + 1, dtype=torch.int32), + ) + assert md.decode_schedules == [] + + +def test_tep8_q_split_on_skipped_and_mixed_steps(tep8): + """What `Indexer.prepare_for_chunked_prefill` does on the two steps the target must not + misread. (a) A step whose every context is within index_topk 2048: `prepare_for_skip_indexer` + sets `skip_indexer_for_ctx_reqs` and fills the dense top-k rows, and the engine STILL sets + `indexer_prefill_q_split` once the context tokens reach 8192 (and plans each rank's share); + only the full-KV gather mappings are skipped. A target that gathers on the split must not do + so on a skipped step. (b) A mixed step, one chunk with 15 decode rows behind it: the split + counts the context tokens only and covers the context rows; the decode rows keep their own + schedule.""" + for rank in range(_TEP8_RANKS): + skipped = _indexer_step(tep8, rank, ctx=[(0, 1000)] * 9) + assert skipped.skip_indexer_for_ctx_reqs is True + assert skipped.indexer_prefill_q_split == (1125 * rank, 1125 * (rank + 1), [1125] * 8) + # Rank r's share runs from inside prompt r to inside prompt r + 1 (rank 0 holds prompt 0 + # whole, rank 7 prompt 8): two chunks, cut at the prompt boundary 1000 (r + 1). + boundary = 1000 * (rank + 1) + assert [(c.token_start, c.token_end) for c in skipped.indexer_prefill_chunks] == [ + (1125 * rank, boundary), + (boundary, 1125 * (rank + 1)), + ] + assert skipped.slot_mapping_fp8_fullkv is skipped.slot_mapping_fp8 + dense = skipped.host_topk_indices_buffer[:9000] + assert torch.equal(dense[999, :1000], torch.arange(1000, dtype=torch.int32)) + assert bool((dense[999, 1000:] == -1).all()) + + cached, new = 2 * 32768, 32768 - 32 + decode = [1048000 - 64 * i for i in range(15)] + mixed = _indexer_step(tep8, 3, ctx=[(cached, new)], gen=decode) + assert (mixed.num_contexts, mixed.num_generations, mixed.num_tokens) == (1, 15, 32751) + assert mixed.indexer_prefill_q_split == (3 * 4092, 4 * 4092, _TEP8_SPLITS[32736]) + assert _chunk_ranges(mixed) == [(12276, 16368, 0, cached + 16368)] + assert mixed.decode_schedules == [15] + assert mixed.max_gen_seq_len == 1 + assert mixed.host_gen_kv_indptr[-1].item() == sum(decode) + 15 + + +def test_packed_chunks_pass_max_seq_len_keys(tep8): + """`plan_mla_chunked_prefill_chunks` (dsa/indexer.py) packs consecutive whole context requests + into one indexer chunk while sum(q) x sum(kv) stays within 1 << 28, and the chunk's keys are + every packed request's whole KV, cached and new: a chunk can carry more than max_seq_len keys. + (a) Two requests' 5-token last chunks over 983040-token prefixes (10 context tokens, no + split): one chunk of n_k 1966090 on every rank. (b) A 50-token last chunk over a 1048480-token + prefix beside a fresh 200-token prompt: one chunk of n_k 1048730. (c) The two tails of (a) + beside a fresh 8182-token prompt, 8192 context tokens: split 1024 a rank; rank 0's share + holds both tails whole, packed into the 1966090-key chunk, then the fresh prompt's first 1014 + rows as a chunk of their own; ranks 1..7 each plan a slice of the fresh prompt over its own + keys, past the tails' 1966090. Each packed chunk's n_k is bounded by the budget over its new + tokens and by the KV pool, not by max_seq_len; the gather mappings cover every key.""" + tails = [(983040, 5), (983040, 5)] + for rank in range(_TEP8_RANKS): + packed = _indexer_step(tep8, rank, ctx=tails) + assert packed.indexer_prefill_q_split is None + assert _chunk_ranges(packed) == [(0, 10, 0, 1966090)] + assert 10 * 1966090 <= 1 << 28 + assert tuple(packed.slot_mapping_fp8_fullkv.shape) == (1966090,) + table = packed.host_indexer_k_cache_block_offsets + for request, position in ((0, 0), (0, 983044), (1, 0), (1, 983044)): + fp8, scale = _key_slots(table, request, torch.tensor([position])) + flat = request * 983045 + position + assert int(packed.slot_mapping_fp8_fullkv[flat]) == int(fp8) + assert int(packed.slot_mapping_scale_fullkv[flat]) == int(scale) + + beside = _indexer_step(tep8, 0, ctx=[(1048480, 50), (0, 200)]) + assert beside.indexer_prefill_q_split is None + assert _chunk_ranges(beside) == [(0, 250, 0, 1048730)] + assert 1048730 > _MAX_SEQ_LEN and 250 * 1048730 <= 1 << 28 + assert 1048480 % _TOKENS_PER_BLOCK == 0 and 1048480 + 50 < _MAX_SEQ_LEN + + for rank in range(_TEP8_RANKS): + split = _indexer_step(tep8, rank, ctx=[*tails, (0, 8182)]) + assert split.indexer_prefill_q_split == ( + 1024 * rank, + 1024 * (rank + 1), + _TEP8_SPLITS[8192], + ) + if rank == 0: + assert _chunk_ranges(split) == [(0, 10, 0, 1966090), (10, 1024, 1966090, 1967104)] + else: + q_end = 1024 * (rank + 1) + assert _chunk_ranges(split) == [(1024 * rank, q_end, 1966090, 1966090 + q_end - 10)] + assert tuple(split.slot_mapping_fp8_fullkv.shape) == (1966090 + 8182,) + + +def test_tep8_indexer_chunking_under_each_envelope(tep8, tep8_32k): + """`Indexer.prepare_for_chunked_prefill` (dsa/indexer.py) on the largest step of each engine. + Delivered (32768 unchunked, no cached KV) it takes the indexer's own chunking: + `indexer_max_chunk_size` resolves to its 32768 default, `select_indexer_chunk_size` keeps it + below 256K keys, and a whole 32768-token prompt (or the warmup's and KV estimation's 32767 + + 1) is one chunk, so no chunk list is built and the query is never split. At the 1M config the + MLA-chunked-prefill branch plans the chunks instead, by request and packed logits, and the + warmup's and KV estimation's one 32768-token prompt is itself a split step: each of the eight + ranks plans 4096 of its rows, so every boot runs the split before the pool is sized. + `indexer_max_chunk_size` is not read there.""" + params = _metadata_params(tep8_32k) + assert params.indexer_max_chunk_size == 32768 >= tep8_32k.max_num_tokens + assert select_indexer_chunk_size(params.indexer_max_chunk_size, 32768) == 32768 + groups = split_prefill_chunks(torch.tensor([32768], dtype=torch.int32), 32768) + assert groups == [[(0, 0, 32768, 0)]] + for ctx in ([(0, 32768)], [(0, 32767), (0, 1)]): + for rank in (0, _TEP8_RANKS - 1): + step = _indexer_step(tep8_32k, rank, ctx=ctx) + assert step.indexer_prefill_chunks is None and step.indexer_prefill_q_split is None + assert step.slot_mapping_fp8_fullkv is step.slot_mapping_fp8 + + for rank in range(_TEP8_RANKS): + step = _indexer_step(tep8, rank, ctx=[(0, 32768)]) + assert step.indexer_prefill_q_split == (4096 * rank, 4096 * (rank + 1), [4096] * 8) + assert _chunk_ranges(step) == [(4096 * rank, 4096 * (rank + 1), 0, 4096 * (rank + 1))] + assert step.slot_mapping_fp8_fullkv is step.slot_mapping_fp8 + + +def test_tep8_context_rows_take_the_path_of_their_kv(tep8): + """The target's context-row path (`PrefillTarget._ctx_workspace`) on the engine's own host + metadata at the 1M config, on the eight TP ranks, over the held workspace's rows the target + takes from this engine (`_ctx_workspace_rows`: 2**19). It reads a step's context KV as + cached + new over its context requests (`_ctx_kv_offsets`, the host + `ctx_kv_indptr[num_contexts]`), never `num_ctx_kv_tokens`, which counts the new tokens only; + the query split changes only which rows a rank selects for, so every rank takes the same + path: + + * the KV-capacity estimate's step, one fresh 32768-token prompt split 4096 a rank, takes the + workspace path, and the first context step allocates the held bf16 [2**19, 576] workspace; + * the 1M gate prompt's chunks, each a query-split step with the indexer running: the first + 16 (context KV up to 2**19) on the workspace path, from the 17th on the decode path in + sub-batches of 8192, the last (32751 rows behind 1015808 cached) 8192 x 3 and 8175; + * one row of context KV either side of 2**19, with a decode row behind; + * a packed step of four chunks over 1032192, 524288, 0 and 2048 cached keys with twelve + decode rows behind them: the decode path over the 32752 context rows only; + * nine prompts within index_topk (the context indexer skipped, the split still planned): the + workspace path. + + A workspace-path step keeps layer 35's gather beside its rows when its context KV fits the + held rows twice, and gathers it again for each sharer otherwise. The workspace is never + reallocated after the first step.""" + rows_b = tep4_modeling._ctx_workspace_rows(_target_sizing(tep8)) + assert (rows_b, tep4_modeling._CTX_DECODE_ROWS) == (2**19, 8192) + + def path(target, rank, ctx, gen=()): + md = _indexer_step(tep8, rank, ctx=ctx, gen=gen) + step = dict( + num_contexts=md.num_contexts, + num_ctx_tokens=md.num_ctx_tokens, + **tep4_modeling._ctx_kv_offsets(md, md.num_contexts), + ) + n_kv = sum(c + n for c, n in ctx) + assert int(md.host_ctx_kv_indptr[md.num_contexts]) == n_kv + assert not any(c for c, _ in ctx) or md.num_ctx_kv_tokens == md.num_ctx_tokens < n_kv + ws = target._ctx_workspace(step, md.num_ctx_tokens, torch.device("cpu")) + split = md.indexer_prefill_q_split + if ws.ws_buf is not None: + assert n_kv <= rows_b and ws.ws_buf.shape == (n_kv, 576) + assert ws.ws_buf.data_ptr() == target._ws_buf.data_ptr() + # Layer 35's gather is kept beside the step's rows where they fit twice. + assert (ws.kept is not None) == (2 * n_kv <= rows_b) + return "workspace", split, None + assert n_kv > rows_b and sorted(ws.sched) == sorted({hi - lo for lo, hi in ws.sub_batches}) + return "decode", split, ws.sub_batches + + full = [(0, 8192), (8192, 16384), (16384, 24576), (24576, 32768)] + targets = [] + for rank in range(_TEP8_RANKS): + target = tep4_modeling.PrefillTarget.__new__(tep4_modeling.PrefillTarget) + target.core = SimpleNamespace(lat_dim=576, ctx_workspace_rows=rows_b) + share = (4096 * rank, 4096 * (rank + 1), [4096] * 8) + assert path(target, rank, [(0, _MAX_NUM_TOKENS)]) == ("workspace", share, None) + held = target._ws_buf + assert held.shape == (rows_b, 576) and held.dtype == torch.bfloat16 + targets.append((target, held)) + for k, new in enumerate(_CHUNKS): + # Every rank on the chunks either side of 2**19 and on the last; one rank on the rest. + for rank in range(_TEP8_RANKS) if k in (15, 16, 31) else [k % _TEP8_RANKS]: + target = targets[rank][0] + got = path(target, rank, [(sum(_CHUNKS[:k]), new)]) + q_begin, q_end, sizes = got[1] + assert sizes == _TEP8_SPLITS[new] and q_end - q_begin == sizes[rank] + if k < 16: + assert got[0] == "workspace", k + elif k < 31: + assert got[0::2] == ("decode", full), k + else: + assert got[0::2] == ("decode", full[:3] + [(24576, 32751)]) + target, held = targets[0] + edge = rows_b - 32767 + assert path(target, 0, [(edge, 32767)], gen=[4096])[0] == "workspace" + past = path(target, 0, [(edge + 1, 32767)], gen=[4096]) + assert past[0::2] == ("decode", full[:3] + [(24576, 32767)]) + packed = [(1032192, 16384), (524288, 8192), (0, 4096), (2048, 4080)] + for rank, (target, _) in enumerate(targets): + got = path(target, rank, packed, gen=[1048000 - 64 * i for i in range(12)]) + assert got[0::2] == ("decode", full[:3] + [(24576, 32752)]) + assert got[1] is not None + skipped = path(target, rank, [(0, 1000)] * 9) + assert skipped == ("workspace", (1125 * rank, 1125 * (rank + 1), [1125] * 8), None) + assert all(target._ws_buf is held for target, held in targets) + + +# ------------------------------------------------------- tep8 (5): the scheduler's chunks + + +def test_tep8_scheduler_admits_the_envelope_prompt_in_chunks(tep8): + """`KVCacheV2Scheduler` (pyexecutor/scheduler/scheduler_v2.py), the scheduler of a V2 KV + manager, with the chunking `_create_py_executor` configures. One scheduler serves the eight + ranks. It schedules the decode rows first, then cuts the context chunk from what the step's + 32768 tokens leave, rounded down to the 32-token unit unless it is the prompt's last. The + configured GUARANTEED_NO_EVICT is served as MAX_UTILIZATION. Alone, the 1048559-token prompt + takes 32 steps, 31 of 32768 and a last of 32751; beside 15 decode rows each chunk is 32736, + context and decode rows 32751 of the 32768, and the prompt takes 33 steps, the last 1007.""" + assert tep8.llm_args.scheduler_config.capacity_scheduler_policy == ( + CapacitySchedulerPolicy.GUARANTEED_NO_EVICT + ) + alone = _schedule_prompt(tep8, _ENVELOPE_PROMPT, 0) + assert alone == [(c, 0) for c in _CHUNKS] + assert len(alone) == 32 and alone[-1] == (32751, 0) + + beside = _schedule_prompt(tep8, _ENVELOPE_PROMPT, 15) + assert beside == [(32736, 15)] * 32 + [(1007, 15)] + assert (32768 - 15) // 32 * 32 == 32736 + assert all(chunk + rows <= _MAX_NUM_TOKENS for chunk, rows in beside) + assert sum(chunk for chunk, _ in beside) == _ENVELOPE_PROMPT + + +# ------------------------------------------------------- tep8 (6): KV pool, what max_seq_len sizes + + +def test_kv_pool_as_at_tep4(tep8, tep8_gate, tep8_32k, tep8_32k_gate, tep4): + """MLA's latent and the indexer keys are not head-sharded, and every rank holds all 16 + requests: a token costs the same on each of the eight ranks as on each tep4 rank, at the 1M + config as delivered (`MistralLarge4CacheManager.get_cache_size_per_token`, + attention/backends/sparse/mistral_large4/cache_manager.py) -- an fp8_ds_mla latent of 656 + bytes (512 e4m3, four fp32 tile scales, 64 bf16 rope) on each of the 63 KV owners (the nine + sharers keep no latent) plus a 128-byte fp8 indexer key and its fp32 scale on each of the 44 + full-indexer layers, in pages of 32, in the manager the sparse registry picks for a + MistralLarge4Config, with no per-request intercept. The delivered pool is that + times 16 x 32768 tokens; at 1M the pool is sized by free memory, not by 16 x + max_seq_len.""" + for resolved in (tep8, tep8_gate, tep8_32k, tep8_32k_gate, tep4): + model_config, kv_cache_config = resolved.model_config, resolved.llm_args.kv_cache_config + assert get_kv_cache_manager_cls(model_config, kv_cache_config) is ( + MistralLarge4CacheManager + ) + assert kv_cache_config.tokens_per_block == 32 + + cfg, sparse = tep8.model_config.pretrained_config, tep8.model_config.sparse_attention_config + latent = MistralLarge4CacheManager.latent_token_bytes + owners = sum(1 for i, s in enumerate(cfg.kv_source_layers) if s == i) + indexer_key = sparse.index_head_dim + sparse.index_head_dim // 128 * 4 + full_indexer_layers = cfg.index_topk_pattern.count("F") + per_token = latent * owners + indexer_key * full_indexer_layers + assert (latent, owners, indexer_key, full_indexer_layers) == (656, 63, 132, 44) + assert per_token == _KV_BYTES_PER_TOKEN == 47136 + + for rank in range(_TEP8_RANKS): + cost = _kv_cost(tep8, rank) + assert (cost.slope, cost.intercept) == (per_token, 0) + assert cost == _kv_cost(tep8_gate, rank) == _kv_cost(tep8_32k, rank) + assert cost == _kv_cost(tep8_32k_gate, rank) == _kv_cost(tep4, rank % 4) + assert cost.bytes_for_tokens(32) == 32 * per_token == 1_508_352 + + tokens = tep8_32k.max_batch_size * tep8_32k.max_seq_len + pool = _kv_cost(tep8_32k, 0).bytes_for_tokens(tokens) + assert pool == tokens * per_token == 24_712_839_168 + assert math.isclose(pool / 2**30, 23.016, abs_tol=5e-4) + request = _kv_cost(tep8, 0).bytes_for_tokens(_MAX_SEQ_LEN) + assert request == 49_425_678_336 and math.isclose(request / 2**30, 46.03, abs_tol=5e-3) + + +def test_tep8_buffers_sized_by_max_seq_len(tep8, tep8_32k): + """The KV manager's page tables are max_seq_len wide: `KVCacheManagerV2.__init__` + (pyexecutor/kv_cache/kv_cache_manager_v2.py) takes max_seq_len plus one token of generation + headroom (`_get_generation_kv_capacity`) in pages of 32 and pads to a multiple of 4: 32772 + columns at 1048576 (1028 at 32768). The attention metadata the target reads + (`kv_cache_block_offsets`, the DSA `block_table`, `indexer_k_cache_block_offsets`, + `block_table_expanded`) is 16 sequences by that width. Every other buffer -- the selection + buffer `topk_indices_buffer` [32768, 2048], the radix top-k's `radix_aux_indices` / + `radix_aux_logits` [16, 10, 2048], DeepGEMM's `scheduler_metadata_buffer` [num_sms + 1, 2], + the slot mappings and query bounds [32768] -- is the same at both configs; none is + max_batch_size x max_seq_len, and no buffer has a max_seq_len dimension.""" + assert _get_generation_kv_capacity(None, is_draft=False) == (0, 1) + assert _max_blocks_per_seq(_MAX_SEQ_LEN) == 32772 and _max_blocks_per_seq(32768) == 1028 + wide, narrow = _metadata_buffers(tep8), _metadata_buffers(tep8_32k) + + tables = { + "kv_cache_block_offsets": ((1, 16, 2), torch.int32), + "block_table": ((16,), torch.int32), + "indexer_k_cache_block_offsets": ((16,), torch.int32), + "block_table_expanded": ((16,), torch.int32), + } + for name, (rows, dtype) in tables.items(): + assert wide[name] == (rows + (32772,), dtype) + assert narrow[name] == (rows + (1028,), dtype) + assert wide["topk_indices_buffer"] == ((32768, 2048), torch.int32) + assert wide["radix_aux_indices"] == ( + (16, dsa_metadata._MAX_RADIX_BLOCKS_PER_ROW, 2048), + torch.int32, + ) + assert wide["radix_aux_logits"] == ((16, 10, 2048), torch.float32) + # `_metadata_buffers` stands in 148 for the device's SM count. + assert wide["scheduler_metadata_buffer"] == ((149, 2), torch.int32) + assert wide["slot_mapping_fp8"] == wide["slot_mapping_scale"] == ((32768,), torch.int64) + assert wide["cu_seqlen_ks"] == wide["cu_seqlen_ke"] == ((32768,), torch.int32) + assert wide["shared_topk_indices"] == ((32768, 2048), torch.int32) + assert wide["ctx_cached_token_indptr"] == wide["ctx_kv_indptr"] == ((17,), torch.int64) + + # Under cached-KV MLA the base metadata also keeps the uncached prefix sums. + assert set(wide) - set(narrow) == {"ctx_uncached_token_indptr"} + assert set(narrow) <= set(wide) + for name, (shape, dtype) in wide.items(): + assert _MAX_SEQ_LEN not in shape, name + assert _MAX_BATCH_SIZE * _MAX_SEQ_LEN not in shape, name + if name not in tables and name in narrow: + assert (shape, dtype) == narrow[name], name + + +# The KV estimation's temporary pool at the 1M config, in pages and tokens. +_ESTIMATION_PAGES = 32784 +_ESTIMATION_TOKENS = _ESTIMATION_PAGES * _TOKENS_PER_BLOCK + + +def test_tep8_kv_estimation_profiles_one_full_step(tep8, tep8_32k): + """Before the pool exists `KvCacheCreator` (pyexecutor/_util.py) profiles dummy context + requests as long as a request may be, net max_seq_len - 1, but no more than max_num_tokens in + all, for the one scheduler the eight ranks share: at the 1M config one 32768-token prompt (the + delivered engine: a 32767- and a 1-token prompt). The DSA config keeps it off the chunked-MLA + profile (`_get_mla_chunked_profile_length`), so the profiled step is one 32768-token context + with nothing cached -- a split step at the 1M config (see the indexer chunking test). + `_get_token_num_for_estimation` sizes the temporary pool for the larger of those prompts and + CUDA-graph capture's longest decode row: ceil((max_seq_len + 1) / 32) + 16 - 1 pages, 32784 at + the 1M config -- 1049088 tokens, 46.05 GiB at 47136 bytes a token. It holds a max_seq_len + request, so the estimation KV manager does not lower max_seq_len (`KVCacheManagerV2.__init__` + caps it by `get_num_available_tokens`, which a V2 estimate caps by these max_tokens, and + `KvCacheCreator._create_kv_cache_manager` would carry a lowered value to every later + manager). No buffer is max_batch_size x max_seq_len.""" + expected = { + id(tep8): ([32768], _ESTIMATION_PAGES), + id(tep8_32k): ([32767, 1], 1040), + } + for resolved in (tep8, tep8_32k): + creator = _kv_creator(resolved) + requests = creator._create_dummy_context_requests(max(1, resolved.max_seq_len - 1)) + assert creator._mla_chunked_profile_length is None + lengths = [len(request.input_token_ids) for request in requests] + prompts, pages = expected[id(resolved)] + assert lengths == prompts and sum(lengths) == resolved.max_num_tokens + creator._dummy_reqs = requests + tokens = creator._get_token_num_for_estimation() + assert tokens == pages * _TOKENS_PER_BLOCK + assert tokens >= resolved.max_seq_len + 1 + assert _ESTIMATION_TOKENS == 1_049_088 < _MAX_BATCH_SIZE * _MAX_SEQ_LEN + assert math.isclose(_ESTIMATION_TOKENS * _KV_BYTES_PER_TOKEN / 2**30, 46.05, abs_tol=5e-3) + + +def test_tep8_kv_estimation_reserves_no_workspace(tep8): + """What `KvCacheCreator.configure_kv_cache_capacity` (pyexecutor/_util.py) takes out of the KV + budget beside the profiled peak, at the 1M config on sm_103, on every rank's mapping: the + indexer MQA-logits workspace (`get_indexer_mqa_logits_workspace_bytes`, a tile of + `_INDEXER_MQA_LOGITS_ELEM_BUDGET`) is reserved only when the model engine's sparse attention + config is the deepseek_v4 algorithm, and this one is DSA; the TRTLLM backend declares no + context-MLA workspace for fp8 DSA on sm_103 + (`TrtllmAttention.runtime_workspace_bytes_per_token`, attention/backends/trtllm.py: sparse + MLA stages no buffer), so `get_mla_context_workspace_kv_len_cap` installs no attended-KV cap + though chunked prefill is on (with no workspace the creator keeps the chunk-bounded default + and asks no chunk profile), `KvCacheCreator._fp8_ctx_mla_kv_len_cap` stays None and no fixed + capacity-sized reserve is taken. The pool is the profiled budget whole.""" + model_config = tep8.model_config + assert model_config.sparse_attention_config.algorithm == "dsa" != "deepseek_v4" + quant_mode = model_config.quant_config.quant_mode + assert quant_mode.has_fp8_kv_cache() and not quant_mode.has_fp4_kv_cache() + with pytest.MonkeyPatch.context() as mp: + mp.setattr(trtllm_backend, "get_sm_version", lambda: 103) + mp.delenv("TRTLLM_MLA_SHORT_SEQ_MHA_THRESHOLD", raising=False) + for rank in range(_TEP8_RANKS): + mapping = _rank_mapping(tep8.mapping, rank) + w = get_attention_workspace_bytes_per_token(model_config, mapping) + fixed = get_attention_workspace_fixed_bytes( + model_config, + mapping, + max_batch_size=_MAX_BATCH_SIZE, + max_num_tokens=_MAX_NUM_TOKENS, + max_seq_len=_MAX_SEQ_LEN, + enable_cuda_graph=True, + ) + assert (w, fixed) == (0, 0) + assert get_attention_workspace_is_chunked_prefill_bounded(model_config) is True + cap = get_mla_context_workspace_kv_len_cap( + tep8.llm_args.kv_cache_config, + _MAX_BATCH_SIZE, + _MAX_NUM_TOKENS, + _MAX_SEQ_LEN, + tep8.llm_args.enable_chunked_prefill, + True, + chunked_workspace_profiled=False, + require_chunked_workspace_profile=w > 0, + ) + assert cap is None + assert tep8.llm_args.kv_cache_config.fp8_context_mla_kv_len_cap is None + + +@pytest.mark.parametrize("tep8_envelope", ["1m", "32k"]) +def test_warmup_runs_one_full_step_on_every_rank(request, tep8_envelope): + """The warmup's shapes at tep8 (`DecoderWarmup._get_full_general_warmup_requests`, + pyexecutor/engine/runners/decoder/warmup.py), on the estimation's pool at the 1M config (the + warmup runs in the profiling executor first, then again in the final one). Each rank proposes + its shapes from its own free KV, the eight ranks keep the ones all proposed + (`WarmupAgreement.agree_shapes`), and every rank enters each shape's forward. The largest is + min(max_num_tokens, max_batch_size x (max_seq_len - 1)) = 32768 tokens of context in one step, + which `DummyRequestFactory.create_warmup_request` (decoder/dummy_requests.py) builds as prompts + of at most max_seq_len - 1: one 32768-token prompt at the 1M config (a 32767- and a 1-token + prompt delivered), never a 1M prompt -- warmup bypasses the scheduler. All eight ranks run it on + the same tokens, so each rank's routed experts see 32768 tokens, and when every token's top-4 + falls in one rank's 16 experts, 4 x 32768 = 131072 permuted rows on that rank. No prefill CUDA + graph is captured (`filter_prefill_capture_num_tokens`, pyexecutor/engine/cuda_graph.py: its 16 + x (max_seq_len - 1) ceiling caps only capture candidates).""" + wide = tep8_envelope == "1m" + resolved = request.getfixturevalue("tep8" if wide else "tep8_32k") + prompts = [32768] if wide else [32767, 1] + pool_tokens = _ESTIMATION_TOKENS if wide else 16 * resolved.max_seq_len + config = _runner_config(resolved) + group = _Group() + + def rank_warmup(rank): + resources = _Resources(_KvPool(pool_tokens, config.max_seq_len)) + warmup = DecoderWarmup.__new__(DecoderWarmup) + warmup._config = config + agreement = WarmupAgreement(_RankDist(group, rank), _rank_mapping(resolved.mapping, rank)) + assert agreement.is_distributed_forward() + shapes = agreement.agree_shapes(warmup._get_full_general_warmup_requests(resources)) + max_context = max(tokens for tokens, gen in shapes if gen == 0) + batch = _dummy_factory(resolved).create_warmup_request(resources, max_context, 0) + enters = agreement.should_run_warmup_batch(batch, max_context, "max context") + return shapes, enters, [prompt.token_num for prompt in batch.context_requests] + + results = group.run(rank_warmup) + for shapes, enters, rank_prompts in results: + assert shapes == [(1, 0), (1, 1), (32768, 0), (16, 16), (2, 0)] + assert enters is True + assert rank_prompts == prompts + + cfg = resolved.model_config.pretrained_config + rows = sum(results[0][2]) + assert rows == resolved.max_num_tokens == 32768 + local_experts = cfg.n_routed_experts // resolved.mapping.moe_ep_size + assert local_experts == 16 >= cfg.num_experts_per_tok == 4 + assert rows * cfg.num_experts_per_tok == 131072 + + kept, unrecordable = filter_prefill_capture_num_tokens( + config.cuda_graph_batch_sizes, + resolved.max_num_tokens, + _MAX_BATCH_SIZE, + resolved.max_seq_len, + ) + assert (kept, unrecordable) == ([1, 2, 4, 8, 16], []) + assert resolved.llm_args.prefill_cuda_graph_backend == "disabled" + + +@pytest.mark.parametrize("tep8_envelope", ["1m", "32k"]) +def test_tep8_cuda_graph_capture_decodes_at_max_seq_len(request, tep8_envelope): + """`DecoderCudaGraphs.warmup_and_capture` (pyexecutor/engine/runners/decoder/graphs.py) + captures every graph batch (1, 2, 4, 8, 16) twice for DSA, whose config asks for separate + short and long graphs (`needs_separate_short_long_cuda_graphs`, its threshold index_topk + 2048): once at max_seq_len and once at 2048 less the one token a decode step adds. + `create_cuda_graph_warmup_request` (decoder/dummy_requests.py) fills each batch with one-token + decode rows and one row of the family's longest history (`add_longest_dummy_request`): + 1048575 tokens at the 1M config -- every boot captures the decode kernels at KV length 1048576 + -- and 2046 (32767 and 2046 delivered). At batch 16 the long family needs 32768 + 15 pages, + within the 32784 the estimation pool holds, so capture sends no request the pool cannot + hold.""" + wide = tep8_envelope == "1m" + resolved = request.getfixturevalue("tep8" if wide else "tep8_32k") + sparse = copy.deepcopy(resolved.model_config.sparse_attention_config) + assert sparse.needs_separate_short_long_cuda_graphs() is True + families = [resolved.max_seq_len, sparse.seq_len_threshold - 1] + longest_rows = (resolved.max_seq_len - 1, 2046) + assert families == ([1048576, 2047] if wide else [32768, 2047]) + + config = _runner_config(resolved) + assert config.cuda_graph_batch_sizes == [1, 2, 4, 8, 16] + for family, longest in zip(families, longest_rows): + for batch_size in config.cuda_graph_batch_sizes: + pool = _KvPool(_ESTIMATION_TOKENS, resolved.max_seq_len) + batch = _dummy_factory(resolved).create_cuda_graph_warmup_request( + _Resources(pool), batch_size, 0, family + ) + rows = [row.token_num for row in batch.generation_requests] + assert rows == [longest] + [1] * (batch_size - 1) + assert all(row.is_gen for row in batch.generation_requests) + pages = math.ceil((longest + 1) / _TOKENS_PER_BLOCK) + (batch_size - 1) + assert pages <= _ESTIMATION_PAGES + assert math.ceil(1048576 / _TOKENS_PER_BLOCK) + 15 == 32783 + + +def test_tep8_pool_holds_the_envelope_load(monkeypatch, tep8, tep8_gate, tep8_32k): + """Arithmetic on the pools measured at tep8 on the fp8_ds_mla cache, at 47136 bytes a token + (`MistralLarge4CacheManager.get_cache_size_per_token`): 3,402,400 tokens a rank at 32768 at the + accuracy gate's KV fraction 0.85 (core.yaml's), and 3,581,536 at the 1M config at the boot + gate's 0.9 (the quickstart's). Each is also read at `KvCacheConfig.max_util_for_resume` (0.95) + of its size, the share of its quota the V2 manager (`KVCacheManagerV2.__init__`) syncs as the + ranks' resumable token capacity, in case the scheduler holds requests to it. The V2 manager + gives a request ceil((tokens + 1) / 32) pages. The 1M config adds width-sized allocations the + measured 32768 engine did not have (the 1M pool has its rope tables out already; charging them + again only understates what is left): the target's rope tables at 1048576 rows (indexer 64 x + fp32, identity 64 x fp32: 512 MiB against their 16 MiB at 32768 rows; the 16 MiB `mla` layout + that engine also held, which the target no longer builds, not credited), one gathered indexer + key block of 1048576 x 132 bytes, the indexer logits tile at the budget the target takes from + each engine (`_prefill_logits_budget`: 1 << 28, 1 GiB, at 1M against 1 << 24, 64 MiB, at 32768), + the decode scorer's [16, 1048576] fp32 arena, and the held context workspace + (`PrefillTarget._ctx_workspace`, `_ctx_workspace_rows` at 1M): 2**19 rows of bf16 576 with the + int32 request and position maps of a step that fills it (a step with more context KV attends its + context rows straight off the pool). Charged to the pool whole, they cost 49,517 tokens. What is + left of either pool, read either way, holds the 1M gate's load -- the 1048559-token prompt + generating 16 tokens beside 15 short requests (its probes and fillers), each bounded here by a + full step -- and three requests of max_seq_len, though not four: the latent is replicated over + the eight ranks, so the engine holds three 1M requests at once.""" + fractions = { + tep8_gate.llm_args.kv_cache_config.free_gpu_memory_fraction: _TEP8_BOOT_POOL_TOKENS, + tep8.llm_args.kv_cache_config.free_gpu_memory_fraction: _TEP8_ACCURACY_POOL_TOKENS, + } + assert fractions == {0.9: 3_581_536, 0.85: 3_402_400} + resume = tep8.llm_args.kv_cache_config.max_util_for_resume + assert resume == tep8_gate.llm_args.kv_cache_config.max_util_for_resume == 0.95 + + mib = 2**20 + monkeypatch.delenv("TLLM_INDEXER_MQA_LOGITS_ELEM_BUDGET", raising=False) + wide, narrow = _target_sizing(tep8), _target_sizing(tep8_32k) + budgets = [ + tep4_modeling._prefill_logits_budget(md.max_num_tokens, md.max_seq_len) + for md in (wide, narrow) + ] + assert budgets == [1 << 28, 1 << 24] + held = tep4_modeling._ctx_workspace_rows(wide) + grown = { + "rope tables": (1048576 - 32768) * (64 + 64) * 4, + "gathered keys": (1048576 - 32768) * 132, + "logits tile": (budgets[0] - budgets[1]) * 4, + "decode arena": 16 * (1048576 - 32768) * 4, + "context workspace": held * (576 * 2 + 2 * 4), + } + assert 1048576 * (64 + 64) * 4 == 512 * mib + assert held * 576 * 2 == 576 * mib + assert sum(grown.values()) == 2_333_999_104 < 2.2 * 2**30 + lost = math.ceil(sum(grown.values()) / _KV_BYTES_PER_TOKEN) + assert lost == 49_517 + + gate = _pages(_ENVELOPE_PROMPT + 16) + 15 * _pages(_MAX_NUM_TOKENS + 16) + assert gate * _TOKENS_PER_BLOCK == 1_540_576 + three = 3 * _pages(_MAX_SEQ_LEN) * _TOKENS_PER_BLOCK + four = 4 * _pages(_MAX_SEQ_LEN) * _TOKENS_PER_BLOCK + assert (three, four) == (3_145_824, 4_194_432) + for measured in (_TEP8_BOOT_POOL_TOKENS, _TEP8_ACCURACY_POOL_TOKENS): + assert measured < four + for pool in (measured - lost, measured * 95 // 100 - lost): + assert gate * _TOKENS_PER_BLOCK <= pool + assert three <= pool < four + # The tightest reading: the accuracy gate's pool at 0.95, 36,939 tokens past three. + assert _TEP8_ACCURACY_POOL_TOKENS * 95 // 100 - lost - three == 36_939 + + +# ------------------------------------------------------- tep8 (7): the engine's node layout + + +def test_shell_lm_head_at_tp8(tep8, tep4): + """The causal-LM shell every target subclasses builds the vocab-parallel lm_head from the TP + width: on each rank an eighth of the 204800-token vocabulary, 25600 rows of 6144, gathered + over the TP group of all eight ranks (``allgather`` over ``mapping.tp_group``) and with no + AllReduce of its own. The node layout does not enter. Built on the meta device: no weights.""" + + def lm_head(resolved, rank): + config = dataclasses.replace( + resolved.model_config, mapping=_rank_mapping(resolved.mapping, rank) + ) + cfg = config.pretrained_config + with torch.device("meta"): + shell = DecoderModelForCausalLM( + DecoderModel(config), + config=config, + hidden_size=cfg.hidden_size, + vocab_size=cfg.vocab_size, + ) + head = shell.lm_head + return dict( + tp=(head.tp_size, head.tp_rank, head.tp_mode), + shape=tuple(head.weight.shape), + gather=(head.gather_output, config.lm_head_gather_output), + all_reduce=head.all_reduce, + group=head.mapping.tp_group, + quant=head.quant_config, + pp=(shell.pp_rank, shell.pp_size), + ) + + for rank in range(_TEP8_RANKS): + head = lm_head(tep8, rank) + assert head["tp"] == (8, rank, TensorParallelMode.COLUMN) + assert head["shape"] == (204800 // 8, 6144) == (25600, 6144) + assert head["gather"] == (True, True) and head["all_reduce"] is None + assert head["group"] == list(range(_TEP8_RANKS)) and head["pp"] == (0, 1) + tep4_head = lm_head(tep4, 1) + assert tep4_head["tp"] == (4, 1, TensorParallelMode.COLUMN) + assert tep4_head["shape"] == (51200, 6144) and tep4_head["all_reduce"] is None + + +def test_collective_paths_read_the_node_layout(tep8, tep4): + """Where the engine's collectives read the node layout at tep8, on each rank's mapping: + + * the rank's mapping is an ``MpiTopology`` and its communicator an ``MPIDist`` (the MPI + launch leaves TLLM_DISABLE_MPI unset; ``DeviceMeshTopology`` and ``TorchDist`` are the + non-MPI path); + * the MNNVL communicator (``MnnvlMemory.get_comm``) splits by pipeline, context and MoE TP + rank, keyed by TP rank: one communicator of all eight ranks, in TP order; an MNNVL buffer + sits on device ``mapping.local_rank``, the device the worker set; + * an engine ``AllReduce`` at the default AUTO strategy takes MNNVL only for a group that spans + nodes (``MNNVLAllReduce.is_mnnvl`` asks ``mapping.is_multi_node()``: true at tep8, false at + tep4), and the one-node IPC workspace (``IpcMemory``) stays closed past one node's devices. + No userbuffers: the strategy is AUTO, not NCCL_SYMMETRIC, and torch.compile is off.""" + llm_args = tep8.llm_args + assert llm_args.allreduce_strategy == "AUTO" and llm_args.torch_compile_config is None + signature = MnnvlMemory.comm_layout_signature(tep8.mapping) + assert signature == (8, 1, 1, 8, 1, 8, 1) + with pytest.MonkeyPatch.context() as mp: + mp.delenv("TLLM_DISABLE_MPI", raising=False) + for rank in range(_TEP8_RANKS): + rank_mapping = _rank_mapping(tep8.mapping, rank) + assert type(rank_mapping) is MpiTopology + assert type(Distributed.get(rank_mapping)) is MPIDist + assert MnnvlMemory.comm_split_color_key(rank_mapping) == (0, rank) + assert distributed_ops._mnnvl_device_index(rank_mapping) == rank % 4 + assert rank_mapping.is_multi_node() is True + assert IpcMemory(rank_mapping, 1024).open_ipc is False + for rank in range(4): + rank_mapping = _rank_mapping(tep4.mapping, rank) + assert MnnvlMemory.comm_split_color_key(rank_mapping) == (0, rank) + assert rank_mapping.is_multi_node() is False + + +def test_moe_autotuner_at_sixteen_local_experts(tep8, tep8_32k, tep4): + """The autotuner's tuning group is the TP group, ``(tp_size, tp_rank)``: all eight ranks over + both nodes. The NVFP4 MoE runner tunes INDEPENDENT, each rank alone, so tuning sends nothing + between the nodes; its expert-parallel size, num_experts // local_num_experts, is 8 at tep8 + and moves no bucket without attention DP: the buckets and the token-to-bucket rule are tep4's, + 32768 its top bucket when the target tunes to max_num_tokens. ModelConfig's MoE token budget + reads the attention-DP width only (32768, at the 1M config as delivered: a chunk is at most + max_num_tokens), and no MoE load balancer is built.""" + cfg = tep8.model_config.pretrained_config + for rank in range(_TEP8_RANKS): + tuner = SimpleNamespace(mapping=_rank_mapping(tep8.mapping, rank)) + assert AutoTuner._tuning_group(tuner) == (8, rank) + + ep_size = cfg.n_routed_experts // (cfg.n_routed_experts // tep8.mapping.moe_ep_size) + assert ep_size == 8 + tep8_tuning = FP4BlockScaleMoERunner.get_tuning_config( + ep_size, tune_max_num_tokens=tep8.max_num_tokens, use_dp=False + ) + tep4_tuning = FP4BlockScaleMoERunner.get_tuning_config( + 4, tune_max_num_tokens=tep4.max_num_tokens, use_dp=False + ) + assert tep8_tuning.distributed_tuning_strategy == DistributedTuningStrategy.INDEPENDENT + (tep8_spec,) = tep8_tuning.dynamic_tensor_specs + (tep4_spec,) = tep4_tuning.dynamic_tensor_specs + assert tep8_spec.gen_tuning_buckets == tep4_spec.gen_tuning_buckets + assert max(tep8_spec.gen_tuning_buckets) == 32768 + for tokens in (1, 3, 16, 17, 4097, 8192, 8193, 16384, 32767, 32768): + assert tep8_spec.map_to_tuning_buckets(tokens) == tep4_spec.map_to_tuning_buckets(tokens) + assert tep8_spec.map_to_tuning_buckets(32768) == 32768 + + for resolved in (tep8, tep8_32k): + model_config = resolved.model_config + assert model_config.moe_max_num_tokens == resolved.max_num_tokens == 32768 + assert resolved.mapping.dp_size == 1 + assert resolved.llm_args.moe_config.load_balancer is None + assert model_config.moe_load_balancer is None + for rank in range(_TEP8_RANKS): + with maybe_create_moe_load_balancer( + model_config, _rank_mapping(resolved.mapping, rank) + ) as lb: + assert lb is None + + +def test_kv_host_tier_is_shared_by_the_node_four_ways(): + """The V2 KV manager provisions a host tier per rank (no host_cache_size is set): its device + quota, capped at half the node's available physical pages divided by the ranks on the node -- + ``Distributed.local_world_size``, the size of the MPI communicator split by host, four at + tep8 -- then the minimum over all eight ranks (`_compute_auto_host_tier_quota`, + pyexecutor/kv_cache/kv_cache_manager_v2.py). So a tep8 node's four tiers together hold at most + half the host memory available when the pool is built.""" + gib = 2**30 + quota = 160 * gib + assert _compute_auto_host_tier_quota(quota, 4, 800 * gib, float("inf")) == 100 * gib + assert _compute_auto_host_tier_quota(quota, 4, 2000 * gib, float("inf")) == quota + assert _compute_auto_host_tier_quota(quota, 4, float("inf"), float("inf")) == quota + + +# ------------------------------------------------------- tep8 (8): beside tep4 and dep4 + + +def test_tep4_and_dep4_route_on_one_node_of_four(tep4, dep4): + """The NVFP4 checkpoint at tep4 and at dep4, built as the tep8 arguments are, on their own + base options (32768 tokens, chunked prefill off), resolve and route on one node of four: each + claimed by its own target (`route`, models/mistral_large4/routing.py), the routed experts + expert-parallel four ways and not split along their width, each on the checkpoint's anchor, + the same 47136-byte token of the fp8_ds_mla pool and the DSA config unchanged.""" + cases = ( + (tep4, False, _NVFP4_TEP4_TARGET), + (dep4, True, "ModelingV2MistralLarge4Nvfp4Sm103Dep"), + ) + for resolved, attention_dp, target in cases: + seen = resolved.routed[0] + mapping = seen.mapping + assert (mapping.world_size, mapping.tp_size, mapping.pp_size) == (4, 4, 1) + assert (mapping.moe_ep_size, mapping.moe_tp_size, mapping.enable_attention_dp) == ( + 4, + 1, + attention_dp, + ) + assert mapping.gpus_per_node == 4 and mapping.is_multi_node() is False + for rank in range(4): + rank_mapping = _rank_mapping(mapping, rank) + assert (rank_mapping.moe_ep_rank, rank_mapping.moe_tp_rank) == (rank, 0) + assert (rank_mapping.local_rank, rank_mapping.node_rank) == (rank, 0) + assert _route(seen.pretrained_config, mapping).__name__ == target + assert resolved.llm_args.quant_config.quant_algo == QuantAlgo.MIXED_PRECISION + assert resolved.llm_args.quant_config.kv_cache_quant_algo == QuantAlgo.FP8 + assert _anchor(resolved.llm_args) == 91.926 + assert (_kv_cost(resolved, 0).slope, _kv_cost(resolved, 0).intercept) == (47136, 0) + assert resolved.llm_args.enable_chunked_prefill is False + sparse = resolved.model_config.sparse_attention_config + assert (sparse.algorithm, sparse.q_split_threshold, sparse.index_topk) == ( + "dsa", + 8192, + 2048, + ) + assert (tep4.max_num_tokens, tep4.max_seq_len, tep4.max_batch_size) == (32768, 32768, 16) + assert (dep4.max_num_tokens, dep4.max_seq_len, dep4.max_batch_size) == (32768, 32768, 16) + + +@pytest.mark.parametrize("tep8_envelope", ["1m", "32k"]) +def test_tep8_differs_from_tep4_in_the_world_only(request, tep8_envelope): + """The same checkpoint at tep8 and at tep4 on the same options resolves to the same + configuration -- quantization, per-layer quantization, model config, DSA config, limits, + chunked prefill, CUDA graphs, KV config, the MoE config and token budget, what the router + reads -- except the mapping's world: its size, the TP and attention-TP widths and the EP + width; at the 1M config (tep4 on its 1M gate's options) as delivered.""" + wide = tep8_envelope == "1m" + tep8 = request.getfixturevalue("tep8" if wide else "tep8_32k") + tep4 = request.getfixturevalue("tep4_1m" if wide else "tep4") + assert _summary(tep8) == _summary(tep4) + tep8_mapping, tep4_mapping = tep8.mapping.to_dict(), tep4.mapping.to_dict() + assert {k for k in tep8_mapping if tep8_mapping[k] != tep4_mapping[k]} == { + "world_size", + "tp_size", + "attn_tp_size", + "moe_ep_size", + } + + +def test_tep4_branch_claims_tep8_at_both_envelopes(deployment): + """The routing (`route`, models/mistral_large4/routing.py), on what each tep8 engine hands it: + the tep4 branch, whose target serves eight ranks as well as four, claims the NVFP4 export at + tep8, at the 1M config as delivered, from core.yaml and from the boot gate's command -- no + criterion reads max_seq_len or chunked prefill.""" + resolved = deployment[0] + seen = resolved.routed[0] + target = _route(seen.pretrained_config, seen.mapping) + assert target.__name__ == _NVFP4_TEP4_TARGET + assert routing._nvfp4_mixed_precision(seen.pretrained_config) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_head_padding.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_head_padding.py new file mode 100644 index 000000000000..45308447c6f8 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_head_padding.py @@ -0,0 +1,148 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The absorbed query's head layout on the target's own scratch, per kernel that reads it. + +A rank holds 16 of the model's 64 query heads at tep4 (four ranks) and 8 at tep8 (eight ranks +over two nodes of four). FlashMLA's sm100 sparse prefill takes 64 or 128 query heads, so the +prefill target keeps the absorbed query in a `[rows, 64, 576]` bf16 scratch, zeroed once per +step, and writes only its rank's heads 0..h-1 (the absorption BMM into the 512-wide head, the +rope tail written or zeroed by the SSMax op, the owner's rope op in place). The generation +rows' sparse decode, `cute_dsl_ds_mla_sparse_decode`, runs the rank's h heads unpadded: the +decode target's scratch is `[rows, h, 576]`, every element written by every layer (the SSMax +op's tail write included), and the prefill target's generation rows hand it the 64-head +scratch, of which it reads heads 0..h-1. Both kernels write only those h heads of their output +into the compact `[rows, h, 512]` latent output the V-B BMM reads. That the kernels leave the +live heads unchanged under zero dead heads, and read h of a wider query, is their own +certification (`flash_mla_sparse_prefill`'s and `cute_dsl_ds_mla_sparse_decode`'s cells); this +checks the layout around them, through `PrefillTarget._scratch` and `DecodeTarget._scratch`, +at both world sizes: + +* the prefill scratch's query is 64 heads of 576, bf16, every byte zero (+0.0, not -0.0); the + decode scratch's is the rank's h heads of 576, bf16, contiguous; +* the rank's heads are a view of heads 0..h-1 with contiguous heads, the layout the rope op and + the kernels address, so writing them never reaches heads h..63; +* the latent output is h heads wide, a NoPE owner's latent rows are 576 wide and zero (the + fused glue's; the unfused glue, the core's `unfused_glue` (`_UNFUSED_GLUE`), builds them per + call), and no FlashMLA scheduler rides on the step. + +`_GEN_ATTN_FLASH_MLA` (the core's `gen_attn_flash_mla`) puts the generation rows back on +FlashMLA's 64-head decode for A/B verification; its layout (the decode scratch at 64 heads, +zeroed, one scheduler shared by the step's layers, and a latent output only for the unfused +glue: the fused glue reads FlashMLA's live heads in place) is checked too, under both glues, and +that the shipped default is the native kernel. Neither switch depends on max_seq_len. +""" + +from __future__ import annotations + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_103__tep import ( # noqa: E501 + modeling as M, +) + +pytestmark = pytest.mark.cpu_only + +DEV = "cpu" +KV_LORA, ROPE, V_DIM = 512, 64, 128 +LAT = KV_LORA + ROPE +#: World size -> the query heads a rank holds: the model's 64 over the ranks. +HEADS = {4: 16, 8: 8} +GLUES = pytest.mark.parametrize("unfused_glue", [False, True], ids=["fused", "unfused"]) + + +def _core(heads: int, flash_mla: bool, unfused_glue: bool) -> SimpleNamespace: + """The widths the targets' scratch reads off the core, with `heads` of the 64 on the rank, + its generation rows' kernel (`flash_mla`: `_GEN_ATTN_FLASH_MLA`) and its glue + (`unfused_glue`: `_UNFUSED_GLUE`).""" + return SimpleNamespace( + heads=heads, + v_dim=V_DIM, + lat_dim=LAT, + kv_lora=KV_LORA, + rope_dim=ROPE, + gen_attn_flash_mla=flash_mla, + unfused_glue=unfused_glue, + ) + + +def _target(cls, heads: int, flash_mla: bool, unfused_glue: bool): + target = object.__new__(cls) + target.core = _core(heads, flash_mla, unfused_glue) + return target + + +def _check_scratch(s, rows: int, heads: int, width: int, lat_out: bool, unfused_glue: bool) -> None: + assert s.fused_q.dtype == torch.bfloat16 + assert tuple(s.fused_q.shape) == (rows, width, LAT) + if width > heads: + assert not s.fused_q.view(torch.int16).any(), "a head is not +0.0" + if lat_out: + assert tuple(s.lat_out.shape) == (rows, heads, KV_LORA) + else: + assert s.lat_out is None, "decode reads FlashMLA's output in place" + assert tuple(s.attn_out.shape) == (rows, heads * V_DIM) + if unfused_glue: + assert s.nope_lat is None, "the unfused glue builds a NoPE owner's latent per call" + else: + # A NoPE owner's latent rows: RMSNorm(c_kv) lands in the first 512 columns, the rope + # key stays +0.0 for the step. + assert s.nope_lat.dtype == torch.bfloat16 and tuple(s.nope_lat.shape) == (rows, LAT) + assert not s.nope_lat.view(torch.int16).any(), "a NoPE owner's rope key is not +0.0" + + # The rank's heads as the attention block takes them: heads 0..h-1, contiguous heads. + fq = s.fused_q[:, :heads] + assert fq.stride(-1) == 1 and fq.stride(1) == LAT + assert fq.data_ptr() == s.fused_q.data_ptr() + g = torch.Generator().manual_seed(rows) + fq[..., :KV_LORA].copy_(torch.randn(rows, heads, KV_LORA, generator=g)) + fq[..., KV_LORA:].copy_(torch.randn(rows, heads, ROPE, generator=g)) + if width > heads: + assert not s.fused_q[:, heads:].view(torch.int16).any(), "a dead head was written" + else: + assert s.fused_q.is_contiguous() + + +def test_the_native_decode_is_the_shipped_default(): + assert M._GEN_ATTN_FLASH_MLA is False + assert M._FLASH_MLA_HEADS == 64 + + +@GLUES +@pytest.mark.parametrize("world", sorted(HEADS)) +@pytest.mark.parametrize("rows", [1, 2, 7, 16]) +def test_decode_scratch(rows, world, unfused_glue): + heads = HEADS[world] + s = _target(M.DecodeTarget, heads, False, unfused_glue)._scratch(rows, torch.bfloat16, DEV) + _check_scratch(s, rows, heads, heads, lat_out=True, unfused_glue=unfused_glue) + assert s.sched is None + + +@GLUES +@pytest.mark.parametrize("world", sorted(HEADS)) +@pytest.mark.parametrize("rows", [1, 16]) +def test_decode_scratch_under_flash_mla(rows, world, unfused_glue): + heads = HEADS[world] + s = _target(M.DecodeTarget, heads, True, unfused_glue)._scratch(rows, torch.bfloat16, DEV) + _check_scratch( + s, rows, heads, M._FLASH_MLA_HEADS, lat_out=unfused_glue, unfused_glue=unfused_glue + ) + assert s.sched is not None + + +@GLUES +@pytest.mark.parametrize("flash_mla", [False, True]) +@pytest.mark.parametrize("world", sorted(HEADS)) +@pytest.mark.parametrize("tc, gen", [(5, 0), (31, 1), (1031, 15), (2048, 16)]) +def test_prefill_scratch(tc, gen, world, flash_mla, unfused_glue): + rows = tc + gen + heads = HEADS[world] + target = _target(M.PrefillTarget, heads, flash_mla, unfused_glue) + s = target._scratch(rows, torch.bfloat16, DEV) + _check_scratch(s, rows, heads, M._FLASH_MLA_HEADS, lat_out=True, unfused_glue=unfused_glue) + # The context and generation rows are row slices of the one buffer: the dead heads of + # either slice stay zero after the other is written. + assert s.fused_q[:tc].data_ptr() == s.fused_q.data_ptr() + assert (s.sched is not None) == flash_mla diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_indexer_packed_chunks.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_indexer_packed_chunks.py new file mode 100644 index 000000000000..1f979dd41487 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_indexer_packed_chunks.py @@ -0,0 +1,344 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The tep target's context indexer on prefill chunks the engine packs past max_seq_len keys. + +Under chunked prefill the engine plans the indexer's prefill chunks with +`plan_mla_chunked_prefill_chunks`: consecutive context requests share one chunk while (summed +new tokens) x (summed cached + new keys) stays within the packed-logits budget, 2^28 at +max_num_tokens 32768 and max_seq_len 1048576. A short last chunk of a near-1M request beside a +short prompt is therefore one chunk of more than 1048576 keys, past the key scorer's +(`fp8_mqa_logits`) and the prefill top-k's cells. The target cuts a chunk of more than +max_seq_len keys, and only such a chunk, at its request boundaries (`_indexer_blocks`) and scores +each request's rows against its own keys (`PrefillTarget._select_ctx`) in tiles under the logits +budget it takes from the engine (`_prefill_logits_budget`: 2**28 at max_num_tokens 32768 and +max_seq_len 1048576). + +On one GPU, each case's chunk is the one the engine's own planner and chunk builder +(`Indexer.prepare_one_prefill_chunk`) make at that config, over an indexer key pool the +case's full-KV gather mappings address through a page table (the engine's +`_compute_slot_mappings`, each request's pages drawn from one permutation of the pool), with +the step's offsets through the target's `_ctx_kv_offsets` and `_ctx_host_offsets`. Checked per +case: + +* the planner packs every request into one chunk, of the expected key count; +* the target's blocks: one per request where the chunk holds more than max_seq_len keys, the + chunk itself where it does not, each row covered once; +* no key gather, scorer or top-k call sees more than 1048576 keys, and every top-k call's + selection is an exact top-k of the scores it was given (rewritten canonically: ascending, + `-1` last, a tie at the k-th score given to the lowest-index keys, since the top-k promises + neither order nor which tied keys it keeps); +* every row is written, and each request's selections are exactly those of the same request + planned and selected alone. + +Skipped off sm_103 or without a GPU. +""" + +from __future__ import annotations + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch.attention.backends.sparse.dsa import indexer as engine +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_103__tep import ( # noqa: E501 + modeling as M, +) +from tensorrt_llm._utils import get_sm_version + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or get_sm_version() != 103, + reason="the mistral_large4 modeling_v2 targets are certified on sm_103 only", +) + +MAX_SEQ_LEN = 1 << 20 +MAX_NUM_TOKENS = 32768 +TOKENS_PER_BLOCK = 32 +INDEX_HEADS = 32 +INDEX_DIM = 128 +INDEX_TOPK = 2048 +#: The step's widest block table at max_seq_len 1048576 (V2: ceil((L + 1) / 32), to 4). +TABLE_WIDTH = 32772 +SENTINEL = -(2**31) +#: Elements the exact-top-k check holds at once. +_CHECK_ELEMENTS = 1 << 25 + +#: (name, ((cached, new) per context request, in batch order), the packed chunk's keys). +CASES = ( + ( + "a 1048559-token prompt's 100-token tail beside a 150-token prompt", + ((1048459, 100), (0, 150)), + 1048709, + ), + ( + "two 5-token tails over 983040 cached keys", + ((983040, 5), (983040, 5)), + 1966090, + ), + ( + "a 64-token tail of 1015872, a 100-token prompt and a 50-token tail of 32818", + ((1015808, 64), (0, 100), (32768, 50)), + 1048790, + ), + ( + "a 128-token tail and a 128-token prompt, exactly max_seq_len keys", + ((1048320, 128), (0, 128)), + 1048576, + ), + ( + "16 tails of 16 tokens over 61440 cached keys", + tuple((61440, 16) for _ in range(16)), + 983296, + ), +) + + +def _md(requests): + """The metadata the engine's planner, its chunk builder and the target's offsets read, for + a step whose context requests are `requests` (no decode rows).""" + cached = [c for c, _ in requests] + new = [n for _, n in requests] + kv = [c + n for c, n in requests] + + def indptr(values): + return torch.tensor([0, *torch.tensor(values).cumsum(0).tolist()], dtype=torch.int64) + + return SimpleNamespace( + compress_ratios=[1], + cu_seqlen_ks=torch.empty(0, dtype=torch.int32, device="cuda"), + max_num_tokens=MAX_NUM_TOKENS, + kv_cache_manager=SimpleNamespace(max_seq_len=MAX_SEQ_LEN), + enable_context_mla_with_cached_kv=True, + host_ctx_cached_token_indptr=indptr(cached), + host_ctx_uncached_token_indptr=indptr(new), + host_ctx_kv_indptr=indptr(kv), + ctx_cached_token_indptr=indptr(cached).cuda(), + ctx_kv_indptr=indptr(kv).cuda(), + max_ctx_seq_len=max(new), + num_ctx_cached_tokens=sum(cached), + ) + + +def _chunks(md, requests): + """The engine's indexer prefill chunks for the step, unsplit (fewer rows than + q_split_threshold).""" + q_lens = [n for _, n in requests] + kv_lens = [c + n for c, n in requests] + budget = engine._packed_logits_elem_budget(md) + assert budget == 1 << 28, budget + return [ + engine.Indexer.prepare_one_prefill_chunk(md, specs) + for specs in engine.plan_mla_chunked_prefill_chunks(q_lens, kv_lens, budget) + ] + + +def _step(md, requests, chunks, pool): + """The target's step dict for the indexer's context selection, through its own offsets and + block plan.""" + tc = sum(n for _, n in requests) + step = dict( + num_ctx_tokens=tc, + num_ctx_kv_tokens=sum(c + n for c, n in requests), + cu_seqlen_ks=None, + cu_seqlen_ke=None, + max_seq_len=MAX_SEQ_LEN, + slot_mapping_fp8_fullkv=pool.fp8, + slot_mapping_scale_fullkv=pool.scale, + **M._ctx_kv_offsets(md, len(requests)), + **M._ctx_host_offsets(md, len(requests)), + ) + step["indexer_blocks"] = M._indexer_blocks(chunks, step) + return step + + +def _pool(requests, gen, cuda_gen): + """An indexer key pool holding every request's keys at random e4m3 values and positive + scales, and the step's full-KV gather mappings (every request's cached and new keys, + request after request), each request's pages drawn from one permutation of the pool.""" + kv = [c + n for c, n in requests] + pages = [-(-k // TOKENS_PER_BLOCK) for k in kv] + num_pages = sum(pages) + 7 + order = torch.randperm(num_pages, generator=gen) + table = torch.zeros(len(requests), TABLE_WIDTH, dtype=torch.int32) + at = 0 + for r, p in enumerate(pages): + table[r, :p] = order[at : at + p].to(torch.int32) + at += p + req = torch.repeat_interleave(torch.arange(len(requests)), torch.tensor(kv)) + starts = torch.tensor([0, *torch.tensor(kv).cumsum(0).tolist()[:-1]]) + positions = torch.arange(sum(kv)) - starts.repeat_interleave(torch.tensor(kv)) + fp8, scale = engine._compute_slot_mappings( + positions, table, req, INDEX_DIM, TOKENS_PER_BLOCK, INDEX_DIM + ) + row = TOKENS_PER_BLOCK * (INDEX_DIM + 4) + flat = torch.empty(num_pages, row, dtype=torch.uint8, device="cuda") + data = torch.randn(num_pages, TOKENS_PER_BLOCK * INDEX_DIM, generator=cuda_gen, device="cuda") + flat[:, : TOKENS_PER_BLOCK * INDEX_DIM] = data.to(torch.float8_e4m3fn).view(torch.uint8) + scales = 0.5 + torch.rand(num_pages, TOKENS_PER_BLOCK, generator=cuda_gen, device="cuda") + flat[:, TOKENS_PER_BLOCK * INDEX_DIM :] = scales.view(torch.uint8) + return SimpleNamespace( + k_cache=flat.view(num_pages, TOKENS_PER_BLOCK, 1, INDEX_DIM + 4), + fp8=fp8.to(torch.int64).cuda(), + scale=scale.to(torch.int64).cuda(), + kv=kv, + ) + + +def _canonicalize(values, ks, ke, out, top_k: int) -> None: + """Check that `out` is an exact top-k of each row's window `[ks, ke)` of `values` + (window-local indices; a window within top_k kept whole, then `-1`) and rewrite it + canonically: ascending, `-1` last, a tie at the k-th score given to the lowest-index + keys.""" + n, width = values.shape + dev = values.device + cols = torch.arange(width, device=dev, dtype=torch.int64) + ident = torch.arange(top_k, device=dev, dtype=torch.int64) + big = torch.iinfo(torch.int64).max + step = max(1, _CHECK_ELEMENTS // max(width, 1)) + for r0 in range(0, n, step): + r1 = min(n, r0 + step) + s = ks[r0:r1].long()[:, None] + e = ke[r0:r1].long()[:, None] + length = (e - s)[:, 0] + assert int(e.max()) <= width, (int(e.max()), width) + raw = out[r0:r1].long() + raw_sorted = torch.where(raw < 0, big, raw).sort(dim=1).values + raw_sorted = torch.where(raw_sorted == big, -1, raw_sorted) + long_row = length > top_k + short = torch.where(ident < length[:, None], ident, -1) + bad_short = (~long_row) & (raw_sorted != short).any(1) + assert not bad_short.any(), f"{int(bad_short.sum())} short windows not kept whole" + canon = short + if long_row.any(): + sc = torch.where((cols >= s) & (cols < e), values[r0:r1].float(), float("-inf")) + kth = sc.topk(top_k, dim=1).values[:, -1:] + above = (sc > kth).sum(1) + at = sc == kth + inside = (raw_sorted >= 0) & (raw_sorted < length[:, None]) + distinct = (raw_sorted[:, 1:] != raw_sorted[:, :-1]).all(1) + picked = sc.gather(1, (s + raw_sorted.clamp(min=0)).clamp(max=width - 1)) + exact = ( + inside.all(1) + & distinct + & (picked.min(1).values == kth[:, 0]) + & ((picked > kth).sum(1) == above) + ) + bad = long_row & ~exact + assert not bad.any(), f"{int(bad.sum())} rows are not an exact top-{top_k}" + take = (sc > kth) | (at & (at.int().cumsum(1) <= (top_k - above)[:, None])) + best = torch.where(take, cols - s, big).sort(dim=1).values[:, :top_k] + canon = torch.where(long_row[:, None], best, short) + out[r0:r1].copy_(canon.to(out.dtype)) + + +@pytest.fixture +def calls(monkeypatch): + """Every key gather's, scorer's and top-k's key count, through the module names the target + calls them by; each top-k selection checked exact and made canonical where it is made.""" + seen = SimpleNamespace(gather=[], logits=[], topk=[]) + gather, logits = M.indexer_k_cache_gather_op, M.fp8_mqa_logits + topk = M.cute_dsl_gvr_topk_prefill + + def gather_(k_cache, fp8, scale, k_token_start, num_tokens, head_dim): + seen.gather.append((k_token_start, num_tokens)) + return gather(k_cache, fp8, scale, k_token_start, num_tokens, head_dim) + + def logits_(q, kv, w, ks, ke, *args, **kwargs): + seen.logits.append((q.shape[0], kv[0].shape[0])) + return logits(q, kv, w, ks, ke, *args, **kwargs) + + def topk_(values, ks, ke, out, top_k, *args, **kwargs): + seen.topk.append((values.shape[0], values.shape[1])) + result = topk(values, ks, ke, out, top_k, *args, **kwargs) + _canonicalize(values, ks, ke, out, top_k) + return result + + monkeypatch.setattr(M, "indexer_k_cache_gather_op", gather_) + monkeypatch.setattr(M, "fp8_mqa_logits", logits_) + monkeypatch.setattr(M, "cute_dsl_gvr_topk_prefill", topk_) + return seen + + +def _select(requests, chunks, md, pool, q_fp8, weights): + """The target's context selection over one step, on a core with the logits budget + `_derive_first_step` takes from this engine (`_prefill_logits_budget` at max_num_tokens + 32768 and max_seq_len 1048576: 2**28); the selection buffer it fills.""" + step = _step(md, requests, chunks, pool) + tc = step["num_ctx_tokens"] + topk_buf = torch.full([tc, INDEX_TOPK], SENTINEL, dtype=torch.int32, device="cuda") + budget = M._prefill_logits_budget(MAX_NUM_TOKENS, MAX_SEQ_LEN) + assert budget == M._PREFILL_LOGITS_CEILING == 1 << 28, budget + core = SimpleNamespace( + idx_dim=INDEX_DIM, idx_topk=INDEX_TOPK, prefill_logits_budget=budget, + _max_ctx_len=MAX_SEQ_LEN, + ) # fmt: skip + target = M.PrefillTarget.__new__(M.PrefillTarget) + target.core = core + M.PrefillTarget._select_ctx(target, step, pool.k_cache, q_fp8, weights, topk_buf) + torch.cuda.synchronize() + return step, topk_buf + + +@pytest.mark.parametrize("name,requests,packed_keys", CASES, ids=[c[0] for c in CASES]) +def test_packed_chunk_selects_as_each_request_alone(name, requests, packed_keys, calls): + seed = len(name) * 7919 + packed_keys + gen = torch.Generator().manual_seed(seed) + cuda_gen = torch.Generator(device="cuda").manual_seed(seed) + md = _md(requests) + chunks = _chunks(md, requests) + # The engine packs every request into one chunk over all their keys. + tc = sum(n for _, n in requests) + assert len(chunks) == 1, [(c.token_start, c.token_end) for c in chunks] + (chunk,) = chunks + assert (chunk.token_start, chunk.token_end) == (0, tc) + assert chunk.k_token_end - chunk.k_token_start == packed_keys + + pool = _pool(requests, gen, cuda_gen) + q = torch.randn(tc, INDEX_HEADS, INDEX_DIM, generator=cuda_gen, device="cuda") + q_fp8 = q.to(torch.float8_e4m3fn) + weights = torch.randn(tc, INDEX_HEADS, generator=cuda_gen, device="cuda") + step, packed = _select(requests, chunks, md, pool, q_fp8, weights) + + # The blocks: the chunk itself within max_seq_len keys, else one per request. + blocks = step["indexer_blocks"] + cu_new, cu_kv = step["host_cu_ctx_new"], step["host_cu_ctx_kv"] + if packed_keys <= MAX_SEQ_LEN: + assert len(blocks) == 1 and blocks[0] is chunk, blocks + else: + expect = [(cu_new[r], cu_new[r + 1], cu_kv[r], cu_kv[r + 1]) for r in range(len(requests))] + got = [(b.token_start, b.token_end, b.k_token_start, b.k_token_end) for b in blocks] + assert got == expect, (got, expect) + for r, b in enumerate(blocks): + cached = requests[r][0] + assert torch.equal(b.cu_seqlen_ks.cpu(), torch.zeros(requests[r][1], dtype=torch.int32)) + ends = torch.arange(cached + 1, cached + requests[r][1] + 1, dtype=torch.int32) + assert torch.equal(b.cu_seqlen_ke.cpu(), ends) + # No call sees more than max_seq_len keys. + widest = max(n for _, n in calls.gather) + assert widest <= MAX_SEQ_LEN, calls.gather + assert max(n for _, n in calls.logits) <= MAX_SEQ_LEN, calls.logits + assert max(w for _, w in calls.topk) <= MAX_SEQ_LEN, calls.topk + assert int((packed == SENTINEL).any(dim=1).sum()) == 0, "rows left unwritten" + + # Each request alone: its own chunk over its own keys (the same pool slots). + for r, (cached, new) in enumerate(requests): + alone_req = ((cached, new),) + alone_md = _md(alone_req) + alone_chunks = _chunks(alone_md, alone_req) + lo, hi = cu_kv[r], cu_kv[r + 1] + alone_pool = SimpleNamespace( + k_cache=pool.k_cache, + fp8=pool.fp8[lo:hi].contiguous(), + scale=pool.scale[lo:hi].contiguous(), + ) + rows = slice(cu_new[r], cu_new[r + 1]) + _, alone = _select( + alone_req, alone_chunks, alone_md, alone_pool, q_fp8[rows], weights[rows] + ) + differ = int((packed[rows] != alone).any(dim=1).sum()) + assert differ == 0, f"{name}: request {r}, {differ} of {new} rows select otherwise alone" + print( + f"PACKED {name}: chunk keys {packed_keys}, blocks " + f"{[(b.token_start, b.token_end, b.k_token_end - b.k_token_start) for b in blocks]}, " + f"widest gather {widest}, top-k calls (rows, width) {sorted(set(calls.topk))}" + ) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_indexer_q_split.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_indexer_q_split.py new file mode 100644 index 000000000000..83973cf625e0 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_indexer_q_split.py @@ -0,0 +1,1339 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The tep target's context indexer under the engine's query split, at tep4 (four ranks) and at +tep8 (eight ranks over two nodes of four). + +Under chunked prefill the engine splits a step's context queries evenly across the TP ranks +once they reach `q_split_threshold` (8192): `Indexer.prepare_for_chunked_prefill` sets +`indexer_prefill_q_split = (q_begin, q_end, q_sizes)` and plans `indexer_prefill_chunks` over +this rank's `[q_begin, q_end)` only. The target selects those rows and all-gathers every +rank's share into rows `[0, tc)` of its selection buffer before a layer reads them. The engine +also packs consecutive whole requests into one chunk while `sum(q) * sum(kv)` stays within its +logits budget, so short tails over long prefixes make a chunk of more keys than `max_seq_len`; +the target scores such a chunk, and only such a chunk, per request, each over its own keys +(`_indexer_blocks`). + +The ranks, on the MPI session's communicator a tep4 or tep8 engine under mpirun reaches: + +* tep4: four ranks this pytest starts on its node with mpirun (the launching allocation's + PMI/PMIx/Open MPI/Slurm variables removed, as the launch prefix does); +* tep8: eight ranks over two nodes of four, each a pytest process an MPI launcher started -- + `bash /scripts/mn/mn_mpirun.sh 4 python -m pytest -k tep8 ` -- not + trtllm-llmapi-launch, under which only rank 0 runs pytest. + +Each rank builds the target's core on the meta device with layers 1 (a RoPE indexer), 3 (a NoPE +indexer) and 17 (a layer that reuses the last selection) materialized from +Mistral-Large-4-1T-A52B-NVFP4 through the production weight table at its own mapping, at the +engine's max_seq_len, and drives `PrefillTarget._attention` -- the code the forward runs -- over +the engine's own DSA metadata (`MistralLarge4CacheManager`'s fp8_ds_mla pool on the KV owners at +max_seq_len 65536 at tep4, 32768 at tep8, the most keys a scoring block holds; chunked prefill +on, the world's mapping at that rank) and the context workspace the forward builds per step +(`PrefillTarget._ctx_workspace`), one step after another, so every key and latent a step reads +was appended at its position by an earlier one. The core takes its sizes from the first step's +metadata (`_derive_first_step`): at max_num_tokens 32768 the indexer's logits budget is 2**26 at +tep4's max_seq_len and 2**24 at tep8's (`_prefill_logits_budget`), and with chunked prefill on +the held context workspace is 2**19 rows (`_ctx_workspace_rows`). The steps: + +* 3500 tokens (under the threshold: no split, the chunks cover every row); +* 20480 tokens, then 12288 more over those 20480 cached (keys to 32768); +* 8193 tokens of one request, every share boundary inside it (shares 2048, 2048, 2048, 2049 at + tep4; seven shares of 1024 and one of 1025 at tep8); +* four requests summing 8200, one over a cached prefix, with a decode row behind them; +* five requests within `index_topk` summing 8500: the split is planned, the context indexer + skipped; + +and at tep8, whose packed chunks pass its pool's max_seq_len of 32768: + +* two requests of 16384, one 32768-token chunk (4096 rows per rank); +* two 5-token tails over those 16384-token prefixes and an 8200-token prompt: the split, and on + rank 0 a chunk the engine packs from the two tails over 32778 keys; +* two 5-token tails over 16389-token prefixes: no split, one packed chunk over 32788 keys. + +Every step runs as the engine planned it (split) and as the same step with no split (the +chunks `plan_mla_chunked_prefill_chunks` gives over every context row, every row on every +rank); a step that packs requests also runs with each request in a chunk of its own (the +planner at a zero budget). Checked on each rank, per layer: + +* every row the indexer selects for starts as a sentinel and none is left after the split; +* each row's scores (its whole window of fp8_mqa_logits output, by two positional checksums of + the fp32 bits) are bitwise the unsplit ones on this rank's share, and the per-request ones; +* the split's selections are bitwise the unsplit ones and the per-request ones, and the same on + every rank; +* the layer's sparse-MLA output is bitwise the unsplit one and the per-request one; +* one gather per selecting layer, of this rank's share, over every rank, none on the reusing + layer, none without a split and none when the context indexer is skipped; +* no key gather, scorer or top-k call is over more than `max_seq_len` keys, though the engine's + packed chunks may be; each packed chunk's parts are the per-request chunks exactly. + +Both top-k entries promise neither an order nor which of the keys tied at the k-th score they +keep (their catalog notes); measured here, three calls of the prefill top-k on one input kept +different tied keys on 2 of 5120 rows. So a wrapper around each top-k call first checks that the +kernel's selection is an exact top-k of the scores it was given (every key above the k-th score +kept, the rest at it, no duplicate, short windows whole), then rewrites it canonically: each +row ascending, `-1` last, a tie at the k-th score given to the lowest-index tied keys. The +rewrite happens before the gather, so the gather moves it. How many rows tied, and on how many +of those the kernel kept other tied keys, is reported. Also: the split without the gather leaves +the other ranks' rows as the sentinel and the coverage check catches them (negative control); +the selections at the budget the core took from this engine are bitwise those at the 1M +config's `1 << 28` and at the 32768 config's `1 << 24` (`_prefill_logits_budget`), over up to +32768 keys; the gather of a full 32768-token chunk's shares is timed (over the two nodes at +tep8); and the (query rows, keys) tiles the target cuts at the 1M config (max_seq_len 1048576, +its budget 2**28) for one 32768-token chunk of a 1048576-token prompt, planned by the engine's +own planner over cached prefixes up to 1015808, for an unsplit step of 8191 tokens over a +near-1M prefix, and for two 5-token tails over 983040-token prefixes, are reported and checked +against the budget and max_seq_len. + +Skipped where the checkpoint is absent (looked up as `mistral-large-4-nvfp4` or +`Mistral-Large-4-1T-A52B-NVFP4` under `LLM_MODELS_ROOT`); tep4 where fewer than four GPUs are +visible, inside a launch of another world size or in a pytest that is itself a launched rank; +tep8 unless this pytest is one of eight launched ranks. A tep8 rank that fails prints its +traceback and leaves at once, which ends the launcher's job; after a pass every tep8 rank +finalizes MPI once pytest has reported. +""" + +from __future__ import annotations + +import faulthandler +import hashlib +import itertools +import json +import os +import shutil +import signal +import socket +import subprocess +import sys +import time +import traceback +from dataclasses import dataclass +from pathlib import Path +from types import MethodType, SimpleNamespace + +import pytest + +from tensorrt_llm.executor.utils import get_spawn_proxy_process_env +from tensorrt_llm.llmapi.mpi_session import get_mpi_world_size + +#: The ranks of a node: tep4 is one node of four, tep8 two. +RANKS_PER_NODE = 4 +CHECKPOINTS = ("mistral-large-4-nvfp4", "Mistral-Large-4-1T-A52B-NVFP4") +SHARD_INDEX = "consolidated.safetensors.index.json" +#: A world's whole run, compiles and the checkpoint reads included: past it tep4's mpirun and its +#: ranks are ended, and each tep8 rank dumps its stacks and exits, which ends the launcher's job. +DEADLINE_S = 2400 +_RANK_FLAG = "--q-split-rank" + +LAYERS = 72 +TOKENS_PER_BLOCK = 32 +LATENT = 576 +INDEX_TOPK = 2048 +#: The run's chunk (max_num_tokens), batch and the engine's default split threshold. +MAX_NUM_TOKENS = 32768 +MAX_BATCH = 16 +Q_SPLIT_THRESHOLD = 8192 +#: The pool's and the attention metadata's max_seq_len, and the pool's blocks, by world; the +#: core is built at the same max_seq_len (`_build`). At tep8 every top-k call stays within the +#: 32768 columns the prefill top-k entry's cells hold, so max_seq_len is 32768 there, and its +#: packed steps (`PACKED`) pass it. +MAX_SEQ_LEN = {4: 65536, 8: 32768} +NUM_BLOCKS = {4: 2560, 8: 4096} +#: Layer 1 selects with RoPE, layer 3 without; layer 17 reuses the selection in the buffer. +DRIVEN = (1, 3, 17) +SENTINEL = -(2**31) +_INT32_MAX = 2**31 - 1 +#: Rows of the checkpoint's embedding the tokens are drawn from: one contiguous read, every +#: token distinct, so a NoPE key never ties another of its window by repeating a token. +FIRST_TOKEN = 1000 +#: Rows the top-k check and rewrite handle at once (bounds its temporaries at 32768 columns). +_CHECK_ROWS = 1024 + +#: The 1M config's max_seq_len, its longest prompt, at which the tiles the target cuts are +#: reported (`_envelope_tiles`), and the cached prefixes before one 32768-token chunk whose last +#: key is 32768, 65536, ..., 1048576. +ENVELOPE_MAX_SEQ_LEN = 1 << 20 +ENVELOPE_CACHED = (0, 32768, 98304, 229376, 491520, 1015808) +#: An unsplit step just under the threshold whose last key is max_seq_len. +ENVELOPE_UNSPLIT = Q_SPLIT_THRESHOLD - 1 +#: Two 5-token tails over 983040-token prefixes: one packed chunk of 1966090 keys. +ENVELOPE_TAILS = ((983040, 5), (983040, 5)) + +#: Each request's prompt length. +PROMPTS = {0: 32768, 1: 8193, 2: 6000, 3: 3000, 4: 1800, 5: 900, 6: 1700} +PROMPTS.update({r: 1700 for r in range(7, 11)}) +#: The requests of tep8's packed steps. +PACKED_PROMPTS = {11: 16394, 12: 16394, 13: 8200} + + +@dataclass(frozen=True) +class Step: + name: str + #: (request, cached tokens, new tokens) per context request, in batch order. + ctx: tuple + #: Requests decoding one token behind the context rows. + gen: tuple = () + + +STEPS = ( + Step("3500 tokens, under the threshold", ((2, 0, 3500),)), + Step("20480 tokens", ((0, 0, 20480),)), + Step("12288 tokens over 20480 cached", ((0, 20480, 12288),)), + Step("8193 tokens of one request", ((1, 0, 8193),)), + Step( + "four requests summing 8200 and a decode row", + ((2, 3500, 2500), (3, 0, 3000), (4, 0, 1800), (5, 0, 900)), + gen=(1,), + ), + Step( + "five requests within index_topk summing 8500", + tuple((r, 0, 1700) for r in range(6, 11)), + ), +) +#: tep8's further steps, over its pool's max_seq_len of 32768. +PACKED = ( + Step("two requests of 16384, one 32768-token chunk", ((11, 0, 16384), (12, 0, 16384))), + Step( + "two 5-token tails over 16384 cached and 8200 new tokens", + ((11, 16384, 5), (12, 16384, 5), (13, 0, 8200)), + ), + Step("two 5-token tails over 16389 cached", ((11, 16389, 5), (12, 16389, 5))), +) +#: The steps each world runs, and the requests they draw their tokens for. +WORLD_STEPS = {4: STEPS, 8: STEPS + PACKED} +WORLD_PROMPTS = {4: PROMPTS, 8: {**PROMPTS, **PACKED_PROMPTS}} +#: The step the negative control and the budget comparison run on. +NEGATIVE_STEP = 3 +BUDGET_STEP = 2 +#: The steps whose engine-packed chunk passes max_seq_len keys (on rank 0 only under the split): +#: tep8's, at its 32768. +PACKED_STEPS = (7, 8) + + +def _launched_elsewhere(world: int) -> bool: + """A launcher started an MPI world of another size: the ranks of `world` do not start + inside it.""" + return get_mpi_world_size() not in (1, world) + + +def _a_rank_of_a_launch() -> bool: + """This pytest is itself one rank of an MPI launch, not a process on its own (unlaunched, or + trtllm-llmapi-launch's proxy, which runs with the launch's MPI variables removed).""" + return get_mpi_world_size() > 1 and not get_spawn_proxy_process_env() + + +#: The world sizes, each started its own way: at tep4 this pytest, a process on its own, starts +#: the four ranks on its node under mpirun; at tep8 each of the eight ranks over two nodes of +#: four is a pytest process an MPI launcher started. +WORLDS = [ + pytest.param( + 4, + marks=[ + pytest.mark.skipif( + _launched_elsewhere(4), reason="tep4 needs a launch of four ranks or none" + ), + pytest.mark.skipif( + _a_rank_of_a_launch(), + reason="tep4's ranks are started by one pytest under mpirun, not by a launch", + ), + ], + id="tep4", + ), + pytest.param( + 8, + marks=[ + pytest.mark.skipif( + get_mpi_world_size() != 8 or not _a_rank_of_a_launch(), + reason="tep8 needs eight ranks over two nodes of four, each a pytest process an " + "MPI launcher started (mn_mpirun.sh 4); under trtllm-llmapi-launch only rank 0 " + "runs pytest", + ), + ], + id="tep8", + ), +] + +pytestmark = [ + # The ranks' communicators and the CuTe DSL compiles keep threads past the test. + pytest.mark.threadleak(enabled=False), +] + + +# ---------------------------------------------------------------- the rank body + + +def _canonicalize(values, starts, ends, out, top_k: int): + """Check that `out` is an exact top-k of each row's window `[starts, ends)` of `values` + (window-local indices), then rewrite it in canonical form: ascending, `-1` last, a tie at + the k-th score given to the lowest-index tied keys. Returns per row: whether its window has + a tie at the k-th score, whether the kernel kept other tied keys than the canonical ones, + and two positional checksums of the window's fp32 score bits.""" + import torch + + n, width = values.shape + dev = values.device + cols = torch.arange(width, device=dev, dtype=torch.int64) + ident = torch.arange(top_k, device=dev, dtype=torch.int64) + big = torch.iinfo(torch.int64).max + tie_all, moved_all, sums_all = [], [], [] + for r0 in range(0, n, _CHECK_ROWS): + r1 = min(n, r0 + _CHECK_ROWS) + s = starts[r0:r1].long()[:, None] + e = ends[r0:r1].long()[:, None] + length = (e - s)[:, 0] + inside = (cols >= s) & (cols < e) + v = values[r0:r1] + bits = torch.where(inside, v.view(torch.int32).long(), 0) + sums = torch.stack([bits.sum(1), (bits * (cols - s + 1)).sum(1)], dim=1) + sc = torch.where(inside, v.float(), float("-inf")) + raw = out[r0:r1].long() + raw_sorted = torch.where(raw < 0, big, raw).sort(dim=1).values + raw_sorted = torch.where(raw_sorted == big, -1, raw_sorted) + long_row = length > top_k + # A window within top_k is kept whole: every position once, then -1. + short = torch.where(ident < length[:, None], ident, -1) + bad_short = (~long_row) & (raw_sorted != short).any(1) + assert not bad_short.any(), f"{int(bad_short.sum())} short windows not kept whole" + canon = short + tie = torch.zeros_like(long_row) + moved = torch.zeros_like(long_row) + if long_row.any(): + kth = sc.topk(top_k, dim=1).values[:, -1:] + above = (sc > kth).sum(1) + at = sc == kth + inside_idx = (raw_sorted >= 0) & (raw_sorted < length[:, None]) + distinct = (raw_sorted[:, 1:] != raw_sorted[:, :-1]).all(1) + picked = sc.gather(1, (s + raw_sorted.clamp(min=0)).clamp(max=width - 1)) + exact = ( + inside_idx.all(1) + & distinct + & (picked.min(1).values == kth[:, 0]) + & ((picked > kth).sum(1) == above) + ) + bad = long_row & ~exact + assert not bad.any(), f"{int(bad.sum())} rows are not an exact top-{top_k}" + take = (sc > kth) | (at & (at.int().cumsum(1) <= (top_k - above)[:, None])) + best = torch.where(take, cols - s, big).sort(dim=1).values[:, :top_k] + canon = torch.where(long_row[:, None], best, short) + tie = long_row & (above + at.sum(1) > top_k) + moved = long_row & (raw_sorted != canon).any(1) + out[r0:r1].copy_(canon.to(out.dtype)) + tie_all.append(tie) + moved_all.append(moved) + sums_all.append(sums) + return torch.cat(tie_all), torch.cat(moved_all), torch.cat(sums_all) + + +def _unwritten_rows(raw, lo: int, hi: int) -> int: + """Rows of `raw[lo:hi]` still holding the sentinel anywhere.""" + return int((raw[lo:hi] == SENTINEL).any(dim=1).sum()) + + +def _assert_overwritten(raw, lo: int, hi: int, where: str) -> None: + left = _unwritten_rows(raw, lo, hi) + assert left == 0, f"{where}: {left} of rows [{lo}, {hi}) still hold the sentinel" + + +class _Checkpoint: + """Raw tensors by Mistral name, read lazily from the shard files, and the configuration the + engine adapts from its params.json.""" + + def __init__(self, path: Path) -> None: + from tensorrt_llm._torch.models.checkpoints.mistral.config_loader import MistralConfigLoader + + self.path = path + self.index = json.loads((path / SHARD_INDEX).read_text())["weight_map"] + self.config = MistralConfigLoader().load_pretrained_config(str(path)) + + def entry(self, key: str): + from safetensors import safe_open + + return safe_open(self.path / self.index[key], framework="pt", device="cpu").get_slice(key) + + +def _materialize(core, ckpt: _Checkpoint, keys, W) -> None: + """Fill the named parameters of a meta-declared core through the production manifest: + its slices (this rank's heads), multi-source rows and load-time transforms.""" + import torch + + manifest = W.MODEL_WEIGHTS.manifest(core) + for key in keys: + meta = core.w[key] + dst = torch.empty(meta.shape, dtype=meta.dtype, device="cuda") + for ckpt_key, index, transform, shard in manifest[key]: + entry = ckpt.entry(ckpt_key) + src = (entry[...] if shard is None else shard(entry)).to("cuda").contiguous() + if transform is not None: + src = transform(src, core) + (dst if index is None else dst[index]).copy_(src) + core.w[key] = torch.nn.Parameter(dst, requires_grad=False) + + +def _sparse_config(): + from tensorrt_llm.llmapi.llm_args import DeepSeekSparseAttentionConfig + + return DeepSeekSparseAttentionConfig( + index_n_heads=32, index_head_dim=128, index_topk=INDEX_TOPK, indexer_rope_interleave=True + ) + + +def _manager(pretrained_config, world: int): + """The engine's KV cache manager for this checkpoint (the sparse registry's pick for a + MistralLarge4Config): fp8_ds_mla latent pages on the 63 KV owners, the indexer keys on the + 44 full-indexer layers; this rank's own pool (the latent and the indexer keys are + replicated over the ranks), at the world's max_seq_len.""" + from tensorrt_llm._torch.attention.backends.sparse.mistral_large4 import ( + MistralLarge4CacheManager, + ) + from tensorrt_llm.bindings import DataType + from tensorrt_llm.bindings.internal.batch_manager import CacheType + from tensorrt_llm.llmapi.llm_args import KvCacheConfig + from tensorrt_llm.mapping import Mapping + + return MistralLarge4CacheManager( + KvCacheConfig( + enable_block_reuse=False, + max_tokens=NUM_BLOCKS[world] * TOKENS_PER_BLOCK, + host_cache_size=0, + ), + CacheType.SELFKONLY, + num_layers=LAYERS, + num_kv_heads=1, + head_dim=LATENT, + tokens_per_block=TOKENS_PER_BLOCK, + max_seq_len=MAX_SEQ_LEN[world], + max_batch_size=MAX_BATCH, + mapping=Mapping(world_size=1, rank=0, tp_size=1, pp_size=1), + dtype=DataType.FP8, + sparse_attention_config=_sparse_config(), + pretrained_config=pretrained_config, + max_num_tokens=MAX_NUM_TOKENS, + ) + + +def _allocate(env, step: Step) -> None: + """Grow the pool for one step as the scheduler does: a first chunk adds the request, a + later chunk or a decode token extends it.""" + manager = env.manager + for rid, cached, new in step.ctx: + if cached == 0: + manager.add_dummy_requests([rid], token_nums=[new]) + else: + cache = manager.kv_cache_map[rid] + assert cache.resize(cache.capacity + new) + for rid in step.gen: + cache = manager.kv_cache_map[rid] + assert cache.resize(cache.capacity + 1) + + +def _metadata(env, step: Step): + """`DSAtrtllmAttentionMetadata` for one step, prepared as a tep4 or tep8 engine with + chunked prefill prepares it: the cached-KV MLA path on, this rank's mapping.""" + import torch + + from tensorrt_llm._torch.attention.backends.interface import AttentionRuntimeFeatures + from tensorrt_llm._torch.attention.backends.sparse.dsa import DSAtrtllmAttentionMetadata + from tensorrt_llm._torch.metadata import KVCacheParams + + rids = [r for r, _, _ in step.ctx] + list(step.gen) + md = DSAtrtllmAttentionMetadata( + seq_lens=torch.tensor([n for _, _, n in step.ctx] + [1] * len(step.gen), dtype=torch.int), + request_ids=rids, + max_num_requests=MAX_BATCH, + num_contexts=len(step.ctx), + prompt_lens=[env.prompts[r] for r in rids], + max_num_tokens=MAX_NUM_TOKENS, + kv_cache_manager=env.manager, + kv_cache_params=KVCacheParams( + use_cache=True, + num_cached_tokens_per_seq=[c for _, c, _ in step.ctx] + [env.kv[r] for r in step.gen], + ), + mapping=env.mapping, + sparse_attention_config=_sparse_config(), + runtime_features=AttentionRuntimeFeatures( + chunked_prefill=True, cache_reuse=False, chunk_size=MAX_NUM_TOKENS + ), + enable_context_mla_with_cached_kv=True, + ) + md.prepare() + return md + + +def _planned_chunks(md, step: Step, budget: int): + """The same step's chunks with no split, by the engine's planner over every context row at + the given packed-logits budget: the engine's own at `_packed_logits_elem_budget`, one request + per chunk at zero.""" + from tensorrt_llm._torch.attention.backends.sparse.dsa import indexer as engine + + q_lens = [n for _, _, n in step.ctx] + kv_lens = [c + n for _, c, n in step.ctx] + specs = engine.plan_mla_chunked_prefill_chunks(q_lens, kv_lens, budget) + return [engine.Indexer.prepare_one_prefill_chunk(md, s) for s in specs], specs + + +def _step_inputs(env, step: Step): + """The step's token rows (context rows in batch order, then the decode rows) and their + positions.""" + import torch + + ids, pos = [], [] + for rid, cached, new in step.ctx: + ids.append(env.ids[rid][cached : cached + new]) + pos.append(torch.arange(cached, cached + new, dtype=torch.int32)) + for rid in step.gen: + ids.append(env.ids[rid][env.kv[rid] : env.kv[rid] + 1]) + pos.append(torch.tensor([env.kv[rid]], dtype=torch.int32)) + return torch.cat(ids), torch.cat(pos).cuda() + + +def _layer_input(env, rows, i: int): + """Layer `i`'s input: the tokens' embeddings through its own attention norm.""" + import torch + + xf = env.embed[rows].float() + xf = xf * torch.rsqrt(xf.pow(2).mean(-1, keepdim=True) + env.eps) + return xf.to(torch.bfloat16) * env.core.w[f"l{i}_attn_norm"] + + +def _run(env, md, step: Step, chunks, q_split, *, indexer_only: bool = False): + """One pass of the step over the driven layers through the target's own code, with the + given chunk plan. Returns per layer: the selection buffer as the indexer left it (None on + a layer that selects nothing), the selection attention read, the layer output, and per + row the scores' checksums and whether a top-k call wrote them (and tied, and moved).""" + import torch + + prefill = env.prefill + md.indexer_prefill_chunks, md.indexer_prefill_q_split = chunks, q_split + s = prefill.step_args(md) + rows, tc = s["num_tokens"], s["num_ctx_tokens"] + gen = rows - tc + skip_ctx = s["skip_ctx"] + skip_gen = s["skip_gen"] if gen else True + token_rows, pos = env.step_rows, env.step_pos + req_idx = s["req_idx_per_token"][:rows] + topk_buf = prefill._topk_buf(s, rows, tc, skip_ctx, skip_gen) + scratch = prefill._scratch(rows, torch.bfloat16, "cuda") + ctx = prefill._ctx_workspace(s, tc, "cuda") if tc else None + env.topk_base = topk_buf + out = {} + for i in DRIVEN: + L = env.layers[i] + x = _layer_input(env, token_rows, i) + selects = L.indexer is not None and not (skip_ctx and skip_gen) + if selects: + if not skip_ctx: + topk_buf[:tc].fill_(SENTINEL) + if not skip_gen: + topk_buf[tc:].fill_(SENTINEL) + env.raw = None + env.rows_seen = SimpleNamespace( + scored=torch.zeros(rows, dtype=torch.bool, device="cuda"), + tie=torch.zeros(rows, dtype=torch.bool, device="cuda"), + moved=torch.zeros(rows, dtype=torch.bool, device="cuda"), + sums=torch.zeros(rows, 2, dtype=torch.int64, device="cuda"), + ) + if indexer_only: + if L.indexer is not None: + prefill._indexer(s, x, pos, L, topk_buf, tc, skip_ctx, skip_gen) + o = None + else: + o = prefill._attention( + s, L, x, pos, req_idx, topk_buf, scratch, ctx, tc, skip_ctx, skip_gen + ).clone() + out[i] = SimpleNamespace( + raw=env.raw if selects else None, + sel=topk_buf.clone(), + out=o, + rows=rows, + tc=tc, + seen=env.rows_seen, + ) + torch.cuda.synchronize() + return out + + +def _digest(t) -> str: + return hashlib.sha256(t.contiguous().cpu().numpy().tobytes()).hexdigest() + + +def _build(ckpt_path: Path, rank: int, world: int): + """The rank's core at the `world` mapping and the world's max_seq_len, its prefill target + with the driven layers materialized, the pool at the same max_seq_len and the tokens.""" + import torch + + from tensorrt_llm._torch.model_config import ModelConfig + from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_103__tep import ( # noqa: E501 + modeling as M, + ) + from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_103__tep import ( # noqa: E501 + weights as W, + ) + from tensorrt_llm.mapping import Mapping + + ck = _Checkpoint(ckpt_path) + mapping = Mapping( + world_size=world, + rank=rank, + gpus_per_node=RANKS_PER_NODE, + tp_size=world, + pp_size=1, + moe_ep_size=world, + moe_tp_size=1, + ) + # The core at the world's max_seq_len (`MAX_SEQ_LEN`), the pool's and the metadata's: the + # step's `max_seq_len`, the most keys `_indexer_blocks` leaves in one block, the decode + # scorer's columns and the rope tables' rows, which cover every position the pool holds. Its + # logits budget and held context workspace are sized on the first step, from the metadata. + model_config = ModelConfig( + pretrained_config=ck.config, + mapping=mapping, + max_num_tokens=MAX_NUM_TOKENS, + max_seq_len=MAX_SEQ_LEN[world], + ) + with torch.device("meta"): + core = M.MistralLarge4TepModelingV2(model_config) + keys = ["lerope_alpha", "final_norm"] + skip = ("fc", "router", "mlp", "wo", "ffn", "isc", "ws2") + for i in DRIVEN: + keys += [ + k + for k in core.w.keys() + if k.startswith(f"l{i}_") and not k.split("_", 1)[1].startswith(skip) + ] + _materialize(core, ck, keys, W) + core.build_kernels(torch.cuda.current_device()) + prefill = M.PrefillTarget(core) + + # Every request's tokens, distinct, drawn in one seeded order shared by all ranks. + prompts = WORLD_PROMPTS[world] + total = sum(prompts.values()) + 1 + assert FIRST_TOKEN + total <= ck.config.vocab_size, total + table = ck.entry("tok_embeddings.weight")[FIRST_TOKEN : FIRST_TOKEN + total] + order = torch.randperm(total, generator=torch.Generator().manual_seed(1051)) + embed = table[order].cuda() + ids, at = {}, 0 + for rid, length in prompts.items(): + n = length + (1 if rid == 1 else 0) + ids[rid] = torch.arange(at, at + n) + at += n + + env = SimpleNamespace( + M=M, + core=core, + prefill=prefill, + layers={L.index: L for L in prefill._layers}, + mapping=mapping, + world=world, + max_seq_len=MAX_SEQ_LEN[world], + prompts=prompts, + steps=WORLD_STEPS[world], + manager=_manager(model_config.pretrained_config, world), + embed=embed, + ids=ids, + kv={}, + eps=ck.config.rms_norm_eps, + raw=None, + ) + # Record the selection buffer as the indexer leaves it, after its gather. + indexer = M.PrefillTarget._indexer + + def recorded(self, step, x, pos, L, topk_buf, tc, skip_ctx, skip_gen): + indexer(self, step, x, pos, L, topk_buf, tc, skip_ctx, skip_gen) + env.raw = topk_buf.clone() + + prefill._indexer = MethodType(recorded, prefill) + return env + + +def _instrument(env): + """Count the target's gathers, record the keys its key gather, prefill scorer and top-k get, + and check and canonicalize every top-k call's selection where it is made, through the module + names the target calls them by: the decode rows' selector is the trtllm op, or, at a + max_seq_len cute_dsl_radix_topk_decode serves (both worlds' here), that entry.""" + import torch + + M = env.M + env.gathers, env.logits_calls, env.key_calls = [], [], [] + gather, logits, topk = M.allgather, M.fp8_mqa_logits, M.cute_dsl_gvr_topk_prefill + key_gather, topk_decode = M.indexer_k_cache_gather_op, M.indexer_topk_decode + radix_decode = M.cute_dsl_radix_topk_decode + + def counted_gather(input, sizes, group, process_group=None): + env.gathers.append( + (tuple(input.shape), str(input.dtype), tuple(sizes), tuple(group), process_group) + ) + return gather(input, sizes, group, process_group) + + def recorded_key_gather(k_pool, slots_fp8, slots_scale, k_token_start, n_k, *args, **kwargs): + env.key_calls.append(("gather", int(n_k))) + return key_gather(k_pool, slots_fp8, slots_scale, k_token_start, n_k, *args, **kwargs) + + def recorded_logits(q, kv, weights, ks, ke, *args, **kwargs): + result = logits(q, kv, weights, ks, ke, *args, **kwargs) + env.logits_calls.append((q.shape[0], kv[0].shape[0], result.stride(0))) + env.key_calls.append(("logits", int(kv[0].shape[0]))) + return result + + def first_row(out) -> int: + base = env.topk_base + offset = out.data_ptr() - base.data_ptr() + row_bytes = base.stride(0) * base.element_size() + assert offset >= 0 and offset % row_bytes == 0, (offset, row_bytes) + return offset // row_bytes + + def note(out, tie, moved, sums) -> None: + r0 = first_row(out) + r1 = r0 + out.shape[0] + seen = env.rows_seen + assert not seen.scored[r0:r1].any(), f"rows [{r0}, {r1}) selected twice" + seen.scored[r0:r1] = True + seen.tie[r0:r1] = tie + seen.moved[r0:r1] = moved + seen.sums[r0:r1] = sums + + def checked_topk(values, ks, ke, out, top_k, *args, **kwargs): + env.logits_calls[-1] += ("topk", values.shape[0], values.shape[1]) + env.key_calls.append(("topk", int(values.shape[1]))) + result = topk(values, ks, ke, out, top_k, *args, **kwargs) + note(out, *_canonicalize(values, ks, ke, out, top_k)) + return result + + def checked_topk_decode(logits_, seq_lens, indices, next_n, index_topk, *args, **kwargs): + assert next_n == 1, next_n + result = topk_decode(logits_, seq_lens, indices, next_n, index_topk, *args, **kwargs) + starts = torch.zeros_like(seq_lens) + note(indices, *_canonicalize(logits_, starts, seq_lens, indices, index_topk)) + return result + + def checked_radix_decode(logits_, seq_lens, indices, *, next_n, compiled): + assert next_n == 1, next_n + result = radix_decode(logits_, seq_lens, indices, next_n=next_n, compiled=compiled) + starts = torch.zeros_like(seq_lens) + note(indices, *_canonicalize(logits_, starts, seq_lens, indices, indices.shape[1])) + return result + + M.allgather = counted_gather + M.indexer_k_cache_gather_op = recorded_key_gather + M.fp8_mqa_logits = recorded_logits + M.cute_dsl_gvr_topk_prefill = checked_topk + M.indexer_topk_decode = checked_topk_decode + M.cute_dsl_radix_topk_decode = checked_radix_decode + + +def _chunks_summary(chunks): + return [(c.token_start, c.token_end, c.k_token_end - c.k_token_start) for c in chunks or ()] + + +def _max_keys(chunks) -> int: + return max((c.k_token_end - c.k_token_start for c in chunks or ()), default=0) + + +def _same_chunks(a, b) -> bool: + """Two chunk plans with the same rows, keys and windows.""" + import torch + + return len(a) == len(b) and all( + (x.token_start, x.token_end, x.k_token_start, x.k_token_end) + == (y.token_start, y.token_end, y.k_token_start, y.k_token_end) + and torch.equal(x.cu_seqlen_ks, y.cu_seqlen_ks) + and torch.equal(x.cu_seqlen_ke, y.cu_seqlen_ke) + for x, y in zip(a, b) + ) + + +def _compare(k: int, i: int, what: str, got, ref, rows_mask) -> dict: + """Selections, layer output and (on `rows_mask`) scores of one layer of two runs.""" + import torch + + bad_rows = int((got.sel != ref.sel).any(dim=1).sum()) + result = dict( + same_selection=torch.equal(got.sel, ref.sel), + rows_differing=bad_rows, + same_output=torch.equal(got.out, ref.out), + ) + if got.raw is not None: + result["same_scores"] = torch.equal(got.seen.sums[rows_mask], ref.seen.sums[rows_mask]) + assert result["same_scores"], f"step {k} layer {i}: scores differ from {what}" + assert result["same_selection"], f"step {k} layer {i}: {bad_rows} rows differ from {what}" + assert result["same_output"], f"step {k} layer {i}: the attention output differs from {what}" + return result + + +def _check_step(env, k: int, step: Step, log) -> dict: + """Run step `k` split, unsplit and (when the engine packs requests) one request per chunk; + every assertion of the step, on this rank.""" + import torch + + from tensorrt_llm._torch.attention.backends.sparse.dsa import indexer as engine + + M = env.M + rank = env.mapping.tp_rank + _allocate(env, step) + md = _metadata(env, step) + env.core._derive_first_step(md) + env.step_rows, env.step_pos = _step_inputs(env, step) + tc = md.num_ctx_tokens + ctx = md.num_contexts + assert tc == sum(n for _, _, n in step.ctx) + assert md.max_seq_len == env.max_seq_len, md.max_seq_len + + # The engine's split, re-derived from its own helper: rank r's share of the context rows. + split = md.indexer_prefill_q_split + expect = engine._split_prefill_queries(tc, env.mapping, Q_SPLIT_THRESHOLD) + if expect[2] is None: + assert split is None, split + else: + assert split == expect, (split, expect) + q_begin, q_end, q_sizes = split + assert sum(q_sizes) == tc and q_end - q_begin == q_sizes[rank] and len(q_sizes) == env.world + split_chunks = md.indexer_prefill_chunks + full_chunks, full_specs = _planned_chunks(md, step, engine._packed_logits_elem_budget(md)) + lo, hi = (0, tc) if split is None else split[:2] + covered = sorted((c.token_start, c.token_end) for c in split_chunks) + assert covered[0][0] == lo and covered[-1][1] == hi, (covered, lo, hi) + assert all(a[1] == b[0] for a, b in zip(covered, covered[1:])), covered + full = sorted((c.token_start, c.token_end) for c in full_chunks) + assert ( + full[0][0] == 0 and full[-1][1] == tc and all(a[1] == b[0] for a, b in zip(full, full[1:])) + ), full + + # What `_indexer_blocks` reads of the step: the target's scoring blocks for a chunk plan, + # a chunk past max_seq_len keys cut at its requests. + plan = dict( + num_ctx_tokens=tc, + num_ctx_kv_tokens=md.num_ctx_kv_tokens, + cu_seqlen_ks=md.cu_seqlen_ks, + cu_seqlen_ke=md.cu_seqlen_ke, + max_seq_len=md.max_seq_len, + **M._ctx_host_offsets(md, ctx), + ) + + def bounded(chunks): + return M._indexer_blocks(chunks, plan) + + packs = any(len(specs) > 1 for specs in full_specs) + alone_chunks = _planned_chunks(md, step, 0)[0] if packs else None + if k in PACKED_STEPS: + # The engine's packed chunk passes max_seq_len keys; the target's parts are the chunks + # the engine plans for each request alone, rows, keys and windows. + assert _max_keys(full_chunks) > env.max_seq_len, _chunks_summary(full_chunks) + assert _same_chunks(bounded(full_chunks), alone_chunks), ( + _chunks_summary(bounded(full_chunks)), + _chunks_summary(alone_chunks), + ) + if rank == 0 and k in PACKED_STEPS: + assert _max_keys(split_chunks) > env.max_seq_len, _chunks_summary(split_chunks) + + env.key_calls.clear() + env.gathers.clear() + env.logits_calls.clear() + got = _run(env, md, step, split_chunks, split) + split_gathers, split_logits = list(env.gathers), list(env.logits_calls) + env.gathers.clear() + env.logits_calls.clear() + ref = _run(env, md, step, full_chunks, None) + full_gathers, full_logits = list(env.gathers), list(env.logits_calls) + alone = None + if alone_chunks is not None: + env.gathers.clear() + alone = _run(env, md, step, alone_chunks, None) + assert not env.gathers, env.gathers + + skip_ctx = bool(md.skip_indexer_for_ctx_reqs) + selecting = [i for i in DRIVEN if env.layers[i].indexer is not None] + expected_gathers = 0 if split is None or skip_ctx else len(selecting) + assert len(split_gathers) == expected_gathers, split_gathers + assert not full_gathers, full_gathers + for shape, dtype, sizes, group, pg in split_gathers: + assert shape == (split[1] - split[0], INDEX_TOPK) and dtype == "torch.int32", shape + assert sizes == tuple(split[2]) and group == tuple(range(env.world)) and pg is None + # No call reads more keys than a request can hold, though the engine's plans may pack more. + widest = max((n for _, n in env.key_calls), default=0) + assert widest <= env.max_seq_len, (widest, sorted(set(env.key_calls))) + + report = dict( + step=k, + name=step.name, + tc=tc, + rows=got[DRIVEN[0]].rows, + split=None if split is None else (split[0], split[1], list(split[2])), + skip_ctx=skip_ctx, + split_chunks=_chunks_summary(split_chunks), + split_chunks_as_run=_chunks_summary(bounded(split_chunks)), + full_chunks=_chunks_summary(full_chunks), + full_chunks_as_run=_chunks_summary(bounded(full_chunks)), + engine_max_keys=max(_max_keys(split_chunks), _max_keys(full_chunks)), + widest_call_keys=widest, + gathers=len(split_gathers), + split_logits=sorted(set(split_logits)), + full_logits=sorted(set(full_logits)), + layers={}, + ) + for i in DRIVEN: + g, r = got[i], ref[i] + layer = dict(selects=g.raw is not None) + mine = torch.zeros(g.rows, dtype=torch.bool, device="cuda") + every = torch.zeros(g.rows, dtype=torch.bool, device="cuda") + if g.raw is not None: + # Every row the indexer selects for was written: the context rows through the + # gather, the decode rows by the decode scorer. + _assert_overwritten(g.raw, 0, g.rows, f"step {k} layer {i} split") + _assert_overwritten(r.raw, 0, r.rows, f"step {k} layer {i} unsplit") + # This rank scored exactly its share (and the decode rows); the unsplit run every + # row. On the rows this rank scored, the scores are the unsplit ones bit for bit. + if not skip_ctx: + mine[lo:hi] = True + every[:tc] = True + if g.rows > tc and not bool(md.skip_indexer_for_gen_reqs): + mine[tc:] = True + every[tc:] = True + assert torch.equal(g.seen.scored, mine), f"step {k} layer {i}: rows scored, split" + assert torch.equal(r.seen.scored, every), f"step {k} layer {i}: rows scored, unsplit" + layer.update( + scored_rows=int(mine.sum()), + tie_rows_split=int(g.seen.tie.sum()), + tie_rows_unsplit=int(r.seen.tie.sum()), + kernel_kept_other_tied_split=int(g.seen.moved.sum()), + kernel_kept_other_tied_unsplit=int(r.seen.moved.sum()), + ) + layer.update(_compare(k, i, "unsplit", g, r, mine)) + if alone is not None: + a = alone[i] + if a.raw is not None: + _assert_overwritten(a.raw, 0, a.rows, f"step {k} layer {i} per request") + assert torch.equal(a.seen.scored, every), f"step {k} layer {i}: rows, per request" + layer["per_request"] = _compare(k, i, "per request", r, a, every) + layer["digest"] = _digest(g.sel) + report["layers"][i] = layer + log(f"step {k} layer {i}: {layer}") + + if k == NEGATIVE_STEP: + # The split's chunks without the gather: the rows of the other ranks' shares are left + # as the sentinel, and the coverage check sees them. + neg = _run(env, md, step, split_chunks, None, indexer_only=True) + for i in selecting: + left = _unwritten_rows(neg[i].raw, 0, tc) + assert left == tc - (split[1] - split[0]), (left, tc, split) + with pytest.raises(AssertionError, match="still hold the sentinel"): + _assert_overwritten(neg[i].raw, 0, tc, "negative control") + report[f"negative_layer{i}_sentinel_rows"] = left + assert not env.gathers + if k == BUDGET_STEP: + # The selections do not move with the logits budget: the 1M config's (fewer, longer + # tiles) and the 32768 config's (more, shorter ones) against the core's own. + budget = env.core.prefill_logits_budget + others = { + M._prefill_logits_budget(MAX_NUM_TOKENS, ENVELOPE_MAX_SEQ_LEN), + M._prefill_logits_budget(MAX_NUM_TOKENS, 32768), + } + assert others == {1 << 28, 1 << 24}, others + for other in sorted(others - {budget}): + at = f"2^{other.bit_length() - 1}" + env.logits_calls.clear() + env.core.prefill_logits_budget = other + try: + moved = _run(env, md, step, split_chunks, split, indexer_only=True) + finally: + env.core.prefill_logits_budget = budget + report[f"budget_{at}_logits"] = sorted(set(env.logits_calls)) + for i in selecting: + _assert_overwritten(moved[i].raw, 0, tc, f"step {k} layer {i} at {at}") + same = torch.equal(moved[i].sel, got[i].sel) + same_scores = torch.equal(moved[i].seen.sums, got[i].seen.sums) + report[f"budget_{at}_same_layer{i}"] = (same, same_scores) + assert same, f"layer {i}: selections at {at} differ from the core's {budget}" + assert same_scores, f"layer {i}: scores at {at} differ from the core's {budget}" + for rid, cached, new in step.ctx: + env.kv[rid] = cached + new + for rid in step.gen: + env.kv[rid] += 1 + return report + + +def _time_gather(env, comm) -> dict: + """What one gather of a full chunk's shares costs: 32768 context rows split over the ranks, + 8192 x 2048 int32 (64 MiB) per rank at tep4 and 4096 x 2048 (32 MiB) at tep8, over the two + nodes, into rows [0, 32768) of a selection buffer.""" + import torch + + from tensorrt_llm._torch.attention.backends.sparse.dsa import indexer as engine + + core = env.core + tc = MAX_NUM_TOKENS + q_begin, q_end, q_sizes = engine._split_prefill_queries(tc, env.mapping, Q_SPLIT_THRESHOLD) + buf = torch.randint(0, tc, (tc, INDEX_TOPK), dtype=torch.int32, device="cuda") + times, gather_only = [], [] + for it in range(25): + comm.Barrier() + a, b, c = (torch.cuda.Event(enable_timing=True) for _ in range(3)) + a.record() + gathered = env.M.allgather(buf[q_begin:q_end], q_sizes, core.tp_group, core.tp_pg) + b.record() + buf[:tc].copy_(gathered) + c.record() + torch.cuda.synchronize() + if it >= 5: + gather_only.append(a.elapsed_time(b)) + times.append(a.elapsed_time(c)) + times.sort() + gather_only.sort() + return dict( + share_rows=q_end - q_begin, + share_mib=(q_end - q_begin) * INDEX_TOPK * 4 / 2**20, + median_ms=times[len(times) // 2], + min_ms=times[0], + gather_median_ms=gather_only[len(gather_only) // 2], + gather_min_ms=gather_only[0], + ) + + +def _envelope_md(cached: list[int], new: list[int]): + """The planner's view of a step at the 1M config (max_seq_len 1048576, max_num_tokens + 32768): context requests with these cached and new lengths.""" + import torch + + cu_cached = [0] + cu_kv = [0] + for c, n in zip(cached, new): + cu_cached.append(cu_cached[-1] + c) + cu_kv.append(cu_kv[-1] + c + n) + return SimpleNamespace( + cu_seqlen_ks=torch.empty(0, dtype=torch.int32, device="cuda"), + compress_ratios=[1], + host_ctx_cached_token_indptr=torch.tensor(cu_cached), + host_ctx_kv_indptr=torch.tensor(cu_kv), + max_num_tokens=MAX_NUM_TOKENS, + kv_cache_manager=SimpleNamespace(max_seq_len=ENVELOPE_MAX_SEQ_LEN), + ) + + +def _tiles_row(M, chunk, budget: int, **extra) -> dict: + """The target's tiles of one chunk as run: the (query rows, keys) of each scorer and top-k + call, checked against the budget.""" + n_k = chunk.k_token_end - chunk.k_token_start + assert n_k <= ENVELOPE_MAX_SEQ_LEN, n_k + assert int(chunk.cu_seqlen_ke[-1]) <= n_k and int(chunk.cu_seqlen_ks[0]) >= 0 + tiles = M._prefill_tiles(chunk.token_start, chunk.token_end, n_k, budget) + assert tiles[0][0] == chunk.token_start and tiles[-1][1] == chunk.token_end + assert all(a[1] == b[0] for a, b in zip(tiles, tiles[1:])), tiles + full_rows = min(chunk.token_end - chunk.token_start, budget // n_k) + assert all(b - a == full_rows for a, b in tiles[:-1]), tiles + assert all((b - a) * n_k <= budget for a, b in tiles), tiles + return dict( + extra, + rows=(chunk.token_start, chunk.token_end), + n_k=n_k, + tiles=len(tiles), + tile_rows=sorted({b - a for a, b in tiles}, reverse=True), + topk_bucket=1 << (n_k - 1).bit_length(), + tile_mib=full_rows * n_k * 4 / 2**20, + ) + + +def _envelope_tiles(env) -> list: + """The (query rows, keys) tiles this rank's fp8_mqa_logits and prefill top-k get at the 1M + config, from the chunks the engine's own planner gives at max_seq_len 1048576 and + max_num_tokens 32768, run through the target's own cut and tiling under the budget it takes + from that engine (`_prefill_logits_budget`: 2**28): one 32768-token chunk of a + 1048576-token prompt split over the ranks, over each cached prefix; an unsplit step of 8191 + tokens whose last key is the 1048576th; two 5-token tails over 983040-token prefixes. The + top-k compiles one kernel per next power of two of the keys.""" + from tensorrt_llm._torch.attention.backends.sparse.dsa import indexer as engine + + M = env.M + budget = M._prefill_logits_budget(MAX_NUM_TOKENS, ENVELOPE_MAX_SEQ_LEN) + assert budget == M._PREFILL_LOGITS_CEILING == 1 << 28, budget + rows = [] + + def plan(cached, new, q_begin=0, q_end=None): + md = _envelope_md(cached, new) + specs = engine.plan_mla_chunked_prefill_chunks( + new, + [c + n for c, n in zip(cached, new)], + engine._packed_logits_elem_budget(md), + q_begin, + q_end, + ) + chunks = [engine.Indexer.prepare_one_prefill_chunk(md, s) for s in specs] + kv = [c + n for c, n in zip(cached, new)] + step = dict( + num_ctx_tokens=sum(new), + max_seq_len=ENVELOPE_MAX_SEQ_LEN, + host_cu_ctx_new=[0, *itertools.accumulate(new)], + host_cu_ctx_kv=[0, *itertools.accumulate(kv)], + ) + return chunks, M._indexer_blocks(chunks, step) + + new = MAX_NUM_TOKENS + q_begin, q_end, q_sizes = engine._split_prefill_queries(new, env.mapping, Q_SPLIT_THRESHOLD) + assert q_sizes == [new // env.world] * env.world, q_sizes + for cached in ENVELOPE_CACHED: + chunks, as_run = plan([cached], [new], q_begin, q_end) + assert len(chunks) == 1 and len(as_run) == 1, _chunks_summary(chunks) + chunk = as_run[0] + assert (chunk.token_start, chunk.token_end) == (q_begin, q_end) + n_k = chunk.k_token_end - chunk.k_token_start + assert n_k == cached + q_end, (n_k, cached, q_end) + assert int(chunk.cu_seqlen_ke[-1]) == n_k + rows.append(_tiles_row(M, chunk, budget, case="split 32768", cached=cached)) + + cached = ENVELOPE_MAX_SEQ_LEN - ENVELOPE_UNSPLIT + unsplit = engine._split_prefill_queries(ENVELOPE_UNSPLIT, env.mapping, Q_SPLIT_THRESHOLD) + assert unsplit[2] is None, unsplit + chunks, as_run = plan([cached], [ENVELOPE_UNSPLIT]) + assert len(as_run) == 1 and as_run[0].k_token_end - as_run[0].k_token_start == 1 << 20 + rows.append(_tiles_row(M, as_run[0], budget, case=f"unsplit {ENVELOPE_UNSPLIT}", cached=cached)) + + tails_cached = [c for c, _ in ENVELOPE_TAILS] + tails_new = [n for _, n in ENVELOPE_TAILS] + chunks, as_run = plan(tails_cached, tails_new) + packed = _max_keys(chunks) + assert len(chunks) == 1 and packed == sum(c + n for c, n in ENVELOPE_TAILS), packed + assert len(as_run) == len(ENVELOPE_TAILS), _chunks_summary(as_run) + for part in as_run: + rows.append(_tiles_row(M, part, budget, case="tails", cached=tails_cached, packed=packed)) + return rows + + +def _spawned_rank(world: int): + """This process as a rank of the `world` this pytest started under mpirun (`_rank_entry`), + on its own device: its rank and the MPI session's communicator.""" + rank = int(os.environ["OMPI_COMM_WORLD_RANK"]) + size = int(os.environ["OMPI_COMM_WORLD_SIZE"]) + import torch + + torch.cuda.set_device(int(os.environ["OMPI_COMM_WORLD_LOCAL_RANK"])) + from tensorrt_llm._utils import mpi_comm, mpi_rank, mpi_world_size + + comm = mpi_comm() + assert os.environ.get("TLLM_DISABLE_MPI") != "1", "the gather runs on the MPI session" + assert size == world == mpi_world_size() == comm.Get_size(), (size, mpi_world_size()) + assert rank == mpi_rank() == comm.Get_rank(), (rank, mpi_rank()) + return rank, comm + + +def _launched_rank(world: int): + """This pytest process as a rank of the `world` an MPI launcher started, on its own device: + its rank and the MPI session's communicator.""" + import torch + + from tensorrt_llm._utils import mpi_comm, mpi_disabled, mpi_rank + + comm = mpi_comm() + rank, size = mpi_rank(), comm.Get_size() + torch.cuda.set_device(int(os.environ["OMPI_COMM_WORLD_LOCAL_RANK"])) + assert not mpi_disabled(), "the gather runs on the MPI session" + assert size == world == get_mpi_world_size(), (size, get_mpi_world_size()) + return rank, comm + + +def _rank_main(ckpt_path: Path, world: int, rank: int, comm) -> None: + import torch + + device = torch.cuda.current_device() + devices = comm.allgather(str(torch.cuda.get_device_properties(device).uuid)) + assert len(set(devices)) == world, "every rank must own its own device" + hosts = comm.allgather(socket.gethostname()) + + def log(msg: str) -> None: + print(f"[rank {rank}] {msg}", flush=True) + + log(f"on {hosts[rank]} device {device}; ranks over {len(set(hosts))} node(s): {hosts}") + started = time.monotonic() + env = _build(ckpt_path, rank, world) + _instrument(env) + assert env.core.tp_group == list(range(world)) and env.core.tp_pg is None + # Nothing is sized before the first step (`_check_step` hands the core its metadata). + assert (env.core.prefill_logits_budget, env.core.ctx_workspace_rows) == (0, 0) + assert _sparse_config().q_split_threshold == Q_SPLIT_THRESHOLD + L = env.layers + assert L[1].indexer.rope and not L[3].indexer.rope and L[17].indexer is None + log(f"built in {time.monotonic() - started:.1f}s") + reports = [] + for k, step in enumerate(env.steps): + t0 = time.monotonic() + reports.append(_check_step(env, k, step, log)) + log(f"step {k} ({step.name}) passed in {time.monotonic() - t0:.1f}s") + # The sizes the first step's metadata gave: the logits budget at max_num_tokens 32768 and + # the world's max_seq_len, and, chunked prefill on, the held workspace's 2**19 rows. + budget = env.M._prefill_logits_budget(MAX_NUM_TOKENS, env.max_seq_len) + assert env.core.prefill_logits_budget == budget == {4: 1 << 26, 8: 1 << 24}[world], budget + assert env.core.ctx_workspace_rows == env.M._CTX_WORKSPACE_ROWS == 1 << 19 + assert env.prefill._ws_buf.shape == (1 << 19, LATENT) + log(f"sized on the first step: logits budget {budget}, held context workspace 2**19 rows") + timing = _time_gather(env, comm) + tiles = _envelope_tiles(env) + + everyone = comm.allgather( + dict(host=hosts[rank], reports=reports, timing=timing, envelope_tiles=tiles) + ) + for k in range(len(env.steps)): + for i in DRIVEN: + digests = {r["reports"][k]["layers"][i]["digest"] for r in everyone} + assert len(digests) == 1, f"step {k} layer {i}: the ranks hold different selections" + if rank == 0: + print(json.dumps(dict(ranks=everyone), indent=1, default=str), flush=True) + for r, member in enumerate(everyone): + for t in member["envelope_tiles"]: + print(f"[envelope tiles] rank {r} {t}", flush=True) + print(f"[gather] rank {r} {member['host']} {member['timing']}", flush=True) + print(f"[q-split] every check passed on every rank, all {world}", flush=True) + comm.Barrier() + + +def _rank_entry() -> None: + """A rank `_start_ranks` started: ` --q-split-rank `.""" + try: + flag = sys.argv.index(_RANK_FLAG) + world = int(sys.argv[flag + 2]) + rank, comm = _spawned_rank(world) + _rank_main(Path(sys.argv[flag + 1]), world, rank, comm) + sys.stdout.flush() + # mpirun fails a rank that exits without finalizing. + from mpi4py import MPI + + MPI.Finalize() + except BaseException: + print(f"[rank {os.environ.get('OMPI_COMM_WORLD_RANK')}] FAILED", flush=True) + traceback.print_exc() + sys.stdout.flush() + sys.stderr.flush() + os._exit(13) + os._exit(0) + + +def _finalize(config, session) -> None: + """After pytest's report: MPI finalized (mpirun fails a rank that leaves without it), then + out at once with pytest's own status, ahead of the interpreter's teardown of the ranks' + communicators.""" + from mpi4py import MPI + + config.get_terminal_writer().flush() + for stream in (sys.stdout, sys.stderr, sys.__stdout__, sys.__stderr__): + stream.flush() + MPI.Finalize() + os._exit(int(session.exitstatus)) + + +# ---------------------------------------------------------------- the launcher (pytest) + + +def _checkpoint_dir() -> Path | None: + from test_common.llm_data import llm_models_root + + root = llm_models_root() + if root is None: + return None + for name in CHECKPOINTS: + path = Path(root) / name + if (path / "params.json").is_file() and (path / SHARD_INDEX).is_file(): + return path + return None + + +def _session_members(sid: int) -> list[int]: + """Every process in session `sid`, read from /proc.""" + members = [] + for entry in Path("/proc").iterdir(): + if not entry.name.isdigit(): + continue + try: + stat = (entry / "stat").read_text() + except OSError: + continue + fields = stat[stat.rindex(")") + 2 :].split() + if int(fields[3]) == sid: + members.append(int(entry.name)) + return members + + +def _end_job(job) -> None: + """End mpirun and every rank it started (each rank sits in a process group of its own, + inside mpirun's session).""" + if job.poll() is None: + job.terminate() + try: + job.wait(timeout=30) + except subprocess.TimeoutExpired: + pass + for pid in _session_members(job.pid): + try: + os.kill(pid, signal.SIGKILL) + except ProcessLookupError: + pass + if job.poll() is None: + job.kill() + job.wait() + + +def _start_ranks(path: Path, world: int, tmp_path: Path) -> None: + """tep4: the `world` ranks on this node, started under mpirun as `_rank_entry`, their log + read back.""" + import torch + + if torch.cuda.device_count() < world: + pytest.skip(f"needs {world} GPUs, {torch.cuda.device_count()} visible") + mpirun = shutil.which("mpirun") + assert mpirun is not None, "the ranks start under mpirun; none on PATH" + repo_root = Path(__file__).resolve().parents[7] + assert (repo_root / "tensorrt_llm").is_dir(), repo_root + env = { + k: v + for k, v in os.environ.items() + if not k.startswith(("PMIX_", "PMI_", "OMPI_", "SLURM_")) and k != "TLLM_DISABLE_MPI" + } + env["PYTHONPATH"] = os.pathsep.join(p for p in (str(repo_root), env.get("PYTHONPATH", "")) if p) + command = [mpirun, "-n", str(world), "--oversubscribe", "--allow-run-as-root"] + command += [sys.executable, str(Path(__file__).resolve()), _RANK_FLAG, str(path), str(world)] + log_path = tmp_path / "ranks.log" + with open(log_path, "w") as log: + job = subprocess.Popen( + command, env=env, stdout=log, stderr=subprocess.STDOUT, start_new_session=True + ) + try: + code = job.wait(timeout=DEADLINE_S) + except subprocess.TimeoutExpired: + _end_job(job) + code = None + output = log_path.read_text() + print(output) + assert code is not None, ( + f"the {world} ranks did not finish in {DEADLINE_S}s:\n{output[-20000:]}" + ) + assert code == 0, f"mpirun exited {code}:\n{output[-20000:]}" + assert "[q-split] every check passed on every rank" in output + + +def _run_as_rank(path: Path, world: int, request) -> None: + """tep8: this pytest process as one of the `world` ranks.""" + request.config.add_cleanup(lambda: _finalize(request.config, request.session)) + faulthandler.dump_traceback_later(DEADLINE_S, exit=True) + try: + rank, comm = _launched_rank(world) + _rank_main(path, world, rank, comm) + except BaseException: + # The other ranks wait in a collective for this one: leave now, so the launcher ends + # the job instead of the deadline. + print(f"[rank {os.environ.get('OMPI_COMM_WORLD_RANK')}] FAILED", flush=True) + traceback.print_exc() + sys.stdout.flush() + sys.stderr.flush() + os._exit(13) + finally: + faulthandler.cancel_dump_traceback_later() + + +@pytest.mark.no_xdist +@pytest.mark.parametrize("world", WORLDS) +def test_indexer_q_split_on_every_rank(world, request): + """Every check above on every rank of the world: at tep4 the four ranks this pytest starts, + at tep8 the eight launched pytest processes, this one among them.""" + path = _checkpoint_dir() + if path is None: + pytest.skip("checkpoint Mistral-Large-4-1T-A52B-NVFP4 is not under LLM_MODELS_ROOT") + if world > RANKS_PER_NODE: + _run_as_rank(path, world, request) + else: + _start_ranks(path, world, request.getfixturevalue("tmp_path")) + + +if __name__ == "__main__" and _RANK_FLAG in sys.argv: + _rank_entry() diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_long_context.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_long_context.py new file mode 100644 index 000000000000..32a0d37c8064 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_long_context.py @@ -0,0 +1,259 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The long-context witness of Mistral-Large-4-1T-A52B-NVFP4 on its sm_103 tep target, at tep4 +(four ranks) and at tep8 (eight ranks over two nodes of four). + +GSM8K's prompts never pass the indexer's `index_topk` (2048 positions). Below it the engine +skips the indexer and sparse MLA attends over the causal window, so the accuracy gate never +runs the indexer's selection, nor SSMax, LeRoPE and YaRN at long positions, nor the FlashMLA +sparse kernels on a selection past it (each called on the rank's heads -- 16 at tep4, 8 at +tep8 -- first in the 64-head query they take). Here the whole model runs under its serving +config -- max_seq_len = max_num_tokens = 32768, chunked prefill off, the latent KV cache at +its default (the fp8_ds_mla pool), CUDA graphs over batches 1, 2, 4, 8 and 16, padded -- and +greedy decoding must recall a passkey hidden in plain filler text: + +* in prompts of about 4K, 8K, 16K and 31K tokens, at three depths each, one prompt at a time: + each prompt is one context step of one request (read back from the engine's iteration + statistics), then decodes in the batch-1 graph; +* in sixteen prompts whose lengths sum to nearly 32768 tokens, six of them past `index_topk` + and ten within it, admitted together: they are one context step of sixteen requests (read + back the same way) holding the whole token budget, then decode in the batch-16 graph. + +Text only: no image, no chat template. Two engine options beyond the serving ones make the +steps observable and do not change what a step computes: iteration statistics, and a request +queue that waits up to `BATCH_WAIT_MS` for a full batch so the sixteen prompts are admitted +in one step rather than split by their arrival. + +Runs with the ranks already in place, on four GB300s at tep4 or on eight over two nodes of four +at tep8: pytest under `trtllm-llmapi-launch`, as the whole-model accuracy class runs, with the +checkpoint under `LLM_MODELS_ROOT` as `Mistral-Large-4-1T-A52B-NVFP4` (or +`mistral-large-4-nvfp4`) as published, Mistral-native (params.json, tekken.json and the +consolidated safetensors), which the engine loads with checkpoint_format "mistral". One +scheduler serves all the ranks, so every step below is one step on every rank. Skipped where the +checkpoint is absent or off sm_103; each world size is skipped below its own ranks (four GPUs at +tep4, eight ranks at tep8) and inside a launch of another world size. +""" + +from __future__ import annotations + +import random +from pathlib import Path + +import pytest +import torch +from test_common.llm_data import llm_models_root +from utils.util import skip_num_gpus_less_than + +from tensorrt_llm import LLM, SamplingParams +from tensorrt_llm._torch.modeling_v2 import MODELING_V2_VALIDATE_ENV +from tensorrt_llm._utils import get_sm_version +from tensorrt_llm.inputs import TokensPrompt +from tensorrt_llm.llmapi import CudaGraphConfig, KvCacheConfig +from tensorrt_llm.llmapi.mpi_session import get_mpi_world_size + + +def _ranks() -> int: + """The ranks this process can serve: the MPI world a launcher started (eight over two nodes + of four, where each node sees four devices), or, unlaunched, the devices the LLM API spawns + its workers on -- as the integration conftest's `skip_less_mpi_world_size` counts.""" + world = get_mpi_world_size() + return torch.cuda.device_count() if world == 1 else world + + +def _launched_elsewhere(world: int) -> bool: + """A launcher started an MPI world of another size: an engine of `world` ranks does not + start inside it.""" + return get_mpi_world_size() not in (1, world) + + +#: The engine's world sizes, each gated on its own ranks: tep4 on four GPUs, tep8 on eight +#: ranks over two nodes of four. +WORLDS = [ + pytest.param( + 4, + marks=[ + skip_num_gpus_less_than(4), + pytest.mark.skipif( + _launched_elsewhere(4), reason="tep4 needs a launch of four ranks or none" + ), + ], + id="tep4", + ), + pytest.param( + 8, + marks=[ + pytest.mark.skipif(_ranks() < 8, reason="tep8 needs eight ranks"), + pytest.mark.skipif( + _launched_elsewhere(8), reason="tep8 needs a launch of eight ranks or none" + ), + ], + id="tep8", + ), +] + +pytestmark = [ + pytest.mark.skipif( + get_sm_version() != 103, + reason="the mistral_large4 modeling_v2 targets are certified on sm_103 only", + ), + # One engine serves the module: the LLM API starts its result-dispatch thread on the first + # request, inside the first test, and stops it when the fixture shuts the engine down. + pytest.mark.threadleak(enabled=False), +] + +CHECKPOINTS = ("Mistral-Large-4-1T-A52B-NVFP4", "mistral-large-4-nvfp4") + +# The serving config: tep4 or tep8 (attention tensor parallel and the routed experts expert +# parallel over all the ranks), sixteen sequences, unchunked steps of up to 32768 tokens. +MAX_NUM_TOKENS = 32768 +ENGINE = dict( + enable_attention_dp=False, + max_batch_size=16, + max_num_tokens=MAX_NUM_TOKENS, + max_seq_len=MAX_NUM_TOKENS, + enable_chunked_prefill=False, + cuda_graph_config=CudaGraphConfig(max_batch_size=16, enable_padding=True), + kv_cache_config=KvCacheConfig(enable_block_reuse=False, free_gpu_memory_fraction=0.85), +) +# How long the request queue holds an iteration for more requests while fewer than +# max_batch_size are waiting; every iteration pays it, decode steps included. The sixteen +# prompts are submitted together as token ids and arrive within milliseconds of each other. +BATCH_WAIT_MS = 1000 + +INDEX_TOPK = 2048 +# Prompts of about 4K, 8K, 16K and 31K tokens, a passkey at three depths of each. +LENGTHS = (4096, 8192, 16384, 31744) +DEPTHS = (0.1, 0.5, 0.9) +# Sixteen prompts filling one 32768-token context step: six past index_topk, ten within it. +# Each is built to at most its length and within one filler sentence of it, so the step holds +# between 98% of the budget and 32700 tokens. +BATCH_PAST_TOPK = (8500, 4000, 3000, 2600, 2300, 2100) +BATCH_WITHIN_TOPK = (1900, 1700, 1500, 1300, 1100, 900, 700, 500, 350, 250) +BATCH_LENGTHS = BATCH_PAST_TOPK + BATCH_WITHIN_TOPK +BATCH_FILL = 0.98 +BATCH_DEPTHS = (0.1, 0.3, 0.5, 0.7, 0.9) + +HEAD = ( + "There is an important piece of information hidden inside a lot of irrelevant text. " + "Find it and memorize it. I will quiz you about it.\n\n" +) +FILLER = ( + "The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again. " +) +QUESTION = "\nWhat is the pass key? The pass key is" +# A five-digit key and its closing punctuation fit with room to spare. +ANSWER = SamplingParams(max_tokens=12, temperature=0.0) + + +def _checkpoint_dir() -> Path | None: + """The checkpoint's directory, found by its Mistral-native config, params.json.""" + root = llm_models_root() + if root is None: + return None + for name in CHECKPOINTS: + if (Path(root) / name / "params.json").is_file(): + return Path(root) / name + return None + + +def _prompt(reps: int, depth: float, key: int) -> str: + before = round(reps * depth) + needle = f"\nThe pass key is {key}. Remember it. {key} is the pass key.\n" + return HEAD + FILLER * before + needle + FILLER * (reps - before) + QUESTION + + +def _passkey_ids(tokenizer, tokens: int, depth: float, key: int) -> list[int]: + """The prompt's token ids, BOS included as the LLM API's own tokenization adds it, at most + `tokens` long and within one filler sentence of it.""" + unit = len(tokenizer.encode(FILLER * 64)) / 64 + bare = len(tokenizer.encode(_prompt(0, depth, key))) + reps = max(0, int((tokens - bare) / unit)) + ids = tokenizer.encode(_prompt(reps, depth, key)) + while len(ids) > tokens and reps > 0: + reps -= 1 + ids = tokenizer.encode(_prompt(reps, depth, key)) + return ids + + +def _keys(count: int, seed: int) -> list[int]: + return random.Random(seed).sample(range(10000, 100000), count) + + +def _context_steps(llm: LLM) -> list[tuple[int, int]]: + """(context requests, context tokens) of each iteration since the last call that ran any + context rows.""" + steps = [] + for record in llm.get_stats(timeout=2): + ifb = record.get("inflightBatchingStats") or {} + if ifb.get("numContextRequests", 0): + steps.append((ifb["numContextRequests"], ifb["numCtxTokens"])) + return steps + + +@pytest.fixture(scope="module", params=WORLDS) +def llm(request): + world = request.param + path = _checkpoint_dir() + if path is None: + pytest.skip(f"checkpoint {CHECKPOINTS} is not under LLM_MODELS_ROOT ({llm_models_root()})") + with LLM( + str(path), + checkpoint_format="mistral", + modeling_v2="require", + env_overrides={MODELING_V2_VALIDATE_ENV: "1"}, + enable_iter_perf_stats=True, + batch_wait_timeout_ms=BATCH_WAIT_MS, + tensor_parallel_size=world, + moe_expert_parallel_size=world, + **ENGINE, + ) as engine: + capacity = engine.get_kv_cache_capacity() + print(f"LONGCTX kv_cache_capacity {capacity}", flush=True) + _context_steps(engine) + yield engine + + +@pytest.mark.parametrize("tokens", LENGTHS) +def test_passkey_alone_in_one_context_step(llm, tokens): + """A passkey at three depths of a prompt of about `tokens` tokens, each prompt one context + step of one request, recalled by greedy decoding.""" + misses = [] + for depth, key in zip(DEPTHS, _keys(len(DEPTHS), seed=tokens)): + ids = _passkey_ids(llm.tokenizer, tokens, depth, key) + assert INDEX_TOPK < len(ids) <= tokens, (tokens, len(ids)) + answer = llm.generate(TokensPrompt(prompt_token_ids=ids), ANSWER).outputs[0].text + steps = _context_steps(llm) + print( + f"LONGCTX alone tokens={len(ids)} depth={depth} key={key} answer={answer!r} " + f"context_steps={steps}", + flush=True, + ) + assert steps == [(1, len(ids))], (len(ids), depth, steps) + if str(key) not in answer: + misses.append((len(ids), depth, key, answer)) + assert not misses, misses + + +def test_sixteen_prompts_fill_one_context_step(llm): + """Sixteen passkey prompts admitted together are one context step of sixteen requests + holding nearly the whole 32768-token budget, and every one recalls its key.""" + keys = _keys(len(BATCH_LENGTHS), seed=MAX_NUM_TOKENS) + prompts = [ + _passkey_ids(llm.tokenizer, tokens, BATCH_DEPTHS[i % len(BATCH_DEPTHS)], key) + for i, (tokens, key) in enumerate(zip(BATCH_LENGTHS, keys)) + ] + lengths = [len(ids) for ids in prompts] + total = sum(lengths) + assert sum(n > INDEX_TOPK for n in lengths) == len(BATCH_PAST_TOPK), lengths + assert BATCH_FILL * MAX_NUM_TOKENS <= total <= MAX_NUM_TOKENS, (total, lengths) + + outputs = llm.generate([TokensPrompt(prompt_token_ids=ids) for ids in prompts], ANSWER) + steps = _context_steps(llm) + answers = [out.outputs[0].text for out in outputs] + for n, key, answer in zip(lengths, keys, answers): + print(f"LONGCTX batch tokens={n} key={key} answer={answer!r}", flush=True) + print(f"LONGCTX batch total={total} context_steps={steps}", flush=True) + + assert steps == [(len(prompts), total)], (total, steps) + misses = [(n, key, a) for n, key, a in zip(lengths, keys, answers) if str(key) not in a] + assert not misses, misses diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_retrieval_at_length.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_retrieval_at_length.py new file mode 100644 index 000000000000..150c276b80b6 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_retrieval_at_length.py @@ -0,0 +1,606 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Retrieval at length: Mistral-Large-4-1T-A52B-NVFP4 on its sm_103 tep target at the 1M config, +at tep4 (four ranks) and at tep8 (eight ranks over two nodes of four). + +The 1M config: max_seq_len 1048576 (the checkpoint's max_position_embeddings), chunked prefill in +steps of max_num_tokens 32768, sixteen sequences, the fp8_ds_mla latent KV cache at its +default dtype, CUDA graphs over batches 1, 2, 4, 8 and 16, padded. A passkey hidden in plain +filler text at early, middle and late depths (0.1, 0.5, 0.9) of prompts of exactly 32768, +131072, 524288 and 1040000 tokens at tep4 (1040384 at tep8), one request at a time, must be +recalled by greedy decoding through the LLM API. Every chunk after a prompt's first is a +context step over a cached prefix: the rope append at the prefix's positions, the indexer's key +gather and scoring over the whole prefix, sparse MLA over selections that reach back to +position 0. Each full chunk is 32768 rows, so the engine splits its indexer queries over every +TP rank (q_split_threshold 8192; 8192 rows a rank at tep4, 4096 at tep8) and the target +all-gathers the selections, across both nodes at tep8; the longest prompts run fourteen such +chunks over cached prefixes past 524288. + +Sparse MLA takes one of two paths per step. While the step's context KV (the request's cached and +new rows) fits the held context workspace, 2**19 rows at this config (`_ctx_workspace_rows`: chunked +prefill lets a step attend cached prefixes), those rows are dequantized out of the fp8_ds_mla pool +into it for the sparse prefill kernel; the 524288-token prompt's last chunk reaches exactly 2**19 +and stays there. Past it -- the longest prompt's last 16 chunks -- every context row attends +straight off the pool as a one-row sequence of the sparse decode kernel, in sub-batches of 8192 +rows. At tep8 the longest prompt is 1040384 tokens, 31 full chunks and a last one of three full +sub-batches, the sub-batch size the decode kernel's tep8 1M cells certify; at tep4 it is 1040000, +whose last chunk ends on a sub-batch of 7808 rows. + +Per prompt, reported: the time to the first token and the decode time per token (sixteen +tokens, streamed), the context steps (read back from the engine's iteration statistics), and +the peak device memory of every rank while it runs, KV pool included. Torch's allocator +statistics miss the pool, so device-wide memory is read: this process polls +`torch.cuda.mem_get_info` on the four devices of its node (ranks 0-3), and every rank's own +process reads its device through NVML (`_retrieval_at_length_ranks.py`; at tep8 ranks 4-7 are +on the other node). The decode scorer's column count is frozen at max_seq_len, so a decode step +may cost more at this config than at the delivered one whatever the sequence's length: two short +prompts (one within the indexer's `index_topk`, one past it) are decoded for 128 tokens at this +config and at the delivered configuration (max_seq_len = max_num_tokens = 32768, chunked prefill +off), one engine after the other. The engines are named "envelope" (the 1M config) and +"delivered" in the printed lines. + +What each rank's context steps received is read through an observe-only record on every rank, +installed and collected through the launcher's MPI session (the engines' own channel, idle +between them): the query split, the indexer chunk plan, the cached prefixes, the selection +gathers, and the most keys any indexer key gather, scorer or prefill top-k call received. +Outside the launcher there is no such session: the engines still run, and the checks on the +records are skipped. + +Text only: no image, no chat template. Runs with the ranks already in place, on four GB300s at +tep4 or on eight over two nodes of four at tep8: pytest under `trtllm-llmapi-launch`, as the +whole-model accuracy class runs, with the checkpoint under `LLM_MODELS_ROOT` as +`Mistral-Large-4-1T-A52B-NVFP4` (or `mistral-large-4-nvfp4`) in its Mistral-native format, +which both engines load with checkpoint_format "mistral". Skipped where the checkpoint +is absent or off sm_103; each world size is skipped below its own ranks (four GPUs at tep4, eight +ranks at tep8) and inside a launch of another world size. About 25 minutes, two engine loads +included. +""" + +from __future__ import annotations + +import random +import site +import threading +import time +from pathlib import Path +from types import SimpleNamespace + +import pytest +import torch +from test_common.llm_data import llm_models_root +from utils.util import skip_num_gpus_less_than + +from tensorrt_llm import LLM, SamplingParams +from tensorrt_llm._torch.modeling_v2 import MODELING_V2_VALIDATE_ENV +from tensorrt_llm._torch.models.checkpoints.mistral.config_loader import MistralConfigLoader +from tensorrt_llm._utils import get_sm_version +from tensorrt_llm.executor.utils import create_mpi_comm_session, get_spawn_proxy_process_env +from tensorrt_llm.inputs import TokensPrompt +from tensorrt_llm.llmapi import CudaGraphConfig, KvCacheConfig +from tensorrt_llm.llmapi.mpi_session import get_mpi_world_size + +from . import _retrieval_at_length_ranks as rank_side + + +def _ranks() -> int: + """The ranks this process can serve: the MPI world a launcher started (eight over two nodes + of four, where each node sees four devices), or, unlaunched, the devices the LLM API spawns + its workers on -- as the integration conftest's `skip_less_mpi_world_size` counts.""" + world = get_mpi_world_size() + return torch.cuda.device_count() if world == 1 else world + + +def _launched_elsewhere(world: int) -> bool: + """A launcher started an MPI world of another size: an engine of `world` ranks does not + start inside it.""" + return get_mpi_world_size() not in (1, world) + + +#: The engines' world sizes, each gated on its own ranks: tep4 on four GPUs, tep8 on eight +#: ranks over two nodes of four. +WORLDS = [ + pytest.param( + 4, + marks=[ + skip_num_gpus_less_than(4), + pytest.mark.skipif( + _launched_elsewhere(4), reason="tep4 needs a launch of four ranks or none" + ), + ], + id="tep4", + ), + pytest.param( + 8, + marks=[ + pytest.mark.skipif(_ranks() < 8, reason="tep8 needs eight ranks"), + pytest.mark.skipif( + _launched_elsewhere(8), reason="tep8 needs a launch of eight ranks or none" + ), + ], + id="tep8", + ), +] + +pytestmark = [ + pytest.mark.skipif( + get_sm_version() != 103, + reason="the mistral_large4 modeling_v2 targets are certified on sm_103 only", + ), + # The module's engines start the LLM API's result-dispatch thread on their first request + # and stop it when the fixture shuts them down; the memory poller is a daemon thread. + pytest.mark.threadleak(enabled=False), +] + +CHECKPOINTS = ("Mistral-Large-4-1T-A52B-NVFP4", "mistral-large-4-nvfp4") + +MAX_SEQ_LEN = 1048576 +CHUNK = 32768 +DELIVERED_LEN = 32768 +INDEX_TOPK = 2048 +# sparse_attention_config.q_split_threshold's default (llmapi/llm_args.py): a step of at least +# this many context rows is split over the TP ranks. +Q_SPLIT_THRESHOLD = 8192 +# A cached prefix past this is "at length" for the split's check. +FAR_PREFIX = 524288 + +COMMON = dict( + enable_attention_dp=False, + max_batch_size=16, + cuda_graph_config=CudaGraphConfig(max_batch_size=16, enable_padding=True), + kv_cache_config=KvCacheConfig(enable_block_reuse=False, free_gpu_memory_fraction=0.85), +) +ENGINES = { + "envelope": dict(max_seq_len=MAX_SEQ_LEN, max_num_tokens=CHUNK, enable_chunked_prefill=True), + "delivered": dict( + max_seq_len=DELIVERED_LEN, max_num_tokens=DELIVERED_LEN, enable_chunked_prefill=False + ), +} + +# Prompt lengths in tokens, BOS included, by world; the longest leaves room for the answer +# within max_seq_len. At tep8 the longest one's last chunk is 3 x 8192 rows, so every +# decode-path sub-batch is full. +LENGTHS = {4: (32768, 131072, 524288, 1040000), 8: (32768, 131072, 524288, 1040384)} +# The lengths' test ids, one per place in LENGTHS at either world. +LENGTH_IDS = ("32k", "128k", "512k", "longest") +DEPTHS = (0.1, 0.5, 0.9) +# Short prompts decoded at both engines: one within index_topk (the engine skips the indexer +# and its decode graph has no scorer), one past it (the decode scorer runs over max_seq_len +# columns). +SHORT = (1000, 8000) + +HEAD = ( + "There is an important piece of information hidden inside a lot of irrelevant text. " + "Find it and memorize it. I will quiz you about it.\n\n" +) +FILLER = ( + "The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again. " +) +QUESTION = "\nWhat is the pass key? The pass key is" +# The key's tokens come first; past them the model's text is a near tie no engine repeats, so +# only the key is checked. A fixed count of tokens times the decode. +NEEDLE = SamplingParams(max_tokens=16, temperature=0.0, ignore_eos=True) +SHORT_DECODE = SamplingParams(max_tokens=128, temperature=0.0, ignore_eos=True) + +GIB = 2**30 + + +# ------------------------------------------------------------------- every rank, between engines + + +def _reach_ranks(session) -> None: + """Make the rank side importable on every rank: a task's function is unpickled by module + path before the session hands the rank this process's `sys.path`, so first add the + directory that holds the rank side's top-level package.""" + base = Path(rank_side.__file__).resolve().parents[rank_side.__name__.count(".")] + session.submit_sync(site.addsitedir, str(base)) + + +def _probe(session, action: str, world: int) -> dict[int, dict] | None: + """`rank_probe(action)` on every rank, by rank; None without the launcher's session.""" + if session is None: + return None + reports = {r["rank"]: r for r in session.submit_sync(rank_side.rank_probe, action)} + assert sorted(reports) == list(range(world)), sorted(reports) + return reports + + +def _peak_used(changes: list[tuple[float, int]], start: float, end: float) -> int | None: + """The most bytes a device held over [start, end]: the value in effect at `start` and every + change up to `end`.""" + before = [used for t, used in changes if t <= start] + inside = [used for t, used in changes if start < t <= end] + values = before[-1:] + inside + return max(values) if values else None + + +# ------------------------------------------------------------------------------- the prompts + + +def _checkpoint_dir() -> Path | None: + """The checkpoint's directory, found by its Mistral-native config, params.json.""" + root = llm_models_root() + if root is None: + return None + for name in CHECKPOINTS: + if (Path(root) / name / "params.json").is_file(): + return Path(root) / name + return None + + +def _prompt(reps: int, depth: float, key: int) -> str: + before = round(reps * depth) + needle = f"\nThe pass key is {key}. Remember it. {key} is the pass key.\n" + return HEAD + FILLER * before + needle + FILLER * (reps - before) + QUESTION + + +def _passkey_ids(tokenizer, tokens: int, depth: float, key: int) -> list[int]: + """Exactly `tokens` ids, BOS included as the LLM API's own tokenization adds it: the prompt + with the fewest whole filler sentences that reach `tokens`, tokenized as one text, then the + few ids past `tokens` taken off the filler right after the head. A long run of filler + tokenizes a little shorter per sentence than a short one, so the count is topped up from + what the whole text gives.""" + head = tokenizer.encode(HEAD) + unit = len(tokenizer.encode(FILLER * 65)) - len(tokenizer.encode(FILLER * 64)) + bare = len(tokenizer.encode(_prompt(0, depth, key))) + reps = max(0, -(-(tokens - bare) // unit)) + ids = tokenizer.encode(_prompt(reps, depth, key)) + while len(ids) < tokens: + reps += max(1, (tokens - len(ids)) // unit) + ids = tokenizer.encode(_prompt(reps, depth, key)) + excess = len(ids) - tokens + assert ids[: len(head)] == head and excess < 4 * unit <= round(reps * depth) * unit, ( + tokens, + excess, + unit, + ) + return ids[: len(head)] + ids[len(head) + excess :] + + +def _keys(count: int, seed: int) -> list[int]: + return random.Random(seed).sample(range(10000, 100000), count) + + +def _chunks(n: int) -> list[tuple[int, int]]: + """The context steps of one `n`-token prompt alone at the 1M config.""" + return [(1, min(CHUNK, n - lo)) for lo in range(0, n, CHUNK)] + + +def _shares(tc: int, rank: int, world: int) -> tuple[int, int, list[int]]: + """`_split_prefill_queries` at tp `world`, attention DP off: rank r's + [tc*r//world, tc*(r+1)//world).""" + bounds = [tc * r // world for r in range(world + 1)] + return bounds[rank], bounds[rank + 1], [bounds[r + 1] - bounds[r] for r in range(world)] + + +def _context_steps(llm: LLM) -> list[tuple[int, int]]: + """(context requests, context tokens) of each iteration since the last call that ran any + context rows.""" + steps = [] + for record in llm.get_stats(timeout=2): + ifb = record.get("inflightBatchingStats") or {} + if ifb.get("numContextRequests", 0): + steps.append((ifb["numContextRequests"], ifb["numCtxTokens"])) + return steps + + +class _DeviceMemory: + """The least free memory of every device of this node since the last `reset`, polled with + `torch.cuda.mem_get_info` from a thread of this process (not an engine's): device-wide, so + it counts what every rank's process holds, the KV pool and the workspaces outside torch's + allocator included. Device d of this node runs rank d.""" + + def __init__(self, period_s: float = 0.02) -> None: + self.devices = list(range(torch.cuda.device_count())) + self.total = [torch.cuda.mem_get_info(d)[1] for d in self.devices] + self._period = period_s + self._lock = threading.Lock() + self._stop = threading.Event() + self._least = self._free() + self._thread = threading.Thread(target=self._run, daemon=True) + self._thread.start() + + def _free(self) -> list[int]: + return [torch.cuda.mem_get_info(d)[0] for d in self.devices] + + def _run(self) -> None: + while not self._stop.is_set(): + free = self._free() + with self._lock: + self._least = [min(a, b) for a, b in zip(self._least, free)] + self._stop.wait(self._period) + + def reset(self) -> list[float]: + """Start a new window; the used GiB per device now.""" + free = self._free() + with self._lock: + self._least = list(free) + return [round((t - f) / GIB, 2) for t, f in zip(self.total, free)] + + def peak(self) -> list[float]: + """The most GiB used per device in this window.""" + free = self._free() + with self._lock: + least = [min(a, b) for a, b in zip(self._least, free)] + return [round((t - f) / GIB, 2) for t, f in zip(self.total, least)] + + def stop(self) -> None: + self._stop.set() + self._thread.join() + + +def _generate(llm: LLM, ids: list[int], params: SamplingParams, memory: _DeviceMemory): + """One request alone, streamed: its text, its tokens, the client-side time to the first + token and decode time per token, the engine's own first-token and last-token times, its + context steps, this node's devices' peak memory while it ran and its wall-clock window.""" + before = memory.reset() + start, wall_start = time.perf_counter(), time.time() + first = None + result = llm.generate_async(TokensPrompt(prompt_token_ids=ids), params, streaming=True) + for partial in result: + if first is None and partial.outputs[0].token_ids: + first = time.perf_counter() + end, wall_end = time.perf_counter(), time.time() + peak = memory.peak() + out = result.outputs[0] + n = len(out.token_ids) + engine_ttft = engine_tpot = None + metrics = out.request_perf_metrics + if metrics is not None and metrics.timing_metrics is not None: + timing = metrics.timing_metrics + arrival = timing.arrival_time.total_seconds() + first_token = timing.first_token_time.total_seconds() + last_token = timing.last_token_time.total_seconds() + if first_token > 0 and arrival > 0: + engine_ttft = round(first_token - arrival, 3) + if n > 1 and last_token > first_token > 0: + engine_tpot = round((last_token - first_token) / (n - 1), 5) + return SimpleNamespace( + tokens=len(ids), + text=out.text, + generated=n, + ttft_s=round(first - start, 3), + tpot_s=round((end - first) / max(n - 1, 1), 5), + engine_ttft_s=engine_ttft, + engine_tpot_s=engine_tpot, + steps=_context_steps(llm), + used_before_gib=before, + peak_gib=peak, + window=(wall_start, wall_end), + rank_peak_gib=None, + ) + + +def _engine(path: Path, name: str, world: int) -> LLM: + return LLM( + str(path), + checkpoint_format="mistral", + modeling_v2="require", + env_overrides={MODELING_V2_VALIDATE_ENV: "1"}, + enable_iter_perf_stats=True, + return_perf_metrics=True, + tensor_parallel_size=world, + moe_expert_parallel_size=world, + **COMMON, + **ENGINES[name], + ) + + +def _rank_peaks( + reports: dict[int, dict] | None, window: tuple[float, float] +) -> list[float | None] | None: + """Every rank's most device memory over `window`, GiB, from its own NVML record; None + without the records.""" + if reports is None: + return None + peaks = [] + for rank in sorted(reports): + used = _peak_used(reports[rank]["memory"], *window) + peaks.append(None if used is None else round(used / GIB, 2)) + return peaks + + +def _report(label: str, r) -> None: + print( + f"RETRIEVAL {label} tokens={r.tokens} ttft_s={r.ttft_s} tpot_s={r.tpot_s} " + f"engine_ttft_s={r.engine_ttft_s} engine_tpot_s={r.engine_tpot_s} " + f"generated={r.generated} context_steps={len(r.steps)} " + f"steps={r.steps if len(r.steps) <= 4 else (r.steps[:2], '...', r.steps[-2:])} " + f"used_before_gib(this node, mem_get_info)={r.used_before_gib} " + f"peak_gib(this node, mem_get_info)={r.peak_gib} " + f"peak_gib(every rank, nvml)={r.rank_peak_gib} text={r.text!r}", + flush=True, + ) + + +def _short_decodes(llm: LLM, memory: _DeviceMemory) -> dict: + results = {} + for tokens, key in zip(SHORT, _keys(len(SHORT), seed=7)): + r = _generate(llm, _passkey_ids(llm.tokenizer, tokens, 0.5, key), SHORT_DECODE, memory) + r.key = key + results[tokens] = r + return results + + +def _print_steps(label: str, reports: dict[int, dict] | None) -> None: + """Each rank's context steps over a cached prefix past FAR_PREFIX, the first and last of + them in full, and the most keys each indexer prefill entry received in any step.""" + if reports is None: + return + for rank, report in sorted(reports.items()): + steps = report["steps"] + far = [s for s in steps if s["cached"] and s["cached"][0] > FAR_PREFIX] + most = { + name: max((s["keys"][name] for s in steps), default=0) + for name in ( + "indexer_k_cache_gather_op", + "fp8_mqa_logits", + "cute_dsl_gvr_topk_prefill", + ) + } + print( + f"RETRIEVAL steps {label} rank {rank} {report['host']}:{report['device']}: " + f"{len(steps)} context steps, {len(far)} over a cached prefix past {FAR_PREFIX}; " + f"most keys per call {most}; nvml_error {report['nvml_error']}", + flush=True, + ) + for s in far[:1] + far[-1:]: + print(f"RETRIEVAL step {label} rank {rank} {s}", flush=True) + + +@pytest.fixture(scope="module", params=WORLDS) +def runs(request): + world = request.param + path = _checkpoint_dir() + if path is None: + pytest.skip(f"checkpoint {CHECKPOINTS} is not under LLM_MODELS_ROOT ({llm_models_root()})") + # The checkpoint's config as the engines read it from params.json. + config = MistralConfigLoader().load_pretrained_config(str(path)) + # The engines' own session under trtllm-llmapi-launch: a singleton every LLM in this + # process reuses. Unlaunched, each engine spawns its own workers and no record runs. + session = None + if get_spawn_proxy_process_env(): + session = create_mpi_comm_session(world) + _reach_ranks(session) + _probe(session, "install", world) + memory = _DeviceMemory() + needles, short = {}, {} + with _engine(path, "envelope", world) as llm: + print(f"RETRIEVAL envelope kv_cache_capacity {llm.get_kv_cache_capacity()}", flush=True) + print(f"RETRIEVAL envelope used_gib_when_up(this node) {memory.reset()}", flush=True) + _context_steps(llm) + for tokens in LENGTHS[world]: + for depth, key in zip(DEPTHS, _keys(len(DEPTHS), seed=tokens)): + ids = _passkey_ids(llm.tokenizer, tokens, depth, key) + r = _generate(llm, ids, NEEDLE, memory) + r.key, r.depth = key, depth + needles[(tokens, depth)] = r + print(f"RETRIEVAL needle tokens={tokens} depth={depth} key={key} " + f"ttft_s={r.ttft_s} text={r.text!r}", flush=True) # fmt: skip + short["envelope"] = _short_decodes(llm, memory) + envelope_ranks = _probe(session, "collect", world) + with _engine(path, "delivered", world) as llm: + print(f"RETRIEVAL delivered kv_cache_capacity {llm.get_kv_cache_capacity()}", flush=True) + _context_steps(llm) + short["delivered"] = _short_decodes(llm, memory) + delivered_ranks = _probe(session, "remove", world) + memory.stop() + + for (tokens, depth), r in needles.items(): + r.rank_peak_gib = _rank_peaks(envelope_ranks, r.window) + _report(f"needle depth={depth} key={r.key}", r) + for name, reports in (("envelope", envelope_ranks), ("delivered", delivered_ranks)): + for tokens, r in short[name].items(): + r.rank_peak_gib = _rank_peaks(reports, r.window) + _report(f"{name} short key={r.key}", r) + for tokens in SHORT: + env, dlv = short["envelope"][tokens], short["delivered"][tokens] + print( + f"RETRIEVAL short decode tokens={env.tokens} tpot_s envelope {env.tpot_s} " + f"delivered {dlv.tpot_s} (engine {env.engine_tpot_s} / {dlv.engine_tpot_s})", + flush=True, + ) + _print_steps("envelope", envelope_ranks) + return SimpleNamespace( + world=world, + needles=needles, + short=short, + steps=None + if envelope_ranks is None + else {rank: report["steps"] for rank, report in envelope_ranks.items()}, + ranks=envelope_ranks, + selecting_layers=sum(role == "full" for role in config.index_sharing), + ) + + +# ------------------------------------------------------------------------------------ tests + + +def _recorded(runs) -> None: + if runs.ranks is None: + pytest.skip("the per-rank record runs through trtllm-llmapi-launch's MPI session") + + +@pytest.mark.parametrize("length", range(len(LENGTH_IDS)), ids=LENGTH_IDS) +def test_passkey_recalled_at_every_depth(runs, length): + """The key at depths 0.1, 0.5 and 0.9 of an exactly `tokens`-token prompt (the world's + `length`-th in LENGTHS), sent alone, is in the greedy answer; the prompt ran in chunks of + 32768 tokens, one context step each.""" + tokens = LENGTHS[runs.world][length] + misses = [] + for depth in DEPTHS: + r = runs.needles[(tokens, depth)] + assert r.tokens == tokens, (r.tokens, tokens) + assert r.steps == _chunks(tokens), (tokens, depth, r.steps) + assert r.generated == NEEDLE.max_tokens, r.generated + if str(r.key) not in r.text: + misses.append((tokens, depth, r.key, r.text)) + assert not misses, misses + + +def test_split_chunks_ran_over_long_prefixes(runs): + """The longest prompts (1040000 tokens at tep4, 1040384 at tep8) ran 32768-token context + steps over cached prefixes past 524288 tokens: steps of at least q_split_threshold rows, + which the engine splits over every rank.""" + for depth in DEPTHS: + steps = runs.needles[(LENGTHS[runs.world][-1], depth)].steps + cached = [CHUNK * k for k in range(len(steps))] + far = [c for c, (_, rows) in zip(cached, steps) if c > 524288 and rows == CHUNK] + assert CHUNK >= Q_SPLIT_THRESHOLD and len(far) == 14, (depth, far) + + +def test_split_chunks_ran_over_long_prefixes_on_every_rank(runs): + """On every rank, each longest prompt ran fourteen 32768-token context steps over a cached + prefix past 524288 tokens. Each carried the engine's query split with this rank's share + (8192 rows at tep4, 4096 at tep8); the rank gathered keys and scored them over the prefix and + its share's rows (cached + the share's end keys) and made one selection gather per selecting + indexer layer.""" + _recorded(runs) + for rank, steps in runs.steps.items(): + far = [ + s + for s in steps + if s["contexts"] == 1 and s["new"] == [CHUNK] and s["cached"][0] > FAR_PREFIX + ] + assert len(far) == 14 * len(DEPTHS), (rank, [(s["cached"], s["new"]) for s in far]) + for s in far: + lo, hi, sizes = _shares(CHUNK, rank, runs.world) + keys = s["cached"][0] + hi + assert s["q_split"] == (lo, hi, sizes), (rank, s) + assert s["gathers"] == runs.selecting_layers, (rank, s) + assert s["keys"]["indexer_k_cache_gather_op"] == keys, (rank, s) + assert s["keys"]["fp8_mqa_logits"] == keys, (rank, s) + assert keys <= s["keys"]["cute_dsl_gvr_topk_prefill"] <= MAX_SEQ_LEN, (rank, s) + + +def test_no_indexer_prefill_call_past_max_seq_len_keys(runs): + """No key gather, scorer or prefill top-k call on any rank received more than max_seq_len + keys, and the longest prompt's last chunk took its scorer to all of its keys.""" + _recorded(runs) + most = {} + for steps in runs.steps.values(): + for s in steps: + for name, keys in s["keys"].items(): + most[name] = max(most.get(name, 0), keys) + assert all(keys <= MAX_SEQ_LEN for keys in most.values()), most + longest = LENGTHS[runs.world][-1] + assert most["indexer_k_cache_gather_op"] == most["fp8_mqa_logits"] == longest, most + + +def test_every_rank_reported_its_device_memory(runs): + """Every rank's NVML record covers each needle's window.""" + _recorded(runs) + for rank, report in runs.ranks.items(): + assert report["nvml_error"] is None and report["memory"], (rank, report["nvml_error"]) + for key, r in runs.needles.items(): + assert None not in r.rank_peak_gib, (key, r.rank_peak_gib) + + +@pytest.mark.parametrize("tokens", SHORT) +def test_short_prompt_decodes_at_both_envelopes(runs, tokens): + """A short prompt's key is recalled and 128 tokens are decoded at the 1M config and at the + delivered 32768 one; their decode times per token are reported side by side.""" + for name in ("envelope", "delivered"): + r = runs.short[name][tokens] + assert r.generated == SHORT_DECODE.max_tokens, (name, r.generated) + assert r.steps == [(1, r.tokens)], (name, r.steps) + assert str(r.key) in r.text, (name, r.key, r.text) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_routed_experts.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_routed_experts.py new file mode 100644 index 000000000000..aa58b58d7740 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_routed_experts.py @@ -0,0 +1,566 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Real layers' routed experts through the prepared layout, on one GPU, at tep4 and at tep8. + +The NVFP4 MoE runner reads its expert stacks in a layout no checkpoint stores -- FC1's halves +interleaved, every stack's rows block-shuffled, the block scales 128x4-swizzled -- and nothing +checks it: a wrong layout costs hundreds to thousands of bf16 ulp and raises nothing. Each rank +holds n whole experts, `[n r, n r + n)`: 32 at tep4 (four ranks), 16 at tep8 (eight ranks over +two nodes of four), and the ranks' partials sum in the closing all-reduce. + +One window, at tep4: on one MoE layer and one rank's 32-expert window of +Mistral-Large-4-1T-A52B-NVFP4 itself (layers 1 and 45, which hold the checkpoint's largest and +smallest FC2 global scale, on the first and the last window): + +* the window's stacks are filled from the shard files by the weight table's own manifest rows + (sources, destination indices, transforms) and laid out by the rows' own `layout`, the steps + the table's `load` takes (the CPU manifest test checks `load` itself against an independent + construction of the layout); that layout is byte for byte the engine's own TRTLLM-Gen + preparation of the same stored bytes (`trtllmgen_maybe_get_cached_w3_w1_permute_indices`, + `shuffle_matrix`, `block_scale_interleave`); +* each target's `_mlp` -- the layer's router through `renorm_moe_routing_op`, `fp4_quantize` + under the layer's static scale, the pre-routed runner with `_moe_scalars` -- is run on bf16 + activations at the magnitudes the layer's input scale was calibrated for, with the shared + expert zeroed so the routed half is all that is measured, and compared with an fp32 + reference on the window's dequantized checkpoint experts (`e2m1 * weight_scale * + weight_scale_2`, activations `e2m1 * scale * input_scale`, the SwiGLU output requantized to + NVFP4 under `1 / input_scale(w2)` as the reference implementation does), at the runner + entry's gate: 16 ulp per element of the row's scale, 2 ulp relative RMS; +* the runner on the stored bytes without the layout, and on FC1 with its halves swapped, + lands far outside that gate (measured 260.8 and 22.5 rms ulp at layer 1, 376.9 and 289.2 at + layer 45, where the right layout lands at 0.70-0.76): the comparison sees a wrong layout. + +Every window, at tep4 and at tep8: on an early, a middle and a late MoE layer (1, 35 and 71), +for every rank r of the world: + +* the window's stacks are filled and laid out as above, and that layout is byte for byte the + engine's own TRTLLM-Gen preparation of the same stored bytes; +* each target's `_mlp` -- the layer's router through `renorm_moe_routing_op`, `fp4_quantize` + under the layer's static scale, the pre-routed runner with `_moe_scalars` -- is run on bf16 + activations at the magnitudes the layer's input scale was calibrated for (16 rows through + `DecodeTarget`, 1031 and 32768 through `PrefillTarget`), with the shared expert zeroed so the + routed half is all that is measured, and compared with an fp32 reference on the window's + dequantized checkpoint experts (`e2m1 * weight_scale * weight_scale_2`, activations + `e2m1 * scale * input_scale`, the SwiGLU output requantized to NVFP4 under + `1 / input_scale(w2)` as the reference implementation does), at the runner entry's gate: 16 + ulp per element of the row's scale, 2 ulp relative RMS; +* the ranks' partials, summed in fp32 in rank order as the all-reduce sums them, are compared at + the same gate with the same reference over all 128 experts, built from the checkpoint's + expert keys 0..127 directly rather than from the table's windows, so a window the table + misplaces, doubles or drops is seen; leaving any one partial out lands far outside; +* with every token's top-4 skewed into one window (the router's column 0 biased for that + window's experts, the activations' element 0 held positive), that window's partial alone + meets the 128-expert reference and every other window's is exactly zero; +* the runner on the stored bytes without the layout, on FC1 with its halves swapped, and on a + neighbouring rank's laid-out window under this rank's offset lands far outside the gate. + +Skipped where the checkpoint is absent (looked up as `mistral-large-4-nvfp4` or +`Mistral-Large-4-1T-A52B-NVFP4` under `LLM_MODELS_ROOT`). +""" + +from __future__ import annotations + +import collections +import json +from pathlib import Path +from types import SimpleNamespace + +import pytest +import torch +from safetensors import safe_open +from test_common.llm_data import llm_models_root + +from tensorrt_llm._torch.modeling_v2.catalog.gemm.cublas_mm import cublas_mm +from tensorrt_llm._torch.modeling_v2.catalog.moe.fp4_block_scale_moe_runner import ( # noqa: E501 + fp4_block_scale_moe_runner, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_103__tep import ( # noqa: E501 + modeling as M, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_103__tep import ( # noqa: E501 + weights as W, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.moe.renorm_moe_routing_op import ( + renorm_moe_routing_op, +) +from tensorrt_llm._torch.models.checkpoints.mistral.config_loader import MistralConfigLoader +from tensorrt_llm._torch.moe.fused_moe.quantization import ( + trtllmgen_maybe_get_cached_w2_permute_indices, + trtllmgen_maybe_get_cached_w3_w1_permute_indices, +) +from tensorrt_llm.mapping import Mapping + +__extra_import_path__ = ["../../.."] +from _precision import true_fp32_matmul # noqa: E402 + +DEV = "cuda" +#: World size -> the routed experts of each rank's window: 128 over the ranks. +WINDOWS = {4: 32, 8: 16} +#: GB300 nodes here hold four devices: tep4 is one of them, tep8 spans two. +GPUS_PER_NODE = 4 +CHECKPOINTS = ("mistral-large-4-nvfp4", "Mistral-Large-4-1T-A52B-NVFP4") +#: The expert parameters of one MoE layer, in the table's names. +EXPERT_PARAMS = ("fc1_w", "fc1_s", "fc2_w", "fc2_s", "isc1", "isc2", "ws2_13", "ws2_2") +#: The runner's epilogue tile, which fixes its 32-row shuffle (the engine's own constant). +EPILOGUE_TILE_M = 128 +#: One bf16 ulp, relative: 2^-8 of the binade top. +ULP = 2.0**-8 +#: The largest e2m1 magnitude and the largest finite e4m3 one. +E2M1_MAX, E4M3_MAX = 6.0, 448.0 +#: The token counts each window is driven at: a decode step, a ragged prefill, a full step. +STEPS = (("DecodeTarget", 16), ("PrefillTarget", 1031), ("PrefillTarget", 32768)) +#: An early, a middle and a late MoE layer. +LAYERS = (1, 35, 71) +#: World size -> the windows every token's top-4 is skewed into: the first, the last, one between. +SKEWS = {4: (0, 3, 1), 8: (0, 7, 3)} + + +def _checkpoint_dir() -> Path | None: + root = llm_models_root() + if root is None: + return None + for name in CHECKPOINTS: + path = root / name + if (path / "params.json").is_file() and (path / W.SHARD_INDEX).is_file(): + return path + return None + + +@pytest.fixture(scope="module") +def checkpoint(): + path = _checkpoint_dir() + if path is None: + pytest.skip(f"checkpoint {CHECKPOINTS} is not under LLM_MODELS_ROOT ({llm_models_root()})") + assert torch.cuda.is_available(), "the routed-expert check needs a GPU" + weight_map = json.loads((path / W.SHARD_INDEX).read_text())["weight_map"] + # The configuration the engine adapts from the checkpoint's params.json. + config = MistralConfigLoader().load_pretrained_config(str(path)) + return SimpleNamespace(path=path, weight_map=weight_map, config=config) + + +def _core(config, rank: int, world: int) -> SimpleNamespace: + """The surface the weight table reads off the core, for rank `rank` of tep4 or tep8.""" + mapping = Mapping( + world_size=world, + rank=rank, + gpus_per_node=GPUS_PER_NODE, + tp_size=world, + moe_tp_size=1, + moe_ep_size=world, + ) + assert (mapping.tp_rank, mapping.moe_ep_rank, mapping.moe_tp_rank) == (rank, rank, 0) + return SimpleNamespace( + model_config=SimpleNamespace( + pretrained_config=config, + mapping=mapping, + torch_dtype=torch.bfloat16, + ) + ) + + +def _read(checkpoint, keys) -> dict[str, torch.Tensor]: + """The given checkpoint tensors, off the shard files, one file at a time.""" + by_file = collections.defaultdict(list) + for key in keys: + by_file[checkpoint.weight_map[key]].append(key) + out = {} + for name, names in by_file.items(): + with safe_open(str(checkpoint.path / name), framework="pt", device="cpu") as handle: + for key in names: + out[key] = handle.get_tensor(key) + return out + + +def _window(checkpoint, layer: int, rank: int, world: int): + """Layer `layer`'s expert parameters on rank `rank` of `world`, as the table's `load` leaves + them: every manifest source copied to its index through its transform, then each row's + `layout` on every expert. Also returns the stored tensors it read and the router.""" + core = _core(checkpoint.config, rank, world) + d = W.MODEL_WEIGHTS.dims(core) + manifest = W.MODEL_WEIGHTS.manifest(core) + rows = {r.name: r for r in W.MODEL_WEIGHTS.WEIGHTS if r.name in EXPERT_PARAMS} + keys = { + k + for name in EXPERT_PARAMS + for ckpt, *_ in manifest[f"l{layer}_{name}"] + for k in W._keys(ckpt) + } + router_key = f"layers.{layer}.gate.weight" + stored = _read(checkpoint, sorted(keys | {router_key})) + params = {} + for name in EXPERT_PARAMS: + row = rows[name] + param = torch.empty(row.shape(d), dtype=row.dtype or d.dtype, device=DEV) + for ckpt, index, transform, shard in manifest[f"l{layer}_{name}"]: + dst = param if index is None else param[index] + if isinstance(ckpt, tuple): + src = transform(tuple(W._materialize(stored[k], shard).to(DEV) for k in ckpt), core) + else: + src = W._materialize(stored[ckpt], shard).to(DEV) + if transform is not None: + src = transform(src.contiguous(), core) + assert tuple(dst.shape) == tuple(src.shape) and dst.dtype == src.dtype, (name, ckpt) + dst.copy_(src) + if row.layout is not None: + for slot in range(param.shape[0]): + param[slot].copy_(row.layout(param[slot])) + params[name] = param + return d, params, stored, stored[router_key].to(DEV) + + +def _stored_experts(stored: dict, layer: int, experts) -> dict[str, list[torch.Tensor]]: + """The stored bytes of the given global experts: `[w3 ; w1]` codes and block scales, w2's, + and each expert's `weight_scale_2` pair.""" + out = collections.defaultdict(list) + for expert in experts: + e = f"layers.{layer}.experts.{expert}" + out["fc1_w"].append(torch.cat([stored[f"{e}.w3.weight"], stored[f"{e}.w1.weight"]])) + out["fc1_s"].append( + torch.cat([stored[f"{e}.w3.weight_scale"], stored[f"{e}.w1.weight_scale"]]) + ) + out["fc2_w"].append(stored[f"{e}.w2.weight"]) + out["fc2_s"].append(stored[f"{e}.w2.weight_scale"]) + out["ws2_13"].append(stored[f"{e}.w1.weight_scale_2"]) + out["ws2_2"].append(stored[f"{e}.w2.weight_scale_2"]) + return out + + +def _engine_layout(stored: dict[str, list[torch.Tensor]]) -> dict[str, torch.Tensor]: + """The engine's own TRTLLM-Gen NVFP4 MoE preparation of the stored bytes, expert by expert: + its gated-activation row reorder chained with its row shuffle, `shuffle_matrix`, and + `block_scale_interleave` on the scales.""" + cache: dict = {} + sf = torch.uint8 + out = collections.defaultdict(list) + for w31, s31, w2, s2 in zip(stored["fc1_w"], stored["fc1_s"], stored["fc2_w"], stored["fc2_s"]): + w31, s31, w2, s2 = (t.to(DEV) for t in (w31, s31, w2, s2)) + perm = trtllmgen_maybe_get_cached_w3_w1_permute_indices(w31, cache, EPILOGUE_TILE_M) + out["fc1_w"].append(torch.ops.trtllm.shuffle_matrix(w31, perm)) + perm = trtllmgen_maybe_get_cached_w3_w1_permute_indices( + s31.view(sf), cache, EPILOGUE_TILE_M, num_elts_per_sf=16 + ) + shuffled = torch.ops.trtllm.shuffle_matrix(s31.view(sf), perm) + out["fc1_s"].append( + torch.ops.trtllm.block_scale_interleave(shuffled.reshape(s31.shape)).reshape(s31.shape) + ) + perm = trtllmgen_maybe_get_cached_w2_permute_indices(w2, cache, EPILOGUE_TILE_M) + out["fc2_w"].append(torch.ops.trtllm.shuffle_matrix(w2, perm)) + perm = trtllmgen_maybe_get_cached_w2_permute_indices( + s2.view(sf), cache, EPILOGUE_TILE_M, num_elts_per_sf=16 + ) + shuffled = torch.ops.trtllm.shuffle_matrix(s2.view(sf), perm) + out["fc2_s"].append( + torch.ops.trtllm.block_scale_interleave(shuffled.reshape(s2.shape)).reshape(s2.shape) + ) + return {name: torch.stack(t) for name, t in out.items()} + + +def _unpack(codes: torch.Tensor) -> torch.Tensor: + """`[..., K/2]` packed e2m1 -> `[..., K]` codes, the even element in the low nibble.""" + return torch.stack([codes & 0x0F, codes >> 4], dim=-1).reshape(*codes.shape[:-1], -1) + + +def _reference(x_q, x_s, isc1, isc2, ids, weights, stored, first: int, inter: int): + """The routed half in fp32 over the given experts' dequantized checkpoint bytes, global + experts `[first, first + len(stored))`: the runner entry's `reference`, handed the stored + codes (unpacked) and their block scales with each expert's stored `weight_scale_2` + multiplied in, and the activation's exact NVFP4 value with its global scale undone + (`code * scale * input_scale`). fp32; the caller rounds.""" + rows = x_q.shape[0] + act = _unpack(x_q).long() + act_scale = x_s.view(torch.float8_e4m3fn).float().reshape(rows, -1) + e2m1 = torch.tensor( + [0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0, -0.0, -0.5, -1.0, -1.5, -2.0, -3.0, -4.0, -6.0], + device=DEV, + ) + hidden = e2m1[act] * act_scale.repeat_interleave(16, dim=1) * isc1.float() + count = len(stored["fc1_w"]) + + def operand(name, half, global_scales): + codes = torch.stack([_unpack(t.to(DEV))[half] for t in stored[name + "_w"]]) + scales = torch.stack([t.to(DEV).float()[half] for t in stored[name + "_s"]]) + g = torch.stack([s.to(DEV).float() for s in global_scales]) + return codes, scales * g.view(-1, 1, 1) + + up = operand("fc1", slice(0, inter), stored["ws2_13"]) + gate = operand("fc1", slice(inter, 2 * inter), stored["ws2_13"]) + down = operand("fc2", slice(None), stored["ws2_2"]) + with true_fp32_matmul(): + return fp4_block_scale_moe_runner.reference( + hidden, + ids, + weights, + up, + gate, + down, + fc2_global_scale=1.0 / float(isc2), + local_expert_offset=first, + local_num_experts=count, + ) + + +def _deviation(out: torch.Tensor, ref: torch.Tensor) -> tuple[float, float]: + """(worst element deviation against its row's scale, relative RMS deviation), in ulp.""" + o, r = out.float(), ref.float() + row = r.abs().amax(dim=1, keepdim=True).clamp_min(1e-30) + elt = ((o - r).abs() / row).max().item() / ULP + rms = ((o - r).pow(2).mean().sqrt() / r.pow(2).mean().sqrt().clamp_min(1e-30)).item() / ULP + return elt, rms + + +def _gate(out: torch.Tensor, ref: torch.Tensor, what: str) -> tuple[float, float]: + """The runner entry's gate on the rows `ref` is nonzero in: 16 ulp per element of the row's + scale, 2 ulp relative RMS.""" + live = (ref != 0).any(dim=1) + assert bool(live.any()), f"{what}: no token routed here, nothing is measured" + elt, rms = _deviation(out[live], ref[live].to(torch.bfloat16)) + print(f"{what}: {int(live.sum())} tokens, {elt:.2f} elt / {rms:.3f} rms ulp") + assert rms <= 2.0, f"{what}: relative RMS {rms:.2f} ulp > 2" + assert elt <= 16.0, f"{what}: element deviation {elt:.2f} ulp > 16" + return elt, rms + + +def _layer(d, params, router) -> SimpleNamespace: + """The `L` a target's `_mlp` reads: a shared expert whose down projection is zero (its + partial is exactly 0, so the result is the routed half alone), the router's operand, the + laid-out window and the call scalars from `_moe_scalars`.""" + shared = d.shared_inter + unit_sf = lambda n, k: W._deepgemm_sf( # noqa: E731 + torch.ones(n // W.BLOCK, k // W.BLOCK, device=DEV), None + ).t() + return SimpleNamespace( + mlp_gu=(torch.randn(2 * shared, d.hidden, device=DEV) * 0.1).to(torch.float8_e4m3fn), + mlp_gu_s=unit_sf(2 * shared, d.hidden), + mlp_dn=torch.zeros(d.hidden, shared, device=DEV).to(torch.float8_e4m3fn), + mlp_dn_s=unit_sf(d.hidden, shared), + moe=SimpleNamespace( + router=router.t(), + fc1_w=params["fc1_w"], + fc1_s=params["fc1_s"], + fc2_w=params["fc2_w"], + fc2_s=params["fc2_s"], + **vars( + M._moe_scalars(params["isc1"], params["isc2"], params["ws2_13"], params["ws2_2"]) + ), + ), + ) + + +def _target(cls, d, max_num_tokens: int): + """A `PrefillTarget` or `DecodeTarget` over the surface `_mlp` reads off the core.""" + target = object.__new__(cls) + target.core = SimpleNamespace( + topk=4, + num_experts=d.num_experts, + moe_inter=d.moe_inter, + expert_offset=d.expert_offset, + local_experts=d.local_experts, + max_num_tokens=max_num_tokens, + ) + return target + + +def _routing(o, router): + """The routing every rank computes for the step, and the step's NVFP4 activations.""" + logits = cublas_mm(o, router.t(), None, torch.float32) + return renorm_moe_routing_op(logits, 4, torch.bfloat16) + + +#: (layer, tep4 rank): the layers holding the checkpoint's largest and smallest FC2 global +#: scale (1 / input_scale(w2): 6898.53 at layer 1, 1.36898 at layer 45), on the first and the +#: last expert window. +CASES = [(1, 0), (45, 3)] + + +@pytest.mark.parametrize("layer, rank", CASES, ids=[f"layer{n}-rank{r}" for n, r in CASES]) +def test_routed_experts_of_one_real_window(checkpoint, layer, rank): + torch.manual_seed(1000 * layer + rank) + d, params, stored_tensors, router = _window(checkpoint, layer, rank, 4) + stored = _stored_experts( + stored_tensors, layer, range(d.expert_offset, d.expert_offset + d.local_experts) + ) + + # The table's layout is the engine's own preparation of the same bytes. + engine = _engine_layout(stored) + for name in ("fc1_w", "fc1_s", "fc2_w", "fc2_s"): + mine = params[name].view(torch.uint8) + assert torch.equal(mine, engine[name].view(torch.uint8)), name + + isc1, isc2 = params["isc1"], params["isc2"] + inter = d.moe_inter + # Activations at the amax the layer's static input scale was calibrated for (448 * 6 * + # input_scale): a gaussian whose largest elements reach about it. + calibrated = E4M3_MAX * E2M1_MAX * float(isc1) + L = _layer(d, params, router) + for cls, rows in ((M.DecodeTarget, 16), (M.PrefillTarget, 1031), (M.PrefillTarget, 32768)): + o = (torch.randn(rows, d.hidden, device=DEV) * (calibrated / 5)).to(torch.bfloat16) + out = _target(cls, d, 32768)._mlp(o, L) + + logits = cublas_mm(o, L.moe.router, None, torch.float32) + ids, weights = renorm_moe_routing_op(logits, 4, torch.bfloat16) + local = ids.long() - d.expert_offset + live = ((local >= 0) & (local < d.local_experts)).any(dim=1) + assert bool(live.any()), "no token routed into the window: nothing is measured" + x_q, x_s = M.fp4_quantize(o, L.moe.gscale, 16, False, False) + ref = _reference(x_q, x_s, isc1, isc2, ids, weights, stored, d.expert_offset, inter) + ref = ref.to(torch.bfloat16) + assert bool((out[~live] == 0).all()), "a token with no slot in the window must be zero" + elt, rms = _deviation(out[live], ref[live]) + print( + f"layer {layer} rank {rank} {cls.__name__} T={rows}: {int(live.sum())} tokens in " + f"the window, {elt:.2f} elt / {rms:.3f} rms ulp" + ) + assert rms <= 2.0, f"relative RMS {rms:.2f} ulp > 2" + assert elt <= 16.0, f"element deviation {elt:.2f} ulp > 16" + + # A wrong layout runs without complaint and lands far away. + rows = 1031 + o = (torch.randn(rows, d.hidden, device=DEV) * (calibrated / 5)).to(torch.bfloat16) + logits = cublas_mm(o, L.moe.router, None, torch.float32) + ids, weights = renorm_moe_routing_op(logits, 4, torch.bfloat16) + x_q, x_s = M.fp4_quantize(o, L.moe.gscale, 16, False, False) + ref = _reference(x_q, x_s, isc1, isc2, ids, weights, stored, d.expert_offset, inter) + ref = ref.to(torch.bfloat16) + live = ((ids.long() - d.expert_offset >= 0) & (ids.long() - d.expert_offset < 32)).any(dim=1) + as_stored = {name: torch.stack(t).to(DEV) for name, t in stored.items()} + swapped = torch.cat([as_stored["fc1_w"][:, inter:], as_stored["fc1_w"][:, :inter]], dim=1) + swapped_layout = torch.stack([W._fc1_codes(t) for t in swapped]) + wrong = { + "no layout": as_stored, + "halves swapped": dict(params, fc1_w=swapped_layout), + } + for what, ops in wrong.items(): + s = L.moe + out = fp4_block_scale_moe_runner( + None, None, x_q, x_s.view(torch.float8_e4m3fn), ops["fc1_w"].contiguous(), + ops["fc1_s"].contiguous(), None, None, None, None, ops["fc2_w"].contiguous(), + ops["fc2_s"].contiguous(), None, s.out1, s.out1_gate, s.out2, d.num_experts, 4, None, + None, d.moe_inter, d.expert_offset, d.local_experts, None, 1, True, 0, + topk_weights=weights, topk_ids=ids, tune_max_num_tokens=32768, + )[0] # fmt: skip + _, rms = _deviation(out[live], ref[live]) + print(f"layer {layer} rank {rank} {what}: {rms:.1f} rms ulp") + assert rms > 5 * 2.0, f"{what}: {rms:.2f} rms ulp is not told apart from the right layout" + + +@pytest.mark.parametrize("layer", LAYERS, ids=[f"layer{n}" for n in LAYERS]) +@pytest.mark.parametrize("world", sorted(WINDOWS), ids=[f"tep{w}" for w in sorted(WINDOWS)]) +def test_routed_experts_of_every_window(checkpoint, world, layer): + torch.manual_seed(1000 * layer) + n = WINDOWS[world] + windows = [_window(checkpoint, layer, rank, world) for rank in range(world)] + d0, params0, _, router = windows[0] + inter = d0.moe_inter + # Every window is whole experts at n r, and the windows tile the 128 routed experts. + assert [w[0].expert_offset for w in windows] == [n * r for r in range(world)] + assert {w[0].local_experts for w in windows} == {n} + + # The table's layout is the engine's own preparation of the same bytes, on every window. + stored_all: dict[str, torch.Tensor] = {} + for rank, (d, params, stored_tensors, _) in enumerate(windows): + stored = _stored_experts(stored_tensors, layer, range(n * rank, n * rank + n)) + engine = _engine_layout(stored) + for name in ("fc1_w", "fc1_s", "fc2_w", "fc2_s"): + assert torch.equal(params[name].view(torch.uint8), engine[name].view(torch.uint8)), ( + rank, + name, + ) + stored_all.update(stored_tensors) + # Every expert of the layer, by its checkpoint keys: what the 128-expert reference reads. + full = _stored_experts(stored_all, layer, range(d0.num_experts)) + isc1, isc2 = params0["isc1"], params0["isc2"] + for _, params, _, _ in windows: + assert torch.equal(params["isc1"], isc1) and torch.equal(params["isc2"], isc2) + + # Activations at the amax the layer's static input scale was calibrated for (448 * 6 * + # input_scale): a gaussian whose largest elements reach about it. + calibrated = E4M3_MAX * E2M1_MAX * float(isc1) + layers = [_layer(d, params, router) for d, params, _, _ in windows] + for cls_name, rows in STEPS: + cls = getattr(M, cls_name) + o = (torch.randn(rows, d0.hidden, device=DEV) * (calibrated / 5)).to(torch.bfloat16) + ids, weights = _routing(o, router) + x_q, x_s = M.fp4_quantize(o, layers[0].moe.gscale, 16, False, False) + parts = [] + for rank, ((d, params, stored_tensors, _), L) in enumerate(zip(windows, layers)): + out = _target(cls, d, 32768)._mlp(o, L) + stored = _stored_experts(stored_tensors, layer, range(n * rank, n * rank + n)) + ref = _reference(x_q, x_s, isc1, isc2, ids, weights, stored, n * rank, inter) + local = ids.long() - n * rank + outside = ~((local >= 0) & (local < n)).any(dim=1) + assert bool((out[outside] == 0).all()), "a token with no slot in the window is zero" + _gate(out, ref, f"layer {layer} rank {rank} {cls_name} T={rows}") + parts.append(out) + # The all-reduce's sum: fp32 in rank order, rounded once. + summed = sum(p.float() for p in parts).to(torch.bfloat16) + ref_full = _reference(x_q, x_s, isc1, isc2, ids, weights, full, 0, inter) + assert bool((ref_full != 0).any(dim=1).all()), "every token routes somewhere" + _gate(summed, ref_full, f"layer {layer} {world} windows summed {cls_name} T={rows}") + if rows >= 1031: + for drop in range(world): + partial = sum(p.float() for i, p in enumerate(parts) if i != drop) + _, rms = _deviation(partial.to(torch.bfloat16), ref_full.to(torch.bfloat16)) + assert rms > 25.0, f"T={rows} without window {drop}: only {rms:.1f} rms ulp" + + # Skew: every token's top-4 in one window. Column 0 of the router is biased up for that + # window's experts and down for the rest, and every token's element 0 is positive, so the + # bias dominates the logits; the ranking inside the window is the router's own. + rows = 4096 + for skew in SKEWS[world]: + o = (torch.randn(rows, d0.hidden, device=DEV) * (calibrated / 5)).to(torch.bfloat16) + o[:, 0] = o[:, 0].abs() + calibrated / 10 + base = cublas_mm(o, router.t(), None, torch.float32) + bias = 64.0 * float(base.abs().max()) / float(o[:, 0].float().min()) + skewed = router.clone() + inside = torch.zeros(d0.num_experts, dtype=torch.bool, device=DEV) + inside[n * skew : n * skew + n] = True + skewed[inside, 0] += bias + skewed[~inside, 0] -= bias + ids, weights = _routing(o, skewed) + assert bool(((ids >= n * skew) & (ids < n * skew + n)).all()), "skew missed" + x_q, x_s = M.fp4_quantize(o, layers[0].moe.gscale, 16, False, False) + ref_full = _reference(x_q, x_s, isc1, isc2, ids, weights, full, 0, inter) + parts = [] + for rank, (d, params, _, _) in enumerate(windows): + L = _layer(d, params, skewed) + out = _target(M.PrefillTarget, d, 32768)._mlp(o, L) + if rank != skew: + assert bool((out == 0).all()), f"skew {skew}: window {rank} wrote" + parts.append(out) + _gate(parts[skew], ref_full, f"layer {layer} skew into window {skew}") + summed = sum(p.float() for p in parts).to(torch.bfloat16) + assert torch.equal(summed, parts[skew]), "the other windows' zeros change the sum" + + # A wrong layout, or a neighbour's window under this rank's offset, runs without complaint + # and lands far away. + rows = 1031 + o = (torch.randn(rows, d0.hidden, device=DEV) * (calibrated / 5)).to(torch.bfloat16) + ids, weights = _routing(o, router) + x_q, x_s = M.fp4_quantize(o, layers[0].moe.gscale, 16, False, False) + for rank in (0, world - 1): + d, params, stored_tensors, _ = windows[rank] + stored = _stored_experts(stored_tensors, layer, range(n * rank, n * rank + n)) + ref = _reference(x_q, x_s, isc1, isc2, ids, weights, stored, n * rank, inter) + live = (ref != 0).any(dim=1) + as_stored = {name: torch.stack(stored[name]).to(DEV) for name in ("fc1_w", "fc1_s")} + as_stored.update({name: torch.stack(stored[name]).to(DEV) for name in ("fc2_w", "fc2_s")}) + swapped = torch.cat([as_stored["fc1_w"][:, inter:], as_stored["fc1_w"][:, :inter]], dim=1) + swapped_layout = torch.stack([W._fc1_codes(t) for t in swapped]) + neighbour = windows[(rank + 1) % world][1] + wrong = { + "no layout": dict(params, **as_stored), + "halves swapped": dict(params, fc1_w=swapped_layout), + f"window {(rank + 1) % world}'s experts": dict(neighbour), + } + s = layers[rank].moe + for what, ops in wrong.items(): + out = fp4_block_scale_moe_runner( + None, None, x_q, x_s.view(torch.float8_e4m3fn), ops["fc1_w"].contiguous(), + ops["fc1_s"].contiguous(), None, None, None, None, ops["fc2_w"].contiguous(), + ops["fc2_s"].contiguous(), None, s.out1, s.out1_gate, s.out2, d.num_experts, 4, + None, None, inter, d.expert_offset, d.local_experts, None, 1, True, 0, + topk_weights=weights, topk_ids=ids, tune_max_num_tokens=32768, + )[0] # fmt: skip + _, rms = _deviation(out[live], ref[live].to(torch.bfloat16)) + print(f"layer {layer} rank {rank} {what}: {rms:.1f} rms ulp") + assert rms > 5 * 2.0, f"{what}: {rms:.2f} rms ulp is not told apart from the right one" diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_step_paths.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_step_paths.py new file mode 100644 index 000000000000..33b1061abc32 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_step_paths.py @@ -0,0 +1,1010 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The per-step paths of the Mistral-Large-4-1T-A52B NVFP4 tep target, and the sizes it takes from +the engine. + +No path is selected by the serving configuration. The core reads nothing off max_seq_len when it +is built: `shared_source` is structural (layer 35 alone, the owner the NoPE layers 39, 43, ..., 71 +attend over), and the indexer's logits budget and the held context workspace's rows are 0 until +the first step, when `_derive_first_step` sizes them from the engine's quantities on the attention +metadata (max_num_tokens, max_num_requests, max_seq_len and runtime_features): + +* the logits budget, `_prefill_logits_budget(max_num_tokens, max_seq_len)`: mainline's reachable + MQA-logits tile (`get_indexer_mqa_logits_workspace_bytes`, which + TLLM_INDEXER_MQA_LOGITS_ELEM_BUDGET can only lower), within `_PREFILL_LOGITS_CEILING` (2**28) + and within max_seq_len / 64 query rows at max_seq_len keys: 2**24 at max_seq_len 32768 (512-row + tiles at 32768 keys), the ceiling from max_seq_len 131072 on (2048-row tiles at 131072 keys, 256 + at 1048576); +* the held workspace's rows, `_ctx_workspace_rows`: twice the most context KV one step can attend + (`_most_ctx_kv`: max_num_tokens with neither chunked prefill nor block reuse, else + min(max_num_requests, max_num_tokens) x max_seq_len), up to `_CTX_WORKSPACE_ROWS` (2**19): + 65536 for the 32768 serving config (max_num_tokens = max_seq_len = 32768, chunked prefill and + block reuse off), 2**19 with either at 16 requests of max_seq_len 32768 or more. + +Chunked prefill and block reuse are the metadata's runtime_features alone, which the executor +builds once for the service (`AttentionRuntimeFeatures(chunked_prefill=enable_chunked_prefill, +cache_reuse=kv_cache_config.enable_block_reuse)` in py_executor_creator, turned off again where +it disables either): the KV cache manager is not consulted for them, and a user's +fp8_context_mla_kv_len_cap is not read. + +Each context step then picks its own path by its context KV n_kv, every context request's cached +and new rows (`PrefillTarget._ctx_workspace`): while n_kv fits the held rows, FlashMLA's sparse +prefill over those rows dequantized into the held workspace's first n_kv rows, layer 35's gather +kept in the next n_kv rows for its sharers when n_kv fits twice and gathered again by each sharer +when it fits once (`_attn_ctx`); past the held rows, the sparse decode over the pool in sub-batches +of at most 8192 context rows. The workspace is allocated once, on the first context step, and its +storage is the one every later step uses. The context indexer cuts an engine chunk at its requests +only when it holds more than max_seq_len keys, over the host prefix sums `step_args` carries on +every context step (`_ctx_host_offsets`), scores each block in tiles under the budget, and fills +the rows of a block with no key with -1 (`_select_ctx`). The rope tables hold no `mla` table. + +No device: the core is built on the meta device from the checkpoint's native config, params.json +(skipped where it is absent under LLM_MODELS_ROOT); everything else runs on host values, with +recording stand-ins for the kernels the paths call. +""" + +from __future__ import annotations + +import inspect +from types import SimpleNamespace + +import pytest +import torch +from test_common.llm_data import llm_models_root + +from tensorrt_llm._torch.attention.backends.interface import AttentionRuntimeFeatures +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_103__tep import ( # noqa: E501 + modeling as M, +) + +pytestmark = pytest.mark.cpu_only + +CHECKPOINTS = ("Mistral-Large-4-1T-A52B-NVFP4", "mistral-large-4-nvfp4") +#: The checkpoint's layers, the owner its NoPE layers 39, 43, ..., 71 attend over, each layer's +#: latent source, the 63 KV owners and the latent row's width. +LAYERS = 72 +SHARED_SOURCE = 35 +KV_SOURCE = [SHARED_SOURCE if i > SHARED_SOURCE and i % 4 == 3 else i for i in range(LAYERS)] +OWNERS = [i for i in range(LAYERS) if KV_SOURCE[i] == i] +LAT_DIM = 576 +#: Engine max_seq_lens a core is built at -- below, at and one past 32768, 1048576, and unset +#: (the engine infers one of at least the checkpoint's max_position_embeddings, 1048576) -- none +#: of which moves anything the core holds before its first step. +MAX_SEQ_LENS = [8192, 32768, 32769, 1048576, None] +#: The checkpoint's rope surface (its native config, adapted from params.json), what +#: `_rope_tables` reads of it. +ROPE_CFG = SimpleNamespace( + rope_parameters=dict( + rope_type="yarn", + rope_theta=10000.0, + factor=2048.0, + original_max_position_embeddings=512, + beta_fast=32.0, + beta_slow=1.0, + mscale_all_dim=1, + attention_factor=1.0, + ), + qk_rope_head_dim=64, +) +#: Mainline's override of the reachable MQA-logits tile, in logits. +ENV_BUDGET = "TLLM_INDEXER_MQA_LOGITS_ELEM_BUDGET" +#: Serving configs by their numbers, (max_num_tokens, max_seq_len, max_batch_size, chunked +#: prefill, block reuse), and what the target takes from each: the most context KV one step +#: attends (`_most_ctx_kv`), the held workspace's rows (`_ctx_workspace_rows`) and the logits +#: budget (`_prefill_logits_budget`). +CONFIGS = { + "32k": ((32768, 32768, 16, False, False), (32768, 1 << 16, 1 << 24)), + "16k_tokens-32k": ((16384, 32768, 16, False, False), (16384, 1 << 15, 1 << 24)), + "32k-block_reuse": ((32768, 32768, 16, False, True), (1 << 19, 1 << 19, 1 << 24)), + "1m-chunked": ((32768, 1048576, 16, True, False), (1 << 24, 1 << 19, 1 << 28)), + "1m-chunked-block_reuse": ((32768, 1048576, 16, True, True), (1 << 24, 1 << 19, 1 << 28)), + "131072-chunked": ((32768, 131072, 16, True, False), (1 << 21, 1 << 19, 1 << 28)), + "131072-unchunked": ((32768, 131072, 16, False, False), (32768, 1 << 16, 1 << 28)), +} +#: The 32768 config with neither chunked prefill nor block reuse and the 1M config, by their +#: numbers as above. +PLAIN_32K, CHUNKED_1M = (CONFIGS[c][0] for c in ("32k", "1m-chunked")) + + +def _sizing(max_num_tokens, max_seq_len, max_batch_size, chunked, reuse, **fields): + """The engine's quantities on the attention metadata, as `build_attention_metadata` fills + them (max_num_requests is the engine's max_batch_size, the runtime features its chunked + prefill and block reuse) and the KV cache manager states max_seq_len.""" + return SimpleNamespace( + max_num_tokens=max_num_tokens, + max_num_requests=max_batch_size, + max_seq_len=max_seq_len, + runtime_features=AttentionRuntimeFeatures( + chunked_prefill=chunked, cache_reuse=reuse, chunk_size=max_num_tokens + ), + **fields, + ) + + +def _core(max_seq_len: int | None): + """The core of rank 0 at four ranks, on the meta device, for an engine at `max_seq_len`.""" + from tensorrt_llm._torch.model_config import ModelConfig + from tensorrt_llm._torch.models.checkpoints.mistral.config_loader import MistralConfigLoader + from tensorrt_llm.mapping import Mapping + + root = llm_models_root() + paths = [root / name for name in CHECKPOINTS] if root is not None else [] + path = next((p for p in paths if (p / "params.json").is_file()), None) + if path is None: + pytest.skip(f"none of {CHECKPOINTS} is under LLM_MODELS_ROOT ({root})") + mapping = Mapping( + world_size=4, rank=0, gpus_per_node=4, tp_size=4, pp_size=1, moe_ep_size=4, moe_tp_size=1 + ) + model_config = ModelConfig( + pretrained_config=MistralConfigLoader().load_pretrained_config(str(path)), + mapping=mapping, + max_num_tokens=32768, + max_seq_len=max_seq_len, + ) + with torch.device("meta"): + return M.MistralLarge4TepModelingV2(model_config) + + +def _manager(): + """The fp8_ds_mla KV cache manager's surface `_derive_first_step` reads: its layout, the + layers' latent sources and owner rows and the KV owners. Nothing else: it holds no + KvCacheConfig, so a read of one fails.""" + return SimpleNamespace( + latent_layout=M._LATENT_LAYOUT, + kv_source_layers=KV_SOURCE, + latent_pool_row=lambda i: OWNERS.index(KV_SOURCE[i]), + kv_owner_layers=OWNERS, + ) + + +@pytest.fixture +def built(monkeypatch) -> list: + """Stand-ins for what `build_kernels` and `_derive_first_step` build on a device. The rope + tables are recorded by their rows through a stand-in with `_rope_tables`'s signature and no + other parameter, so a call asking for any other table fails.""" + rows = [] + + def tables(cfg, alpha, positions, dtype, device): + rows.append(positions) + return SimpleNamespace(rows=positions) + + monkeypatch.setattr(M, "_rope_tables", tables) + monkeypatch.setattr(M, "build_ix_q_side", lambda dev, is_neox: None) + monkeypatch.setattr(M, "build_ix_k_cache", lambda dev, is_neox, cache_contiguous: None) + monkeypatch.setattr(M, "build_ds_mla_decode", lambda dev, num_heads: None) + monkeypatch.setattr( + M, "build_ix_topk_decode", lambda dev, next_n: SimpleNamespace(next_n=next_n) + ) + monkeypatch.setattr(torch.cuda, "current_device", lambda: 0) + return rows + + +# ---- the core: nothing selected off max_seq_len, its sizes from the first step + + +def test_module_holds_no_switch(): + """The switch and the budgets only it chose between are gone; what remains is fixed: the + budget's ceiling (2**28 logits, 256 rows at 1048576 keys), the held workspace's ceiling + (2**19 rows) and the decode path's sub-batch (8192 rows). `_indexer_blocks` takes the chunk + plan and the step only, and `_rope_tables` builds no `mla` table on request.""" + removed = ("_LONG_CONTEXT_THRESHOLD", "_LONG_PREFILL_LOGITS_BUDGET", "_PREFILL_LOGITS_BUDGET") + assert not [name for name in removed if hasattr(M, name)] + assert M._PREFILL_LOGITS_CEILING == 1 << 28 + assert (M._CTX_WORKSPACE_ROWS, M._CTX_DECODE_ROWS) == (1 << 19, 8192) + assert list(inspect.signature(M._indexer_blocks).parameters) == ["chunks", "step"] + assert list(inspect.signature(M._rope_tables).parameters) == [ + "cfg", + "alpha", + "positions", + "dtype", + "device", + ] + + +@pytest.mark.parametrize("max_seq_len", MAX_SEQ_LENS, ids=[f"max_seq_len{n}" for n in MAX_SEQ_LENS]) +def test_core_reads_nothing_off_max_seq_len(max_seq_len): + """Whatever the engine's max_seq_len, the core is built the same: no switch (`long_context` + is gone), layer 35 the one shared source, and no logits budget or held workspace rows until + the first step. The max_seq_len sets the rope tables' first size only.""" + core = _core(max_seq_len) + assert not hasattr(core, "long_context") + # The fused attention glue at every max_seq_len, unless `_UNFUSED_GLUE`. + assert core.unfused_glue is M._UNFUSED_GLUE + # The native generation-row decode at every max_seq_len, unless `_GEN_ATTN_FLASH_MLA`. + assert core.gen_attn_flash_mla is M._GEN_ATTN_FLASH_MLA + assert core.kv_source == KV_SOURCE + assert [i for i, shared in enumerate(core.shared_source) if shared] == [SHARED_SOURCE] + assert (core.prefill_logits_budget, core.ctx_workspace_rows) == (0, 0) + assert core.rope_rows == (max_seq_len or 1048576) + + +@pytest.mark.parametrize("config", list(CONFIGS)) +def test_first_step_sizes_the_core(monkeypatch, built, config): + """On the core built from the checkpoint's config at the engine's max_seq_len: `build_kernels` + builds the rope tables at max_seq_len rows with no `mla` table asked for, and + `_derive_first_step`, handed the engine's metadata on the first step, sets the logits budget + and the held workspace's rows from max_num_tokens, max_num_requests, max_seq_len and the + runtime features -- 2**24 and 65536 at the 32768 serving config, 2**28 and 2**19 at the 1M + config (max_seq_len 1048576, chunked prefill) -- without growing the tables.""" + monkeypatch.delenv(ENV_BUDGET, raising=False) + sizes, (_, rows, budget) = CONFIGS[config] + core = _core(sizes[1]) + core.build_kernels(0) + assert built == [sizes[1]] + core._derive_first_step(_sizing(*sizes, kv_cache_manager=_manager(), tokens_per_block=32)) + assert (core.prefill_logits_budget, core.ctx_workspace_rows) == (budget, rows) + assert core._max_ctx_len == sizes[1] and core._stride_factor == 63 * 32 + assert built == [sizes[1]] + + +def test_first_step_grows_the_tables_and_rederives_for_a_new_manager(monkeypatch, built): + """On a stand-in core with what `build_kernels` and `_derive_first_step` read: the tables + built at 16 rows grow to an engine max_seq_len of 64 on the first step, neither call asking + for an `mla` table. The sizes are fixed for the KV cache manager they were derived with -- a + later step's metadata on the same manager moves nothing -- and derived again for a replaced + one.""" + monkeypatch.delenv(ENV_BUDGET, raising=False) + core = SimpleNamespace( + model_config=SimpleNamespace(pretrained_config=ROPE_CFG), + w={"lerope_alpha": None, "final_norm": torch.empty(0, dtype=torch.bfloat16)}, + rope_rows=16, + kv_source=KV_SOURCE, + num_layers=LAYERS, + _derived_for=None, + _ix_k=None, + prefill_logits_budget=0, + ctx_workspace_rows=0, + unfused_glue=False, + heads=16, + gen_attn_flash_mla=False, + ) + M.MistralLarge4TepModelingV2.build_kernels(core, 0) + mgr = _manager() + small = (32768, 64, 16, False, False) + M.MistralLarge4TepModelingV2._derive_first_step( + core, _sizing(*small, kv_cache_manager=mgr, tokens_per_block=32) + ) + assert built == [16, 64] and core._rope.rows == 64 + # 64 rows of 64 keys: max_seq_len x (max_seq_len >> 6) binds. + assert (core.prefill_logits_budget, core.ctx_workspace_rows) == (64, 1 << 16) + + sizes, (_, rows, budget) = CONFIGS["1m-chunked"] + M.MistralLarge4TepModelingV2._derive_first_step( + core, _sizing(*sizes, kv_cache_manager=mgr, tokens_per_block=32) + ) + assert (core.prefill_logits_budget, core.ctx_workspace_rows) == (64, 1 << 16) + assert built == [16, 64] + M.MistralLarge4TepModelingV2._derive_first_step( + core, _sizing(*sizes, kv_cache_manager=_manager(), tokens_per_block=32) + ) + assert (core.prefill_logits_budget, core.ctx_workspace_rows) == (budget, rows) + assert built == [16, 64, 1048576] + + +def test_first_step_sizes_the_held_rows_from_the_runtime_features(monkeypatch, built): + """On the core built from the checkpoint's config at max_seq_len 32768, each first step of + the 32768 config (32768 tokens, 16 requests) on a new KV cache manager: the held rows follow + that step's metadata's runtime features, derived again for each manager. Block reuse on: the + never-stall worst case, 16 x 32768 rows of context KV, 2**19 held; off: every context row + new, 65536 held; chunked prefill on without block reuse: 2**19 again. The logits budget, + from max_num_tokens and max_seq_len alone, stays 2**24.""" + monkeypatch.delenv(ENV_BUDGET, raising=False) + core = _core(32768) + core.build_kernels(0) + for chunked, reuse, rows in ( + (False, True, 1 << 19), + (False, False, 1 << 16), + (True, False, 1 << 19), + (False, True, 1 << 19), + ): + mgr = _manager() + md = _sizing(*PLAIN_32K[:3], chunked, reuse, kv_cache_manager=mgr, tokens_per_block=32) + core._derive_first_step(md) + assert core._derived_for is mgr + assert (core.prefill_logits_budget, core.ctx_workspace_rows) == (1 << 24, rows), ( + chunked, + reuse, + ) + assert built == [32768] + + +# ---- the sizes, from the engine's quantities + + +@pytest.mark.parametrize("config", list(CONFIGS)) +def test_held_workspace_rows_from_the_runtime_features(config): + """`_most_ctx_kv` and `_ctx_workspace_rows` on each config's metadata. With neither chunked + prefill nor block reuse every context row of a step is new: max_num_tokens of KV, twice that + held (65536 at 32768 tokens, whatever max_seq_len). Either one lets a step attend cached + prefixes: min(max_num_requests, max_num_tokens) x max_seq_len, mainline's never-stall worst + case, and the held rows reach their 2**19 ceiling (the 32768 config with block reuse at + max_batch_size 16, the 1M config, 131072 with chunked prefill).""" + sizes, (most, rows, _) = CONFIGS[config] + md = _sizing(*sizes) + assert M._most_ctx_kv(md) == most + assert M._ctx_workspace_rows(md) == rows == min(2 * most, M._CTX_WORKSPACE_ROWS) + + +def test_block_reuse_is_read_from_the_runtime_features(): + """Block reuse is the metadata's `runtime_features.cache_reuse`, the executor's + `kv_cache_config.enable_block_reuse` (turned off again where it disables reuse). On the + 32768 config (16 requests, chunked prefill off): on, the never-stall worst case, 16 x 32768 + of context KV and 2**19 held rows; off, max_num_tokens and 65536 rows. Under chunked prefill + (the 1M config) the worst case holds either way. The metadata carries no KV cache manager + here: nothing else is read for it.""" + for reuse, most in ((True, 1 << 19), (False, 32768)): + md = _sizing(*PLAIN_32K[:4], reuse) + assert not hasattr(md, "kv_cache_manager") + assert md.runtime_features.cache_reuse is reuse + assert M._most_ctx_kv(md) == most, reuse + assert M._ctx_workspace_rows(md) == min(2 * most, 1 << 19) + md = _sizing(*CHUNKED_1M[:4], reuse) + assert M._most_ctx_kv(md) == 1 << 24 and M._ctx_workspace_rows(md) == 1 << 19 + # The same on the runtime features the executor flips in place when it disables reuse. + md = _sizing(*PLAIN_32K[:4], True) + md.runtime_features.cache_reuse = False + assert M._most_ctx_kv(md) == 32768 and M._ctx_workspace_rows(md) == 1 << 16 + + +@pytest.mark.parametrize( + "max_num_tokens, max_seq_len, budget, rows", + [ + pytest.param(32768, 32768, 1 << 24, 512, id="32k"), + pytest.param(16384, 32768, 1 << 24, 512, id="16k_tokens-32k"), + pytest.param(32768, 131072, 1 << 28, 2048, id="131072"), + pytest.param(8192, 1048576, 1 << 28, 256, id="8k_tokens-1m"), + pytest.param(32768, 1048576, 1 << 28, 256, id="1m"), + ], +) +def test_logits_budget_from_the_engine(monkeypatch, max_num_tokens, max_seq_len, budget, rows): + """`_prefill_logits_budget`: the reachable tile (max_num_tokens x max_seq_len, at most + mainline's 2**31 default) within the 2**28 ceiling and within max_seq_len / 64 rows at + max_seq_len keys. 2**24 at max_seq_len 32768, at 32768 or 16384 tokens: 512-row tiles of + 32768 keys. The ceiling from 131072 on: 2048 rows of 131072 keys, 256 of 1048576. A step of + max_num_tokens rows against max_seq_len keys is scored in tiles of those rows, the last one + short.""" + monkeypatch.delenv(ENV_BUDGET, raising=False) + assert M._prefill_logits_budget(max_num_tokens, max_seq_len) == budget + assert budget <= M._PREFILL_LOGITS_CEILING + tiles = M._prefill_tiles(0, max_num_tokens, max_seq_len, budget) + assert budget // max_seq_len == rows and len(tiles) == -(-max_num_tokens // rows) + assert all(hi - lo == rows for lo, hi in tiles[:-1]) and tiles[-1][1] == max_num_tokens + + +def test_logits_budget_env_override_only_lowers(monkeypatch): + """TLLM_INDEXER_MQA_LOGITS_ELEM_BUDGET lowers the reachable tile: at 2**20 every config's + budget is 2**20 (32-row tiles at 32768 keys). Above the other bounds it raises none: at 2**34 + the 32768 config stays at 2**24 and the 1M config at the ceiling.""" + monkeypatch.setenv(ENV_BUDGET, str(1 << 20)) + for sizes in ((32768, 32768), (16384, 32768), (32768, 131072), (32768, 1048576)): + assert M._prefill_logits_budget(*sizes) == 1 << 20, sizes + assert M._prefill_tiles(0, 1024, 32768, 1 << 20)[0] == (0, 32) + monkeypatch.setenv(ENV_BUDGET, str(1 << 34)) + assert M._prefill_logits_budget(32768, 32768) == 1 << 24 + assert M._prefill_logits_budget(32768, 1048576) == 1 << 28 + + +def test_rope_tables_hold_no_mla_table(): + """`_rope_tables` builds the indexer's `[cos | sin]` rows and the NoPE layers' identity rows, + nothing else: the MLA rope ops' duplicated layout, which no op reads, is not built, and + asking for it is an error.""" + alpha = torch.linspace(-0.5, 0.5, 32) + tables = M._rope_tables(ROPE_CFG, alpha, 300, torch.bfloat16, "cpu") + assert sorted(vars(tables)) == ["identity", "indexer", "rows"] + assert tables.indexer.shape == tables.identity.shape == (300, 64) + with pytest.raises(TypeError): + M._rope_tables(ROPE_CFG, alpha, 300, torch.bfloat16, "cpu", with_mla=True) + + +# ---- the generation rows' sparse decode + + +@pytest.mark.parametrize("max_seq_len", [8192, 32768, 32769, 1048576]) +@pytest.mark.parametrize("flash_mla", [False, True], ids=["native", "flash_mla"]) +def test_native_decode_built_at_every_max_seq_len(monkeypatch, max_seq_len, flash_mla): + """`build_kernels` compiles cute_dsl_ds_mla_sparse_decode at the rank's heads for the + generation rows whatever the engine's max_seq_len -- the 32K and the 1M configs alike -- and + builds no native handle only under `_GEN_ATTN_FLASH_MLA` (the core's `gen_attn_flash_mla`), + which keeps FlashMLA's 64-head decode. The shipped default is the native kernel.""" + assert M._GEN_ATTN_FLASH_MLA is False + built = [] + monkeypatch.setattr(M, "_rope_tables", lambda *a: SimpleNamespace(rows=a[2])) + monkeypatch.setattr(M, "build_ix_q_side", lambda dev, is_neox: None) + monkeypatch.setattr( + M, "build_ds_mla_decode", lambda dev, num_heads: built.append(num_heads) or "native" + ) + core = SimpleNamespace( + model_config=SimpleNamespace(pretrained_config=ROPE_CFG), + w={"lerope_alpha": None, "final_norm": torch.empty(0, dtype=torch.bfloat16)}, + rope_rows=max_seq_len, + heads=16, + unfused_glue=False, + gen_attn_flash_mla=flash_mla, + ) + M.MistralLarge4TepModelingV2.build_kernels(core, 0) + assert core._rope.rows == max_seq_len + assert core._mla_decode == (None if flash_mla else "native") + assert built == ([] if flash_mla else [16]) + + +# ---- the decode top-k's selector + + +#: Engine max_seq_lens, the decode scorer's column count, and whether cute_dsl_radix_topk_decode +#: serves it (`_TOPK_DECODE_COLUMNS`, 4096..65536): either side of both ends, the 32K and the 1M +#: configs and 131072. +TOPK_WIDTHS = [ + (2048, False), + (4095, False), + (4096, True), + (8192, True), + (32768, True), + (65536, True), + (65537, False), + (131072, False), + (1048576, False), +] + + +def _topk_md(max_seq_len: int, chunked: bool, reuse: bool) -> SimpleNamespace: + """A first step's metadata at `max_seq_len` (32768 tokens, 16 requests) on a fresh KV cache + manager, with chunked prefill and block reuse as given.""" + return SimpleNamespace( + max_num_tokens=32768, + max_num_requests=16, + max_seq_len=max_seq_len, + runtime_features=AttentionRuntimeFeatures( + chunked_prefill=chunked, cache_reuse=reuse, chunk_size=32768 + ), + kv_cache_manager=SimpleNamespace( + latent_layout=M._LATENT_LAYOUT, + kv_source_layers=KV_SOURCE, + latent_pool_row=lambda i: OWNERS.index(KV_SOURCE[i]), + kv_owner_layers=OWNERS, + ), + tokens_per_block=32, + ) + + +@pytest.mark.parametrize("max_seq_len, serves", TOPK_WIDTHS, ids=[str(w) for w, _ in TOPK_WIDTHS]) +def test_decode_topk_selector_by_the_scorer_width(monkeypatch, max_seq_len, serves): + """`_derive_first_step` holds cute_dsl_radix_topk_decode's handle exactly when the decode + scorer's column count, the engine's max_seq_len, is one the selector serves (4096..65536); + otherwise none, and both decode call sites run the trtllm indexer_topk_decode op. Nothing + else decides it: chunked prefill and block reuse, on or off, keep the one handle, built + once. A replaced KV cache manager at a width on the other side of the range flips it, and + `_TOPK_DECODE_CUTE_DSL` False builds none at any width.""" + assert M._TOPK_DECODE_CUTE_DSL is True + assert M._TOPK_DECODE_COLUMNS == (4096, 65536) + calls = [] + monkeypatch.setattr(M, "_rope_tables", lambda *a: SimpleNamespace(rows=a[2])) + monkeypatch.setattr(M, "build_ix_k_cache", lambda dev, is_neox, cache_contiguous: None) + monkeypatch.setattr( + M, "build_ix_topk_decode", lambda dev, next_n: calls.append(next_n) or object() + ) + monkeypatch.setattr(torch.cuda, "current_device", lambda: 0) + core = SimpleNamespace( + model_config=SimpleNamespace(pretrained_config=ROPE_CFG), + w={"lerope_alpha": None, "final_norm": torch.empty(0, dtype=torch.bfloat16)}, + rope_rows=16, + kv_source=KV_SOURCE, + num_layers=LAYERS, + _derived_for=None, + _ix_k=None, + _ix_topk=None, + _rope=SimpleNamespace(rows=16), + ) + derive = M.MistralLarge4TepModelingV2._derive_first_step + handles = [] + for chunked, reuse in ((False, False), (True, False), (False, True), (True, True)): + derive(core, _topk_md(max_seq_len, chunked, reuse)) + handles.append(core._ix_topk) + if serves: + assert calls == [1] and handles[0] is not None + assert all(h is handles[0] for h in handles) + else: + assert calls == [] and handles == [None] * 4 + derive(core, _topk_md(1048576 if serves else 32768, False, False)) + assert (core._ix_topk is None) is serves + monkeypatch.setattr(M, "_TOPK_DECODE_CUTE_DSL", False) + derive(core, _topk_md(32768, False, False)) + assert core._ix_topk is None + + +# ---- the indexer's context selection + + +def _packed(): + """Two context requests of 40 cached + 8 new and 30 cached + 6 new tokens, packed by the + engine into one prefill chunk of 14 rows over 84 keys: the rows' windows `[ks, ke)` in the + chunk's keys, and the metadata's host prefix sums.""" + ks = torch.tensor([0] * 8 + [48] * 6, dtype=torch.int32) + ke = torch.tensor([*range(41, 49), *range(79, 85)], dtype=torch.int32) + chunk = SimpleNamespace( + token_start=0, + token_end=14, + k_token_start=0, + k_token_end=84, + cu_seqlen_ks=ks, + cu_seqlen_ke=ke, + ) + md = SimpleNamespace( + enable_context_mla_with_cached_kv=True, + host_ctx_uncached_token_indptr=torch.tensor([0, 8, 14]), + host_ctx_kv_indptr=torch.tensor([0, 48, 84]), + ) + return chunk, md + + +def _spans(blocks) -> list[tuple[int, int, int, int]]: + return [(b.token_start, b.token_end, b.k_token_start, b.k_token_end) for b in blocks] + + +def test_ctx_offsets(): + """`_ctx_kv_offsets` hands over the context rope/append op's device offsets; the host prefix + sums of the new and the total lengths come from `_ctx_host_offsets`, off the host + `seq_lens` without cached KV and off the metadata's host copies with it.""" + md = SimpleNamespace( + enable_context_mla_with_cached_kv=False, + mla_prepare_ctx_cu_seqlens=lambda: torch.tensor([0, 8, 14], dtype=torch.int32), + seq_lens=torch.tensor([8, 6, 1], dtype=torch.int32), + ) + assert set(M._ctx_kv_offsets(md, 2)) == { + "cu_ctx_cached", + "cu_ctx_kv", + "max_ctx_new", + "ctx_cached_tokens", + } + assert M._ctx_host_offsets(md, 2) == dict(host_cu_ctx_new=[0, 8, 14], host_cu_ctx_kv=[0, 8, 14]) + _, md = _packed() + assert M._ctx_host_offsets(md, 2) == dict( + host_cu_ctx_new=[0, 8, 14], host_cu_ctx_kv=[0, 48, 84] + ) + + +def test_indexer_blocks_cut_only_past_max_seq_len(): + """A chunk of more than max_seq_len keys (84 past 64) is cut at its requests, each request's + rows against its own keys with their windows re-based to its first key. A chunk within + max_seq_len (84 at 84 and at 128) is passed through as the one block it is, though it holds + two requests. With no chunk planned, one chunk over every context row, cut by the same + rule.""" + chunk, md = _packed() + step = dict(num_ctx_tokens=14, num_ctx_kv_tokens=84, **M._ctx_host_offsets(md, 2)) + step.update(cu_seqlen_ks=chunk.cu_seqlen_ks, cu_seqlen_ke=chunk.cu_seqlen_ke) + + for chunks in ([chunk], None): + blocks = M._indexer_blocks(chunks, dict(step, max_seq_len=64)) + assert _spans(blocks) == [(0, 8, 0, 48), (8, 14, 48, 84)] + assert blocks[0].cu_seqlen_ks.tolist() == [0] * 8 + assert blocks[0].cu_seqlen_ke.tolist() == list(range(41, 49)) + assert blocks[1].cu_seqlen_ks.tolist() == [0] * 6 + assert blocks[1].cu_seqlen_ke.tolist() == list(range(31, 37)) + + for max_seq_len in (84, 128): + (whole,) = M._indexer_blocks([chunk], dict(step, max_seq_len=max_seq_len)) + assert whole is chunk + (whole,) = M._indexer_blocks(None, dict(step, max_seq_len=max_seq_len)) + assert _spans([whole]) == [(0, 14, 0, 84)] + assert torch.equal(whole.cu_seqlen_ke, chunk.cu_seqlen_ke) + + +def _step_md(max_seq_len: int, cached_kv: bool): + """A context step's metadata, every field `step_args` reads. With cached KV, the packed + chunk of `_packed` and the host prefix sums; without, two prompts of 8 and 6 tokens the + engine planned no chunk for.""" + i64 = dict(dtype=torch.int64) + md = SimpleNamespace( + kv_cache_manager=SimpleNamespace( + get_latent_pool=lambda: torch.zeros(2, 3, 32, 656, dtype=torch.uint8), + get_indexer_k_cache_buffers=lambda i: None, + ), + num_seqs=2, + num_tokens=14, + kv_lens_cuda_runtime=None, + kv_lens_runtime=None, + prompt_lens_cpu_runtime=None, + kv_cache_block_offsets=None, + host_kv_cache_pool_pointers=None, + host_kv_cache_pool_mapping=None, + tokens_per_block=32, + max_seq_len=max_seq_len, + block_table=None, + req_idx_per_token=None, + slot_mapping_fp8=None, + slot_mapping_scale=None, + skip_indexer_for_gen_reqs=False, + enable_indexer_skip=False, + indexer_k_cache_block_offsets=None, + scheduler_metadata_buffer=None, + radix_aux_indices=None, + radix_aux_logits=None, + num_contexts=2, + num_ctx_tokens=14, + seq_lens=torch.tensor([8, 6], dtype=torch.int32), + skip_indexer_for_ctx_reqs=False, + indexer_prefill_q_split=None, + num_ctx_kv_tokens=14, + slot_mapping_fp8_fullkv=None, + slot_mapping_scale_fullkv=None, + enable_context_mla_with_cached_kv=cached_kv, + ) + if cached_kv: + chunk, host = _packed() + vars(md).update( + cu_seqlen_ks=chunk.cu_seqlen_ks, + cu_seqlen_ke=chunk.cu_seqlen_ke, + indexer_prefill_chunks=[chunk], + ctx_cached_token_indptr=torch.tensor([0, 40, 70], **i64), + ctx_kv_indptr=torch.tensor([0, 48, 84], **i64), + max_ctx_seq_len=8, + num_ctx_cached_tokens=70, + host_ctx_uncached_token_indptr=host.host_ctx_uncached_token_indptr, + host_ctx_kv_indptr=host.host_ctx_kv_indptr, + ) + return md + vars(md).update( + cu_seqlen_ks=torch.tensor([0] * 8 + [8] * 6, dtype=torch.int32), + cu_seqlen_ke=torch.tensor([*range(1, 9), *range(9, 15)], dtype=torch.int32), + indexer_prefill_chunks=None, + mla_prepare_ctx_cu_seqlens=lambda: torch.tensor([0, 8, 14], dtype=torch.int32), + ) + return md + + +def _prefill_target(**core) -> M.PrefillTarget: + target = M.PrefillTarget.__new__(M.PrefillTarget) + # The selection's top-k warm-up reads the engine's max_seq_len off the core. + core.setdefault("_max_ctx_len", 1 << 20) + target.core = SimpleNamespace(**core) + return target + + +def test_step_args_carry_the_host_prefix_sums(): + """`step_args` takes the host prefix sums on every context step and hands over the scoring + blocks they cut. With cached KV (the packed chunk of two requests over 84 keys): cut at + max_seq_len 64, the engine's chunk itself at 128. Without (two fresh prompts of 8 and 6 + tokens, no chunk planned, the sums off the host `seq_lens`): cut at max_seq_len 8, one block + over the 14 rows at 32.""" + target = _prefill_target(indexer_layers=[1]) + + step = target.step_args(_step_md(64, cached_kv=True)) + assert step["ctx_cached_tokens"] == 70 and int(step["cu_ctx_kv"][-1]) == 84 + assert (step["host_cu_ctx_new"], step["host_cu_ctx_kv"]) == ([0, 8, 14], [0, 48, 84]) + assert _spans(step["indexer_blocks"]) == [(0, 8, 0, 48), (8, 14, 48, 84)] + md = _step_md(128, cached_kv=True) + step = target.step_args(md) + assert (step["host_cu_ctx_new"], step["host_cu_ctx_kv"]) == ([0, 8, 14], [0, 48, 84]) + (whole,) = step["indexer_blocks"] + assert whole is md.indexer_prefill_chunks[0] + + step = target.step_args(_step_md(8, cached_kv=False)) + assert step["ctx_cached_tokens"] == 0 and step["cu_ctx_kv"].tolist() == [0, 8, 14] + assert (step["host_cu_ctx_new"], step["host_cu_ctx_kv"]) == ([0, 8, 14], [0, 8, 14]) + blocks = step["indexer_blocks"] + assert _spans(blocks) == [(0, 8, 0, 8), (8, 14, 8, 14)] + assert blocks[1].cu_seqlen_ks.tolist() == [0] * 6 + assert blocks[1].cu_seqlen_ke.tolist() == list(range(1, 7)) + step = target.step_args(_step_md(32, cached_kv=False)) + assert (step["host_cu_ctx_new"], step["host_cu_ctx_kv"]) == ([0, 8, 14], [0, 8, 14]) + assert _spans(step["indexer_blocks"]) == [(0, 14, 0, 14)] + + +def _recorded_kernels(monkeypatch) -> list: + """Stand-ins for the context selection's kernels, recording each key gather's + `(k_token_start, n_k)`, each scored tile's rows and keys and each top-k call's windows. + Each tile's split-KV schedule is a marker of the windows and keys it was built for, which + its logits call must be handed (checked here), and the top-k's scratch a marker the target + must hand every top-k call.""" + calls = [] + + def gather(pool, fp8, scale, k_token_start, n_k, dim): + calls.append(("gather", k_token_start, n_k)) + return torch.empty(n_k, dim), torch.empty(n_k, 1) + + def schedule(ks, ke, n_k, heads): + return ("schedule", ks.tolist(), ke.tolist(), n_k, heads) + + def logits(q, kv, weights, ks, ke, schedule_meta=None): + assert schedule_meta == ("schedule", ks.tolist(), ke.tolist(), kv[0].shape[0], q.shape[1]) + calls.append(("logits", q.shape[0], kv[0].shape[0])) + return torch.empty(0) + + def topk(scores, ks, ke, out, k, workspace=None): + assert workspace == "gvr-scratch" + calls.append(("topk", ks.tolist(), ke.tolist())) + + monkeypatch.setattr(M, "indexer_k_cache_gather_op", gather) + monkeypatch.setattr(M, "mqa_logits_schedule", schedule) + monkeypatch.setattr(M, "fp8_mqa_logits", logits) + monkeypatch.setattr(M, "cute_dsl_gvr_topk_prefill", topk) + monkeypatch.setattr(M, "gvr_workspace", lambda device: "gvr-scratch") + monkeypatch.setattr(M, "gvr_warmup", lambda top_k, cols, elems, device, ws: 0) + return calls + + +@pytest.mark.parametrize( + "max_num_tokens, max_seq_len, env_budget, tile", + [ + pytest.param(32768, 32768, None, 512, id="32k"), + pytest.param(16384, 32768, None, 512, id="16k_tokens-32k"), + pytest.param(32768, 131072, None, 8192, id="131072"), + pytest.param(32768, 1048576, None, 8192, id="1m"), + pytest.param(32768, 32768, 1 << 20, 32, id="32k-env_2_20"), + ], +) +def test_select_ctx_tiles_under_the_engines_budget( + monkeypatch, max_num_tokens, max_seq_len, env_budget, tile +): + """1024 context rows over 32768 keys, scored in tiles of budget / 32768 rows under the + budget the core takes from the engine: two 512-row tiles under 2**24 (the 32768 configs), + one under 2**28 (131072 and 1M), 32 tiles of 32 rows with the override at 2**20. The keys + are gathered once for the block.""" + if env_budget is None: + monkeypatch.delenv(ENV_BUDGET, raising=False) + else: + monkeypatch.setenv(ENV_BUDGET, str(env_budget)) + calls = _recorded_kernels(monkeypatch) + rows, keys = 1024, 32768 + block = SimpleNamespace( + token_start=0, + token_end=rows, + k_token_start=0, + k_token_end=keys, + cu_seqlen_ks=torch.zeros(rows, dtype=torch.int32), + cu_seqlen_ke=torch.arange(keys - rows + 1, keys + 1, dtype=torch.int32), + ) + step = dict( + indexer_blocks=[block], slot_mapping_fp8_fullkv=None, slot_mapping_scale_fullkv=None + ) + budget = M._prefill_logits_budget(max_num_tokens, max_seq_len) + target = _prefill_target(idx_dim=128, idx_topk=2048, prefill_logits_budget=budget) + q, w, out = torch.empty(rows, 1), torch.empty(rows, 1), torch.empty(rows, 1) + target._select_ctx(step, None, q, w, out) + assert [c for c in calls if c[0] == "gather"] == [("gather", 0, keys)] + tiles = [c[1] for c in calls if c[0] == "logits"] + assert tiles == [min(tile, rows)] * -(-rows // tile) + assert all(c[2] == keys for c in calls if c[0] == "logits") + + +def test_select_ctx_fills_a_block_with_no_key(monkeypatch): + """A block with rows but no key (n_k = 0) gathers, scores and selects nothing: its rows' + selection slots are filled with -1, which every attention kernel skips. A block with no row + is skipped and writes nothing; a block with keys runs as ever.""" + calls = _recorded_kernels(monkeypatch) + blocks = [ + SimpleNamespace( + token_start=0, + token_end=3, + k_token_start=5, + k_token_end=5, + cu_seqlen_ks=torch.zeros(3, dtype=torch.int32), + cu_seqlen_ke=torch.zeros(3, dtype=torch.int32), + ), + SimpleNamespace( + token_start=3, + token_end=3, + k_token_start=0, + k_token_end=8, + cu_seqlen_ks=torch.zeros(0, dtype=torch.int32), + cu_seqlen_ke=torch.zeros(0, dtype=torch.int32), + ), + SimpleNamespace( + token_start=3, + token_end=6, + k_token_start=5, + k_token_end=13, + cu_seqlen_ks=torch.zeros(3, dtype=torch.int32), + cu_seqlen_ke=torch.tensor([6, 7, 8], dtype=torch.int32), + ), + ] + step = dict(indexer_blocks=blocks, slot_mapping_fp8_fullkv=None, slot_mapping_scale_fullkv=None) + target = _prefill_target(idx_dim=128, idx_topk=4, prefill_logits_budget=1 << 24) + out = torch.full((6, 4), 7, dtype=torch.int32) + target._select_ctx(step, None, torch.empty(6, 1), torch.empty(6, 1), out) + assert bool((out[:3] == -1).all()) and bool((out[3:] == 7).all()) + assert calls == [("gather", 5, 8), ("logits", 3, 8), ("topk", [0, 0, 0], [6, 7, 8])] + + +# ---- the context path, the held workspace and the context attention + + +def _attn_target(held: int) -> M.PrefillTarget: + """A prefill target over what its context workspace and attention read off the core: the + held workspace's rows as `_derive_first_step` sets them, and layer 35 the one shared source, + as the core builds it (`test_core_reads_nothing_off_max_seq_len`).""" + return _prefill_target( + lat_dim=LAT_DIM, + ctx_workspace_rows=held, + shared_source=[i == SHARED_SOURCE for i in range(LAYERS)], + softmax_scale=1.0, + idx_topk=4, + _pool_row=list(range(LAYERS)), + _stride_factor=1, + ) + + +def _ctx_step(kv_lens: list[int], new: int) -> dict: + """A context step's entries the context path reads: requests of `kv_lens` cached and new rows + each, `new` of them this step's.""" + return dict( + num_contexts=len(kv_lens), + num_ctx_tokens=new, + ctx_cached_tokens=sum(kv_lens) - new, + cu_ctx_kv=torch.tensor([0, *kv_lens]).cumsum(0), + latent_pool=None, + latent_kv=None, + block_table=None, + tokens_per_block=32, + ) + + +def _row_ptr(buf: torch.Tensor, row: int) -> int: + return buf.data_ptr() + row * buf.stride(0) * buf.element_size() + + +def test_context_path_by_the_steps_kv(): + """Steps on one target whose held workspace is 64 rows, by their context KV n_kv: 16 and 32 + fit twice (the workspace path over rows [0, n_kv), rows [n_kv, 2 n_kv) kept); 33 and 64 fit + once (the workspace path, nothing kept); 65 does not fit (the decode path, the context rows + in one sub-batch). The first step allocates the bf16 [64, 576] workspace, and every later + step, either path, uses that very storage. Each request's rows start at its cumulative KV + offset.""" + target = _attn_target(64) + cpu = torch.device("cpu") + first = target._ctx_workspace(_ctx_step([10, 6], 9), 9, cpu) + buf = target._ws_buf + assert buf.shape == (64, LAT_DIM) and buf.dtype == torch.bfloat16 + assert first.kv_req.tolist() == [0] * 10 + [1] * 6 + assert first.kv_pos.tolist() == [*range(10), *range(6)] + assert first.ws_off.tolist() == [0, 10] and first.ws == {} + for kv_lens, new, kept in ( + ([10, 6], 9, True), + ([20, 12], 12, True), + ([30, 3], 5, False), + ([40, 24], 24, False), + ): + n_kv = sum(kv_lens) + ctx = target._ctx_workspace(_ctx_step(kv_lens, new), new, cpu) + assert target._ws_buf is buf + assert ctx.ws_buf.data_ptr() == buf.data_ptr() and ctx.ws_buf.shape == (n_kv, LAT_DIM) + assert ctx.kv_req.shape == ctx.kv_pos.shape == (n_kv,) + if kept: + assert ctx.kept.shape == (n_kv, LAT_DIM) + assert ctx.kept.data_ptr() == _row_ptr(buf, n_kv) + else: + assert ctx.kept is None + past = target._ctx_workspace(_ctx_step([40, 25], 25), 25, cpu) + assert past.ws_buf is None and past.sub_batches == [(0, 25)] and sorted(past.sched) == [25] + assert target._ws_buf is buf + + +def test_decode_path_sub_batches_and_its_first_step_allocates(): + """A first context step past the held rows (20000 new rows over 4096 cached, held 2**14) + takes the decode path and still allocates the held workspace, so the KV-capacity estimate's + peak holds it whichever path its step takes. The context rows run in sub-batches of 8192, one + FlashMLA scheduler per sub-batch size.""" + target = _attn_target(1 << 14) + ctx = target._ctx_workspace(_ctx_step([24096], 20000), 20000, torch.device("cpu")) + assert target._ws_buf.shape == (1 << 14, LAT_DIM) + assert ctx.ws_buf is None + assert ctx.sub_batches == [(0, 8192), (8192, 16384), (16384, 20000)] + assert sorted(ctx.sched) == [3616, 8192] + + +@pytest.mark.parametrize( + "config, kv_lens, new, path", + [ + # A full step of the 32768 config: n_kv 32768 fits its 65536 rows twice, exactly. + pytest.param("32k", [32768], 32768, "kept", id="32k-full_step"), + pytest.param("32k", [7, 33], 40, "kept", id="32k-two_prompts"), + # The 1M config's first chunk, its 16th (n_kv 2**19), its 17th. + pytest.param("1m-chunked", [32768], 32768, "kept", id="1m-first_chunk"), + pytest.param("1m-chunked", [1 << 19], 32768, "workspace", id="1m-16th_chunk"), + pytest.param("1m-chunked", [(1 << 19) + 32768], 32768, "decode", id="1m-17th_chunk"), + # The 32768 config with block reuse: a full step over a cached prefix. + pytest.param("32k-block_reuse", [32768] * 4, 32768, "kept", id="32k_reuse-4x32k"), + ], +) +def test_serving_configs_steps(config, kv_lens, new, path): + """Steps of the serving configs on a target holding the rows `_ctx_workspace_rows` gives the + config: the 32768 config's 65536, the 1M config's and the 32768 config with block reuse's + 2**19. The path each step takes, and whether layer 35's gather is kept beside its rows.""" + sizes, (_, rows, _) = CONFIGS[config] + assert M._ctx_workspace_rows(_sizing(*sizes)) == rows + target = _attn_target(rows) + ctx = target._ctx_workspace(_ctx_step(kv_lens, new), new, torch.device("cpu")) + n_kv = sum(kv_lens) + assert target._ws_buf.shape == (rows, LAT_DIM) + if path == "decode": + assert n_kv > rows and ctx.ws_buf is None + assert ctx.sub_batches == [(lo, lo + 8192) for lo in range(0, new, 8192)] + return + assert n_kv <= rows and ctx.ws_buf.shape == (n_kv, LAT_DIM) + assert ctx.ws_buf.data_ptr() == target._ws_buf.data_ptr() + assert (ctx.kept is not None) == (path == "kept") == (2 * n_kv <= rows) + + +def _recorded_attention(monkeypatch) -> SimpleNamespace: + """Stand-ins for the context attention's kernels: each dequantizing gather's source row and + output buffer, each sparse prefill's workspace, each slot conversion's source row and each + sparse decode call's rows and scheduler.""" + seen = SimpleNamespace(gathers=[], prefills=[], converts=[], decodes=[]) + + def gather(pool, row, kv_req, kv_pos, block_table, out): + seen.gathers.append((row, out)) + return out + + def convert(req_idx, block_table, sel, tokens_per_block, topk, stride, row): + seen.converts.append(row) + return torch.zeros(sel.shape, dtype=torch.int32) + + def decode(q, kv, slots, sched, scale, out): + seen.decodes.append((q.shape[0], sched)) + + monkeypatch.setattr(M, "ds_mla_dequant_gather", gather) + monkeypatch.setattr( + M, "flash_mla_sparse_prefill", lambda q, ws, idx, scale, out: seen.prefills.append(ws) + ) + monkeypatch.setattr(M, "convert_req_index_to_global", convert) + monkeypatch.setattr(M, "flash_mla_sparse_decode", decode) + return seen + + +def _attend(target, step, ctx, tc: int) -> None: + """Layers 35, 39 (attending over 35's latent), 40 and 43 (over 35's) on the step's context + rows, as the forward runs them.""" + req_idx = torch.zeros(tc, dtype=torch.long) + sel = torch.zeros(tc, 4, dtype=torch.int32) + for layer in (SHARED_SOURCE, 39, 40, 43): + L = SimpleNamespace(kv_source=KV_SOURCE[layer]) + target._attn_ctx(step, L, torch.empty(tc, 1), torch.empty(tc, 1), req_idx, sel, ctx) + + +def test_shared_source_kept_when_it_fits_twice_else_gathered_again(monkeypatch): + """Two steps on one target holding 64 rows, each running layers 35, 39, 40 and 43. Context KV + 16 fits twice: layer 35's rows are gathered once, into the kept rows [16, 32), and remembered + for the step; 39 and 43 read them without a gather of their own, and 40 gathers its own + source into rows [0, 16). Context KV 40 fits once: nothing is kept, every layer gathers its + source's rows into rows [0, 40), 35's again for 39 and for 43. Both steps use the one held + buffer.""" + seen = _recorded_attention(monkeypatch) + target = _attn_target(64) + step = _ctx_step([10, 6], 9) + ctx = target._ctx_workspace(step, 9, torch.device("cpu")) + buf = target._ws_buf + _attend(target, step, ctx, 9) + assert [row for row, _ in seen.gathers] == [SHARED_SOURCE, 40] + assert seen.gathers[0][1] is ctx.kept and seen.gathers[1][1] is ctx.ws_buf + assert ctx.kept.data_ptr() == _row_ptr(buf, 16) + assert ctx.ws == {SHARED_SOURCE: ctx.kept} + assert [ws is ctx.kept for ws in seen.prefills] == [True, True, False, True] + assert seen.prefills[2] is ctx.ws_buf + + seen.gathers.clear() + seen.prefills.clear() + step = _ctx_step([30, 10], 9) + ctx = target._ctx_workspace(step, 9, torch.device("cpu")) + assert target._ws_buf is buf and ctx.kept is None + _attend(target, step, ctx, 9) + assert [row for row, _ in seen.gathers] == [SHARED_SOURCE, SHARED_SOURCE, 40, SHARED_SOURCE] + assert all(out is ctx.ws_buf for _, out in seen.gathers) + assert ctx.ws_buf.data_ptr() == buf.data_ptr() + assert all(ws is ctx.ws_buf for ws in seen.prefills) and ctx.ws == {} + assert not seen.converts and not seen.decodes + + +def test_decode_path_attends_off_the_pool(monkeypatch): + """A step past the held rows (9000 new rows over 9000 keys, held 64): no gather and no + prefill; each layer converts its selections to pool slots of its source's owner row, then + runs the sparse decode over its sub-batches of 8192 and 808 rows, each on that size's + scheduler.""" + seen = _recorded_attention(monkeypatch) + target = _attn_target(64) + step = _ctx_step([9000], 9000) + ctx = target._ctx_workspace(step, 9000, torch.device("cpu")) + _attend(target, step, ctx, 9000) + assert not seen.gathers and not seen.prefills + assert seen.converts == [SHARED_SOURCE, SHARED_SOURCE, 40, SHARED_SOURCE] + expect = [(8192, ctx.sched[8192]), (808, ctx.sched[808])] * 4 + assert [rows for rows, _ in seen.decodes] == [rows for rows, _ in expect] + assert all(sched is want for (_, sched), (_, want) in zip(seen.decodes, expect)) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_weight_manifest.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_weight_manifest.py new file mode 100644 index 000000000000..46c9e00baab4 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tep/test_modeling_v2_weight_manifest.py @@ -0,0 +1,1496 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The Mistral Large 4 NVFP4 tep4 weight table against the checkpoint it reads, at tep4 (four ranks: +tp 4, moe_ep 4, moe_tp 1, one node) and at tep8 (eight ranks: tp 8, moe_ep 8, moe_tp 1, two +nodes of four). + +No GPU. Three kinds of evidence, each at both world sizes: + +* the table at the checkpoint's own language-model configuration (written out below, and + compared with the configuration adapted from the checkpoint's `params.json` when it is + present), on the ranks of the engine's mapping: which layers carry which weight at what + per-rank shape, and that each rank's manifest reads only its own routed experts, 32 at tep4 + and 16 at tep8; +* the checkpoint itself, through its index and safetensors headers, as meta tensors, plus the + values of its small scale tensors: the union of the ranks' sources is exactly the index's + keys but the DSpark draft's, which every rank declares unconsumed; each rank consumes every + one of the 1,721 language-model keys, the 276 vision-tower keys and its window's expert keys, + 27,264 and 146.42 GiB at tep4, 13,632 and 76.78 GiB at tep8, 3.17 GiB of it the vision tower + every rank reads whole; the index the production `load` takes of every key on every rank -- + the ranks' head, row, column and vocabulary ranges and expert windows tile each sliced tensor + exactly once, everything else is read whole; the production `load` -- every slice, transform, + shape and dtype assert, the expert-scalar asserts on the checkpoint's own scalars and the + two-way coverage assert -- passes on each rank; every FP8 block scale and every expert + scalar satisfies the invariants `load` asserts; and the packed UE8M0 scale operand at each K + a rank's slice gives (at tep8 `wo` at 1024, the dense `w2` at 3072, the shared expert's `w2` + at 768: 6 blocks, a half-padding second word; at tep4 2048, 6144 and 1536, whole words) + equals an independent byte-level packing and the engine's + `transform_sf_into_required_layout`. Skipped where the checkpoint is absent (looked up as + `mistral-large-4-nvfp4` or `Mistral-Large-4-1T-A52B-NVFP4` under `LLM_MODELS_ROOT`); +* a scaled-down configuration per world size with real CPU tensors, loaded on every rank and + compared element for element with an independent construction of what each parameter must + hold: the slices, the `wkv_b` regroup, both half orders, the packed scale operand padding + included (at tep8 the dense `w2` at a 6-block K too), the expert window in the MoE runner's + layout and its scalars, and the DSpark draft tensors left alone; plus what the streaming path + reads off the shard files, and each assert's refusal. +""" + +from __future__ import annotations + +import json +import re +import struct +from pathlib import Path +from types import SimpleNamespace + +import pytest +import torch +import torch.nn as nn +from safetensors import safe_open +from test_common.llm_data import llm_models_root + +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_103__tep import ( # noqa: E501 + weights as W, +) +from tensorrt_llm._torch.models.checkpoints.mistral.config_loader import MistralConfigLoader +from tensorrt_llm.mapping import Mapping +from tensorrt_llm.quantization.utils.fp8_utils import ( + check_sf_layout, + inverse_transform_sf, + transform_sf_into_required_layout, +) + +pytestmark = pytest.mark.cpu_only + +#: The world sizes the table serves: tep4 on one node, tep8 over two. +WORLDS = (4, 8) +#: GB300 nodes carry four devices: tep4 is one node, tep8 spans two. +GPUS_PER_NODE = 4 +CHECKPOINTS = ("mistral-large-4-nvfp4", "Mistral-Large-4-1T-A52B-NVFP4") + +_LAYERS = 72 +#: The BF16 vision tower -- encoder, pre-projector norm, patch merger and language adapter -- +#: which every rank reads whole: its keys and bytes in the checkpoint (3.17 GiB). +_VISION_KEYS, _VISION_BYTES = 276, 3_408_437_248 +#: The DSpark draft's keys, all in the checkpoint's last shard, which no rank reads. +_DRAFT_KEYS = 38 + +#: The language-model configuration surface the table reads, at the checkpoint's values. The +#: checkpoint's configuration adds its vision encoder, whose rows are checked against the +#: checkpoint below. +_CONFIG = dict( + num_hidden_layers=_LAYERS, + hidden_size=6144, + num_attention_heads=64, + qk_nope_head_dim=64, + qk_rope_head_dim=64, + v_head_dim=128, + kv_lora_rank=512, + nope_layer_flags=[i % 4 == 3 for i in range(_LAYERS)], + kv_source_layers=[35 if i >= 39 and i % 4 == 3 else i for i in range(_LAYERS)], + index_sharing=["shared" if i >= 17 and (i - 17) % 4 < 2 else "full" for i in range(_LAYERS)], + index_n_heads=32, + index_head_dim=128, + first_k_dense_replace=1, + n_routed_experts=128, + moe_intermediate_size=6144, + n_shared_experts=1, + intermediate_size=24576, + vocab_size=204800, +) + +_SAFETENSORS_DTYPES = { + "BF16": torch.bfloat16, + "F32": torch.float32, + "F8_E4M3": torch.float8_e4m3fn, + "U8": torch.uint8, +} + +_EXPERT_PARAMS = ("fc1_w", "fc1_s", "fc2_w", "fc2_s", "isc1", "isc2", "ws2_13", "ws2_2") +_EXPERT_SUFFIXES = { + f"{proj}.{s}" + for proj in ("w1", "w2", "w3") + for s in ("weight", "weight_scale", "weight_scale_2", "input_scale") +} + + +@pytest.fixture(autouse=True) +def _no_device_memory(): + """Nothing here allocates on a GPU. The context alone proves nothing: `import tensorrt_llm` + creates one wherever a device is visible.""" + yield + if torch.cuda.is_initialized(): + assert torch.cuda.memory_reserved() == 0, "the weight table test allocated on a GPU" + + +def _mapping(world: int, rank: int) -> Mapping: + m = Mapping( + world_size=world, + rank=rank, + gpus_per_node=GPUS_PER_NODE, + tp_size=world, + moe_tp_size=1, + moe_ep_size=world, + ) + # A topology that ignores `rank=` would hand every rank rank 0's slices. + assert (m.tp_rank, m.moe_ep_rank, m.moe_tp_rank) == (rank, rank, 0), m + return m + + +def _core(config: dict, world: int, rank: int) -> SimpleNamespace: + return SimpleNamespace( + model_config=SimpleNamespace( + pretrained_config=SimpleNamespace(**config), + mapping=_mapping(world, rank), + torch_dtype=torch.bfloat16, + ) + ) + + +class _Core(nn.Module): + """The surface `load` reads off the core: its config and its declared weights.""" + + def __init__(self, config: dict, world: int, rank: int, device: str): + super().__init__() + self.model_config = _core(config, world, rank).model_config + with torch.device(device): + self.w = W.MODEL_WEIGHTS.declare(W.MODEL_WEIGHTS.dims(self)) + + +class _Shell(nn.Module): + """The causal-LM shell: the core as `model`, a vocab-parallel `lm_head`.""" + + def __init__(self, config: dict, world: int, rank: int, device: str): + super().__init__() + self.model = _Core(config, world, rank, device) + self.lm_head = nn.Module() + rows = config["vocab_size"] // world + self.lm_head.weight = nn.Parameter( + torch.empty(rows, config["hidden_size"], dtype=torch.bfloat16, device=device), + requires_grad=False, + ) + + +def _source_keys(manifest: dict) -> set: + """Every checkpoint key the manifest reads, key tuples flattened.""" + return {k for sources in manifest.values() for ckpt, *_ in sources for k in W._keys(ckpt)} + + +def _is_vision(key: str) -> bool: + return key.startswith(W.VISION_PREFIXES) + + +def _is_draft(key: str) -> bool: + """A DSpark draft key: the speculative-decoding head, which these targets do not run.""" + return key.startswith("mtp_dspark.") + + +def _is_expert(key: str) -> bool: + return ".experts." in key + + +def _is_expert_scalar(key: str) -> bool: + return _is_expert(key) and key.endswith((".weight_scale_2", ".input_scale")) + + +def _role(key: str) -> str: + """A parameter key's weight role: `l{i}_` per layer, the key itself otherwise.""" + head, _, rest = key.partition("_") + return rest if head[:1] == "l" and head[1:].isdigit() else key + + +def _expert_of(key: str) -> tuple[int, int, str]: + """`layers.N.experts.E.` -> (N, E, suffix).""" + parts = key.split(".") + assert parts[0] == "layers" and parts[2] == "experts", key + return int(parts[1]), int(parts[3]), ".".join(parts[4:]) + + +def _packed(grid: torch.Tensor) -> torch.Tensor: + """Independent construction of the packed scale storage from fp32 `[N/128, K/128]` block + scales: `[ceil(K/512), N]` int32, word `[w, n]` holding the UE8M0 bytes (the fp32 exponent + field) of K blocks 4w..4w+3 of row n's block, low byte first, and zero bytes past the last K + block. Integer arithmetic only, so it shares no code with the engine's packer.""" + n_blk, k_blk = grid.shape + words = -(-k_blk // 4) + exps = torch.zeros(n_blk, 4 * words, dtype=torch.int64) + exps[:, :k_blk] = (grid.contiguous().view(torch.int32).to(torch.int64) >> 23) & 0xFF + packed = exps[:, 0::4] | exps[:, 1::4] << 8 | exps[:, 2::4] << 16 | exps[:, 3::4] << 24 + packed = torch.where(packed >= 2**31, packed - 2**32, packed).to(torch.int32) + return packed.repeat_interleave(128, dim=0).t().contiguous() + + +# --------------------------------------------------------------------- the configuration + + +def _layers_of(declared, name: str) -> dict[int, tuple]: + """Layer index -> declared shape, for every `l{i}_`.""" + out = {} + for key, param in declared.items(): + head, _, rest = key.partition("_") + if head.startswith("l") and head[1:].isdigit() and rest == name: + out[int(head[1:])] = tuple(param.shape) + return out + + +#: The per-rank shapes, counts and bytes of the checkpoint's configuration that depend on the +#: world size. tep4: 16 of the 64 heads, the dense MLP at 6144 and the shared expert at 1536 of +#: their widths, 32 routed experts, an lm_head of 51200 rows. tep8: 8 heads, 3072, 768, 16 +#: routed experts, 25600 rows. `gib` is a rank's language model, `read_gib` what it reads off +#: the checkpoint: that and the vision tower's 3.17 GiB. +_AT = { + 4: SimpleNamespace( + heads=16, + dense=6144, + shared=1536, + experts=32, + lm_rows=51200, + wq=(2048, 6144), + wq_s=(12, 2048), + ssmax=(16,), + # 16 heads x [k_nope 128 | v 128] on NoPE layers, [k_nope 64 | v 128] on RoPE layers. + wkv_b_nope=(4096, 512), + wkv_b_rope=(3072, 512), + wo=(6144, 2048), + wo_s=(4, 6144), + gu_dense=(12288, 6144), + gu_shared=(3072, 6144), + gu_s_dense=(12, 12288), + gu_s_shared=(12, 3072), + dn_dense=(6144, 6144), + dn_shared=(6144, 1536), + dn_s_dense=(12, 6144), + dn_s_shared=(3, 6144), + expert_keys=27264, + gib=143.24, + gib_hi=143.30, + expert_gib=134.79, + read_gib=146.42, + ), + 8: SimpleNamespace( + heads=8, + dense=3072, + shared=768, + experts=16, + lm_rows=25600, + wq=(1024, 6144), + wq_s=(12, 1024), + ssmax=(8,), + # 8 heads x [k_nope 128 | v 128] on NoPE layers, [k_nope 64 | v 128] on RoPE layers. + wkv_b_nope=(2048, 512), + wkv_b_rope=(1536, 512), + wo=(6144, 1024), + # K 1024: 8 blocks, two whole words. + wo_s=(2, 6144), + gu_dense=(6144, 6144), + gu_shared=(1536, 6144), + gu_s_dense=(12, 6144), + gu_s_shared=(12, 1536), + dn_dense=(6144, 3072), + dn_shared=(6144, 768), + # K 3072: 24 blocks, six words; K 768: 6 blocks, two words, the second half padding. + dn_s_dense=(6, 6144), + dn_s_shared=(2, 6144), + expert_keys=13632, + gib=73.61, + gib_hi=73.66, + expert_gib=67.39, + read_gib=76.78, + ), +} + + +@pytest.mark.parametrize("world, rank", [(w, r) for w in WORLDS for r in (0, w - 1)]) +def test_layer_sets_and_per_rank_shapes(world, rank): + """Which layers carry which weight, at what per-rank shape: at tep4 16 of the 64 heads + (`wq` 2048 rows, `wo` 2048 columns, `wkv_b` 16 heads, SSMax 16), the dense MLP at 6144 and + the shared expert at 1536 of their widths, 32 routed experts; at tep8 8 heads (1024, 1024, + 8, 8), 3072, 768 and 16 routed experts. The NoPE layers are 3, 7, ..., 71; the KV sharers + 39, 43, ..., 71; the 'shared' indexer layers 17, 18, 21, 22, ..., 69, 70.""" + at = _AT[world] + core = _core(_CONFIG, world, rank) + with torch.device("meta"): + declared = W.MODEL_WEIGHTS.declare(W.MODEL_WEIGHTS.dims(core)) + nope = {i for i in range(_LAYERS) if i % 4 == 3} + sharers = {i for i in range(39, _LAYERS, 4)} + full = {i for i in range(_LAYERS) if i < 17 or (i - 17) % 4 >= 2} + moe = range(1, _LAYERS) + assert len(nope) == 18 and len(sharers) == 9 and len(full) == 44 + + every = {i: None for i in range(_LAYERS)} + expect = { + "attn_norm": {i: (6144,) for i in every}, + "wq": {i: at.wq for i in every}, + "wq_s": {i: at.wq_s for i in every}, + "ssmax_s": {i: at.ssmax for i in every}, + "ssmax_b": {i: at.ssmax for i in every}, + "wkv_a": {i: (512 if i in nope else 576, 6144) for i in range(_LAYERS) if i not in sharers}, + "kv_a_norm": {i: (512,) for i in range(_LAYERS) if i not in sharers}, + "wkv_b": {i: at.wkv_b_nope if i in nope else at.wkv_b_rope for i in every}, + "wo": {i: at.wo for i in every}, + "wo_s": {i: at.wo_s for i in every}, + "ffn_norm": {i: (6144,) for i in every}, + "ix_q_norm_w": {i: (6144,) for i in full}, + "ix_q_norm_b": {i: (6144,) for i in full}, + "ix_wq_b": {i: (4096, 6144) for i in full}, + "ix_wq_b_s": {i: (12, 4096) for i in full}, + "ix_wk_wp": {i: (160, 6144) for i in full}, + "ix_k_norm_w": {i: (128,) for i in full}, + "ix_k_norm_b": {i: (128,) for i in full}, + "ix_ssmax_s": {i: (32,) for i in full}, + "ix_ssmax_b": {i: (32,) for i in full}, + "mlp_gu": {i: at.gu_dense if i == 0 else at.gu_shared for i in every}, + "mlp_gu_s": {i: at.gu_s_dense if i == 0 else at.gu_s_shared for i in every}, + "mlp_dn": {i: at.dn_dense if i == 0 else at.dn_shared for i in every}, + "mlp_dn_s": {i: at.dn_s_dense if i == 0 else at.dn_s_shared for i in every}, + "router": {i: (128, 6144) for i in moe}, + "fc1_w": {i: (at.experts, 12288, 3072) for i in moe}, + "fc1_s": {i: (at.experts, 12288, 384) for i in moe}, + "fc2_w": {i: (at.experts, 6144, 3072) for i in moe}, + "fc2_s": {i: (at.experts, 6144, 384) for i in moe}, + "isc1": {i: (1,) for i in moe}, + "isc2": {i: (1,) for i in moe}, + "ws2_13": {i: (at.experts,) for i in moe}, + "ws2_2": {i: (at.experts,) for i in moe}, + } + for name, want in expect.items(): + assert _layers_of(declared, name) == want, name + per_layer = {k for k in declared if k.startswith("l") and k.split("_")[0][1:].isdigit()} + assert len(per_layer) == sum(len(v) for v in expect.values()), "an undeclared-for role" + assert {k: tuple(v.shape) for k, v in declared.items() if k not in per_layer} == { + "final_norm": (6144,), + "embed": (204800, 6144), + "lerope_alpha": (32,), + } + dtypes = {_role(k): v.dtype for k, v in declared.items()} + assert dtypes["wq"] == dtypes["mlp_dn"] == dtypes["ix_wq_b"] == torch.float8_e4m3fn + assert dtypes["wq_s"] == dtypes["mlp_gu_s"] == dtypes["ix_wq_b_s"] == torch.int32 + assert dtypes["fc1_w"] == dtypes["fc2_w"] == torch.uint8 + assert dtypes["fc1_s"] == dtypes["fc2_s"] == torch.float8_e4m3fn + assert dtypes["isc1"] == dtypes["isc2"] == dtypes["ws2_13"] == dtypes["ws2_2"] == torch.float32 + assert dtypes["ix_k_norm_w"] == dtypes["lerope_alpha"] == torch.float32 + assert dtypes["wkv_b"] == dtypes["router"] == dtypes["embed"] == torch.bfloat16 + + # Per rank, with the shell's lm_head (51200 rows at tep4, 25600 at tep8). The routed + # experts are exactly their checkpoint bytes; the FP8 scale operands, one int32 word per + # four 128-blocks per output row, are a few MiB more than the fp32 grids they are built from. + nbytes = {k: p.numel() * p.element_size() for k, p in declared.items()} + experts = sum(n for k, n in nbytes.items() if _role(k) in _EXPERT_PARAMS) + total = sum(nbytes.values()) + at.lm_rows * 6144 * 2 + assert round(experts / 2**30, 2) == at.expert_gib, experts / 2**30 + assert at.gib * 2**30 < total < at.gib_hi * 2**30, total / 2**30 + + +@pytest.mark.parametrize("world", WORLDS) +def test_each_rank_reads_only_its_expert_window(world): + """Rank r's manifest names routed experts [n r, n r + n) and nothing else -- n is 32 at + tep4, 16 at tep8 -- on every MoE layer, local slot s <- global expert n r + s, all twelve + tensors of each; the one-per-layer scalars read the window's copies. The windows tile the + 128 experts. The known-not-loaded set is the other 96 experts at tep4, 112 at tep8: their + scalars predicted, their codes and block scales only where found.""" + n = _AT[world].experts + seen = [] + for rank in range(world): + core = _core(_CONFIG, world, rank) + manifest = W.MODEL_WEIGHTS.manifest(core) + d = W.MODEL_WEIGHTS.dims(core) + assert (d.local_experts, d.expert_offset) == (n, n * rank) + lo, hi = n * rank, n * rank + n + experts: dict = {} + for key, sources in manifest.items(): + for ckpt, index, *_ in sources: + keys = W._keys(ckpt) + if not any(_is_expert(k) for k in keys): + continue + assert all(_is_expert(k) for k in keys), (key, ckpt) + param = _role(key) + assert param in _EXPERT_PARAMS, key + for k in keys: + layer, e, suffix = _expert_of(k) + assert lo <= e < hi, (rank, key, k) + assert key == f"l{layer}_{param}", (key, k) + if param in ("isc1", "isc2"): + assert index == (0,), (key, index) + else: + assert index[0] == e - lo, (key, k, index) + experts.setdefault((layer, e), set()).add(suffix) + assert set(experts) == {(i, e) for i in range(1, _LAYERS) for e in range(lo, hi)} + assert all(v == _EXPERT_SUFFIXES for v in experts.values()) + seen.extend(experts) + + predicted = W.MODEL_WEIGHTS.expected_unconsumed(core, {}) + assert len(predicted) == 71 * (128 - n) * 6 + assert all(_is_expert_scalar(k) for k in predicted) + assert all(not lo <= _expert_of(k)[1] < hi for k in predicted) + # Exactly once: no (layer, expert) in two windows, none in no window. + assert sorted(seen) == [(i, e) for i in range(1, _LAYERS) for e in range(128)] + + +@pytest.mark.parametrize("world", WORLDS) +def test_manifest_and_declaration_agree(world): + """A declared weight with no source allocates memory nothing fills.""" + for rank in range(world): + core = _core(_CONFIG, world, rank) + manifest = W.MODEL_WEIGHTS.manifest(core) + with torch.device("meta"): + declared = W.MODEL_WEIGHTS.declare(W.MODEL_WEIGHTS.dims(core)) + assert set(manifest) == set(declared) + assert all(manifest.values()) + + +# --------------------------------------------------------------------- the checkpoint + + +def _checkpoint_dir() -> Path | None: + root = llm_models_root() + if root is None: + return None + for name in CHECKPOINTS: + path = root / name + if (path / "params.json").is_file() and (path / W.SHARD_INDEX).is_file(): + return path + return None + + +@pytest.fixture(scope="module") +def checkpoint(): + path = _checkpoint_dir() + if path is None: + pytest.skip(f"checkpoint {CHECKPOINTS} is not under LLM_MODELS_ROOT ({llm_models_root()})") + index = json.loads((path / W.SHARD_INDEX).read_text())["weight_map"] + headers = {} + for name in sorted(set(index.values())): + with open(path / name, "rb") as f: + (size,) = struct.unpack(" dict: + return { + key: torch.empty(shape, dtype=dtype, device="meta") + for key, (dtype, shape) in checkpoint.headers.items() + } + + +def test_checkpoint_config_is_the_one_tested(checkpoint): + """The configuration above is the checkpoint's, on every key the table reads.""" + assert {k: checkpoint.config[k] for k in _CONFIG} == _CONFIG + + +#: The FP8 128x128-block linears (`FP8_PB_WO`): e4m3 codes, fp32 `[N/128, 1, K/128, 1]` scales. +_FP8_LINEAR = re.compile( + r"layers\.\d+\.(attention\.(wq|wo|indexer\.wq_b)|feed_forward\.w[123]|shared_experts\.w[123])" +) + + +def test_checkpoint_dtypes_by_role(checkpoint): + """What each key stores, which is what the table's rows read: the FP8 linears e4m3 with + fp32 block scales, the routed experts NVFP4 (packed codes, e4m3 block scales, fp32 + per-tensor scalars), every other language-model tensor and the vision tower bf16.""" + for key, (dtype, shape) in checkpoint.headers.items(): + if _is_draft(key): + continue + module, _, leaf = key.rpartition(".") + if _is_expert(key): + want = { + "weight": torch.uint8, + "weight_scale": torch.float8_e4m3fn, + "weight_scale_2": torch.float32, + "input_scale": torch.float32, + }[leaf] + assert dtype == want, key + assert (shape == ()) == _is_expert_scalar(key), (key, shape) + elif _FP8_LINEAR.fullmatch(module): + want = {"weight": torch.float8_e4m3fn, "weight_scale": torch.float32}[leaf] + assert dtype == want, key + else: + assert dtype == torch.bfloat16, key + + +@pytest.mark.parametrize("world", WORLDS) +def test_union_over_ranks_is_every_key_but_the_draft(checkpoint, world): + """Every key of the index but the DSpark draft's 38 is read by some rank (lm_head's + `output.weight` by the shell); per rank, read and known-not-loaded split the index, the + known-not-loaded the draft and the other ranks' experts, and a rank reads every one of the + 1,721 language-model keys, the 276 vision-tower keys and its window's expert keys: 27,264 + at tep4 (71 layers x 32 experts x 3 projections x 4 tensors), 13,632 at tep8 + (71 x 16 x 3 x 4).""" + at = _AT[world] + keys = set(checkpoint.index) + draft = {k for k in keys if _is_draft(k)} + vision = {k for k in keys if _is_vision(k)} + lm = {k for k in keys - draft - vision if not _is_expert(k)} + assert len(keys) == 111091 and len(draft) == _DRAFT_KEYS and len(vision) == _VISION_KEYS + assert len(lm) == 1721 + union = set() + for rank in range(world): + core = _core(checkpoint.config, world, rank) + read = _source_keys(W.MODEL_WEIGHTS.manifest(core)) | {"output.weight"} + skipped = W.MODEL_WEIGHTS.expected_unconsumed(core, checkpoint.index) + assert not read & set(skipped) + assert read | set(skipped) == keys + assert {k for k in read if not _is_expert(k)} == lm | vision + assert len({k for k in read if _is_expert(k)}) == at.expert_keys == 71 * at.experts * 12 + assert {k for k in skipped if not _is_expert(k)} == draft + assert {skipped[k] for k in draft} == {"DSpark draft disabled"} + assert len(skipped) == _DRAFT_KEYS + 71 * (128 - at.experts) * 12 + union |= read + assert union == keys - draft + + +@pytest.mark.parametrize("world", WORLDS) +def test_per_rank_bytes_from_the_headers(checkpoint, world): + """What each rank moves, from the safetensors headers through the production slices: the + same 146.42 GiB on every rank at tep4, 134.79 GiB of it its routed-expert window; 76.78 GiB + at tep8, 67.39 GiB of it its window; at both, 3.17 GiB of it the vision tower, whole, and + 143.24 and 73.61 GiB the language model.""" + at = _AT[world] + for rank in range(world): + core = _core(checkpoint.config, world, rank) + d = W.MODEL_WEIGHTS.dims(core) + meta = {} + + def entry(key): + if key not in meta: + dtype, shape = checkpoint.headers[key] + meta[key] = torch.empty(shape, dtype=dtype, device="meta") + return meta[key] + + total = experts = vision = 0 + for sources in W.MODEL_WEIGHTS.manifest(core).values(): + for ckpt, _, _, shard in sources: + for key in W._keys(ckpt): + t = W._materialize(entry(key), shard) + n = t.numel() * t.element_size() + total += n + experts += n if _is_expert(key) else 0 + vision += n if _is_vision(key) else 0 + dtype, (vocab, hidden) = checkpoint.headers["output.weight"] + total += vocab // d.tp_size * hidden * torch.empty((), dtype=dtype).element_size() + assert vision == _VISION_BYTES, (rank, vision) + assert round(total / 2**30, 2) == at.read_gib, (rank, total / 2**30) + assert round((total - vision) / 2**30, 2) == at.gib, (rank, (total - vision) / 2**30) + assert round(experts / 2**30, 2) == at.expert_gib, (rank, experts / 2**30) + + +@pytest.mark.parametrize("world, rank", [(w, r) for w in WORLDS for r in range(w)]) +def test_load_on_meta_matches_the_headers(checkpoint, expert_scalars, world, rank): + """The production `load` over meta tensors shaped like the safetensors headers, with the + checkpoint's own expert scalars: the scalar asserts pass on every layer, every source is + sliced and transformed to its declared per-rank shape and dtype, every declared parameter + is filled, and every key is consumed or known-not-loaded.""" + weights = _meta(checkpoint) + weights.update(expert_scalars) + model = _Shell(checkpoint.config, world, rank, "meta") + W.MODEL_WEIGHTS.load(model, weights) + + +def test_scale_invariants_hold_on_the_checkpoint(checkpoint, expert_scalars): + """Every FP8 block scale goes through the production transform (squeezed from + `[N/128, 1, K/128, 1]`, asserted UE8M0, packed), and every expert scalar passes the + production check: per layer, w1 == w3 on both scalars of every expert, and one input_scale + per projection over all 128 experts.""" + fp8 = sorted( + k + for k, (dtype, _) in checkpoint.headers.items() + if k.endswith(".weight_scale") and dtype == torch.float32 + ) + assert len(fp8) == 404 == 72 + 72 + 44 + 71 * 3 + 3 + by_file: dict = {} + for key in fp8: + by_file.setdefault(checkpoint.index[key], []).append(key) + for name, keys in sorted(by_file.items()): + with safe_open(str(checkpoint.path / name), framework="pt", device="cpu") as handle: + for key in keys: + s = handle.get_tensor(key) + n_blk, _, k_blk, _ = s.shape + packed = W._block_sf(s, None) + assert packed.shape == (W._sf_words(k_blk * 128), n_blk * 128), key + + assert len(expert_scalars) == 71 * 128 * 6 + for world in WORLDS: + W._check_expert_scalars( + expert_scalars, W.MODEL_WEIGHTS.dims(_core(checkpoint.config, world, 0)) + ) + for i in range(1, _LAYERS): + isc1 = expert_scalars[f"layers.{i}.experts.0.w1.input_scale"].item() + isc2 = expert_scalars[f"layers.{i}.experts.0.w2.input_scale"].item() + assert 0 < isc1 < 1 and 0 < isc2 < 1, (i, isc1, isc2) + + +class _Cut: + """A checkpoint entry that records each index taken of it -- what `load` reads of that key + -- and hands back a meta tensor of the header's shape so indexed.""" + + def __init__(self, key: str, dtype: torch.dtype, shape: tuple, log: list): + self.key, self.dtype, self.shape, self.log = key, dtype, shape, log + + def __getitem__(self, index): + self.log.append((self.key, index)) + return torch.empty(self.shape, dtype=self.dtype, device="meta")[index] + + +def _region(index, shape: tuple) -> tuple[tuple[int, int], ...]: + """The `[start, stop)` an index of basic slices takes on each dimension of `shape`.""" + if index is Ellipsis: + index = () + index = index if isinstance(index, tuple) else (index,) + assert all(isinstance(s, slice) for s in index), index + out = [] + for dim, size in enumerate(shape): + start, stop, step = (index[dim] if dim < len(index) else slice(None)).indices(size) + assert step == 1, index + out.append((start, stop)) + return tuple(out) + + +def _sliced(key: str, world: int) -> tuple[int, int] | None: + """`(dimension, per-rank width)` of a key the ranks split, None for one each rank reads + whole: h heads of 128 (`wq` rows, `wo` columns), of `wkv_b`'s per-head rows (192 on RoPE + layers, 256 on NoPE layers) and of SSMax; the dense and shared-expert rows (`w1`, `w3`) or + columns (`w2`), their block scales on the same 128-blocks (scale axis 0 for rows, 2 for + columns); the vocabulary rows of `output`. At tep4 h is 16, the widths 6144 and 1536 and the + vocabulary rows 51200; at tep8 8, 3072, 768 and 25600.""" + at = _AT[world] + h = at.heads + if key == "output.weight": + return 0, at.lm_rows + m = re.fullmatch(r"layers\.(\d+)\.(.+)", key) + if m is None: + return None + layer, rest = int(m.group(1)), m.group(2) + table = { + "attention.wq.weight": (0, h * 128), + "attention.wq.weight_scale": (0, h), + "attention.wo.weight": (1, h * 128), + "attention.wo.weight_scale": (2, h), + "attention.ssmax_scaling.s": (0, h), + "attention.ssmax_scaling.b": (0, h), + "attention.wkv_b.weight": (0, h * (256 if _CONFIG["nope_layer_flags"][layer] else 192)), + } + for module, width in (("feed_forward", at.dense), ("shared_experts", at.shared)): + for w in ("w1", "w3"): + table[f"{module}.{w}.weight"] = (0, width) + table[f"{module}.{w}.weight_scale"] = (0, width // 128) + table[f"{module}.w2.weight"] = (1, width) + table[f"{module}.w2.weight_scale"] = (2, width // 128) + return table.get(rest) + + +@pytest.mark.parametrize("world", WORLDS) +def test_the_ranks_tile_every_sliced_tensor(checkpoint, world): + """The index the production `load` takes of every checkpoint key, on each rank, at the + headers' shapes. Each key is read at most once per rank. Every language-model key is read + by every rank: a sliced one -- heads, rows, columns, vocabulary rows, block scales on the + same boundary -- along exactly one dimension, rank r at [r w, (r + 1) w), so the ranks tile + it exactly once; any other whole. A routed expert's codes and block scales are read whole + by the one rank whose window holds it, and its per-tensor scalars whole by every rank, for + the asserts. Every vision key is read whole by every rank, no DSpark draft key by any. The + bytes so read are the loader's own per-rank total, as the headers give: 146.42 GiB at tep4, + 76.78 GiB at tep8.""" + at = _AT[world] + regions: dict = {} + for rank in range(world): + log: list = [] + weights = { + key: _Cut(key, dtype, shape, log) for key, (dtype, shape) in checkpoint.headers.items() + } + W.MODEL_WEIGHTS.load(_Shell(checkpoint.config, world, rank, "meta"), weights) + keys = [key for key, _ in log] + assert len(keys) == len(set(keys)), "a key was read twice" + total = window = 0 + for key, index in log: + dtype, shape = checkpoint.headers[key] + region = _region(index, shape) + regions.setdefault(key, {})[rank] = region + n = torch.empty((), dtype=dtype).element_size() + for start, stop in region: + n *= stop - start + total += n + window += n if _is_expert(key) and not _is_expert_scalar(key) else 0 + assert round(total / 2**30, 2) == at.read_gib, (rank, total / 2**30) + assert round(window / 2**30, 2) == at.expert_gib, (rank, window / 2**30) + + sliced = 0 + for key, (_, shape) in checkpoint.headers.items(): + whole = tuple((0, size) for size in shape) + if _is_draft(key): + assert key not in regions, key + elif _is_expert_scalar(key): + assert regions[key] == {r: whole for r in range(world)}, key + elif _is_expert(key): + assert regions[key] == {_expert_of(key)[1] // at.experts: whole}, key + elif (cut := _sliced(key, world)) is None: + assert regions[key] == {r: whole for r in range(world)}, key + else: + dim, width = cut + assert width * world == shape[dim], (key, shape) + for r in range(world): + want = list(whole) + want[dim] = (r * width, (r + 1) * width) + assert regions[key][r] == tuple(want), (key, r, regions[key][r]) + sliced += 1 + # 72 x (wq, wo, their scales, SSMax s and b, wkv_b) + 6 dense + 71 x 6 shared + output. + assert sliced == 72 * 7 + 6 + 71 * 6 + 1 + + +@pytest.mark.parametrize( + "world, param, module, k, words", + [ + (4, "mlp_dn_s", "shared_experts.w2", 1536, 3), + (4, "mlp_dn_s", "feed_forward.w2", 6144, 12), + (4, "wo_s", "attention.wo", 2048, 4), + (8, "mlp_dn_s", "shared_experts.w2", 768, 2), + (8, "mlp_dn_s", "feed_forward.w2", 3072, 6), + (8, "wo_s", "attention.wo", 1024, 2), + ], +) +def test_packed_k_operand_of_each_rank_on_the_checkpoint( + checkpoint, world, param, module, k, words +): + """The DeepGEMM scale operand of each down projection at the K a rank's column slice + gives, built by the production path (`fill`'s steps: the row's shard on the lazy entry, + then its transform) from the checkpoint's own scales, on every layer and rank. It equals + an independent byte-level packing of the test's own slice of the stored grid and + `transform_sf_into_required_layout(recipe=(1, 128, 128))` on that slice, layout and strides + included. At tep8's K 768 that is 6 blocks in 2 words: bytes 0..3 of word 0 and 0..1 of + word 1 hold blocks 0..5, and bytes 2..3 of word 1 are zero; tep4's slices fill whole words.""" + k_blk = k // 128 + cores = [_core(checkpoint.config, world, rank) for rank in range(world)] + manifests = [W.MODEL_WEIGHTS.manifest(core) for core in cores] + checked = 0 + for i in range(_LAYERS): + key = f"layers.{i}.{module}.weight_scale" + if key not in checkpoint.index: + continue + with safe_open( + str(checkpoint.path / checkpoint.index[key]), framework="pt", device="cpu" + ) as handle: + stored = handle.get_tensor(key) + built = [] + for core, manifest in zip(cores, manifests): + [(ckpt, index, transform, shard)] = manifest[f"l{i}_{param}"] + assert (ckpt, index) == (key, None) + entry = W._materialize(handle.get_slice(key), shard) + built.append(transform(entry.contiguous(), core)) + for rank, got in enumerate(built): + n = stored.shape[0] * 128 + grid = stored[:, 0, rank * k_blk : (rank + 1) * k_blk, 0] + assert got.shape == (words, n) and got.dtype == torch.int32, (key, got.shape) + assert torch.equal(got, _packed(grid)), (key, rank) + operand = got.t() + engine = transform_sf_into_required_layout( + grid.contiguous(), mn=n, k=k, recipe=(1, 128, 128), is_sfa=False + ) + assert engine.shape == operand.shape and engine.stride() == operand.stride() + assert torch.equal(engine, operand), (key, rank) + check_sf_layout( + operand, + mn=n, + k=k, + gran=(1, 128), + num_groups=None, + tma_stride_check=True, + type_check=torch.int, + ) + assert torch.equal(inverse_transform_sf(operand, mn=n, k=k), grid), (key, rank) + # Byte slot 4w + j of a row is byte j (low first) of its word w: K block 4w + j. + slots = got.view(torch.uint8).view(words, n, 4).permute(1, 0, 2).reshape(n, 4 * words) + assert bool((slots[:, :k_blk] > 0).all()), (key, rank) + assert bool((slots[:, k_blk:] == 0).all()), (key, rank) + checked += 1 + layers = {"shared_experts.w2": 71, "feed_forward.w2": 1, "attention.wo": 72}[module] + assert checked == world * layers + + +# --------------------------------------------------------------------- values, scaled down + +_TINY_LAYERS = 8 +#: Every structural feature at a size a CPU fills in a second, at tep4: a dense layer, RoPE and +#: NoPE layers (3, 7), a KV sharer (7, reading 3), 'shared' indexer layers (2, 5, 6), two heads +#: and two experts per rank, one 128-block per rank on the shared expert. +_TINY_TEP4 = dict( + num_hidden_layers=_TINY_LAYERS, + hidden_size=256, + num_attention_heads=8, + qk_nope_head_dim=64, + qk_rope_head_dim=64, + v_head_dim=128, + kv_lora_rank=128, + nope_layer_flags=[i % 4 == 3 for i in range(_TINY_LAYERS)], + kv_source_layers=[0, 1, 2, 3, 4, 5, 6, 3], + index_sharing=["full", "full", "shared", "full", "full", "shared", "shared", "full"], + index_n_heads=4, + index_head_dim=128, + first_k_dense_replace=1, + n_routed_experts=8, + moe_intermediate_size=512, + n_shared_experts=1, + intermediate_size=1024, + vocab_size=512, +) +#: World size -> the scaled-down configuration. At tep8 the same per rank -- two heads, two +#: experts, one 128-block of the shared expert -- and the dense layer at 6 128-blocks per rank, +#: so its `w2` scale operand has tep8's half-padding second word. +_TINY = { + 4: _TINY_TEP4, + 8: dict( + _TINY_TEP4, + num_attention_heads=16, + n_routed_experts=16, + moe_intermediate_size=1024, + intermediate_size=6144, + ), +} + + +def _tiny_checkpoint(world: int, seed: int = 0) -> dict: + """A Mistral-named checkpoint at `_TINY[world]`: e4m3 weights with power-of-two fp32 block + scales stored `[N/128, 1, K/128, 1]` where the real one has them, NVFP4 routed experts -- + packed codes, e4m3 block scales, a weight_scale_2 per expert that w1 and w3 share, one + input_scale per projection and layer -- bf16 elsewhere, and two DSpark draft tensors.""" + c = SimpleNamespace(**_TINY[world]) + g = torch.Generator().manual_seed(seed) + bf16 = lambda *s: torch.randn(*s, generator=g).to(torch.bfloat16) # noqa: E731 + scalar = lambda: torch.rand((), generator=g) * 1e-3 + 1e-5 # noqa: E731 + + def fp8(name, n, k): + codes = (torch.rand(n, k, generator=g) * 800 - 400).to(torch.float8_e4m3fn) + exps = torch.randint(-15, -7, (n // 128, 1, k // 128, 1), generator=g).float() + return {f"{name}.weight": codes, f"{name}.weight_scale": torch.exp2(exps)} + + def nvfp4(name, n, k, ws2, isc): + codes = torch.randint(0, 256, (n, k // 2), dtype=torch.uint8, generator=g) + sf = (torch.rand(n, k // 16, generator=g) * 400 + 1).to(torch.float8_e4m3fn) + return { + f"{name}.weight": codes, + f"{name}.weight_scale": sf, + f"{name}.weight_scale_2": ws2.clone(), + f"{name}.input_scale": isc.clone(), + } + + h, qk = c.hidden_size, c.qk_nope_head_dim + c.qk_rope_head_dim + ck = {"tok_embeddings.weight": bf16(c.vocab_size, h), "norm.weight": bf16(h)} + ck["output.weight"] = bf16(c.vocab_size, h) + ck["lerope.alpha"] = bf16(c.qk_rope_head_dim // 2) + ck["mtp_dspark.backbone.final_norm.weight"] = bf16(32) + ck["mtp_dspark.backbone.layers.0.attention.wq.weight"] = bf16(32, 64) + for i in range(c.num_hidden_layers): + p = f"layers.{i}" + nope = c.nope_layer_flags[i] + ck[f"{p}.attention_norm.weight"] = bf16(h) + ck[f"{p}.ffn_norm.weight"] = bf16(h) + ck.update(fp8(f"{p}.attention.wq", c.num_attention_heads * qk, h)) + ck.update(fp8(f"{p}.attention.wo", h, c.num_attention_heads * c.v_head_dim)) + ck[f"{p}.attention.ssmax_scaling.s"] = bf16(c.num_attention_heads) + ck[f"{p}.attention.ssmax_scaling.b"] = bf16(c.num_attention_heads) + per_head = (qk if nope else c.qk_nope_head_dim) + c.v_head_dim + ck[f"{p}.attention.wkv_b.weight"] = bf16(c.num_attention_heads * per_head, c.kv_lora_rank) + if c.kv_source_layers[i] == i: + lat = c.kv_lora_rank + (0 if nope else c.qk_rope_head_dim) + ck[f"{p}.attention.wkv_a_with_mqa.weight"] = bf16(lat, h) + ck[f"{p}.attention.kv_a_norm.weight"] = bf16(c.kv_lora_rank) + if c.index_sharing[i] == "full": + x = f"{p}.attention.indexer" + ck.update(fp8(f"{x}.wq_b", c.index_n_heads * c.index_head_dim, h)) + ck[f"{x}.wk.weight"] = bf16(c.index_head_dim, h) + ck[f"{x}.weights_proj.weight"] = bf16(c.index_n_heads, h) + for norm, n in (("q_input_norm", h), ("k_norm", c.index_head_dim)): + ck[f"{x}.{norm}.weight"] = bf16(n) + ck[f"{x}.{norm}.bias"] = bf16(n) + ck[f"{x}.ssmax_scaling.s"] = bf16(c.index_n_heads) + ck[f"{x}.ssmax_scaling.b"] = bf16(c.index_n_heads) + if i < c.first_k_dense_replace: + for w, (n, k) in (("w1", (c.intermediate_size, h)), ("w3", (c.intermediate_size, h))): + ck.update(fp8(f"{p}.feed_forward.{w}", n, k)) + ck.update(fp8(f"{p}.feed_forward.w2", h, c.intermediate_size)) + continue + ck[f"{p}.gate.weight"] = bf16(c.n_routed_experts, h) + inter = c.moe_intermediate_size + m = f"{p}.shared_experts" + ck.update(fp8(f"{m}.w1", inter, h)) + ck.update(fp8(f"{m}.w3", inter, h)) + ck.update(fp8(f"{m}.w2", h, inter)) + isc1, isc2 = scalar(), scalar() + for e in range(c.n_routed_experts): + m = f"{p}.experts.{e}" + ws2_13 = scalar() + ck.update(nvfp4(f"{m}.w1", inter, h, ws2_13, isc1)) + ck.update(nvfp4(f"{m}.w3", inter, h, ws2_13, isc1)) + ck.update(nvfp4(f"{m}.w2", h, inter, scalar(), isc2)) + return ck + + +def _bytes(t: torch.Tensor) -> torch.Tensor: + """Compare by bit pattern: e4m3 has no `==`, and it is exactness that is asserted.""" + return t.contiguous().view(torch.uint8) if t.element_size() == 1 else t + + +def _grid(s: torch.Tensor) -> torch.Tensor: + """An FP8 block scale as stored, `[N/128, 1, K/128, 1]`, as its `[N/128, K/128]` grid.""" + return s.reshape(s.shape[0], s.shape[2]) + + +def _decode_sf(storage: torch.Tensor, n: int, k_blocks: int) -> torch.Tensor: + """Independent reading of the packed scale storage `[ceil(k_blocks/4), n]` int32: word + `[w, row]` holds the UE8M0 bytes of K blocks 4w..4w+3 of `row`, low byte first.""" + words = storage.t().contiguous() # [n, words] + out = torch.empty(n, k_blocks, dtype=torch.float32) + for kb in range(k_blocks): + byte = (words[:, kb // 4] >> (8 * (kb % 4))) & 0xFF + out[:, kb] = torch.exp2(byte.float() - 127) + return out + + +def _runner_rows(n: int, interleave: bool) -> torch.Tensor: + """The stored row each row of an expert's runner operand holds, one row at a time: row + `32b + 8v + u` takes row `32b + 4u + v` of the interleaved stack, whose row `2i` is up row + `i` (stored row `i`) and row `2i + 1` gate row `i` (stored row `n/2 + i`).""" + rows = [] + for r in range(n): + b, within = divmod(r, 32) + v, u = divmod(within, 8) + j = 32 * b + 4 * u + v + rows.append((j // 2 if j % 2 == 0 else n // 2 + j // 2) if interleave else j) + return torch.tensor(rows) + + +def _swizzle_offsets(m: int, c: int) -> torch.Tensor: + """`[m, c]`: where scale byte `(row, col)` lands in the 128x4-swizzled buffer.""" + r = torch.arange(m).view(-1, 1) + k = torch.arange(c).view(1, -1) + return (r // 128) * 128 * c + (k // 4) * 512 + (r % 32) * 16 + ((r % 128) // 32) * 4 + k % 4 + + +def _swizzled(s: torch.Tensor) -> torch.Tensor: + """`[m, c]` scale bytes scattered to their swizzled offsets, viewed `[m, c]` again.""" + m, c = s.shape + out = torch.empty(m * c, dtype=torch.uint8) + out[_swizzle_offsets(m, c).reshape(-1)] = s.reshape(-1) + return out.view(m, c) + + +def _unswizzled(s: torch.Tensor) -> torch.Tensor: + m, c = s.shape + return s.reshape(-1)[_swizzle_offsets(m, c)] + + +def _expected(ck: dict, world: int, rank: int) -> dict: + """What each parameter of `rank` must hold, built from the checkpoint by plain slicing, + with fp32 block-scale grids standing for the packed scale storage (`*_s` int32).""" + c = SimpleNamespace(**_TINY[world]) + heads = c.num_attention_heads // world + qk, v = c.qk_nope_head_dim + c.qk_rope_head_dim, c.v_head_dim + local = c.n_routed_experts // world + out = {} + + def rows(t, n): + return t[rank * n : (rank + 1) * n] + + def cols(t, n): + return t[:, rank * n : (rank + 1) * n] + + for i in range(c.num_hidden_layers): + p, q = f"layers.{i}", f"l{i}_" + nope = qk if c.nope_layer_flags[i] else c.qk_nope_head_dim + out[q + "attn_norm"] = ck[f"{p}.attention_norm.weight"] + out[q + "ffn_norm"] = ck[f"{p}.ffn_norm.weight"] + out[q + "wq"] = rows(ck[f"{p}.attention.wq.weight"], heads * qk) + out[q + "wq_s"] = rows(_grid(ck[f"{p}.attention.wq.weight_scale"]), heads * qk // 128) + out[q + "wo"] = cols(ck[f"{p}.attention.wo.weight"], heads * v) + out[q + "wo_s"] = cols(_grid(ck[f"{p}.attention.wo.weight_scale"]), heads * v // 128) + for s in ("s", "b"): + out[q + f"ssmax_{s}"] = rows(ck[f"{p}.attention.ssmax_scaling.{s}"], heads) + kvb = rows(ck[f"{p}.attention.wkv_b.weight"], heads * (nope + v)) + per_head = [kvb[h * (nope + v) : (h + 1) * (nope + v)] for h in range(heads)] + out[q + "wkv_b"] = torch.cat([t[:nope] for t in per_head] + [t[nope:] for t in per_head]) + if c.kv_source_layers[i] == i: + out[q + "wkv_a"] = ck[f"{p}.attention.wkv_a_with_mqa.weight"] + out[q + "kv_a_norm"] = ck[f"{p}.attention.kv_a_norm.weight"] + if c.index_sharing[i] == "full": + x = f"{p}.attention.indexer" + out[q + "ix_q_norm_w"] = ck[f"{x}.q_input_norm.weight"] + out[q + "ix_q_norm_b"] = ck[f"{x}.q_input_norm.bias"] + out[q + "ix_wq_b"] = ck[f"{x}.wq_b.weight"] + out[q + "ix_wq_b_s"] = _grid(ck[f"{x}.wq_b.weight_scale"]) + out[q + "ix_wk_wp"] = torch.cat([ck[f"{x}.wk.weight"], ck[f"{x}.weights_proj.weight"]]) + out[q + "ix_k_norm_w"] = ck[f"{x}.k_norm.weight"].float() + out[q + "ix_k_norm_b"] = ck[f"{x}.k_norm.bias"].float() + out[q + "ix_ssmax_s"] = ck[f"{x}.ssmax_scaling.s"] + out[q + "ix_ssmax_b"] = ck[f"{x}.ssmax_scaling.b"] + if i < c.first_k_dense_replace: + m, inter = f"{p}.feed_forward", c.intermediate_size // world + else: + m, inter = f"{p}.shared_experts", c.moe_intermediate_size // world + # Gate rows first. + out[q + "mlp_gu"] = torch.cat( + [_bytes(rows(ck[f"{m}.{w}.weight"], inter)) for w in ("w1", "w3")] + ) + out[q + "mlp_gu_s"] = torch.cat( + [rows(_grid(ck[f"{m}.{w}.weight_scale"]), inter // 128) for w in ("w1", "w3")] + ) + out[q + "mlp_dn"] = cols(ck[f"{m}.w2.weight"], inter) + out[q + "mlp_dn_s"] = cols(_grid(ck[f"{m}.w2.weight_scale"]), inter // 128) + if i < c.first_k_dense_replace: + continue + out[q + "router"] = ck[f"{p}.gate.weight"] + window = [f"{p}.experts.{rank * local + s}" for s in range(local)] + # Up rows first, interleaved and block-shuffled; the block scales also swizzled. + fc1 = _runner_rows(2 * c.moe_intermediate_size, True) + fc2 = _runner_rows(c.hidden_size, False) + + def up_gate(e, suffix): + return torch.cat([_bytes(ck[f"{e}.{w}.{suffix}"]) for w in ("w3", "w1")])[fc1] + + out[q + "fc1_w"] = torch.stack([up_gate(e, "weight") for e in window]) + out[q + "fc1_s"] = torch.stack([_swizzled(up_gate(e, "weight_scale")) for e in window]) + out[q + "fc2_w"] = torch.stack([ck[f"{e}.w2.weight"][fc2] for e in window]) + out[q + "fc2_s"] = torch.stack( + [_swizzled(_bytes(ck[f"{e}.w2.weight_scale"])[fc2]) for e in window] + ) + out[q + "isc1"] = ck[f"{window[0]}.w1.input_scale"].reshape(1) + out[q + "isc2"] = ck[f"{window[0]}.w2.input_scale"].reshape(1) + out[q + "ws2_13"] = torch.stack([ck[f"{e}.w1.weight_scale_2"] for e in window]) + out[q + "ws2_2"] = torch.stack([ck[f"{e}.w2.weight_scale_2"] for e in window]) + out["final_norm"] = ck["norm.weight"] + out["embed"] = ck["tok_embeddings.weight"] + out["lerope_alpha"] = ck["lerope.alpha"].float() + return out + + +@pytest.mark.parametrize("world", WORLDS) +def test_tiny_scale_operands_cover_whole_and_partial_words(world): + """The scaled-down table's scale operands at a rank, by K blocks: 2 for every K of the + hidden size 256 and for `wo` (2 heads x 128), 1 for the shared `w2` (128 of 512 at tep4, of + 1024 at tep8), and for the dense `w2` 2 at tep4 (256 of 1024) and 6 at tep8 (768 of 6144) + -- the shared expert's case at tep8's full size, 6 blocks in 2 words. Every one has a + partly padded last word, which the values test below compares whole.""" + d = W.MODEL_WEIGHTS.dims(_core(_TINY[world], world, 0)) + dense, dense_words = {4: (256, 1), 8: (768, 2)}[world] + assert (d.heads, d.dense_inter, d.shared_inter, d.local_experts) == (2, dense, 128, 2) + with torch.device("meta"): + declared = W.MODEL_WEIGHTS.declare(d) + assert tuple(declared["l0_mlp_dn_s"].shape) == (dense_words, 256) + assert tuple(declared["l1_mlp_dn_s"].shape) == (1, 256) + assert tuple(declared["l1_wo_s"].shape) == (1, 256) + + +@pytest.mark.parametrize("world, rank", [(w, r) for w in WORLDS for r in range(w)]) +def test_tiny_load_values(world, rank): + """Every parameter of every rank, bit for bit against `_expected`; each scale operand's + whole int32 storage, padding bytes included, against `_packed`.""" + ck = _tiny_checkpoint(world) + model = _Shell(_TINY[world], world, rank, "cpu") + W.MODEL_WEIGHTS.load(model, dict(ck)) + got = dict(model.model.w) + want = _expected(ck, world, rank) + assert set(got) == set(want) + for key, ref in want.items(): + param = got[key].data + if key.endswith("_s") and param.dtype == torch.int32: + # DeepGEMM's operand is the transpose of the storage: MN-major, row stride N. + n = param.shape[1] + k = ref.shape[1] * 128 + operand = param.t() + check_sf_layout( + operand, + mn=n, + k=k, + gran=(1, 128), + num_groups=None, + tma_stride_check=True, + type_check=torch.int, + ) + assert torch.equal(param, _packed(ref)), key + per_row = ref.repeat_interleave(128, dim=0) + assert torch.equal(_decode_sf(param, n, ref.shape[1]), per_row), key + assert torch.equal(inverse_transform_sf(operand, mn=n, k=k), ref), key + if key.endswith(("wq_s", "wo_s", "ix_wq_b_s", "mlp_dn_s")): + engine = transform_sf_into_required_layout( + ref.contiguous(), mn=n, k=k, recipe=(1, 128, 128), is_sfa=False + ) + assert engine.shape == operand.shape and engine.stride() == operand.stride() + assert torch.equal(engine, operand), key + continue + if ref.dtype == torch.uint8 and param.dtype == torch.float8_e4m3fn: + assert key.endswith(("mlp_gu", "fc1_s", "fc2_s")), key + else: + assert param.dtype == ref.dtype, (key, param.dtype, ref.dtype) + assert torch.equal(_bytes(param), _bytes(ref)), key + lm_rows = _TINY[world]["vocab_size"] // world + assert torch.equal( + model.lm_head.weight.data, ck["output.weight"][rank * lm_rows : (rank + 1) * lm_rows] + ) + + +@pytest.mark.parametrize("world", WORLDS) +def test_tiny_expert_halves_are_up_then_gate(world): + """Undo the block shuffle (and the scales' swizzle) of FC1: its even rows are w3 (up), its + odd rows w1 (gate), codes and block scales alike -- the order the NVFP4 runner's FC1 reads, + up first, the opposite of `mlp_gu`'s gate first.""" + ck = _tiny_checkpoint(world) + model = _Shell(_TINY[world], world, 1, "cpu") + W.MODEL_WEIGHTS.load(model, dict(ck)) + inter, e = _TINY[world]["moe_intermediate_size"], 2 # rank 1's local slot 0 + w = model.model.w + shuffled = _runner_rows(2 * inter, False) + unshuffle = torch.empty_like(shuffled) + unshuffle[shuffled] = torch.arange(2 * inter) + codes = w["l4_fc1_w"][0][unshuffle] + assert torch.equal(codes[0::2], ck[f"layers.4.experts.{e}.w3.weight"]) + assert torch.equal(codes[1::2], ck[f"layers.4.experts.{e}.w1.weight"]) + scales = _unswizzled(_bytes(w["l4_fc1_s"][0]))[unshuffle] + assert torch.equal(scales[0::2], _bytes(ck[f"layers.4.experts.{e}.w3.weight_scale"])) + assert torch.equal(scales[1::2], _bytes(ck[f"layers.4.experts.{e}.w1.weight_scale"])) + + +@pytest.mark.parametrize("world", WORLDS) +def test_tiny_layout_is_applied_once_per_load(world): + """A second `load` into the same parameters starts again from the stored bytes, so it lands + on the same layout rather than shuffling the first one's result again.""" + ck = _tiny_checkpoint(world) + model = _Shell(_TINY[world], world, world - 1, "cpu") + W.MODEL_WEIGHTS.load(model, dict(ck)) + first = {k: _bytes(v.data).clone() for k, v in model.model.w.items() if k.startswith("l5_fc")} + W.MODEL_WEIGHTS.load(model, dict(ck)) + for key, value in first.items(): + assert torch.equal(_bytes(model.model.w[key].data), value), key + + +def _write_shards(ck: dict, directory: Path, files: int = 3) -> None: + """`ck` as Mistral-named shard files with their index, keys dealt round-robin, so one + expert's tensors -- and w1's and w3's scalars -- land in different files.""" + from safetensors.torch import save_file + + names = [f"consolidated-{i + 1:05d}-of-{files:05d}.safetensors" for i in range(files)] + weight_map = {key: names[i % files] for i, key in enumerate(sorted(ck))} + for name in names: + part = {k: v.contiguous() for k, v in ck.items() if weight_map[k] == name} + save_file(part, str(directory / name)) + (directory / W.SHARD_INDEX).write_text(json.dumps({"weight_map": weight_map})) + + +class _ReadLog: + """`safe_open` that records every key read through it.""" + + def __init__(self, real): + self.real, self.keys = real, [] + + def __call__(self, *args, **kwargs): + log = self + + class Handle: + def __enter__(self): + self.h = log.real(*args, **kwargs).__enter__() + return self + + def __exit__(self, *exc): + return self.h.__exit__(*exc) + + def get_slice(self, key): + log.keys.append(key) + return self.h.get_slice(key) + + return Handle() + + +@pytest.mark.parametrize("world, rank", [(w, r) for w in WORLDS for r in range(w)]) +def test_tiny_load_streams_from_the_shard_files(world, rank, tmp_path, monkeypatch): + """With the checkpoint's shard files found through the directory the engine records on + the config, every source is read from them -- the given dict is never read: here it + holds meta tensors, which cannot be copied out -- and the result is bit for bit the + dict path's. Each key is read once; of the other ranks' experts only the scalars are read, + and nothing of the DSpark draft. Each file's cached pages are dropped once read.""" + ck = _tiny_checkpoint(world) + _write_shards(ck, tmp_path) + log = _ReadLog(W.safe_open) + dropped = [] + drop = W._drop_page_cache + monkeypatch.setattr(W, "safe_open", log) + monkeypatch.setattr(W, "_drop_page_cache", lambda path: (dropped.append(path), drop(path))) + streamed = _Shell(_TINY[world], world, rank, "cpu") + streamed.model.model_config.pretrained_config._name_or_path = str(tmp_path) + W.MODEL_WEIGHTS.load(streamed, {k: torch.empty_like(v, device="meta") for k, v in ck.items()}) + monkeypatch.undo() + direct = _Shell(_TINY[world], world, rank, "cpu") + W.MODEL_WEIGHTS.load(direct, dict(ck)) + got, want = dict(streamed.named_parameters()), dict(direct.named_parameters()) + assert set(got) == set(want) + for key in want: + assert torch.equal(_bytes(got[key].data), _bytes(want[key].data)), key + + read = set(log.keys) + assert len(read) == len(log.keys), "a key was read twice" + lo, hi = 2 * rank, 2 * rank + 2 + off = {k for k in ck if _is_expert(k) and not lo <= _expert_of(k)[1] < hi} + own = {k for k in ck if not _is_draft(k)} - off + assert read == own | {k for k in off if _is_expert_scalar(k)} + assert sorted(p.name for p in dropped) == sorted(p.name for p in tmp_path.glob("*.safetensors")) + + +@pytest.mark.parametrize("world", WORLDS) +@pytest.mark.parametrize("with_index", [True, False]) +def test_tiny_load_checkpoint_reads_the_directory(world, with_index, tmp_path): + """Given the checkpoint directory alone, as the engine gives it, the result is bit for bit + the dict path's: the files come from the directory's index, or without one from the shard + files' headers.""" + ck = _tiny_checkpoint(world) + _write_shards(ck, tmp_path) + if not with_index: + (tmp_path / W.SHARD_INDEX).unlink() + loaded = _Shell(_TINY[world], world, 2, "cpu") + W.MODEL_WEIGHTS.load_checkpoint(loaded, tmp_path) + direct = _Shell(_TINY[world], world, 2, "cpu") + W.MODEL_WEIGHTS.load(direct, dict(ck)) + got, want = dict(loaded.named_parameters()), dict(direct.named_parameters()) + assert set(got) == set(want) + for key in want: + assert torch.equal(_bytes(got[key].data), _bytes(want[key].data)), key + + +@pytest.mark.parametrize("world", WORLDS) +def test_tiny_load_falls_back_to_the_dict_without_an_index(world, tmp_path): + """A recorded directory without the shard index is not the checkpoint: the dict is read.""" + ck = _tiny_checkpoint(world) + model = _Shell(_TINY[world], world, 2, "cpu") + model.model.model_config.pretrained_config._name_or_path = str(tmp_path) + W.MODEL_WEIGHTS.load(model, dict(ck)) + + +@pytest.mark.parametrize("world", WORLDS) +def test_tiny_load_without_draft_tensors(world): + """The DSpark draft is never asserted on: a checkpoint without it loads too.""" + ck = {k: v for k, v in _tiny_checkpoint(world).items() if not _is_draft(k)} + W.MODEL_WEIGHTS.load(_Shell(_TINY[world], world, 1, "cpu"), ck) + + +@pytest.mark.parametrize("world", WORLDS) +def test_tiny_load_without_off_window_codes(world): + """The other ranks' expert codes and block scales are never read nor asserted present.""" + ck = _tiny_checkpoint(world) + for e in range(2, _TINY[world]["n_routed_experts"]): + for proj in ("w1", "w2", "w3"): + del ck[f"layers.3.experts.{e}.{proj}.weight"] + del ck[f"layers.3.experts.{e}.{proj}.weight_scale"] + W.MODEL_WEIGHTS.load(_Shell(_TINY[world], world, 0, "cpu"), ck) + + +def _load_raises(ck: dict, world: int, match: str, rank: int = 0) -> None: + with pytest.raises(AssertionError, match=match): + W.MODEL_WEIGHTS.load(_Shell(_TINY[world], world, rank, "cpu"), ck) + + +@pytest.mark.parametrize("world", WORLDS) +@pytest.mark.parametrize( + "key", + [ + "layers.3.experts.1.w3.weight", + "layers.3.experts.1.w1.weight_scale", + "layers.3.experts.0.w3.weight_scale_2", + "layers.3.experts.1.w2.input_scale", + ], +) +def test_tiny_load_refuses_a_missing_in_window_tensor(key, world): + ck = _tiny_checkpoint(world) + del ck[key] + _load_raises(ck, world, f"checkpoint key missing: {key}") + + +@pytest.mark.parametrize("world, expert", [(4, 5), (8, 13)]) +def test_tiny_load_refuses_a_missing_off_window_scalar(world, expert): + """The other ranks' scalars are read for the per-layer asserts, so they must be there.""" + ck = _tiny_checkpoint(world) + del ck[f"layers.3.experts.{expert}.w2.input_scale"] + _load_raises(ck, world, f"checkpoint key missing: layers.3.experts.{expert}.w2.input_scale") + + +@pytest.mark.parametrize("world", WORLDS) +def test_tiny_load_refuses_an_unknown_tensor(world): + ck = _tiny_checkpoint(world) + ck["layers.2.attention.wq_extra.weight"] = torch.zeros(4) + _load_raises(ck, world, "checkpoint coverage") + + +@pytest.mark.parametrize("world", WORLDS) +@pytest.mark.parametrize( + "key", + [ + "layers.0.attention.wq.weight_scale", + "layers.0.feed_forward.w3.weight_scale", + "layers.4.shared_experts.w2.weight_scale", + "layers.4.attention.indexer.wq_b.weight_scale", + "layers.5.attention.wo.weight_scale", + ], +) +def test_tiny_load_refuses_a_scale_that_is_not_ue8m0(key, world): + ck = _tiny_checkpoint(world) + ck[key] = ck[key].clone() + ck[key].view(-1)[0] *= 1.5 + _load_raises(ck, world, "not a positive normal power of two") + + +@pytest.mark.parametrize("world", WORLDS) +def test_tiny_load_refuses_a_two_dimensional_fp8_scale(world): + """A `[N/128, K/128]` block-scale layout is not this one's.""" + ck = _tiny_checkpoint(world) + key = "layers.4.attention.indexer.wq_b.weight_scale" + ck[key] = _grid(ck[key]).contiguous() + _load_raises(ck, world, r"not \[N/128, 1, K/128, 1\]") + + +@pytest.mark.parametrize("world, expert", [(4, 0), (4, 6), (8, 0), (8, 13)]) +@pytest.mark.parametrize("scalar", ["weight_scale_2", "input_scale"]) +def test_tiny_load_refuses_w1_and_w3_scalars_that_differ(world, expert, scalar): + """On this rank's experts and on the others' alike.""" + ck = _tiny_checkpoint(world) + key = f"layers.5.experts.{expert}.w3.{scalar}" + ck[key] = ck[key] * 2 + experts = _TINY[world]["n_routed_experts"] + want = f"layers.5: w1 and w3 {scalar} differ on 1 of {experts} experts, first expert {expert}:" + _load_raises(ck, world, want) + + +@pytest.mark.parametrize("world, expert", [(4, 7), (8, 15)]) +@pytest.mark.parametrize("proj", ["w1", "w2"]) +def test_tiny_load_refuses_an_input_scale_that_varies_over_the_experts(world, expert, proj): + """Rank 0 sees the other ranks' experts' input_scale: one off-window expert disagreeing is + a quantization rank 0's window would not share.""" + ck = _tiny_checkpoint(world) + for p in {proj, "w3"} if proj == "w1" else {proj}: + key = f"layers.6.experts.{expert}.{p}.input_scale" + ck[key] = ck[key] * 2 + want = f"layers.6: {proj}.input_scale is not one value over the experts: expert {expert}" + _load_raises(ck, world, want) + + +@pytest.mark.parametrize("world", WORLDS) +@pytest.mark.parametrize("value", [0.0, -1e-4, float("inf"), float("nan")]) +def test_tiny_load_refuses_a_scalar_that_is_not_a_positive_finite_scale(value, world): + ck = _tiny_checkpoint(world) + ck["layers.2.experts.3.w2.weight_scale_2"] = torch.tensor(value) + _load_raises( + ck, world, "layers.2: w2.weight_scale_2 of expert 3 is .*not a positive finite scale" + ) + + +def test_shared_scalar_refuses_copies_that_differ(): + """The row transform asserts on its own copies too, whatever ran before it.""" + one = torch.tensor(0.5) + assert W._shared_scalar((one, one.clone(), one.clone()), None) is one + with pytest.raises(AssertionError, match="1 of 3 copies"): + W._shared_scalar((one, one.clone(), torch.tensor(0.25)), None) + + +@pytest.mark.parametrize( + "value, exact", + [ + (1.0, True), + (2.0**-126, True), + (2.0**127, True), + (2.0**-15, True), + (0.0, False), + (-(2.0**-12), False), + (3.0 * 2.0**-12, False), + (2.0**-130, False), # subnormal + (float("inf"), False), + (float("nan"), False), + ], +) +def test_ue8m0_assert(value, exact): + s = torch.full((2, 3), 2.0**-10) + s[1, 2] = value + if exact: + W._assert_ue8m0(s) + else: + with pytest.raises(AssertionError, match="1 of 6"): + W._assert_ue8m0(s) + + +def test_packed_reference_itself(): + """`_packed` on a hand-written grid: 6 K blocks with exponent bytes 120..125 in one + 128-row block -> word 0 = 120 | 121 << 8 | 122 << 16 | 123 << 24, word 1 = 124 | 125 << 8, + the same for all 128 rows; a byte of 128 or more lands in the sign bit.""" + grid = torch.exp2(torch.arange(120, 126, dtype=torch.float32) - 127).view(1, 6) + got = _packed(grid) + assert got.shape == (2, 128) + assert bool((got[0] == 120 | 121 << 8 | 122 << 16 | 123 << 24).all()) + assert bool((got[1] == 124 | 125 << 8).all()) + high = _packed(torch.tensor([[1.0, 1.0, 1.0, 2.0]])) + assert bool((high[0] == (127 | 127 << 8 | 127 << 16 | 128 << 24) - 2**32).all()) + + +@pytest.mark.parametrize("world", [4, 8]) +def test_native_dspark_manifest_shards_and_off_coverage(world): + native = { + "n_layers": 3, + "depth": 3, + "target_activation_layers": "1,18,36,54,71", + "dspark": {"markov_rank": 512}, + } + # Tensor dimensions from the checkpoint's safetensors header, independently + # of the table's shape lambdas. Values encode the sliced axis so a row / + # column or head-boundary error is visible without allocating the weights. + projections = { + "wq": (8192, 6144, 0), + "wkv_b": (12288, 512, 0), + "wo": (6144, 8192, 1), + "w1": (24576, 6144, 0), + "w3": (24576, 6144, 0), + "w2": (6144, 24576, 1), + } + shards = {key: [] for key in projections} + for rank in range(world): + core = _core(_CONFIG, world=world, rank=rank) + off = W.MODEL_WEIGHTS.manifest(core) + core.model_config.spec_config = SimpleNamespace( + _draft_model_config=native, target_layer_ids=[1, 18, 36, 54, 71] + ) + manifest = W.MODEL_WEIGHTS.manifest(core) + added = set(manifest) - set(off) + assert len(added) == 38 + keys = {manifest[name][0][0] for name in added} + assert len(keys) == 38 and all(key.startswith("mtp_dspark.") for key in keys) + assert not keys.intersection(W.MODEL_WEIGHTS.expected_unconsumed(core, dict.fromkeys(keys))) + with torch.device("meta"): + declared = W.MODEL_WEIGHTS.declare(W.MODEL_WEIGHTS.dims(core)) + assert tuple(declared["draft_target_projection"].shape) == (6144, 30720) + assert tuple(declared["draft_joint"].shape) == (1536, 7168) + assert tuple(declared["draft_joint_bias"].shape) == (1536,) + assert tuple(declared["draft_mask"].shape) == (2, 6144) + assert all(declared[name].dtype == torch.bfloat16 for name in added) + for role, (rows, cols, axis) in projections.items(): + ids = torch.arange((rows, cols)[axis]) + source = ( + ids[:, None].expand(rows, cols) if axis == 0 else ids[None, :].expand(rows, cols) + ) + key, index, transform, shard = manifest[f"l0_draft_{role}"][0] + assert index is None and transform is None + sliced = source if shard is None else shard(source) + assert tuple(sliced.shape) == tuple(declared[f"l0_draft_{role}"].shape) + shards[role].append(sliced[:, 0] if axis == 0 else sliced[0]) + core.model_config.spec_config = None + assert keys.issubset(W.MODEL_WEIGHTS.expected_unconsumed(core, dict.fromkeys(keys))) + for role, parts in shards.items(): + rows, cols, axis = projections[role] + expected = torch.arange((rows, cols)[axis]) + torch.testing.assert_close(torch.cat(parts), expected) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/__init__.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/__init__.py new file mode 100644 index 000000000000..46c7fb1a43fa --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/__init__.py @@ -0,0 +1,7 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Package marker. + +These test basenames can repeat other targets'. Without packages, pytest's +default import mode loads both under one module name and refuses the second. +""" diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_ctx_workspace.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_ctx_workspace.py new file mode 100644 index 000000000000..777822a2a249 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_ctx_workspace.py @@ -0,0 +1,104 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The context workspace of the Mistral-Large-4-1T-A52B NVFP4 tp target: the KV rows its +dequantizing gather and prefill kernel address. Under KV block reuse a context request's KV +is its cached prefix plus its new tokens, so the workspace covers both, while the engine's +`num_ctx_kv_tokens` counts the new tokens only. The workspace is the first rows of the bf16 +workspace the target holds, `[core.ctx_workspace_rows, 576]`, allocated on the first context +step; its rows are sized from the engine's quantities (`_ctx_workspace_rows`), checked at two +of them (ids `32k` and `1m`): the 32768 serving config (max_num_tokens = max_seq_len = 32768, +neither chunked prefill nor block reuse), 65536 rows, and the 1M config (max_seq_len 1048576, +chunked prefill), 2**19. The steps here fit either twice, so the next rows are kept beside them +for the shared owner. No device: the step is built from host-side metadata.""" + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_103__tp import ( # noqa: E501 + modeling as M, +) + +pytestmark = pytest.mark.cpu_only + +_LAT_DIM = 576 +#: The engine quantities the held workspace is sized from: (max_num_tokens, max_seq_len, +#: chunked prefill, block reuse) at max_batch_size 16, and the rows they give. +_CONFIGS = [ + pytest.param((32768, 32768, False, False), 65536, id="32k"), + pytest.param((32768, 1048576, True, False), 1 << 19, id="1m"), +] + + +def _cumsum(lens): + return torch.tensor([0] + lens, dtype=torch.int64).cumsum(0) + + +def _metadata(cached, new): + """The fields `_ctx_kv_offsets` reads, as the engine fills them for context requests with + `cached` reused and `new` scheduled tokens each.""" + if any(cached): + return SimpleNamespace( + enable_context_mla_with_cached_kv=True, + ctx_cached_token_indptr=_cumsum(cached), + ctx_kv_indptr=_cumsum([c + n for c, n in zip(cached, new)]), + max_ctx_seq_len=max(new), + num_ctx_cached_tokens=sum(cached), + ) + return SimpleNamespace( + enable_context_mla_with_cached_kv=False, + mla_prepare_ctx_cu_seqlens=lambda: _cumsum(new).to(torch.int32), + seq_lens=torch.tensor(new, dtype=torch.int32), + ) + + +def _held_rows(max_num_tokens, max_seq_len, chunked, reuse) -> int: + """`_ctx_workspace_rows` for an engine at these quantities, off the attention metadata + fields it reads (block reuse is the runtime features' cache_reuse).""" + return M._ctx_workspace_rows( + SimpleNamespace( + max_num_tokens=max_num_tokens, + max_num_requests=16, + max_seq_len=max_seq_len, + runtime_features=SimpleNamespace(chunked_prefill=chunked, cache_reuse=reuse), + ) + ) + + +@pytest.mark.parametrize("engine, rows", _CONFIGS) +@pytest.mark.parametrize( + "cached, new", + [ + ([0, 0], [7, 33]), + ([64, 0, 96], [5, 40, 1]), + ([32], [1]), + ], + ids=["no_reuse", "mixed_prefixes", "one_new_token"], +) +def test_workspace_covers_the_cached_prefix_and_the_new_tokens(cached, new, engine, rows): + step = dict( + num_contexts=len(new), + num_ctx_tokens=sum(new), + num_ctx_kv_tokens=sum(new), + **M._ctx_kv_offsets(_metadata(cached, new), len(new)), + ) + target = M.PrefillTarget.__new__(M.PrefillTarget) + target.core = SimpleNamespace(lat_dim=_LAT_DIM, ctx_workspace_rows=_held_rows(*engine)) + assert target.core.ctx_workspace_rows == rows + ws = target._ctx_workspace(step, sum(new), torch.device("cpu")) + + kv_lens = [c + n for c, n in zip(cached, new)] + n_kv = sum(kv_lens) + assert ws.ws_buf.shape == (n_kv, _LAT_DIM) + assert ws.kv_req.tolist() == [r for r, n in enumerate(kv_lens) for _ in range(n)] + assert ws.kv_pos.tolist() == [p for n in kv_lens for p in range(n)] + assert ws.ws_off.tolist() == _cumsum(kv_lens)[:-1].tolist() + # The workspaces the step keeps by owner layer: none before any layer attends. + assert ws.ws == {} + # The first rows of the held workspace, and the next as many kept beside them. + assert target._ws_buf.shape == (rows, _LAT_DIM) + assert target._ws_buf.dtype == torch.bfloat16 + assert ws.ws_buf.data_ptr() == target._ws_buf.data_ptr() + assert ws.kept.shape == (n_kv, _LAT_DIM) + assert ws.kept.data_ptr() == target._ws_buf[n_kv].data_ptr() diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_head_padding.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_head_padding.py new file mode 100644 index 000000000000..a740d4252b16 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_head_padding.py @@ -0,0 +1,148 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The absorbed query's head layout on the target's own scratch, per kernel that reads it. + +A rank holds 16 of the model's 64 query heads at tp4 (four ranks) and 8 at tp8 (eight ranks +over two nodes of four). FlashMLA's sm100 sparse prefill takes 64 or 128 query heads, so the +prefill target keeps the absorbed query in a `[rows, 64, 576]` bf16 scratch, zeroed once per +step, and writes only its rank's heads 0..h-1 (the absorption BMM into the 512-wide head, the +rope tail written or zeroed by the SSMax op, the owner's rope op in place). The generation +rows' sparse decode, `cute_dsl_ds_mla_sparse_decode`, runs the rank's h heads unpadded: the +decode target's scratch is `[rows, h, 576]`, every element written by every layer (the SSMax +op's tail write included), and the prefill target's generation rows hand it the 64-head +scratch, of which it reads heads 0..h-1. Both kernels write only those h heads of their output +into the compact `[rows, h, 512]` latent output the V-B BMM reads. That the kernels leave the +live heads unchanged under zero dead heads, and read h of a wider query, is their own +certification (`flash_mla_sparse_prefill`'s and `cute_dsl_ds_mla_sparse_decode`'s cells); this +checks the layout around them, through `PrefillTarget._scratch` and `DecodeTarget._scratch`, +at both world sizes: + +* the prefill scratch's query is 64 heads of 576, bf16, every byte zero (+0.0, not -0.0); the + decode scratch's is the rank's h heads of 576, bf16, contiguous; +* the rank's heads are a view of heads 0..h-1 with contiguous heads, the layout the rope op and + the kernels address, so writing them never reaches heads h..63; +* the latent output is h heads wide, a NoPE owner's latent rows are 576 wide and zero (the + fused glue's; the unfused glue, the core's `unfused_glue` (`_UNFUSED_GLUE`), builds them per + call), and no FlashMLA scheduler rides on the step. + +`_GEN_ATTN_FLASH_MLA` (the core's `gen_attn_flash_mla`) puts the generation rows back on +FlashMLA's 64-head decode for A/B verification; its layout (the decode scratch at 64 heads, +zeroed, one scheduler shared by the step's layers, and a latent output only for the unfused +glue: the fused glue reads FlashMLA's live heads in place) is checked too, under both glues, and +that the shipped default is the native kernel. Neither switch depends on max_seq_len. +""" + +from __future__ import annotations + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_103__tp import ( # noqa: E501 + modeling as M, +) + +pytestmark = pytest.mark.cpu_only + +DEV = "cpu" +KV_LORA, ROPE, V_DIM = 512, 64, 128 +LAT = KV_LORA + ROPE +#: World size -> the query heads a rank holds: the model's 64 over the ranks. +HEADS = {4: 16, 8: 8} +GLUES = pytest.mark.parametrize("unfused_glue", [False, True], ids=["fused", "unfused"]) + + +def _core(heads: int, flash_mla: bool, unfused_glue: bool) -> SimpleNamespace: + """The widths the targets' scratch reads off the core, with `heads` of the 64 on the rank, + its generation rows' kernel (`flash_mla`: `_GEN_ATTN_FLASH_MLA`) and its glue + (`unfused_glue`: `_UNFUSED_GLUE`).""" + return SimpleNamespace( + heads=heads, + v_dim=V_DIM, + lat_dim=LAT, + kv_lora=KV_LORA, + rope_dim=ROPE, + gen_attn_flash_mla=flash_mla, + unfused_glue=unfused_glue, + ) + + +def _target(cls, heads: int, flash_mla: bool, unfused_glue: bool): + target = object.__new__(cls) + target.core = _core(heads, flash_mla, unfused_glue) + return target + + +def _check_scratch(s, rows: int, heads: int, width: int, lat_out: bool, unfused_glue: bool) -> None: + assert s.fused_q.dtype == torch.bfloat16 + assert tuple(s.fused_q.shape) == (rows, width, LAT) + if width > heads: + assert not s.fused_q.view(torch.int16).any(), "a head is not +0.0" + if lat_out: + assert tuple(s.lat_out.shape) == (rows, heads, KV_LORA) + else: + assert s.lat_out is None, "decode reads FlashMLA's output in place" + assert tuple(s.attn_out.shape) == (rows, heads * V_DIM) + if unfused_glue: + assert s.nope_lat is None, "the unfused glue builds a NoPE owner's latent per call" + else: + # A NoPE owner's latent rows: RMSNorm(c_kv) lands in the first 512 columns, the rope + # key stays +0.0 for the step. + assert s.nope_lat.dtype == torch.bfloat16 and tuple(s.nope_lat.shape) == (rows, LAT) + assert not s.nope_lat.view(torch.int16).any(), "a NoPE owner's rope key is not +0.0" + + # The rank's heads as the attention block takes them: heads 0..h-1, contiguous heads. + fq = s.fused_q[:, :heads] + assert fq.stride(-1) == 1 and fq.stride(1) == LAT + assert fq.data_ptr() == s.fused_q.data_ptr() + g = torch.Generator().manual_seed(rows) + fq[..., :KV_LORA].copy_(torch.randn(rows, heads, KV_LORA, generator=g)) + fq[..., KV_LORA:].copy_(torch.randn(rows, heads, ROPE, generator=g)) + if width > heads: + assert not s.fused_q[:, heads:].view(torch.int16).any(), "a dead head was written" + else: + assert s.fused_q.is_contiguous() + + +def test_the_native_decode_is_the_shipped_default(): + assert M._GEN_ATTN_FLASH_MLA is False + assert M._FLASH_MLA_HEADS == 64 + + +@GLUES +@pytest.mark.parametrize("world", sorted(HEADS)) +@pytest.mark.parametrize("rows", [1, 2, 7, 16]) +def test_decode_scratch(rows, world, unfused_glue): + heads = HEADS[world] + s = _target(M.DecodeTarget, heads, False, unfused_glue)._scratch(rows, torch.bfloat16, DEV) + _check_scratch(s, rows, heads, heads, lat_out=True, unfused_glue=unfused_glue) + assert s.sched is None + + +@GLUES +@pytest.mark.parametrize("world", sorted(HEADS)) +@pytest.mark.parametrize("rows", [1, 16]) +def test_decode_scratch_under_flash_mla(rows, world, unfused_glue): + heads = HEADS[world] + s = _target(M.DecodeTarget, heads, True, unfused_glue)._scratch(rows, torch.bfloat16, DEV) + _check_scratch( + s, rows, heads, M._FLASH_MLA_HEADS, lat_out=unfused_glue, unfused_glue=unfused_glue + ) + assert s.sched is not None + + +@GLUES +@pytest.mark.parametrize("flash_mla", [False, True]) +@pytest.mark.parametrize("world", sorted(HEADS)) +@pytest.mark.parametrize("tc, gen", [(5, 0), (31, 1), (1031, 15), (2048, 16)]) +def test_prefill_scratch(tc, gen, world, flash_mla, unfused_glue): + rows = tc + gen + heads = HEADS[world] + target = _target(M.PrefillTarget, heads, flash_mla, unfused_glue) + s = target._scratch(rows, torch.bfloat16, DEV) + _check_scratch(s, rows, heads, M._FLASH_MLA_HEADS, lat_out=True, unfused_glue=unfused_glue) + # The context and generation rows are row slices of the one buffer: the dead heads of + # either slice stay zero after the other is written. + assert s.fused_q[:tc].data_ptr() == s.fused_q.data_ptr() + assert (s.sched is not None) == flash_mla diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_long_context.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_long_context.py new file mode 100644 index 000000000000..990f7e8ced85 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_long_context.py @@ -0,0 +1,267 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The long-context witness of Mistral-Large-4-1T-A52B-NVFP4 on its sm_103 tp target, at tp4 +(four ranks) and at tp8 (eight ranks over two nodes of four). + +GSM8K's prompts never pass the indexer's `index_topk` (2048 positions). Below it the engine +skips the indexer and sparse MLA attends over the causal window, so the accuracy gate never +runs the indexer's selection, nor SSMax, LeRoPE and YaRN at long positions, nor the FlashMLA +sparse kernels on a selection past it (each called on the rank's heads -- 16 at tp4, 8 at tp8 -- +first in the 64-head query they take), nor the routed experts at the largest step. Here the +whole model runs under its serving configuration -- max_seq_len = max_num_tokens = 32768, chunked +prefill off, the latent KV cache at its default (the fp8_ds_mla pool), CUDA graphs over batches +1, 2, 4, 8 and 16, padded -- with the routed experts tensor parallel (every rank holds all 128 +experts, each cut to its slice of the 6144-wide intermediate: 1536 at tp4, 768 at tp8), and +greedy decoding must recall a passkey hidden in plain filler text: + +* in prompts of about 4K, 8K, 16K and 31K tokens, at three depths each, one prompt at a time: + each prompt is one context step of one request (read back from the engine's iteration + statistics), then decodes in the batch-1 graph; +* in sixteen prompts whose lengths sum to nearly 32768 tokens, six of them past `index_topk` + and ten within it, admitted together: they are one context step of sixteen requests (read + back the same way) holding the whole token budget, then decode in the batch-16 graph. + +Every rank sees every token of a step, so the ~31K prompt and the sixteen-prompt step each run +every rank's experts over the whole step: up to 32768 rows, 131072 routed slots, over 128 local +experts of width 1536 at tp4 and 768 at tp8. + +Text only: no image, no chat template. Two engine options beyond the serving ones make the +steps observable and do not change what a step computes: iteration statistics, and a request +queue that waits up to `BATCH_WAIT_MS` for a full batch so the sixteen prompts are admitted +in one step rather than split by their arrival. + +Runs with the ranks already in place, on four GB300s at tp4 or on eight over two nodes of four +at tp8: pytest under `trtllm-llmapi-launch`, as the whole-model accuracy class runs, with the +checkpoint under `LLM_MODELS_ROOT` as `Mistral-Large-4-1T-A52B-NVFP4` (or +`mistral-large-4-nvfp4`) as published, Mistral-native (params.json, tekken.json and the +consolidated safetensors), which the engine loads with checkpoint_format "mistral". One +scheduler serves all the ranks, so every step below is one step on every rank. Skipped where the +checkpoint is absent or off sm_103; each world size is skipped below its own ranks (four GPUs at +tp4, eight ranks at tp8) and inside a launch of another world size. +""" + +from __future__ import annotations + +import random +from pathlib import Path + +import pytest +import torch +from test_common.llm_data import llm_models_root +from utils.util import skip_num_gpus_less_than + +from tensorrt_llm import LLM, SamplingParams +from tensorrt_llm._torch.modeling_v2 import MODELING_V2_VALIDATE_ENV +from tensorrt_llm._utils import get_sm_version +from tensorrt_llm.inputs import TokensPrompt +from tensorrt_llm.llmapi import CudaGraphConfig, KvCacheConfig +from tensorrt_llm.llmapi.mpi_session import get_mpi_world_size + + +def _ranks() -> int: + """The ranks this process can serve: the MPI world a launcher started (eight over two nodes + of four, where each node sees four devices), or, unlaunched, the devices the LLM API spawns + its workers on -- as the integration conftest's `skip_less_mpi_world_size` counts.""" + world = get_mpi_world_size() + return torch.cuda.device_count() if world == 1 else world + + +def _launched_elsewhere(world: int) -> bool: + """A launcher started an MPI world of another size: an engine of `world` ranks does not + start inside it.""" + return get_mpi_world_size() not in (1, world) + + +#: The engine's world sizes, each gated on its own ranks: tp4 on four GPUs, tp8 on eight ranks +#: over two nodes of four. +WORLDS = [ + pytest.param( + 4, + marks=[ + skip_num_gpus_less_than(4), + pytest.mark.skipif( + _launched_elsewhere(4), reason="tp4 needs a launch of four ranks or none" + ), + ], + id="tp4", + ), + pytest.param( + 8, + marks=[ + pytest.mark.skipif(_ranks() < 8, reason="tp8 needs eight ranks"), + pytest.mark.skipif( + _launched_elsewhere(8), reason="tp8 needs a launch of eight ranks or none" + ), + ], + id="tp8", + ), +] + +pytestmark = [ + pytest.mark.skipif( + get_sm_version() != 103, + reason="the mistral_large4 modeling_v2 targets are certified on sm_103 only", + ), + # One engine serves the module: the LLM API starts its result-dispatch thread on the first + # request, inside the first test, and stops it when the fixture shuts the engine down. + pytest.mark.threadleak(enabled=False), +] + +CHECKPOINTS = ("Mistral-Large-4-1T-A52B-NVFP4", "mistral-large-4-nvfp4") + +# The serving configuration: tp4 or tp8 (attention and the routed experts tensor parallel over all +# the ranks, the experts not expert-parallel), sixteen sequences, unchunked steps of up to 32768 +# tokens. +MAX_NUM_TOKENS = 32768 +ENGINE = dict( + moe_expert_parallel_size=1, + enable_attention_dp=False, + max_batch_size=16, + max_num_tokens=MAX_NUM_TOKENS, + max_seq_len=MAX_NUM_TOKENS, + enable_chunked_prefill=False, + cuda_graph_config=CudaGraphConfig(max_batch_size=16, enable_padding=True), + kv_cache_config=KvCacheConfig(enable_block_reuse=False, free_gpu_memory_fraction=0.85), +) +# How long the request queue holds an iteration for more requests while fewer than +# max_batch_size are waiting; every iteration pays it, decode steps included. The sixteen +# prompts are submitted together as token ids and arrive within milliseconds of each other. +BATCH_WAIT_MS = 1000 + +INDEX_TOPK = 2048 +# Prompts of about 4K, 8K, 16K and 31K tokens, a passkey at three depths of each. +LENGTHS = (4096, 8192, 16384, 31744) +DEPTHS = (0.1, 0.5, 0.9) +# Sixteen prompts filling one 32768-token context step: six past index_topk, ten within it. +# Each is built to at most its length and within one filler sentence of it, so the step holds +# between 98% of the budget and 32700 tokens. +BATCH_PAST_TOPK = (8500, 4000, 3000, 2600, 2300, 2100) +BATCH_WITHIN_TOPK = (1900, 1700, 1500, 1300, 1100, 900, 700, 500, 350, 250) +BATCH_LENGTHS = BATCH_PAST_TOPK + BATCH_WITHIN_TOPK +BATCH_FILL = 0.98 +BATCH_DEPTHS = (0.1, 0.3, 0.5, 0.7, 0.9) + +HEAD = ( + "There is an important piece of information hidden inside a lot of irrelevant text. " + "Find it and memorize it. I will quiz you about it.\n\n" +) +FILLER = ( + "The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again. " +) +QUESTION = "\nWhat is the pass key? The pass key is" +# A five-digit key and its closing punctuation fit with room to spare. +ANSWER = SamplingParams(max_tokens=12, temperature=0.0) + + +def _checkpoint_dir() -> Path | None: + """The checkpoint's directory, found by its Mistral-native config, params.json.""" + root = llm_models_root() + if root is None: + return None + for name in CHECKPOINTS: + if (Path(root) / name / "params.json").is_file(): + return Path(root) / name + return None + + +def _prompt(reps: int, depth: float, key: int) -> str: + before = round(reps * depth) + needle = f"\nThe pass key is {key}. Remember it. {key} is the pass key.\n" + return HEAD + FILLER * before + needle + FILLER * (reps - before) + QUESTION + + +def _passkey_ids(tokenizer, tokens: int, depth: float, key: int) -> list[int]: + """The prompt's token ids, BOS included as the LLM API's own tokenization adds it, at most + `tokens` long and within one filler sentence of it.""" + unit = len(tokenizer.encode(FILLER * 64)) / 64 + bare = len(tokenizer.encode(_prompt(0, depth, key))) + reps = max(0, int((tokens - bare) / unit)) + ids = tokenizer.encode(_prompt(reps, depth, key)) + while len(ids) > tokens and reps > 0: + reps -= 1 + ids = tokenizer.encode(_prompt(reps, depth, key)) + return ids + + +def _keys(count: int, seed: int) -> list[int]: + return random.Random(seed).sample(range(10000, 100000), count) + + +def _context_steps(llm: LLM) -> list[tuple[int, int]]: + """(context requests, context tokens) of each iteration since the last call that ran any + context rows.""" + steps = [] + for record in llm.get_stats(timeout=2): + ifb = record.get("inflightBatchingStats") or {} + if ifb.get("numContextRequests", 0): + steps.append((ifb["numContextRequests"], ifb["numCtxTokens"])) + return steps + + +@pytest.fixture(scope="module", params=WORLDS) +def llm(request): + world = request.param + path = _checkpoint_dir() + if path is None: + pytest.skip(f"checkpoint {CHECKPOINTS} is not under LLM_MODELS_ROOT ({llm_models_root()})") + with LLM( + str(path), + checkpoint_format="mistral", + modeling_v2="require", + env_overrides={MODELING_V2_VALIDATE_ENV: "1"}, + enable_iter_perf_stats=True, + batch_wait_timeout_ms=BATCH_WAIT_MS, + tensor_parallel_size=world, + moe_tensor_parallel_size=world, + **ENGINE, + ) as engine: + capacity = engine.get_kv_cache_capacity() + print(f"LONGCTX kv_cache_capacity {capacity}", flush=True) + _context_steps(engine) + yield engine + + +@pytest.mark.parametrize("tokens", LENGTHS) +def test_passkey_alone_in_one_context_step(llm, tokens): + """A passkey at three depths of a prompt of about `tokens` tokens, each prompt one context + step of one request, recalled by greedy decoding.""" + misses = [] + for depth, key in zip(DEPTHS, _keys(len(DEPTHS), seed=tokens)): + ids = _passkey_ids(llm.tokenizer, tokens, depth, key) + assert INDEX_TOPK < len(ids) <= tokens, (tokens, len(ids)) + answer = llm.generate(TokensPrompt(prompt_token_ids=ids), ANSWER).outputs[0].text + steps = _context_steps(llm) + print( + f"LONGCTX alone tokens={len(ids)} depth={depth} key={key} answer={answer!r} " + f"context_steps={steps}", + flush=True, + ) + assert steps == [(1, len(ids))], (len(ids), depth, steps) + if str(key) not in answer: + misses.append((len(ids), depth, key, answer)) + assert not misses, misses + + +def test_sixteen_prompts_fill_one_context_step(llm): + """Sixteen passkey prompts admitted together are one context step of sixteen requests + holding nearly the whole 32768-token budget, and every one recalls its key.""" + keys = _keys(len(BATCH_LENGTHS), seed=MAX_NUM_TOKENS) + prompts = [ + _passkey_ids(llm.tokenizer, tokens, BATCH_DEPTHS[i % len(BATCH_DEPTHS)], key) + for i, (tokens, key) in enumerate(zip(BATCH_LENGTHS, keys)) + ] + lengths = [len(ids) for ids in prompts] + total = sum(lengths) + assert sum(n > INDEX_TOPK for n in lengths) == len(BATCH_PAST_TOPK), lengths + assert BATCH_FILL * MAX_NUM_TOKENS <= total <= MAX_NUM_TOKENS, (total, lengths) + + outputs = llm.generate([TokensPrompt(prompt_token_ids=ids) for ids in prompts], ANSWER) + steps = _context_steps(llm) + answers = [out.outputs[0].text for out in outputs] + for n, key, answer in zip(lengths, keys, answers): + print(f"LONGCTX batch tokens={n} key={key} answer={answer!r}", flush=True) + print(f"LONGCTX batch total={total} context_steps={steps}", flush=True) + + assert steps == [(len(prompts), total)], (total, steps) + misses = [(n, key, a) for n, key, a in zip(lengths, keys, answers) if str(key) not in a] + assert not misses, misses diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_routed_experts.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_routed_experts.py new file mode 100644 index 000000000000..4e54f4dea6c3 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_routed_experts.py @@ -0,0 +1,527 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""One real layer's routed experts, width-sliced four or eight ways through the prepared layout, on +one GPU. + +At tp4 every rank holds all 128 routed experts of a layer, each cut to its 1536-wide slice of +the 6144-wide intermediate, and at tp8 (eight ranks over two nodes of four) to its 768-wide +slice: `w1`/`w3` by output rows, `w2` by input columns (a strided byte read of the packed codes, +and of the block scales one per 16 inputs), `weight_scale_2` and `input_scale` whole. The NVFP4 +MoE runner reads each slice in a layout no checkpoint stores -- FC1's halves interleaved, every +stack's rows block-shuffled, the block scales 128x4-swizzled -- and nothing checks either the +cut or the layout: a wrong one costs hundreds of bf16 ulp and raises nothing. So, on an early, a +middle and a late MoE layer (1, 35, 71) of Mistral-Large-4-1T-A52B-NVFP4 itself, at each world +size W (4 and 8), for each rank r in 0..W-1 in turn: + +* the rank's stacks are filled from the layer's stored tensors by the weight table's own manifest + rows (sources, shards, destination indices, transforms) for the core the target builds on rank + r, and laid out by the rows' own `layout`, the steps the table's `load` takes; that layout is + byte for byte the engine's own TRTLLM-Gen preparation of the same slice + (`trtllmgen_maybe_get_cached_w3_w1_permute_indices`, `shuffle_matrix`, + `block_scale_interleave`); +* the target's `_mlp` -- the layer's router through `renorm_moe_routing_op`, `fp4_quantize` under + the layer's static scale, the pre-routed runner over all 128 experts at the core's own + `moe_inter` with `_moe_scalars` -- runs on bf16 activations at the magnitudes the layer's input + scale was calibrated for (16 rows through `DecodeTarget`, 1031 and 32768 through + `PrefillTarget`, the last 131072 permuted rows over the 128 local experts), with the shared + expert zeroed so the routed partial is all that is measured, and is compared with an fp32 + reference on rank r's slice of the dequantized checkpoint experts, cut here from the whole + stored tensors (`e2m1 * weight_scale * weight_scale_2`, activations `e2m1 * scale * + input_scale`, the SwiGLU output requantized to NVFP4 under `1 / input_scale(w2)` as the + reference implementation does), at the gate the runner's element band certifies a slice of + width I at: 16 * sqrt(6144 / I) ulp per element of the row's scale (32 at 1536, 45.25 at + 768), 2 ulp relative RMS; + +then the W ranks' partials, summed in fp32 in rank order as the closing all-reduce sums them, +are compared with the same reference over the whole experts at the whole-width gate, 16 ulp per +element and 2 ulp RMS. With every token's top-4 skewed onto four experts (124..127, the router's +column 0 biased for them and the activations' element 0 held positive), 32768 tokens put 32768 +rows on each of the four; every rank's partial and the W ranks' sum meet the same gates. +Wrong readings run without complaint and must land outside the slice's gate: the stored slice +without the layout, FC1 with its halves swapped, and `w2`'s codes or its block scales taken from +the neighbouring rank's column range; and on the sum, one rank's partial taken twice in place of +its neighbour's. + +Skipped where the checkpoint is absent (looked up as `mistral-large-4-nvfp4` or +`Mistral-Large-4-1T-A52B-NVFP4` under `LLM_MODELS_ROOT`).""" + +from __future__ import annotations + +import collections +import json +import math +from pathlib import Path +from types import SimpleNamespace + +import pytest +import torch +from safetensors import safe_open +from test_common.llm_data import llm_models_root + +from tensorrt_llm._torch.model_config import ModelConfig +from tensorrt_llm._torch.modeling_v2.catalog.gemm.cublas_mm import cublas_mm +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_103__tp import ( # noqa: E501 + modeling as M, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_103__tp import ( # noqa: E501 + weights as W, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.moe.renorm_moe_routing_op import ( + renorm_moe_routing_op, +) +from tensorrt_llm._torch.models.checkpoints.mistral.config_loader import MistralConfigLoader +from tensorrt_llm._torch.moe.fused_moe.quantization import ( + trtllmgen_maybe_get_cached_w2_permute_indices, + trtllmgen_maybe_get_cached_w3_w1_permute_indices, +) +from tensorrt_llm.mapping import Mapping + +__extra_import_path__ = ["../../.."] +from _precision import true_fp32_matmul # noqa: E402 + +DEV = "cuda" +#: The world sizes and each one's slice of every expert's 6144-wide intermediate. +SLICE_WIDTH = {4: 1536, 8: 768} +#: GB300 nodes here hold four devices; the eight-rank world spans two of them. +GPUS_PER_NODE = 4 +CHECKPOINTS = ("mistral-large-4-nvfp4", "Mistral-Large-4-1T-A52B-NVFP4") +#: The expert parameters of one MoE layer, in the table's names. +EXPERT_PARAMS = ("fc1_w", "fc1_s", "fc2_w", "fc2_s", "isc1", "isc2", "ws2_13", "ws2_2") +#: The engine bound the target is built for: the runner's tuner bucket and the largest step. +MAX_NUM_TOKENS = 32768 +#: The runner's epilogue tile, which fixes its 32-row shuffle (the engine's own constant). +EPILOGUE_TILE_M = 128 +#: One bf16 ulp, relative: 2^-8 of the binade top. +ULP = 2.0**-8 +#: The largest e2m1 magnitude and the largest finite e4m3 one. +E2M1_MAX, E4M3_MAX = 6.0, 448.0 +#: The gates the runner certifies a width slice at: an element of one rank's I-wide partial reads +#: a marginal FC2-input flip up to sqrt(6144 / I) larger against its row's scale than the whole +#: width does -- 2x at 1536, 2.83x at 768 -- so 16 * sqrt(6144 / I) ulp per element there (32 and +#: 45.25) and 16 at the whole width; RMS 2 ulp at both. +ELT_WHOLE, RMS = 16.0, 2.0 +ELT_SLICE = {world: ELT_WHOLE * math.sqrt(6144 / width) for world, width in SLICE_WIDTH.items()} +#: A wrong reading must land this far past the RMS gate. +WRONG_RMS = 5 * RMS +#: An early, a middle and a late MoE layer, as the runner's cells spread the layer scalars. +LAYERS = (1, 35, 71) +#: (target, rows): a decode step at the largest captured batch, an eager context step, and the +#: largest step the engine sends (max_num_tokens, unchunked). +STEPS = (("DecodeTarget", 16), ("PrefillTarget", 1031), ("PrefillTarget", MAX_NUM_TOKENS)) +#: The rows the wrong readings run at. +WRONG_ROWS = 1031 +#: The skew: every token's top-4 on these experts, at the largest step. +SKEW_EXPERTS = (124, 125, 126, 127) +SKEW_ROWS = MAX_NUM_TOKENS + + +def _checkpoint_dir() -> Path | None: + root = llm_models_root() + if root is None: + return None + for name in CHECKPOINTS: + path = root / name + if (path / "params.json").is_file() and (path / W.SHARD_INDEX).is_file(): + return path + return None + + +@pytest.fixture(scope="module") +def checkpoint(): + path = _checkpoint_dir() + if path is None: + pytest.skip(f"checkpoint {CHECKPOINTS} is not under LLM_MODELS_ROOT ({llm_models_root()})") + assert torch.cuda.is_available(), "the routed-expert check needs a GPU" + weight_map = json.loads((path / W.SHARD_INDEX).read_text())["weight_map"] + # The configuration the engine adapts from the checkpoint's params.json. + config = MistralConfigLoader().load_pretrained_config(str(path)) + return SimpleNamespace(path=path, weight_map=weight_map, config=config) + + +def _core(config, world: int, rank: int) -> M.MistralLarge4TpModelingV2: + """The target's own core for rank `rank` of `world` (tp4 or tp8), declared on the meta + device: its geometry is the core's derivation off the mapping, and the weight table reads it + as `load` does.""" + mapping = Mapping( + world_size=world, + rank=rank, + gpus_per_node=GPUS_PER_NODE, + tp_size=world, + moe_tp_size=world, + moe_ep_size=1, + ) + assert (mapping.tp_rank, mapping.moe_tp_rank, mapping.moe_ep_rank) == (rank, rank, 0), mapping + model_config = ModelConfig( + pretrained_config=config, + mapping=mapping, + max_num_tokens=MAX_NUM_TOKENS, + max_seq_len=MAX_NUM_TOKENS, + ) + with torch.device("meta"): + return M.MistralLarge4TpModelingV2(model_config) + + +def _read(checkpoint, keys) -> dict[str, torch.Tensor]: + """The given checkpoint tensors, whole, off the shard files, one file at a time.""" + by_file = collections.defaultdict(list) + for key in keys: + by_file[checkpoint.weight_map[key]].append(key) + out = {} + for name in sorted(by_file): + with safe_open(str(checkpoint.path / name), framework="pt", device="cpu") as handle: + for key in by_file[name]: + out[key] = handle.get_tensor(key) + return out + + +def _layer_keys(core, layer: int) -> list[str]: + """Every checkpoint tensor the layer's routed experts and router load from, on any rank.""" + manifest = W.MODEL_WEIGHTS.manifest(core) + keys = { + k + for name in EXPERT_PARAMS + for ckpt, *_ in manifest[f"l{layer}_{name}"] + for k in W._keys(ckpt) + } + return sorted(keys | {f"layers.{layer}.gate.weight"}) + + +def _rank_params(core, stored: dict, layer: int) -> dict[str, torch.Tensor]: + """Layer `layer`'s expert parameters on the core's rank, as the table's `load` leaves them: + every manifest source cut by its shard, copied to its index through its transform, then each + row's `layout` on every expert.""" + d = W.MODEL_WEIGHTS.dims(core) + manifest = W.MODEL_WEIGHTS.manifest(core) + rows = {r.name: r for r in W.MODEL_WEIGHTS.WEIGHTS if r.name in EXPERT_PARAMS} + params = {} + for name in EXPERT_PARAMS: + row = rows[name] + param = torch.empty(row.shape(d), dtype=row.dtype or d.dtype, device=DEV) + for ckpt, index, transform, shard in manifest[f"l{layer}_{name}"]: + dst = param if index is None else param[index] + if isinstance(ckpt, tuple): + src = transform(tuple(W._materialize(stored[k], shard).to(DEV) for k in ckpt), core) + else: + src = W._materialize(stored[ckpt], shard).to(DEV) + if transform is not None: + src = transform(src.contiguous(), core) + assert tuple(dst.shape) == tuple(src.shape) and dst.dtype == src.dtype, (name, ckpt) + dst.copy_(src) + if row.layout is not None: + for slot in range(param.shape[0]): + param[slot].copy_(row.layout(param[slot])) + params[name] = param + return params + + +def _stored_slices(stored: dict, layer: int, experts: int, inter: int, rank: int): + """Rank `rank`'s stored bytes per expert, cut here from the whole tensors: `[w3 ; w1]` rows + `[inter * rank, + inter)` of the codes and of the block scales, `w2`'s input columns over the + same range (two per code byte, sixteen per block scale).""" + rows = slice(inter * rank, inter * (rank + 1)) + code_cols = slice(inter * rank // 2, inter * (rank + 1) // 2) + scale_cols = slice(inter * rank // 16, inter * (rank + 1) // 16) + out = collections.defaultdict(list) + for e in range(experts): + p = f"layers.{layer}.experts.{e}" + out["fc1_w"].append( + torch.cat([stored[f"{p}.w3.weight"][rows], stored[f"{p}.w1.weight"][rows]]) + ) + out["fc1_s"].append( + torch.cat([stored[f"{p}.w3.weight_scale"][rows], stored[f"{p}.w1.weight_scale"][rows]]) + ) + out["fc2_w"].append(stored[f"{p}.w2.weight"][:, code_cols].contiguous()) + out["fc2_s"].append(stored[f"{p}.w2.weight_scale"][:, scale_cols].contiguous()) + return out + + +def _engine_layout(stored: dict[str, list[torch.Tensor]]) -> dict[str, torch.Tensor]: + """The engine's own TRTLLM-Gen NVFP4 MoE preparation of the stored slices, expert by expert: + its gated-activation row reorder chained with its row shuffle, `shuffle_matrix`, and + `block_scale_interleave` on the scales.""" + cache: dict = {} + sf = torch.uint8 + out = collections.defaultdict(list) + for w31, s31, w2, s2 in zip(stored["fc1_w"], stored["fc1_s"], stored["fc2_w"], stored["fc2_s"]): + w31, s31, w2, s2 = (t.to(DEV) for t in (w31, s31, w2, s2)) + perm = trtllmgen_maybe_get_cached_w3_w1_permute_indices(w31, cache, EPILOGUE_TILE_M) + out["fc1_w"].append(torch.ops.trtllm.shuffle_matrix(w31, perm)) + perm = trtllmgen_maybe_get_cached_w3_w1_permute_indices( + s31.view(sf), cache, EPILOGUE_TILE_M, num_elts_per_sf=16 + ) + shuffled = torch.ops.trtllm.shuffle_matrix(s31.view(sf), perm) + out["fc1_s"].append( + torch.ops.trtllm.block_scale_interleave(shuffled.reshape(s31.shape)).reshape(s31.shape) + ) + perm = trtllmgen_maybe_get_cached_w2_permute_indices(w2, cache, EPILOGUE_TILE_M) + out["fc2_w"].append(torch.ops.trtllm.shuffle_matrix(w2, perm)) + perm = trtllmgen_maybe_get_cached_w2_permute_indices( + s2.view(sf), cache, EPILOGUE_TILE_M, num_elts_per_sf=16 + ) + shuffled = torch.ops.trtllm.shuffle_matrix(s2.view(sf), perm) + out["fc2_s"].append( + torch.ops.trtllm.block_scale_interleave(shuffled.reshape(s2.shape)).reshape(s2.shape) + ) + return {name: torch.stack(t) for name, t in out.items()} + + +def _unpack(codes: torch.Tensor) -> torch.Tensor: + """`[..., K/2]` packed e2m1 -> `[..., K]` codes, the even element in the low nibble.""" + return torch.stack([codes & 0x0F, codes >> 4], dim=-1).reshape(*codes.shape[:-1], -1) + + +def _whole_operands(stored: dict, layer: int, experts: int, ws2_13, ws2_2) -> SimpleNamespace: + """The layer's whole experts as the reference reads them: unpacked e2m1 codes `[E, N, K]` and + fp32 block scales `[E, N, K/16]` with each expert's `weight_scale_2` multiplied in, for the up + (`w3`), gate (`w1`) and down (`w2`) projections.""" + + def operand(proj, global_scale): + p = f"layers.{layer}.experts" + codes = torch.stack( + [_unpack(stored[f"{p}.{e}.{proj}.weight"].to(DEV)) for e in range(experts)] + ) + scales = torch.stack( + [stored[f"{p}.{e}.{proj}.weight_scale"].to(DEV).float() for e in range(experts)] + ) + return codes, scales * global_scale.view(-1, 1, 1) + + return SimpleNamespace( + up=operand("w3", ws2_13), gate=operand("w1", ws2_13), down=operand("w2", ws2_2) + ) + + +def _reference(x_q, x_s, isc1, isc2, ids, weights, ops, inter: int, rank: int | None): + """The routed half in fp32 over the dequantized checkpoint experts -- rank `rank`'s + intermediate slice `[inter * rank, + inter)` of each, or the whole experts for None -- + rounded to bf16: the runner entry's `reference`, handed the activation's exact NVFP4 value + with its global scale undone (`code * scale * input_scale`).""" + rows = x_q.shape[0] + act = _unpack(x_q).long() + act_scale = x_s.view(torch.float8_e4m3fn).float().reshape(rows, -1) + e2m1 = torch.tensor( + [0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0, -0.0, -0.5, -1.0, -1.5, -2.0, -3.0, -4.0, -6.0], + device=DEV, + ) + hidden = e2m1[act] * act_scale.repeat_interleave(16, dim=1) * isc1.float() + if rank is None: + up, gate, down = ops.up, ops.gate, ops.down + else: + out_rows = slice(inter * rank, inter * (rank + 1)) + in_scales = slice(inter * rank // 16, inter * (rank + 1) // 16) + up = (ops.up[0][:, out_rows], ops.up[1][:, out_rows]) + gate = (ops.gate[0][:, out_rows], ops.gate[1][:, out_rows]) + down = (ops.down[0][:, :, out_rows], ops.down[1][:, :, in_scales]) + with true_fp32_matmul(): + out = M.fp4_block_scale_moe_runner.reference( + hidden, + ids, + weights, + up, + gate, + down, + fc2_global_scale=1.0 / float(isc2), + local_expert_offset=0, + local_num_experts=up[0].shape[0], + ) + return out.to(torch.bfloat16) + + +def _deviation(out: torch.Tensor, ref: torch.Tensor) -> tuple[float, float]: + """(worst element deviation against its row's scale, relative RMS deviation), in ulp.""" + o, r = out.float(), ref.float() + row = r.abs().amax(dim=1, keepdim=True).clamp_min(1e-30) + elt = ((o - r).abs() / row).max().item() / ULP + rms = ((o - r).pow(2).mean().sqrt() / r.pow(2).mean().sqrt().clamp_min(1e-30)).item() / ULP + return elt, rms + + +def _layer(d, params, router) -> SimpleNamespace: + """The `L` a target's `_mlp` reads: a shared expert whose down projection is zero (its + partial is exactly 0, so the result is the routed partial alone), the router's operand, the + rank's laid-out slices and the call scalars from `_moe_scalars`.""" + shared = d.shared_inter + unit_sf = lambda n, k: W._deepgemm_sf( # noqa: E731 + torch.ones(n // W.BLOCK, k // W.BLOCK, device=DEV), None + ).t() + return SimpleNamespace( + mlp_gu=(torch.randn(2 * shared, d.hidden, device=DEV) * 0.1).to(torch.float8_e4m3fn), + mlp_gu_s=unit_sf(2 * shared, d.hidden), + mlp_dn=torch.zeros(d.hidden, shared, device=DEV).to(torch.float8_e4m3fn), + mlp_dn_s=unit_sf(d.hidden, shared), + moe=SimpleNamespace( + router=router.t(), + fc1_w=params["fc1_w"], + fc1_s=params["fc1_s"], + fc2_w=params["fc2_w"], + fc2_s=params["fc2_s"], + **vars( + M._moe_scalars(params["isc1"], params["isc2"], params["ws2_13"], params["ws2_2"]) + ), + ), + ) + + +def _target(name: str, core): + """A `PrefillTarget` or `DecodeTarget` over the core, its per-layer views not taken (the + meta core holds no weights): `_mlp` reads only the core's geometry and the `L` it is given.""" + target = object.__new__(getattr(M, name)) + target.core = core + return target + + +def _run(L, core, x_q, x_s, ids, weights, ops: dict) -> torch.Tensor: + """The runner as `_mlp` calls it, on the given (possibly wrong) expert operands.""" + s = L.moe + return M.fp4_block_scale_moe_runner( + None, None, x_q, x_s.view(torch.float8_e4m3fn), ops["fc1_w"].contiguous(), + ops["fc1_s"].contiguous(), None, None, None, None, ops["fc2_w"].contiguous(), + ops["fc2_s"].contiguous(), None, s.out1, s.out1_gate, s.out2, core.num_experts, + core.topk, None, None, core.moe_inter, core.expert_offset, core.local_experts, None, 1, + True, 0, topk_weights=weights, topk_ids=ids, tune_max_num_tokens=core.max_num_tokens, + )[0] # fmt: skip + + +@pytest.mark.parametrize("layer", LAYERS, ids=[f"layer{n}" for n in LAYERS]) +@pytest.mark.parametrize("world", sorted(SLICE_WIDTH), ids=[f"tp{w}" for w in sorted(SLICE_WIDTH)]) +def test_width_slices_of_one_real_layer(checkpoint, world, layer): + torch.manual_seed(1000 * layer) + elt_slice = ELT_SLICE[world] + cores = [_core(checkpoint.config, world, rank) for rank in range(world)] + geometry = {(c.moe_inter, c.local_experts, c.expert_offset, c.num_experts) for c in cores} + assert geometry == {(SLICE_WIDTH[world], 128, 0, 128)}, geometry + inter, experts = cores[0].moe_inter, cores[0].local_experts + d = W.MODEL_WEIGHTS.dims(cores[0]) + + stored = _read(checkpoint, _layer_keys(cores[0], layer)) + router = stored[f"layers.{layer}.gate.weight"].to(DEV) + params = [_rank_params(core, stored, layer) for core in cores] + slices = [_stored_slices(stored, layer, experts, inter, rank) for rank in range(world)] + + for rank in range(world): + # The table's layout is the engine's own preparation of the same slice. + engine = _engine_layout(slices[rank]) + for name in ("fc1_w", "fc1_s", "fc2_w", "fc2_s"): + mine = params[rank][name].view(torch.uint8) + assert torch.equal(mine, engine[name].view(torch.uint8)), (rank, name) + # The per-tensor scalars are whole: every rank holds every expert's. + for name in ("isc1", "isc2", "ws2_13", "ws2_2"): + assert torch.equal(params[rank][name], params[0][name]), (rank, name) + + isc1, isc2 = params[0]["isc1"], params[0]["isc2"] + ops = _whole_operands(stored, layer, experts, params[0]["ws2_13"], params[0]["ws2_2"]) + del stored + layers = [_layer(d, params[rank], router) for rank in range(world)] + # Activations at the amax the layer's static input scale was calibrated for (448 * 6 * + # input_scale): a gaussian whose largest elements reach about it. + calibrated = E4M3_MAX * E2M1_MAX * float(isc1) + kept = {} + for name, rows in STEPS: + o = (torch.randn(rows, d.hidden, device=DEV) * (calibrated / 5)).to(torch.bfloat16) + # The router and the quantization are replicated: every rank routes every row the same. + logits = cublas_mm(o, layers[0].moe.router, None, torch.float32) + ids, weights = renorm_moe_routing_op(logits, cores[0].topk, torch.bfloat16) + x_q, x_s = M.fp4_quantize(o, layers[0].moe.gscale, 16, False, False) + partials, refs = [], [] + for rank in range(world): + out = _target(name, cores[rank])._mlp(o, layers[rank]) + ref = _reference(x_q, x_s, isc1, isc2, ids, weights, ops, inter, rank) + elt, rms = _deviation(out, ref) + print( + f"layer {layer} rank {rank} {name} T={rows}: slice partial {elt:.2f} elt / " + f"{rms:.3f} rms ulp", + flush=True, + ) + assert rms <= RMS, f"rank {rank}: relative RMS {rms:.2f} ulp > {RMS}" + assert elt <= elt_slice, f"rank {rank}: element deviation {elt:.2f} ulp > {elt_slice}" + partials.append(out) + refs.append(ref) + whole = _reference(x_q, x_s, isc1, isc2, ids, weights, ops, inter, None) + summed = partials[0].float() + for partial in partials[1:]: + summed = summed + partial.float() + elt, rms = _deviation(summed.to(torch.bfloat16), whole) + print( + f"layer {layer} {name} T={rows}: {world} partials summed {elt:.2f} elt / {rms:.3f} " + "rms ulp against the whole experts", + flush=True, + ) + assert rms <= RMS, f"{world}-rank sum: relative RMS {rms:.2f} ulp > {RMS}" + assert elt <= ELT_WHOLE, f"{world}-rank sum: element deviation {elt:.2f} ulp > {ELT_WHOLE}" + # One rank's partial taken twice, in place of its neighbour's, is told apart. + doubled = partials[0].float() * 2 + for partial in partials[2:]: + doubled = doubled + partial.float() + _, rms = _deviation(doubled.to(torch.bfloat16), whole) + print(f"layer {layer} {name} T={rows}: rank 0 summed twice, no rank 1: {rms:.1f} rms ulp") + assert rms > WRONG_RMS, f"a doubled slice at {rms:.2f} rms ulp is not told apart" + if rows == WRONG_ROWS: + kept = dict(x_q=x_q, x_s=x_s, ids=ids, weights=weights, refs=refs) + + # Wrong readings run without complaint and land far away. + for rank in range(world): + neighbour = (rank + 1) % world + as_stored = {name: torch.stack(t).to(DEV) for name, t in slices[rank].items()} + swapped = torch.cat([as_stored["fc1_w"][:, inter:], as_stored["fc1_w"][:, :inter]], dim=1) + wrong = { + "no layout": as_stored, + "FC1 halves swapped": dict( + params[rank], fc1_w=torch.stack([W._fc1_codes(t) for t in swapped]) + ), + f"w2 codes of rank {neighbour}": dict(params[rank], fc2_w=params[neighbour]["fc2_w"]), + f"w2 block scales of rank {neighbour}": dict( + params[rank], fc2_s=params[neighbour]["fc2_s"] + ), + } + for what, wrong_ops in wrong.items(): + out = _run( + layers[rank], cores[rank], kept["x_q"], kept["x_s"], kept["ids"], kept["weights"], + wrong_ops, + ) # fmt: skip + elt, rms = _deviation(out, kept["refs"][rank]) + print(f"layer {layer} rank {rank} {what}: {elt:.1f} elt / {rms:.1f} rms ulp") + assert rms > WRONG_RMS, f"{what}: {rms:.2f} rms ulp is not told apart from the slice" + + # Skew: every token's top-4 on SKEW_EXPERTS. Column 0 of the router is biased up for those + # experts and down for the rest, and every token's element 0 is positive, so the bias + # dominates the logits; the ranking among the four is the router's own. + o = (torch.randn(SKEW_ROWS, d.hidden, device=DEV) * (calibrated / 5)).to(torch.bfloat16) + o[:, 0] = o[:, 0].abs() + calibrated / 10 + base = cublas_mm(o, router.t(), None, torch.float32) + bias = 64.0 * float(base.abs().max()) / float(o[:, 0].float().min()) + skewed = router.clone() + hot = torch.zeros(experts, dtype=torch.bool, device=DEV) + hot[list(SKEW_EXPERTS)] = True + skewed[hot, 0] += bias + skewed[~hot, 0] -= bias + logits = cublas_mm(o, skewed.t(), None, torch.float32) + ids, weights = renorm_moe_routing_op(logits, cores[0].topk, torch.bfloat16) + assert {int(e) for e in ids.unique()} == set(SKEW_EXPERTS), ids.unique() + x_q, x_s = M.fp4_quantize(o, layers[0].moe.gscale, 16, False, False) + partials = [] + for rank in range(world): + out = _target("PrefillTarget", cores[rank])._mlp(o, _layer(d, params[rank], skewed)) + ref = _reference(x_q, x_s, isc1, isc2, ids, weights, ops, inter, rank) + elt, rms = _deviation(out, ref) + print( + f"layer {layer} rank {rank} skew onto {SKEW_EXPERTS} T={SKEW_ROWS}: slice partial " + f"{elt:.2f} elt / {rms:.3f} rms ulp", + flush=True, + ) + assert rms <= RMS, f"skew, rank {rank}: relative RMS {rms:.2f} ulp > {RMS}" + assert elt <= elt_slice, f"skew, rank {rank}: element deviation {elt:.2f} ulp > {elt_slice}" + partials.append(out) + whole = _reference(x_q, x_s, isc1, isc2, ids, weights, ops, inter, None) + summed = partials[0].float() + for partial in partials[1:]: + summed = summed + partial.float() + elt, rms = _deviation(summed.to(torch.bfloat16), whole) + print( + f"layer {layer} skew T={SKEW_ROWS}: {world} partials summed {elt:.2f} elt / {rms:.3f} " + "rms ulp against the whole experts", + flush=True, + ) + assert rms <= RMS, f"skew, {world}-rank sum: relative RMS {rms:.2f} ulp > {RMS}" + assert elt <= ELT_WHOLE, ( + f"skew, {world}-rank sum: element deviation {elt:.2f} ulp > {ELT_WHOLE}" + ) + del o, base, logits, x_q, x_s, partials, summed, whole diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_tp4_1m_engine_config.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_tp4_1m_engine_config.py new file mode 100644 index 000000000000..29bf7d9df624 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_tp4_1m_engine_config.py @@ -0,0 +1,1554 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Mistral-Large-4-1T-A52B-NVFP4 at tp4 in the checkpoint's own context: the engine at max_seq_len +1048576 with chunked prefill in 32768-token steps, as a served engine reaches it. + +No GPU and no weights. The checkpoint goes through the engine's own configuration path with the +LLM-API options of the 1M core unit (four ranks, tensor parallel throughout, the routed experts cut +along their width, 16 requests and 32768 tokens per step, max_seq_len 1048576, chunked prefill on, +block reuse off, the KV cache at its default dtype), and in the spelling the quickstart gates use +(``--kv_cache_dtype=fp8``). On the configuration it resolves to, the engine's own host code is run +wherever a CPU reaches it: + +* (1) the limits on their way to the executor and the attention metadata, the quantization key, + the KV cache manager both spellings build (``MistralLarge4CacheManager``, the fp8_ds_mla latent + cache), the DSA sparse config and the attention runtime features that turn on context MLA over + cached KV; +* (2) the DSA indexer's prefill planning (``Indexer.prepare_for_chunked_prefill``, which calls + ``_split_prefill_queries`` and ``plan_mla_chunked_prefill_chunks``) on each of the four ranks' + host metadata: the query split across the TP ranks that the target must gather back; +* (3) the KV-capacity estimate (``KvCacheCreator``): its dummy requests, the reservations it makes + at this config and the pool and admission cap it leaves of the 32K boot's budget; the + warmup's steps and CUDA-graph decode rows; the target's held context workspace, sized from the + engine's quantities, and the context-row path it takes on each of these steps and on a 1M + prompt's chunks; +* (4) the DSA metadata buffers that grow with max_seq_len, and the per-step context gather + mappings. + +The delivered 32K configuration (pinned in test_modeling_v2_tp4_engine_config.py) is resolved +beside it as the contrast. The checkpoint is Mistral-native, as published (params.json, +tekken.json, consolidated safetensors), loaded with checkpoint_format mistral in every +spelling. It is looked up under ``LLM_MODELS_ROOT`` (as ``Mistral-Large-4-1T-A52B-NVFP4`` or +``mistral-large-4-nvfp4``) by its params.json, and the tests are skipped where it is +absent. A CPU job may still see the node's GPUs: nothing here opens a CUDA context, +allocates pinned memory or reads a device property, and the SM is stated as sm_103 wherever the +engine asks for it. +""" + +from __future__ import annotations + +import copy +import importlib +import itertools +import json +import math +import threading +import types +from pathlib import Path +from types import SimpleNamespace + +import pytest +import torch +import yaml +from test_common.llm_data import llm_models_root + +from tensorrt_llm._torch.attention.backends import trtllm as trtllm_backend +from tensorrt_llm._torch.attention.backends.interface import AttentionRuntimeFeatures +from tensorrt_llm._torch.attention.backends.sparse.dsa import indexer as dsa_indexer +from tensorrt_llm._torch.attention.backends.sparse.dsa import metadata as dsa_metadata +from tensorrt_llm._torch.attention.backends.sparse.dsa.indexer import Indexer +from tensorrt_llm._torch.attention.backends.sparse.dsa.metadata import DSAtrtllmAttentionMetadata +from tensorrt_llm._torch.attention.backends.sparse.mistral_large4 import MistralLarge4CacheManager +from tensorrt_llm._torch.attention.backends.sparse.params import ( + get_indexer_mqa_logits_workspace_bytes, +) +from tensorrt_llm._torch.modeling_v2 import _router_index +from tensorrt_llm._torch.modeling_v2.models.mistral_large4 import routing +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_103__tp import ( # noqa: E501 + modeling as tp4_modeling, +) +from tensorrt_llm._torch.models.modeling_utils import DecoderModelForCausalLM +from tensorrt_llm._torch.pyexecutor import _util, model_loader, py_executor_creator +from tensorrt_llm._torch.pyexecutor._util import KvCacheCreator +from tensorrt_llm._torch.pyexecutor.engine.metadata import build_attention_metadata +from tensorrt_llm._torch.pyexecutor.engine.runners.decoder.dummy_requests import DummyRequestFactory +from tensorrt_llm._torch.pyexecutor.engine.runners.decoder.warmup import ( + DecoderWarmup, + WarmupAgreement, +) +from tensorrt_llm._torch.pyexecutor.kv_cache import kv_cache_manager_v2 +from tensorrt_llm._torch.pyexecutor.model_engine import ModelEngine +from tensorrt_llm._torch.pyexecutor.model_loader import ModelLoader as RankModelLoader +from tensorrt_llm._torch.pyexecutor.py_executor import PyExecutor +from tensorrt_llm._torch.pyexecutor.resource_manager import ResourceManagerType +from tensorrt_llm.bindings.internal import thop +from tensorrt_llm.llmapi.llm_args import SeqLenAwareSparseAttentionConfig, TorchLlmArgs +from tensorrt_llm.llmapi.llm_utils import ModelLoader as FrontEndModelLoader +from tensorrt_llm.models.modeling_utils import QuantAlgo +from tensorrt_llm.sampling_params import SamplingParams + +pytestmark = pytest.mark.cpu_only + +_SM103 = (10, 3) +_RANKS = 4 +_MAX_SEQ_LEN = 1048576 +# max_num_tokens: the chunk, and the step budget context and decode tokens share. +_CHUNK = 32768 +_MAX_BATCH = 16 +_PAGE = 32 +# A token's KV on every rank: a 656-byte fp8_ds_mla latent row (512 e4m3, four fp32 tile scales, +# 64 bf16 rope) on each of the 63 KV owners, a 128-byte fp8 indexer key and its fp32 scale on each +# of the 44 full-indexer layers. +_LATENT_ROW_BYTES = 656 +_KV_OWNERS = 63 +_INDEXER_KEY_BYTES = 132 +_FULL_INDEXER_LAYERS = 44 +_TOKEN_BYTES = 47136 +_TOPK = 2048 +_Q_SPLIT_THRESHOLD = 8192 +# The pool each rank of a boot at this config on the fp8_ds_mla cache held, the native +# checkpoint's vision tower loaded, in tokens (66756 pages of 32): the budget handed to +# the estimate. +_BOOT_POOL_TOKENS = 2_136_192 +# The last chunk of a 1048576-token request reads a cached prefix of 1015808 tokens. +_LAST_CHUNK_PREFIX = _MAX_SEQ_LEN - _CHUNK +# The envelope gate's long prompt: max_seq_len less its 16 output tokens and one. +_ENVELOPE_PROMPT = _MAX_SEQ_LEN - 16 - 1 + +_TP4 = dict( + tensor_parallel_size=4, + moe_tensor_parallel_size=4, + moe_expert_parallel_size=1, + enable_attention_dp=False, +) +# The quickstart gates (boot, envelope, witness) state only --tp_size=4, with the MoE sizes as the +# -1 sentinels, and run the KV pool at the quickstart's --kv_cache_fraction default 0.9. +_TP4_QUICKSTART = dict( + tensor_parallel_size=4, + moe_expert_parallel_size=-1, + moe_tensor_parallel_size=-1, + moe_cluster_parallel_size=-1, + enable_attention_dp=False, +) + + +def _options( + topology: dict, max_seq_len: int, chunked: bool, fraction: float = 0.85, **kv_dtype +) -> dict: + """The LLM-API options of a Mistral Large 4 tp4 gate, the checkpoint in its Mistral-native format: + the 1M core unit at 1048576 with chunked prefill, the delivered 32K engine at 32768 without + it. The KV cache dtype is left at its default (auto) as the CI class leaves it, unless + ``kv_dtype`` names one.""" + return dict( + checkpoint_format="mistral", + **topology, + max_batch_size=_MAX_BATCH, + max_num_tokens=_CHUNK, + max_seq_len=max_seq_len, + enable_chunked_prefill=chunked, + cuda_graph_config=dict(enable_padding=True, max_batch_size=_MAX_BATCH), + kv_cache_config=dict( + enable_block_reuse=False, free_gpu_memory_fraction=fraction, **kv_dtype + ), + ) + + +_NVFP4_NAMES = ("Mistral-Large-4-1T-A52B-NVFP4", "mistral-large-4-nvfp4") + +# The GSM8K reference rows, and the key the harness reads them by. +_GSM8K_REFERENCES = ( + Path(__file__).resolve().parents[6] / "integration/defs/accuracy/references/gsm8k.yaml" +) +_MODEL_NAME = "mistralai/Mistral-Large-4-1T-A52B" + +_TP4_TARGET = "ModelingV2MistralLarge4Nvfp4Sm103Tp" + + +def _checkpoint_dir(names: tuple[str, ...]) -> Path: + """The first of ``names`` under LLM_MODELS_ROOT holding a Mistral-native params.json.""" + root = llm_models_root() + for name in names: + if root is not None and (root / name / "params.json").is_file(): + return root / name + pytest.skip(f"none of {names} is under LLM_MODELS_ROOT ({root})") + + +class _Routed: + """What the recording `route` returns: no target is built here.""" + + +def _route(pretrained_config, mapping): + """`route` for this config on ``mapping``, on sm_103 without speculative decoding.""" + model_config = SimpleNamespace( + pretrained_config=pretrained_config, mapping=mapping, spec_config=None + ) + return routing.route(model_config, None, _SM103) + + +def _resolve(checkpoint_dir: Path, options: dict) -> SimpleNamespace: + """The configuration a served engine reaches for this checkpoint, front end then rank.""" + model = str(checkpoint_dir) + llm_args = TorchLlmArgs(model=model, modeling_v2="require", **options) + prequantized = FrontEndModelLoader(llm_args)._update_from_hf_quant_config() + + routed = [] + + def route(model_config, llm_args, compute_capability): + pretrained_config = model_config.pretrained_config + routed.append( + SimpleNamespace( + pretrained_config=pretrained_config, + quantization_config=copy.deepcopy(pretrained_config.quantization_config), + mapping=model_config.mapping, + compute_capability=compute_capability, + spec_config=model_config.spec_config, + ) + ) + return _Routed + + with pytest.MonkeyPatch.context() as mp: + # The rank's config pass pins the CUDA memory fraction first; there is no device here. + mp.setattr(torch.cuda, "set_per_process_memory_fraction", lambda *args, **kwargs: None) + mp.setattr(_router_index, "_device_compute_capability", lambda: _SM103) + mp.setattr(_router_index, "registered_routes", lambda: {"MistralLarge4ForCausalLM": route}) + # One process stands in for every rank, so the rank exchange is the local resolution. + mp.setattr( + model_loader, "modeling_v2_resolve_on_all_ranks", _router_index.modeling_v2_resolve + ) + llm_args, checkpoint_loader = py_executor_creator._load_config_and_create_checkpoint_loader( + llm_args, model + ) + _, max_num_tokens, max_seq_len, max_batch_size = llm_args.get_runtime_sizes() + mapping = llm_args.parallel_config.to_mapping() + model_config = RankModelLoader( + llm_args=llm_args, + mapping=mapping, + spec_config=llm_args.speculative_config, + sparse_attention_config=llm_args.sparse_attention_config, + max_num_tokens=max_num_tokens, + max_seq_len=max_seq_len, + )._load_and_validate_config(model, checkpoint_loader) + return SimpleNamespace( + path=checkpoint_dir, + llm_args=llm_args, + prequantized=prequantized, + model_config=model_config, + mapping=mapping, + routed=routed, + max_num_tokens=max_num_tokens, + max_seq_len=max_seq_len, + max_batch_size=max_batch_size, + ) + + +@pytest.fixture(scope="module") +def tp4(): + """The 1M core unit, as the CI class builds it: the KV cache dtype at its default.""" + return _resolve(_checkpoint_dir(_NVFP4_NAMES), _options(_TP4, _MAX_SEQ_LEN, True)) + + +@pytest.fixture(scope="module") +def tp4_fp8(): + """The 1M core unit as the gates' options file states it: the KV cache dtype fp8.""" + return _resolve(_checkpoint_dir(_NVFP4_NAMES), _options(_TP4, _MAX_SEQ_LEN, True, dtype="fp8")) + + +@pytest.fixture(scope="module") +def tp4_quickstart(): + """The 1M config as the quickstart gates build it (``--kv_cache_dtype=fp8``).""" + return _resolve( + _checkpoint_dir(_NVFP4_NAMES), + _options(_TP4_QUICKSTART, _MAX_SEQ_LEN, True, fraction=0.9, dtype="fp8"), + ) + + +@pytest.fixture(scope="module") +def tp4_32k(): + """The delivered engine: 32768, unchunked.""" + return _resolve(_checkpoint_dir(_NVFP4_NAMES), _options(_TP4, _CHUNK, False)) + + +def _anchor(llm_args) -> float: + """The reference row `AccuracyTask.get_hypothesis_testing_params` selects for this engine.""" + quant = llm_args.quant_config + acc_specs = dict( + dtype=llm_args.dtype, + quant_algo=quant.quant_algo, + kv_cache_quant_algo=quant.kv_cache_quant_algo, + spec_dec_algo=None, + extra_acc_spec=None, + ) + for entry in yaml.safe_load(_GSM8K_REFERENCES.read_text())[_MODEL_NAME]: + if all( + entry.get(key, "auto" if key == "dtype" else None) == value + for key, value in acc_specs.items() + ): + return entry["accuracy"] + raise AssertionError(f"no {_MODEL_NAME} GSM8K reference row for {acc_specs}") + + +def _rank_mapping(mapping, rank: int): + rank_mapping = copy.deepcopy(mapping) + rank_mapping.rank = rank + return rank_mapping + + +def _pin_sm103(mp: pytest.MonkeyPatch) -> None: + """The engine reads the SM off the device in these modules; state it as sm_103.""" + for module in (_util, trtllm_backend, dsa_indexer, dsa_metadata): + mp.setattr(module, "get_sm_version", lambda: 103) + + +def _host_only(mp: pytest.MonkeyPatch) -> None: + """Keep the engine's host-side preparation on the host: no pinned staging buffers, and the + copies it hands to the device stay where they are.""" + for module in (dsa_indexer, dsa_metadata): + mp.setattr(module, "prefer_pinned", lambda: False) + mp.setattr(dsa_indexer, "maybe_pin_memory", lambda tensor: tensor) + mp.setattr(torch.Tensor, "cuda", lambda self, *args, **kwargs: self) + + +def _block_table_width(max_seq_len: int) -> int: + """``max_blocks_per_seq`` of ``KVCacheManagerV2``, the width of every block table the metadata + sizes by it: the pages of max_seq_len plus the generation headroom, padded to a multiple of 4.""" + pages = math.ceil((max_seq_len + kv_cache_manager_v2.BASE_GENERATION_TOKEN_COUNT) / _PAGE) + return (pages + 3) // 4 * 4 + + +# ------------------------------------------------------- four ranks in one process + + +class _TpGroup: + """The four ranks of the TP group in one process: rank r is a thread, and every exchange is a + barrier, so each rank's call sees what every rank sent, as over MPI.""" + + def __init__(self, size: int = _RANKS): + self.size = size + self._barrier = threading.Barrier(size, timeout=60) + self._slots = [None] * size + + def exchange(self, rank: int, obj): + self._slots[rank] = obj + self._barrier.wait() + gathered = list(self._slots) + self._barrier.wait() + return gathered + + def run(self, step): + """``step(rank)`` on every rank at once; the per-rank results in rank order.""" + results, errors = [None] * self.size, [None] * self.size + + def body(rank): + try: + results[rank] = step(rank) + except BaseException as error: # noqa: BLE001 - re-raised below, after every rank + errors[rank] = error + self._barrier.abort() + + threads = [threading.Thread(target=body, args=(rank,)) for rank in range(self.size)] + for thread in threads: + thread.start() + for thread in threads: + thread.join() + # The rank that failed first, not a peer it left waiting at the barrier. + raised = [error for error in errors if error is not None] + for error in raised: + if not isinstance(error, threading.BrokenBarrierError): + raise error + if raised: + raise raised[0] + return results + + +class _RankDist: + """One rank's communicator: the world gather the warmup agrees its shapes over.""" + + def __init__(self, group: _TpGroup, rank: int): + self.group = group + self.rank = self.tp_rank = rank + self.world_size = self.tp_size = group.size + + def allgather(self, obj): + return self.group.exchange(self.rank, obj) + + +class _KvPool: + """The KV manager surface the warmup's dummy requests read: ``tokens`` tokens in pages of 32, + and dummies recorded as the token counts they were asked for. The longest sequence it offers + follows ``KVCacheManagerV2.get_num_available_tokens``: what is left once batch_size - 1 + one-page sequences are set aside.""" + + tokens_per_block = _PAGE + num_extra_kv_tokens = 0 + + def __init__(self, tokens: int, max_seq_len: int): + self.tokens = tokens + self.max_seq_len = max_seq_len + + def get_num_available_tokens( + self, *, token_num_upper_bound, batch_size=1, max_num_draft_tokens=0, **_ + ): + pages = self.tokens // self.tokens_per_block - (batch_size - 1) + return max(0, min(token_num_upper_bound, pages * self.tokens_per_block)) + + def get_num_free_blocks(self) -> int: + return self.tokens // self.tokens_per_block + + def add_dummy_requests(self, request_ids, token_nums=None, is_gen=False, **kwargs): + if token_nums is None: + token_nums = [1 + kwargs.get("max_num_draft_tokens", 0)] * len(request_ids) + return [ + SimpleNamespace( + py_request_id=request_id, + token_num=token_num, + is_gen=is_gen, + is_last_context_chunk=True, + is_attention_dp_dummy=False, + py_draft_tokens=[], + py_needs_onehot_draft_probs=False, + ) + for request_id, token_num in zip(request_ids, token_nums) + ] + + def free_resources(self, request) -> None: + pass + + +class _Resources: + def __init__(self, kv: _KvPool): + self.kv = kv + + def get_resource_manager(self, key): + return self.kv if key == ResourceManagerType.KV_CACHE_MANAGER else None + + +def _runner_config(resolved) -> SimpleNamespace: + """The decoder runner's sizes for this engine. No speculative decoding: every draft width is + zero, and beams are one wide.""" + graphs = resolved.llm_args.cuda_graph_config + return SimpleNamespace( + max_num_tokens=resolved.max_num_tokens, + max_seq_len=resolved.max_seq_len, + max_batch_size=resolved.max_batch_size, + cuda_graph_batch_sizes=graphs.batch_sizes, + max_beam_width=1, + spec_config=None, + use_mrope=False, + max_draft_len=0, + original_max_draft_len=0, + max_total_draft_tokens=0, + max_draft_loop_tokens=0, + ) + + +def _dummy_factory(resolved) -> DummyRequestFactory: + return DummyRequestFactory( + config=_runner_config(resolved), + model=SimpleNamespace(model_config=resolved.model_config), + get_runtime_tokens_per_gen_step=lambda draft_len: 1 + draft_len, + release_request_state=None, + ) + + +# ------------------------------------------------------- the engine's attention metadata + + +class _RecordingBackend: + """An attention backend whose metadata is the keyword arguments it is built with.""" + + Metadata = SimpleNamespace + + +def _runtime_features(resolved) -> AttentionRuntimeFeatures: + """``AttentionRuntimeFeatures`` as ``create_py_executor`` builds them for this engine: no + speculative drafter, and the chunk is max_num_tokens.""" + llm_args = resolved.llm_args + return AttentionRuntimeFeatures( + chunked_prefill=llm_args.enable_chunked_prefill, + cache_reuse=llm_args.kv_cache_config.enable_block_reuse, + has_speculative_draft_tokens=False, + chunk_size=resolved.max_num_tokens, + ) + + +def _engine_metadata(resolved, rank: int) -> SimpleNamespace: + """What ``build_attention_metadata`` hands the attention backend's metadata on ``rank``.""" + return build_attention_metadata( + resolved.model_config, + max_batch_size=resolved.max_batch_size, + max_num_tokens=resolved.max_num_tokens, + max_beam_width=1, + attention_backend=_RecordingBackend, + attention_runtime_features=_runtime_features(resolved), + mapping=_rank_mapping(resolved.mapping, rank), + cache_indirection=None, + ) + + +def _host_metadata(resolved, rank: int, q_lens, cached, gen_kv=()) -> SimpleNamespace: + """One rank's DSA metadata for one step, host side: the engine's metadata arguments for the + rank, the step's requests, and what ``DSAtrtllmAttentionMetadata.prepare`` derives from them + before it plans the indexer (the context and generation indptrs, the indexer-skip decision). + ``q_lens`` and ``cached`` are the context rows' new and cached tokens; ``gen_kv`` the KV + lengths of the decode rows behind them.""" + engine = _engine_metadata(resolved, rank) + params = engine.sparse_metadata_params + num_contexts, num_generations = len(q_lens), len(gen_kv) + seq_lens = torch.tensor([*q_lens, *[1] * num_generations], dtype=torch.int32) + cached_tokens = [*cached, *(kv - 1 for kv in gen_kv)] + width = _block_table_width(resolved.max_seq_len) + + def indptr(): + return torch.zeros(engine.max_num_requests + 1, dtype=torch.int64) + + md = SimpleNamespace(**vars(engine)) + md.kv_cache_manager = SimpleNamespace( + index_head_dim=params.index_head_dim, + max_seq_len=resolved.max_seq_len, + tokens_per_block=_PAGE, + ) + md.indexer_max_chunk_size = params.indexer_max_chunk_size + md.num_sparse_topk = params.max_sparse_topk + md.enable_indexer_skip = params.enable_indexer_skip + md.indexer_head_dim = params.index_head_dim + md.indexer_quant_block_size = 128 + md._tokens_per_block = _PAGE + md.compress_ratios = [1] + md.num_contexts, md.num_generations = num_contexts, num_generations + md.num_seqs = num_contexts + num_generations + md.num_ctx_tokens = sum(q_lens) + md.num_tokens = md.num_ctx_tokens + num_generations + md.seq_lens = seq_lens + md.request_ids = list(range(md.num_seqs)) + md.kv_cache_params = SimpleNamespace( + num_cached_tokens_per_seq=cached_tokens, num_extra_kv_tokens=0 + ) + for name in ("ctx_cached_token", "ctx_kv", "gen_cached_token", "gen_kv"): + setattr(md, f"{name}_indptr", indptr()) + setattr(md, f"host_{name}_indptr", indptr()) + md.cu_seqlen_ks = torch.empty(0, dtype=torch.int32) + md.host_indexer_k_cache_block_offsets = torch.arange( + engine.max_num_sequences * width, dtype=torch.int32 + ).view(engine.max_num_sequences, width) + md.slot_mapping_fp8 = torch.empty(engine.max_num_tokens, dtype=torch.int64) + md.slot_mapping_scale = torch.empty(engine.max_num_tokens, dtype=torch.int64) + md.indexer_prefill_q_split = None + md.indexer_prefill_chunks = None + + md.prepare_dense_topk_indices = types.MethodType( + DSAtrtllmAttentionMetadata.prepare_dense_topk_indices, md + ) + + cached_lens = torch.tensor(cached_tokens, dtype=torch.int) + kv_lens = cached_lens + seq_lens + DSAtrtllmAttentionMetadata.prepare_for_mla_rope_append(md, cached_lens, kv_lens) + DSAtrtllmAttentionMetadata.prepare_for_skip_indexer(md, kv_lens) + return md + + +def _plan(resolved, rank: int, q_lens, cached, gen_kv=()) -> SimpleNamespace: + """The indexer's prefill plan for one step on ``rank``, by the engine's own planning.""" + md = _host_metadata(resolved, rank, q_lens, cached, gen_kv) + with pytest.MonkeyPatch.context() as mp: + _host_only(mp) + Indexer.prepare_for_chunked_prefill(md, Indexer.build_indexer_params(md)) + return md + + +def _target_ctx_kv_offsets(md, q_lens) -> dict: + """The target's context offsets, `_ctx_kv_offsets` and `_ctx_host_offsets`, on a step `_plan` + prepared, given the uncached prefix sums the engine's `TrtllmAttentionMetadata.prepare` fills + beside the DSA ones.""" + md.host_ctx_uncached_token_indptr = torch.tensor( + [0, *itertools.accumulate(q_lens)], dtype=torch.int64 + ) + return dict( + **tp4_modeling._ctx_kv_offsets(md, md.num_contexts), + **tp4_modeling._ctx_host_offsets(md, md.num_contexts), + ) + + +def _target_blocks(md, q_lens) -> list: + """The target's indexer scoring blocks (`_indexer_blocks`) for a step `_plan` prepared on this + engine, from the entries `PrefillTarget.step_args` hands it.""" + # The engine plans chunks under chunked prefill, so the whole-step fallback's entries + # (num_ctx_kv_tokens, the step's windows) are never read here. + assert md.indexer_prefill_chunks is not None + step = dict( + num_ctx_tokens=md.num_ctx_tokens, + max_seq_len=md.kv_cache_manager.max_seq_len, + **_target_ctx_kv_offsets(md, q_lens), + ) + return tp4_modeling._indexer_blocks(md.indexer_prefill_chunks, step) + + +def _target_sizes(resolved) -> SimpleNamespace: + """What the target sizes its buffers from on the engine's attention metadata + (`_derive_first_step`): max_num_requests, max_num_tokens and runtime_features as + ``build_attention_metadata`` hands them over (the features as ``create_py_executor`` builds + them, `_runtime_features`: chunked prefill from enable_chunked_prefill, cache reuse from the + KvCacheConfig's enable_block_reuse), and max_seq_len, which ``TrtllmAttentionMetadata`` reads + off the KV cache manager the engine builds for it.""" + md = _engine_metadata(resolved, 0) + return SimpleNamespace( + max_num_requests=md.max_num_requests, + max_num_tokens=md.max_num_tokens, + max_seq_len=resolved.max_seq_len, + runtime_features=md.runtime_features, + ) + + +def _spans(md) -> list[tuple[int, int, int, int]]: + """Each planned chunk's query rows [token_start, token_end) and keys [k_start, k_end).""" + return [ + (chunk.token_start, chunk.token_end, chunk.k_token_start, chunk.k_token_end) + for chunk in md.indexer_prefill_chunks + ] + + +# ------------------------------------------------------- (1) topology, limits, quantization + + +def test_tp4_mapping_as_at_base(tp4, tp4_quickstart, tp4_32k): + """The tp4 mapping exactly as the delivered 32K engine has it, in both spellings: attention TP + 4 with no attention DP, the routed experts tensor parallel four ways and not expert-parallel, + one pipeline stage, rank r MoE TP rank r and MoE EP rank 0. The router is handed it, and the + family's routing claims the configuration with the tp target: max_seq_len is no routing + criterion.""" + llm_args, mapping = tp4.llm_args, tp4.mapping + assert ( + llm_args.tensor_parallel_size, + llm_args.moe_tensor_parallel_size, + llm_args.moe_expert_parallel_size, + llm_args.enable_attention_dp, + ) == (4, 4, 1, False) + assert (mapping.world_size, mapping.tp_size, mapping.pp_size, mapping.cp_size) == (4, 4, 1, 1) + assert (mapping.moe_tp_size, mapping.moe_ep_size, mapping.moe_cluster_size) == (4, 1, 1) + assert mapping.enable_attention_dp is False and mapping.dp_size == 1 + assert mapping.attn_tp_size == 4 + assert mapping.to_dict() == tp4_32k.mapping.to_dict() + assert tp4_quickstart.mapping.to_dict() == mapping.to_dict() + for rank in range(_RANKS): + rank_mapping = _rank_mapping(mapping, rank) + assert ( + rank_mapping.tp_rank, + rank_mapping.moe_tp_rank, + rank_mapping.moe_ep_rank, + rank_mapping.moe_cluster_rank, + ) == (rank, rank, 0, 0) + assert _rank_mapping(tp4_quickstart.mapping, rank) == rank_mapping + assert _rank_mapping(tp4_32k.mapping, rank) == rank_mapping + + for resolved in (tp4, tp4_quickstart): + seen = resolved.routed[0] + assert seen.mapping is resolved.model_config.mapping + assert seen.mapping.to_dict() == mapping.to_dict() + assert seen.compute_capability == _SM103 and seen.spec_config is None + target = _route(seen.pretrained_config, seen.mapping) + assert target.__name__ == _TP4_TARGET + + +def test_limits_reach_the_engine_unclamped(tp4, tp4_quickstart, monkeypatch): + """max_seq_len 1048576 is the checkpoint's max_position_embeddings, and nothing on the way + from the LLM API to the executor lowers it: the LLM args and the rank's ModelConfig carry it; + ``ModelEngine`` keeps it, since the causal-LM shell's ``infer_max_seq_len`` reads the + same 1048576 off this config; and ``create_py_executor`` keeps chunked context on for MLA on + sm_103. 32768 tokens and 16 requests per step, chunked prefill on, block reuse off, in both + spellings.""" + cfg = tp4.model_config.pretrained_config + assert cfg.max_position_embeddings == _MAX_SEQ_LEN + for resolved in (tp4, tp4_quickstart): + llm_args = resolved.llm_args + assert (llm_args.max_seq_len, llm_args.max_num_tokens, llm_args.max_batch_size) == ( + _MAX_SEQ_LEN, + _CHUNK, + _MAX_BATCH, + ) + assert llm_args.enable_chunked_prefill is True + assert llm_args.kv_cache_config.enable_block_reuse is False + assert (resolved.max_num_tokens, resolved.max_seq_len, resolved.max_batch_size) == ( + _CHUNK, + _MAX_SEQ_LEN, + _MAX_BATCH, + ) + model_config = resolved.model_config + assert (model_config.max_num_tokens, model_config.max_seq_len) == (_CHUNK, _MAX_SEQ_LEN) + assert tp4.llm_args.kv_cache_config.free_gpu_memory_fraction == 0.85 + assert tp4_quickstart.llm_args.kv_cache_config.free_gpu_memory_fraction == 0.9 + + monkeypatch.delenv("TLLM_ALLOW_LONG_MAX_MODEL_LEN", raising=False) + shell = SimpleNamespace(config=cfg) + assert DecoderModelForCausalLM.infer_max_seq_len(shell) == _MAX_SEQ_LEN + engine = SimpleNamespace( + model=SimpleNamespace( + infer_max_seq_len=lambda: DecoderModelForCausalLM.infer_max_seq_len(shell) + ), + max_seq_len=tp4.max_seq_len, + max_num_tokens=tp4.max_num_tokens, + batch_size=tp4.max_batch_size, + ) + ModelEngine._init_max_seq_len(engine) + ModelEngine._init_max_num_tokens(engine) + assert (engine.max_seq_len, engine.max_num_tokens) == (_MAX_SEQ_LEN, _CHUNK) + + assert 103 in py_executor_creator._MLA_CHUNKED_PREFILL_SUPPORTED_SM_VERSIONS + + +def test_llm_api_admits_a_1m_prompt(tp4, tp4_32k): + """``BaseLLM._check_arguments`` refuses a prompt longer than max_num_tokens only with chunked + prefill off: the envelope gate's 1048559-token prompt passes it at 1M, where the delivered 32K + engine refuses 32769 tokens.""" + llm_module = importlib.import_module("tensorrt_llm.llmapi.llm") + + def check(resolved, prompt_len): + llm = SimpleNamespace( + args=resolved.llm_args, _check_one_model_speculative_sampling=lambda params: None + ) + llm_module.BaseLLM._check_arguments( + llm, prompt_len, SamplingParams(max_tokens=16), is_gen_only=False + ) + + check(tp4, _ENVELOPE_PROMPT) + check(tp4_32k, _CHUNK) + with pytest.raises(llm_module.RequestError, match="should not exceed max_num_tokens"): + check(tp4_32k, _CHUNK + 1) + + +def test_quantization_is_the_anchor_key(tp4): + """At 1M as at 32K: MIXED_PRECISION with the fp8 KV cache (the checkpoint's fp8_kv_cache under + the default dtype), the GSM8K anchor row's key, and the router handed params.json's + quantization_config with its 475 quantized_layers intact plus the FP8 KV-cache algorithm the + native loader records from fp8_kv_cache.""" + assert tp4.prequantized is True + for quant in (tp4.llm_args.quant_config, tp4.model_config.quant_config): + assert quant.quant_algo == QuantAlgo.MIXED_PRECISION + assert quant.kv_cache_quant_algo == QuantAlgo.FP8 + assert tp4.llm_args.kv_cache_config.dtype == "auto" + assert _anchor(tp4.llm_args) == 91.926 + + on_disk = json.loads((tp4.path / "params.json").read_text())["quantization_config"] + assert len(on_disk["quantized_layers"]) == 475 + assert len(tp4.routed) == 1 + seen = tp4.routed[0] + assert seen.quantization_config == dict(on_disk, kv_cache_quant_algo="FP8") + assert tp4.model_config.modeling_v2_target is _Routed + + +def test_kv_cache_is_fp8_ds_mla(tp4, tp4_fp8, tp4_quickstart): + """Every spelling builds the fp8_ds_mla latent cache: the CI class's default dtype (auto, the + checkpoint's fp8_kv_cache), the gates' options file's fp8 and the quickstart gates' fp8 each + select + ``MistralLarge4CacheManager`` (the sparse registry picks it by the MistralLarge4Config), and + the engine hands it the FP8 element type. Per token on every rank: a 656-byte row on each of + the 63 KV owners and the 132-byte indexer key on each of the 44 full-indexer layers, 47136 B, + in pages of 32. The target refuses any other latent layout on its first step.""" + cfg = tp4.model_config.pretrained_config + owners = sum(1 for i, s in enumerate(cfg.kv_source_layers) if s == i) + assert (owners, cfg.index_topk_pattern.count("F")) == (_KV_OWNERS, _FULL_INDEXER_LAYERS) + assert _TOKEN_BYTES == ( + _KV_OWNERS * _LATENT_ROW_BYTES + _FULL_INDEXER_LAYERS * _INDEXER_KEY_BYTES + ) + for resolved, dtype in ((tp4, "auto"), (tp4_fp8, "fp8"), (tp4_quickstart, "fp8")): + kv_cache_config = resolved.llm_args.kv_cache_config + assert kv_cache_config.dtype == dtype + assert kv_cache_config.tokens_per_block == _PAGE + assert resolved.model_config.quant_config.quant_mode.has_fp8_kv_cache() + manager = _util.get_kv_cache_manager_cls(resolved.model_config, kv_cache_config) + assert manager is MistralLarge4CacheManager + assert manager.latent_layout == "fp8_ds_mla" + assert manager.latent_token_bytes == _LATENT_ROW_BYTES + per_token = manager.get_cache_size_per_token(resolved.model_config, resolved.mapping) + assert per_token == _TOKEN_BYTES + + +def test_dsa_sparse_config(tp4, tp4_32k, monkeypatch): + """The DSA config the metadata is built from: the checkpoint's index_topk 2048, q_split_threshold + at its default 8192 (nothing sets -1), no indexer_max_chunk_size (32768 in the metadata), the + short-sequence indexer skip on, the shared-indexer layers seen, and neither the CuTe DSL top-k + nor the heuristic top-k. The two logits caps the prefill planning reads are their defaults: + 2^31 elements per logits call, 2^28 packed per chunk. The target tiles its own indexer prefill + logits under `_prefill_logits_budget` of this engine: the ceiling, 2^28 (256 rows at 1048576 + keys), where the delivered 32K engine's is 2^24 (512 rows at 32768 keys).""" + cfg = tp4.model_config.pretrained_config + sparse = tp4.model_config.sparse_attention_config + assert sparse.algorithm == "dsa" + assert sparse.q_split_threshold == _Q_SPLIT_THRESHOLD + assert sparse.indexer_max_chunk_size is None + assert sparse.skip_indexer_for_short_seqs is True + assert sparse.use_cute_dsl_topk is False and sparse.enable_heuristic_topk is False + assert cfg.index_topk == _TOPK + + params = sparse.to_sparse_metadata_params(pretrained_config=cfg) + assert params.max_sparse_topk == _TOPK + assert params.q_split_threshold == _Q_SPLIT_THRESHOLD + assert params.indexer_max_chunk_size == 32768 + assert params.index_head_dim == 128 + assert params.enable_indexer_skip is True + assert params.has_shared_indexer_layers is True + + user = tp4.llm_args.sparse_attention_config + assert user is None or user.q_split_threshold == _Q_SPLIT_THRESHOLD + + assert dsa_indexer._INDEXER_MQA_LOGITS_ELEM_BUDGET == 1 << 31 + assert dsa_indexer._INDEXER_PACKED_LOGITS_ELEM_BUDGET == 1 << 28 + + monkeypatch.delenv("TLLM_INDEXER_MQA_LOGITS_ELEM_BUDGET", raising=False) + budget = tp4_modeling._prefill_logits_budget(tp4.max_num_tokens, tp4.max_seq_len) + assert budget == tp4_modeling._PREFILL_LOGITS_CEILING == 1 << 28 + assert budget // _MAX_SEQ_LEN == 256 + assert tp4_modeling._prefill_logits_budget(tp4_32k.max_num_tokens, tp4_32k.max_seq_len) == ( + 1 << 24 + ) + + +def test_context_mla_over_cached_kv_turns_on(tp4, tp4_32k): + """``create_py_executor`` builds the attention runtime features with chunked_prefill on and the + chunk max_num_tokens, and ``build_attention_metadata`` turns on context MLA over cached KV for + an MLA model with chunked prefill: every chunk after a request's first is a context request with + a cached prefix. The delivered 32K engine (no chunking, no reuse) has it off.""" + for rank in range(_RANKS): + md = _engine_metadata(tp4, rank) + assert md.enable_context_mla_with_cached_kv is True + features = md.runtime_features + assert (features.chunked_prefill, features.cache_reuse, features.chunk_size) == ( + True, + False, + _CHUNK, + ) + assert features.has_speculative_draft_tokens is False + assert (md.max_num_requests, md.max_num_tokens, md.max_num_sequences) == ( + _MAX_BATCH, + _CHUNK, + _MAX_BATCH, + ) + assert md.mapping.tp_rank == rank and md.mapping.enable_attention_dp is False + assert md.sparse_metadata_params.q_split_threshold == _Q_SPLIT_THRESHOLD + assert md.enable_flash_mla is False + assert _engine_metadata(tp4_32k, rank).enable_context_mla_with_cached_kv is False + + +# ------------------------------------------------------- (2) the query split across the TP ranks + +# Each step size is planned with the context rows composed four ways. +_SPLIT_STEPS = (8191, 8192, 32768, 32769) +# Per-rank query counts of a split step, in rank order; None below the threshold. +_SPLIT_SIZES = { + 8191: None, + 8192: [2048] * 4, + 32753: [8188, 8188, 8188, 8189], + 32768: [8192] * 4, + 32769: [8192, 8192, 8192, 8193], +} +# Decode rows behind the context rows, by KV length: one at max_seq_len, the rest spread. +_DECODE_KV = ( + 1048576, + 1048575, + 524288, + 131073, + 65536, + 32769, + 32768, + 8192, + 2049, + 2048, + 2047, + 1024, + 33, + 32, + 2, +) + + +def _compositions(tc: int) -> dict: + """The context rows of a ``tc``-token step: (new tokens, cached prefixes, decode KV lengths). + One fresh prompt; one chunk over the longest cached prefix a 1048576-token request leaves it + (1015808 below 32769 tokens); four requests over different prefixes, cut by the rank + boundaries; one chunk with 15 decode rows behind it.""" + last = min(_LAST_CHUNK_PREFIX, _MAX_SEQ_LEN - tc) + a, b, c = tc * 3 // 8, tc // 8, tc // 4 + return { + "fresh": ([tc], [0], ()), + "last_chunk": ([tc], [last], ()), + "prefixes": ([a, b, c, tc - a - b - c], [131072, 0, 524288, 917504], ()), + "behind_decode": ([tc], [262144], _DECODE_KV), + } + + +def _check_plan(md, q_lens, cached, rank: int) -> tuple[int, int]: + """One rank's plan against the rules it must keep; returns the query rows it covers.""" + tc = sum(q_lens) + sizes = _SPLIT_SIZES[tc] + if sizes is None: + assert md.indexer_prefill_q_split is None + begin, end = 0, tc + else: + begin = sum(sizes[:rank]) + end = begin + sizes[rank] + assert md.indexer_prefill_q_split == (begin, end, sizes) + + chunks = md.indexer_prefill_chunks + assert chunks, "a context step always plans its indexer chunks under chunked MLA prefill" + assert chunks[0].token_start == begin and chunks[-1].token_end == end + assert all(a.token_end == b.token_start for a, b in zip(chunks, chunks[1:])) + + q_starts = [sum(q_lens[:i]) for i in range(len(q_lens))] + kv_lens = [c + q for c, q in zip(cached, q_lens)] + for chunk in chunks: + rows = chunk.token_end - chunk.token_start + assert rows > 0 + assert chunk.cu_seqlen_ks.numel() == chunk.cu_seqlen_ke.numel() == rows + first = max(i for i, start in enumerate(q_starts) if start <= chunk.token_start) + last = max(i for i, start in enumerate(q_starts) if start < chunk.token_end) + assert chunk.k_token_start == sum(kv_lens[:first]) + if first == last: + # One request, whole or cut: its queries see its cached prefix and its own rows up to + # the chunk's last one. + seen = cached[first] + chunk.token_end - q_starts[first] + assert chunk.k_token_end - chunk.k_token_start == seen + assert int(chunk.cu_seqlen_ks.max()) == 0 + else: + # Whole requests packed together: the keys of each in turn. + assert chunk.token_start == q_starts[first] + assert chunk.token_end == q_starts[last] + q_lens[last] + assert chunk.k_token_end - chunk.k_token_start == sum(kv_lens[first : last + 1]) + assert int(chunk.cu_seqlen_ke[-1]) == chunk.k_token_end - chunk.k_token_start + assert bool((chunk.cu_seqlen_ke > chunk.cu_seqlen_ks).all()) + return begin, end + + +@pytest.mark.parametrize("tc", _SPLIT_STEPS) +def test_query_split_tiles_the_step(tp4, tc): + """The engine's own planning on each rank's host metadata, at 8191, 8192, 32768 and 32769 + context query tokens, each composed as one fresh prompt, one chunk over a cached prefix of + 1015808 (the last chunk of a 1048576-token request; 1015807 at 32769 tokens), four requests + over different cached prefixes and one chunk with 15 decode rows behind it. Below 8192 tokens + there is no split: every rank plans every row. From 8192 on each rank gets + [tc r / 4, tc (r + 1) / 4) and the four ranges tile [0, tc) exactly once, in rank order; each + rank's chunks tile its range in order, a request cut by a rank boundary is a chunk of its own + over its cached prefix and its rows up to the chunk's end, and whole requests are grouped by + the packed-logits budget. Decode rows neither move the split nor the chunks. A step of 32769 + context tokens exceeds max_num_tokens 32768 and is never scheduled; it is planned here for the + uneven split's sake (32753 tokens behind 15 decode rows is the reachable one).""" + for name, (q_lens, cached, gen_kv) in _compositions(tc).items(): + covered = [] + for rank in range(_RANKS): + md = _plan(tp4, rank, q_lens, cached, gen_kv) + covered.append(_check_plan(md, q_lens, cached, rank)) + if gen_kv: + alone = _plan(tp4, rank, q_lens, cached) + assert md.indexer_prefill_q_split == alone.indexer_prefill_q_split + assert _spans(md) == _spans(alone) + if _SPLIT_SIZES[tc] is None: + assert covered == [(0, tc)] * _RANKS, name + else: + assert covered[0][0] == 0 and covered[-1][1] == tc, name + assert all(a[1] == b[0] for a, b in zip(covered, covered[1:])), name + + +def test_query_split_contract(tp4, tp4_32k): + """The plans the target is checked against, exactly. ``indexer_prefill_q_split`` is + (q_begin, q_end, q_sizes) and each chunk (token_start, token_end, k_token_start, k_token_end): + + * one fresh 32768-token prompt: rank r scores rows [8192 r, 8192 r + 8192) against keys + [0, 8192 r + 8192); + * the last chunk of a 1048576-token request (32768 rows over 1015808 cached): rank r scores + rows [8192 r, 8192 r + 8192) against keys [0, 1015808 + 8192 r + 8192), the first row's + window ending at 1015808 + 8192 r + 1; + * 32769 rows over 1015807 cached: sizes 8192, 8192, 8192, 8193; + * 32753 rows behind 15 decode rows (a full 32768-token step): sizes 8188, 8188, 8188, 8189; + * four requests (12288, 4096, 8192, 8192 rows over 131072, 0, 524288, 917504 cached): rank 0 + the first request's rows [0, 8192), rank 1 its rows [8192, 12288) and the second request + whole, ranks 2 and 3 the third and the fourth whole; + * 8191 rows over 1015808 cached, below the threshold: no split, one chunk of every row on + every rank; + * and the delivered 32K engine, with no context MLA over cached KV: no split and no chunks at + 32768 rows. + """ + for rank in range(_RANKS): + lo, hi = 8192 * rank, 8192 * (rank + 1) + + md = _plan(tp4, rank, [_CHUNK], [0]) + assert md.indexer_prefill_q_split == (lo, hi, [8192] * 4) + assert _spans(md) == [(lo, hi, 0, hi)] + + md = _plan(tp4, rank, [_CHUNK], [_LAST_CHUNK_PREFIX]) + assert md.indexer_prefill_q_split == (lo, hi, [8192] * 4) + assert _spans(md) == [(lo, hi, 0, _LAST_CHUNK_PREFIX + hi)] + (chunk,) = md.indexer_prefill_chunks + assert int(chunk.cu_seqlen_ke[0]) == _LAST_CHUNK_PREFIX + lo + 1 + assert int(chunk.cu_seqlen_ke[-1]) == _LAST_CHUNK_PREFIX + hi + + md = _plan(tp4, rank, [_CHUNK + 1], [_LAST_CHUNK_PREFIX - 1]) + sizes = [8192, 8192, 8192, 8193] + assert md.indexer_prefill_q_split == (lo, lo + sizes[rank], sizes) + assert _spans(md) == [(lo, lo + sizes[rank], 0, _LAST_CHUNK_PREFIX - 1 + lo + sizes[rank])] + + md = _plan(tp4, rank, [_CHUNK - 15], [0], _DECODE_KV) + sizes = [8188, 8188, 8188, 8189] + begin = 8188 * rank + assert md.num_ctx_tokens + md.num_generations == _CHUNK + assert md.indexer_prefill_q_split == (begin, begin + sizes[rank], sizes) + assert _spans(md) == [(begin, begin + sizes[rank], 0, begin + sizes[rank])] + + md = _plan(tp4, rank, [8191], [_LAST_CHUNK_PREFIX]) + assert md.indexer_prefill_q_split is None + assert _spans(md) == [(0, 8191, 0, _LAST_CHUNK_PREFIX + 8191)] + + md = _plan(tp4_32k, rank, [_CHUNK], [0]) + assert md.indexer_prefill_q_split is None and md.indexer_prefill_chunks is None + + q_lens, cached = [12288, 4096, 8192, 8192], [131072, 0, 524288, 917504] + plans = [_spans(_plan(tp4, rank, q_lens, cached)) for rank in range(_RANKS)] + assert plans == [ + [(0, 8192, 0, 139264)], + [(8192, 12288, 0, 143360), (12288, 16384, 143360, 147456)], + [(16384, 24576, 147456, 679936)], + [(24576, 32768, 679936, 1605632)], + ] + + +def test_query_split_follows_the_attention_mapping(tp4): + """The split reads the attention mapping alone: tp_size 4 and tp_rank, attention DP off. The + MoE split never reaches it, and with attention DP on (one rank's own requests) nothing is + split.""" + for rank in range(_RANKS): + mapping = _rank_mapping(tp4.mapping, rank) + bounds = dsa_indexer._split_prefill_queries(_CHUNK, mapping, _Q_SPLIT_THRESHOLD) + assert bounds == (8192 * rank, 8192 * (rank + 1), [8192] * 4) + adp = SimpleNamespace(enable_attention_dp=True, tp_size=4, tp_rank=rank) + assert dsa_indexer._split_prefill_queries(_CHUNK, adp, _Q_SPLIT_THRESHOLD) == ( + 0, + _CHUNK, + None, + ) + assert dsa_indexer._split_prefill_queries(_CHUNK, tp4.mapping, -1) == (0, _CHUNK, None) + + +# The steps whose indexer chunk packs past max_seq_len keys: (new, cached) tokens per context +# request. Two 5-token last chunks over 983040-token prefixes (10 context rows, no split: one +# chunk of 1966090 keys); a 50-token last chunk over a 1048480-token prefix beside a fresh +# 200-token prompt (one chunk of 1048730 keys). +_PACKED_STEPS = { + "two-tails-over-983040": ([5, 5], [983040, 983040]), + "tail-beside-a-prompt": ([50, 200], [1048480, 0]), +} + + +@pytest.mark.parametrize("name", list(_PACKED_STEPS)) +def test_packed_chunks_are_scored_request_by_request(tp4, name): + """`plan_mla_chunked_prefill_chunks` packs consecutive whole context requests into one indexer + chunk while sum(q) x sum(kv) stays within 1 << 28, and the chunk's keys are every packed + request's whole KV: past max_seq_len keys here, wider than the key scorer's and the prefill + top-k's cells. `_indexer_blocks` cuts it at its request boundaries: each request's rows + against its own keys only, its windows re-based to its first key, every block within + max_seq_len keys and every window inside its block.""" + q_lens, cached = _PACKED_STEPS[name] + md = _plan(tp4, 0, q_lens, cached) + kv = [c + q for c, q in zip(cached, q_lens)] + q_at = [0, *itertools.accumulate(q_lens)] + kv_at = [0, *itertools.accumulate(kv)] + assert md.indexer_prefill_q_split is None + assert _spans(md) == [(0, q_at[-1], 0, kv_at[-1])] + assert kv_at[-1] > _MAX_SEQ_LEN and q_at[-1] * kv_at[-1] <= 1 << 28 + (chunk,) = md.indexer_prefill_chunks + blocks = _target_blocks(md, q_lens) + assert [(b.token_start, b.token_end, b.k_token_start, b.k_token_end) for b in blocks] == [ + (q_at[r], q_at[r + 1], kv_at[r], kv_at[r + 1]) for r in range(len(q_lens)) + ] + for r, block in enumerate(blocks): + rows = slice(q_at[r], q_at[r + 1]) + n_k = block.k_token_end - block.k_token_start + assert n_k == kv[r] <= _MAX_SEQ_LEN + assert torch.equal(block.cu_seqlen_ks, chunk.cu_seqlen_ks[rows] - kv_at[r]) + assert torch.equal(block.cu_seqlen_ke, chunk.cu_seqlen_ke[rows] - kv_at[r]) + assert int(block.cu_seqlen_ks.min()) >= 0 and int(block.cu_seqlen_ke.max()) <= n_k + + +def test_chunks_within_max_seq_len_stay_whole_under_the_split(tp4): + """A chunk of at most max_seq_len keys is one scoring block, unchanged, on every rank of the + query split: the envelope gate's prompt's last chunk over its 1015808-token prefix (8192 rows a + rank); and the two 983040-prefix tails beside a fresh 8182-token prompt, 8192 context rows + split 2048 a rank, where rank 0's share packs both tails into a 1966090-key chunk -- cut at + the tails -- and then the fresh prompt's first 2038 rows, a block of their own like every + other rank's slice of it.""" + for rank in range(_RANKS): + md = _plan(tp4, rank, [_CHUNK], [_LAST_CHUNK_PREFIX]) + assert md.indexer_prefill_q_split == (8192 * rank, 8192 * (rank + 1), [8192] * _RANKS) + blocks = _target_blocks(md, [_CHUNK]) + assert len(blocks) == len(md.indexer_prefill_chunks) + assert all(b is c for b, c in zip(blocks, md.indexer_prefill_chunks)) + + q_lens, cached = [5, 5, 8182], [983040, 983040, 0] + for rank in range(_RANKS): + md = _plan(tp4, rank, q_lens, cached) + q_begin, q_end = 2048 * rank, 2048 * (rank + 1) + assert md.indexer_prefill_q_split == (q_begin, q_end, [2048] * _RANKS) + blocks = _target_blocks(md, q_lens) + spans = [(b.token_start, b.token_end, b.k_token_start, b.k_token_end) for b in blocks] + if rank == 0: + assert _spans(md) == [(0, 10, 0, 1966090), (10, 2048, 1966090, 1968128)] + assert spans == [(0, 5, 0, 983045), (5, 10, 983045, 1966090), *_spans(md)[1:]] + assert blocks[2] is md.indexer_prefill_chunks[1] + else: + assert spans == _spans(md) == [(q_begin, q_end, 1966090, 1966090 + q_end - 10)] + assert all(b.k_token_end - b.k_token_start <= _MAX_SEQ_LEN for b in blocks) + + +# ------------------------------------------------------- (3) KV estimate, warmup + + +def _model_engine(resolved) -> SimpleNamespace: + """The fields of ``ModelEngine`` the KV-cache creator reads, for this engine.""" + graphs = resolved.llm_args.cuda_graph_config + return SimpleNamespace( + model=SimpleNamespace(model_config=resolved.model_config), + attn_runtime_features=_runtime_features(resolved), + batch_size=resolved.max_batch_size, + max_cuda_graph_batch_size=max(graphs.batch_sizes), + max_seq_len=resolved.max_seq_len, + mm_encoder_cache=None, + sparse_attention_config=resolved.llm_args.sparse_attention_config, + ) + + +def _kv_creator(resolved) -> KvCacheCreator: + """``KvCacheCreator`` as ``create_py_executor`` builds it for this engine (no speculation, one + beam, no connector, not disaggregated).""" + llm_args = resolved.llm_args + engine = _model_engine(resolved) + kv_cache_config = copy.deepcopy(llm_args.kv_cache_config) + return KvCacheCreator( + model_engine=engine, + mapping=resolved.mapping, + net_max_seq_len=resolved.max_seq_len, + kv_connector_manager=None, + max_num_tokens=resolved.max_num_tokens, + max_beam_width=1, + tokens_per_block=kv_cache_config.tokens_per_block, + max_seq_len=resolved.max_seq_len, + max_batch_size=resolved.max_batch_size, + kv_cache_config=kv_cache_config, + llm_args=llm_args, + speculative_config=None, + sparse_attention_config=engine.sparse_attention_config, + profiling_stage_data=None, + is_disagg=False, + ) + + +def test_kv_estimation_runs_one_full_step(tp4, tp4_fp8, tp4_quickstart, tp4_32k): + """The estimation pass sends one fresh 32768-token prompt at 1M (the delivered 32K engine: a + 32767-token and a 1-token prompt), with no chunk-aware MLA profile, which is skipped for any + sparse algorithm but skip_softmax. Its throwaway KV pool (``MistralLarge4CacheManager``, a + KVCacheManagerV2) is sized for the CUDA-graph warmup's longest decode row: + ceil((1048576 + 1) / 32) + 15 pages, 1049088 tokens, where 32K needed 33280. The same in + every KV dtype spelling.""" + + def estimation(resolved): + creator = _kv_creator(resolved) + assert creator._kv_cache_manager_cls is MistralLarge4CacheManager + tokens = creator._get_token_num_for_estimation() + assert creator._mla_chunked_profile_length is None + assert creator._get_mla_chunked_profile_length(resolved.max_seq_len - 1) is None + return tokens, [len(request.input_token_ids) for request in creator._dummy_reqs] + + tokens, prompts = estimation(tp4) + assert prompts == [_CHUNK] + assert tokens == (math.ceil((_MAX_SEQ_LEN + 1) / _PAGE) + _MAX_BATCH - 1) * _PAGE == 1_049_088 + assert estimation(tp4_fp8) == estimation(tp4_quickstart) == (tokens, prompts) + assert estimation(tp4_32k) == (33_280, [_CHUNK - 1, 1]) + + +def test_kv_capacity_reserves_nothing_and_caps_nothing(tp4): + """The estimate, handed the budget a boot at this config on the fp8_ds_mla cache measured + (2136192 tokens of 47136 B per rank, 93.78 GiB), on sm_103: + + * the DeepSeek-V4 indexer-logits reserve applies to algorithm deepseek_v4 only, and this is + dsa (it would set aside 8 GiB: min(2^31, 32768 x 1048576) fp32 logits); + * the context-MLA workspace: the backend declares w = 0 B/token, since DSA on sm_103 runs + sparse MLA and the engine stages nothing for it (the dense fp8 path would stage 16 heads x + (64 + 64 + 128) = 4096 B/token); chunked prefill bounds it; with w = 0 there is no + kv_len_cap, no reserve and no attended-KV admission cap. The target's own context + workspace (bf16 [2**19, 576], 576 MiB, `_ctx_workspace_rows` of this engine) is not the + engine's to know of and needs no reserve: the target allocates it on its first context + step, the estimation's own, and holds it, so the measured peak includes it + (test_context_rows_path_per_step); + * so the KV budget leaves the estimate unchanged and the pool is 2136192 tokens per rank, + one 1048576-token request with a chunk (two before every rank held the vision tower); + the scheduler, with no cap, keeps a second context request beside the last chunk of a + 1M one. + + Were a staging rate declared here (the dense path's 4096 B/token), chunked prefill would not + exempt it on sm_103 without a chunk-aware profile, and the cap it installed, + budget / (47136 + 4096) = 1965403 tokens, would still admit a 1M request and a chunk.""" + creator = _kv_creator(tp4) + model_config, mapping = tp4.model_config, tp4.mapping + kv_cache_config = creator._kv_cache_config + fraction = kv_cache_config.free_gpu_memory_fraction + budget = _BOOT_POOL_TOKENS * _TOKEN_BYTES + free = math.ceil(budget / fraction) + total = free + 160 * 2**30 + + with pytest.MonkeyPatch.context() as mp: + _pin_sm103(mp) + mp.setattr(torch.cuda, "empty_cache", lambda: None) + mp.setattr(torch.cuda, "reset_peak_memory_stats", lambda *args, **kwargs: None) + mp.setattr(torch.cuda, "mem_get_info", lambda *args, **kwargs: (free, total)) + mp.setattr( + torch.cuda, + "memory_stats", + lambda *args, **kwargs: {"allocated_bytes.all.current": total - free}, + ) + w = _util.get_attention_workspace_bytes_per_token(model_config, mapping) + bounded = _util.get_attention_workspace_is_chunked_prefill_bounded(model_config) + assert creator._get_kv_size_per_token().slope == _TOKEN_BYTES + creator.configure_kv_cache_capacity(None) + + assert model_config.sparse_attention_config.algorithm == "dsa" + assert getattr(creator._sparse_attention_config, "algorithm", None) != "deepseek_v4" + assert get_indexer_mqa_logits_workspace_bytes(_CHUNK, _MAX_SEQ_LEN) == 8 * 2**30 + + staged = dict( + num_attn_heads=model_config.pretrained_config.num_attention_heads // mapping.tp_size, + qk_rope_head_dim=64, + qk_nope_head_dim=64, + v_head_dim=128, + fp8_context_mla=True, + separate_q_and_kv_input=True, + ) + assert thop.get_context_mla_workspace_bytes_per_token(**staged, sparse_mla=False) == 4096 + assert thop.get_context_mla_workspace_bytes_per_token(**staged, sparse_mla=True) == 0 + assert w == 0 and bounded is True + assert ( + _util.get_mla_context_workspace_kv_len_cap( + kv_cache_config, + _MAX_BATCH, + _CHUNK, + _MAX_SEQ_LEN, + True, + bounded, + chunked_workspace_profiled=False, + require_chunked_workspace_profile=False, + ) + is None + ) + assert creator._fp8_ctx_mla_kv_len_cap is None + pool_bytes = kv_cache_config.max_gpu_total_bytes + assert pool_bytes == int(free * fraction) + pool = pool_bytes // _TOKEN_BYTES + assert pool == _BOOT_POOL_TOKENS + assert pool // (_MAX_SEQ_LEN + _CHUNK) == 1 + + executor = SimpleNamespace( + is_warmup=False, + max_seq_len=tp4.max_seq_len, + kv_cache_manager=SimpleNamespace(fp8_ctx_mla_kv_len_cap=creator._fp8_ctx_mla_kv_len_cap), + _context_attended_kv_len=PyExecutor._context_attended_kv_len, + ) + for name in ("_get_ctx_mla_kv_len_cap", "_warn_if_ctx_mla_kv_len_cap_degenerate"): + setattr(executor, name, types.MethodType(getattr(PyExecutor, name), executor)) + last_chunk = SimpleNamespace( + context_current_position=_LAST_CHUNK_PREFIX, + is_first_context_chunk=False, + estimated_reusable_tokens=0, + context_chunk_size=_CHUNK, + orig_prompt_len=_MAX_SEQ_LEN, + ) + first_chunk = SimpleNamespace( + context_current_position=0, + is_first_context_chunk=True, + estimated_reusable_tokens=0, + context_chunk_size=_CHUNK, + orig_prompt_len=_MAX_SEQ_LEN, + ) + scheduled = PyExecutor._cap_context_by_total_kv_len(executor, [last_chunk, first_chunk]) + assert scheduled == [last_chunk, first_chunk] + assert executor._ctx_mla_kv_len_cap is None + + reserve, cap = _util.get_mla_context_workspace_reserve( + budget, _TOKEN_BYTES, 4096, _MAX_BATCH * _MAX_SEQ_LEN + ) + assert ( + _util.get_mla_context_workspace_kv_len_cap( + kv_cache_config, + _MAX_BATCH, + _CHUNK, + _MAX_SEQ_LEN, + True, + True, + chunked_workspace_profiled=False, + require_chunked_workspace_profile=True, + ) + == _MAX_BATCH * _MAX_SEQ_LEN + ) + assert cap == budget // (_TOKEN_BYTES + 4096) == 1_965_403 + assert cap >= _MAX_SEQ_LEN + _CHUNK + + +def test_warmup_runs_one_32768_token_prompt(tp4, tp4_32k): + """The warmup's shapes at 1M, agreed by the four ranks: 1-token context and decode, the + largest context step of 32768 tokens, 16 decode rows, a 2-token context, as at 32K. A prompt + may now be up to 1048575 tokens, so the largest step is one 32768-token prompt (at 32K: a + 32767-token and a 1-token one). It is a fresh prompt: from 8192 context tokens on, the warmup's + and the estimation's largest step run the query split on every rank.""" + + def max_context_prompts(resolved): + config = _runner_config(resolved) + group = _TpGroup() + factory = _dummy_factory(resolved) + + def rank_warmup(rank): + resources = _Resources(_KvPool(_BOOT_POOL_TOKENS, config.max_seq_len)) + warmup = DecoderWarmup.__new__(DecoderWarmup) + warmup._config = config + agreement = WarmupAgreement( + _RankDist(group, rank), _rank_mapping(resolved.mapping, rank) + ) + shapes = agreement.agree_shapes(warmup._get_full_general_warmup_requests(resources)) + max_context = max(tokens for tokens, gen in shapes if gen == 0) + batch = factory.create_warmup_request(resources, max_context, 0) + return shapes, [request.token_num for request in batch.context_requests] + + return group.run(rank_warmup) + + for shapes, prompts in max_context_prompts(tp4): + assert shapes == [(1, 0), (1, 1), (_CHUNK, 0), (_MAX_BATCH, _MAX_BATCH), (2, 0)] + assert prompts == [_CHUNK] + for shapes, prompts in max_context_prompts(tp4_32k): + assert prompts == [_CHUNK - 1, 1] + + for rank in range(_RANKS): + md = _plan(tp4, rank, [_CHUNK], [0]) + assert md.indexer_prefill_q_split == (8192 * rank, 8192 * (rank + 1), [8192] * 4) + + +def test_context_rows_path_per_step(tp4, tp4_fp8, tp4_quickstart, tp4_32k): + """The target's held context workspace and its context-row path + (``PrefillTarget._ctx_workspace``) on the engine's own host metadata for rank 0 at this + config, in every KV dtype spelling. The held workspace's rows are `_ctx_workspace_rows` of the + engine's quantities, off the runtime features the executor builds from the LLM-API options + (chunked prefill on, block reuse off): under chunked prefill a step can attend up to 16 cached + prefixes of max_seq_len, so 2**19 (`_CTX_WORKSPACE_ROWS`), where the delivered 32K engine + (neither chunked prefill nor block reuse) holds twice max_num_tokens, 65536. The path reads a + step's context KV as cached + new over its context requests (``_ctx_kv_offsets`` off the + engine's indptrs, the host ``ctx_kv_indptr[num_contexts]``), never the engine's + ``num_ctx_kv_tokens``, which counts the new tokens only: + + * the KV-capacity estimate's step, one fresh 32768-token prompt (N = 32768, split 8192 per + rank), takes the workspace path, and the first context step allocates the target's bf16 + [2**19, 576] workspace (576 MiB), held from then on: the estimate measures its peak with the + workspace in it, so it needs no reserve (w = 0, above); + * a 1048576-token prompt's chunks, each a query-split step with the indexer running: through + the 16th (N up to 2**19) on the workspace path, keeping layer 35's rows beside the step's + through the 8th (2N up to 2**19), from the 17th on the decode path in 4 sub-batches of + 8192, the most FlashMLA's decode scheduler is driven at (it exits the process past 11622); + * the envelope gate's 1048559-token prompt: its last chunk is 32751 rows behind 1015808 cached + keys, in sub-batches of 8192 x 3 and 8175; + * a step packing 16384 rows behind 1015808 with a fresh 16369-token prompt and 15 decode rows + behind them (32768 rows) takes the decode path; the decode rows are not context rows. + + The workspace is never reallocated after the estimate's step. + """ + rows_b = tp4_modeling._CTX_WORKSPACE_ROWS + assert (rows_b, tp4_modeling._CTX_DECODE_ROWS) == (2**19, 8192) + assert rows_b * 576 * torch.bfloat16.itemsize == 576 * 2**20 + sizes_32k = _target_sizes(tp4_32k) + features = sizes_32k.runtime_features + assert (features.chunked_prefill, features.cache_reuse) == (False, False) + assert tp4_modeling._most_ctx_kv(sizes_32k) == _CHUNK + assert tp4_modeling._ctx_workspace_rows(sizes_32k) == 2 * _CHUNK + cpu = torch.device("cpu") + for resolved in (tp4, tp4_fp8, tp4_quickstart): + creator = _kv_creator(resolved) + creator._get_token_num_for_estimation() + assert [len(request.input_token_ids) for request in creator._dummy_reqs] == [_CHUNK] + sizes = _target_sizes(resolved) + llm_args = resolved.llm_args + features = sizes.runtime_features + assert (features.chunked_prefill, features.cache_reuse) == ( + llm_args.enable_chunked_prefill, + llm_args.kv_cache_config.enable_block_reuse, + ) + assert (features.chunked_prefill, features.cache_reuse) == (True, False) + assert tp4_modeling._most_ctx_kv(sizes) == _MAX_BATCH * _MAX_SEQ_LEN + assert tp4_modeling._ctx_workspace_rows(sizes) == rows_b + target = tp4_modeling.PrefillTarget.__new__(tp4_modeling.PrefillTarget) + target.core = SimpleNamespace( + lat_dim=576, ctx_workspace_rows=tp4_modeling._ctx_workspace_rows(sizes) + ) + + def path(q_lens, cached, gen_kv=()): + md = _plan(resolved, 0, q_lens, cached, gen_kv) + step = dict( + num_contexts=md.num_contexts, + num_ctx_tokens=md.num_ctx_tokens, + **_target_ctx_kv_offsets(md, q_lens), + ) + n_kv = int(md.host_ctx_kv_indptr[md.num_contexts]) + assert n_kv == sum(q_lens) + sum(cached) + assert step["ctx_cached_tokens"] + step["num_ctx_tokens"] == n_kv + return md, n_kv, target._ctx_workspace(step, md.num_ctx_tokens, cpu) + + md, n_kv, ctx = path([_CHUNK], [0]) + held = target._ws_buf + assert held.shape == (rows_b, 576) and held.dtype == torch.bfloat16 + assert ctx.ws_buf.shape == (_CHUNK, 576) and ctx.ws_buf.data_ptr() == held.data_ptr() + assert ctx.kept.data_ptr() == held[_CHUNK].data_ptr() + assert md.indexer_prefill_q_split == (0, 8192, [8192] * 4) + + for chunk in (2, 8, 9, 16, 17, 32): + md, n_kv, ctx = path([_CHUNK], [(chunk - 1) * _CHUNK]) + assert n_kv == chunk * _CHUNK + assert md.indexer_prefill_q_split == (0, 8192, [8192] * 4) + assert not md.skip_indexer_for_ctx_reqs + if chunk <= 16: + assert ctx.ws_buf.shape[0] == n_kv and ctx.ws_buf.data_ptr() == held.data_ptr() + assert (ctx.kept is not None) == (chunk <= 8) + else: + assert ctx.ws_buf is None + assert ctx.sub_batches == [(lo, lo + 8192) for lo in range(0, _CHUNK, 8192)] + + last = _ENVELOPE_PROMPT - _LAST_CHUNK_PREFIX + assert last == 32751 + md, n_kv, ctx = path([last], [_LAST_CHUNK_PREFIX]) + assert n_kv == _ENVELOPE_PROMPT and ctx.ws_buf is None + assert [hi - lo for lo, hi in ctx.sub_batches] == [8192, 8192, 8192, 8175] + assert sorted(ctx.sched) == [8175, 8192] + + md, n_kv, ctx = path([16384, 16369], [_LAST_CHUNK_PREFIX, 0], _DECODE_KV) + assert md.num_tokens == _CHUNK and n_kv == _LAST_CHUNK_PREFIX + 32753 > rows_b + assert ctx.ws_buf is None + assert ctx.sub_batches[-1] == (24576, 32753) + assert target._ws_buf is held + + +def test_cuda_graph_warmup_decode_rows(tp4): + """The decode graphs (batches 1, 2, 4, 8, 16) are captured twice, the DSA config asking for + separate short and long graphs at its index_topk: once at max_seq_len and once at 2047 (2048 + less the one token a decode step adds). Each capture batch is one longest dummy and batch - 1 + one-token rows: the longest is 1048575 tokens (KV length 1048575, 32768 pages, its decode + token at position 1048574), or 2046 on the short pass. That holds in the estimation's + 1049088-token pool and in the served one.""" + sparse = tp4.model_config.sparse_attention_config + assert isinstance(sparse, SeqLenAwareSparseAttentionConfig) + assert sparse.needs_separate_short_long_cuda_graphs() is True + assert sparse.seq_len_threshold == _TOPK + capture_lengths = [_MAX_SEQ_LEN, sparse.seq_len_threshold - 1] + + graphs = sorted(tp4.llm_args.cuda_graph_config.batch_sizes, reverse=True) + assert graphs == [16, 8, 4, 2, 1] + factory = _dummy_factory(tp4) + for pool_tokens in (1_049_088, _BOOT_POOL_TOKENS): + for batch_size in graphs: + for max_seq_len in capture_lengths: + resources = _Resources(_KvPool(pool_tokens, tp4.max_seq_len)) + batch = factory.create_cuda_graph_warmup_request( + resources, batch_size, 0, max_seq_len + ) + rows = [request.token_num for request in batch.generation_requests] + assert rows == [max_seq_len - 1] + [1] * (batch_size - 1) + assert all(request.is_gen for request in batch.generation_requests) + assert len(batch.context_requests) == 0 + assert math.ceil((_MAX_SEQ_LEN - 1) / _PAGE) == 32768 + + +# ------------------------------------------------------- (4) buffers sized by max_seq_len + + +def _indexer_buffers(resolved, width: int) -> tuple[dict, dict]: + """The DSA metadata's indexer buffers as ``create_buffers_for_indexer`` allocates them for a + KV manager ``width`` pages wide: name -> (shape, dtype), the device ones recorded on the meta + device, the host ones as allocated. ``num_sms`` sizes only the DeepGEMM schedule buffers.""" + params = resolved.model_config.sparse_attention_config.to_sparse_metadata_params( + pretrained_config=resolved.model_config.pretrained_config + ) + device = {} + + def get_empty(buffers, tensor_shape, dtype, cache_name, capture_graph=False): + device[cache_name] = (tuple(tensor_shape), dtype) + return torch.empty(tensor_shape, dtype=dtype, device="meta") + + md = SimpleNamespace( + sparse_metadata_params=params, + cuda_graph_buffers=None, + get_empty=get_empty, + max_num_sequences=resolved.max_batch_size, + max_num_tokens=resolved.max_num_tokens, + kv_cache_manager=SimpleNamespace(max_blocks_per_seq=width), + draft_kv_cache_manager=None, + use_fp8_ds_mla=False, + num_sparse_topk=params.max_sparse_topk, + enable_indexer_skip=params.enable_indexer_skip, + needs_gvr_prior=False, + num_sms=1, + _radix_rows_per_sequence=1, + _draft_sizing_cap=0, + enable_context_mla_with_cached_kv=True, + enable_ragged_verification=False, + ) + for name in ( + "_create_kv_lens_2d_buffer", + "_create_radix_aux_buffers", + "create_expanded_buffers", + ): + setattr(md, name, types.MethodType(getattr(DSAtrtllmAttentionMetadata, name), md)) + with pytest.MonkeyPatch.context() as mp: + _host_only(mp) + DSAtrtllmAttentionMetadata.create_buffers_for_indexer(md, capture_graph=False) + host = { + name: (tuple(value.shape), value.dtype) + for name, value in vars(md).items() + if name.startswith("host_") and isinstance(value, torch.Tensor) + } + return device, host + + +def test_metadata_buffers_sized_by_max_seq_len(tp4, tp4_32k): + """What grows with max_seq_len per rank, on the engine's own allocation code: + + * the block tables, at KVCacheManagerV2's width ceil((max_seq_len + 1) / 32) padded to 4: + 32772 columns at 1M (1028 at 32K). The DSA metadata's indexer_k_cache_block_offsets, + block_table and block_table_expanded are [16, 32772] int32, 2097408 B each, and the first and + third have host copies of the same size; the latent pool's kv_cache_block_offsets + (TrtllmAttentionMetadata) is [pools, 16, 2, 32772] int32, 4194816 B per pool; + * nothing else of the indexer's: topk_indices_buffer and shared_topk_indices stay + [32768, 2048] int32 (max_num_tokens), radix_aux_indices and radix_aux_logits + [16, 10, 2048] (max_num_sequences), scheduler_metadata_buffer [num_sms + 1, 2]; + * the context gather mappings, built every context step over every context request's cached + and new keys on every rank (the query split does not narrow them): the last chunk of a 1M + request gathers 1048576 int64 offsets each for slot_mapping_fp8_fullkv and + slot_mapping_scale_fullkv, 16 MiB together, plus their host staging; a step with no cached + prefix reuses the per-step slot mappings.""" + width, width_32k = _block_table_width(_MAX_SEQ_LEN), _block_table_width(_CHUNK) + assert (width, width_32k) == (32772, 1028) + + device, host = _indexer_buffers(tp4, width) + device_32k, host_32k = _indexer_buffers(tp4_32k, width_32k) + assert device.keys() == device_32k.keys() and host.keys() == host_32k.keys() + grown = {name for name in device if device[name] != device_32k[name]} + assert grown == {"indexer_k_cache_block_offsets", "block_table", "block_table_expanded"} + for name in grown: + assert device[name] == ((_MAX_BATCH, width), torch.int32) + assert device_32k[name] == ((_MAX_BATCH, width_32k), torch.int32) + assert _MAX_BATCH * width * 4 == 2_097_408 + grown_host = {name for name in host if host[name] != host_32k[name]} + assert grown_host == {"host_indexer_k_cache_block_offsets", "host_block_table_expanded"} + for name in grown_host: + assert host[name] == ((_MAX_BATCH, width), torch.int32) + assert _MAX_BATCH * 2 * width * 4 == 4_194_816 + + assert device["topk_indices_buffer"] == ((_CHUNK, _TOPK), torch.int32) + assert device["shared_topk_indices"] == ((_CHUNK, _TOPK), torch.int32) + assert device["radix_aux_indices"] == ((_MAX_BATCH, 10, _TOPK), torch.int32) + assert device["radix_aux_logits"] == ((_MAX_BATCH, 10, _TOPK), torch.float32) + assert device["scheduler_metadata_buffer"] == ((2, 2), torch.int32) + + for rank in range(_RANKS): + md = _plan(tp4, rank, [_CHUNK], [_LAST_CHUNK_PREFIX]) + for name in ("slot_mapping_fp8_fullkv", "slot_mapping_scale_fullkv"): + mapping = getattr(md, name) + assert (tuple(mapping.shape), mapping.dtype) == ((_MAX_SEQ_LEN,), torch.int64) + assert 2 * _MAX_SEQ_LEN * 8 == 16 * 2**20 + fresh = _plan(tp4, 0, [_CHUNK], [0]) + assert fresh.slot_mapping_fp8_fullkv is fresh.slot_mapping_fp8 + assert fresh.slot_mapping_scale_fullkv is fresh.slot_mapping_scale diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_tp4_chunk_equivalence.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_tp4_chunk_equivalence.py new file mode 100644 index 000000000000..f1f2321bcd5d --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_tp4_chunk_equivalence.py @@ -0,0 +1,418 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Chunked prefill against whole-prompt prefill on Mistral-Large-4-1T-A52B-NVFP4's sm_103 tp +target, at tp4 (four ranks) and at tp8 (eight ranks over two nodes of four). + +At the checkpoint's own context the target serves prompts longer than one step: the engine +prefills them in chunks of up to max_num_tokens, and every chunk after a request's first is a +context request over a cached prefix -- its rope/append offsets past the prefix, its indexer +selecting over the cached and the new keys (the engine's chunk plan and its full-KV gather +mappings), sparse MLA attending over the pool rows the earlier chunks wrote, layer 35's latent +read by the NoPE layers 39..71 and every shared-indexer layer reusing the selection of the full +one before it. A chunk of 8192 or more context tokens is split over the TP ranks: each rank +selects for its share of the queries (2048 rows of a full 8192-token chunk at tp4, 1024 at tp8) +and every full indexer layer gathers every rank's share. A step may carry another request's +decode rows behind its context rows. + +So the same prompts run through two engines of one world, one after the other, and greedy +decoding must give the same tokens: + +* the delivered one, `test_modeling_v2_long_context.py`'s `ENGINE`: max_seq_len = + max_num_tokens = 32768, chunked prefill off, so every prompt is prefilled in one step; +* the same engine with max_num_tokens 8192 and chunked prefill on: every full chunk is 8192 + context tokens, so the query split runs on each. + +Both are at max_seq_len 32768, so the target tiles its indexer logits under the same budget in +both, 2**24 (`_prefill_logits_budget`: 512 rows at 32768 keys). It sizes its held context +workspace from each engine's quantities (`_ctx_workspace_rows`): 65536 rows, twice +max_num_tokens, for the unchunked engine; 2**19 under chunked prefill, where a step attends +cached prefixes. Every step of either fits it twice, so in both layer 35's rows are kept beside +the step's for the NoPE layers; an indexer chunk is scored as planned unless it packs past +32768 keys, when it is cut at its requests. What differs is the chunking above. + +The prompts are passkey prompts (the long-context witness's) of 4096, 12288, 20480 and 32752 +tokens, every one past `index_topk` (2048), each key in an earlier chunk than its question (but +the 4096-token prompt's, one chunk), so the answer is read across chunk boundaries. Each runs +alone, twice, on each engine; then the four run together as one batch, so that a chunk carries +two requests and later steps carry chunks behind decode rows. Each step's composition is read +back from the engine's iteration statistics. + +The comparison is gated by the engine's own noise. The target's outputs are not bitwise +reproducible from run to run: the indexer's top-k breaks ties at its 2048th score arbitrarily, so +two runs of one engine on one input may attend over different keys of equal score, and greedy +tokens flip wherever the top two are close. So each prompt's tokens are compared over its leading +steps whose top-1 / top-2 log-probability margin is at least `NOISE` in both engines, and those +steps must hold the key. Every step's margins are printed, and where the tokens differ later, both +engines' top two at the first differing step in the prompt's alone runs, and whether either +engine's repeat differs from itself there. + +Both engines run in this one process, the second once the first has shut down, and the second +sizes its KV pool from the memory the first left free. Under the launcher the ranks' processes +outlive each engine, so before the first engine boots and after each shuts down every rank +reports its device's free memory through the launcher's MPI session (the engines' own channel, +idle between them; `_rank_probe` runs there, imported from this module by its path): the first +engine's shutdown must give back what it took. + +Runs with the ranks already in place, on four GB300s at tp4 or on eight over two nodes of four +at tp8: pytest under `trtllm-llmapi-launch`, as the whole-model accuracy class runs, with the +checkpoint under `LLM_MODELS_ROOT` as `Mistral-Large-4-1T-A52B-NVFP4` (or +`mistral-large-4-nvfp4`) in its Mistral-native format, which both engines load with +checkpoint_format "mistral". Skipped where the checkpoint is absent or off sm_103; each +world size is skipped below its own ranks (four GPUs at tp4, eight ranks at tp8) and inside a +launch of another world size; the release check is skipped outside the launcher. +""" + +from __future__ import annotations + +import gc +import random +import site +import socket +from pathlib import Path +from types import SimpleNamespace + +import pytest +import torch +from test_common.llm_data import llm_models_root + +from tensorrt_llm import LLM, SamplingParams +from tensorrt_llm._torch.modeling_v2 import MODELING_V2_VALIDATE_ENV +from tensorrt_llm._utils import get_sm_version, global_mpi_rank +from tensorrt_llm.executor.utils import create_mpi_comm_session, get_spawn_proxy_process_env +from tensorrt_llm.inputs import TokensPrompt + +from . import test_modeling_v2_long_context as long_context + +#: The engines' world sizes, gated as the long-context witness gates its engine: tp4 on four +#: GPUs, tp8 on eight ranks over two nodes of four, each skipped inside a launch of the other. +WORLDS = long_context.WORLDS + +pytestmark = [ + pytest.mark.skipif( + get_sm_version() != 103, + reason="the mistral_large4 modeling_v2 targets are certified on sm_103 only", + ), + pytest.mark.threadleak(enabled=False), + # Two engine loads, one after the other, about 10 minutes each. + pytest.mark.timeout(3600), +] + +#: The delivered engine: whole prompts of up to 32768 tokens in one step. Its TP sizes are the +#: world's, set per engine. +UNCHUNKED = long_context.ENGINE +CHUNK = 8192 +#: The same engine prefilling in steps of at most CHUNK tokens. +CHUNKED = dict(UNCHUNKED, max_num_tokens=CHUNK, enable_chunked_prefill=True) +#: The engine's default: a step of this many context rows or more splits its queries over the +#: TP ranks. +Q_SPLIT_THRESHOLD = 8192 +INDEX_TOPK = long_context.INDEX_TOPK +#: How long the request queue holds an iteration for more requests while fewer than +#: max_batch_size are waiting, so the batch's prompts, submitted together as token ids, are +#: admitted together; every iteration pays it. +BATCH_WAIT_MS = 200 + +#: (tokens, depth of the key): the key in an earlier chunk of 8192 than the question, but for +#: the one-chunk prompt; the longest leaves room for the answer within max_seq_len 32768. +PROMPTS = ((4096, 0.5), (12288, 0.3), (20480, 0.6), (32752, 0.1)) +#: Greedy, with the top two log-probabilities at each step. +ANSWER = SamplingParams(max_tokens=12, temperature=0.0, logprobs=2) +#: The top-1 / top-2 log-probability margin from which a step's greedy token is compared: below +#: it a token may flip between two runs of one engine on one input. +NOISE = 1.5 +#: What the first engine may keep of a rank's device once it is shut down: its free memory must +#: come back to within this of what the rank had before that engine booted, or the second engine +#: would size its pool from a device the first still holds. The engine itself is far above it +#: (tens of GiB of weights a rank, and a KV pool of most of the rest); what a process kept of an +#: engine it had shut down was 1.82 to 2.56 GiB a rank on two GB300 nodes, measured by the tep8 +#: deployment's chunk-equivalence witness on the same engine machinery. +RELEASE_SLACK = 4 << 30 + +HEAD = long_context.HEAD +FILLER = long_context.FILLER +QUESTION = long_context.QUESTION + + +def _checkpoint(): + path = long_context._checkpoint_dir() + if path is None: + pytest.skip( + f"checkpoint {long_context.CHECKPOINTS} is not under LLM_MODELS_ROOT " + f"({llm_models_root()})" + ) + return path + + +def _engine(engine: dict, world: int) -> LLM: + return LLM( + str(_checkpoint()), + checkpoint_format="mistral", + modeling_v2="require", + env_overrides={MODELING_V2_VALIDATE_ENV: "1"}, + enable_iter_perf_stats=True, + batch_wait_timeout_ms=BATCH_WAIT_MS, + tensor_parallel_size=world, + moe_tensor_parallel_size=world, + **engine, + ) + + +# ------------------------------------------------------------------- every rank, between engines + + +def _rank_probe() -> dict: + """This rank's device free memory after a garbage collection and a cache release, and what + of the device this process's torch allocator still holds. Runs on every rank of the + launcher's MPI world through the engines' session while no engine is up (the session runs + one task at a time).""" + rank = global_mpi_rank() + # The engine's device for this rank: its local rank among the node's devices. + device = rank % torch.cuda.device_count() + torch.cuda.set_device(device) + gc.collect() + torch.cuda.empty_cache() + free, total = torch.cuda.mem_get_info(device) + return dict( + rank=rank, + host=socket.gethostname(), + device=device, + free=free, + total=total, + allocated=torch.cuda.memory_allocated(device), + reserved=torch.cuda.memory_reserved(device), + ) + + +def _reach_ranks(session) -> None: + """Make this module importable on every rank: a task's function is unpickled by module path + before the session hands the rank this process's `sys.path`, so first add the directory that + holds this module's top-level package.""" + base = Path(__file__).resolve().parents[__name__.count(".")] + session.submit_sync(site.addsitedir, str(base)) + + +def _probe(session, world: int, label: str, baseline: dict[int, dict] | None) -> dict[int, dict]: + """`_rank_probe` on every rank, by rank; printed beside `baseline`.""" + reports = {r["rank"]: r for r in session.submit_sync(_rank_probe)} + assert sorted(reports) == list(range(world)), sorted(reports) + for rank, report in sorted(reports.items()): + delta = "" + if baseline is not None: + delta = f" ({report['free'] - baseline[rank]['free']:+d} B against the first probe)" + print( + f"CHUNKEQ probe {label} rank {rank} {report['host']}:{report['device']} free " + f"{report['free'] / 2**30:.3f} of {report['total'] / 2**30:.3f} GiB{delta}, torch " + f"allocated {report['allocated']} B reserved {report['reserved']} B", + flush=True, + ) + return reports + + +# ------------------------------------------------------------------------- the prompts and the runs + + +def _passkey_ids(tokenizer, tokens: int, depth: float, key: int) -> list[int]: + """Exactly `tokens` ids: BOS, then the head, the filler with the needle at `depth` of it, and + the question, each part tokenized on its own and the filler cut at the token that gives the + length.""" + + def encode(text: str) -> list[int]: + ids = tokenizer.encode(text) + return ids[1:] if ids and ids[0] == tokenizer.bos_token_id else ids + + head, question = encode(HEAD), encode(QUESTION) + needle = encode(f"\nThe pass key is {key}. Remember it. {key} is the pass key.\n") + filler = encode(FILLER * (tokens // 8)) + room = tokens - 1 - len(head) - len(needle) - len(question) + assert 0 < room <= len(filler), (tokens, room, len(filler)) + before = int(room * depth) + ids = [tokenizer.bos_token_id, *head, *filler[:before], *needle] + ids += [*filler[: room - before], *question] + assert len(ids) == tokens, (len(ids), tokens) + return ids + + +def _requests(tokenizer) -> list[SimpleNamespace]: + keys = random.Random(CHUNK).sample(range(10000, 100000), len(PROMPTS)) + requests = [] + for (tokens, depth), key in zip(PROMPTS, keys): + ids = _passkey_ids(tokenizer, tokens, depth, key) + assert INDEX_TOPK < len(ids), (tokens, len(ids)) + requests.append(SimpleNamespace(key=key, ids=ids, depth=depth)) + return requests + + +def _steps(llm: LLM) -> list[tuple[int, int, int]]: + """(context requests, context tokens, generation requests) of each iteration since the last + call that ran any context rows.""" + steps = [] + for record in llm.get_stats(timeout=2): + ifb = record.get("inflightBatchingStats") or {} + if ifb.get("numContextRequests", 0): + steps.append( + (ifb["numContextRequests"], ifb["numCtxTokens"], ifb.get("numGenRequests", 0)) + ) + return steps + + +def _generate(llm: LLM, requests: list) -> tuple[list[SimpleNamespace], list]: + """`requests` through `llm` in one generate call: per request its token ids, its top two + log-probabilities at each step and its text; and the context steps they ran in.""" + _steps(llm) + outputs = llm.generate([TokensPrompt(prompt_token_ids=r.ids) for r in requests], ANSWER) + steps = _steps(llm) + results = [] + for out in outputs: + o = out.outputs[0] + top2 = [ + sorted(((lp.logprob, tok) for tok, lp in step.items()), reverse=True)[:2] + for step in o.logprobs + ] + results.append(SimpleNamespace(ids=list(o.token_ids), top2=top2, text=o.text)) + print(f"CHUNKEQ run tokens={len(out.prompt_token_ids)}: {o.text!r} {list(o.token_ids)}") + return results, steps + + +def _run(llm: LLM, requests: list) -> dict: + """Each prompt alone, twice, then all of them as one batch.""" + alone = [[], []] + alone_steps = [] + for r in requests: + for repeat in range(2): + (result,), steps = _generate(llm, [r]) + alone[repeat].append(result) + if repeat == 0: + alone_steps.append(steps) + batch, batch_steps = _generate(llm, requests) + return dict(alone=alone[0], again=alone[1], alone_steps=alone_steps, batch=batch, + batch_steps=batch_steps) # fmt: skip + + +def _margins(top2: list) -> list[float]: + """Each step's top-1 / top-2 log-probability gap.""" + return [s[0][0] - s[1][0] if len(s) > 1 else float("inf") for s in top2] + + +def _first_difference(a: list[int], b: list[int]) -> int | None: + return next((i for i, (x, y) in enumerate(zip(a, b)) if x != y), None) + + +@pytest.fixture(scope="module", params=WORLDS) +def ran(request): + """Both engines at the world, one after the other, each over the same prompts; under the + launcher, every rank's device memory before the first boots and after each shuts down.""" + world = request.param + _checkpoint() + session, probes = None, {} + if get_spawn_proxy_process_env(): + # The engines' own session: a singleton every LLM in this process reuses. + session = create_mpi_comm_session(world) + _reach_ranks(session) + probes["before"] = _probe(session, world, f"tp{world} before the whole engine", None) + with _engine(UNCHUNKED, world) as llm: + tokenizer = llm.tokenizer + requests = _requests(tokenizer) + runs = {"whole": _run(llm, requests)} + if session is not None: + label = f"tp{world} after the whole engine" + probes["after_whole"] = _probe(session, world, label, probes["before"]) + with _engine(CHUNKED, world) as llm: + runs["chunked"] = _run(llm, requests) + if session is not None: + label = f"tp{world} after the chunked engine" + probes["after_chunked"] = _probe(session, world, label, probes["before"]) + return SimpleNamespace(tokenizer=tokenizer, requests=requests, runs=runs, probes=probes) + + +def test_chunked_prefill_gives_the_whole_prompt_tokens(ran): + tokenizer, requests, runs = ran.tokenizer, ran.requests, ran.runs + + problems = [] + lengths = [len(r.ids) for r in requests] + for engine, run in runs.items(): + for n, steps in zip(lengths, run["alone_steps"]): + print(f"CHUNKEQ {engine} alone tokens={n}: (ctx requests, ctx tokens, gen requests) " + f"{steps}") # fmt: skip + print(f"CHUNKEQ {engine} batch tokens={lengths}: {run['batch_steps']}") + for n, steps in zip(lengths, run["alone_steps"]): + if sum(t for _, t, _ in steps) != n or any(c != 1 for c, _, _ in steps): + problems.append(f"{engine} alone tokens={n}: context steps {steps}") + if sum(t for _, t, _ in run["batch_steps"]) != sum(lengths): + problems.append(f"{engine} batch: context steps {run['batch_steps']}") + + # The whole-prompt engine prefills each prompt in one step; the chunked one in full chunks of + # CHUNK context rows (the query split) and a last one of the rest. + for n, steps in zip(lengths, runs["whole"]["alone_steps"]): + if [t for _, t, _ in steps] != [n]: + problems.append(f"whole alone tokens={n}: not one step: {steps}") + for n, steps in zip(lengths, runs["chunked"]["alone_steps"]): + want = [CHUNK] * (n // CHUNK) + ([n % CHUNK] if n % CHUNK else []) + if [t for _, t, _ in steps] != want: + problems.append(f"chunked alone tokens={n}: chunks {steps}, not {want}") + split = sum( + 1 for s in runs["chunked"]["alone_steps"] for _, t, _ in s if t >= Q_SPLIT_THRESHOLD + ) + print(f"CHUNKEQ chunked alone: {split} context steps of >= {Q_SPLIT_THRESHOLD} rows (split)") + batch_steps = runs["chunked"]["batch_steps"] + if not any(c > 1 for c, _, _ in batch_steps): + problems.append(f"chunked batch: no step carries two context requests: {batch_steps}") + if not any(g > 0 for _, _, g in batch_steps): + problems.append(f"chunked batch: no step mixes context and decode rows: {batch_steps}") + + whole, chunked = runs["whole"], runs["chunked"] + for kind in ("alone", "batch"): + for i, r in enumerate(requests): + a, b = whole[kind][i], chunked[kind][i] + ma, mb = _margins(a.top2), _margins(b.top2) + n = min(len(a.ids), len(b.ids)) + gated = next((s for s in range(n) if min(ma[s], mb[s]) < NOISE), n) + shown = tokenizer.decode(a.ids[:gated]) + print( + f"CHUNKEQ {kind} tokens={len(r.ids)} key={r.key}: whole {a.text!r}, chunked " + f"{b.text!r}; {gated} steps at margin >= {NOISE} in both ({shown!r}); margins " + f"whole {[round(m, 2) for m in ma]} chunked {[round(m, 2) for m in mb]}" + ) + for label, out in (("whole", a), ("chunked", b)): + if str(r.key) not in out.text: + problems.append( + f"{kind} tokens={len(r.ids)}: {label} lost the key: {out.text!r}" + ) + if a.ids[:gated] != b.ids[:gated]: + problems.append( + f"{kind} tokens={len(r.ids)}: differ within the {gated} steps above the " + f"noise: {a.ids[:gated]} vs {b.ids[:gated]}" + ) + elif str(r.key) not in shown: + problems.append( + f"{kind} tokens={len(r.ids)}: the key {r.key} is not within the {gated} steps " + f"above the noise ({shown!r})" + ) + k = _first_difference(a.ids, b.ids) + if k is None: + continue + # The first differing step measured on each engine alone, and each engine's repeat. + for label, run in (("whole", whole), ("chunked", chunked)): + alone, again = run["alone"][i], run["again"][i] + self_k = _first_difference(alone.ids, again.ids) + top = alone.top2[k] if k < len(alone.top2) else None + print( + f"CHUNKEQ {kind} tokens={len(r.ids)}: first differing step {k}; {label} alone " + f"top-2 {top}, its repeat first differs from it at step {self_k}" + ) + assert not problems, problems + + +def test_the_first_engine_gives_its_devices_back(ran): + """After the whole-prompt engine shuts down, every rank's device is back within RELEASE_SLACK + of its free memory before that engine booted, so the chunked engine sizes its pool from the + same devices.""" + if not ran.probes: + pytest.skip("the per-rank probe runs through trtllm-llmapi-launch's MPI session") + for rank, before in ran.probes["before"].items(): + after = ran.probes["after_whole"][rank] + assert (after["host"], after["device"]) == (before["host"], before["device"]) + assert before["free"] - after["free"] <= RELEASE_SLACK, ( + rank, + before["free"], + after["free"], + ) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_tp4_chunked_split.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_tp4_chunked_split.py new file mode 100644 index 000000000000..33f0b403911c --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_tp4_chunked_split.py @@ -0,0 +1,261 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The indexer's query split under chunked prefill, end to end, on Mistral-Large-4-1T-A52B-NVFP4's +sm_103 tp target at the 1M config, at tp4 (four ranks) and at tp8 (eight ranks over two nodes of +four). + +GSM8K's prompts stay within the indexer's `index_topk` (2048 positions), where the engine skips +the context indexer: the accuracy gate never runs the split of a step's context queries over the +TP ranks, nor the gather of their selections. Here the whole model runs under its serving +configuration -- max_seq_len 1048576, chunked prefill in steps of max_num_tokens 32768, sixteen +sequences, the fp8_ds_mla latent cache at the default KV dtype, CUDA graphs over batches 1, 2, 4, +8 and 16, padded -- and two passkey prompts submitted together in one generate call are +scheduled, first come first served, as two context steps: + +* all of prompt A (20480 tokens) and the first 12288 tokens of prompt B, 32768 context rows, B's + chunk first (the scheduler puts a step's last chunks after the others): the engine splits them + evenly over the ranks, 8192 rows a rank at tp4 (rank 1's share runs from B's chunk into A, one + indexer chunk per request) and 4096 at tp8 (B's chunk is exactly ranks 0-2's shares, A ranks + 3-7's); each rank selects for its own rows and every full indexer layer gathers every rank's + share; +* the rest of B, 9003 tokens over its 12288-token cached prefix, with A's first decode row behind + them: an uneven split of a chunk over a cached prefix in a mixed step -- (2250, 2251, 2251, + 2251) at tp4, (1125, 1125, 1126, 1125, 1125, 1126, 1125, 1126) at tp8. + +A's passkey sits at 0.7 of its filler, in rank 3's share of the first step at tp4 and rank 6's at +tp8; B's at 0.3, in its first chunk (rank 0's share at tp4, rank 1's at tp8), so B's question rows +reach it only through the cached prefix. Greedy decoding must recall both. Every key count stays +at or below 32768. The step composition is read back from the engine's iteration statistics, and +each step's shares at the world are printed beside it. + +Text only: no image, no chat template. Two engine options beyond the serving ones make the steps +observable and do not change what a step computes: iteration statistics, and a request queue that +waits up to `BATCH_WAIT_MS` for a full batch so the two prompts are admitted in one step rather +than split by their arrival. + +Runs with the ranks already in place, on four GB300s at tp4 or on eight over two nodes of four +at tp8: pytest under `trtllm-llmapi-launch`, as the whole-model accuracy class runs, with the +checkpoint under `LLM_MODELS_ROOT` as `Mistral-Large-4-1T-A52B-NVFP4` (or +`mistral-large-4-nvfp4`) in its Mistral-native format, which the engine loads with +checkpoint_format "mistral". Skipped where the checkpoint is absent or off sm_103; each +world size is skipped below its own ranks (four GPUs at tp4, eight ranks at tp8) and inside a +launch of another world size. +""" + +from __future__ import annotations + +import random +from pathlib import Path + +import pytest +import torch +from test_common.llm_data import llm_models_root +from utils.util import skip_num_gpus_less_than + +from tensorrt_llm import LLM, SamplingParams +from tensorrt_llm._torch.modeling_v2 import MODELING_V2_VALIDATE_ENV +from tensorrt_llm._utils import get_sm_version +from tensorrt_llm.inputs import TokensPrompt +from tensorrt_llm.llmapi import CudaGraphConfig, KvCacheConfig +from tensorrt_llm.llmapi.mpi_session import get_mpi_world_size + + +def _ranks() -> int: + """The ranks this process can serve: the MPI world a launcher started (eight over two nodes + of four, where each node sees four devices), or, unlaunched, the devices the LLM API spawns + its workers on -- as the integration conftest's `skip_less_mpi_world_size` counts.""" + world = get_mpi_world_size() + return torch.cuda.device_count() if world == 1 else world + + +def _launched_elsewhere(world: int) -> bool: + """A launcher started an MPI world of another size: an engine of `world` ranks does not + start inside it.""" + return get_mpi_world_size() not in (1, world) + + +#: The engine's world sizes, each gated on its own ranks: tp4 on four GPUs, tp8 on eight ranks +#: over two nodes of four. +WORLDS = [ + pytest.param( + 4, + marks=[ + skip_num_gpus_less_than(4), + pytest.mark.skipif( + _launched_elsewhere(4), reason="tp4 needs a launch of four ranks or none" + ), + ], + id="tp4", + ), + pytest.param( + 8, + marks=[ + pytest.mark.skipif(_ranks() < 8, reason="tp8 needs eight ranks"), + pytest.mark.skipif( + _launched_elsewhere(8), reason="tp8 needs a launch of eight ranks or none" + ), + ], + id="tp8", + ), +] + +pytestmark = [ + pytest.mark.skipif( + get_sm_version() != 103, + reason="the mistral_large4 modeling_v2 targets are certified on sm_103 only", + ), + # One engine serves the module: the LLM API starts its result-dispatch thread on the first + # request, inside the first test, and stops it when the fixture shuts the engine down. + pytest.mark.threadleak(enabled=False), +] + +CHECKPOINTS = ("Mistral-Large-4-1T-A52B-NVFP4", "mistral-large-4-nvfp4") + +# The serving configuration: tp4 or tp8 (attention and the routed experts tensor parallel over all +# the ranks, the experts not expert-parallel), sixteen sequences, the checkpoint's own context +# prefilled in chunked steps of up to 32768 tokens. +MAX_NUM_TOKENS = 32768 +ENGINE = dict( + moe_expert_parallel_size=1, + enable_attention_dp=False, + max_batch_size=16, + max_num_tokens=MAX_NUM_TOKENS, + max_seq_len=1048576, + enable_chunked_prefill=True, + cuda_graph_config=CudaGraphConfig(max_batch_size=16, enable_padding=True), + kv_cache_config=KvCacheConfig(enable_block_reuse=False, free_gpu_memory_fraction=0.85), +) +# How long the request queue holds an iteration for more requests while fewer than +# max_batch_size are waiting; every iteration pays it, decode steps included. The two prompts +# are submitted together as token ids and arrive within milliseconds of each other. +BATCH_WAIT_MS = 1000 + +# The engine's defaults: the split from this many context rows in a step, and the KV page, which +# is also the unit a chunk is cut down to when the step's budget trims it. +Q_SPLIT_THRESHOLD = 8192 +TOKENS_PER_BLOCK = 32 + +# Prompt A fills its first step whole; the rest of the step's budget is B's first chunk, and the +# rest of B is the next step's one chunk, behind which A decodes. +A_TOKENS = 20480 +B_TOKENS = 21291 +B_FIRST_CHUNK = MAX_NUM_TOKENS - A_TOKENS +B_LAST_CHUNK = B_TOKENS - B_FIRST_CHUNK +A_DEPTH, B_DEPTH = 0.7, 0.3 + +HEAD = ( + "There is an important piece of information hidden inside a lot of irrelevant text. " + "Find it and memorize it. I will quiz you about it.\n\n" +) +FILLER = ( + "The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again. " +) +QUESTION = "\nWhat is the pass key? The pass key is" +# A five-digit key and its closing punctuation fit with room to spare. +ANSWER = SamplingParams(max_tokens=12, temperature=0.0) + + +def _checkpoint_dir() -> Path | None: + """The checkpoint's directory, found by its Mistral-native config, params.json.""" + root = llm_models_root() + if root is None: + return None + for name in CHECKPOINTS: + if (Path(root) / name / "params.json").is_file(): + return Path(root) / name + return None + + +def _split(rows: int, world: int) -> list[int]: + """The engine's per-rank shares of a step's context rows over `world` ranks, in rank order: + rank r takes `[rows * r // world, rows * (r + 1) // world)`.""" + return [rows * (r + 1) // world - rows * r // world for r in range(world)] + + +def _passkey_ids(tokenizer, tokens: int, depth: float, key: int) -> list[int]: + """Exactly `tokens` ids: BOS, then the head, the filler with the needle at `depth` of it, and + the question, each part tokenized on its own and the filler cut at the token that gives the + length.""" + + def encode(text: str) -> list[int]: + ids = tokenizer.encode(text) + return ids[1:] if ids and ids[0] == tokenizer.bos_token_id else ids + + head, question = encode(HEAD), encode(QUESTION) + needle = encode(f"\nThe pass key is {key}. Remember it. {key} is the pass key.\n") + filler = encode(FILLER * (tokens // 8)) + room = tokens - 1 - len(head) - len(needle) - len(question) + assert 0 < room <= len(filler), (tokens, room, len(filler)) + before = int(room * depth) + ids = [tokenizer.bos_token_id, *head, *filler[:before], *needle] + ids += [*filler[: room - before], *question] + assert len(ids) == tokens, (len(ids), tokens) + return ids + + +def _steps(llm: LLM) -> list[tuple[int, int, int]]: + """(context requests, context tokens, generation requests) of each iteration since the last + call that ran any context rows.""" + steps = [] + for record in llm.get_stats(timeout=2): + ifb = record.get("inflightBatchingStats") or {} + if ifb.get("numContextRequests", 0): + steps.append( + (ifb["numContextRequests"], ifb["numCtxTokens"], ifb.get("numGenRequests", 0)) + ) + return steps + + +@pytest.fixture(scope="module", params=WORLDS) +def world(request) -> int: + """The engine's world size: four ranks at tp4, eight at tp8.""" + return request.param + + +@pytest.fixture(scope="module") +def llm(world): + path = _checkpoint_dir() + if path is None: + pytest.skip(f"checkpoint {CHECKPOINTS} is not under LLM_MODELS_ROOT ({llm_models_root()})") + with LLM( + str(path), + checkpoint_format="mistral", + modeling_v2="require", + env_overrides={MODELING_V2_VALIDATE_ENV: "1"}, + enable_iter_perf_stats=True, + batch_wait_timeout_ms=BATCH_WAIT_MS, + tensor_parallel_size=world, + moe_tensor_parallel_size=world, + **ENGINE, + ) as engine: + print(f"SPLIT tp{world} kv_cache_capacity {engine.get_kv_cache_capacity()}", flush=True) + _steps(engine) + yield engine + + +def test_two_prompts_split_over_the_ranks(llm, world): + """Two passkey prompts in one generate call: a full 32768-row step split evenly over the + world's ranks, then a chunk over a cached prefix split unevenly with a decode row behind it; + both keys are recalled.""" + assert A_TOKENS % TOKENS_PER_BLOCK == 0 and B_FIRST_CHUNK % TOKENS_PER_BLOCK == 0 + assert _split(MAX_NUM_TOKENS, world) == [MAX_NUM_TOKENS // world] * world + assert B_LAST_CHUNK >= Q_SPLIT_THRESHOLD and B_LAST_CHUNK % world != 0 + keys = random.Random(MAX_NUM_TOKENS + 1).sample(range(10000, 100000), 2) + prompts = [ + _passkey_ids(llm.tokenizer, A_TOKENS, A_DEPTH, keys[0]), + _passkey_ids(llm.tokenizer, B_TOKENS, B_DEPTH, keys[1]), + ] + outputs = llm.generate([TokensPrompt(prompt_token_ids=ids) for ids in prompts], ANSWER) + steps = _steps(llm) + answers = [out.outputs[0].text for out in outputs] + for name, ids, key, answer in zip("AB", prompts, keys, answers): + print(f"SPLIT prompt {name} tokens={len(ids)} key={key} answer={answer!r}", flush=True) + print( + f"SPLIT context_steps={steps} (context requests, context tokens, generation requests); " + f"q_sizes at tp{world} {_split(MAX_NUM_TOKENS, world)} then {_split(B_LAST_CHUNK, world)}", + flush=True, + ) + + assert steps == [(2, MAX_NUM_TOKENS, 0), (1, B_LAST_CHUNK, 1)], steps + misses = [(n, key, a) for n, key, a in zip("AB", keys, answers) if str(key) not in a] + assert not misses, misses diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_tp4_ctx_bound.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_tp4_ctx_bound.py new file mode 100644 index 000000000000..3cc7c59c8585 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_tp4_ctx_bound.py @@ -0,0 +1,700 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The tp target's context rows at the 1M config (max_seq_len 1048576, chunked prefill in steps +of max_num_tokens 32768, max_batch_size 16): the per-step choice between the held workspace and +the pool, at tp4 (four ranks) and at tp8 (eight ranks over two nodes of four). + +`PrefillTarget._ctx_workspace` picks the step's path once, on host ints, from N, its context KV +(every context request's cached and new rows; the engine's `num_ctx_kv_tokens` counts the new +ones only), against B, the held workspace's rows: `_ctx_workspace_rows` of the engine's +quantities, 2**19 (`_CTX_WORKSPACE_ROWS`) at this config, where a step can attend cached prefixes. +Up to B rows: `ds_mla_dequant_gather` writes the rows into the first N rows of one bf16 [B, 576] +workspace, allocated on the first context step and held -- layer 35's, which the NoPE layers +39, 43, ..., 71 attend over, into the N rows after them while 2N fits B, and into the first N +again for each of those layers past that --, then `flash_mla_sparse_prefill` attends over it. +Past B: `convert_req_index_to_global` turns the selections into pool slots and +`flash_mla_sparse_decode` attends with every context row a one-row sequence, in sub-batches of +at most `_CTX_DECODE_ROWS` (8192) rows with one FlashMLA scheduler per sub-batch size. +`PrefillTarget._attn_ctx` runs the path. Both kernels take the rank's heads first in a 64-head +query whose other heads are zero: 16 heads at tp4, 8 at tp8. The choice and its paths are the +same at both worlds; the heads are not, so every GPU test below runs at each world's heads (ids +`tp4` and `tp8`), on one GPU either way. + +On one GPU, over one fp8_ds_mla pool at the 63-owner layout (52000 blocks, 68.8 GB; every byte +0xFF, a NaN in every field, but the owner rows of layers 1, 35 and 36), with the engine's +[16, 32772] block table and selections shaped like the indexer's (request-local positions, every +earlier position below index_topk, else 2048 distinct ones with the row's own): + +* the KV-capacity estimate's step (one fresh 32768-token prompt) allocates the workspace; no + later step reallocates it, no step past B reaches the gather, and every gather writes the + held buffer: a step's first N rows, or layer 35's N rows after them; +* one packed step of four chunks at N = B (workspace path) and, with its last chunk one row + longer, at N = B + 1 (decode path, 4 x 8192): the shared rows' outputs are bitwise the same; +* the envelope gate's 1048559-token prompt's last chunk, 32751 rows behind 1015808 cached keys: + sub-batches 8192 x 3 + 8175 on two schedulers, the short one's split rows within the survey's + band; +* a q-split step (32768 context rows, the indexer running, rank 0's share the first 8192 rows at + tp4, 4096 at tp8) behind 1015808 cached keys takes the decode path and attends every row the + gather filled; its per-layer peak memory against the estimate's step on the workspace path; +* mixed steps on both paths (generation rows behind the context rows, which the context + attention leaves alone), and layer 35's sharer (39) reading row 35: where the step kept it + beside its own rows (2N within B), and gathered again after another owner (36) overwrote the + workspace (2N past B, N within it); +* the packed four-chunk step of the 1M config (cached 1032192 / 524288 / 0 / 2048), on the path + the step picks: per-layer time and peak memory, printed. + +Every decode-path output is checked against the same kernel over an unbounded workspace (the +workspace path's computation without the bound): bitwise in each full 8192-row sub-batch, and +within relative L2 1e-3 overall. Sampled rows of every output are checked against an fp32 softmax +over keys read straight from the pool through the block table. The CPU test checks +the choice on host metadata alone. +""" + +from __future__ import annotations + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch.modeling_v2.catalog._op import true_fp32_matmul +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_103__tp import ( # noqa: E501 + modeling as M, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.convert_req_index_to_global import ( # noqa: E501 + convert_req_index_to_global, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.flash_mla_sparse_prefill import ( # noqa: E501 + flash_mla_sparse_prefill, +) +from tensorrt_llm.flash_mla import get_mla_metadata + +_LAYERS = 72 +_TPB = 32 +_TOPK = 2048 +#: The checkpoint's MLA heads, split over the ranks: a rank's share goes first in the kernels' +#: 64-head query, the other heads zero. +_MODEL_HEADS = 64 +_KERNEL_HEADS = 64 +#: The target's worlds: tp4 (four ranks, 16 heads a rank) and tp8 (eight ranks over two nodes of +#: four, 8 heads a rank). One GPU runs a rank's kernels at either. +WORLDS = [pytest.param(4, id="tp4"), pytest.param(8, id="tp8")] +_LAT_DIM, _NOPE, _TILE, _ROW_BYTES = 576, 512, 128, 656 +#: Every fourth layer from 39 on attends over layer 35's latent (the checkpoint's +#: kv_source_layers); the pool holds the 63 owners' rows. +_SOURCES = [35 if i >= 39 and i % 4 == 3 else i for i in range(_LAYERS)] +_OWNERS = [i for i in range(_LAYERS) if _SOURCES[i] == i] +#: The owner layers whose rows the pool holds: a RoPE owner, the NoPE source of the sharers, +#: and the owner after it. +_WRITTEN = (1, 35, 36) +#: Pool blocks: the packed four-chunk step of the 1M config holds 49728 of them. +_BLOCKS = 52000 +#: md.block_table at max_seq_len 1048576 on the V2 manager: [max_batch_size, 32772]. +_TABLE = (16, 32772) +_SCALE = 128**-0.5 +_SEED = 1051 +#: What the target reads of the 1M config's attention metadata to size its held workspace +#: (chunked prefill on and block reuse off in its runtime features). +_ENGINE_1M = SimpleNamespace( + max_num_tokens=32768, + max_num_requests=16, + max_seq_len=1048576, + runtime_features=SimpleNamespace(chunked_prefill=True, cache_reuse=False), +) +#: The held workspace's rows there: 2**19, the ceiling. +_B = M._ctx_workspace_rows(_ENGINE_1M) +_SUB = M._CTX_DECODE_ROWS +#: The last chunk of a 1048576-token prompt reads a cached prefix of 1015808 tokens. +_PREFIX_1M = 1048576 - 32768 +#: Four chunks (cached, new) packed to a context KV of exactly B in 32767 rows, and the same +#: step with its last chunk one row longer: B + 1 in 32768 rows. +_AT_B = [(358401, 16384), (131072, 8192), (2048, 4096), (0, 4095)] +_PAST_B = _AT_B[:-1] + [(0, 4096)] +#: The agreement between the paths: bitwise in a full sub-batch of 8192 rows; a shorter one +#: (a chunk's tail) has FlashMLA split some rows' keys over SMs, which moves those rows within the +#: survey's measured band for split sub-batches (relative L2 1.4e-04 to 8.7e-04). +_AGREE = 1e-3 +#: An fp32 softmax over the selected keys against the bf16 kernels. +_FP32_BAND = 1e-2 +_SAMPLED = 192 +_GIB = 2**30 + + +def _cumsum(lens: list[int], dev=None) -> torch.Tensor: + return torch.tensor([0] + list(lens), dtype=torch.int64, device=dev).cumsum(0) + + +#: cute_dsl_ds_mla_sparse_decode's handles by head count, built once. +_NATIVE = {} + + +def _native_decode(heads: int): + """The generation rows' kernel handle at the rank's `heads`, as `build_kernels` builds it.""" + if heads not in _NATIVE: + _NATIVE[heads] = M.build_ds_mla_decode(torch.cuda.current_device(), num_heads=heads) + return _NATIVE[heads] + + +def _target(heads: int | None = None) -> M.PrefillTarget: + """The prefill target with only what its context and generation attention read off the + core: the held workspace's rows at the 1M config, the shared source as the core builds it + (layer 35 alone), the 64-head kernels' softmax scale, index_topk, the 63-owner pool's + addressing, and the generation rows' native decode at the rank's `heads` (none on the host: + no generation row is attended there).""" + target = M.PrefillTarget.__new__(M.PrefillTarget) + target.core = SimpleNamespace( + ctx_workspace_rows=_B, + shared_source=[i == 35 for i in range(_LAYERS)], + # The generation rows on cute_dsl_ds_mla_sparse_decode at the rank's heads, as shipped. + gen_attn_flash_mla=False, + _mla_decode=None if heads is None else _native_decode(heads), + lat_dim=_LAT_DIM, + softmax_scale=_SCALE, + idx_topk=_TOPK, + _stride_factor=len(_OWNERS) * _TPB, + _pool_row=[_OWNERS.index(_SOURCES[i]) for i in range(_LAYERS)], + ) + return target + + +def _host_step(contexts, gens=(), **extra) -> dict: + """The host fields `_ctx_workspace` reads, as `PrefillTarget.step_args` builds them under + chunked prefill (`_ctx_kv_offsets` and `_ctx_host_offsets` over the engine's indptrs) for + context requests of (cached, new) tokens with generation rows behind them.""" + cached = [c for c, _ in contexts] + new = [n for _, n in contexts] + md = SimpleNamespace( + enable_context_mla_with_cached_kv=True, + ctx_cached_token_indptr=_cumsum(cached), + ctx_kv_indptr=_cumsum([c + n for c, n in contexts]), + max_ctx_seq_len=max(new), + num_ctx_cached_tokens=sum(cached), + host_ctx_uncached_token_indptr=_cumsum(new), + host_ctx_kv_indptr=_cumsum([c + n for c, n in contexts]), + ) + step = dict( + num_contexts=len(contexts), + num_ctx_tokens=sum(new), + # The engine's count: the step's new context keys only. + num_ctx_kv_tokens=sum(new), + num_tokens=sum(new) + len(gens), + skip_ctx=False, + indexer_prefill_q_split=None, + **M._ctx_kv_offsets(md, len(contexts)), + **M._ctx_host_offsets(md, len(contexts)), + ) + step.update(extra) + return step + + +# ---- host only: the per-step choice ---- + + +@pytest.mark.cpu_only +def test_switch_reads_cached_plus_new_on_host_ints(): + """The held workspace is 2**19 rows at the 1M config. The path follows cached + new at B + and B + 1, whatever the engine's new-token count, the query split or the generation rows + say; the decode path's sub-batches tile the context rows at 8192 with one scheduler per + size; layer 35's rows are kept beside a step's while 2N fits B; the workspace is allocated + once.""" + assert _B == M._CTX_WORKSPACE_ROWS == 2**19 + target = _target() + cpu = torch.device("cpu") + + at_b = _host_step(_AT_B) + assert at_b["ctx_cached_tokens"] + at_b["num_ctx_tokens"] == _B + ctx = target._ctx_workspace(at_b, at_b["num_ctx_tokens"], cpu) + buf = target._ws_buf + assert buf.shape == (_B, _LAT_DIM) and buf.dtype == torch.bfloat16 + assert ctx.ws_buf.shape == (_B, _LAT_DIM) and ctx.ws_buf.data_ptr() == buf.data_ptr() + assert ctx.kv_req.shape == (_B,) and int(ctx.kv_pos[-1]) == 4094 + assert ctx.kept is None + + past_b = _host_step(_PAST_B) + assert past_b["ctx_cached_tokens"] + past_b["num_ctx_tokens"] == _B + 1 + ctx = target._ctx_workspace(past_b, past_b["num_ctx_tokens"], cpu) + assert ctx.ws_buf is None and not hasattr(ctx, "kv_req") + assert ctx.sub_batches == [(0, 8192), (8192, 16384), (16384, 24576), (24576, 32768)] + assert sorted(ctx.sched) == [8192] and not ctx.sched[8192].have_initialized + + # The envelope gate's prompt's last chunk (1048559 = 1015808 + 32751): its new tokens alone + # are far below B. + ragged = _host_step([(_PREFIX_1M, 32751)]) + assert ragged["num_ctx_kv_tokens"] == 32751 < _B + ctx = target._ctx_workspace(ragged, 32751, cpu) + assert ctx.ws_buf is None + assert ctx.sub_batches == [(0, 8192), (8192, 16384), (16384, 24576), (24576, 32751)] + assert sorted(ctx.sched) == [8175, 8192] + + # A q-split step (32768 context rows, the indexer running, this rank's share 0..8191) over a + # cached prefix, with generation rows behind it. + q_split = _host_step( + [(_PREFIX_1M, 32768 - 3)], gens=(524288, 3001, 17), + indexer_prefill_q_split=(0, 8192, [8192, 8192, 8192, 8189]), + ) # fmt: skip + ctx = target._ctx_workspace(q_split, q_split["num_ctx_tokens"], cpu) + assert ctx.ws_buf is None and ctx.sub_batches[-1] == (24576, 32765) + + # A dense step (every request within index_topk) and the estimate's fresh 32768-token prompt + # keep layer 35's rows beside their own; a step of B / 2 + 1 rows fits B only once. + for contexts in ([(0, 2048)] * 16, [(0, 32768)], [(2047, 1)] * 16): + step = _host_step(contexts) + ctx = target._ctx_workspace(step, step["num_ctx_tokens"], cpu) + assert ctx.ws_buf is not None and ctx.ws_buf.data_ptr() == buf.data_ptr() + assert ctx.kept.shape == (32768, _LAT_DIM) and ctx.kept.data_ptr() == buf[32768].data_ptr() + half = _host_step([(_B // 2 - 32767, 32768)]) + ctx = target._ctx_workspace(half, 32768, cpu) + assert ctx.ws_buf.shape == (_B // 2 + 1, _LAT_DIM) and ctx.kept is None + assert target._ws_buf is buf + + +# ---- one GPU ---- + + +def _dev_rows(n: int, g: torch.Generator, nope: bool) -> torch.Tensor: + """`n` uint8 656-byte rows as the fp8_ds_mla writer stores them: e4m3 codes, power-of-two + fp32 tile scales 2^-10..2^-4, bf16 rope (zero on a NoPE owner).""" + dev = g.device + codes = ((torch.rand(n, _NOPE, generator=g, device=dev) * 2 - 1) * 448).to(torch.float8_e4m3fn) + scale = torch.exp2(torch.randint(-10, -3, (n, _NOPE // _TILE), generator=g, device=dev).float()) + if nope: + rope = torch.zeros(n, _LAT_DIM - _NOPE, device=dev) + else: + rope = torch.randn(n, _LAT_DIM - _NOPE, generator=g, device=dev) * 0.5 + return torch.cat( + [ + codes.view(torch.uint8), + scale.view(torch.uint8), + rope.to(torch.bfloat16).view(torch.uint8), + ], + dim=1, + ) + + +def _dequant(rows: torch.Tensor) -> torch.Tensor: + """fp8_ds_mla rows `[n, 656]` to bf16 `[n, 576]`: e4m3 times its tile's scale (exact in + bf16 for power-of-two scales), then the rope.""" + codes = rows[:, :_NOPE].contiguous().view(torch.float8_e4m3fn).float() + scale = rows[:, _NOPE : _NOPE + 16].contiguous().view(torch.float32) + nope = (codes.view(-1, _NOPE // _TILE, _TILE) * scale[..., None]).view(-1, _NOPE) + rope = rows[:, _NOPE + 16 :].contiguous().view(torch.bfloat16) + return torch.cat([nope.to(torch.bfloat16), rope], dim=1) + + +def _selections(pos: torch.Tensor, g: torch.Generator) -> torch.Tensor: + """int32 `[rows, 2048]` request-local selections for rows at positions `pos`: below + index_topk every earlier position in order then -1, else one draw from each of 2048 equal + strata of [0, p] with p itself in the last, so distinct.""" + p = pos.long()[:, None] + col = torch.arange(_TOPK, device=pos.device)[None] + lo = col * (p + 1) // _TOPK + hi = (col + 1) * (p + 1) // _TOPK + draw = lo + (torch.rand(pos.numel(), _TOPK, generator=g, device=pos.device) * (hi - lo)).long() + draw = torch.minimum(draw, hi - 1) + draw[:, -1] = pos + return torch.where(p < _TOPK, torch.where(col <= p, col, -1), draw).to(torch.int32) + + +@pytest.fixture(scope="module", params=WORLDS) +def env(request): + """The pool, the target, and the KV-capacity estimate's step already run through the + per-step choice, at the world's heads; every gather the module's tests make is recorded + (observe only).""" + if not torch.cuda.is_available(): + pytest.skip("needs a GPU") + from tensorrt_llm._utils import get_sm_version + + if get_sm_version() != 103: + pytest.skip("the mistral_large4 modeling_v2 targets are certified on sm_103 only") + dev = torch.device("cuda") + g = torch.Generator(device=dev).manual_seed(_SEED) + pool = torch.empty((_BLOCKS, len(_OWNERS), _TPB, _ROW_BYTES), dtype=torch.uint8, device=dev) + pool.fill_(0xFF) + rows = {layer: _OWNERS.index(layer) for layer in _WRITTEN} + for layer, row in rows.items(): + for lo in range(0, _BLOCKS, 1 << 15): + hi = min(lo + (1 << 15), _BLOCKS) + drawn = _dev_rows((hi - lo) * _TPB, g, nope=layer == 35) + pool[lo:hi, row] = drawn.view(hi - lo, _TPB, _ROW_BYTES) + del drawn + gathers = [] + real_gather = M.ds_mla_dequant_gather + + def recording_gather(pool_, row, kv_req, kv_pos, block_table, out): + gathers.append( + SimpleNamespace(rows=int(kv_req.shape[0]), out_ptr=out.data_ptr(), pool_row=row) + ) + return real_gather(pool_, row, kv_req, kv_pos, block_table, out) + + with pytest.MonkeyPatch.context() as mp: + mp.setattr(M, "ds_mla_dequant_gather", recording_gather) + e = SimpleNamespace( + dev=dev, pool=pool, kv=pool.view(-1, _TPB, 1, _ROW_BYTES), rows=rows, + target=_target(_MODEL_HEADS // request.param), gathers=gathers, world=request.param, + heads=_MODEL_HEADS // request.param, + ) # fmt: skip + assert e.target._ws_buf is None + # The KV-capacity estimate's step at 1M: one fresh 32768-token prompt. + e.estimate = _step(e, [(0, 32768)], seed=1) + e.estimate_ctx = e.target._ctx_workspace(e.estimate.step, e.estimate.tc, dev) + e.buf = e.target._ws_buf + assert e.buf.shape == (_B, _LAT_DIM) and e.estimate_ctx.ws_buf is not None + yield e + assert e.target._ws_buf is e.buf + + +def _step(env, contexts, gens=(), seed: int = 0, **extra) -> SimpleNamespace: + """A step of context requests of (cached, new) tokens with generation rows of the given KV + lengths behind them: each request's pages distinct blocks of the pool, its table row + padded to the engine's width, a query per row (this rank's heads drawn -- 16 at tp4, 8 at + tp8 -- the rest zero) and its selection.""" + dev, heads = env.dev, env.heads + g = torch.Generator(device=dev).manual_seed(_SEED + seed) + kv_lens = [c + n for c, n in contexts] + list(gens) + pages = [-(-k // _TPB) for k in kv_lens] + assert sum(pages) <= _BLOCKS and max(pages) <= _TABLE[1] + order = torch.randperm(_BLOCKS, generator=g, device=dev).to(torch.int32) + table = torch.zeros(_TABLE, dtype=torch.int32, device=dev) + used = 0 + for r, n in enumerate(pages): + table[r, :n] = order[used : used + n] + used += n + tc = sum(n for _, n in contexts) + req_idx = torch.cat( + [torch.full((n,), r, dtype=torch.int32, device=dev) for r, (_, n) in enumerate(contexts)] + + [torch.arange(len(contexts), len(kv_lens), dtype=torch.int32, device=dev)] + ) + pos = torch.cat( + [torch.arange(c, c + n, device=dev) for c, n in contexts] + + [torch.tensor(gens, device=dev, dtype=torch.int64) - 1] + ) + q = torch.zeros(pos.numel(), _KERNEL_HEADS, _LAT_DIM, dtype=torch.bfloat16, device=dev) + q[:, :heads] = torch.randn(pos.numel(), heads, _LAT_DIM, generator=g, device=dev) * 0.1 + step = _host_step(contexts, gens, **extra) + step = {k: v.to(dev) if isinstance(v, torch.Tensor) else v for k, v in step.items()} + step.update(block_table=table, tokens_per_block=_TPB, latent_pool=env.pool, latent_kv=env.kv) + return SimpleNamespace( + step=step, contexts=list(contexts), gens=list(gens), kv_lens=kv_lens, tc=tc, + rows=pos.numel(), req_idx=req_idx, pos=pos, topk=_selections(pos, g), q=q, table=table, + ) # fmt: skip + + +def _request_kv(env, s, layer: int, r: int) -> torch.Tensor: + """Request r's KV rows 0..kv_len-1 of `layer`, bf16, read through its block table.""" + n = s.kv_lens[r] + out = torch.empty(n, _LAT_DIM, dtype=torch.bfloat16, device=env.dev) + for lo in range(0, n, 1 << 19): + p = torch.arange(lo, min(lo + (1 << 19), n), device=env.dev) + out[lo : lo + p.numel()] = _dequant( + env.pool[s.table[r, p // _TPB].long(), env.rows[layer], p % _TPB] + ) + return out + + +def _unbounded(env, s, layer: int) -> torch.Tensor: + """The workspace path's computation without its bound: every context request's rows in one + bf16 buffer of N rows, the selections offset into it, the same prefill kernel.""" + ws = torch.cat([_request_kv(env, s, layer, r) for r in range(len(s.contexts))]) + off = _cumsum([c + n for c, n in s.contexts], env.dev)[:-1] + sel = s.topk[: s.tc] + idx = torch.where(sel >= 0, sel + off[s.req_idx[: s.tc].long()][:, None], sel).to(torch.int32) + out = torch.empty(s.tc, env.heads, _NOPE, dtype=torch.bfloat16, device=env.dev) + flash_mla_sparse_prefill(s.q[: s.tc], ws, idx, _SCALE, out) + return out + + +def _fp32_check(env, s, layer: int, out: torch.Tensor, rows: torch.Tensor) -> float: + """`out`'s rows `rows` against an fp32 softmax over each row's selected keys read straight + from the pool through its request's table row; returns the relative L2 error.""" + sel = s.topk[rows].long() + req = s.req_idx[rows].long() + valid = sel >= 0 + key = sel.clamp_min(0) + blocks = s.table[req[:, None], key // _TPB].long() + k = _dequant(env.pool[blocks, env.rows[layer], key % _TPB].view(-1, _ROW_BYTES)).float() + k = k.view(rows.numel(), _TOPK, _LAT_DIM) + with true_fp32_matmul(): + logits = torch.einsum("rhd,rkd->rhk", s.q[rows, : env.heads].float(), k) * _SCALE + logits = logits.masked_fill(~valid[:, None, :], float("-inf")) + exact = torch.einsum("rhk,rkd->rhd", torch.softmax(logits, -1), k[..., :_NOPE]) + got = out[rows].float() + assert bool(torch.isfinite(got).all()), "a sampled row read an unwritten pool byte" + return float((got - exact).norm() / exact.norm()) + + +def _sample(rows: int, g_seed: int, must=()) -> torch.Tensor: + g = torch.Generator().manual_seed(g_seed) + picked = torch.randperm(rows, generator=g)[:_SAMPLED].tolist() + keep = sorted(set(picked) | {r for r in must if 0 <= r < rows}) + return torch.tensor(keep, device="cuda") + + +def _attend(env, s, layer: int, ctx=None, out=None): + """`_attn_ctx` for `layer` over the step's context rows: the step's path (made here unless + given) and the output (NaN-filled first unless given).""" + if ctx is None: + ctx = env.target._ctx_workspace(s.step, s.tc, env.dev) + if out is None: + out = torch.full((s.rows, env.heads, _NOPE), float("nan"), dtype=torch.bfloat16, + device=env.dev) # fmt: skip + L = SimpleNamespace(kv_source=_SOURCES[layer]) + env.target._attn_ctx(s.step, L, s.q[: s.tc], out[: s.tc], s.req_idx, s.topk, ctx) + return ctx, out + + +def _agreement(got: torch.Tensor, ref: torch.Tensor) -> tuple[float, int]: + rel = float((got.float() - ref.float()).norm() / ref.float().norm()) + return rel, int((got.view(torch.int16) != ref.view(torch.int16)).sum()) + + +def _check_decode_path(env, s, ctx, out, layer: int, label: str) -> None: + """A decode-path output: finite, within the agreement of the unbounded workspace + computation (bitwise in every full sub-batch), and the sampled rows within the fp32 band.""" + lat = out[: s.tc] + assert bool(torch.isfinite(lat).all()), f"{label}: a context row read an unwritten pool byte" + ref = _unbounded(env, s, layer) + rel, differ = _agreement(lat, ref) + rows_differ = [ + int((lat[lo:hi].view(torch.int16) != ref[lo:hi].view(torch.int16)).flatten(1).any(1).sum()) + for lo, hi in ctx.sub_batches + ] + edges = [lo for lo, _ in ctx.sub_batches] + [hi - 1 for _, hi in ctx.sub_batches] + err = _fp32_check(env, s, layer, out, _sample(s.tc, layer, edges)) + print(f"CTXBOUND {label} layer {layer}: decode path, sub-batches " + f"{[hi - lo for lo, hi in ctx.sub_batches]}, against the unbounded workspace rel L2 " + f"{rel:.3e} ({differ} of {lat.numel()} elements differ; rows differing per sub-batch " + f"{rows_differ}); fp32 rel L2 {err:.3e}") # fmt: skip + assert rel <= _AGREE, (label, rel) + for (lo, hi), n in zip(ctx.sub_batches, rows_differ): + if hi - lo == _SUB: + assert n == 0, (label, lo, hi, n) + assert err < _FP32_BAND, (label, err) + + +def _measure(fn, iters: int = 5) -> tuple[float, int]: + """(ms per call, peak bytes above the allocation before the call) of a warm `fn`.""" + fn() + torch.cuda.synchronize() + base = torch.cuda.memory_allocated() + torch.cuda.reset_peak_memory_stats() + fn() + torch.cuda.synchronize() + peak = torch.cuda.max_memory_allocated() - base + start, end = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True) + start.record() + for _ in range(iters): + fn() + end.record() + torch.cuda.synchronize() + return start.elapsed_time(end) / iters, peak + + +def test_same_packed_step_at_b_and_b_plus_one(env): + """Four chunks packed to N = B take the workspace path; the same step with its last chunk + one row longer (N = B + 1, 32768 rows) takes the decode path in four sub-batches of 8192. + The 32767 shared rows' outputs are the same.""" + past = _step(env, _PAST_B, seed=2) + assert sum(past.kv_lens) == _B + 1 + ctx_past, out_past = _attend(env, past, 1) + assert ctx_past.ws_buf is None + assert ctx_past.sub_batches == [(lo, lo + _SUB) for lo in range(0, 32768, _SUB)] + _check_decode_path(env, past, ctx_past, out_past, 1, "N = B + 1") + + # The same step, its last row dropped: the same pool, table, queries and selections. + at = SimpleNamespace(**vars(past)) + at.contexts = list(_AT_B) + at.kv_lens = [c + n for c, n in at.contexts] + at.tc = at.rows = past.tc - 1 + at.step = dict(past.step, **_host_step(at.contexts)) + at.step = {k: v.to(env.dev) if isinstance(v, torch.Tensor) else v for k, v in at.step.items()} + assert sum(at.kv_lens) == _B + gathered = len(env.gathers) + ctx_at, out_at = _attend(env, at, 1) + assert ctx_at.ws_buf is not None and ctx_at.ws_buf.data_ptr() == env.buf.data_ptr() + assert [x.rows for x in env.gathers[gathered:]] == [_B] + rel, differ = _agreement(out_at[: at.tc], out_past[: at.tc]) + print(f"CTXBOUND N = B (workspace) against N = B + 1 (decode) on the {at.tc} shared rows: " + f"rel L2 {rel:.3e}, {differ} elements differ") # fmt: skip + assert rel <= _AGREE and differ == 0, (rel, differ) + err = _fp32_check(env, at, 1, out_at, _sample(at.tc, 7, (0, 16383, 16384, at.tc - 1))) + assert err < _FP32_BAND, err + # The workspace holds every request's rows, request after request. + off = 0 + for r, n in enumerate(at.kv_lens): + assert torch.equal(env.buf[off : off + n], _request_kv(env, at, 1, r)), r + off += n + + +def test_ragged_last_chunk_of_a_1048559_token_prompt(env): + """The envelope gate's 1048559-token prompt's last chunk, 32751 rows behind 1015808 cached + keys: sub-batches 8192 x 3 + 8175, on two schedulers the 72 layers would share.""" + s = _step(env, [(_PREFIX_1M, 32751)], seed=3) + gathered = len(env.gathers) + ctx, out = _attend(env, s, 1) + assert [hi - lo for lo, hi in ctx.sub_batches] == [8192, 8192, 8192, 8175] + assert sorted(ctx.sched) == [8175, 8192] + assert all(m.have_initialized for m in ctx.sched.values()) + assert (ctx.sched[8192].config.b, ctx.sched[8175].config.b) == (8192, 8175) + _check_decode_path(env, s, ctx, out, 1, "ragged 32751 behind 1015808") + # The next layer reuses the filled schedulers. + again = _attend(env, s, 1, ctx=ctx)[1] + assert torch.equal(again[: s.tc], out[: s.tc]) + assert len(env.gathers) == gathered + + +def test_q_split_step_behind_a_1m_prefix_takes_the_decode_path(env): + """The q-split step of a 1M prompt's last chunk: 32768 context rows with the indexer running + and rank 0's share its first 32768 / world rows (8192 at tp4, 4096 at tp8), after the gather + every row's selection in the buffer. It takes the decode path and attends every row. Its + per-layer peak memory stays under the KV-capacity estimate's step on the workspace path, + whose peak sizes the pool.""" + share = 32768 // env.world + s = _step( + env, [(_PREFIX_1M, 32768)], seed=4, + indexer_prefill_q_split=(0, share, [share] * env.world), + ) # fmt: skip + assert s.step["skip_ctx"] is False and s.tc >= 8192 + gathered = len(env.gathers) + ctx, out = _attend(env, s, 1) + assert ctx.ws_buf is None and len(ctx.sub_batches) == 4 and len(env.gathers) == gathered + _check_decode_path(env, s, ctx, out, 1, "q-split 32768 behind 1015808") + assert bool(torch.isfinite(out[share : s.tc]).all()) + + est = env.estimate + lat_est = torch.empty(est.tc, env.heads, _NOPE, dtype=torch.bfloat16, device=env.dev) + lat_dec = torch.empty(s.tc, env.heads, _NOPE, dtype=torch.bfloat16, device=env.dev) + L = SimpleNamespace(kv_source=1) + ws_ms, ws_peak = _measure( + lambda: env.target._attn_ctx( + est.step, L, est.q, lat_est, est.req_idx, est.topk, env.estimate_ctx + ) + ) + dec_ms, dec_peak = _measure( + lambda: env.target._attn_ctx(s.step, L, s.q, lat_dec, s.req_idx, s.topk, ctx) + ) + print(f"CTXBOUND per layer: the estimate's step (32768 fresh rows, workspace path) " + f"{ws_ms:.3f} ms, peak {ws_peak / _GIB:.3f} GiB above the held workspace; " + f"32768 rows behind 1015808 (decode path) {dec_ms:.3f} ms, peak " + f"{dec_peak / _GIB:.3f} GiB; margin {(ws_peak - dec_peak) / _GIB:.3f} GiB") # fmt: skip + assert dec_peak <= ws_peak, (dec_peak, ws_peak) + + +def test_mixed_steps_and_layer_35s_sharers(env): + """Generation rows behind the context rows on each path: the context attention writes the + context rows only, and the generation rows' own call (`_attn_gen`, the native decode at the + rank's heads) reads the + pool. Layer 39 attends over layer 35's rows: on a step whose 2N fits B, where layer 35 + gathered them, the N rows kept after the step's first N, with no gather of its own, layer 36 + having gathered into the first N; on a step of N past B / 2, from its own gather of row 35 + into the first N rows, which layer 36 overwrote; on the decode path, from the pool. Each is + bitwise layer 35's output.""" + target = env.target + for label, contexts, gens in ( + ("decode", [(_PREFIX_1M, 16384)], (524288, 3001, 17)), + ("workspace, kept", [(4096, 640), (0, 300)], (3001, 17)), + ("workspace, gathered again", [(_B // 2, 640), (0, 300)], (3001, 17)), + ): + s = _step(env, contexts, gens, seed=5) + n_kv = sum(c + n for c, n in contexts) + ctx = target._ctx_workspace(s.step, s.tc, env.dev) + assert (ctx.ws_buf is None) == (label == "decode") + if ctx.ws_buf is not None: + assert ctx.ws_buf.data_ptr() == env.buf.data_ptr() + assert (ctx.kept is not None) == (label == "workspace, kept") == (2 * n_kv <= _B) + outs = {} + for layer in (35, 36, 39): + gathered = len(env.gathers) + ctx, out = _attend(env, s, layer, ctx=ctx) + assert bool(torch.isnan(out[s.tc :].float()).all()), (label, layer) + wrote = [(x.rows, x.out_ptr) for x in env.gathers[gathered:]] + if ctx.ws_buf is None: + assert wrote == [], (label, layer) + else: + # Where the source's rows are now, request after request: layer 35's in the + # kept rows when the step has them, every other gather's in the first N. + src = _SOURCES[layer] + kept = src == 35 and ctx.kept is not None + ws = ctx.kept if kept else ctx.ws_buf + if kept: + assert ws.data_ptr() == env.buf[n_kv].data_ptr() + expect = [] if kept and layer == 39 else [(n_kv, ws.data_ptr())] + assert wrote == expect, (label, layer) + off = 0 + for r in range(len(contexts)): + n = s.kv_lens[r] + assert torch.equal(ws[off : off + n], _request_kv(env, s, src, r)) + off += n + slots = convert_req_index_to_global( + s.req_idx[s.tc :], s.table, s.topk[s.tc :], _TPB, _TOPK, + len(_OWNERS) * _TPB, target.core._pool_row[_SOURCES[layer]], + ) # fmt: skip + L = SimpleNamespace(kv_source=_SOURCES[layer]) + target._attn_gen(s.step, L, s.q[s.tc :], out[s.tc :], slots, get_mla_metadata()[0]) + outs[layer] = out + assert torch.equal(outs[39], outs[35]), label + assert not torch.equal(outs[36], outs[35]), label + for layer in (35, 36): + rows = _sample(s.rows, layer, range(s.tc - 2, s.rows)) + err = _fp32_check(env, s, layer, outs[layer], rows) + print(f"CTXBOUND mixed step, {label} path, layer {layer}: {s.tc} context rows and " + f"{len(gens)} generation rows, fp32 rel L2 {err:.3e}") # fmt: skip + assert err < _FP32_BAND, (label, layer, err) + if label == "decode": + _check_decode_path(env, s, ctx, outs[35], 35, "mixed, layer 35") + else: + # Each context row is the same kernel over its own request's gather. + start = 0 + for r, (cached, new) in enumerate(contexts): + kv = _request_kv(env, s, 35, r) + own = torch.empty(new, env.heads, _NOPE, dtype=torch.bfloat16, device=env.dev) + flash_mla_sparse_prefill( + s.q[start : start + new], kv, s.topk[start : start + new], _SCALE, own + ) + assert torch.equal(outs[35][start : start + new], own), r + start += new + + +def test_packed_four_chunks_at_1m(env): + """Four chunks of 16384 / 8192 / 4096 / 4096 rows packed behind 1032192 / 524288 / 0 / 2048 + cached keys (N = 1591296): the path the step picks, its per-layer time and peak memory.""" + contexts = [(1032192, 16384), (524288, 8192), (0, 4096), (2048, 4096)] + s = _step(env, contexts, seed=6) + ctx, out = _attend(env, s, 1) + path = "workspace" if ctx.ws_buf is not None else "decode" + assert path == "decode" and sum(s.kv_lens) == 1591296 + _check_decode_path(env, s, ctx, out, 1, "packed four chunks") + lat = torch.empty(s.tc, env.heads, _NOPE, dtype=torch.bfloat16, device=env.dev) + L = SimpleNamespace(kv_source=1) + ms, peak = _measure(lambda: env.target._attn_ctx(s.step, L, s.q, lat, s.req_idx, s.topk, ctx)) + print(f"CTXBOUND packed four-chunk step (cached 1032192/524288/0/2048, new " + f"16384/8192/4096/4096, N {sum(s.kv_lens)}): {path} path, {ms:.3f} ms per layer, " + f"peak {peak / _GIB:.3f} GiB above the held workspace") # fmt: skip + + +def test_workspace_held_and_no_gather_past_b(env): + """After every step above and a sequence of its own (sixteen requests filling B exactly, + then B + 1, then a 1M chunk), the workspace is the estimate's step's allocation, every + gather wrote into it -- a step's first N rows, or layer 35's N rows kept after them on a + step whose 2N fits B --, and no step past B reached the gather.""" + for contexts, path in ( + ([(30720, 2048)] * 16, "workspace"), + ([(30720, 2048)] * 15 + [(30721, 2048)], "decode"), + ([(_PREFIX_1M, 32768)], "decode"), + ([(5, 300)], "workspace"), + ): + s = _step(env, contexts, seed=7) + gathered = len(env.gathers) + ctx, out = _attend(env, s, 36) + assert (ctx.ws_buf is None) == (path == "decode"), (contexts[0], path) + assert len(env.gathers) - gathered == (path == "workspace") + assert bool(torch.isfinite(out[: s.tc]).all()) + assert env.target._ws_buf is env.buf + assert env.gathers and all(x.rows <= _B for x in env.gathers) + base, row_bytes = env.buf.data_ptr(), _LAT_DIM * env.buf.element_size() + kept = [x for x in env.gathers if x.out_ptr != base] + for x in kept: + assert x.out_ptr == base + x.rows * row_bytes and 2 * x.rows <= _B, vars(x) + assert x.pool_row == _OWNERS.index(35), vars(x) + print(f"CTXBOUND {len(env.gathers)} gathers, at most {max(x.rows for x in env.gathers)} " + f"rows, all into the workspace allocated on the estimate's step ({len(kept)} of " + f"layer 35 into the rows kept after a step's)") # fmt: skip diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_tp4_engine_config.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_tp4_engine_config.py new file mode 100644 index 000000000000..8c634f9cce57 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_tp4_engine_config.py @@ -0,0 +1,1358 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Mistral-Large-4-1T-A52B-NVFP4 at tp4 and at tp8: the engine's configuration with the routed +experts tensor parallel, as a served engine reaches it, on four ranks of one GB300 node and on +eight ranks over two nodes of four. + +No GPU and no weights. The checkpoint goes through the engine's own configuration path with the +LLM-API options of the tp4 and tp8 core units (four or eight ranks, tensor parallel throughout, +attention DP off, every routed expert cut along its width four or eight ways and none +expert-parallel, 32768 tokens and 16 requests over the whole engine, the latent KV cache at its +default -- the checkpoint's fp8_kv_cache, the fp8_ds_mla pool of MistralLarge4CacheManager): + +* the LLM API front end, ``TorchLlmArgs`` and ``ModelLoader._update_from_hf_quant_config`` + (``llmapi/llm_utils.py``), which settle the key the accuracy harness looks its anchor up by; +* each rank, ``_load_config_and_create_checkpoint_loader`` (``py_executor_creator``) then the + model loader's ``_load_and_validate_config``, which builds the ``ModelConfig`` and asks the + modeling_v2 router for a target. The router's ``route`` is replaced by a recorder, so what it + is handed is observed rather than judged. + +The same deployment is reached from ``tensor_parallel_size`` alone: at tp4 as a gate command +carrying only ``--tp_size=4`` builds it, at tp8 as the boot gate's command (``--tp_size=8`` and +no MoE split) is parsed and turned into ``LLM`` keyword arguments by the tree's own +``examples/llm-api/quickstart_advanced.py`` (``parse_arguments`` and ``setup_llm``), plus the +``modeling_v2`` argument the harness adds. ``Mapping`` gives the MoE the whole TP width when +neither MoE split is stated. + +No gate states ``gpus_per_node``: ``TorchLlmArgs`` takes it from ``torch.cuda.device_count()`` on +rank 0, which on a GB300 node of this cluster is 4. The count is set to 4 where the arguments are +built rather than read from the test host, which may see no device or a whole node. What the +engine reads of the MoE split and of the node layout on its way to a modeling_v2 target -- the +mapping's ranks and groups, the MoE token budget and load balancer, the autotuner and the FP4 MoE +runner's tuning, the weight-sharing qualification, the causal-LM shell's lm_head, and at tp8 the +MNNVL communicator splits and the device each rank selects -- and the largest step its warmup and +KV-capacity estimation send are pinned on the engine's own host code, the ranks simulated in one +process where they exchange anything. + +The checkpoint is Mistral-native, as published (params.json, tekken.json, consolidated +safetensors), and is loaded with checkpoint_format mistral: the gates' LLM-API options +state it and the boot gate's command passes --checkpoint_format=mistral. The checkpoint +at tep4, at dep4 and at tep8 is checked to resolve and route as it did before this target. The +checkpoint is looked up under ``LLM_MODELS_ROOT`` (as ``Mistral-Large-4-1T-A52B-NVFP4`` or +``mistral-large-4-nvfp4``) by its params.json, and its tests are skipped where it is absent. +""" + +from __future__ import annotations + +import copy +import dataclasses +import importlib.util +import json +import math +import sys +import threading +from pathlib import Path +from types import SimpleNamespace + +import pytest +import torch +import yaml +from test_common.llm_data import llm_models_root + +from tensorrt_llm._ipc_utils import IpcMemory +from tensorrt_llm._torch.attention.backends.sparse.mistral_large4 import MistralLarge4CacheManager +from tensorrt_llm._torch.autotuner import AutoTuner, DistributedTuningStrategy +from tensorrt_llm._torch.custom_ops.trtllm_gen_custom_ops import FP4BlockScaleMoERunner +from tensorrt_llm._torch.distributed import ops as distributed_ops +from tensorrt_llm._torch.distributed.communicator import Distributed, MPIDist +from tensorrt_llm._torch.distributed.mnnvl_memory import MnnvlMemory +from tensorrt_llm._torch.modeling_v2 import _router_index +from tensorrt_llm._torch.modeling_v2._router_index import NoTarget +from tensorrt_llm._torch.modeling_v2.models.mistral_large4 import routing +from tensorrt_llm._torch.models.modeling_utils import DecoderModel, DecoderModelForCausalLM +from tensorrt_llm._torch.modules.linear import TensorParallelMode +from tensorrt_llm._torch.moe.fused_moe.moe_load_balancer import maybe_create_moe_load_balancer +from tensorrt_llm._torch.moe.fused_moe.routing import RoutingMethodType +from tensorrt_llm._torch.pyexecutor import model_loader, py_executor_creator +from tensorrt_llm._torch.pyexecutor._util import CacheCost, KvCacheCreator, get_kv_cache_manager_cls +from tensorrt_llm._torch.pyexecutor.engine.runners.decoder.dummy_requests import DummyRequestFactory +from tensorrt_llm._torch.pyexecutor.engine.runners.decoder.warmup import ( + DecoderWarmup, + WarmupAgreement, +) +from tensorrt_llm._torch.pyexecutor.model_loader import ModelLoader as RankModelLoader +from tensorrt_llm._torch.pyexecutor.resource_manager import ResourceManagerType +from tensorrt_llm._torch.pyexecutor.scheduler.adp_router import ADPRouter, DefaultADPRouter +from tensorrt_llm._torch.utils import ActType_TrtllmGen +from tensorrt_llm._torch.weight_sharing.post_transform_profiles import ( + PostTransformConfigIdentity, + PostTransformQualificationReason, + PostTransformRuntimeConfig, +) +from tensorrt_llm.executor import base_worker +from tensorrt_llm.executor.base_worker import BaseWorker +from tensorrt_llm.llmapi.llm_args import ( + AttentionDpConfig, + LoadFormat, + TorchLlmArgs, + _ParallelConfig, +) +from tensorrt_llm.llmapi.llm_utils import ModelLoader as FrontEndModelLoader +from tensorrt_llm.mapping import Mapping, MpiTopology +from tensorrt_llm.models.modeling_utils import QuantAlgo +from tensorrt_llm.usage import usage_lib + +pytestmark = pytest.mark.cpu_only + +_SM103 = (10, 3) +# Devices per GB300 node on this cluster: one node holds the four ranks, two hold the eight. +_GPUS_PER_NODE = 4 +# The gates' max_seq_len, which is also their max_num_tokens: 32768. +_MAX_SEQ_LEN = 32768 + +# The topologies. tp4 and tp8 are the core units' (spec-tp4/options/core.yaml, +# spec-tp8/options/core.yaml); a gate command carrying only --tp_size=4 states no MoE split and no +# attention DP, and quickstart_advanced.py spells that as the -1 sentinels. tep4 is where the +# tep target already serves, tep8 the same target on the same eight ranks as tp8 +# (spec-tep8/options/core.yaml), dep4 the NVFP4 attention-DP target (spec-dep4/options/core.yaml). +_TP4 = dict( + tensor_parallel_size=4, + moe_tensor_parallel_size=4, + moe_expert_parallel_size=1, + enable_attention_dp=False, +) +_TP4_BARE = dict(tensor_parallel_size=4) +_TP4_QUICKSTART = dict( + tensor_parallel_size=4, + moe_expert_parallel_size=-1, + moe_tensor_parallel_size=-1, + moe_cluster_parallel_size=-1, + enable_attention_dp=False, +) +_TP8 = dict( + tensor_parallel_size=8, + moe_tensor_parallel_size=8, + moe_expert_parallel_size=1, + enable_attention_dp=False, +) +_TEP4 = dict(tensor_parallel_size=4, moe_expert_parallel_size=4, enable_attention_dp=False) +_TEP8 = dict(tensor_parallel_size=8, moe_expert_parallel_size=8, enable_attention_dp=False) +_DEP4 = dict(tensor_parallel_size=4, moe_expert_parallel_size=4, enable_attention_dp=True) + +# The tp8 boot gate's quickstart command after --model_dir (gates/geometry/geometry.core.tp8.json): +# the checkpoint's Mistral-native format, --tp_size=8 and no MoE split, the core unit's CUDA-graph +# capture list, task.yaml's runtime flags. No --kv_cache_dtype: the latent cache is the +# checkpoint's default (quickstart's "auto"), the fp8_ds_mla pool. +_GATE_ARGV = ( + "--checkpoint_format=mistral", + "--max_tokens=2", + "--tp_size=8", + "--use_cuda_graph", + "--cuda_graph_batch_sizes", + "1", + "2", + "4", + "8", + "16", + "--disable_kv_cache_reuse", + "--max_seq_len=32768", + "--max_batch_size=16", + "--max_num_tokens=32768", + "--cuda_graph_padding_enabled", +) +_QUICKSTART = Path(__file__).resolve().parents[7] / "examples/llm-api/quickstart_advanced.py" + + +def _options(topology: dict, max_seq_len: int) -> dict: + """The LLM-API options of a Mistral Large 4 accuracy gate on ``topology``, the checkpoint in its + Mistral-native format, max_num_tokens = max_seq_len, with the latent KV cache at its + default: the checkpoint's fp8_kv_cache, the fp8_ds_mla pool.""" + return dict( + checkpoint_format="mistral", + **topology, + max_batch_size=16, + max_num_tokens=max_seq_len, + max_seq_len=max_seq_len, + enable_chunked_prefill=False, + cuda_graph_config=dict(enable_padding=True, max_batch_size=16), + kv_cache_config=dict(enable_block_reuse=False, free_gpu_memory_fraction=0.85), + ) + + +_NVFP4_NAMES = ("Mistral-Large-4-1T-A52B-NVFP4", "mistral-large-4-nvfp4") + +# The GSM8K reference rows, and the key the harness reads them by. +_GSM8K_REFERENCES = ( + Path(__file__).resolve().parents[6] / "integration/defs/accuracy/references/gsm8k.yaml" +) +_MODEL_NAME = "mistralai/Mistral-Large-4-1T-A52B" + +_TP4_TARGET = "ModelingV2MistralLarge4Nvfp4Sm103Tp" +_OTHER_TARGETS = ( + "ModelingV2MistralLarge4Nvfp4Sm103Tep", + "ModelingV2MistralLarge4Nvfp4Sm103Dep", +) + +# By world size: the fixtures of the core unit as its gates spell it with the MoE split stated +# (core.yaml; at tp8 also the boot gate's command, which states none), and of the +# expert-parallel deployment on the same ranks. +_SPELLINGS = {4: ("tp4",), 8: ("tp8", "tp8_gate")} +_TEP = {4: "tep4", 8: "tep8"} +# By world size: the deployments on the same ranks whose KV cost the core unit's is compared with. +_KV_PEERS = {4: ("tep4", "dep4"), 8: ("tep8",)} +# By world size: the query heads a rank holds, and its width slice of every routed expert's 6144 +# (w1/w3 rows, w2 columns) with w2's block-scale columns of 384, one per 16 inputs. +_HEADS = {4: 16, 8: 8} +_SLICE = {4: (1536, 96), 8: (768, 48)} +_WORLDS = pytest.mark.parametrize("world", [4, 8], ids=["tp4", "tp8"]) + + +def _checkpoint_dir(names: tuple[str, ...]) -> Path: + """The first of ``names`` under LLM_MODELS_ROOT holding a Mistral-native params.json.""" + root = llm_models_root() + for name in names: + if root is not None and (root / name / "params.json").is_file(): + return root / name + pytest.skip(f"none of {names} is under LLM_MODELS_ROOT ({root})") + + +def _llm_args(devices: int = _GPUS_PER_NODE, **kwargs) -> TorchLlmArgs: + """``TorchLlmArgs`` as the LLM constructor builds it on rank 0 of a node with ``devices`` + visible devices (outside Ray).""" + with pytest.MonkeyPatch.context() as mp: + mp.delenv("RAY_LOCAL_WORLD_SIZE", raising=False) + mp.setattr(torch.cuda, "device_count", lambda: devices) + return TorchLlmArgs(**kwargs) + + +def _gate_kwargs(checkpoint_dir: Path, argv: tuple[str, ...]) -> dict: + """The ``LLM`` keyword arguments of a quickstart gate command: the tree's own + quickstart_advanced.py parses ``argv`` and builds them in ``setup_llm``, and the gate harness + adds ``modeling_v2``. Every KV-cache key of core.yaml has a quickstart flag, so the harness + hands over no other.""" + if not _QUICKSTART.is_file(): + pytest.skip(f"{_QUICKSTART} is not in this checkout") + spec = importlib.util.spec_from_file_location("quickstart_advanced", _QUICKSTART) + quickstart = importlib.util.module_from_spec(spec) + spec.loader.exec_module(quickstart) + built = {} + + def llm(**kwargs): + built.update(kwargs) + return SimpleNamespace() + + with pytest.MonkeyPatch.context() as mp: + mp.setattr(sys, "argv", ["quickstart_advanced.py", f"--model_dir={checkpoint_dir}", *argv]) + args = quickstart.parse_arguments() + mp.setattr(quickstart, "LLM", llm) + quickstart.setup_llm(args) + return dict(built, modeling_v2="require") + + +class _Routed: + """What the recording `route` returns: no target is built here.""" + + +def _route(pretrained_config, mapping): + """`route` for this config on ``mapping``, on sm_103 without speculative decoding.""" + model_config = SimpleNamespace( + pretrained_config=pretrained_config, mapping=mapping, spec_config=None + ) + return routing.route(model_config, None, _SM103) + + +def _resolve(checkpoint_dir: Path, llm_args: TorchLlmArgs) -> SimpleNamespace: + """The configuration a served engine reaches for this checkpoint, front end then rank.""" + model = str(checkpoint_dir) + prequantized = FrontEndModelLoader(llm_args)._update_from_hf_quant_config() + + routed = [] + + def route(model_config, llm_args, compute_capability): + pretrained_config = model_config.pretrained_config + routed.append( + SimpleNamespace( + pretrained_config=pretrained_config, + quantization_config=copy.deepcopy(pretrained_config.quantization_config), + mapping=model_config.mapping, + compute_capability=compute_capability, + spec_config=model_config.spec_config, + ) + ) + return _Routed + + with pytest.MonkeyPatch.context() as mp: + # The rank's config pass pins the CUDA memory fraction first; there is no device here. + mp.setattr(torch.cuda, "set_per_process_memory_fraction", lambda *args, **kwargs: None) + mp.setattr(_router_index, "_device_compute_capability", lambda: _SM103) + mp.setattr(_router_index, "registered_routes", lambda: {"MistralLarge4ForCausalLM": route}) + # One process stands in for every rank, so the rank exchange is the local resolution. + mp.setattr( + model_loader, "modeling_v2_resolve_on_all_ranks", _router_index.modeling_v2_resolve + ) + llm_args, checkpoint_loader = py_executor_creator._load_config_and_create_checkpoint_loader( + llm_args, model + ) + _, max_num_tokens, max_seq_len, max_batch_size = llm_args.get_runtime_sizes() + mapping = llm_args.parallel_config.to_mapping() + model_config = RankModelLoader( + llm_args=llm_args, + mapping=mapping, + spec_config=llm_args.speculative_config, + sparse_attention_config=llm_args.sparse_attention_config, + max_num_tokens=max_num_tokens, + max_seq_len=max_seq_len, + )._load_and_validate_config(model, checkpoint_loader) + return SimpleNamespace( + path=checkpoint_dir, + llm_args=llm_args, + checkpoint_loader=checkpoint_loader, + prequantized=prequantized, + model_config=model_config, + mapping=mapping, + routed=routed, + max_num_tokens=max_num_tokens, + max_seq_len=max_seq_len, + max_batch_size=max_batch_size, + ) + + +def _resolve_options(names: tuple[str, ...], topology: dict, max_seq_len: int) -> SimpleNamespace: + path = _checkpoint_dir(names) + options = _options(topology, max_seq_len) + return _resolve(path, _llm_args(model=str(path), modeling_v2="require", **options)) + + +@pytest.fixture(scope="module") +def tp4(): + """The tp4 accuracy and serve gates' engine: spec-tp4/options/core.yaml.""" + return _resolve_options(_NVFP4_NAMES, _TP4, _MAX_SEQ_LEN) + + +@pytest.fixture(scope="module") +def tp4_spellings(): + """``tensor_parallel_size`` 4 with no MoE or attention-DP option: bare, and as + quickstart_advanced.py passes ``--tp_size=4``, the MoE sizes as -1.""" + return { + "bare": _resolve_options(_NVFP4_NAMES, _TP4_BARE, _MAX_SEQ_LEN), + "quickstart": _resolve_options(_NVFP4_NAMES, _TP4_QUICKSTART, _MAX_SEQ_LEN), + } + + +@pytest.fixture(params=["bare", "quickstart"]) +def tp4_bare(request, tp4_spellings): + return tp4_spellings[request.param] + + +@pytest.fixture(scope="module") +def tp8(): + """The tp8 accuracy and serve gates' engine: spec-tp8/options/core.yaml.""" + return _resolve_options(_NVFP4_NAMES, _TP8, _MAX_SEQ_LEN) + + +@pytest.fixture(scope="module") +def tp8_gate(): + """The tp8 boot gate's engine: its quickstart command, --tp_size=8 and no MoE split.""" + path = _checkpoint_dir(_NVFP4_NAMES) + return _resolve(path, _llm_args(**_gate_kwargs(path, _GATE_ARGV))) + + +@pytest.fixture(scope="module") +def tep4(): + return _resolve_options(_NVFP4_NAMES, _TEP4, _MAX_SEQ_LEN) + + +@pytest.fixture(scope="module") +def tep8(): + return _resolve_options(_NVFP4_NAMES, _TEP8, _MAX_SEQ_LEN) + + +@pytest.fixture(scope="module") +def dep4(): + return _resolve_options(_NVFP4_NAMES, _DEP4, _MAX_SEQ_LEN) + + +def _core_unit(request, world: int) -> list: + """The core unit at ``world`` ranks, as each of its gates spells it with the MoE split.""" + return [request.getfixturevalue(name) for name in _SPELLINGS[world]] + + +def _unsplit(request, world: int) -> list: + """The core unit at ``world`` ranks spelled with ``tensor_parallel_size`` alone: at tp4 bare + and as quickstart_advanced.py passes ``--tp_size=4``, at tp8 the boot gate's command.""" + if world == 4: + return list(request.getfixturevalue("tp4_spellings").values()) + return [request.getfixturevalue("tp8_gate")] + + +def _anchor(llm_args) -> float: + """The reference row `AccuracyTask.get_hypothesis_testing_params` selects for this engine.""" + quant = llm_args.quant_config + acc_specs = dict( + dtype=llm_args.dtype, + quant_algo=quant.quant_algo, + kv_cache_quant_algo=quant.kv_cache_quant_algo, + spec_dec_algo=None, + extra_acc_spec=None, + ) + for entry in yaml.safe_load(_GSM8K_REFERENCES.read_text())[_MODEL_NAME]: + if all( + entry.get(key, "auto" if key == "dtype" else None) == value + for key, value in acc_specs.items() + ): + return entry["accuracy"] + raise AssertionError(f"no {_MODEL_NAME} GSM8K reference row for {acc_specs}") + + +def _rank_mapping(mapping, rank: int): + rank_mapping = copy.deepcopy(mapping) + rank_mapping.rank = rank + return rank_mapping + + +def _kv_cost(resolved, rank: int) -> CacheCost: + """The engine's affine KV cost on ``rank``, as the KV-cache creator sizes the pool with it.""" + model_config, llm_args = resolved.model_config, resolved.llm_args + kv_cache_config = llm_args.kv_cache_config + manager = get_kv_cache_manager_cls(model_config, kv_cache_config) + return CacheCost.from_raw( + manager.get_cache_size_per_token( + model_config, + _rank_mapping(resolved.mapping, rank), + tokens_per_block=kv_cache_config.tokens_per_block, + max_seq_len=resolved.max_seq_len, + max_batch_size=resolved.max_batch_size, + max_num_tokens=resolved.max_num_tokens, + kv_cache_config=kv_cache_config, + spec_config=llm_args.speculative_config, + is_draft=False, + ) + ) + + +def _summary(resolved) -> dict: + """Everything the engine resolved for a checkpoint except its mapping.""" + pretrained = resolved.model_config.pretrained_config.to_dict() + pretrained.pop("_name_or_path", None) + llm_args, model_config = resolved.llm_args, resolved.model_config + return dict( + llm_quant=repr(llm_args.quant_config), + quant=repr(model_config.quant_config), + layers={k: repr(v) for k, v in (model_config.quant_config_dict or {}).items()}, + pretrained=pretrained, + sparse=model_config.sparse_attention_config.model_dump(), + routed=resolved.routed[0].quantization_config, + sizes=(resolved.max_num_tokens, resolved.max_seq_len, resolved.max_batch_size), + cuda_graphs=llm_args.cuda_graph_config.model_dump(), + kv_cache=llm_args.kv_cache_config.model_dump(), + moe=llm_args.moe_config.model_dump(), + moe_backend=model_config.moe_backend, + moe_max_num_tokens=model_config.moe_max_num_tokens, + ) + + +def _flat(value, prefix: str = "") -> dict: + """``value`` as dotted keys to leaves, so two resolutions compare field by field.""" + if isinstance(value, dict): + flat = {} + for key, item in value.items(): + flat.update(_flat(item, f"{prefix}{key}.")) + return flat + return {prefix[:-1]: value} + + +def _differs(a: dict, b: dict) -> set: + a, b = _flat(a), _flat(b) + return {key for key in a.keys() | b.keys() if a.get(key, a) != b.get(key, b)} + + +def _width_slice(resolved, rank: int) -> tuple[range, range]: + """Rank ``rank``'s cut of every routed expert, from the mapping and the model config: the + intermediate rows of w1/w3 (= the input columns of w2) and w2's NVFP4 block-scale columns, + one per ``group_size`` inputs.""" + cfg = resolved.model_config.pretrained_config + rank_mapping = _rank_mapping(resolved.mapping, rank) + width = cfg.moe_intermediate_size // rank_mapping.moe_tp_size + group = resolved.model_config.quant_config.group_size + start = width * rank_mapping.moe_tp_rank + return range(start, start + width), range(start // group, (start + width) // group) + + +# ------------------------------------------------------- the ranks in one process + + +class _Group: + """The ranks in one process: rank r is a thread, and every exchange is a barrier, so each + rank's call sees what every rank sent, as over MPI.""" + + def __init__(self, size: int): + self.size = size + self._barrier = threading.Barrier(size, timeout=60) + self._slots = [None] * size + + def exchange(self, rank: int, obj): + self._slots[rank] = obj + self._barrier.wait() + gathered = list(self._slots) + self._barrier.wait() + return gathered + + def run(self, step): + """``step(rank)`` on every rank at once; the per-rank results in rank order.""" + results, errors = [None] * self.size, [None] * self.size + + def body(rank): + try: + results[rank] = step(rank) + except BaseException as error: # noqa: BLE001 - re-raised below, after every rank + errors[rank] = error + self._barrier.abort() + + threads = [threading.Thread(target=body, args=(rank,)) for rank in range(self.size)] + for thread in threads: + thread.start() + for thread in threads: + thread.join() + # The rank that failed first, not a peer it left waiting at the barrier. + raised = [error for error in errors if error is not None] + for error in raised: + if not isinstance(error, threading.BrokenBarrierError): + raise error + if raised: + raise raised[0] + return results + + +class _RankDist: + """One rank's communicator: the world gather the warmup agrees its shapes over.""" + + def __init__(self, group: _Group, rank: int): + self.group = group + self.rank = self.tp_rank = rank + self.world_size = self.tp_size = group.size + + def allgather(self, obj): + return self.group.exchange(self.rank, obj) + + +class _KvPool: + """The KV manager surface the warmup's dummy requests read: ``tokens`` free tokens in pages of + 32, and dummies recorded as the token counts they were asked for.""" + + tokens_per_block = 32 + num_extra_kv_tokens = 0 + + def __init__(self, tokens: int, max_seq_len: int): + self.tokens = tokens + self.max_seq_len = max_seq_len + + def get_num_available_tokens(self, *, token_num_upper_bound, max_num_draft_tokens=0, **_): + return min(token_num_upper_bound, self.tokens) + + def get_num_free_blocks(self) -> int: + return self.tokens // self.tokens_per_block + + def add_dummy_requests(self, request_ids, token_nums=None, is_gen=False, **kwargs): + if token_nums is None: + token_nums = [1 + kwargs.get("max_num_draft_tokens", 0)] * len(request_ids) + return [ + SimpleNamespace( + py_request_id=request_id, + token_num=token_num, + is_gen=is_gen, + is_last_context_chunk=True, + is_attention_dp_dummy=False, + py_draft_tokens=[], + py_needs_onehot_draft_probs=False, + ) + for request_id, token_num in zip(request_ids, token_nums) + ] + + def free_resources(self, request) -> None: + pass + + +class _Resources: + def __init__(self, kv: _KvPool): + self.kv = kv + + def get_resource_manager(self, key): + return self.kv if key == ResourceManagerType.KV_CACHE_MANAGER else None + + +def _runner_config(resolved) -> SimpleNamespace: + """The decoder runner's sizes for this engine. No speculative decoding: every draft width is + zero, and beams are one wide.""" + graphs = resolved.llm_args.cuda_graph_config + return SimpleNamespace( + max_num_tokens=resolved.max_num_tokens, + max_seq_len=resolved.max_seq_len, + max_batch_size=resolved.max_batch_size, + cuda_graph_batch_sizes=graphs.batch_sizes, + max_beam_width=1, + spec_config=None, + use_mrope=False, + max_draft_len=0, + original_max_draft_len=0, + max_total_draft_tokens=0, + max_draft_loop_tokens=0, + ) + + +def _shell(resolved, rank: int, cls=DecoderModelForCausalLM): + """The causal-LM shell every target subclasses, on ``rank``, built on the meta device.""" + config = dataclasses.replace( + resolved.model_config, mapping=_rank_mapping(resolved.mapping, rank) + ) + cfg = config.pretrained_config + with torch.device("meta"): + return cls( + DecoderModel(config), + config=config, + hidden_size=cfg.hidden_size, + vocab_size=cfg.vocab_size, + ) + + +# ------------------------------------------------------- (1) topology, node layout, limits + + +@_WORLDS +def test_topology_and_limits(request, world): + """Four ranks at tp4, eight at tp8, tensor parallel throughout: attention TP over every rank + with no attention DP (16 of the 64 heads per rank at tp4, 8 at tp8), the routed experts tensor + parallel as many ways and not expert-parallel, one pipeline stage -- at tp8 from core.yaml and + from the boot gate's command alike. Rank r is TP rank r and MoE TP rank r, MoE EP rank 0: its + window is all 128 experts at offset 0, and its share of each is the width slice [I r, I r + I) + of 6144 (w1/w3 rows, w2 columns) with w2's block scales at columns [I/16 r, I/16 r + I/16) of + 384, I = 1536 at tp4 and 768 at tp8; the slices tile every expert once. One scheduler drives + the ranks: 16 requests and 32768 tokens per step in all, unchunked.""" + core = request.getfixturevalue(f"tp{world}") + llm_args = core.llm_args + assert ( + llm_args.tensor_parallel_size, + llm_args.moe_tensor_parallel_size, + llm_args.moe_expert_parallel_size, + llm_args.enable_attention_dp, + ) == (world, world, 1, False) + parallel = llm_args.parallel_config + assert (parallel.tp_size, parallel.moe_tp_size, parallel.moe_ep_size, parallel.pp_size) == ( + world, + world, + 1, + 1, + ) + assert llm_args.attention_dp_config is None + + width, scale_width = _SLICE[world] + for resolved in _core_unit(request, world): + llm_args, mapping = resolved.llm_args, resolved.mapping + parallel = llm_args.parallel_config + assert (parallel.tp_size, parallel.pp_size, parallel.cp_size) == (world, 1, 1) + assert llm_args.enable_attention_dp is False and parallel.enable_attention_dp is False + + assert (mapping.world_size, mapping.tp_size, mapping.pp_size, mapping.cp_size) == ( + world, + world, + 1, + 1, + ) + assert (mapping.moe_tp_size, mapping.moe_ep_size, mapping.moe_cluster_size) == (world, 1, 1) + assert mapping.enable_attention_dp is False and mapping.dp_size == 1 + assert mapping.attn_tp_size == world and mapping.enable_lm_head_tp_in_adp is False + assert mapping.has_moe_tp() and not mapping.has_moe_ep() + assert resolved.model_config.mapping is mapping + + cfg = resolved.model_config.pretrained_config + assert (cfg.num_attention_heads, cfg.n_routed_experts, cfg.num_experts_per_tok) == ( + 64, + 128, + 4, + ) + assert (cfg.moe_intermediate_size, resolved.model_config.quant_config.group_size) == ( + 6144, + 16, + ) + local_experts = cfg.n_routed_experts // mapping.moe_ep_size + assert (cfg.num_attention_heads // mapping.tp_size, local_experts) == (_HEADS[world], 128) + + rows, scales = set(), set() + for rank in range(world): + rank_mapping = _rank_mapping(mapping, rank) + assert ( + rank_mapping.tp_rank, + rank_mapping.moe_tp_rank, + rank_mapping.moe_ep_rank, + rank_mapping.moe_cluster_rank, + ) == (rank, rank, 0, 0) + assert rank_mapping.ep_experts(cfg.n_routed_experts) == list(range(128)) + assert local_experts * rank_mapping.moe_ep_rank == 0 + assert rank_mapping.tp_group == rank_mapping.moe_tp_group == list(range(world)) + assert rank_mapping.moe_ep_group == [rank] + rank_rows, scale_columns = _width_slice(resolved, rank) + assert rank_rows == range(width * rank, width * rank + width) + assert scale_columns == range(scale_width * rank, scale_width * rank + scale_width) + assert rows.isdisjoint(rank_rows) and scales.isdisjoint(scale_columns) + rows.update(rank_rows) + scales.update(scale_columns) + assert rows == set(range(6144)) and scales == set(range(6144 // 16)) + + assert (resolved.max_batch_size, resolved.max_num_tokens, resolved.max_seq_len) == ( + 16, + 32768, + 32768, + ) + model_config = resolved.model_config + assert (model_config.max_num_tokens, model_config.max_seq_len) == (32768, 32768) + assert llm_args.enable_chunked_prefill is False + + +def test_tp_size_4_alone_builds_the_same_deployment(tp4, tp4_bare): + """``tensor_parallel_size`` 4 with no MoE or attention-DP option -- also as + quickstart_advanced.py passes ``--tp_size=4``, the MoE sizes as -1 -- resolves to the core + unit's mapping on every rank and the same configuration throughout, and the router is handed + an equal mapping. The only trace of the spelling is ``Mapping.moe_tp_ep_user_specified``, + which neither ``to_dict`` nor equality carries.""" + assert tp4_bare.mapping.to_dict() == tp4.mapping.to_dict() + for rank in range(4): + bare, core = _rank_mapping(tp4_bare.mapping, rank), _rank_mapping(tp4.mapping, rank) + assert bare == core + assert (bare.moe_tp_rank, bare.moe_ep_rank) == (core.moe_tp_rank, core.moe_ep_rank) + assert _summary(tp4_bare) == _summary(tp4) + assert tp4_bare.routed[0].mapping == tp4.routed[0].mapping + assert tp4_bare.routed[0].mapping.to_dict() == tp4.routed[0].mapping.to_dict() + assert tp4.mapping.moe_tp_ep_user_specified is True + assert tp4_bare.mapping.moe_tp_ep_user_specified is False + + +def test_gpus_per_node_is_the_visible_device_count(tp8, tp8_gate): + """At tp8 neither gate states gpus_per_node. ``TorchLlmArgs.gpus_per_node`` defaults to None + and its validator (outside Ray) replaces None with ``torch.cuda.device_count()`` of the + process that builds the arguments, rank 0; each node's ranks see that node's four held + devices, so it is 4. The mapping every rank builds from ``parallel_config`` carries it: rank r + is local rank r % 4 on node r // 4, and the deployment is multi-node. The LLM constructor's + device check reads it too: world_size_per_node is 4, which four visible devices satisfy. + Stating gpus_per_node 4 builds the same mapping. + + The defaults of ``_ParallelConfig`` and ``Mapping`` are 8, not the device count: an + eight-rank mapping built without it reads one node of eight -- local rank r, never multi-node + -- and would demand eight devices on rank 0's node.""" + field = TorchLlmArgs.model_fields["gpus_per_node"] + assert field.default is None and field.validate_default is True + assert "gpus_per_node" not in _options(_TP8, _MAX_SEQ_LEN) + assert "gpus_per_node" not in _gate_kwargs(tp8_gate.path, _GATE_ARGV) + + for resolved in (tp8, tp8_gate): + llm_args, mapping = resolved.llm_args, resolved.mapping + assert llm_args.gpus_per_node == _GPUS_PER_NODE == 4 + assert llm_args.parallel_config.gpus_per_node == 4 + assert mapping.gpus_per_node == 4 and resolved.model_config.mapping.gpus_per_node == 4 + assert llm_args.parallel_config.world_size_per_node == 4 + for rank in range(8): + rank_mapping = _rank_mapping(mapping, rank) + assert (rank_mapping.local_rank, rank_mapping.node_rank) == (rank % 4, rank // 4) + assert rank_mapping.get_node_rank(rank) == rank // 4 + assert rank_mapping.is_multi_node() is True + + path = str(tp8.path) + for devices in (4, 8): + llm_args = _llm_args(devices, model=path, **_options(_TP8, _MAX_SEQ_LEN)) + assert llm_args.gpus_per_node == devices + assert llm_args.parallel_config.world_size_per_node == devices + stated = _llm_args(1, model=path, gpus_per_node=4, **_options(_TP8, _MAX_SEQ_LEN)) + assert stated.gpus_per_node == 4 + assert stated.parallel_config.to_mapping().to_dict() == tp8.mapping.to_dict() + + assert _ParallelConfig.model_fields["gpus_per_node"].default == 8 + assert _ParallelConfig(tp_size=8, moe_tp_size=8, moe_ep_size=1).world_size_per_node == 8 + unstated = Mapping(world_size=8, rank=5, tp_size=8, moe_tp_size=8, moe_ep_size=1) + assert unstated.gpus_per_node == 8 + assert (unstated.local_rank, unstated.node_rank, unstated.is_multi_node()) == (5, 0, False) + + +def test_each_rank_selects_its_own_nodes_device(tp8): + """The device each worker sets before building its executor (``BaseWorker._configure_device``, + executor/base_worker.py) at tp8: its global MPI rank modulo the devices its node shows, so rank + r drives device r % 4 of its node -- the mapping's local rank. Each rank checks that its + device's architecture is supported and pins its CPU affinity to it.""" + group = _Group(8) + local = threading.local() + with pytest.MonkeyPatch.context() as mp: + mp.setattr(torch.cuda, "device_count", lambda: _GPUS_PER_NODE) + mp.setattr(torch.cuda, "set_device", lambda device: setattr(local, "device", device)) + mp.setattr( + base_worker, + "check_cuda_architecture_supported", + lambda device: setattr(local, "checked", device), + ) + mp.setattr( + base_worker, "configure_cpu_affinity", lambda device: setattr(local, "cpus", device) + ) + + def step(rank): + BaseWorker._configure_device(SimpleNamespace(global_rank=rank)) + return local.checked, local.device, local.cpus + + results = group.run(step) + for rank, (checked, device, cpus) in enumerate(results): + assert checked == device == cpus == rank % 4 + assert device == _rank_mapping(tp8.mapping, rank).local_rank + + +def test_core_yaml_and_boot_command_build_one_deployment(tp8, tp8_gate): + """At tp8, core.yaml and the boot gate's command, which states --tp_size=8 and no MoE split, + resolve to the same mapping on every rank and the same model, quantization, DSA config, + limits, CUDA graphs, KV dtype, page size, block-reuse setting, KV manager and MoE config; the + router is handed equal mappings. ``Mapping`` gives the MoE the whole TP width when neither MoE + size is stated (moe_tp 8, moe_ep 1), so the command's -1 sentinels land where core.yaml's 8 + and 1 do. + + What differs, field by field, and nothing else: + + * the MoE sizes as stated: core.yaml's 8 and 1, the command's -1 and -1, in ``TorchLlmArgs`` + and in ``parallel_config`` -- and so in the usage telemetry, which reports them (8 and 1 + against 0 and 0) and nothing reads back; + * the mapping's ``moe_tp_ep_user_specified``: True from core.yaml, False from the command; + neither ``to_dict`` nor equality carries it; + * the KV fraction: the quickstart's --kv_cache_fraction default, 0.9, against core.yaml's + 0.85, so the boot gate's pool is sized at 0.9 of the free memory and the accuracy gate's at + 0.85; + * ``kv_cache_config.mamba_ssm_cache_dtype``: the quickstart's default 'bfloat16' against + 'auto', which the model loader resolves to the checkpoint's bfloat16 -- the same value in + the quantization config, read only by a recurrent-state pool this model has none of; + * ``attention_dp_config``: None from core.yaml, the quickstart's ``AttentionDpConfig`` from the + command -- balancing and KV-aware routing off, so the executor builds the same default ADP + router for either and never reads the quickstart's balancing waits, and attention DP is off + besides.""" + assert tp8_gate.mapping.to_dict() == tp8.mapping.to_dict() + assert tp8_gate.routed[0].mapping == tp8.routed[0].mapping + assert tp8_gate.routed[0].mapping.to_dict() == tp8.routed[0].mapping.to_dict() + for rank in range(8): + gate, core = _rank_mapping(tp8_gate.mapping, rank), _rank_mapping(tp8.mapping, rank) + assert gate == core + assert (gate.moe_tp_rank, gate.moe_ep_rank) == (core.moe_tp_rank, core.moe_ep_rank) + gate, core = _summary(tp8_gate), _summary(tp8) + assert _differs(gate, core) == { + "kv_cache.free_gpu_memory_fraction", + "kv_cache.mamba_ssm_cache_dtype", + } + assert gate["moe_max_num_tokens"] == core["moe_max_num_tokens"] == 32768 + for resolved in (tp8, tp8_gate): + graphs = resolved.llm_args.cuda_graph_config + assert graphs.batch_sizes == [1, 2, 4, 8, 16] and graphs.enable_padding is True + kv = resolved.llm_args.kv_cache_config + assert (kv.dtype, kv.enable_block_reuse, kv.tokens_per_block) == ("auto", False, 32) + assert kv.use_kv_cache_manager_v2 is True + assert resolved.llm_args.disable_overlap_scheduler is False + assert tp8.llm_args.kv_cache_config.free_gpu_memory_fraction == 0.85 + assert tp8_gate.llm_args.kv_cache_config.free_gpu_memory_fraction == 0.9 + + # Field by field. + stated = ("moe_tensor_parallel_size", "moe_expert_parallel_size") + assert tuple(getattr(tp8.llm_args, name) for name in stated) == (8, 1) + assert tuple(getattr(tp8_gate.llm_args, name) for name in stated) == (-1, -1) + dumps = ( + args.model_dump(exclude={"attention_dp_config"}) + for args in (tp8_gate.llm_args, tp8.llm_args) + ) + assert _differs(*dumps) == { + "moe_tensor_parallel_size", + "moe_expert_parallel_size", + "kv_cache_config.free_gpu_memory_fraction", + "kv_cache_config.mamba_ssm_cache_dtype", + } + parallel = tp8_gate.llm_args.parallel_config, tp8.llm_args.parallel_config + assert _differs(*(p.model_dump() for p in parallel)) == {"moe_tp_size", "moe_ep_size"} + assert [(p.moe_tp_size, p.moe_ep_size) for p in parallel] == [(-1, -1), (8, 1)] + assert _differs(vars(tp8_gate.mapping), vars(tp8.mapping)) == {"moe_tp_ep_user_specified"} + assert (tp8_gate.mapping.moe_tp_ep_user_specified, tp8.mapping.moe_tp_ep_user_specified) == ( + False, + True, + ) + telemetry = [usage_lib._extract_trtllm_config(r.llm_args) for r in (tp8_gate, tp8)] + assert [(t["moe_tensor_parallel_size"], t["moe_expert_parallel_size"]) for t in telemetry] == [ + (0, 0), + (8, 1), + ] + + assert [r.llm_args.kv_cache_config.mamba_ssm_cache_dtype for r in (tp8_gate, tp8)] == [ + "bfloat16", + "auto", + ] + for resolved in (tp8_gate, tp8): + assert resolved.model_config.quant_config.mamba_ssm_cache_dtype == torch.bfloat16 + + assert tp8.llm_args.attention_dp_config is None + adp = tp8_gate.llm_args.attention_dp_config + assert type(adp) is AttentionDpConfig + assert ( + adp.enable_balance, + adp.enable_kv_cache_aware_routing, + adp.kv_cache_routing_conversation_affinity, + ) == (False, False, False) + # The quickstart's balancing waits (its flags' defaults, not the class's), read only when + # balancing is on. + assert (adp.timeout_iters, adp.batching_wait_iters) == (0, 0) + for resolved in (tp8_gate, tp8): + config = resolved.llm_args.attention_dp_config + router = ADPRouter.create( + dist=None, has_seq_slot_headroom=False, attention_dp_config=config + ) + assert type(router) is DefaultADPRouter + assert not (config is not None and config.enable_balance) + + +# ------------------------------------------------------- (2) quantization, the anchor's key + + +@_WORLDS +def test_quantization_is_the_anchor_key(request, world): + """At tp4 and tp8 as at tep4, dep4 and tep8: MIXED_PRECISION with the FP8 KV cache, the GSM8K + anchor row's key, and the router handed params.json's quantization_config with its 475 + quantized_layers intact plus the FP8 KV-cache algorithm the native loader records from + fp8_kv_cache, on the core unit's mapping -- at tp8 from core.yaml and from the boot gate's + command alike.""" + resolutions = _core_unit(request, world) + on_disk = json.loads((resolutions[0].path / "params.json").read_text())["quantization_config"] + assert len(on_disk["quantized_layers"]) == 475 + routed = dict(on_disk, kv_cache_quant_algo="FP8") + for resolved in resolutions: + assert resolved.prequantized is True + for quant in (resolved.llm_args.quant_config, resolved.model_config.quant_config): + assert quant.quant_algo == QuantAlgo.MIXED_PRECISION + assert quant.kv_cache_quant_algo == QuantAlgo.FP8 + assert quant.group_size == 16 + assert resolved.llm_args.kv_cache_config.dtype == "auto" + assert _anchor(resolved.llm_args) == 91.926 + + assert len(resolved.routed) == 1 + seen = resolved.routed[0] + assert seen.quantization_config == routed + assert resolved.model_config.pretrained_config.quantization_config == routed + assert seen.pretrained_config is resolved.model_config.pretrained_config + assert resolved.model_config.modeling_v2_target is _Routed + assert seen.compute_capability == _SM103 and seen.spec_config is None + assert seen.mapping is resolved.model_config.mapping + assert (seen.mapping.world_size, seen.mapping.tp_size, seen.mapping.pp_size) == ( + world, + world, + 1, + ) + assert (seen.mapping.moe_tp_size, seen.mapping.moe_ep_size) == (world, 1) + assert seen.mapping.enable_attention_dp is False + + +# ------------------------------------------------------- (3) the KV pool, per rank + + +@_WORLDS +def test_kv_pool_as_at_tep(request, world): + """MLA's latent and the indexer keys are not head-sharded, the MoE split never reaches the KV + cost, and every rank holds all 16 requests: a token costs the same on every rank at tp4 as at + tep4 and dep4, and on each of the eight ranks at tp8 as on each tep8 rank, an fp8_ds_mla + latent of 656 bytes (512 e4m3, four fp32 tile scales, 64 bf16 rope) on each of the 63 KV + owners plus a 128-byte fp8 indexer key and its fp32 scale on each of the 44 full-indexer + layers, in pages of 32, in MistralLarge4CacheManager (the nine sharers keep no latent). A + rank's pool for 16 requests of max_seq_len 32768 is that times 16 x 32768 tokens.""" + core = _core_unit(request, world) + peers = [request.getfixturevalue(name) for name in _KV_PEERS[world]] + for resolved in core + peers: + model_config, kv_cache_config = resolved.model_config, resolved.llm_args.kv_cache_config + assert get_kv_cache_manager_cls(model_config, kv_cache_config) is ( + MistralLarge4CacheManager + ) + assert kv_cache_config.tokens_per_block == 32 + + model_config = core[0].model_config + cfg, sparse = model_config.pretrained_config, model_config.sparse_attention_config + latent = 656 # fp8_ds_mla: 512 e4m3 + 4 fp32 tile scales + 64 bf16 rope + owners = sum(1 for i, s in enumerate(cfg.kv_source_layers) if s == i) + indexer_key = sparse.index_head_dim + sparse.index_head_dim // 128 * 4 + full_indexer_layers = cfg.index_topk_pattern.count("F") + per_token = latent * owners + indexer_key * full_indexer_layers + assert (latent, owners, indexer_key, full_indexer_layers) == (656, 63, 132, 44) + assert per_token == 47136 + + for rank in range(world): + cost = _kv_cost(core[0], rank) + assert (cost.slope, cost.intercept) == (per_token, 0) + for resolved in core[1:] + peers: + assert cost == _kv_cost(resolved, rank) + assert cost.bytes_for_tokens(32) == 32 * per_token == 1_508_352 + + tokens = core[0].max_batch_size * core[0].max_seq_len + assert tokens == 16 * 32768 + pool = _kv_cost(core[0], 0).bytes_for_tokens(tokens) + assert pool == tokens * per_token + assert pool == 524288 * 47136 == 24_712_839_168 + assert math.isclose(pool / 2**30, 23.016, abs_tol=5e-4) + + +@_WORLDS +def test_indexer_chunk_holds_a_whole_prompt(request, world): + """The indexer's [T, N] logits are cut at indexer_max_chunk_size: a whole 32K prompt, the + largest unchunked step, is one chunk.""" + resolved = request.getfixturevalue(f"tp{world}") + model_config = resolved.model_config + params = model_config.sparse_attention_config.to_sparse_metadata_params( + pretrained_config=model_config.pretrained_config + ) + assert params.indexer_max_chunk_size >= 32768 + assert params.indexer_max_chunk_size >= resolved.max_num_tokens + + +# ------------------------------------------------------- (4) the largest step + + +@_WORLDS +def test_warmup_runs_max_num_tokens_on_every_rank(request, world): + """The warmup's shapes at tp4 and at tp8. Each rank proposes its shapes from its own free KV, + the ranks keep the ones all proposed, and every rank enters each shape's forward. The largest + is max_num_tokens of context in one step -- a 32767-token and a 1-token prompt, since a prompt + leaves room for one decode token -- run by all the ranks on the same tokens. Every rank's + window is all 128 experts, so each rank's routed experts see all 32768 tokens and every one of + their top-4 slots: 4 x 32768 = 131072 permuted rows on every rank, all on local experts.""" + resolved = request.getfixturevalue(f"tp{world}") + config = _runner_config(resolved) + model = SimpleNamespace(model_config=resolved.model_config) + group = _Group(world) + + def rank_warmup(rank): + # The pool holds at least max_batch_size requests of max_seq_len. + resources = _Resources( + _KvPool(config.max_batch_size * config.max_seq_len, config.max_seq_len) + ) + warmup = DecoderWarmup.__new__(DecoderWarmup) + warmup._config = config + agreement = WarmupAgreement(_RankDist(group, rank), _rank_mapping(resolved.mapping, rank)) + assert agreement.is_distributed_forward() + shapes = agreement.agree_shapes(warmup._get_full_general_warmup_requests(resources)) + factory = DummyRequestFactory( + config=config, + model=model, + get_runtime_tokens_per_gen_step=lambda draft_len: 1 + draft_len, + release_request_state=None, + ) + max_context = max(tokens for tokens, gen in shapes if gen == 0) + batch = factory.create_warmup_request(resources, max_context, 0) + enters = agreement.should_run_warmup_batch(batch, max_context, "max context") + prompts = [request.token_num for request in batch.context_requests] + return shapes, enters, prompts + + results = group.run(rank_warmup) + assert len(results) == world + for shapes, enters, prompts in results: + assert shapes == [(1, 0), (1, 1), (32768, 0), (16, 16), (2, 0)] + assert enters is True + assert prompts == [32767, 1] + + cfg = resolved.model_config.pretrained_config + rows = sum(results[0][2]) + assert rows == resolved.max_num_tokens == 32768 + for rank in range(world): + window = _rank_mapping(resolved.mapping, rank).ep_experts(cfg.n_routed_experts) + assert window == list(range(cfg.n_routed_experts)) + assert rows * cfg.num_experts_per_tok == 131072 + local_experts = cfg.n_routed_experts // resolved.mapping.moe_ep_size + assert local_experts == cfg.n_routed_experts + + +@_WORLDS +def test_kv_estimation_runs_one_max_num_tokens_step(request, world, dep4): + """Before the real pool exists the engine runs a profiling pass on dummy context requests as + long as a request may be, net max_seq_len - 1. Without attention DP it submits max_num_tokens + of them once (a 32767-token and a 1-token prompt) for the one scheduler the four ranks of tp4 + or the eight of tp8 share; dep4 submits a copy per rank. The DSA config keeps the pass off the + chunked-MLA profile.""" + resolved = request.getfixturevalue(f"tp{world}") + + def lengths(resolved): + creator = KvCacheCreator.__new__(KvCacheCreator) + creator._model_engine = SimpleNamespace( + model=SimpleNamespace(model_config=resolved.model_config) + ) + creator._max_num_tokens = resolved.max_num_tokens + creator._max_beam_width = 1 + creator._mapping = resolved.mapping + requests = creator._create_dummy_context_requests(max(1, resolved.max_seq_len - 1)) + assert creator._mla_chunked_profile_length is None + return [len(request.input_token_ids) for request in requests] + + assert lengths(resolved) == [32767, 1] + assert sum(lengths(resolved)) == resolved.max_num_tokens + assert lengths(dep4) == [32767, 1] * 4 + + +# ------------------------------------------------------- (5) the engine's reads of the MoE split + + +@_WORLDS +def test_engine_reads_of_the_moe_split(request, world): + """What the engine derives from the MoE split before a target runs. ModelConfig sizes the MoE + token budget from the attention-DP width, not the MoE split: max_num_tokens x dp_size = 32768 + at tp4 as at tep4, and at tp8 (from core.yaml and from the boot gate's command alike) as at + tep8. No MoE load balancer is built on any rank: none is configured, and + ``maybe_create_moe_load_balancer`` reads only the EP split, which is 1 at tp4 and tp8. The + autotuner's tuning group is the TP group -- ``tp_size`` and ``tp_rank`` -- so it is every + rank, (4, r) at tp4 and tep4 and (8, r) over both nodes at tp8 and tep8.""" + for resolved in _core_unit(request, world) + [request.getfixturevalue(_TEP[world])]: + model_config = resolved.model_config + assert model_config.moe_max_num_tokens == resolved.max_num_tokens * resolved.mapping.dp_size + assert model_config.moe_max_num_tokens == 32768 + assert resolved.llm_args.moe_config.load_balancer is None + assert model_config.moe_load_balancer is None + for rank in range(world): + rank_mapping = _rank_mapping(resolved.mapping, rank) + with maybe_create_moe_load_balancer(model_config, rank_mapping) as balancer: + assert balancer is None + tuner = SimpleNamespace(mapping=rank_mapping) + assert AutoTuner._tuning_group(tuner) == (world, rank) + + +@_WORLDS +def test_moe_autotuner_at_128_local_experts(request, world): + """The NVFP4 MoE runner the target calls (128 experts, all 128 local at offset 0, width 1536 + at tp4 and 768 at tp8, top-4, ``tune_max_num_tokens`` = max_num_tokens, ``use_dp`` False) + tunes INDEPENDENT, each rank alone, so tuning sends nothing between the ranks (or, at tp8, the + nodes). Its expert-parallel size, num_experts // local_num_experts, is 1 here and the world + size at tep4 and tep8, and without attention DP neither moves a bucket: the buckets and the + token-to-bucket rule are tep's, 32768 the top bucket. The tuner's cache key (top_k, + intermediate_size, local_num_experts, act_type) keeps the tp entries apart from the tep ones. + The tuning group is the TP group, ``(tp_size, tp_rank)``, at tp and tep alike.""" + tp, tep = request.getfixturevalue(f"tp{world}"), request.getfixturevalue(_TEP[world]) + for rank in range(world): + tuner = SimpleNamespace(mapping=_rank_mapping(tp.mapping, rank)) + assert AutoTuner._tuning_group(tuner) == (world, rank) + tuner = SimpleNamespace(mapping=_rank_mapping(tep.mapping, rank)) + assert AutoTuner._tuning_group(tuner) == (world, rank) + + def runner(resolved): + cfg, mapping = resolved.model_config.pretrained_config, resolved.mapping + local_experts = cfg.n_routed_experts // mapping.moe_ep_size + return FP4BlockScaleMoERunner( + num_experts=cfg.n_routed_experts, + top_k=cfg.num_experts_per_tok, + n_group=None, + topk_group=None, + intermediate_size=cfg.moe_intermediate_size // mapping.moe_tp_size, + local_expert_offset=local_experts * _rank_mapping(mapping, world - 1).moe_ep_rank, + local_num_experts=local_experts, + routed_scaling_factor=None, + routing_method_type=RoutingMethodType.Renormalize, + do_finalize=True, + act_type=ActType_TrtllmGen.SwiGlu.value, + tune_max_num_tokens=resolved.max_num_tokens, + ) + + width, _ = _SLICE[world] + tp_runner, tep_runner = runner(tp), runner(tep) + assert (tp_runner.local_expert_offset, tp_runner.local_num_experts) == (0, 128) + assert tp_runner.intermediate_size == width + assert tp_runner.num_experts // tp_runner.local_num_experts == 1 + assert tep_runner.num_experts // tep_runner.local_num_experts == world + # The runner passes every argument of the cached builder, decode_tuning (undeclared) included. + assert tp_runner.tuning_config is FP4BlockScaleMoERunner.get_tuning_config( + 1, tune_max_num_tokens=32768, use_dp=False, decode_tuning=False + ) + swiglu = ActType_TrtllmGen.SwiGlu.value + assert tp_runner.unique_id() == (4, width, 128, swiglu) + assert tep_runner.unique_id() == (4, 6144, 128 // world, swiglu) + + tp_tuning, tep_tuning = tp_runner.tuning_config, tep_runner.tuning_config + assert tp_tuning.distributed_tuning_strategy == DistributedTuningStrategy.INDEPENDENT + assert tp_tuning.tune_max_num_tokens == 32768 + (tp_spec,) = tp_tuning.dynamic_tensor_specs + (tep_spec,) = tep_tuning.dynamic_tensor_specs + assert tp_spec.gen_tuning_buckets == tep_spec.gen_tuning_buckets + assert max(tp_spec.gen_tuning_buckets) == 32768 + for tokens in (1, 3, 16, 17, 4097, 8192, 8193, 16384, 32767, 32768): + assert tp_spec.map_to_tuning_buckets(tokens) == tep_spec.map_to_tuning_buckets(tokens) + assert tp_spec.map_to_tuning_buckets(32768) == 32768 + + +def test_mnnvl_and_collectives_at_moe_tp_8(tp8, tep8): + """Where the engine's collectives read the MoE split and the node layout at tp8, on each + rank's mapping: + + * the rank's mapping is an ``MpiTopology`` and its communicator an ``MPIDist`` (the MPI + launch leaves TLLM_DISABLE_MPI unset); ``DeviceMeshTopology`` -- whose mesh has no moe_tp + dimension unless moe_ep_size > 1 -- and ``TorchDist`` are the non-MPI path; + * ``MnnvlMemory.get_comm`` splits by pipeline, context and MoE TP rank, keyed by TP rank: at + moe_tp 8 every rank is its own colour, eight one-rank communicators (tep8: one of all eight). + Only the MoE all-to-all workspaces (and Helix's subclass) build an ``MnnvlMemory``, and the + all-to-all is refused at moe_tp > 1; + * the MNNVL AllReduce's workspace communicator (``_get_mnnvl_workspace_comm``) splits by + pipeline and context rank only, keyed by TP rank: one communicator of all eight ranks, as at + tep8. An engine ``AllReduce`` at the default AUTO strategy takes MNNVL only for a group that + spans nodes (``MNNVLAllReduce.is_mnnvl`` asks ``mapping.is_multi_node()``: true here), and + the one-node IPC workspace (``IpcMemory``) stays closed past one node's devices; an MNNVL + buffer sits on device ``mapping.local_rank``. No userbuffers: the strategy is AUTO, not + NCCL_SYMMETRIC, and torch.compile is off.""" + llm_args = tp8.llm_args + assert llm_args.allreduce_strategy == "AUTO" and llm_args.torch_compile_config is None + assert MnnvlMemory.comm_layout_signature(tp8.mapping) == (8, 1, 1, 8, 8, 1, 1) + assert MnnvlMemory.comm_layout_signature(tep8.mapping) == (8, 1, 1, 8, 1, 8, 1) + + splits = [] + with pytest.MonkeyPatch.context() as mp: + mp.delenv("TLLM_DISABLE_MPI", raising=False) + mp.setattr( + distributed_ops, + "mpi_comm", + lambda: SimpleNamespace(Split=lambda color, key: splits.append((color, key))), + ) + for rank in range(8): + rank_mapping = _rank_mapping(tp8.mapping, rank) + assert type(rank_mapping) is MpiTopology + assert type(Distributed.get(rank_mapping)) is MPIDist + assert MnnvlMemory.comm_split_color_key(rank_mapping) == (rank, rank) + assert MnnvlMemory.comm_split_color_key(_rank_mapping(tep8.mapping, rank)) == (0, rank) + distributed_ops._get_mnnvl_workspace_comm(rank_mapping) + assert distributed_ops._mnnvl_device_index(rank_mapping) == rank % 4 + assert rank_mapping.is_multi_node() is True + assert IpcMemory(rank_mapping, 1024).open_ipc is False + assert splits == [(0, rank) for rank in range(8)] + + +@_WORLDS +def test_weight_sharing_reads(request, world): + """``pyexecutor/model_loader.py`` imports ``_torch/weight_sharing`` and reads the MoE split + there on every load. ``ModelLoader.load`` qualifies the constructed model against the + post-transform profile registry with a ``PostTransformRuntimeConfig`` that records moe_tp 4, + moe_ep 1 on one node at tp4 and moe_tp 8, moe_ep 1 and multi-node at tp8: no profile names a + modeling_v2 target's class, so the decision is ROOT_MODEL_CLASS_NOT_REGISTERED and nothing is + shared. ``SourceIdentity``, which hashes moe_tp_size and each rank's moe_tp_rank + (``check_weight_sharing_compatibility`` compares it), is built only for GMS or an MX + checkpoint, and the gates load AUTO from this checkpoint's files.""" + + class _Target(DecoderModelForCausalLM): + """Stands for the target class the router returns: no profile registers it.""" + + tp, tep = request.getfixturevalue(f"tp{world}"), request.getfixturevalue(_TEP[world]) + for resolved, moe_split in ((tp, (world, 1)), (tep, (1, world))): + runtime = PostTransformRuntimeConfig.from_model_config(resolved.model_config) + assert (runtime.tp_size, runtime.pp_size, runtime.cp_size) == (world, 1, 1) + assert (runtime.moe_tp_size, runtime.moe_ep_size) == moe_split + assert (runtime.attention_tp_size, runtime.attention_dp, runtime.multi_node) == ( + world, + False, + world > _GPUS_PER_NODE, + ) + + shell = _shell(resolved, 3, _Target) + decision = RankModelLoader._qualify_post_transform_profile( + shell, + config_identity=PostTransformConfigIdentity.from_model_config(resolved.model_config), + speculative_mode=None, + loads_draft_weights=False, + ) + assert decision.qualified is False and decision.profile is None + assert decision.reason == PostTransformQualificationReason.ROOT_MODEL_CLASS_NOT_REGISTERED + + assert resolved.llm_args.load_format == LoadFormat.AUTO + assert resolved.checkpoint_loader.checkpoint_format != "MX" + assert ( + RankModelLoader._needs_source_identity(resolved.checkpoint_loader, LoadFormat.AUTO) + is False + ) + + +@_WORLDS +def test_shell_lm_head_as_at_tep(request, world): + """The causal-LM shell every target subclasses reads the TP width, not the MoE split: on each + rank ``DecoderModelForCausalLM`` builds the same vocab-parallel lm_head at tp4 as at tep4 and + at tp8 as at tep8, a quarter (51200 rows of 6144) or an eighth (25600 rows) of the + 204800-token vocabulary, gathered over the TP group of every rank and with no AllReduce of its + own. Built on the meta device: no weights.""" + tp, tep = request.getfixturevalue(f"tp{world}"), request.getfixturevalue(_TEP[world]) + + def lm_head(resolved, rank): + shell = _shell(resolved, rank) + head = shell.lm_head + return dict( + type=type(head), + tp=(head.tp_size, head.tp_rank, head.tp_mode), + shape=tuple(head.weight.shape), + gather=(head.gather_output, shell.model_config.lm_head_gather_output), + all_reduce=head.all_reduce, + group=head.mapping.tp_group, + quant=head.quant_config, + pp=(shell.pp_rank, shell.pp_size), + ) + + vocab_rows = {4: 51200, 8: 25600}[world] + for rank in range(world): + head = lm_head(tp, rank) + assert head == lm_head(tep, rank) + assert head["tp"] == (world, rank, TensorParallelMode.COLUMN) + assert head["shape"] == (204800 // world, 6144) == (vocab_rows, 6144) + assert head["gather"] == (True, True) and head["all_reduce"] is None + assert head["group"] == list(range(world)) and head["pp"] == (0, 1) + + +# ------------------------------------------------------- (6) tep4, dep4 and tep8 as before + + +def test_tep4_dep4_and_tep8_route_as_before(tep4, dep4, tep8): + """The NVFP4 checkpoint at tep4, at dep4 and at tep8 resolves and routes as before this + target: each claimed by its own target -- tep8 by the tep target, which serves eight ranks + over two nodes of four too -- the routed experts expert-parallel and not split along their + width (MoE TP rank 0 everywhere), each on the checkpoint's anchor and the same 47136-byte + token.""" + cases = ( + (tep4, 4, False, "ModelingV2MistralLarge4Nvfp4Sm103Tep"), + (dep4, 4, True, "ModelingV2MistralLarge4Nvfp4Sm103Dep"), + (tep8, 8, False, "ModelingV2MistralLarge4Nvfp4Sm103Tep"), + ) + for resolved, world, attention_dp, target in cases: + seen = resolved.routed[0] + mapping = seen.mapping + assert (mapping.world_size, mapping.tp_size, mapping.pp_size) == (world, world, 1) + assert (mapping.moe_ep_size, mapping.moe_tp_size, mapping.enable_attention_dp) == ( + world, + 1, + attention_dp, + ) + assert mapping.gpus_per_node == 4 and mapping.is_multi_node() is (world > 4) + for rank in range(world): + rank_mapping = _rank_mapping(mapping, rank) + assert (rank_mapping.moe_ep_rank, rank_mapping.moe_tp_rank) == (rank, 0) + assert (rank_mapping.local_rank, rank_mapping.node_rank) == (rank % 4, rank // 4) + assert _route(seen.pretrained_config, mapping).__name__ == target + assert resolved.llm_args.quant_config.quant_algo == QuantAlgo.MIXED_PRECISION + assert resolved.llm_args.quant_config.kv_cache_quant_algo == QuantAlgo.FP8 + assert _anchor(resolved.llm_args) == 91.926 + assert (_kv_cost(resolved, 0).slope, _kv_cost(resolved, 0).intercept) == (47136, 0) + assert (resolved.max_num_tokens, resolved.max_seq_len, resolved.max_batch_size) == ( + 32768, + 32768, + 16, + ) + + +@_WORLDS +def test_tp_differs_from_tep_in_the_moe_split_only(request, world): + """The same checkpoint at tp4 and at tep4, and at tp8 and at tep8, resolves to the same + configuration -- quantization, per-layer quantization, model config, DSA config, limits, CUDA + graphs, KV config, the MoE config and token budget, what the router reads -- except the + mapping's MoE split. No tep4 or dep4 branch of the routing claims tp4 or tp8 in any + spelling (core.yaml; ``tensor_parallel_size`` alone, at tp4 bare and as quickstart_advanced.py + passes it, at tp8 the boot gate's command): it is refused, naming moe_ep=1 and moe_tp at the + world size, or claimed by the tp target.""" + tp, tep = request.getfixturevalue(f"tp{world}"), request.getfixturevalue(_TEP[world]) + assert _summary(tp) == _summary(tep) + tp_mapping, tep_mapping = tp.mapping.to_dict(), tep.mapping.to_dict() + assert {k for k in tp_mapping if tp_mapping[k] != tep_mapping[k]} == { + "moe_tp_size", + "moe_ep_size", + } + deployment = f"world_size={world} tp={world} pp=1 moe_ep=1 moe_tp={world} attention_dp=False" + for resolved in [tp] + _unsplit(request, world): + seen = resolved.routed[0] + try: + target = _route(seen.pretrained_config, seen.mapping) + except NoTarget as refusal: + assert deployment in str(refusal) + else: + assert target.__name__ == _TP4_TARGET + assert target.__name__ not in _OTHER_TARGETS diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_tp4_indexer_packed_chunks.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_tp4_indexer_packed_chunks.py new file mode 100644 index 000000000000..d408ce504b12 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_tp4_indexer_packed_chunks.py @@ -0,0 +1,344 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The tp target's context indexer at the 1M config (max_seq_len 1048576, chunked prefill in +steps of max_num_tokens 32768) on prefill chunks the engine packs past max_seq_len keys. + +Under chunked prefill the engine plans the indexer's prefill chunks with +`plan_mla_chunked_prefill_chunks`: consecutive context requests share one chunk while (summed +new tokens) x (summed cached + new keys) stays within the packed-logits budget, 2^28 at +max_num_tokens 32768 and max_seq_len 1048576. A short last chunk of a near-1M request beside a +short prompt is therefore one chunk of more than 1048576 keys, past the key scorer's +(`fp8_mqa_logits`) and the prefill top-k's cells. The target cuts a chunk past max_seq_len keys, +and only such a chunk, at its request boundaries (`_indexer_blocks`) and scores each request's +rows against its own keys (`PrefillTarget._select_ctx`, in tiles under the engine's +`_prefill_logits_budget`, 1 << 28 here). + +On one GPU, each case's chunk is the one the engine's own planner and chunk builder +(`Indexer.prepare_one_prefill_chunk`) make at that config, over an indexer key pool the +case's full-KV gather mappings address through a page table (the engine's +`_compute_slot_mappings`, each request's pages drawn from one permutation of the pool), with +the step's offsets through the target's `_ctx_kv_offsets` and `_ctx_host_offsets`. Checked +per case: + +* the planner packs every request into one chunk, of the expected key count; +* the target's blocks: one per request where the chunk holds more than max_seq_len keys, the + chunk itself where it does not, each row covered once; +* no key gather, scorer or top-k call sees more than 1048576 keys, and every top-k call's + selection is an exact top-k of the scores it was given (rewritten canonically: ascending, + `-1` last, a tie at the k-th score given to the lowest-index keys, since the top-k promises + neither order nor which tied keys it keeps); +* every row is written, and each request's selections are exactly those of the same request + planned and selected alone. + +Skipped off sm_103 or without a GPU. +""" + +from __future__ import annotations + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch.attention.backends.sparse.dsa import indexer as engine +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_103__tp import ( # noqa: E501 + modeling as M, +) +from tensorrt_llm._utils import get_sm_version + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or get_sm_version() != 103, + reason="the mistral_large4 modeling_v2 targets are certified on sm_103 only", +) + +MAX_SEQ_LEN = 1 << 20 +MAX_NUM_TOKENS = 32768 +TOKENS_PER_BLOCK = 32 +INDEX_HEADS = 32 +INDEX_DIM = 128 +INDEX_TOPK = 2048 +#: The step's widest block table at max_seq_len 1048576 (V2: ceil((L + 1) / 32), to 4). +TABLE_WIDTH = 32772 +SENTINEL = -(2**31) +#: Elements the exact-top-k check holds at once. +_CHECK_ELEMENTS = 1 << 25 + +#: (name, ((cached, new) per context request, in batch order), the packed chunk's keys). +CASES = ( + ( + "a 1048559-token prompt's 100-token tail beside a 150-token prompt", + ((1048459, 100), (0, 150)), + 1048709, + ), + ( + "two 5-token tails over 983040 cached keys", + ((983040, 5), (983040, 5)), + 1966090, + ), + ( + "a 64-token tail of 1015872, a 100-token prompt and a 50-token tail of 32818", + ((1015808, 64), (0, 100), (32768, 50)), + 1048790, + ), + ( + "a 128-token tail and a 128-token prompt, exactly max_seq_len keys", + ((1048320, 128), (0, 128)), + 1048576, + ), + ( + "16 tails of 16 tokens over 61440 cached keys", + tuple((61440, 16) for _ in range(16)), + 983296, + ), +) + + +def _md(requests): + """The metadata the engine's planner, its chunk builder and the target's offsets read, for + a step whose context requests are `requests` (no decode rows).""" + cached = [c for c, _ in requests] + new = [n for _, n in requests] + kv = [c + n for c, n in requests] + + def indptr(values): + return torch.tensor([0, *torch.tensor(values).cumsum(0).tolist()], dtype=torch.int64) + + return SimpleNamespace( + compress_ratios=[1], + cu_seqlen_ks=torch.empty(0, dtype=torch.int32, device="cuda"), + max_num_tokens=MAX_NUM_TOKENS, + kv_cache_manager=SimpleNamespace(max_seq_len=MAX_SEQ_LEN), + enable_context_mla_with_cached_kv=True, + host_ctx_cached_token_indptr=indptr(cached), + host_ctx_uncached_token_indptr=indptr(new), + host_ctx_kv_indptr=indptr(kv), + ctx_cached_token_indptr=indptr(cached).cuda(), + ctx_kv_indptr=indptr(kv).cuda(), + max_ctx_seq_len=max(new), + num_ctx_cached_tokens=sum(cached), + ) + + +def _chunks(md, requests): + """The engine's indexer prefill chunks for the step, unsplit (fewer rows than + q_split_threshold).""" + q_lens = [n for _, n in requests] + kv_lens = [c + n for c, n in requests] + budget = engine._packed_logits_elem_budget(md) + assert budget == 1 << 28, budget + return [ + engine.Indexer.prepare_one_prefill_chunk(md, specs) + for specs in engine.plan_mla_chunked_prefill_chunks(q_lens, kv_lens, budget) + ] + + +def _step(md, requests, chunks, pool): + """The target's step dict for the indexer's context selection, through its own offsets and + block plan.""" + tc = sum(n for _, n in requests) + step = dict( + num_ctx_tokens=tc, + num_ctx_kv_tokens=sum(c + n for c, n in requests), + cu_seqlen_ks=None, + cu_seqlen_ke=None, + max_seq_len=MAX_SEQ_LEN, + slot_mapping_fp8_fullkv=pool.fp8, + slot_mapping_scale_fullkv=pool.scale, + **M._ctx_kv_offsets(md, len(requests)), + **M._ctx_host_offsets(md, len(requests)), + ) + step["indexer_blocks"] = M._indexer_blocks(chunks, step) + return step + + +def _pool(requests, gen, cuda_gen): + """An indexer key pool holding every request's keys at random e4m3 values and positive + scales, and the step's full-KV gather mappings (every request's cached and new keys, + request after request), each request's pages drawn from one permutation of the pool.""" + kv = [c + n for c, n in requests] + pages = [-(-k // TOKENS_PER_BLOCK) for k in kv] + num_pages = sum(pages) + 7 + order = torch.randperm(num_pages, generator=gen) + table = torch.zeros(len(requests), TABLE_WIDTH, dtype=torch.int32) + at = 0 + for r, p in enumerate(pages): + table[r, :p] = order[at : at + p].to(torch.int32) + at += p + req = torch.repeat_interleave(torch.arange(len(requests)), torch.tensor(kv)) + starts = torch.tensor([0, *torch.tensor(kv).cumsum(0).tolist()[:-1]]) + positions = torch.arange(sum(kv)) - starts.repeat_interleave(torch.tensor(kv)) + fp8, scale = engine._compute_slot_mappings( + positions, table, req, INDEX_DIM, TOKENS_PER_BLOCK, INDEX_DIM + ) + row = TOKENS_PER_BLOCK * (INDEX_DIM + 4) + flat = torch.empty(num_pages, row, dtype=torch.uint8, device="cuda") + data = torch.randn(num_pages, TOKENS_PER_BLOCK * INDEX_DIM, generator=cuda_gen, device="cuda") + flat[:, : TOKENS_PER_BLOCK * INDEX_DIM] = data.to(torch.float8_e4m3fn).view(torch.uint8) + scales = 0.5 + torch.rand(num_pages, TOKENS_PER_BLOCK, generator=cuda_gen, device="cuda") + flat[:, TOKENS_PER_BLOCK * INDEX_DIM :] = scales.view(torch.uint8) + return SimpleNamespace( + k_cache=flat.view(num_pages, TOKENS_PER_BLOCK, 1, INDEX_DIM + 4), + fp8=fp8.to(torch.int64).cuda(), + scale=scale.to(torch.int64).cuda(), + kv=kv, + ) + + +def _canonicalize(values, ks, ke, out, top_k: int) -> None: + """Check that `out` is an exact top-k of each row's window `[ks, ke)` of `values` + (window-local indices; a window within top_k kept whole, then `-1`) and rewrite it + canonically: ascending, `-1` last, a tie at the k-th score given to the lowest-index + keys.""" + n, width = values.shape + dev = values.device + cols = torch.arange(width, device=dev, dtype=torch.int64) + ident = torch.arange(top_k, device=dev, dtype=torch.int64) + big = torch.iinfo(torch.int64).max + step = max(1, _CHECK_ELEMENTS // max(width, 1)) + for r0 in range(0, n, step): + r1 = min(n, r0 + step) + s = ks[r0:r1].long()[:, None] + e = ke[r0:r1].long()[:, None] + length = (e - s)[:, 0] + assert int(e.max()) <= width, (int(e.max()), width) + raw = out[r0:r1].long() + raw_sorted = torch.where(raw < 0, big, raw).sort(dim=1).values + raw_sorted = torch.where(raw_sorted == big, -1, raw_sorted) + long_row = length > top_k + short = torch.where(ident < length[:, None], ident, -1) + bad_short = (~long_row) & (raw_sorted != short).any(1) + assert not bad_short.any(), f"{int(bad_short.sum())} short windows not kept whole" + canon = short + if long_row.any(): + sc = torch.where((cols >= s) & (cols < e), values[r0:r1].float(), float("-inf")) + kth = sc.topk(top_k, dim=1).values[:, -1:] + above = (sc > kth).sum(1) + at = sc == kth + inside = (raw_sorted >= 0) & (raw_sorted < length[:, None]) + distinct = (raw_sorted[:, 1:] != raw_sorted[:, :-1]).all(1) + picked = sc.gather(1, (s + raw_sorted.clamp(min=0)).clamp(max=width - 1)) + exact = ( + inside.all(1) + & distinct + & (picked.min(1).values == kth[:, 0]) + & ((picked > kth).sum(1) == above) + ) + bad = long_row & ~exact + assert not bad.any(), f"{int(bad.sum())} rows are not an exact top-{top_k}" + take = (sc > kth) | (at & (at.int().cumsum(1) <= (top_k - above)[:, None])) + best = torch.where(take, cols - s, big).sort(dim=1).values[:, :top_k] + canon = torch.where(long_row[:, None], best, short) + out[r0:r1].copy_(canon.to(out.dtype)) + + +@pytest.fixture +def calls(monkeypatch): + """Every key gather's, scorer's and top-k's key count, through the module names the target + calls them by; each top-k selection checked exact and made canonical where it is made.""" + seen = SimpleNamespace(gather=[], logits=[], topk=[]) + gather, logits = M.indexer_k_cache_gather_op, M.fp8_mqa_logits + topk = M.cute_dsl_gvr_topk_prefill + + def gather_(k_cache, fp8, scale, k_token_start, num_tokens, head_dim): + seen.gather.append((k_token_start, num_tokens)) + return gather(k_cache, fp8, scale, k_token_start, num_tokens, head_dim) + + def logits_(q, kv, w, ks, ke, *args, **kwargs): + seen.logits.append((q.shape[0], kv[0].shape[0])) + return logits(q, kv, w, ks, ke, *args, **kwargs) + + def topk_(values, ks, ke, out, top_k, *args, **kwargs): + seen.topk.append((values.shape[0], values.shape[1])) + result = topk(values, ks, ke, out, top_k, *args, **kwargs) + _canonicalize(values, ks, ke, out, top_k) + return result + + monkeypatch.setattr(M, "indexer_k_cache_gather_op", gather_) + monkeypatch.setattr(M, "fp8_mqa_logits", logits_) + monkeypatch.setattr(M, "cute_dsl_gvr_topk_prefill", topk_) + return seen + + +def _select(requests, chunks, md, pool, q_fp8, weights): + """The target's context selection over one step, off a core sized for this config (the + logits budget `_derive_first_step` sets from max_num_tokens and max_seq_len); the selection + buffer it fills.""" + step = _step(md, requests, chunks, pool) + tc = step["num_ctx_tokens"] + topk_buf = torch.full([tc, INDEX_TOPK], SENTINEL, dtype=torch.int32, device="cuda") + core = SimpleNamespace( + prefill_logits_budget=M._prefill_logits_budget(MAX_NUM_TOKENS, MAX_SEQ_LEN), + idx_dim=INDEX_DIM, + idx_topk=INDEX_TOPK, + _max_ctx_len=MAX_SEQ_LEN, + ) + target = M.PrefillTarget.__new__(M.PrefillTarget) + target.core = core + M.PrefillTarget._select_ctx(target, step, pool.k_cache, q_fp8, weights, topk_buf) + torch.cuda.synchronize() + return step, topk_buf + + +@pytest.mark.parametrize("name,requests,packed_keys", CASES, ids=[c[0] for c in CASES]) +def test_packed_chunk_selects_as_each_request_alone(name, requests, packed_keys, calls): + seed = len(name) * 7919 + packed_keys + gen = torch.Generator().manual_seed(seed) + cuda_gen = torch.Generator(device="cuda").manual_seed(seed) + md = _md(requests) + chunks = _chunks(md, requests) + # The engine packs every request into one chunk over all their keys. + tc = sum(n for _, n in requests) + assert len(chunks) == 1, [(c.token_start, c.token_end) for c in chunks] + (chunk,) = chunks + assert (chunk.token_start, chunk.token_end) == (0, tc) + assert chunk.k_token_end - chunk.k_token_start == packed_keys + + pool = _pool(requests, gen, cuda_gen) + q = torch.randn(tc, INDEX_HEADS, INDEX_DIM, generator=cuda_gen, device="cuda") + q_fp8 = q.to(torch.float8_e4m3fn) + weights = torch.randn(tc, INDEX_HEADS, generator=cuda_gen, device="cuda") + step, packed = _select(requests, chunks, md, pool, q_fp8, weights) + + # The blocks: the chunk itself within max_seq_len keys, else one per request. + blocks = step["indexer_blocks"] + cu_new, cu_kv = step["host_cu_ctx_new"], step["host_cu_ctx_kv"] + if packed_keys <= MAX_SEQ_LEN: + assert len(blocks) == 1 and blocks[0] is chunk, blocks + else: + expect = [(cu_new[r], cu_new[r + 1], cu_kv[r], cu_kv[r + 1]) for r in range(len(requests))] + got = [(b.token_start, b.token_end, b.k_token_start, b.k_token_end) for b in blocks] + assert got == expect, (got, expect) + for r, b in enumerate(blocks): + cached = requests[r][0] + assert torch.equal(b.cu_seqlen_ks.cpu(), torch.zeros(requests[r][1], dtype=torch.int32)) + ends = torch.arange(cached + 1, cached + requests[r][1] + 1, dtype=torch.int32) + assert torch.equal(b.cu_seqlen_ke.cpu(), ends) + # No call sees more than max_seq_len keys. + widest = max(n for _, n in calls.gather) + assert widest <= MAX_SEQ_LEN, calls.gather + assert max(n for _, n in calls.logits) <= MAX_SEQ_LEN, calls.logits + assert max(w for _, w in calls.topk) <= MAX_SEQ_LEN, calls.topk + assert int((packed == SENTINEL).any(dim=1).sum()) == 0, "rows left unwritten" + + # Each request alone: its own chunk over its own keys (the same pool slots). + for r, (cached, new) in enumerate(requests): + alone_req = ((cached, new),) + alone_md = _md(alone_req) + alone_chunks = _chunks(alone_md, alone_req) + lo, hi = cu_kv[r], cu_kv[r + 1] + alone_pool = SimpleNamespace( + k_cache=pool.k_cache, + fp8=pool.fp8[lo:hi].contiguous(), + scale=pool.scale[lo:hi].contiguous(), + ) + rows = slice(cu_new[r], cu_new[r + 1]) + _, alone = _select( + alone_req, alone_chunks, alone_md, alone_pool, q_fp8[rows], weights[rows] + ) + differ = int((packed[rows] != alone).any(dim=1).sum()) + assert differ == 0, f"{name}: request {r}, {differ} of {new} rows select otherwise alone" + print( + f"PACKED {name}: chunk keys {packed_keys}, blocks " + f"{[(b.token_start, b.token_end, b.k_token_end - b.k_token_start) for b in blocks]}, " + f"widest gather {widest}, top-k calls (rows, width) {sorted(set(calls.topk))}" + ) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_tp4_q_split.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_tp4_q_split.py new file mode 100644 index 000000000000..922b122fee47 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_tp4_q_split.py @@ -0,0 +1,1004 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The tp target's indexer query split across its TP ranks: at tp4 (four ranks, on one node of +four GPUs) and at tp8 (eight ranks over two nodes of four). + +Under chunked prefill, once a step's context rows reach `q_split_threshold` (8192), the engine +splits them over the TP ranks: `Indexer.prepare_for_chunked_prefill` (`_split_prefill_queries`, +then `plan_mla_chunked_prefill_chunks` over this rank's share) sets +`indexer_prefill_q_split = (q_begin, q_end, q_sizes)` and plans `indexer_prefill_chunks` over +rows `[q_begin, q_end)` only. `PrefillTarget._indexer` selects for those rows, then every full +indexer layer all-gathers every rank's share -- four at tp4, eight at tp8 -- (int32 +`[rows, 2048]`, uneven sizes in rank order) into rows `[0, tc)` of the step's selection buffer, +through the catalog's `comm/allgather` on the MPI session, before any layer reads it. + +Each rank builds the tp4 core on the meta device at its world's mapping and fills, through the +weight table's own manifest (its tensor-parallel slices -- 16 heads a rank at tp4, 8 at tp8 -- +and load-time transforms), the attention and indexer weights of three layers of the checkpoint: +1 (a full-indexer RoPE layer), 3 (a full-indexer NoPE layer: its indexer reads the identity +table) and 17 (a 'shared' layer, which attends with layer 3's selection). The inputs are seeded: +real token embeddings of seeded random ids from an 8192-token block of the vocabulary, through +each layer's own attention norm. The KV pools are the engine's `MistralLarge4CacheManager` built +from the checkpoint's own config (the fp8_ds_mla latent pool on the 63 KV owners, the indexer key +pool on the 44 full-indexer layers), and every step's metadata is `DSAtrtllmAttentionMetadata` +prepared by the engine itself (`prepare()` runs the indexer's planning) with chunked prefill on, +as the engine builds it (`enable_context_mla_with_cached_kv`, `chunk_size` 32768). Every key and +latent in the pools was written by the target's own appends (`ds_mla_rope_append` for the +latent), over the steps before. The engine is at max_seq_len 131072 with chunked prefill, and +the target sizes itself from that on the first step (`_derive_first_step`): its indexer logits +budget `_prefill_logits_budget` is the ceiling, 1 << 28 (2048-row tiles at 131072 keys), and its +held context workspace `_ctx_workspace_rows` 2**19 rows, since a step can attend cached prefixes. +Every step's context KV fits it, so the context rows attend on the workspace path: each +request's cached and new latent rows dequantized into the held workspace +(`PrefillTarget._ctx_workspace`, `ds_mla_dequant_gather`), then FlashMLA's sparse prefill over +it; every step after R's first chunk reads a cached prefix there. + +What "the same selection" can mean here is set by the top-k entries, not by the split. The +decode top-k returns a long row's 2048 winners in unspecified order and may break a tie at the +2048th score either way, and the prefill top-k (`cute_dsl_gvr_topk_prefill`) keeps the lowest +columns among such a tie, which another top-k need not; `fp8_mqa_logits` is bitwise per query +tile. So every step runs three times from identical pools, each through the target's +`_attention`: on the engine's split plan at this rank's TP mapping, and twice on the unsplit +plan (the same metadata with `q_split_threshold` past any step, so the chunks cover every +context row); the second unsplit run is the kernel's own run-to-run variation. An observe-only +wrapper around the target module's two top-k entries keeps, for every row it selects, what it +wrote and the scores of the keys it chose, sorted. Asserted for every step, layer and rank: + +* the engine's split is the expected one and its chunks tile exactly this rank's rows; +* the context workspace holds every context request's cached and new rows (the engine's + `num_ctx_kv_tokens` counts the new ones only); +* with the selection buffer filled with a sentinel before each full layer, no sentinel survives + anywhere in the step's rows; +* after the gather, this rank's rows are bitwise what its own top-k calls wrote, and the + context rows of the gathered buffer are bitwise the same on every rank; +* per row, the selected keys' scores are bitwise the unsplit computation's, as a multiset: the + two are the same top-k answer, up to the kernel's order and its choice among keys tied at the + 2048th score. Decode rows behind the context rows, which every rank scores itself, likewise; +* each layer's attention output (this rank's heads), recomputed through `_attention` with the + layer's indexer detached over each selection put in ascending order, is bitwise the unsplit + one on every row whose selection is the same set; the rows that differ (exact ties) are + counted and their output difference printed beside the unsplit-vs-unsplit control's. + +The steps, in order, on one pool, with the expected q_sizes at tp4; at tp8 they are the same +split over eight ranks (rank r takes `[tc * r // 8, tc * (r + 1) // 8)`): + +* request R, 98304 tokens in three 32768-row chunks (even split, 8192 rows per rank at tp4, 4096 + at tp8): over no prefix, a 32768-token prefix, then a 65536-token prefix, where each rank's + keys reach past 65536 and the 1 << 28 logits budget cuts its share into 3 or 4 tiles at tp4, 2 + at tp8; +* three first chunks in one step, 4096 + 20000 + 3000 = 27096 rows (6774 per rank at tp4, 3387 + at tp8): rank boundaries cut a request; +* 32766 context rows over three requests with cached prefixes of 4096, 0 and 20000 tokens + (q_sizes 8191, 8192, 8191, 8192; at tp8 4095, then 4096 three times, twice over), with two + decode rows behind them (R at 98304 and a 3000-token request): 32768 rows, a full step; +* 32769 context rows (q_sizes 8192, 8192, 8192, 8193; at tp8 seven of 4096 and one of 4097), one + request over a 30766-token prefix and a fresh one: one row past max_num_tokens, the split's + uneven-by-one case; +* 8191 context rows, just below the threshold, with R's next decode row: no split; +* 8192 context rows, at the threshold: split 2048 per rank at tp4, 1024 at tp8. + +A negative control on R's third chunk runs layer 1's indexer with the split's chunks and the +gather withheld: the other ranks' rows keep the sentinel, which is what the target without the +gather attended over. + +At tp4 the test starts the ranks itself: one `mpirun -n 4 --oversubscribe --allow-run-as-root` +job on this node (the launch line's MPI world, without `trtllm-llmapi-launch`, which runs a +command on rank 0 only), its environment scrubbed of the allocation's SLURM/PMI/Open MPI +variables, and reads the four ranks' results back; it runs unlaunched or inside a launch of four +ranks, on four GPUs. At tp8 each of the eight ranks is a pytest process an MPI launcher started +over two nodes of four, not under `trtllm-llmapi-launch` (e.g. `srun --mpi=pmix --nodes=2 +--ntasks-per-node=4 python -m pytest -k tp8`): each runs its rank in place on a device +of its node and every rank gathers every rank's results. A tp8 rank that fails prints its +traceback and leaves at once, which ends the launcher's job; after a pass every rank finalizes +MPI once pytest has reported. Each world is skipped inside a launch of another. Needs sm_103 and +the checkpoint under `LLM_MODELS_ROOT`; the per-step numbers are printed (run with -rA). +""" + +from __future__ import annotations + +import faulthandler +import hashlib +import json +import os +import shutil +import signal +import socket +import subprocess +import sys +import time +import traceback +from pathlib import Path +from types import SimpleNamespace + +import pytest +import torch + +#: A node's GPUs: tp4 runs its four ranks on one node, tp8 its eight over two. +RANKS_PER_NODE = 4 +#: The worlds: tp4, four ranks one mpirun job this test starts runs; tp8, eight ranks over two +#: nodes, each a pytest process an MPI launcher started. +WORLDS = [pytest.param(4, id="tp4"), pytest.param(8, id="tp8")] +CHECKPOINTS = ("Mistral-Large-4-1T-A52B-NVFP4", "mistral-large-4-nvfp4") +SHARD_INDEX = "consolidated.safetensors.index.json" + +LAYERS = 72 +TOKENS_PER_BLOCK = 32 +LATENT = 576 +INDEX_TOPK = 2048 +#: The checkpoint's index_sharing as the engine's DSA pattern: 17 indexer layers, then each +#: indexer pair followed by two layers that reuse its selection. +PATTERN = "F" * 17 + "SSFF" * 13 + "SSF" +#: The engine's token budget, which is also its chunked-prefill chunk. +MAX_NUM_TOKENS = 32768 +#: The engine's default `q_split_threshold`: a step of this many context rows or more has its +#: indexer queries split over the TP ranks. +Q_SPLIT_THRESHOLD = 8192 +#: The metadata's token capacity: room for the 32769-row step, one row past the engine's +#: max_num_tokens, which the split's arithmetic asks for. +STEP_TOKENS = 32776 +MAX_SEQ_LEN = 131072 +NUM_BLOCKS = 8192 +#: The layers driven, in order: full-indexer RoPE, full-indexer NoPE, and a 'shared' layer. +DRIVEN = (1, 3, 17) +#: The attention and indexer weight roles of a driven layer. +ATTN_ROLES = ( + "attn_norm", "wq", "wq_s", "ssmax_s", "ssmax_b", "wkv_a", "kv_a_norm", "wkv_b", + "ix_q_norm_w", "ix_q_norm_b", "ix_wq_b", "ix_wq_b_s", "ix_wk_wp", "ix_k_norm_w", + "ix_k_norm_b", "ix_ssmax_s", "ix_ssmax_b", +) # fmt: skip +#: int32, neither -1 (an empty slot) nor a position. +SENTINEL = -(2**31) + 7 +#: A threshold no step reaches: the engine then plans every context row on every rank. +UNSPLIT_THRESHOLD = 1 << 30 +SEED = 1051 +#: The token embeddings the inputs are drawn from: a contiguous block of the vocabulary. +EMB_LO, EMB_ROWS = 1000, 8192 +#: The tp4 job, and each tp8 rank's whole run (compiles and checkpoint reads included); past it a +#: tp8 rank dumps its stacks and exits, which ends the launcher's job. +DEADLINE_S = 3600 +_RANK_FLAG = "--q-split-rank" + +#: Request id -> prompt length, allocated when the request is first scheduled. +PROMPTS = {1: 98304, 2: 14096, 3: 47151, 4: 3000, 5: 12000, 6: 16384, 7: 8191, 8: 8192} +#: (name, context rows [(request, cached, new)], decode rows [(request, cached)], expected +#: q_sizes at tp4 or None; `_q_sizes` gives them at any world). +STEPS = ( + ("R chunk 1, no prefix", [(1, 0, 32768)], [], [8192] * 4), + ("R chunk 2, 32768 prefix", [(1, 32768, 32768)], [], [8192] * 4), + ("R chunk 3, 65536 prefix", [(1, 65536, 32768)], [], [8192] * 4), + ("three first chunks", [(2, 0, 4096), (3, 0, 20000), (4, 0, 3000)], [], [6774] * 4), + ( + "three prefixes + 2 decode rows", + [(2, 4096, 10000), (5, 0, 12000), (3, 20000, 10766)], + [(1, 98304), (4, 3000)], + [8191, 8192, 8191, 8192], + ), + ("32769 rows", [(3, 30766, 16385), (6, 0, 16384)], [], [8192, 8192, 8192, 8193]), + ("below the threshold + 1 decode row", [(7, 0, 8191)], [(1, 98305)], None), + ("at the threshold", [(8, 0, 8192)], [], [2048] * 4), +) +#: The step whose layer-1 indexer also runs with the gather withheld. +CONTROL_STEP = 2 + + +# ===================================================================== the ranks + + +def _mapping(rank: int, world: int): + """The engine's mapping at tp4 or tp8: tensor parallel throughout over `world` ranks on nodes + of four, the routed experts tensor parallel, no attention DP.""" + from tensorrt_llm.mapping import Mapping + + return Mapping( + world_size=world, + rank=rank, + gpus_per_node=RANKS_PER_NODE, + tp_size=world, + moe_tp_size=world, + moe_ep_size=1, + ) + + +def _sparse_config(**kwargs): + from tensorrt_llm.llmapi.llm_args import DeepSeekSparseAttentionConfig + + return DeepSeekSparseAttentionConfig( + index_n_heads=32, + index_head_dim=128, + index_topk=INDEX_TOPK, + indexer_rope_interleave=True, + **kwargs, + ) + + +def _manager(pretrained_config): + """The engine's KV cache manager for this checkpoint, from its own config: the fp8_ds_mla + latent pool (656 B per token on each of the 63 KV owners), page 32, and the 44-layer indexer + key pool.""" + from tensorrt_llm._torch.attention.backends.sparse.mistral_large4 import ( + MistralLarge4CacheManager, + ) + from tensorrt_llm.bindings import DataType + from tensorrt_llm.bindings.internal.batch_manager import CacheType + from tensorrt_llm.llmapi.llm_args import KvCacheConfig + from tensorrt_llm.mapping import Mapping + + assert pretrained_config.index_topk_pattern == PATTERN + return MistralLarge4CacheManager( + KvCacheConfig( + enable_block_reuse=False, max_tokens=NUM_BLOCKS * TOKENS_PER_BLOCK, host_cache_size=0 + ), + CacheType.SELFKONLY, + num_layers=LAYERS, + num_kv_heads=1, + head_dim=LATENT, + tokens_per_block=TOKENS_PER_BLOCK, + max_seq_len=MAX_SEQ_LEN, + max_batch_size=16, + mapping=Mapping(world_size=1, rank=0, tp_size=1, pp_size=1), + dtype=DataType.FP8, + sparse_attention_config=_sparse_config(), + pretrained_config=pretrained_config, + max_num_tokens=STEP_TOKENS, + ) + + +def _prepare(manager, ctx, gen, rank: int, world: int, **sparse): + """`DSAtrtllmAttentionMetadata` for one step, prepared as the engine prepares it under + chunked prefill: contexts first, each over its cached prefix, then the decode rows.""" + from tensorrt_llm._torch.attention.backends.interface import AttentionRuntimeFeatures + from tensorrt_llm._torch.attention.backends.sparse.dsa import DSAtrtllmAttentionMetadata + from tensorrt_llm._torch.metadata import KVCacheParams + + rids = [r for r, _, _ in ctx] + [r for r, _ in gen] + md = DSAtrtllmAttentionMetadata( + seq_lens=torch.tensor([n for _, _, n in ctx] + [1] * len(gen), dtype=torch.int), + request_ids=rids, + max_num_requests=16, + max_num_sequences=16, + num_contexts=len(ctx), + prompt_lens=[PROMPTS[r] for r in rids], + max_num_tokens=STEP_TOKENS, + kv_cache_manager=manager, + kv_cache_params=KVCacheParams( + use_cache=True, + num_cached_tokens_per_seq=[c for _, c, _ in ctx] + [c for _, c in gen], + ), + mapping=_mapping(rank, world), + runtime_features=AttentionRuntimeFeatures( + chunked_prefill=True, cache_reuse=False, chunk_size=MAX_NUM_TOKENS + ), + enable_context_mla_with_cached_kv=True, + sparse_attention_config=_sparse_config(**sparse), + ) + md.prepare() + return md + + +class _Checkpoint: + """Raw tensors by Mistral name, read lazily from the safetensors files, and the + configuration the engine adapts from its params.json.""" + + def __init__(self, path: Path) -> None: + from tensorrt_llm._torch.models.checkpoints.mistral.config_loader import MistralConfigLoader + + self.path = path + self.index = json.loads((path / SHARD_INDEX).read_text())["weight_map"] + self.config = MistralConfigLoader().load_pretrained_config(str(path)) + + def entry(self, key: str): + from safetensors import safe_open + + f = safe_open(self.path / self.index[key], framework="pt", device="cpu") + return f.get_slice(key) + + +def _materialize(W, core, ck: _Checkpoint, keys) -> None: + """Fill the named parameters of a meta-declared core through the weight table's manifest: + this rank's slices, multi-source rows and load-time transforms.""" + manifest = W.MODEL_WEIGHTS.manifest(core) + for key in keys: + meta = core.w[key] + dst = torch.empty(meta.shape, dtype=meta.dtype, device="cuda") + for ckpt_key, index, transform, shard in manifest[key]: + entry = ck.entry(ckpt_key) + src = entry[...] if shard is None else shard(entry) + src = src.to("cuda").contiguous() + if transform is not None: + src = transform(src, core) + (dst if index is None else dst[index]).copy_(src) + core.w[key] = torch.nn.Parameter(dst, requires_grad=False) + + +class _Recorder: + """Observe-only wrapper of the target module's two top-k entries. + + While armed with a step's selection buffer, it keeps per absolute row of that buffer what + each top-k call wrote there (`local`) and the scores of the keys it chose, sorted + descending (`scores`, -inf for an empty slot): the multiset that makes two top-k answers + the same up to the kernel's order and its choice among keys tied at the 2048th score. + Rows no call wrote stay SENTINEL / NaN.""" + + def __init__(self, M) -> None: + self.prefill_topk = M.cute_dsl_gvr_topk_prefill + self.decode_topk = M.indexer_topk_decode + M.cute_dsl_gvr_topk_prefill = self._prefill + M.indexer_topk_decode = self._decode + self.buf = None + + def arm(self, buf: torch.Tensor) -> None: + self.buf = buf + self.local = torch.full_like(buf, SENTINEL) + self.scores = torch.full(buf.shape, float("nan"), dtype=torch.float32, device=buf.device) + + def take(self) -> SimpleNamespace: + self.buf = None + return SimpleNamespace(local=self.local, scores=self.scores) + + def _record(self, out, logits, cols, valid) -> None: + if self.buf is None: + return + row_bytes = self.buf.stride(0) * self.buf.element_size() + r0 = (out.data_ptr() - self.buf.data_ptr()) // row_bytes + n = out.shape[0] + self.local[r0 : r0 + n] = out + chosen = torch.where(valid, torch.gather(logits, 1, cols), float("-inf")) + self.scores[r0 : r0 + n] = torch.sort(chosen, dim=1, descending=True).values + + def _prefill(self, logits, ks, ke, out, top_k, *args, **kwargs): + res = self.prefill_topk(logits, ks, ke, out, top_k, *args, **kwargs) + idx = out.long() + # Window-local indices: the score's column is the index plus the row's window start. + self._record(out, logits, (idx + ks.long()[:, None]).clamp(min=0), idx >= 0) + return res + + def _decode(self, logits, kv_lens, out, *args, **kwargs): + res = self.decode_topk(logits, kv_lens, out, *args, **kwargs) + idx = out.long() + self._record(out, logits, idx.clamp(min=0), idx >= 0) + return res + + +def _rmsnorm(x: torch.Tensor, w: torch.Tensor, eps: float) -> torch.Tensor: + xf = x.float() + xf = xf * torch.rsqrt(xf.pow(2).mean(-1, keepdim=True) + eps) + return xf.to(torch.bfloat16) * w + + +def _digest(t: torch.Tensor) -> str: + return hashlib.sha256(t.contiguous().cpu().numpy().tobytes()).hexdigest()[:16] + + +def _sentinel_rows(sel: torch.Tensor) -> int: + return int((sel == SENTINEL).any(dim=1).sum()) + + +def _canonical(sel: torch.Tensor) -> torch.Tensor: + """Each row's selection in ascending order, empty slots (-1) last.""" + big = torch.iinfo(torch.int32).max + ordered = sel.masked_fill(sel < 0, big).sort(dim=1).values + return ordered.masked_fill(ordered == big, -1) + + +def _chunks(step, budget: int) -> list[list[int]]: + """`[token_start, token_end, n_k, tiles, largest logits tile in bytes]` per engine chunk.""" + out = [] + for c in step["indexer_prefill_chunks"] or []: + rows, n_k = c.token_end - c.token_start, c.k_token_end - c.k_token_start + tile = max(1, budget // max(1, n_k)) + out.append([c.token_start, c.token_end, n_k, -(-rows // tile), min(tile, rows) * n_k * 4]) + return out + + +def _setup(prefill, md): + """The step's set-up, as the target's forward does it.""" + step = prefill.step_args(md) + rows, tc = int(step["num_tokens"]), int(step["num_ctx_tokens"]) + gen = rows - tc + skip_ctx = step["skip_ctx"] + skip_gen = step["skip_gen"] if gen else True + req_idx = step["req_idx_per_token"][:rows] + return SimpleNamespace( + step=step, + rows=rows, + tc=tc, + skip_ctx=skip_ctx, + skip_gen=skip_gen, + req_idx=req_idx, + scratch=prefill._scratch(rows, torch.bfloat16, "cuda"), + ctx=prefill._ctx_workspace(step, tc, "cuda") if tc else None, + ) + + +def _drive(env, md, xs, pos) -> SimpleNamespace: + """One step through the driven layers with the target's own `_attention`; the selection + buffer is refilled with the sentinel, and the recorder armed, before each full layer.""" + prefill = env.prefill + s = _setup(prefill, md) + topk_buf = prefill._topk_buf(s.step, s.rows, s.tc, s.skip_ctx, s.skip_gen) + outs, sels, recs, peak, ms = {}, {}, {}, {}, {} + for i in DRIVEN: + L = env.layers[i] + if L.indexer is not None: + topk_buf.fill_(SENTINEL) + env.recorder.arm(topk_buf) + torch.cuda.synchronize() + base = torch.cuda.memory_allocated() + torch.cuda.reset_peak_memory_stats() + started = time.monotonic() + out = prefill._attention( + s.step, L, xs[i], pos, s.req_idx, topk_buf, s.scratch, s.ctx, s.tc, s.skip_ctx, + s.skip_gen, + ) # fmt: skip + torch.cuda.synchronize() + ms[i] = (time.monotonic() - started) * 1e3 + peak[i] = torch.cuda.max_memory_allocated() - base + if L.indexer is not None: + recs[i] = env.recorder.take() + outs[i] = out.clone() + sels[i] = topk_buf.clone() + s.outs, s.sels, s.recs, s.peak, s.ms = outs, sels, recs, peak, ms + return s + + +def _attend(env, md, xs, pos, sels) -> dict: + """Each driven layer's attention through `_attention` with its indexer detached, over the + given selection put in ascending order.""" + prefill = env.prefill + s = _setup(prefill, md) + outs = {} + for i in DRIVEN: + detached = SimpleNamespace(**{**vars(env.layers[i]), "indexer": None}) + buf = _canonical(sels[i]) + out = prefill._attention( + s.step, detached, xs[i], pos, s.req_idx, buf, s.scratch, s.ctx, s.tc, s.skip_ctx, + s.skip_gen, + ) # fmt: skip + outs[i] = out.clone() + return outs + + +def _sets_differ(a: torch.Tensor, b: torch.Tensor) -> torch.Tensor: + return (_canonical(a) != _canonical(b)).any(dim=1) + + +def _max_abs(a: torch.Tensor, b: torch.Tensor) -> float: + if a.numel() == 0: + return 0.0 + return float((a.float() - b.float()).abs().max()) + + +def _rank_run(rank: int, world: int, ckpt_path: Path) -> dict: + from tensorrt_llm._torch.model_config import ModelConfig + from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_103__tp import ( # noqa: E501 + modeling as M, + ) + from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_103__tp import ( # noqa: E501 + weights as W, + ) + + ck = _Checkpoint(ckpt_path) + model_config = ModelConfig( + pretrained_config=ck.config, + mapping=_mapping(rank, world), + max_num_tokens=MAX_NUM_TOKENS, + max_seq_len=MAX_SEQ_LEN, + ) + with torch.device("meta"): + core = M.MistralLarge4TpModelingV2(model_config) + assert core.tp_group == list(range(world)) and core.tp_pg is None + # The engine's sizes are fixed on the first step (`_derive_first_step`), not at construction. + assert (core.prefill_logits_budget, core.ctx_workspace_rows) == (0, 0) + assert _sparse_config().q_split_threshold == Q_SPLIT_THRESHOLD + keys = ["lerope_alpha", "final_norm"] + keys += [f"l{i}_{r}" for i in DRIVEN for r in ATTN_ROLES if f"l{i}_{r}" in core.w] + _materialize(W, core, ck, keys) + core.build_kernels(torch.cuda.current_device()) + prefill = M.PrefillTarget(core) + env = SimpleNamespace( + prefill=prefill, + layers={L.index: L for L in prefill._layers}, + recorder=_Recorder(M), + ) + assert [env.layers[i].indexer is not None for i in DRIVEN] == [True, True, False] + assert env.layers[3].indexer.rope is False and env.layers[1].indexer.rope is True + + emb = ck.entry("tok_embeddings.weight")[EMB_LO : EMB_LO + EMB_ROWS].to("cuda") + eps = ck.config.rms_norm_eps + norms = {i: core.w[f"l{i}_attn_norm"] for i in DRIVEN} + gen_ids = torch.Generator().manual_seed(SEED) + ids = {r: torch.randint(0, EMB_ROWS, (n + 8,), generator=gen_ids) for r, n in PROMPTS.items()} + + manager = _manager(model_config.pretrained_config) + allocated: set[int] = set() + result = dict(rank=rank, budget=0, held_rows=0, steps=[]) + try: + for n_step, (name, ctx, gen, _) in enumerate(STEPS): + for r, _, _ in ctx: + if r not in allocated: + manager.add_dummy_requests([r], token_nums=[PROMPTS[r]]) + allocated.add(r) + for r, cached in gen: + # One more token, as a decode step allocates it. + cache = manager.kv_cache_map[r] + if cache.capacity < cached + 1: + assert cache.resize(cached + 1) + md_split = _prepare(manager, ctx, gen, rank, world) + md_ref = _prepare(manager, ctx, gen, rank, world, q_split_threshold=UNSPLIT_THRESHOLD) + assert md_split.enable_context_mla_with_cached_kv + assert md_split.runtime_features.chunked_prefill + core._derive_first_step(md_split) + # At max_seq_len 131072 with chunked prefill: the logits budget's ceiling, and the + # held workspace's 2**19 rows. + assert md_split.max_seq_len == MAX_SEQ_LEN + assert core.prefill_logits_budget == M._prefill_logits_budget( + md_split.max_num_tokens, MAX_SEQ_LEN + ) + assert core.ctx_workspace_rows == M._ctx_workspace_rows(md_split) == 1 << 19 + result["budget"] = core.prefill_logits_budget + result["held_rows"] = core.ctx_workspace_rows + tok = torch.cat( + [ids[r][c : c + n] for r, c, n in ctx] + [ids[r][c : c + 1] for r, c in gen] + ).to("cuda") + pos = torch.cat( + [torch.arange(c, c + n) for _, c, n in ctx] + [torch.tensor([c]) for _, c in gen] + ).to(device="cuda", dtype=torch.int32) + h = emb[tok] + xs = {i: _rmsnorm(h, norms[i], eps) for i in DRIVEN} + + split = _drive(env, md_split, xs, pos) + ref = _drive(env, md_ref, xs, pos) + ref2 = _drive(env, md_ref, xs, pos) + q_split = split.step["indexer_prefill_q_split"] + assert ref.step["indexer_prefill_q_split"] is None + rows, tc = split.rows, split.tc + begin, end = (0, tc) if q_split is None else (q_split[0], q_split[1]) + canon = dict( + split=_attend(env, md_split, xs, pos, split.sels), + ref=_attend(env, md_ref, xs, pos, ref.sels), + ) + rec = dict( + name=name, + rows=rows, + tc=tc, + skip_ctx=bool(split.skip_ctx), + q_split=None if q_split is None else [q_split[0], q_split[1], list(q_split[2])], + chunks_split=_chunks(split.step, core.prefill_logits_budget), + chunks_ref=_chunks(ref.step, core.prefill_logits_budget), + gather_bytes=0 if q_split is None else tc * INDEX_TOPK * 4, + # The context workspace's rows (every context request's cached + new latent + # rows) beside the engine's count of the step's new context keys. + workspace_rows=0 if split.ctx is None else int(split.ctx.ws_buf.shape[0]), + num_ctx_kv_tokens=int(split.step["num_ctx_kv_tokens"]), + layers={}, + ) + for i in DRIVEN: + full = env.layers[i].indexer is not None + a, b, b2 = split.sels[i], ref.sels[i], ref2.sels[i] + differ = _sets_differ(a, b) + same = ~differ + lay = dict( + full=full, + sentinel_rows=[_sentinel_rows(a), _sentinel_rows(b), _sentinel_rows(b2)], + rows_order_differ=int((a != b).any(dim=1).sum()), + rows_sets_differ=int(differ.sum()), + control_rows_order_differ=int((b2 != b).any(dim=1).sum()), + control_rows_sets_differ=int(_sets_differ(b2, b).sum()), + digest_split_ctx=_digest(a[:tc]), + out_native_max_abs=_max_abs(split.outs[i], ref.outs[i]), + control_out_native_max_abs=_max_abs(ref2.outs[i], ref.outs[i]), + out_canonical_same_rows_bitwise=bool( + torch.equal(canon["split"][i][same], canon["ref"][i][same]) + ), + out_canonical_same_rows=int(same.sum()), + out_canonical_tie_rows_max_abs=_max_abs( + canon["split"][i][differ], canon["ref"][i][differ] + ), + peak=[int(split.peak[i]), int(ref.peak[i])], + ms=[round(split.ms[i], 1), round(ref.ms[i], 1)], + ) + if full: + rs, rr, rr2 = split.recs[i], ref.recs[i], ref2.recs[i] + lay.update( + # This rank's rows of the gathered buffer are what its own calls wrote. + share_is_local=bool(torch.equal(a[begin:end], rs.local[begin:end])), + gen_is_local=bool(torch.equal(a[tc:], rs.local[tc:])), + scores_share_equal=bool( + torch.equal(rs.scores[begin:end], rr.scores[begin:end]) + ), + scores_gen_equal=bool(torch.equal(rs.scores[tc:], rr.scores[tc:])), + scores_ref_complete=bool(not torch.isnan(rr.scores[:rows]).any()), + control_scores_equal=bool(torch.equal(rr2.scores, rr.scores)), + ) + rec["layers"][str(i)] = lay + if n_step == CONTROL_STEP: + # The split's chunks with the gather withheld: the target without the gather. + withheld = dict(split.step, indexer_prefill_q_split=None) + buf = torch.full([rows, INDEX_TOPK], SENTINEL, dtype=torch.int32, device="cuda") + env.recorder.arm(buf) + prefill._indexer(withheld, xs[1], pos, env.layers[1], buf, tc, False, True) + kept = env.recorder.take() + rec["control"] = dict( + unwritten_rows=int((buf[:tc] == SENTINEL).all(dim=1).sum()), + expected=tc - (end - begin), + share_scores_equal=bool( + torch.equal(kept.scores[begin:end], split.recs[1].scores[begin:end]) + ), + ) + result["steps"].append(rec) + print(f"[rank {rank}] step {n_step} ({name}) done", flush=True) + del split, ref, ref2, canon, xs, h + finally: + manager.shutdown() + return result + + +def _rank_main(ckpt_dir: str, out_dir: str) -> None: + """Body of one rank of tp4's mpirun job. A failing rank exits nonzero without finalizing, + and mpirun then ends its siblings and fails.""" + rank = int(os.environ["OMPI_COMM_WORLD_RANK"]) + world = int(os.environ["OMPI_COMM_WORLD_SIZE"]) + torch.cuda.set_device(int(os.environ["OMPI_COMM_WORLD_LOCAL_RANK"])) + from mpi4py import MPI + + from tensorrt_llm._utils import mpi_comm + + comm = mpi_comm() + try: + assert world == RANKS_PER_NODE == comm.Get_size() and comm.Get_rank() == rank, ( + world, + comm.Get_size(), + comm.Get_rank(), + rank, + ) + result = _rank_run(rank, world, Path(ckpt_dir)) + (Path(out_dir) / f"rank{rank}.json").write_text(json.dumps(result, indent=1)) + except BaseException: + print(f"[rank {rank}] FAILED", flush=True) + traceback.print_exc() + sys.stdout.flush() + sys.stderr.flush() + os._exit(13) + comm.Barrier() + print(f"[rank {rank}] finished", flush=True) + sys.stdout.flush() + MPI.Finalize() + os._exit(0) + + +def _rank_in_place(ckpt_path: Path, world: int) -> tuple[int, list[dict]]: + """tp8: this pytest process is one rank of the launcher's world, two nodes of four. It runs + its rank on a device of its node, and every rank's results are gathered on every rank.""" + from mpi4py import MPI + + from tensorrt_llm._utils import mpi_comm, mpi_disabled + + comm = mpi_comm() + rank = comm.Get_rank() + # The node's ranks, one device each. + local = comm.Split_type(MPI.COMM_TYPE_SHARED) + torch.cuda.set_device(local.Get_rank() % torch.cuda.device_count()) + assert not mpi_disabled(), "the gather runs on the MPI session" + assert comm.Get_size() == world, (comm.Get_size(), world) + device = torch.cuda.current_device() + devices = comm.allgather(str(torch.cuda.get_device_properties(device).uuid)) + assert len(set(devices)) == world, f"every rank must own its own device: {devices}" + hosts = comm.allgather(socket.gethostname()) + print( + f"[rank {rank}] on {hosts[rank]} device {device}; ranks over {len(set(hosts))} node(s): " + f"{hosts}", + flush=True, + ) + result = _rank_run(rank, world, ckpt_path) + # As tp4's ranks hand theirs over: through JSON. + results = [json.loads(json.dumps(r)) for r in comm.allgather(result)] + print(f"[rank {rank}] finished", flush=True) + return rank, results + + +def _finalize(config, session) -> None: + """tp8, after pytest's report: MPI finalized (the launcher fails a rank that leaves without + it), then out at once with pytest's own status, ahead of the interpreter's teardown of the + ranks' communicators.""" + from mpi4py import MPI + + config.get_terminal_writer().flush() + for stream in (sys.stdout, sys.stderr, sys.__stdout__, sys.__stderr__): + stream.flush() + MPI.Finalize() + os._exit(int(session.exitstatus)) + + +def _join_ranks(request, ckpt_path: Path, world: int) -> tuple[int, list[dict]]: + """tp8: this rank's run among the launcher's ranks. A rank that fails prints its traceback + and leaves at once (the others wait in a collective for it), so the launcher ends the job + instead of the deadline.""" + from tensorrt_llm._utils import mpi_rank + + request.config.add_cleanup(lambda: _finalize(request.config, request.session)) + faulthandler.dump_traceback_later(DEADLINE_S, exit=True) + try: + return _rank_in_place(ckpt_path, world) + except BaseException: + print(f"[rank {mpi_rank()}] FAILED", flush=True) + traceback.print_exc() + sys.stdout.flush() + sys.stderr.flush() + os._exit(13) + finally: + faulthandler.cancel_dump_traceback_later() + + +# ===================================================================== the launcher + + +def _checkpoint_dir() -> Path | None: + from test_common.llm_data import llm_models_root + + root = llm_models_root() + if root is None: + return None + for name in CHECKPOINTS: + path = Path(root) / name + if (path / "params.json").is_file() and (path / SHARD_INDEX).is_file(): + return path + return None + + +def _session_members(sid: int) -> list[int]: + """Every process in session `sid`, read from /proc.""" + members = [] + for entry in Path("/proc").iterdir(): + if not entry.name.isdigit(): + continue + try: + stat = (entry / "stat").read_text() + except OSError: + continue + fields = stat[stat.rindex(")") + 2 :].split() + if int(fields[3]) == sid: + members.append(int(entry.name)) + return members + + +def _end_job(job: subprocess.Popen) -> None: + """End mpirun and every rank it started: mpirun is asked first, then its session's members + are killed (each rank sits in a process group of its own inside mpirun's session).""" + if job.poll() is None: + job.terminate() + try: + job.wait(timeout=30) + except subprocess.TimeoutExpired: + pass + for pid in _session_members(job.pid): + try: + os.kill(pid, signal.SIGKILL) + except ProcessLookupError: + pass + if job.poll() is None: + job.kill() + job.wait() + + +def _devices(world: int) -> str: + visible = os.environ.get("CUDA_VISIBLE_DEVICES") + if visible: + devices = [d.strip() for d in visible.split(",") if d.strip()] + else: + devices = [str(i) for i in range(torch.cuda.device_count())] + return ",".join(devices[:world]) + + +def _launch(ckpt: Path, out_dir: Path, world: int) -> str: + """tp4: the four ranks as one mpirun job on this node; its log.""" + mpirun = shutil.which("mpirun") + assert mpirun is not None, "the ranks are started by mpirun; none on PATH" + env = { + k: v + for k, v in os.environ.items() + if not k.startswith(("PMIX_", "PMI_", "OMPI_", "SLURM_")) and k != "TLLM_DISABLE_MPI" + } + env["CUDA_VISIBLE_DEVICES"] = _devices(world) + # tests/unittest/_torch/modeling_v2/models/mistral_large4// -> the repo root. + repo_root = Path(__file__).resolve().parents[7] + assert (repo_root / "tensorrt_llm").is_dir(), repo_root + env["PYTHONPATH"] = os.pathsep.join(p for p in (str(repo_root), env.get("PYTHONPATH")) if p) + command = [mpirun, "-n", str(world), "--oversubscribe", "--allow-run-as-root"] + command += [sys.executable, str(Path(__file__).resolve()), _RANK_FLAG, str(ckpt), str(out_dir)] + job = subprocess.Popen( + command, + env=env, + start_new_session=True, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + text=True, + ) + try: + log, _ = job.communicate(timeout=DEADLINE_S) + except subprocess.TimeoutExpired: + _end_job(job) + raise AssertionError(f"the {world}-rank job did not finish in {DEADLINE_S} s") from None + if job.returncode != 0: + _end_job(job) + raise AssertionError(f"the {world}-rank job failed: mpirun exited {job.returncode}\n{log}") + return log + + +def _q_sizes(ctx, tp4_sizes, world: int) -> list[int] | None: + """A step's expected q_sizes at `world` ranks: `_split_prefill_queries` over its `tc` context + rows, rank r taking `[tc * r // world, tc * (r + 1) // world)` from Q_SPLIT_THRESHOLD rows on + and no split below. At four ranks it is the step's listed tp4 expectation.""" + tc = sum(n for _, _, n in ctx) + if tc < Q_SPLIT_THRESHOLD: + sizes = None + else: + sizes = [tc * (r + 1) // world - tc * r // world for r in range(world)] + if world == RANKS_PER_NODE: + assert sizes == tp4_sizes, (ctx, sizes, tp4_sizes) + return sizes + + +def _expected_split(sizes, rank: int): + if sizes is None: + return None + begin = sum(sizes[:rank]) + return [begin, begin + sizes[rank], list(sizes)] + + +def _report(results: list[dict]) -> str: + lines = [ + f"prefill_logits_budget {results[0]['budget']}, held context workspace rows " + f"{results[0]['held_rows']}" + ] + for n_step, (name, *_rest) in enumerate(STEPS): + r0 = results[0]["steps"][n_step] + lines.append( + f"step {n_step}: {name}: rows {r0['rows']}, context rows {r0['tc']}, context " + f"workspace rows {r0['workspace_rows']} (num_ctx_kv_tokens {r0['num_ctx_kv_tokens']})" + ) + for res in results: + st = res["steps"][n_step] + tiles = sum(c[3] for c in st["chunks_split"]) + n_k = max((c[2] for c in st["chunks_split"]), default=0) + tile_mib = max((c[4] for c in st["chunks_split"]), default=0) / 2**20 + lines.append( + f" rank {res['rank']}: q_split {st['q_split']}; {len(st['chunks_split'])} " + f"chunk(s) {[c[:2] for c in st['chunks_split']]}, largest n_k {n_k}, {tiles} " + f"tile(s), largest tile {tile_mib:.0f} MiB, gathered block " + f"{st['gather_bytes'] / 2**20:.0f} MiB; unsplit {len(st['chunks_ref'])} " + f"chunk(s), {sum(c[3] for c in st['chunks_ref'])} tile(s)" + ) + for i, lay in st["layers"].items(): + head = f" layer {i}{'' if lay['full'] else ' (shared)'}" + lines.append( + f"{head}: sentinel rows {lay['sentinel_rows']}; rows differing in order " + f"{lay['rows_order_differ']} (control {lay['control_rows_order_differ']}), as " + f"sets {lay['rows_sets_differ']} (control {lay['control_rows_sets_differ']}); " + f"native output max |diff| {lay['out_native_max_abs']} (control " + f"{lay['control_out_native_max_abs']}); canonical output bitwise on " + f"{lay['out_canonical_same_rows']} same-set rows: " + f"{lay['out_canonical_same_rows_bitwise']}, tie rows max |diff| " + f"{lay['out_canonical_tie_rows_max_abs']}; peak {lay['peak'][0] / 2**30:.2f} " + f"/ {lay['peak'][1] / 2**30:.2f} GiB, {lay['ms'][0]} / {lay['ms'][1]} ms" + ) + if lay["full"]: + lines.append( + f" share is local {lay['share_is_local']}, decode rows local " + f"{lay['gen_is_local']}, scores equal: share {lay['scores_share_equal']}, " + f"decode rows {lay['scores_gen_equal']}, control " + f"{lay['control_scores_equal']}" + ) + if "control" in st: + lines.append(f" gather withheld (layer 1): {st['control']}") + return "\n".join(lines) + + +def _problems(results: list[dict], world: int) -> list[str]: + problems = [] + assert sorted(res["rank"] for res in results) == list(range(world)) + for n_step, (name, ctx, _gen, tp4_sizes) in enumerate(STEPS): + sizes = _q_sizes(ctx, tp4_sizes, world) + tc = sum(n for _, _, n in ctx) + for res in results: + rank, st = res["rank"], res["steps"][n_step] + where = f"step {n_step} ({name}) rank {rank}" + want = _expected_split(sizes, rank) + if st["q_split"] != want: + problems.append(f"{where}: q_split {st['q_split']}, expected {want}") + begin, end = (0, tc) if want is None else want[:2] + kv_rows = sum(c + n for _, c, n in ctx) + if st["workspace_rows"] != kv_rows: + problems.append( + f"{where}: the context workspace holds {st['workspace_rows']} rows, the " + f"context requests' cached + new KV is {kv_rows}" + ) + covered = sorted(c[:2] for c in st["chunks_split"]) + spans_ok = bool(covered) and covered[0][0] == begin and covered[-1][1] == end + spans_ok = spans_ok and all(a[1] == b[0] for a, b in zip(covered, covered[1:])) + if not spans_ok: + problems.append(f"{where}: chunks {covered} do not tile [{begin}, {end})") + for i, lay in st["layers"].items(): + checks = [any(lay["sentinel_rows"]), not lay["out_canonical_same_rows_bitwise"]] + if lay["full"]: + checks += [ + not lay["share_is_local"], + not lay["gen_is_local"], + not lay["scores_share_equal"], + not lay["scores_gen_equal"], + not lay["scores_ref_complete"], + not lay["control_scores_equal"], + ] + if any(checks): + problems.append(f"{where} layer {i}: {lay}") + # Gathered rows are one buffer on every rank; without the split each rank selects every + # row itself, in the kernel's own order. + for i in map(str, DRIVEN if sizes is not None else ()): + digests = {res["steps"][n_step]["layers"][i]["digest_split_ctx"] for res in results} + if len(digests) != 1: + problems.append(f"step {n_step} layer {i}: gathered rows differ by rank {digests}") + # Past 65536 keys the 1 << 28 budget cuts each rank's share of R's third chunk into tiles. + for res in results: + chunks = res["steps"][CONTROL_STEP]["chunks_split"] + if not (len(chunks) == 1 and chunks[0][2] > 65536 and chunks[0][3] > 1): + problems.append(f"rank {res['rank']}: R chunk 3 is not tiled past 65536: {chunks}") + control = res["steps"][CONTROL_STEP]["control"] + if not (control["unwritten_rows"] == control["expected"] > 0): + problems.append(f"rank {res['rank']}: the withheld gather left {control}") + if not control["share_scores_equal"]: + problems.append(f"rank {res['rank']}: the withheld gather's share {control}") + return problems + + +def _skip_unless_launched_for(world: int) -> None: + """tp4 starts its four ranks itself, so it runs unlaunched or inside a launch of four (pytest + on rank 0), on four GPUs. tp8 runs inside its ranks: this process must be one of eight MPI + ranks a launcher started, each a pytest process.""" + from tensorrt_llm._utils import mpi_world_size + from tensorrt_llm.executor.utils import get_spawn_proxy_process_env + from tensorrt_llm.llmapi.mpi_session import get_mpi_world_size + + launched = get_mpi_world_size() + if world == RANKS_PER_NODE: + if launched not in (1, world): + pytest.skip(f"tp4 starts its {world} ranks itself; this is a launch of {launched}") + if torch.cuda.device_count() < world: + pytest.skip(f"needs {world} GPUs, {torch.cuda.device_count()} visible") + return + in_place = mpi_world_size() + if get_spawn_proxy_process_env() or in_place != world: + pytest.skip( + f"tp8 runs in its {world} ranks, each a pytest process an MPI launcher started over " + f"two nodes of {RANKS_PER_NODE} (not under trtllm-llmapi-launch, which runs pytest on " + f"rank 0 only); this process is one of {in_place} (launched world {launched})" + ) + + +@pytest.mark.parametrize("world", WORLDS) +def test_q_split_matches_the_unsplit_computation(world, request, tmp_path): + _skip_unless_launched_for(world) + from tensorrt_llm._utils import get_sm_version + + if get_sm_version() != 103: + pytest.skip("the mistral_large4 modeling_v2 targets are certified on sm_103 only") + ckpt = _checkpoint_dir() + if ckpt is None: + pytest.skip(f"checkpoint {CHECKPOINTS} is not under LLM_MODELS_ROOT") + + if world == RANKS_PER_NODE: + log = _launch(ckpt, tmp_path, world) + results = [json.loads((tmp_path / f"rank{r}.json").read_text()) for r in range(world)] + print("\n" + _report(results)) + print("\n--- rank log (tail) ---\n" + "\n".join(log.splitlines()[-40:])) + else: + rank, results = _join_ranks(request, ckpt, world) + if rank == 0: + print("\n" + _report(results), flush=True) + + assert all(res["budget"] == 1 << 28 and res["held_rows"] == 1 << 19 for res in results) + problems = _problems(results, world) + assert not problems, "\n".join(problems) + + +if __name__ == "__main__" and _RANK_FLAG in sys.argv: + _at = sys.argv.index(_RANK_FLAG) + _rank_main(sys.argv[_at + 1], sys.argv[_at + 2]) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_tp4_retrieval_1m.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_tp4_retrieval_1m.py new file mode 100644 index 000000000000..d990634e11ab --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_tp4_retrieval_1m.py @@ -0,0 +1,244 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Retrieval at length on Mistral-Large-4-1T-A52B-NVFP4's sm_103 tp target at the 1M config, at +tp4 (four ranks) and at tp8 (eight ranks over two nodes of four). + +The checkpoint's own context is 1048576 positions (YaRN 512 x 2048, LeRoPE, SSMax). Here the whole +model runs under its serving configuration -- max_seq_len 1048576, chunked prefill in steps of +max_num_tokens 32768, sixteen sequences, the fp8_ds_mla latent cache at the default KV dtype, CUDA +graphs over batches 1, 2, 4, 8 and 16, padded -- and greedy decoding must recall a passkey hidden in +plain filler text, in prompts of 32768, 131072, 524288 and 1040384 tokens, at an early, a middle +and a late depth of each, one request at a time. + +Each prompt is prefilled in ceil(tokens / 32768) context steps of one request, read back from the +engine's iteration statistics: every chunk after the first runs over a cached prefix, its 32768 +queries split over the TP ranks (8192 a rank at tp4, 4096 at tp8, every full indexer layer +gathering the shares), the indexer selecting over the cached and the new keys and sparse MLA +attending over the selected latent rows of every earlier chunk and its own. While a step's +context KV (the request's cached and new rows) is at most 2**19 rows, the target's held workspace +at this configuration (`_ctx_workspace_rows`), those rows are dequantized out of the pool into it +for the sparse prefill kernel; the 524288-token prompt's last chunk reaches exactly 2**19 and +stays there. Past it -- the 1040384-token prompt's last 16 chunks -- every context row attends +straight off the fp8 pool as a one-row sequence of the sparse decode kernel, in sub-batches of +8192 rows: 1040384 is 31 full chunks and a last one of three full sub-batches, the sub-batch +size the decode kernel's 1M cells certify. That last chunk reads keys to 1040384 and positions +to 1040383. Then the answer decodes in the batch-1 graph over the whole prompt. Each prompt's +time to first token, its context steps and the KV pool blocks it held are printed. + +Text only: no image, no chat template. Iteration statistics are on to read the steps back; they +do not change what a step computes. + +Runs with the ranks already in place, on four GB300s at tp4 or on eight over two nodes of four +at tp8: pytest under `trtllm-llmapi-launch`, as the whole-model accuracy class runs, with the +checkpoint under `LLM_MODELS_ROOT` as `Mistral-Large-4-1T-A52B-NVFP4` (or +`mistral-large-4-nvfp4`) in its Mistral-native format, which the engine loads with +checkpoint_format "mistral". Skipped where the checkpoint is absent or off sm_103; each +world size is skipped below its own ranks (four GPUs at tp4, eight ranks at tp8) and inside a +launch of another world size. +""" + +from __future__ import annotations + +import math +import random +import time +from pathlib import Path + +import pytest +import torch +from test_common.llm_data import llm_models_root +from utils.util import skip_num_gpus_less_than + +from tensorrt_llm import LLM, SamplingParams +from tensorrt_llm._torch.modeling_v2 import MODELING_V2_VALIDATE_ENV +from tensorrt_llm._utils import get_sm_version +from tensorrt_llm.inputs import TokensPrompt +from tensorrt_llm.llmapi import CudaGraphConfig, KvCacheConfig +from tensorrt_llm.llmapi.mpi_session import get_mpi_world_size + + +def _ranks() -> int: + """The ranks this process can serve: the MPI world a launcher started (eight over two nodes + of four, where each node sees four devices), or, unlaunched, the devices the LLM API spawns + its workers on -- as the integration conftest's `skip_less_mpi_world_size` counts.""" + world = get_mpi_world_size() + return torch.cuda.device_count() if world == 1 else world + + +def _launched_elsewhere(world: int) -> bool: + """A launcher started an MPI world of another size: an engine of `world` ranks does not + start inside it.""" + return get_mpi_world_size() not in (1, world) + + +#: The engine's world sizes, each gated on its own ranks: tp4 on four GPUs, tp8 on eight ranks +#: over two nodes of four. +WORLDS = [ + pytest.param( + 4, + marks=[ + skip_num_gpus_less_than(4), + pytest.mark.skipif( + _launched_elsewhere(4), reason="tp4 needs a launch of four ranks or none" + ), + ], + id="tp4", + ), + pytest.param( + 8, + marks=[ + pytest.mark.skipif(_ranks() < 8, reason="tp8 needs eight ranks"), + pytest.mark.skipif( + _launched_elsewhere(8), reason="tp8 needs a launch of eight ranks or none" + ), + ], + id="tp8", + ), +] + +pytestmark = [ + pytest.mark.skipif( + get_sm_version() != 103, + reason="the mistral_large4 modeling_v2 targets are certified on sm_103 only", + ), + # One engine serves the module: the LLM API starts its result-dispatch thread on the first + # request, inside the first test, and stops it when the fixture shuts the engine down. + pytest.mark.threadleak(enabled=False), + # The first test of each world also loads its engine (about 10 minutes); a 1040000-token + # prompt took 105 s to its first token on four GB300s. + pytest.mark.timeout(1800), +] + +CHECKPOINTS = ("Mistral-Large-4-1T-A52B-NVFP4", "mistral-large-4-nvfp4") + +# The serving configuration (the core unit's LLM-API options): tp4 or tp8 (attention and the routed +# experts tensor parallel over all the ranks, the experts not expert-parallel), sixteen +# sequences, the checkpoint's own context prefilled in chunked steps of up to 32768 tokens. +MAX_NUM_TOKENS = 32768 +MAX_SEQ_LEN = 1048576 +ENGINE = dict( + moe_expert_parallel_size=1, + enable_attention_dp=False, + max_batch_size=16, + max_num_tokens=MAX_NUM_TOKENS, + max_seq_len=MAX_SEQ_LEN, + enable_chunked_prefill=True, + cuda_graph_config=CudaGraphConfig(max_batch_size=16, enable_padding=True), + kv_cache_config=KvCacheConfig(enable_block_reuse=False, free_gpu_memory_fraction=0.85), +) +TOKENS_PER_BLOCK = 32 + +# The longest: 31 chunks of 32768 and a last one of 3 x 8192, so that every decode-path sub-batch +# is a full 8192 rows (1040000 would end on one of 7808). +LENGTHS = (32768, 131072, 524288, 1040384) +DEPTHS = (0.1, 0.5, 0.9) + +HEAD = ( + "There is an important piece of information hidden inside a lot of irrelevant text. " + "Find it and memorize it. I will quiz you about it.\n\n" +) +FILLER = ( + "The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again. " +) +QUESTION = "\nWhat is the pass key? The pass key is" +# A five-digit key and its closing punctuation fit with room to spare. +ANSWER = SamplingParams(max_tokens=12, temperature=0.0) + + +def _checkpoint_dir() -> Path | None: + """The checkpoint's directory, found by its Mistral-native config, params.json.""" + root = llm_models_root() + if root is None: + return None + for name in CHECKPOINTS: + if (Path(root) / name / "params.json").is_file(): + return Path(root) / name + return None + + +def _passkey_ids(tokenizer, tokens: int, depth: float, key: int) -> list[int]: + """Exactly `tokens` ids: BOS, then the head, the filler with the needle at `depth` of it, and + the question, each part tokenized on its own and the filler cut at the token that gives the + length.""" + + def encode(text: str) -> list[int]: + ids = tokenizer.encode(text) + return ids[1:] if ids and ids[0] == tokenizer.bos_token_id else ids + + head, question = encode(HEAD), encode(QUESTION) + needle = encode(f"\nThe pass key is {key}. Remember it. {key} is the pass key.\n") + # A long run of the filler tokenizes a little shorter per sentence than a short one. + per_filler = len(encode(FILLER * 64)) / 64 + filler = encode(FILLER * math.ceil(1.05 * tokens / per_filler)) + room = tokens - 1 - len(head) - len(needle) - len(question) + assert 0 < room <= len(filler), (tokens, room, len(filler)) + before = int(room * depth) + ids = [tokenizer.bos_token_id, *head, *filler[:before], *needle] + ids += [*filler[: room - before], *question] + assert len(ids) == tokens, (len(ids), tokens) + return ids + + +def _stats(llm: LLM) -> tuple[list[tuple[int, int]], int]: + """Since the last call: (context requests, context tokens) of each iteration that ran context + rows, and the most KV pool blocks in use at any iteration.""" + steps, used = [], 0 + for record in llm.get_stats(timeout=2): + ifb = record.get("inflightBatchingStats") or {} + if ifb.get("numContextRequests", 0): + steps.append((ifb["numContextRequests"], ifb["numCtxTokens"])) + used = max(used, (record.get("kvCacheStats") or {}).get("usedNumBlocks", 0)) + return steps, used + + +@pytest.fixture(scope="module", params=WORLDS) +def llm(request): + world = request.param + path = _checkpoint_dir() + if path is None: + pytest.skip(f"checkpoint {CHECKPOINTS} is not under LLM_MODELS_ROOT ({llm_models_root()})") + with LLM( + str(path), + checkpoint_format="mistral", + modeling_v2="require", + env_overrides={MODELING_V2_VALIDATE_ENV: "1"}, + enable_iter_perf_stats=True, + tensor_parallel_size=world, + moe_tensor_parallel_size=world, + **ENGINE, + ) as engine: + print(f"RETRIEVAL tp{world} kv_cache_capacity {engine.get_kv_cache_capacity()}", flush=True) + _stats(engine) + yield engine + + +@pytest.mark.parametrize("depth", DEPTHS) +@pytest.mark.parametrize("tokens", LENGTHS) +def test_passkey_at_length(llm, tokens, depth): + """A passkey at `depth` of a `tokens`-token prompt, prefilled in 32768-token chunks, recalled + by greedy decoding.""" + key = random.Random(tokens + int(depth * 10)).randrange(10000, 100000) + ids = _passkey_ids(llm.tokenizer, tokens, depth, key) + + started_at = time.strftime("%H:%M:%S", time.gmtime()) + print(f"RETRIEVAL start tokens={len(ids)} depth={depth} at {started_at} UTC") + started = time.monotonic() + ttft, text = None, "" + for output in llm.generate_async(TokensPrompt(prompt_token_ids=ids), ANSWER, streaming=True): + if ttft is None and output.outputs[0].token_ids: + ttft = time.monotonic() - started + text = output.outputs[0].text + total = time.monotonic() - started + steps, used_blocks = _stats(llm) + print( + f"RETRIEVAL tokens={len(ids)} depth={depth} key={key} answer={text!r} " + f"ttft_s={ttft:.2f} total_s={total:.2f} context_steps={len(steps)} " + f"chunks={sorted(set(t for _, t in steps))} kv_blocks_used_max={used_blocks} " + f"({used_blocks * TOKENS_PER_BLOCK} tokens)", + flush=True, + ) + + assert all(n == 1 and t <= MAX_NUM_TOKENS for n, t in steps), steps + assert sum(t for _, t in steps) == len(ids), (len(ids), steps) + assert len(steps) == math.ceil(len(ids) / MAX_NUM_TOKENS), (len(ids), steps) + assert str(key) in text, (len(ids), depth, key, text) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_tp4_step_paths.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_tp4_step_paths.py new file mode 100644 index 000000000000..598e208e2684 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_tp4_step_paths.py @@ -0,0 +1,940 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""What the Mistral-Large-4-1T-A52B NVFP4 tp target chooses per step, and the engine sizes it +chooses against. No path is selected by the serving configuration: each context step picks its +own from its sizes, and the two sizes that bound it are derived on the first step +(`_derive_first_step`) from the engine's quantities on the attention metadata alone +(max_num_tokens, max_num_requests, max_seq_len, and runtime_features: chunked prefill and block +reuse, as the executor builds them from the LLM-API options); both are 0 until then. + +* The context path, by the step's context KV n_kv (every context request's cached and new rows): + while n_kv fits the held workspace, the workspace path over its first n_kv rows; past it, the + decode path, the context rows in sub-batches of at most `_CTX_DECODE_ROWS` with one FlashMLA + scheduler per sub-batch size and no workspace. The held workspace is allocated once, on the + first context step whichever path it takes, and every later step reuses its storage. +* Layer 35, the owner the NoPE layers 39, 43, ..., 71 attend over (`shared_source`, layer 35 + alone at every max_seq_len), by fit: while 2 n_kv fits the held rows it is gathered once into + the n_kv rows kept beside the step's own (`kept`), where 39 and 43 read it without a gather; + past that, with n_kv still within the held rows, every layer gathers its source into the + first n_kv rows, 35 again for 39 and for 43. +* The held workspace's rows (`_ctx_workspace_rows`): the most context KV one step can attend + (`_most_ctx_kv`) twice over, up to `_CTX_WORKSPACE_ROWS` (2**19). With neither chunked + prefill nor block reuse every context row is new, so it is twice max_num_tokens: 65536 at the + 32768 serving config (max_num_tokens = max_seq_len = 32768), 32768 at the Pareto config + (max_num_tokens 16384, max_seq_len 32768), 65536 at max_seq_len 131072 and max_num_tokens + 32768. With either, a step can attend each request's cached prefix up to max_seq_len, and it + is mainline's never-stall worst case min(max_num_requests, max_num_tokens) * max_seq_len twice + over, so 2**19: the 32768 config with block reuse (max_batch_size 16), the 1M config + (max_seq_len 1048576, chunked prefill) and max_seq_len 131072 with chunked prefill. Block + reuse is the runtime features' cache_reuse; neither size reads the KV cache manager. +* The indexer's prefill logits budget (`_prefill_logits_budget`): mainline's reachable + MQA-logits tile, within `_PREFILL_LOGITS_CEILING` (2**28) and within max_seq_len / 64 rows of + max_seq_len keys. 2**24 at max_seq_len 32768 (512-row tiles at 32768 keys), the 32768 and the + Pareto config alike; 2**28 at 131072 (2048-row tiles) and at 1048576 (256-row tiles). + TLLM_INDEXER_MQA_LOGITS_ELEM_BUDGET lowers it and cannot raise it. `_select_ctx` scores each + block's rows in tiles of budget // keys. +* The indexer's scoring blocks (`_indexer_blocks`): an engine chunk is one block, as planned, + while it holds at most max_seq_len keys, several requests included; one past max_seq_len + keys is cut at its requests over the host prefix sums `step_args` carries on every context + step. A block with no key fills its rows' selection slots with -1. +* The rope tables: `indexer` and `identity`, no `mla` table; `build_kernels` and + `_derive_first_step` build them with no such option. + +`_derive_first_step` runs on the core the target builds from the checkpoint's native config, +params.json, on the meta device (skipped where the checkpoint is absent under LLM_MODELS_ROOT); +everything else runs on host values, with recording stand-ins for the kernels the paths call. +""" + +from __future__ import annotations + +from types import SimpleNamespace + +import pytest +import torch +from test_common.llm_data import llm_models_root + +from tensorrt_llm._torch.attention.backends.interface import AttentionRuntimeFeatures +from tensorrt_llm._torch.attention.backends.sparse.params import ( + get_indexer_mqa_logits_workspace_bytes, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_103__tp import ( # noqa: E501 + modeling as M, +) + +pytestmark = pytest.mark.cpu_only + +CHECKPOINTS = ("Mistral-Large-4-1T-A52B-NVFP4", "mistral-large-4-nvfp4") +#: The checkpoint's layers, the owner its NoPE layers 39, 43, ..., 71 attend over, each layer's +#: latent source, the KV owners, and the latent row's width. +LAYERS = 72 +SHARED_SOURCE = 35 +KV_SOURCE = [SHARED_SOURCE if i > SHARED_SOURCE and i % 4 == 3 else i for i in range(LAYERS)] +KV_OWNERS = [i for i in range(LAYERS) if KV_SOURCE[i] == i] +LAT_DIM = 576 +#: The engine's max_batch_size, max_num_requests on the attention metadata. +MAX_BATCH = 16 +#: The mainline knob that lowers the reachable MQA-logits tile. +ELEM_BUDGET_ENV = "TLLM_INDEXER_MQA_LOGITS_ELEM_BUDGET" +#: Serving configs by what the target sizes from -- (max_num_tokens, max_seq_len, chunked +#: prefill, block reuse) at max_batch_size 16 -- with the held workspace's rows and the logits +#: budget they give. +CONFIGS = [ + pytest.param((32768, 32768, False, False), 65536, 1 << 24, id="32k"), + pytest.param((16384, 32768, False, False), 32768, 1 << 24, id="pareto"), + pytest.param((32768, 32768, False, True), 1 << 19, 1 << 24, id="32k_block_reuse"), + pytest.param((32768, 1048576, True, False), 1 << 19, 1 << 28, id="1m"), + pytest.param((32768, 131072, True, False), 1 << 19, 1 << 28, id="131072_chunked"), + pytest.param((32768, 131072, False, False), 65536, 1 << 28, id="131072"), +] +#: The checkpoint's rope surface (its native config, adapted from params.json), what +#: `_rope_tables` reads of it. +ROPE_CFG = SimpleNamespace( + rope_parameters=dict( + rope_type="yarn", + rope_theta=10000.0, + factor=2048.0, + original_max_position_embeddings=512, + beta_fast=32.0, + beta_slow=1.0, + mscale_all_dim=1, + attention_factor=1.0, + ), + qk_rope_head_dim=64, +) +CPU = torch.device("cpu") + + +def _core(max_num_tokens: int, max_seq_len: int | None): + """The core of rank 0 at four ranks, on the meta device, for an engine at `max_num_tokens` + and `max_seq_len`.""" + from tensorrt_llm._torch.model_config import ModelConfig + from tensorrt_llm._torch.models.checkpoints.mistral.config_loader import MistralConfigLoader + from tensorrt_llm.mapping import Mapping + + root = llm_models_root() + paths = [root / name for name in CHECKPOINTS] if root is not None else [] + path = next((p for p in paths if (p / "params.json").is_file()), None) + if path is None: + pytest.skip(f"none of {CHECKPOINTS} is under LLM_MODELS_ROOT ({root})") + mapping = Mapping( + world_size=4, rank=0, gpus_per_node=4, tp_size=4, pp_size=1, moe_ep_size=1, moe_tp_size=4 + ) + model_config = ModelConfig( + pretrained_config=MistralConfigLoader().load_pretrained_config(str(path)), + mapping=mapping, + max_num_tokens=max_num_tokens, + max_seq_len=max_seq_len, + ) + with torch.device("meta"): + return M.MistralLarge4TpModelingV2(model_config) + + +def _engine(max_num_tokens: int, max_seq_len: int, chunked: bool, reuse: bool) -> SimpleNamespace: + """What the target reads of the engine's attention metadata to size its buffers, with the + runtime features as `create_py_executor` builds them (chunked prefill `chunked`, block reuse + `reuse`, no drafter, the chunk max_num_tokens), and the KV cache manager surface the first + step checks.""" + return SimpleNamespace( + max_num_tokens=max_num_tokens, + max_num_requests=MAX_BATCH, + max_seq_len=max_seq_len, + runtime_features=AttentionRuntimeFeatures( + chunked_prefill=chunked, + cache_reuse=reuse, + has_speculative_draft_tokens=False, + chunk_size=max_num_tokens, + ), + kv_cache_manager=SimpleNamespace( + latent_layout=M._LATENT_LAYOUT, + kv_source_layers=KV_SOURCE, + latent_pool_row=lambda i: KV_OWNERS.index(KV_SOURCE[i]), + kv_owner_layers=KV_OWNERS, + ), + tokens_per_block=32, + ) + + +def _no_device(monkeypatch) -> list: + """Stand-ins for what `build_kernels` and `_derive_first_step` compile or read off a device. + The rope tables' stand-in takes `_rope_tables`'s parameters and no other, so a call with an + `mla` option fails; it records each table's rows.""" + built = [] + + def tables(cfg, alpha, positions, dtype, device): + built.append(positions) + return SimpleNamespace(rows=positions) + + monkeypatch.setattr(M, "_rope_tables", tables) + monkeypatch.setattr(M, "build_ix_q_side", lambda dev, is_neox: None) + monkeypatch.setattr(M, "build_ix_k_cache", lambda dev, is_neox, cache_contiguous: None) + monkeypatch.setattr(M, "build_ds_mla_decode", lambda dev, num_heads: None) + monkeypatch.setattr( + M, "build_ix_topk_decode", lambda dev, next_n: SimpleNamespace(next_n=next_n) + ) + monkeypatch.setattr(torch.cuda, "current_device", lambda: 0) + return built + + +def _cumsum(lens) -> torch.Tensor: + return torch.tensor([0, *lens], dtype=torch.int64).cumsum(0) + + +# ---- the engine sizes ---- + + +@pytest.mark.parametrize( + "max_seq_len", [8192, 32768, 32769, 1048576, None], ids=lambda n: f"max_seq_len{n}" +) +def test_core_selects_nothing_by_max_seq_len(max_seq_len): + """Built at any max_seq_len (None: the engine infers one, at least the checkpoint's + max_position_embeddings, 1048576), the core is the same: layer 35 is its one shared source, + and the logits budget and the held workspace's rows are 0 until the first step fixes them. + Nothing of a static switch is left: no `long_context`, no threshold, no fixed budget.""" + core = _core(32768, max_seq_len) + assert core.kv_source == KV_SOURCE + assert [i for i, keep in enumerate(core.shared_source) if keep] == [SHARED_SOURCE] + assert (core.prefill_logits_budget, core.ctx_workspace_rows) == (0, 0) + assert not hasattr(core, "long_context") + # The fused attention glue at every max_seq_len, unless `_UNFUSED_GLUE`. + assert core.unfused_glue is M._UNFUSED_GLUE + # The native generation-row decode at every max_seq_len, unless `_GEN_ATTN_FLASH_MLA`. + assert core.gen_attn_flash_mla is M._GEN_ATTN_FLASH_MLA + for name in ( + "_LONG_CONTEXT_THRESHOLD", + "_LONG_PREFILL_LOGITS_BUDGET", + "_PREFILL_LOGITS_BUDGET", + ): + assert not hasattr(M, name), name + + +@pytest.mark.parametrize("engine, rows, budget", CONFIGS) +def test_ctx_workspace_rows_from_runtime_features(engine, rows, budget): + """`_most_ctx_kv` is max_num_tokens with neither chunked prefill nor block reuse in the + runtime features, else mainline's never-stall worst case min(max_num_requests, + max_num_tokens) * max_seq_len; the held workspace has room for it twice, up to 2**19 rows.""" + max_num_tokens, max_seq_len, chunked, reuse = engine + md = _engine(*engine) + features = md.runtime_features + assert (features.chunked_prefill, features.cache_reuse) == (chunked, reuse) + most = min(MAX_BATCH, max_num_tokens) * max_seq_len if chunked or reuse else max_num_tokens + assert M._most_ctx_kv(md) == most + assert M._CTX_WORKSPACE_ROWS == 1 << 19 + assert M._ctx_workspace_rows(md) == min(2 * most, M._CTX_WORKSPACE_ROWS) == rows + + +@pytest.mark.parametrize( + "engine, rows", + [ + pytest.param((32768, 32768, False), (65536, 1 << 19), id="32k"), + pytest.param((16384, 32768, False), (32768, 1 << 19), id="pareto"), + pytest.param((32768, 1048576, True), (1 << 19, 1 << 19), id="1m"), + ], +) +def test_block_reuse_read_from_the_runtime_features(engine, rows): + """Block reuse is the attention metadata's `runtime_features.cache_reuse`, which the executor + sets from the KvCacheConfig's enable_block_reuse (`create_py_executor`, which keeps the two in + step where it turns reuse off); the sizes read the attention metadata alone, here one with no + KV cache manager at all. Without chunked prefill, cache_reuse off holds twice max_num_tokens + (65536 at the 32768 config, 32768 at the Pareto config) and on the worst case of 16 cached + prefixes of max_seq_len, 2**19 rows; under chunked prefill (the 1M config) the worst case + either way.""" + max_num_tokens, max_seq_len, chunked = engine + for reuse, want in zip((False, True), rows): + md = _engine(max_num_tokens, max_seq_len, chunked, reuse) + assert md.runtime_features.cache_reuse is reuse + metadata = SimpleNamespace( + max_num_tokens=md.max_num_tokens, + max_num_requests=md.max_num_requests, + max_seq_len=md.max_seq_len, + runtime_features=md.runtime_features, + ) + most = MAX_BATCH * max_seq_len if chunked or reuse else max_num_tokens + assert M._most_ctx_kv(metadata) == M._most_ctx_kv(md) == most + assert M._ctx_workspace_rows(metadata) == M._ctx_workspace_rows(md) == want + + +#: (max_num_tokens, max_seq_len, budget, rows a tile of max_seq_len keys holds). +BUDGETS = [ + pytest.param(32768, 32768, 1 << 24, 512, id="32k"), + pytest.param(16384, 32768, 1 << 24, 512, id="pareto"), + pytest.param(32768, 131072, 1 << 28, 2048, id="131072"), + pytest.param(32768, 1048576, 1 << 28, 256, id="1m"), + # A step budget under max_seq_len / 64 tokens: mainline's reachable tile binds. + pytest.param(256, 32768, 1 << 23, 256, id="reachable"), +] + + +@pytest.mark.parametrize("max_num_tokens, max_seq_len, budget, tile_rows", BUDGETS) +def test_prefill_logits_budget(monkeypatch, max_num_tokens, max_seq_len, budget, tile_rows): + """The budget is the least of mainline's reachable tile (min(2**31, max_num_tokens x + max_seq_len) logits), the ceiling 2**28, and max_seq_len / 64 rows of max_seq_len keys.""" + monkeypatch.delenv(ELEM_BUDGET_ENV, raising=False) + reachable = get_indexer_mqa_logits_workspace_bytes(max_num_tokens, max_seq_len) // 4 + assert reachable == min(1 << 31, max_num_tokens * max_seq_len) + got = M._prefill_logits_budget(max_num_tokens, max_seq_len) + assert got == min(reachable, M._PREFILL_LOGITS_CEILING, max_seq_len * (max_seq_len >> 6)) + assert got == budget and got // max_seq_len == tile_rows + + +def test_prefill_logits_budget_ceiling_and_env_override(monkeypatch): + """`_PREFILL_LOGITS_CEILING` is 2**28 logits, one 1 GiB fp32 tile. Under + TLLM_INDEXER_MQA_LOGITS_ELEM_BUDGET 2**20 every config's budget is 2**20 (32-row tiles at + 32768 keys); set past the ceiling it raises nothing.""" + assert M._PREFILL_LOGITS_CEILING == 1 << 28 + configs = [(32768, 32768), (16384, 32768), (32768, 131072), (32768, 1048576)] + monkeypatch.delenv(ELEM_BUDGET_ENV, raising=False) + unset = [M._prefill_logits_budget(n, s) for n, s in configs] + assert unset == [1 << 24, 1 << 24, 1 << 28, 1 << 28] + monkeypatch.setenv(ELEM_BUDGET_ENV, str(1 << 20)) + assert [M._prefill_logits_budget(n, s) for n, s in configs] == [1 << 20] * 4 + assert M._prefill_logits_budget(32768, 32768) // 32768 == 32 + monkeypatch.setenv(ELEM_BUDGET_ENV, str(1 << 30)) + assert [M._prefill_logits_budget(n, s) for n, s in configs] == unset + + +@pytest.mark.parametrize("engine, rows, budget", CONFIGS) +def test_derive_first_step_sets_the_engine_sizes(monkeypatch, engine, rows, budget): + """On the core built from the checkpoint's config at the engine's max_num_tokens and + max_seq_len, `_derive_first_step` sets the logits budget and the held workspace's rows to + the module helpers' values for the step's metadata (`build_kernels` built the rope tables at + max_seq_len, and they are not built again). The target's first context step, one fresh + request of max_num_tokens, then allocates the held workspace at those rows and keeps layer + 35's rows beside its own: the most context KV fits twice by construction.""" + monkeypatch.delenv(ELEM_BUDGET_ENV, raising=False) + built = _no_device(monkeypatch) + max_num_tokens, max_seq_len, _, _ = engine + core = _core(max_num_tokens, max_seq_len) + core.build_kernels(0) + md = _engine(*engine) + core._derive_first_step(md) + assert built == [max_seq_len] + assert core.prefill_logits_budget == M._prefill_logits_budget(max_num_tokens, max_seq_len) + assert core.prefill_logits_budget == budget + assert core.ctx_workspace_rows == M._ctx_workspace_rows(md) == rows + assert core._derived_for is md.kv_cache_manager and core._max_ctx_len == max_seq_len + + target = M.PrefillTarget.__new__(M.PrefillTarget) + target.core = core + step = _host_step([(0, max_num_tokens)]) + ctx = target._ctx_workspace(step, max_num_tokens, CPU) + assert target._ws_buf.shape == (rows, LAT_DIM) and target._ws_buf.dtype == torch.bfloat16 + assert ctx.ws_buf.shape == (max_num_tokens, LAT_DIM) and ctx.kept is not None + + +# ---- the rope tables ---- + + +def test_rope_tables_hold_no_mla_table(): + """`_rope_tables` returns its rows, the indexer's `[cos | sin]` table and the NoPE indexer + layers' identity table, nothing else, and takes no `mla` option; a longer table is the same + table with more rows.""" + alpha = torch.linspace(-0.5, 0.5, 32) + tables = M._rope_tables(ROPE_CFG, alpha, 300, torch.bfloat16, "cpu") + assert set(vars(tables)) == {"rows", "indexer", "identity"} + assert tables.rows == 300 + assert tables.indexer.shape == (300, 64) and tables.indexer.dtype == torch.float32 + assert torch.equal(tables.identity, torch.cat([torch.ones(300, 32), torch.zeros(300, 32)], 1)) + longer = M._rope_tables(ROPE_CFG, alpha, 600, torch.bfloat16, "cpu") + assert torch.equal(longer.indexer[:300], tables.indexer) + with pytest.raises(TypeError): + M._rope_tables(ROPE_CFG, alpha, 300, torch.bfloat16, "cpu", with_mla=True) + + +def test_rope_tables_built_and_grown_with_no_mla_option(monkeypatch): + """`build_kernels` builds the tables at the core's rows, and `_derive_first_step` grows them + to the engine's max_seq_len, both through `_rope_tables`'s own parameters alone.""" + built = _no_device(monkeypatch) + core = SimpleNamespace( + model_config=SimpleNamespace(pretrained_config=ROPE_CFG), + w={"lerope_alpha": None, "final_norm": torch.empty(0, dtype=torch.bfloat16)}, + rope_rows=16, + kv_source=KV_SOURCE, + num_layers=LAYERS, + _derived_for=None, + _ix_k=None, + unfused_glue=False, + heads=16, + gen_attn_flash_mla=False, + ) + M.MistralLarge4TpModelingV2.build_kernels(core, 0) + md = _engine(64, 64, False, False) + M.MistralLarge4TpModelingV2._derive_first_step(core, md) + assert built == [16, 64] + assert core.rope_rows == 64 and core._rope.rows == 64 + + +# ---- the generation rows' sparse decode + + +@pytest.mark.parametrize("max_seq_len", [8192, 32768, 32769, 1048576]) +@pytest.mark.parametrize("flash_mla", [False, True], ids=["native", "flash_mla"]) +def test_native_decode_built_at_every_max_seq_len(monkeypatch, max_seq_len, flash_mla): + """`build_kernels` compiles cute_dsl_ds_mla_sparse_decode at the rank's heads for the + generation rows whatever the engine's max_seq_len -- the 32K and the 1M configs alike -- and + builds no native handle only under `_GEN_ATTN_FLASH_MLA` (the core's `gen_attn_flash_mla`), + which keeps FlashMLA's 64-head decode. The shipped default is the native kernel.""" + assert M._GEN_ATTN_FLASH_MLA is False + built = [] + monkeypatch.setattr(M, "_rope_tables", lambda *a: SimpleNamespace(rows=a[2])) + monkeypatch.setattr(M, "build_ix_q_side", lambda dev, is_neox: None) + monkeypatch.setattr( + M, "build_ds_mla_decode", lambda dev, num_heads: built.append(num_heads) or "native" + ) + core = SimpleNamespace( + model_config=SimpleNamespace(pretrained_config=ROPE_CFG), + w={"lerope_alpha": None, "final_norm": torch.empty(0, dtype=torch.bfloat16)}, + rope_rows=max_seq_len, + heads=16, + unfused_glue=False, + gen_attn_flash_mla=flash_mla, + ) + M.MistralLarge4TpModelingV2.build_kernels(core, 0) + assert core._rope.rows == max_seq_len + assert core._mla_decode == (None if flash_mla else "native") + assert built == ([] if flash_mla else [16]) + + +# ---- the decode top-k's selector + + +#: Engine max_seq_lens, the decode scorer's column count, and whether cute_dsl_radix_topk_decode +#: serves it (`_TOPK_DECODE_COLUMNS`, 4096..65536): either side of both ends, the 32K and the 1M +#: configs and 131072. +TOPK_WIDTHS = [ + (2048, False), + (4095, False), + (4096, True), + (8192, True), + (32768, True), + (65536, True), + (65537, False), + (131072, False), + (1048576, False), +] + + +def _topk_md(max_seq_len: int, chunked: bool, reuse: bool) -> SimpleNamespace: + """A first step's metadata at `max_seq_len` (32768 tokens, 16 requests) on a fresh KV cache + manager, with chunked prefill and block reuse as given.""" + return SimpleNamespace( + max_num_tokens=32768, + max_num_requests=16, + max_seq_len=max_seq_len, + runtime_features=AttentionRuntimeFeatures( + chunked_prefill=chunked, cache_reuse=reuse, chunk_size=32768 + ), + kv_cache_manager=SimpleNamespace( + latent_layout=M._LATENT_LAYOUT, + kv_source_layers=KV_SOURCE, + latent_pool_row=lambda i: KV_OWNERS.index(KV_SOURCE[i]), + kv_owner_layers=KV_OWNERS, + ), + tokens_per_block=32, + ) + + +@pytest.mark.parametrize("max_seq_len, serves", TOPK_WIDTHS, ids=[str(w) for w, _ in TOPK_WIDTHS]) +def test_decode_topk_selector_by_the_scorer_width(monkeypatch, max_seq_len, serves): + """`_derive_first_step` holds cute_dsl_radix_topk_decode's handle exactly when the decode + scorer's column count, the engine's max_seq_len, is one the selector serves (4096..65536); + otherwise none, and both decode call sites run the trtllm indexer_topk_decode op. Nothing + else decides it: chunked prefill and block reuse, on or off, keep the one handle, built + once. A replaced KV cache manager at a width on the other side of the range flips it, and + `_TOPK_DECODE_CUTE_DSL` False builds none at any width.""" + assert M._TOPK_DECODE_CUTE_DSL is True + assert M._TOPK_DECODE_COLUMNS == (4096, 65536) + calls = [] + monkeypatch.setattr(M, "_rope_tables", lambda *a: SimpleNamespace(rows=a[2])) + monkeypatch.setattr(M, "build_ix_k_cache", lambda dev, is_neox, cache_contiguous: None) + monkeypatch.setattr( + M, "build_ix_topk_decode", lambda dev, next_n: calls.append(next_n) or object() + ) + monkeypatch.setattr(torch.cuda, "current_device", lambda: 0) + core = SimpleNamespace( + model_config=SimpleNamespace(pretrained_config=ROPE_CFG), + w={"lerope_alpha": None, "final_norm": torch.empty(0, dtype=torch.bfloat16)}, + rope_rows=16, + kv_source=KV_SOURCE, + num_layers=LAYERS, + _derived_for=None, + _ix_k=None, + _ix_topk=None, + _rope=SimpleNamespace(rows=16), + ) + derive = M.MistralLarge4TpModelingV2._derive_first_step + handles = [] + for chunked, reuse in ((False, False), (True, False), (False, True), (True, True)): + derive(core, _topk_md(max_seq_len, chunked, reuse)) + handles.append(core._ix_topk) + if serves: + assert calls == [1] and handles[0] is not None + assert all(h is handles[0] for h in handles) + else: + assert calls == [] and handles == [None] * 4 + derive(core, _topk_md(1048576 if serves else 32768, False, False)) + assert (core._ix_topk is None) is serves + monkeypatch.setattr(M, "_TOPK_DECODE_CUTE_DSL", False) + derive(core, _topk_md(32768, False, False)) + assert core._ix_topk is None + + +# ---- the indexer's context selection ---- + + +def _packed(): + """Two context requests of 40 cached + 8 new and 30 cached + 6 new tokens, packed by the + engine into one prefill chunk of 14 rows over 84 keys: the rows' windows `[ks, ke)` in the + chunk's keys, and the metadata's host prefix sums.""" + ks = torch.tensor([0] * 8 + [48] * 6, dtype=torch.int32) + ke = torch.tensor([*range(41, 49), *range(79, 85)], dtype=torch.int32) + chunk = SimpleNamespace( + token_start=0, + token_end=14, + k_token_start=0, + k_token_end=84, + cu_seqlen_ks=ks, + cu_seqlen_ke=ke, + ) + md = SimpleNamespace( + enable_context_mla_with_cached_kv=True, + host_ctx_uncached_token_indptr=torch.tensor([0, 8, 14]), + host_ctx_kv_indptr=torch.tensor([0, 48, 84]), + ) + return chunk, md + + +def _spans(blocks) -> list: + return [(b.token_start, b.token_end, b.k_token_start, b.k_token_end) for b in blocks] + + +def test_ctx_offsets_device_and_host(): + """`_ctx_kv_offsets` hands over mainline's device offsets only; `_ctx_host_offsets` the host + prefix sums of the new and of the total lengths: the engine's host indptrs under context MLA + over cached KV, else the prefix sum of the host seq_lens for both.""" + md = SimpleNamespace( + enable_context_mla_with_cached_kv=False, + mla_prepare_ctx_cu_seqlens=lambda: torch.tensor([0, 8, 14], dtype=torch.int32), + seq_lens=torch.tensor([8, 6, 1], dtype=torch.int32), + ) + assert set(M._ctx_kv_offsets(md, 2)) == { + "cu_ctx_cached", + "cu_ctx_kv", + "max_ctx_new", + "ctx_cached_tokens", + } + assert M._ctx_host_offsets(md, 2) == dict(host_cu_ctx_new=[0, 8, 14], host_cu_ctx_kv=[0, 8, 14]) + _, md = _packed() + assert M._ctx_host_offsets(md, 2) == dict( + host_cu_ctx_new=[0, 8, 14], host_cu_ctx_kv=[0, 48, 84] + ) + + +def test_indexer_blocks_cut_only_past_max_seq_len(): + """The packed chunk of 84 keys: at max_seq_len 64 cut at its requests, each request's + windows re-based to its own first key; at 84 and at 128 the very chunk, one block over both + requests. With no chunk planned, one chunk over every context row, cut the same way: two + fresh prompts of 40 and 30 tokens are one block within max_seq_len 128 and two past 64.""" + chunk, md = _packed() + step = dict(num_ctx_tokens=14, num_ctx_kv_tokens=84, **M._ctx_host_offsets(md, 2)) + step.update(cu_seqlen_ks=chunk.cu_seqlen_ks, cu_seqlen_ke=chunk.cu_seqlen_ke) + + blocks = M._indexer_blocks([chunk], dict(step, max_seq_len=64)) + assert _spans(blocks) == [(0, 8, 0, 48), (8, 14, 48, 84)] + assert blocks[0].cu_seqlen_ks.tolist() == [0] * 8 + assert blocks[0].cu_seqlen_ke.tolist() == list(range(41, 49)) + assert blocks[1].cu_seqlen_ks.tolist() == [0] * 6 + assert blocks[1].cu_seqlen_ke.tolist() == list(range(31, 37)) + for max_seq_len in (84, 128): + (whole,) = M._indexer_blocks([chunk], dict(step, max_seq_len=max_seq_len)) + assert whole is chunk + + fresh = dict( + num_ctx_tokens=70, + num_ctx_kv_tokens=70, + host_cu_ctx_new=[0, 40, 70], + host_cu_ctx_kv=[0, 40, 70], + cu_seqlen_ks=torch.tensor([0] * 40 + [40] * 30, dtype=torch.int32), + cu_seqlen_ke=torch.arange(1, 71, dtype=torch.int32), + ) + (whole,) = M._indexer_blocks(None, dict(fresh, max_seq_len=128)) + assert _spans([whole]) == [(0, 70, 0, 70)] + assert torch.equal(whole.cu_seqlen_ke, fresh["cu_seqlen_ke"]) + blocks = M._indexer_blocks(None, dict(fresh, max_seq_len=64)) + assert _spans(blocks) == [(0, 40, 0, 40), (40, 70, 40, 70)] + assert blocks[1].cu_seqlen_ks.tolist() == [0] * 30 + assert blocks[1].cu_seqlen_ke.tolist() == list(range(1, 31)) + + +def _step_md(**fields) -> SimpleNamespace: + """The metadata `step_args` reads, for the packed step of `_packed` (two context requests + over cached prefixes, one engine chunk), with `fields` overriding it.""" + chunk, host = _packed() + i64 = dict(dtype=torch.int64) + md = SimpleNamespace( + kv_cache_manager=SimpleNamespace( + get_latent_pool=lambda: torch.zeros(2, 3, 32, 656, dtype=torch.uint8), + get_indexer_k_cache_buffers=lambda i: None, + ), + num_seqs=2, + num_tokens=14, + kv_lens_cuda_runtime=None, + kv_lens_runtime=None, + prompt_lens_cpu_runtime=None, + kv_cache_block_offsets=None, + host_kv_cache_pool_pointers=None, + host_kv_cache_pool_mapping=None, + tokens_per_block=32, + max_seq_len=64, + block_table=None, + req_idx_per_token=None, + slot_mapping_fp8=None, + slot_mapping_scale=None, + skip_indexer_for_gen_reqs=False, + enable_indexer_skip=False, + indexer_k_cache_block_offsets=None, + scheduler_metadata_buffer=None, + radix_aux_indices=None, + radix_aux_logits=None, + num_contexts=2, + num_ctx_tokens=14, + seq_lens=torch.tensor([8, 6], dtype=torch.int32), + skip_indexer_for_ctx_reqs=False, + cu_seqlen_ks=chunk.cu_seqlen_ks, + cu_seqlen_ke=chunk.cu_seqlen_ke, + indexer_prefill_q_split=None, + indexer_prefill_chunks=[chunk], + num_ctx_kv_tokens=14, + slot_mapping_fp8_fullkv=None, + slot_mapping_scale_fullkv=None, + enable_context_mla_with_cached_kv=True, + ctx_cached_token_indptr=torch.tensor([0, 40, 70], **i64), + ctx_kv_indptr=torch.tensor([0, 48, 84], **i64), + max_ctx_seq_len=8, + num_ctx_cached_tokens=70, + host_ctx_uncached_token_indptr=host.host_ctx_uncached_token_indptr, + host_ctx_kv_indptr=host.host_ctx_kv_indptr, + ) + for name, value in fields.items(): + setattr(md, name, value) + return md + + +@pytest.mark.parametrize("max_seq_len, cut", [(64, True), (84, False), (128, False)]) +def test_step_args_carries_the_host_prefix_sums(max_seq_len, cut): + """`step_args` over the packed chunk of 84 keys carries the host prefix sums on every + context step and cuts the chunk only past max_seq_len keys.""" + md = _step_md(max_seq_len=max_seq_len) + step = _prefill_target(indexer_layers=[1]).step_args(md) + assert step["ctx_cached_tokens"] == 70 and int(step["cu_ctx_kv"][-1]) == 84 + assert (step["host_cu_ctx_new"], step["host_cu_ctx_kv"]) == ([0, 8, 14], [0, 48, 84]) + if cut: + assert _spans(step["indexer_blocks"]) == [(0, 8, 0, 48), (8, 14, 48, 84)] + else: + (whole,) = step["indexer_blocks"] + assert whole is md.indexer_prefill_chunks[0] + + +@pytest.mark.parametrize( + "max_seq_len, spans", + [(16, [(0, 14, 0, 14)]), (10, [(0, 8, 0, 8), (8, 14, 8, 14)])], + ids=["within_max_seq_len", "past_max_seq_len"], +) +def test_step_args_host_prefix_sums_without_cached_kv(max_seq_len, spans): + """Without context MLA over cached KV (neither chunked prefill nor block reuse) `step_args` + takes the host prefix sums off the host seq_lens, and the whole step's block, two fresh + prompts of 8 and 6 tokens planned by no chunk, is cut at its requests only past max_seq_len + keys, the second request's windows re-based to its own first key.""" + md = _step_md( + max_seq_len=max_seq_len, + enable_context_mla_with_cached_kv=False, + mla_prepare_ctx_cu_seqlens=lambda: torch.tensor([0, 8, 14], dtype=torch.int32), + indexer_prefill_chunks=None, + cu_seqlen_ks=torch.tensor([0] * 8 + [8] * 6, dtype=torch.int32), + cu_seqlen_ke=torch.arange(1, 15, dtype=torch.int32), + ) + step = _prefill_target(indexer_layers=[1]).step_args(md) + assert step["ctx_cached_tokens"] == 0 and step["cu_ctx_kv"].tolist() == [0, 8, 14] + assert (step["host_cu_ctx_new"], step["host_cu_ctx_kv"]) == ([0, 8, 14], [0, 8, 14]) + blocks = step["indexer_blocks"] + assert _spans(blocks) == spans + if len(blocks) == 2: + assert blocks[1].cu_seqlen_ks.tolist() == [0] * 6 + assert blocks[1].cu_seqlen_ke.tolist() == list(range(1, 7)) + + +def _recorded_kernels(monkeypatch) -> list: + """Stand-ins for the context selection's kernels, recording each key gather's + `(k_token_start, n_k)`, each scored tile's `(rows, keys)` and each top-k call's window + ends. The gathered keys are zero-stride views: nothing of n_k x 128 is allocated. Each + tile's split-KV schedule is a marker of the windows and keys it was built for, which its + logits call must be handed (checked here), and the top-k's scratch a marker the target + must hand every top-k call.""" + calls = [] + + def gather(pool, fp8, scale, k_token_start, n_k, dim): + calls.append(("gather", k_token_start, n_k)) + return torch.zeros(1, dim).expand(n_k, dim), torch.zeros(1, 1).expand(n_k, 1) + + def schedule(ks, ke, n_k, heads): + return ("schedule", ks.tolist(), ke.tolist(), n_k, heads) + + def logits(q, kv, weights, ks, ke, schedule_meta=None): + assert schedule_meta == ("schedule", ks.tolist(), ke.tolist(), kv[0].shape[0], q.shape[1]) + calls.append(("logits", q.shape[0], kv[0].shape[0])) + return torch.empty(0) + + def topk(scores, ks, ke, out, k, workspace=None): + assert workspace == "gvr-scratch" + calls.append(("topk", ks.shape[0], int(ke[-1]))) + + monkeypatch.setattr(M, "indexer_k_cache_gather_op", gather) + monkeypatch.setattr(M, "mqa_logits_schedule", schedule) + monkeypatch.setattr(M, "fp8_mqa_logits", logits) + monkeypatch.setattr(M, "cute_dsl_gvr_topk_prefill", topk) + monkeypatch.setattr(M, "gvr_workspace", lambda device: "gvr-scratch") + monkeypatch.setattr(M, "gvr_warmup", lambda top_k, cols, elems, device, ws: 0) + return calls + + +def _prefill_target(**core) -> M.PrefillTarget: + target = M.PrefillTarget.__new__(M.PrefillTarget) + # The selection's top-k warm-up reads the engine's max_seq_len off the core. + core.setdefault("_max_ctx_len", 1 << 20) + target.core = SimpleNamespace(**core) + return target + + +@pytest.mark.parametrize( + "max_num_tokens, max_seq_len, env, rows, tiles", + [ + pytest.param(32768, 32768, None, 1024, [512, 512], id="32k"), + pytest.param(16384, 32768, None, 1024, [512, 512], id="pareto"), + pytest.param(32768, 131072, None, 4096, [2048, 2048], id="131072"), + pytest.param(32768, 1048576, None, 512, [256, 256], id="1m"), + pytest.param(32768, 32768, 1 << 20, 64, [32, 32], id="env_2**20"), + ], +) +def test_select_ctx_tiles_under_the_budget( + monkeypatch, max_num_tokens, max_seq_len, env, rows, tiles +): + """`rows` context rows over max_seq_len keys, scored in tiles of the engine's budget // + keys: 512 rows at 32768 keys under 2**24, 2048 at 131072 and 256 at 1048576 under 2**28, + 32 at 32768 under TLLM_INDEXER_MQA_LOGITS_ELEM_BUDGET 2**20. The keys are gathered once, + every tile is within the budget, and every row is selected once.""" + if env is None: + monkeypatch.delenv(ELEM_BUDGET_ENV, raising=False) + else: + monkeypatch.setenv(ELEM_BUDGET_ENV, str(env)) + budget = M._prefill_logits_budget(max_num_tokens, max_seq_len) + calls = _recorded_kernels(monkeypatch) + keys = max_seq_len + block = SimpleNamespace( + token_start=0, + token_end=rows, + k_token_start=0, + k_token_end=keys, + cu_seqlen_ks=torch.zeros(rows, dtype=torch.int32), + cu_seqlen_ke=torch.arange(keys - rows + 1, keys + 1, dtype=torch.int32), + ) + step = dict( + indexer_blocks=[block], slot_mapping_fp8_fullkv=None, slot_mapping_scale_fullkv=None + ) + target = _prefill_target(idx_dim=128, idx_topk=2048, prefill_logits_budget=budget) + q, w, out = torch.empty(rows, 1), torch.empty(rows, 1), torch.empty(rows, 1) + target._select_ctx(step, None, q, w, out) + assert [c for c in calls if c[0] == "gather"] == [("gather", 0, keys)] + scored = [c[1:] for c in calls if c[0] == "logits"] + assert [n for n, _ in scored] == tiles + assert all(n * k <= budget for n, k in scored) + assert sum(c[1] for c in calls if c[0] == "topk") == rows + + +def test_select_ctx_block_with_no_key(monkeypatch): + """A block of rows with no key fills its rows' selection slots with -1 and leaves every + other row's as they are; nothing is gathered, scored or selected for it.""" + calls = _recorded_kernels(monkeypatch) + block = SimpleNamespace( + token_start=2, + token_end=5, + k_token_start=7, + k_token_end=7, + cu_seqlen_ks=torch.zeros(3, dtype=torch.int32), + cu_seqlen_ke=torch.zeros(3, dtype=torch.int32), + ) + step = dict( + indexer_blocks=[block], slot_mapping_fp8_fullkv=None, slot_mapping_scale_fullkv=None + ) + target = _prefill_target(idx_dim=128, idx_topk=4, prefill_logits_budget=1 << 24) + out = torch.full((6, 4), 9, dtype=torch.int32) + target._select_ctx(step, None, torch.empty(6, 1), torch.empty(6, 1), out) + assert calls == [] + assert out[:2].eq(9).all() and out[5:].eq(9).all() + assert out[2:5].eq(-1).all() + + +# ---- the context workspace and the context attention ---- + + +def _host_step(contexts) -> dict: + """The step fields `_ctx_workspace` and `_attn_ctx` read, for context requests of + (cached, new) tokens.""" + return dict( + num_contexts=len(contexts), + num_ctx_tokens=sum(n for _, n in contexts), + ctx_cached_tokens=sum(c for c, _ in contexts), + cu_ctx_kv=_cumsum([c + n for c, n in contexts]), + latent_pool=None, + latent_kv=None, + block_table=None, + tokens_per_block=32, + ) + + +def _attn_target(held: int) -> M.PrefillTarget: + """A prefill target over what its context workspace and attention read off the core: the + held workspace's rows, and `shared_source` as the core builds it, layer 35 alone (checked + on the core itself by `test_core_selects_nothing_by_max_seq_len`).""" + return _prefill_target( + lat_dim=LAT_DIM, + ctx_workspace_rows=held, + shared_source=[i == SHARED_SOURCE for i in range(LAYERS)], + softmax_scale=1.0, + idx_topk=4, + _pool_row=list(range(LAYERS)), + _stride_factor=1, + ) + + +def test_context_path_by_the_steps_context_kv(): + """Held rows 64. Steps whose context KV (cached + new over the context requests) is 16, 32, + 40 and 64 take the workspace path, the held buffer's first n_kv rows, with the next n_kv + kept beside them at 16 and 32 (2 n_kv within 64) and none at 40 and 64; one row past, the + decode path, with no workspace. Every step reads the one buffer the first allocated.""" + target = _attn_target(64) + assert target._ws_buf is None + buf = None + for contexts, kept in ( + ([(10, 6)], True), + ([(20, 4), (0, 8)], True), + ([(30, 10)], False), + ([(60, 4)], False), + ): + step = _host_step(contexts) + n_kv = step["ctx_cached_tokens"] + step["num_ctx_tokens"] + ctx = target._ctx_workspace(step, step["num_ctx_tokens"], CPU) + buf = target._ws_buf if buf is None else buf + assert target._ws_buf is buf + assert buf.shape == (64, LAT_DIM) and buf.dtype == torch.bfloat16 + assert ctx.ws_buf.shape == (n_kv, LAT_DIM) and ctx.ws_buf.data_ptr() == buf.data_ptr() + kv = [c + n for c, n in contexts] + assert ctx.kv_req.tolist() == [r for r, n in enumerate(kv) for _ in range(n)] + assert ctx.kv_pos.tolist() == [p for n in kv for p in range(n)] + assert ctx.ws_off.tolist() == _cumsum(kv)[:-1].tolist() and ctx.ws == {} + if kept: + assert ctx.kept.shape == (n_kv, LAT_DIM) + assert ctx.kept.data_ptr() == buf[n_kv].data_ptr() + else: + assert ctx.kept is None + + ctx = target._ctx_workspace(_host_step([(60, 5)]), 5, CPU) + assert ctx.ws_buf is None and not hasattr(ctx, "kv_req") + assert ctx.sub_batches == [(0, 5)] and sorted(ctx.sched) == [5] + assert target._ws_buf is buf + + +def test_held_workspace_allocated_on_a_first_step_past_it(): + """The held workspace is allocated on the first context step even when that step takes the + decode path: 20000 new rows past 64 held, in sub-batches of 8192 with one scheduler per + sub-batch size. The next step, within the held rows, reads that buffer.""" + target = _attn_target(64) + ctx = target._ctx_workspace(_host_step([(0, 20000)]), 20000, CPU) + buf = target._ws_buf + assert buf.shape == (64, LAT_DIM) and buf.dtype == torch.bfloat16 + assert ctx.ws_buf is None + assert M._CTX_DECODE_ROWS == 8192 + assert ctx.sub_batches == [(0, 8192), (8192, 16384), (16384, 20000)] + assert sorted(ctx.sched) == [3616, 8192] + assert not any(m.have_initialized for m in ctx.sched.values()) + ctx = target._ctx_workspace(_host_step([(4, 4)]), 4, CPU) + assert target._ws_buf is buf and ctx.ws_buf.data_ptr() == buf.data_ptr() + + +@pytest.mark.parametrize("held, kept", [(64, True), (24, False)], ids=["fits_twice", "fits_once"]) +def test_nope_source_kept_or_gathered_again_by_fit(monkeypatch, held, kept): + """Layers 35, 39 (attending over 35's latent), 40 and 43 (over 35's) on two steps of two + context requests of 10 and 6 KV rows (16). Held 64, so 2 x 16 fits: layer 35 is gathered + once into the 16 rows kept beside the step's, which 39 and 43 read without a gather, and 40 + into the step's first 16. Held 24, so only 16 fits: every layer gathers its source into the + step's first 16 rows, 35 again for 39 and for 43. Both steps run on the one held buffer.""" + gathers, prefills = [], [] + + def gather(pool, row, kv_req, kv_pos, block_table, out): + gathers.append((row, out)) + return out + + monkeypatch.setattr(M, "ds_mla_dequant_gather", gather) + monkeypatch.setattr( + M, "flash_mla_sparse_prefill", lambda q, ws, idx, scale, out: prefills.append(ws) + ) + target = _attn_target(held) + step = _host_step([(7, 3), (0, 6)]) + req_idx = torch.tensor([0] * 3 + [1] * 6) + sel = torch.zeros(9, 4, dtype=torch.int32) + buf = None + for _ in range(2): + gathers.clear() + prefills.clear() + ctx = target._ctx_workspace(step, 9, CPU) + buf = target._ws_buf if buf is None else buf + assert target._ws_buf is buf and buf.shape == (held, LAT_DIM) + assert ctx.ws_buf.shape == (16, LAT_DIM) and ctx.ws_buf.data_ptr() == buf.data_ptr() + assert ctx.ws == {} + for layer in (SHARED_SOURCE, 39, 40, 43): + L = SimpleNamespace(kv_source=KV_SOURCE[layer]) + target._attn_ctx(step, L, torch.empty(9, 1), torch.empty(9, 1), req_idx, sel, ctx) + if kept: + assert ctx.kept.shape == (16, LAT_DIM) and ctx.kept.data_ptr() == buf[16].data_ptr() + assert [row for row, _ in gathers] == [SHARED_SOURCE, 40] + assert gathers[0][1] is ctx.kept and gathers[1][1] is ctx.ws_buf + assert ctx.ws == {SHARED_SOURCE: ctx.kept} + assert [ws is ctx.kept for ws in prefills] == [True, True, False, True] + assert prefills[2] is ctx.ws_buf + else: + assert ctx.kept is None + assert [row for row, _ in gathers] == [SHARED_SOURCE, SHARED_SOURCE, 40, SHARED_SOURCE] + assert all(out is ctx.ws_buf for _, out in gathers) + assert all(ws is ctx.ws_buf for ws in prefills) and ctx.ws == {} + + +def test_decode_path_attends_off_the_pool_in_sub_batches(monkeypatch): + """Past the held rows `_attn_ctx` gathers nothing: the selections become pool slots once + per layer, at the layer's source row, and every sub-batch is one sparse decode call on its + own size's scheduler; layer 39 reads layer 35's pool row.""" + converted, decoded = [], [] + + def convert(req_idx, block_table, topk, tpb, k, stride, pool_row): + converted.append(pool_row) + return topk + + def decode(q, kv, slots, sched, scale, out): + decoded.append((q.shape[0], slots.shape[0], sched)) + + def no_gather(*args): + raise AssertionError("the decode path gathers nothing") + + monkeypatch.setattr(M, "convert_req_index_to_global", convert) + monkeypatch.setattr(M, "flash_mla_sparse_decode", decode) + monkeypatch.setattr(M, "ds_mla_dequant_gather", no_gather) + target = _attn_target(64) + tc = 20000 + step = _host_step([(5, tc)]) + ctx = target._ctx_workspace(step, tc, CPU) + assert ctx.ws_buf is None + sel = torch.zeros(tc, 4, dtype=torch.int32) + req_idx = torch.zeros(tc, dtype=torch.int64) + for layer in (36, 39): + decoded.clear() + L = SimpleNamespace(kv_source=KV_SOURCE[layer]) + target._attn_ctx(step, L, torch.empty(tc, 1), torch.empty(tc, 1), req_idx, sel, ctx) + assert decoded == [(n, n, ctx.sched[n]) for n in (8192, 8192, 3616)] + assert converted == [36, SHARED_SOURCE] diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_weight_manifest.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_weight_manifest.py new file mode 100644 index 000000000000..17c12487468f --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_103__tp/test_modeling_v2_weight_manifest.py @@ -0,0 +1,2015 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The Mistral Large 4 NVFP4 tp4 weight table against the checkpoint it reads, at tp4 (four ranks on +one node) and at tp8 (eight ranks over two nodes of four). + +No GPU. Against the tep4 table only the routed experts move: every rank holds all 128 experts of +a layer, each cut along its intermediate width to rank r's I -- `w1`/`w3` codes and block scales +by output rows `[Ir, Ir + I)`, `w2`'s codes by input bytes `[Ir/2, (I/2)(r + 1))` and its block +scales by columns `[Ir/16, (I/16)(r + 1))` -- and the MoE runner's layout is applied to that +slice. I is 1536 at tp4 (768 code bytes, 96 scale columns) and 768 at tp8 (384 and 48). Four +kinds of evidence, at each world size: + +* the table at the checkpoint's own language-model configuration (written out below, and + compared with the configuration adapted from the checkpoint's `params.json` when it is + present) on each rank of the engine's mapping (tp and moe_tp at the world size, moe_ep 1, four + ranks per node): which layers carry which weight at what per-rank shape, that every rank's + manifest reads every expert through its width slice, and that the ranks' slices tile each + expert exactly once; +* the checkpoint itself, through its index and safetensors headers, as meta tensors, plus the + values of its small scale tensors: every rank reads all 1,721 language-model keys, all + 109,056 expert keys and all 276 vision-tower keys, each at the per-rank shape and dtype + `_PER_RANK` states, 157,212,207,424 bytes (146.42 GiB) at tp4 and 82,442,488,768 bytes + (76.78 GiB) at tp8 -- the language model's 153,803,770,176 and 79,034,051,520 and the vision + tower's 3,408,437,248, whole -- which is also what the production `load` itself reads; the + index `load` takes of every key on every rank -- the ranks' head, row, column and vocabulary + ranges and expert width slices tile each sliced tensor exactly once, everything else is read + whole; the production `load` -- every slice, transform, shape and dtype assert, the + expert-scalar asserts on the checkpoint's own scalars and the two-way coverage assert, the + DSpark draft its only known-not-loaded set -- passes on each rank; every FP8 block scale and + every expert scalar satisfies the invariants `load` asserts; + and the packed UE8M0 scale operand at each K a rank's slice gives (at tp8 `wo` at 1024, the + dense `w2` at 3072, the shared expert's `w2` at 768: 6 blocks, a half-padding second word) + equals an independent byte-level packing and the engine's `transform_sf_into_required_layout`; +* the checkpoint's bytes for sampled experts of layers 1, 35 and 71: the production `load` puts + each rank's slice of them, in the runner layout, exactly where the stock TRTLLM-Gen NVFP4 MoE + preparation at moe_tp_size 4 or 8 puts it (its own slicing, permutations and scale + interleave, run on CPU), and where the whole stored expert, cut here and laid out by this + file's own construction, puts it; the ranks' slices, read through the production shards, + reassemble the stored tensors; and the whole expert laid out before slicing differs wherever + the layout depends on the slice (the negative control). Skipped where the checkpoint is + absent (looked up as `mistral-large-4-nvfp4` or `Mistral-Large-4-1T-A52B-NVFP4` under + `LLM_MODELS_ROOT`); +* a scaled-down configuration with real CPU tensors, loaded on every rank and compared element + for element with an independent construction of what each parameter must hold: the slices, + the `wkv_b` regroup, both half orders, the packed scale operand padding included (at tp8 the + dense `w2` at a 6-block K here too), every expert's width slice in the MoE runner's layout and + its scalars, and the DSpark draft tensors left alone; plus what the streaming path reads off + the shard files -- each expert code and block scale as this rank's slice of the entry, never + the whole, and each file's cached pages dropped once read -- and each assert's refusal. +""" + +from __future__ import annotations + +import contextlib +import json +import re +import struct +from collections import Counter +from pathlib import Path +from types import SimpleNamespace + +import pytest +import torch +import torch.nn as nn +from safetensors import safe_open +from test_common.llm_data import llm_models_root + +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_103__tp import ( # noqa: E501 + weights as W, +) +from tensorrt_llm._torch.models.checkpoints.mistral.config_loader import MistralConfigLoader +from tensorrt_llm._torch.modules.linear import TensorParallelMode, load_weight_shard +from tensorrt_llm._torch.moe.fused_moe.quantization import ( + NVFP4TRTLLMGenFusedMoEMethod, + trtllmgen_maybe_get_cached_w2_permute_indices, + trtllmgen_maybe_get_cached_w3_w1_permute_indices, +) +from tensorrt_llm.mapping import Mapping +from tensorrt_llm.quantization.utils.fp8_utils import ( + check_sf_layout, + inverse_transform_sf, + transform_sf_into_required_layout, +) + +pytestmark = pytest.mark.cpu_only + +#: The world sizes: tp4, four ranks on one node, and tp8, eight ranks over two. +WORLDS = (4, 8) +#: GB300 nodes carry four devices. +GPUS_PER_NODE = 4 +CHECKPOINTS = ("mistral-large-4-nvfp4", "Mistral-Large-4-1T-A52B-NVFP4") + +_LAYERS = 72 +#: The DSpark draft's keys, all in the checkpoint's last shard, which no rank reads. +_DRAFT_KEYS = 38 + +#: The language-model configuration surface the table reads, at the checkpoint's values. The +#: checkpoint's configuration adds its vision encoder, whose rows are checked against the +#: checkpoint below. +_CONFIG = dict( + num_hidden_layers=_LAYERS, + hidden_size=6144, + num_attention_heads=64, + qk_nope_head_dim=64, + qk_rope_head_dim=64, + v_head_dim=128, + kv_lora_rank=512, + nope_layer_flags=[i % 4 == 3 for i in range(_LAYERS)], + kv_source_layers=[35 if i >= 39 and i % 4 == 3 else i for i in range(_LAYERS)], + index_sharing=["shared" if i >= 17 and (i - 17) % 4 < 2 else "full" for i in range(_LAYERS)], + index_n_heads=32, + index_head_dim=128, + first_k_dense_replace=1, + n_routed_experts=128, + moe_intermediate_size=6144, + n_shared_experts=1, + intermediate_size=24576, + vocab_size=204800, +) + +_SAFETENSORS_DTYPES = { + "BF16": torch.bfloat16, + "F32": torch.float32, + "F8_E4M3": torch.float8_e4m3fn, + "U8": torch.uint8, +} +_DTYPE_NAMES = {v: k for k, v in _SAFETENSORS_DTYPES.items()} + +_PROJECTIONS = ("w1", "w2", "w3") +_EXPERT_PARAMS = ("fc1_w", "fc1_s", "fc2_w", "fc2_s", "isc1", "isc2", "ws2_13", "ws2_2") +#: The expert parameters the width slice cuts and the runner layout rearranges. +_LAID_OUT = ("fc1_w", "fc1_s", "fc2_w", "fc2_s") +_EXPERT_SUFFIXES = { + f"{proj}.{s}" + for proj in _PROJECTIONS + for s in ("weight", "weight_scale", "weight_scale_2", "input_scale") +} + +#: Per world size, what one rank holds of each tensor-parallel width: query heads (of 64), the +#: dense MLP's intermediate (of 24576), the shared expert's (of 6144), each routed expert's slice +#: (of 6144) and lm_head's vocabulary rows (of 204800). +_GEOMETRY = { + 4: SimpleNamespace(heads=16, dense=6144, shared=1536, inter=1536, vocab=51200), + 8: SimpleNamespace(heads=8, dense=3072, shared=768, inter=768, vocab=25600), +} + +#: Each sliced expert parameter's source as stored (`[out, in/2]` codes, `[out, in/16]` block +#: scales), the axis rank r's slice cuts and its width there. tp4: w1/w3 rows [1536r, + 1536); +#: w2's code bytes [768r, + 768) and block-scale columns [96r, + 96). tp8: w1/w3 rows +#: [768r, + 768); w2's code bytes [384r, + 384) and block-scale columns [48r, + 48). +_SLICED = { + 4: { + "fc1_w": ((6144, 3072), 0, 1536), + "fc1_s": ((6144, 384), 0, 1536), + "fc2_w": ((6144, 3072), 1, 768), + "fc2_s": ((6144, 384), 1, 96), + }, + 8: { + "fc1_w": ((6144, 3072), 0, 768), + "fc1_s": ((6144, 384), 0, 768), + "fc2_w": ((6144, 3072), 1, 384), + "fc2_s": ((6144, 384), 1, 48), + }, +} + +#: Every text key a rank reads, by `(key with layer N and expert E, stored dtype, stored shape, +#: the shape this rank moves)`, with its count; identical on the ranks of a world size. Written +#: from the safetensors headers and the sharding the module docstring states, not from the +#: table's code. 1,721 language-model keys sharded exactly as at tep4 or tep8 (`output.weight` by +#: the shell's vocab-parallel lm_head) and 71 x 128 x 3 x 4 = 109,056 expert keys: the codes and +#: block scales width-sliced, the per-tensor scalars whole. The keys every rank moves whole at +#: either size: +_PER_RANK_WHOLE = { + ("layers.N.attention.indexer.k_norm.bias", "BF16", (128,), (128,)): 44, + ("layers.N.attention.indexer.k_norm.weight", "BF16", (128,), (128,)): 44, + ("layers.N.attention.indexer.q_input_norm.bias", "BF16", (6144,), (6144,)): 44, + ("layers.N.attention.indexer.q_input_norm.weight", "BF16", (6144,), (6144,)): 44, + ("layers.N.attention.indexer.ssmax_scaling.b", "BF16", (32,), (32,)): 44, + ("layers.N.attention.indexer.ssmax_scaling.s", "BF16", (32,), (32,)): 44, + ("layers.N.attention.indexer.weights_proj.weight", "BF16", (32, 6144), (32, 6144)): 44, + ("layers.N.attention.indexer.wk.weight", "BF16", (128, 6144), (128, 6144)): 44, + ("layers.N.attention.indexer.wq_b.weight", "F8_E4M3", (4096, 6144), (4096, 6144)): 44, + ("layers.N.attention.indexer.wq_b.weight_scale", "F32", (32, 1, 48, 1), (32, 1, 48, 1)): 44, + ("layers.N.attention.kv_a_norm.weight", "BF16", (512,), (512,)): 63, + ("layers.N.attention.wkv_a_with_mqa.weight", "BF16", (512, 6144), (512, 6144)): 9, + ("layers.N.attention.wkv_a_with_mqa.weight", "BF16", (576, 6144), (576, 6144)): 54, + ("layers.N.attention_norm.weight", "BF16", (6144,), (6144,)): 72, + ("layers.N.experts.E.w1.input_scale", "F32", (), ()): 9088, + ("layers.N.experts.E.w1.weight_scale_2", "F32", (), ()): 9088, + ("layers.N.experts.E.w2.input_scale", "F32", (), ()): 9088, + ("layers.N.experts.E.w2.weight_scale_2", "F32", (), ()): 9088, + ("layers.N.experts.E.w3.input_scale", "F32", (), ()): 9088, + ("layers.N.experts.E.w3.weight_scale_2", "F32", (), ()): 9088, + ("layers.N.ffn_norm.weight", "BF16", (6144,), (6144,)): 72, + ("layers.N.gate.weight", "BF16", (128, 6144), (128, 6144)): 71, + ("lerope.alpha", "BF16", (32,), (32,)): 1, + ("norm.weight", "BF16", (6144,), (6144,)): 1, + ("tok_embeddings.weight", "BF16", (204800, 6144), (204800, 6144)): 1, +} +#: and the sharded keys, by world size: a quarter or an eighth of the heads, the dense and shared +#: MLP widths, the vocabulary and every expert's width. +_PER_RANK_SHARDED = { + 4: { + ("layers.N.attention.ssmax_scaling.b", "BF16", (64,), (16,)): 72, + ("layers.N.attention.ssmax_scaling.s", "BF16", (64,), (16,)): 72, + ("layers.N.attention.wkv_b.weight", "BF16", (12288, 512), (3072, 512)): 54, + ("layers.N.attention.wkv_b.weight", "BF16", (16384, 512), (4096, 512)): 18, + ("layers.N.attention.wo.weight", "F8_E4M3", (6144, 8192), (6144, 2048)): 72, + ("layers.N.attention.wo.weight_scale", "F32", (48, 1, 64, 1), (48, 1, 16, 1)): 72, + ("layers.N.attention.wq.weight", "F8_E4M3", (8192, 6144), (2048, 6144)): 72, + ("layers.N.attention.wq.weight_scale", "F32", (64, 1, 48, 1), (16, 1, 48, 1)): 72, + ("layers.N.experts.E.w1.weight", "U8", (6144, 3072), (1536, 3072)): 9088, + ("layers.N.experts.E.w1.weight_scale", "F8_E4M3", (6144, 384), (1536, 384)): 9088, + ("layers.N.experts.E.w2.weight", "U8", (6144, 3072), (6144, 768)): 9088, + ("layers.N.experts.E.w2.weight_scale", "F8_E4M3", (6144, 384), (6144, 96)): 9088, + ("layers.N.experts.E.w3.weight", "U8", (6144, 3072), (1536, 3072)): 9088, + ("layers.N.experts.E.w3.weight_scale", "F8_E4M3", (6144, 384), (1536, 384)): 9088, + ("layers.N.feed_forward.w1.weight", "F8_E4M3", (24576, 6144), (6144, 6144)): 1, + ("layers.N.feed_forward.w1.weight_scale", "F32", (192, 1, 48, 1), (48, 1, 48, 1)): 1, + ("layers.N.feed_forward.w2.weight", "F8_E4M3", (6144, 24576), (6144, 6144)): 1, + ("layers.N.feed_forward.w2.weight_scale", "F32", (48, 1, 192, 1), (48, 1, 48, 1)): 1, + ("layers.N.feed_forward.w3.weight", "F8_E4M3", (24576, 6144), (6144, 6144)): 1, + ("layers.N.feed_forward.w3.weight_scale", "F32", (192, 1, 48, 1), (48, 1, 48, 1)): 1, + ("layers.N.shared_experts.w1.weight", "F8_E4M3", (6144, 6144), (1536, 6144)): 71, + ("layers.N.shared_experts.w1.weight_scale", "F32", (48, 1, 48, 1), (12, 1, 48, 1)): 71, + ("layers.N.shared_experts.w2.weight", "F8_E4M3", (6144, 6144), (6144, 1536)): 71, + ("layers.N.shared_experts.w2.weight_scale", "F32", (48, 1, 48, 1), (48, 1, 12, 1)): 71, + ("layers.N.shared_experts.w3.weight", "F8_E4M3", (6144, 6144), (1536, 6144)): 71, + ("layers.N.shared_experts.w3.weight_scale", "F32", (48, 1, 48, 1), (12, 1, 48, 1)): 71, + ("output.weight", "BF16", (204800, 6144), (51200, 6144)): 1, + }, + 8: { + ("layers.N.attention.ssmax_scaling.b", "BF16", (64,), (8,)): 72, + ("layers.N.attention.ssmax_scaling.s", "BF16", (64,), (8,)): 72, + ("layers.N.attention.wkv_b.weight", "BF16", (12288, 512), (1536, 512)): 54, + ("layers.N.attention.wkv_b.weight", "BF16", (16384, 512), (2048, 512)): 18, + ("layers.N.attention.wo.weight", "F8_E4M3", (6144, 8192), (6144, 1024)): 72, + ("layers.N.attention.wo.weight_scale", "F32", (48, 1, 64, 1), (48, 1, 8, 1)): 72, + ("layers.N.attention.wq.weight", "F8_E4M3", (8192, 6144), (1024, 6144)): 72, + ("layers.N.attention.wq.weight_scale", "F32", (64, 1, 48, 1), (8, 1, 48, 1)): 72, + ("layers.N.experts.E.w1.weight", "U8", (6144, 3072), (768, 3072)): 9088, + ("layers.N.experts.E.w1.weight_scale", "F8_E4M3", (6144, 384), (768, 384)): 9088, + ("layers.N.experts.E.w2.weight", "U8", (6144, 3072), (6144, 384)): 9088, + ("layers.N.experts.E.w2.weight_scale", "F8_E4M3", (6144, 384), (6144, 48)): 9088, + ("layers.N.experts.E.w3.weight", "U8", (6144, 3072), (768, 3072)): 9088, + ("layers.N.experts.E.w3.weight_scale", "F8_E4M3", (6144, 384), (768, 384)): 9088, + ("layers.N.feed_forward.w1.weight", "F8_E4M3", (24576, 6144), (3072, 6144)): 1, + ("layers.N.feed_forward.w1.weight_scale", "F32", (192, 1, 48, 1), (24, 1, 48, 1)): 1, + ("layers.N.feed_forward.w2.weight", "F8_E4M3", (6144, 24576), (6144, 3072)): 1, + ("layers.N.feed_forward.w2.weight_scale", "F32", (48, 1, 192, 1), (48, 1, 24, 1)): 1, + ("layers.N.feed_forward.w3.weight", "F8_E4M3", (24576, 6144), (3072, 6144)): 1, + ("layers.N.feed_forward.w3.weight_scale", "F32", (192, 1, 48, 1), (24, 1, 48, 1)): 1, + ("layers.N.shared_experts.w1.weight", "F8_E4M3", (6144, 6144), (768, 6144)): 71, + ("layers.N.shared_experts.w1.weight_scale", "F32", (48, 1, 48, 1), (6, 1, 48, 1)): 71, + ("layers.N.shared_experts.w2.weight", "F8_E4M3", (6144, 6144), (6144, 768)): 71, + ("layers.N.shared_experts.w2.weight_scale", "F32", (48, 1, 48, 1), (48, 1, 6, 1)): 71, + ("layers.N.shared_experts.w3.weight", "F8_E4M3", (6144, 6144), (768, 6144)): 71, + ("layers.N.shared_experts.w3.weight_scale", "F32", (48, 1, 48, 1), (6, 1, 48, 1)): 71, + ("output.weight", "BF16", (204800, 6144), (25600, 6144)): 1, + }, +} +#: Besides, every rank reads the BF16 vision tower whole -- encoder, pre-projector norm, patch +#: merger and language adapter -- at either size: 276 keys, the transformer's 9 on each of its 30 +#: layers. +_PER_RANK_VISION = { + ("patch_merger.merging_layer.weight", "BF16", (2048, 8192), (2048, 8192)): 1, + ("pre_mm_projector_norm.weight", "BF16", (2048,), (2048,)): 1, + ("vision_encoder.ln_pre.weight", "BF16", (2048,), (2048,)): 1, + ("vision_encoder.patch_conv.weight", "BF16", (2048, 3, 14, 14), (2048, 3, 14, 14)): 1, + ("vision_language_adapter.w_in.weight", "BF16", (6144, 2048), (6144, 2048)): 1, + ("vision_language_adapter.w_out.weight", "BF16", (6144, 6144), (6144, 6144)): 1, + **{ + (f"vision_encoder.transformer.layers.N.{name}.weight", "BF16", shape, shape): 30 + for name, shape in ( + ("attention.wq", (2048, 2048)), + ("attention.wk", (2048, 2048)), + ("attention.wv", (2048, 2048)), + ("attention.wo", (2048, 2048)), + ("attention_norm", (2048,)), + ("feed_forward.w1", (6144, 2048)), + ("feed_forward.w2", (2048, 6144)), + ("feed_forward.w3", (6144, 2048)), + ("ffn_norm", (2048,)), + ) + }, +} +_PER_RANK = { + world: {**_PER_RANK_WHOLE, **_PER_RANK_SHARDED[world], **_PER_RANK_VISION} for world in WORLDS +} + +#: What each rank moves of the language model, from the headers (also plain arithmetic on them, +#: apart from the table: a quarter or an eighth of every expert code and block scale, every +#: expert scalar, the rest as at tep4 or tep8). tep8's rank moves 79,033,860,672 bytes of it, +#: its 16 whole experts 72,364,354,176 of them: the same bytes up to the per-tensor scalars of +#: the 112 experts a tep8 rank does not hold, 71 x 112 x 6 x 4 = 190,848 bytes. +_RANK_LM_BYTES = {4: 153_803_770_176, 8: 79_034_051_520} +_RANK_EXPERT_BYTES = {4: 144_728_871_936, 8: 72_364_545_024} +#: The same in GiB, to the second decimal: (rank, routed experts). +_RANK_GIB = {4: (143.24, 134.79), 8: (73.61, 67.39)} +#: The vision tower's keys and bytes (3.17 GiB), and all a rank moves: the language model and +#: the vision tower, 146.42 GiB at tp4 and 76.78 GiB at tp8. +_VISION_KEYS, _VISION_BYTES = 276, 3_408_437_248 +_RANK_BYTES = {world: n + _VISION_BYTES for world, n in _RANK_LM_BYTES.items()} +_RANK_READ_GIB = {4: 146.42, 8: 76.78} + + +@pytest.fixture(autouse=True) +def _no_device_memory(): + """Nothing here allocates on a GPU. The context alone proves nothing: `import tensorrt_llm` + creates one wherever a device is visible.""" + yield + if torch.cuda.is_initialized(): + assert torch.cuda.memory_reserved() == 0, "the weight table test allocated on a GPU" + + +def _mapping(world: int, rank: int) -> Mapping: + """The engine's mapping at tp4 or tp8: tensor parallel throughout, the routed experts tensor + parallel (moe_tp_size the world size, moe_ep_size 1), no attention DP, four ranks per + node.""" + m = Mapping( + world_size=world, + rank=rank, + gpus_per_node=GPUS_PER_NODE, + tp_size=world, + moe_tp_size=world, + moe_ep_size=1, + ) + # A topology that ignores `rank=` would hand every rank rank 0's slices. + assert (m.tp_rank, m.moe_tp_rank, m.moe_ep_rank) == (rank, rank, 0), m + return m + + +def _core(config: dict, world: int, rank: int) -> SimpleNamespace: + return SimpleNamespace( + model_config=SimpleNamespace( + pretrained_config=SimpleNamespace(**config), + mapping=_mapping(world, rank), + torch_dtype=torch.bfloat16, + ) + ) + + +class _Core(nn.Module): + """The surface `load` reads off the core: its config and its declared weights.""" + + def __init__(self, config: dict, world: int, rank: int, device: str): + super().__init__() + self.model_config = _core(config, world, rank).model_config + with torch.device(device): + self.w = W.MODEL_WEIGHTS.declare(W.MODEL_WEIGHTS.dims(self)) + + +class _Shell(nn.Module): + """The causal-LM shell: the core as `model`, a vocab-parallel `lm_head`.""" + + def __init__(self, config: dict, world: int, rank: int, device: str): + super().__init__() + self.model = _Core(config, world, rank, device) + self.lm_head = nn.Module() + rows = config["vocab_size"] // world + self.lm_head.weight = nn.Parameter( + torch.empty(rows, config["hidden_size"], dtype=torch.bfloat16, device=device), + requires_grad=False, + ) + + +def _source_keys(manifest: dict) -> set: + """Every checkpoint key the manifest reads, key tuples flattened.""" + return {k for sources in manifest.values() for ckpt, *_ in sources for k in W._keys(ckpt)} + + +def _is_vision(key: str) -> bool: + return key.startswith(W.VISION_PREFIXES) + + +def _is_draft(key: str) -> bool: + """A DSpark draft key: the speculative-decoding head, which these targets do not run.""" + return key.startswith("mtp_dspark.") + + +def _is_expert(key: str) -> bool: + return ".experts." in key + + +def _is_expert_scalar(key: str) -> bool: + return _is_expert(key) and key.endswith((".weight_scale_2", ".input_scale")) + + +def _role(key: str) -> str: + """A parameter key's weight role: `l{i}_` per layer, the key itself otherwise.""" + head, _, rest = key.partition("_") + return rest if head[:1] == "l" and head[1:].isdigit() else key + + +def _expert_of(key: str) -> tuple[int, int, str]: + """`layers.N.experts.E.` -> (N, E, suffix).""" + parts = key.split(".") + assert parts[0] == "layers" and parts[2] == "experts", key + return int(parts[1]), int(parts[3]), ".".join(parts[4:]) + + +def _bytes(t: torch.Tensor) -> torch.Tensor: + """Compare by bit pattern: e4m3 has no `==`, and it is exactness that is asserted.""" + return t.contiguous().view(torch.uint8) if t.element_size() == 1 else t + + +def _packed(grid: torch.Tensor) -> torch.Tensor: + """Independent construction of the packed scale storage from fp32 `[N/128, K/128]` block + scales: `[ceil(K/512), N]` int32, word `[w, n]` holding the UE8M0 bytes (the fp32 exponent + field) of K blocks 4w..4w+3 of row n's block, low byte first, and zero bytes past the last K + block. Integer arithmetic only, so it shares no code with the engine's packer.""" + n_blk, k_blk = grid.shape + words = -(-k_blk // 4) + exps = torch.zeros(n_blk, 4 * words, dtype=torch.int64) + exps[:, :k_blk] = (grid.contiguous().view(torch.int32).to(torch.int64) >> 23) & 0xFF + packed = exps[:, 0::4] | exps[:, 1::4] << 8 | exps[:, 2::4] << 16 | exps[:, 3::4] << 24 + packed = torch.where(packed >= 2**31, packed - 2**32, packed).to(torch.int32) + return packed.repeat_interleave(128, dim=0).t().contiguous() + + +#: Parametrizations: each world size, and each rank of each. +_WORLDS = pytest.mark.parametrize("world", WORLDS, ids=[f"tp{w}" for w in WORLDS]) +_WORLD_RANKS = pytest.mark.parametrize( + "world, rank", [pytest.param(w, r, id=f"tp{w}-{r}") for w in WORLDS for r in range(w)] +) + + +@pytest.fixture(scope="module") +def manifests() -> dict: + """Each rank's manifest at the checkpoint's configuration, by world size and rank.""" + return { + world: { + rank: W.MODEL_WEIGHTS.manifest(_core(_CONFIG, world, rank)) for rank in range(world) + } + for world in WORLDS + } + + +# --------------------------------------------------------------------- the configuration + + +def _layers_of(declared, name: str) -> dict[int, tuple]: + """Layer index -> declared shape, for every `l{i}_`.""" + out = {} + for key, param in declared.items(): + head, _, rest = key.partition("_") + if head.startswith("l") and head[1:].isdigit() and rest == name: + out[int(head[1:])] = tuple(param.shape) + return out + + +@_WORLD_RANKS +def test_the_width_slice_is_read_off_the_mapping(world, rank): + """A gate at tp4 passes tensor_parallel_size 4 alone, one at tp8 tensor_parallel_size 8 alone, + and the engine's Mapping then derives moe_tp_size at the world size and moe_ep_size 1 itself: + the same mapping as the explicit one. Rank r is moe_tp_rank r of an expert window that is + every expert, so the table reads expert_offset 0, local_experts 128, and an I-wide slice at Ir + (I = 1536 at tp4, 768 at tp8); the rest of its geometry is tep4's at tp4 and tep8's at tp8.""" + g = _GEOMETRY[world] + bare = Mapping(world_size=world, rank=rank, gpus_per_node=GPUS_PER_NODE, tp_size=world) + for m in (_mapping(world, rank), bare): + assert (m.tp_size, m.tp_rank, m.moe_tp_size, m.moe_tp_rank) == (world, rank, world, rank) + assert (m.moe_ep_size, m.moe_ep_rank, m.enable_attention_dp) == (1, 0, False) + d = W.MODEL_WEIGHTS.dims(_core(_CONFIG, world, rank)) + assert (d.local_experts, d.expert_offset, d.moe_inter, d.moe_tp_rank) == (128, 0, g.inter, rank) + assert (d.tp_rank, d.tp_size, d.heads, d.shared_inter, d.dense_inter) == ( + rank, + world, + g.heads, + g.shared, + g.dense, + ) + + +@pytest.mark.parametrize( + "world, rank", [pytest.param(w, r, id=f"tp{w}-{r}") for w in WORLDS for r in (0, w - 1)] +) +def test_layer_sets_and_per_rank_shapes(world, rank): + """Which layers carry which weight, at what per-rank shape: 16 of the 64 heads at tp4 and 8 + at tp8 (`wq` 128 rows and `wo` 128 columns a head, `wkv_b` and SSMax per head), the dense MLP + at 6144 or 3072 and the shared expert at 1536 or 768 of their widths, all 128 routed experts + each at this rank's 1536- or 768-wide slice. The NoPE layers are 3, 7, ..., 71; the KV sharers + 39, 43, ..., 71; the 'shared' indexer layers 17, 18, 21, 22, ..., 69, 70. Every row but the + routed experts' is tep4's at tp4 and tep8's at tp8.""" + g = _GEOMETRY[world] + core = _core(_CONFIG, world, rank) + with torch.device("meta"): + declared = W.MODEL_WEIGHTS.declare(W.MODEL_WEIGHTS.dims(core)) + nope = {i for i in range(_LAYERS) if i % 4 == 3} + sharers = {i for i in range(39, _LAYERS, 4)} + full = {i for i in range(_LAYERS) if i < 17 or (i - 17) % 4 >= 2} + moe = range(1, _LAYERS) + assert len(nope) == 18 and len(sharers) == 9 and len(full) == 44 + + every = {i: None for i in range(_LAYERS)} + # Packed scale operands: one int32 word per four 128-blocks of K per output row, the last + # word partly padding where the block count is not a multiple of four. + words = {k: -(-k // 512) for k in (g.heads * 128, g.dense, g.shared)} + expect = { + "attn_norm": {i: (6144,) for i in every}, + "wq": {i: (g.heads * 128, 6144) for i in every}, + "wq_s": {i: (12, g.heads * 128) for i in every}, + "ssmax_s": {i: (g.heads,) for i in every}, + "ssmax_b": {i: (g.heads,) for i in every}, + "wkv_a": {i: (512 if i in nope else 576, 6144) for i in range(_LAYERS) if i not in sharers}, + "kv_a_norm": {i: (512,) for i in range(_LAYERS) if i not in sharers}, + # Per head [k_nope 128 | v 128] on NoPE layers, [k_nope 64 | v 128] on RoPE layers. + "wkv_b": {i: (g.heads * (256 if i in nope else 192), 512) for i in every}, + "wo": {i: (6144, g.heads * 128) for i in every}, + # K 2048 at tp4: 16 blocks, four whole words; K 1024 at tp8: 8 blocks, two whole words. + "wo_s": {i: (words[g.heads * 128], 6144) for i in every}, + "ffn_norm": {i: (6144,) for i in every}, + "ix_q_norm_w": {i: (6144,) for i in full}, + "ix_q_norm_b": {i: (6144,) for i in full}, + "ix_wq_b": {i: (4096, 6144) for i in full}, + "ix_wq_b_s": {i: (12, 4096) for i in full}, + "ix_wk_wp": {i: (160, 6144) for i in full}, + "ix_k_norm_w": {i: (128,) for i in full}, + "ix_k_norm_b": {i: (128,) for i in full}, + "ix_ssmax_s": {i: (32,) for i in full}, + "ix_ssmax_b": {i: (32,) for i in full}, + "mlp_gu": {i: (2 * (g.dense if i == 0 else g.shared), 6144) for i in every}, + "mlp_gu_s": {i: (12, 2 * (g.dense if i == 0 else g.shared)) for i in every}, + "mlp_dn": {i: (6144, g.dense if i == 0 else g.shared) for i in every}, + # tp4: K 6144, 48 blocks in 12 words; K 1536, 12 blocks in 3. tp8: K 3072, 24 blocks in + # six words; K 768, 6 blocks in two words, the second half padding. + "mlp_dn_s": {i: (words[g.dense if i == 0 else g.shared], 6144) for i in every}, + "router": {i: (128, 6144) for i in moe}, + "fc1_w": {i: (128, 2 * g.inter, 3072) for i in moe}, + "fc1_s": {i: (128, 2 * g.inter, 384) for i in moe}, + "fc2_w": {i: (128, 6144, g.inter // 2) for i in moe}, + "fc2_s": {i: (128, 6144, g.inter // 16) for i in moe}, + "isc1": {i: (1,) for i in moe}, + "isc2": {i: (1,) for i in moe}, + "ws2_13": {i: (128,) for i in moe}, + "ws2_2": {i: (128,) for i in moe}, + } + for name, want in expect.items(): + assert _layers_of(declared, name) == want, name + per_layer = {k for k in declared if k.startswith("l") and k.split("_")[0][1:].isdigit()} + assert len(per_layer) == sum(len(v) for v in expect.values()), "an undeclared-for role" + assert {k: tuple(v.shape) for k, v in declared.items() if k not in per_layer} == { + "final_norm": (6144,), + "embed": (204800, 6144), + "lerope_alpha": (32,), + } + dtypes = {_role(k): v.dtype for k, v in declared.items()} + assert dtypes["wq"] == dtypes["mlp_dn"] == dtypes["ix_wq_b"] == torch.float8_e4m3fn + assert dtypes["wq_s"] == dtypes["mlp_gu_s"] == dtypes["ix_wq_b_s"] == torch.int32 + assert dtypes["fc1_w"] == dtypes["fc2_w"] == torch.uint8 + assert dtypes["fc1_s"] == dtypes["fc2_s"] == torch.float8_e4m3fn + assert dtypes["isc1"] == dtypes["isc2"] == dtypes["ws2_13"] == dtypes["ws2_2"] == torch.float32 + assert dtypes["ix_k_norm_w"] == dtypes["lerope_alpha"] == torch.float32 + assert dtypes["wkv_b"] == dtypes["router"] == dtypes["embed"] == torch.bfloat16 + + # Per rank, with the shell's lm_head (51200 rows at tp4, 25600 at tp8). The routed experts + # are exactly a quarter (tp4) or an eighth (tp8) of their checkpoint bytes per layer, + # 128 x (3072 x 3072 + 3072 x 384 + 6144 x 768 + 6144 x 96) at tp4 -- what tep4's 32 whole + # experts come to -- and 128 x (1536 x 3072 + 1536 x 384 + 6144 x 384 + 6144 x 48) at tp8 -- + # what tep8's 16 do -- and 1,032 bytes of scalars; the FP8 scale operands, one int32 word per + # four 128-blocks per output row, are a few MiB more than the fp32 grids they are built from. + per_expert, expert_bytes, low, high = { + 4: (15925248, 144_728_727_096, 143.24, 143.30), + 8: (7962624, 72_364_400_184, 73.61, 73.66), + }[world] + nbytes = {k: p.numel() * p.element_size() for k, p in declared.items()} + experts = sum(n for k, n in nbytes.items() if _role(k) in _EXPERT_PARAMS) + total = sum(nbytes.values()) + g.vocab * 6144 * 2 + assert experts == 71 * (128 * per_expert + 1032) == expert_bytes + assert round(experts / 2**30, 2) == _RANK_GIB[world][1], experts / 2**30 + assert low * 2**30 < total < high * 2**30, total / 2**30 + + +@_WORLDS +def test_each_rank_reads_every_expert_through_its_width_slice(manifests, world): + """Rank r's manifest names all 128 routed experts on every MoE layer, local slot e <- global + expert e, all twelve tensors of each: the codes and block scales through the row's shard, + the per-tensor scalars whole; the one-per-layer scalars read every expert's copies. The + known-not-loaded set predicts nothing: no expert is outside the window.""" + for rank, manifest in manifests[world].items(): + experts: dict = {} + for key, sources in manifest.items(): + for ckpt, index, _, shard in sources: + keys = W._keys(ckpt) + if not any(_is_expert(k) for k in keys): + continue + assert all(_is_expert(k) for k in keys), (key, ckpt) + param = _role(key) + assert param in _EXPERT_PARAMS, key + assert (shard is not None) == (param in _LAID_OUT), key + for k in keys: + layer, e, suffix = _expert_of(k) + assert key == f"l{layer}_{param}", (key, k) + if param in ("isc1", "isc2"): + assert index == (0,), (key, index) + else: + assert index[0] == e, (key, k, index) + experts.setdefault((layer, e), set()).add(suffix) + assert set(experts) == {(i, e) for i in range(1, _LAYERS) for e in range(128)} + assert all(v == _EXPERT_SUFFIXES for v in experts.values()) + assert W.MODEL_WEIGHTS.expected_unconsumed(_core(_CONFIG, world, rank), {}) == {} + + +@_WORLDS +def test_width_slices_tile_each_expert_exactly_once(manifests, world): + """Every source of an expert row, on every MoE layer, shares the row's one shard (FC1's w3 + and w1 alike). Applied to a counter shaped like the stored tensor, rank r's shard is a view + at exactly rank r's range -- at tp4 w1/w3 rows [1536r, + 1536), w2's code bytes [768r, + 768) + and block-scale columns [96r, + 96); at tp8 w1/w3 rows [768r, + 768), w2's code bytes + [384r, + 384) and block-scale columns [48r, + 48); the other axis whole -- and adding one + through each rank's view leaves every element of the stored tensor at exactly one.""" + for role, (shape, axis, width) in _SLICED[world].items(): + counts = torch.zeros(shape, dtype=torch.int8) + for rank, manifest in manifests[world].items(): + sources = [src for i in range(1, _LAYERS) for src in manifest[f"l{i}_{role}"]] + assert len(sources) == 71 * 128 * (2 if role.startswith("fc1") else 1) + shard = sources[0][3] + assert all(src[3] is shard for src in sources), role + view = shard(counts) + want = list(shape) + want[axis] = width + assert tuple(view.shape) == tuple(want), (role, rank) + assert view.data_ptr() - counts.data_ptr() == rank * width * counts.stride(axis) + view.add_(1) + assert bool((counts == 1).all()), role + + +@_WORLDS +def test_the_per_rank_slices_are_laid_out_unpadded(world): + """The cut at 1536r (tp4) or 768r (tp8) splits no 16-input scale block (768 = 48 x 16) and no + code byte. The production layouts take each per-rank slot as it is: FC1's 2 x 1536 = 3072 or + 2 x 768 = 1536 rows and FC2's 6144 are whole 32-row shuffle blocks and 128-row swizzle tiles, + FC1's 384 and FC2's 96 or 48 scale columns whole 4-column tiles (each layout asserts its own + rule). The stock TRTLLM-Gen NVFP4 method resolves this geometry, hidden 6144 and per-rank + intermediate 1536 or 768, to weight and hidden alignment 32 -- 768 is a multiple of 32, and + of the 128 its scale shuffle needs -- which both meet, so it pads nothing either: the + unpadded chain `_stock` runs below is what it builds here.""" + d = W.MODEL_WEIGHTS.dims(_core(_CONFIG, world, 0)) + assert d.moe_inter % W.NVFP4_BLOCK == 0 and d.moe_inter % W.E2M1_PER_BYTE == 0 + per_rank = { + 4: { + "fc1_w": (3072, 3072), + "fc1_s": (3072, 384), + "fc2_w": (6144, 768), + "fc2_s": (6144, 96), + }, + 8: { + "fc1_w": (1536, 3072), + "fc1_s": (1536, 384), + "fc2_w": (6144, 384), + "fc2_s": (6144, 48), + }, + }[world] + laid_out = {} + for entry in W.MODEL_WEIGHTS.WEIGHTS: + if entry.layout is not None: + slot = torch.empty(per_rank[entry.name], dtype=entry.dtype, device="meta") + laid_out[entry.name] = tuple(entry.layout(slot).shape) + assert laid_out == per_rank + weight, hidden = NVFP4TRTLLMGenFusedMoEMethod.resolve_alignments(d.hidden, d.moe_inter) + assert (weight, hidden) == (32, 32) + assert d.moe_inter % weight == 0 and d.hidden % hidden == 0 + + +@_WORLDS +def test_manifest_and_declaration_agree(manifests, world): + """A declared weight with no source allocates memory nothing fills.""" + for rank, manifest in manifests[world].items(): + with torch.device("meta"): + declared = W.MODEL_WEIGHTS.declare(W.MODEL_WEIGHTS.dims(_core(_CONFIG, world, rank))) + assert set(manifest) == set(declared) + assert all(manifest.values()) + + +# --------------------------------------------------------------------- the checkpoint + + +def _checkpoint_dir() -> Path | None: + root = llm_models_root() + if root is None: + return None + for name in CHECKPOINTS: + path = root / name + if (path / "params.json").is_file() and (path / W.SHARD_INDEX).is_file(): + return path + return None + + +@pytest.fixture(scope="module") +def checkpoint(): + path = _checkpoint_dir() + if path is None: + pytest.skip(f"checkpoint {CHECKPOINTS} is not under LLM_MODELS_ROOT ({llm_models_root()})") + index = json.loads((path / W.SHARD_INDEX).read_text())["weight_map"] + headers = {} + for name in sorted(set(index.values())): + with open(path / name, "rb") as f: + (size,) = struct.unpack(" dict[str, torch.Tensor]: + """The given tensors off the shard files, whole, one file at a time.""" + by_file: dict = {} + for key in keys: + by_file.setdefault(checkpoint.index[key], []).append(key) + out = {} + for name, names in sorted(by_file.items()): + with safe_open(str(checkpoint.path / name), framework="pt", device="cpu") as handle: + for key in names: + out[key] = handle.get_tensor(key) + return out + + +@pytest.fixture(scope="module") +def expert_scalars(checkpoint): + """Every routed expert's per-tensor scalars, read off the shard files by the production + reader. Every rank at either world size holds every expert, so tp4's rank 0 names them all.""" + d = W.MODEL_WEIGHTS.dims(_core(checkpoint.config, 4, 0)) + keys = W._expert_scalar_keys(d) + return W._read_scalars(keys, checkpoint.index, (checkpoint.path, checkpoint.index)) + + +def _meta(checkpoint) -> dict: + return { + key: torch.empty(shape, dtype=dtype, device="meta") + for key, (dtype, shape) in checkpoint.headers.items() + } + + +def _normalized(key: str) -> str: + """A checkpoint key with its layer index (the language model's or the vision transformer's) + as N and its expert index as E.""" + key = re.sub(r"^((vision_encoder\.transformer\.)?layers)\.\d+\.", r"\1.N.", key) + return re.sub(r"\.experts\.\d+\.", ".experts.E.", key) + + +def test_checkpoint_config_is_the_one_tested(checkpoint): + """The configuration above is the checkpoint's, on every key the table reads.""" + assert {k: checkpoint.config[k] for k in _CONFIG} == _CONFIG + + +#: The FP8 128x128-block linears (`FP8_PB_WO`): e4m3 codes, fp32 `[N/128, 1, K/128, 1]` scales. +_FP8_LINEAR = re.compile( + r"layers\.\d+\.(attention\.(wq|wo|indexer\.wq_b)|feed_forward\.w[123]|shared_experts\.w[123])" +) + + +def test_checkpoint_dtypes_by_role(checkpoint): + """What each key stores, which is what the table's rows read: the FP8 linears e4m3 with + fp32 block scales, the routed experts NVFP4 (packed codes, e4m3 block scales, fp32 + per-tensor scalars), every other language-model tensor and the vision tower bf16.""" + for key, (dtype, shape) in checkpoint.headers.items(): + if _is_draft(key): + continue + module, _, leaf = key.rpartition(".") + if _is_expert(key): + want = { + "weight": torch.uint8, + "weight_scale": torch.float8_e4m3fn, + "weight_scale_2": torch.float32, + "input_scale": torch.float32, + }[leaf] + assert dtype == want, key + assert (shape == ()) == _is_expert_scalar(key), (key, shape) + elif _FP8_LINEAR.fullmatch(module): + want = {"weight": torch.float8_e4m3fn, "weight_scale": torch.float32}[leaf] + assert dtype == want, key + else: + assert dtype == torch.bfloat16, key + + +@_WORLDS +def test_every_rank_reads_every_key_but_the_draft(checkpoint, world): + """The index holds 111,091 keys, 38 of them the DSpark draft's and 276 the vision tower's; of + the rest, 1,721 are the language model's and 109,056 = 71 layers x 128 experts x 3 + projections x 4 tensors the routed experts'. Every rank reads every one of the 111,053 + (lm_head's `output.weight` by the shell) and no draft key, and its known-not-loaded set is + exactly the draft, declared as the disabled DSpark draft.""" + keys = set(checkpoint.index) + draft = {k for k in keys if _is_draft(k)} + vision = {k for k in keys if _is_vision(k)} + text = keys - draft - vision + lm = {k for k in text if not _is_expert(k)} + assert len(keys) == 111091 and len(draft) == _DRAFT_KEYS and len(vision) == _VISION_KEYS + assert len(lm) == 1721 and len(text - lm) == 109056 == 71 * 128 * 3 * 4 + for rank in range(world): + core = _core(checkpoint.config, world, rank) + read = _source_keys(W.MODEL_WEIGHTS.manifest(core)) | {"output.weight"} + skipped = W.MODEL_WEIGHTS.expected_unconsumed(core, checkpoint.index) + assert read == text | vision + assert set(skipped) == draft + assert set(skipped.values()) == {"DSpark draft disabled"} + + +@_WORLD_RANKS +def test_per_rank_shape_and_dtype_of_every_key(checkpoint, world, rank): + """Every key the rank reads, through the production manifest and shards on header-shaped + meta tensors, by stored dtype and shape and the shape this rank moves, is `_PER_RANK`: the + slices keep the dtype, the shell's lm_head takes `output.weight`'s vocabulary quarter at + tp4 and eighth at tp8, and the vision tower is moved whole.""" + core = _core(checkpoint.config, world, rank) + got: Counter = Counter() + for sources in W.MODEL_WEIGHTS.manifest(core).values(): + for ckpt, _, _, shard in sources: + for key in W._keys(ckpt): + dtype, shape = checkpoint.headers[key] + moved = W._materialize(torch.empty(shape, dtype=dtype, device="meta"), shard) + assert moved.dtype == dtype, key + got[(_normalized(key), _DTYPE_NAMES[dtype], shape, tuple(moved.shape))] += 1 + dtype, shape = checkpoint.headers["output.weight"] + lm_head = _Shell(checkpoint.config, world, rank, "meta").lm_head.weight + got[("output.weight", _DTYPE_NAMES[dtype], shape, tuple(lm_head.shape))] += 1 + assert dict(got) == _PER_RANK[world] + assert sum(n for (k, *_), n in got.items() if not _is_expert(k) and not _is_vision(k)) == 1721 + assert sum(n for (k, *_), n in got.items() if _is_expert(k)) == 109056 + assert sum(n for (k, *_), n in got.items() if _is_vision(k)) == _VISION_KEYS + + +@_WORLDS +def test_per_rank_bytes_from_the_headers(checkpoint, world): + """What each rank moves, from the safetensors headers through the production slices. At tp4 + the same 157,212,207,424 bytes (146.42 GiB) on every rank: the language model's + 153,803,770,176 (143.24 GiB), 144,728,871,936 (134.79 GiB) of it its width slices of the + routed experts and their scalars -- what tep4's 32 whole experts per rank came to, to the + second decimal of a GiB -- and the vision tower's 3,408,437,248 (3.17 GiB), whole. At tp8 + the same 82,442,488,768 bytes (76.78 GiB) on every rank: the language model's 79,034,051,520 + (73.61 GiB), 72,364,545,024 (67.39 GiB) of it its width slices of the routed experts and + their scalars -- a tep8 rank's bytes and 190,848 more, the scalars of the 112 experts outside + a tep8 window -- and the same vision tower.""" + expert_bytes = vision_bytes = 0 + for key, (dtype, shape) in checkpoint.headers.items(): + n = torch.empty(shape, dtype=dtype, device="meta").nbytes + if _is_expert(key): + expert_bytes += n if _is_expert_scalar(key) else n // world + vision_bytes += n if _is_vision(key) else 0 + assert expert_bytes == _RANK_EXPERT_BYTES[world] + assert vision_bytes == _VISION_BYTES + for rank in range(world): + core = _core(checkpoint.config, world, rank) + d = W.MODEL_WEIGHTS.dims(core) + total = experts = vision = 0 + for sources in W.MODEL_WEIGHTS.manifest(core).values(): + for ckpt, _, _, shard in sources: + for key in W._keys(ckpt): + dtype, shape = checkpoint.headers[key] + t = W._materialize(torch.empty(shape, dtype=dtype, device="meta"), shard) + total += t.nbytes + experts += t.nbytes if _is_expert(key) else 0 + vision += t.nbytes if _is_vision(key) else 0 + dtype, (vocab, hidden) = checkpoint.headers["output.weight"] + total += vocab // d.tp_size * hidden * torch.empty((), dtype=dtype).element_size() + assert (total, experts, vision) == ( + _RANK_BYTES[world], + _RANK_EXPERT_BYTES[world], + _VISION_BYTES, + ), rank + if world == 8: + assert _RANK_LM_BYTES[8] - 79_033_860_672 == 71 * 112 * 6 * 4 + assert round(_RANK_LM_BYTES[world] / 2**30, 2) == _RANK_GIB[world][0] + assert round(_RANK_EXPERT_BYTES[world] / 2**30, 2) == _RANK_GIB[world][1] + assert round(_RANK_BYTES[world] / 2**30, 2) == _RANK_READ_GIB[world] + + +class _Cut: + """A checkpoint entry that records each index taken of it -- what `load` reads of that key + -- and hands back a meta tensor of the header's shape so indexed.""" + + def __init__(self, key: str, dtype: torch.dtype, shape: tuple, log: list): + self.key, self.dtype, self.shape, self.log = key, dtype, shape, log + + def __getitem__(self, index): + self.log.append((self.key, index)) + return torch.empty(self.shape, dtype=self.dtype, device="meta")[index] + + +def _region(index, shape: tuple) -> tuple[tuple[int, int], ...]: + """The `[start, stop)` an index of basic slices takes on each dimension of `shape`.""" + if index is Ellipsis: + index = () + index = index if isinstance(index, tuple) else (index,) + assert all(isinstance(s, slice) for s in index), index + out = [] + for dim, size in enumerate(shape): + start, stop, step = (index[dim] if dim < len(index) else slice(None)).indices(size) + assert step == 1, index + out.append((start, stop)) + return tuple(out) + + +def _sliced(key: str, world: int) -> tuple[int, int] | None: + """`(dimension, per-rank width)` of a key the ranks split, None for one each rank reads whole: + 16 heads of 128 at tp4, 8 at tp8 (`wq` rows, `wo` columns), of `wkv_b`'s per-head rows (192 + on RoPE layers, 256 on NoPE layers) and of SSMax; 6144 or 3072 dense and 1536 or 768 + shared-expert rows (`w1`, `w3`) or columns (`w2`), their block scales on the same 128-blocks + (scale axis 0 for rows, 2 for columns); 51200 or 25600 vocabulary rows of `output`; and every + routed expert's 1536- or 768-wide slice: `w1`/`w3` rows, codes and block scales alike, `w2`'s + 768 or 384 code bytes and 96 or 48 scale columns.""" + g = _GEOMETRY[world] + if key == "output.weight": + return 0, g.vocab + m = re.fullmatch(r"layers\.(\d+)\.(.+)", key) + if m is None: + return None + layer, rest = int(m.group(1)), m.group(2) + expert = re.fullmatch(r"experts\.\d+\.(w[123])\.(weight|weight_scale)", rest) + if expert is not None: + proj, leaf = expert.groups() + if proj != "w2": + return 0, g.inter + return 1, g.inter // 2 if leaf == "weight" else g.inter // 16 + table = { + "attention.wq.weight": (0, g.heads * 128), + "attention.wq.weight_scale": (0, g.heads), + "attention.wo.weight": (1, g.heads * 128), + "attention.wo.weight_scale": (2, g.heads), + "attention.ssmax_scaling.s": (0, g.heads), + "attention.ssmax_scaling.b": (0, g.heads), + "attention.wkv_b.weight": ( + 0, + g.heads * (256 if _CONFIG["nope_layer_flags"][layer] else 192), + ), + } + for module, width in (("feed_forward", g.dense), ("shared_experts", g.shared)): + for w in ("w1", "w3"): + table[f"{module}.{w}.weight"] = (0, width) + table[f"{module}.{w}.weight_scale"] = (0, width // 128) + table[f"{module}.w2.weight"] = (1, width) + table[f"{module}.w2.weight_scale"] = (2, width // 128) + return table.get(rest) + + +@_WORLDS +def test_ranks_tile_every_sliced_tensor(checkpoint, world): + """The index the production `load` takes of every checkpoint key, on each of the four ranks + at tp4 and the eight at tp8, at the headers' shapes. Each key is read at most once per rank. + Every text key is read by every rank: a sliced one -- heads, rows, columns, vocabulary rows, + block scales on the same boundary, each expert's width slice -- along exactly one dimension, + rank r at [r w, (r + 1) w), so the ranks tile it exactly once; any other whole, the routed + experts' per-tensor scalars and every vision key included. No DSpark draft key is read. The + bytes so read are the loader's own per-rank total, and it is the headers' 157,212,207,424 at + tp4 and 82,442,488,768 at tp8.""" + regions: dict = {} + for rank in range(world): + log: list = [] + weights = { + key: _Cut(key, dtype, shape, log) for key, (dtype, shape) in checkpoint.headers.items() + } + W.MODEL_WEIGHTS.load(_Shell(checkpoint.config, world, rank, "meta"), weights) + keys = [key for key, _ in log] + assert len(keys) == len(set(keys)), "a key was read twice" + total = experts = 0 + for key, index in log: + dtype, shape = checkpoint.headers[key] + region = _region(index, shape) + regions.setdefault(key, {})[rank] = region + n = torch.empty((), dtype=dtype).element_size() + for start, stop in region: + n *= stop - start + total += n + experts += n if _is_expert(key) else 0 + assert (total, experts) == (_RANK_BYTES[world], _RANK_EXPERT_BYTES[world]), rank + + sliced = 0 + for key, (_, shape) in checkpoint.headers.items(): + whole = tuple((0, size) for size in shape) + if _is_draft(key): + assert key not in regions, key + elif (cut := _sliced(key, world)) is None: + assert regions[key] == {r: whole for r in range(world)}, key + else: + dim, width = cut + assert width * world == shape[dim], (key, shape) + for r in range(world): + want = list(whole) + want[dim] = (r * width, (r + 1) * width) + assert regions[key][r] == tuple(want), (key, r, regions[key][r]) + sliced += 1 + # 72 x (wq, wo, their scales, SSMax s and b, wkv_b) + 6 dense + 71 x 6 shared + output, and + # 71 x 128 x 3 x (codes, block scales) of the routed experts. + assert sliced == 72 * 7 + 6 + 71 * 6 + 1 + 71 * 128 * 6 + + +@_WORLD_RANKS +def test_load_on_meta_matches_the_headers(checkpoint, expert_scalars, world, rank): + """The production `load` over meta tensors shaped like the safetensors headers, with the + checkpoint's own expert scalars: the scalar asserts pass on every layer, every source is + sliced and transformed to its declared per-rank shape and dtype, every declared parameter + is filled, and every key is consumed or known-not-loaded.""" + weights = _meta(checkpoint) + weights.update(expert_scalars) + model = _Shell(checkpoint.config, world, rank, "meta") + W.MODEL_WEIGHTS.load(model, weights) + + +def test_scale_invariants_hold_on_the_checkpoint(checkpoint, expert_scalars): + """Every FP8 block scale goes through the production transform (squeezed from + `[N/128, 1, K/128, 1]`, asserted UE8M0, packed), and every expert scalar passes the + production check: per layer, w1 == w3 on both scalars of every expert, and one input_scale + per projection over all 128 experts.""" + fp8 = sorted( + k + for k, (dtype, _) in checkpoint.headers.items() + if k.endswith(".weight_scale") and dtype == torch.float32 + ) + assert len(fp8) == 404 == 72 + 72 + 44 + 71 * 3 + 3 + by_file: dict = {} + for key in fp8: + by_file.setdefault(checkpoint.index[key], []).append(key) + for name, keys in sorted(by_file.items()): + with safe_open(str(checkpoint.path / name), framework="pt", device="cpu") as handle: + for key in keys: + s = handle.get_tensor(key) + n_blk, _, k_blk, _ = s.shape + packed = W._block_sf(s, None) + assert packed.shape == (W._sf_words(k_blk * 128), n_blk * 128), key + + d = W.MODEL_WEIGHTS.dims(_core(checkpoint.config, 4, 0)) + assert len(expert_scalars) == 71 * 128 * 6 + W._check_expert_scalars(expert_scalars, d) + for i in range(1, _LAYERS): + isc1 = expert_scalars[f"layers.{i}.experts.0.w1.input_scale"].item() + isc2 = expert_scalars[f"layers.{i}.experts.0.w2.input_scale"].item() + assert 0 < isc1 < 1 and 0 < isc2 < 1, (i, isc1, isc2) + + +@pytest.mark.parametrize( + "world, param, module, k, words", + [ + pytest.param(4, "mlp_dn_s", "shared_experts.w2", 1536, 3, id="tp4-shared"), + pytest.param(4, "mlp_dn_s", "feed_forward.w2", 6144, 12, id="tp4-dense"), + pytest.param(4, "wo_s", "attention.wo", 2048, 4, id="tp4-wo"), + pytest.param(8, "mlp_dn_s", "shared_experts.w2", 768, 2, id="tp8-shared"), + pytest.param(8, "mlp_dn_s", "feed_forward.w2", 3072, 6, id="tp8-dense"), + pytest.param(8, "wo_s", "attention.wo", 1024, 2, id="tp8-wo"), + ], +) +def test_packed_k_operand_of_each_rank_on_the_checkpoint( + checkpoint, world, param, module, k, words +): + """The DeepGEMM scale operand of each down projection at the K a rank's column slice gives, + built by the production path (`fill`'s steps: the row's shard on the lazy entry, then its + transform) from the checkpoint's own scales, on every layer and rank. It equals an + independent byte-level packing of the test's own slice of the stored grid and + `transform_sf_into_required_layout(recipe=(1, 128, 128))` on that slice, layout and strides + included. At tp4 every K is whole words (12, 48 and 16 blocks); at tp8 the shared expert's + K 768 is 6 blocks in 2 words: bytes 0..3 of word 0 and 0..1 of word 1 hold blocks 0..5, and + bytes 2..3 of word 1 are zero.""" + k_blk = k // 128 + cores = [_core(checkpoint.config, world, rank) for rank in range(world)] + manifests = [W.MODEL_WEIGHTS.manifest(core) for core in cores] + checked = 0 + for i in range(_LAYERS): + key = f"layers.{i}.{module}.weight_scale" + if key not in checkpoint.index: + continue + with safe_open( + str(checkpoint.path / checkpoint.index[key]), framework="pt", device="cpu" + ) as handle: + stored = handle.get_tensor(key) + built = [] + for core, manifest in zip(cores, manifests): + [(ckpt, index, transform, shard)] = manifest[f"l{i}_{param}"] + assert (ckpt, index) == (key, None) + entry = W._materialize(handle.get_slice(key), shard) + built.append(transform(entry.contiguous(), core)) + for rank, got in enumerate(built): + n = stored.shape[0] * 128 + grid = stored[:, 0, rank * k_blk : (rank + 1) * k_blk, 0] + assert got.shape == (words, n) and got.dtype == torch.int32, (key, got.shape) + assert torch.equal(got, _packed(grid)), (key, rank) + operand = got.t() + engine = transform_sf_into_required_layout( + grid.contiguous(), mn=n, k=k, recipe=(1, 128, 128), is_sfa=False + ) + assert engine.shape == operand.shape and engine.stride() == operand.stride() + assert torch.equal(engine, operand), (key, rank) + check_sf_layout( + operand, + mn=n, + k=k, + gran=(1, 128), + num_groups=None, + tma_stride_check=True, + type_check=torch.int, + ) + assert torch.equal(inverse_transform_sf(operand, mn=n, k=k), grid), (key, rank) + # Byte slot 4w + j of a row is byte j (low first) of its word w: K block 4w + j. + slots = got.view(torch.uint8).view(words, n, 4).permute(1, 0, 2).reshape(n, 4 * words) + assert bool((slots[:, :k_blk] > 0).all()), (key, rank) + assert bool((slots[:, k_blk:] == 0).all()), (key, rank) + checked += 1 + layers = {"shared_experts.w2": 71, "feed_forward.w2": 1, "attention.wo": 72}[module] + assert checked == world * layers + + +# --------------------------------------------------------------------- real experts, sliced + +#: Sampled routed experts: an early, a middle and a late layer, each at both ends of the expert +#: axis and once between. +_SAMPLED = {1: (0, 77, 127), 35: (3, 64, 126), 71: (0, 101, 127)} + +#: The stock TRTLLM-Gen NVFP4 method's `epilogue_tile_m`, which picks its 32-row shuffle. +_EPILOGUE_TILE_M = 128 + + +@pytest.fixture(scope="module") +def stored_experts(checkpoint): + """The sampled experts' codes and block scales, read whole: the independent side of the + comparisons below (the production loader never reads an expert whole).""" + keys = [ + W._expert_key(i, e, f"{proj}.{s}") + for i, experts in _SAMPLED.items() + for e in experts + for proj in _PROJECTIONS + for s in ("weight", "weight_scale") + ] + return _read_whole(checkpoint, keys) + + +def _stock(stored: dict, i: int, e: int, rank: int, tp: int) -> dict: + """One expert as the stock TRTLLM-Gen NVFP4 MoE method prepares it for moe_tp_rank `rank` + of moe_tp_size `tp`: `load_weight_shard` slices w1/w3 as COLUMN (output rows) and w2 as ROW + (input columns), codes and block scales alike; FC1 stacks `[w3 ; w1]`; then the method's + own permutations, `shuffle_matrix` and, on the block scales, `block_scale_interleave`, + composed as its `_shuffle_*` methods compose them. Those methods first move the buffer to a + GPU; the ops they call run on CPU, so this calls them directly.""" + cache: dict = {} + + def shard(proj, s, mode): + t = stored[W._expert_key(i, e, f"{proj}.{s}")] + return load_weight_shard(t, tp, rank, mode).contiguous() + + col, row = TensorParallelMode.COLUMN, TensorParallelMode.ROW + w31 = torch.cat([shard("w3", "weight", col), shard("w1", "weight", col)]) + perm = trtllmgen_maybe_get_cached_w3_w1_permute_indices(w31, cache, _EPILOGUE_TILE_M) + fc1_w = torch.ops.trtllm.shuffle_matrix(w31, perm) + + s31 = torch.cat([shard("w3", "weight_scale", col), shard("w1", "weight_scale", col)]) + s31 = s31.view(torch.uint8) + perm = trtllmgen_maybe_get_cached_w3_w1_permute_indices( + s31, cache, _EPILOGUE_TILE_M, num_elts_per_sf=W.NVFP4_BLOCK + ) + fc1_s = torch.ops.trtllm.block_scale_interleave(torch.ops.trtllm.shuffle_matrix(s31, perm)) + + w2 = shard("w2", "weight", row) + perm = trtllmgen_maybe_get_cached_w2_permute_indices(w2, cache, _EPILOGUE_TILE_M) + fc2_w = torch.ops.trtllm.shuffle_matrix(w2, perm) + + s2 = shard("w2", "weight_scale", row).view(torch.uint8) + perm = trtllmgen_maybe_get_cached_w2_permute_indices( + s2, cache, _EPILOGUE_TILE_M, num_elts_per_sf=W.NVFP4_BLOCK + ) + fc2_s = torch.ops.trtllm.block_scale_interleave(torch.ops.trtllm.shuffle_matrix(s2, perm)) + return dict( + fc1_w=fc1_w, + fc1_s=fc1_s.reshape(s31.shape), + fc2_w=fc2_w, + fc2_s=fc2_s.reshape(s2.shape), + ) + + +def _formula(stored: dict, i: int, e: int, world: int, rank: int) -> dict: + """The same expert slice from the whole stored expert: cut here in torch, then laid out by + this file's own row-at-a-time construction (`_runner_rows`, `_swizzled`), which the + scaled-down tests below also use.""" + + def part(proj, s, axis): + t = _bytes(stored[W._expert_key(i, e, f"{proj}.{s}")]) + n = t.shape[axis] // world + return t.narrow(axis, rank * n, n) + + fc1 = _runner_rows(2 * _GEOMETRY[world].inter, True) + fc2 = _runner_rows(6144, False) + return dict( + fc1_w=torch.cat([part("w3", "weight", 0), part("w1", "weight", 0)])[fc1], + fc1_s=_swizzled( + torch.cat([part("w3", "weight_scale", 0), part("w1", "weight_scale", 0)])[fc1] + ), + fc2_w=part("w2", "weight", 1)[fc2], + fc2_s=_swizzled(part("w2", "weight_scale", 1)[fc2]), + ) + + +def _load_with_real_experts(checkpoint, expert_scalars, world: int, rank: int): + """The production `load` of `rank` on header-shaped meta tensors, except for the sampled + layers' eight expert parameters: real CPU tensors, first filled with 0xab bytes and NaN, + whose sources are the sampled experts' lazy entries of the shard files and, for every other + expert, a zero stand-in of the stored shape. Returns the core.""" + model = _Shell(checkpoint.config, world, rank, "meta") + w = model.model.w + for i in _SAMPLED: + for role in _EXPERT_PARAMS: + p = w[f"l{i}_{role}"] + if p.element_size() == 1: + real = torch.full(p.shape, 0xAB, dtype=torch.uint8).view(p.dtype) + else: + real = torch.full(p.shape, float("nan"), dtype=p.dtype) + w[f"l{i}_{role}"] = nn.Parameter(real, requires_grad=False) + weights = _meta(checkpoint) + weights.update(expert_scalars) + with contextlib.ExitStack() as stack: + handles: dict = {} + for i, sampled in _SAMPLED.items(): + for e in range(128): + for proj in _PROJECTIONS: + for s in ("weight", "weight_scale"): + key = W._expert_key(i, e, f"{proj}.{s}") + dtype, shape = checkpoint.headers[key] + if e not in sampled: + zeros = torch.zeros(shape[-1], dtype=torch.uint8).view(dtype) + weights[key] = zeros.expand(shape) + continue + name = checkpoint.index[key] + if name not in handles: + handles[name] = stack.enter_context( + safe_open(str(checkpoint.path / name), framework="pt", device="cpu") + ) + weights[key] = handles[name].get_slice(key) + W.MODEL_WEIGHTS.load(model, weights) + return model.model + + +@_WORLD_RANKS +def test_real_experts_slice_then_layout(checkpoint, expert_scalars, stored_experts, world, rank): + """The production `load`, reading the sampled experts' real entries through its shards, + holds in slot e of each sampled layer's FC1/FC2 codes and block scales exactly the stock + preparation's bytes of expert e at moe_tp_size 4 or 8 -- which this file's own construction + from the whole stored expert reproduces too -- and in every other slot its zero stand-in's, so + every slot was written by its own expert. The per-tensor scalars are whole: every expert's + `weight_scale_2`, the layer's one FC1 and FC2 `input_scale`.""" + core = _load_with_real_experts(checkpoint, expert_scalars, world, rank) + for i, sampled in _SAMPLED.items(): + for e in sampled: + stock = _stock(stored_experts, i, e, rank, world) + formula = _formula(stored_experts, i, e, world, rank) + for role in _LAID_OUT: + assert torch.equal(stock[role], formula[role]), (i, e, role) + got = _bytes(core.w[f"l{i}_{role}"].data[e]) + assert torch.equal(got, stock[role]), (i, e, role, rank) + for role in _LAID_OUT: + param = _bytes(core.w[f"l{i}_{role}"].data) + nonzero = torch.count_nonzero(param.reshape(128, -1), dim=1) + assert {e for e in range(128) if nonzero[e]} == set(sampled), (i, role) + + def scalars(name): + return torch.stack([expert_scalars[W._expert_key(i, e, name)] for e in range(128)]) + + assert torch.equal(core.w[f"l{i}_ws2_13"].data, scalars("w1.weight_scale_2")) + assert torch.equal(core.w[f"l{i}_ws2_2"].data, scalars("w2.weight_scale_2")) + assert torch.equal(core.w[f"l{i}_isc1"].data, scalars("w1.input_scale")[:1]) + assert torch.equal(core.w[f"l{i}_isc2"].data, scalars("w2.input_scale")[:1]) + + +@_WORLDS +def test_real_slices_reassemble_the_stored_experts(checkpoint, stored_experts, manifests, world): + """Each rank's production shard read off the shard file -- a row read of w1/w3, a strided + column read of w2 -- gives that rank's slice of a sampled expert's codes and block scales, + and the four (tp4) or eight (tp8) slices joined on the cut axis are the stored tensor, byte + for byte.""" + for i, sampled in _SAMPLED.items(): + for e in sampled: + for proj in _PROJECTIONS: + for s, kind in (("weight", "_w"), ("weight_scale", "_s")): + key = W._expert_key(i, e, f"{proj}.{s}") + role = ("fc2" if proj == "w2" else "fc1") + kind + shape, axis, width = _SLICED[world][role] + parts = [] + path = str(checkpoint.path / checkpoint.index[key]) + with safe_open(path, framework="pt", device="cpu") as handle: + for rank in range(world): + shard = manifests[world][rank][f"l{i}_{role}"][0][3] + part = W._materialize(handle.get_slice(key), shard) + assert ( + part.shape[axis] == width + and part.shape[1 - axis] == shape[1 - axis] + ) + parts.append(_bytes(part)) + assert torch.equal(torch.cat(parts, axis), _bytes(stored_experts[key])), key + + +@_WORLDS +def test_real_experts_layout_then_slice(stored_experts, world): + """The negative control: each sampled expert laid out whole -- the stock preparation at + moe_tp_size 1, the layout a rank holding whole experts uses -- then cut to rank r, against + the per-rank layout the tests above pin. The cut differs where the layout depends on the + slice: on FC2's block scales, whose 128x4 swizzle orders bytes by the column count (384 + whole, 96 per rank at tp4, 48 at tp8), and on FC1 cut at the checkpoint's ranges -- rows + [Ir, + I) of the w3 half and of the w1 half, I = 1536 at tp4 and 768 at tp8 -- which the + interleave has spread over the stack. It coincides where the layout leaves the cut alone: FC1 + cut into contiguous 2I-row chunks (the interleave pairs up row i with gate row i, so a rank's + pairs are one run of the stack, which starts on a 32-row shuffle block and a 128-row swizzle + tile), and FC2's codes, whose shuffle moves rows while the cut takes columns.""" + inter = _GEOMETRY[world].inter + for i, sampled in _SAMPLED.items(): + for e in sampled: + whole = _stock(stored_experts, i, e, rank=0, tp=1) + for r in range(world): + mine = _stock(stored_experts, i, e, r, world) + for role in ("fc1_w", "fc1_s"): + t = whole[role] + halves = [t[h * 6144 + r * inter : h * 6144 + (r + 1) * inter] for h in (0, 1)] + assert not torch.equal(torch.cat(halves), mine[role]), (i, e, r, role) + rows = slice(r * 2 * inter, (r + 1) * 2 * inter) + assert torch.equal(t[rows], mine[role]), (i, e, r, role) + fc2_s = whole["fc2_s"][:, r * inter // 16 : (r + 1) * inter // 16] + assert not torch.equal(fc2_s, mine["fc2_s"]), (i, e, r) + fc2_w = whole["fc2_w"][:, r * inter // 2 : (r + 1) * inter // 2] + assert torch.equal(fc2_w, mine["fc2_w"]), (i, e, r) + + +# --------------------------------------------------------------------- values, scaled down + +_TINY_LAYERS = 8 +#: Every structural feature at a size a CPU fills in a second, by world size: a dense layer, +#: RoPE and NoPE layers (3, 7), a KV sharer (7, reading 3), 'shared' indexer layers (2, 5, 6), +#: two heads per rank, every expert cut to 128 per rank (FC1 256 rows, FC2 64 code bytes and 8 +#: scale columns per rank: the shuffle and swizzle tile them), one 128-block per rank on the +#: shared expert. Eight experts and the dense layer at 2 128-blocks per rank at tp4; sixteen +#: experts and the dense layer at 6 128-blocks per rank at tp8, so its `w2` scale operand has the +#: half-padding second word the shared expert's has at full size. +_TINY_COMMON = dict( + num_hidden_layers=_TINY_LAYERS, + hidden_size=256, + qk_nope_head_dim=64, + qk_rope_head_dim=64, + v_head_dim=128, + kv_lora_rank=128, + nope_layer_flags=[i % 4 == 3 for i in range(_TINY_LAYERS)], + kv_source_layers=[0, 1, 2, 3, 4, 5, 6, 3], + index_sharing=["full", "full", "shared", "full", "full", "shared", "shared", "full"], + index_n_heads=4, + index_head_dim=128, + first_k_dense_replace=1, + n_shared_experts=1, + vocab_size=512, +) +_TINY = { + 4: dict( + _TINY_COMMON, + num_attention_heads=8, + n_routed_experts=8, + moe_intermediate_size=512, + intermediate_size=1024, + ), + 8: dict( + _TINY_COMMON, + num_attention_heads=16, + n_routed_experts=16, + moe_intermediate_size=1024, + intermediate_size=6144, + ), +} +#: Each routed expert's per-rank slice in the scaled-down table, at either world size. +_TINY_INTER = 128 + + +def _tiny_checkpoint(world: int, seed: int = 0) -> dict: + """A Mistral-named checkpoint at `_TINY`: e4m3 weights with power-of-two fp32 block scales + stored `[N/128, 1, K/128, 1]` where the real one has them, NVFP4 routed experts -- packed + codes, e4m3 block scales, a weight_scale_2 per expert that w1 and w3 share, one input_scale + per projection and layer -- bf16 elsewhere, and two DSpark draft tensors.""" + c = SimpleNamespace(**_TINY[world]) + g = torch.Generator().manual_seed(seed) + bf16 = lambda *s: torch.randn(*s, generator=g).to(torch.bfloat16) # noqa: E731 + scalar = lambda: torch.rand((), generator=g) * 1e-3 + 1e-5 # noqa: E731 + + def fp8(name, n, k): + codes = (torch.rand(n, k, generator=g) * 800 - 400).to(torch.float8_e4m3fn) + exps = torch.randint(-15, -7, (n // 128, 1, k // 128, 1), generator=g).float() + return {f"{name}.weight": codes, f"{name}.weight_scale": torch.exp2(exps)} + + def nvfp4(name, n, k, ws2, isc): + codes = torch.randint(0, 256, (n, k // 2), dtype=torch.uint8, generator=g) + sf = (torch.rand(n, k // 16, generator=g) * 400 + 1).to(torch.float8_e4m3fn) + return { + f"{name}.weight": codes, + f"{name}.weight_scale": sf, + f"{name}.weight_scale_2": ws2.clone(), + f"{name}.input_scale": isc.clone(), + } + + h, qk = c.hidden_size, c.qk_nope_head_dim + c.qk_rope_head_dim + ck = {"tok_embeddings.weight": bf16(c.vocab_size, h), "norm.weight": bf16(h)} + ck["output.weight"] = bf16(c.vocab_size, h) + ck["lerope.alpha"] = bf16(c.qk_rope_head_dim // 2) + ck["mtp_dspark.backbone.final_norm.weight"] = bf16(32) + ck["mtp_dspark.backbone.layers.0.attention.wq.weight"] = bf16(32, 64) + for i in range(c.num_hidden_layers): + p = f"layers.{i}" + nope = c.nope_layer_flags[i] + ck[f"{p}.attention_norm.weight"] = bf16(h) + ck[f"{p}.ffn_norm.weight"] = bf16(h) + ck.update(fp8(f"{p}.attention.wq", c.num_attention_heads * qk, h)) + ck.update(fp8(f"{p}.attention.wo", h, c.num_attention_heads * c.v_head_dim)) + ck[f"{p}.attention.ssmax_scaling.s"] = bf16(c.num_attention_heads) + ck[f"{p}.attention.ssmax_scaling.b"] = bf16(c.num_attention_heads) + per_head = (qk if nope else c.qk_nope_head_dim) + c.v_head_dim + ck[f"{p}.attention.wkv_b.weight"] = bf16(c.num_attention_heads * per_head, c.kv_lora_rank) + if c.kv_source_layers[i] == i: + lat = c.kv_lora_rank + (0 if nope else c.qk_rope_head_dim) + ck[f"{p}.attention.wkv_a_with_mqa.weight"] = bf16(lat, h) + ck[f"{p}.attention.kv_a_norm.weight"] = bf16(c.kv_lora_rank) + if c.index_sharing[i] == "full": + x = f"{p}.attention.indexer" + ck.update(fp8(f"{x}.wq_b", c.index_n_heads * c.index_head_dim, h)) + ck[f"{x}.wk.weight"] = bf16(c.index_head_dim, h) + ck[f"{x}.weights_proj.weight"] = bf16(c.index_n_heads, h) + for norm, n in (("q_input_norm", h), ("k_norm", c.index_head_dim)): + ck[f"{x}.{norm}.weight"] = bf16(n) + ck[f"{x}.{norm}.bias"] = bf16(n) + ck[f"{x}.ssmax_scaling.s"] = bf16(c.index_n_heads) + ck[f"{x}.ssmax_scaling.b"] = bf16(c.index_n_heads) + if i < c.first_k_dense_replace: + for w, (n, k) in (("w1", (c.intermediate_size, h)), ("w3", (c.intermediate_size, h))): + ck.update(fp8(f"{p}.feed_forward.{w}", n, k)) + ck.update(fp8(f"{p}.feed_forward.w2", h, c.intermediate_size)) + continue + ck[f"{p}.gate.weight"] = bf16(c.n_routed_experts, h) + inter = c.moe_intermediate_size + m = f"{p}.shared_experts" + ck.update(fp8(f"{m}.w1", inter, h)) + ck.update(fp8(f"{m}.w3", inter, h)) + ck.update(fp8(f"{m}.w2", h, inter)) + isc1, isc2 = scalar(), scalar() + for e in range(c.n_routed_experts): + m = f"{p}.experts.{e}" + ws2_13 = scalar() + ck.update(nvfp4(f"{m}.w1", inter, h, ws2_13, isc1)) + ck.update(nvfp4(f"{m}.w3", inter, h, ws2_13, isc1)) + ck.update(nvfp4(f"{m}.w2", h, inter, scalar(), isc2)) + return ck + + +def _grid(s: torch.Tensor) -> torch.Tensor: + """An FP8 block scale as stored, `[N/128, 1, K/128, 1]`, as its `[N/128, K/128]` grid.""" + return s.reshape(s.shape[0], s.shape[2]) + + +def _decode_sf(storage: torch.Tensor, n: int, k_blocks: int) -> torch.Tensor: + """Independent reading of the packed scale storage `[ceil(k_blocks/4), n]` int32: word + `[w, row]` holds the UE8M0 bytes of K blocks 4w..4w+3 of `row`, low byte first.""" + words = storage.t().contiguous() # [n, words] + out = torch.empty(n, k_blocks, dtype=torch.float32) + for kb in range(k_blocks): + byte = (words[:, kb // 4] >> (8 * (kb % 4))) & 0xFF + out[:, kb] = torch.exp2(byte.float() - 127) + return out + + +def _runner_rows(n: int, interleave: bool) -> torch.Tensor: + """The stored row each row of an expert's runner operand holds, one row at a time: row + `32b + 8v + u` takes row `32b + 4u + v` of the interleaved stack, whose row `2i` is up row + `i` (stored row `i`) and row `2i + 1` gate row `i` (stored row `n/2 + i`).""" + rows = [] + for r in range(n): + b, within = divmod(r, 32) + v, u = divmod(within, 8) + j = 32 * b + 4 * u + v + rows.append((j // 2 if j % 2 == 0 else n // 2 + j // 2) if interleave else j) + return torch.tensor(rows) + + +def _swizzle_offsets(m: int, c: int) -> torch.Tensor: + """`[m, c]`: where scale byte `(row, col)` lands in the 128x4-swizzled buffer.""" + r = torch.arange(m).view(-1, 1) + k = torch.arange(c).view(1, -1) + return (r // 128) * 128 * c + (k // 4) * 512 + (r % 32) * 16 + ((r % 128) // 32) * 4 + k % 4 + + +def _swizzled(s: torch.Tensor) -> torch.Tensor: + """`[m, c]` scale bytes scattered to their swizzled offsets, viewed `[m, c]` again.""" + m, c = s.shape + out = torch.empty(m * c, dtype=torch.uint8) + out[_swizzle_offsets(m, c).reshape(-1)] = s.reshape(-1) + return out.view(m, c) + + +def _unswizzled(s: torch.Tensor) -> torch.Tensor: + m, c = s.shape + return s.reshape(-1)[_swizzle_offsets(m, c)] + + +def _expected(ck: dict, world: int, rank: int) -> dict: + """What each parameter of `rank` must hold, built from the checkpoint by plain slicing, + with fp32 block-scale grids standing for the packed scale storage (`*_s` int32).""" + c = SimpleNamespace(**_TINY[world]) + heads = c.num_attention_heads // world + qk, v = c.qk_nope_head_dim + c.qk_rope_head_dim, c.v_head_dim + out = {} + + def rows(t, n): + return t[rank * n : (rank + 1) * n] + + def cols(t, n): + return t[:, rank * n : (rank + 1) * n] + + for i in range(c.num_hidden_layers): + p, q = f"layers.{i}", f"l{i}_" + nope = qk if c.nope_layer_flags[i] else c.qk_nope_head_dim + out[q + "attn_norm"] = ck[f"{p}.attention_norm.weight"] + out[q + "ffn_norm"] = ck[f"{p}.ffn_norm.weight"] + out[q + "wq"] = rows(ck[f"{p}.attention.wq.weight"], heads * qk) + out[q + "wq_s"] = rows(_grid(ck[f"{p}.attention.wq.weight_scale"]), heads * qk // 128) + out[q + "wo"] = cols(ck[f"{p}.attention.wo.weight"], heads * v) + out[q + "wo_s"] = cols(_grid(ck[f"{p}.attention.wo.weight_scale"]), heads * v // 128) + for s in ("s", "b"): + out[q + f"ssmax_{s}"] = rows(ck[f"{p}.attention.ssmax_scaling.{s}"], heads) + kvb = rows(ck[f"{p}.attention.wkv_b.weight"], heads * (nope + v)) + per_head = [kvb[h * (nope + v) : (h + 1) * (nope + v)] for h in range(heads)] + out[q + "wkv_b"] = torch.cat([t[:nope] for t in per_head] + [t[nope:] for t in per_head]) + if c.kv_source_layers[i] == i: + out[q + "wkv_a"] = ck[f"{p}.attention.wkv_a_with_mqa.weight"] + out[q + "kv_a_norm"] = ck[f"{p}.attention.kv_a_norm.weight"] + if c.index_sharing[i] == "full": + x = f"{p}.attention.indexer" + out[q + "ix_q_norm_w"] = ck[f"{x}.q_input_norm.weight"] + out[q + "ix_q_norm_b"] = ck[f"{x}.q_input_norm.bias"] + out[q + "ix_wq_b"] = ck[f"{x}.wq_b.weight"] + out[q + "ix_wq_b_s"] = _grid(ck[f"{x}.wq_b.weight_scale"]) + out[q + "ix_wk_wp"] = torch.cat([ck[f"{x}.wk.weight"], ck[f"{x}.weights_proj.weight"]]) + out[q + "ix_k_norm_w"] = ck[f"{x}.k_norm.weight"].float() + out[q + "ix_k_norm_b"] = ck[f"{x}.k_norm.bias"].float() + out[q + "ix_ssmax_s"] = ck[f"{x}.ssmax_scaling.s"] + out[q + "ix_ssmax_b"] = ck[f"{x}.ssmax_scaling.b"] + if i < c.first_k_dense_replace: + m, inter = f"{p}.feed_forward", c.intermediate_size // world + else: + m, inter = f"{p}.shared_experts", c.moe_intermediate_size // world + # Gate rows first. + out[q + "mlp_gu"] = torch.cat( + [_bytes(rows(ck[f"{m}.{w}.weight"], inter)) for w in ("w1", "w3")] + ) + out[q + "mlp_gu_s"] = torch.cat( + [rows(_grid(ck[f"{m}.{w}.weight_scale"]), inter // 128) for w in ("w1", "w3")] + ) + out[q + "mlp_dn"] = cols(ck[f"{m}.w2.weight"], inter) + out[q + "mlp_dn_s"] = cols(_grid(ck[f"{m}.w2.weight_scale"]), inter // 128) + if i < c.first_k_dense_replace: + continue + out[q + "router"] = ck[f"{p}.gate.weight"] + experts = [f"{p}.experts.{e}" for e in range(c.n_routed_experts)] + # Every expert, this rank's 128 rows of w3 and w1 (up rows first), interleaved and + # block-shuffled, and its 64 code bytes / 8 scale columns of w2, block-shuffled; the + # block scales also swizzled. + fc1 = _runner_rows(2 * _TINY_INTER, True) + fc2 = _runner_rows(c.hidden_size, False) + + def up_gate(e, suffix): + return torch.cat( + [_bytes(rows(ck[f"{e}.{w}.{suffix}"], _TINY_INTER)) for w in ("w3", "w1")] + )[fc1] + + def down(e, suffix, per_column): + return _bytes(cols(ck[f"{e}.w2.{suffix}"], _TINY_INTER // per_column))[fc2] + + out[q + "fc1_w"] = torch.stack([up_gate(e, "weight") for e in experts]) + out[q + "fc1_s"] = torch.stack([_swizzled(up_gate(e, "weight_scale")) for e in experts]) + out[q + "fc2_w"] = torch.stack([down(e, "weight", 2) for e in experts]) + out[q + "fc2_s"] = torch.stack([_swizzled(down(e, "weight_scale", 16)) for e in experts]) + out[q + "isc1"] = ck[f"{experts[0]}.w1.input_scale"].reshape(1) + out[q + "isc2"] = ck[f"{experts[0]}.w2.input_scale"].reshape(1) + out[q + "ws2_13"] = torch.stack([ck[f"{e}.w1.weight_scale_2"] for e in experts]) + out[q + "ws2_2"] = torch.stack([ck[f"{e}.w2.weight_scale_2"] for e in experts]) + out["final_norm"] = ck["norm.weight"] + out["embed"] = ck["tok_embeddings.weight"] + out["lerope_alpha"] = ck["lerope.alpha"].float() + return out + + +@_WORLDS +def test_tiny_scale_operands_cover_whole_and_partial_words(world): + """The scaled-down table's scale operands at a rank, by K blocks: 2 for every K of the + hidden size 256 and for `wo` (2 heads x 128), 1 for the shared `w2` (128 of 512 at tp4, of + 1024 at tp8), and for the dense `w2` 2 at tp4 (256 of 1024) and 6 at tp8 (768 of 6144) -- at + tp8 the shared expert's case at full size, 6 blocks in 2 words. Every one has a partly padded + last word, which the values test below compares whole. Every rank holds all 8 (tp4) or 16 + (tp8) experts, each at its 128-wide slice.""" + d = W.MODEL_WEIGHTS.dims(_core(_TINY[world], world, 0)) + experts, dense, dense_words = {4: (8, 256, 1), 8: (16, 768, 2)}[world] + assert (d.heads, d.dense_inter, d.shared_inter) == (2, dense, 128) + assert (d.local_experts, d.expert_offset, d.moe_inter) == (experts, 0, _TINY_INTER) + with torch.device("meta"): + declared = W.MODEL_WEIGHTS.declare(d) + assert tuple(declared["l0_mlp_dn_s"].shape) == (dense_words, 256) + assert tuple(declared["l1_mlp_dn_s"].shape) == (1, 256) + assert tuple(declared["l1_wo_s"].shape) == (1, 256) + + +@_WORLD_RANKS +def test_tiny_load_values(world, rank): + """Every parameter of every rank, bit for bit against `_expected`; each scale operand's + whole int32 storage, padding bytes included, against `_packed`.""" + ck = _tiny_checkpoint(world) + model = _Shell(_TINY[world], world, rank, "cpu") + W.MODEL_WEIGHTS.load(model, dict(ck)) + got = dict(model.model.w) + want = _expected(ck, world, rank) + assert set(got) == set(want) + for key, ref in want.items(): + param = got[key].data + if key.endswith("_s") and param.dtype == torch.int32: + # DeepGEMM's operand is the transpose of the storage: MN-major, row stride N. + n = param.shape[1] + k = ref.shape[1] * 128 + operand = param.t() + check_sf_layout( + operand, + mn=n, + k=k, + gran=(1, 128), + num_groups=None, + tma_stride_check=True, + type_check=torch.int, + ) + per_row = ref.repeat_interleave(128, dim=0) + assert torch.equal(_decode_sf(param, n, ref.shape[1]), per_row), key + assert torch.equal(param, _packed(ref)), key + assert torch.equal(inverse_transform_sf(operand, mn=n, k=k), ref), key + if key.endswith(("wq_s", "wo_s", "ix_wq_b_s", "mlp_dn_s")): + engine = transform_sf_into_required_layout( + ref.contiguous(), mn=n, k=k, recipe=(1, 128, 128), is_sfa=False + ) + assert engine.shape == operand.shape and engine.stride() == operand.stride() + assert torch.equal(engine, operand), key + continue + if ref.dtype == torch.uint8 and param.dtype == torch.float8_e4m3fn: + assert key.endswith(("mlp_gu", "fc1_s", "fc2_s")), key + else: + assert param.dtype == ref.dtype, (key, param.dtype, ref.dtype) + assert torch.equal(_bytes(param), _bytes(ref)), key + lm_rows = _TINY[world]["vocab_size"] // world + assert torch.equal( + model.lm_head.weight.data, ck["output.weight"][rank * lm_rows : (rank + 1) * lm_rows] + ) + + +@_WORLDS +def test_tiny_expert_halves_are_up_then_gate(world): + """Undo the block shuffle (and the scales' swizzle) of FC1: its even rows are this rank's + rows of w3 (up), its odd rows the same rows of w1 (gate), codes and block scales alike -- + the order the NVFP4 runner's FC1 reads, up first, the opposite of `mlp_gu`'s gate first.""" + ck = _tiny_checkpoint(world) + rank, e = 1, 2 + model = _Shell(_TINY[world], world, rank, "cpu") + W.MODEL_WEIGHTS.load(model, dict(ck)) + w = model.model.w + mine = slice(rank * _TINY_INTER, (rank + 1) * _TINY_INTER) + shuffled = _runner_rows(2 * _TINY_INTER, False) + unshuffle = torch.empty_like(shuffled) + unshuffle[shuffled] = torch.arange(2 * _TINY_INTER) + codes = w["l4_fc1_w"][e][unshuffle] + assert torch.equal(codes[0::2], ck[f"layers.4.experts.{e}.w3.weight"][mine]) + assert torch.equal(codes[1::2], ck[f"layers.4.experts.{e}.w1.weight"][mine]) + scales = _unswizzled(_bytes(w["l4_fc1_s"][e]))[unshuffle] + assert torch.equal(scales[0::2], _bytes(ck[f"layers.4.experts.{e}.w3.weight_scale"][mine])) + assert torch.equal(scales[1::2], _bytes(ck[f"layers.4.experts.{e}.w1.weight_scale"][mine])) + + +@_WORLDS +def test_tiny_layout_is_applied_once_per_load(world): + """A second `load` into the same parameters starts again from the stored bytes, so it lands + on the same layout rather than shuffling the first one's result again.""" + ck = _tiny_checkpoint(world) + model = _Shell(_TINY[world], world, world - 1, "cpu") + W.MODEL_WEIGHTS.load(model, dict(ck)) + first = {k: _bytes(v.data).clone() for k, v in model.model.w.items() if k.startswith("l5_fc")} + W.MODEL_WEIGHTS.load(model, dict(ck)) + for key, value in first.items(): + assert torch.equal(_bytes(model.model.w[key].data), value), key + + +def _write_shards(ck: dict, directory: Path, files: int = 3) -> None: + """`ck` as Mistral-named shard files with their index, keys dealt round-robin, so one + expert's tensors -- and w1's and w3's scalars -- land in different files.""" + from safetensors.torch import save_file + + names = [f"consolidated-{i + 1:05d}-of-{files:05d}.safetensors" for i in range(files)] + weight_map = {key: names[i % files] for i, key in enumerate(sorted(ck))} + for name in names: + part = {k: v.contiguous() for k, v in ck.items() if weight_map[k] == name} + save_file(part, str(directory / name)) + (directory / W.SHARD_INDEX).write_text(json.dumps({"weight_map": weight_map})) + + +class _ReadLog: + """`safe_open` that records every key read through it, and for each the index its lazy + entry was read at and the shape that read returned.""" + + def __init__(self, real): + self.real, self.keys, self.reads = real, [], [] + + def __call__(self, *args, **kwargs): + log = self + + class Entry: + def __init__(self, key, entry): + self.key, self.entry = key, entry + + def __getitem__(self, index): + out = self.entry[index] + log.reads.append((self.key, index, tuple(out.shape))) + return out + + class Handle: + def __enter__(self): + self.h = log.real(*args, **kwargs).__enter__() + return self + + def __exit__(self, *exc): + return self.h.__exit__(*exc) + + def get_slice(self, key): + log.keys.append(key) + return Entry(key, self.h.get_slice(key)) + + return Handle() + + +@_WORLD_RANKS +def test_tiny_load_streams_from_the_shard_files(world, rank, tmp_path, monkeypatch): + """With the checkpoint's shard files found through the directory the engine records on + the config, every source is read from them -- the given dict is never read: here it + holds meta tensors, which cannot be copied out -- and the result is bit for bit the dict + path's. Each key is read once, every key but the DSpark draft's, nothing of the draft. Every + expert's codes and block scales are read as this rank's slice of the entry -- w1/w3 rows + [128r, + 128), w2's code bytes [64r, + 64) and scale columns [8r, + 8) -- never whole; the + per-tensor scalars whole. Each file's cached pages are dropped once read.""" + ck = _tiny_checkpoint(world) + _write_shards(ck, tmp_path) + log = _ReadLog(W.safe_open) + dropped = [] + drop = W._drop_page_cache + monkeypatch.setattr(W, "safe_open", log) + monkeypatch.setattr(W, "_drop_page_cache", lambda path: (dropped.append(path), drop(path))) + streamed = _Shell(_TINY[world], world, rank, "cpu") + streamed.model.model_config.pretrained_config._name_or_path = str(tmp_path) + W.MODEL_WEIGHTS.load(streamed, {k: torch.empty_like(v, device="meta") for k, v in ck.items()}) + monkeypatch.undo() + direct = _Shell(_TINY[world], world, rank, "cpu") + W.MODEL_WEIGHTS.load(direct, dict(ck)) + got, want = dict(streamed.named_parameters()), dict(direct.named_parameters()) + assert set(got) == set(want) + for key in want: + assert torch.equal(_bytes(got[key].data), _bytes(want[key].data)), key + + read = set(log.keys) + assert len(read) == len(log.keys), "a key was read twice" + assert read == {k for k in ck if not _is_draft(k)} + reads = {} + for key, index, shape in log.reads: + assert key not in reads, f"{key}: its entry was read twice" + reads[key] = (index, shape) + for key in ck: + if not _is_expert(key): + continue + index, shape = reads[key] + stored = tuple(ck[key].shape) + if _is_expert_scalar(key): + assert (index, shape) == (Ellipsis, ()), key + elif ".w2." in key: + width = _TINY_INTER // (2 if key.endswith(".weight") else 16) + assert index == (slice(None), slice(rank * width, (rank + 1) * width)), (key, index) + assert shape == (stored[0], width), key + else: + assert index == slice(rank * _TINY_INTER, (rank + 1) * _TINY_INTER), (key, index) + assert shape == (_TINY_INTER, stored[1]), key + assert sorted(p.name for p in dropped) == sorted(p.name for p in tmp_path.glob("*.safetensors")) + + +@_WORLDS +@pytest.mark.parametrize("with_index", [True, False]) +def test_tiny_load_checkpoint_reads_the_directory(world, with_index, tmp_path): + """Given the checkpoint directory alone, as the engine gives it, the result is bit for bit + the dict path's: the files come from the directory's index, or without one from the shard + files' headers.""" + ck = _tiny_checkpoint(world) + _write_shards(ck, tmp_path) + if not with_index: + (tmp_path / W.SHARD_INDEX).unlink() + loaded = _Shell(_TINY[world], world, 2, "cpu") + W.MODEL_WEIGHTS.load_checkpoint(loaded, tmp_path) + direct = _Shell(_TINY[world], world, 2, "cpu") + W.MODEL_WEIGHTS.load(direct, dict(ck)) + got, want = dict(loaded.named_parameters()), dict(direct.named_parameters()) + assert set(got) == set(want) + for key in want: + assert torch.equal(_bytes(got[key].data), _bytes(want[key].data)), key + + +@_WORLDS +def test_tiny_load_falls_back_to_the_dict_without_an_index(world, tmp_path): + """A recorded directory without the shard index is not the checkpoint: the dict is read.""" + ck = _tiny_checkpoint(world) + model = _Shell(_TINY[world], world, 2, "cpu") + model.model.model_config.pretrained_config._name_or_path = str(tmp_path) + W.MODEL_WEIGHTS.load(model, dict(ck)) + + +@_WORLDS +def test_tiny_load_without_draft_tensors(world): + """The DSpark draft is never asserted on: a checkpoint without it loads too.""" + ck = {k: v for k, v in _tiny_checkpoint(world).items() if not _is_draft(k)} + W.MODEL_WEIGHTS.load(_Shell(_TINY[world], world, 1, "cpu"), ck) + + +def _load_raises(ck: dict, match: str, world: int, rank: int = 0) -> None: + with pytest.raises(AssertionError, match=match): + W.MODEL_WEIGHTS.load(_Shell(_TINY[world], world, rank, "cpu"), ck) + + +@pytest.mark.parametrize( + "world, key", + [ + (4, "layers.3.experts.6.w3.weight"), + (4, "layers.3.experts.1.w1.weight_scale"), + (4, "layers.3.experts.7.w2.weight"), + (4, "layers.3.experts.0.w3.weight_scale_2"), + (4, "layers.3.experts.5.w2.input_scale"), + (8, "layers.3.experts.6.w3.weight"), + (8, "layers.3.experts.1.w1.weight_scale"), + (8, "layers.3.experts.13.w2.weight"), + (8, "layers.3.experts.0.w3.weight_scale_2"), + (8, "layers.3.experts.15.w2.input_scale"), + ], +) +def test_tiny_load_refuses_a_missing_expert_tensor(world, key): + """Every rank reads every expert, so any expert's tensor missing is refused on rank 0.""" + ck = _tiny_checkpoint(world) + del ck[key] + _load_raises(ck, f"checkpoint key missing: {key}", world) + + +@_WORLDS +def test_tiny_load_refuses_an_unknown_tensor(world): + ck = _tiny_checkpoint(world) + ck["layers.2.attention.wq_extra.weight"] = torch.zeros(4) + _load_raises(ck, "checkpoint coverage", world) + + +@pytest.mark.parametrize( + "key", + [ + "layers.0.attention.wq.weight_scale", + "layers.0.feed_forward.w3.weight_scale", + "layers.4.shared_experts.w2.weight_scale", + "layers.4.attention.indexer.wq_b.weight_scale", + "layers.5.attention.wo.weight_scale", + ], +) +@_WORLDS +def test_tiny_load_refuses_a_scale_that_is_not_ue8m0(world, key): + ck = _tiny_checkpoint(world) + ck[key] = ck[key].clone() + ck[key].view(-1)[0] *= 1.5 + _load_raises(ck, "not a positive normal power of two", world) + + +@_WORLDS +def test_tiny_load_refuses_a_two_dimensional_fp8_scale(world): + """A `[N/128, K/128]` block-scale layout is not this one's.""" + ck = _tiny_checkpoint(world) + key = "layers.4.attention.indexer.wq_b.weight_scale" + ck[key] = _grid(ck[key]).contiguous() + _load_raises(ck, r"not \[N/128, 1, K/128, 1\]", world) + + +@pytest.mark.parametrize( + "world, expert", [(4, 0), (4, 6), (8, 0), (8, 13)], ids=["tp4-0", "tp4-6", "tp8-0", "tp8-13"] +) +@pytest.mark.parametrize("scalar", ["weight_scale_2", "input_scale"]) +def test_tiny_load_refuses_w1_and_w3_scalars_that_differ(world, expert, scalar): + ck = _tiny_checkpoint(world) + key = f"layers.5.experts.{expert}.w3.{scalar}" + ck[key] = ck[key] * 2 + experts = _TINY[world]["n_routed_experts"] + want = f"layers.5: w1 and w3 {scalar} differ on 1 of {experts} experts, first expert {expert}:" + _load_raises(ck, want, world) + + +@_WORLDS +@pytest.mark.parametrize("proj", ["w1", "w2"]) +def test_tiny_load_refuses_an_input_scale_that_varies_over_the_experts(world, proj): + """One quantization of the replicated activations serves every expert's slice on every + rank: one expert disagreeing -- the last, 7 at tp4 and 15 at tp8 -- is refused.""" + ck = _tiny_checkpoint(world) + last = _TINY[world]["n_routed_experts"] - 1 + for p in {proj, "w3"} if proj == "w1" else {proj}: + key = f"layers.6.experts.{last}.{p}.input_scale" + ck[key] = ck[key] * 2 + want = f"layers.6: {proj}.input_scale is not one value over the experts: expert {last}" + _load_raises(ck, want, world) + + +@_WORLDS +@pytest.mark.parametrize("value", [0.0, -1e-4, float("inf"), float("nan")]) +def test_tiny_load_refuses_a_scalar_that_is_not_a_positive_finite_scale(world, value): + ck = _tiny_checkpoint(world) + ck["layers.2.experts.3.w2.weight_scale_2"] = torch.tensor(value) + want = "layers.2: w2.weight_scale_2 of expert 3 is .*not a positive finite scale" + _load_raises(ck, want, world) + + +def test_shared_scalar_refuses_copies_that_differ(): + """The row transform asserts on its own copies too, whatever ran before it.""" + one = torch.tensor(0.5) + assert W._shared_scalar((one, one.clone(), one.clone()), None) is one + with pytest.raises(AssertionError, match="1 of 3 copies"): + W._shared_scalar((one, one.clone(), torch.tensor(0.25)), None) + + +@pytest.mark.parametrize( + "value, exact", + [ + (1.0, True), + (2.0**-126, True), + (2.0**127, True), + (2.0**-15, True), + (0.0, False), + (-(2.0**-12), False), + (3.0 * 2.0**-12, False), + (2.0**-130, False), # subnormal + (float("inf"), False), + (float("nan"), False), + ], +) +def test_ue8m0_assert(value, exact): + s = torch.full((2, 3), 2.0**-10) + s[1, 2] = value + if exact: + W._assert_ue8m0(s) + else: + with pytest.raises(AssertionError, match="1 of 6"): + W._assert_ue8m0(s) + + +def test_packed_reference_itself(): + """`_packed` on a hand-written grid: 6 K blocks with exponent bytes 120..125 in one + 128-row block -> word 0 = 120 | 121 << 8 | 122 << 16 | 123 << 24, word 1 = 124 | 125 << 8, + the same for all 128 rows; a byte of 128 or more lands in the sign bit.""" + grid = torch.exp2(torch.arange(120, 126, dtype=torch.float32) - 127).view(1, 6) + got = _packed(grid) + assert got.shape == (2, 128) + assert bool((got[0] == 120 | 121 << 8 | 122 << 16 | 123 << 24).all()) + assert bool((got[1] == 124 | 125 << 8).all()) + high = _packed(torch.tensor([[1.0, 1.0, 1.0, 2.0]])) + assert bool((high[0] == (127 | 127 << 8 | 127 << 16 | 128 << 24) - 2**32).all()) + + +@pytest.mark.parametrize("world", [4, 8]) +def test_native_dspark_manifest_shards_and_off_coverage(world): + native = { + "n_layers": 3, + "depth": 3, + "target_activation_layers": "1,18,36,54,71", + "dspark": {"markov_rank": 512}, + } + # Tensor dimensions from the checkpoint's safetensors header, independently + # of the table's shape lambdas. Values encode the sliced axis so a row / + # column or head-boundary error is visible without allocating the weights. + projections = { + "wq": (8192, 6144, 0), + "wkv_b": (12288, 512, 0), + "wo": (6144, 8192, 1), + "w1": (24576, 6144, 0), + "w3": (24576, 6144, 0), + "w2": (6144, 24576, 1), + } + shards = {key: [] for key in projections} + for rank in range(world): + core = _core(_CONFIG, world=world, rank=rank) + off = W.MODEL_WEIGHTS.manifest(core) + core.model_config.spec_config = SimpleNamespace( + _draft_model_config=native, target_layer_ids=[1, 18, 36, 54, 71] + ) + manifest = W.MODEL_WEIGHTS.manifest(core) + added = set(manifest) - set(off) + assert len(added) == 38 + keys = {manifest[name][0][0] for name in added} + assert len(keys) == 38 and all(key.startswith("mtp_dspark.") for key in keys) + assert not keys.intersection(W.MODEL_WEIGHTS.expected_unconsumed(core, dict.fromkeys(keys))) + with torch.device("meta"): + declared = W.MODEL_WEIGHTS.declare(W.MODEL_WEIGHTS.dims(core)) + assert tuple(declared["draft_target_projection"].shape) == (6144, 30720) + assert tuple(declared["draft_joint"].shape) == (1536, 7168) + assert tuple(declared["draft_joint_bias"].shape) == (1536,) + assert tuple(declared["draft_mask"].shape) == (2, 6144) + assert all(declared[name].dtype == torch.bfloat16 for name in added) + for role, (rows, cols, axis) in projections.items(): + ids = torch.arange((rows, cols)[axis]) + source = ( + ids[:, None].expand(rows, cols) if axis == 0 else ids[None, :].expand(rows, cols) + ) + key, index, transform, shard = manifest[f"l0_draft_{role}"][0] + assert index is None and transform is None + sliced = source if shard is None else shard(source) + assert tuple(sliced.shape) == tuple(declared[f"l0_draft_{role}"].shape) + shards[role].append(sliced[:, 0] if axis == 0 else sliced[0]) + core.model_config.spec_config = None + assert keys.issubset(W.MODEL_WEIGHTS.expected_unconsumed(core, dict.fromkeys(keys))) + for role, parts in shards.items(): + rows, cols, axis = projections[role] + expected = torch.arange((rows, cols)[axis]) + torch.testing.assert_close(torch.cat(parts), expected) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/__init__.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/__init__.py new file mode 100644 index 000000000000..46c7fb1a43fa --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/__init__.py @@ -0,0 +1,7 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Package marker. + +These test basenames can repeat other targets'. Without packages, pytest's +default import mode loads both under one module name and refuses the second. +""" diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/gemm/__init__.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/gemm/__init__.py new file mode 100644 index 000000000000..46c7fb1a43fa --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/gemm/__init__.py @@ -0,0 +1,7 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Package marker. + +These test basenames can repeat other targets'. Without packages, pytest's +default import mode loads both under one module name and refuses the second. +""" diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/gemm/test_modeling_v2_cute_dsl_fused_fp8_block_scale_linear.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/gemm/test_modeling_v2_cute_dsl_fused_fp8_block_scale_linear.py new file mode 100644 index 000000000000..586537088b14 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/gemm/test_modeling_v2_cute_dsl_fused_fp8_block_scale_linear.py @@ -0,0 +1,214 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Fused K128 quantization/GEMM numerical and graph replay coverage.""" + +import pytest +import torch + +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_107__tp.ops.gemm.cute_dsl_fused_fp8_block_scale_linear import ( # noqa: E501 + cute_dsl_fused_fp8_block_scale_linear as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import assert_within_ulp, certifying # noqa: E402,F401 + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="requires VR200/SM107", +) + + +def _input(m, k, seed): + generator = torch.Generator(device="cuda").manual_seed(seed) + storage = torch.randn(m, k + 16, dtype=torch.bfloat16, device="cuda", generator=generator) + view = storage[:, 8 : k + 8] + exponents = torch.randint(-20, 21, (m, k // 128, 1), device="cuda", generator=generator) + view.copy_(torch.ldexp(view.reshape(m, k // 128, 128).float(), exponents).reshape(m, k)) + view[0, :128] = 0 + return view + + +def _weights(n, k, seed): + # The fused kernel reads weights before its PDL wait, so tests create them + # before the activations instead of directly ahead of the GEMM. + weight = _input(n, k, seed).contiguous() + return torch.ops.trtllm.fp8_quantize_1x128_packed_ue8m0(weight, True) + + +@pytest.mark.parametrize("m", (1, 3, 4, 8, 12, 16)) +@pytest.mark.parametrize("k", (768, 1024, 1536, 2048, 3072, 6144)) +@pytest.mark.parametrize("seed", (11, 23, 47)) +def test_fused_matches_k128_reference(m, k, seed): + weight, scale = _weights(128, k, seed + 1) + x = _input(m, k, seed) + from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_107__tp.ops.gemm._fused_fp8_quant_gemv import ( # noqa: E501 + fused_fp8_quant_gemv, + ) + + actual = fused_fp8_quant_gemv(x, weight, scale) + expected = op.reference(x, weight, scale) + assert_within_ulp(actual, expected, element_ulp=4, rms_ulp=2) + + +@pytest.mark.parametrize("m,k", ((4, 768), (16, 1536), (4, 6144), (16, 6144))) +def test_fused_changing_input_graph(m, k): + weight, scale = _weights(128, k, 23) + x = _input(m, k, 11) + from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_107__tp.ops.gemm._fused_fp8_quant_gemv import ( # noqa: E501 + fused_fp8_quant_gemv, + ) + + for _ in range(3): + fused_fp8_quant_gemv(x, weight, scale) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + output = fused_fp8_quant_gemv(x, weight, scale) + for seed in range(101): + x.copy_(_input(m, k, seed)) + graph.replay() + expected = op.reference(x, weight, scale) + assert_within_ulp(output, expected, element_ulp=4, rms_ulp=2) + + +@pytest.mark.parametrize("m", (1, 3, 4, 8, 12, 16)) +@pytest.mark.parametrize("k", (768, 1024, 1536, 2048, 3072, 6144)) +@pytest.mark.parametrize("seed", (11, 23, 47)) +def test_fused_quantization_byte_contract(m, k, seed): + from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_107__tp.ops.gemm._fused_fp8_quant_gemv import ( # noqa: E501 + debug_quantization, + ) + + x = _input(m, k, seed) + codes, scale = debug_quantization(x) + expected_codes, physical = torch.ops.trtllm.fp8_quantize_1x128_packed_ue8m0( + x.contiguous(), True + ) + logical = ( + physical.reshape(-1, k // 128, 32, 4, 4) + .permute(0, 3, 2, 1, 4) + .reshape(-1, k // 32)[:m, ::4] + ) + assert torch.equal(codes.view(torch.uint8), expected_codes.view(torch.uint8)) + assert torch.equal(scale, logical) + + +def test_fused_quantization_finite_extremes(): + from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_107__tp.ops.gemm._fused_fp8_quant_gemv import ( # noqa: E501 + debug_quantization, + ) + + values = torch.tensor( + [ + 0.0, + 1.0e-40, + -1.0e-40, + torch.finfo(torch.bfloat16).max, + -torch.finfo(torch.bfloat16).max, + 7.0, + 7.03125, + 6.96875, + ], + device="cuda", + dtype=torch.bfloat16, + ) + x = values.repeat(4 * 768 // values.numel()).reshape(4, 768) + actual, scales = debug_quantization(x) + expected, physical = torch.ops.trtllm.fp8_quantize_1x128_packed_ue8m0(x, True) + logical = physical.reshape(-1, 6, 32, 4, 4).permute(0, 3, 2, 1, 4).reshape(-1, 24)[:4, ::4] + assert torch.equal(actual.view(torch.uint8), expected.view(torch.uint8)) + assert torch.equal(scales, logical) + + +def test_fused_opposite_scale_exponents(): + # The dequantized operands and their output are finite even though applying + # the activation scale alone to an unscaled K128 subtotal would overflow. + weight = torch.full((128, 768), 448.0, device="cuda", dtype=torch.float8_e4m3fn) + logical = torch.full((128, 24), 17, device="cuda", dtype=torch.uint8) + scale = torch.ops.trtllm.block_scale_interleave(logical) + x = torch.full((4, 768), 2.0**118, device="cuda", dtype=torch.bfloat16) + from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_107__tp.ops.gemm._fused_fp8_quant_gemv import ( # noqa: E501 + fused_fp8_quant_gemv, + ) + + actual = fused_fp8_quant_gemv(x, weight, scale) + expected = op.reference(x, weight, scale) + assert bool(torch.isfinite(actual).all()) + assert_within_ulp(actual, expected, element_ulp=4, rms_ulp=2) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[cell.why for cell in op.CELLS]) +@pytest.mark.parametrize("draw", range(3)) +def test_certified_cells(cell, draw, certifying) -> None: # noqa: F811 — pytest fixture + certifying(op) + m, n, k = (cell.spec[name] for name in ("m", "n", "k")) + seed = cell.seed_base() + 2 * draw + weight, scale = _weights(n, k, seed + 1) + x = _input(m, k, seed) + actual = op(x, weight, scale) + expected = op.reference(x, weight, scale) + assert_within_ulp(actual, expected, element_ulp=4, rms_ulp=2) + + +@pytest.mark.parametrize("m", (64, 128, 129, 256, 2048, 16384)) +def test_fused_catalog_protection(m): + x = _input(m, 768, 11).contiguous() + weight, scale = _weights(128, 768, 23) + actual = op(x, weight, scale) + expected = op.reference(x, weight, scale) + assert_within_ulp(actual, expected, element_ulp=4, rms_ulp=2) + + +@pytest.mark.parametrize("n,k", ((1024, 6144), (1536, 6144), (6144, 768), (6144, 1024))) +def test_fused_catalog_changing_input_graph(n, k): + weight, scale = _weights(n, k, 23) + x = _input(4, k, 11).contiguous() + for _ in range(3): + op(x, weight, scale) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + output = op(x, weight, scale) + for seed in range(101): + x.copy_(_input(4, k, seed)) + graph.replay() + expected = op.reference(x, weight, scale) + assert_within_ulp(output, expected, element_ulp=4, rms_ulp=2) + + +# The wq projection's per-rank width is 1024 under TP8 and 2048 under TP4. Both are tuned cells +# up to 16 rows; 32 rows exceed the fused kernel and stay on the MXFP8 kernel, as does the +# indexer's wq_b (4096 wide) at 16 rows, where the table has no cell. +@pytest.mark.parametrize( + "m,n,fused", + ((4, 1024, True), (16, 2048, True), (32, 1024, False), (16, 4096, False)), + ids=("tp8", "tp4", "m32", "untuned"), +) +def test_decode_target_dispatch(m, n, fused): + from importlib import import_module + + from torch.utils._python_dispatch import TorchDispatchMode + + modeling = import_module( + "tensorrt_llm._torch.modeling_v2.models.mistral_large4." + "mistral_large_4_nvfp4__sm_107__tp.modeling" + ) + + class Operations(TorchDispatchMode): + def __init__(self): + super().__init__() + self.seen = [] + + def __torch_dispatch__(self, func, types, args=(), kwargs=None): + self.seen.append(str(func)) + return func(*args, **(kwargs or {})) + + target = object.__new__(modeling.DecodeTarget) + weight, scale = _weights(n, 6144, 23) + x = _input(m, 6144, 11).contiguous() + with Operations() as operations: + actual = target._linear_fp8(x, weight, scale) + seen = any("trtllm.cute_dsl_fused_fp8_block_scale_linear" in name for name in operations.seen) + assert seen == fused, operations.seen + if not fused: + assert any("trtllm.cute_dsl_mxfp8_gemm_rubin" in name for name in operations.seen) + expected = op.reference(x, weight, scale) + assert_within_ulp(actual, expected, element_ulp=4, rms_ulp=2) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/gemm/test_modeling_v2_low_m_cublas_mm.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/gemm/test_modeling_v2_low_m_cublas_mm.py new file mode 100644 index 000000000000..5ae361c6538a --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp/gemm/test_modeling_v2_low_m_cublas_mm.py @@ -0,0 +1,130 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the sm_107 tp target's low_m_cublas_mm entry.""" + +import pytest +import torch + +from tensorrt_llm._torch.modeling_v2.catalog.gemm.cublas_mm import cublas_mm as op +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_107__tp.ops.gemm.low_m_cublas_mm import ( # noqa: E501 + _LOW_M_DIRECT_TACTICS, + _low_m_direct_tactic, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_107__tp.ops.gemm.low_m_cublas_mm import ( # noqa: E501 + low_m_cublas_mm as low_m_op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import ( # noqa: E402,F401 — needs the path declared above + ULP, + assert_within_ulp, + certifying, + validating, +) + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available(), reason="low_m_cublas_mm requires a CUDA device" +) + + +def _compare(out: torch.Tensor, ref: torch.Tensor) -> None: + """The cublas_mm entry's band, derived in its test: a bf16 output within 4 ulp element-wise + and 2 ulp of relative RMS, an fp32 output within 512 and 256 of fp32's ulp.""" + if out.dtype is torch.float32: + element_ulp, rms_ulp, ulp = 512.0, 256.0, ULP[torch.float32] + else: + element_ulp, rms_ulp, ulp = 4.0, 2.0, ULP[torch.bfloat16] + assert_within_ulp(out, ref, element_ulp=element_ulp, rms_ulp=rms_ulp, ulp=ulp) + + +@pytest.mark.parametrize("cell", low_m_op.CELLS, ids=[cell.why for cell in low_m_op.CELLS]) +@pytest.mark.parametrize("seed", [42, 123, 2026]) +def test_low_m_fixed_tactic_graph_and_offsets(cell, seed, certifying) -> None: # noqa: F811 + if torch.cuda.get_device_capability() != (10, 7): + pytest.skip("Fixed low-M tactics require SM107") + certifying(low_m_op) + spec = cell.spec + torch.manual_seed(seed) + m, n, k = spec["m"], spec["n"], spec["k"] + a = torch.randn(m + 1, k, device="cuda", dtype=torch.bfloat16)[1:] + b = torch.randn(n + 1, k, device="cuda", dtype=torch.bfloat16)[1:].t() + assert a.storage_offset() and b.storage_offset() + with validating(low_m_op): + _compare(low_m_op(a, b), low_m_op.reference(a, b)) + low_m_op(a, b) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + out = low_m_op(a, b) + for replay in range(100): + a.normal_() + a[-1].zero_() + graph.replay() + _compare(out, low_m_op.reference(a, b)) + assert torch.equal(out, low_m_op(a, b)) + assert not out[-1].view(torch.int16).any() + a.zero_() + a[:, :3] = 1 + b.zero_() + b[0].fill_(1) + b[1].fill_(-1) + b[2].fill_(2**-10) + graph.replay() + _compare(out, low_m_op.reference(a, b)) + + +@pytest.mark.parametrize( + "m,n", + [(m, 160) for m in (1, 2, 3, 12, 32, 64, 128, 256, 2048, 16384)] + + [(16, 512), (16, 576), (24, 576), (32, 576)], +) +def test_low_m_protection_fallback(m, n) -> None: + assert (m, n, 6144) not in _LOW_M_DIRECT_TACTICS + a = torch.randn(m, 6144, device="cuda", dtype=torch.bfloat16) + b = torch.randn(n, 6144, device="cuda", dtype=torch.bfloat16).t() + actual = low_m_op(a, b) + assert torch.equal(actual, op(a, b)) + _compare(actual, low_m_op.reference(a, b)) + + +@pytest.mark.parametrize("out_dtype", [torch.bfloat16, torch.float32]) +@pytest.mark.parametrize("has_bias", [False, True]) +def test_low_m_output_dtype_and_bias_fallback(out_dtype, has_bias) -> None: + a = torch.randn(4, 6144, device="cuda", dtype=torch.bfloat16) + n = 128 if out_dtype == torch.float32 else 160 + b = torch.randn(n, 6144, device="cuda", dtype=torch.bfloat16).t() + bias = torch.randn(n, device="cuda", dtype=out_dtype) if has_bias else None + actual = low_m_op(a, b, bias, out_dtype) + if has_bias or n == 128: + assert torch.equal(actual, op(a, b, bias, out_dtype)) + _compare(actual, low_m_op.reference(a, b, bias, out_dtype)) + + +@pytest.mark.parametrize("buffer_kind,group", [(1, None), (0, []), (0, [0, 1])]) +def test_low_m_collective_buffer_arguments_are_forwarded(monkeypatch, buffer_kind, group) -> None: + a = torch.empty(4, 6144, device="cuda", dtype=torch.bfloat16) + b = torch.empty(160, 6144, device="cuda", dtype=torch.bfloat16).t() + calls = [] + + def fallback(*args): + calls.append(args) + return a.new_empty((4, 160)) + + monkeypatch.setattr(torch.ops.trtllm, "cublas_mm", fallback) + low_m_op(a, b, output_buffer_kind=buffer_kind, group=group) + assert len(calls) == 1 + assert calls[0][4] == buffer_kind and calls[0][5] is group + + +@pytest.mark.parametrize("offset", [1, 8, 16, 64]) +def test_low_m_operand_alignment(offset) -> None: + if torch.cuda.get_device_capability() != (10, 7): + pytest.skip("Fixed low-M tactics require SM107") + a = torch.randn(4 * 6144 + offset, device="cuda", dtype=torch.bfloat16)[offset:].view(4, 6144) + b = ( + torch.randn(160 * 6144 + offset, device="cuda", dtype=torch.bfloat16)[offset:] + .view(160, 6144) + .t() + ) + selected = _low_m_direct_tactic(a, b, None, None, 0, None) + assert (selected is not None) == (offset % 16 == 0) + _compare(low_m_op(a, b), low_m_op.reference(a, b)) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/__init__.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/__init__.py new file mode 100644 index 000000000000..a5b2988e425e --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/__init__.py @@ -0,0 +1,8 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Tests of the family's operations. + +A package, so their basenames can repeat another family's own +(`test_modeling_v2_convert_req_index_to_global.py`) under pytest's default +import mode. +""" diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/activation/__init__.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/activation/__init__.py new file mode 100644 index 000000000000..52a7a9daf028 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/activation/__init__.py @@ -0,0 +1,2 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/activation/test_modeling_v2_gelu_erf.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/activation/test_modeling_v2_gelu_erf.py new file mode 100644 index 000000000000..e34b374d3439 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/activation/test_modeling_v2_gelu_erf.py @@ -0,0 +1,307 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the gelu_erf entry. + +The entry owns what is certified: `CELLS` is the list, `reference` is the +gate's other side. This file owns turning a spec into real tensors, the band, +and the measurements the entry's note states. + +The entry's identity is *which* GELU it is. The wrong answer is the tanh +GELU, which qwen4_exp's block MLPs use in the same tower as its merger. It +stays within about one bf16 ulp of the right answer at the output's scale, +so the gate here is per element, in ulp of each element's own value. That is +what lets bf16 tell the two apart. + +bf16 and fp16 have 65536 values each, so `test_every_finite_bf16_and_fp16_input` +drives the whole input domain of the two half types through the op once. The +cells then certify the shapes, layouts and launch sizes. +""" + +import math +import zlib + +import pytest +import torch + +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.activation.gelu_erf import ( + gelu_erf as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402,F401 + +assert torch.cuda.is_available(), "gelu_erf requires a CUDA device" + +MERGER_WIDTH = 4608 +TOWER_HIDDEN = 1152 +#: qwen4_exp model.visual.merger, measured from that checkpoint: +#: linear_fc1.weight std 0.010927, linear_fc1.bias std 0.002407, norm.weight +#: mean 1.174581 / std 0.140536. The real tensors give an activand at std +#: 0.875 / absmax 4.23. +FC1_WEIGHT_STD = 0.010927 +FC1_BIAS_STD = 0.002407 +NORM_GAMMA_MEAN = 1.174581 +NORM_GAMMA_STD = 0.140536 + +SQRT_2_OVER_PI = math.sqrt(2.0 / math.pi) + +#: torch.testing's untouched per-dtype defaults, restated only so the +#: discrimination tests can say how far outside that gate each variant lands. +_RTOL = {torch.bfloat16: 1.6e-2, torch.float16: 1e-3, torch.float32: 1.3e-6} +_ATOL = 1e-5 + +#: Explicit mantissa bits, for the ulp of a value in each dtype. +_MANTISSA = {torch.bfloat16: 7, torch.float16: 10, torch.float32: 23} + +#: Absolute allowance per unit |x| for the fp32 `1 + erf` term (see `_gate`). +_CANCEL = 2.0**-21 + +#: bf16 only: the most elements with |x| <= 4 that may differ from the +#: correctly rounded value (see `_gate`). +_BF16_NOT_CR_MAX = 1e-3 +_CR_REGION = 4.0 + + +def _seed(text: str) -> int: + return zlib.crc32(text.encode()) + + +def _uniform(shape, lo, hi, dtype, seed): + g = torch.Generator(device="cuda").manual_seed(seed) + x = torch.empty(shape, device="cuda", dtype=torch.float32).uniform_(lo, hi, generator=g) + return x.to(dtype) + + +def _build(spec, seed): + shape, dtype = spec["shape"], spec["dtype"] + if spec.get("dist", "uniform") == "merger": + g = torch.Generator(device="cuda").manual_seed(seed) + rows = shape[0] + gamma = NORM_GAMMA_MEAN + NORM_GAMMA_STD * torch.randn( + TOWER_HIDDEN, generator=g, device="cuda" + ) + hidden = torch.randn(4 * rows, TOWER_HIDDEN, generator=g, device="cuda") + grouped = (hidden * gamma.to(dtype).float()).reshape(rows, MERGER_WIDTH) + weight = FC1_WEIGHT_STD * torch.randn( + MERGER_WIDTH, MERGER_WIDTH, generator=g, device="cuda" + ) + bias = FC1_BIAS_STD * torch.randn(MERGER_WIDTH, generator=g, device="cuda") + return (grouped @ weight.to(dtype).float().t() + bias.to(dtype).float()).to(dtype) + layout = spec.get("layout", "contiguous") + lo, hi = spec["lo"], spec["hi"] + if layout == "column_slice": + rows, width = shape + x = _uniform((rows, 2 * width), lo, hi, dtype, seed)[:, :width] + assert not x.is_contiguous() and x.stride(-1) == 1 + return x + if layout == "transposed": + rows, width = shape + x = _uniform((width, rows), lo, hi, dtype, seed).t() + assert x.stride(-1) != 1 + return x + return _uniform(shape, lo, hi, dtype, seed) + + +def _ulp(value: torch.Tensor, dtype: torch.dtype) -> torch.Tensor: + """One ulp of `dtype` at |value| (fp64 in, fp64 out); subnormals share the + smallest normal's ulp.""" + magnitude = value.abs().clamp_min(torch.finfo(dtype).tiny) + _, exponent = torch.frexp(magnitude) + return torch.ldexp(torch.ones_like(magnitude), exponent - 1 - _MANTISSA[dtype]) + + +def _gate(out: torch.Tensor, x: torch.Tensor) -> dict: + """How far `out` is from the exact value, against the band. Does not assert. + + `default` is the worst ratio to torch's own dtype-aware band, against the + reference rounded to `x.dtype`. + + `ulp` is the worst ratio to the per-element allowance, which is one ulp of + the exact value plus |x| * 2^-21. The kernel evaluates + `0.5 * x * (1 + erf(x / sqrt(2)))` in fp32 and rounds once, so: + - the final rounding contributes half an ulp; + - the fp32 sum `1 + erf` carries an absolute error of a few 2^-24 (half an + fp32 ulp at 1, plus erf's own error at |erf| near 1), and `0.5 * x` + scales that. + Once `1 + erf` cancels (x below about -3 in fp32, -4 in bf16), that second + term is all of the result: many ulp of a tiny value, but under |x| * 2^-23 + in absolute terms. The allowance takes 2x on the rounding and 4x on the + cancellation. Measured at the worst element: 0.500 in bf16 and fp16 (the + rounding), 0.19 in fp32. + + `not_cr` is the fraction, among elements with |x| <= 4, that differ from + the correctly rounded value. It is gated in bf16 only. There the fp32 + evaluation is about 2^16 times finer than a bf16 ulp outside the + cancellation, so a misrounding needs the exact value that close to a + rounding tie. Measured: 0 of the 33026 bf16 values in [-4, 4]. fp16 has 3 + more mantissa bits, and 59 of its 34818 values in [-4, 4] misround, so the + count is reported for fp16 and not gated. The ulp gate cannot see an erf + scale of 1/1.414 for 1/sqrt(2) in bf16; it moves 3.87% of elements over + [-3, 3], which is 39x this bound. + """ + exact = op.reference(x.double()) + rounded = op.reference(x) + dtype = x.dtype + err = (out.double() - exact).abs() + allowance = _ulp(exact, dtype) + x.double().abs() * _CANCEL + default_band = _ATOL + _RTOL[dtype] * rounded.double().abs() + region = x.double().abs() <= _CR_REGION + differs = (out != rounded) & region + in_region = int(region.sum()) + worst = (err / allowance).flatten() + return dict( + ulp=worst.max().item() if worst.numel() else 0.0, + worst_x=x.flatten()[int(worst.argmax())].item() if worst.numel() else 0.0, + default=((out.double() - rounded.double()).abs() / default_band).max().item() + if out.numel() + else 0.0, + not_cr=int(differs.sum()) / in_region if in_region else 0.0, + not_cr_count=int(differs.sum()), + in_region=in_region, + ) + + +def _compare(out: torch.Tensor, x: torch.Tensor) -> dict: + """The band: torch's default against the rounded reference, the per-element + ulp allowance against the exact value, and in bf16 the correctly-rounded + count. Each is derived in `_gate`.""" + torch.testing.assert_close(out, op.reference(x)) + stats = _gate(out, x) + assert stats["ulp"] <= 1.0, stats + if x.dtype == torch.bfloat16: + assert stats["not_cr"] <= _BF16_NOT_CR_MAX, stats + return stats + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying): # noqa: F811 + certifying(op) + x = _build(cell.spec, seed=_seed(cell.why)) + before = x.clone() + out = op(x) + assert out.shape == x.shape and out.dtype == x.dtype and out.device == x.device + if x.numel(): + assert out.data_ptr() != x.data_ptr(), "the entry is out-of-place" + assert torch.equal(before, x), "the entry must not mutate its input" + if cell.spec.get("layout") == "transposed": + assert out.stride() == x.stride(), "the output keeps the input's memory format" + assert torch.equal(out, op(x.contiguous())) + if cell.spec.get("layout") == "column_slice": + assert out.is_contiguous() + assert torch.equal(out, op(x.contiguous())) + stats = _compare(out, x) + print(f"\n {cell.why}: {stats}") + + +def _all_finite(dtype: torch.dtype) -> torch.Tensor: + bits = torch.arange(-32768, 32768, dtype=torch.int32, device="cuda").to(torch.int16) + values = bits.view(dtype) + return values[torch.isfinite(values)].contiguous() + + +def test_every_finite_bf16_and_fp16_input() -> None: + """The whole input domain of each half type, once: every finite value is + inside the band, and bf16 is correctly rounded on all of [-4, 4]. Also + pins where the fp32 `1 + erf` cancellation turns the bf16 output into + -0.0, which the note states.""" + for dtype, count, in_region in ( + (torch.bfloat16, 65280, 33026), + (torch.float16, 63488, 34818), + ): + x = _all_finite(dtype) + assert x.numel() == count + out = op(x) + stats = _compare(out, x) + assert stats["in_region"] == in_region + print(f"\n exhaustive {dtype}: {stats}") + if dtype == torch.bfloat16: + assert stats["not_cr_count"] == 0, "every bf16 input in [-4, 4] rounds correctly" + + x = _all_finite(torch.bfloat16) + out = op(x) + deep = x.double() < -1.0 + negative_zero = (out == 0) & torch.signbit(out) + onset = x[deep & negative_zero].double().max().item() + exact_at_onset = op.reference(x.new_tensor([onset]).double()).item() + print(f" bf16: -0.0 from x = {onset} down; the exact value there is {exact_at_onset:.3e}") + assert -5.7 < onset < -5.4, onset + below = x.double() <= onset + assert bool(negative_zero[below].all()), "every bf16 input from the onset down returns -0.0" + tail = int(((out != op.reference(x)) & (x.double() < -_CR_REGION)).sum()) + print(f" bf16: {tail} inputs below -4 are not correctly rounded") + + +def _erf64(x): + return op.reference(x.double()) + + +def _tanh64(x): + """The *other* GELU -- `gelu_pytorch_tanh`, qwen4_exp's block-MLP activation.""" + x64 = x.double() + return 0.5 * x64 * (1.0 + torch.tanh(SQRT_2_OVER_PI * (x64 + 0.044715 * x64 * x64 * x64))) + + +def test_identity_is_the_erf_form_and_not_tanh() -> None: + """Measured in fp32, where the deviation from the exact value is far below + the gap between the two GELUs.""" + x32 = _uniform((512, MERGER_WIDTH), -4.0, 4.0, torch.float32, 24) + out = op(x32).double() + ref = _erf64(x32) + correct = (out - ref).abs().max().item() + gate = _ATOL + _RTOL[torch.float32] * ref.abs().max().item() + + # The entry's own gate refuses the tanh form in fp32. + with pytest.raises(AssertionError): + torch.testing.assert_close(out.float(), _tanh64(x32).float()) + + x64 = x32.double() + variants = { + "tanh gelu (gelu_pytorch_tanh)": _tanh64(x32), + "sigmoid approx x*sigmoid(1.702x)": x64 * torch.sigmoid(1.702 * x64), + "silu": x64 * torch.sigmoid(x64), + "relu": torch.clamp(x64, min=0.0), + } + print(f"\n fp32: gate {gate:.3e}; correct-path deviation {correct:.3e}") + for name, wrong in variants.items(): + dev = (out - wrong).abs().max().item() + print( + f" {name:36s} dev {dev:.6e} ({dev / correct:9.1f}x correct, {dev / gate:7.1f}x gate)" + ) + assert dev > 100.0 * correct, (name, dev, correct) + assert dev > 10.0 * gate, (name, dev, gate) + + # erf's scale truncated from 1/sqrt(2) to 1/1.414 -- a plausible + # transcription slip, weak by construction, so a lower multiple. + perturbed = 0.5 * x64 * (1.0 + torch.erf(x64 / 1.414)) + dev = (out - perturbed).abs().max().item() + print(f" {'erf scale 1/sqrt(2) -> 1/1.414':36s} dev {dev:.6e} ({dev / gate:7.1f}x gate)") + assert dev > 20.0 * correct, (dev, correct) + assert dev > 1.5 * gate, (dev, gate) + + +def test_extremes_and_non_finite() -> None: + """Saturation, signed-zero underflow, no overflow at finfo.max, -inf -> nan.""" + for dtype in (torch.bfloat16, torch.float16, torch.float32): + finfo = torch.finfo(dtype) + probe = torch.tensor( + [0.0, -0.0, 1e-3, -1e-3, 20.0, -20.0, 1000.0, -1000.0, finfo.max, -finfo.max], + device="cuda", + dtype=dtype, + ) + out = op(probe) + torch.testing.assert_close(out, op.reference(probe)) + assert out[4].item() == 20.0 and out[6].item() == 1000.0 + assert out[8].item() == finfo.max, "no overflow at the top of the dtype's range" + for index in (5, 7, 9): + assert out[index].item() == 0.0 and math.copysign(1.0, out[index].item()) < 0 + + nonfinite = torch.tensor( + [float("inf"), float("-inf"), float("nan")], device="cuda", dtype=dtype + ) + got = op(nonfinite) + assert got[0].item() == float("inf") + assert math.isnan(got[1].item()), "gelu_erf(-inf) is nan" + assert math.isnan(got[2].item()) + ref = _erf64(nonfinite) + assert ref[0].item() == float("inf"), "the closed form agrees: the function, not the kernel" + assert math.isnan(ref[1].item()) and math.isnan(ref[2].item()) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/activation/test_modeling_v2_tanh_softcap.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/activation/test_modeling_v2_tanh_softcap.py new file mode 100644 index 000000000000..3b30815bbab1 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/activation/test_modeling_v2_tanh_softcap.py @@ -0,0 +1,230 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the tanh_softcap entry. + +Every cell is gated twice against the entry's fp64 reference: torch's fp32 +default band, and a per-element ulp bound derived at `MAX_ULP`. The second is +the one that bites near zero, where the default atol of 1e-5 admits any +relative error at all. Cells are compared `_CHUNK_ROWS` rows at a time, so the +context-logits cells check every element without an fp64 copy of the whole. +""" + +import math +import zlib + +import pytest +import torch + +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.activation.tanh_softcap import ( + tanh_softcap as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402,F401 + +assert torch.cuda.is_available(), "tanh_softcap requires a CUDA device" + +CAP = 30.0 +VOCAB = 204800 + +#: The smallest fp32 whose cap is exactly 30.0: the fp32 tanh rounds to 1 there. +SATURATION = 270.327392578125 + +#: Per-element bound, in fp32 ulps of the entry's reference (the fp64 closed +#: form rounded once to fp32). The op computes q = fl(x * fl(1 / cap)), two +#: roundings, so q = (x / cap)(1 + e) with |e| <= 2u, u = 2^-24; t = tanhf(q), +#: within the CUDA math library's documented 2 ulp, at most 4u relative; +#: o = fl(t * cap), at most u. tanh's relative condition number 2y / sinh(2y) +#: is at most 1, so e passes through no larger. That is 7u relative, and one +#: ulp of an fp32 value v exceeds u * |v|: 7 ulp from the fp64 value, plus 0.5 +#: for rounding the reference to fp32. +MAX_ULP = 7.5 + +#: torch.testing's untouched fp32 defaults, restated only so the +#: discrimination test can say how far outside that gate each variant lands. +_RTOL, _ATOL = 1.3e-6, 1e-5 + +#: Rows compared at once: 1024 x 204800 is 0.8 GiB in fp32, 1.6 GiB in fp64. +_CHUNK_ROWS = 1024 + + +def _seed(text: str) -> int: + return zlib.crc32(text.encode()) + + +def _uniform(shape, lo, hi, dtype, seed): + g = torch.Generator(device="cuda").manual_seed(seed) + x = torch.empty(shape, device="cuda", dtype=torch.float32).uniform_(lo, hi, generator=g) + return x.to(dtype) + + +def _ulps(out: torch.Tensor, ref: torch.Tensor) -> torch.Tensor: + """|out - ref| in units of one fp32 ulp of `ref`.""" + mag = ref.abs() + ulp = torch.nextafter(mag, torch.full_like(mag, math.inf)) - mag + return (out.double() - ref.double()).abs() / ulp.double() + + +def _compare(out: torch.Tensor, ref: torch.Tensor) -> float: + """Torch's fp32 default band, then the derived `MAX_ULP` bound; returns the worst ulp.""" + assert out.dtype == ref.dtype == torch.float32, (out.dtype, ref.dtype) + assert out.shape == ref.shape, (out.shape, ref.shape) + torch.testing.assert_close(out, ref) + worst = _ulps(out, ref).max().item() + assert worst <= MAX_ULP, f"{worst:.3f} ulp from the reference exceeds {MAX_ULP}" + return worst + + +def _cell_params(): + return [ + pytest.param(c, marks=pytest.mark.large) if c.spec.get("large") else c for c in op.CELLS + ] + + +@pytest.mark.parametrize("cell", _cell_params(), ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying): # noqa: F811 + certifying(op) + spec = cell.spec + shape, cap, seed = (spec["rows"], spec["vocab"]), spec["cap"], _seed(cell.why) + lo, hi, dtype = spec["lo"], spec["hi"], spec["dtype"] + x = _uniform(shape, lo, hi, dtype, seed) + peak = "" + if spec["graph"]: + side = torch.cuda.Stream() + side.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(side): + for _ in range(2): + op(x, cap) + torch.cuda.current_stream().wait_stream(side) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + out = op(x, cap) + seed += 1 + x.copy_(_uniform(shape, lo, hi, dtype, seed)) + graph.replay() + torch.cuda.synchronize() + assert torch.equal(out, op(x, cap)), "a replay is bitwise the eager call on its logits" + else: + torch.cuda.synchronize() + held = torch.cuda.memory_allocated() + torch.cuda.reset_peak_memory_stats() + out = op(x, cap) + torch.cuda.synchronize() + extra = torch.cuda.max_memory_allocated() - held + size = x.numel() * x.element_size() + peak = f"; peak {extra / 2**30:.3f} GiB beyond the input ({extra / size:.2f}x its size)" + + assert out.shape == x.shape and out.dtype == x.dtype and out.device == x.device + assert out.data_ptr() != x.data_ptr(), "the entry is out-of-place" + assert torch.equal(x, _uniform(shape, lo, hi, dtype, seed)), "the entry must not mutate x" + + for a, b in spec.get("windows", ()): + assert torch.equal(op(x[a:b], cap), out[a:b]), ( + f"rows [{a}, {b}) alone differ from the same rows of the {shape[0]}-row call" + ) + + worst, saturated = 0.0, 0 + for a in range(0, shape[0], _CHUNK_ROWS): + xs, outs = x[a : a + _CHUNK_ROWS], out[a : a + _CHUNK_ROWS] + worst = max(worst, _compare(outs, op.reference(xs, cap))) + sat = xs.abs() >= SATURATION + assert torch.equal(outs[sat], torch.sign(xs[sat]) * cap), "saturates to exactly cap" + saturated += sat.sum().item() + if not spec["graph"]: + peak += f"; the cell's peak {torch.cuda.max_memory_allocated() / 2**30:.3f} GiB" + print(f"\n {cell.why}: worst {worst:.3f} ulp; {saturated} saturated{peak}") + + +def test_band_rejects_wrong_variants() -> None: + """Every plausible wrong soft-cap lands far outside the gate, the bf16 path included.""" + x = _uniform((16, VOCAB), -60.0, 60.0, torch.float32, 31) + out = op(x, CAP) + ref = op.reference(x, CAP) + correct = (out - ref).abs().max().item() + gate = _ATOL + _RTOL * ref.abs().max().item() + xb = x.to(torch.bfloat16) + variants = { + "bf16 logits path, each step rounded": (torch.tanh(xb / CAP) * CAP).float(), + "hard clamp to [-cap, cap]": x.clamp(-CAP, CAP), + "no cap": x.clone(), + "tanh(x) * cap": torch.tanh(x) * CAP, + "cap 30.5": torch.tanh(x / 30.5) * 30.5, + } + print(f"\n gate {gate:.3e} at the output's scale; correct path {correct:.3e}") + for name, wrong in variants.items(): + dev = (wrong - ref).abs().max().item() + print(f" {name:38s} dev {dev:.4e} ({dev / gate:10.1f}x gate)") + assert dev > 100.0 * gate, (name, dev, gate) + with pytest.raises(AssertionError): + _compare(wrong, ref) + + +def test_saturation_and_non_finite() -> None: + finfo = torch.finfo(torch.float32) + below = torch.nextafter(torch.tensor(SATURATION), torch.tensor(0.0)).item() + probe = torch.tensor( + [0.0, -0.0, SATURATION, -SATURATION, below, -below, finfo.max, -finfo.max], + device="cuda", + ) + out = op(probe, CAP) + assert out[0].item() == 0.0 and not torch.signbit(out[0]) + assert out[1].item() == 0.0 and torch.signbit(out[1]) + assert out[2].item() == CAP and out[3].item() == -CAP + assert out[4].item() < CAP and out[5].item() > -CAP, "SATURATION is the first fp32 at the cap" + assert out[6].item() == CAP and out[7].item() == -CAP + _compare(out, op.reference(probe, CAP)) + + nonfinite = torch.tensor([math.inf, -math.inf, math.nan], device="cuda") + got = op(nonfinite, CAP) + assert got[0].item() == CAP and got[1].item() == -CAP + assert math.isnan(got[2].item()) + + # A subnormal quotient: relative precision goes, the absolute error stays tiny. + tiny = torch.tensor([1e-40, -1e-40, finfo.tiny], device="cuda") + got, ref = op(tiny, CAP), op.reference(tiny, CAP) + assert torch.equal(torch.sign(got), torch.sign(ref)) + assert (got.double() - ref.double()).abs().max().item() < 1e-43 + + +def test_python_float_cap_is_a_reciprocal_multiply() -> None: + """The class docstring's rounding sequence, and why `cap` stays a Python float.""" + x = _uniform((16, VOCAB), -400.0, 400.0, torch.float32, 32) + inv = torch.tensor(1.0 / CAP, dtype=torch.float32).item() + assert torch.equal(x / CAP, x * inv) + true_div = x / torch.tensor(CAP, device="cuda") + assert not torch.equal(x / CAP, true_div) + assert _ulps(x / CAP, true_div).max().item() <= 1.0 + + +def test_padded_rows_never_reach_live_rows() -> None: + """A padded decode capture runs dummy rows beside the live ones.""" + x = _uniform((16, VOCAB), -400.0, 400.0, torch.float32, 33) + live = x[:3].clone() + x[3:] = math.nan + x[3:, ::2] = math.inf + assert torch.equal(op(x, CAP)[:3], op(live, CAP)) + + +@pytest.mark.parametrize( + "cap", [0.0, -0.0, math.inf, -math.inf, math.nan], ids=["0", "-0", "inf", "-inf", "nan"] +) +def test_guard_refuses_degenerate_cap(cap) -> None: + x = _uniform((2, VOCAB), -60.0, 60.0, torch.float32, 34) + x[0, 0] = 0.0 + # Unguarded, the op accepts the cap and answers with zeros and nan. + got = op(x, cap) + if cap == 0.0: + assert math.isnan(got[0, 0].item()) + finite = got[~torch.isnan(got)] + assert finite.numel() == got.numel() - 1 and (finite == 0.0).all() + else: + assert torch.isnan(got).all() + with validating(op): + with pytest.raises(ValueError, match="cap must be finite and non-zero"): + op(x, cap) + + +def test_guard_admits_a_negative_cap_which_computes_its_magnitude() -> None: + x = _uniform((16, VOCAB), -400.0, 400.0, torch.float32, 35) + with validating(op): + assert torch.equal(op(x, -CAP), op(x, CAP)) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/__init__.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/__init__.py new file mode 100644 index 000000000000..52a7a9daf028 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/__init__.py @@ -0,0 +1,2 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/_ds_mla_pool.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/_ds_mla_pool.py new file mode 100644 index 000000000000..eb885244ed77 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/_ds_mla_pool.py @@ -0,0 +1,15 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Synthetic fp8_ds_mla pools for the entry tests: rows as the writer stores them.""" + +import torch + + +def ds_mla_rows(n: int, g: torch.Generator, nope: bool = False) -> torch.Tensor: + """`n` uint8 [656] rows: e4m3 codes in (224, 448] magnitude as the writer leaves them, + power-of-two tile scales 2^-13..2^0, bf16 rope (zero for a NoPE layer).""" + codes = (torch.rand(n, 512, generator=g) * 2 - 1) * 448 + codes = codes.to(torch.float8_e4m3fn).view(torch.uint8) + scale = torch.exp2(torch.randint(-13, 1, (n, 4), generator=g).float()) + rope = torch.zeros(n, 64) if nope else torch.randn(n, 64, generator=g) * 3 + return torch.cat([codes, scale.view(torch.uint8), rope.to(torch.bfloat16).view(torch.uint8)], 1) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_convert_req_index_to_global.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_convert_req_index_to_global.py new file mode 100644 index 000000000000..37afd73bd5a0 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_convert_req_index_to_global.py @@ -0,0 +1,785 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the convert_req_index_to_global entry. + +The cells are the Mistral Large 4 FP8 target's at its 9216 envelope, the +Mistral Large 4 NVFP4 targets' at their 32768 envelope (the tep8, tp8 and dep8 ones +on the fp8_ds_mla pool with page ids from a 147456-block pool), the NVFP4 tep4, +tp4 and dep4 deployments' at max_seq_len 1048576 with chunked prefill, and those +three targets' and the tep8 deployment's on the 63-owner fp8_ds_mla pool at that +envelope (dep4's under attention DP, tep8's from the 147456-block pool). Every +value is integer address arithmetic, so each output is gated bitwise against +the entry's reference. +""" + +import math +import zlib + +import pytest +import torch + +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.convert_req_index_to_global import ( # noqa: E501 + _ML4_1M_DEP4_DS, + _ML4_1M_DS, + _ML4_1M_TEP8_DS, + _ML4_32K_DS, + _ML4_32K_DS_8GPU, + _ML4_DS_STEP, + _ML4_OWNERS, + _ML4_STEP, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.convert_req_index_to_global import ( # noqa: E501 + convert_req_index_to_global as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402,F401 — needs the path declared above + +assert torch.cuda.is_available(), "convert_req_index_to_global requires a CUDA device" + +#: Fills output memory the op is expected to leave unwritten. +_SENTINEL = 0x7EADBEEF + +#: The dep4 envelope's max_seq_len: the boot captures each decode graph on a longest dummy of +#: one key less, and the engine's table is ceil((max_seq_len + 1) / 32) columns, padded to 4. +_DEP4_MAX_SEQ_LEN = 1048576 + + +def _compare(out: torch.Tensor, ref: torch.Tensor) -> None: + """Exact: this is integer address arithmetic, and an off-by-one row is a + different token's KV.""" + assert out.dtype == ref.dtype == torch.int32, (out.dtype, ref.dtype) + assert out.shape == ref.shape, (out.shape, ref.shape) + assert torch.equal(out, ref), ( + f"{int((out != ref).sum())} of {out.numel()} pool rows differ from the reference" + ) + + +def _layer_ids(spec) -> tuple: + if spec.get("layer_ids") == "all": + return tuple(range(spec["pool_layers"])) + if spec.get("layer_ids") == "owner_rows_0_62": + # Every owner row of the 63-owner fp8_ds_mla page. + assert spec["pool_layers"] == len(_ML4_OWNERS) == 63, spec["pool_layers"] + return tuple(range(63)) + return tuple(spec["layer_ids"]) + + +def _table(spec, g: torch.Generator) -> torch.Tensor: + """The block table, and in it each request's pages. + + It is the cache manager's: `table_rows` x `table_width`, pages drawn from a + `pool_blocks` pool (its top ids with `far_end`), and every entry no request + uses holds a stale page id, so reading the wrong row or column shows. + + A step of several 1M contexts names more pages than the pool has (no rank's + pool holds them at once; the cell bounds what the table can name, not what + the scheduler admits). Its pages then run through fresh permutations of the + whole pool, so the top ids are in use and ids repeat across requests. + + With `first_seq`, request `s` holds table row `first_seq + s`, as a mixed + step's generation requests follow its context requests. With `pad`, the + last `pad` requests are the CUDA-graph padding dummy, one request repeated: + each keeps a row of its own, and every one of those rows holds the first's + pages. + """ + bs = spec["block_size"] + need = [math.ceil(n / bs) for n in spec["kv_lens"]] + first = spec.get("first_seq", 0) + rows = spec["table_rows"] + assert first + len(need) <= rows, "every request needs a row of the table" + width = spec["table_width"] + pool = spec["pool_blocks"] + used = [min(n, width) for n in need] + if sum(used) > pool: + cycles = math.ceil(sum(used) / pool) + pages = torch.cat([torch.randperm(pool, generator=g) for _ in range(cycles)]) + elif spec.get("far_end"): + pages = pool - 1 - torch.randperm(sum(used), generator=g) + else: + pages = torch.randperm(pool, generator=g)[: sum(used)] + table = torch.randint(0, pool, (rows, width), generator=g) + start = 0 + for s, n in enumerate(used): + table[first + s, :n] = pages[start : start + n] + start += n + dummies = spec.get("pad", 0) + if dummies: + assert spec["kv_lens"][-dummies:] == [1] * dummies, "the padding dummy holds one key" + dummy = first + len(need) - dummies + table[dummy + 1 : dummy + dummies] = table[dummy] + return table.to(torch.int32) + + +def _positions( + ends: torch.Tensor, + topk: int, + order: str, + seed: int, + last: bool = False, + starts: torch.Tensor | None = None, +) -> torch.Tensor: + """Row `i`: `min(ends[i], topk)` distinct positions of `[0, ends[i])`, then -1. + + The shape the indexer's top-k hands over: in no order, or in the ascending + order of the engine's dense causal index when the indexer is skipped + (`order="causal"`). With `last`, each row's newest position `ends[i] - 1` + is among its selections (first), so a context that fills the block table + reads its last column. `order="chunk"` draws by `_chunk`, which also reads + each row's cached prefix, `starts`. + + A random draw is one key per position of every row; past 2**30 keys (a + 32768-row step of 1M contexts would take 2**35) the rows are drawn by + `_strata` instead. + """ + if order == "chunk": + return _chunk(ends, starts, topk, seed, last) + width = int(ends.max()) + if ends.numel() * width > 2**30: + assert order == "random", "only the random order is drawn by strata" + return _strata(ends, topk, seed, last) + col = torch.arange(width, device="cuda").unsqueeze(0) + if order == "causal": + keys = col.expand(ends.numel(), width).float() + else: + g = torch.Generator(device="cuda").manual_seed(seed) + keys = torch.rand(ends.numel(), width, device="cuda", generator=g) + if last: + keys.scatter_(1, (ends - 1).unsqueeze(1), -1.0) + keys = keys.masked_fill(col >= ends.unsqueeze(1), float("inf")) + k = min(topk, width) + pick = keys.topk(k, dim=1, largest=False).indices + out = torch.full((ends.numel(), topk), -1, dtype=torch.int32, device="cuda") + out[:, :k] = torch.where(col[:, :k] < ends.unsqueeze(1), pick, -1).to(torch.int32) + return out + + +def _strata(ends: torch.Tensor, topk: int, seed: int, last: bool) -> torch.Tensor: + """`_positions` in random order without a key per position. + + A row with `ends[i] <= topk` holds every position of `[0, ends[i])`, then + -1, as `_positions` gives it. A longer row cuts `[0, ends[i])` into `topk` + strata `[s * ends[i] // topk, (s + 1) * ends[i] // topk)`, each at least one + position wide, and takes one position at random from each: `topk` distinct + positions over the whole context. Columns are shuffled, the -1 padding + last; with `last`, the last stratum gives `ends[i] - 1`, placed first. + """ + g = torch.Generator(device="cuda").manual_seed(seed) + end = ends.long().unsqueeze(1) + k = torch.arange(topk, device="cuda").unsqueeze(0) + lo, hi = k * end // topk, (k + 1) * end // topk + u = torch.rand(ends.numel(), topk, device="cuda", generator=g) + pick = torch.minimum(lo + (u * (hi - lo)).long(), hi - 1) + wide = end > topk + if last: + pick[:, -1] = end[:, 0] - 1 + valid = k < end + pos = torch.where(valid, torch.where(wide, pick, k), -1) + keys = torch.rand(ends.numel(), topk, device="cuda", generator=g) + keys = keys.masked_fill(~valid, float("inf")) + if last: + keys.scatter_(1, torch.where(wide, topk - 1, end - 1), -1.0) + return pos.gather(1, keys.argsort(dim=1)).to(torch.int32) + + +def _chunk(ends: torch.Tensor, starts: torch.Tensor, topk: int, seed: int, last: bool): + """`_positions` for `order="chunk"`: a chunk row's selections on both sides of + its request's cached prefix. + + Row `i` sees `[0, ends[i])`, and `[starts[i], ends[i])` is its chunk's own rows + up to it (`starts[i]` is the request's cached prefix). It takes + `min(ends[i] - starts[i], max(topk // 2, topk - starts[i]))` positions from the + chunk and `min(starts[i], topk - that)` from `[0, starts[i])`. Each part is cut + into that many strata, with one position drawn from each, which gives + `min(ends[i], topk)` distinct positions. Columns are shuffled and the -1 + padding comes last; with `last`, the chunk part's last stratum gives + `ends[i] - 1`, placed first. + """ + g = torch.Generator(device="cuda").manual_seed(seed) + end, start = ends.long().unsqueeze(1), starts.long().unsqueeze(1) + new = torch.minimum(end - start, torch.clamp(topk - start, min=topk // 2)) + old = torch.minimum(start, topk - new) + k = torch.arange(topk, device="cuda").unsqueeze(0) + in_new = k < new + # Stratum `s` of `count` over `[base, base + span)`; a part is never narrower + # than its count, so every stratum is at least one position wide. + s = torch.where(in_new, k, k - new) + count = torch.where(in_new, new, old).clamp(min=1) + base = torch.where(in_new, start, 0) + span = torch.where(in_new, end - start, start) + lo, hi = base + s * span // count, base + (s + 1) * span // count + u = torch.rand(ends.numel(), topk, device="cuda", generator=g) + pick = torch.minimum(lo + (u * (hi - lo)).long(), hi - 1) + newest = k == new - 1 + if last: + pick = torch.where(newest, end - 1, pick) + valid = k < new + old + pos = torch.where(valid, pick, -1) + keys = torch.rand(ends.numel(), topk, device="cuda", generator=g) + keys = keys.masked_fill(~valid, float("inf")) + if last: + keys = keys.masked_fill(newest, -1.0) + return pos.gather(1, keys.argsort(dim=1)).to(torch.int32) + + +def _build(spec, seed: int): + """`(req_id, block_table, token_indices, block_size, topk, stride_factor)`. + + Request `s` contributes `seq_rows[s]` query rows; row `j` of it sees + positions `[0, kv_lens[s] - seq_rows[s] + j + 1)`, and `req_id` names the + request's row of the table (`first_seq + s`), as the engine's per-token + request index does. + """ + assert len(spec["seq_rows"]) == len(spec["kv_lens"]), "one seq_rows entry per request" + first = spec.get("first_seq", 0) + req, ends, starts = [], [], [] + for s, (r_count, kv) in enumerate(zip(spec["seq_rows"], spec["kv_lens"])): + assert 1 <= r_count <= kv, f"request {s}: {r_count} rows over {kv} keys" + req += [first + s] * r_count + ends += [kv - r_count + j + 1 for j in range(r_count)] + starts += [kv - r_count] * r_count + g = torch.Generator().manual_seed(seed) + bs, topk = spec["block_size"], spec["topk"] + return ( + torch.tensor(req, dtype=torch.int32, device="cuda"), + _table(spec, g).cuda(), + _positions( + torch.tensor(ends, device="cuda"), + topk, + spec.get("order", "random"), + seed, + spec.get("last_position", False), + starts=torch.tensor(starts, device="cuda"), + ), + bs, + topk, + spec["pool_layers"] * bs, + ) + + +def _check_padding(out: torch.Tensor, tok: torch.Tensor) -> None: + # The padding has to survive as -1, not as a row computed from -1. + assert bool((out[tok < 0] == -1).all()) + + +def _check_reach(spec, outs) -> None: + """A cell drawn from a cache manager's pool names rows past 2**24 + 1, where an + index carried through fp32 stops being exact; the note's fp32 row rests on it.""" + if "pool_blocks" in spec: + top = max(int(o.max()) for o in outs) + assert top > 2**24 + 1, f"the cell's highest pool row is {top}, not past 2**24 + 1" + + +def _check_newest(spec, tok: torch.Tensor, out: torch.Tensor) -> None: + """With `last_position`, the longest context's newest position is selected and + resolves to a pool row: the cell reads the last table column that context reaches, + which a `why` naming the last column rests on.""" + if spec.get("last_position"): + newest = tok == max(spec["kv_lens"]) - 1 + assert bool(newest.any()), "the longest context's newest position is not selected" + assert bool((out[newest] >= 0).all()), "the newest position resolved to -1" + + +def _check_resolved(spec, tok: torch.Tensor, out: torch.Tensor) -> None: + """In a cell on a cache manager's pool every selected position has a page in the + table, so each one resolves to a pool row. A cell whose pages went negative would + read -1 in the kernel and in the reference alike, and so pass on nothing; this + check fails it.""" + if "pool_blocks" in spec: + assert max(spec["kv_lens"]) <= spec["table_width"] * spec["block_size"] + assert not bool(((tok >= 0) & (out < 0)).any()), "a selected position resolved to -1" + + +def _check_dummy_rows(spec, out: torch.Tensor) -> None: + """The padding dummy's rows (the last `pad`, one query row each) are one + request repeated, so each maps to the same pool rows as the first of them.""" + dummies = spec.get("pad", 0) + if dummies > 1: + rows = out[-dummies:] + assert torch.equal(rows, rows[:1].expand_as(rows)), "padding rows differ" + + +def _check_boot(spec) -> None: + """A `boot_capture` cell captures what the dep4 boot captures for the long + sequence family: the warm-up batch's longest dummy, max_seq_len - 1 keys, + first and one-key fillers, each a request of its own, on the engine's table + width; one of its replays is attention DP's single live row at max_seq_len + behind the padding dummy.""" + if not spec.get("boot_capture"): + return + b, bs = len(spec["kv_lens"]), spec["block_size"] + assert spec["kv_lens"] == [_DEP4_MAX_SEQ_LEN - 1] + [1] * (b - 1), spec["kv_lens"] + assert spec["seq_rows"] == [1] * b and not spec.get("pad"), "fillers are requests of their own" + assert spec["table_width"] == -(-math.ceil((_DEP4_MAX_SEQ_LEN + 1) / bs) // 4) * 4 + replays = zip(spec["replay_kv_lens"], spec["replay_pad"]) + assert any(pad == b - 1 and kv[0] == _DEP4_MAX_SEQ_LEN for kv, pad in replays) + + +def _drive_eager(spec, seed: int) -> None: + args = _build(spec, seed) + snap = [a.clone() for a in args[:3]] + for layer in _layer_ids(spec): + out = op(*args, layer) + _compare(out, op.reference(*args, layer)) + _check_padding(out, args[2]) + _check_resolved(spec, args[2], out) + _check_dummy_rows(spec, out) + _check_reach(spec, [out]) + _check_newest(spec, args[2], out) + for a, s in zip(args[:3], snap): + assert torch.equal(a, s), "an input tensor was mutated" + if spec.get("prefix_sweep"): + # The grid is one block row per query row, so a shorter call is a + # different launch; each must be bitwise the leading rows of the long one. + # Compared as int64 pairs with `any`: at 32768 rows a count over int32 + # elements cost about as much as the 32768 calls themselves. + req, table, tok, bs, topk, stride = args + full = out.view(torch.int64) + bad = torch.zeros((), dtype=torch.bool, device="cuda") + for m in range(1, req.shape[0] + 1): + short = op(req[:m], table, tok[:m], bs, topk, stride, layer) + bad |= torch.ne(short.view(torch.int64), full[:m]).any() + if bool(bad): + first = next( + ( + m + for m in range(1, req.shape[0] + 1) + if not torch.equal( + op(req[:m], table, tok[:m], bs, topk, stride, layer), out[:m] + ) + ), + None, + ) + raise AssertionError( + f"a shorter call differs from the full call's leading rows (re-run: first at {first})" + ) + + +def _drive_captured(spec, seed: int) -> None: + """Capture the step's calls, then replay them on new batch data. + + The replay data reverses the batch's contexts and draws new pages and + positions; it is written into the captured inputs, as the engine writes a + step's metadata into its graph buffers. Each replayed output must be + bitwise an eager call on that data, and the reference. With + `replay_kv_lens` the graph is replayed once per entry, on those contexts, + each drawn anew, and `replay_pad` names how many of each replay's trailing + requests are the padding dummy. A cell with `replay_kv_lens` reaches past + 2**24 + 1 over the capture data and every replay together, any other in its + replay alone. + """ + _check_boot(spec) + layers = _layer_ids(spec) + req, table, tok, bs, topk, stride = _build(spec, seed) + peaks, replay_peaks = [], [] + for layer in layers: # eager on the capture data, validated in full + out = op(req, table, tok, bs, topk, stride, layer) + _compare(out, op.reference(req, table, tok, bs, topk, stride, layer)) + _check_newest(spec, tok, out) + _check_resolved(spec, tok, out) + _check_dummy_rows(spec, out) + peaks.append(out.max()) + side = torch.cuda.Stream() + side.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(side): + op(req, table, tok, bs, topk, stride, layers[0]) + torch.cuda.current_stream().wait_stream(side) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + outs = [op(req, table, tok, bs, topk, stride, layer) for layer in layers] + + replays = spec.get("replay_kv_lens", (spec["kv_lens"][::-1],)) + pads = spec.get("replay_pad", (0,) * len(replays)) + assert len(pads) == len(replays), "one replay_pad entry per replay" + for k, (kv_lens, pad) in enumerate(zip(replays, pads)): + again = dict(spec, seq_rows=spec["seq_rows"][::-1], kv_lens=list(kv_lens), pad=pad) + new_req, new_table, new_tok, *_ = _build(again, seed + 1 + k) + assert new_tok.shape == tok.shape and not torch.equal(new_tok, tok) + req.copy_(new_req) + table.copy_(new_table) + tok.copy_(new_tok) + graph.replay() + for layer, out in zip(layers, outs): + eager = op(req, table, tok, bs, topk, stride, layer) + assert torch.equal(out, eager), f"layer_id {layer}: graph replay differs from eager" + _compare(out, op.reference(req, table, tok, bs, topk, stride, layer)) + _check_padding(out, tok) + _check_newest(again, tok, out) + _check_resolved(again, tok, out) + _check_dummy_rows(again, out) + peaks.append(out.max()) + replay_peaks.append(out.max()) + _check_reach(spec, peaks if "replay_kv_lens" in spec else replay_peaks) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 + certifying(op) + seed = zlib.crc32(cell.why.encode()) % 2**31 + if cell.spec.get("graph"): + _drive_captured(cell.spec, seed) + else: + _drive_eager(cell.spec, seed) + + +def _small(): + req = torch.tensor([0, 1, 1, 0], dtype=torch.int32, device="cuda") + table = torch.tensor([[3, 1], [0, 2]], dtype=torch.int32, device="cuda") + tok = torch.tensor( + [[0, 33, -1, 5], [1, 2, 63, -1], [40, 0, -1, -1], [31, 32, 7, 8]], + dtype=torch.int32, + device="cuda", + ) + return req, table, tok + + +def test_hand_worked_rows() -> None: + """Four rows worked out by hand, independent of `reference`.""" + req, table, tok = _small() + out = op(req, table, tok, 32, 4, 64, 1) + # row 0, req 0: p=0 -> page 3: 3*64 + 0 + 32 = 224; p=33 -> page 1: 64 + 1 + 32 = 97 + assert out[0].tolist() == [224, 97, -1, 229] + # row 1, req 1: pages [0, 2]; p=63 -> page 2, offset 31: 128 + 31 + 32 = 191 + assert out[1].tolist() == [33, 34, 191, -1] + _compare(out, op.reference(req, table, tok, 32, 4, 64, 1)) + + +def test_strata_draw_the_selection_shape() -> None: + """`_strata`, which draws the 1M steps' selections, hands over what `_positions` + does: per row `min(end, topk)` distinct positions below `end`, the -1 padding + after them, and with `last` the newest position first.""" + topk = 2048 + ends = torch.tensor( + [1, 2, 33, 2047, 2048, 2049, 4096, 4097, 1015809, 1048575, 1048576], device="cuda" + ) + for last in (False, True): + tok = _strata(ends, topk, 7, last).long() + count = torch.clamp(ends, max=topk) + col = torch.arange(topk, device="cuda").unsqueeze(0) + valid = tok >= 0 + assert torch.equal(valid, col < count.unsqueeze(1)), "padding is not the row's tail" + assert bool((tok < ends.unsqueeze(1)).all()), "a position at or past the row's end" + ordered = torch.where(valid, tok, ends.unsqueeze(1) + col).sort(dim=1).values + assert bool((ordered[:, 1:] > ordered[:, :-1]).all()), "a position drawn twice" + if last: + assert torch.equal(tok[:, 0], ends - 1), "the newest position is not first" + # Rows wider than top-k reach both ends of their context. + tok = _strata(torch.full((64,), 1048576, device="cuda"), topk, 8, False) + assert int(tok.min()) < 512 and int(tok.max()) >= 1048576 - 512 + + +def test_chunk_draw_splits_the_prefix_and_the_chunk() -> None: + """`_chunk`, which draws the `order="chunk"` selections, gives what `_positions` + gives: per row `min(end, topk)` distinct positions below `end`, the -1 padding + after them, and with `last` the newest position first. It splits them: at most + half of top-k (more when the cached prefix is shorter) from the row's own chunk + `[start, end)`, and the rest from the cached prefix `[0, start)`.""" + topk = 2048 + starts = [0, 0, 0, 1, 1000, 1024, 1500, 2048, 4096, 1015808, 1015808, 1015808, 1032192] + rows = [1, 2048, 3000, 1, 5, 1024, 1500, 1, 4096, 1, 1024, 32768, 16384] + starts = torch.tensor(starts, device="cuda") + ends = starts + torch.tensor(rows, device="cuda") + for last in (False, True): + tok = _chunk(ends, starts, topk, 9, last).long() + count = torch.clamp(ends, max=topk) + col = torch.arange(topk, device="cuda").unsqueeze(0) + valid = tok >= 0 + assert torch.equal(valid, col < count.unsqueeze(1)), "padding is not the row's tail" + assert bool((tok < ends.unsqueeze(1)).all()), "a position at or past the row's end" + ordered = torch.where(valid, tok, ends.unsqueeze(1) + col).sort(dim=1).values + assert bool((ordered[:, 1:] > ordered[:, :-1]).all()), "a position drawn twice" + own = (tok >= starts.unsqueeze(1)).sum(dim=1) + share = torch.minimum(ends - starts, torch.clamp(topk - starts, min=topk // 2)) + assert torch.equal(own, share), "the chunk's share of the selections" + if last: + assert torch.equal(tok[:, 0], ends - 1), "the newest position is not first" + # The last row of a 1M prompt's last chunk reaches both ends of the cached + # prefix (1024 strata of 992 positions) and of its chunk (1024 of 32). + row = tok[11] + prefix, chunk = row[row < 1015808], row[row >= 1015808] + assert int(prefix.min()) < 992 and int(prefix.max()) >= 1015808 - 992 + assert int(chunk.min()) < 1015808 + 32 and int(chunk.max()) == 1048575 + + +def test_resolved_check_sees_pages_gone_negative() -> None: + """The control for `_check_resolved`. Far-end pages drawn as `pool - 1 - randperm` + for more pages than the pool holds go negative, and the kernel and the reference + then agree on -1, so the bitwise gate passes. The check refuses such a table and + passes the same call on the pool's own pages.""" + spec = dict(block_size=32, topk=4, table_rows=1, table_width=4, pool_blocks=8, kv_lens=[128]) + req = torch.zeros(1, dtype=torch.int32, device="cuda") + tok = torch.tensor([[0, 33, 64, 127]], dtype=torch.int32, device="cuda") + good = torch.tensor([[7, 6, 5, 4]], dtype=torch.int32, device="cuda") + _check_resolved(spec, tok, op(req, good, tok, 32, 4, 2304, 0)) + bad = good - 6 + out = op(req, bad, tok, 32, 4, 2304, 0) + _compare(out, op.reference(req, bad, tok, 32, 4, 2304, 0)) + assert out[0].tolist() == [2304, 1, -1, -1] + with pytest.raises(AssertionError, match="resolved to -1"): + _check_resolved(spec, tok, out) + + +def test_resolved_check_sees_pages_gone_negative_at_the_owner_stride() -> None: + """The control for `_check_resolved`. Far-end pages drawn as `pool - 1 - + randperm` for more pages than the pool holds go negative; the kernel and the + reference then agree on -1, so the bitwise gate passes. The check refuses + that table and passes the same call on the pool's own pages.""" + spec = dict(block_size=32, topk=4, table_rows=1, table_width=4, pool_blocks=8, kv_lens=[128]) + req = torch.zeros(1, dtype=torch.int32, device="cuda") + tok = torch.tensor([[0, 33, 64, 127]], dtype=torch.int32, device="cuda") + good = torch.tensor([[7, 6, 5, 4]], dtype=torch.int32, device="cuda") + _check_resolved(spec, tok, op(req, good, tok, 32, 4, 2016, 0)) + bad = good - 6 + out = op(req, bad, tok, 32, 4, 2016, 0) + _compare(out, op.reference(req, bad, tok, 32, 4, 2016, 0)) + assert out[0].tolist() == [2016, 1, -1, -1] + with pytest.raises(AssertionError, match="resolved to -1"): + _check_resolved(spec, tok, out) + + +def test_padding_dummy_rows_share_their_pool_rows() -> None: + """With `pad` the last requests are the one CUDA-graph padding dummy: rows of + the table of their own (distinct `req_id`) holding the same page, so the op + maps every one of them to the same pool rows, which `_check_dummy_rows` + asserts. The same batch drawn without `pad` gives each one-key request a page + of its own, which the check refuses: it can see the difference.""" + spec = dict(_ML4_1M_DEP4_DS, seq_rows=[1] * 8, kv_lens=[1048576, 9217] + [1] * 6, pad=6) + req, table, tok, bs, topk, stride = _build(spec, 11) + assert req.tolist() == list(range(8)), "each padding row keeps a request index of its own" + assert bool((table[3:8] == table[2]).all()), "the padding rows hold the dummy's pages" + out = op(req, table, tok, bs, topk, stride, 35) + _compare(out, op.reference(req, table, tok, bs, topk, stride, 35)) + _check_dummy_rows(spec, out) + assert bool((out[2:, 1:] == -1).all()) and int(out[2, 0]) == int(table[2, 0]) * 2016 + 35 * 32 + own = dict(spec, pad=0) + req, table, tok, *_ = _build(own, 11) + out = op(req, table, tok, bs, topk, stride, 35) + with pytest.raises(AssertionError, match="padding rows differ"): + _check_dummy_rows(spec, out) + + +def test_dep4_fp8_ds_mla_cells_capture_every_graph_batch() -> None: + """The dep4 engine captures decode graphs at 1/2/4/8/16 padded; every one is a + boot-capture cell and a live-batch cell at stride 2016, and the ones that pad + replay behind the padding dummy.""" + ds = [c.spec for c in op.CELLS if c.why.startswith("dep4 1M fp8_ds_mla")] + graphs = [s for s in ds if s.get("graph")] + for boot in (True, False): + batches = sorted(len(s["kv_lens"]) for s in graphs if bool(s.get("boot_capture")) == boot) + assert batches == [1, 2, 4, 8, 16], (boot, batches) + for spec in graphs: + assert ( + spec["layer_ids"] == _ML4_DS_STEP and spec["pool_layers"] * spec["block_size"] == 2016 + ) + live = {len(s["kv_lens"]) - p for s in graphs for p in s.get("replay_pad", ()) if p} + assert {1, 3, 5, 9, 13} <= live, sorted(live) + + +def test_owner_pool_far_end_cells_fit_their_pool() -> None: + """A far-end cell hands its requests the pool's top pages only while its pages + fit the pool; past it `_table` cycles through the whole pool instead. Every + far-end cell on the 63-owner pool fits, capture and replays alike, so its + pages are the top ones its `why` names.""" + far = [c for c in op.CELLS if c.spec.get("far_end") and c.spec.get("pool_layers") == 63] + assert len(far) == 21, [c.why for c in far] + for cell in far: + spec = cell.spec + for kv_lens in (spec["kv_lens"], *spec.get("replay_kv_lens", ())): + pages = sum(math.ceil(n / spec["block_size"]) for n in kv_lens) + assert pages <= spec["pool_blocks"], (cell.why, pages) + + +def _replayed_into_sentinel(*args) -> torch.Tensor: + """One unguarded call, captured, its output filled with `_SENTINEL` and + replayed: what the call leaves unwritten keeps the sentinel. The output is + the graph's own, so this owns every byte it inspects.""" + op(*args) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + out = op(*args) + out.fill_(_SENTINEL) + graph.replay() + torch.cuda.synchronize() + return out + + +def test_guard_refuses_a_column_count_mismatch() -> None: + """Narrower leaves each row's trailing columns unwritten, shown below. + Wider writes each row's surplus over the next row and the last row's past + the output: refused, never driven unguarded.""" + req, table, tok = _small() + ref = op.reference(req, table, tok, 32, 4, 64, 1) + out = _replayed_into_sentinel(req, table, tok, 32, 2, 64, 1) + assert torch.equal(out[:, :2], ref[:, :2]) + assert bool((out[:, 2:] == _SENTINEL).all()), "the op wrote past num_topk_tokens" + for bad in (2, 8): + with pytest.raises(AssertionError, match="num_topk_tokens must equal"): + with validating(op): + op(req, table, tok, 32, bad, 64, 0) + + +def test_guard_refuses_a_req_id_length_mismatch() -> None: + """The kernel's row count is req_id's, not token_indices'. Shorter leaves + the trailing output rows unwritten, shown below; longer reads past + token_indices and writes past the output: refused, never driven unguarded.""" + req, table, tok = _small() + ref = op.reference(req, table, tok, 32, 4, 64, 1) + out = _replayed_into_sentinel(req[:2].contiguous(), table, tok, 32, 4, 64, 1) + assert torch.equal(out[:2], ref[:2]) + assert bool((out[2:] == _SENTINEL).all()), "the op wrote rows past req_id" + for bad in (req[:2].contiguous(), torch.cat([req, req])): + with pytest.raises(AssertionError, match="exactly one entry per row"): + with validating(op): + op(bad, table, tok, 32, 4, 64, 0) + + +def test_guard_refuses_a_req_id_outside_the_table() -> None: + """`block_table[req_id[i]]` is read unchecked. Both calls read rows of + `parent`, which the test owns: past a sliced table they are the parent's + next row, and -1 is the row before the slice.""" + parent = torch.tensor([[1, 2], [3, 4], [7, 8]], dtype=torch.int32, device="cuda") + tok = torch.tensor([[0, 33]], dtype=torch.int32, device="cuda") + past = op(torch.tensor([2], dtype=torch.int32, device="cuda"), parent[:2], tok, 32, 2, 2304, 0) + assert past.tolist() == [[7 * 2304, 8 * 2304 + 1]] + before = op( + torch.tensor([-1], dtype=torch.int32, device="cuda"), parent[1:], tok, 32, 2, 2304, 0 + ) + assert before.tolist() == [[1 * 2304, 2 * 2304 + 1]] + for req, table in ((2, parent[:2]), (-1, parent[1:])): + with pytest.raises(AssertionError, match="must be rows of block_table"): + with validating(op): + op( + torch.tensor([req], dtype=torch.int32, device="cuda"), + table, + tok, + 32, + 2, + 2304, + 0, + ) + + +def test_guard_refuses_a_layer_outside_the_page() -> None: + """At stride 2304 a page holds layers 0..71. Layer 72 lands on the next + page's layer 0, and -1 on the previous page's layer 71: pages this request + was never given.""" + req = torch.zeros(1, dtype=torch.int32, device="cuda") + table = torch.tensor([[5, 9]], dtype=torch.int32, device="cuda") + tok = torch.tensor([[0, 1, 31, 32]], dtype=torch.int32, device="cuda") + beyond = op(req, table, tok, 32, 4, 2304, 72) + assert beyond.tolist() == [[6 * 2304, 6 * 2304 + 1, 6 * 2304 + 31, 10 * 2304]] + before = op(req, table, tok, 32, 4, 2304, -1) + assert before.tolist() == [ + [4 * 2304 + 71 * 32, 4 * 2304 + 71 * 32 + 1, 5 * 2304 - 1, 8 * 2304 + 71 * 32] + ] + with validating(op): + for layer in (0, 71): + _compare( + op(req, table, tok, 32, 4, 2304, layer), + op.reference(req, table, tok, 32, 4, 2304, layer), + ) + for layer in (72, -1): + with pytest.raises(AssertionError, match="outside a page of 72 layers"): + with validating(op): + op(req, table, tok, 32, 4, 2304, layer) + + +def test_guard_refuses_an_int32_overflow() -> None: + """The pool row is int32 on the device. At stride 2304, page 932067's rows + start 1280 below 2**31: layer 39 ends exactly at 2**31 - 1, and layer 40 + starts at 2**31, which the kernel wraps to -2**31 without raising.""" + req = torch.zeros(1, dtype=torch.int32, device="cuda") + table = torch.tensor([[932067]], dtype=torch.int32, device="cuda") + tok = torch.arange(32, dtype=torch.int32, device="cuda").view(1, 32) + offsets = torch.arange(32, dtype=torch.int64, device="cuda") + with validating(op): + edge = op(req, table, tok, 32, 32, 2304, 39) + assert torch.equal(edge.long()[0], 932067 * 2304 + 39 * 32 + offsets) + assert int(edge.max()) == 2**31 - 1 + wrapped = op(req, table, tok, 32, 32, 2304, 40) + assert torch.equal(wrapped.long()[0], -(2**31) + offsets), "the op no longer wraps" + for layer, block in ((40, 932067), (0, 932068)): + with pytest.raises(AssertionError, match="does not fit int32"): + with validating(op): + op(req, torch.full_like(table, block), tok, 32, 32, 2304, layer) + small_req, small_table, small_tok = _small() + with pytest.raises(AssertionError, match="does not fit int32"): + with validating(op): + op(small_req, small_table, small_tok, 32, 4, 2**30, 0) + + +def test_guard_at_64_token_pages() -> None: + """The fp8_ds_mla pool at 64-token pages: stride 63 x 64 = 4032, a page of 63 owner + rows. Page 532610's rows start 128 below 2**31: owner row 1 ends exactly at 2**31 - 1, + and owner row 2 starts at 2**31, which the kernel wraps to -2**31 without raising. Row 63 + is the next page's row 0.""" + req = torch.zeros(1, dtype=torch.int32, device="cuda") + table = torch.tensor([[532610]], dtype=torch.int32, device="cuda") + tok = torch.arange(64, dtype=torch.int32, device="cuda").view(1, 64) + offsets = torch.arange(64, dtype=torch.int64, device="cuda") + with validating(op): + edge = op(req, table, tok, 64, 64, 4032, 1) + last = op(req, torch.full_like(table, 532609), tok, 64, 64, 4032, 62) + assert torch.equal(edge.long()[0], 532610 * 4032 + 64 + offsets) + assert int(edge.max()) == 2**31 - 1 + assert torch.equal(last.long()[0], 532609 * 4032 + 62 * 64 + offsets) + wrapped = op(req, table, tok, 64, 64, 4032, 2) + assert torch.equal(wrapped.long()[0], -(2**31) + offsets), "the op no longer wraps" + for layer, block in ((2, 532610), (0, 532611)): + with pytest.raises(AssertionError, match="does not fit int32"): + with validating(op): + op(req, torch.full_like(table, block), tok, 64, 64, 4032, layer) + with pytest.raises(AssertionError, match="outside a page of 63 layers"): + with validating(op): + op(req, table, tok, 64, 64, 4032, 63) + + +def test_guard_refuses_an_int32_overflow_at_the_owner_stride() -> None: + """At the 63-owner fp8_ds_mla pool's stride 2016, page 1065220's rows start 128 + below 2**31: owner row 3 ends exactly at 2**31 - 1, and row 4 starts at 2**31, + which the kernel wraps to -2**31 without raising.""" + stride = len(_ML4_OWNERS) * 32 + req = torch.zeros(1, dtype=torch.int32, device="cuda") + table = torch.tensor([[1065220]], dtype=torch.int32, device="cuda") + tok = torch.arange(32, dtype=torch.int32, device="cuda").view(1, 32) + offsets = torch.arange(32, dtype=torch.int64, device="cuda") + with validating(op): + edge = op(req, table, tok, 32, 32, stride, 3) + assert torch.equal(edge.long()[0], 1065220 * 2016 + 3 * 32 + offsets) + assert int(edge.max()) == 2**31 - 1 + wrapped = op(req, table, tok, 32, 32, stride, 4) + assert torch.equal(wrapped.long()[0], -(2**31) + offsets), "the op no longer wraps" + for layer, block in ((4, 1065220), (0, 1065221)): + with pytest.raises(AssertionError, match="does not fit int32"): + with validating(op): + op(req, torch.full_like(table, block), tok, 32, 32, stride, layer) + # The last owner row of the dep4, tp4, tep4 and tep8 boots' pools (64633, 69011, + # 72757 and 114326 blocks) and of the cells' 107917- and 147456-block bounds are far + # inside. + for blocks, top in ( + (64633, 130300127), + (69011, 139126175), + (72757, 146678111), + (107917, 217560671), + (114326, 230481215), + (147456, 297271295), + ): + with validating(op): + out = op(req, torch.full_like(table, blocks - 1), tok, 32, 32, stride, 62) + assert int(out.max()) == (blocks - 1) * 2016 + 62 * 32 + 31 == top + + +def test_owner_row_map() -> None: + """The fp8_ds_mla cells' `layer_ids`: each layer's source's index among the 63 KV + owners. The closed form the entry writes is the owner-index form, every owner row + 0..62 is some layer's, and each sharer 39, 43, ..., 71 reads owner row 35.""" + assert _ML4_DS_STEP == tuple(_ML4_OWNERS.index(_ML4_STEP[i]) for i in range(72)) + assert len(_ML4_OWNERS) == _ML4_1M_DS["pool_layers"] == _ML4_32K_DS["pool_layers"] == 63 + assert _ML4_1M_DEP4_DS["pool_layers"] == _ML4_1M_TEP8_DS["pool_layers"] == 63 + assert _ML4_32K_DS_8GPU["pool_layers"] == 63 + assert sorted(set(_ML4_DS_STEP)) == list(range(63)) + sharers = [i for i in range(72) if _ML4_STEP[i] != i] + assert sharers == list(range(39, 72, 4)) and {_ML4_DS_STEP[i] for i in sharers} == {35} + assert _ML4_1M_DS["pool_layers"] * _ML4_1M_DS["block_size"] == 2016 + assert _ML4_1M_DEP4_DS["pool_layers"] * _ML4_1M_DEP4_DS["block_size"] == 2016 + assert _ML4_1M_TEP8_DS["pool_layers"] * _ML4_1M_TEP8_DS["block_size"] == 2016 diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_cute_dsl_ds_mla_sparse_decode.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_cute_dsl_ds_mla_sparse_decode.py new file mode 100644 index 000000000000..5eb39eff6145 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_cute_dsl_ds_mla_sparse_decode.py @@ -0,0 +1,641 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the cute_dsl_ds_mla_sparse_decode entry. + +Against `reference`: the kernel computes in bf16 (query, dequantized keys, P) with fp32 +accumulation; the reference is fp32 throughout. Band: relative L2 per (row, head) under 1e-2 +and the whole call under 4e-3 -- bf16's 2^-8 relative step, softened by the 512-wide sums; +flash_mla_sparse_decode's band, derived the same way, since the arithmetic is FlashMLA's. + +Against flash_mla_sparse_decode, the kernel the targets ran before, on identical inputs (its +query padded to 64 heads): bit-identical wherever the two split a row's selections alike -- +FlashMLA's own scheduler output is read back to show the partition is the same -- and, at the +split this entry dispatches, within the same band of each other. + +The "1M" cells and `test_bitwise_equal_to_flash_mla_at_1m_where_it_splits_alike` run the 1M +config's pools on the device (`_pool_1m`): every byte 0xFF (NaN as e4m3, as a tile's fp32 scale +and as bf16 rope) but one owner row of the pages 16 decode sequences hold, keys up to 1048576, +those pages at the pool's top so the slots read pass 2**31 bytes; every output row is first +checked finite (a key read from anywhere else turns its row NaN), then gated as above. +""" + +import math +import zlib +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.kernels import ( + _ds_mla_sparse_decode_kernel as kernel, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_ds_mla_sparse_decode import ( # noqa: E501 + GLOBAL_SPLITS, + SPLITS, + CompiledDsMlaSparseDecode, + build, + splits_for, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_ds_mla_sparse_decode import ( # noqa: E501 + cute_dsl_ds_mla_sparse_decode as op, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.flash_mla_sparse_decode import ( # noqa: E501 + flash_mla_sparse_decode as flash_op, +) +from tensorrt_llm.flash_mla import get_mla_metadata + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402,F401 — needs the path declared above + +from ._ds_mla_pool import ds_mla_rows # noqa: E402 + +assert torch.cuda.is_available(), "cute_dsl_ds_mla_sparse_decode requires a CUDA device" + +SCALE = 128**-0.5 +BLOCKS = 2048 // 64 # FlashMLA's 64-token blocks per row +_COMPILED = {} + + +def _compiled(heads): + dev = torch.cuda.current_device() + if (dev, heads) not in _COMPILED: + _COMPILED[dev, heads] = build(dev, num_heads=heads) + return _COMPILED[dev, heads] + + +def _build(spec, seed): + g = torch.Generator().manual_seed(seed) + rows, page, owners, pages = spec["rows"], spec["page_size"], spec["owners"], spec["pages"] + blocks = math.ceil(pages / owners) + 1 + kv = ds_mla_rows(blocks * owners * page, g, nope=spec.get("nope", False)) + kv = kv.view(blocks * owners, page, 1, 656) + row = 1 + kv_len = spec["kv_len"] + per_seq = math.ceil(kv_len / page) + idx = torch.full((rows, spec["topk"]), -1, dtype=torch.int32) + for r in range(rows): + blk = torch.randint(0, blocks, (per_seq,), generator=g) + pos = torch.randperm(kv_len, generator=g)[: min(kv_len, spec["topk"])] + slots = (blk[pos // page] * owners + row) * page + pos % page + if spec.get("holes"): + slots[torch.rand(slots.shape, generator=g) < 0.2] = -1 + idx[r, : slots.numel()] = slots.int() + heads = spec["heads"] + q = torch.zeros(rows, spec.get("h_q", heads), 576) + q[:, :heads] = torch.randn(rows, heads, 576, generator=g) + if spec.get("nope"): + q[..., 512:] = 0 + out = torch.full((rows, heads, 512), float("nan"), dtype=torch.bfloat16) + return q.to(torch.bfloat16).cuda(), kv.cuda(), idx.cuda(), out.cuda() + + +def _gate(out, ref): + o, r = out.float(), ref.float() + per = (o - r).norm(dim=-1) / r.norm(dim=-1).clamp_min(1e-30) + whole = float((o - r).norm() / r.norm()) + assert float(per.max()) < 1e-2 and whole < 4e-3, (float(per.max()), whole) + + +# ---- the 1M config's pools, on the device + +#: Pool rows drawn per batch while writing the sequences' pages. +_DRAW_ROWS = 1 << 18 +#: The step scratch a `q_offset` cell's views come from. +_STEP_ROWS = 32768 + + +def _lens_1m(kv_len: int) -> list[int]: + """The 16 decode sequences' key counts: `kv_len`, one 31 keys short, its 1/8, 1/16 and 1/32 + plus one, rows across the sub-chunk sizes, around top-k 2048 and down to one key. At 1048576 + they hold 2364359 keys on 73898 pages, distinct on every 1M pool.""" + spread = (kv_len, kv_len - 31, kv_len // 8 + 1, kv_len // 16 + 1, kv_len // 32 + 1, 16385, + 9217, 4097, 2049, 2048, 2047, 1025, 700, 257, 33, 1) # fmt: skip + return [max(1, min(kv_len, n)) for n in spread] + + +def _draw_rows(n: int, g: torch.Generator, nope: bool) -> torch.Tensor: + """`ds_mla_rows` drawn on g's device: e4m3 codes, power-of-two tile scales, bf16 rope.""" + dev = g.device + codes = ((torch.rand(n, 512, generator=g, device=dev) * 2 - 1) * 448).to(torch.float8_e4m3fn) + scale = torch.exp2(torch.randint(-13, 1, (n, 4), generator=g, device=dev).float()) + rope = torch.zeros(n, 64, device=dev) if nope else torch.randn(n, 64, generator=g, device=dev) + rope = (rope * 3).to(torch.bfloat16) + return torch.cat([codes.view(torch.uint8), scale.view(torch.uint8), rope.view(torch.uint8)], 1) + + +def _pool_1m(spec, g) -> SimpleNamespace: + """The pool `[blocks * owners, page, 1, 656]`, every byte 0xFF but the owner row (63's last, + or 35 on a NoPE layer) of the 16 sequences' pages: with `far` the pool's top pages, + distinct, the first sequence's last page on the top block; else drawn over the pool.""" + dev = g.device + blocks, owners, page = spec["blocks"], spec["owners"], spec["page_size"] + lens = _lens_1m(spec["kv_len"]) + counts = [-(-n // page) for n in lens] + need = sum(counts) + row = 35 if spec.get("nope") else owners - 1 + pool = torch.full((blocks, owners, page, 656), 0xFF, dtype=torch.uint8, device=dev) + if spec.get("far"): + assert need <= blocks, (need, blocks) + pages = blocks - need + torch.randperm(need, generator=g, device=dev) + top, last = int((pages == blocks - 1).nonzero()[0, 0]), counts[0] - 1 + pages[[top, last]] = pages[[last, top]] + else: + pages = torch.randint(0, blocks, (need,), generator=g, device=dev) + step = _DRAW_ROWS // page + for lo in range(0, need, step): + part = pages[lo : lo + step] + pool[part, row] = _draw_rows(part.numel() * page, g, spec.get("nope", False)).view( + -1, page, 656 + ) + first = torch.tensor([0, *counts[:-1]], device=dev).cumsum(0) + last_slot = (int(pages[counts[0] - 1]) * owners + row) * page + (lens[0] - 1) % page + return SimpleNamespace( + kv=pool.view(-1, page, 1, 656), pages=pages, first=first, lens=lens, row=row, + last_slot=last_slot, + ) # fmt: skip + + +def _selections_1m(spec, pool: SimpleNamespace, rows: int, g) -> torch.Tensor: + """int32 `[rows, topk]` slots of decode rows, row r one of sequence r % 16 at its last + position p: below topk keys every key 0..p then -1; else topk distinct keys, one in each of + topk equal strata of [0, p], p itself the last. Each row's valid keys in a random order, -1 + last, each key's slot through its sequence's pages.""" + dev = g.device + topk, page, owners = spec["topk"], spec["page_size"], spec["owners"] + req = torch.arange(rows, device=dev) % 16 + pos = torch.tensor(pool.lens, device=dev)[req] - 1 + n_keys = pos[:, None] + 1 + col = torch.arange(topk + 1, device=dev)[None] + edge = col * n_keys // topk + draw = ( + edge[:, :-1] + + (torch.rand(rows, topk, generator=g, device=dev) * (edge[:, 1:] - edge[:, :-1])).long() + ) + draw[:, -1] = pos + short = torch.where(col[:, :-1] < n_keys, col[:, :-1], -1) + sel = torch.where(n_keys > topk, draw, short) + sel = sel.gather(1, (torch.rand(sel.shape, generator=g, device=dev) + (sel < 0)).argsort(1)) + key = sel.clamp_min(0) + slot = (pool.pages[pool.first[req][:, None] + key // page] * owners + pool.row) * page + slot = torch.where(sel >= 0, slot + key % page, -1) + assert int(slot.max()) < 2**31 + return slot.to(torch.int32) + + +def _queries_1m(rows: int, spec, g, scratch: int = 0) -> torch.Tensor: + """bf16 `[rows, h_q, 576]`, the first `heads` heads drawn and the rest zero (rope zero on a + NoPE layer); with `scratch`, the tail view of a scratch that many rows long.""" + total = max(rows, scratch) + q = torch.zeros(total, spec["h_q"], 576, dtype=torch.bfloat16, device=g.device) + q[:, : spec["heads"]] = torch.randn(total, spec["heads"], 576, generator=g, device=g.device) + if spec.get("nope"): + q[..., 512:] = 0 + return q[total - rows :] + + +def _check_1m(q, kv, idx, out) -> None: + bad = ~torch.isfinite(out.float()).flatten(1).all(1) + assert not bool(bad.any()), f"{int(bad.sum())} of {out.shape[0]} rows read an unheld byte" + _gate(out, op.reference(q, kv, idx, SCALE, out)) + + +def _run_1m(spec, seed) -> None: + """A "1M" cell: per batch, the decode rows' slots over the device pool, `layers` calls into + a NaN-filled `out` (tail views with `q_offset`, the rows ahead left alone), every row finite + and in the band; with `captured`, the batch captured, replayed on new queries and slots, and + the replay bitwise an eager call.""" + g = torch.Generator(device="cuda").manual_seed(seed) + pool = _pool_1m(spec, g) + print( + f"\n pool of {spec['blocks']} blocks ({pool.kv.numel() / 1e9:.1f} GB), the first " + f"sequence's last key at slot {pool.last_slot} (byte {pool.last_slot * 656:.4g})" + ) + compiled = _compiled(spec["heads"]) + scratch = _STEP_ROWS if "q_offset" in spec else 0 + for b in spec.get("graph_batches", (spec["rows"],)): + idx = _selections_1m(spec, pool, b, g) + if spec.get("far"): + assert int(idx.max()) == pool.last_slot, (int(idx.max()), pool.last_slot) + assert int(idx.max()) * 656 >= 2**31 + q = _queries_1m(b, spec, g, scratch) + out_buf = torch.full((max(b, scratch), spec["heads"], 512), -1, dtype=torch.int16) + out_buf = out_buf.cuda().view(torch.bfloat16) + out = out_buf[out_buf.shape[0] - b :] + if scratch: + assert out_buf.shape[0] - b == spec["q_offset"] + for _ in range(spec["layers"]): + op(q, pool.kv, idx, SCALE, out, compiled=compiled) + _check_1m(q, pool.kv, idx, out) + if scratch: + ahead = out_buf[: spec["q_offset"]].view(torch.int16) + assert bool((ahead == -1).all()), "the call wrote ahead of its view" + if not spec.get("captured"): + continue + side = torch.cuda.Stream() + side.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(side): + op(q, pool.kv, idx, SCALE, out, compiled=compiled) + torch.cuda.current_stream().wait_stream(side) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + for _ in range(spec["layers"]): + op(q, pool.kv, idx, SCALE, out, compiled=compiled) + q.copy_(_queries_1m(b, spec, g)) + idx.copy_(_selections_1m(spec, pool, b, g)) + out.view(torch.int16).fill_(-1) + graph.replay() + torch.cuda.synchronize() + _check_1m(q, pool.kv, idx, out) + eager = op(q, pool.kv, idx, SCALE, torch.empty_like(out), compiled=compiled) + assert torch.equal(eager.view(torch.int16), out.view(torch.int16)), "replay != eager" + del graph + + +@pytest.mark.parametrize( + "cell", + [pytest.param(c, marks=[pytest.mark.large] if c.spec.get("large") else []) for c in op.CELLS], + ids=lambda c: c.why[:60], +) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — fixture by name + certifying(op) + spec = cell.spec + seed = zlib.crc32(cell.why.encode()) + if "blocks" in spec: + try: + _run_1m(spec, seed) + finally: + torch.cuda.empty_cache() + return + q, kv, idx, out = _build(spec, seed) + compiled = _compiled(spec["heads"]) + for _ in range(spec["layers"]): + op(q, kv, idx, SCALE, out, compiled=compiled) + _gate(out, op.reference(q, kv, idx, SCALE, out)) + if spec.get("captured"): + s = torch.cuda.Stream() + s.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(s): + op(q, kv, idx, SCALE, out, compiled=compiled) + torch.cuda.current_stream().wait_stream(s) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + for _ in range(spec["layers"]): + op(q, kv, idx, SCALE, out, compiled=compiled) + q2, kv2, idx2, _ = _build(spec, seed + 1) + q.copy_(q2), kv.copy_(kv2), idx.copy_(idx2) + out.fill_(float("nan")) + graph.replay() + torch.cuda.synchronize() + _gate(out, op.reference(q, kv, idx, SCALE, out)) + + +def _flash_mla(q, kv, idx, heads): + """flash_mla_sparse_decode on the same inputs, the query padded to 64 heads; its output + and the per-row partition its scheduler chose, as (first block, end block) ranges.""" + rows = q.shape[0] + q64 = torch.zeros(rows, 64, 576, dtype=q.dtype, device=q.device) + q64[:, :heads] = q[:, :heads] + out = torch.empty(rows, heads, 512, dtype=torch.bfloat16, device=q.device) + meta = get_mla_metadata()[0] + flash_op(q64, kv, idx, meta, SCALE, out) + parts = {r: [] for r in range(rows)} + for b_req, e_req, b_blk, e_blk, *_ in meta.tile_scheduler_metadata.cpu().tolist(): + for r in range(b_req, min(e_req, rows - 1) + 1): + lo = b_blk if r == b_req else 0 + hi = e_blk if r == e_req else BLOCKS + if hi > lo: + parts[r].append((lo, hi)) + return out, {r: sorted(v) for r, v in parts.items()} + + +def _uniform(ns): + return [(k * BLOCKS // ns, (k + 1) * BLOCKS // ns) for k in range(ns)] + + +_ALIGNED = dict(topk=2048, kv_len=9000, page_size=32, owners=63, pages=512, layers=1) + + +@pytest.mark.parametrize( + "rows,heads,ns,extra", + [ + (16, 16, 8, {}), + (32, 16, 4, {}), + (64, 16, 2, {}), + (128, 16, 1, {}), + (16, 8, 8, {}), + (32, 8, 4, {}), + (64, 8, 2, {}), + (128, 8, 1, {}), + (64, 16, 2, dict(holes=True)), + (32, 8, 4, dict(holes=True, kv_len=1500)), + (16, 16, 8, dict(nope=True)), + (128, 16, 1, dict(h_q=64)), + ], +) +def test_bitwise_equal_to_flash_mla_where_it_splits_alike(rows, heads, ns, extra) -> None: + """FlashMLA's scheduler gives a 2048-selection row 32 blocks + 5 of overhead and every + partition ceil(37 rows / 152) + 5: at 16, 32, 64 and 128 rows that cuts each row into 8, + 4, 2 and 1 equal parts. Run at that split, this kernel must reproduce FlashMLA's output + bit for bit: same conversions, same accumulators, same summation orders.""" + if torch.cuda.get_device_properties(0).multi_processor_count != 152: + pytest.skip("the partitions coincide at these batches on 152 SMs") + spec = dict(_ALIGNED, rows=rows, heads=heads, **extra) + q, kv, idx, out = _build(spec, rows * 31 + heads) + ref_out, parts = _flash_mla(q, kv, idx, heads) + assert all(p == _uniform(ns) for p in parts.values()), parts[0] + kernel.launch(_compiled(heads).fns[ns], q, kv, idx, out, SCALE) + torch.cuda.synchronize() + diff = int((out.view(torch.int16) != ref_out.view(torch.int16)).sum()) + assert diff == 0, f"{diff} of {out.numel()} outputs differ from FlashMLA's" + + +@pytest.mark.parametrize( + "rows,heads,ns,h_q", + [ + (16, 16, 8, 16), + (32, 16, 4, 16), + (64, 16, 2, 16), + (128, 16, 1, 16), + (16, 8, 8, 8), + (32, 8, 4, 8), + (64, 8, 2, 8), + (128, 8, 1, 8), + (16, 16, 8, 64), + ], +) +def test_bitwise_equal_to_flash_mla_at_1m_where_it_splits_alike(rows, heads, ns, h_q) -> None: + """At the 1M config's shapes -- the boot pool (74183 blocks, slots past 2**31 bytes, the + pool's top slot read), keys up to 1048576 down to one, -1 tails -- FlashMLA's scheduler + still cuts every row of a 16 / 32 / 64 / 128-row batch into 8 / 4 / 2 / 1 equal parts (read + back from its metadata), and this kernel run at that split reproduces its output bit for + bit, at 16 and 8 heads and reading 16 of a 64-head query.""" + if torch.cuda.get_device_properties(0).multi_processor_count != 152: + pytest.skip("the partitions coincide at these batches on 152 SMs") + spec = dict(op.CELLS[0].spec, **dict( + topk=2048, kv_len=1048576, page_size=32, owners=63, blocks=74183, far=True, + heads=heads, h_q=h_q, + )) # fmt: skip + g = torch.Generator(device="cuda").manual_seed(rows * 131 + heads + h_q) + try: + pool = _pool_1m(spec, g) + idx = _selections_1m(spec, pool, rows, g) + assert int(idx.max()) == pool.last_slot and pool.last_slot * 656 >= 2**31 + q = _queries_1m(rows, spec, g) + ref_out, parts = _flash_mla(q, pool.kv, idx, heads) + assert all(p == _uniform(ns) for p in parts.values()), parts[0] + out = torch.full((rows, heads, 512), float("nan"), dtype=torch.bfloat16, device="cuda") + kernel.launch(_compiled(heads).fns[ns], q, pool.kv, idx, out, SCALE) + torch.cuda.synchronize() + _check_1m(q, pool.kv, idx, out) + diff = int((out.view(torch.int16) != ref_out.view(torch.int16)).sum()) + assert diff == 0, f"{diff} of {out.numel()} outputs differ from FlashMLA's" + finally: + torch.cuda.empty_cache() + + +@pytest.mark.parametrize("heads", [16, 8]) +@pytest.mark.parametrize("rows", [1, 4, 16, 24, 64, 200]) +def test_matches_flash_mla_at_the_dispatched_split(rows, heads) -> None: + """At the split this entry dispatches the partition can differ from FlashMLA's (32 + one-block splits at 1-4 rows, mixed ones at 200): the same arithmetic over a different + split of the selections. Both stay in the reference band, and in it of each other.""" + spec = dict(_ALIGNED, rows=rows, heads=heads) + q, kv, idx, out = _build(spec, rows * 7 + heads) + compiled = _compiled(heads) + op(q, kv, idx, SCALE, out, compiled=compiled) + ref_out, _ = _flash_mla(q, kv, idx, heads) + assert splits_for(rows, compiled) in (*SPLITS, *GLOBAL_SPLITS) + ref = op.reference(q, kv, idx, SCALE, out) + _gate(out, ref) + _gate(ref_out, ref) + _gate(out, ref_out) + + +def test_is_valid_refuses_padded_heads_and_strided_queries() -> None: + spec = dict(_ALIGNED, rows=2, heads=16) + q, kv, idx, out = _build(spec, 1) + compiled = _compiled(16) + op.is_valid(q, kv, idx, SCALE, out, compiled=compiled) + q64 = torch.zeros(2, 64, 576, dtype=q.dtype, device=q.device) + out64 = torch.empty(2, 64, 512, dtype=out.dtype, device=out.device) + with pytest.raises(ValueError, match="heads"): + op.is_valid(q64, kv, idx, SCALE, out64, compiled=compiled) + with pytest.raises(ValueError, match="contiguous"): + op.is_valid(q64[:, :16], kv, idx, SCALE, out, compiled=compiled) + + +def _small_poisoned_global_inputs( + rows: int, heads: int, nope: bool, seed: int +) -> tuple[ + torch.Tensor, + torch.Tensor, + torch.Tensor, + torch.Tensor, + torch.Tensor, + list[torch.Tensor], + torch.Generator, +]: + """Two legal page owners; every other owner's FP8/scale/BF16 bytes are NaN.""" + gen = torch.Generator(device="cuda").manual_seed(seed) + blocks, owners, page = 67, 63, 32 + pool = torch.full((blocks, owners, page, 656), 255, dtype=torch.uint8, device="cuda") + legal = [] + for owner in (35 if nope else 62, 49): + slots = ( + (torch.arange(blocks, device="cuda")[:, None] * owners + owner) * page + + torch.arange(page, device="cuda")[None] + ).flatten() + pool.view(-1, 656)[slots] = _draw_rows(slots.numel(), gen, nope) + legal.append(slots) + q_parent = torch.randn(rows + 3, heads, 576, device="cuda", generator=gen).bfloat16() + if nope: + q_parent[..., 512:] = 0 + out_parent = torch.full( + (rows + 3, heads, 512), float("nan"), device="cuda", dtype=torch.bfloat16 + ) + indices = torch.full((rows, 2048), -1, device="cuda", dtype=torch.int32) + return ( + q_parent[3:], + pool.view(-1, page, 1, 656), + indices, + out_parent[3:], + out_parent, + legal, + gen, + ) + + +@pytest.mark.parametrize("heads", [8, 16]) +@pytest.mark.parametrize("rows", [1, 2, 3, 4, 5, 6, 7, 8, 16]) +@pytest.mark.parametrize("nope", [False, True]) +def test_global_split_boundaries_and_dynamic_graph(heads: int, rows: int, nope: bool) -> None: + """Global partials preserve owner, hole, offset and dynamic graph output contracts.""" + q, kv, idx, out, parent, legal, gen = _small_poisoned_global_inputs(rows, heads, nope, 7241) + fns = kernel._compile_global_variant(heads, 16, torch.cuda.current_device()) + workspace = torch.full((rows, 16, heads, 516), float("nan"), device="cuda", dtype=torch.float32) + counts = (0, 1, 31, 32, 33, 2047, 2048) + for start in (0, 4): + idx.fill_(-1) + for row in range(rows): + count = counts[(start + row) % len(counts)] + columns = torch.randperm(2048, generator=gen, device="cuda")[:count] + slots = legal[row % 2] + picks = torch.randperm(slots.numel() - 1, generator=gen, device="cuda")[:count] + if count: + picks[-1] = slots.numel() - 1 + idx[row, columns] = slots[picks].int() + kernel._launch_global(fns, q, kv, idx, out, SCALE, workspace) + _gate(out, op.reference(q, kv, idx, SCALE, out)) + assert bool(torch.isfinite(out).all()) + assert bool(torch.isnan(parent[:3]).all()) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + kernel._launch_global(fns, q, kv, idx, out, SCALE, workspace) + for replay in range(100): + q.copy_(torch.randn(q.shape, generator=gen, device="cuda", dtype=q.dtype)) + if nope: + q[..., 512:] = 0 + idx.copy_(idx.roll(1, dims=-1)) + slots = legal[replay % 2][:32] + kv.view(-1, 656)[slots] = _draw_rows(slots.numel(), gen, nope) + out.fill_(float("nan")) + workspace.fill_(float("nan")) + graph.replay() + _gate(out, op.reference(q, kv, idx, SCALE, out)) + assert bool(torch.isfinite(out).all()) + assert bool(torch.isnan(parent[:3]).all()) + + +@pytest.mark.parametrize("heads", [8, 16]) +@pytest.mark.parametrize("use_wrapper", [False, True]) +def test_global_split_concurrent_graph_workspaces(heads: int, use_wrapper: bool) -> None: + """A shared compiled variant runs independent stream/graph workspaces without cross-talk.""" + rows = 4 + fns = kernel._compile_global_variant(heads, 16, torch.cuda.current_device()) + streams = [torch.cuda.Stream(), torch.cuda.Stream()] + graphs = [torch.cuda.CUDAGraph(), torch.cuda.CUDAGraph()] + inputs = [_small_poisoned_global_inputs(rows, heads, False, seed) for seed in (42, 314)] + workspaces = [ + torch.empty(rows, 16, heads, 516, device="cuda", dtype=torch.float32) for _ in streams + ] + for data in inputs: + q, kv, idx, out, _, legal, _ = data + idx.copy_(legal[0][:2048].int()[None].expand(rows, -1)) + for stream, graph, data, scratch in zip(streams, graphs, inputs, workspaces): + q, kv, idx, out, *_ = data + stream.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(stream): + if use_wrapper: + op(q, kv, idx, SCALE, out, compiled=_compiled(heads)) + else: + kernel._launch_global(fns, q, kv, idx, out, SCALE, scratch) + with torch.cuda.graph(graph, stream=stream): + if use_wrapper: + op(q, kv, idx, SCALE, out, compiled=_compiled(heads)) + else: + kernel._launch_global(fns, q, kv, idx, out, SCALE, scratch) + for replay in range(100): + for stream, graph, data, scratch in zip(streams, graphs, inputs, workspaces): + q, _, idx, out, _, _, gen = data + with torch.cuda.stream(stream): + q.copy_(torch.randn(q.shape, generator=gen, device="cuda", dtype=q.dtype)) + idx.copy_(idx.roll(1, dims=-1)) + out.fill_(float("nan")) + scratch.fill_(float("nan")) + graph.replay() + torch.cuda.synchronize() + for q, kv, idx, out, parent, *_ in inputs: + _gate(out, op.reference(q, kv, idx, SCALE, out)) + assert bool(torch.isfinite(out).all()) + assert bool(torch.isnan(parent[:3]).all()) + + +@pytest.mark.large +@pytest.mark.parametrize("heads", [8, 16]) +@pytest.mark.parametrize("nope", [False, True]) +def test_global_split_at_1m_large_address(heads: int, nope: bool) -> None: + """The candidate reads a real 1M pool's far owner slots beyond 32-bit byte addresses.""" + rows = 4 if heads == 8 else 16 + spec = dict( + op.CELLS[0].spec, + topk=2048, + kv_len=1048576, + page_size=32, + owners=63, + blocks=74183, + far=True, + heads=heads, + h_q=heads, + nope=nope, + ) + gen = torch.Generator(device="cuda").manual_seed(heads * 131 + int(nope)) + try: + pool = _pool_1m(spec, gen) + indices = _selections_1m(spec, pool, rows, gen) + assert int(indices.max()) * 656 > 2**32 + q = _queries_1m(rows, spec, gen) + out = torch.empty(rows, heads, 512, device="cuda", dtype=torch.bfloat16) + scratch = torch.empty(rows, 16, heads, 516, device="cuda", dtype=torch.float32) + fns = kernel._compile_global_variant(heads, 16, torch.cuda.current_device()) + kernel._launch_global(fns, q, pool.kv, indices, out, SCALE, scratch) + _check_1m(q, pool.kv, indices, out) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + kernel._launch_global(fns, q, pool.kv, indices, out, SCALE, scratch) + q.copy_(_queries_1m(rows, spec, gen)) + indices.copy_(_selections_1m(spec, pool, rows, gen)) + out.fill_(float("nan")) + scratch.fill_(float("nan")) + graph.replay() + _check_1m(q, pool.kv, indices, out) + finally: + torch.cuda.empty_cache() + + +@pytest.mark.parametrize( + "num_sms,global_fns,global_rows", + [(212, (None, None), range(1, 9)), (100, (None, None), range(1, 7)), (212, None, ())], +) +def test_global_split_row_gate(num_sms, global_fns, global_rows) -> None: + """Rows up to GLOBAL_MAX_ROWS run global16 while their CTAs fit in one wave of the SMs; + every other call keeps the cluster split.""" + compiled = CompiledDsMlaSparseDecode( + device_index=0, num_heads=8, num_sms=num_sms, fns={}, global_fns=global_fns + ) + for rows in range(33): + expected = GLOBAL_SPLITS[0] if rows in global_rows else kernel.pick_splits(rows, num_sms) + assert splits_for(rows, compiled) == expected + + +@pytest.mark.parametrize("heads", [8, 16]) +@pytest.mark.parametrize("rows", [1, 2, 3, 4, 5, 6, 7, 8, 9, 16]) +def test_low_row_wrapper_padded_query_and_graph(heads: int, rows: int) -> None: + """The actual dispatch reads only real heads of a 64-head query and owns graph scratch.""" + q, kv, idx, out, parent, legal, gen = _small_poisoned_global_inputs(rows, heads, False, 2026) + q64 = torch.full((rows + 3, 64, 576), float("nan"), device="cuda", dtype=torch.bfloat16)[3:] + q64[:, :heads] = q + idx.copy_(legal[0][:2048].int()[None].expand(rows, -1)) + compiled = _compiled(heads) + graph = torch.cuda.CUDAGraph() + op(q64, kv, idx, SCALE, out, compiled=compiled) + with torch.cuda.graph(graph): + op(q64, kv, idx, SCALE, out, compiled=compiled) + for replay in range(100): + q64[:, :heads] = torch.randn(q.shape, device="cuda", generator=gen, dtype=q.dtype) + idx.copy_(idx.roll(1, dims=-1)) + out.fill_(float("nan")) + graph.replay() + _gate(out, op.reference(q64, kv, idx, SCALE, out)) + assert bool(torch.isfinite(out).all()) + assert bool(torch.isnan(q64[:, heads:]).all()) + assert bool(torch.isnan(parent[:3]).all()) + if splits_for(rows, compiled) in SPLITS: + expected = kernel.launch( + compiled.fns[kernel.pick_splits(rows, compiled.num_sms)], + q64, + kv, + idx, + torch.empty_like(out), + SCALE, + ) + assert torch.equal(out.view(torch.int16), expected.view(torch.int16)) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_cute_dsl_fp8_paged_mqa_logits.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_cute_dsl_fp8_paged_mqa_logits.py new file mode 100644 index 000000000000..31f0b614247b --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_cute_dsl_fp8_paged_mqa_logits.py @@ -0,0 +1,720 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the cute_dsl_fp8_paged_mqa_logits entry. + +The cells are the Mistral Large 4 FP8 target's at its 9216 envelope, the +Mistral Large 4 NVFP4 target's at its 32768 envelope (on 32- and 64-token pages) +and the NVFP4 tp4, tep4 and dep4 deployments' at max_seq_len 1048576 with chunked +prefill. +""" + +import math +import zlib +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm import deep_gemm +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_fp8_paged_mqa_logits import ( # noqa: E501 + cute_dsl_fp8_paged_mqa_logits as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402,F401 — needs the path declared above + +assert torch.cuda.is_available(), "cute_dsl_fp8_paged_mqa_logits requires a CUDA device" + +E4M3_MAX = 448.0 +#: The schedule block the DSA indexer builds its decode schedule with. +SCHEDULE_BLOCK_KV = 64 +#: Every byte of a Mistral Large 4 cell's pool that the step does not own: 0xFF is +#: an e4m3 NaN code and, four at a time, an fp32 NaN scale, so a score read +#: from a page, slot or layer the call has no business reading comes out NaN. +POISON = 0xFF +#: A cell whose sequences need more pages than its pool has (lengths no rank's +#: pool holds at once) keeps this many of the pool's last blocks free, so the +#: stale block-table entries still name poisoned pages. +SHARED_POOL_FREE = 1024 + +# Gate, in fp32 ulps of the term magnitude. See `_compare`. +ULP = 2.0**-23 +ELEMENT_ULP = 8 +RMS_ULP = 8 + +#: Mistral Large 4's index_sharing (params.json's DSA sharing), one letter per layer: layers 0..16 +#: run their own indexer, then each 4 layers are 2 'shared' and 2 'full', and +#: layer 71 is full. 44 'F' in all. +_ML4_PATTERN = "F" * 17 + "SSFF" * 13 + "SSF" +assert len(_ML4_PATTERN) == 72 and _ML4_PATTERN.count("F") == 44 + + +def _keys(kv_fused, block_table, b: int, n: int, head_dim: int): + """Sequence `b`'s `n` keys as fp64 `(k [n, D], k_scale [n])`.""" + page = kv_fused.shape[1] + pages = block_table[b, : (n + page - 1) // page].long() + codes, scale = op.unpack_cache(kv_fused[pages], head_dim) + k = codes.reshape(-1, head_dim)[:n].contiguous().view(torch.float8_e4m3fn).double() + return k, scale.reshape(-1)[:n].double() + + +def _magnitude(args) -> torch.Tensor: + """`M[row, j] = |k_scale[j]| * sum_h |weights[row, h]| * sum_d |q[b, t, h, d] * k[j, d]|`. + + The size of the terms logit `[row, j]` sums, without the cancellation + between them: what any evaluation's rounding error scales with. Every term + is non-negative and every fp8 code is exact in fp64. 1 outside the keys. + """ + q, kv_fused, weights, context_lens, block_table, _, max_context_len = args[:7] + batch, next_n, _, dim = q.shape + m = torch.ones(batch * next_n, max_context_len, dtype=torch.float64, device=q.device) + for b, n in enumerate(context_lens.tolist()): + k, s = _keys(kv_fused, block_table, b, min(n, max_context_len), dim) + for t in range(next_n): + row = b * next_n + t + terms = q[b, t].float().double().abs() @ k.abs().t() + weighted = terms * weights[row].double().abs().unsqueeze(1) + m[row, : k.shape[0]] = weighted.sum(0) * s.abs() + return m + + +def _compare(out: torch.Tensor, ref: torch.Tensor, magnitude: torch.Tensor): + """8 fp32 ulps of each logit's term magnitude `M`, and 8 ulps relative RMS, + over the causal part of each row only. Returns both measured, in ulps. + + fp8 x fp8 products are exact in fp32, so kernel and fp64 reference differ + only where the kernel rounds: its tensor-core fp32 sum of 128 products per + head, relu, its fp32 FMA chain over the 32 weighted heads (four partial + sums), and the product with the key scale. Each rounding is bounded by the + magnitude of what it sums, so the element gate is stated in ulps of `M` + (`_magnitude`) rather than of the logit, whose own size can be one + cancelling sum. Measured on sm_103 over every cell, rows of 32768 and + 1048576 keys included: at most 0.212 ulp of `M` per element and 0.686 ulp + relative RMS (about what rounding the output to fp32 costs); the gate is 8 + and 8, 38x and 12x above. A wrong + evaluation lands far outside, in ulps of `M` per element on a 16-sequence + batch of every edge length: the op's own fp16 epilogue + (`epi_dtype=torch.float16`) at 758 and head weights merely rounded to fp16 + at 169 (`test_the_band_rejects_an_fp16_epilogue` keeps both); relu after + the head sum at 7.6e5, a neighbouring row's head weights at 9.3e5 and no + key scale at 1.4e8 (probed once, on the same lengths). + + Columns past the causal end are not compared: NaN in `ref` means + unspecified (`test_columns_past_the_causal_end` pins what they hold). + """ + assert out.dtype == ref.dtype == torch.float32, (out.dtype, ref.dtype) + assert out.shape == ref.shape == magnitude.shape, (out.shape, ref.shape, magnitude.shape) + live = ~torch.isnan(ref) + assert bool(torch.isfinite(out[live]).all()), "a causal entry is not finite" + o = torch.where(live, out, torch.zeros_like(out)).double() + r = torch.where(live, ref, torch.zeros_like(ref)).double() + err = float(((o - r).abs() / magnitude.clamp_min(1e-300)).max()) / ULP + assert err <= ELEMENT_ULP, ( + f"max error {err:.3g} ulp of the term magnitude exceeds {ELEMENT_ULP}" + ) + rms = float(((o - r) ** 2).sum().sqrt() / (r**2).sum().sqrt().clamp_min(1e-300)) / ULP + assert rms <= RMS_ULP, f"relative RMS {rms:.3g} ulp exceeds {RMS_ULP}" + return err, rms + + +def _schedule(context_lens: torch.Tensor) -> torch.Tensor: + """The decode schedule, built the way the DSA indexer builds it.""" + return deep_gemm.get_paged_mqa_logits_metadata( + context_lens.view(-1, 1), SCHEDULE_BLOCK_KV, deep_gemm.get_num_sms() + ) + + +def _same_bytes(a: torch.Tensor, b: torch.Tensor) -> bool: + """`torch.equal`, 8 bytes at a time where the buffer allows it: its + elementwise temporary is then an eighth of a 5 GB pool, not the pool.""" + if a.is_contiguous() and b.is_contiguous() and a.numel() % 8 == 0: + a, b = a.reshape(-1).view(torch.int64), b.reshape(-1).view(torch.int64) + return torch.equal(a, b) + + +class _Built(SimpleNamespace): + """A cell's arguments, the memory behind `kv_fused` the test owns, and how + to release it. A manager's views must be gone before it shuts down.""" + + def close(self) -> None: + manager = self.manager + self.args = self.owned = self.manager = None + if manager is not None: + manager.shutdown() + + +def _build_contiguous(spec, seed) -> _Built: + """`(q, kv_fused, weights, context_lens, block_table, schedule, max_context_len)` + over a contiguous single-layer cache. + + Every key is quantized per token to e4m3 with an fp32 scale and written to + a random page, data first and scales after, as the indexer cache stores + them. Unused block-table entries point at page 0. + """ + g = torch.Generator(device="cuda").manual_seed(seed) + cpu = torch.Generator().manual_seed(seed) + heads, dim, tpb = spec["heads"], spec["head_dim"], spec["tokens_per_block"] + next_n = spec["next_n"] + lens = spec["context_lens"] + batch = len(lens) + pages = [math.ceil(n / tpb) for n in lens] + num_pages = sum(pages) + 1 + order = torch.randperm(num_pages, generator=cpu).to(torch.int32) + table = torch.zeros(batch, max(pages), dtype=torch.int32) + used = 0 + for b, n in enumerate(pages): + table[b, :n] = order[used : used + n] + used += n + + flat = torch.zeros(num_pages, tpb * (dim + 4), dtype=torch.uint8, device="cuda") + keys = torch.randn(num_pages, tpb, dim, generator=g, device="cuda") + scale = (keys.abs().amax(-1) / E4M3_MAX).clamp_min(1e-12) + codes = (keys / scale.unsqueeze(-1)).to(torch.float8_e4m3fn) + flat[:, : tpb * dim] = codes.view(torch.uint8).reshape(num_pages, -1) + flat[:, tpb * dim :] = scale.contiguous().view(torch.uint8).reshape(num_pages, -1) + kv_fused = flat.view(num_pages, tpb, 1, dim + 4) + + q = torch.randn(batch, next_n, heads, dim, generator=g, device="cuda").to(torch.float8_e4m3fn) + w = torch.randn(batch * next_n, heads, generator=g, device="cuda") * (dim**-0.5 * heads**-0.5) + context_lens = torch.tensor(lens, dtype=torch.int32, device="cuda") + args = ( + q, + kv_fused, + w.contiguous(), + context_lens, + table.cuda(), + _schedule(context_lens), + spec["max_context_len"], + ) + return _Built(args=args, owned=flat, manager=None) + + +def _ml4_pool(spec): + """`(parent, kv_fused, page_scale)`: the cell's per-layer view of a pool + whose every byte is POISON, laid out as the cell's manager lays it out.""" + tpb, dim = spec["tokens_per_block"], spec["head_dim"] + page = tpb * (dim + 4) + rows, row, blocks = spec["pool_layers"], spec["pool_row"], spec["num_blocks"] + if spec["layout"] == "v1": + parent = torch.full((blocks, rows, page), POISON, dtype=torch.uint8, device="cuda") + kv_fused = parent[:, row].view(blocks, tpb, 1, dim + 4) + assert kv_fused.stride(0) == rows * page + return parent, kv_fused, 1 + parent = torch.full((blocks * rows * page,), POISON, dtype=torch.uint8, device="cuda") + kv_fused = parent[row * page :].view(blocks * rows - row, tpb, 1, dim + 4) + return parent, kv_fused, rows + + +def _ml4_manager(spec): + """A real DSA cache manager at Mistral Large 4's geometry (72 layers, latent 576 + fp8, the 44-row indexer mask): `(manager, kv_fused, page_table)` with the + view `get_indexer_k_cache_buffers` hands out and the page indices the DSA + metadata copies into `indexer_k_cache_block_offsets` (pool slots from + `get_pool_block_indices`, times `indexer_k_cache_page_scale`). The manager + is built for `max_seq_len`, the row width unless the cell says.""" + from tensorrt_llm._torch.attention.backends.sparse.dsa.cache_manager import ( + DSACacheManager, + DSACacheManagerV2, + ) + from tensorrt_llm.bindings import DataType + from tensorrt_llm.bindings.executor import KvCacheConfig as BindingKvCacheConfig + from tensorrt_llm.bindings.internal.batch_manager import CacheType + from tensorrt_llm.llmapi.llm_args import DeepSeekSparseAttentionConfig, KvCacheConfig + from tensorrt_llm.mapping import Mapping + + v2 = spec["layout"] == "manager_v2" + # The manager's pool is allocated outside the caching allocator: hand it back + # what earlier cells' pools left cached (the 1M manager's pool is ~100 GiB + # in tep4's cell, ~50 GiB in dep4's). + torch.cuda.empty_cache() + if v2: + kv_cache_config = KvCacheConfig( + enable_block_reuse=False, max_tokens=spec["max_tokens"], host_cache_size=0 + ) + else: + kv_cache_config = BindingKvCacheConfig( + enable_block_reuse=False, max_tokens=spec["max_tokens"] + ) + max_seq_len = spec.get("max_seq_len", spec["max_context_len"]) + manager = (DSACacheManagerV2 if v2 else DSACacheManager)( + kv_cache_config=kv_cache_config, + kv_cache_type=CacheType.SELFKONLY, + num_layers=72, + num_kv_heads=1, + head_dim=576, + tokens_per_block=spec["tokens_per_block"], + max_seq_len=max_seq_len, + max_batch_size=16, + mapping=Mapping(world_size=1, rank=0, tp_size=1, pp_size=1), + dtype=DataType.FP8, + sparse_attention_config=DeepSeekSparseAttentionConfig(index_head_dim=spec["head_dim"]), + pretrained_config=SimpleNamespace( + num_hidden_layers=72, qk_rope_head_dim=64, index_topk_pattern=list(_ML4_PATTERN) + ), + max_num_tokens=spec.get("max_num_tokens", 9216), + ) + try: + assert manager.max_seq_len == max_seq_len, (manager.max_seq_len, max_seq_len) + assert sum(manager.indexer_k_cache_local_layer_mask) == spec["pool_layers"] + assert manager.indexer_k_cache_page_scale == (spec["pool_layers"] if v2 else 1) + assert manager.max_blocks_per_seq == spec["table_width"], manager.max_blocks_per_seq + lens = spec["context_lens"] + ids = list(range(len(lens))) + # A manager refuses a population past its `max_tokens` quota by returning None. + added = manager.add_dummy_requests(ids, lens, is_gen=False) + assert added is not None, f"the manager refused {lens} at max_tokens {spec['max_tokens']}" + if not v2: + # V1 reads the block ids back from the offsets the attention metadata copies. + manager.copy_batch_block_offsets( + torch.zeros_like(manager.host_kv_cache_block_offsets, device="cuda"), + ids, + 1, + len(ids), + len(ids), + ) + blocks = manager.get_pool_block_indices(len(ids), request_ids=ids, num_contexts=len(ids)) + page_table = (blocks.long() * manager.indexer_k_cache_page_scale).to(torch.int32) + kv_fused = manager.get_indexer_k_cache_buffers(spec["layer"]) + assert kv_fused.is_contiguous() == v2 + print( + f"\n[{spec['layout']}] bytes per token {manager.get_cache_bytes_per_token()}, " + f"view {tuple(kv_fused.shape)} stride {kv_fused.stride()}, " + f"max_seq_len {manager.max_seq_len}, table {tuple(page_table.shape)}", + flush=True, + ) + except BaseException: + manager.shutdown() + raise + return manager, kv_fused, page_table + + +def _write_keys(kv_fused, pages, n: int, gen) -> None: + """`n` random keys, quantized per token to e4m3 with an fp32 scale, written + token `i` to page `pages[i // page]`, slot `i % page`: a page holds its + tokens' data first, then their scales, as the indexer cache stores them.""" + tpb, dim = kv_fused.shape[1], kv_fused.shape[3] - 4 + keys = torch.randn(n, dim, generator=gen, device="cuda") + scale = (keys.abs().amax(-1) / E4M3_MAX).clamp_min(1e-12) + codes = (keys / scale.unsqueeze(-1)).to(torch.float8_e4m3fn).view(torch.uint8) + flat = kv_fused.view(kv_fused.shape[0], -1) + tok = torch.arange(n, device="cuda") + page = torch.as_tensor(pages, device="cuda").long()[tok // tpb].unsqueeze(1) + slot = (tok % tpb).unsqueeze(1) + flat[page, slot * dim + torch.arange(dim, device="cuda")] = codes + flat[page, tpb * dim + slot * 4 + torch.arange(4, device="cuda")] = scale.view( + torch.uint8 + ).view(n, 4) + + +def _fill_sw(built: _Built, spec, lens, seed: int) -> None: + """Write one decode step into the cell's buffers in place: keys into a + poisoned pool, the block table's rows, the lengths, the query and weight + rows, and the schedule. Block-table entries past a sequence's pages name + pages of blocks the step leaves free (poisoned), as the engine leaves stale + in-range entries there; padding rows repeat the dummy sequence's row and + length. A manager cell's table is the manager's, as the metadata copies it.""" + g = torch.Generator(device="cuda").manual_seed(seed) + cpu = torch.Generator().manual_seed(seed) + tpb, heads, dim = spec["tokens_per_block"], spec["heads"], spec["head_dim"] + q, kv_fused, weights, context_lens, block_table, schedule, _ = built.args + pad = spec.get("padding", 0) + live = len(lens) - pad + seqs = list(lens[:live]) + list(lens[live : live + 1]) + need = [math.ceil(n / tpb) for n in seqs] + if built.manager is not None: + page_lists = [built.page_table[b, :k] for b, k in enumerate(need)] + kv_fused.fill_(POISON) + table = torch.zeros_like(built.table) + table[: built.page_table.shape[0], : built.page_table.shape[1]] = built.page_table + table.clamp_(min=0) + else: + blocks, scale = spec["num_blocks"], built.page_scale + if sum(need) <= blocks: + if spec.get("tail"): + ids = torch.arange(blocks - sum(need), blocks) + ids = ids[torch.randperm(sum(need), generator=cpu)] + else: + ids = torch.randperm(blocks, generator=cpu)[: sum(need)] + else: + # Each sequence draws distinct pages from all but the pool's last + # SHARED_POOL_FREE blocks, so sequences share pages; a shared page + # holds the later sequence's keys, which the call and the reference + # both read. + usable = blocks - SHARED_POOL_FREE + assert not spec.get("tail") and max(need) <= usable, (need, blocks) + ids = torch.cat([torch.randperm(usable, generator=cpu)[:k] for k in need]) + page_lists = [(p * scale).to(torch.int32) for p in torch.split(ids, need)] + built.owned.fill_(POISON) + free = torch.ones(blocks, dtype=torch.bool) + free[ids] = False + free = free.nonzero().flatten() + assert free.numel(), "no free block left for the stale entries" + stale = torch.randint(0, free.numel(), tuple(built.table.shape), generator=cpu) + table = (free[stale] * scale).to(torch.int32) + first = spec.get("first_seq", 0) + for b in range(len(lens)): + pages = page_lists[min(b, live)] + table[first + b, : pages.numel()] = pages + built.table.copy_(table) + for pages, n in zip(page_lists, seqs): + _write_keys(kv_fused, pages.tolist(), n, g) + context_lens.copy_(torch.tensor(lens, dtype=torch.int32)) + q.copy_(torch.randn(q.shape, generator=g, device="cuda").to(torch.float8_e4m3fn)) + weights.copy_( + torch.randn(weights.shape, generator=g, device="cuda") * (dim**-0.5 * heads**-0.5) + ) + schedule.copy_(_schedule(context_lens)) + + +def _rows(spec, lens) -> list: + """The call's rows: `lens`, then under `padded_to` the cell's `padding` rows + of a 1-key dummy, as a padded graph batch carries them after its live rows. + Without `padded_to`, `lens` already ends in its padding rows.""" + if "padded_to" not in spec: + return list(lens) + assert len(lens) + spec["padding"] == spec["padded_to"], (len(lens), spec["padded_to"]) + return list(lens) + [1] * spec["padding"] + + +def _build_sw(spec, seed) -> _Built: + """Mistral Large 4's cells, in buffers shaped the way the engine hands them + over: block-table, length, query and weight rows sliced out of larger + buffers (`table_rows`, `first_seq`, `q_offset`), the table the manager's + `max_blocks_per_seq` wide (`table_width`), and the cache a per-layer view + of a pool.""" + heads, dim, tpb, next_n = ( + spec["heads"], + spec["head_dim"], + spec["tokens_per_block"], + spec["next_n"], + ) + batch = len(_rows(spec, spec["context_lens"])) + rows = spec.get("table_rows", batch) + first = spec.get("first_seq", 0) + q_off = spec.get("q_offset", 0) + manager = page_table = None + page_scale = 1 + if spec["layout"].startswith("manager"): + manager, kv_fused, page_table = _ml4_manager(spec) + owned = kv_fused + else: + owned, kv_fused, page_scale = _ml4_pool(spec) + assert spec["table_width"] * tpb >= spec["max_context_len"] + table = torch.zeros(rows, spec["table_width"], dtype=torch.int32, device="cuda") + lens = torch.zeros(rows, dtype=torch.int32, device="cuda") + g = torch.Generator(device="cuda").manual_seed(seed ^ 0x5A5A) + q_rows = torch.randn(q_off + batch * next_n, heads, dim, generator=g, device="cuda") + q_rows = q_rows.to(torch.float8_e4m3fn) + w_rows = torch.randn(q_off + batch * next_n, heads, generator=g, device="cuda") + schedule = torch.zeros(deep_gemm.get_num_sms() + 1, 2, dtype=torch.int32, device="cuda") + args = ( + q_rows[q_off:].view(batch, next_n, heads, dim), + kv_fused, + w_rows[q_off:], + lens[first : first + batch], + table[first : first + batch], + schedule, + spec["max_context_len"], + ) + built = _Built( + args=args, + owned=owned, + manager=manager, + page_table=page_table, + page_scale=page_scale, + table=table, + ) + try: + _fill_sw(built, spec, _rows(spec, spec["context_lens"]), seed) + except BaseException: + built.close() + raise + return built + + +def _last_page_byte(built: _Built) -> int: + """The byte offset, into the pool, of the last byte of the highest page the + call's block-table rows name for its sequences' keys.""" + q, kv_fused, _, context_lens, block_table, *_ = built.args + tpb = kv_fused.shape[1] + top = max( + int(block_table[b, : (n + tpb - 1) // tpb].max()) + for b, n in enumerate(context_lens.tolist()) + ) + base = kv_fused.data_ptr() - built.owned.data_ptr() + return base + top * kv_fused.stride(0) + tpb * kv_fused.shape[3] - 1 + + +def _cell_ids(cells) -> list: + """Each cell's test id, `why[:44]`; a cell whose 44-character prefix an + earlier cell already took (three of tep4's 1M cells share one with tp4's) + is named by its whole `why`, so no id repeats and no earlier id changes. + The seed is `crc32(why)` either way.""" + ids = [] + for c in cells: + ids.append(c.why if c.why[:44] in ids else c.why[:44]) + assert len(set(ids)) == len(ids), ids + return ids + + +@pytest.mark.parametrize( + "cell", + [pytest.param(c, marks=pytest.mark.large) if c.spec.get("large") else c for c in op.CELLS], + ids=_cell_ids(op.CELLS), +) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 + certifying(op) + spec = cell.spec + seed = zlib.crc32(cell.why.encode()) % 2**31 + built = _build_sw(spec, seed) + graph = captured = None + try: + args = built.args + if "capture_lens" in spec: + # Captured where the engine captures it, at other lengths, tables + # and keys; the replay then runs on this cell's step. + _fill_sw(built, spec, _rows(spec, spec["capture_lens"]), seed + 1) + op(*args) # compile outside the capture + torch.cuda.synchronize() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + captured = op(*args) + _fill_sw(built, spec, _rows(spec, spec["context_lens"]), seed) + before = built.owned.clone() + if graph is not None: + graph.replay() + out = captured.clone() + eager = op(*args) + assert eager.data_ptr() == captured.data_ptr(), "the replay and eager call share it" + eager = eager.clone() + else: + out = op(*args) + assert _same_bytes(built.owned, before), "scoring modified the cache" + del before + batch, next_n = args[0].shape[:2] + assert tuple(out.shape) == (batch * next_n, args[6]) + ref = op.reference(*args) + if graph is not None: + live = ~torch.isnan(ref) + assert torch.equal(out.view(torch.int32)[live], eager.view(torch.int32)[live]), ( + "the replay's causal logits are not bitwise those of an eager call" + ) + err, rms = _compare(out, ref, _magnitude(args)) + print(f"\n[{cell.why[:44]}] max {err:.3f} ulp of M, RMS {rms:.3f} ulp", flush=True) + if "past" in spec: + last = _last_page_byte(built) + print(f"last byte of the highest page read: {last}", flush=True) + assert last >= spec["past"], (last, spec["past"]) + finally: + # The cache view goes before a manager shuts down. + args = graph = captured = None + built.close() + + +def test_the_result_is_a_shared_buffer() -> None: + """Two calls of one shape return the same storage, and the second call + overwrites the first result. A caller that keeps a result across calls + reads the later scores without an error.""" + spec = dict( + heads=32, + head_dim=128, + tokens_per_block=32, + next_n=1, + context_lens=[500, 300], + max_context_len=1024, + ) + first_args = _build_contiguous(spec, seed=21).args + second_args = _build_contiguous(spec, seed=22).args + first = op(*first_args) + _compare(first, op.reference(*first_args), _magnitude(first_args)) + second = op(*second_args) + assert first.data_ptr() == second.data_ptr() + _compare(first, op.reference(*second_args), _magnitude(second_args)) + + +@pytest.mark.parametrize( + "max_context_len, table_width, num_blocks, context_lens", + [ + (9216, 292, 1024, [1, 33, 255, 256, 257, 8960, 8961, 9216]), + (32768, 1028, 8192, [1, 257, 16384, 32512, 32513, 32767, 32768]), + (1048576, 32772, 74183, [1, 257, 1048320, 1048321]), + (1048576, 32772, 74157, [1, 257, 32513, 1048321, 1048576]), + ], + ids=["9216", "32768", "1048576_tp4", "1048576"], +) +def test_columns_past_the_causal_end(max_context_len, table_width, num_blocks, context_lens): + """For sequence b the kernel writes every column below + ceil(context_lens[b] / 256) * 256 and none after it: the scores of the + poisoned slots and pages past the last key come out NaN, and the rest of + the row keeps what the buffer held. The decode top-k reads none of it; + this pins what the note says about it, on rows of every Mistral Large 4 + width (the 1048576 row on tp4's 74183-block boot pool, `1048576_tp4`, and + on the delivered 74157-block pool).""" + spec = dict( + heads=32, + head_dim=128, + tokens_per_block=32, + next_n=1, + max_context_len=max_context_len, + layout="v2", + pool_layers=44, + pool_row=5, + num_blocks=num_blocks, + table_width=table_width, + context_lens=context_lens, + ) + built = _build_sw(spec, seed=26) + try: + sentinel = 12345.0 + op(*built.args).fill_(sentinel) + out = op(*built.args) + for b, n in enumerate(spec["context_lens"]): + written = out[b] != sentinel + end = -(-n // 256) * 256 + assert bool(written[:end].all()) and not bool(written[end:].any()), (n, end) + assert bool(torch.isfinite(out[b, :n]).all()) + assert n == end or bool(torch.isnan(out[b, n])) + finally: + built.close() + + +@pytest.mark.parametrize("next_n,width", [(1, 1048576), (4, 1048584)]) +def test_a_1m_row_reads_its_last_page_through_the_table(next_n, width) -> None: + """Each causal row reads its last page through the table. Redirecting that + page to poisoned or unrelated keys must fail the same accuracy gate.""" + spec = dict( + heads=32, + head_dim=128, + tokens_per_block=32, + next_n=next_n, + max_context_len=width, + layout="v2", + pool_layers=44, + pool_row=43, + num_blocks=33792, + table_width=32772, + context_lens=[width], + ) + built = _build_sw(spec, seed=28) + try: + args = built.args + table = args[4] + ref, mag = op.reference(*args), _magnitude(args) + good = op(*args).clone() + print(f"\nas built: {_compare(good, ref, mag)}", flush=True) + last_page = (width - 1) // 32 + page_start = last_page * 32 + for column, refusal in ((last_page + 1, "not finite"), (0, "ulp")): + table[0, last_page] = table[0, column] + out = op(*args).clone() + assert torch.equal(out[:, :page_start], good[:, :page_start]), ( + "only the last page moved" + ) + for row in range(next_n): + end = width - next_n + row + 1 + assert bool(torch.isnan(out[row, page_start:end]).all()) == ( + refusal == "not finite" + ) + with pytest.raises(AssertionError, match=refusal) as refused: + _compare(out, ref, mag) + print(f"last page from column {column}: {refused.value}", flush=True) + finally: + built.close() + + +def test_the_band_rejects_an_fp16_epilogue() -> None: + """The gate's discrimination, kept: on the 16-sequence edge batch the op's + own fp16 epilogue and head weights rounded to fp16 each land outside the + band that the fp32 call sits well inside.""" + spec = dict( + heads=32, + head_dim=128, + tokens_per_block=32, + next_n=1, + max_context_len=9216, + layout="v2", + pool_layers=44, + pool_row=3, + num_blocks=2048, + table_width=292, + context_lens=[1, 31, 32, 33, 63, 64, 65, 127, 128, 129, 255, 256, 257, 2049, 9215, 9216], + ) + built = _build_sw(spec, seed=27) + try: + args = built.args + ref, mag = op.reference(*args), _magnitude(args) + print(f"\nfp32 epilogue: {_compare(op(*args).clone(), ref, mag)}", flush=True) + fp16_epilogue = op(*args, 1, torch.float16).clone() + with pytest.raises(AssertionError, match="ulp") as refused: + _compare(fp16_epilogue, ref, mag) + print(f"fp16 epilogue: {refused.value}", flush=True) + q, kv, w, *rest = args + fp16_weights = op(q, kv, w.half().float(), *rest).clone() + with pytest.raises(AssertionError, match="ulp") as refused: + _compare(fp16_weights, ref, mag) + print(f"fp16 head weights: {refused.value}", flush=True) + finally: + built.close() + + +def _ok(): + spec = dict( + heads=32, + head_dim=128, + tokens_per_block=32, + next_n=1, + context_lens=[100, 64], + max_context_len=256, + ) + return _build_contiguous(spec, seed=25).args + + +def test_guard_refuses_a_schedule_for_another_sm_count() -> None: + """Shown first without the guard: 16 sequences of 9216 keys scored with a + schedule built for twice the SMs. The op launches one CTA per SM and each + takes one schedule partition, so the partitions past the SM count are never + scored: those logits keep what the buffer held, and nothing raises. A + schedule for fewer SMs makes the last CTAs read rows past its end and + faults the context (an illegal memory access, measured in a separate + process), so it is only refused here.""" + spec = dict( + heads=32, + head_dim=128, + tokens_per_block=32, + next_n=1, + context_lens=[9216] * 16, + max_context_len=9216, + ) + q, kv, w, lens, table, schedule, max_len = _build_contiguous(spec, seed=24).args + wide = deep_gemm.get_paged_mqa_logits_metadata( + lens.view(-1, 1), SCHEDULE_BLOCK_KV, 2 * deep_gemm.get_num_sms() + ) + op(q, kv, w, lens, table, schedule, max_len).fill_(float("nan")) + out = op(q, kv, w, lens, table, wide, max_len) + unwritten = int(torch.isnan(out).sum()) + print(f"\nunwritten causal logits with a {tuple(wide.shape)} schedule: {unwritten}", flush=True) + assert 0 < unwritten < out.numel(), unwritten + + for rows in (schedule.shape[0] - 1, schedule.shape[0] + 1, wide.shape[0]): + bad = torch.zeros(rows, 2, dtype=torch.int32, device="cuda") + with pytest.raises(AssertionError, match="SM count \\+ 1 rows"): + with validating(op): + op(q, kv, w, lens, table, bad, max_len) + + +def test_guard_refuses_a_context_past_max_context_len() -> None: + """Shown first without the guard: a 100-key context scored into a row of + 64 columns. The 64 columns hold keys 0..63's scores, right to the band; + keys 64..99 were written into the row's padding (rows are 256 columns wide + here) and are gone, so a top-k over the row can never select them.""" + args = _ok() + q, kv, w, lens, table, schedule, _ = args + full = op.reference(*args) + out = op(q, kv, w, lens, table, schedule, 64) + assert tuple(out.shape) == (2, 64) + _compare(out, full[:, :64], _magnitude(args)[:, :64]) + assert not bool(torch.isnan(full[0, 64:100]).any()), "keys 64..99 have scores" + + with pytest.raises(AssertionError, match="exceeds max_context_len"): + with validating(op): + op(q, kv, w, lens, table, schedule, 64) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_cute_dsl_gvr_topk_prefill.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_cute_dsl_gvr_topk_prefill.py new file mode 100644 index 000000000000..fcef00ee7e0a --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_cute_dsl_gvr_topk_prefill.py @@ -0,0 +1,249 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the cute_dsl_gvr_topk_prefill entry. + +Every cell is gated exactly: a short row (`len <= top_k`) must be `0..len-1` then -1, order +included, and a long row must be the deterministic engine's set, the scores above the row's +k-th score plus the lowest columns among those equal to it, in the same order on a second +call. +""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch.kernels.blackwell.top_k import gvr_topk_decode_self_sampling as gvr_kernels +from tensorrt_llm._torch.kernels.blackwell.top_k import ( + gvr_topk_decode_self_sampling_host as gvr_host, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_gvr_topk_prefill import ( # noqa: E501 + cute_dsl_gvr_topk_prefill as op, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_gvr_topk_prefill import ( # noqa: E501 + gvr_warmup, + gvr_workspace, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.fp8_mqa_logits import ( + fp8_mqa_logits, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402,F401 — needs the path declared above + +assert torch.cuda.is_available(), "the entry's cells drive a CuTe DSL kernel on the GPU" + +#: Written everywhere outside a row's window, its row padding included. A +#: kernel that read outside would rank these above every real score. +POISON = 1.0e30 +HEADS, HEAD_DIM = 32, 128 +#: Rows argsorted per call while drawing the permutations. +_SORT_ROWS = 1024 + + +def _windows(requests): + """`(row_starts, row_ends, width, rows)` for prompts packed back to back in one K block. + + Prompt `(cached, new, taken)` owns `cached + new` keys starting where the previous one + ended; its `j`-th new token sees `[offset, offset + cached + j + 1)`. `taken` keeps only + the prompt's last rows, like one row tile. + """ + starts, ends, offset = [], [], 0 + for cached, new, taken in requests: + first = 0 if taken is None else new - taken + for j in range(first, new): + starts.append(offset) + ends.append(offset + cached + j + 1) + offset += cached + new + as_t = lambda v: torch.tensor(v, dtype=torch.int32, device="cuda") # noqa: E731 + return as_t(starts), as_t(ends), offset + + +def _scores(rows: int, width: int, values: str, g: torch.Generator) -> torch.Tensor: + """Per-row scores; "distinct", "narrow", "spread" and "wide" are distinct within a row.""" + if values == "ties": + return torch.randint(0, 4, (rows, width), generator=g, device="cuda").float() + perm = torch.rand(rows, width, generator=g, device="cuda") + for lo in range(0, rows, _SORT_ROWS): + perm[lo : lo + _SORT_ROWS] = perm[lo : lo + _SORT_ROWS].argsort(dim=1).float() + if values == "distinct": + return perm + if values == "narrow": + # Distinct in fp32, one value in fp16: every key in one coarse bin. + return perm.mul_(2.0**-20).add_(1.0) + assert values == "spread", values + # 2^15 at the top, falling by a factor 2^(1/8192) per rank. + return perm.sub_(width - 1).mul_(2.0**-13).add_(15.0).exp2_() + + +def _build(requests, values: str, top_k: int, seed: int): + """`(input_values, row_starts, row_ends, output_indices)` for one K block. + + `input_values` is a `[rows, W]` view of rows padded as fp8_mqa_logits pads them, + align(W + 256, 256) wide. "logits" scores the block with fp8_mqa_logits itself over + random e4m3 queries and keys (-inf outside the windows, as the target hands it over); + every other pattern writes `POISON` outside the windows. `output_indices` starts at -7 + so an unwritten slot shows. + """ + g = torch.Generator(device="cuda").manual_seed(seed) + row_starts, row_ends, width = _windows(requests) + rows = row_starts.shape[0] + if values == "logits": + q = torch.randn(rows, HEADS, HEAD_DIM, generator=g, device="cuda").to(torch.float8_e4m3fn) + k = torch.randn(width, HEAD_DIM, generator=g, device="cuda").to(torch.float8_e4m3fn) + k_scale = torch.rand(width, generator=g, device="cuda") + 0.5 + w = torch.randn(rows, HEADS, generator=g, device="cuda") * (HEAD_DIM * HEADS) ** -0.5 + input_values = fp8_mqa_logits(q, (k, k_scale), w, row_starts, row_ends) + else: + cols = torch.arange(width, device="cuda") + outside = (cols < row_starts[:, None]) | (cols >= row_ends[:, None]) + scores = _scores(rows, width, values, g).masked_fill_(outside, POISON) + del outside + stride = (width + 255) // 256 * 256 + 256 + base = torch.full((rows, stride), POISON, device="cuda") + base[:, :width] = scores + del scores + input_values = base[:, :width] + out = torch.full((rows, top_k), -7, dtype=torch.int32, device="cuda") + return input_values, row_starts, row_ends, out + + +def _deterministic(input_values, row_starts, row_ends, top_k: int) -> torch.Tensor: + """Each row's deterministic set, sorted, window-local (`[rows, top_k]`, int64): the + scores above the k-th plus the lowest columns among those equal to it; for a short row + `0..len-1` then -1.""" + rows = input_values.shape[0] + out = torch.full((rows, top_k), -1, dtype=torch.int64, device=input_values.device) + for r in range(rows): + s, e = int(row_starts[r]), int(row_ends[r]) + n = e - s + if n <= top_k: + out[r, : max(n, 0)] = torch.arange(max(n, 0), device=out.device) + continue + win = input_values[r, s:e] + kth = torch.topk(win, top_k).values[-1] + above = (win > kth).nonzero().flatten() + tied = (win == kth).nonzero().flatten()[: top_k - above.numel()] + out[r] = torch.cat([above, tied]).sort().values + return out + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 + certifying(op) + top_k = cell.spec["top_k"] + for b, requests in enumerate(cell.spec["blocks"]): + seed = zlib.crc32(f"{cell.why}/{b}".encode()) % 2**31 + values, starts, ends, out = _build(requests, cell.spec["values"], top_k, seed) + ws = gvr_workspace(values.device) + got = op(values, starts, ends, out, top_k, ws) + assert got is out + ref = op.reference(values, starts, ends, out, top_k) + want = _deterministic(values, starts, ends, top_k) + lengths = (ends - starts).long() + short = lengths <= top_k + assert torch.equal(got[short], ref[short]), "a short row is not 0..len-1 then -1" + assert not bool((got[~short] < 0).any()), "a long row holds -1 or an unwritten slot" + assert torch.equal(got.long().sort(dim=1).values[~short], want[~short]), ( + "a long row is not the deterministic set" + ) + again = op(values, starts, ends, torch.full_like(out, -7), top_k, ws) + assert torch.equal(again, got), "a second call gives another order" + print( + f"GVR {cell.why[:60]}: block {b}, {values.shape[0]} rows x {values.shape[1]}, " + f"{int(short.sum())} short rows" + ) + + +def _ok(): + return _build([(4096, 64, None)], "distinct", 2048, seed=3) + + +def test_engine_workspace_default_selects_the_same() -> None: + """Without a slab the engine takes its per-device one: the same selection.""" + values, starts, ends, out = _ok() + mine = op(values, starts, ends, out, 2048, gvr_workspace(values.device)) + theirs = op(values, starts, ends, torch.full_like(out, -7), 2048) + assert torch.equal(mine, theirs) + + +def test_guard_refuses_another_width() -> None: + """k is the output's width, not `top_k`.""" + values, starts, ends, _ = _ok() + wide = torch.full((values.shape[0], 4096), -7, dtype=torch.int32, device="cuda") + with pytest.raises(AssertionError, match="exactly top_k"): + with validating(op): + op(values, starts, ends, wide, 2048) + + +def test_guard_refuses_a_window_past_the_block() -> None: + """The engine reads past W into the row padding and ranks it.""" + values, starts, ends, out = _ok() + with pytest.raises(AssertionError, match="past the"): + with validating(op): + op(values, starts, ends + 300, out, 2048) + with pytest.raises(AssertionError, match="starts before its row"): + with validating(op): + op(values, starts - 1, ends, out, 2048) + + +def _engine_caches() -> tuple: + """The engine's launcher keys and compiled variants: what a call that compiles adds to.""" + return set(gvr_host._PREFILL_CACHE), set(gvr_kernels._COMPILE_CACHE) + + +@pytest.mark.parametrize("width", [1, 7, 2047, 2048]) +def test_a_block_within_top_k_is_answered_without_the_engine(width) -> None: + """Every row of a block of at most top_k columns is short: the identity rows, exactly the + reference's, with no engine launch (nothing compiles for the narrow widths).""" + values, starts, ends, out = _build([(0, width, None), (0, 3, None)], "distinct", 2048, 5) + before = _engine_caches() + got = op(values, starts, ends, out, 2048, gvr_workspace(values.device)) + assert got is out and _engine_caches() == before + assert torch.equal(got, op.reference(values, starts, ends, torch.full_like(out, -7), 2048)) + + +#: The targets' engines: (max_seq_len, logits budget): dep4 at the 1M config, tep4 at the 1M +#: config, both at the 32K config. +_ENGINES = {"dep4-1m": (1048576, 2**26), "tep4-1m": (1048576, 2**28), "32k": (32768, 2**24)} + + +@pytest.mark.parametrize("engine", list(_ENGINES)) +def test_warmup_leaves_nothing_to_compile(engine, monkeypatch) -> None: + """After `gvr_warmup`, the target's tile shapes compile nothing: every tier, each tier-0 + width bucket and both sides of each tier's stride bound (65536 / 131072 / 262144 columns), + full tiles of floor(budget / W) rows and short remainder tiles, up to max_seq_len keys. + Each engine starts from empty engine caches, as a server process does: what another + engine's warmup or an earlier test compiled must not hide a launch this one misses.""" + monkeypatch.setattr(gvr_host, "_PREFILL_CACHE", {}) + monkeypatch.setattr(gvr_kernels, "_COMPILE_CACHE", {}) + max_cols, budget = _ENGINES[engine] + dev = torch.device("cuda") + ws = gvr_workspace(dev) + calls = gvr_warmup(2048, max_cols, budget, dev, ws) + before = _engine_caches() + widths = [2049, 3000, 4096, 4097, 8192, 8448, 8449, 16384, 16385, 32768, 65280, 65281, + 65536, 98304, 131072, 131073, 225000, 262144, 262145, 450000, 524288, 557056, + 1015808, 1048559, 1048576] # fmt: skip + shapes = [] + for width in (w for w in widths if w <= max_cols): + full = max(1, budget // width) + for rows in {full, 1, 75, 148, 149, 296, 297}: + if rows <= full: + shapes.append((rows, width)) + for rows, width in shapes: + stride = (width + 255) // 256 * 256 + 256 + scores = torch.rand(rows, stride, device=dev) + starts = torch.zeros(rows, dtype=torch.int32, device=dev) + ends = (width - rows + 1 + torch.arange(rows, device=dev)).clamp(min=1).to(torch.int32) + out = torch.empty(rows, 2048, dtype=torch.int32, device=dev) + op(scores[:, :width], starts, ends, out, 2048, ws) + del scores + torch.cuda.synchronize() + after = _engine_caches() + print(f"GVR warmup {engine}: {calls} warm calls, then {len(shapes)} tile shapes; launcher keys " + f"{sorted(after[0])}, compiled variants {len(after[1])}") # fmt: skip + assert after == before, ( + f"a tile shape compiled after the warmup: new launchers {after[0] - before[0]}, " + f"new variants {len(after[1] - before[1])}" + ) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_cute_dsl_indexer_k_cache_update.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_cute_dsl_indexer_k_cache_update.py new file mode 100644 index 000000000000..431fde0f3884 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_cute_dsl_indexer_k_cache_update.py @@ -0,0 +1,1030 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the cute_dsl_indexer_k_cache_update entry.""" + +import math +import zlib +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch.attention.backends.sparse.dsa.cache_manager import DSACacheManagerV2 +from tensorrt_llm._torch.attention.backends.sparse.dsa.indexer import _compute_slot_mappings +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_indexer_k_cache_update import ( # noqa: E501 + NORM_EPS, + build, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_indexer_k_cache_update import ( # noqa: E501 + cute_dsl_indexer_k_cache_update as op, +) +from tensorrt_llm.bindings import DataType +from tensorrt_llm.bindings.internal.batch_manager import CacheType +from tensorrt_llm.llmapi.llm_args import DeepSeekSparseAttentionConfig, KvCacheConfig +from tensorrt_llm.mapping import Mapping + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402,F401 — needs the path declared above + +assert torch.cuda.is_available(), ( + "the CuTe DSL kernel runs on a GPU; this test cannot run without one" +) + +HEAD_DIM, ROPE_DIM = 128, 64 +#: Page size and per-token bytes of the indexer K pool (128 codes + one fp32 scale). +PAGE, TOKEN_BYTES = 32, 132 +#: The layer of the pool a cell writes unless its spec names one: neither the +#: first nor the last, so a stray write on either side shows. +LAYER = 3 +POISON = 0xA7 +#: Mistral Large 4's `rope_parameters` (interleaved cells): YaRN with no magnitude +#: scaling, and LeRoPE's per-frequency exp(alpha); its max_seq_len bounds the table. +#: These are the FP8 checkpoint's factor and the FP8 target's max_seq_len; a +#: cell's `yarn_factor` and `max_seq_len` replace them (the NVFP4 checkpoint's +#: factor is 2048, its target's max_seq_len 32768, or 1048576 at its own context: the +#: `_ML4_1M_TP4` and `_ML4_1M` cells). +ML4_THETA, ML4_FACTOR, ML4_ORIGINAL_MAX, ML4_BETA_FAST, ML4_BETA_SLOW = 1.0e4, 512.0, 512, 32.0, 1.0 +ML4_MAX_SEQ_LEN = 9216 +#: The engine's token budget per step at the 1M envelope (the chunk size); a +#: manager at max_seq_len 32768 gets the same. +MAX_NUM_TOKENS = 32768 +#: Tables up to this many rows are evaluated on the host; longer ones (the +#: 1048576-row tables, 2^26 cos and sin) in fp64 on the device. A row depends +#: only on its index. tp4's `table_dtype` cells evaluate theirs on the host at +#: every length, as tp4's test did (`_table`). +HOST_TABLE_ROWS = 32768 +#: Mistral Large 4's index_sharing (params.json's DSA sharing), one letter per layer: layers 0..16 +#: run their own indexer, then each 4 layers are 2 'shared' and 2 'full', and +#: layer 71 is full. Pool row r of the indexer K cache is the r-th full layer. +ML4_INDEX_PATTERN = "F" * 17 + "SSFF" * 13 + "SSF" +ML4_FULL_LAYERS = [i for i, role in enumerate(ML4_INDEX_PATTERN) if role == "F"] +assert len(ML4_INDEX_PATTERN) == 72 and len(ML4_FULL_LAYERS) == 44 +GIB = 2**30 + +_COMPILED: dict = {} + + +def _compiled(is_neox: bool, cache_contiguous: bool): + key = (bool(is_neox), bool(cache_contiguous)) + if key not in _COMPILED: + _COMPILED[key] = build( + torch.cuda.current_device(), is_neox=is_neox, cache_contiguous=cache_contiguous + ) + return _COMPILED[key] + + +def _as_table(angles: torch.Tensor) -> torch.Tensor: + """`[positions, cos(32) | sin(32)]` in fp32, viewed `[positions, 2, 32]` as the op reads it.""" + table = torch.cat([angles.cos(), angles.sin()], dim=1).float().cuda().contiguous() + return table.view(-1, 2, ROPE_DIM // 2) + + +def _table_device(positions: int) -> str: + return "cpu" if positions <= HOST_TABLE_ROWS else "cuda" + + +def _yarn_table( + positions: int, gen: torch.Generator, factor: float = ML4_FACTOR, device: str | None = None +) -> torch.Tensor: + """Mistral Large 4's indexer table: YaRN frequencies, each scaled by a LeRoPE exp(alpha). + + The YaRN ramp runs between the correction dims of beta_fast and beta_slow + and is taken on the unscaled frequencies; alpha is a checkpoint tensor, so a + seeded draw stands in for it. The angles and their cos and sin are taken on + `device`, by default `_table_device(positions)`. + """ + half = ROPE_DIM // 2 + pos_freqs = ML4_THETA ** (torch.arange(0, ROPE_DIM, 2, dtype=torch.float64) / ROPE_DIM) + lerope = torch.exp(0.1 * torch.randn(half, generator=gen, dtype=torch.float64)) + extrapolation, interpolation = lerope / pos_freqs, lerope / (factor * pos_freqs) + + def correction_dim(rotations: float) -> float: + return (ROPE_DIM * math.log(ML4_ORIGINAL_MAX / (rotations * 2 * math.pi))) / ( + 2 * math.log(ML4_THETA) + ) + + low = max(math.floor(correction_dim(ML4_BETA_FAST)), 0) + high = min(math.ceil(correction_dim(ML4_BETA_SLOW)), ROPE_DIM - 1) + ramp = ((torch.arange(half, dtype=torch.float64) - low) / (high - low)).clamp(0, 1) + inv = (interpolation * ramp + extrapolation * (1 - ramp)).to(device or _table_device(positions)) + t = torch.arange(positions, dtype=torch.float64, device=inv.device) + return _as_table(t[:, None] * inv[None, :]) + + +def _identity_table(positions: int, device: str | None = None) -> torch.Tensor: + """A NoPE layer's table: cos 1, sin 0 at every position.""" + dev = device or _table_device(positions) + return _as_table(torch.zeros(positions, ROPE_DIM // 2, dtype=torch.float64, device=dev)) + + +def _keys_and_norm(spec, tokens: int, g: torch.Generator): + """`(indexer_k, norm_weight, norm_bias)`: bf16 keys (misaligned when the spec says so).""" + keys = torch.randn(tokens, HEAD_DIM, generator=g, device="cuda") * 3.0 + 0.5 + if spec["misalign"]: + storage = torch.empty(tokens * HEAD_DIM + 1, dtype=torch.bfloat16, device="cuda") + indexer_k = storage[1:].view(tokens, HEAD_DIM) + indexer_k.copy_(keys) + assert indexer_k.data_ptr() % 8 != 0 + else: + indexer_k = keys.to(torch.bfloat16) + norm_weight = 1.0 + 0.1 * torch.randn(HEAD_DIM, generator=g, device="cuda") + norm_bias = 0.1 * torch.randn(HEAD_DIM, generator=g, device="cuda") + return indexer_k, norm_weight, norm_bias + + +def _table(spec, cpu: torch.Generator): + """`(table, rows)`: the cell's rope table and its row count, the positions it serves. + + `table_dtype="bfloat16"` rounds the values to bf16 and keeps them in fp32, as + the NVFP4 target holds its tables (the op reads fp32). Only tp4's cells carry + it; their table is evaluated on the host at every length, as tp4's test did, + and every other cell's on `_table_device`'s device, as tep4's and dep4's did + (the host up to `HOST_TABLE_ROWS`, as the mainline's). + """ + max_pos = spec.get("max_seq_len", ML4_MAX_SEQ_LEN) + device = "cpu" if "table_dtype" in spec else None + if spec["table"] == "yarn": + table = _yarn_table(max_pos, cpu, spec.get("yarn_factor", ML4_FACTOR), device) + else: + table = _identity_table(max_pos, device) + if spec.get("table_dtype") == "bfloat16": + table = table.to(torch.bfloat16).float() + return table, max_pos + + +def _positions(spec) -> torch.Tensor: + """A step's positions from `requests` and `cached`: request i's rows at + `cached[i]..cached[i] + requests[i] - 1`, in request order. (tep4's and dep4's + `_request_positions`, the same function under another name.)""" + new = list(spec["requests"]) + cached = list(spec.get("cached", (0,) * len(new))) + assert sum(new) == spec["tokens"], (sum(new), spec["tokens"]) + return torch.cat([torch.arange(c, c + n) for c, n in zip(cached, new)]) + + +def _build(spec, seed): + """`(args, pool, layer)` for one cell; `args` is the op's positional argument list. + + Without a `layout` the pool is `[blocks, layers, page * 132]` uint8, + block-major over layers as the V1 DSA cache manager allocates the indexer K + cache, and the cache is one layer's `[blocks, page, 1, 132]` view of it. With + `layout="v2"` the pool is the same bytes, flat, and the cache is the view + `DSACacheManagerV2` hands out (measured in the `manager_v2` cells): the + contiguous `[blocks * layers - layer, page, 1, 132]` from the layer's first + page, block b's page at `b * layers` in it. Each token gets a distinct + (block, slot) and the TRT-LLM slot pair: codes at `page * page_size * 132 + + slot * 128`, the scale at `page * page_size * 132 + page_size * 128 + slot * + 4`. Padding rows get -1. The slot mappings have `slot_rows` entries + (default: one per token); past the pool's `blocks * page` slots the entries + are -1, a tail no row reads. `far_end` puts the tokens in the pool's last + blocks: ascending in V1's layout, and in V2's in the order V2 hands them out, + a request's first page in the highest block. With `requests` the positions + are each request's cached..cached+new-1 (`_positions`); without, 0..tokens-1 + above 64 tokens and random below. + """ + assert spec.get("layout") in (None, "v2"), spec.get("layout") + v2 = spec.get("layout") == "v2" + g = torch.Generator(device="cuda").manual_seed(seed) + cpu = torch.Generator().manual_seed(seed) + tokens, blocks, layers = spec["tokens"], spec["blocks"], spec["pool_layers"] + layer = spec.get("layer", min(LAYER, layers - 1)) + if v2: + page_bytes = PAGE * TOKEN_BYTES + pool = torch.full((blocks * layers * page_bytes,), POISON, dtype=torch.uint8, device="cuda") + k_cache = pool[layer * page_bytes :].view(blocks * layers - layer, PAGE, 1, TOKEN_BYTES) + else: + pool = torch.full( + (blocks, layers, PAGE * TOKEN_BYTES), POISON, dtype=torch.uint8, device="cuda" + ) + k_cache = pool[:, layer, :].view(blocks, PAGE, 1, TOKEN_BYTES) + + indexer_k, norm_weight, norm_bias = _keys_and_norm(spec, tokens, g) + table, max_pos = _table(spec, cpu) + if "requests" in spec: + pos = _positions(spec).cuda() + elif tokens > 64: + pos = torch.arange(tokens, device="cuda") + else: + pos = torch.randint(0, max_pos, (tokens,), generator=g, device="cuda") + if spec.get("top_position"): + pos[0] = max_pos - 1 + + slot_rows = spec.get("slot_rows", tokens) + drawn = min(slot_rows, blocks * PAGE) + assert drawn >= tokens, "the pool has fewer slots than the cell has rows" + if spec.get("far_end") and v2: + n = torch.arange(drawn) + flat = (blocks - 1 - n // PAGE) * PAGE + n % PAGE + elif spec.get("far_end"): + flat = torch.arange(blocks * PAGE - drawn, blocks * PAGE) + else: + flat = torch.randperm(blocks * PAGE, generator=cpu)[:drawn] + block, slot = flat // PAGE, flat % PAGE + page = block * layers if v2 else block + fp8_slot = page * PAGE * TOKEN_BYTES + slot * HEAD_DIM + scale_slot = page * PAGE * TOKEN_BYTES + PAGE * HEAD_DIM + slot * 4 + if drawn < slot_rows: + tail = torch.full((slot_rows - drawn,), -1, dtype=fp8_slot.dtype) + fp8_slot, scale_slot = torch.cat([fp8_slot, tail]), torch.cat([scale_slot, tail]) + if spec["pad_rows"]: + fp8_slot[tokens - spec["pad_rows"] : tokens] = -1 + scale_slot[tokens - spec["pad_rows"] : tokens] = -1 + args = [ + indexer_k, + norm_weight, + norm_bias, + pos.to(torch.int32), + table, + k_cache, + fp8_slot.cuda(), + scale_slot.cuda(), + tokens, + ] + return args, pool, layer + + +def _dsa_manager_v2(blocks: int, max_seq_len: int, max_num_tokens: int) -> DSACacheManagerV2: + """The manager the engine builds for a Mistral Large 4 target by default + (`use_kv_cache_manager_v2` defaults to True): `DSACacheManagerV2` over the + 72-layer fp8 latent pool beside the 44-row indexer pool, with a quota of + `blocks` pages (`max_tokens`), at the target's max_seq_len and the engine's + max_num_tokens (equal at 32768; 32768 at 1048576).""" + return DSACacheManagerV2( + kv_cache_config=KvCacheConfig( + enable_block_reuse=False, max_tokens=blocks * PAGE, host_cache_size=0 + ), + kv_cache_type=CacheType.SELFKONLY, + num_layers=72, + num_kv_heads=1, + head_dim=576, + tokens_per_block=PAGE, + max_seq_len=max_seq_len, + max_batch_size=16, + mapping=Mapping(world_size=1, rank=0, tp_size=1, pp_size=1), + dtype=DataType.FP8, + sparse_attention_config=DeepSeekSparseAttentionConfig(index_head_dim=HEAD_DIM), + pretrained_config=SimpleNamespace( + num_hidden_layers=72, + qk_rope_head_dim=ROPE_DIM, + index_topk_pattern=list(ML4_INDEX_PATTERN), + ), + max_num_tokens=max_num_tokens, + ) + + +def _build_manager(spec, seed): + """`(args, manager, pool, head)` for a `manager_v2` cell. + + A real `DSACacheManagerV2` holds one request per `requests` entry (that + many new tokens after its `cached` ones); the cache is the view + `get_indexer_k_cache_buffers(model_layer)` hands out and the slot mappings + are the engine's own arithmetic (`_compute_slot_mappings` over the block ids + times the manager's page scale), as the DSA metadata builds them. The + manager runs at the spec's `max_num_tokens` (tp4's 1M cells name 32768) or, + when it names none, at `min(max_seq_len, MAX_NUM_TOKENS)`: 32768 for every + cell, as each member's test built it. The context count passed with the + block ids is each member's own: in a cell naming `max_num_tokens` (tp4's), a + request with no cached tokens or more than one new one is a context request, + as tp4's test counted; in every other cell, a request with no cached tokens, + as the mainline's, tep4's and dep4's tests counted (the two agree on the + mainline's cells). `pool` is the first indexer row's view, which spans the + whole indexer pool and is poisoned first; `head` is the byte the cell's view + starts at in it. The view's geometry is asserted to be the one the `v2` cells + rebuild. The pool is allocated outside torch's caching allocator, so what + torch still caches from earlier cells is returned to the device first. + """ + g = torch.Generator(device="cuda").manual_seed(seed) + cpu = torch.Generator().manual_seed(seed) + new = list(spec["requests"]) + cached = list(spec.get("cached", (0,) * len(new))) + pos = _positions(spec) + max_num_tokens = spec.get("max_num_tokens", min(spec["max_seq_len"], MAX_NUM_TOKENS)) + torch.cuda.empty_cache() + manager = _dsa_manager_v2(spec["blocks"], spec["max_seq_len"], max_num_tokens) + try: + assert sum(manager.indexer_k_cache_local_layer_mask) == spec["pool_layers"] + assert manager.indexer_k_cache_page_scale == spec["pool_layers"] + ids = list(range(len(new))) + added = manager.add_dummy_requests(ids, [c + n for c, n in zip(cached, new)], is_gen=False) + assert added is not None, "the pool cannot hold the cell's requests" + if "max_num_tokens" in spec: + contexts = sum(1 for c, n in zip(cached, new) if c == 0 or n > 1) + else: + contexts = sum(1 for c in cached if c == 0) + blocks = manager.get_pool_block_indices(len(ids), request_ids=ids, num_contexts=contexts) + page_table = blocks.long() * manager.indexer_k_cache_page_scale + req = torch.repeat_interleave(torch.arange(len(new)), torch.tensor(new)) + fp8_slot, scale_slot = _compute_slot_mappings(pos, page_table, req, HEAD_DIM, PAGE, 128) + pool = manager.get_indexer_k_cache_buffers(ML4_FULL_LAYERS[0]) + pool.fill_(POISON) + k_cache = manager.get_indexer_k_cache_buffers(spec["model_layer"]) + row = ML4_FULL_LAYERS.index(spec["model_layer"]) + head = row * PAGE * TOKEN_BYTES + assert k_cache.is_contiguous(), k_cache.stride() + assert tuple(k_cache.shape) == (pool.shape[0] - row, PAGE, 1, TOKEN_BYTES), k_cache.shape + assert k_cache.data_ptr() - pool.data_ptr() == head + print( + f"\n[manager] quota {spec['blocks']} pages, {manager.num_blocks} slots, block table " + f"{tuple(blocks.shape)}, max_seq_len {manager.max_seq_len}, max_num_tokens " + f"{max_num_tokens}, {contexts} contexts; view {tuple(k_cache.shape)}, " + f"{k_cache.numel()} bytes, slots {int(fp8_slot.min())}..{int(fp8_slot.max())}" + ) + indexer_k, norm_weight, norm_bias = _keys_and_norm(spec, spec["tokens"], g) + table, _ = _table(spec, cpu) + except BaseException: + manager.shutdown() + raise + args = [ + indexer_k, + norm_weight, + norm_bias, + pos.to(device="cuda", dtype=torch.int32), + table, + k_cache, + fp8_slot.cuda(), + scale_slot.cuda(), + spec["tokens"], + ] + return args, manager, pool, head + + +def _byte_index(fp8_slot, scale_slot, rows: slice): + """Flat cache-byte indices of the codes and the scales the live rows in `rows` write.""" + f, s = fp8_slot[rows], scale_slot[rows] + live = f >= 0 + code_at = (f[live].unsqueeze(1) + torch.arange(HEAD_DIM, device="cuda")).flatten() + scale_at = (s[live].unsqueeze(1) + torch.arange(4, device="cuda")).flatten() + return code_at, scale_at + + +def _compare(args, expected, is_neox: bool) -> None: + """Bytes the op must not write: exact. Scales: exact. Codes: within the + band `_code_excess` derives, and rarely off at all. + + `args` is the call's argument list; `expected` the reference's cache image. + + The kernel normalises in fp32 with its own summation order and an + approximate rsqrt; the reference normalises in fp64. The two round a + normed element to the other bf16 neighbour about once in 10**4 values + (measured: 2 to 9 codes of 4194304 differ in a 32768-row cell), so at most + 2**-10 of the codes (at least one) may differ. Where one does, the + difference is bounded per code (`_code_excess`); rope carries a moved + neighbour into the rotated pair, and a pair that cancels turns it into + several e4m3 steps at the small result (measured: a code 3 steps off, at + x13 * cos + x12 * sin = 0.68897 - 0.70631, reproduced by moving x12 one + bf16 ulp). The scale is a power of two from the row's amax; one ulp on the + amax only moves it when the amax sits on a power-of-two boundary, which + these inputs do not hit, so scales are exact. + + The bytes are compared a slice at a time and only the differing ones are + kept, so a view of a whole pool costs no full-size mask. + """ + stats = _code_excess(args, expected, is_neox) + print(f"\n[codes] {stats}") + assert stats["scales_equal"], "a row scale differs" + assert stats["outside_slots"] == 0, ( + f"the op wrote outside its slots ({stats['outside_slots']} bytes)" + ) + assert stats["outside_band"] == 0, ( + f"{stats['outside_band']} codes are outside the band, the worst at " + f"{stats['worst']:.3g} times it" + ) + assert stats["differ"] <= stats["allowed"], ( + f"{stats['differ']} of {stats['codes']} codes differ; at most {stats['allowed']} may" + ) + + +def _ulp_bf16(x: torch.Tensor) -> torch.Tensor: + """One bf16 unit in the last place at |x| (fp64), the smallest normal's below it.""" + _, exp = torch.frexp(x.abs().clamp(min=2.0**-126)) + return torch.ldexp(torch.ones_like(x), exp - 8) + + +def _step_e4m3(v: torch.Tensor) -> torch.Tensor: + """The e4m3 spacing at |v| (fp64, in units of the row scale): 2**-9 below 2**-6.""" + _, exp = torch.frexp(v.abs().clamp(min=2.0**-6)) + return torch.ldexp(torch.ones_like(v), exp - 4) + + +def _code_excess(args, expected, is_neox: bool) -> dict: + """How the written cache departs from the reference's, and how far. + + For a code that differs, at row t and column c, the band in the cache's + values (the e4m3 value times the row scale s) is + + |x_c'| u_c + |x_p'| u_p + ulp_bf16(|r| + that) + s * step_e4m3(|v| / s) + + for a rotated column (partner p, the table's cos and sin at the row's + position as the weights `|x'|` stand for), and `u_c + s * step` for an + unrotated one. `u` is one bf16 ulp of the normed element plus 2**-20 of + its LayerNorm terms, the most the kernel's fp32 LayerNorm moves it from + the reference's fp64 one (a few fp32 ulps, which cross at most one bf16 + rounding, plus their absolute size where the terms cancel); the rotation + carries each by its weight |cos| or |sin|, plus 2**-22 of the products for + the kernel's fp32 arithmetic; the rotated value `r` and the quantized one + `v` may then each round to the other neighbour, one bf16 ulp and one e4m3 + step. The normed elements, the rotation and its magnitude are the test's + fp64 evaluation of the inputs; the reference being checked is the entry's. + """ + k_cache, fp8_slot, scale_slot, n = args[5], args[6], args[7], int(args[8]) + got = k_cache.contiguous().view(-1) + want = expected.view(-1) + code_at, scale_at = _byte_index(fp8_slot, scale_slot, slice(0, n)) + off_at = _differing_bytes(got, want) + in_codes = torch.isin(off_at, code_at) + stats = dict( + codes=code_at.numel(), + allowed=max(1, code_at.numel() >> 10), + scales_equal=torch.equal(got[scale_at], want[scale_at]), + outside_slots=int((~in_codes).sum()), + differ=int(in_codes.sum()), + outside_band=0, + worst=0.0, + ) + off_at = off_at[in_codes] + if off_at.numel() == 0: + return stats + f = fp8_slot[:n] + live = torch.nonzero(f >= 0).flatten() + order = torch.argsort(f[live]) + rows = live[order[torch.searchsorted(f[live][order], off_at, right=True) - 1]] + cols = off_at - f[rows] + s = want[scale_slot[rows].unsqueeze(1) + torch.arange(4, device="cuda")] + s = s.contiguous().view(torch.float32).flatten().double() + + x = args[0][:n].double() + mean = x.mean(dim=-1, keepdim=True) + centered = (x - mean) * torch.rsqrt((x - mean).pow(2).mean(dim=-1, keepdim=True) + NORM_EPS) + terms = (centered * args[1].double()).abs() + args[2].double().abs() + normed = (centered * args[1].double() + args[2].double()).float().to(torch.bfloat16).double() + u = _ulp_bf16(normed) + 2.0**-20 * terms + + move = u[rows, cols] + rotated = cols < ROPE_DIM + if bool(rotated.any()): + rc = cols[rotated] + partner = (rc + ROPE_DIM // 2) % ROPE_DIM if is_neox else rc ^ 1 + col = rc % (ROPE_DIM // 2) if is_neox else rc // 2 + rr = rows[rotated] + pos = args[3][rr].long() + cos = args[4][pos, 0, col].double().abs() + sin = args[4][pos, 1, col].double().abs() + xc, xp = normed[rr, rc].abs(), normed[rr, partner].abs() + carried = cos * u[rr, rc] + sin * u[rr, partner] + 2.0**-22 * (xc * cos + xp * sin) + first = rc < ROPE_DIM // 2 if is_neox else rc % 2 == 0 + sign = torch.where(first, -1.0, 1.0).double() + r = ( + normed[rr, rc] * args[4][pos, 0, col].double() + + sign * normed[rr, partner] * args[4][pos, 1, col].double() + ).abs() + move = move.clone() + move[rotated] = carried + _ulp_bf16(r + carried) + a = got[off_at].view(torch.float8_e4m3fn).double() * s + b = want[off_at].view(torch.float8_e4m3fn).double() * s + band = move + s * _step_e4m3(torch.maximum(a.abs(), b.abs()) / s) + ratio = (a - b).abs() / band + stats["outside_band"] = int((ratio > 1).sum()) + stats["worst"] = float(ratio.max()) + return stats + + +def _differing_bytes(got: torch.Tensor, want: torch.Tensor, chunk: int = 1 << 28) -> torch.Tensor: + """Flat indices at which two equally long byte vectors differ.""" + found = [ + torch.nonzero(got[s : s + chunk] != want[s : s + chunk]).flatten() + s + for s in range(0, got.numel(), chunk) + ] + return torch.cat(found) if found else torch.empty(0, dtype=torch.long, device=got.device) + + +def _assert_other_layers_untouched(pool, before, layer) -> None: + assert torch.equal(pool[:, :layer], before[:, :layer]), "a lower layer was written" + assert torch.equal(pool[:, layer + 1 :], before[:, layer + 1 :]), "a higher layer was written" + + +def _assert_head_untouched(pool, head: int) -> None: + """A V2 view runs from its layer's first page to the pool's end, so the other + layers' pages after it are in the view (`_compare` holds them); the bytes + before it are not.""" + assert bool((pool.view(-1)[:head] == POISON).all()), "the op wrote below the layer's view" + + +def _assert_replay_matches_eager(spec, compiled, seed) -> None: + """Capture the call at this batch, replay it on new data, and require the eager bytes. + + A capture freezes the launcher, the grid and every pointer; what changes + between replays is the data behind them. So the static inputs take a second + set of keys, weights, positions, table and slots, the pool goes back to + poison, and the replay must write bitwise what an eager call on that data + writes into a pool of its own. `validate=False`, as in every capture. + """ + static, pool, _ = _build(spec, seed) + fresh, _, _ = _build(spec, seed + 1) + eager, eager_pool, _ = _build(spec, seed + 1) + kwargs = dict(is_neox=spec["is_neox"], compiled=compiled, validate=False) + op(*static, **kwargs) + stream = torch.cuda.Stream() + stream.wait_stream(torch.cuda.current_stream()) + graph = torch.cuda.CUDAGraph() + with torch.cuda.stream(stream), torch.cuda.graph(graph, stream=stream): + op(*static, **kwargs) + torch.cuda.current_stream().wait_stream(stream) + for i in (0, 1, 2, 3, 4, 6, 7): + static[i].copy_(fresh[i]) + pool.fill_(POISON) + graph.replay() + op(*eager, **kwargs) + torch.cuda.synchronize() + assert torch.equal(pool, eager_pool), "the replay wrote other bytes than the eager call" + + +#: The steps a `boot_capture` cell's graph runs, in order: the capture batch, then replays. +BOOT_STEPS = ("warmup", "live", "idle", "drawn") + + +def _boot_inputs(spec, base, step: str, seed: int) -> dict: + """One step's data through a `boot_capture` cell's graph: `{arg index: tensor}` for the + keys, positions and both slot mappings. + + `warmup` is the engine's capture batch: row 0 the longest dummy's new key at position + max_seq_len - 2 (its KV length max_seq_len - 1), rows 1.. one-key fillers at position 0, + each in a page of its own. `live` is attention DP's one live row at max_seq_len - 1 ahead of + the cell's `pad_rows` padding rows, and `idle` an idle rank's one-key dummy (position 0) + ahead of them; the padding rows are one request repeated, so they carry one key, position + 0 and one slot pair. `drawn` is a batch of live rows at drawn positions. A row's slots are + `_compute_slot_mappings`'s: its page (a block of its own times the 44 rows) and its + position's place in the page. The mappings keep `base`'s length (`slot_rows`); the entries + past the batch are `_build`'s, and the op must not read them. + """ + rows, top = spec["tokens"], spec["max_seq_len"] - 1 + g = torch.Generator(device="cuda").manual_seed(seed) + cpu = torch.Generator().manual_seed(seed) + keys = (torch.randn(rows, HEAD_DIM, generator=g, device="cuda") * 3.0 + 0.5).to(torch.bfloat16) + block = torch.randperm(spec["blocks"], generator=cpu)[:rows] + if step == "drawn": + pos = torch.randint(0, top + 1, (rows,), generator=cpu) + else: + pos = torch.zeros(rows, dtype=torch.long) + pos[0] = {"warmup": top - 1, "live": top, "idle": 0}[step] + pad = spec["pad_rows"] if step in ("live", "idle") else 0 + if pad: + keys[rows - pad :] = keys[rows - pad] + block[rows - pad :] = block[rows - pad] + page = block * spec["pool_layers"] * PAGE * TOKEN_BYTES + fp8_slot, scale_slot = base[6].clone(), base[7].clone() + fp8_slot[:rows] = (page + (pos % PAGE) * HEAD_DIM).cuda() + scale_slot[:rows] = (page + PAGE * HEAD_DIM + (pos % PAGE) * 4).cuda() + return {0: keys, 3: pos.to(device="cuda", dtype=torch.int32), 6: fp8_slot, 7: scale_slot} + + +def _drive_boot_capture(spec, seed, certifying) -> None: # noqa: F811 + """One `boot_capture` cell: a long-family decode graph as the dep4 engine captures and + replays it (`_boot_inputs`). + + The warm-up batch is called once and captured (`validate=False`, as the engine does); + the graph is then replayed on each later step's data into a pool poisoned again, and each + replay must write bitwise what an eager call on that data writes into a pool of its own. + Armed, every step's data, the warm-up batch first, is the eager, validated call gated + against the reference. The table and the norm weights stay as captured, as the engine's + do. + """ + static, pool, layer = _build(dict(spec, pad_rows=0), seed) + eager, eager_pool, _ = _build(dict(spec, pad_rows=0), seed) + compiled = _compiled(spec["is_neox"], static[5].is_contiguous()) + kwargs = dict(is_neox=spec["is_neox"], compiled=compiled, validate=False) + steps = {s: _boot_inputs(spec, static, s, seed + 1 + i) for i, s in enumerate(BOOT_STEPS)} + + def load(args, step): + for i, t in steps[step].items(): + args[i].copy_(t) + + load(static, "warmup") + op(*static, **kwargs) + stream = torch.cuda.Stream() + stream.wait_stream(torch.cuda.current_stream()) + graph = torch.cuda.CUDAGraph() + with torch.cuda.stream(stream), torch.cuda.graph(graph, stream=stream): + op(*static, **kwargs) + torch.cuda.current_stream().wait_stream(stream) + for step in BOOT_STEPS[1:]: + load(static, step) + pool.fill_(POISON) + graph.replay() + load(eager, step) + eager_pool.fill_(POISON) + op(*eager, **kwargs) + torch.cuda.synchronize() + assert torch.equal(pool, eager_pool), f"the {step} replay wrote other bytes than eager" + certifying(op) + for step in BOOT_STEPS: + load(static, step) + pool.fill_(POISON) + expected = op.reference(*static, is_neox=spec["is_neox"], compiled=compiled) + out = op(*static, is_neox=spec["is_neox"], compiled=compiled, validate=True) + assert out is static[5], "the op writes the caller's cache and returns it" + _assert_head_untouched(pool, layer * PAGE * TOKEN_BYTES) + print(f"\n[boot {step}] positions {static[3].tolist()}") + _compare(static, expected, spec["is_neox"]) + + +def _seed(cell) -> int: + return zlib.crc32(cell.why.encode()) % 2**31 + + +def _drive_manager_cell(spec, seed, certifying) -> None: # noqa: F811 + """One `manager_v2` cell: the eager, validated call on a real manager's view. + + The manager's views must be gone before it releases its pool, and a failed + check's traceback holds them in its frames; so a failure is carried out of + the block as its message alone and raised once the manager is shut down. + The device memory in use with the manager up is printed: its pool is + allocated outside torch's caching allocator. + """ + args, manager, pool, head = _build_manager(spec, seed) + failure = None + try: + compiled = _compiled(spec["is_neox"], args[5].is_contiguous()) + certifying(op) + expected = op.reference(*args, is_neox=spec["is_neox"], compiled=compiled) + out = op(*args, is_neox=spec["is_neox"], compiled=compiled, validate=True) + assert out is args[5], "the op writes the caller's cache and returns it" + _assert_head_untouched(pool, head) + _compare(args, expected, spec["is_neox"]) + free, total = torch.cuda.mem_get_info() + print(f"[device] {(total - free) / GIB:.2f} GiB in use of {total / GIB:.2f}") + except Exception as exc: # noqa: BLE001 -- re-raised below, without its frames + failure = f"{type(exc).__name__}: {exc}" + finally: + torch.cuda.synchronize() + args = pool = expected = out = None + manager.shutdown() + assert failure is None, failure + + +def _drive_position_sweep(spec, seed, certifying) -> None: # noqa: F811 + """One `position_sweep` cell: the cell's keys and slots, run once per chunk of + `tokens` positions across `[lo, hi)`, so every table row there is read at its + own position. Each call is the eager, validated one on a pool poisoned + again, gated against the reference like any cell.""" + args, pool, layer = _build(spec, seed) + compiled = _compiled(spec["is_neox"], args[5].is_contiguous()) + certifying(op) + lo, hi = spec["position_sweep"] + ramp = torch.arange(spec["tokens"], dtype=torch.int32, device="cuda") + for start in range(lo, hi, spec["tokens"]): + pool.fill_(POISON) + args[3] = ramp + start + expected = op.reference(*args, is_neox=spec["is_neox"], compiled=compiled) + out = op(*args, is_neox=spec["is_neox"], compiled=compiled, validate=True) + assert out is args[5], "the op writes the caller's cache and returns it" + _assert_head_untouched(pool, layer * PAGE * TOKEN_BYTES) + _compare(args, expected, spec["is_neox"]) + assert start + spec["tokens"] == hi, "the sweep stopped short of its last position" + + +def _cell_ids(cells) -> list[str]: + """A test id per cell: the first 44 characters of its `why`, as each source's test + named it. Where two cells' ids would be one (tp4's and tep4's 1M decode-graph cells + share a why, or its first 44 characters, and differ in the table: tp4's carries + `table_dtype`), the cell carrying `table_dtype` gets it appended; every other id + stays as its source's test had it.""" + short = [c.why[:44] for c in cells] + count = {i: short.count(i) for i in short} + ids = [ + f"{i} table_dtype={c.spec['table_dtype']}" + if count[i] > 1 and "table_dtype" in c.spec + else i + for i, c in zip(short, cells) + ] + assert len(set(ids)) == len(ids), [i for i in ids if ids.count(i) > 1] + return ids + + +@pytest.mark.parametrize( + "cell", + [pytest.param(c, marks=pytest.mark.large) if c.spec.get("large") else c for c in op.CELLS], + ids=_cell_ids(op.CELLS), +) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 + spec = cell.spec + if spec.get("layout") == "manager_v2": + _drive_manager_cell(spec, _seed(cell), certifying) + return + if spec.get("position_sweep"): + _drive_position_sweep(spec, _seed(cell), certifying) + return + if spec.get("boot_capture"): + _drive_boot_capture(spec, _seed(cell), certifying) + return + args, pool, layer = _build(spec, seed=_seed(cell)) + k_cache = args[5] + compiled = _compiled(spec["is_neox"], k_cache.is_contiguous()) + if spec.get("graph"): + # Before arming: the guard reads positions and slots back to the host, + # which a capture forbids. + _assert_replay_matches_eager(spec, compiled, _seed(cell)) + certifying(op) + # The reference reads the bytes it does not write, so it runs first. + expected = op.reference(*args, is_neox=spec["is_neox"], compiled=compiled) + v2 = spec.get("layout") == "v2" + pool_before = None if v2 else pool.clone() + # The target's first, eager call validates on the device; later ones do not. + out = op(*args, is_neox=spec["is_neox"], compiled=compiled, validate=True) + assert out is k_cache, "the op writes the caller's cache and returns it" + if v2: + _assert_head_untouched(pool, layer * PAGE * TOKEN_BYTES) + else: + _assert_other_layers_untouched(pool, pool_before, layer) + _compare(args, expected, spec["is_neox"]) + + +def _cell_32768_v2(): + """The non-large 32768-row cell on a rebuilt V2 view at max_seq_len 32768 (slots past 2^31 + and 2^32 bytes).""" + return next( + c + for c in op.CELLS + if c.spec.get("layout") == "v2" + and c.spec["tokens"] == 32768 + and c.spec.get("max_seq_len") == 32768 + and not c.spec.get("large") + ) + + +def _cell_1m_chunk_v2(): + """tp4's 1M-table 32768-row chunk on a rebuilt V2 view: positions 1015808..1048575 over + the bf16-rounded table (`table_dtype`, which tep4's and dep4's cells do not carry).""" + return next( + c + for c in op.CELLS + if c.spec.get("layout") == "v2" + and c.spec.get("max_seq_len") == 1048576 + and c.spec.get("requests") == (32768,) + and c.spec["table"] == "yarn" + and c.spec.get("table_dtype") == "bfloat16" + ) + + +def _cell_1m_v2(): + """The 1M envelope's last chunk on a rebuilt V2 view: positions 1015808..1048575 (tep4's + and dep4's cell, over the fp32 table: tp4's, with `table_dtype`, is `_cell_1m_chunk_v2`).""" + return next( + c + for c in op.CELLS + if c.spec.get("layout") == "v2" + and c.spec.get("max_seq_len") == 1048576 + and c.spec["table"] == "yarn" + and c.spec.get("cached") == (1015808,) + and "table_dtype" not in c.spec + ) + + +@pytest.mark.parametrize( + "tokens, prefixes", + [(9216, (1024, 1025)), (32768, (1024, 1025, 9216, 16385, 32767))], + ids=["9216", "32768"], +) +def test_rows_do_not_depend_on_the_launch_shape(tokens, prefixes) -> None: + """The launcher and the grid change no byte a row writes. + + One input of a full-step cell -- the FP8 target's 9216-row interleaved cell + on the strided view, or the NVFP4 target's 32768-row cell on a V2 view (the + contiguous launchers) -- run at shorter num_tokens: 1024 (2 rows per CTA), + 1025 (8 rows per CTA) and, at 32768, also 9216, 16385 and 32767. The rows a + shorter run writes are bitwise the full run's, and the rows at and past its + num_tokens stay unwritten. With the bucket-edge cells and the maximum + measured against the reference, every row count between them writes what + they measured. + """ + if tokens == 9216: + cell = next(c for c in op.CELLS if c.spec["tokens"] == 9216 and c.spec["table"] == "yarn") + else: + cell = _cell_32768_v2() + full, _, _ = _build(cell.spec, _seed(cell)) + compiled = _compiled(False, full[5].is_contiguous()) + op(*full, is_neox=False, compiled=compiled) + want = full[5].contiguous().view(-1) + for n in prefixes: + args, _, _ = _build(cell.spec, _seed(cell)) + args[8] = n + op(*args, is_neox=False, compiled=compiled) + got = args[5].contiguous().view(-1) + head = torch.cat(_byte_index(args[6], args[7], slice(0, n))) + tail = torch.cat(_byte_index(args[6], args[7], slice(n, None))) + assert torch.equal(got[head], want[head]), f"num_tokens {n} wrote other bytes" + assert bool((got[tail] == POISON).all()), f"num_tokens {n} wrote rows past it" + args = got = None + + +@pytest.mark.parametrize("layout", [None, "v2"], ids=["interleaved", "interleaved-v2"]) +def test_a_capturable_call_matches_the_validated_one(layout) -> None: + """`validate=False` (every call inside a CUDA graph) writes the same bytes.""" + spec = dict( + tokens=16, + pool_layers=4, + blocks=8, + pad_rows=2, + misalign=False, + is_neox=False, + table="yarn", + layout=layout, + ) + args, _, _ = _build(spec, seed=41) + again, _, _ = _build(spec, seed=41) + compiled = _compiled(False, args[5].is_contiguous()) + op(*args, is_neox=False, compiled=compiled, validate=True) + op(*again, is_neox=False, compiled=compiled, validate=False) + assert torch.equal(args[5], again[5]) + + +@pytest.mark.parametrize( + "wrong", + [ + "offsets wrapped at 2^32", + "positions one back", + "rotation reversed", + "1M positions one back", + "1M positions wrapped at 2^15", + ], +) +def test_the_gate_sees_a_wrapped_offset_or_a_wrong_rotation(wrong) -> None: + """The 32768-row V2 cell's gate is not blind to what a 32-bit offset, an + off-by-one position or a reversed rotation would do, nor the 1M chunk cell's + to a position one back or one read through a 32768-row table. + + The op runs on that cell's input with every slot taken modulo 2^32 (what an + unsigned 32-bit byte offset would write; the rows past 2^32 bytes land + inside the view, among other rows), with every position one back (the + first row stays at 0), or with the table's sines negated; on the 1M chunk + (positions 1015808..1048575) with every position one back, or modulo 2^15 + (the rows a 32768-row table would hand out cyclically). `_compare`, against + the reference of the true input, must refuse each; what it measured is + printed. + """ + cell = _cell_1m_v2() if wrong.startswith("1M") else _cell_32768_v2() + args, _, _ = _build(cell.spec, _seed(cell)) + compiled = _compiled(False, args[5].is_contiguous()) + expected = op.reference(*args, is_neox=False, compiled=compiled) + call = list(args) + if wrong == "offsets wrapped at 2^32": + moved = int((args[6] >= 2**32).sum()) + assert moved > 0, "no slot of the cell lies past 2^32 bytes" + call[6], call[7] = args[6] % 2**32, args[7] % 2**32 + elif wrong.endswith("positions one back"): + call[3] = (args[3] - 1).clamp(min=0) + elif wrong == "1M positions wrapped at 2^15": + assert int(args[3].min()) >= 2**15, "the cell has a position below 2^15" + call[3] = args[3] % 2**15 + else: + call[4] = args[4].clone() + call[4][:, 1].neg_() + op(*call, is_neox=False, compiled=compiled) + print(f"\n[{wrong}] {_code_excess(args, expected, is_neox=False)}") + with pytest.raises(AssertionError, match="scale differs|outside its slots|outside the band"): + _compare(args, expected, is_neox=False) + + +@pytest.mark.parametrize("wrong", ["positions one back", "positions wrapped at 2^15"]) +def test_the_gate_sees_a_far_position_read_from_the_wrong_row(wrong) -> None: + """The 1M chunk cell's gate is not blind to a key roped at the wrong row. + + The op runs on that cell's input (positions 1015808..1048575 over the + bf16-rounded 1048576-row table) with every position one back, or taken + modulo 2^15 (the row a position wrapped at the 32768-row envelope would + read). `_compare`, against the reference of the true input, must refuse + each; what it measured is printed. + """ + cell = _cell_1m_chunk_v2() + args, _, _ = _build(cell.spec, _seed(cell)) + compiled = _compiled(False, args[5].is_contiguous()) + expected = op.reference(*args, is_neox=False, compiled=compiled) + call = list(args) + if wrong == "positions one back": + call[3] = args[3] - 1 + else: + call[3] = args[3] % 2**15 + op(*call, is_neox=False, compiled=compiled) + print(f"\n[{wrong}] {_code_excess(args, expected, is_neox=False)}") + with pytest.raises(AssertionError, match="scale differs|outside its slots|outside the band"): + _compare(args, expected, is_neox=False) + + +_SMALL = dict( + tokens=8, pool_layers=4, blocks=8, pad_rows=0, misalign=False, is_neox=False, table="yarn" +) + + +def _small(): + return _build(_SMALL, seed=43) + + +def test_guard_refuses_a_rope_mode_the_launchers_were_not_built_for() -> None: + args, _, _ = _small() + neox = _compiled(True, False) + expected = op.reference(*args, is_neox=False, compiled=neox) + # Unguarded, launchers built for NeoX rotate interleaved keys the NeoX way: + # well-formed rows whose rope codes are mostly wrong. + op(*args, is_neox=False, compiled=neox) + rope_at = (args[6].unsqueeze(1) + torch.arange(ROPE_DIM, device="cuda")).flatten() + wrong = (args[5].contiguous().view(-1)[rope_at] != expected.view(-1)[rope_at]).float().mean() + assert float(wrong) > 0.5, f"only {float(wrong):.2f} of the rope codes moved" + with pytest.raises(AssertionError, match="compiled for is_neox"): + with validating(op): + op(*_small()[0], is_neox=False, compiled=neox) + + +def test_guard_refuses_contiguous_addressing_on_a_strided_view() -> None: + args, pool, layer = _small() + assert not args[5].is_contiguous() + dense = _compiled(False, True) + # Unguarded, the contiguous launchers address the layer's view as if dense, + # and the rows of later blocks land in other layers of the same pool. + before = pool.clone() + op(*args, is_neox=False, compiled=dense) + assert not torch.equal(pool[:, :layer], before[:, :layer]), "no other layer was written" + args, _, _ = _small() + with pytest.raises(AssertionError, match="strided view"): + with validating(op): + op(*args, is_neox=False, compiled=dense) + + +def test_guard_refuses_a_position_past_the_rope_table() -> None: + # Unguarded, the kernel reads past the table: not driven here. + args, _, _ = _small() + args[3] = args[3].clone() + args[3][0] = args[4].shape[0] + with pytest.raises(AssertionError, match="rope table"): + with validating(op): + op(*args, is_neox=False, compiled=_compiled(False, False)) + + +def test_guard_refuses_a_misaligned_slot() -> None: + # Unguarded, the kernel stores a 32-bit word at an unaligned byte: not driven here. + args, _, _ = _small() + args[6] = args[6].clone() + args[6][1] += 2 + with pytest.raises(AssertionError, match="4-byte aligned"): + with validating(op): + op(*args, is_neox=False, compiled=_compiled(False, False)) + + +def test_guard_refuses_slots_that_disagree_on_validity() -> None: + args, _, _ = _small() + args[7] = args[7].clone() + args[7][2] = -1 + # Unguarded, a row with a code slot and no scale slot is silently skipped. + op(*args, is_neox=False, compiled=_compiled(False, False)) + row_codes = args[6][2] + torch.arange(HEAD_DIM, device="cuda") + assert bool((args[5].contiguous().view(-1)[row_codes] == POISON).all()) + args, _, _ = _small() + args[7] = args[7].clone() + args[7][2] = -1 + with pytest.raises(AssertionError, match="disagree"): + with validating(op): + op(*args, is_neox=False, compiled=_compiled(False, False)) + + +def _strided_keys(args): + """Mistral Large 4's keys as the projection leaves them: the first 128 columns of `[T, 160]`.""" + kw = torch.zeros(args[0].shape[0], HEAD_DIM + 32, dtype=torch.bfloat16, device="cuda") + kw[:, :HEAD_DIM] = args[0] + args[0] = kw[:, :HEAD_DIM] + + +@pytest.mark.parametrize( + "mutate, message", + [ + (_strided_keys, r"Mismatched indexer_k\.strides\[0\]"), + (lambda a: a.__setitem__(1, a[1].to(torch.bfloat16)), r"argument #1 .*dtype=float32"), + (lambda a: a.__setitem__(2, a[2].to(torch.bfloat16)), r"argument #2 .*dtype=float32"), + (lambda a: a.__setitem__(4, a[4].to(torch.bfloat16)), r"argument #4 .*dtype=float32"), + (lambda a: a.__setitem__(3, a[3].long()), r"argument #3 .*dtype=int32"), + (lambda a: a.__setitem__(6, a[6].int()), r"argument #6 .*dtype=int64"), + (lambda a: a.__setitem__(7, a[7].int()), r"argument #7 .*dtype=int64"), + (lambda a: a.__setitem__(3, torch.cat([a[3], a[3]])), r"Mismatched position_ids\.shape"), + (lambda a: a.__setitem__(6, a[6][:4]), r"Mismatched slot_mapping_fp8\.shape"), + ], + ids=[ + "strided keys", + "bf16 norm weight", + "bf16 norm bias", + "bf16 table", + "int64 positions", + "int32 code slots", + "int32 scale slots", + "positions longer than keys", + "slots shorter than keys", + ], +) +def test_the_launcher_refuses_what_it_was_not_compiled_for(mutate, message) -> None: + """The note's refusals, pinned: raised before any launch, with `validate` off. + + The pool is unchanged afterwards, so nothing ran. + """ + args, pool, _ = _small() + before = pool.clone() + mutate(args) + with pytest.raises(ValueError, match=message): + op(*args, is_neox=False, compiled=_compiled(False, False), validate=False) + torch.cuda.synchronize() + assert torch.equal(pool, before), "a refused call wrote the pool" diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_cute_dsl_indexer_q_side_fused.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_cute_dsl_indexer_q_side_fused.py new file mode 100644 index 000000000000..83360399f7d7 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_cute_dsl_indexer_q_side_fused.py @@ -0,0 +1,473 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the cute_dsl_indexer_q_side_fused entry.""" + +import math +import zlib + +import pytest +import torch + +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_indexer_q_side_fused import ( # noqa: E501 + build, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_indexer_q_side_fused import ( # noqa: E501 + cute_dsl_indexer_q_side_fused as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402,F401 — needs the path declared above + +assert torch.cuda.is_available(), ( + "the CuTe DSL kernel runs on a GPU; this test cannot run without one" +) + +#: The indexer geometry and its folded softmax scale (fp32 2**-6 exactly). +HEADS, HEAD_DIM, ROPE_DIM = 32, 128, 64 +WEIGHT_SCALE = HEAD_DIM**-0.5 * HEADS**-0.5 +#: e4m3 NaN: the poison both code buffers start from. +POISON = 0x7F +#: Mistral Large 4's `rope_parameters` (interleaved cells): YaRN with no magnitude +#: scaling, and LeRoPE's per-frequency exp(alpha); its max_seq_len bounds the table. +#: The FP8 checkpoint's factor is the default; a cell's `factor` key overrides it +#: (the NVFP4 checkpoint's is 2048, its max_seq_len 32768). +ML4_THETA, ML4_FACTOR, ML4_ORIGINAL_MAX, ML4_BETA_FAST, ML4_BETA_SLOW = 1.0e4, 512.0, 512, 32.0, 1.0 +ML4_MAX_SEQ_LEN = 9216 +NVFP4_FACTOR, NVFP4_MAX_SEQ_LEN = 2048.0, 32768 +#: The NVFP4 checkpoint's own context: max_seq_len 1048576 in chunk steps of 32768 rows. +LONG_MAX_SEQ_LEN = 1048576 + +_COMPILED: dict = {} + + +def _compiled(is_neox: bool): + if is_neox not in _COMPILED: + _COMPILED[is_neox] = build(torch.cuda.current_device(), is_neox=is_neox) + return _COMPILED[is_neox] + + +def _as_table(angles: torch.Tensor) -> torch.Tensor: + """`[positions, 64]` fp32 `[cos | sin]`, built in fp64 and rounded once.""" + return torch.cat([angles.cos(), angles.sin()], dim=1).float().cuda().contiguous() + + +def _yarn_table(positions: int, gen: torch.Generator, factor: float = ML4_FACTOR) -> torch.Tensor: + """Mistral Large 4's indexer table: YaRN frequencies, each scaled by a LeRoPE exp(alpha). + + The YaRN ramp runs between the correction dims of beta_fast and beta_slow + and is taken on the unscaled frequencies (so it does not depend on the + factor); alpha is a checkpoint tensor, so a seeded draw stands in for it. + """ + half = ROPE_DIM // 2 + pos_freqs = ML4_THETA ** (torch.arange(0, ROPE_DIM, 2, dtype=torch.float64) / ROPE_DIM) + lerope = torch.exp(0.1 * torch.randn(half, generator=gen, dtype=torch.float64)) + extrapolation, interpolation = lerope / pos_freqs, lerope / (factor * pos_freqs) + + def correction_dim(rotations: float) -> float: + return (ROPE_DIM * math.log(ML4_ORIGINAL_MAX / (rotations * 2 * math.pi))) / ( + 2 * math.log(ML4_THETA) + ) + + low = max(math.floor(correction_dim(ML4_BETA_FAST)), 0) + high = min(math.ceil(correction_dim(ML4_BETA_SLOW)), ROPE_DIM - 1) + ramp = ((torch.arange(half, dtype=torch.float64) - low) / (high - low)).clamp(0, 1) + inv = interpolation * ramp + extrapolation * (1 - ramp) + return _as_table(torch.arange(positions, dtype=torch.float64)[:, None] * inv[None, :]) + + +def _identity_table(positions: int) -> torch.Tensor: + """A NoPE layer's table: cos 1, sin 0 at every position.""" + return _as_table(torch.zeros(positions, ROPE_DIM // 2, dtype=torch.float64)) + + +def _table(spec, seed) -> torch.Tensor: + """The cell's `[max_pos, 64]` fp32 rope table; a yarn table's LeRoPE alpha is drawn from `seed`. + + `table_dtype="bfloat16"` rounds the table's cos and sin through bf16, as the + targets hold theirs (in the activation dtype, widened to fp32 for this op). + """ + max_pos = spec["max_pos"] + if spec["table"] == "yarn": + cpu = torch.Generator().manual_seed(seed) + table = _yarn_table(max_pos, cpu, spec.get("factor", ML4_FACTOR)) + else: + table = _identity_table(max_pos) + if spec.get("table_dtype") == "bfloat16": + table = table.to(torch.bfloat16).float() + return table + + +def _build(spec, seed, table=None): + """`[q, position_ids, cos_sin, wq, weight_scale, q_fp8_out, weights_out]` for one cell. + + Both outputs start poisoned, so a row the kernel skipped cannot pass. + Mistral Large 4's head weights come out of a bf16 projection and are cast to + fp32, so its cells draw bf16-representable `wq`. `cached` starts an `arange` + step at that position, as a chunk after a request's first runs behind its + cached prefix. `tail` draws each row from the table's last `tail_rows` rows + (64 unless given). `first` puts row 0 at position `first` and every other + row at 0: a decode batch whose other rows are one-key dummies (the boot + warm-up's fillers, attention DP's padding rows). `table` passes a table + built once for several calls. + """ + g = torch.Generator(device="cuda").manual_seed(seed) + tokens, max_pos = spec["tokens"], spec["max_pos"] + q = (torch.randn(tokens, HEADS, HEAD_DIM, generator=g, device="cuda") * 2.0).to(torch.bfloat16) + if spec["positions"] == "arange": + cached = spec.get("cached", 0) + assert cached == 0 or cached + tokens <= max_pos, (cached, tokens, max_pos) + pos = (cached + torch.arange(tokens, device="cuda")) % max_pos + elif spec["positions"] == "tail": + rows = spec.get("tail_rows", 64) + pos = max_pos - 1 - torch.randint(0, rows, (tokens,), generator=g, device="cuda") + elif spec["positions"] == "first": + pos = torch.zeros(tokens, dtype=torch.int64, device="cuda") + pos[0] = spec["first"] + else: + pos = torch.randint(0, max_pos, (tokens,), generator=g, device="cuda") + wq = torch.randn(tokens, HEADS, generator=g, device="cuda") + wq = wq.to(torch.bfloat16).float() + if table is None: + table = _table(spec, seed) + q_fp8_out = torch.full((tokens, HEADS, HEAD_DIM), POISON, dtype=torch.uint8, device="cuda") + q_fp8_out = q_fp8_out.view(torch.float8_e4m3fn) + weights_out = torch.full((tokens, HEADS), float("nan"), device="cuda") + return [q, pos.to(torch.int32), table, wq, WEIGHT_SCALE, q_fp8_out, weights_out] + + +def _kw(spec) -> dict: + return dict(is_neox=spec["is_neox"], compiled=_compiled(spec["is_neox"])) + + +def _compare(out, ref) -> None: + """Bit-exact codes and weights. + + The reference reproduces every rounding the kernel makes: the rope FMA + (fp64 rounded once to fp32), the bf16 store of the rotated half, the + power-of-two scale from exponent bits, an exact multiply by its inverse, + the e4m3 cast, and two fp32 products for the weights. Nothing is left to + accumulation order, so the band is zero; a wrong rope mode or table moves + most codes (the rope-mode guard and the identity-table tests measure it). + """ + (q_fp8, weights), (ref_fp8, ref_weights) = out, ref + codes, want = q_fp8.view(torch.uint8), ref_fp8.view(torch.uint8) + bad = codes != want + assert not bool(bad.any()), ( + f"{int(bad.sum())} of {bad.numel()} e4m3 codes differ; first at " + f"{tuple(int(i) for i in bad.nonzero()[0])}" + ) + wbad = weights.view(torch.int32) != ref_weights.view(torch.int32) + assert not bool(wbad.any()), f"{int(wbad.sum())} of {wbad.numel()} weights differ" + + +def _rope_codes_moved(out, ref) -> float: + """Fraction of the rope half's e4m3 codes that differ.""" + got, want = out[0].view(torch.uint8), ref[0].view(torch.uint8) + return float((got[..., :ROPE_DIM] != want[..., :ROPE_DIM]).float().mean()) + + +def _assert_replay_matches_eager(spec, seed) -> None: + """Capture the call at this batch, replay it on new data, and require the eager bytes. + + A capture freezes the launcher, the grid and every pointer; what changes + between replays is the data behind them. So the static inputs take a second + set of queries, positions, table and head weights, the outputs go back to + poison, and the replay must write bitwise what an eager call on that data + writes into buffers of its own. A boot capture replays once per `_replays` + entry, each on data of its own. + """ + static = _build(spec, seed) + kw = _kw(spec) + op(*static, **kw) + stream = torch.cuda.Stream() + stream.wait_stream(torch.cuda.current_stream()) + graph = torch.cuda.CUDAGraph() + with torch.cuda.stream(stream), torch.cuda.graph(graph, stream=stream): + op(*static, **kw) + torch.cuda.current_stream().wait_stream(stream) + for k, replay in enumerate(_replays(spec)): + fresh, eager = _build(replay, seed + 1 + k), _build(replay, seed + 1 + k) + for i in (0, 1, 2, 3): + static[i].copy_(fresh[i]) + static[5].view(torch.uint8).fill_(POISON) + static[6].fill_(float("nan")) + graph.replay() + op(*eager, **kw) + torch.cuda.synchronize() + assert torch.equal(static[5].view(torch.uint8), eager[5].view(torch.uint8)), ( + f"replay {k}: the replay wrote other codes than the eager call" + ) + assert torch.equal(static[6].view(torch.int32), eager[6].view(torch.int32)), ( + f"replay {k}: the replay wrote other weights than the eager call" + ) + + +def _replays(spec) -> list: + """The data a captured cell replays on, in order. + + A boot capture (`positions="first"`: the warm-up batch, its longest dummy in + row 0 beside one-key fillers) replays as serving runs the graph: attention + DP's one live row at the table's last position behind one-key padding rows + at 0, then a full batch at positions spread over the table. Any other + captured cell replays once, on data drawn as its own. + """ + if spec["positions"] == "first": + return [dict(spec, first=spec["max_pos"] - 1), dict(spec, positions="spread")] + return [spec] + + +def _seed(cell) -> int: + return zlib.crc32(cell.why.encode()) % 2**31 + + +def _position_sweep(spec, seed) -> str: + """`position_sweep` (lo, hi): every table row lo .. hi - 1, `tokens` rows per call. + + Call k reads positions lo + k * tokens onward, on fresh queries and head + weights over the one table, and is gated as a cell is: the caller's buffers + written and returned, `q` untouched, codes and weights bitwise the + reference. The table must come out of the sweep unwritten too. Returns a + summary line. + """ + lo, hi = spec["position_sweep"] + rows = spec["tokens"] + assert (hi - lo) % rows == 0 and hi <= spec["max_pos"], (lo, hi, rows) + table, kw = _table(spec, seed), _kw(spec) + table_before = table.clone() + read = torch.zeros(spec["max_pos"], dtype=torch.bool, device="cuda") + for k in range((hi - lo) // rows): + args = _build(dict(spec, cached=lo + k * rows), seed + 1 + k, table=table) + q_before = args[0].clone() + out = op(*args, **kw) + assert out[0] is args[5] and out[1] is args[6], "the op writes the caller's buffers" + assert torch.equal(args[0], q_before), "q must not be written" + _compare(out, op.reference(*args, **kw)) + read[args[1].long()] = True + assert torch.equal(table, table_before), "the rope table must not be written" + assert bool(read[lo:hi].all()), "a table row of the sweep was never read" + return f"{(hi - lo) // rows} calls, every row {lo}..{hi - 1} read, codes and weights bitwise" + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 + spec = cell.spec + if spec.get("graph"): + # Before arming: the guard reads the positions back to the host, which a + # capture forbids. `graph` is True (capture at the cell's rows) or the + # batches to capture, each on draws of its own. + batches = (spec["tokens"],) if spec["graph"] is True else spec["graph"] + for i, batch in enumerate(batches): + _assert_replay_matches_eager(dict(spec, tokens=batch), _seed(cell) + 2 * i) + certifying(op) + if "position_sweep" in spec: + print(f"\n {cell.why}: {_position_sweep(spec, _seed(cell))}") + return + args, kw = _build(spec, seed=_seed(cell)), _kw(spec) + q_before = args[0].clone() + out = op(*args, **kw) + assert out[0] is args[5] and out[1] is args[6], "the op writes the caller's buffers" + assert torch.equal(args[0], q_before), "q must not be written" + _compare(out, op.reference(*args, **kw)) + + +def test_a_fresh_launch_overwrites_every_row() -> None: + """Two calls into the same buffers: the second result is the second input's.""" + spec = dict( + tokens=5, + positions="spread", + max_pos=ML4_MAX_SEQ_LEN, + is_neox=False, + table="yarn", + ) + first, second = _build(spec, seed=31), _build(spec, seed=32) + kw = _kw(spec) + out_fp8, out_w = first[5], first[6] + op(*first, **kw) + op(*second[:5], out_fp8, out_w, **kw) + _compare((out_fp8, out_w), op.reference(*second, **kw)) + + +#: Interleaved calls a non-cell test makes: a NoPE layer's, and a small RoPE layer's. +_IDENTITY = dict(is_neox=False, table="identity", max_pos=ML4_MAX_SEQ_LEN, positions="spread") +_SMALL = dict(is_neox=False, table="yarn", max_pos=ML4_MAX_SEQ_LEN, positions="spread", tokens=8) + + +def test_an_identity_table_is_no_rope() -> None: + """A NoPE layer's call is a plain per-(token, head) ue8m0 quantization of `q`. + + With bf16 subnormals in both halves and a -0 in the rope half, the call is + first bit-exact against the entry's reference, then checked against a + quantization written here from `q` alone: weights bitwise, codes value for + value. The one bitwise difference is the sign of a zero code in the rope + half: the kernel flushes a rope-half subnormal to a signed zero and the + rotation then adds the partner's `x * 0`, so a negative subnormal or -0 + whose partner product is +0 comes out +0 (code 0x00 where plain + quantization writes 0x80). Both decode to zero. And the same call against + the roped reference moves most rope codes, so the NoPE cells would see a + table that rotates. + """ + spec = dict(_IDENTITY, tokens=256) + args, kw = _build(spec, seed=35), _kw(spec) + q = args[0] + tiny = torch.tensor([2.0**-130, -(2.0**-130), 2.0**-127, -(2.0**-133)], dtype=torch.bfloat16) + q[3, 5, :4] = tiny.cuda() + q[3, 5, 70:74] = tiny.cuda() + q[7, 0, :] = tiny.cuda().repeat(HEAD_DIM // 4) + q[3, 5, 8:10] = torch.tensor([1.0, -0.0], dtype=torch.bfloat16).cuda() + out = op(*args, **kw) + _compare(out, op.reference(*args, **kw)) + + row = q.float() + amax = row.abs().amax(dim=-1) + raw = torch.maximum(amax, torch.tensor(1.0e-12, device="cuda")) * torch.tensor( + 1.0 / 448.0, dtype=torch.float32, device="cuda" + ) + # Round up to a power of two through frexp (raw = m * 2**e, 0.5 <= m < 1): + # exact, where CUDA's fp64 log2 of an exact power of two is not. + mantissa, exponent = torch.frexp(raw) + scale = torch.ldexp(torch.ones_like(raw), torch.where(mantissa == 0.5, exponent - 1, exponent)) + codes = (row / scale[..., None]).to(torch.float8_e4m3fn) + weights = (args[3] * scale) * torch.tensor(WEIGHT_SCALE, dtype=torch.float32, device="cuda") + assert torch.equal(out[1].view(torch.int32), weights.view(torch.int32)), "weights differ" + assert torch.equal(out[0].float(), codes.float()), "a code decodes to another value" + got, want = out[0].view(torch.uint8), codes.view(torch.uint8) + bits = q.view(torch.int16) + negative_zero_or_subnormal = ((bits & 0x7F80) == 0) & (bits < 0) + off = got != want + assert not bool(off[..., ROPE_DIM:].any()), "a nope code differs" + assert not bool((off[..., :ROPE_DIM] & ~negative_zero_or_subnormal[..., :ROPE_DIM]).any()), ( + "a rope code differs away from a negative zero or subnormal" + ) + assert int(got[3, 5, 9]) == 0x00 and int(want[3, 5, 9]) == 0x80, "-0 beside a +1 partner" + assert bool((got[off] == 0x00).all() and (want[off] == 0x80).all()), ( + "a differing code is not a zero of the other sign" + ) + + roped = list(args) + roped[2] = _yarn_table(ML4_MAX_SEQ_LEN, torch.Generator().manual_seed(35)) + moved = _rope_codes_moved(out, op.reference(*roped, **kw)) + assert moved > 0.5, f"only {moved:.2f} of the rope codes moved under a rotating table" + + +def test_a_32768_step_sees_a_wrong_table_row_past_9215() -> None: + """The NVFP4 32768-row cells pass only if every row reads its own table row. + + A read wrapped at the FP8 target's 9216 rows, or one position early, is the + slip a kernel or a table sized for the old envelope would make. On the + factor 2048 table each moves at least one rope code in every row from 9216 + up (about 0.6 and 0.3 of them), so the cells' bitwise gate refuses it. + """ + spec = dict( + is_neox=False, + table="yarn", + factor=NVFP4_FACTOR, + max_pos=NVFP4_MAX_SEQ_LEN, + positions="arange", + tokens=NVFP4_MAX_SEQ_LEN, + ) + args, kw = _build(spec, seed=37), _kw(spec) + out = op(*args, **kw) + pos = args[1] + for name, wrong in ( + ("wrapped at 9216", pos % ML4_MAX_SEQ_LEN), + ("one position early", (pos - 1).clamp(min=0)), + ): + wrong_args = list(args) + wrong_args[1] = wrong + ref = op.reference(*wrong_args, **kw) + got, want = out[0].view(torch.uint8), ref[0].view(torch.uint8) + moved = (got[..., :ROPE_DIM] != want[..., :ROPE_DIM]).flatten(1).any(dim=1) + unseen = int((~moved[ML4_MAX_SEQ_LEN:]).sum()) + assert unseen == 0, f"{name}: {unseen} rows from 9216 up would pass the bitwise gate" + + +def test_a_1m_chunk_sees_a_wrong_table_row() -> None: + """The NVFP4 1M chunk cells pass only if every row reads its own row of the 1048576 rows. + + A read wrapped at the delivered target's 32768 rows, or one position early, + is the slip a table or a cached-prefix offset sized for the old envelope + would make. At the last chunk of a 1M prompt (positions 1015808..1048575), + on the factor 2048 table rounded through bf16 as the target holds it, each + moves at least one rope code in every row (about 0.9 and 0.3 of them), so + the cells' bitwise gate refuses it. + """ + spec = dict( + is_neox=False, + table="yarn", + factor=NVFP4_FACTOR, + max_pos=LONG_MAX_SEQ_LEN, + positions="arange", + tokens=NVFP4_MAX_SEQ_LEN, + cached=LONG_MAX_SEQ_LEN - NVFP4_MAX_SEQ_LEN, + table_dtype="bfloat16", + ) + args, kw = _build(spec, seed=38), _kw(spec) + out = op(*args, **kw) + _compare(out, op.reference(*args, **kw)) + pos = args[1] + for name, wrong in ( + ("wrapped at 32768", pos % NVFP4_MAX_SEQ_LEN), + ("one position early", pos - 1), + ): + wrong_args = list(args) + wrong_args[1] = wrong + ref = op.reference(*wrong_args, **kw) + got, want = out[0].view(torch.uint8), ref[0].view(torch.uint8) + moved = got[..., :ROPE_DIM] != want[..., :ROPE_DIM] + unseen = int((~moved.flatten(1).any(dim=1)).sum()) + assert unseen == 0, f"{name}: {unseen} of the chunk's rows would pass the bitwise gate" + print(f"\n {name}: {float(moved.float().mean()):.3f} of the rope codes move") + + +def test_guard_refuses_a_rope_mode_the_launcher_was_not_built_for() -> None: + args = _build(_SMALL, seed=33) + neox = _compiled(True) + # Unguarded, the NeoX launcher rotates interleaved queries the NeoX way: + # well-formed rows whose rope codes are mostly wrong. + out = op(*args, is_neox=False, compiled=neox) + moved = _rope_codes_moved(out, op.reference(*args, is_neox=False, compiled=neox)) + assert moved > 0.5, f"only {moved:.2f} of the rope codes moved" + with pytest.raises(AssertionError, match="compiled for is_neox"): + with validating(op): + op(*_build(_SMALL, seed=33), is_neox=False, compiled=neox) + + +def test_guard_refuses_a_position_past_the_rope_table() -> None: + # Unguarded, the kernel reads past the table: not driven here. + args = _build(_SMALL, seed=34) + args[1][2] = args[2].shape[0] + with pytest.raises(AssertionError, match="rope table"): + with validating(op): + op(*args, **_kw(_SMALL)) + + +def _strided_head_weights(args): + """Head weights as a `[T, 160]` wk|weights_proj output leaves them: its last 32 columns.""" + kw = torch.zeros(args[3].shape[0], HEAD_DIM + HEADS, device="cuda") + kw[:, HEAD_DIM:] = args[3] + args[3] = kw[:, HEAD_DIM:] + + +def _cast(index: int, dtype: torch.dtype): + return lambda a: a.__setitem__(index, a[index].to(dtype)) + + +@pytest.mark.parametrize( + "mutate, error, message", + [ + (_cast(3, torch.bfloat16), TypeError, r"wq must be torch\.float32, got torch\.bfloat16"), + (_strided_head_weights, ValueError, r"wq must be contiguous, got stride \(160, 1\)"), + (_cast(2, torch.bfloat16), TypeError, r"cos_sin must be torch\.float32"), + (_cast(1, torch.int64), TypeError, r"position_ids must be torch\.int32"), + ], + ids=["bf16 head weights", "strided head weights", "bf16 table", "int64 positions"], +) +def test_the_launcher_refuses_what_it_was_not_compiled_for(mutate, error, message) -> None: + """The note's refusals, pinned: raised before any launch, so both outputs stay poisoned.""" + args = _build(_SMALL, seed=36) + mutate(args) + with pytest.raises(error, match=message): + op(*args, **_kw(_SMALL)) + torch.cuda.synchronize() + assert bool((args[5].view(torch.uint8) == POISON).all()), "a refused call wrote codes" + assert bool(args[6].isnan().all()), "a refused call wrote weights" diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_cute_dsl_radix_topk_decode.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_cute_dsl_radix_topk_decode.py new file mode 100644 index 000000000000..f1c6891720cc --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_cute_dsl_radix_topk_decode.py @@ -0,0 +1,302 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the cute_dsl_radix_topk_decode entry, and its equivalence to the +trtllm indexer_topk_decode op it replaces.""" + +import struct +import zlib + +import pytest +import torch + +if not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 3): + pytest.skip("the CuTe DSL kernel is certified on sm_103", allow_module_level=True) + +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_radix_topk_decode import ( # noqa: E402, E501 + TOP_K, + blocks_per_row, + build, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_radix_topk_decode import ( # noqa: E402, E501 + cute_dsl_radix_topk_decode as op, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.indexer_topk_decode import ( # noqa: E402, E501 + indexer_topk_decode as old_op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402,F401 — needs the path declared above + +#: Written past each row's causal end. A kernel that read past the end would select these. +POISON = 1.0e9 +#: What `indices` holds before a call; a slot the op should not write keeps it. +UNWRITTEN = -7 +#: The CUDA-graph padding dummy's KV length (one token, no speculation). +PAD_SEQ_LEN = 1 +#: key(-FLT_MAX), the op's padding entries' key. +PAD_KEY = 0xFF7FFFFF +#: Blocks per row the DSA metadata's radix scratch reserves for the trtllm op. +AUX_BLOCKS = 10 + + +def _f32(bits: int) -> float: + return struct.unpack(" int: + return max(0, min(width, seq_lens[row // next_n] - next_n + row % next_n + 1)) + + +def _scores(kind: str, row: int, end: int, g: torch.Generator) -> torch.Tensor: + """One row's causal scores (see the cells' `scores`).""" + if kind == "perm": + return torch.randperm(end, generator=g).float() + if kind == "indexer": + # Head-weighted relu sums: about half the columns have every head at relu's floor. + s = (torch.randn(8, end, generator=g) - 2.0).clamp_min_(0) + w = torch.randn(8, 1, generator=g) + return (s * w).sum(0) + if kind == "ties": + if row % 3 == 0: + return torch.randint(0, 16, (end,), generator=g).float() + zeros = torch.zeros(end) + if row % 3 == 2: + zeros[torch.rand(end, generator=g) < 0.5] = -0.0 + return zeros + if kind == "wide": + exponent = torch.randperm(end, generator=g).double() / max(end, 1) * 200.0 - 100.0 + sign = torch.where(torch.rand(end, generator=g) < 0.5, -1.0, 1.0).double() + return (sign * torch.exp2(exponent)).float() + if kind == "special": + # A few finite scores; the rest special patterns, so the op's padding decides. + vals = torch.randperm(end, generator=g).float() - end / 2 + pick = torch.randint(0, len(SPECIAL), (end,), generator=g) + bits = torch.tensor(SPECIAL, dtype=torch.int64)[pick] + special = (bits - (bits >= 2**31).long() * 2**32).to(torch.int32).view(torch.float32) + frac = 0.7 if row % 2 == 0 else 0.95 + return torch.where(torch.rand(end, generator=g) < frac, special, vals) + if kind == "negnan": + # Distinct scores (even rows) or a floor of +0.0 under 1000 scores (odd rows), with + # negative NaNs (two payloads) and a few positive ones sprinkled through. + if row % 2 == 0: + vals = torch.randperm(end, generator=g).float() - end / 3 + else: + vals = torch.zeros(end) + vals[torch.randperm(end, generator=g)[:1000]] = torch.rand(1000, generator=g) + 1.0 + nan_bits = torch.tensor([0xFFC00000, 0xFF800001, 0x7FC00000], dtype=torch.int64) + pick = nan_bits[torch.randint(0, 3, (end,), generator=g)] + nans = (pick - (pick >= 2**31).long() * 2**32).to(torch.int32).view(torch.float32) + return torch.where(torch.rand(end, generator=g) < 0.03, nans, vals) + if kind == "crowded": + if row % 2 == 0: + # 1000 positive scores over a floor of +0.0: the cut lands in a group of equals. + vals = torch.zeros(end) + vals[torch.randperm(end, generator=g)[:1000]] = torch.rand(1000, generator=g) + 1.0 + return vals + # Distinct scores in [1.0, 1.25): one 11-bit top digit for all of them. + return 1.0 + torch.randperm(end, generator=g).float() / (4.0 * max(end, 1)) + raise ValueError(kind) + + +def _logits(spec, seq_lens, g: torch.Generator) -> torch.Tensor: + next_n, width = spec["next_n"], spec["width"] + rows = len(seq_lens) * next_n + parent = POISON + torch.arange(width, dtype=torch.float32).repeat(rows, 1) + for r in range(rows): + end = _end(seq_lens, next_n, r, width) + parent[r, :end] = _scores(spec.get("scores", "perm"), r, end, g) + return parent.cuda() + + +def _keys(values: torch.Tensor) -> torch.Tensor: + bits = values.contiguous().view(torch.int32).long() & 0xFFFFFFFF + return torch.where(bits >= 0x80000000, bits, (~bits) & 0x7FFFFFFF) + + +def _compare(out, ref, logits, seq_lens, next_n) -> None: + """Equal up to exact ties: per row, the same multiset of keys over all top_k slots. + + A -1 counts as a pad, keyed like -FLT_MAX. Columns must be distinct and causal. A + selection that differs only in which of several bit-identical scores (or a pad and a + -FLT_MAX column) it took at the cut passes; any other difference changes the multiset. + """ + assert out.dtype == ref.dtype == torch.int32 and out.shape == ref.shape + width = logits.shape[1] + keys = _keys(logits) + for r in range(out.shape[0]): + end = _end(seq_lens, next_n, r, width) + sets = [] + for name, sel in (("op", out[r]), ("reference", ref[r])): + cols = sel[sel >= 0].long() + assert bool(((sel >= 0) | (sel == -1)).all()), f"{name} row {r}: unwritten slots" + assert cols.unique().numel() == cols.numel(), f"{name} row {r}: repeated columns" + assert bool((cols < end).all()), f"{name} row {r}: a column past the end {end}" + pads = torch.full((int((sel == -1).sum()),), PAD_KEY, device=keys.device) + k = torch.cat([keys[r, cols], pads]) + sets.append(k.sort().values) + assert torch.equal(sets[0], sets[1]), f"row {r} (end {end}): a different selection" + + +def _old(logits, lens, indices, next_n): + """The trtllm op, as the targets called it, with the DSA metadata's radix scratch.""" + rows = logits.shape[0] + shape = (max(16, rows), AUX_BLOCKS, TOP_K) + aux_i = torch.empty(shape, dtype=torch.int32, device="cuda") + aux_l = torch.empty(shape, dtype=torch.float32, device="cuda") + return old_op(logits, lens, indices, next_n, TOP_K, 1, aux_i, aux_l) + + +def _drive(spec, seq_lens, logits, kw) -> None: + rows = logits.shape[0] + lens = torch.tensor(seq_lens, dtype=torch.int32, device="cuda") + tail_of = spec.get("tail_of") + if tail_of is None: + buf = torch.full((rows, TOP_K), UNWRITTEN, dtype=torch.int32, device="cuda") + indices = buf + else: + # A mixed step: the context rows' selection sits ahead of these rows in one buffer. + buf = torch.full((tail_of, TOP_K), UNWRITTEN, dtype=torch.int32, device="cuda") + indices = buf[tail_of - rows :] + out = op(logits, lens, indices, **kw) + assert out is indices, "the op writes the caller's buffer and returns it" + _compare(out, op.reference(logits, lens, indices, **kw), logits, seq_lens, kw["next_n"]) + if tail_of is not None: + assert bool((buf[: tail_of - rows] == UNWRITTEN).all()), "a row ahead was written" + + +def _drive_graph(spec, g, kw) -> None: + """Capture once per batch and replay on new scores and seq_lens.""" + next_n, width = spec["next_n"], spec["width"] + for batch in spec["graph_batches"]: + capture_lens = spec["seq_lens"][:batch] + logits = _logits(spec, capture_lens, g) + lens = torch.tensor(capture_lens, dtype=torch.int32, device="cuda") + indices = torch.full((batch * next_n, TOP_K), UNWRITTEN, dtype=torch.int32, device="cuda") + side = torch.cuda.Stream() + side.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(side): + op(logits, lens, indices, **kw) + torch.cuda.current_stream().wait_stream(side) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + op(logits, lens, indices, **kw) + for _ in range(spec["replays"]): + new_lens = torch.randint(1, width + 1, (batch,), generator=g).tolist() + if batch > 2: + new_lens[-1] = PAD_SEQ_LEN + new = _logits(spec, new_lens, g) + logits.copy_(new) + lens.copy_(torch.tensor(new_lens, dtype=torch.int32)) + indices.fill_(UNWRITTEN) + graph.replay() + torch.cuda.synchronize() + _compare(indices, op.reference(new, lens, indices, **kw), new, new_lens, next_n) + + +def _cell_inputs(cell): + spec = cell.spec + g = torch.Generator().manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + return spec, g + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 + certifying(op) + spec, g = _cell_inputs(cell) + kw = dict(next_n=spec["next_n"], compiled=_compiled(spec["next_n"])) + if "graph_batches" in spec: + _drive_graph(spec, g, kw) + return + logits = _logits(spec, spec["seq_lens"], g) + for batch in spec.get("batches", (len(spec["seq_lens"]),)): + _drive(spec, spec["seq_lens"][:batch], logits[: batch * spec["next_n"]], kw) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_selects_the_trtllm_ops_set(cell) -> None: + """Old against new on identical scores: the trtllm indexer_topk_decode op (the family's + indexer_topk_decode entry) and this entry select the same entries up + to exact ties, -1 padding included, at every cell's shapes and scores.""" + spec, g = _cell_inputs(cell) + next_n = spec["next_n"] + kw = dict(next_n=next_n, compiled=_compiled(next_n)) + batches = spec.get("graph_batches", spec.get("batches", (len(spec["seq_lens"]),))) + for batch in batches: + seq_lens = spec["seq_lens"][:batch] + logits = _logits(spec, seq_lens, g) + lens = torch.tensor(seq_lens, dtype=torch.int32, device="cuda") + rows = logits.shape[0] + new = op(logits, lens, torch.full((rows, TOP_K), UNWRITTEN, dtype=torch.int32, + device="cuda"), **kw) # fmt: skip + old = _old(logits, lens, torch.full_like(new, UNWRITTEN), next_n) + _compare(new, old, logits, seq_lens, next_n) + + +def test_short_rows_list_their_columns_in_order() -> None: + """A row no longer than top_k lists its columns in order, then -1.""" + spec = dict(next_n=1, width=32768) + logits = _logits(spec, [5, 40], torch.Generator().manual_seed(11)) + lens = torch.tensor([5, 40], dtype=torch.int32, device="cuda") + out = torch.full((2, TOP_K), UNWRITTEN, dtype=torch.int32, device="cuda") + op(logits, lens, out, next_n=1, compiled=_compiled()) + assert out[0].tolist() == list(range(5)) + [-1] * (TOP_K - 5) + assert out[1].tolist() == list(range(40)) + [-1] * (TOP_K - 40) + + +def test_padding_outranks_minus_inf_as_in_the_op() -> None: + """Where the op's pads exist, a column scored -inf or -NaN loses to them: a short row + drops it, and a long row of 16 rows (9 blocks of 2366 columns, no pads) keeps it.""" + width = 32768 + logits = torch.full((16, width), POISON, device="cuda") + logits[0, :100] = torch.arange(100, dtype=torch.float32, device="cuda") + logits[0, 50] = float("-inf") + logits[0, 60] = _f32(0xFFC00000) + logits[1, :21300] = float("-inf") + logits[1, 21300:21310] = 1.0 + lens = torch.tensor([100, 21300 + 10] + [1] * 14, dtype=torch.int32, device="cuda") + assert blocks_per_row(16, width, torch.cuda.get_device_properties(0).multi_processor_count) == 9 + out = torch.full((16, TOP_K), UNWRITTEN, dtype=torch.int32, device="cuda") + op(logits, lens, out, next_n=1, compiled=_compiled()) + assert set(out[0].tolist()) == set(range(100)) - {50, 60} | {-1} + assert int((out[1] >= 0).sum()) == TOP_K + old = _old(logits, lens, torch.full_like(out, UNWRITTEN), 1) + _compare(out, old, logits, lens.tolist(), 1) + + +def test_refuses_a_shape_the_op_runs_as_one_block() -> None: + """Under 4096 columns the op takes its single-block path, which this does not reproduce.""" + logits = torch.zeros((1, 2048), device="cuda") + lens = torch.tensor([2048], dtype=torch.int32, device="cuda") + out = torch.empty((1, TOP_K), dtype=torch.int32, device="cuda") + with pytest.raises(ValueError, match="one block per row"): + op(logits, lens, out, next_n=1, compiled=_compiled()) + + +def test_next_n_and_compiled_are_keyword_only() -> None: + logits = torch.zeros((1, 4096), device="cuda") + lens = torch.tensor([4096], dtype=torch.int32, device="cuda") + out = torch.empty((1, TOP_K), dtype=torch.int32, device="cuda") + with pytest.raises(TypeError): + op(logits, lens, out, 1, _compiled()) + + +def test_guard_refuses_a_causal_end_past_the_row() -> None: + """The kernel would clamp it to the row and answer for the truncated row.""" + logits = torch.zeros((1, 32768), device="cuda") + lens = torch.tensor([32769], dtype=torch.int32, device="cuda") + out = torch.empty((1, TOP_K), dtype=torch.int32, device="cuda") + with pytest.raises(AssertionError, match="exceeds the 32768-column logits"): + with validating(op): + op(logits, lens, out, next_n=1, compiled=_compiled()) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_ds_mla_dequant_gather.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_ds_mla_dequant_gather.py new file mode 100644 index 000000000000..8d896ebb8c59 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_ds_mla_dequant_gather.py @@ -0,0 +1,199 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the ds_mla_dequant_gather entry: bitwise against the reference +(e4m3 times a power of two is exact in bf16, and the rope half is a copy).""" + +import math +import zlib + +import pytest +import torch + +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ds_mla_dequant_gather import ( + ds_mla_dequant_gather as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402,F401 — needs the path declared above + +assert torch.cuda.is_available(), "ds_mla_dequant_gather requires a CUDA device" + +#: Every pool byte the cell's requests do not hold: NaN as an e4m3 code, as an fp32 tile scale +#: and as a bf16 rope value, so a read from another page, owner row or offset shows in the bits. +_UNHELD = 0xFF +#: The int16 bits of every output-buffer element the call must leave alone (a bf16 NaN). +_UNWRITTEN = -1 + + +def ds_mla_rows(n: int, g: torch.Generator, nope: bool = False) -> torch.Tensor: + """`n` uint8 [656] rows on g's device: e4m3 codes drawn over (-448, 448), power-of-two + tile scales 2^-13..2^0, bf16 rope (zero for a NoPE layer). The shared tests' + `_ds_mla_pool.ds_mla_rows`, drawing where the generator lives.""" + dev = g.device + codes = (torch.rand(n, 512, generator=g, device=dev) * 2 - 1) * 448 + codes = codes.to(torch.float8_e4m3fn).view(torch.uint8) + scale = torch.exp2(torch.randint(-13, 1, (n, 4), generator=g, device=dev).float()) + if nope: + rope = torch.zeros(n, 64, device=dev) + else: + rope = torch.randn(n, 64, generator=g, device=dev) * 3 + return torch.cat([codes, scale.view(torch.uint8), rope.to(torch.bfloat16).view(torch.uint8)], 1) + + +def _build(spec, seed): + """The call's arguments, built on the device, and the buffer `out` leads. + + The pool is `_UNHELD` but for the owner `row` of the pages the requests hold, which + hold drawn rows: distinct pages anywhere in the pool, or with `far` its top ones. The + block table has `table_rows` x `table_width` int32 entries (the requests' rows first, + as the engine's context requests, a rank's under attention DP, lead md.block_table), + random block ids past the requests' pages. `out` is the first N rows of a + `_UNWRITTEN` buffer of `out_rows`. + """ + dev = torch.device("cuda") + g = torch.Generator(device=dev).manual_seed(seed) + tpb, blocks, owners = spec["tokens_per_block"], spec["blocks"], spec["owners"] + totals = [c + n for c, n in zip(spec["cached"], spec["new"])] + counts = [math.ceil(t / tpb) for t in totals] + need = sum(counts) + if spec.get("far"): + pages = blocks - need + torch.randperm(need, generator=g, device=dev) + else: + pages = torch.randperm(blocks, generator=g, device=dev)[:need] + rows_, width = spec.get("table_rows", len(totals)), spec.get("table_width", max(counts)) + table = torch.randint(0, blocks, (rows_, width), generator=g, device=dev, dtype=torch.int32) + used = 0 + for r, k in enumerate(counts): + table[r, :k] = pages[used : used + k] + used += k + pool = torch.full((blocks, owners, tpb, 656), _UNHELD, dtype=torch.uint8, device=dev) + pool[pages, spec["row"]] = ds_mla_rows(need * tpb, g, spec.get("nope", False)).view( + need, tpb, 656 + ) + n = sum(totals) + lens = torch.tensor(totals, device=dev) + kv_req = torch.repeat_interleave( + torch.arange(len(totals), dtype=torch.int32, device=dev), lens, output_size=n + ) + kv_pos = (torch.arange(n, device=dev) - (torch.cumsum(lens, 0) - lens)[kv_req]).int() + buf = torch.full((spec.get("out_rows", n), 576), _UNWRITTEN, dtype=torch.int16, device=dev) + buf = buf.view(torch.bfloat16) + return (pool, spec["row"], kv_req, kv_pos, table, buf[:n]), buf + + +def _bits(t: torch.Tensor) -> torch.Tensor: + return t.view(torch.int16) + + +def _assert_bitwise(out: torch.Tensor, ref: torch.Tensor) -> None: + assert out.shape == ref.shape and out.dtype == ref.dtype, (out.shape, ref.shape) + bad = _bits(out) != _bits(ref) + assert not bad.any(), f"{int(bad.sum())} of {bad.numel()} elements differ" + + +def _untouched(rows: torch.Tensor) -> bool: + return bool((_bits(rows) == _UNWRITTEN).all()) + + +def _prefix_counts(n: int) -> list[int]: + """The row counts below `n` the prefix check drives: 1..64, 2**k - 1 / 2**k / 2**k + 1 + from 2**7, strides of 4093 and of 4099 (the tep4 and the tp4 cells' certifying runs + each drove one; the tep8 cells' drove 4093), and n - 1.""" + counts = {*range(1, 65), *(2**k + d for k in range(7, 20) for d in (-1, 0, 1))} + counts |= {*range(4093, n, 4093), *range(4099, n, 4099), n - 1} + return sorted(m for m in counts if 0 < m < n) + + +def _check_prefixes(args, full: torch.Tensor) -> int: + """Each count of `_prefix_counts(N)` gathers the leading kv_req / kv_pos rows into a fresh + `_UNWRITTEN` buffer: its rows are the full call's leading rows bitwise and the buffer's + later rows stay unwritten. One sync for all counts; a failure re-runs them one by one to + name the count. Returns how many counts ran.""" + pool, row, kv_req, kv_pos, table, _ = args + scratch = torch.empty_like(full) + counts = _prefix_counts(full.shape[0]) + bad = torch.zeros((), dtype=torch.bool, device=full.device) + for m in counts: + _bits(scratch).fill_(_UNWRITTEN) + got = op(pool, row, kv_req[:m], kv_pos[:m], table, scratch[:m]) + bad |= (_bits(got) != _bits(full[:m])).any() | (_bits(scratch[m:]) != _UNWRITTEN).any() + if bad.item(): + for m in counts: + _bits(scratch).fill_(_UNWRITTEN) + op(pool, row, kv_req[:m], kv_pos[:m], table, scratch[:m]) + _assert_bitwise(scratch[:m], full[:m]) + assert _untouched(scratch[m:]), f"N = {m}: rows past N were written" + return len(counts) + + +_CELLS = [ + pytest.param(c, id=c.why[:60], marks=[pytest.mark.large] if c.spec.get("large") else []) + for c in op.CELLS +] + + +@pytest.mark.parametrize("cell", _CELLS) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — fixture by name + certifying(op) + args, buf = _build(cell.spec, zlib.crc32(cell.why.encode())) + n = args[2].shape[0] + out = op(*args) + ref = op.reference(*args) + assert out.data_ptr() == buf.data_ptr() + _assert_bitwise(out, ref) + assert not torch.isnan(ref.float()).any(), "the reference read a page the requests lack" + del ref + assert _untouched(buf[n:]), "the call wrote rows of the buffer past N" + if cell.spec.get("prefixes"): + assert _check_prefixes(args, out) > 64 + + +def test_is_valid_refuses_a_bf16_pool() -> None: + args, _ = _build(dict(cached=[0], new=[64], blocks=4, owners=2, tokens_per_block=32, row=1), 1) + args = list(args) + args[0] = args[0].view(torch.bfloat16) + with pytest.raises(ValueError, match="uint8"): + op.is_valid(*args) + + +def test_is_valid_refuses_an_out_shorter_than_kv_req() -> None: + """Unguarded, an `out` of 40 rows for 64 kv_req entries comes back right for its 40 rows, + and the kernel writes the other 24 past its end: here into the rest of the 64-row buffer + the test owns, where they are the reference's rows 40..63. Guarded, the call raises + before the kernel runs.""" + args, buf = _build( + dict(cached=[5], new=[59], blocks=6, owners=3, tokens_per_block=32, row=2), 2 + ) + pool, row, kv_req, kv_pos, table, _ = args + ref = op.reference(*args) + short = op(pool, row, kv_req, kv_pos, table, buf[:40]) + assert short.shape[0] == 40 + _assert_bitwise(short, ref[:40]) + _assert_bitwise(buf[40:], ref[40:]) + _bits(buf).fill_(_UNWRITTEN) + with validating(op), pytest.raises(ValueError, match="40 rows for 64 kv_req entries"): + op(pool, row, kv_req, kv_pos, table, buf[:40]) + assert _untouched(buf), "refused, yet the kernel wrote" + with validating(op): + _assert_bitwise(op(*args), ref) + + +def test_is_valid_refuses_48_out_rows_for_64_kv_req_entries() -> None: + """Unguarded, an `out` of 48 rows for 64 kv_req entries comes back right for its 48 rows + while the kernel writes the other 16 past its end: here into the rest of the 64-row + buffer the test owns, so the stray rows land where the test can read them.""" + args, buf = _build( + dict(cached=[0], new=[64], blocks=4, owners=2, tokens_per_block=32, row=1), 2 + ) + pool, row, kv_req, kv_pos, table, _ = args + ref = op.reference(*args) + short = op(pool, row, kv_req, kv_pos, table, buf[:48]) + assert short.shape[0] == 48 + _assert_bitwise(short, ref[:48]) + _assert_bitwise(buf[48:], ref[48:]) + _bits(buf).fill_(_UNWRITTEN) + with validating(op), pytest.raises(ValueError, match="48 rows for 64 kv_req"): + op(pool, row, kv_req, kv_pos, table, buf[:48]) + assert _untouched(buf), "refused, yet the kernel ran" + with validating(op): + _assert_bitwise(op(*args), ref) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_ds_mla_rope_append.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_ds_mla_rope_append.py new file mode 100644 index 000000000000..aaf1ba0fae4e --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_ds_mla_rope_append.py @@ -0,0 +1,659 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the ds_mla_rope_append entry. + +Every cell gates the pool bitwise against the entry's reference on the codes and the tile +scales and on every byte the call must not touch; the rope halves (key in the pool, query in +place) within one bf16 unit in the last place, the room an FMA in the rotation leaves. + +The entry's cells draw their pool on the host and compare all of it. The `_ML4_1M`, +`_ML4_1M_DEP4` and `_ML4_1M_TEP8` cells build theirs on the device, because their pools reach +195.0 GB (142.7 GB at tep4, tp4 and dep4): every byte a sentinel but the blocks the call +writes (one per request page, random bytes). The reference runs on a copy of just those +blocks under the block table renumbered to them; after the call they must equal it, and once +they are set back to the sentinel the whole pool must read as the sentinel, so a write +anywhere else shows. Their rope table is built on the device as the target builds its +`_rope.indexer`, 1048576 rows, with a seeded LeRoPE alpha in place of the checkpoint's. +""" + +import math +import zlib +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ds_mla_rope_append import ( + ROPE_OFFSET, + TOKEN_BYTES, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ds_mla_rope_append import ( + ds_mla_rope_append as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402,F401 — needs the path declared above + +assert torch.cuda.is_available(), "ds_mla_rope_append requires a CUDA device" + +_Q_HEADS = 64 # the absorbed query's kernel width; the cell's heads are its first ones +_POSITIONS = 32768 + 64 + + +def _every_bf16_amax(tokens: int) -> torch.Tensor: + """`tokens` x 4 tiles whose amaxes are every finite non-negative bf16 value once (the rest + of each tile below it): zero, the subnormals, the 1e-4 floor, powers of two, the largest.""" + bits = torch.arange(0, 0x7F80, dtype=torch.int32) # +0 .. largest finite + amax = bits.to(torch.int16).view(torch.bfloat16).float() + n = tokens * 4 + amax = torch.cat([amax, amax[: max(0, n - amax.numel())]])[:n] + g = torch.Generator().manual_seed(7) + tiles = torch.rand(n, 128, generator=g) * 2 - 1 + tiles = tiles * amax[:, None] + tiles[torch.arange(n), torch.randint(0, 128, (n,), generator=g)] = amax + return tiles.view(tokens, 512) + + +def _build(spec, seed: int): + g = torch.Generator().manual_seed(seed) + t, tpb, blocks, owners = ( + spec["tokens"], + spec["tokens_per_block"], + spec["blocks"], + spec["owners"], + ) + if spec["phase"] == "decode": + # One request per row; its token's page is its own, every other entry a stale page. + width = math.ceil(_POSITIONS / tpb) + pos = torch.randint(0, width * tpb, (t,), generator=g) + req = torch.arange(t) + table = torch.randint(0, blocks, (t, width), generator=g) + table[req, pos // tpb] = torch.randperm(blocks, generator=g)[:t] + else: + pos = spec["cached"] + torch.arange(t) + req = torch.zeros(t, dtype=torch.long) + width = math.ceil((spec["cached"] + t) / tpb) + table = torch.randperm(blocks, generator=g)[:width].view(1, width) + if spec.get("amax") == "every_bf16": + nope = _every_bf16_amax(t) + else: + mag = torch.exp2(torch.rand(t, 4, 1, generator=g) * 24 - 16) + nope = (torch.randn(t, 4, 128, generator=g) * mag).view(t, 512) + lat = torch.cat([nope, torch.randn(t, 64, generator=g) * 4], 1).to(torch.bfloat16) + ang = torch.arange(_POSITIONS).float()[:, None] * torch.rand(32, generator=g)[None] + table_cs = torch.cat([ang.cos(), ang.sin()], 1).to(torch.bfloat16).float() + pool = torch.randint(0, 256, (blocks, owners, tpb, 656), dtype=torch.uint8, generator=g) + q = None + if spec["heads"]: + q = torch.randn(t, _Q_HEADS, 576, generator=g).to(torch.bfloat16) + cuda = lambda x: None if x is None else x.cuda() # noqa: E731 + return dict( + latent=cuda(lat), + q_full=cuda(q), + pos=cuda(pos.to(torch.int32)), + table=cuda(table_cs), + req_idx=cuda(req.to(torch.int32)), + block_table=cuda(table.to(torch.int32)), + pool=cuda(pool), + row=spec["row"], + rope=spec["rope"], + heads=spec["heads"], + ) + + +def _args(b): + q = None if b["q_full"] is None else b["q_full"][:, : b["heads"]] + return (b["latent"], q, b["pos"], b["table"], b["req_idx"], b["block_table"], b["pool"], + b["row"], b["rope"]) # fmt: skip + + +def _ulp_close(a: torch.Tensor, b: torch.Tensor) -> None: + a, b = a.float(), b.float() + ulp = torch.exp2(torch.floor(torch.log2(b.abs().clamp_min(2.0**-126))) - 7) + assert bool(((a - b).abs() <= ulp).all()), float(((a - b).abs() / ulp).max()) + + +def _check(b, pool_before, q_before) -> None: + args = list(_args(b)) + args[6] = pool_before.clone() + if args[1] is not None: + args[1] = q_before[:, : b["heads"]].clone() + ref_pool, ref_q = op.reference(*args) + pool = b["pool"] + # The written rows: their codes and scales bitwise, their rope halves within an ulp. + # Every other byte of the pool bitwise (it starts as random bytes, so a stray write shows). + tpb = pool.shape[2] + pos = b["pos"].long() + block = b["block_table"][b["req_idx"].long(), pos // tpb].long() + written = (block, torch.full_like(block, b["row"]), pos % tpb) + assert torch.equal(pool[..., :ROPE_OFFSET], ref_pool[..., :ROPE_OFFSET]), ( + f"{int((pool[..., :ROPE_OFFSET] != ref_pool[..., :ROPE_OFFSET]).sum())} code/scale " + "bytes differ" + ) + _ulp_close( + pool[written][:, ROPE_OFFSET:].contiguous().view(torch.bfloat16), + ref_pool[written][:, ROPE_OFFSET:].contiguous().view(torch.bfloat16), + ) + untouched, ref_untouched = pool.clone(), ref_pool.clone() + untouched[written] = 0 + ref_untouched[written] = 0 + assert torch.equal(untouched, ref_untouched), "bytes outside the written rows changed" + if b["q_full"] is not None: + h = b["heads"] + _ulp_close(b["q_full"][:, :h], ref_q) + assert torch.equal(b["q_full"][:, h:], q_before[:, h:]), ( + "heads past the call's were written" + ) + + +# -------------------------------------------- the _ML4_1M, _ML4_1M_DEP4 and _ML4_1M_TEP8 cells + +#: The checkpoint's rope_parameters: YaRN, rope_theta 1e4, factor 2048, original +#: max_position_embeddings 512, beta_fast 32, beta_slow 1. +_YARN = dict(theta=1.0e4, factor=2048.0, original=512.0, beta_fast=32.0, beta_slow=1.0) +#: Under `positions` "tail" a decode row's position is one of the table's last this many. +_TAIL = 64 +_SENTINEL = 0x5A +_SENTINEL_WORD = int.from_bytes(bytes([_SENTINEL]) * 8, "little", signed=True) +_TABLES: dict = {} + + +def _indexer_table(rows: int) -> torch.Tensor: + """The target's `_rope.indexer` at `rows` positions, built as `_rope_tables` builds it: + YaRN's blend of the extrapolated and the interpolated inverse frequency, both scaled by + LeRoPE's exp(alpha), over a ramp taken without LeRoPE (fp32); the fp32 angle grid; cos + and sin rounded through bf16; `[cos(32) | sin(32)]` fp32 on the device. alpha is a + checkpoint tensor, so a seeded draw stands in for it. Built once per process (256 MiB at + 1048576 rows).""" + if rows not in _TABLES: + y, dim = _YARN, 64 + pos_freqs = y["theta"] ** (torch.arange(0, dim, 2, dtype=torch.float32) / dim) + alpha = 0.1 * torch.randn(dim // 2, generator=torch.Generator().manual_seed(11)) + lerope = torch.exp(alpha) + extrapolated = 1.0 / pos_freqs * lerope + interpolated = 1.0 / (y["factor"] * pos_freqs) * lerope + + def correction_dim(rotations: float) -> float: + turns = y["original"] / (rotations * 2 * math.pi) + return dim * math.log(turns) / (2 * math.log(y["theta"])) + + low = max(math.floor(correction_dim(y["beta_fast"])), 0) + high = min(math.ceil(correction_dim(y["beta_slow"])), dim - 1) + ramp = ((torch.arange(dim // 2, dtype=torch.float32) - low) / (high - low)).clamp(0, 1) + extrapolation = 1 - ramp + inv = interpolated * (1 - extrapolation) + extrapolated * extrapolation + angle = torch.einsum("i,j -> ij", torch.arange(rows, dtype=torch.float32), inv) + cos = angle.cos().to(torch.bfloat16).float() + sin = angle.sin().to(torch.bfloat16).float() + _TABLES[rows] = torch.cat([cos, sin], dim=1).cuda().contiguous() + return _TABLES[rows] + + +def _requests(spec, gen: torch.Generator, chunk: int = 0, batch: int = 0) -> list: + """`(cached, new)` per request of the call, in its row order; request i is block-table + row i.""" + phase = spec["phase"] + if phase == "context": + if "sweep_positions" in spec: + return [(spec["sweep_positions"][0] + chunk * spec["tokens"], spec["tokens"])] + return [(spec["cached"], spec["tokens"])] + if phase == "packed": + return [tuple(c) for c in spec["contexts"]] + if phase == "mixed": + return [tuple(c) for c in spec["contexts"]] + [(p, 1) for p in spec["decode_positions"]] + assert phase == "decode" and spec["positions"] == "tail", spec + rows = batch or spec["tokens"] + top = spec["max_pos"] - 1 + pos = (top - torch.randint(0, _TAIL, (rows,), generator=gen)).tolist() + pos[int(torch.randint(0, rows, (1,), generator=gen))] = top + return [(p, 1) for p in pos] + + +def _step(spec, reqs: list, gen: torch.Generator, gpu: torch.Generator, pool, table, pad=0): + """One call's inputs. Each request page the call writes gets a block of its own (under + `far` the pool's top blocks, the step's last position on its last block); every other + block-table entry points at a block the call does not own. The last `pad` requests are + attention DP's padding dummy repeated: one row each, each its own block-table row holding + the first one's pages, and one latent and query row repeated, so they all write one slot. + The pool is reset to the sentinel and the owned blocks drawn at random; `compact` is the + block table renumbered to the owned blocks in `own` order, for the reference.""" + tpb, width, blocks = spec["tokens_per_block"], spec["table_width"], spec["blocks"] + pos = torch.cat([torch.arange(c, c + n) for c, n in reqs]) + req = torch.repeat_interleave(torch.arange(len(reqs)), torch.tensor([n for _, n in reqs])) + top = int(pos.max()) + assert len(reqs) <= 16 and top < spec["max_pos"] and top // tpb < width, (len(reqs), top) + first = len(reqs) - pad + assert all(n == 1 for _, n in reqs[first:]), reqs + key = req.clamp(max=first) * width + pos // tpb + pages = torch.unique(key) + n = pages.numel() + assert n < blocks, f"{n} pages leave no stale block in {blocks}" + if spec.get("far"): + own = blocks - n + torch.randperm(n, generator=gen) + i = int(torch.searchsorted(pages, key[int(pos.argmax())])) + j = int((own == blocks - 1).nonzero()) + own[[i, j]] = own[[j, i]] + else: + own = torch.randperm(blocks, generator=gen)[:n] + free = torch.ones(blocks, dtype=torch.bool) + free[own] = False + stale = free.nonzero().flatten() + block_table = stale[torch.randint(0, stale.numel(), (16 * width,), generator=gen)] + block_table[pages] = own + compact = torch.zeros(16 * width, dtype=torch.int32) + compact[pages] = torch.arange(n, dtype=torch.int32) + block_table, compact = block_table.view(16, width), compact.view(16, width) + if pad > 1: + block_table[first + 1 : len(reqs)] = block_table[first] + compact[first + 1 : len(reqs)] = compact[first] + t = pos.numel() + mag = torch.exp2(torch.rand(t, 4, 1, generator=gpu, device="cuda") * 24 - 16) + nope = (torch.randn(t, 4, 128, generator=gpu, device="cuda") * mag).view(t, 512) + k_pe = torch.randn(t, 64, generator=gpu, device="cuda") * 4 + latent = torch.cat([nope, k_pe], 1).to(torch.bfloat16) + q_full = torch.randn(t, _Q_HEADS, 576, generator=gpu, device="cuda", dtype=torch.bfloat16) + if pad: + latent[t - pad :] = latent[t - pad] + q_full[t - pad :] = q_full[t - pad] + own = own.cuda() + pool.fill_(_SENTINEL) + pool[own] = torch.randint( + 0, 256, (n, *pool.shape[1:]), dtype=torch.uint8, device="cuda", generator=gpu + ) + return SimpleNamespace( + latent=latent, + q_full=q_full, + pos=pos.to(torch.int32).cuda(), + req=req.to(torch.int32).cuda(), + block_table=block_table.to(torch.int32).cuda(), + compact=compact.cuda(), + own=own, + table=table, + pool=pool, + row=spec["row"], + rope=spec["rope"], + heads=spec["heads"], + ) + + +def _args_1m(s, block_table=None, table=None, heads=None): + q = s.q_full[:, : s.heads if heads is None else heads] + return (s.latent, q, s.pos, s.table if table is None else table, s.req, + s.block_table if block_table is None else block_table, s.pool, s.row, + s.rope) # fmt: skip + + +def _assert_sentinel(pool: torch.Tensor) -> None: + """Every byte of `pool` reads as the sentinel: int64 words, 2 GiB at a time.""" + words = pool.view(-1).view(torch.int64) + step = 1 << 28 + bad = torch.zeros((), dtype=torch.int64, device=pool.device) + for i in range(0, words.numel(), step): + bad += torch.count_nonzero(words[i : i + step] != _SENTINEL_WORD) + assert int(bad) == 0, f"{int(bad)} 8-byte words outside the written blocks changed" + + +def _check_1m(s, snap: torch.Tensor, q_before: torch.Tensor) -> int: + """Gate the call against `reference` on `snap` (the owned blocks before it) under the + renumbered table: the owned blocks' codes and scales and every byte outside the written + rows' rope halves bitwise, those within a bf16 ulp; the rest of the pool the sentinel; + the call's heads' rope tails within an ulp, the rest of `q_full` bitwise. Returns the + largest pool byte offset a row was written at.""" + h = s.heads + ref_pool, ref_q = op.reference( + s.latent, q_before[:, :h], s.pos, s.table, s.req, s.compact, snap, s.row, s.rope + ) + after = s.pool[s.own] + tpb, owners = s.pool.shape[2], s.pool.shape[1] + pos = s.pos.long() + slot = s.compact[s.req.long(), pos // tpb].long() + off = pos % tpb + assert torch.equal(after[..., :ROPE_OFFSET], ref_pool[..., :ROPE_OFFSET]), ( + f"{int((after[..., :ROPE_OFFSET] != ref_pool[..., :ROPE_OFFSET]).sum())} code/scale " + "bytes of the written blocks differ" + ) + _ulp_close( + after[slot, s.row, off, ROPE_OFFSET:].contiguous().view(torch.bfloat16), + ref_pool[slot, s.row, off, ROPE_OFFSET:].contiguous().view(torch.bfloat16), + ) + after[slot, s.row, off, ROPE_OFFSET:] = 0 + ref_pool[slot, s.row, off, ROPE_OFFSET:] = 0 + assert torch.equal(after, ref_pool), "bytes of the written blocks outside its rows changed" + s.pool[s.own] = _SENTINEL + _assert_sentinel(s.pool) + assert torch.equal(s.q_full[:, h:], q_before[:, h:]), "heads past the call's were written" + assert torch.equal(s.q_full[:, :h, :512], q_before[:, :h, :512]), "q's latent part changed" + if s.rope: + _ulp_close(s.q_full[:, :h, 512:], ref_q[..., 512:]) + else: + assert torch.equal(s.q_full, q_before), "q changed on a NoPE owner" + block = s.block_table[s.req.long(), pos // tpb].long() + return int((((block * owners + s.row) * tpb + off) * TOKEN_BYTES).max()) + + +def _eager_1m(s) -> int: + snap, q_before = s.pool[s.own], s.q_full.clone() + op(*_args_1m(s)) + return _check_1m(s, snap, q_before) + + +def _capture(s) -> torch.cuda.CUDAGraph: + """The engine's capture: one eager call on a side stream, then the captured call.""" + side = torch.cuda.Stream() + side.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(side): + op(*_args_1m(s)) + torch.cuda.current_stream().wait_stream(side) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + op(*_args_1m(s)) + return graph + + +def _replay(s, new, graph, rows: int) -> tuple: + """Write `new`'s rows, positions and pages into the captured inputs, replay and gate it; + then run the same inputs eagerly from the same pool and require the replay's bytes. + Returns the top byte offset written, and the pool snapshot, the replayed blocks and the + query before the replay.""" + for name in ("latent", "q_full", "pos", "req", "block_table"): + getattr(s, name).copy_(getattr(new, name)) + s.compact, s.own = new.compact, new.own + snap, q_before = s.pool[s.own], s.q_full.clone() + graph.replay() + torch.cuda.synchronize() + replayed, replayed_q = s.pool[s.own], s.q_full.clone() + written = _check_1m(s, snap, q_before) + s.pool[s.own] = snap + s.q_full.copy_(q_before) + op(*_args_1m(s)) + assert torch.equal(s.pool[s.own], replayed), f"batch {rows}: the replay differs from eager" + assert torch.equal(s.q_full, replayed_q), f"batch {rows}: replayed q differs from eager" + return written, snap, replayed, q_before + + +def _graphs_1m(spec, seed: int, pool, table) -> int: + """At each of the cell's batches: capture the call, replay it on new rows, positions and + pages, gate it and require the eager call's bytes.""" + top = 0 + for rows in spec.get("graph_batches", (spec["tokens"],)): + gen = torch.Generator().manual_seed(seed * 100 + rows) + gpu = torch.Generator(device="cuda").manual_seed(seed * 100 + rows) + s = _step(spec, _requests(spec, gen, batch=rows), gen, gpu, pool, table) + graph = _capture(s) + new = _step(spec, _requests(spec, gen, batch=rows), gen, gpu, pool, table) + written = _replay(s, new, graph, rows)[0] + print(f"graph batch {rows}: replay gated, bitwise eager, top byte offset {written}") + top = max(top, written) + del graph + return top + + +def _boot_1m(spec, seed: int, pool, table) -> int: + """The dep4 engine's decode graphs under attention DP, at each of the cell's batches b: + captured on the CUDA-graph warm-up batch (the longest dummy's new key at max_pos - 2 in + row 0 ahead of b - 1 one-key fillers at 0, each a request on a page of its own; that + eager call gated), then replayed with `live` rows ahead of b - live rows of the padding + dummy at position 0 on one slot -- the live row at the table's last position, at 0 (an + idle rank's attention-DP dummy) and at a drawn one. Each replay is gated and bitwise the + eager call; with two or more padding rows, also bitwise the eager call with the padding + cut to its first row, so the repeated rows leave the slot as one dummy row's write + does.""" + top, live, tpb = 0, spec["live"], spec["tokens_per_block"] + last = spec["max_pos"] - 1 + for rows in spec["graph_batches"]: + gen = torch.Generator().manual_seed(seed * 100 + rows) + gpu = torch.Generator(device="cuda").manual_seed(seed * 100 + rows) + s = _step(spec, [(last - 1, 1)] + [(0, 1)] * (rows - 1), gen, gpu, pool, table) + snap, q_before = pool[s.own], s.q_full.clone() + graph = _capture(s) + top = max(top, _check_1m(s, snap, q_before)) + pad = rows - live + drawn = int(torch.randint(0, spec["max_pos"], (1,), generator=gen)) + for at in (last, 0, drawn): + reqs = [(at, 1)] * live + [(0, 1)] * pad + new = _step(spec, reqs, gen, gpu, pool, table, pad=pad) + slots = new.block_table[new.req.long(), new.pos.long() // tpb] + assert slots[live:].unique().numel() == min(pad, 1), "padding rows on one slot" + assert slots[:live].unique().numel() == live and not bool( + torch.isin(slots[:live], slots[live:]).any() + ), "live rows on slots of their own" + written, snap, replayed, q_before = _replay(s, new, graph, rows) + top = max(top, written) + if pad > 1: + k = live + 1 + pool[s.own] = snap + q_cut = q_before[:k].clone() + op(s.latent[:k], q_cut, s.pos[:k], s.table, s.req[:k], s.block_table, pool, + s.row, s.rope) # fmt: skip + assert torch.equal(pool[s.own], replayed), ( + f"batch {rows}: {pad} padding rows on one slot differ from one row's write" + ) + assert torch.equal(q_cut, s.q_full[:k]), f"batch {rows}: the cut call's q" + cut = ", bitwise the call cut to one padding row" if pad > 1 else "" + print(f"boot graph batch {rows}: live row at {at} ahead of {pad} padding rows on " + f"one slot, gated, bitwise eager{cut}") # fmt: skip + del graph + return top + + +def _run_1m(spec, seed: int) -> None: + torch.cuda.reset_peak_memory_stats() + table = _indexer_table(spec["max_pos"]) + pool = torch.empty( + (spec["blocks"], spec["owners"], spec["tokens_per_block"], TOKEN_BYTES), + dtype=torch.uint8, + device="cuda", + ) + if spec["phase"] == "decode": + drive = _boot_1m if spec["positions"] == "boot" else _graphs_1m + top = drive(spec, seed, pool, table) + else: + chunks = 1 + if "sweep_positions" in spec: + lo, hi = spec["sweep_positions"] + chunks = (hi - lo) // spec["tokens"] + assert lo + chunks * spec["tokens"] == hi, spec["sweep_positions"] + top = 0 + for chunk in range(chunks): + gen = torch.Generator().manual_seed(seed * 100 + chunk) + gpu = torch.Generator(device="cuda").manual_seed(seed * 100 + chunk) + s = _step(spec, _requests(spec, gen, chunk=chunk), gen, gpu, pool, table) + top = max(top, _eager_1m(s)) + if pool.numel() > 2**31: + assert top >= 2**31, f"no row was written past 2^31 bytes (top byte offset {top})" + if spec.get("far"): + # The step's last position is the table's last, on the pool's last block. + tpb = spec["tokens_per_block"] + last = (((spec["blocks"] - 1) * spec["owners"] + spec["row"]) * tpb + + (spec["max_pos"] - 1) % tpb) * TOKEN_BYTES # fmt: skip + assert top == last, (top, last) + peak = torch.cuda.max_memory_allocated() / 2**30 + print(f"pool {pool.numel()} bytes, top written row at byte offset {top}, peak {peak:.1f} GiB") + del pool + torch.cuda.empty_cache() + + +@pytest.mark.parametrize( + "cell", + [pytest.param(c, marks=pytest.mark.large) if c.spec.get("large") else c for c in op.CELLS], + ids=lambda c: c.why[:60], +) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — fixture by name + certifying(op) + spec = cell.spec + if "max_pos" in spec: + _run_1m(spec, seed=zlib.crc32(cell.why.encode()) % 1000) + return + b = _build(spec, seed=zlib.crc32(cell.why.encode()) % 1000) + pool0 = b["pool"].clone() + q0 = None if b["q_full"] is None else b["q_full"].clone() + if spec.get("captured"): + # Capture on the cell's data, replay on new rows, positions and slots written into + # the captured inputs. + s = torch.cuda.Stream() + s.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(s): + op(*_args(b)) + torch.cuda.current_stream().wait_stream(s) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + op(*_args(b)) + nb = _build(spec, seed=zlib.crc32(cell.why.encode()) % 1000 + 1) + for k in ("latent", "pos", "table", "req_idx", "block_table", "pool", "q_full"): + if b[k] is not None: + b[k].copy_(nb[k]) + pool0 = b["pool"].clone() + q0 = None if b["q_full"] is None else b["q_full"].clone() + graph.replay() + torch.cuda.synchronize() + else: + op(*_args(b)) + _check(b, pool0, q0) + + +def test_1m_harness_refuses_misplaced_writes() -> None: + """Controls for the 1M harness at the `_ML4_1M` cells' 16 heads: a call whose block table + sends one page to a stale block, a call that reads its rope one table row early, a call + that ropes a 17th query head, and a call followed by one stray byte at the pool's far end + each fail `_check_1m`, by the check meant for it.""" + spec = dict(phase="context", tokens=64, cached=1015808, heads=16, rope=True, blocks=256, + owners=63, tokens_per_block=32, row=62, max_pos=1048576, + table_width=32772) # fmt: skip + table = _indexer_table(spec["max_pos"]) + pool = torch.empty((256, 63, 32, TOKEN_BYTES), dtype=torch.uint8, device="cuda") + gen, gpu = torch.Generator().manual_seed(1), torch.Generator(device="cuda").manual_seed(1) + + def run(**kw): + s = _step(spec, _requests(spec, gen), gen, gpu, pool, table) + snap, q_before = pool[s.own], s.q_full.clone() + if kw.pop("misplace", False): + kw["block_table"] = s.block_table.clone() + kw["block_table"][0, 1015808 // 32] = s.block_table[1, 0] + op(*_args_1m(s, **kw)) + return s, snap, q_before + + with pytest.raises(AssertionError, match="code/scale bytes"): + _check_1m(*run(misplace=True)) + with pytest.raises(AssertionError): + _check_1m(*run(table=torch.roll(table, 1, dims=0))) + with pytest.raises(AssertionError, match="heads past the call's"): + _check_1m(*run(heads=17)) + s, snap, q_before = run() + stray = max(set(range(256)) - set(s.own.tolist())) + pool[stray, -1, -1, -1] = _SENTINEL ^ 1 + with pytest.raises(AssertionError, match="8-byte words outside"): + _check_1m(s, snap, q_before) + + +def test_1m_dep4_harness_refuses_misplaced_writes() -> None: + """Controls for the 1M harness at the `_ML4_1M_DEP4` cells' 64 heads: a call whose block + table sends one page to a stale block, a call that reads its rope one table row early, a + call that ropes 63 of the 64 query heads, a call followed by one stray byte at the pool's + far end, and a padding replay whose shared slot is then changed in one byte each fail + `_check_1m` or the one-row comparison, by the check meant for it.""" + spec = dict(phase="context", tokens=64, cached=1015808, heads=64, rope=True, blocks=256, + owners=63, tokens_per_block=32, row=62, max_pos=1048576, + table_width=32772) # fmt: skip + table = _indexer_table(spec["max_pos"]) + pool = torch.empty((256, 63, 32, TOKEN_BYTES), dtype=torch.uint8, device="cuda") + gen, gpu = torch.Generator().manual_seed(1), torch.Generator(device="cuda").manual_seed(1) + + def run(**kw): + s = _step(spec, _requests(spec, gen), gen, gpu, pool, table) + snap, q_before = pool[s.own], s.q_full.clone() + if kw.pop("misplace", False): + kw["block_table"] = s.block_table.clone() + kw["block_table"][0, 1015808 // 32] = s.block_table[1, 0] + op(*_args_1m(s, **kw)) + return s, snap, q_before + + with pytest.raises(AssertionError, match="code/scale bytes"): + _check_1m(*run(misplace=True)) + with pytest.raises(AssertionError): + _check_1m(*run(table=torch.roll(table, 1, dims=0))) + with pytest.raises(AssertionError): + _check_1m(*run(heads=63)) + s, snap, q_before = run() + stray = max(set(range(256)) - set(s.own.tolist())) + pool[stray, -1, -1, -1] = _SENTINEL ^ 1 + with pytest.raises(AssertionError, match="8-byte words outside"): + _check_1m(s, snap, q_before) + + # One live row ahead of 7 padding rows: the shared slot, one byte off one row's write. + boot = dict(spec, phase="decode", tokens=8, row=1) + s = _step(boot, [(1048575, 1)] + [(0, 1)] * 7, gen, gpu, pool, table, pad=7) + snap, q_before = pool[s.own], s.q_full.clone() + op(*_args_1m(s)) + slot = s.block_table[1, 0].long() + pool[slot, 1, 0, 100] ^= 1 + replayed = pool[s.own] + pool[s.own] = snap + op(s.latent[:2], q_before[:2].clone(), s.pos[:2], s.table, s.req[:2], s.block_table, pool, + s.row, s.rope) # fmt: skip + assert not torch.equal(pool[s.own], replayed), "the one-row comparison is blind" + pool[s.own] = replayed + with pytest.raises(AssertionError, match="code/scale bytes"): + _check_1m(s, snap, q_before) + + +def test_1m_tep8_harness_refuses_misplaced_writes() -> None: + """Controls for the 1M harness at the `_ML4_1M_TEP8` cells' 8 heads: a call whose block + table sends one page to a stale block, a call that reads its rope one table row early, a + call that ropes a 9th query head (at tep8, the first head the kernel's 16-head block + masks), and a call followed by one stray byte at the pool's far end each fail + `_check_1m`, by the check meant for it.""" + spec = dict(phase="context", tokens=64, cached=1015808, heads=8, rope=True, blocks=256, + owners=63, tokens_per_block=32, row=62, max_pos=1048576, + table_width=32772) # fmt: skip + table = _indexer_table(spec["max_pos"]) + pool = torch.empty((256, 63, 32, TOKEN_BYTES), dtype=torch.uint8, device="cuda") + gen, gpu = torch.Generator().manual_seed(1), torch.Generator(device="cuda").manual_seed(1) + + def run(**kw): + s = _step(spec, _requests(spec, gen), gen, gpu, pool, table) + snap, q_before = pool[s.own], s.q_full.clone() + if kw.pop("misplace", False): + kw["block_table"] = s.block_table.clone() + kw["block_table"][0, 1015808 // 32] = s.block_table[1, 0] + op(*_args_1m(s, **kw)) + return s, snap, q_before + + with pytest.raises(AssertionError, match="code/scale bytes"): + _check_1m(*run(misplace=True)) + with pytest.raises(AssertionError): + _check_1m(*run(table=torch.roll(table, 1, dims=0))) + with pytest.raises(AssertionError, match="heads past the call's"): + _check_1m(*run(heads=9)) + s, snap, q_before = run() + stray = max(set(range(256)) - set(s.own.tolist())) + pool[stray, -1, -1, -1] = _SENTINEL ^ 1 + with pytest.raises(AssertionError, match="8-byte words outside"): + _check_1m(s, snap, q_before) + + +def test_scales_are_powers_of_two_with_the_floor() -> None: + """A zero tile stores the 1e-4 floor rounded up (2^-13) and zero codes; an exact power of + two times 448 keeps its own scale.""" + spec = dict(phase="context", tokens=2, cached=0, heads=0, rope=False, blocks=4, owners=1, + tokens_per_block=32, row=0) # fmt: skip + b = _build(spec, seed=3) + b["latent"][0, :512] = 0 + b["latent"][1, :512] = 0 + b["latent"][1, 0] = 448.0 * 2.0**-5 + op(*_args(b)) + blk0 = int(b["block_table"][0, 0]) + rows = b["pool"][blk0, 0, :2] + scales = rows[:, 512:528].contiguous().view(torch.float32) + assert scales[0].tolist() == [2.0**-13] * 4 + assert scales[1, 0].item() == 2.0**-5 + assert int(rows[0, :512].sum()) == 0 + + +def test_is_valid_refuses_a_strided_pool() -> None: + spec = dict(phase="decode", tokens=4, heads=16, rope=True, blocks=64, owners=4, + tokens_per_block=32, row=1) # fmt: skip + b = _build(spec, seed=4) + args = list(_args(b)) + args[6] = b["pool"][:, :2] + with pytest.raises(ValueError, match="contiguous"): + op.is_valid(*args) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_flash_mla_sparse_decode.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_flash_mla_sparse_decode.py new file mode 100644 index 000000000000..4a76d701e499 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_flash_mla_sparse_decode.py @@ -0,0 +1,1078 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the flash_mla_sparse_decode entry. + +The kernel computes in bf16 (query, dequantized keys, P) with fp32 accumulation; the +reference is fp32 throughout. Band: relative L2 per (row, head) under 1e-2 and the whole +call under 4e-3 -- bf16's 2^-8 relative step, softened by the 512-wide sums. The 1M cells +keep that band, each driven by the harness of the run that certified it, named by its why +("1M tp4"; "1M tep4", "1M tep8" and "1M dep4" share one, dep4's extension of the tep4 and +tep8 one). Their pools are 0xFF (e4m3, fp32 and bf16 NaN) wherever the requests hold no +key, so every output row of theirs is first checked finite: a key read from anywhere else +turns its row NaN. A call of more than 1024 rows is gated against the reference on 1024 of +them, its last 128 and 896 drawn (the reference loops over rows); under the tep4 / tep8 / +dep4 harness a context call's rows FlashMLA's scheduler split over their keys +(`num_splits` > 1, the parts merged in fp32) are gated too. A "1M tp4" ragged-tail cell +gates every row of every row count it drives, against one reference run over the largest. +A "1M tep4" or "1M tep8" `ragged_tail` cell runs one sub-batch at many row counts (the +tep8 cells' largest first): the reference runs once over its rows and every row of every +call is gated, and a row the decode scheduler kept in one part is also bitwise the whole +sub-batch's (a split row is merged in fp32 and rounds differently, inside the band). A +"1M dep4" ragged-tail cell runs one sub-batch at many row counts too: the reference and a +whole-sub-batch call run once over its rows, each call's split rows and every row not +bitwise the whole call's are gated, every other row must be bitwise the whole call's, and +each call's rel L2 counts every row. +""" + +import collections +import math +import zlib + +import pytest +import torch + +from tensorrt_llm._torch.modeling_v2.catalog.gemm.bmm_out import bmm_out +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.flash_mla_sparse_decode import ( + flash_mla_sparse_decode as op, +) +from tensorrt_llm.flash_mla import get_mla_metadata + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402,F401 — needs the path declared above + +from ._ds_mla_pool import ds_mla_rows # noqa: E402 + +assert torch.cuda.is_available(), "flash_mla_sparse_decode requires a CUDA device" + +SCALE = 128**-0.5 + + +def _build(spec, seed): + g = torch.Generator().manual_seed(seed) + rows, page, owners, pages = spec["rows"], spec["page_size"], spec["owners"], spec["pages"] + blocks = math.ceil(pages / owners) + 1 + kv = ds_mla_rows(blocks * owners * page, g, nope=spec.get("nope", False)) + kv = kv.view(blocks * owners, page, 1, 656) + row = 1 + kv_len = spec["kv_len"] + per_seq = math.ceil(kv_len / page) + idx = torch.full((rows, spec["topk"]), -1, dtype=torch.int32) + for r in range(rows): + blk = torch.randint(0, blocks, (per_seq,), generator=g) + pos = torch.randperm(kv_len, generator=g)[: min(kv_len, spec["topk"])] + slots = (blk[pos // page] * owners + row) * page + pos % page + if spec.get("holes"): + slots[torch.rand(slots.shape, generator=g) < 0.2] = -1 + idx[r, : slots.numel()] = slots.int() + q = torch.zeros(rows, spec["h_q"], 576) + q[:, : spec["h_out"]] = torch.randn(rows, spec["h_out"], 576, generator=g) + if spec.get("nope"): + q[..., 512:] = 0 + out = torch.full((rows, spec["h_out"], 512), float("nan"), dtype=torch.bfloat16) + return q.to(torch.bfloat16).cuda(), kv.cuda(), idx.cuda(), out.cuda() + + +def _gate(out, ref): + o, r = out.float(), ref.float() + per = (o - r).norm(dim=-1) / r.norm(dim=-1).clamp_min(1e-30) + whole = float((o - r).norm() / r.norm()) + assert float(per.max()) < 1e-2 and whole < 4e-3, (float(per.max()), whole) + + +# --- the "1M tp4" cells: pools built on the device, only the requests' pages written + +#: Every pool byte the cell's requests do not hold: e4m3, fp32 and bf16 NaN, so a key read +#: from anywhere else turns its row's output NaN. +_POOL_FILL = 0xFF +#: Rows of a step's scratch outside the call's tail view: bf16 NaN bits the call must leave. +_OUT_FILL = -1 +_GATED_ROWS = 1024 +#: The step scratch a `q_offset` cell's views tail: max_num_tokens rows. +_STEP_ROWS = 32768 +_ROWS_PER_DRAW = 1 << 20 + + +def _dev_rows(n: int, g: torch.Generator, nope: bool = False) -> torch.Tensor: + """`ds_mla_rows` drawn on g's device.""" + dev = g.device + codes = ((torch.rand(n, 512, generator=g, device=dev) * 2 - 1) * 448).to(torch.float8_e4m3fn) + scale = torch.exp2(torch.randint(-13, 1, (n, 4), generator=g, device=dev).float()) + if nope: + rope = torch.zeros(n, 64, device=dev) + else: + rope = torch.randn(n, 64, generator=g, device=dev) * 3 + return torch.cat( + [ + codes.view(torch.uint8), + scale.view(torch.uint8), + rope.to(torch.bfloat16).view(torch.uint8), + ], + 1, + ) + + +def _write_pages(pool, pages, row, g, nope) -> None: + """Draw the owner `row` of every page in `pages` (block ids), a bounded slab at a time.""" + tpb = pool.shape[2] + step = _ROWS_PER_DRAW // tpb + for lo in range(0, pages.numel(), step): + part = pages[lo : lo + step] + pool[part, row] = _dev_rows(part.numel() * tpb, g, nope).view(-1, tpb, 656) + + +def _kv_lens(kv_len: int, n: int) -> list[int]: + """`n` sequence lengths, the first `kv_len` and the rest spread below it: a row one page + short of it, eighth, sixteenth and 32nd fractions, the rows around topk 2048 (2048 valid + of 2049, all 2048, a -1 tail), and short rows down to 1. At 1048576 the sixteen hold + 2348481 keys, 73398 pages: the delivered 1M boot's 74183-block pool holds them apart.""" + spread = (kv_len, kv_len - 31, kv_len // 8 + 1, kv_len // 16, kv_len // 32 + 1, 9217, 4097, + 2049, 2048, 2047, 1500, 700, 257, 64, 2, 1) # fmt: skip + return [max(1, min(kv_len, n_kv)) for n_kv in spread[:n]] + + +def _requests_tp4(spec, n: int) -> list[tuple[int, int]]: + """(cached, new) per request: a context cell's chunks, else `n` decode rows, each one + query at its sequence's last position.""" + if spec.get("context"): + if "contexts" in spec: + return [tuple(c) for c in spec["contexts"]] + return [(spec["cached"], spec["chunk_rows"])] + return [(n_kv - 1, 1) for n_kv in _kv_lens(spec["kv_len"], n)] + + +def _pool_1m(spec, reqs, g): + """The pool, its owner row, each request's pages (block ids, requests in order) and page + counts, and the slot its first request's last position reads. + + `far`: the pool is `pages` / `owners` blocks exactly, every byte `_POOL_FILL` but the + owner row of the requests' pages, which are distinct and the pool's top ones, the first + request's last page on the top block. Otherwise the pages are drawn over the whole pool + with repeats (its every block's row written), as the shared cells draw them. + """ + dev = g.device + owners, tpb = spec["owners"], spec["page_size"] + blocks = -(-spec["pages"] // owners) + row = 35 if spec.get("nope") else (owners - 1 if spec.get("far") else 1) + counts = [-(-(c + n) // tpb) for c, n in reqs] + need = sum(counts) + pool = torch.empty((blocks, owners, tpb, 656), dtype=torch.uint8, device=dev) + pool.fill_(_POOL_FILL) + if spec.get("far"): + assert need <= blocks, (need, blocks) + pages = blocks - need + torch.randperm(need, generator=g, device=dev) + top = int((pages == blocks - 1).nonzero()[0]) + pages[[top, counts[0] - 1]] = pages[[counts[0] - 1, top]] + _write_pages(pool, pages, row, g, spec.get("nope", False)) + else: + pages = torch.randint(0, blocks, (need,), generator=g, device=dev) + _write_pages(pool, torch.arange(blocks, device=dev), row, g, spec.get("nope", False)) + last = sum(reqs[0]) - 1 + first_last = (int(pages[counts[0] - 1]) * owners + row) * tpb + last % tpb + return pool, row, pages, counts, first_last + + +def _selections_tp4(spec, reqs, pages, counts, row, g) -> torch.Tensor: + """int32 `[rows, topk]` slots, the rows the requests' new tokens in order. A row at + position p sees keys 0..p: below topk all of them in order then -1, else topk draws over + [0, p] (repeats counted by both sides) with p itself first. Each key's slot through its + request's pages.""" + dev = g.device + topk, tpb, owners = spec["topk"], spec["page_size"], spec["owners"] + pos = torch.cat([torch.arange(c, c + n, device=dev) for c, n in reqs]) + req = torch.cat([torch.full((n,), r, device=dev) for r, (_, n) in enumerate(reqs)]) + first = torch.tensor([0] + counts[:-1], device=dev).cumsum(0) + col = torch.arange(topk, device=dev)[None] + p = pos[:, None] + draw = torch.minimum( + (torch.rand(pos.numel(), topk, generator=g, device=dev) * (p + 1)).long(), p + ) + draw[:, 0] = pos + sel = torch.where(p < topk, torch.where(col <= p, col, -1), draw) + del draw + key = sel.clamp_min(0) + slot = (pages[first[req][:, None] + key // tpb] * owners + row) * tpb + key % tpb + slot = torch.where(sel >= 0, slot, -1) + assert int(slot.max()) < 2**31 + return slot.to(torch.int32) + + +def _queries_tp4(n: int, spec, g, scratch_rows: int = 0) -> torch.Tensor: + """bf16 `[n, h_q, 576]`, the first h_out heads drawn and the rest zero (rope zero on a + NoPE layer); with `scratch_rows`, the tail view of a zeroed scratch that many rows long.""" + dev = g.device + q = torch.zeros(max(n, scratch_rows), spec["h_q"], 576, dtype=torch.bfloat16, device=dev) + q = q[q.shape[0] - n :] + q[:, : spec["h_out"]] = torch.randn(n, spec["h_out"], 576, generator=g, device=dev) + if spec.get("nope"): + q[..., 512:] = 0 + return q + + +def _calls_tp4(spec, q, kv, idx, out, metas) -> torch.Tensor: + """Every layer's call, in sub-batches of `sub_batch` rows (one call without it), one + sched meta per sub-batch size shared by every layer.""" + n = q.shape[0] + sub = spec.get("sub_batch", n) + for _ in range(spec["layers"]): + for lo in range(0, n, sub): + hi = min(lo + sub, n) + meta = metas.setdefault(hi - lo, get_mla_metadata()[0]) + op(q[lo:hi], kv, idx[lo:hi], meta, SCALE, out[lo:hi]) + return out + + +def _check_tp4(q, kv, idx, out, g) -> None: + """Every row finite; the rows past `_GATED_ROWS` on a sample against the reference.""" + n = q.shape[0] + nan = out.isnan().flatten(1).any(1) + assert not bool(nan.any()), f"{int(nan.sum())} of {n} rows read an unheld pool byte" + rows = torch.arange(n, device=q.device) + if n > _GATED_ROWS: + picked = torch.randperm(n - 128, generator=g, device=q.device)[: _GATED_ROWS - 128] + rows = torch.cat([rows[-128:], picked]) + _gate(out[rows], op.reference(q[rows], kv, idx[rows], None, SCALE, out[rows])) + + +def _replay_bitwise(spec, q, kv, idx, out, reqs, pages, counts, row, g) -> None: + """Capture the layers' calls as the step does (a fresh sched meta its first call fills), + replay on a new query and new slots over the same pages, gate it, and match an eager + call on the same data bitwise.""" + s = torch.cuda.Stream() + s.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(s): + op(q, kv, idx, get_mla_metadata()[0], SCALE, out) + torch.cuda.current_stream().wait_stream(s) + graph = torch.cuda.CUDAGraph() + meta = get_mla_metadata()[0] + with torch.cuda.graph(graph): + for _ in range(spec["layers"]): + op(q, kv, idx, meta, SCALE, out) + q.copy_(_queries_tp4(q.shape[0], spec, g)) + idx.copy_(_selections_tp4(spec, reqs, pages, counts, row, g)) + graph.replay() + torch.cuda.synchronize() + _check_tp4(q, kv, idx, out, g) + eager = op(q, kv, idx, get_mla_metadata()[0], SCALE, torch.empty_like(out)) + assert torch.equal(eager.view(torch.int16), out.view(torch.int16)), "replay != eager" + del graph + + +def _run_1m_tp4(spec, seed) -> None: + dev = torch.device("cuda") + g = torch.Generator(device=dev).manual_seed(seed) + batches = spec.get("graph_batches", (spec["rows"],)) + reqs = _requests_tp4(spec, max(batches)) + pool, row, pages, counts, first_last = _pool_1m(spec, reqs, g) + kv = pool.view(-1, spec["page_size"], 1, 656) + if spec.get("far"): + top = (pool.shape[0] * spec["owners"] - 1) * spec["page_size"] + spec["page_size"] - 1 + print(f"pool {tuple(pool.shape)}: first request's last slot {first_last} " + f"(byte {first_last * 656}), pool's last {top}") # fmt: skip + if not spec.get("nope"): + assert first_last == top - ( + spec["page_size"] - 1 - (sum(reqs[0]) - 1) % spec["page_size"] + ) + for b in batches: + sub = reqs[: len(reqs) if spec.get("context") else b] + idx = _selections_tp4(spec, sub, pages, counts, row, g) + if spec.get("far"): + assert int(idx.max()) == first_last, (int(idx.max()), first_last) + n = idx.shape[0] + q = _queries_tp4(n, spec, g, _STEP_ROWS if "q_offset" in spec else 0) + out_buf = torch.empty(max(n, _STEP_ROWS if "q_offset" in spec else 0), spec["h_out"], 512, + dtype=torch.bfloat16, device=dev) # fmt: skip + out_buf.view(torch.int16).fill_(_OUT_FILL) + out = out_buf[out_buf.shape[0] - n :] + if "q_offset" in spec: + assert out_buf.shape[0] - n == spec["q_offset"] + metas = {} + _calls_tp4(spec, q, kv, idx, out, metas) + if spec.get("context"): + assert max(metas) == spec["rows"], sorted(metas) + _check_tp4(q, kv, idx, out, g) + if "q_offset" in spec: + head = out_buf[: spec["q_offset"]] + assert bool((head.view(torch.int16) == _OUT_FILL).all()), "wrote outside the view" + if spec.get("captured"): + _replay_bitwise(spec, q, kv, idx, out, sub, pages, counts, row, g) + + +def _ragged_tail_tp4(spec, seed) -> None: + """A chunk's last sub-batch at every row count r of `tail_rows` (else 1..`sub_batch`): rows + `lead_rows` .. `lead_rows` + r - 1 of the chunk behind `cached`, q and out views from that row + of the step's scratch, each r one call on a sched meta of its own. Row i is the same sequence + at every r, so the reference runs once over the `sub_batch` rows and each r's output is gated + against its first r rows in `_gate`'s band (a NaN row fails it); the scratch rows before the + view stay unwritten. Largest r first: rising sizes leave the caching allocator a new block per + call (a rising sweep reserved 254.75 GiB).""" + dev = torch.device("cuda") + g = torch.Generator(device=dev).manual_seed(seed) + sub, lead = spec["sub_batch"], spec["lead_rows"] + reqs = [(spec["cached"] + lead, sub)] + pool, row, pages, counts, _ = _pool_1m(spec, reqs, g) + kv = pool.view(-1, spec["page_size"], 1, 656) + idx = _selections_tp4(spec, reqs, pages, counts, row, g) + q = _queries_tp4(sub, spec, g, lead + sub) + out_buf = torch.empty(lead + sub, spec["h_out"], 512, dtype=torch.bfloat16, device=dev) + out_buf.view(torch.int16).fill_(_OUT_FILL) + out = out_buf[lead:] + ref = op.reference(q, kv, idx, None, SCALE, out).float() + ref_norm = ref.norm(dim=-1) + ref_sq = ref_norm.square().sum(1).cumsum(0) + tails = spec.get("tail_rows", range(1, sub + 1)) + per = torch.empty(len(tails), device=dev) + whole = torch.empty(len(tails), device=dev) + for i in sorted(range(len(tails)), key=lambda i: -tails[i]): + r = tails[i] + o = op(q[:r], kv, idx[:r], get_mla_metadata()[0], SCALE, out[:r]) + err = (o.float() - ref[:r]).square().sum(-1) + per[i] = (err.sqrt() / ref_norm[:r].clamp_min(1e-30)).max() + whole[i] = (err.sum() / ref_sq[r - 1]).sqrt() + bad = ((per < 1e-2) & (whole < 4e-3)).logical_not().nonzero()[:8, 0].tolist() + assert not bad, [(tails[i], float(per[i]), float(whole[i])) for i in bad] + assert bool((out_buf[:lead].view(torch.int16) == _OUT_FILL).all()), "wrote outside the view" + i, j = int(per.argmax()), int(whole.argmax()) + print(f"ragged tail behind {reqs[0][0]}: {len(tails)} row counts {tails[0]}..{tails[-1]}, " + f"per-(row, head) rel L2 max {float(per[i]):.3e} at r {tails[i]}, whole call max " + f"{float(whole[j]):.3e} at r {tails[j]}") # fmt: skip + + +# --- the "1M tep4", "1M dep4" and "1M tep8" cells: pools on the device, one owner row of the +# requests' pages written + +#: Pool rows drawn per batch while writing the requests' pages. +_DRAW_ROWS = 1 << 18 + + +def _draw_rows(n: int, g: torch.Generator, nope: bool) -> torch.Tensor: + """`ds_mla_rows` drawn on g's device.""" + dev = g.device + codes = ((torch.rand(n, 512, generator=g, device=dev) * 2 - 1) * 448).to(torch.float8_e4m3fn) + scale = torch.exp2(torch.randint(-13, 1, (n, 4), generator=g, device=dev).float()) + if nope: + rope = torch.zeros(n, 64, device=dev) + else: + rope = torch.randn(n, 64, generator=g, device=dev) * 3 + rope = rope.to(torch.bfloat16) + return torch.cat([codes.view(torch.uint8), scale.view(torch.uint8), rope.view(torch.uint8)], 1) + + +def _decode_lens(kv_len: int, n: int) -> list[int]: + """`n` decode sequence lengths: `kv_len`, a row 31 keys short of it, its 1/8, 1/16 and + 1/32 plus one, rows across the sub-chunk sizes, around topk 2048 (2049: 2048 of 2049 + valid; 2048: all; 2047: a -1 tail) and short rows down to one key. At 1048576 the sixteen + hold 2364359 keys on 73898 pages: the 74183- and the 147456-block pools hold them on + distinct pages.""" + spread = (kv_len, kv_len - 31, kv_len // 8 + 1, kv_len // 16 + 1, kv_len // 32 + 1, 16385, + 9217, 4097, 2049, 2048, 2047, 1025, 700, 257, 33, 1) # fmt: skip + return [max(1, min(kv_len, n_kv)) for n_kv in spread[:n]] + + +def _requests(spec, n: int) -> list[tuple[int, int]]: + """(cached, new) per request. A context cell: its chunks. A `boot_capture` cell: the live + row (kv_len keys), the warm-up's longest dummy (kv_len - 1 keys), n - 1 one-key fillers, + the padding dummy and an idle rank's dummy (one key each). Otherwise `n` decode rows. A + decode row is one query at its sequence's last position.""" + if spec.get("context"): + if "contexts" in spec: + return [tuple(c) for c in spec["contexts"]] + return [(spec["cached"], spec["chunk_rows"])] + if spec.get("boot_capture"): + kv_len = spec["kv_len"] + return [(kv_len - 1, 1), (kv_len - 2, 1), *[(0, 1)] * (n - 1), (0, 1), (0, 1)] + return [(n_kv - 1, 1) for n_kv in _decode_lens(spec["kv_len"], n)] + + +def _pool(spec, reqs, g): + """The pool [blocks, owners, page, 656], its written owner row, the requests' pages (block + ids, request after request), each request's page count, and the slot of the first + request's last key. + + `far`: the pool is `pages` / `owners` blocks, every byte `_POOL_FILL` but the owner row of + the requests' pages, which are distinct and the pool's top ones, the first request's last + page on the top block. `top_blocks`: the pages are drawn with repeats from the pool's top + `top_blocks` blocks, whose owner row is written. Otherwise they are drawn over the whole + pool and every block's owner row is written. + """ + dev = g.device + owners, page = spec["owners"], spec["page_size"] + row = 35 if spec.get("nope") else owners - 1 + counts = [-(-(c + n) // page) for c, n in reqs] + need = sum(counts) + blocks = -(-spec["pages"] // owners) + pool = torch.full((blocks, owners, page, 656), _POOL_FILL, dtype=torch.uint8, device=dev) + if spec.get("far"): + assert spec["pages"] % owners == 0 and need <= blocks, (need, blocks) + pages = blocks - need + torch.randperm(need, generator=g, device=dev) + top, last = int((pages == blocks - 1).nonzero()[0, 0]), counts[0] - 1 + pages[[top, last]] = pages[[last, top]] + written = pages + elif "top_blocks" in spec: + low = blocks - spec["top_blocks"] + pages = torch.randint(low, blocks, (need,), generator=g, device=dev) + written = torch.arange(low, blocks, device=dev) + else: + pages = torch.randint(0, blocks, (need,), generator=g, device=dev) + written = torch.arange(blocks, device=dev) + step = _DRAW_ROWS // page + for lo in range(0, written.numel(), step): + part = written[lo : lo + step] + rows = _draw_rows(part.numel() * page, g, spec.get("nope", False)) + pool[part, row] = rows.view(-1, page, 656) + first_last = (int(pages[counts[0] - 1]) * owners + row) * page + (sum(reqs[0]) - 1) % page + return pool, row, pages, counts, first_last + + +def _selections(spec, reqs, pages, counts, row, g, which=None) -> torch.Tensor: + """int32 `[rows, topk]` slots, the rows the new tokens of the requests `which` names (all, + in order, by default). A row at position p sees keys 0..p: below topk keys all of them, + then -1; else topk distinct keys, one drawn in each of topk equal strata of [0, p], p + itself the last. Each row's valid keys in a random order, -1 last (as the indexer's top-k + hands them over), each key's slot through its request's pages.""" + dev = g.device + which = range(len(reqs)) if which is None else which + topk, page, owners = spec["topk"], spec["page_size"], spec["owners"] + pos = torch.cat([torch.arange(reqs[r][0], sum(reqs[r]), device=dev) for r in which]) + req = torch.cat([torch.full((reqs[r][1],), r, device=dev) for r in which]) + first = torch.tensor([0, *counts[:-1]], device=dev).cumsum(0) + n_keys = pos[:, None] + 1 + col = torch.arange(topk + 1, device=dev)[None] + edge = col * n_keys // topk + width = edge[:, 1:] - edge[:, :-1] + draw = edge[:, :-1] + (torch.rand(pos.numel(), topk, generator=g, device=dev) * width).long() + draw[:, -1] = pos + del edge, width + short = torch.where(col[:, :-1] < n_keys, col[:, :-1], -1) + sel = torch.where(n_keys > topk, draw, short) + del draw, short + order = (torch.rand(sel.shape, generator=g, device=dev) + (sel < 0)).argsort(1) + sel = sel.gather(1, order) + del order + key = sel.clamp_min(0) + slot = (pages[first[req][:, None] + key // page] * owners + row) * page + key % page + slot = torch.where(sel >= 0, slot, -1) + assert int(slot.max()) < 2**31 + return slot.to(torch.int32) + + +def _queries(n: int, spec, g, scratch: int = 0) -> torch.Tensor: + """bf16 `[n, h_q, 576]`, the first h_out heads drawn and the rest zero (rope zero on a + NoPE layer); with `scratch`, the tail view of a scratch that many rows long whose rows + ahead of the view carry draws of their own.""" + dev = g.device + total = max(n, scratch) + q = torch.zeros(total, spec["h_q"], 576, dtype=torch.bfloat16, device=dev) + q[:, : spec["h_out"]] = torch.randn(total, spec["h_out"], 576, generator=g, device=dev) + if spec.get("nope"): + q[..., 512:] = 0 + return q[total - n :] + + +def _calls(spec, q, kv, idx, out, metas) -> None: + """Every layer's call, in sub-batches of `sub_batch` rows (one call without it), one + sched meta per sub-batch size shared by every layer.""" + n = q.shape[0] + sub = spec.get("sub_batch", n) + for _ in range(spec["layers"]): + for lo in range(0, n, sub): + hi = min(lo + sub, n) + meta = metas.setdefault(hi - lo, get_mla_metadata()[0]) + op(q[lo:hi], kv, idx[lo:hi], meta, SCALE, out[lo:hi]) + + +def _check(q, kv, idx, out, g) -> None: + """Every row finite; up to `_GATED_ROWS` rows against the entry's reference.""" + n = q.shape[0] + bad = ~torch.isfinite(out).flatten(1).all(1) + assert not bool(bad.any()), f"{int(bad.sum())} of {n} rows read an unheld pool byte" + rows = torch.arange(n, device=q.device) + if n > _GATED_ROWS: + drawn = torch.randperm(n - 128, generator=g, device=q.device)[: _GATED_ROWS - 128] + rows = torch.cat([rows[-128:], drawn]) + _gate(out[rows], op.reference(q[rows], kv, idx[rows], None, SCALE, out[rows])) + + +def _split_rows(meta) -> torch.Tensor: + """The rows FlashMLA's decode scheduler split over their keys on the call that filled + `meta`: more than one part in its `num_splits` prefix sums, the parts merged in fp32 by + the combine kernel.""" + starts = meta.num_splits + return (starts[1:] - starts[:-1]) > 1 + + +def _check_split(q, kv, idx, out, metas, sub) -> None: + """Gate the rows each sub-batch's scheduler split against the reference; print how many + per sub-batch size.""" + n = q.shape[0] + rows = torch.cat([lo + _split_rows(metas[min(lo + sub, n) - lo]).nonzero()[:, 0] + for lo in range(0, n, sub)]) # fmt: skip + if rows.numel(): + _gate(out[rows], op.reference(q[rows], kv, idx[rows], None, SCALE, out[rows])) + split = {size: int(_split_rows(m).sum()) for size, m in sorted(metas.items())} + print(f"context sub-batches: split rows per sub-batch size {split}") + + +def _run_ragged_tail(spec, seed, largest_first: bool = False) -> None: + """The last sub-batch of a `_STEP_ROWS`-row chunk behind `cached` keys at each r in `tails` + (every r in 1..`sub_batch` without it; `largest_first`, as the "1M tep8" cells' run walked + it, the largest r first): q, slots and out the views [lo, lo + r) of step buffers, lo = + `_STEP_ROWS` - `sub_batch`, as `_attn_ctx` slices them; each r one call on a sched meta of + its own into rows reset to NaN. Every row finite; a row neither this call's scheduler nor + the whole sub-batch's split is bitwise the whole sub-batch's; every row in the band + against the entry's reference, run once over the sub-batch's rows; rows ahead of the + views untouched. Prints how many rows the scheduler split per r.""" + dev = torch.device("cuda") + torch.cuda.reset_peak_memory_stats() + g = torch.Generator(device=dev).manual_seed(seed) + sub, cached = spec["sub_batch"], spec["cached"] + lo = _STEP_ROWS - sub + pool, row, pages, counts, _ = _pool(spec, [(cached, _STEP_ROWS)], g) + kv = pool.view(-1, spec["page_size"], 1, 656) + idx_buf = torch.full((_STEP_ROWS, spec["topk"]), -1, dtype=torch.int32, device=dev) + idx_buf[lo:] = _selections(spec, [(cached + lo, sub)], pages, counts, row, g) + idx = idx_buf[lo:] + lowest = (pool.shape[0] - spec["top_blocks"]) * spec["owners"] * spec["page_size"] + assert lowest * 656 >= 2**31 and int(idx.min()) >= lowest, lowest + q = _queries(sub, spec, g, _STEP_ROWS) + out_buf = torch.empty(_STEP_ROWS, spec["h_out"], 512, dtype=torch.bfloat16, device=dev) + out_buf.view(torch.int16).fill_(_OUT_FILL) + out = out_buf[lo:] + assert q.storage_offset() == lo * q.stride(0) and out.storage_offset() == lo * out.stride(0) + ref = op.reference(q, kv, idx, None, SCALE, out) + meta = get_mla_metadata()[0] + whole = op(q, kv, idx, meta, SCALE, torch.empty_like(out)) + whole_split = _split_rows(meta) + assert bool(torch.isfinite(whole).all()) + _gate(whole, ref) + split_rows, differ_rows, bitwise_rows = {}, 0, 0 + tails = spec.get("tails", range(1, sub + 1)) + # Largest first: each call's FlashMLA buffers then fit in blocks the caching allocator + # already holds. + for r in sorted(tails, reverse=True) if largest_first else tails: + o = out[:r] + o.view(torch.int16).fill_(_OUT_FILL) + meta = get_mla_metadata()[0] + op(q[:r], kv, idx[:r], meta, SCALE, o) + split = _split_rows(meta) + bad = ~torch.isfinite(o).flatten(1).all(1) + assert not bool(bad.any()), f"r {r}: {int(bad.sum())} rows read an unheld pool byte" + differ = (o.view(torch.int16) != whole[:r].view(torch.int16)).flatten(1).any(1) + unsplit = ~split & ~whole_split[:r] + stray = differ & unsplit + assert not bool(stray.any()), f"r {r}: {int(stray.sum())} unsplit rows differ" + try: + _gate(o, ref[:r]) + except AssertionError as exc: + raise AssertionError(f"r {r}: {exc}") from exc + split_rows[r] = int(split.sum()) + differ_rows += int(differ.sum()) + bitwise_rows += int(unsplit.sum()) + ahead = out_buf[:lo].view(torch.int16) + assert bool((ahead == _OUT_FILL).all()), "a call wrote ahead of its view" + # Both per-row checks saw rows: some calls split rows, some kept rows in one part, and + # some rows neither scheduler split were held bitwise. + assert 0 < sum(split_rows.values()) < sum(split_rows), "no split row, or no unsplit one" + assert bitwise_rows > 0, "no row neither scheduler split" + hist = dict(sorted(collections.Counter(split_rows.values()).items())) + named = {r: split_rows[r] for r in (1, 16, 96, 127, 128, 8175, 8192) if r in split_rows} + print(f"ragged tail behind {cached}: {len(split_rows)} calls, r in {min(split_rows)}.." + f"{max(split_rows)}; the whole sub-batch split {int(whole_split.sum())} rows; " + f"{sum(split_rows.values())} split rows over the calls, {differ_rows} not bitwise the " + f"whole sub-batch's, {bitwise_rows} neither split held bitwise; split rows per call " + f"(count: calls) {hist}; at r {named}; " + f"{torch.cuda.max_memory_reserved() / 2**30:.2f} GiB reserved at most") # fmt: skip + + +def _ragged_tail(spec, seed) -> None: + """A chunk's last sub-batch at each r of `tail_rows` (every r in 1..`sub_batch` without + it), as `_attn_ctx` calls it: rows `lead_rows` + [0, r) of the chunk behind `cached` keys, + q, slots and out views from that row of the step's buffers, each r one call on a sched + meta of its own into rows reset to NaN, largest r first (rising sizes leave the caching + allocator a new block per call). Row i is the same sequence at every r, so the reference + and one whole-sub-batch call run once. Per call: the rows the scheduler split, and any + row not bitwise the whole call's, gated against the reference; a row neither call split + bitwise the whole call's (itself gated); the call's rel L2 over every row. The step's + rows ahead of the views stay unwritten. Prints the rows split per call.""" + dev = torch.device("cuda") + g = torch.Generator(device=dev).manual_seed(seed) + sub, lead, cached = spec["sub_batch"], spec["lead_rows"], spec["cached"] + pool, row, pages, counts, _ = _pool(spec, [(cached, lead + sub)], g) + kv = pool.view(-1, spec["page_size"], 1, 656) + idx_buf = torch.full((lead + sub, spec["topk"]), -1, dtype=torch.int32, device=dev) + idx_buf[lead:] = _selections(spec, [(cached + lead, sub)], pages, counts, row, g) + idx = idx_buf[lead:] + lowest = (pool.shape[0] - spec["top_blocks"]) * spec["owners"] * spec["page_size"] + assert lowest * 656 >= 2**31 and int(idx.min()) >= lowest, lowest + q = _queries(sub, spec, g, lead + sub) + out_buf = torch.empty(lead + sub, spec["h_out"], 512, dtype=torch.bfloat16, device=dev) + out_buf.view(torch.int16).fill_(_OUT_FILL) + out = out_buf[lead:] + assert q.storage_offset() == lead * q.stride(0) and out.storage_offset() == lead * out.stride(0) + ref = op.reference(q, kv, idx, None, SCALE, out).float() + meta = get_mla_metadata()[0] + whole = op(q, kv, idx, meta, SCALE, torch.empty_like(out)) + whole_split = _split_rows(meta) + _gate(whole, ref) + ref_norm = ref.norm(dim=-1) + ref_sq = ref_norm.square().sum(1).cumsum(0) + whole_err = (whole.float() - ref).square().sum(-1) + whole_sq = whole_err.sum(1).cumsum(0) + tails = spec.get("tail_rows", range(1, sub + 1)) + per, rel, stray, n_split, n_differ = (torch.zeros(len(tails), device=dev) for _ in range(5)) + for i in sorted(range(len(tails)), key=lambda i: -tails[i]): + r = tails[i] + o = out[:r] + o.view(torch.int16).fill_(_OUT_FILL) + meta = get_mla_metadata()[0] + op(q[:r], kv, idx[:r], meta, SCALE, o) + split = _split_rows(meta) + differ = torch.ne(o.view(torch.int64), whole[:r].view(torch.int64)).flatten(1).any(1) + gated = (split | differ).nonzero()[:, 0] + err = (o[gated].float() - ref[gated]).square().sum(-1) + per[i] = torch.cat([(err.sqrt() / ref_norm[gated]).flatten(), per[i : i + 1]]).max() + err_sq = whole_sq[r - 1] - whole_err[gated].sum() + err.sum() + rel[i] = (err_sq.clamp_min(0) / ref_sq[r - 1]).sqrt() + stray[i] = (differ & ~split & ~whole_split[:r]).sum() + n_split[i], n_differ[i] = split.sum(), differ.sum() + bad = ((per < 1e-2) & (rel < 4e-3) & (stray == 0)).logical_not().nonzero()[:8, 0].tolist() + assert not bad, [(tails[i], float(per[i]), float(rel[i]), int(stray[i])) for i in bad] + assert bool((out_buf[:lead].view(torch.int16) == _OUT_FILL).all()), "wrote ahead of the view" + splits = n_split.long().tolist() + # Both kinds of row were seen: some calls split rows, some kept rows in one part. + assert 0 < sum(splits) < sum(tails), "no split row, or no unsplit one" + hist = dict(sorted(collections.Counter(splits).items())) + shown = (1, 127, 128, 6445, 8096, 8175, 8192) + named = {r: n for r, n in zip(tails, splits) if r in shown} + print(f"ragged tail behind {cached + lead}: {len(tails)} calls, r {min(tails)}..{max(tails)}; " + f"the whole sub-batch split {int(whole_split.sum())} rows; {sum(splits)} split rows over " + f"the calls, {int(n_differ.sum())} not bitwise the whole call's; per-(row, head) rel L2 " + f"max {float(per.max()):.3e}, call max {float(rel.max()):.3e}; split rows per call " + f"(count: calls) {hist}; at r {named}") # fmt: skip + + +def _capture(spec, q, kv, idx, out) -> torch.cuda.CUDAGraph: + """The decode step's graph over the static q / idx / out: a warm-up call on a side stream, + then every layer's call on one fresh sched meta, which the graph's first call fills.""" + s = torch.cuda.Stream() + s.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(s): + op(q, kv, idx, get_mla_metadata()[0], SCALE, out) + torch.cuda.current_stream().wait_stream(s) + graph = torch.cuda.CUDAGraph() + meta = get_mla_metadata()[0] + with torch.cuda.graph(graph): + for _ in range(spec["layers"]): + op(q, kv, idx, meta, SCALE, out) + return graph + + +def _replay(graph, q, kv, idx, out, q_new, idx_new, g) -> None: + """Replay on new queries and slots into a NaN-filled `out`, check the replay, and match an + eager call on the same data bitwise.""" + q.copy_(q_new) + idx.copy_(idx_new) + out.view(torch.int16).fill_(_OUT_FILL) + graph.replay() + torch.cuda.synchronize() + _check(q, kv, idx, out, g) + eager = op(q, kv, idx, get_mla_metadata()[0], SCALE, torch.empty_like(out)) + assert torch.equal(eager.view(torch.int16), out.view(torch.int16)), "replay != eager" + + +def _run_boot_capture(spec, g, kv, reqs, pages, counts, row, first_last, lowest) -> None: + """Each batch b of `graph_batches` as the dep4 engine runs it: warm-up calls and the capture + on the boot rows (the longest dummy, b - 1 fillers), then two replays as attention DP + pads them: the live row behind b - 1 copies of the padding dummy's row (one slot, 2047 + x -1, one query), and an idle rank's dummy behind the same padding.""" + pad, idle = len(reqs) - 2, len(reqs) - 1 + + def rows(first: int, b: int): + idx = _selections(spec, reqs, pages, counts, row, g, which=[first]) + q = _queries(1, spec, g) + if b > 1: + padding = _selections(spec, reqs, pages, counts, row, g, which=[pad]) + idx = torch.cat([idx, padding.expand(b - 1, -1)]) + q = torch.cat([q, _queries(1, spec, g).expand(b - 1, -1, -1)]) + return q, idx + + for b in spec["graph_batches"]: + boot = [1, *range(2, b + 1)] + idx = _selections(spec, reqs, pages, counts, row, g, which=boot) + assert idx.shape[0] == b and int(idx[idx >= 0].min()) >= lowest + assert int((idx[0] >= 0).sum()) == spec["topk"] and bool((idx[1:, 1:] == -1).all()) + q = _queries(b, spec, g) + out = torch.empty(b, spec["h_out"], 512, dtype=torch.bfloat16, device=q.device) + out.view(torch.int16).fill_(_OUT_FILL) + _calls(spec, q, kv, idx, out, {}) + _check(q, kv, idx, out, g) + graph = _capture(spec, q, kv, idx, out) + q_live, idx_live = rows(0, b) + if spec.get("far"): + assert int(idx_live.max()) == first_last, (int(idx_live.max()), first_last) + assert int((idx_live[1:] >= 0).sum()) == b - 1 + assert bool((idx_live[1:] == idx_live[-1]).all() and (q_live[1:] == q_live[-1]).all()) + _replay(graph, q, kv, idx, out, q_live, idx_live, g) + _replay(graph, q, kv, idx, out, *rows(idle, b), g) + del graph + + +def _run_1m(spec, seed) -> None: + dev = torch.device("cuda") + g = torch.Generator(device=dev).manual_seed(seed) + batches = spec.get("graph_batches", spec.get("row_counts", (spec["rows"],))) + reqs = _requests(spec, max(batches)) + pool, row, pages, counts, first_last = _pool(spec, reqs, g) + page = spec["page_size"] + kv = pool.view(-1, page, 1, 656) + top = pool.shape[0] * spec["owners"] * page - 1 + if spec.get("far"): + print(f"pool {tuple(pool.shape)}: first request's last slot {first_last} " + f"(byte {first_last * 656}), pool's last slot {top}") # fmt: skip + if not spec.get("nope"): + assert first_last == top - (page - 1 - (sum(reqs[0]) - 1) % page) + lowest = 0 + if "top_blocks" in spec: + lowest = (pool.shape[0] - spec["top_blocks"]) * spec["owners"] * page + assert lowest * 656 >= 2**31, lowest + if spec.get("boot_capture"): + _run_boot_capture(spec, g, kv, reqs, pages, counts, row, first_last, lowest) + return + for b in batches: + sub = reqs if spec.get("context") else reqs[:b] + idx = _selections(spec, sub, pages, counts, row, g) + assert int(idx.min()) >= -1 and int(idx[idx >= 0].min()) >= lowest + if spec.get("far"): + assert int(idx.max()) == first_last, (int(idx.max()), first_last) + n = idx.shape[0] + scratch = _STEP_ROWS if "q_offset" in spec else 0 + q = _queries(n, spec, g, scratch) + out_buf = torch.empty(max(n, scratch), spec["h_out"], 512, dtype=torch.bfloat16, device=dev) + out_buf.view(torch.int16).fill_(_OUT_FILL) + out = out_buf[out_buf.shape[0] - n :] + if "q_offset" in spec: + assert out_buf.shape[0] - n == spec["q_offset"] + assert q.storage_offset() == spec["q_offset"] * q.stride(0) + metas = {} + _calls(spec, q, kv, idx, out, metas) + if spec.get("context"): + assert max(metas) == spec["rows"] and n == sum(c[1] for c in reqs), sorted(metas) + else: + assert n == b + _check(q, kv, idx, out, g) + if spec.get("context"): + _check_split(q, kv, idx, out, metas, spec["sub_batch"]) + if "q_offset" in spec: + ahead = out_buf[: spec["q_offset"]].view(torch.int16) + assert bool((ahead == _OUT_FILL).all()), "the call wrote ahead of its view" + if spec.get("captured"): + graph = _capture(spec, q, kv, idx, out) + q_new, idx_new = _queries(n, spec, g), _selections(spec, sub, pages, counts, row, g) + _replay(graph, q, kv, idx, out, q_new, idx_new, g) + del graph + + +def _view_cell(spec, seed) -> None: + """`out` None: the returned `[rows, h_q, 512]` kernel output is gated against the + reference over every head, and its first `h_out` heads are bitwise what a call with `out` + copies out on the same inputs (the kernel is deterministic; measured).""" + q, kv, idx, out = _build(spec, seed) + meta = get_mla_metadata()[0] + for _ in range(spec["layers"]): + res = op(q, kv, idx, meta, SCALE, None) + assert res.shape == (spec["rows"], spec["h_q"], 512), res.shape + _gate(res, op.reference(q, kv, idx, meta, SCALE, None)) + op(q, kv, idx, meta, SCALE, out) + assert torch.equal(res[:, : spec["h_out"]].view(torch.int16), out.view(torch.int16)) + if spec.get("captured"): + meta = get_mla_metadata()[0] + s = torch.cuda.Stream() + s.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(s): + op(q, kv, idx, get_mla_metadata()[0], SCALE, None) + torch.cuda.current_stream().wait_stream(s) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + for _ in range(spec["layers"]): + res = op(q, kv, idx, meta, SCALE, None) + q2, kv2, idx2, _ = _build(spec, seed + 1) + q.copy_(q2), kv.copy_(kv2), idx.copy_(idx2) + graph.replay() + torch.cuda.synchronize() + _gate(res, op.reference(q, kv, idx, None, SCALE, None)) + + +def _run_1m_view(spec, seed) -> None: + """A "1M view" cell: decode rows over the "1M tep4" / "1M dep4" harness's pool, every + layer's call with `out` None on one sched meta (captured and replayed on new queries and + slots when `captured`). The returned kernel output is `[rows, h_q, 512]`; its first `h_out` + heads are every row finite and gated to the reference (`_check`), and bitwise what a call + with `out` copies out on the same inputs.""" + dev = torch.device("cuda") + g = torch.Generator(device=dev).manual_seed(seed) + batches = spec.get("graph_batches", (spec["rows"],)) + reqs = _requests(spec, max(batches)) + pool, row, pages, counts, first_last = _pool(spec, reqs, g) + kv = pool.view(-1, spec["page_size"], 1, 656) + h_out = spec["h_out"] + + def copied(q, idx) -> torch.Tensor: + out = torch.empty(q.shape[0], h_out, 512, dtype=torch.bfloat16, device=dev) + return op(q, kv, idx, get_mla_metadata()[0], SCALE, out).view(torch.int16) + + for b in batches: + idx = _selections(spec, reqs[:b], pages, counts, row, g) + if spec.get("far"): + assert int(idx.max()) == first_last, (int(idx.max()), first_last) + q = _queries(b, spec, g) + meta = get_mla_metadata()[0] + for _ in range(spec["layers"]): + res = op(q, kv, idx, meta, SCALE, None) + assert res.shape == (b, spec["h_q"], 512), res.shape + _check(q, kv, idx, res[:, :h_out], g) + assert torch.equal(res[:, :h_out].contiguous().view(torch.int16), copied(q, idx)) + if not spec.get("captured"): + continue + side = torch.cuda.Stream() + side.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(side): + op(q, kv, idx, get_mla_metadata()[0], SCALE, None) + torch.cuda.current_stream().wait_stream(side) + graph = torch.cuda.CUDAGraph() + meta = get_mla_metadata()[0] + with torch.cuda.graph(graph): + for _ in range(spec["layers"]): + res = op(q, kv, idx, meta, SCALE, None) + q.copy_(_queries(b, spec, g)) + idx.copy_(_selections(spec, reqs[:b], pages, counts, row, g)) + res.view(torch.int16).fill_(_OUT_FILL) + graph.replay() + torch.cuda.synchronize() + _check(q, kv, idx, res[:, :h_out], g) + live = res[:, :h_out].contiguous().view(torch.int16) + assert torch.equal(live, copied(q, idx)), "the replay differs from an eager out= call" + del graph + + +def _is_1m(spec) -> bool: + keys = ("graph_batches", "row_counts", "far", "top_blocks", "context", "q_offset") + return any(k in spec for k in keys) + + +_CELLS = [ + pytest.param(c, id=c.why[:60], marks=[pytest.mark.large] if c.spec.get("large") else []) + for c in op.CELLS +] + + +@pytest.mark.parametrize("cell", _CELLS) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — fixture by name + certifying(op) + spec = cell.spec + seed = zlib.crc32(cell.why.encode()) + if spec.get("view"): + try: + (_run_1m_view if _is_1m(spec) else _view_cell)(spec, seed) + finally: + torch.cuda.empty_cache() + return + if _is_1m(spec): + try: + if cell.why.startswith("1M tp4 "): + ragged = spec.get("ragged_tail") or "tail_rows" in spec + (_ragged_tail_tp4 if ragged else _run_1m_tp4)(spec, seed) + elif cell.why.startswith(("1M tep4 ", "1M tep8 ")) and spec.get("ragged_tail"): + _run_ragged_tail(spec, seed, largest_first=cell.why.startswith("1M tep8 ")) + elif spec.get("ragged_tail") or "tail_rows" in spec: + _ragged_tail(spec, seed) + else: + _run_1m(spec, seed) + finally: + torch.cuda.empty_cache() + return + q, kv, idx, out = _build(spec, seed) + meta = get_mla_metadata()[0] + for _ in range(spec["layers"]): + op(q, kv, idx, meta, SCALE, out) + _gate(out, op.reference(q, kv, idx, meta, SCALE, out)) + if spec.get("captured"): + meta = get_mla_metadata()[0] + s = torch.cuda.Stream() + s.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(s): + op(q, kv, idx, get_mla_metadata()[0], SCALE, out) + torch.cuda.current_stream().wait_stream(s) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + for _ in range(spec["layers"]): + op(q, kv, idx, meta, SCALE, out) + q2, kv2, idx2, _ = _build(spec, seed + 1) + q.copy_(q2), kv.copy_(kv2), idx.copy_(idx2) + graph.replay() + torch.cuda.synchronize() + _gate(out, op.reference(q, kv, idx, None, SCALE, out)) + + +def test_is_valid_refuses_sixteen_kernel_heads() -> None: + q, kv, idx, out = _build(dict(rows=2, h_q=64, h_out=16, topk=2048, kv_len=3000, page_size=32, + owners=4, pages=400), 1) # fmt: skip + with pytest.raises(ValueError, match="h_q"): + op.is_valid(q[:, :16], kv, idx, None, SCALE, out) + + +def test_is_valid_refuses_more_than_8192_rows() -> None: + """Past its shared-memory row limit FlashMLA's scheduler exits the process ("invalid + argument", the note's Hard limits), so nothing here launches: shapes only, on expanded + views of the tep8 call (8 heads out), the tp4 and tep4 call (16 heads out) and the dep4 + call (64 heads in and out). 8192 rows pass is_valid; 8193, 11623, 16384 and 32768 (a + whole chunk in one call) are refused before the call reaches the kernel.""" + dev = torch.device("cuda") + kv = torch.full((64, 32, 1, 656), _POOL_FILL, dtype=torch.uint8, device=dev) + cases = ((8192, False), (8193, True), (11623, True), (16384, True), (32768, True)) + for h_out in (8, 16, 64): + for n, refused in cases: + q = torch.zeros(1, 64, 576, dtype=torch.bfloat16, device=dev).expand(n, 64, 576) + idx = torch.full((1, 2048), -1, dtype=torch.int32, device=dev).expand(n, 2048) + out = torch.empty(1, h_out, 512, dtype=torch.bfloat16, device=dev) + out = out.expand(n, h_out, 512) + if not refused: + op.is_valid(q, kv, idx, None, SCALE, out) + continue + with validating(op), pytest.raises(ValueError, match=f"{n} rows in one call"): + op(q, kv, idx, get_mla_metadata()[0], SCALE, out) + + +def test_1m_tp4_harness_sees_a_misread() -> None: + """The "1M tp4" cells' harness against two wrong slot maps on a 1100-block pool: every key one + page on (the next owner row, unwritten: NaN rows) and every key at the neighbouring + offset of its page (written rows, other keys: outside the band). Both fail, the right + map passes.""" + tail_view = next(c for c in op.CELLS if c.why.startswith("1M tp4 mixed step's tail view")) + spec = {k: v for k, v in tail_view.spec.items() if k != "q_offset"} + assert spec["pages"] == 1100 * 63 and not spec.get("far") + g = torch.Generator(device="cuda").manual_seed(7) + reqs = _requests_tp4(spec, 16) + pool, row, pages, counts, _ = _pool_1m(spec, reqs, g) + kv = pool.view(-1, spec["page_size"], 1, 656) + idx = _selections_tp4(spec, reqs, pages, counts, row, g) + q = _queries_tp4(idx.shape[0], spec, g) + out = torch.empty(idx.shape[0], spec["h_out"], 512, dtype=torch.bfloat16, device="cuda") + next_page = torch.where(idx >= 0, idx + spec["page_size"], idx) + neighbour = torch.where(idx >= 0, idx ^ 1, idx) + for wrong, seen in ((next_page, "unheld pool byte"), (neighbour, r"^\(")): + op(q, kv, wrong, get_mla_metadata()[0], SCALE, out) + with pytest.raises(AssertionError, match=seen): + _check_tp4(q, kv, idx, out, g) + op(q, kv, idx, get_mla_metadata()[0], SCALE, out) + _check_tp4(q, kv, idx, out, g) + + +def test_1m_harness_sees_a_misread_on_a_tep4_pool() -> None: + """The 1M harness against three wrong slot maps over the pre-merge context cell's pool, + whose requests' keys all sit past 2**31 bytes: every key one owner row down (row 61 of + its block, unwritten: NaN rows), every key's byte offset wrapped at 2**31 (what an + offset kept in 31 bits reads: the unwritten low blocks), and every key at the neighbouring + offset of its page (written rows, other keys: outside the band). Each is refused; the + right map passes.""" + cell = next(c for c in op.CELLS if c.why.startswith("1M tep4 pre-merge context")) + spec = dict(cell.spec, chunk_rows=1024, cached=1047552) + g = torch.Generator(device="cuda").manual_seed(7) + reqs = _requests(spec, 1) + pool, row, pages, counts, _ = _pool(spec, reqs, g) + kv = pool.view(-1, spec["page_size"], 1, 656) + idx = _selections(spec, reqs, pages, counts, row, g) + q = _queries(idx.shape[0], spec, g) + out = torch.empty(idx.shape[0], spec["h_out"], 512, dtype=torch.bfloat16, device="cuda") + valid = idx >= 0 + row_down = torch.where(valid, idx - spec["page_size"], idx) + wrapped = torch.where(valid, (idx.long() * 656 % 2**31 // 656).int(), idx) + neighbour = torch.where(valid, idx ^ 1, idx) + for wrong, seen in ((row_down, "unheld pool byte"), (wrapped, "unheld pool byte"), + (neighbour, r"^\(")): # fmt: skip + op(q, kv, wrong, get_mla_metadata()[0], SCALE, out) + with pytest.raises(AssertionError, match=seen): + _check(q, kv, idx, out, g) + op(q, kv, idx, get_mla_metadata()[0], SCALE, out) + _check(q, kv, idx, out, g) + + +def test_1m_harness_sees_a_misread() -> None: + """The 1M harness against three wrong slot maps over the pre-merge context cell's pool past + 2**31 bytes, its requests' keys all there: every key one owner row down (row 61 of its + block, unwritten: NaN rows), every key's byte offset wrapped at 2**31 (what an offset kept + in 31 bits reads: the unwritten low blocks), and every key at the neighbouring offset of + its page (written rows, other keys: outside the band). Each is refused; the right map + passes.""" + cell = next(c for c in op.CELLS if c.why.startswith("1M dep4 pre-merge context sub-batches")) + spec = dict(cell.spec, chunk_rows=1024, cached=1047552) + g = torch.Generator(device="cuda").manual_seed(7) + reqs = _requests(spec, 1) + pool, row, pages, counts, _ = _pool(spec, reqs, g) + kv = pool.view(-1, spec["page_size"], 1, 656) + idx = _selections(spec, reqs, pages, counts, row, g) + q = _queries(idx.shape[0], spec, g) + out = torch.empty(idx.shape[0], spec["h_out"], 512, dtype=torch.bfloat16, device="cuda") + valid = idx >= 0 + row_down = torch.where(valid, idx - spec["page_size"], idx) + wrapped = torch.where(valid, (idx.long() * 656 % 2**31 // 656).int(), idx) + neighbour = torch.where(valid, idx ^ 1, idx) + for wrong, seen in ((row_down, "unheld pool byte"), (wrapped, "unheld pool byte"), + (neighbour, r"^\(")): # fmt: skip + op(q, kv, wrong, get_mla_metadata()[0], SCALE, out) + with pytest.raises(AssertionError, match=seen): + _check(q, kv, idx, out, g) + op(q, kv, idx, get_mla_metadata()[0], SCALE, out) + _check(q, kv, idx, out, g) + + +def test_1m_harness_sees_a_misread_on_a_tep8_pool() -> None: + """The 1M harness against three wrong slot maps over the pre-merge context cell's pool, + whose requests' keys all sit past 2**31 bytes: every key one owner row down (row 61 of + its block, unwritten: NaN rows), every key's byte offset wrapped at 2**31 (what an + offset kept in 31 bits reads: the unwritten low blocks), and every key at the neighbouring + offset of its page (written rows, other keys: outside the band). Each is refused; the + right map passes.""" + cell = next(c for c in op.CELLS if c.why.startswith("1M tep8 pre-merge context")) + spec = dict(cell.spec, chunk_rows=1024, cached=1047552) + g = torch.Generator(device="cuda").manual_seed(7) + reqs = _requests(spec, 1) + pool, row, pages, counts, _ = _pool(spec, reqs, g) + kv = pool.view(-1, spec["page_size"], 1, 656) + idx = _selections(spec, reqs, pages, counts, row, g) + q = _queries(idx.shape[0], spec, g) + out = torch.empty(idx.shape[0], spec["h_out"], 512, dtype=torch.bfloat16, device="cuda") + valid = idx >= 0 + row_down = torch.where(valid, idx - spec["page_size"], idx) + wrapped = torch.where(valid, (idx.long() * 656 % 2**31 // 656).int(), idx) + neighbour = torch.where(valid, idx ^ 1, idx) + for wrong, seen in ((row_down, "unheld pool byte"), (wrapped, "unheld pool byte"), + (neighbour, r"^\(")): # fmt: skip + op(q, kv, wrong, get_mla_metadata()[0], SCALE, out) + with pytest.raises(AssertionError, match=seen): + _check(q, kv, idx, out, g) + op(q, kv, idx, get_mla_metadata()[0], SCALE, out) + _check(q, kv, idx, out, g) + + +@pytest.mark.parametrize("rows", [1, 16, 64, 256]) +@pytest.mark.parametrize("h_out", [8, 16, 64]) +def test_v_b_over_the_view_bit_equal_to_over_the_copy(h_out, rows) -> None: + """Old versus new on identical inputs, `torch.equal` on the bits: the decode targets' V + up-projection BMM over the live heads of the returned 64-head output, read in place, + equals the same BMM over those heads copied out (`out`), with `b` the transposed view of + the V rows as the targets pass it.""" + spec = dict(rows=rows, h_q=64, h_out=h_out, topk=2048, kv_len=4096, page_size=32, + owners=63, pages=512, layers=1) # fmt: skip + q, kv, idx, out = _build(spec, 1000 * h_out + rows) + meta = get_mla_metadata()[0] + view = op(q, kv, idx, meta, SCALE, None)[:, :h_out] + op(q, kv, idx, meta, SCALE, out) + assert torch.equal(view.view(torch.int16), out.view(torch.int16)) + g = torch.Generator(device="cuda").manual_seed(rows) + v_b_t = torch.transpose( + torch.randn(h_out, 128, 512, generator=g, device="cuda").to(torch.bfloat16), 1, 2 + ) + new = torch.empty(rows, h_out * 128, dtype=torch.bfloat16, device="cuda") + old = torch.empty_like(new) + bmm_out(torch.transpose(view, 0, 1), v_b_t, torch.transpose(new.view(rows, h_out, 128), 0, 1)) + bmm_out(torch.transpose(out, 0, 1), v_b_t, torch.transpose(old.view(rows, h_out, 128), 0, 1)) + assert torch.equal(new.view(torch.int16), old.view(torch.int16)) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_flash_mla_sparse_prefill.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_flash_mla_sparse_prefill.py new file mode 100644 index 000000000000..d916b51d32f2 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_flash_mla_sparse_prefill.py @@ -0,0 +1,567 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the flash_mla_sparse_prefill entry. + +bf16 inputs, bf16 P, fp32 accumulation against an fp32 reference: relative L2 per (row, head) +under 1e-2, whole call under 4e-3; the reference is a per-row loop and the kernel runs every +row. The workspace cells (`kv_rows`) are each driven by the harness of the run that certified +them, named by the why ("1M tp4"; "tep4 1M", "dep4 1M" and "tep8 1M" share one, dep4's +extension of tep4's, which tep8's is too). + +A cell of more than 1024 rows outside the "tep4 1M", "dep4 1M" and "tep8 1M" ones is gated on +its last 128 rows, 896 drawn ones, the first and last 8 of each request and, when the query +reaches it, the 16 rows around its 2**31-byte offset. A "1M tp4" cell also checks every row: +finite (a read of the stale NaN rows behind the workspace view would show), each 4096-row +window re-run bitwise equal to the full call's rows, and the rows of `out`'s buffer past the +call untouched. +The 1M cells of 16 rows or more measured per-row 4.1e-3..4.8e-3 and whole-call +1.8e-3..2.1e-3 on sm_103, the fresh 2- and 7-row dense steps at most 2.5e-3 per row, the +1-row step (one key) exactly 0; the two tests after the cells show the kernel skips an index +past the view without reading its row, and that the gate sees the workspace defects next to +this call. + +A "tep4 1M", "dep4 1M" or "tep8 1M" workspace cell is gated on 1024 sampled rows plus each +request's first and last rows and the rows where q's byte offset passes 2**31. It holds every +row finite (the buffer's stale rows and the rows after q and out are NaN) and bitwise equal to +the same rows called alone in windows of `_WINDOW`. A `row_sweep` cell calls every leading row +count of its call and a `row_counts` cell the counts it lists, each bitwise the full call's +leading rows and writing none of out's rows past its own; `large` cells are marked for the +stage that runs them. +""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.flash_mla_sparse_prefill import ( + flash_mla_sparse_prefill as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402,F401 — needs the path declared above + +assert torch.cuda.is_available(), "flash_mla_sparse_prefill requires a CUDA device" + +SCALE = 128**-0.5 +_GATED_ROWS = 1024 +#: The bits (bf16 NaN) of every workspace row past the view and of `out`'s buffer tail. +_STALE = -1 +_OUT_TAIL = 16 +_WINDOW_TP4 = 4096 +#: Query row whose bytes straddle 2**31 in a [rows, 64, 576] bf16 query (29127). +_Q_BYTE_EDGE = 2**31 // (64 * 576 * 2) +#: NaN rows after the q scratch's and the out buffer's rows, so a read or a write past the +#: call's rows shows. +_PAD = 16 +#: Row-independence windows: 4093 rows, so no window starts or ends on a power of two. +_WINDOW = 4093 +#: q is [rows, 64, 576] bf16, 73728 bytes a row: its byte offset passes 2**31 inside this row. +_Q_2GB_ROW = 2**31 // (64 * 576 * 2) + + +def _positions(spec): + """Each query row's position in its request and its request's run offset in `kv`.""" + kv_lens = spec["kv_len"] + new = spec.get("new", kv_lens) + pos, base = [], [] + off = 0 + for n_kv, n_new in zip(kv_lens, new): + pos.append(torch.arange(n_kv - n_new, n_kv, device="cuda")) + base.append(torch.full((n_new,), off, device="cuda")) + off += n_kv + return torch.cat(pos), torch.cat(base) + + +def _build(spec, seed): + g = torch.Generator(device="cuda").manual_seed(seed) + kv_lens = spec["kv_len"] + kv = (torch.randn(sum(kv_lens), 576, device="cuda", generator=g) * 2).to(torch.bfloat16) + # Row j of a request sees positions [0, p_j]: min(p_j + 1, topk) draws of them (with + # repeats, which the kernel and the reference both count), offset to the request's run. + pos, base = _positions(spec) + col = torch.arange(spec["topk"], device="cuda")[None] + draw = torch.rand(pos.numel(), spec["topk"], device="cuda", generator=g) * (pos[:, None] + 1) + idx = torch.where(col <= pos[:, None], draw.long() + base[:, None], -1) + if spec.get("holes"): + idx[torch.rand(idx.shape, device="cuda", generator=g) < 0.2] = -1 + idx = idx.to(torch.int32) + assert idx.shape[0] == spec["rows"] + q = torch.zeros(spec["rows"], spec["h_q"], 576, device="cuda") + q[:, : spec["h_out"]] = torch.randn( + spec["rows"], spec["h_out"], 576, device="cuda", generator=g + ) + out = torch.full((spec["rows"], spec["h_out"], 512), float("nan"), dtype=torch.bfloat16, + device="cuda") # fmt: skip + return q.to(torch.bfloat16), kv, idx, out + + +def _selection(pos: torch.Tensor, topk: int, g: torch.Generator, dense: bool) -> torch.Tensor: + """Request-local selections, int64 [rows, topk], for rows at positions `pos`. + + `dense`: the engine's dense causal index for a skipped indexer, 0..p then -1 (DSA + metadata `_get_dense_topk_indices`). Otherwise as the indexer's top-k returns them: a + row with p + 1 <= topk candidates lists 0..p in order then -1; a longer one holds topk + distinct positions, one per stratum of [0, p] (0 and p among them), in shuffled order. + """ + col = torch.arange(topk, device=pos.device)[None] + causal = torch.where(col <= pos[:, None], col, -1) + if dense: + return causal + n = pos[:, None] + 1 + lo = col * n // topk + hi = (col + 1) * n // topk + draw = torch.rand(lo.shape, device=pos.device, generator=g) + pick = torch.minimum(lo + (draw * (hi - lo)).long(), hi - 1) + pick[:, 0] = 0 + pick[:, -1] = pos + order = torch.rand(lo.shape, device=pos.device, generator=g).argsort(dim=1) + return torch.where(n > topk, pick.gather(1, order), causal) + + +def _build_1m(spec, seed): + """The call's arguments, and the buffer `out` leads. + + `kv` is the first N rows of a `kv_rows`-row workspace whose later rows are NaN, as the + target's view of its 2**19-row buffer; `q` is zero past `h_out` heads; `out` leads a + NaN buffer `_OUT_TAIL` rows longer, as the context rows lead the step's `lat_out`. + """ + g = torch.Generator(device="cuda").manual_seed(seed) + n = sum(spec["kv_len"]) + buf = torch.empty(spec["kv_rows"], 576, dtype=torch.bfloat16, device="cuda") + buf.view(torch.int16).fill_(_STALE) + buf[:n] = (torch.randn(n, 576, device="cuda", generator=g) * 2).to(torch.bfloat16) + pos, base = _positions(spec) + sel = _selection(pos, spec["topk"], g, spec.get("dense", False)) + idx = torch.where(sel >= 0, sel + base[:, None], sel).to(torch.int32) + rows = spec["rows"] + assert idx.shape[0] == rows + q = torch.zeros(rows, spec["h_q"], 576, dtype=torch.bfloat16, device="cuda") + q[:, : spec["h_out"]] = torch.randn(rows, spec["h_out"], 576, device="cuda", generator=g) + out_buf = torch.empty(rows + _OUT_TAIL, spec["h_out"], 512, dtype=torch.bfloat16, + device="cuda") # fmt: skip + out_buf.view(torch.int16).fill_(_STALE) + return (q, buf[:n], idx, out_buf[:rows]), out_buf + + +def _gated_rows_tp4(spec) -> torch.Tensor: + rows = torch.arange(spec["rows"], device="cuda") + if spec["rows"] <= _GATED_ROWS: + return rows + g = torch.Generator(device="cuda").manual_seed(0) + picked = torch.randperm(spec["rows"], device="cuda", generator=g)[: _GATED_ROWS - 128] + edges = [] + start = 0 + for n_new in spec.get("new", spec["kv_len"]): + edges += [*range(start, start + min(8, n_new)), *range(max(start, start + n_new - 8), + start + n_new)] # fmt: skip + start += n_new + if spec["rows"] > _Q_BYTE_EDGE + 8 and spec["h_q"] == 64: + edges += range(_Q_BYTE_EDGE - 8, _Q_BYTE_EDGE + 8) + extra = torch.tensor(edges, device="cuda", dtype=rows.dtype) + return torch.unique(torch.cat([rows[-128:], picked, extra])) + + +def _errors_tp4(o: torch.Tensor, r: torch.Tensor) -> tuple[float, float]: + """Largest relative L2 per (row, head), and the whole call's.""" + o, r = o.float(), r.float() + per = (o - r).norm(dim=-1) / r.norm(dim=-1).clamp_min(1e-30) + return float(per.max()), float((o - r).norm() / r.norm()) + + +def _assert_close(o: torch.Tensor, r: torch.Tensor) -> None: + per, whole = _errors_tp4(o, r) + assert per < 1e-2 and whole < 4e-3, (per, whole) + + +def _check_windows_tp4(q, kv, idx, out) -> int: + """Re-run every `_WINDOW_TP4` rows at query rows 0.. of their own call: each bitwise the full + call's rows, so no row's answer depends on where in the step it sits. Returns how many + calls ran.""" + rows = q.shape[0] + scratch = torch.empty(_WINDOW_TP4, out.shape[1], 512, dtype=out.dtype, device=out.device) + bad = torch.zeros((), dtype=torch.bool, device=out.device) + calls = 0 + for a in range(0, rows, _WINDOW_TP4): + b = min(a + _WINDOW_TP4, rows) + got = op(q[a:b], kv, idx[a:b], SCALE, scratch[: b - a]) + bad |= (got.view(torch.int16) != out[a:b].view(torch.int16)).any() + calls += 1 + assert not bad.item(), "a window's rows differ from the full call's" + return calls + + +def _runs(spec): + """Each query row's request-local position, and its request's offset in the workspace (the + run of a request starts where the previous request's `kv_len` ends).""" + pos, base = [], [] + off = 0 + for n_kv, n_new in zip(spec["kv_len"], spec["new"]): + pos.append(torch.arange(n_kv - n_new, n_kv, device="cuda")) + base.append(torch.full((n_new,), off, device="cuda")) + off += n_kv + return torch.cat(pos), torch.cat(base) + + +def _topk_selection(pos, topk, g): + """The indexer's top-`topk` output shape for rows at request-local positions `pos`. A row + that sees more than `topk` keys selects `topk` distinct ones, one in each of `topk` equal + strata of [0, p] (0 and p among them, so the run's first and last rows are read); a row that + sees fewer selects all of them, then -1. The valid selections are in a shuffled order.""" + col = torch.arange(topk, device="cuda")[None] + span = pos[:, None] + 1 + lo = col * span // topk + width = (col + 1) * span // topk - lo + pick = lo + (torch.rand(lo.shape, device="cuda", generator=g) * width).long() + pick[:, 0] = 0 + pick[:, -1] = pos + sel = torch.where(span > topk, pick, torch.where(col < span, col, -1)) + del lo, width, pick + key = torch.rand(sel.shape, device="cuda", generator=g).masked_fill_(sel < 0, 2.0) + return torch.gather(sel, 1, key.argsort(dim=1)) + + +def _dense_selection(pos, topk): + """DSA metadata's `_get_dense_topk_indices` for rows at request-local positions `pos` (the + cached prefix counted): column j holds j where j <= p, else -1.""" + col = torch.arange(topk, device="cuda")[None] + return torch.where(col <= pos[:, None], col, -1) + + +def _build_ws(spec, seed): + """A workspace cell's arguments, and the buffers they lead. + + kv is the first N = sum(kv_len) rows of a `kv_rows`-row bf16 buffer whose later rows are + NaN, as the bounded workspace's rows past the step's KV hold earlier steps' rows. The + indices are request-local selections plus each request's offset in the workspace, -1 kept. + q is the first `rows` rows of the 64-head scratch, `h_out` live heads (all 64 at dep4) and + zeros after them, followed by `_PAD` NaN rows; out leads a NaN buffer `_PAD` rows longer. + """ + g = torch.Generator(device="cuda").manual_seed(seed) + rows, n = spec["rows"], sum(spec["kv_len"]) + assert n <= spec["kv_rows"] + buf = torch.full((spec["kv_rows"], 576), float("nan"), dtype=torch.bfloat16, device="cuda") + buf[:n] = (torch.randn(n, 576, device="cuda", generator=g) * 2).to(torch.bfloat16) + pos, base = _runs(spec) + assert pos.numel() == rows + if spec.get("dense"): + sel = _dense_selection(pos, spec["topk"]) + else: + assert spec.get("selection", "topk") == "topk" + sel = _topk_selection(pos, spec["topk"], g) + idx = torch.where(sel >= 0, sel + base[:, None], -1).to(torch.int32) + del sel + h_q, h_out = spec["h_q"], spec["h_out"] + q_buf = torch.full((rows + _PAD, h_q, 576), float("nan"), dtype=torch.bfloat16, device="cuda") + q_buf[:rows] = 0 + q_buf[:rows, :h_out] = torch.randn(rows, h_out, 576, device="cuda", generator=g) + out_buf = torch.full((rows + _PAD, h_out, 512), float("nan"), dtype=torch.bfloat16, + device="cuda") # fmt: skip + return (q_buf[:rows], buf[:n], idx, out_buf[:rows]), (q_buf, buf, out_buf) + + +def _gated_rows(spec, rows): + """The rows the reference gates: every row up to `_GATED_ROWS`, else the last 128 and 896 + drawn; a workspace cell adds each request's first and last 4 rows and the 8 rows around + `_Q_2GB_ROW`.""" + every = torch.arange(rows, device="cuda") + if rows <= _GATED_ROWS: + return every + g = torch.Generator(device="cuda").manual_seed(0) + picked = torch.randperm(rows, device="cuda", generator=g)[: _GATED_ROWS - 128] + gated = [every[-128:], picked] + if "kv_rows" not in spec: + return torch.cat(gated) + start = 0 + for n_new in spec["new"]: + gated += [every[start : start + 4], every[start + n_new - 4 : start + n_new]] + start += n_new + if rows > _Q_2GB_ROW + 4: + gated.append(every[_Q_2GB_ROW - 4 : _Q_2GB_ROW + 4]) + return torch.unique(torch.cat(gated)) + + +def _errors(out, ref): + """Largest per-(row, head) relative L2 error, and the whole call's.""" + o, r = out.float(), ref.float() + per = (o - r).norm(dim=-1) / r.norm(dim=-1).clamp_min(1e-30) + return float(per.max()), float((o - r).norm() / r.norm()) + + +def _bits(t): + return t.view(torch.int16) + + +def _check_windows(q, kv, idx, out): + """Every `_WINDOW` rows called alone answer bitwise as in the full call: no row depends on + the rows beside it or on where its call starts. One sync for all windows.""" + differ = torch.zeros((), dtype=torch.bool, device="cuda") + for s in range(0, q.shape[0], _WINDOW): + e = min(s + _WINDOW, q.shape[0]) + part = op(q[s:e], kv, idx[s:e], SCALE, torch.empty_like(out[s:e])) + differ |= (_bits(part) != _bits(out[s:e])).any() + assert not differ.item(), "a window of rows answered other than in the full call" + + +def _sweep(q, kv, idx, full, counts, call=op): + """For each m in `counts`, the call on q[:m] and idx[:m] into the first m rows of a NaN + buffer answers bitwise as the full call's first m rows, and no call writes a row of the + buffer past its own: each m's rows go back to NaN after their compare, so every row is NaN + at the end. Largest m first, so the kernel's own output reuses the first call's block. Rows + compare as int64 words, one flag per m, one sync at the end. `call` stands in for the op in + the harness control.""" + rows = q.shape[0] + buf = torch.full((rows + _PAD, *full.shape[1:]), float("nan"), dtype=full.dtype, + device="cuda") # fmt: skip + words = buf.view(rows + _PAD, -1).view(torch.int64) + want = full.reshape(rows, -1).view(torch.int64) + nan = int(words[-1, 0]) + order = sorted(counts, reverse=True) + bad = torch.zeros(len(order), dtype=torch.bool, device="cuda") + for i, m in enumerate(order): + call(q[:m], kv, idx[:m], SCALE, buf[:m]) + bad[i] = torch.ne(words[:m], want[:m]).any() + words[:m].fill_(nan) + failed = [m for m, b in zip(order, bad.tolist()) if b] + assert not failed, f"{len(failed)} row counts answered other than the full call: {failed[:8]}" + assert not bool(torch.ne(words, nan).any()), "a call wrote out's rows past its own" + + +def _run_ws(spec, seed) -> None: + """A "tep4 1M", "dep4 1M" or "tep8 1M" workspace cell, as its run certified it.""" + (q, kv, idx, out), (_, _, out_buf) = _build_ws(spec, seed) + got = op(q, kv, idx, SCALE, out) + assert got.data_ptr() == out.data_ptr() + rows = _gated_rows(spec, q.shape[0]) + ref = op.reference(q[rows], kv, idx[rows], SCALE, out[rows]) + per, whole = _errors(out[rows], ref) + assert per < 1e-2 and whole < 4e-3, (per, whole) + assert bool(torch.isfinite(out).all()), "a row read the buffer's stale rows or q's padding" + assert bool(torch.isnan(out_buf[q.shape[0] :]).all()), "the call wrote past out's rows" + if q.shape[0] > _WINDOW: + _check_windows(q, kv, idx, out) + if spec.get("row_sweep"): + _sweep(q, kv, idx, out, range(1, q.shape[0] + 1)) + elif "row_counts" in spec: + _sweep(q, kv, idx, out, spec["row_counts"]) + + +@pytest.mark.parametrize( + "cell", + [ + pytest.param(c, id=c.why[:60], marks=[pytest.mark.large] if c.spec.get("large") else []) + for c in op.CELLS + ], +) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — fixture by name + certifying(op) + seed = zlib.crc32(cell.why.encode()) + if "kv_rows" in cell.spec and not cell.why.startswith("1M tp4 "): + _run_ws(cell.spec, seed) + return + one_m = "kv_rows" in cell.spec + if one_m: + (q, kv, idx, out), out_buf = _build_1m(cell.spec, seed) + else: + q, kv, idx, out = _build(cell.spec, seed) + got = op(q, kv, idx, SCALE, out) + assert got.data_ptr() == out.data_ptr() + rows = _gated_rows_tp4(cell.spec) + ref = op.reference(q[rows], kv, idx[rows], SCALE, out[rows]) + _assert_close(out[rows], ref) + if one_m: + assert bool(torch.isfinite(out).all()), "a row came back non-finite" + assert not (out_buf[q.shape[0] :].view(torch.int16) != _STALE).any(), "wrote past out" + if q.shape[0] > _WINDOW_TP4: + assert _check_windows_tp4(q, kv, idx, out) > 1 + + +_SHORT = dict(rows=512, kv_len=[8192], new=[512], h_q=64, h_out=16, topk=2048, kv_rows=16384, + selection="topk") # fmt: skip + + +def test_indices_past_kv_are_skipped_not_read() -> None: + """An index at or past kv.shape[0] is skipped as -1 is (FlashMLA's contract for its + indices): bitwise the answer with -1 in its place, and the NaN rows it names behind the + view are never read. Control: the same indices over the whole buffer read those rows, + and the rows come back NaN, so the stale fill is visible to this harness.""" + (q, kv, idx, out), _ = _build_1m(_SHORT, 11) + n = kv.shape[0] + full = kv.as_strided((_SHORT["kv_rows"], 576), kv.stride()) + past = idx.clone() + past[:, 1::97] = n + past[:, 2::97] = _SHORT["kv_rows"] - 1 + past[:, 3::97] = 2**31 - 1 + dropped = torch.where(past >= n, -1, past) + got = op(q, kv, past, SCALE, torch.empty_like(out)) + want = op(q, kv, dropped, SCALE, torch.empty_like(out)) + assert bool(torch.isfinite(got).all()) + assert not (got.view(torch.int16) != want.view(torch.int16)).any() + stale = torch.where(past == 2**31 - 1, -1, past) + seen = op(q, full, stale, SCALE, torch.empty_like(out)) + assert bool(torch.isnan(seen).all(dim=-1).all()), "the stale rows were not read" + + +def test_gate_sees_a_short_workspace_and_wrong_offsets() -> None: + """The gate's sight of the workspace defects next to this call: a view of only the new + tokens' rows (the step's KV counted as new tokens, not cached + new) drops every + selection past it without an error, and runs offset by the new tokens instead of the + whole KV read the wrong rows. Both must land above 0.3, 30x the per-row band and 75x + the whole-call one (on the 1M cells' builds both measured 0.95-2.2 whole-call).""" + spec = dict(rows=64, kv_len=[4096, 6144], new=[32, 32], h_q=64, h_out=16, topk=2048, + kv_rows=16384, selection="topk") # fmt: skip + (q, kv, idx, out), _ = _build_1m(spec, 13) + ref = op.reference(q, kv, idx, SCALE, out) + _assert_close(op(q, kv, idx, SCALE, torch.empty_like(out)), ref) + short = op(q, kv[:64], idx, SCALE, torch.empty_like(out)) + per, whole = _errors_tp4(short, ref) + assert per > 0.3 and whole > 0.3, (per, whole) + wrong = torch.where(idx[32:] >= 0, idx[32:] - 4096 + 32, idx[32:]) + moved = op(q[32:], kv, wrong.contiguous(), SCALE, torch.empty_like(out[32:])) + per, whole = _errors_tp4(moved, ref[32:]) + assert per > 0.3 and whole > 0.3, (per, whole) + + +def test_indices_at_or_past_n_are_skipped() -> None: + """An index at or past kv.shape[0] (N, the buffer's last row, 2**31 - 1) answers bitwise as + -1 does: the row is skipped and never read, so a workspace view cut short drops selections + without an error or a NaN. Control: the same indices over the whole buffer, whose rows past + N are NaN, read them.""" + spec = dict(rows=64, kv_len=[3000], new=[64], h_q=64, h_out=16, topk=2048, kv_rows=4096) + (q, kv, idx, out), (_, buf, _) = _build_ws(spec, 7) + n = kv.shape[0] + past = idx.clone() + past[:, 5::64] = n + past[:, 6::64] = buf.shape[0] - 1 + past[:, 7::64] = 2**31 - 1 + skipped = torch.where(past >= n, -1, past).to(torch.int32) + got = op(q, kv, past, SCALE, torch.empty_like(out)) + want = op(q, kv, skipped, SCALE, torch.empty_like(out)) + assert bool(torch.isfinite(got).all()) + assert bool((_bits(got) == _bits(want)).all()), "an index past N changed the answer" + seen = op(q, buf, torch.where(past == 2**31 - 1, -1, past), SCALE, torch.empty_like(out)) + assert bool(torch.isnan(seen).all()), "the control did not read the NaN rows past N" + + +def test_gate_refuses_request_local_indices() -> None: + """The gate sees an offset error: the dense three-request cell's selections passed without + their requests' workspace offsets answer far outside the band.""" + cell = next(c for c in op.CELLS if c.spec.get("dense") and len(c.spec["kv_len"]) == 3) + (q, kv, idx, out), _ = _build_ws(cell.spec, zlib.crc32(cell.why.encode())) + _, base = _runs(cell.spec) + local = torch.where(idx >= 0, idx - base[:, None], -1).to(torch.int32) + wrong = op(q, kv, local, SCALE, torch.empty_like(out)) + per, whole = _errors(wrong, op.reference(q, kv, idx, SCALE, out)) + assert whole > 25 * 4e-3, (per, whole) + + +def test_sweep_sees_a_wrong_row_and_a_stray_write() -> None: + """The row sweep's harness sees what it certifies: against a full answer one bit off in row + 40 every m past 40 fails and none up to it, and a call that also writes the row after its + own fails the stray-write check.""" + spec = dict(rows=64, kv_len=[3000], new=[64], h_q=64, h_out=16, topk=2048, kv_rows=4096) + (q, kv, idx, out), _ = _build_ws(spec, 11) + op(q, kv, idx, SCALE, out) + _sweep(q, kv, idx, out, range(1, 65)) + off = out.clone() + off.view(torch.int16)[40, 3, 7] ^= 1 + with pytest.raises(AssertionError, match=r"^24 row counts .*: \[64, 63, 62"): + _sweep(q, kv, idx, off, range(1, 65)) + + def spill(q, kv, idx, scale, o): + op(q, kv, idx, scale, o) + after = o.storage_offset() + o.shape[0] * o.stride(0) + torch.as_strided(o, (1, *o.shape[1:]), o.stride(), after).zero_() + return o + + with pytest.raises(AssertionError, match="past its own"): + _sweep(q, kv, idx, out, range(1, 65), call=spill) + + +def test_indices_at_or_past_n_are_skipped_at_64_heads() -> None: + """An index at or past kv.shape[0] (N, the buffer's last row, 2**31 - 1) answers bitwise as + -1 does: the row is skipped and never read, so a workspace view cut short drops selections + without an error or a NaN. Control: the same indices over the whole buffer, whose rows past + N are NaN, read them.""" + spec = dict(rows=64, kv_len=[3000], new=[64], h_q=64, h_out=64, topk=2048, kv_rows=4096) + (q, kv, idx, out), (_, buf, _) = _build_ws(spec, 7) + n = kv.shape[0] + past = idx.clone() + past[:, 5::64] = n + past[:, 6::64] = buf.shape[0] - 1 + past[:, 7::64] = 2**31 - 1 + skipped = torch.where(past >= n, -1, past).to(torch.int32) + got = op(q, kv, past, SCALE, torch.empty_like(out)) + want = op(q, kv, skipped, SCALE, torch.empty_like(out)) + assert bool(torch.isfinite(got).all()) + assert bool((_bits(got) == _bits(want)).all()), "an index past N changed the answer" + seen = op(q, buf, torch.where(past == 2**31 - 1, -1, past), SCALE, torch.empty_like(out)) + assert bool(torch.isnan(seen).all()), "the control did not read the NaN rows past N" + + +def test_sweep_sees_a_wrong_row_and_a_stray_write_at_64_heads() -> None: + """The row sweep's harness sees what it certifies, at 64 live heads: against a full answer + one bit off in row 40 every m past 40 fails and none up to it, and a call that also writes + the row after its own fails the stray-write check.""" + spec = dict(rows=64, kv_len=[3000], new=[64], h_q=64, h_out=64, topk=2048, kv_rows=4096) + (q, kv, idx, out), _ = _build_ws(spec, 11) + op(q, kv, idx, SCALE, out) + _sweep(q, kv, idx, out, range(1, 65)) + off = out.clone() + off.view(torch.int16)[40, 63, 511] ^= 1 + with pytest.raises(AssertionError, match=r"^24 row counts .*: \[64, 63, 62"): + _sweep(q, kv, idx, off, range(1, 65)) + + def spill(q, kv, idx, scale, o): + op(q, kv, idx, scale, o) + after = o.storage_offset() + o.shape[0] * o.stride(0) + torch.as_strided(o, (1, *o.shape[1:]), o.stride(), after).zero_() + return o + + with pytest.raises(AssertionError, match="past its own"): + _sweep(q, kv, idx, out, range(1, 65), call=spill) + + +def test_indices_at_or_past_n_are_skipped_at_8_heads() -> None: + """An index at or past kv.shape[0] (N, the buffer's last row, 2**31 - 1) answers bitwise as + -1 does at the tep8 call's 8 live heads: the row is skipped and never read, so a workspace + view cut short drops selections without an error or a NaN. Control: the same indices over + the whole buffer, whose rows past N are NaN, read them.""" + spec = dict(rows=64, kv_len=[3000], new=[64], h_q=64, h_out=8, topk=2048, kv_rows=4096) + (q, kv, idx, out), (_, buf, _) = _build_ws(spec, 7) + n = kv.shape[0] + past = idx.clone() + past[:, 5::64] = n + past[:, 6::64] = buf.shape[0] - 1 + past[:, 7::64] = 2**31 - 1 + skipped = torch.where(past >= n, -1, past).to(torch.int32) + got = op(q, kv, past, SCALE, torch.empty_like(out)) + want = op(q, kv, skipped, SCALE, torch.empty_like(out)) + assert bool(torch.isfinite(got).all()) + assert bool((_bits(got) == _bits(want)).all()), "an index past N changed the answer" + seen = op(q, buf, torch.where(past == 2**31 - 1, -1, past), SCALE, torch.empty_like(out)) + assert bool(torch.isnan(seen).all()), "the control did not read the NaN rows past N" + + +def test_sweep_sees_a_wrong_row_and_a_stray_write_at_8_heads() -> None: + """The row sweep's harness sees what it certifies, at the tep8 call's 8 live heads: against + a full answer one bit off in row 40 (the last live head's last element) every m past 40 + fails and none up to it, and a call that also writes the row after its own fails the + stray-write check.""" + spec = dict(rows=64, kv_len=[3000], new=[64], h_q=64, h_out=8, topk=2048, kv_rows=4096) + (q, kv, idx, out), _ = _build_ws(spec, 11) + op(q, kv, idx, SCALE, out) + _sweep(q, kv, idx, out, range(1, 65)) + off = out.clone() + off.view(torch.int16)[40, 7, 511] ^= 1 + with pytest.raises(AssertionError, match=r"^24 row counts .*: \[64, 63, 62"): + _sweep(q, kv, idx, off, range(1, 65)) + + def spill(q, kv, idx, scale, o): + op(q, kv, idx, scale, o) + after = o.storage_offset() + o.shape[0] * o.stride(0) + torch.as_strided(o, (1, *o.shape[1:]), o.stride(), after).zero_() + return o + + with pytest.raises(AssertionError, match="past its own"): + _sweep(q, kv, idx, out, range(1, 65), call=spill) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_fp8_mqa_logits.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_fp8_mqa_logits.py new file mode 100644 index 000000000000..795b6da70ffc --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_fp8_mqa_logits.py @@ -0,0 +1,708 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the fp8_mqa_logits entry.""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.fp8_mqa_logits import ( + fp8_mqa_logits as op, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.fp8_mqa_logits import ( + mqa_logits_schedule, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402,F401 — needs the path declared above + +assert torch.cuda.is_available(), "fp8_mqa_logits requires a CUDA device" + +E4M3_MAX = 448.0 + +# Gate, in fp32 ulps. See `_compare`. +ULP = 2.0**-23 +ELEMENT_ULP = 8 +RMS_ULP = 8 + +#: The logits budget a tiling caller cuts its queries to: floor(2**24 / keys) +#: rows per call, 1820 at 9216 keys and 512 at 32768. +LOGITS_BUDGET = 2**24 + +#: The tep4, tp4 and tep8 deployments' budget at their 1M envelope: floor(2**28 / keys) +#: rows per call, 256 at 1048576 keys and 8192 at 32768, about 1 GiB of fp32 +#: logits. +LOGITS_BUDGET_1M = 2**28 + +#: Tiles per wider call in `test_2_28_tiles_are_bitwise_rows_of_a_wider_call`: +#: four 2**28-element tiles, about 4 GiB of fp32 logits. +WIDE_TILES = 4 + +#: Query rows per reference evaluation in `_check`. See there. +REF_ROWS = 2048 + + +def _magnitude(q, kv, weights) -> torch.Tensor: + """`M[i, j] = k_scale[j] * sum_h |weights[i, h]| * sum_d |q[i, h, d] * k[j, d]|`. + + The size of the terms logit `[i, j]` sums, without the cancellation between + them: what any evaluation's rounding error scales with. Every term is + non-negative and every fp8 code is exact in fp32, so an fp32 matmul is + exact up to its accumulation, ample for a scale. + """ + k, k_scale = kv + qa, ka, wa = q.float().abs(), k.float().abs(), weights.float().abs() + m = torch.zeros(q.shape[0], k.shape[0], device=q.device) + for h in range(q.shape[1]): + m += (qa[:, h] @ ka.t()) * wa[:, h : h + 1] + return m * k_scale.float().abs().unsqueeze(0) + + +def _errors(out: torch.Tensor, ref: torch.Tensor, args) -> tuple: + """Assert the -inf pattern exactly; return the in-window error terms. + + `(max error in ulps of M, sum of squared errors, sum of squared reference + values)`, so that `_check` can sum a call's blocks into one RMS. + """ + assert out.dtype == ref.dtype == torch.float32, (out.dtype, ref.dtype) + assert out.shape == ref.shape, (out.shape, ref.shape) + must_inf = torch.isneginf(ref) + assert bool(torch.isneginf(out[must_inf]).all()), "an out-of-window entry is not -inf" + live = torch.isfinite(ref) + assert bool(torch.isfinite(out[live]).all()), "an in-window entry is not finite" + o = torch.where(live, out, torch.zeros_like(out)).double() + r = torch.where(live, ref, torch.zeros_like(ref)).double() + m = _magnitude(*args[:3]).double().clamp_min(1e-30) + err = float(((o - r).abs() / m).max()) / ULP if o.numel() else 0.0 + return err, float(((o - r) ** 2).sum()), float((r**2).sum()) + + +def _assert_band(err: float, sq_err: float, sq_ref: float) -> None: + assert err <= ELEMENT_ULP, ( + f"max error {err:.3g} ulp of the term magnitude exceeds {ELEMENT_ULP}" + ) + rms = sq_err**0.5 / max(sq_ref**0.5, 1e-30) / ULP + assert rms <= RMS_ULP, f"relative RMS {rms:.3g} ulp exceeds {RMS_ULP}" + + +def _compare(out: torch.Tensor, ref: torch.Tensor, args) -> None: + """8 fp32 ulps of each logit's term magnitude `M`, and 8 ulps relative RMS. + + fp8 x fp8 products are exact in fp32, so kernel and fp64 reference differ + only where the kernel rounds: its tensor-core fp32 sum of 128 products per + head, its fp32 FMA chain over the 32 weighted heads (relu taken as + (a + |a|) / 2, which is exact), and the final product with the key scale. + Each rounding is bounded by the magnitude of what it sums, so the element + gate is stated in ulps of `M` (`_magnitude`) rather than of the logit: + a row whose window holds one key -- the first token of every request -- + has a logit that is one cancelling sum, and a gate relative to it is + ill-conditioned (measured up to 25.8 ulp of the row's largest |logit| on + the 16-request cells). Measured on sm_103 over every cell and every + 1820-row tile of the two 9216-key steps: at most 0.20 ulp of `M` per + element, 0.57 ulp relative RMS (about what rounding the output to fp32 + costs); over the two 32768-key steps, whole and as 512-row tiles, at most + 0.23 ulp and 0.572 ulp; over the sixteen 1M-envelope cells (tiles at 32768 + to 1048576 keys, packed chunks at 2097152 and 2373713), at most 0.220 ulp + and 0.604 ulp (the 2-row packed chunk), and over the twelve of them tep4 + certified (tiles at 32768 to 1048576 keys) at most 0.220 ulp and 0.582 ulp; + over dep4's 1M envelope cells (32768 to 1966090 keys: the 2**28, 2**26 and + 2**24 tiles and the packed blocks) at most 0.245 ulp and 0.577 ulp, and + 0.250 and 0.590 at other seeds; over the fifteen tep8 1M cells (share tiles + at 4096 to 1048576 keys) at most 0.276 ulp and 0.573 ulp. The gate is 29x + and 13x above the largest of those (35x and 13x above tp4's, 35x and 14x + above tep4's, 29x and 14x above tep8's). A wrong + formula lands far outside: no key scale at 1.5e8 ulp of `M`, a + neighbouring row's head weights at 1.3e6, relu after the head sum at + 8.4e5, the second kernel fp16 weights select at 458 (RMS 4000 ulp), and + merely rounding the head weights to fp16 at 195 (RMS 1777 ulp). + + The -inf pattern outside the windows is exact: the top-k that follows + treats those columns as absent, so a window off by one key fails here. + NaN in `ref` means unspecified, -inf must be -inf, and a finite entry must + sit within the band. + """ + _assert_band(*_errors(out, ref, args)) + + +def _check(out: torch.Tensor, args) -> None: + """`_compare` of a whole call against `op.reference`, `REF_ROWS` query rows at a time. + + The reference is row-separable: row `i` reads `q[i]`, `weights[i]` and the + keys of its own window, and is -inf everywhere else. So each block of rows + is evaluated over its key span `[c0, c1)` -- the smallest window start to + the largest window end, clipped to the keys -- with the block's windows + shifted by `c0`; outside the span every window of the block is empty, the + reference holds -inf there, and the output must too, exactly. The blocks' + error terms are summed, so the RMS gate is the whole call's. This keeps the + fp64 transients at `[REF_ROWS, c1 - c0]` and skips the keys no row of a + block can see -- most of a causal step's upper triangle, all but the + diagonal of a packed one. fp64 matmuls are slow on this GPU, so their + work decides what the 32768-key cells cost: measured 4.87 s and + 3.06 GiB for the one-request call, against 8.73 s and 6.12 GiB over every + key in 4096-row blocks; an unblocked reference holds several [T, N] fp64 + tensors, 8 GiB each at T = N = 32768. A 2**28-element tile costs about the + same whatever its keys: [256, 1048576] in one block 2.2 s and 11.5 GiB + above its inputs, [8192, 32768] in four 2.0 s and 2.9 GiB (measured). + `test_the_blocked_gate_sees_what_it_skips` shows the skipped entries are + still checked. + """ + q, (k, k_scale), w, ks, ke = args + n = k.shape[0] + assert out.dtype == torch.float32 and tuple(out.shape) == (q.shape[0], n), out.shape + err, sq_err, sq_ref = 0.0, 0.0, 0.0 + for lo in range(0, q.shape[0], REF_ROWS): + rows = slice(lo, lo + REF_ROWS) + c0 = max(0, min(int(ks[rows].min()), n)) + c1 = max(min(int(ke[rows].max()), n), c0) + block = out[rows] + outside = torch.isneginf(block[:, :c0]).all() & torch.isneginf(block[:, c1:]).all() + assert bool(outside), "an out-of-window entry is not -inf" + if c1 == c0: + continue + span = (q[rows], (k[c0:c1], k_scale[c0:c1]), w[rows], ks[rows] - c0, ke[rows] - c0) + e, se, sr = _errors(block[:, c0:c1], op.reference(*span), span) + err, sq_err, sq_ref = max(err, e), sq_err + se, sq_ref + sr + _assert_band(err, sq_err, sq_ref) + + +def _quantize_rows(x: torch.Tensor): + """Per-row e4m3 with an fp32 scale, as the indexer quantizes its keys.""" + scale = (x.float().abs().amax(dim=-1) / E4M3_MAX).clamp_min(1e-12) + return (x.float() / scale.unsqueeze(-1)).to(torch.float8_e4m3fn), scale.contiguous() + + +def _step(spec, seed): + """`(q, (k, k_scale), weights, ks, ke)` for every query row of one context step. + + The weights carry the indexer's own scale, `D**-0.5 * H**-0.5`, so the + logits sit at the magnitudes the target produces. + """ + g = torch.Generator(device="cuda").manual_seed(seed) + heads, dim = spec["heads"], spec["head_dim"] + starts, ends, col = [], [], 0 + for cached, new in spec["seqs"]: + for t in range(new): + starts.append(col) + ends.append(col + cached + t + 1) + col += cached + new + rows = len(starts) + q = torch.randn(rows, heads, dim, generator=g, device="cuda").to(torch.float8_e4m3fn) + k, k_scale = _quantize_rows(torch.randn(col, dim, generator=g, device="cuda")) + w = torch.randn(rows, heads, generator=g, device="cuda") * (dim**-0.5 * heads**-0.5) + ks = torch.tensor(starts, dtype=torch.int32, device="cuda") + ke = torch.tensor(ends, dtype=torch.int32, device="cuda") + return q, (k, k_scale), w.contiguous(), ks, ke + + +def _build(spec, seed): + """One call's arguments: the step's query tile `rows`, as views, or the whole step.""" + q, kv, w, ks, ke = _step(spec, seed) + lo, hi = spec.get("rows", (0, q.shape[0])) + return q[lo:hi], kv, w[lo:hi], ks[lo:hi], ke[lo:hi] + + +def _schedule(spec, args): + """The split-KV schedule a `split_kv` cell's call takes (`mqa_logits_schedule` over its + own windows and keys), else None: the grid-stride call.""" + if not spec.get("split_kv"): + return None + q, (k, _), _, ks, ke = args + return mqa_logits_schedule(ks, ke, k.shape[0], q.shape[1]) + + +@pytest.mark.parametrize( + "cell", + [pytest.param(c, marks=pytest.mark.large) if c.spec.get("large") else c for c in op.CELLS], + ids=[c.why[:44] for c in op.CELLS], +) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 + certifying(op) + args = _build(cell.spec, seed=zlib.crc32(cell.why.encode()) % 2**31) + out = op(*args, schedule_meta=_schedule(cell.spec, args)) + q, (k, _), *_ = args + n = k.shape[0] + assert tuple(out.shape) == (q.shape[0], n) + # A view of rows padded to align(N + 256, 256): the prefill top-k reads it + # at stride(0) as it stands. + assert out.stride(1) == 1 and out.stride(0) == (n + 256 + 255) // 256 * 256, out.stride() + _check(out, args) + + +#: The whole step of every cell that scores a query tile of one, at the 2**24 +#: budget: steps whose untiled call is itself a cell. The `budget` cells' steps +#: (dep4's) are not: at 32768 to 1048576 keys a whole chunk is [32768, n_k], up +#: to 2**35 logits, which no caller computes in one call +#: (`test_budget_tiles_are_bitwise_rows_of_the_2_28_tile`); nor are the +#: "Mistral Large 4 1M" cells' (tp4's and tep4's, `_TILED_STEPS_1M`), nor the +#: `share` cells' (tep8's, `_TEP8_SHARES`). tp4's and tep4's files left out +#: their 1M cells by the why prefix, dep4's its 1M cells by the `budget` key, +#: tep8's its 1M cells by the why prefix and the `share` key; no tp4, tep4 or +#: tep8 cell carries `budget`, no dep4 why starts with "Mistral Large 4 1M" and no +#: tp4, tep4 or dep4 cell carries `share`, so each member's cells select here +#: exactly as in its own file (the base cells' four steps). +_TILED_STEPS = { + f"{len(c.spec['seqs'])}-request-{sum(a + b for a, b in c.spec['seqs'])}-key": { + k: v for k, v in c.spec.items() if k != "rows" + } + for c in op.CELLS + if "rows" in c.spec + and "budget" not in c.spec + and "share" not in c.spec + and not c.why.startswith("Mistral Large 4 1M") +} + +#: The whole step of every 1M-envelope cell that scores a 2**28 tile of one: +#: the "Mistral Large 4 1M" cells (tp4's and tep4's). dep4's 1M cells, whose whys +#: start otherwise, are not here; `_TILES_2_28` takes its `budget` cells. Nor +#: are tep8's ("Mistral Large 4 tep8 1M"), whose tiles are cut from their rank's +#: share: `_TEP8_SHARES` takes those. +_TILED_STEPS_1M = { + f"{len(c.spec['seqs'])}-request-{sum(a + b for a, b in c.spec['seqs'])}-key": { + k: v for k, v in c.spec.items() if k != "rows" + } + for c in op.CELLS + if "rows" in c.spec and c.why.startswith("Mistral Large 4 1M") +} + + +@pytest.mark.parametrize("spec", list(_TILED_STEPS.values()), ids=list(_TILED_STEPS)) +def test_query_tiles_are_bitwise_rows_of_one_call(spec) -> None: + """Tiling the queries changes no bit: a row's logits depend on its own window only. + + The untiled call of each tiled step is a certified cell, so this carries + the certification to every tile a 2**24-element logits budget cuts (at + 9216 keys five of 1820 rows and the 116-row tail, at 32768 keys sixty-four + of 512 rows), and shows a repeated call is bitwise the same. + """ + q, kv, w, ks, ke = _build(spec, seed=13) + whole = op(q, kv, w, ks, ke) + assert torch.equal(op(q, kv, w, ks, ke), whole), "a repeated call is not bitwise the same" + tile = LOGITS_BUDGET // kv[0].shape[0] + for lo in range(0, q.shape[0], tile): + hi = min(lo + tile, q.shape[0]) + out = op(q[lo:hi], kv, w[lo:hi], ks[lo:hi], ke[lo:hi]) + assert torch.equal(out, whole[lo:hi]), f"rows [{lo}, {hi}) differ from the untiled call" + + +@pytest.mark.parametrize("spec", list(_TILED_STEPS_1M.values()), ids=list(_TILED_STEPS_1M)) +def test_2_28_tiles_are_bitwise_rows_of_a_wider_call(spec) -> None: + """The tiles the 1M envelope's budget cuts change no bit either. + + The target cuts a chunk's queries into floor(2**28 / keys)-row tiles from + the chunk's first row. A call over the step's last `WIDE_TILES` of them + spans about 2**30 fp32 logits: at 32768 keys that is the whole step, whose + untiled call is a certified cell, and at 1048576 keys rows 31744..32767. + Each tile is bitwise the wider call's rows, the short tail included and + the tiles that start inside the wider call's 4-row query blocks (262 rows + at 1024000 keys, 4095 at 65537); a repeated wider call is bitwise the same. + """ + q, kv, w, ks, ke = _build(spec, seed=13) + rows = q.shape[0] + tile = LOGITS_BUDGET_1M // kv[0].shape[0] + starts = list(range(0, rows, tile))[-WIDE_TILES:] + w0 = starts[0] + wide = op(q[w0:], kv, w[w0:], ks[w0:], ke[w0:]) + again = op(q[w0:], kv, w[w0:], ks[w0:], ke[w0:]) + assert torch.equal(again, wide), "a repeated call is not bitwise the same" + del again + for lo in starts: + hi = min(lo + tile, rows) + out = op(q[lo:hi], kv, w[lo:hi], ks[lo:hi], ke[lo:hi]) + assert torch.equal(out, wide[lo - w0 : hi - w0]), ( + f"rows [{lo}, {hi}) differ from the call over rows [{w0}, {rows})" + ) + + +def test_a_1m_tile_gate_sees_a_window_end_moved() -> None: + """At 1048576 keys the gate refuses a window whose far end moved. + + The last 16 rows of the 1M chunk, windows of 1048561..1048576 keys, scored + with their ends wrapped at 2**16 or at 2**15 (what a 16-bit column, or a + window clamped to a 32768-key layout, would read), one key short and one + key long. `_check` against the true windows refuses each, on the -inf + pattern; the true windows pass. + """ + args = _build(dict(heads=32, head_dim=128, seqs=[(1015808, 32768)], rows=(32752, 32768)), 7) + q, kv, w, ks, ke = args + _check(op(*args), args) + moved = { + "ends wrapped at 2**16": ke % 2**16, + "ends wrapped at 2**15": ke % 2**15, + "one key short": ke - 1, + "one key long": ke + 1, + } + for tag, bad in moved.items(): + try: + _check(op(q, kv, w, ks, bad), args) + except AssertionError as e: + assert "is not -inf" in str(e) or "is not finite" in str(e), (tag, str(e)) + continue + raise AssertionError(f"the gate accepted window {tag}") + + +def test_a_packed_gate_sees_a_window_moved_across_requests() -> None: + """In a packed chunk past 2**20 keys the gate refuses a window that reads the wrong request. + + Two 1M prompts' 1-token final chunks, 2 rows against 2097152 keys: the + second row's window [1048576, 2097152) starts at 2**20 and ends at 2**21. + Scored with its start wrapped at 2**20 (reading the first request's keys, + as a packed chunk read as one request would), its end wrapped at 2**21 (an + inverted window), its start one key late, and the first row's end one key + long (its window reaching the second request's first key), `_check` against + the true windows refuses each; the true windows pass. + """ + spec = dict(heads=32, head_dim=128, seqs=[(1048575, 1), (1048575, 1)], rows=(0, 2)) + args = _build(spec, 8) + q, kv, w, ks, ke = args + _check(op(*args), args) + assert (int(ks[1]), int(ke[1])) == (2**20, 2**21), (ks, ke) + first, second = torch.tensor([1, 0], device="cuda"), torch.tensor([0, 1], device="cuda") + moved = { + "second start wrapped at 2**20": (ks % 2**20, ke), + "second end wrapped at 2**21": (ks, ke % 2**21), + "second start one key late": (ks + second, ke), + "first end one key long": (ks, ke + first), + } + for tag, (bad_ks, bad_ke) in moved.items(): + bad_ks, bad_ke = bad_ks.to(torch.int32), bad_ke.to(torch.int32) + try: + _check(op(q, kv, w, bad_ks, bad_ke), args) + except AssertionError as e: + assert "is not -inf" in str(e) or "is not finite" in str(e), (tag, str(e)) + continue + raise AssertionError(f"the gate accepted window {tag}") + + +def _keys(spec) -> int: + return sum(a + b for a, b in spec["seqs"]) + + +def _tile_rows(spec) -> int: + lo, hi = spec.get("rows", (0, sum(b for _, b in spec["seqs"]))) + return hi - lo + + +#: Every cell that is one whole query tile of the 2**28 budget and that the +#: 2**24 budget cuts into more than one. +_TILES_2_28 = [ + c + for c in op.CELLS + if c.spec.get("budget") == 2**28 and _tile_rows(c.spec) > 2**24 // _keys(c.spec) +] + + +@pytest.mark.parametrize("cell", _TILES_2_28, ids=[c.why[:44] for c in _TILES_2_28]) +def test_budget_tiles_are_bitwise_rows_of_the_2_28_tile(cell) -> None: + """Each tile the 2**26 and 2**24 budgets cut is bitwise the rows of the 2**28 tile. + + Above 32768 keys no untiled chunk fits one call, so the 2**28 tile is the + certified cell these are compared with: at 1048576 keys its 256 rows as + four 64-row and sixteen 16-row calls, at 32768 keys its 8192 rows as four + of 2048 and sixteen of 512, and in the packed blocks tiles that end at and + inside request boundaries (88 and 22 rows at 762256 keys). This carries + the certification of each 2**28 tile cell to the 2**26 and 2**24 tiles of + the same rows, and shows a repeated call is bitwise the same. + """ + q, kv, w, ks, ke = _build(cell.spec, seed=17) + whole = op(q, kv, w, ks, ke) + assert torch.equal(op(q, kv, w, ks, ke), whole), "a repeated call is not bitwise the same" + n = kv[0].shape[0] + for budget in (2**26, 2**24): + tile = budget // n + for lo in range(0, q.shape[0], tile): + hi = min(lo + tile, q.shape[0]) + out = op(q[lo:hi], kv, w[lo:hi], ks[lo:hi], ke[lo:hi]) + assert torch.equal(out, whole[lo:hi]), ( + f"budget {budget}: rows [{lo}, {hi}) differ from the 2**28 tile" + ) + + +def test_a_packed_chunk_split_per_request_changes_no_bit() -> None: + """Each request's own call is bitwise its rows and keys of the packed call. + + The engine packs requests into one indexer chunk while sum(new) x + sum(cached + new) <= 2**28, so two 5-token last chunks over 983040 cached + keys each gather 1966090 keys, past max_seq_len. Split per request, each + call scores its own 983045 keys with windows rebased to key 0, as the + split cell does; this shows the split changes no bit of either request's + logits, so certifying the split call certifies what the target reads. + Each request's scales are a buffer of their own, as a per-request gather + allocates them: a view at key 983045 is not 16-byte aligned, which the op + refuses (`test_a_k_scale_view_off_16_bytes_raises`). + """ + q, (k, s), w, ks, ke = _build(dict(heads=32, head_dim=128, seqs=[(983040, 5)] * 2), seed=21) + packed = op(q, (k, s), w, ks, ke) + row, col = 0, 0 + for cached, new in [(983040, 5)] * 2: + n, rows = cached + new, slice(row, row + new) + kv = (k[col : col + n], s[col : col + n].clone()) + own = op(q[rows], kv, w[rows], ks[rows] - col, ke[rows] - col) + assert torch.equal(own, packed[rows, col : col + n]), f"request at key {col} differs" + row, col = row + new, col + n + + +def test_a_k_scale_view_off_16_bytes_raises() -> None: + """A `k_scale` view at a key offset that is not a multiple of 4 raises. + + The op asserts a 16-byte-aligned `k_scale`; at offsets 4 and 8 the view + scores bitwise like a copy. This is what a caller that splits a packed + chunk by slicing its gathered scales meets. The `k` rows are 128 bytes, so + their views are always aligned. + """ + q, (k, s), w, ks, ke = _ok() + for off in (1, 2, 3, 4, 8): + kv = (k[off:], s[off:]) + args = (q, kv, w, (ks - off).clamp_min(0), (ke - off).clamp_min(0)) + if off % 4: + with pytest.raises(RuntimeError, match=r"data_ptr\(\)\) % 16u == 0u"): + op(*args) + else: + assert torch.equal(op(*args), op(q, (k[off:], s[off:].clone()), *args[2:])) + + +#: The step of every tep8 share a cell tiles in more than one call, keyed by its +#: requests and the share's rows. +_TEP8_SHARES = { + "+".join(f"{a}:{b}" for a, b in c.spec["seqs"]) + + f"-rows-{c.spec['share'][0]}-{c.spec['share'][1]}": { + k: v for k, v in c.spec.items() if k != "rows" + } + for c in op.CELLS + if "share" in c.spec + and c.spec["share"][1] - c.spec["share"][0] > LOGITS_BUDGET_1M // _keys(c.spec) +} + + +@pytest.mark.parametrize("spec", list(_TEP8_SHARES.values()), ids=list(_TEP8_SHARES)) +def test_tep8_share_tiles_are_bitwise_rows_of_wider_calls(spec) -> None: + """Every tile of a tep8 rank's share is bitwise its rows of a call over up to four tiles. + + The tep8 deployment cuts floor(2**28 / keys)-row tiles from its share's first + row, not the chunk's. The share is scored in calls of `WIDE_TILES` tiles + from it, and every tile is bitwise the rows of the call it falls in, the + first and last tiles the cells gate included, and the tiles that start + inside that call's 4-row query blocks (263 rows at 1019904 keys, 541 at + 495616, the 32751-token chunk's 256 from row 28657); the first wider call + repeated is bitwise the same. + """ + q, kv, w, ks, ke = _build(spec, seed=13) + s0, s1 = spec["share"] + tile = LOGITS_BUDGET_1M // kv[0].shape[0] + starts = list(range(s0, s1, tile)) + for g in range(0, len(starts), WIDE_TILES): + w0, w1 = starts[g], min(starts[g] + WIDE_TILES * tile, s1) + wide = op(q[w0:w1], kv, w[w0:w1], ks[w0:w1], ke[w0:w1]) + if g == 0: + again = op(q[w0:w1], kv, w[w0:w1], ks[w0:w1], ke[w0:w1]) + assert torch.equal(again, wide), "a repeated call is not bitwise the same" + del again + for lo in starts[g : g + WIDE_TILES]: + hi = min(lo + tile, s1) + out = op(q[lo:hi], kv, w[lo:hi], ks[lo:hi], ke[lo:hi]) + assert torch.equal(out, wide[lo - w0 : hi - w0]), ( + f"rows [{lo}, {hi}) differ from the call over rows [{w0}, {w1})" + ) + del out + del wide + + +#: A 1048576-token prompt's last chunk behind its 1015808 cached tokens, and the +#: 1048559-token envelope prompt's: the steps the split test divides eight ways. +_SPLIT_CHUNKS_1M = {"32768-token": (1015808, 32768), "32751-token": (1015808, 32751)} + + +@pytest.mark.parametrize("chunk", list(_SPLIT_CHUNKS_1M.values()), ids=list(_SPLIT_CHUNKS_1M)) +def test_a_tep8_split_tile_is_bitwise_the_unsplit_rows_on_its_keys(chunk) -> None: + """Under tep8's query split a rank's tile scores bitwise as the unsplit chunk's rows. + + Rank r of eight scores its share [count r // 8, count (r + 1) // 8) of the + chunk (`_split_prefill_queries` in the engine's DSA indexer) against the + keys up to the share's end, cached + count (r + 1) // 8, where the unsplit + chunk scores the same rows against all its keys. For every rank's first + and last floor(2**28 / keys)-row tile, the split call is bitwise the + unsplit call's rows on the share's keys, and the unsplit call holds -inf on + every key past them: the top-k after it reads the same scores either way. + """ + cached, count = chunk + q, (k, k_scale), w, ks, ke = _build(dict(heads=32, head_dim=128, seqs=[chunk]), seed=13) + for r in range(8): + s0, s1 = count * r // 8, count * (r + 1) // 8 + n = cached + s1 + tile = LOGITS_BUDGET_1M // n + starts = list(range(s0, s1, tile)) + for lo in sorted({starts[0], starts[-1]}): + hi = min(lo + tile, s1) + split = op(q[lo:hi], (k[:n], k_scale[:n]), w[lo:hi], ks[lo:hi], ke[lo:hi]) + whole = op(q[lo:hi], (k, k_scale), w[lo:hi], ks[lo:hi], ke[lo:hi]) + assert torch.equal(split, whole[:, :n]), ( + f"rank {r} rows [{lo}, {hi}) differ on the share's {n} keys" + ) + assert bool(torch.isneginf(whole[:, n:]).all()), ( + f"rank {r} rows [{lo}, {hi}) score keys past the share's {n}" + ) + del split, whole + + +def test_the_gate_rejects_wrong_formulas() -> None: + """The band is only worth something if a plausible mistake misses it.""" + args = _build(dict(heads=32, head_dim=128, seqs=[(256, 64)]), seed=4) + out = op(*args) + q, (k, k_scale), w, ks, ke = args + # relu after the head sum instead of per head + dots = torch.einsum("qhd,kd->qhk", q.float().double(), k.float().double()) + after = (dots * w.double().unsqueeze(-1)).sum(1).relu() * k_scale.double().unsqueeze(0) + cols = torch.arange(k.shape[0], device="cuda").unsqueeze(0) + after = after.masked_fill((cols < ks.unsqueeze(1)) | (cols >= ke.unsqueeze(1)), float("-inf")) + wrong = { + "no key scale": op.reference(q, (k, torch.ones_like(k_scale)), w, ks, ke), + "relu after the head sum": after.float(), + "weights of the neighbouring row": op.reference(q, (k, k_scale), w.roll(1, 0), ks, ke), + "windows one key short": op.reference(q, (k, k_scale), w, ks, ke - 1), + # the near miss: one half-precision rounding, of the head weights + "head weights rounded to fp16": op.reference(q, (k, k_scale), w.half().float(), ks, ke), + } + for tag, ref in wrong.items(): + try: + _compare(out, ref, args) + except AssertionError: + continue + raise AssertionError(f"the gate accepted a wrong formula: {tag}") + + +def test_the_blocked_gate_sees_what_it_skips() -> None: + """`_check` still checks every entry, though it evaluates only each block's key span. + + Left and right of a span, and every block of rows. Two requests of 300 and + 4000 tokens: three blocks of rows, the second's span the keys 300..4095 of + the second request. + """ + args = _build(dict(heads=32, head_dim=128, seqs=[(0, 300), (0, 4000)]), seed=5) + out = op(*args) + _check(out, args) + q, _, _, ks, ke = args + last = q.shape[0] - 1 + corrupt = { + "a key left of a block's span": (3000, 100, 0.0, "out-of-window entry is not -inf"), + "a key right of a block's span": (2100, 4200, 0.0, "out-of-window entry is not -inf"), + "an in-window logit of the last block": (last, last, None, "max error"), + } + for tag, (row, col, value, match) in corrupt.items(): + bad = out.clone() + bad[row, col] = value if value is not None else bad[row, col] * (1 + 2**-10) + 1.0 + with pytest.raises(AssertionError, match=match): + _check(bad, args) + assert int(ks[3000]) == 300 and int(ke[2100]) == 2101, "the corruptions miss their spans" + + +def _ok(): + """A cached prefix of 64 keys and 8 new tokens: every window starts at key 0.""" + return _build(dict(heads=32, head_dim=128, seqs=[(64, 8)]), seed=6) + + +def test_row_padded_weights_are_read_at_their_stride() -> None: + """The op reads `weights` rows at stride(0), so a padded view needs no copy.""" + q, kv, w, ks, ke = _ok() + padded = torch.zeros(w.shape[0], 64, device="cuda")[:, : w.shape[1]] + padded.copy_(w) + with validating(op): + out = op(q, kv, padded, ks, ke) + assert torch.equal(out, op(q, kv, w, ks, ke)) + + +def test_a_window_past_the_keys_is_clamped_to_them() -> None: + """A window that ends beyond the last key scores the keys there are.""" + q, kv, w, ks, ke = _ok() + far = ke + 1000 + with validating(op): + out = op(q, kv, w, ks, far) + _compare(out, op.reference(q, kv, w, ks, far), (q, kv, w)) + + +def test_guard_refuses_half_weights() -> None: + """fp16 weights select a second kernel, which answers at half precision.""" + q, kv, w, ks, ke = _ok() + half = w.half() + out = op(q, kv, half, ks, ke) + with pytest.raises(AssertionError, match="max error"): + _compare(out, op.reference(q, kv, half.float(), ks, ke), (q, kv, half)) + with pytest.raises(AssertionError, match="weights must be fp32"): + with validating(op): + op(q, kv, half, ks, ke) + + +def test_guard_refuses_a_negative_window_start() -> None: + """A start of -1 is not read as 0: every logit of the call comes back -inf.""" + q, kv, w, ks, ke = _ok() + out = op(q, kv, w, ks - 1, ke) + assert bool(torch.isneginf(out).all()) + with pytest.raises(AssertionError, match="in-window entry is not finite"): + _compare(out, op.reference(q, kv, w, ks - 1, ke), (q, kv, w)) + with pytest.raises(AssertionError, match="starts before key 0"): + with validating(op): + op(q, kv, w, ks - 1, ke) + + +def test_guard_refuses_an_inverted_window() -> None: + """An inverted window is not read as empty: its row comes back with finite logits.""" + q, kv, w, ks, ke = _ok() + inverted = ks.clone() + inverted[3] = ke[3] + 2 # still inside the 72 keys + out = op(q, kv, w, inverted, ke) + assert bool(torch.isfinite(out[3]).any()) + with pytest.raises(AssertionError, match="out-of-window entry is not -inf"): + _compare(out, op.reference(q, kv, w, inverted, ke), (q, kv, w)) + with pytest.raises(AssertionError, match="ends before it starts"): + with validating(op): + op(q, kv, w, inverted, ke) + + +#: The dep4 deployment's split-KV calls. +_SPLIT_KV = [c for c in op.CELLS if c.spec.get("split_kv")] + + +@pytest.mark.parametrize("cell", _SPLIT_KV, ids=[c.why[:44] for c in _SPLIT_KV]) +def test_split_kv_schedule_is_bitwise_the_grid_stride_call(cell) -> None: + """The schedule moves no bit: with it each 4-row query block's 256-key splits are spread + over several SMs, without it one SM scans them all, and every logit comes out the same. + + The kernel computes a logit inside one 256-key split of its query block: a tensor-core + product over the 128 dims, the fixed-order sum over the 32 weighted heads and the key + scale (`sm100_mqa_logits.cuh`), and the splits sit at the same keys either way (the + block's smallest window start rounded down to 4, then every 256). The cleaner that + writes -inf outside the windows runs grid-stride in both. So the whole [T, N] view, + -inf entries and row padding stride included, is bitwise the schedule-less call's, and + the dep4 deployment's 2**26 tiles keep the certification of the grid-stride path. A + schedule is reused across calls: the target builds one per tile and step and scores + every indexer layer's keys with it, so a second call with the same schedule and + other keys of the same count is bitwise its own grid-stride call too. + """ + q, kv, w, ks, ke = args = _build(cell.spec, seed=zlib.crc32(cell.why.encode()) % 2**31) + sched = _schedule(cell.spec, args) + plain = op(*args) + with validating(op): + split = op(*args, schedule_meta=sched) + assert split.shape == plain.shape and split.stride() == plain.stride() + assert torch.equal(split.view(torch.int32), plain.view(torch.int32)), ( + "the split-KV schedule changed a logit" + ) + del plain, split + g = torch.Generator(device="cuda").manual_seed(29) + k2 = _quantize_rows(torch.randn(kv[0].shape[0], kv[0].shape[1], generator=g, device="cuda")) + again = op(q, k2, w, ks, ke, schedule_meta=sched) + assert torch.equal(again.view(torch.int32), op(q, k2, w, ks, ke).view(torch.int32)), ( + "a reused schedule changed a logit" + ) + + +def test_guard_refuses_a_schedule_built_for_other_windows() -> None: + """The kernel trusts the schedule; one built for another call's windows scores the wrong + key spans.""" + q, kv, w, ks, ke = _build(dict(heads=32, head_dim=128, seqs=[(4096, 64)]), seed=8) + good = mqa_logits_schedule(ks, ke, kv[0].shape[0], q.shape[1]) + with validating(op): + op(q, kv, w, ks, ke, schedule_meta=good) + other = mqa_logits_schedule(ks, ke - 3000, kv[0].shape[0], q.shape[1]) + with pytest.raises(AssertionError, match="was not built for this call"): + with validating(op): + op(q, kv, w, ks, ke, schedule_meta=other) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_indexer_k_cache_gather_op.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_indexer_k_cache_gather_op.py new file mode 100644 index 000000000000..208da3c68554 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_indexer_k_cache_gather_op.py @@ -0,0 +1,814 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the indexer_k_cache_gather_op entry. + +The "dep4 1M" cells are the ones the Mistral Large 4 NVFP4 dep4 deployment passes at +max_seq_len 1048576 with chunked prefill, where a chunk gathers its requests' +whole KV (`cached` + new keys) from the step's full-KV mapping. +""" + +import math +import zlib +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.indexer_k_cache_gather_op import ( # noqa: E501 + indexer_k_cache_gather_op as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402,F401 — needs the path declared above + +assert torch.cuda.is_available(), "indexer_k_cache_gather_op requires a CUDA device" + +#: Mistral Large 4's index_sharing (params.json's DSA sharing), one letter per layer: layers 0..16 +#: run their own indexer, then each 4 layers are 2 'shared' and 2 'full', and +#: layer 71 is full. 44 'F' in all. +_ML4_PATTERN = "F" * 17 + "SSFF" * 13 + "SSF" +assert len(_ML4_PATTERN) == 72 and _ML4_PATTERN.count("F") == 44 + +#: The engine's token budget per step at the 1M envelope (the chunk size); a +#: manager at max_seq_len 32768 or below gets its max_seq_len. +MAX_NUM_TOKENS = 32768 +#: The tep4 and tp4 deployments' TP ranks: from q_split_threshold context rows on, the +#: engine splits a 32768-row chunk 8192 rows per rank. +TP_RANKS = 4 +#: The tep8 deployment's: 4096 rows of a 32768-row chunk per rank (a cell's +#: `qsplit_ranks`). +TEP8_RANKS = 8 +#: The block-table width a V2 manager hands out (`max_blocks_per_seq`), measured +#: at the max_seq_len the 1M cells build it at. +_V2_TABLE_WIDTH = {524288: 16388, 1048576: 32772} + + +def _compare(out: torch.Tensor, ref: torch.Tensor) -> None: + """Bit-exact: a gather moves bytes and computes nothing. Call once for + the data and once for the scale.""" + assert out.dtype == ref.dtype, (out.dtype, ref.dtype) + assert out.shape == ref.shape, (out.shape, ref.shape) + assert torch.equal(out.contiguous().view(torch.uint8), ref.contiguous().view(torch.uint8)), ( + "gathered bytes differ from the reference" + ) + + +def _slot_mappings(pages_per_seq, positions_per_seq, head_dim, tpb, page_scale=1): + """Flat byte offsets of each listed token's data and scale, in sequence order. + + The same arithmetic the DSA indexer uses: a page is `tpb * (head_dim + 4)` + bytes, all data first, then all scales, and a block's page index is its id + times the manager's page scale. + """ + page_bytes = tpb * (head_dim + 4) + fp8, scale = [], [] + for pages, pos in zip(pages_per_seq, positions_per_seq): + page = pages[pos // tpb].long() * page_scale + fp8.append(page * page_bytes + (pos % tpb) * head_dim) + scale.append(page * page_bytes + tpb * head_dim + (pos % tpb) * 4) + return torch.cat(fp8).cuda(), torch.cat(scale).cuda() + + +class _Built(SimpleNamespace): + """A cell's arguments, the memory the test owns behind `k_cache`, and how to + release it. A manager's views must be gone before the manager shuts down.""" + + def close(self) -> None: + manager = self.manager + self.args = self.owned = self.manager = None + if manager is not None: + manager.shutdown() + + +def _build_interleaved(spec, seed) -> _Built: + """The cache is one layer's view of a pool that interleaves `pool_layers` + layers page by page, so dim 0 is not contiguous when there is more than + one -- the layout the K cache manager hands out.""" + g = torch.Generator().manual_seed(seed) + hd, tpb = spec["head_dim"], spec["tokens_per_block"] + pages = [math.ceil(n / tpb) for n in spec["seq_lens"]] + num_pages = 2 * sum(pages) + 1 + pool = torch.randperm(num_pages, generator=g).to(torch.int32) + table = torch.zeros(len(pages), max(pages), dtype=torch.int32) + used = 0 + for s, n in enumerate(pages): + table[s, :n] = pool[used : used + n] + used += n + layers = spec["pool_layers"] + parent = torch.randint( + 0, 256, (num_pages, layers, tpb * (hd + 4)), generator=g, dtype=torch.uint8 + ).cuda() + k_cache = parent[:, layers // 2].view(num_pages, tpb, 1, hd + 4) + assert layers == 1 or not k_cache.is_contiguous() + fp8, scale = _slot_mappings(list(table), [torch.arange(n) for n in spec["seq_lens"]], hd, tpb) + args = (k_cache, fp8, scale, spec["k_token_start"], spec["num_tokens"], hd) + return _Built(args=args, owned=parent, manager=None) + + +def _build_view(spec, seed) -> _Built: + """Mistral Large 4's rebuilt views, the geometry both managers were measured to + hand out: "v1" is row `pool_row` of a [num_blocks, pool_layers, page] + pool, "v2" a contiguous view starting at that row's first page, with page + indices scaled by `pool_layers`. The prompts' tokens come first in the + mapping, then one new-token slot per decode row: `decode_pos` lists their + positions, `decode_rows` draws that many below 4096. With `cycle` the + step's pages may outnumber the pool: they are consecutive fresh + permutations of it, so a page repeats but no stretch of the mapping + repeats another. With `cached` a prompt's keys are its cached prefix and + its new keys, positions 0..cached + new - 1, as the full-KV mapping lists + them, and a decode row owns only the page its new key sits on (its earlier + pages are never read). + + `cycle` (only the 1M, 1M tp4 and 1M tep8 cells carry it) and `cached` (only + the dep4 1M cells) are never set together; a cell without `cached` takes + prefix 0, so its positions, page counts, draws and mapping are those of + the helper without the `cached` lines.""" + g = torch.Generator().manual_seed(seed) + hd, tpb = spec["head_dim"], spec["tokens_per_block"] + page_bytes = tpb * (hd + 4) + rows, row, blocks = spec["pool_layers"], spec["pool_row"], spec["num_blocks"] + decode = torch.randint(1, 4096, (spec.get("decode_rows", 0),), generator=g).tolist() + decode += spec.get("decode_pos", []) + prefix = spec.get("cached", [0] * len(spec["seq_lens"])) + prompts = [torch.arange(c + n) for c, n in zip(prefix, spec["seq_lens"])] + positions = prompts + [torch.tensor([p]) for p in decode] + need = [int(p[-1]) // tpb + 1 for p in positions] + if "cached" in spec: + need[len(prompts) :] = [1] * len(decode) + assert not (spec.get("cycle") and "cached" in spec), "cycle and cached are never combined" + if spec.get("cycle"): + perms = math.ceil(sum(need) / blocks) + ids = torch.cat([torch.randperm(blocks, generator=g) for _ in range(perms)])[: sum(need)] + else: + assert sum(need) <= blocks, (sum(need), blocks) + if spec.get("tail"): + ids = torch.arange(blocks - sum(need), blocks)[torch.randperm(sum(need), generator=g)] + else: + ids = torch.randperm(blocks, generator=g)[: sum(need)] + pages = list(torch.split(ids, need)) + if "cached" in spec: + for i, p in enumerate(decode, start=len(prompts)): + # Only the page holding position p is real; the earlier entries are never read. + pages[i] = torch.cat([torch.full((p // tpb,), -1, dtype=ids.dtype), pages[i]]) + gen = torch.Generator(device="cuda").manual_seed(seed) + if spec["layout"] == "v1": + parent = torch.randint( + 0, 256, (blocks, rows, page_bytes), dtype=torch.uint8, device="cuda", generator=gen + ) + k_cache = parent[:, row].view(blocks, tpb, 1, hd + 4) + assert k_cache.stride(0) == rows * page_bytes + page_scale = 1 + else: + parent = torch.randint( + 0, 256, (blocks * rows * page_bytes,), dtype=torch.uint8, device="cuda", generator=gen + ) + k_cache = parent[row * page_bytes :].view(blocks * rows - row, tpb, 1, hd + 4) + page_scale = rows + fp8, scale = _slot_mappings(pages, positions, hd, tpb, page_scale) + assert fp8.numel() == sum(prefix) + sum(spec["seq_lens"]) + len(decode) + if "cached" in spec: + print( + f"\n[view] {spec['layout']} row {row} of {blocks} blocks x {rows} rows: mapping " + f"{fp8.numel()} keys, data slots {int(fp8.min())}..{int(fp8.max())}" + ) + args = (k_cache, fp8, scale, spec["k_token_start"], spec["num_tokens"], hd) + return _Built(args=args, owned=parent, manager=None) + + +def _build_manager(spec, seed) -> _Built: + """A real DSA cache manager at Mistral Large 4's geometry (72 layers, latent 576 + fp8, the 44-row indexer mask): the view `get_indexer_k_cache_buffers` hands + out, the block ids `get_pool_block_indices` reads for the metadata, and the + indexer's own slot arithmetic. The pages the window reads are filled with + seeded random bytes through the view. `max_seq_len` (default 9216, the FP8 + target's) is also the manager's max_num_tokens up to 32768, the 1M + envelope's chunk. With `cached` each prompt holds its cached and new keys, + and the mapping lists them all, as the full-KV mapping does; the manager + must then keep the cell's max_seq_len (V2 lowers it to what a small quota + holds). The pool is allocated outside torch's caching allocator, so what + torch still caches from earlier cells is returned to the device first. + + Only the dep4 1M cells carry `cached`; every `cached` line is skipped + without it (the prompts' lengths are then `seq_lens`). The dep4 run + emptied torch's cache before building the config objects, the tp4, tep4 + and tep8 runs after; the configs allocate no device memory, so the manager + is built on the same device state either way.""" + from tensorrt_llm._torch.attention.backends.sparse.dsa.cache_manager import ( + DSACacheManager, + DSACacheManagerV2, + ) + from tensorrt_llm._torch.attention.backends.sparse.dsa.indexer import _compute_slot_mappings + from tensorrt_llm.bindings import DataType + from tensorrt_llm.bindings.executor import KvCacheConfig as BindingKvCacheConfig + from tensorrt_llm.bindings.internal.batch_manager import CacheType + from tensorrt_llm.llmapi.llm_args import DeepSeekSparseAttentionConfig, KvCacheConfig + from tensorrt_llm.mapping import Mapping + + hd, tpb = spec["head_dim"], spec["tokens_per_block"] + max_seq_len = spec.get("max_seq_len", 9216) + v2 = spec["layout"] == "manager_v2" + if v2: + kv_cache_config = KvCacheConfig( + enable_block_reuse=False, max_tokens=spec["max_tokens"], host_cache_size=0 + ) + else: + kv_cache_config = BindingKvCacheConfig( + enable_block_reuse=False, max_tokens=spec["max_tokens"] + ) + torch.cuda.empty_cache() + manager = (DSACacheManagerV2 if v2 else DSACacheManager)( + kv_cache_config=kv_cache_config, + kv_cache_type=CacheType.SELFKONLY, + num_layers=72, + num_kv_heads=1, + head_dim=576, + tokens_per_block=tpb, + max_seq_len=max_seq_len, + max_batch_size=16, + mapping=Mapping(world_size=1, rank=0, tp_size=1, pp_size=1), + dtype=DataType.FP8, + sparse_attention_config=DeepSeekSparseAttentionConfig(index_head_dim=hd), + pretrained_config=SimpleNamespace( + num_hidden_layers=72, qk_rope_head_dim=64, index_topk_pattern=list(_ML4_PATTERN) + ), + max_num_tokens=min(max_seq_len, MAX_NUM_TOKENS), + ) + try: + assert sum(manager.indexer_k_cache_local_layer_mask) == spec["pool_layers"] + assert manager.indexer_k_cache_page_scale == (spec["pool_layers"] if v2 else 1) + lens = [ + c + n for c, n in zip(spec.get("cached", [0] * len(spec["seq_lens"])), spec["seq_lens"]) + ] + if "cached" in spec: + assert manager.max_seq_len == max_seq_len, (manager.max_seq_len, max_seq_len) + ids = list(range(len(lens))) + added = manager.add_dummy_requests(ids, lens, is_gen=False) + assert added is not None, "the manager's quota refused the cell's prompts" + if not v2: + # V1 reads the block ids back from the offsets the attention metadata copies. + manager.copy_batch_block_offsets( + torch.zeros_like(manager.host_kv_cache_block_offsets, device="cuda"), + ids, + 1, + len(ids), + len(ids), + ) + blocks = manager.get_pool_block_indices(len(ids), request_ids=ids, num_contexts=len(ids)) + if "cached" in spec: + assert blocks.shape[1] == _V2_TABLE_WIDTH[max_seq_len], tuple(blocks.shape) + page_table = blocks.long() * manager.indexer_k_cache_page_scale + req = torch.repeat_interleave(torch.arange(len(lens)), torch.tensor(lens)) + pos = torch.cat([torch.arange(n) for n in lens]) + fp8, scale = _compute_slot_mappings(pos, page_table, req, hd, tpb, 128) + fp8, scale = fp8.cuda(), scale.cuda() + k_cache = manager.get_indexer_k_cache_buffers(spec["layer"]) + assert k_cache.is_contiguous() == v2 + print( + f"\n[manager] {type(manager).__name__} quota {spec['max_tokens']} tokens, " + f"max_seq_len {manager.max_seq_len}, table {tuple(blocks.shape)}; layer " + f"{spec['layer']} view {tuple(k_cache.shape)}, data slots " + f"{int(fp8.min())}..{int(fp8.max())}" + ) + pages = torch.unique(fp8 // (tpb * (hd + 4))) + gen = torch.Generator(device="cuda").manual_seed(seed) + k_cache[pages] = torch.randint( + 0, + 256, + (pages.numel(), *k_cache.shape[1:]), + dtype=torch.uint8, + device="cuda", + generator=gen, + ) + if "cached" in spec: + first = [int(blocks[i, 0]) for i in ids] + last = [int(blocks[i, (n - 1) // tpb]) for i, n in zip(ids, lens)] + print( + f"\n[manager] {type(manager).__name__} quota {spec['max_tokens']} tokens, " + f"{manager.num_blocks} slots, max_seq_len {manager.max_seq_len}, table " + f"{tuple(blocks.shape)}; prompts' slots first {first}, last {last}; layer " + f"{spec['layer']} view {tuple(k_cache.shape)}, mapping {fp8.numel()} keys, data " + f"slots {int(fp8.min())}..{int(fp8.max())}" + ) + except BaseException: + manager.shutdown() + raise + args = (k_cache, fp8, scale, spec["k_token_start"], spec["num_tokens"], hd) + return _Built(args=args, owned=k_cache, manager=manager) + + +def _build(spec, seed) -> _Built: + if spec["layout"].startswith("manager"): + return _build_manager(spec, seed) + return _build_view(spec, seed) + + +def _gather_once(args) -> None: + data, scale = op(*args) + assert data.dtype == torch.float8_e4m3fn and scale.dtype == torch.float32 + assert data.is_contiguous() and scale.is_contiguous() + ref_data, ref_scale = op.reference(*args) + _compare(data, ref_data) + _compare(scale, ref_scale) + + +#: Key counts `_sweep` compares at once: near 32768 keys a group's outputs and +#: the rows they must equal take about 0.5 GiB. +_SWEEP_GROUP = 32 + + +def _sweep(args) -> None: + """Every key count n of the cell's window, read twice: as its first n keys + and as its last n. The kernel's grid is one block per key, so this is every + grid the window can launch; the expected rows are the entry's reference of + the whole window, sliced. + + The loop is launch-bound. On a GB300 at the 32768-key window a call is + dominated by host time, and the guard's host read and two per-call + compares add more than the call itself. So the outputs of `_SWEEP_GROUP` + key counts are concatenated and compared bitwise with the rows they must + equal at once, and only a group that differs is read again window by + window to name the windows. The groups run from the largest key count + down, so the caching allocator reuses each group's blocks for the next: + in rising order it reserved 32.8 GiB for a 1.5 GiB peak.""" + k_cache, fp8, scale, start, total, hd = args + ref_data, ref_scale = op.reference(*args) + + def windows(counts): + return [(n, side, first) for n in counts for side, first in enumerate((0, total - n))] + + def gathered(first, n): + return op(k_cache, fp8, scale, start + first, n, hd) + + def expected(first, n): + return ref_data[first : first + n], ref_scale[first : first + n] + + def same(got, want) -> bool: + return all( + torch.equal(torch.cat(g).view(torch.int32), torch.cat(w).view(torch.int32)) + for g, w in zip(got, want) + ) + + wrong = [] + for high in range(total, 0, -_SWEEP_GROUP): + group = windows(range(high, max(high - _SWEEP_GROUP, 0), -1)) + got = zip(*(gathered(first, n) for n, _, first in group)) + if not same(got, zip(*(expected(first, n) for n, _, first in group))): + wrong += [ + (n, side) + for n, side, first in group + if not same(zip(gathered(first, n)), zip(expected(first, n))) + ] + assert not wrong, f"(key count, prefix=0/suffix=1) windows that differ: {sorted(wrong)[:8]}" + + +def _qsplit_shares(seq_lens, ranks=TP_RANKS) -> list[tuple[int, int]]: + """`(start, count)` of each TP rank's gather when the engine's query split cuts + the last 32768-key chunk of a request of that many keys or more over `ranks` + ranks: rank r gathers the request's cached keys and its share's rows, cached + + 8192 (r + 1) keys at 4 ranks and cached + 4096 (r + 1) at 8, from the + request's offset in the mapping.""" + shares, offset = [], 0 + for n in seq_lens: + if n >= MAX_NUM_TOKENS: + cached, share = n - MAX_NUM_TOKENS, MAX_NUM_TOKENS // ranks + shares += [(offset, cached + share * (r + 1)) for r in range(ranks)] + offset += n + return shares + + +def _edge_windows(total: int) -> list[tuple[int, int]]: + """`(start, count)` windows across the integer edges a 1M mapping reaches: 2^k - 1, + 2^k and 2^k + 1 keys (k = 15..21) and the whole mapping, at its start and at + its end; and 1, 2049 and every remaining key from keys 2^20 and 2^21, each +-1.""" + counts = [2**k + d for k in range(15, 22) for d in (-1, 0, 1) if 2**k + d < total] + [total] + starts = [2**k + d for k in (20, 21) for d in (-1, 0, 1) if 2**k + d < total] + return ( + [(0, n) for n in counts] + + [(total - n, n) for n in counts] + + [(s, n) for s in starts for n in (1, 2049, total - s)] + ) + + +def _wrong_windows(args, windows) -> list[tuple[int, int]]: + """The `(start, count)` windows whose gather differs from the rows of the + entry's reference over the whole mapping. Each window is one call and one + compare (int32 words: a scale's random bytes may read as NaN).""" + k_cache, fp8, scale, _, _, hd = args + ref_data, ref_scale = op.reference(k_cache, fp8, scale, 0, fp8.numel(), hd) + wrong = [] + for first, n in windows: + data, sc = op(k_cache, fp8, scale, first, n, hd) + if not ( + torch.equal(data.view(torch.int32), ref_data[first : first + n].view(torch.int32)) + and torch.equal(sc.view(torch.int32), ref_scale[first : first + n].view(torch.int32)) + ): + wrong.append((first, n)) + return wrong + + +def _windows(args, seq_lens, ranks=TP_RANKS) -> None: + """The cell's window, each q-split share of its requests over `ranks` ranks and + the integer-edge windows of its mapping (`_qsplit_shares`, `_edge_windows`).""" + _, fp8, _, start, total, _ = args + windows = [(start, total), *_qsplit_shares(seq_lens, ranks), *_edge_windows(fp8.numel())] + wrong = _wrong_windows(args, windows) + assert not wrong, f"(start, count) windows that differ: {wrong}" + + +def _wrong_counts(args, counts) -> list[tuple[int, int]]: + """The `(key count, prefix=0/suffix=1)` windows of the listed counts whose + gather differs from the rows of the entry's reference over the cell's whole + window. One call and one compare per window (int32 words: a scale's random + bytes may read as NaN).""" + k_cache, fp8, scale, start, total, hd = args + ref_data, ref_scale = op.reference(*args) + wrong = [] + for n in counts: + for side, first in enumerate((0, total - n)): + data, sc = op(k_cache, fp8, scale, start + first, n, hd) + if not ( + torch.equal(data.view(torch.int32), ref_data[first : first + n].view(torch.int32)) + and torch.equal( + sc.view(torch.int32), ref_scale[first : first + n].view(torch.int32) + ) + ): + wrong.append((n, side)) + return wrong + + +def _sweep_counts(args, counts) -> None: + """Each listed key count n of the cell's window, read twice: as its first n + keys and as its last n. A 1M window's every count is about 2M calls, so the + cells name the counts at its edges instead (one key, index_topk, the 32768 + chunk and one past it, one past each power of two, the window less one key, + the whole window).""" + wrong = _wrong_counts(args, counts) + assert not wrong, f"(key count, prefix=0/suffix=1) windows that differ: {wrong}" + + +def _same_bytes(a: torch.Tensor, b: torch.Tensor) -> bool: + """`torch.equal`, compared 8 bytes at a time where the buffer allows it: its + elementwise temporary is then an eighth of a 5 GB pool, not the pool's size.""" + if a.is_contiguous() and b.is_contiguous() and a.numel() % 8 == 0: + a, b = a.reshape(-1).view(torch.int64), b.reshape(-1).view(torch.int64) + return torch.equal(a, b) + + +@pytest.mark.parametrize( + "cell", + [pytest.param(c, marks=pytest.mark.large) if c.spec.get("large") else c for c in op.CELLS], + ids=[c.why[:44] for c in op.CELLS], +) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 + certifying(op) + built = _build(cell.spec, seed=zlib.crc32(cell.why.encode()) % 2**31) + try: + before = built.owned.clone() + # `sweep="windows"` is only on the 1M cells (their q-split shares over + # `qsplit_ranks` ranks, 4 unless given), `sweep_counts` only on a dep4 1M + # cell, and no cell carries both: each reaches its own member's branch. + if cell.spec.get("sweep") == "windows": + _windows(built.args, cell.spec["seq_lens"], cell.spec.get("qsplit_ranks", TP_RANKS)) + elif cell.spec.get("sweep_counts"): + _sweep_counts(built.args, cell.spec["sweep_counts"]) + elif cell.spec.get("sweep"): + _sweep(built.args) + else: + _gather_once(built.args) + assert _same_bytes(built.owned, before), "the gather wrote into memory behind the cache" + del before + finally: + built.close() + + +def _ok(seed=5, seq_lens=(64,), pool_layers=1): + return _build_interleaved( + dict( + head_dim=128, + tokens_per_block=32, + seq_lens=list(seq_lens), + k_token_start=0, + num_tokens=sum(seq_lens), + pool_layers=pool_layers, + ), + seed=seed, + ).args + + +def test_zero_tokens_returns_empty() -> None: + """Nothing but head_dim is read: a CPU int32 mapping and a 2-D one pass.""" + k_cache, fp8, scale, _, _, hd = _ok(seed=3, seq_lens=(8,)) + data, sc = op(k_cache, fp8.cpu().int(), scale[:1].view(1, 1), 0, 0, hd) + assert tuple(data.shape) == (0, hd) and data.dtype == torch.float8_e4m3fn + assert tuple(sc.shape) == (0, 1) and sc.dtype == torch.float32 + with pytest.raises(RuntimeError, match="head_dim must be 128 \\(FP8\\) or 64"): + op(k_cache, fp8, scale, 0, 0, 96) + + +@pytest.mark.parametrize( + "wrong, message", + [ + ("window past the mapping", "slot_mapping_fp8 too short for k_token_start \\+ num_tokens"), + ("int32 mapping", "slot_mapping_fp8 must be int64"), + ("strided mapping", "slot_mapping_fp8 must be contiguous"), + ("fp8-typed cache", "expected scalar type Byte but found Float8_e4m3fn"), + ("3-D cache", "k_cache must be a 4D Tensor"), + ("head_dim 96", "head_dim must be 128 \\(FP8\\) or 64"), + ], +) +def test_the_op_refuses_loudly(wrong, message) -> None: + """What the op validates itself, so `is_valid` does not restate it.""" + k_cache, fp8, scale, start, n, hd = _ok(seed=13) + call = { + "window past the mapping": lambda: op(k_cache, fp8, scale, 1, n, hd), + "int32 mapping": lambda: op(k_cache, fp8.int(), scale, start, n, hd), + "strided mapping": lambda: op(k_cache, torch.stack([fp8, fp8], 1)[:, 0], scale, 0, n, hd), + "fp8-typed cache": lambda: op(k_cache.view(torch.float8_e4m3fn), fp8, scale, 0, n, hd), + "3-D cache": lambda: op(k_cache.reshape(k_cache.shape[0], 32, -1), fp8, scale, 0, n, hd), + "head_dim 96": lambda: op(k_cache, fp8, scale, start, n, 96), + }[wrong] + with validating(op), pytest.raises(RuntimeError, match=message): + call() + + +def test_guard_refuses_a_negative_start() -> None: + """The op checks only the upper end of the window. Unguarded, a start of -8 + reads the eight int64 words in front of the mapping as slots.""" + k_cache, fp8, scale, _, _, hd = _ok(seed=17, seq_lens=(200,)) + # The words in front belong to the test: tokens 100..107's slots. + fp8_buf, scale_buf = torch.cat([fp8[100:108], fp8]), torch.cat([scale[100:108], scale]) + fp8_view, scale_view = fp8_buf[8:], scale_buf[8:] + data, _ = op(k_cache, fp8_view, scale_view, -8, 16, hd) + front, _ = op.reference(k_cache, fp8, scale, 100, 8, hd) + assert torch.equal(data[:8].view(torch.uint8), front.view(torch.uint8)), ( + "a negative start no longer reads the words in front of the mapping" + ) + with pytest.raises(AssertionError, match="k_token_start must be >= 0"): + with validating(op): + op(k_cache, fp8_view, scale_view, -8, 16, hd) + + +def test_guard_refuses_a_negative_slot() -> None: + """The kernel skips the row and leaves torch.empty's contents in it.""" + k_cache, fp8, scale, _, _, hd = _ok(seed=19) + bad = fp8.clone() + bad[3] = -1 + data, _ = op(k_cache, bad, scale, 0, 64, hd) + ref, _ = op.reference(k_cache, fp8, scale, 0, 64, hd) + rows, ref_rows = data.view(torch.uint8), ref.view(torch.uint8) + assert torch.equal(torch.cat([rows[:3], rows[4:]]), torch.cat([ref_rows[:3], ref_rows[4:]])) + assert not torch.equal(rows[3], ref_rows[3]), "a negative slot's row came back correct" + with pytest.raises(AssertionError, match="must be >= 0"): + with validating(op): + op(k_cache, bad, scale, 0, 64, hd) + + +def test_guard_refuses_a_slot_past_the_view() -> None: + """Nothing bounds a slot from above. Unguarded, a slot one page past a view + that is the leading slice of a larger buffer reads that buffer's tail.""" + k_cache, fp8, scale, _, _, hd = _ok(seed=23) + blocks = k_cache.shape[0] + page_bytes = k_cache[0].numel() + buf = torch.full((blocks + 4, page_bytes), 0xEE, dtype=torch.uint8, device="cuda") + buf[:blocks] = k_cache.reshape(blocks, page_bytes) + view = buf[:blocks].view(k_cache.shape) + fp8 = fp8.clone() + fp8[0] = blocks * page_bytes + 128 # token 1 of the first page past the view + data, _ = op(view, fp8, scale, 0, 64, hd) + assert bool((data[0].view(torch.uint8) == 0xEE).all()), "the row no longer reads past the view" + with pytest.raises(AssertionError, match="must address bytes inside k_cache"): + with validating(op): + op(view, fp8, scale, 0, 64, hd) + + +def test_guard_refuses_a_head_dim_the_cache_rows_do_not_hold() -> None: + """head_dim is the gather width and the op admits 64 (the FP4 cache's) on a + 132-byte FP8 row: unguarded it returns the first half of every key.""" + k_cache, fp8, scale, _, n, _ = _ok(seed=29) + data, sc = op(k_cache, fp8, scale, 0, n, 64) + ref, ref_sc = op.reference(k_cache, fp8, scale, 0, n, 128) + assert tuple(data.shape) == (n, 64) + assert torch.equal(data.view(torch.uint8), ref.view(torch.uint8)[:, :64]) + assert torch.equal(sc, ref_sc) + with pytest.raises(AssertionError, match="does not match the cache's 132-byte rows"): + with validating(op): + op(k_cache, fp8, scale, 0, n, 64) + + +def test_the_compare_sees_a_wrong_layer_or_window() -> None: + """The harness is not blind: the neighbouring layer's view, or a window one + key off, fails the bit-exact compare.""" + spec = dict( + head_dim=128, + tokens_per_block=32, + seq_lens=[300], + k_token_start=0, + num_tokens=300, + pool_layers=3, + ) + built = _build_interleaved(spec, seed=31) + k_cache, fp8, scale, _, n, hd = built.args + ref_data, _ = op.reference(k_cache, fp8, scale, 0, n, hd) + neighbour = built.owned[:, 0].view(k_cache.shape) + with pytest.raises(AssertionError, match="gathered bytes differ"): + _compare(op(neighbour, fp8, scale, 0, n, hd)[0], ref_data) + with pytest.raises(AssertionError, match="gathered bytes differ"): + _compare(op(k_cache, fp8, scale, 1, n - 1, hd)[0], ref_data[: n - 1]) + + +@pytest.mark.parametrize( + "part, row, first_wrong", + [ + ("data", 100, "[(101, 0), (102, 0), (103, 0)"), # prefixes from 101 keys + ("scale", 250, "[(50, 1), (51, 1), (52, 1)"), # suffixes from 300 - 250 keys + ], +) +def test_the_sweep_sees_one_wrong_bit(part, row, first_wrong, monkeypatch) -> None: + """The sweep is not blind: against a reference one bit off in one key's + data, or in its scale, exactly the windows holding that key fail.""" + spec = dict( + head_dim=128, + tokens_per_block=32, + seq_lens=[300], + k_token_start=0, + num_tokens=300, + pool_layers=1, + ) + built = _build_interleaved(spec, seed=37) + reference = type(op).reference + + def one_bit_off(self, *args): + data, scale = reference(self, *args) + (data if part == "data" else scale).view(torch.uint8)[row, 0] ^= 1 + return data, scale + + monkeypatch.setattr(type(op), "reference", one_bit_off) + with pytest.raises(AssertionError, match="windows that differ") as failed: + _sweep(built.args) + assert first_wrong in str(failed.value), str(failed.value) + + +#: A 1M packed mapping on a pool its pages outnumber 400 times over: the controls' +#: cells reach the 1M cells' offsets in 12 MB. +_CYCLED_1M = dict( + head_dim=128, + tokens_per_block=32, + layout="v2", + pool_layers=44, + pool_row=43, + num_blocks=64, + cycle=True, + seq_lens=[1048576, 1048576, 275872], + k_token_start=0, + num_tokens=2373024, +) + + +@pytest.mark.parametrize("part, key", [("data", 2**20 + 5), ("scale", 2**21 - 1)]) +def test_the_windows_see_one_wrong_bit(part, key, monkeypatch) -> None: + """The 1M windows are not blind: against a reference one bit off in one key's + data, or in its scale, exactly the windows holding that key fail.""" + built = _build_view(_CYCLED_1M, seed=41) + reference = type(op).reference + + def one_bit_off(self, *args): + data, scale = reference(self, *args) + (data if part == "data" else scale).view(torch.uint8)[key, 0] ^= 1 + return data, scale + + windows = [*_qsplit_shares(_CYCLED_1M["seq_lens"]), *_edge_windows(2373024)] + holding = [(s, n) for s, n in windows if s <= key < s + n] + assert 0 < len(holding) < len(windows), (len(holding), len(windows)) + monkeypatch.setattr(type(op), "reference", one_bit_off) + assert _wrong_windows(built.args, windows) == holding + built.close() + + +def test_the_compare_sees_a_1m_start_one_early_or_wrapped() -> None: + """Pages that outnumber the pool do not make a 1M window read like another: the + second request's first 2049 keys, gathered from key 2^20 - 1 or from key 0 (the + start wrapped at 2^20), fail the bit-exact compare against those from 2^20.""" + built = _build_view(_CYCLED_1M, seed=43) + k_cache, fp8, scale, _, _, hd = built.args + ref_data, ref_scale = op.reference(k_cache, fp8, scale, 2**20, 2049, hd) + _compare(op(k_cache, fp8, scale, 2**20, 2049, hd)[0], ref_data) + for start in (2**20 - 1, 0): + data, sc = op(k_cache, fp8, scale, start, 2049, hd) + with pytest.raises(AssertionError, match="gathered bytes differ"): + _compare(data, ref_data) + with pytest.raises(AssertionError, match="gathered bytes differ"): + _compare(sc, ref_scale) + built.close() + + +#: The 1M windows on a one-row pool: every key its own slot, the operands in 0.2 GB. +_ONE_ROW_1M = dict( + head_dim=128, + tokens_per_block=32, + layout="v2", + pool_layers=1, + pool_row=0, + num_blocks=36864, + seq_lens=[32768], + cached=[1015808], + k_token_start=0, + num_tokens=1048576, +) +#: The edge counts of the "dep4 1M edge key counts" cell. +_EDGE_COUNTS = next(c.spec["sweep_counts"] for c in op.CELLS if c.spec.get("sweep_counts")) + + +@pytest.mark.parametrize("part, key", [("data", 2048), ("scale", 1015807)]) +def test_the_count_sweep_sees_one_wrong_bit(part, key, monkeypatch) -> None: + """The edge-count sweep is not blind: against a reference one bit off in one + key's data, or in its scale, exactly the listed windows holding that key fail. + Key 1015807 is the last cached key of a 1M prompt's last chunk: the 32768-key + suffix (the chunk's new keys) misses it, the 32769-key one holds it.""" + built = _build_view(_ONE_ROW_1M, seed=47) + reference = type(op).reference + + def one_bit_off(self, *args): + data, scale = reference(self, *args) + (data if part == "data" else scale).view(torch.uint8)[key, 0] ^= 1 + return data, scale + + total = _ONE_ROW_1M["num_tokens"] + holding = [ + (n, side) + for n in _EDGE_COUNTS + for side, first in enumerate((0, total - n)) + if first <= key < first + n + ] + assert 0 < len(holding) < 2 * len(_EDGE_COUNTS), holding + monkeypatch.setattr(type(op), "reference", one_bit_off) + assert _wrong_counts(built.args, _EDGE_COUNTS) == holding + built.close() + + +def test_the_compare_sees_a_split_window_one_key_off_or_its_start_lost() -> None: + """The split packed chunk's second window is not read like another: its first + 2049 keys gathered from key 983045 match the reference, and from one key + early or late, from key 0 (the start dropped: the first request's keys) or + from key 5 (983045 narrowed to 16 bits) fail the bit-exact compare.""" + spec = dict(_ONE_ROW_1M, num_blocks=61442, seq_lens=[5, 5], cached=[983040, 983040]) + built = _build_view(dict(spec, k_token_start=983045, num_tokens=983045), seed=53) + k_cache, fp8, scale, start, _, hd = built.args + assert fp8.numel() == 1966090 and start == 983045 + ref_data, ref_scale = op.reference(k_cache, fp8, scale, start, 2049, hd) + data, sc = op(k_cache, fp8, scale, start, 2049, hd) + _compare(data, ref_data) + _compare(sc, ref_scale) + for wrong in (start - 1, start + 1, 0, start % 2**16): + data, sc = op(k_cache, fp8, scale, wrong, 2049, hd) + with pytest.raises(AssertionError, match="gathered bytes differ"): + _compare(data, ref_data) + with pytest.raises(AssertionError, match="gathered bytes differ"): + _compare(sc, ref_scale) + built.close() + + +def test_tep8_windows_are_the_engines_chunks() -> None: + """The "1M tep8" windows are the chunks the engine plans. Under its query split + over eight ranks (`_split_prefill_queries`, at q_split_threshold's default 8192), + rank r's chunk of a 1M prompt's last 32768 rows (`plan_mla_chunked_prefill_chunks`, + `Indexer.prepare_one_prefill_chunk`) is `_qsplit_shares`' window: 1019904 keys on + rank 0, 1032192 on rank 3. Two 983045-key requests' 5-token tails pack into one + chunk of 1966090 keys, past max_seq_len, which the tep8 deployment gathers per + request, the second from key 983045.""" + from tensorrt_llm._torch.attention.backends.sparse.dsa.indexer import ( + Indexer, + _packed_logits_elem_budget, + _split_prefill_queries, + plan_mla_chunked_prefill_chunks, + ) + + max_seq_len = 1048576 + budget = _packed_logits_elem_budget( + SimpleNamespace( + max_num_tokens=MAX_NUM_TOKENS, + kv_cache_manager=SimpleNamespace(max_seq_len=max_seq_len), + ) + ) + + def windows(new, cached, q_begin=0, q_end=None): + kv = [c + n for c, n in zip(cached, new)] + md = SimpleNamespace( + cu_seqlen_ks=torch.empty(0, dtype=torch.int32), + compress_ratios=[], + host_ctx_cached_token_indptr=torch.tensor([0, *cached]).cumsum(0), + host_ctx_kv_indptr=torch.tensor([0, *kv]).cumsum(0), + ) + planned = plan_mla_chunked_prefill_chunks(new, kv, budget, q_begin, q_end) + chunks = [Indexer.prepare_one_prefill_chunk(md, specs) for specs in planned] + return [(c.k_token_start, c.k_token_end - c.k_token_start) for c in chunks] + + shares = [] + for rank in range(TEP8_RANKS): + mapping = SimpleNamespace(tp_size=TEP8_RANKS, tp_rank=rank, enable_attention_dp=False) + q_begin, q_end, sizes = _split_prefill_queries(MAX_NUM_TOKENS, mapping, 8192) + assert sizes == [MAX_NUM_TOKENS // TEP8_RANKS] * TEP8_RANKS, sizes + shares += windows([MAX_NUM_TOKENS], [max_seq_len - MAX_NUM_TOKENS], q_begin, q_end) + assert shares == _qsplit_shares([max_seq_len], TEP8_RANKS), shares + assert shares[0] == (0, 1019904) and shares[3] == (0, 1032192), shares + packed = windows([5, 5], [983040, 983040]) + assert packed == [(0, 1966090)] and packed[0][1] > max_seq_len, packed diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_indexer_topk_decode.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_indexer_topk_decode.py new file mode 100644 index 000000000000..3c78077475a4 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_indexer_topk_decode.py @@ -0,0 +1,685 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the indexer_topk_decode entry. + +The `_ML4_1M_DEP4` and `_ML4_1M_DSPARK` cells use scorer widths 1048576 and +1048584, respectively, and run through the scratch-footprint drivers +(`_scratch_dep4`, `_drive_dep4`, `_drive_graph_dep4`). There +the radix scratch is the size each cell names, the leading view of a buffer +with a canary tail. At every call such a cell gates (eager, a graph's warm-up +and each replay) the test checks that the op's first launch wrote a leading +prefix of rows x blocks x index_topk elements and nothing past it, and that the +split is 10 blocks per row at every row count, the whole 16-request arena at +16 single-token or 64 verification rows. Every other cell uses `_scratch`, +`_drive` and `_drive_graph`, whose arena has a guard slot that no call may write. +""" + +import math +import re +import zlib + +import pytest +import torch + +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.indexer_topk_decode import ( # noqa: E501 + MAX_INDEX_TOPK_SINGLE_BLOCK, + MAX_INDEX_TOPK_SPLIT_WORK, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.indexer_topk_decode import ( # noqa: E501 + indexer_topk_decode as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402,F401 — needs the path declared above + +assert torch.cuda.is_available(), "indexer_topk_decode requires a CUDA device" + +#: Written past each row's causal end. A kernel that read past the end would +#: select these before any real score. +POISON = 1.0e9 +#: What `indices` holds before a call; a row the op should not write keeps it. +UNWRITTEN = -7 +#: Blocks per row the DSA metadata's radix scratch reserves for each sequence +#: slot (`_MAX_RADIX_BLOCKS_PER_ROW`), the most the op ever splits a row into. +AUX_BLOCKS = 10 +#: The CUDA-graph padding dummy's KV length (one token, no speculation). +PAD_SEQ_LEN = 1 +#: What `_scratch`'s radix arena holds before a call, and the slot past it keeps. +AUX_SENTINEL = -24301 +#: Columns the `edges` scores plant at each end of every split block: one +#: 16-byte vector of fp32, so a block that starts off that grid hands over its +#: unaligned head and tail columns among them. +EDGE_COLUMNS = 4 +#: Elements past the end of each `_scratch_dep4` radix scratch buffer, which +#: the op is never given and must never write. +CANARY = 4096 +#: What the dep4 drivers' radix index scratch holds before a call (`_arm`): no +#: column, and not the -1 the first launch writes for a block's padding. The +#: logits scratch holds NaN. +AUX_UNWRITTEN = -(2**31) + + +def _ends(seq_lens, next_n: int, rows: int) -> list[int]: + return [max(0, seq_lens[r // next_n] - next_n + r % next_n + 1) for r in range(rows)] + + +def _edge_plan(end: int, k: int) -> tuple[list[int], list[int], list[int]]: + """The `edges` scores' planted columns of a row ending at `end`, as `(top, inside, outside)`. + + The row is cut as the op cuts it at `AUX_BLOCKS` blocks: floor(end / blocks) + columns each, the last block the rest. Each block's first and last + `EDGE_COLUMNS` columns are planted, alternately at the top of the row (`top`) + and around its k-th score: `inside` just above it, so among the winners, and + `outside` just below it, so not. + """ + span = end // AUX_BLOCKS + edges = set() + for b in range(AUX_BLOCKS): + lo = b * span + hi = end if b == AUX_BLOCKS - 1 else lo + span + edges.update(range(lo, min(lo + EDGE_COLUMNS, hi))) + edges.update(range(max(hi - EDGE_COLUMNS, lo), hi)) + edges = sorted(edges) + top, around = edges[0::2], edges[1::2] + inside, outside = around[0::2], around[1::2] + assert end > k + len(outside), (end, k) + return top, inside, outside + + +def _compare(out: torch.Tensor, ref: torch.Tensor, logits, seq_lens, next_n, by_score) -> None: + """Exact per row: the winners first, then -1 padding; the op does not order its winners. + + A top-k picks the same keys or it does not; there is no rounding to absorb. + With distinct scores the winning set is unique, so each row is sorted and + compared to the reference's (`by_score=False`). With ties at the k-th + score the set is not unique: then the selected columns must be distinct + and causal, and the multiset of their fp32 scores must be the reference's + exactly, which every correct choice among the tied columns satisfies and + a wrong winner does not. + """ + assert out.dtype == ref.dtype == torch.int32, (out.dtype, ref.dtype) + assert out.shape == ref.shape, (out.shape, ref.shape) + k = out.shape[1] + ends = _ends(seq_lens, next_n, out.shape[0]) + for r, end in enumerate(ends): + n = min(k, end) + row = out[r] + assert bool((row[n:] == -1).all()) and bool((row[:n] >= 0).all()), ( + f"row {r} (end {end}): the {n} winners must come first and -1 after; " + f"got {int((row >= 0).sum())} columns and {int((row == -1).sum())} -1s" + ) + if not by_score: + continue + won = row[:n].long() + assert won.unique().numel() == n and bool((won < end).all()), ( + f"row {r}: selected columns repeat or pass the causal end {end}" + ) + got = logits[r, won].sort().values + want = logits[r, ref[r, :n].long()].sort().values + assert torch.equal(got, want), f"row {r}: the selected scores are not the top {n}" + if not by_score: + got, want = out.sort(dim=1).values, ref.sort(dim=1).values + bad = (got != want).any(dim=1) + assert not bool(bad.any()), ( + f"{int(bad.sum())} of {out.shape[0]} rows select a different key set; " + f"first at row {int(bad.nonzero()[0])}" + ) + + +def _scores(kind: str, row: int, end: int, g: torch.Generator, k: int = 2048) -> torch.Tensor: + if kind == "edges": + # A random permutation of distinct integers with every split block's + # first and last columns moved to the ranks `_edge_plan` gives them: + # the top ranks, and the ranks either side of the k-th score. + scores = torch.randperm(end, generator=g) + where = torch.empty(end, dtype=torch.int64) + where[scores] = torch.arange(end) + top, inside, outside = _edge_plan(end, k) + ranks = list(range(len(top))) + ranks += [k - 1 - j for j in range(len(inside))] + [k + j for j in range(len(outside))] + for col, rank in zip(top + inside + outside, ranks): + value, held = end - 1 - rank, int(scores[col]) + other = int(where[value]) + scores[col], scores[other] = value, held + where[value], where[held] = col, other + return scores.float() + if kind == "perm": + # A random permutation of distinct integers: the top-k set is unique. + return torch.randperm(end, generator=g).float() + if kind == "ties": + # Ties at the k-th value: 16 levels, all +0.0 (every head's relu at its + # floor), and +0.0 mixed with -0.0, which compare equal. + if row % 3 == 0: + return torch.randint(0, 16, (end,), generator=g).float() + zeros = torch.zeros(end) + if row % 3 == 2: + zeros[torch.rand(end, generator=g) < 0.5] = -0.0 + return zeros + if kind == "wide": + # Distinct magnitudes from 2^-100 to 2^100, both signs: most are outside + # fp16's range or below its smallest subnormal. + exponent = torch.randperm(end, generator=g).double() / end * 200.0 - 100.0 + sign = torch.where(torch.rand(end, generator=g) < 0.5, -1.0, 1.0).double() + return (sign * torch.exp2(exponent)).float() + raise ValueError(kind) + + +def _logits(spec, seq_lens, next_n: int, g: torch.Generator) -> torch.Tensor: + width, pad = spec["width"], spec.get("row_pad", 0) + rows = len(seq_lens) * next_n + parent = torch.empty(rows, width + pad, dtype=torch.float32) + for r, end in enumerate(_ends(seq_lens, next_n, rows)): + parent[r] = POISON + torch.arange(width + pad, dtype=torch.float32) + parent[r, :end] = _scores(spec.get("scores", "perm"), r, end, g, spec["index_topk"]) + logits = parent.cuda()[:, :width] + assert pad == 0 or not logits.is_contiguous() + return logits + + +def _scratch(spec, rows: int): + """The radix scratch, as `((indices, logits), (guard_indices, guard_logits))`. + + The engine's per-sequence arena when the cell names its size (`aux_rows`), + else sized for the batch. The arena is the leading `slots` of a buffer one + sequence slot longer, all of it `AUX_SENTINEL`; the extra slot is the + guard, which no call may write. + """ + slots = spec.get("aux_rows", rows) + assert slots >= rows + shape = (slots + 1, AUX_BLOCKS, spec["index_topk"]) + idx = torch.full(shape, AUX_SENTINEL, dtype=torch.int32, device="cuda") + val = torch.full(shape, AUX_SENTINEL, dtype=torch.float32, device="cuda") + return (idx[:slots], val[:slots]), (idx[slots:], val[slots:]) + + +def _assert_guard_untouched(guard) -> None: + assert bool((guard[0] == AUX_SENTINEL).all()) and bool((guard[1] == AUX_SENTINEL).all()), ( + "the call wrote past the radix arena" + ) + + +def _scratch_dep4(spec, rows: int): + """The dep4 deployment's radix scratch (its file's `_scratch`), for its drivers + and its own tests: the engine's per-sequence arena when the cell names its + size (`aux_rows`), else sized for the batch. + + Each buffer is exactly that size, `[slots, 10, index_topk]` as the DSA + metadata allocates it, and the leading view of an allocation whose + CANARY-element tail the op is never given, so a write past the arena shows. + """ + slots = spec.get("aux_rows", rows) + assert slots >= rows + shape = (slots, AUX_BLOCKS, spec["index_topk"]) + n = math.prod(shape) + return tuple( + torch.empty(n + CANARY, dtype=dtype, device="cuda")[:n].view(shape) + for dtype in (torch.int32, torch.float32) + ) + + +def _arm(aux) -> None: + """Fill both scratch buffers and their canaries before a call.""" + aux[0]._base.fill_(AUX_UNWRITTEN) + aux[1]._base.fill_(float("nan")) + + +def _check_scratch(aux, rows: int, k: int, blocks=None) -> int: + """What the call's first launch wrote: rows x blocks x index_topk elements of + each buffer, a leading prefix of the arena, and nothing after it or in the + canary. Returns the blocks per row that footprint implies (0: one block a + row, no scratch).""" + idx, val = aux[0]._base, aux[1]._base + w_i, w_l = idx != AUX_UNWRITTEN, ~torch.isnan(val) + gaps = [(w[1:] & ~w[:-1]).sum() for w in (w_i, w_l)] + n_i, n_l, gap_i, gap_l = torch.stack([w_i.sum(), w_l.sum(), *gaps]).tolist() + assert not gap_i and not gap_l, "the scratch written is not a leading prefix" + assert n_i == n_l and n_i <= aux[0].numel(), ( + f"the scratch footprint is {n_i} / {n_l} elements against an arena of {aux[0].numel()}" + ) + assert n_i % (rows * k) == 0, (n_i, rows, k) + got = n_i // (rows * k) + assert blocks is None or got == blocks, f"{rows} rows split {got} blocks per row, not {blocks}" + return got + + +def _call(logits, seq_lens, indices, next_n, k, aux_idx, aux_val): + # As the targets call it: keyword scratch, compress_ratio left at 1. + return op( + logits, + seq_lens, + indices, + next_n, + k, + radix_aux_indices=aux_idx, + radix_aux_logits=aux_val, + ) + + +def _drive(spec, seq_lens, logits, aux, guard, by_score): + """One eager call over `seq_lens`, gated against the entry's reference.""" + next_n, k = spec["next_n"], spec["index_topk"] + rows = logits.shape[0] + lens = torch.tensor(seq_lens, dtype=torch.int32, device="cuda") + tail_of = spec.get("tail_of") + if tail_of is None: + buf = torch.full((rows, k), UNWRITTEN, dtype=torch.int32, device="cuda") + indices = buf + else: + # A mixed step: the context rows' selection sits ahead of these rows in + # one buffer, and the call writes only its tail. + buf = torch.full((tail_of, k), UNWRITTEN, dtype=torch.int32, device="cuda") + indices = buf[tail_of - rows :] + out = _call(logits, lens, indices, next_n, k, *aux) + assert out is indices, "the op writes the caller's buffer and returns it" + _assert_guard_untouched(guard) + _compare( + out, op.reference(logits, lens, indices, next_n, k), logits, seq_lens, next_n, by_score + ) + if tail_of is not None: + assert bool((buf[: tail_of - rows] == UNWRITTEN).all()), ( + "a row ahead of the slice was written" + ) + if spec.get("scores") == "edges": + # The planted block-edge columns are where the gate looks: the winners + # hold every one scored above the k-th score and none below it. + for r, end in enumerate(_ends(seq_lens, next_n, rows)): + top, inside, outside = _edge_plan(end, k) + won = set(out[r].tolist()) + assert set(top + inside) <= won and not set(outside) & won, ( + f"row {r} (end {end}): a block-edge column is missing or wrongly selected" + ) + + +def _drive_dep4(spec, seq_lens, logits, aux, by_score): + """One eager call over `seq_lens`, gated against the entry's reference. + + The dep4 deployment's `_drive`: its scratch armed before the call and its + footprint checked after it, in place of `_drive`'s guard slot. + """ + next_n, k = spec["next_n"], spec["index_topk"] + rows = logits.shape[0] + lens = torch.tensor(seq_lens, dtype=torch.int32, device="cuda") + tail_of = spec.get("tail_of") + if tail_of is None: + buf = torch.full((rows, k), UNWRITTEN, dtype=torch.int32, device="cuda") + indices = buf + else: + # A mixed step: the context rows' selection sits ahead of these rows in + # one buffer, and the call writes only its tail. + buf = torch.full((tail_of, k), UNWRITTEN, dtype=torch.int32, device="cuda") + indices = buf[tail_of - rows :] + _arm(aux) + out = _call(logits, lens, indices, next_n, k, *aux) + assert out is indices, "the op writes the caller's buffer and returns it" + _check_scratch(aux, rows, k, spec.get("blocks_per_row")) + _compare( + out, op.reference(logits, lens, indices, next_n, k), logits, seq_lens, next_n, by_score + ) + if tail_of is not None: + assert bool((buf[: tail_of - rows] == UNWRITTEN).all()), ( + "a row ahead of the slice was written" + ) + + +def _drive_graph(spec, g, aux, guard, by_score): + """Capture once per batch, as the engine does, and replay on new logits and + seq_lens. The op orders its winners differently on every call, so a replay + is gated as a set, against an eager call on the same data and against the + entry's reference.""" + next_n, k, width = spec["next_n"], spec["index_topk"], spec["width"] + for batch in spec["graph_batches"]: + capture_lens = spec["seq_lens"][:batch] + logits = _logits(spec, capture_lens, next_n, g) + lens = torch.tensor(capture_lens, dtype=torch.int32, device="cuda") + indices = torch.full((batch * next_n, k), UNWRITTEN, dtype=torch.int32, device="cuda") + side = torch.cuda.Stream() + side.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(side): + _call(logits, lens, indices, next_n, k, *aux) + torch.cuda.current_stream().wait_stream(side) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + _call(logits, lens, indices, next_n, k, *aux) + for i in range(spec["replays"]): + if "replay_lens" in spec: + new_lens = list(spec["replay_lens"][i][:batch]) + else: + new_lens = torch.randint(next_n, width + 1, (batch,), generator=g).tolist() + if batch > 2: + new_lens[-1] = PAD_SEQ_LEN + next_n - 1 + new = _logits(spec, new_lens, next_n, g) + logits.copy_(new) + lens.copy_(torch.tensor(new_lens, dtype=torch.int32)) + indices.fill_(UNWRITTEN) + graph.replay() + torch.cuda.synchronize() + _assert_guard_untouched(guard) + eager = _call( + new, + torch.tensor(new_lens, dtype=torch.int32, device="cuda"), + torch.full_like(indices, UNWRITTEN), + next_n, + k, + *aux, + ) + _compare(indices, eager, new, new_lens, next_n, by_score) + ref = op.reference(new, lens, indices, next_n, k) + _compare(indices, ref, new, new_lens, next_n, by_score) + + +def _drive_graph_dep4(spec, g, aux, by_score): + """Capture once per batch, as the engine does, and replay on new logits and + seq_lens. The op orders its winners differently on every call, so a replay + is gated as a set, against an eager call on the same data and against the + entry's reference. + + The dep4 deployment's `_drive_graph`: its scratch armed before the warm-up and + each replay and its footprint checked after them, in place of + `_drive_graph`'s guard slot. The engine's warm-up call ahead of each capture runs eagerly on the + capture's own rows (at boot, the long-family dummy) and is gated too. With + `live`, each replay is attention DP's padded batch: `live` rows of this + rank's own, the first at the full width, one less, then a drawn length, + and the rest of the largest rank's batch padding dummies at `next_n` keys. + """ + next_n, k, width = spec["next_n"], spec["index_topk"], spec["width"] + blocks = spec.get("blocks_per_row") + for batch in spec["graph_batches"]: + capture_lens = spec["seq_lens"][:batch] + logits = _logits(spec, capture_lens, next_n, g) + lens = torch.tensor(capture_lens, dtype=torch.int32, device="cuda") + indices = torch.full((batch * next_n, k), UNWRITTEN, dtype=torch.int32, device="cuda") + _arm(aux) + side = torch.cuda.Stream() + side.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(side): + _call(logits, lens, indices, next_n, k, *aux) + torch.cuda.current_stream().wait_stream(side) + _check_scratch(aux, batch * next_n, k, blocks) + warm = op.reference(logits, lens, indices, next_n, k) + _compare(indices, warm, logits, capture_lens, next_n, by_score) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + _call(logits, lens, indices, next_n, k, *aux) + for replay in range(spec["replays"]): + if "replay_lens" in spec: + new_lens = list(spec["replay_lens"][replay][:batch]) + else: + new_lens = torch.randint(next_n, width + 1, (batch,), generator=g).tolist() + if "live" in spec: + live = min(spec["live"], batch) + if replay < 2: + new_lens[0] = width - replay + new_lens[live:] = [PAD_SEQ_LEN + next_n - 1] * (batch - live) + elif batch > 2: + new_lens[-1] = PAD_SEQ_LEN + next_n - 1 + new = _logits(spec, new_lens, next_n, g) + logits.copy_(new) + lens.copy_(torch.tensor(new_lens, dtype=torch.int32)) + indices.fill_(UNWRITTEN) + _arm(aux) + graph.replay() + torch.cuda.synchronize() + _check_scratch(aux, batch * next_n, k, blocks) + eager = _call( + new, + torch.tensor(new_lens, dtype=torch.int32, device="cuda"), + torch.full_like(indices, UNWRITTEN), + next_n, + k, + *aux, + ) + _compare(indices, eager, new, new_lens, next_n, by_score) + ref = op.reference(new, lens, indices, next_n, k) + _compare(indices, ref, new, new_lens, next_n, by_score) + + +def _is_dep4(spec) -> bool: + """Cells with an explicit split count use the scratch-footprint drivers.""" + return "blocks_per_row" in spec + + +def _cell_id(cell) -> str: + """`why[:44]`, as each target's file named its cells. Five of the dep4 + deployment's whys share their first 44 characters with the tp4 deployment's 1M + cells, so the dep4 deployment's cells carry a ` (dep4)` suffix.""" + return cell.why[:44] + (" (dep4)" if _is_dep4(cell.spec) and cell.spec["next_n"] == 1 else "") + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[_cell_id(c) for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 + certifying(op) + spec = cell.spec + g = torch.Generator().manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + by_score = spec.get("scores") == "ties" + rows = len(spec["seq_lens"]) * spec["next_n"] + # The dep4 deployment's cells take its scratch and drivers (armed scratch, + # footprint and split checked at every call); every other cell the guard + # slot's. + dep4 = _is_dep4(spec) + aux, guard = (_scratch_dep4(spec, rows), None) if dep4 else _scratch(spec, rows) + if "graph_batches" in spec: + if dep4: + _drive_graph_dep4(spec, g, aux, by_score) + else: + _drive_graph(spec, g, aux, guard, by_score) + return + logits = _logits(spec, spec["seq_lens"], spec["next_n"], g) + for batch in spec.get("batches", (len(spec["seq_lens"]),)): + # The leading `batch` sequences: rows are independent, so a smaller + # batch is the same operands' leading rows. + lens = spec["seq_lens"][:batch] + if dep4: + _drive_dep4(spec, lens, logits[: batch * spec["next_n"]], aux, by_score) + else: + _drive(spec, lens, logits[: batch * spec["next_n"]], aux, guard, by_score) + + +@pytest.mark.parametrize("next_n,width,row_pad", [(1, 1048576, 0), (4, 1048584, 248)]) +def test_sw1m_sixteen_requests_need_the_whole_arena(next_n, width, row_pad) -> None: + """Every row at 1M splits 10 ways: one missing scratch element must fail + before any write, both for single-token decode and four-token verification.""" + k, rows = 2048, 16 * next_n + spec = dict(next_n=next_n, width=width, row_pad=row_pad, index_topk=k, aux_rows=rows) + lens = [width] * 16 + logits = _logits(spec, lens, next_n, torch.Generator().manual_seed(14)) + seq_lens = torch.tensor(lens, dtype=torch.int32, device="cuda") + indices = torch.full((rows, k), UNWRITTEN, dtype=torch.int32, device="cuda") + aux = _scratch_dep4(spec, rows) + short = tuple(t.view(-1)[:-1] for t in aux) + _arm(aux) + need = rows * AUX_BLOCKS * k + with pytest.raises(RuntimeError, match=rf"got {need - 1} / {need - 1}, need {need}\)"): + _call(logits, seq_lens, indices, next_n, k, *short) + assert bool((indices == UNWRITTEN).all()) + assert _check_scratch(aux, rows, k) == 0 + _call(logits, seq_lens, indices, next_n, k, *aux) + assert _check_scratch(aux, rows, k) == 10 + _compare( + indices, op.reference(logits, seq_lens, indices, next_n, k), logits, lens, next_n, False + ) + + +def test_short_rows_list_their_columns_in_order() -> None: + """On the single-block path, a row no wider than top-k is the identity, in order, then -1.""" + spec = dict(next_n=2, width=64, index_topk=16) + g = torch.Generator().manual_seed(11) + logits = _logits(spec, [5, 40], 2, g) + indices = torch.full((4, 16), UNWRITTEN, dtype=torch.int32, device="cuda") + out = _call( + logits, torch.tensor([5, 40], dtype=torch.int32, device="cuda"), indices, 2, 16, None, None + ) + assert out[0].tolist() == [0, 1, 2, 3] + [-1] * 12 + assert out[1].tolist() == [0, 1, 2, 3, 4] + [-1] * 11 + + +def _small(): + spec = dict(next_n=1, width=128, index_topk=16) + logits = _logits(spec, [100, 80], 1, torch.Generator().manual_seed(7)) + lens = torch.tensor([100, 80], dtype=torch.int32, device="cuda") + return logits, lens + + +def test_guard_refuses_a_wider_indices_buffer() -> None: + """The kernel writes at a row stride of index_topk, not of indices.size(1).""" + logits, lens = _small() + ref = op.reference(logits, lens, None, 1, 16) + wide = torch.full((2, 32), UNWRITTEN, dtype=torch.int32, device="cuda") + _call(logits, lens, wide, 1, 16, None, None) + # Unguarded: row 1's winners land in row 0's second half, row 1 is never written. + assert set(wide[0, 16:].tolist()) == set(ref[1].tolist()) + assert bool((wide[1] == UNWRITTEN).all()) + with pytest.raises(AssertionError, match="exactly 16 columns wide"): + with validating(op): + _call(logits, lens, wide, 1, 16, None, None) + + +def test_the_op_refuses_int64_seq_lens_itself() -> None: + """Loud, so `is_valid` does not restate it.""" + logits, lens = _small() + indices = torch.full((2, 16), UNWRITTEN, dtype=torch.int32, device="cuda") + with pytest.raises(RuntimeError, match="expected scalar type Int but found Long"): + _call(logits, lens.long(), indices, 1, 16, None, None) + assert bool((indices == UNWRITTEN).all()) + + +#: The Mistral Large 4 targets' scorer widths: the FP8 target's 9216, the NVFP4 +#: target's 32768 and its 1M envelope's 1048576, the dep4 deployment's at the +#: checkpoint's own context (10 blocks per row at the two rows the guards below +#: pass, at both NVFP4 widths). +ML4_WIDTHS = [9216, 32768, 1048576] + + +@pytest.mark.parametrize("width", [32776, 1048584]) +@pytest.mark.parametrize("seq_len", [1, 2]) +def test_guard_refuses_a_negative_uniform_causal_end(seq_len, width) -> None: + spec = dict(next_n=4, width=width, row_pad=248, index_topk=2048, aux_rows=4) + logits = _logits(spec, [4], 4, torch.Generator().manual_seed(9)) + lens = torch.tensor([seq_len], dtype=torch.int32, device="cuda") + indices = torch.full((4, 2048), UNWRITTEN, dtype=torch.int32, device="cuda") + with pytest.raises(AssertionError, match="negative causal end"): + with validating(op): + _call(logits, lens, indices, 4, 2048, *_scratch(spec, 4)) + assert bool((indices == UNWRITTEN).all()) + + +@pytest.mark.parametrize("width", ML4_WIDTHS) +def test_guard_refuses_a_causal_end_past_the_row(width) -> None: + """At Mistral Large 4's widths: an end past the row reads the next row's columns.""" + spec = dict(next_n=1, width=width, index_topk=2048, aux_rows=16) + logits = _logits(spec, [width, width], 1, torch.Generator().manual_seed(9)) + logits[1, :64] = 5.0e8 + torch.arange(64, device="cuda", dtype=torch.float32) + long_lens = torch.tensor([width + 64, width], dtype=torch.int32, device="cuda") + indices = torch.full((2, 2048), UNWRITTEN, dtype=torch.int32, device="cuda") + aux, _ = _scratch(spec, 2) + _call(logits, long_lens, indices, 1, 2048, *aux) + # Unguarded: row 0 selects all 64 planted columns of row 1, as indices past its end. + assert int((indices[0] >= width).sum()) == 64 + with pytest.raises(AssertionError, match=f"exceeds the {width}-column logits"): + with validating(op): + _call(logits, long_lens, indices, 1, 2048, *aux) + + +@pytest.mark.parametrize("width", ML4_WIDTHS) +def test_guard_refuses_a_column_strided_view(width) -> None: + """At Mistral Large 4's widths the split path selects a wrong set, past the row's end.""" + spec = dict(next_n=1, width=width, index_topk=2048, aux_rows=16) + lens = [width, 5000] + logits = _logits(spec, lens, 1, torch.Generator().manual_seed(8)) + holder = torch.full((width, 2), -5.0e8, dtype=torch.float32, device="cuda") + strided = holder.t() + strided.copy_(logits) + assert strided.stride() == (1, 2) + seq_lens = torch.tensor(lens, dtype=torch.int32, device="cuda") + indices = torch.full((2, 2048), UNWRITTEN, dtype=torch.int32, device="cuda") + aux, _ = _scratch(spec, 2) + _call(strided, seq_lens, indices, 1, 2048, *aux) + ref = op.reference(logits, seq_lens, indices, 1, 2048) + # Unguarded: both rows select other columns than the reference, row 0 past its end. + for r in range(2): + assert set(indices[r].tolist()) != set(ref[r].tolist()) + assert int((indices[0] >= width).sum()) > 0 + with pytest.raises(AssertionError, match="unit-stride"): + with validating(op): + _call(strided, seq_lens, indices, 1, 2048, *aux) + + +@pytest.mark.parametrize( + "width,k", + [(128, MAX_INDEX_TOPK_SINGLE_BLOCK + 1)] + + [(w, MAX_INDEX_TOPK_SPLIT_WORK + 1) for w in ML4_WIDTHS], +) +def test_guard_refuses_top_k_past_the_launch_ceiling(width, k) -> None: + spec = dict(next_n=1, width=width, index_topk=k) + lens = [min(width, 100)] if width < 4096 else [width] + logits = _logits(spec, lens, 1, torch.Generator().manual_seed(12)) + seq_lens = torch.tensor(lens, dtype=torch.int32, device="cuda") + indices = torch.full((1, k), UNWRITTEN, dtype=torch.int32, device="cuda") + aux, _ = _scratch(spec, 1) + # Unguarded: the call returns quietly and the launch never ran. Torch's next + # launch does not report it either; the error stays in the CUDA runtime that + # libth_common links statically. + _call(logits, seq_lens, indices, 1, k, *aux) + torch.ones(1, device="cuda").add_(1) + torch.cuda.synchronize() + assert bool((indices == UNWRITTEN).all()) + # One past the ceiling is refused; the ceiling itself launches and answers. + with pytest.raises(AssertionError, match="index_topk must be in"): + with validating(op): + _call(logits, seq_lens, indices, 1, k, *aux) + at = indices[:, : k - 1].contiguous() + with validating(op): + _call(logits, seq_lens, at, 1, k - 1, *aux) + ref = op.reference(logits, seq_lens, at, 1, k - 1) + _compare(at, ref, logits, lens, 1, by_score=False) + + +#: Columns from which the host splits every row across 10 blocks whatever the +#: row count (its forced-split threshold), and the 1M envelope's scorer width. +#: Below the threshold the split also depends on the device's SM count. +FORCED_SPLIT_WIDTHS = [200000, 1048576] + + +def _blocks_per_row(rows: int, width: int, k: int = 2048) -> int: + """The host's split, from the scratch the op asks a one-element arena for. + + The op refuses that arena before any launch. + """ + logits = torch.zeros(rows, width, device="cuda") + lens = torch.full((rows,), width, dtype=torch.int32, device="cuda") + indices = torch.full((rows, k), UNWRITTEN, dtype=torch.int32, device="cuda") + one = ( + torch.empty(1, dtype=torch.int32, device="cuda"), + torch.empty(1, dtype=torch.float32, device="cuda"), + ) + with pytest.raises(RuntimeError, match=r"need \d+\)") as err: + _call(logits, lens, indices, 1, k, *one) + need = int(re.search(r"need (\d+)\)", str(err.value)).group(1)) + assert need % (rows * k) == 0, need + return need // (rows * k) + + +@pytest.mark.parametrize("width", FORCED_SPLIT_WIDTHS) +def test_ten_blocks_per_row_at_every_row_count_from_200000_columns(width) -> None: + """At 1048576 columns every decode batch 1..16 splits its rows 10 ways.""" + assert [_blocks_per_row(rows, width) for rows in range(1, 17)] == [AUX_BLOCKS] * 16 + + +def test_sixteen_rows_at_1m_fill_the_radix_arena_exactly() -> None: + """16 rows x 10 blocks x 2048 is the DSA arena [16, 10, 2048] to the element. + + Every element is written and nothing past it; one element less is refused by the op. + """ + spec = dict(next_n=1, width=1 << 20, index_topk=2048, aux_rows=16) + lens = [1 << 20, (1 << 20) - 1, 2049, 20481, 20490, 1, 524288, 1000000] * 2 + logits = _logits(spec, lens, 1, torch.Generator().manual_seed(13)) + seq_lens = torch.tensor(lens, dtype=torch.int32, device="cuda") + indices = torch.full((16, 2048), UNWRITTEN, dtype=torch.int32, device="cuda") + aux, guard = _scratch(spec, 16) + _call(logits, seq_lens, indices, 1, 2048, *aux) + ref = op.reference(logits, seq_lens, indices, 1, 2048) + _compare(indices, ref, logits, lens, 1, by_score=False) + assert not bool((aux[0] == AUX_SENTINEL).any()), "an arena element was left unwritten" + _assert_guard_untouched(guard) + short = (aux[0].reshape(-1)[:-1], aux[1].reshape(-1)[:-1]) + with pytest.raises(RuntimeError, match=r"got 327679 / 327679, need 327680"): + _call(logits, seq_lens, indices, 1, 2048, *short) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_split_radix_topk_decode.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_split_radix_topk_decode.py new file mode 100644 index 000000000000..3975d7a3ca6f --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_split_radix_topk_decode.py @@ -0,0 +1,295 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""SM107 regression tests for the experimental DSpark partition/radix-merge selector.""" + +from unittest.mock import patch + +import pytest +import torch + +if not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7): + pytest.skip("split radix requires SM107 validation", allow_module_level=True) + +import tensorrt_llm._torch.custom_ops # noqa: E402,F401 +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_radix_topk_decode import ( # noqa: E402 + TOP_K, + blocks_per_row, + build_split, + split_radix_topk_decode, + validate_split_radix_topk_lengths, +) + + +def _buffers(batch, width, next_n=4): + rows = batch * next_n + stride = (width + 255) // 256 * 256 + storage = torch.empty(rows * stride + 8, dtype=torch.float32, device="cuda") + logits = storage[8:].as_strided((rows, width), (stride, 1)) + output_storage = torch.full((rows + 1, TOP_K), -7, dtype=torch.int32, device="cuda") + output = output_storage[1:] + aux_i = torch.full((rows, 10, TOP_K), -7, dtype=torch.int32, device="cuda") + aux_l = torch.full((rows, 10, TOP_K), float("nan"), dtype=torch.float32, device="cuda") + return logits, output, aux_i, aux_l + + +def _fill(logits, lengths, next_n, seed, kind): + generator = torch.Generator(device="cuda").manual_seed(seed) + logits.fill_(1.0e9) + for row in range(logits.shape[0]): + end = lengths[row // next_n] - next_n + row % next_n + 1 + scores = torch.randperm(end, generator=generator, device="cuda").float() + if kind == "ties": + scores.remainder_(7) + elif kind == "boundaries": + for part in range(1, 10): + boundary = end // 10 * part + if boundary < end: + scores[max(0, boundary - 1) : boundary + 1] += end + logits[row, :end] = scores + + +def _check(logits, lengths, output, next_n=4): + prefix = output.as_strided((TOP_K,), (1,), storage_offset=0) + assert bool((prefix == -7).all()), "the output view must not overwrite its leading guard" + for row in range(logits.shape[0]): + end = lengths[row // next_n] - next_n + row % next_n + 1 + indices = output[row] + valid = indices[indices >= 0].long() + assert bool(((indices == -1) | (indices >= 0)).all()) + expected_count = min(TOP_K, end) + assert valid.numel() == expected_count + assert bool((indices[:expected_count] >= 0).all()), "valid columns must form a prefix" + assert bool((indices[expected_count:] == -1).all()), "padding must be a suffix" + assert valid.unique().numel() == valid.numel() + assert bool((valid < end).all()) + selected = logits[row, valid].sort().values + reference = logits[row, :end].topk(min(TOP_K, end)).values.sort().values + torch.testing.assert_close(selected, reference, rtol=0, atol=0) + + +@pytest.mark.parametrize("batch", [1, 2, 4]) +@pytest.mark.parametrize("seed,kind", [(0, "ordinary"), (17, "ties"), (42, "boundaries")]) +@pytest.mark.parametrize("width", [16384, 65536]) +def test_split_radix_causal_selection(batch, seed, kind, width): + compiled = build_split(torch.cuda.current_device()) + logits, output, aux_i, aux_l = _buffers(batch, width) + edge_lengths = [3, 4, 5, 2047, 2048, 2049, 20479, 20480, 20481, width] + for offset in range(0, len(edge_lengths), batch): + lengths = [min(width, edge_lengths[(offset + b) % len(edge_lengths)]) for b in range(batch)] + lens = torch.tensor(lengths, dtype=torch.int32, device="cuda") + _fill(logits, lengths, 4, seed, kind) + validate_split_radix_topk_lengths(logits, lens, next_n=4) + output.fill_(-7) + aux_i.fill_(-7) + aux_l.fill_(float("nan")) + assert ( + split_radix_topk_decode(logits, lens, output, aux_i, aux_l, compiled=compiled) is output + ) + _check(logits, lengths, output) + + +@pytest.mark.parametrize("next_n,batch,width", [(1, 8, 65536), (4, 8, 65536), (4, 1, 1048584)]) +def test_split_radix_protection(next_n, batch, width): + compiled = build_split(torch.cuda.current_device(), next_n=next_n) + logits, output, aux_i, aux_l = _buffers(batch, width, next_n) + lengths = [width] * batch + lens = torch.tensor(lengths, dtype=torch.int32, device="cuda") + _fill(logits, lengths, next_n, 42, "boundaries") + split_radix_topk_decode(logits, lens, output, aux_i, aux_l, compiled=compiled) + _check(logits, lengths, output, next_n) + + +def test_split_radix_graph_dynamic_lengths(): + compiled = build_split(torch.cuda.current_device()) + width = 32776 + assert blocks_per_row(4, width, compiled.sm_count) == 10 + logits, output, aux_i, aux_l = _buffers(1, width) + lens = torch.tensor([width], dtype=torch.int32, device="cuda") + _fill(logits, [width], 4, 0, "ordinary") + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + split_radix_topk_decode(logits, lens, output, aux_i, aux_l, compiled=compiled) + edges = [3, 4, 5, 2047, 2048, 2049, 4096, width - 1, width] + for replay in range(100): + lengths = [edges[replay % len(edges)]] + _fill(logits, lengths, 4, replay, "ties" if replay % 2 else "ordinary") + lens.fill_(lengths[0]) + output.fill_(-7) + aux_i.fill_(-7) + aux_l.fill_(float("nan")) + graph.replay() + _check(logits, lengths, output) + + +@pytest.mark.parametrize("seed,kind", [(0, "ordinary"), (42, "boundaries")]) +def test_split_radix_partition_heads_within_vector(seed, kind): + compiled = build_split(torch.cuda.current_device()) + logits, output, aux_i, aux_l = _buffers(1, 65536) + for residue in range(8): + # Every row's partitions span 4096 + residue columns, so their starts fall at + # every offset inside a 256-bit vector and the aligned reads cover a neighbor's + # columns on both sides. + lengths = [10 * (4096 + residue) + 3] + lens = torch.tensor(lengths, dtype=torch.int32, device="cuda") + _fill(logits, lengths, 4, seed + residue, kind) + output.fill_(-7) + split_radix_topk_decode(logits, lens, output, aux_i, aux_l, compiled=compiled) + _check(logits, lengths, output) + + +@pytest.mark.parametrize("layout", ["unaligned_rows", "unaligned_width"]) +def test_split_radix_unaligned_layout_uses_native(layout): + from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.indexer_topk_decode import ( + indexer_topk_decode as native_topk, + ) + + compiled = build_split(torch.cuda.current_device()) + width = 65536 if layout == "unaligned_rows" else 65535 + _, output, aux_i, aux_l = _buffers(1, 65536) + storage = torch.empty(4 * 65536 + 9, dtype=torch.float32, device="cuda") + offset = 1 if layout == "unaligned_rows" else 8 + logits = storage[offset:].as_strided((4, width), (65536, 1)) + lengths = [width] + lens = torch.tensor(lengths, dtype=torch.int32, device="cuda") + _fill(logits, lengths, 4, 42, "boundaries") + target = ( + "tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention." + "cute_dsl_radix_topk_decode.indexer_topk_decode" + ) + with patch(target, wraps=native_topk) as fallback: + split_radix_topk_decode(logits, lens, output, aux_i, aux_l, compiled=compiled) + fallback.assert_called_once() + _check(logits, lengths, output) + + +def test_split_radix_rejects_invalid_inputs(): + compiled = build_split(torch.cuda.current_device()) + for partitions in (4, 8): + with pytest.raises(ValueError, match="exactly 10"): + build_split(torch.cuda.current_device(), partitions=partitions) + logits, output, aux_i, aux_l = _buffers(1, 16384) + lens = torch.tensor([16384], dtype=torch.int32, device="cuda") + with pytest.raises(ValueError, match="float32"): + split_radix_topk_decode(logits.half(), lens, output, aux_i, aux_l, compiled=compiled) + with pytest.raises(ValueError, match="scratch"): + split_radix_topk_decode( + logits, lens, output, aux_i.flatten()[: 4 * 8 * TOP_K - 1], aux_l, compiled=compiled + ) + for invalid in (2, 16385): + lens.fill_(invalid) + with pytest.raises(ValueError, match="seq_lens"): + validate_split_radix_topk_lengths(logits, lens, next_n=4) + + +@pytest.mark.parametrize("length,seed", [(5, 0), (2049, 17), (65536, 42)]) +def test_split_radix_native_nonfinite_semantics(length, seed): + compiled = build_split(torch.cuda.current_device()) + logits, output, aux_i, aux_l = _buffers(1, 65536) + lengths = [length] + lens = torch.tensor(lengths, dtype=torch.int32, device="cuda") + _fill(logits, lengths, 4, seed, "ordinary") + patterns = torch.tensor( + [0x7FC00000, 0x7F800000, -0x400000, -0x800000, -0x800001, -0x80000000], + dtype=torch.int32, + device="cuda", + ).view(torch.float32) + for row in range(4): + end = length - 3 + row + positions = torch.arange(0, end, 19, device="cuda") + logits[row, positions] = patterns[torch.arange(positions.numel(), device="cuda") % 6] + reference = torch.empty_like(output) + torch.ops.trtllm.indexer_topk_decode(logits, lens, reference, 4, TOP_K, 1, aux_i, aux_l) + aux_i.fill_(-7) + aux_l.fill_(float("nan")) + split_radix_topk_decode(logits, lens, output, aux_i, aux_l, compiled=compiled) + for row in range(4): + selected_keys = [] + for selection in (output[row], reference[row]): + valid = selection[selection >= 0].long() + assert valid.unique().numel() == valid.numel() + assert bool((valid < length - 3 + row).all()) + bits = logits[row, valid].view(torch.int32).long() & 0xFFFFFFFF + keys = torch.where(bits >= 0x80000000, bits, (~bits) & 0x7FFFFFFF) + padding = torch.full( + (int((selection == -1).sum()),), 0xFF7FFFFF, device="cuda", dtype=torch.int64 + ) + selected_keys.append(torch.cat((keys, padding)).sort().values) + assert torch.equal(*selected_keys) + + +def test_split_radix_preserves_native_partition_fallback(): + from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.indexer_topk_decode import ( + indexer_topk_decode as native_topk, + ) + + compiled = build_split(torch.cuda.current_device()) + assert blocks_per_row(4, 16384, compiled.sm_count) == 8 + logits, output, aux_i, aux_l = _buffers(1, 16384) + lens = torch.tensor([20], dtype=torch.int32, device="cuda") + _fill(logits, [20], 4, 42, "ordinary") + logits[:, 1] = -torch.inf + logits[:, 2] = -torch.finfo(torch.float32).max + logits[:, 3] = torch.tensor([-0x3FFFFF], dtype=torch.int32, device="cuda").view(torch.float32) + logits[:, 4] = torch.inf + reference = torch.empty_like(output) + native_topk(logits, lens, reference, 4, TOP_K, 1, aux_i, aux_l) + target = ( + "tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention." + "cute_dsl_radix_topk_decode.indexer_topk_decode" + ) + with patch(target, wraps=native_topk) as fallback: + split_radix_topk_decode(logits, lens, output, aux_i, aux_l, compiled=compiled) + fallback.assert_called_once() + for row in range(4): + selected = [] + for result in (reference[row], output[row]): + col = result[result >= 0].long() + bits = logits[row, col].view(torch.int32) + pads = torch.full((TOP_K - col.numel(),), -0x800001, dtype=torch.int32, device="cuda") + selected.append(torch.cat((bits, pads)).sort().values) + assert torch.equal(*selected) + + +def test_split_radix_sentinel_tie_count_bounds(): + compiled = build_split(torch.cuda.current_device()) + logits, output, aux_i, aux_l = _buffers(1, 1048584) + lens = torch.tensor([16384], dtype=torch.int32, device="cuda") + patterns = torch.tensor( + [0x7F7FFFFF, 0x7F800000, -0x800001, -0x3FFFFF, 0x7FC00001, 0x3F800000, -0x40800000], + dtype=torch.int32, + device="cuda", + ).view(torch.float32) + logits.fill_(1.0e9) + for row in range(4): + end = 16381 + row + logits[row, :end] = -torch.inf + positions = {0, end - 1, 31, 32, 33, 2047, 2048, 2049} + for partitions in (4, 8, 10): + for part in range(1, partitions): + edge = end // partitions * part + positions.update((edge - 1, edge, edge + 1)) + positions = sorted(positions) + logits[row, positions] = patterns[torch.arange(len(positions), device="cuda") % 7] + reference = torch.empty_like(output) + torch.ops.trtllm.indexer_topk_decode(logits, lens, reference, 4, TOP_K, 1, aux_i, aux_l) + split_radix_topk_decode(logits, lens, output, aux_i, aux_l, compiled=compiled) + for row in range(4): + end = 16381 + row + selected = [] + for result in (reference[row], output[row]): + col = result[result >= 0].long() + assert col.unique().numel() == col.numel() + assert bool((col < end).all()) + bits = logits[row, col].view(torch.int32) + selected.append(bits[bits != -0x800001].sort().values) + assert torch.equal(*selected), "only real -FLT_MAX versus padding may differ" + strict_count = selected[0].numel() + real_ties = int((logits[row, :end].view(torch.int32) == -0x800001).sum()) + pads = 10 * TOP_K - end + tie_slots = TOP_K - strict_count + minimum = strict_count + max(0, tie_slots - pads) + maximum = strict_count + min(tie_slots, real_ties) + count = int((output[row] >= 0).sum()) + assert minimum <= count <= maximum + assert bool((output[row, :count] >= 0).all()) + assert bool((output[row, count:] == -1).all()) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_ssmax_indexer_q_side_fused.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_ssmax_indexer_q_side_fused.py new file mode 100644 index 000000000000..f66cf487934c --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_ssmax_indexer_q_side_fused.py @@ -0,0 +1,428 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the ssmax_indexer_q_side_fused entry. + +Every cell is gated bit for bit, codes and weights, against the entry's `reference` (the +composed references of `ssmax_scaling` and `cute_dsl_indexer_q_side_fused`, each bit-exact +for its own op), and against the three calls it replaces run on the same inputs: +`ssmax_scaling`, the head weights' fp32 copy and `cute_dsl_indexer_q_side_fused`. Nothing is +left to accumulation order, so the band is zero. + +The old-versus-new tests state that equivalence on its own, with `torch.equal` on the codes' and +weights' bit patterns: over every cell's random inputs, and over tensors captured from a served +decode step when TRTLLM_MODELING_V2_FUSE_CAPTURE names their directory (`ix_*.pt`, one file per +call: `q`, `positions`, `s`, `b`, the head weights `wq`, `weight_scale` and the name of the rope +table, saved beside them as `table_.pt`), and at the 1M config over the NVFP4 checkpoint's +own indexer layers: each one's s and b and its table (the 1048576-row table the targets build from +the checkpoint's rope parameters and `lerope.alpha`, the identity table on a NoPE indexer layer) +at every position 0..1048575 (skipped where the checkpoint is absent from LLM_MODELS_ROOT). +""" + +import json +import math +import os +import zlib +from pathlib import Path +from types import SimpleNamespace + +import pytest +import torch +from test_common.llm_data import llm_models_root + +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_indexer_q_side_fused import ( # noqa: E501 + build, + cute_dsl_indexer_q_side_fused, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ssmax_indexer_q_side_fused import ( # noqa: E501 + ssmax_indexer_q_side_fused as op, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ssmax_scaling import ( + ssmax_scaling, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402,F401 + +assert torch.cuda.is_available(), "ssmax_indexer_q_side_fused requires a CUDA device" + +HEADS, HEAD_DIM, ROPE_DIM = 32, 128, 64 +WEIGHT_SCALE = HEAD_DIM**-0.5 * HEADS**-0.5 +#: e4m3 NaN: the poison the code buffer starts from. +POISON = 0x7F +#: The NVFP4 checkpoint's `rope_parameters`: YaRN factor 2048 over an original 512 positions, +#: theta 1e4, beta_fast 32, beta_slow 1, and LeRoPE's per-frequency exp(alpha). +THETA, FACTOR, ORIGINAL_MAX, BETA_FAST, BETA_SLOW = 1.0e4, 2048.0, 512, 32.0, 1.0 + +_COMPILED = {} + + +def _compiled(): + if "q_side" not in _COMPILED: + _COMPILED["q_side"] = build(torch.cuda.current_device(), is_neox=False) + return _COMPILED["q_side"] + + +def _yarn_table(positions: int, gen: torch.Generator) -> torch.Tensor: + """The indexer table the targets build, `[positions, 64]` fp32 `[cos | sin]` rounded + through bf16: YaRN frequencies, each scaled by a seeded LeRoPE exp(alpha).""" + half = ROPE_DIM // 2 + pos_freqs = THETA ** (torch.arange(0, ROPE_DIM, 2, dtype=torch.float64) / ROPE_DIM) + lerope = torch.exp(0.3 * torch.randn(half, generator=gen, dtype=torch.float64)) + extrapolation, interpolation = lerope / pos_freqs, lerope / (FACTOR * pos_freqs) + + def correction_dim(rotations: float) -> float: + return (ROPE_DIM * math.log(ORIGINAL_MAX / (rotations * 2 * math.pi))) / ( + 2 * math.log(THETA) + ) + + low = max(math.floor(correction_dim(BETA_FAST)), 0) + high = min(math.ceil(correction_dim(BETA_SLOW)), ROPE_DIM - 1) + ramp = ((torch.arange(half, dtype=torch.float64) - low) / (high - low)).clamp(0, 1) + inv = interpolation * ramp + extrapolation * (1 - ramp) + angles = torch.arange(positions, dtype=torch.float64)[:, None] * inv[None, :] + table = torch.cat([angles.cos(), angles.sin()], dim=1).float() + return table.to(torch.bfloat16).float().cuda().contiguous() + + +def _identity_table(positions: int) -> torch.Tensor: + """A NoPE indexer layer's table: cos 1, sin 0 at every position.""" + return torch.cat([torch.ones(positions, 32), torch.zeros(positions, 32)], 1).cuda() + + +def _queries(shape, kind: str, g: torch.Generator) -> torch.Tensor: + if kind == "normal": + return (2 * torch.randn(shape, device="cuda", generator=g)).to(torch.bfloat16) + exponent = torch.randint(-133, 127, shape, device="cuda", generator=g).float() + mantissa = 1 + torch.rand(shape, device="cuda", generator=g) + sign = 2 * torch.randint(0, 2, shape, device="cuda", generator=g).float() - 1 + return (sign * mantissa * torch.exp2(exponent)).to(torch.bfloat16) + + +def _build(spec, seed, table=None): + """The call's nine arguments, both outputs poisoned. The head weights are the last 32 + columns of a bf16 `[T, 160]` buffer (`kw_bf16`, the targets' view) or a contiguous fp32 + copy (`fp32`); s and b hold the ranges' four corners on heads 0..3. `table`, when given, + is the rope table instead of the spec's own.""" + g = torch.Generator(device="cuda").manual_seed(seed) + cpu = torch.Generator().manual_seed(seed) + tokens, max_pos = spec["tokens"], spec["max_pos"] + q = _queries((tokens, HEADS, HEAD_DIM), spec["q"], g) + if spec["positions"] == "arange": + cached = spec.get("cached", 0) + assert cached + tokens <= max_pos or not cached, (cached, tokens, max_pos) + pos = (cached + torch.arange(tokens, device="cuda")) % max_pos + elif spec["positions"] == "tail": + pos = max_pos - 1 - torch.randint(0, 64, (tokens,), generator=g, device="cuda") + else: + pos = torch.randint(0, max_pos, (tokens,), generator=g, device="cuda") + s, b = ( + torch.empty(HEADS, device="cuda").uniform_(*spec[k], generator=g).bfloat16() + for k in ("s_range", "b_range") + ) + corners = [(sv, bv) for sv in spec["s_range"] for bv in spec["b_range"]] + s[:4] = torch.tensor([c[0] for c in corners], device="cuda").bfloat16() + b[:4] = torch.tensor([c[1] for c in corners], device="cuda").bfloat16() + if table is None: + table = _yarn_table(max_pos, cpu) if spec["table"] == "yarn" else _identity_table(max_pos) + kw = torch.randn(tokens, HEAD_DIM + HEADS, generator=g, device="cuda").bfloat16() + wq = kw[:, HEAD_DIM:] if spec["wq"] == "kw_bf16" else kw[:, HEAD_DIM:].float().contiguous() + q_fp8 = torch.full((tokens, HEADS, HEAD_DIM), POISON, dtype=torch.uint8, device="cuda") + weights = torch.full((tokens, HEADS), float("nan"), device="cuda") + return [ + q, pos.to(torch.int32), s, b, table, wq, WEIGHT_SCALE, + q_fp8.view(torch.float8_e4m3fn), weights, + ] # fmt: skip + + +def _calls(spec, seed): + """The cell's calls' arguments: one call, or for a `position_sweep` (lo, hi) one call of + `tokens` rows at each of lo, lo + tokens, ..., every call on fresh inputs over the cell's + one table.""" + if "position_sweep" not in spec: + yield _build(spec, seed) + return + lo, hi = spec["position_sweep"] + rows = spec["tokens"] + assert (hi - lo) % rows == 0 and hi <= spec["max_pos"], (lo, hi, rows) + table = _build(dict(spec, tokens=1), seed)[4] + for k in range((hi - lo) // rows): + args = _build(spec, seed + 1 + k, table) + args[1] = torch.arange(lo + k * rows, lo + (k + 1) * rows, dtype=torch.int32, device="cuda") + yield args + + +def _replaced(args): + """The three calls the entry replaces, into fresh buffers.""" + q, pos, s, b, table, wq, scale, q_fp8, weights = args + out_fp8, out_w = torch.empty_like(q_fp8), torch.empty_like(weights) + return cute_dsl_indexer_q_side_fused( + ssmax_scaling(q, pos, s, b), pos, table, wq.float().contiguous(), scale, out_fp8, out_w, + is_neox=False, compiled=_compiled(), + ) # fmt: skip + + +def _compare(out, want, what: str) -> None: + (q_fp8, weights), (want_fp8, want_weights) = out, want + bad = q_fp8.view(torch.uint8) != want_fp8.view(torch.uint8) + assert not bool(bad.any()), ( + f"{what}: {int(bad.sum())} of {bad.numel()} e4m3 codes differ; first at " + f"{tuple(int(i) for i in bad.nonzero()[0])}" + ) + wbad = weights.view(torch.int32) != want_weights.view(torch.int32) + assert not bool(wbad.any()), f"{what}: {int(wbad.sum())} of {wbad.numel()} weights differ" + + +def _assert_replay_matches_eager(spec, seed) -> None: + """Capture at this batch, replay on new inputs, and require an eager call's bytes.""" + static, fresh, eager = _build(spec, seed), _build(spec, seed + 1), _build(spec, seed + 1) + op(*static) + stream = torch.cuda.Stream() + stream.wait_stream(torch.cuda.current_stream()) + graph = torch.cuda.CUDAGraph() + with torch.cuda.stream(stream), torch.cuda.graph(graph, stream=stream): + op(*static) + torch.cuda.current_stream().wait_stream(stream) + for i in (0, 1, 2, 3, 4, 5): + static[i].copy_(fresh[i]) + static[7].view(torch.uint8).fill_(POISON) + static[8].fill_(float("nan")) + graph.replay() + op(*eager) + torch.cuda.synchronize() + _compare((static[7], static[8]), (eager[7], eager[8]), "the replay against an eager call") + + +def _seed(cell) -> int: + return zlib.crc32(cell.why.encode()) % 2**31 + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 + spec = cell.spec + if spec.get("graph"): + # Before arming: the guard reads the positions back to the host, which a capture + # forbids. + _assert_replay_matches_eager(spec, _seed(cell)) + certifying(op) + for args in _calls(spec, _seed(cell)): + q_before, wq_before = args[0].clone(), args[5].clone() + out = op(*args) + assert out[0] is args[7] and out[1] is args[8], "the op writes the caller's buffers" + assert torch.equal(args[0], q_before) and torch.equal(args[5], wq_before), ( + "q and the head weights must not be written" + ) + where = f"positions {int(args[1].min())}..{int(args[1].max())}" + _compare(out, op.reference(*args), f"the reference at {where}") + _compare( + out, + _replaced(args), + f"ssmax_scaling + fp32 copy + cute_dsl_indexer_q_side_fused at {where}", + ) + + +def test_ssmax_and_rope_both_move_the_codes() -> None: + """The gate sees the scaling and the table: without SSMax, or on the identity table, a + rope layer's call answers differently in most rows.""" + spec = dict(op.CELLS[0].spec, tokens=64, graph=False) + args = _build(spec, 61) + out = op(*args) + unscaled = list(args) + unscaled[2] = torch.zeros_like(args[2]) + unscaled[3] = torch.ones_like(args[3]) + no_rope = list(args) + no_rope[4] = _identity_table(spec["max_pos"]) + for name, wrong in (("no SSMax", unscaled), ("no rope", no_rope)): + ref = op.reference(*wrong) + moved = (out[0].view(torch.uint8) != ref[0].view(torch.uint8)).any(-1).float().mean() + assert float(moved) > 0.5, f"{name}: only {float(moved):.2f} of the rows moved" + + +def test_a_fresh_launch_overwrites_every_row() -> None: + spec = dict(op.CELLS[0].spec, tokens=5, graph=False) + first, second = _build(spec, 31), _build(spec, 32) + op(*first) + op(*second[:7], first[7], first[8]) + _compare((first[7], first[8]), op.reference(*second), "the second call's inputs") + + +@pytest.mark.parametrize( + "case, match", + [ + ("int64 positions", "positions must be int32"), + ("head weights with a strided head dim", "wq must be"), + ("bf16 table", "cos_sin must be fp32"), + ("s of 16 heads", "s must be"), + ("a position past the table", "rope table"), + ], +) +def test_guard_refuses(case, match) -> None: + spec = dict(op.CELLS[0].spec, tokens=8, graph=False) + args = _build(spec, 36) + if case == "int64 positions": + args[1] = args[1].long() + elif case == "head weights with a strided head dim": + args[5] = torch.zeros(8, HEADS, 2, device="cuda")[..., 0] + elif case == "bf16 table": + args[4] = args[4].bfloat16() + elif case == "s of 16 heads": + args[2] = args[2][:16] + else: + args[1][2] = args[4].shape[0] + with validating(op): + with pytest.raises(ValueError, match=match): + op(*args) + + +#: The directory of tensors captured from a served decode step. +_CAPTURE_ENV = "TRTLLM_MODELING_V2_FUSE_CAPTURE" + + +def _assert_old_equals_new(args, what: str) -> None: + """The fused call into poisoned buffers against the three calls it replaces.""" + q, pos, s, b, table, wq, scale, q_fp8, weights = args + new_fp8 = torch.full_like(q_fp8.view(torch.uint8), POISON).view(torch.float8_e4m3fn) + new_w = torch.full_like(weights, float("nan")) + op(q, pos, s, b, table, wq, scale, new_fp8, new_w) + old_fp8, old_w = _replaced(args) + assert torch.equal(new_fp8.view(torch.uint8), old_fp8.view(torch.uint8)), f"{what}: codes" + assert torch.equal(new_w.view(torch.int32), old_w.view(torch.int32)), f"{what}: weights" + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_bit_equal_to_the_unfused_composition(cell) -> None: + for args in _calls(cell.spec, _seed(cell) + 7): + _assert_old_equals_new(args, cell.why) + + +@pytest.mark.skipif( + not os.environ.get(_CAPTURE_ENV), reason=f"{_CAPTURE_ENV} names no captured decode step" +) +def test_bit_equal_to_the_unfused_composition_on_captured_decode_tensors() -> None: + root = Path(os.environ[_CAPTURE_ENV]) + files = sorted(root.glob("ix_*.pt")) + assert files, f"no ix_*.pt under {root}" + tables, rows = {}, 0 + for f in files: + d = torch.load(f, map_location="cuda") + name = d["table"] + if name not in tables: + tables[name] = torch.load(root / f"table_{name}.pt", map_location="cuda") + tokens = d["q"].shape[0] + # The head weights as the targets pass them: the last 32 columns of [T, 160]. + kw = torch.zeros(tokens, HEAD_DIM + HEADS, dtype=d["wq"].dtype, device="cuda") + kw[:, HEAD_DIM:] = d["wq"] + args = [ + d["q"], d["positions"], d["s"], d["b"], tables[name], kw[:, HEAD_DIM:], + d["weight_scale"], + torch.empty(tokens, HEADS, HEAD_DIM, dtype=torch.float8_e4m3fn, device="cuda"), + torch.empty(tokens, HEADS, device="cuda"), + ] # fmt: skip + _assert_old_equals_new(args, f.name) + rows += tokens + print(f"\n {len(files)} captured indexer calls, {rows} rows: bit-equal") + + +# ---- every indexer layer of the checkpoint at every position of the 1M config + +#: The NVFP4 checkpoint, read under LLM_MODELS_ROOT for its indexer layers' SSMax s and b and +#: the rope table's inputs. +CHECKPOINTS = ("Mistral-Large-4-1T-A52B-NVFP4", "mistral-large-4-nvfp4") +#: Every position the 1M config reaches (max_seq_len 1048576, the table's rows), in calls of +#: one chunked step's 32768 rows. +LONG_MAX_SEQ_LEN, STEP_ROWS = 1048576, 32768 + + +def _checkpoint_dir() -> Path | None: + root = llm_models_root() + for name in CHECKPOINTS if root is not None else (): + path = Path(root) / name + names = ("params.json", "consolidated.safetensors.index.json") + if all((path / f).is_file() for f in names): + return path + return None + + +@pytest.fixture(scope="module") +def checkpoint_indexer() -> list: + """The NVFP4 checkpoint's indexer layers, each (layer, s, b, table): its SSMax s and b + (`[32]` each, as stored) and the table its call reads -- the 1048576-row `[cos | sin]` + indexer table the targets build (`_rope_tables` over the rope parameters the engine adapts + from params.json and `lerope.alpha` widened to fp32, as the weight table loads it), or the + identity table on a NoPE indexer layer.""" + path = _checkpoint_dir() + if path is None: + pytest.skip(f"none of {CHECKPOINTS} is under LLM_MODELS_ROOT ({llm_models_root()})") + from safetensors import safe_open + + from tensorrt_llm._torch.modeling_v2.models.mistral_large4.mistral_large_4_nvfp4__sm_103__tep import ( # noqa: E501 + modeling as M, + ) + from tensorrt_llm._torch.models.checkpoints.mistral.config_loader import MistralConfigLoader + + cfg = MistralConfigLoader().load_pretrained_config(str(path)) + index = json.loads((path / "consolidated.safetensors.index.json").read_text())["weight_map"] + keys = { + (layer, p): f"layers.{layer}.attention.indexer.ssmax_scaling.{p}" + for layer in range(cfg.num_hidden_layers) + for p in ("s", "b") + if f"layers.{layer}.attention.indexer.ssmax_scaling.{p}" in index + } + keys["alpha"] = "lerope.alpha" + got = {} + for shard in sorted({index[k] for k in keys.values()}): + with safe_open(str(path / shard), framework="pt", device="cpu") as handle: + for at, key in keys.items(): + if index[key] == shard: + got[at] = handle.get_tensor(key) + rope_cfg = SimpleNamespace( + rope_parameters=cfg.rope_parameters, qk_rope_head_dim=cfg.qk_rope_head_dim + ) + tables = M._rope_tables( + rope_cfg, got["alpha"].float(), LONG_MAX_SEQ_LEN, torch.bfloat16, torch.device("cuda") + ) + layers = sorted({at[0] for at in keys if at != "alpha"}) + return [ + ( + layer, + got[layer, "s"].cuda(), + got[layer, "b"].cuda(), + tables.identity if cfg.indexer_nope_layer_flags[layer] else tables.indexer, + ) + for layer in layers + ] + + +def test_every_layer_and_position_bit_equal_to_the_unfused_composition(checkpoint_indexer): + """The NVFP4 checkpoint's own indexer layers (their s and b, the 1048576-row table they + read, the identity table on a NoPE indexer layer) at every position 0..1048575 of the 1M + config: the fused call's codes and weights are bitwise the composition it replaces, + `ssmax_scaling`, the head weights' fp32 copy and `cute_dsl_indexer_q_side_fused` + (`torch.equal` on the bit patterns). Each call is one chunked step's 32768 consecutive + positions, half the rows normal queries and half drawn over every bf16 binade, the head + weights the last 32 columns of a bf16 `[T, 160]` buffer, the same inputs at every layer.""" + assert len(checkpoint_indexer) == 44, len(checkpoint_indexer) + calls = 0 + for k in range(LONG_MAX_SEQ_LEN // STEP_ROWS): + g = torch.Generator(device="cuda").manual_seed(1 << 20 | k) + half = STEP_ROWS // 2 + q = torch.cat( + [ + _queries((half, HEADS, HEAD_DIM), "normal", g), + _queries((STEP_ROWS - half, HEADS, HEAD_DIM), "wide", g), + ] + ) + kw = torch.randn(STEP_ROWS, HEAD_DIM + HEADS, generator=g, device="cuda").bfloat16() + start = k * STEP_ROWS + positions = torch.arange(start, start + STEP_ROWS, dtype=torch.int32, device="cuda") + for layer, s, b, table in checkpoint_indexer: + args = [ + q, positions, s, b, table, kw[:, HEAD_DIM:], WEIGHT_SCALE, + torch.empty(STEP_ROWS, HEADS, HEAD_DIM, dtype=torch.float8_e4m3fn, device="cuda"), + torch.empty(STEP_ROWS, HEADS, device="cuda"), + ] # fmt: skip + _assert_old_equals_new(args, f"layer {layer} at positions {start}..") + calls += 1 + print( + f"\n {len(checkpoint_indexer)} indexer layers x positions 0..{LONG_MAX_SEQ_LEN - 1}, " + f"{calls} calls: bit-equal" + ) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_ssmax_scaling.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_ssmax_scaling.py new file mode 100644 index 000000000000..e65818aca9e2 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_ssmax_scaling.py @@ -0,0 +1,888 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the ssmax_scaling entry. + +Every cell is gated twice against the entry: + +* bit for bit against its `reference`. The specified computation is a chain of bf16 + roundings and the reference reproduces each link exactly (its docstring says why), so + there is no band to derive: an implementation of the chain is bit-identical to it, and + anything else is not. The one link that is not a correctly rounded operation is the + device's fp32 log (CUDA logf, documented to 1 ulp), where the reference takes the + correctly rounded one. Measured on sm_103: the two differ in the last fp32 bit at 935 + of the NVFP4 target's 32768 positions (266 of the FP8 target's 9216) and round to the same + bf16 at every one of them, as they must: the exact log(pos + 1) of every position + 1..32767 lies at least 1.748 fp32 ulp from a bf16 rounding midpoint. The context cells + drive every position, so a change on either side fails them. Above 32767 there is no + such margin (38 positions below 2^20 lie within 1 fp32 ulp of a midpoint, measured), so + the six sweep cells (three at 16 MLA heads or the indexer's 32, three at tep8's 8 MLA + heads) and dep4's two 32-chunk cells drive every position 0..1048575 and hold the two + logs' bf16 equal at each directly, beside the gates (`_position_sweep`, `_chunk_sweep`). +* against the closed form ``q * (s * log(pos + 1) + b)`` in fp64, within the chain's + worst-case error derived at `_closed_form_ratio`. Bitwise agreement between the op and a + reference written beside it cannot see a mistake the two share (``log(pos)`` for + ``log(pos + 1)``, another rank's heads); this gate does. + +`test_gates_reject_wrong_variants` measures what each gate rejects. +""" + +import math +import zlib + +import pytest +import torch + +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ssmax_scaling import ( + FP8_MAX_SEQ_LEN, + LONG_MAX_SEQ_LEN, + MAX_NUM_TOKENS, + MAX_SEQ_LEN, + MLA_HEADS, + MLA_PARAM_HEADS, + MLA_TEP8_HEADS, + NVFP4_MLA_LAYER0_RANK1, + NVFP4_MLA_LAYER0_TEP8_RANK2, + NVFP4_MLA_LAYER0_TEP8_RANK3, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ssmax_scaling import ( + ssmax_scaling as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402,F401 + +assert torch.cuda.is_available(), "ssmax_scaling requires a CUDA device" + +#: The extremes of s and b over both Mistral Large 4 checkpoints, every layer that carries them +#: (the NVFP4 checkpoint's read on disk as bf16, the FP8 checkpoint's rounded outward). +_CHECKPOINT_RANGES = { + "mla": dict(s=(-0.078125, 0.574219), b=(0.902, 1.578)), + "indexer": dict(s=(-0.0486, 0.0703), b=(0.973, 1.055)), +} + +#: One rounding's relative error: to fp32 (torch's bf16 arithmetic computes in fp32), +#: then to bf16, whose unit roundoff is 2^-8. +_U = (1 + 2.0**-24) * (1 + 2.0**-8) - 1 +#: The log's: CUDA logf's documented 1 ulp in fp32, then the bf16 rounding. +_U_LOG = (1 + 2.0**-23) * (1 + 2.0**-8) - 1 +#: The last rounding's absolute error where q * scale is subnormal: half the bf16 +#: subnormal spacing, after half the fp32 one. +_SUBNORMAL = 2.0**-134 + 2.0**-150 +#: The smallest magnitude bf16 rounds to infinity: the largest finite bf16, +#: (2 - 2^-7) 2^127, plus half its ulp. +_OVERFLOW = 2.0**128 * (1 - 2.0**-9) + + +def _seed(text: str) -> int: + return zlib.crc32(text.encode()) + + +def _queries(shape, kind: str, dtype: torch.dtype, g: torch.Generator) -> torch.Tensor: + if kind == "normal": + return (4 * torch.randn(shape, device="cuda", generator=g)).to(dtype) + # Every bf16 binade from the smallest subnormal (2^-133) up, both signs; exponents stop + # at 126 so that q itself stays finite. + exponent = torch.randint(-133, 127, shape, device="cuda", generator=g).float() + mantissa = 1 + torch.rand(shape, device="cuda", generator=g) + sign = 2 * torch.randint(0, 2, shape, device="cuda", generator=g).float() - 1 + return (sign * mantissa * torch.exp2(exponent)).to(dtype) + + +def _positions(spec, g: torch.Generator) -> torch.Tensor: + dtype = spec["positions_dtype"] + cached = spec.get("cached", (0,) * len(spec["contexts"])) + assert len(cached) == len(spec["contexts"]), (cached, spec["contexts"]) + parts = [ + torch.arange(c, c + n, dtype=dtype, device="cuda") for n, c in zip(spec["contexts"], cached) + ] + assert all(c + n <= spec["max_seq_len"] for n, c in zip(spec["contexts"], cached)) + if spec["decode_rows"]: + top = spec["max_seq_len"] + rows = torch.randint(1, top, (spec["decode_rows"],), device="cuda", generator=g) + rows[0] = top - 1 + parts.append(rows.to(dtype)) + positions = torch.cat(parts) + assert positions.numel() == spec["tokens"], (positions.numel(), spec["tokens"]) + return positions + + +def _build(spec, seed: int): + """(q, positions, s, b) for one spec. s and b are drawn for all `param_heads` and + sliced to rank `rank`'s `heads`, so they are views of the full parameter; with + `corners`, the slice's heads (4 * rank) % heads .. + 3 hold the ranges' four (s, b) + corners (4 * rank .. 4 * rank + 3 at 16 heads, where no two ranks' coincide), so another + rank's slice differs from this one at every head. `slice_values` (s, b) then replaces the + whole slice with those values (a checkpoint's own heads).""" + g = torch.Generator(device="cuda").manual_seed(seed) + shape = (spec["tokens"], spec["heads"], spec["head_dim"]) + q = _queries(shape, spec["q"], spec["dtype"], g) + positions = _positions(spec, g) + full = [ + torch.empty(spec["param_heads"], device="cuda", dtype=torch.float32) + .uniform_(*spec[key], generator=g) + .to(spec["dtype"]) + for key in ("s_range", "b_range") + ] + lo = spec["rank"] * spec["heads"] + if spec["corners"]: + at = lo + (4 * spec["rank"]) % spec["heads"] + corners = [(s, b) for s in spec["s_range"] for b in spec["b_range"]] + for i, values in enumerate(zip(*corners)): + full[i][at : at + 4] = torch.tensor(values, device="cuda").to(spec["dtype"]) + for i, values in enumerate(spec.get("slice_values", ())): + assert len(values) == spec["heads"], (len(values), spec["heads"]) + exact = torch.tensor(values, dtype=torch.float64) + assert torch.equal(exact.to(spec["dtype"]).double(), exact), "values must be bf16" + full[i][lo : lo + spec["heads"]] = exact.to(device="cuda", dtype=spec["dtype"]) + s, b = (t[lo : lo + spec["heads"]] for t in full) + return q, positions, s, b + + +def _spec_of(site: str, context: int = MAX_SEQ_LEN, heads: int | None = None, **override): + """The certified spec of `site`'s first `context`-row context cell at position 0 (the + first at `heads` heads when given), with `override` applied.""" + base = next( + c.spec + for c in op.CELLS + if c.spec["site"] == site + and c.spec["contexts"] == (context,) + and c.spec["q"] == "normal" + and heads in (None, c.spec["heads"]) + and not any(c.spec.get("cached", ())) + and "position_sweep" not in c.spec + and "chunks" not in c.spec + ) + return dict(base, **override) + + +def _bitwise(out: torch.Tensor, ref: torch.Tensor, what: str = "the reference") -> None: + assert out.dtype == ref.dtype == torch.bfloat16, (out.dtype, ref.dtype) + assert out.shape == ref.shape, (out.shape, ref.shape) + differ = out.view(torch.int16) != ref.view(torch.int16) + n = int(differ.sum()) + if n: + at = tuple(int(i) for i in differ.nonzero()[0]) + raise AssertionError( + f"{n} of {out.numel()} elements differ from {what} bit for bit; " + f"first at {at}: {out[at].item()!r} vs {ref[at].item()!r}" + ) + + +def _closed_form_ratio(out, q, positions, s, b) -> float: + """Worst |out - q * (s * log(pos + 1) + b)| as a fraction of the chain's error bound. + + With L = log(pos + 1) and C = L s + b, and each rounding's relative error at most u + (`_U`; `_U_LOG` for the log): + + lp = rnd(L) = L (1 + a) |a| <= u_log + m = rnd(lp s) |m - L s| <= |L s| e, e = (1 + u_log)(1 + u) - 1 + C' = rnd(m + b) |C' - C| <= |L s| e + (|C| + |L s| e) u =: dC + out = rnd(q C') |out - q C| <= |q| dC (1 + u) + |q C| u + + plus `_SUBNORMAL` where q C' is subnormal. An infinite output passes only where q C' + can reach `_OVERFLOW`, with the closed form's sign; a nan fails. Inputs are finite. + """ + ls = torch.log(positions.double() + 1)[:, None] * s.double()[None, :] + c = ls + b.double()[None, :] + e = (1 + _U_LOG) * (1 + _U) - 1 + dc = (ls.abs() * e + (c.abs() + ls.abs() * e) * _U)[..., None] + qd = q.double() + exact = qd * c[..., None] + bound = qd.abs() * dc * (1 + _U) + exact.abs() * _U + _SUBNORMAL + o = out.double() + if bool(torch.isnan(o).any()): + return math.inf + inf = torch.isinf(o) + if bool(inf.any()): + reach = (exact.abs() + qd.abs() * dc)[inf] * (1 + 2.0**-24) + if not bool((reach >= _OVERFLOW).all()): + return math.inf + if not torch.equal(torch.sign(o[inf]), torch.sign(exact[inf])): + return math.inf + finite = ~inf + return ((o - exact).abs()[finite] / bound[finite]).max().item() + + +def _ulps_from_closed_form(out, q, positions, s, b) -> float: + """Worst |out - q * (s * log(pos + 1) + b)| in bf16 ulps of the closed form (finite inputs).""" + c = torch.log(positions.double() + 1)[:, None] * s.double()[None, :] + b.double()[None, :] + exact = q.double() * c[..., None] + ulp = torch.exp2(torch.floor(torch.log2(exact.abs().clamp_min(2.0**-126))) - 7) + return ((out.double() - exact).abs() / ulp).max().item() + + +def _sweep_counts(spec) -> list[int]: + """The step sizes a sweep cell names (its `sweep_rows` ranges, and 2^k - 1, 2^k and + 2^k + 1 up to `tokens` with `sweep_pow2`), largest first.""" + counts = set() + for lo, hi, step in spec["sweep_rows"]: + counts.update(range(lo, hi + 1, step)) + if spec.get("sweep_pow2"): + k = 1 + while k <= spec["tokens"]: + counts.update(c for c in (k - 1, k, k + 1) if 1 <= c <= spec["tokens"]) + k *= 2 + return sorted(counts, reverse=True) + + +def _prefix_sweep(out, q, positions, s, b, counts) -> None: + """Each step size T in `counts` as its own call, bitwise the leading T rows of `out`. + + Rows are independent, so what can move with T is each launch's grid and its last + block's tail (the entry's note lists the launches). Compared as int64 words with the + verdicts kept on the device and read once, largest T first, so the caching allocator + reuses the largest call's blocks; the smallest failing T is re-run and counted element + by element. + """ + words = out.view(torch.int64) + differ = torch.zeros(len(counts), dtype=torch.bool, device=out.device) + for i, t in enumerate(counts): + differ[i] = torch.ne(op(q[:t], positions[:t], s, b).view(torch.int64), words[:t]).any() + bad = [counts[i] for i in differ.nonzero().flatten().tolist()] + if bad: + t = bad[-1] + _bitwise( + op(q[:t], positions[:t], s, b), + out[:t], + f"the leading {t} rows of the {out.shape[0]}-row call ({len(bad)} step sizes differ)", + ) + + +def _capture(q, positions, s, b): + side = torch.cuda.Stream() + side.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(side): + for _ in range(2): + op(q, positions, s, b) + torch.cuda.current_stream().wait_stream(side) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + out = op(q, positions, s, b) + return graph, out + + +def _log_in_bf16(positions: torch.Tensor, where: str) -> int: + """The device's fp32 log(pos + 1), the op's own first steps, against the correctly rounded + fp32 log, both rounded to bf16 (the step the chain keeps): bitwise there, and the number + of positions where the two differ in fp32.""" + device_log = torch.log(positions.to(torch.float32) + 1) + rounded_log = torch.log(positions.double() + 1).to(torch.float32) + _bitwise( + device_log.to(torch.bfloat16), + rounded_log.to(torch.bfloat16), + f"the correctly rounded log in bf16 at {where}", + ) + return int((device_log != rounded_log).sum()) + + +def _position_sweep(spec, seed: int, s: torch.Tensor, b: torch.Tensor) -> tuple[float, str]: + """`position_sweep` (lo, hi): every position lo .. hi - 1, `tokens` rows per call (call k + at positions lo + k * tokens onward), each call on a fresh q and gated as a cell is: + bitwise to the reference, the closed form within the chain's bound, its inputs untouched. + + Beside the gates it holds the device's log of every position to the correctly rounded + one in bf16 (`_log_in_bf16`): the two differ in fp32 at some positions, and above 32767 + no margin keeps them together in bf16 (the module docstring), so it is measured at + each. Returns the worst closed-form ratio and a summary line. + """ + lo, hi = spec["position_sweep"] + rows = spec["tokens"] + assert (hi - lo) % rows == 0 and hi <= spec["max_seq_len"], (lo, hi, rows) + shape = (rows, spec["heads"], spec["head_dim"]) + worst, fp32_differ, zero = 0.0, 0, 0 + for k in range((hi - lo) // rows): + start = lo + k * rows + where = f"positions {start}..{start + rows - 1}" + g = torch.Generator(device="cuda").manual_seed(seed + 1 + k) + q = _queries(shape, spec["q"], spec["dtype"], g) + positions = torch.arange(start, start + rows, dtype=spec["positions_dtype"], device="cuda") + q_before, positions_before = q.clone(), positions.clone() + out = op(q, positions, s, b) + assert torch.equal(q, q_before) and torch.equal(positions, positions_before), ( + "the entry must not mutate its inputs" + ) + _bitwise(out, op.reference(q, positions, s, b), f"the reference at {where}") + worst = max(worst, _closed_form_ratio(out, q, positions, s, b)) + fp32_differ += _log_in_bf16(positions, where) + zero += int((out == 0).all(-1).sum()) + return worst, ( + f"{(hi - lo) // rows} calls over positions {lo}..{hi - 1}; the device log differs from " + f"the correctly rounded one in fp32 at {fp32_differ} positions, in bf16 at none; " + f"{zero} (position, head) queries zeroed" + ) + + +def _chunk_sweep(spec, seed: int, s: torch.Tensor, b: torch.Tensor) -> tuple[float, str]: + """`chunks`: a whole prompt of chunks * tokens positions as `chunks` calls of `tokens` rows, + chunk k at positions k * tokens onward (the chunk after a cached prefix of k * tokens), + each on a fresh q and gated as a cell is: bitwise to the reference, the closed form within + the chain's bound, its inputs untouched. + + Beside the gates it holds the device's log of every position to the correctly rounded + one in bf16 (`_log_in_bf16`): the two differ in fp32 at some positions, and above 32767 + no margin keeps them together in bf16 (the module docstring), so it is measured at + each. Returns the worst closed-form ratio and a summary line. + """ + rows = spec["tokens"] + top = spec["chunks"] * rows + assert top <= spec["max_seq_len"], (spec["chunks"], rows, spec["max_seq_len"]) + shape = (rows, spec["heads"], spec["head_dim"]) + worst, fp32_differ, zero = 0.0, 0, 0 + for k in range(spec["chunks"]): + start = k * rows + where = f"positions {start}..{start + rows - 1}" + g = torch.Generator(device="cuda").manual_seed(seed + 1 + k) + q = _queries(shape, spec["q"], spec["dtype"], g) + positions = torch.arange(start, start + rows, dtype=spec["positions_dtype"], device="cuda") + q_before, positions_before = q.clone(), positions.clone() + out = op(q, positions, s, b) + assert torch.equal(q, q_before) and torch.equal(positions, positions_before), ( + "the entry must not mutate its inputs" + ) + _bitwise(out, op.reference(q, positions, s, b), f"the reference at {where}") + worst = max(worst, _closed_form_ratio(out, q, positions, s, b)) + fp32_differ += _log_in_bf16(positions, where) + zero += int((out == 0).sum()) + return worst, ( + f"{spec['chunks']} chunk calls over positions 0..{top - 1}; {zero} zero; the device log " + f"differs from the correctly rounded one in fp32 at {fp32_differ} positions, in bf16 at " + "none" + ) + + +@pytest.mark.parametrize( + "cell", + [pytest.param(c, marks=pytest.mark.large) if c.spec.get("large") else c for c in op.CELLS], + ids=[c.why[:44] for c in op.CELLS], +) +def test_certified_cells(cell, certifying): # noqa: F811 + certifying(op) + spec, seed = cell.spec, _seed(cell.why) + q, positions, s, b = _build(spec, seed) + s_before, b_before = s.clone(), b.clone() + if "position_sweep" in spec: + ratio, swept = _position_sweep(spec, seed, s, b) + for now, then in ((s, s_before), (b, b_before)): + assert torch.equal(now, then), "the entry must not mutate its inputs" + assert ratio <= 1.0, f"{ratio:.3f} of the chain's error bound from the closed form" + print(f"\n {cell.why}: bitwise; closed form at {ratio:.3f} of the bound; {swept}") + return + if "chunks" in spec: + ratio, swept = _chunk_sweep(spec, seed, s, b) + for now, then in ((s, s_before), (b, b_before)): + assert torch.equal(now, then), "the entry must not mutate its inputs" + assert ratio <= 1.0, f"{ratio:.3f} of the chain's error bound from the closed form" + print(f"\n {cell.why}: bitwise; closed form at {ratio:.3f} of the bound; {swept}") + return + if spec["graph"]: + graph, out = _capture(q, positions, s, b) + fresh_q, fresh_positions, _, _ = _build(spec, seed + 1) + q.copy_(fresh_q) + positions.copy_(fresh_positions) + q_before, positions_before = q.clone(), positions.clone() + graph.replay() + torch.cuda.synchronize() + _bitwise(out, op(q, positions, s, b), "the eager call on the replayed inputs") + else: + q_before, positions_before = q.clone(), positions.clone() + out = op(q, positions, s, b) + + assert out.shape == q.shape and out.dtype == q.dtype and out.device == q.device + assert out.data_ptr() != q.data_ptr(), "the entry is out-of-place" + for now, then in ((q, q_before), (positions, positions_before), (s, s_before), (b, b_before)): + assert torch.equal(now, then), "the entry must not mutate its inputs" + _bitwise(out, op.reference(q, positions, s, b)) + ratio = _closed_form_ratio(out, q, positions, s, b) + assert ratio <= 1.0, f"{ratio:.3f} of the chain's error bound from the closed form" + swept = "" + if "sweep_rows" in spec: + counts = _sweep_counts(spec) + _prefix_sweep(out, q, positions, s, b, counts) + swept = f"; {len(counts)} step sizes bitwise the leading rows" + print( + f"\n {cell.why}: bitwise; closed form at {ratio:.3f} of the bound; " + f"{int(torch.isinf(out).sum())} overflowed, {int((out == 0).sum())} zero{swept}" + ) + + +def test_the_sweep_sees_one_flipped_bit() -> None: + """The step-size sweep's control: one flipped bit in the full call's last head, last lane, + fails exactly the step sizes whose leading rows hold it.""" + q, positions, s, b = _build( + _spec_of("mla", heads=MLA_PARAM_HEADS, tokens=4096, contexts=(4096,)), 71 + ) + out = op(q, positions, s, b) + _prefix_sweep(out, q, positions, s, b, [4096, 3001, 3000, 7]) + out.view(torch.int16)[3000, -1, -1] ^= 1 + with pytest.raises(AssertionError, match=r"leading 3001 rows .*\(2 step sizes differ\)"): + _prefix_sweep(out, q, positions, s, b, [4096, 3001, 3000, 7]) + + +def test_the_log_check_sees_the_first_parting_position() -> None: + """The log check's control, for the position sweep and dep4's chunk sweep alike: past every + envelope, at position 1695942, the device's log and the correctly rounded one round to + different bf16 (measured): the check fails there and passes at its neighbours.""" + positions = torch.tensor([1695941, 1695942, 1695943], dtype=torch.int32, device="cuda") + _log_in_bf16(positions[[0, 2]], "the neighbours") + with pytest.raises(AssertionError, match=r"1 of 1 elements differ .* first at \(0,\)"): + _log_in_bf16(positions[1:2], "position 1695942") + + +#: What the 16-head checkpoint-slice cell's two crossing heads compute, measured on sm_103 +#: over positions 0..1048575: (local head, first and last position of the run where the +#: chain's factor is exactly 0, the factor at position 1048575). +_CROSSINGS = ((4, 517233, 550591, -0.046875), (10, 108418, 115409, -0.1796875)) + + +def test_checkpoint_heads_cross_zero() -> None: + """The 16-head checkpoint-slice cell's crossings, as the op computes them: with q = 1 the + op returns its bf16 scale. On MLA layer 0's rank-1 slice, heads 4 and 10 hold a factor of + exactly 0 over one run of positions around their closed-form zero and a negative factor + after it; no head's factor takes the opposite sign to the closed form; the other 14 + heads stay positive at every position below 2^20.""" + spec = next( + c.spec for c in op.CELLS if "slice_values" in c.spec and c.spec["heads"] == MLA_HEADS + ) + assert spec["slice_values"] is NVFP4_MLA_LAYER0_RANK1 and spec["rank"] == 1 + _, _, s, b = _build(dict(spec, tokens=1, contexts=(1,)), 81) + positions = torch.arange(LONG_MAX_SEQ_LEN, dtype=torch.int32, device="cuda") + ones = torch.ones(LONG_MAX_SEQ_LEN, spec["heads"], 1, dtype=spec["dtype"], device="cuda") + scale = op(ones, positions, s, b)[:, :, 0] + _bitwise(scale, op.reference(ones, positions, s, b)[:, :, 0]) + closed = torch.log(positions.double() + 1)[:, None] * s.double() + b.double() + opposite = (torch.sign(scale.double()) != torch.sign(closed)) & (scale != 0) + assert not bool(opposite.any()), int(opposite.sum()) + crossing = [h for h, *_ in _CROSSINGS] + others = [h for h in range(spec["heads"]) if h not in crossing] + assert bool((scale[:, others] > 0).all()), "only heads 4 and 10 leave the positive range" + for h, first, last, at_end in _CROSSINGS: + zero = (scale[:, h] == 0).nonzero().flatten() + assert (int(zero[0]), int(zero[-1]), zero.numel()) == (first, last, last - first + 1) + cross = math.exp(-float(b[h]) / float(s[h])) - 1 + assert first <= cross <= last, (h, cross) + assert bool((scale[:first, h] > 0).all()) and bool((scale[last + 1 :, h] < 0).all()) + assert scale[-1, h].item() == at_end, (h, scale[-1, h].item()) + print( + f"\n local head {h} (s {float(s[h])}, b {float(b[h])}): factor 0 at positions " + f"{first}..{last} around the closed-form zero {cross:.1f}, {at_end} at 1048575" + ) + + +#: What the tep8 checkpoint-slice cells' crossing heads compute, measured on sm_103 over +#: positions 0..1048575: rank -> (local head, first and last position of the run where the +#: chain's factor is exactly 0, the factor at position 1048575). +_TEP8_CROSSINGS = {2: (4, 517233, 550591, -0.046875), 3: (2, 108418, 115409, -0.1796875)} + + +@pytest.mark.parametrize("rank", sorted(_TEP8_CROSSINGS)) +def test_tep8_checkpoint_heads_cross_zero(rank: int) -> None: + """The tep8 checkpoint-slice cells' crossings, as the op computes them: with q = 1 the op + returns its bf16 scale. On MLA layer 0's rank-2 slice (heads 16..23) local head 4 (head + 20), and on its rank-3 slice (heads 24..31) local head 2 (head 26), hold a factor of + exactly 0 over one run of positions around their closed-form zero and a negative factor + after it; no head's factor takes the opposite sign to the closed form; the slice's other + 7 heads stay positive at every position below 2^20.""" + spec = next( + c.spec + for c in op.CELLS + if "slice_values" in c.spec and c.spec["heads"] == MLA_TEP8_HEADS and c.spec["rank"] == rank + ) + values = {2: NVFP4_MLA_LAYER0_TEP8_RANK2, 3: NVFP4_MLA_LAYER0_TEP8_RANK3}[rank] + assert spec["slice_values"] is values + _, _, s, b = _build(dict(spec, tokens=1, contexts=(1,)), 81) + positions = torch.arange(LONG_MAX_SEQ_LEN, dtype=torch.int32, device="cuda") + ones = torch.ones(LONG_MAX_SEQ_LEN, spec["heads"], 1, dtype=spec["dtype"], device="cuda") + scale = op(ones, positions, s, b)[:, :, 0] + _bitwise(scale, op.reference(ones, positions, s, b)[:, :, 0]) + closed = torch.log(positions.double() + 1)[:, None] * s.double() + b.double() + opposite = (torch.sign(scale.double()) != torch.sign(closed)) & (scale != 0) + assert not bool(opposite.any()), int(opposite.sum()) + h, first, last, at_end = _TEP8_CROSSINGS[rank] + others = [i for i in range(spec["heads"]) if i != h] + assert bool((scale[:, others] > 0).all()), f"only local head {h} leaves the positive range" + zero = (scale[:, h] == 0).nonzero().flatten() + assert (int(zero[0]), int(zero[-1]), zero.numel()) == (first, last, last - first + 1) + cross = math.exp(-float(b[h]) / float(s[h])) - 1 + assert first <= cross <= last, (h, cross) + assert bool((scale[:first, h] > 0).all()) and bool((scale[last + 1 :, h] < 0).all()) + assert scale[-1, h].item() == at_end, (h, scale[-1, h].item()) + print( + f"\n rank {rank} local head {h} (head {rank * spec['heads'] + h}; s {float(s[h])}, " + f"b {float(b[h])}): factor 0 at positions {first}..{last} around the closed-form zero " + f"{cross:.1f}, {at_end} at 1048575" + ) + + +def test_tep8_gates_at_the_1m_last_chunk() -> None: + """What each gate rejects at far positions on a tep8 rank's 8 heads: the MLA's 1M last + chunk on rank 1, positions 1015808..1048575, where a bf16 step of log(pos + 1) is 0.0625. + + The rounding variants still fail the bitwise gate, each moving at least 0.5% of the + elements, and stay inside the closed form's bound; s and b swapped and rank 2's heads on + rank 1 fail the closed form. A position off by one, either way, moves the chain's log at + one row of the chunk only, the one whose log(pos + 1) and log(pos + 2) round to different + bf16 (position 1028640), so the bitwise gate sees it at that row and at no other: an + off-by-one in the positions a caller passes at far positions is not this entry's to see. + """ + spec = next( + c.spec + for c in op.CELLS + if c.spec["site"] == "mla" + and c.spec["heads"] == MLA_TEP8_HEADS + and c.spec.get("cached") == (LONG_MAX_SEQ_LEN - MAX_NUM_TOKENS,) + and c.spec["q"] == "normal" + ) + assert spec["rank"] == 1 + q, positions, s, b = _build(spec, 37) + _, _, s_other, b_other = _build(dict(spec, rank=2), 37) + assert int(positions[0]) == 1015808 and int(positions[-1]) == 1048575 + ref = op.reference(q, positions, s, b) + bf16 = q.dtype + lp32 = torch.log(positions.to(torch.float32) + 1) + lp = lp32.to(bf16) + rounding = { + "fp32 scale, no intermediate rounding": ( + q.float() * (lp32[:, None] * s.float() + b.float())[..., None] + ).to(bf16), + "log not rounded to bf16": q * ((lp32[:, None] * s.float()).to(bf16) + b)[..., None], + "scale as one fused multiply-add": q + * torch.addcmul(b.float()[None, :], lp.float()[:, None], s.float()[None, :]).to(bf16)[ + ..., None + ], + } + gross = { + "s and b swapped": op(q, positions, b, s), + "rank 2's heads on rank 1": op(q, positions, s_other, b_other), + } + off_by_one = { + "positions off by one": (op(q, positions + 1, s, b), 1028640), + "log(pos) for log(pos + 1)": (op(q, positions - 1, s, b), 1028641), + } + numel = ref.numel() + for name, wrong in {**rounding, **gross, **{k: v for k, (v, _) in off_by_one.items()}}.items(): + differ = wrong.view(torch.int16) != ref.view(torch.int16) + count = int(differ.sum()) + ratio = _closed_form_ratio(wrong, q, positions, s, b) + rows = positions[differ.flatten(1).any(1)].tolist() + print( + f"\n {name:38s} differs in {count:>9d} of {numel} ({100.0 * count / numel:7.3f}%), " + f"{len(rows)} rows; closed form at {ratio:.4g} of the bound" + ) + with pytest.raises(AssertionError): + _bitwise(wrong, ref) + if name in rounding: + assert count >= numel // 200, (name, count) + assert ratio <= 1.0, (name, ratio) + elif name in gross: + assert ratio > 1.0, (name, ratio) + else: + assert rows == [off_by_one[name][1]], (name, rows) + + +def test_draw_ranges_hold_the_checkpoints_parameters() -> None: + """Every checkpoint-shaped cell draws s and b over a range holding both checkpoints' own.""" + checked = 0 + for cell in op.CELLS: + spec = cell.spec + if spec["q"] != "normal": + continue + for i, key in enumerate(("s", "b")): + lo, hi = spec[f"{key}_range"] + want_lo, want_hi = _CHECKPOINT_RANGES[spec["site"]][key] + assert lo <= want_lo and want_hi <= hi, (cell.why, key, (lo, hi)) + for value in spec.get("slice_values", ((), ()))[i]: + assert want_lo <= value <= want_hi, (cell.why, key, value) + checked += 1 + assert checked == len(op.CELLS) - 5, "every cell but the five stress cells is checkpoint-shaped" + + +@pytest.mark.parametrize( + "heads, context", + [ + (MLA_HEADS, FP8_MAX_SEQ_LEN), + (MLA_HEADS, MAX_SEQ_LEN), + (MLA_PARAM_HEADS, MAX_SEQ_LEN), + (MLA_TEP8_HEADS, MAX_SEQ_LEN), + ], +) +def test_gates_reject_wrong_variants(heads: int, context: int) -> None: + """Every plausible wrong chain fails the bitwise gate; the gross ones fail the closed form. + + The rounding variants, which only the bitwise gate can see, move at least 0.5% of the + elements at every width. A gross variant fails the closed form however few elements it + moves: a position off by one moves fewer at 32768 rows than at 9216 (log(pos) and + log(pos + 1) round to the same bf16 at most large positions), and the closed form still + sees the rows where they do not. With 16 heads (tep4) or 8 (tep8, tp8) the wrong heads are + another rank's slice; with all 64 on the rank (dep4) they are the parameter rotated by 16 + heads, so each head meets another's s and b. + """ + if heads in (MLA_HEADS, MLA_TEP8_HEADS): + spec = _spec_of("mla", context, heads, rank=2) + q, positions, s, b = _build(spec, 31) + _, _, s_other, b_other = _build(dict(spec, rank=3), 31) + wrong_heads = "rank 3's heads on rank 2" + else: + q, positions, s, b = _build(_spec_of("mla", context, heads), 31) + s_other, b_other = s.roll(16), b.roll(16) + wrong_heads = "the 64 heads rotated by 16" + ref = op.reference(q, positions, s, b) + bf16 = q.dtype + lp32 = torch.log(positions.to(torch.float32) + 1) + lp = lp32.to(bf16) + fp32_scale = (q.float() * (lp32[:, None] * s.float() + b.float())[..., None]).to(bf16) + rounding = { + "fp32 scale, no intermediate rounding": fp32_scale, + "log not rounded to bf16": q * ((lp32[:, None] * s.float()).to(bf16) + b)[..., None], + "scale as one fused multiply-add": q + * torch.addcmul(b.float()[None, :], lp.float()[:, None], s.float()[None, :]).to(bf16)[ + ..., None + ], + } + gross = { + "log(pos) for log(pos + 1)": op(q, positions - 1, s, b), + "positions off by one": op(q, positions + 1, s, b), + "s and b swapped": op(q, positions, b, s), + wrong_heads: op(q, positions, s_other, b_other), + } + numel = ref.numel() + for name, wrong in {**rounding, **gross}.items(): + differ = int((wrong.view(torch.int16) != ref.view(torch.int16)).sum()) + ratio = _closed_form_ratio(wrong, q, positions, s, b) + print( + f"\n {name:38s} differs in {differ:>9d} of {numel} ({100.0 * differ / numel:7.3f}%);" + f" closed form at {ratio:.4g} of the bound" + ) + with pytest.raises(AssertionError): + _bitwise(wrong, ref) + if name in gross: + assert ratio > 1.0, (name, ratio) + else: + assert differ >= numel // 200, (name, differ) + assert ratio <= 1.0, (name, ratio) + chain = _ulps_from_closed_form(ref, q, positions, s, b) + single = _ulps_from_closed_form(fp32_scale, q, positions, s, b) + print( + f"\n bf16 ulps from the closed form: the specified chain {chain:.3f}, " + f"an fp32 scale rounded once {single:.3f}" + ) + assert single < chain + + +def test_gates_at_the_1m_last_chunk() -> None: + """What each gate rejects at far positions, with all 64 heads (dep4): the MLA's 1M last + chunk, positions 1015808..1048575, where a bf16 step of log(pos + 1) is 0.0625. + + The rounding variants still fail the bitwise gate, each moving at least 0.5% of the + elements, and stay inside the closed form's bound; s and b swapped and the 64 heads + rotated by 16 fail the closed form. A position off by one, either way, moves the chain's + log at one row of the chunk only, the one whose log(pos + 1) and log(pos + 2) round to + different bf16 (position 1028640, measured), so the bitwise gate sees it at that row and + at no other: an off-by-one in the positions a caller passes at far positions is not this + entry's to see. + """ + spec = next( + c.spec + for c in op.CELLS + if c.spec["site"] == "mla" + and c.spec["heads"] == MLA_PARAM_HEADS + and c.spec.get("cached") == (LONG_MAX_SEQ_LEN - MAX_NUM_TOKENS,) + and c.spec["contexts"] == (MAX_NUM_TOKENS,) + and c.spec["q"] == "normal" + ) + q, positions, s, b = _build(spec, 37) + assert int(positions[0]) == 1015808 and int(positions[-1]) == 1048575 + ref = op.reference(q, positions, s, b) + bf16 = q.dtype + lp32 = torch.log(positions.to(torch.float32) + 1) + lp = lp32.to(bf16) + rounding = { + "fp32 scale, no intermediate rounding": ( + q.float() * (lp32[:, None] * s.float() + b.float())[..., None] + ).to(bf16), + "log not rounded to bf16": q * ((lp32[:, None] * s.float()).to(bf16) + b)[..., None], + "scale as one fused multiply-add": q + * torch.addcmul(b.float()[None, :], lp.float()[:, None], s.float()[None, :]).to(bf16)[ + ..., None + ], + } + gross = { + "s and b swapped": op(q, positions, b, s), + "the 64 heads rotated by 16": op(q, positions, s.roll(16), b.roll(16)), + } + off_by_one = { + "positions off by one": (op(q, positions + 1, s, b), 1028640), + "log(pos) for log(pos + 1)": (op(q, positions - 1, s, b), 1028641), + } + numel = ref.numel() + for name, wrong in {**rounding, **gross, **{k: v for k, (v, _) in off_by_one.items()}}.items(): + differ = wrong.view(torch.int16) != ref.view(torch.int16) + count = int(differ.sum()) + ratio = _closed_form_ratio(wrong, q, positions, s, b) + rows = positions[differ.flatten(1).any(1)].tolist() + print( + f"\n {name:38s} differs in {count:>9d} of {numel} ({100.0 * count / numel:7.3f}%), " + f"{len(rows)} rows; closed form at {ratio:.4g} of the bound" + ) + with pytest.raises(AssertionError): + _bitwise(wrong, ref) + if name in rounding: + assert count >= numel // 200, (name, count) + assert ratio <= 1.0, (name, ratio) + elif name in gross: + assert ratio > 1.0, (name, ratio) + else: + assert rows == [off_by_one[name][1]], (name, rows) + + +def _guard_inputs(): + return _build( + _spec_of("mla", FP8_MAX_SEQ_LEN, rank=0, tokens=16, contexts=(), decode_rows=16), 41 + ) + + +def _rowwise_scale(q, positions, s, b) -> torch.Tensor: + """The chain with row t paired with head t's terms: what [H, 1] s and b compute at T == H.""" + lp = torch.log(positions.to(torch.float32) + 1).to(q.dtype) + return q * (lp * s + b)[:, None, None] + + +_SILENT = { + "s and b of length 1": lambda q, p, s, b: ( + (q, p, s[:1], b[:1]), + op(q, p, s[:1].expand(s.numel()), b[:1].expand(b.numel())), + ), + "s and b as [H, 1] at T == H": lambda q, p, s, b: ( + (q, p, s[:, None], b[:, None]), + _rowwise_scale(q, p, s, b), + ), + "positions of length 1": lambda q, p, s, b: ( + (q, p[:1], s, b), + op(q, p[:1].expand(p.numel()), s, b), + ), + "one row of q, positions of length 16": lambda q, p, s, b: ( + (q[:1], p, s, b), + op(q[:1].expand(p.numel(), -1, -1), p, s, b), + ), + "2-D q [T, H * D] at T == H": lambda q, p, s, b: ((q.reshape(q.shape[0], -1), p, s, b), None), +} + + +@pytest.mark.parametrize("case", list(_SILENT), ids=list(_SILENT)) +def test_guard_refuses_a_silent_broadcast(case) -> None: + q, positions, s, b = _guard_inputs() + assert q.shape[0] == q.shape[1] == 16, "T == H is what the pairing cases need" + args, wrong = _SILENT[case](q, positions, s, b) + # Unguarded, the op accepts the arguments and answers with a plausible wrong tensor. + got = op(*args) + right = op(q, positions, s, b) + if wrong is not None: + assert got.shape == right.shape + _bitwise(got, wrong, "the broadcast it silently computes") + assert not torch.equal(got, right) + else: + assert got.shape == (16, 16, 2048), got.shape + with validating(op): + with pytest.raises(ValueError, match="ssmax_scaling: .* must be"): + op(*args) + _bitwise(op(q, positions, s, b), right, "the unguarded call") + + +def test_unabsorbable_mismatches_raise_from_torch() -> None: + """What the note quotes: an unsliced [64] s and b against one rank's 16 heads, a 16-head + slice against all 64 (a tep4 slice at dep4), and a short positions, raise in torch.""" + q, positions, s, b = _guard_inputs() + s64, b64 = torch.cat([s] * 4), torch.cat([b] * 4) + with pytest.raises( + RuntimeError, + match=r"The size of tensor a \(16\) must match the size of tensor b \(64\) " + r"at non-singleton dimension 1", + ): + op(q, positions, s64, b64) + q_all, positions_all, s_all, b_all = _build( + _spec_of("mla", heads=MLA_PARAM_HEADS, tokens=16, contexts=(), decode_rows=16), 42 + ) + with pytest.raises( + RuntimeError, + match=r"The size of tensor a \(64\) must match the size of tensor b \(16\) " + r"at non-singleton dimension 1", + ): + op(q_all, positions_all, s_all[:16], b_all[:16]) + with pytest.raises( + RuntimeError, + match=r"The size of tensor a \(16\) must match the size of tensor b \(9\) " + r"at non-singleton dimension 0", + ): + op(q, positions[:9], s, b) + with validating(op): + with pytest.raises(ValueError, match="s must be"): + op(q, positions, s64, b64) + with pytest.raises(ValueError, match="s must be"): + op(q_all, positions_all, s_all[:16], b_all[:16]) + + +def test_tep8_slice_mismatches() -> None: + """At a tep8 or tp8 rank's 8 heads: s and b as [8, 1] at decode batch 8, a captured batch + where T == H, pair row t with head t's terms silently, and `is_valid` refuses them; the + unsliced [64] parameter and a tep4 rank's 16-head slice raise from torch, as the note + quotes.""" + q, positions, s, b = _build( + _spec_of("mla", heads=MLA_TEP8_HEADS, rank=3, tokens=8, contexts=(), decode_rows=8), 43 + ) + assert q.shape[0] == q.shape[1] == MLA_TEP8_HEADS + right = op(q, positions, s, b) + got = op(q, positions, s[:, None], b[:, None]) + assert got.shape == right.shape + _bitwise(got, _rowwise_scale(q, positions, s, b), "the broadcast it silently computes") + assert not torch.equal(got, right) + s64, b64 = torch.cat([s] * 8), torch.cat([b] * 8) + s16, b16 = torch.cat([s] * 2), torch.cat([b] * 2) + for wide_s, wide_b in ((s64, b64), (s16, b16)): + with pytest.raises( + RuntimeError, + match=rf"The size of tensor a \(8\) must match the size of tensor b \({wide_s.numel()}\) " + r"at non-singleton dimension 1", + ): + op(q, positions, wide_s, wide_b) + with validating(op): + for wrong_s, wrong_b in ((s[:, None], b[:, None]), (s64, b64), (s16, b16)): + with pytest.raises(ValueError, match="s must be"): + op(q, positions, wrong_s, wrong_b) + _bitwise(op(q, positions, s, b), right, "the unguarded call") + + +def test_rows_are_independent() -> None: + """A padded decode capture runs dummy rows beside the live ones.""" + q, positions, s, b = _build( + _spec_of("indexer", FP8_MAX_SEQ_LEN, tokens=16, contexts=(), decode_rows=16), 51 + ) + live = op(q[:3].clone(), positions[:3].clone(), s, b) + q[3:] = math.nan + q[3:, :, ::2] = math.inf + positions[3], positions[4], positions[5:] = -1, -2, 2**30 + _bitwise(op(q, positions, s, b)[:3], live, "the live rows alone") + + # A negative position: log 0 = -inf at -1, nan at -2, on that row and no other. + q, positions, s, b = _build( + _spec_of("indexer", FP8_MAX_SEQ_LEN, tokens=3, contexts=(), decode_rows=3), 52 + ) + positions.copy_(torch.tensor([-1, -2, 5], dtype=positions.dtype)) + out = op(q, positions, s, b) + assert not bool(torch.isfinite(out[0]).any()) + assert bool(torch.isnan(out[1]).all()) + assert bool(torch.isfinite(out[2]).all()) + + +def test_dtypes_that_leave_the_result_unchanged() -> None: + """int64 positions, fp32 s and b holding bf16 values, and a bf16 default dtype, at every + position of the NVFP4 target's envelope (MAX_SEQ_LEN).""" + q, positions, s, b = _build(_spec_of("mla", rank=1), 61) + want = op(q, positions, s, b) + _bitwise(op(q, positions.long(), s, b), want, "the int32-positions result") + _bitwise(op(q, positions, s.float(), b.float()), want, "the bf16-s-and-b result") + previous = torch.get_default_dtype() + torch.set_default_dtype(torch.bfloat16) + try: + got = op(q, positions, s, b) + finally: + torch.set_default_dtype(previous) + _bitwise(got, want, "the result under an fp32 default dtype") diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_ssmax_scaling_fused_q.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_ssmax_scaling_fused_q.py new file mode 100644 index 000000000000..010492c86aa2 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_ssmax_scaling_fused_q.py @@ -0,0 +1,468 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the ssmax_scaling_fused_q entry. + +Every cell is gated bit for bit, three ways: + +* against the entry's `reference` (`ssmax_scaling`'s reference chain, then the tail). The + computation is a chain of bf16 roundings with nothing left to accumulation order, so there + is no band to derive; `ssmax_scaling`'s test derives where the chain sits from the closed + form, and this op computes the same chain. +* against the two calls it replaces, `ssmax_scaling` and the tail's copy (or zero fill), run + on the same inputs: the targets' forward is unchanged bit for bit. +* the fused query's bytes outside the tail -- the other heads, the 512 absorbed columns -- + start poisoned and must come back untouched. + +A `position_sweep` cell (every position 0..1048575 of the 1M config, 32 calls of 32768 rows) is +gated so at every call. The old-versus-new tests state the equivalence on its own: the +unfused composition the targets ran (`ssmax_scaling`, then the tail's copy or zero fill into a +poisoned fused query) and the fused call, on identical inputs, compared with `torch.equal` on +the bit patterns -- over every cell's random inputs (every call of a sweep), and over tensors +captured from a served decode step when +TRTLLM_MODELING_V2_FUSE_CAPTURE names their directory (`mla_*.pt`, one file per call: `q`, +`positions`, `s`, `b`, `rope`). `test_every_layer_and_position_bit_equal_to_the_unfused_composition` +states it at the 1M config over the NVFP4 checkpoint's own s and b: every layer, every position +0..1048575, at each call site's layout of the fused query (skipped where the checkpoint is absent +from LLM_MODELS_ROOT). `test_device_log_is_torchs_log` pins the one transcendental: the kernel's +libdevice logf returns torch.log's fp32 bits, so even the unrounded log agrees. +""" + +import json +import os +import zlib +from pathlib import Path + +import pytest +import torch +import triton +import triton.language as tl +from test_common.llm_data import llm_models_root +from triton.language.extra import libdevice + +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ssmax_scaling import ( + ssmax_scaling, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.ssmax_scaling_fused_q import ( + ssmax_scaling_fused_q as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402,F401 + +assert torch.cuda.is_available(), "ssmax_scaling_fused_q requires a CUDA device" + +#: The fused query FlashMLA reads: 64 kernel heads of [absorbed 512 | rope tail 64]. +FQ_HEADS, LATENT, TAIL = 64, 512, 64 +#: bf16 NaN with a payload no op writes: the poison every output byte starts from. +POISON = 0x7FA5 + + +def _seed(text: str) -> int: + return zlib.crc32(text.encode()) + + +def _queries(shape, kind: str, g: torch.Generator) -> torch.Tensor: + if kind == "normal": + return (4 * torch.randn(shape, device="cuda", generator=g)).to(torch.bfloat16) + # Every bf16 binade from the smallest subnormal (2^-133) up, both signs. + exponent = torch.randint(-133, 127, shape, device="cuda", generator=g).float() + mantissa = 1 + torch.rand(shape, device="cuda", generator=g) + sign = 2 * torch.randint(0, 2, shape, device="cuda", generator=g).float() - 1 + return (sign * mantissa * torch.exp2(exponent)).to(torch.bfloat16) + + +def _poisoned_fq(tokens: int, heads: int = FQ_HEADS) -> torch.Tensor: + fq = torch.full((tokens, heads, LATENT + TAIL), POISON, dtype=torch.int16, device="cuda") + return fq.view(torch.bfloat16) + + +def _build(spec, seed: int): + """`(q, positions, s, b, fq)`: s and b are rank `rank`'s view into a `param_heads` draw, + four of its heads on the ranges' corners; `fq` is a poisoned [T, fq_heads, 576] fused + query (64 heads unless the cell names the native decode's scratch).""" + g = torch.Generator(device="cuda").manual_seed(seed) + tokens, heads = spec["tokens"], spec["heads"] + q = _queries((tokens, heads, 128), spec["q"], g) + cached = spec.get("cached", (0,) * len(spec["contexts"])) + assert len(cached) == len(spec["contexts"]), (cached, spec["contexts"]) + assert all(c + n <= spec["max_seq_len"] for n, c in zip(spec["contexts"], cached)) + parts = [torch.arange(c, c + n, device="cuda") for n, c in zip(spec["contexts"], cached)] + if spec["decode_rows"]: + top = spec["max_seq_len"] + rows = torch.randint(1, top, (spec["decode_rows"],), device="cuda", generator=g) + rows[0] = top - 1 + parts.append(rows) + positions = torch.cat(parts).to(spec["positions_dtype"]) + assert positions.numel() == tokens + full = [ + torch.empty(spec["param_heads"], device="cuda").uniform_(*spec[k], generator=g).bfloat16() + for k in ("s_range", "b_range") + ] + lo = spec["rank"] * heads + if spec["corners"]: + at = lo + (4 * spec["rank"]) % heads + corners = [(s, b) for s in spec["s_range"] for b in spec["b_range"]] + for i, values in enumerate(zip(*corners)): + full[i][at : at + 4] = torch.tensor(values, device="cuda").bfloat16() + s, b = (t[lo : lo + heads] for t in full) + fq = _poisoned_fq(tokens, spec.get("fq_heads", FQ_HEADS)) + return q, positions, s, b, fq + + +def _sweep(spec, seed: int): + """A `position_sweep` (lo, hi) cell's calls, `(q, positions, s, b, fq)` each: every + `tokens` rows of positions lo..hi - 1 in order, each call on a fresh q and a fresh poisoned + fused query, the cell's own s and b.""" + lo, hi = spec["position_sweep"] + rows = spec["tokens"] + assert (hi - lo) % rows == 0 and hi <= spec["max_seq_len"], (lo, hi, rows) + _, _, s, b, _ = _build(spec, seed) + for k in range((hi - lo) // rows): + q, _, _, _, fq = _build(spec, seed + 1 + k) + start = lo + k * rows + positions = torch.arange(start, start + rows, dtype=spec["positions_dtype"], device="cuda") + yield q, positions, s, b, fq + + +def _bits_equal(got: torch.Tensor, want: torch.Tensor, what: str) -> None: + assert got.shape == want.shape, (what, got.shape, want.shape) + differ = got.contiguous().view(torch.int16) != want.contiguous().view(torch.int16) + n = int(differ.sum()) + if n: + at = tuple(int(i) for i in differ.nonzero()[0]) + raise AssertionError(f"{what}: {n} of {got.numel()} elements differ; first at {at}") + + +def _call(q, positions, s, b, fq, spec): + """The targets' call: `q` scaled in place (or into a fresh buffer), the tail into `fq`.""" + out = q if spec["inplace"] else torch.empty_like(q) + return op(q, positions, s, b, out, fq[:, : spec["heads"], LATENT:], spec["rope"]) + + +def _check(spec, q_in, positions, s, b, fq, out): + """Gate one call's results against the reference and the replaced pair of calls.""" + heads = spec["heads"] + ref_out, ref_tail = op.reference( + q_in, positions, s, b, out, fq[:, :heads, LATENT:], spec["rope"] + ) + _bits_equal(out, ref_out, "out against the reference") + _bits_equal(fq[:, :heads, LATENT:], ref_tail, "tail against the reference") + old = ssmax_scaling(q_in, positions, s, b) + _bits_equal(out, old, "out against ssmax_scaling") + want_tail = old[..., TAIL:] if spec["rope"] else torch.zeros_like(old[..., TAIL:]) + _bits_equal(fq[:, :heads, LATENT:], want_tail, "tail against the replaced copy") + rest = fq.view(torch.int16).clone() + rest[:, :heads, LATENT:] = POISON + assert bool((rest == POISON).all()), "a byte of the fused query outside the tail was written" + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell, certifying): # noqa: F811 + spec, seed = cell.spec, _seed(cell.why) + if "position_sweep" in spec: + certifying(op) + calls = 0 + for q, positions, s, b, fq in _sweep(spec, seed): + q_in = q.clone() + out = _call(q, positions, s, b, fq, spec) + _check(spec, q_in, positions, s, b, fq, out) + calls += 1 + print( + f"\n {cell.why}: {calls} calls, each bitwise the reference and ssmax_scaling + the tail" + ) + return + if spec["graph"]: + # Captured first and replayed on new inputs, then held to an eager call on them. + q, positions, s, b, fq = _build(spec, seed) + side = torch.cuda.Stream() + side.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(side): + _call(q.clone(), positions, s, b, fq.clone(), spec) + torch.cuda.current_stream().wait_stream(side) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + out = _call(q, positions, s, b, fq, spec) + fresh_q, fresh_positions, _, _, _ = _build(spec, seed + 1) + q.copy_(fresh_q) + positions.copy_(fresh_positions) + fq.view(torch.int16).fill_(POISON) + q_in = q.clone() + graph.replay() + torch.cuda.synchronize() + eager_fq = torch.full_like(fq.view(torch.int16), POISON).view(torch.bfloat16) + eager = _call(q_in.clone(), positions, s, b, eager_fq, spec) + _bits_equal(out, eager, "the replay against an eager call") + _bits_equal(fq, eager_fq, "the replayed fused query against an eager call") + else: + q, positions, s, b, fq = _build(spec, seed) + q_in = q.clone() + certifying(op) + s_before, b_before, positions_before = s.clone(), b.clone(), positions.clone() + fq.view(torch.int16).fill_(POISON) + if spec["inplace"]: + q.copy_(q_in) + out = _call(q, positions, s, b, fq, spec) + assert out.shape == q_in.shape and out.dtype == torch.bfloat16 + assert (out.data_ptr() == q.data_ptr()) == spec["inplace"] + if not spec["inplace"]: + assert torch.equal(q, q_in), "an out-of-place call must not write q" + for now, then in ((s, s_before), (b, b_before), (positions, positions_before)): + assert torch.equal(now, then), "the entry must not mutate positions, s or b" + _check(spec, q_in, positions, s, b, fq, out) + print( + f"\n {cell.why}: bitwise the reference and ssmax_scaling + the tail " + f"{'copy' if spec['rope'] else 'zero fill'}; {int(torch.isinf(out).sum())} overflowed" + ) + + +def test_rows_are_independent() -> None: + """A padded decode capture runs dummy rows beside the live ones.""" + spec = dict(op.CELLS[0].spec, tokens=16, decode_rows=16) + q, positions, s, b, fq = _build(spec, 51) + live_fq = fq.clone() + live = _call(q[:3].clone(), positions[:3].clone(), s, b, live_fq[:3], spec) + q[3:] = float("nan") + q[3:, :, ::2] = float("inf") + positions[3], positions[4], positions[5:] = -1, -2, 2**30 + out = _call(q, positions, s, b, fq, spec) + _bits_equal(out[:3], live, "the live rows beside poisoned ones") + _bits_equal(fq[:3], live_fq[:3], "the live rows' fused query beside poisoned ones") + + +def test_gates_see_a_wrong_tail() -> None: + """The rope-layer and NoPE-layer tails differ, and the gate tells them apart; so does a + tail written from the wrong half.""" + spec = dict(op.CELLS[0].spec, tokens=16, decode_rows=16, rope=True) + q, positions, s, b, fq = _build(spec, 52) + q_in = q.clone() + out = _call(q, positions, s, b, fq, spec) + _check(spec, q_in, positions, s, b, fq, out) + with pytest.raises(AssertionError, match="tail against the reference"): + _check(dict(spec, rope=False), q_in, positions, s, b, fq, out) + fq[:, : spec["heads"], LATENT:] = out[..., :TAIL] + with pytest.raises(AssertionError, match="tail against the reference"): + _check(spec, q_in, positions, s, b, fq, out) + + +def _guard_inputs(): + spec = dict(op.CELLS[0].spec, tokens=16, decode_rows=16) + q, positions, s, b, fq = _build(spec, 41) + return spec, q, positions, s, b, fq + + +@pytest.mark.parametrize( + "case", + [ + "s of the unsliced 64 heads", + "positions of length 1", + "tail of another width", + "out partly overlapping q", + "tail inside q", + "q with a strided head dim", + ], +) +def test_guard_refuses(case) -> None: + spec, q, positions, s, b, fq = _guard_inputs() + heads = spec["heads"] + out, tail = q, fq[:, :heads, LATENT:] + if case == "s of the unsliced 64 heads": + s = torch.cat([s] * 4) + elif case == "positions of length 1": + positions = positions[:1] + elif case == "tail of another width": + tail = fq[:, :heads, LATENT - TAIL :] + elif case == "out partly overlapping q": + flat = torch.empty(q.numel() + 64, dtype=q.dtype, device="cuda") + flat[: q.numel()] = q.flatten() + q = flat[: q.numel()].view_as(q) + out = flat[64:].view_as(q) + elif case == "tail inside q": + tail = q[..., TAIL:] + else: + q = torch.empty(*q.shape, 2, dtype=q.dtype, device="cuda")[..., 0] + out = q + with validating(op): + with pytest.raises(ValueError, match="ssmax_scaling_fused_q"): + op(q, positions, s, b, out, tail, True) + + +#: The directory of tensors captured from a served decode step. +_CAPTURE_ENV = "TRTLLM_MODELING_V2_FUSE_CAPTURE" + + +def _unfused(q, positions, s, b, rope: bool, fq_heads: int = FQ_HEADS): + """The composition the targets ran: ssmax_scaling, then the tail's copy or zero fill.""" + heads = q.shape[1] + out = ssmax_scaling(q, positions, s, b) + fq = _poisoned_fq(q.shape[0], fq_heads) + if rope: + fq[:, :heads, LATENT:].copy_(out[..., TAIL:]) + else: + fq[:, :heads, LATENT:].zero_() + return out, fq + + +def _fused(q, positions, s, b, rope: bool, fq_heads: int = FQ_HEADS): + """The targets' fused call: `q` scaled in place, the tail written into the fused query.""" + out = q.clone() + fq = _poisoned_fq(q.shape[0], fq_heads) + op(out, positions, s, b, out, fq[:, : q.shape[1], LATENT:], rope) + return out, fq + + +def _assert_old_equals_new( + q, positions, s, b, rope: bool, what: str, fq_heads: int = FQ_HEADS +) -> None: + old_out, old_fq = _unfused(q, positions, s, b, rope, fq_heads) + new_out, new_fq = _fused(q, positions, s, b, rope, fq_heads) + assert torch.equal(new_out.view(torch.int16), old_out.view(torch.int16)), f"{what}: out" + assert torch.equal(new_fq.view(torch.int16), old_fq.view(torch.int16)), f"{what}: fused q" + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_bit_equal_to_the_unfused_composition(cell) -> None: + spec, seed = cell.spec, _seed(cell.why) + 7 + calls = _sweep(spec, seed) if "position_sweep" in spec else [_build(spec, seed)] + for q, positions, s, b, _ in calls: + _assert_old_equals_new( + q, positions, s, b, spec["rope"], cell.why, spec.get("fq_heads", FQ_HEADS) + ) + + +@pytest.mark.skipif( + not os.environ.get(_CAPTURE_ENV), reason=f"{_CAPTURE_ENV} names no captured decode step" +) +def test_bit_equal_to_the_unfused_composition_on_captured_decode_tensors() -> None: + files = sorted(Path(os.environ[_CAPTURE_ENV]).glob("mla_*.pt")) + assert files, f"no mla_*.pt under {os.environ[_CAPTURE_ENV]}" + rows = 0 + for f in files: + d = torch.load(f, map_location="cuda") + _assert_old_equals_new(d["q"], d["positions"], d["s"], d["b"], d["rope"], f.name) + rows += d["q"].shape[0] + print(f"\n {len(files)} captured MLA calls, {rows} rows: bit-equal") + + +@triton.jit +def _logf_kernel(x_ptr, y_ptr, n, BLOCK: tl.constexpr): + i = tl.program_id(0) * BLOCK + tl.arange(0, BLOCK) + x = tl.load(x_ptr + i, mask=i < n) + tl.store(y_ptr + i, libdevice.log(x), mask=i < n) + + +def test_device_log_is_torchs_log() -> None: + """libdevice's logf, compiled as the kernel compiles it (no implicit FMA contraction), + returns torch.log's fp32 bits: at every `pos + 1` for positions 0..2^24 - 1 (every + position fp32 holds exactly), and at 2^24 random positive finite fp32 bit patterns.""" + g = torch.Generator(device="cuda").manual_seed(5) + x = torch.cat( + [ + torch.arange(1, 2**24 + 1, dtype=torch.float32, device="cuda"), + torch.randint( + 1, 0x7F800000, (2**24,), generator=g, device="cuda", dtype=torch.int32 + ).view(torch.float32), + ] + ) + y = torch.empty_like(x) + _logf_kernel[(triton.cdiv(x.numel(), 1024),)]( + x, y, x.numel(), BLOCK=1024, enable_fp_fusion=False + ) + assert torch.equal(y.view(torch.int32), torch.log(x).view(torch.int32)) + + +# ---- every layer of the checkpoint at every position of the 1M config + +#: The NVFP4 checkpoint, read under LLM_MODELS_ROOT for its per-layer SSMax s and b. +CHECKPOINTS = ("Mistral-Large-4-1T-A52B-NVFP4", "mistral-large-4-nvfp4") +#: Every position the 1M config reaches (max_seq_len 1048576), in calls of one chunked step's +#: 32768 rows. +LONG_MAX_SEQ_LEN, STEP_ROWS = 1048576, 32768 +#: The call sites' layouts, (a rank's heads, the fused query's heads): dep4's 64 heads into +#: FlashMLA's 64-head query; a tp4 / tep4 rank's 16 into the native decode's `[T, 16, 576]` +#: scratch and into FlashMLA's padded 64-head one (the prefill scratch, and the decode scratch +#: under `_GEN_ATTN_FLASH_MLA`); a tp8 / tep8 rank's 8 the same. A layout's ranks together +#: hold all 64 heads. +LAYOUTS = { + "dep4": (64, 64), + "tp4_native": (16, 16), + "tp4_padded": (16, 64), + "tp8_native": (8, 8), + "tp8_padded": (8, 64), +} + + +def _checkpoint_dir() -> Path | None: + root = llm_models_root() + for name in CHECKPOINTS if root is not None else (): + path = Path(root) / name + names = ("params.json", "consolidated.safetensors.index.json") + if all((path / f).is_file() for f in names): + return path + return None + + +@pytest.fixture(scope="module") +def checkpoint_mla() -> list: + """Every layer's MLA SSMax s and b (`[64]` each, as stored) and whether it ropes its query + (not a NoPE layer), read off the NVFP4 checkpoint.""" + path = _checkpoint_dir() + if path is None: + pytest.skip(f"none of {CHECKPOINTS} is under LLM_MODELS_ROOT ({llm_models_root()})") + from safetensors import safe_open + + from tensorrt_llm._torch.models.checkpoints.mistral.config_loader import MistralConfigLoader + + cfg = MistralConfigLoader().load_pretrained_config(str(path)) + index = json.loads((path / "consolidated.safetensors.index.json").read_text())["weight_map"] + keys = { + (layer, p): f"layers.{layer}.attention.ssmax_scaling.{p}" + for layer in range(cfg.num_hidden_layers) + for p in ("s", "b") + } + got = {} + for shard in sorted({index[k] for k in keys.values()}): + with safe_open(str(path / shard), framework="pt", device="cpu") as handle: + for at, key in keys.items(): + if index[key] == shard: + got[at] = handle.get_tensor(key).cuda() + return [ + (got[layer, "s"], got[layer, "b"], not cfg.nope_layer_flags[layer]) + for layer in range(cfg.num_hidden_layers) + ] + + +@pytest.mark.parametrize("layout", list(LAYOUTS)) +def test_every_layer_and_position_bit_equal_to_the_unfused_composition(checkpoint_mla, layout): + """The NVFP4 checkpoint's own s and b of every layer (all 72, RoPE and NoPE), at every + position 0..1048575 of the 1M config, in each call site's layout (`LAYOUTS`, every rank's + slice of the 64 heads): the fused call is bitwise the composition it replaces, + `ssmax_scaling` then the tail's copy or zero fill -- the scaled query and the whole fused + query, the tail written and every other byte untouched, `torch.equal` on the bit patterns. + Each call is one chunked step's 32768 consecutive positions, half the rows normal queries + and half drawn over every bf16 binade, the same queries at every layer.""" + heads, fq_heads = LAYOUTS[layout] + assert all(s.shape == b.shape == (64,) for s, b, _ in checkpoint_mla) + calls = 0 + for k in range(LONG_MAX_SEQ_LEN // STEP_ROWS): + g = torch.Generator(device="cuda").manual_seed(1 << 20 | k) + half = STEP_ROWS // 2 + q = torch.cat( + [ + _queries((half, heads, 128), "normal", g), + _queries((STEP_ROWS - half, heads, 128), "wide", g), + ] + ) + start = k * STEP_ROWS + positions = torch.arange(start, start + STEP_ROWS, dtype=torch.int32, device="cuda") + for layer, (s_all, b_all, rope) in enumerate(checkpoint_mla): + for lo in range(0, 64, heads): + what = f"{layout} layer {layer} heads {lo}..{lo + heads - 1} at {start}.." + _assert_old_equals_new( + q, positions, s_all[lo : lo + heads], b_all[lo : lo + heads], rope, what, + fq_heads, + ) # fmt: skip + calls += 1 + print( + f"\n {layout}: {len(checkpoint_mla)} layers x positions 0..{LONG_MAX_SEQ_LEN - 1}, " + f"{calls} calls: bit-equal" + ) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_triton_fp8_mqa_logits.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_triton_fp8_mqa_logits.py new file mode 100644 index 000000000000..407d824e853a --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/attention/test_modeling_v2_triton_fp8_mqa_logits.py @@ -0,0 +1,94 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Local VR200 parity checks for the DSA prefill scorer.""" + +import pytest +import torch + +from tensorrt_llm._torch.modeling_v2.catalog._op import true_fp32_matmul +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.cute_dsl_gvr_topk_prefill import ( # noqa: E501 + cute_dsl_gvr_topk_prefill, + gvr_workspace, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.attention.triton_fp8_mqa_logits import ( # noqa: E501 + triton_fp8_mqa_logits as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying # noqa: E402,F401 + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="requires a VR200/SM107 device", +) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[cell.why for cell in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest fixture + certifying(op) + m, n = cell.spec["m"], cell.spec["n"] + generator = torch.Generator(device="cuda").manual_seed(cell.seed_base()) + q = torch.randn(m, 32, 128, device="cuda", generator=generator).to(torch.float8_e4m3fn) + k = torch.randn(n, 128, device="cuda", generator=generator).to(torch.float8_e4m3fn) + scale = torch.rand(n, device="cuda", generator=generator) + 0.01 + weight = torch.randn(m, 64, device="cuda", generator=generator)[:, ::2] + start = torch.arange(m, device="cuda", dtype=torch.int32) % max(n // 2, 1) + end = (start + torch.arange(m, device="cuda", dtype=torch.int32) * 71 + 1).clamp(max=n + 17) + if m > 1: + end[0] = start[0] + if cell.spec.get("strided"): + + def strided_view(tensor: torch.Tensor) -> torch.Tensor: + storage = torch.zeros(tensor.numel() * 2, dtype=tensor.dtype, device=tensor.device) + storage[::2].copy_(tensor) + return storage[::2] + + scale, start, end = (strided_view(tensor) for tensor in (scale, start, end)) + args = q, (k, scale), weight, start, end + actual = op(*args) + assert actual.stride(0) % 4 == 0 + reference = op.reference(*args) + torch.testing.assert_close(torch.isneginf(actual), torch.isneginf(reference)) + live = torch.isfinite(reference) + assert bool(torch.isfinite(actual[live]).all()) + magnitude = torch.zeros_like(reference) + with true_fp32_matmul(): + for head in range(32): + magnitude += (q[:, head].float().abs() @ k.float().abs().t()) * weight[ + :, head : head + 1 + ].abs() + magnitude *= scale[None, :] + error = (actual[live].double() - reference[live].double()).abs() + assert float((error / magnitude[live].double().clamp_min(1e-30)).max()) <= 8 * 2**-23 + relative_rms = error.square().sum().sqrt() / reference[ + live + ].double().square().sum().sqrt().clamp_min(1e-30) + assert float(relative_rms) <= 8 * 2**-23 + + +@pytest.mark.parametrize("n", [2051, 2530, 2533]) +def test_prefill_topk_consumer(n: int) -> None: + m, top_k = 17, 2048 + generator = torch.Generator(device="cuda").manual_seed(n) + q = torch.randn(m, 32, 128, device="cuda", generator=generator).to(torch.float8_e4m3fn) + k = torch.randn(n, 128, device="cuda", generator=generator).to(torch.float8_e4m3fn) + scale = torch.rand(n, device="cuda", generator=generator) + 0.01 + weight = torch.randn(m, 32, device="cuda", generator=generator) + start = torch.zeros(m, device="cuda", dtype=torch.int32) + end = torch.full_like(start, n) + end[:3] = torch.tensor([0, 31, top_k], device="cuda", dtype=torch.int32) + start[-1] = 32 + logits = op(q, (k, scale), weight, start, end) + output = torch.full((m, top_k), -7, device="cuda", dtype=torch.int32) + cute_dsl_gvr_topk_prefill(logits, start, end, output, top_k, gvr_workspace(q.device)) + for row in range(m): + lo, hi = int(start[row]), int(end[row]) + if hi - lo <= top_k: + expected = torch.full_like(output[row], -1) + expected[: hi - lo] = torch.arange(hi - lo, device="cuda", dtype=torch.int32) + torch.testing.assert_close(output[row], expected, rtol=0, atol=0) + else: + expected = torch.argsort(logits[row, lo:hi], descending=True, stable=True)[:top_k] + torch.testing.assert_close( + output[row].long().sort().values, expected.sort().values, rtol=0, atol=0 + ) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/comm/__init__.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/comm/__init__.py new file mode 100644 index 000000000000..52a7a9daf028 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/comm/__init__.py @@ -0,0 +1,2 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/comm/_cute_dsl_allreduce_add_rmsnorm_op_matrix.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/comm/_cute_dsl_allreduce_add_rmsnorm_op_matrix.py new file mode 100644 index 000000000000..44aee27c5d24 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/comm/_cute_dsl_allreduce_add_rmsnorm_op_matrix.py @@ -0,0 +1,977 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU certification matrix for the `cute_dsl_allreduce_add_rmsnorm` entry. + +Every cell names its world size and the workspace it runs on (`max_rows`), and +each job drives the cells of its own world size: one workspace per distinct +`max_rows` among them is built, the Mistral Large 4 FP8 cells on that target's +9216-row workspace and the Mistral Large 4 NVFP4 cells on those targets' +32768-row one (world 4: tep4 and tp4; world 8: tep8 and tp8), which is how +each target builds it. One check reads the 32768-row workspace's peer buffers +where their offsets pass 2^31 elements +(`check_peer_buffer_offsets_past_2_31_land_in_place`); at world 8 they pass +2^32. + +The Mistral Large 4 cells are driven the way those targets serve: their decode +steps replay captured CUDA graphs and their prefill steps run eagerly, all on +one workspace. So each decode cell is one step's graph -- 144 launches, two +per layer, carrying one residual through them in place -- and its replays are +interleaved with the prefill cells' eager calls: a decode launch then reuses a +slot a two-shot call of up to the whole workspace filled, which is what the +per-slot vector counts in the device state exist to clear. + +A collective cannot be exercised in one process. Two jobs, both this file: + +- the world-4 cells, on one node. Run plainly, this script is its own + launcher: it spawns one worker process per device named in + CUDA_VISIBLE_DEVICES (four), under a deadline the parent enforces by + killing every worker's session. A Lamport collective whose ranks disagree + hangs rather than raising, so the deadline is what keeps a broken kernel + from taking the calling run down with it. The workers are plain processes, + no MPI: the workspace brings its process group up from the rendezvous + address the launcher puts in the environment. + + CUDA_VISIBLE_DEVICES=0,1,2,3 python _cute_dsl_allreduce_add_rmsnorm_op_matrix.py + +- the world-8 cells, over two nodes of four devices. Started once per rank by + an MPI launcher with `--mpi-rank-worker`, as the engine's ranks are + started: each rank takes the device of its node-local rank, drops any + rendezvous address its node's environment holds, and the workspace brings + its process group up from rank 0's address broadcast over + `MPI.COMM_WORLD`. One process cannot place ranks on another node, so the + launcher is the environment's (the collected test takes it from + `MODELING_V2_MPI_LAUNCH`): + + python _cute_dsl_allreduce_add_rmsnorm_op_matrix.py --mpi-rank-worker + + A Lamport collective one rank misses hangs every other rank. So every + rank-local assertion of this job is agreed over `MPI.COMM_WORLD` before any + rank runs another collective: a failure on one rank fails every rank at the + same point, and each exits nonzero. A rank that wedges inside a collective + never reaches the agreement; `MPI_DEADLINE_S` ends it (faulthandler dumps + its stack and exits), and the launcher ends its siblings. + +Not a pytest module, despite the `check_*` bodies: they are one ordered +sequence inside a single job, reading rank state only `_run_one_rank` binds, +and the slot-counter checks depend on every launch before them. The collected +entry point is `test_modeling_v2_cute_dsl_allreduce_add_rmsnorm.py` +beside this file. This file imports only torch and the standard library at +module scope: `_rank_job` executes it to read `DEADLINE_S`. +""" + +import faulthandler +import math +import os +import signal +import socket +import subprocess +import sys +import time +import traceback +import types +import zlib +from contextlib import contextmanager +from pathlib import Path +from typing import Any, Dict, Iterator, List, Optional, Tuple + +import torch +import torch.distributed as dist + +_WORKER_FLAG = "--rank-worker" +_MPI_WORKER_FLAG = "--mpi-rank-worker" +#: The world-4 job: two workspaces of four CuTe DSL compiles each per rank, then the +#: cells, the largest of which build fp64 references over 32768 x 6144: the whole job +#: measured 51 s with a third workspace and its cells besides, so this is some twelve +#: times that. Read by `_rank_job.run`. +DEADLINE_S = 600 +#: The world-8 job's own deadline, each rank's: one workspace, sixteen captured steps +#: replayed three times each between eager calls of up to 32768 rows. Read by the +#: collected test, which allows its launcher this plus a grace. +MPI_DEADLINE_S = 900 + +#: The spawned job's world size: the world-4 cells, on the devices of one node. +WORLD_SIZE = 4 +#: The MPI job's: the world-8 cells, over two nodes. +MPI_WORLD_SIZE = 8 +HIDDEN = 6144 +#: Mistral Large 4's `rms_norm_eps`. +EPS = 1e-5 +#: Launches per captured decode graph: Mistral Large 4's two closing collectives per +#: layer over its 72 layers, one decode step. +STEP_LAUNCHES = 144 +#: Each decode graph is replayed this many times, after a different prefill cell +#: each time. Between two replays of one graph a round launches, per graph cell, +#: one eager call and one 144-launch graph, 145 launches, 1 mod 3: the slot counter +#: moves by the number of graph cells mod 3 (5 graphs per Mistral Large 4 TP4 +#: workspace, 16 at TP8), so while that is not a multiple of 3 the three replays +#: start on three different slots. +GRAPH_ROUNDS = 3 + +# The payload scales. The world's partials sum to std 1 (each std 1/sqrt(world): +# 0.5 at TP4), the residual is std 1, so the sum and the residual are the same +# size and neither hides the other's rounding. The norm weight is 1 +- 1/8, the +# spread the band derivations below assume. +_RESIDUAL_STD = 1.0 +_WEIGHT_SPREAD = 0.25 +#: Scales the payload so a row's mean square is about 2e-6, near eps: see +#: `check_the_workspace_eps_reaches_the_kernel`. +_SMALL = 1e-3 + +# Bound in the rank body, never at module scope: importing the entry pulls in +# tensorrt_llm, and `_rank_job` executes this file to read DEADLINE_S. +RANK = 0 +WORLD = 1 +#: Distinct hosts the job's ranks run on. +NODES = 1 +#: `MPI.COMM_WORLD` in the MPI job, where rank-local failures are agreed; None in the +#: spawned job, whose launcher ends every sibling of a failing rank itself. +COMM: Any = None +TWOSHOT_ABOVE = 0 +op: Any = None +validating: Any = None +assert_within_ulp: Any = None +#: `max_rows` -> (the workspace, its slot-counter mirror). +WORKSPACES: Dict[int, Tuple[Any, "_Mirror"]] = {} +#: How often the entry's own `__call__` and `reference` ran (see `_count_entry_calls`). +SEEN: Dict[str, int] = {"__call__": 0, "reference": 0} + + +# ---- agreeing on a rank-local failure --------------------------------------- + + +def _agree(where: str, error: Optional[str]) -> None: + """Every rank's verdict on a rank-local block; any failure fails every rank here.""" + errors = COMM.allgather(error) + failed = [(r, e) for r, e in enumerate(errors) if e is not None] + if failed: + detail = "\n".join(f" rank {r}: {e[:1500]}" for r, e in failed) + raise AssertionError( + f"{where}: failed on rank(s) {[r for r, _ in failed]} of {WORLD}\n{detail}" + ) + + +@contextmanager +def _rank_local(where: str) -> Iterator[None]: + """A block of rank-local assertions, ahead of the next collective. + + In the MPI job its outcome is agreed across the ranks on the way out, so + every rank leaves it together, raising or not; a failing rank cannot leave + its siblings inside a collective it skipped. In the spawned job an + assertion propagates as it is, and the launcher ends the siblings. + """ + if COMM is None: + yield + return + error = None + try: + yield + except Exception as exc: # noqa: BLE001 -- reported on every rank, re-raised below + error = f"{type(exc).__name__}: {exc}" + print(f"[rank {RANK}] {where}:\n{traceback.format_exc()}", flush=True) + _agree(where, error) + + +# ---- the host mirror of a workspace's device slot counter ---------------- + + +class _Mirror: + """What one workspace's device slot state must read after the launches so far. + + `state` is `[entry counter, current slot, rows*hidden/8 last published per + slot (3), release epoch]`. Every launch of any of the workspace's kernels + takes the current slot, records how many 16-byte vectors it published + there, and moves the slot on by one mod 3; the entry counter returns to 0 + when the launch completes. The epoch moves only under the `strict` PDL + policy, which no kernel of the workspace uses. + """ + + def __init__(self, ws: Any) -> None: + self.ws = ws + self.slot = 0 + self.counts = [0, 0, 0] + + def launched(self, rows: int) -> None: + self.counts[self.slot] = rows * HIDDEN // 8 + self.slot = (self.slot + 1) % 3 + + def check(self, where: str) -> None: + torch.cuda.synchronize() + got = self.ws.state.tolist() + want = [0, self.slot, *self.counts, 0] + assert got == want, f"{where}: device slot state {got}, expected {want}" + + +# ---- payloads ------------------------------------------------------------- + + +def _seed(*parts: Any) -> int: + return zlib.crc32("|".join(str(p) for p in parts).encode()) % 2**31 + + +def _randn(rows: int, seed: int, std: float) -> torch.Tensor: + # Every rank regenerates every rank's payload from the seed: the CUDA + # generator is deterministic per (seed, shape), so the reference is + # arithmetic on this rank rather than a second collective. + g = torch.Generator(device="cuda").manual_seed(seed) + return (torch.randn(rows, HIDDEN, generator=g, device="cuda") * std).to(torch.bfloat16) + + +def _partials(rows: int, seed: int, scale: float = 1.0) -> List[torch.Tensor]: + """Every rank's partial, in rank order.""" + std = 1.0 / math.sqrt(WORLD) + return [_randn(rows, seed * 977 + r + 1, std * scale) for r in range(WORLD)] + + +def _residual(rows: int, seed: int, scale: float = 1.0) -> torch.Tensor: + """The residual stream: replicated, identical on every rank, as in the target.""" + return _randn(rows, seed * 977 + 101, _RESIDUAL_STD * scale) + + +def _weight(seed: int) -> torch.Tensor: + g = torch.Generator(device="cuda").manual_seed(seed * 977 + 202) + u = torch.rand(HIDDEN, generator=g, device="cuda") - 0.5 + return (1.0 + u * _WEIGHT_SPREAD).to(torch.bfloat16) + + +# ---- gates ---------------------------------------------------------------- + + +def _assert_same_on_every_rank(t: torch.Tensor, where: str) -> None: + """Every rank sums in the same order, so every rank's result is bit-identical.""" + bits = t.contiguous().view(torch.uint8).to(torch.int32) + lo, hi = bits.clone(), bits.clone() + dist.all_reduce(lo, op=dist.ReduceOp.MIN) + dist.all_reduce(hi, op=dist.ReduceOp.MAX) + assert torch.equal(lo, hi), f"{where}: the ranks hold different results" + + +def _gate_norm_one_shot(norm: torch.Tensor, ref_norm: torch.Tensor, where: str) -> None: + """A one-shot norm against the fp64 reference: 2 ulp element-wise, 0.5 RMS (see `_gate`).""" + try: + assert_within_ulp(norm, ref_norm, element_ulp=2.0, rms_ulp=0.5) + except AssertionError as exc: + raise AssertionError(f"{where}: {exc}") from None + + +def _gathered(partials: List[torch.Tensor]) -> torch.Tensor: + """What the two-shot gather carries: the fp32 sum in rank order, rounded once to bf16.""" + acc = torch.zeros(partials[0].shape, dtype=torch.float32, device=partials[0].device) + for p in partials: + acc += p.float() + return acc.to(torch.bfloat16) + + +def _refs_per_check(rows: int) -> int: + """How many times `_gate` calls the entry's `reference` at this row count.""" + return 1 if rows <= TWOSHOT_ABOVE else 2 + + +def _gate( + rows: int, + norm: torch.Tensor, + new_residual: torch.Tensor, + inputs: Tuple[List[torch.Tensor], torch.Tensor, torch.Tensor], + ws: Any, + where: str, + report: bool = False, +) -> None: + """Gate one call's outputs against the entry's `reference` on its inputs; the bands, and why. + + One-shot (rows <= TWOSHOT_ABOVE): the residual is bf16(fp32 sum in rank + order + residual), the reference's arithmetic exactly, so it is bit-exact. + The norm differs from the fp64 reference only through the kernel's fp32 + sum of squares, its approximate rsqrt and its fp32 products, about 1e-6 + relative before the bf16 rounding. That moves an element only where it + sits on a bf16 rounding boundary, and then by one ulp of that element: + at most 2^-7 of it, 2 units of 2^-8 against the row's largest. So 2 ulp + element-wise; and since about 1e-6 / 2^-8 of the elements move, 0.5 ulp + relative RMS is some ten times what that rate gives. + + Two-shot (rows > TWOSHOT_ABOVE): the reduced rows are gathered as bf16, so + the residual is bf16(bf16(sum) + residual), rounded twice. Against the + single rounding that is at most half an ulp of the sum plus an ulp of the + result: 1 + 2 units of the row's largest when the sum is no larger than + the residual stream it lands in (both std 1 here, at every world size), so + 4 ulp element-wise. Double rounding moves roughly a quarter to a half of + the elements by one ulp, about 1 ulp relative RMS, so 2. And the residual + must NOT be bit-exact: that is what shows the two-shot kernel ran. The + norm inherits the residual's difference scaled by the weight, at most + 4 x (9/7) units with the weight in 1 +- 1/8, plus its own 2: 8 ulp + element-wise, 2 RMS. + + That band admits a sum taken in bf16 pairwise: measured 3.2-3.4 ulp + element-wise, 0.64 RMS on the residual at 9216 rows (TP4). So the two-shot + call is also gated against its own arithmetic step for step: the entry's + `reference` given the bf16 gathered sum as the one partial computes + bf16(bf16(sum) + residual) and its norm. Measured bit-exact on the + residual at 3073, 4096, 9215 and 9216 rows and on every rank at TP4, with + the norm within 1.35 ulp element-wise, 0.0045 RMS -- the one-shot band + holds. The bf16 pairwise sum differs from it in about a fifth of the + elements. + + `report` prints, on rank 0, how many elements differ from the reference. + """ + partials, residual, weight = inputs + ref_norm, ref_residual = op.reference(partials, residual, weight, norm, ws) + if report and RANK == 0: + print( + f"[rank 0] {where}: {int((norm != ref_norm).sum())} norm and " + f"{int((new_residual != ref_residual).sum())} residual elements of " + f"{norm.numel()} differ from the reference", + flush=True, + ) + if rows <= TWOSHOT_ABOVE: + assert torch.equal(new_residual, ref_residual), f"{where}: one-shot residual not exact" + _gate_norm_one_shot(norm, ref_norm, where) + return + assert_within_ulp(new_residual, ref_residual, element_ulp=4.0, rms_ulp=2.0) + assert not torch.equal(new_residual, ref_residual), ( + f"{where}: a two-shot residual rounded once: the one-shot kernel ran above " + f"{TWOSHOT_ABOVE} rows" + ) + assert_within_ulp(norm, ref_norm, element_ulp=8.0, rms_ulp=2.0) + step_norm, step_residual = op.reference([_gathered(partials)], residual, weight, norm, ws) + if report and RANK == 0: + print( + f"[rank 0] {where}: {int((norm != step_norm).sum())} norm and " + f"{int((new_residual != step_residual).sum())} residual elements differ from " + f"the step-for-step reference", + flush=True, + ) + assert torch.equal(new_residual, step_residual), ( + f"{where}: the two-shot residual is not bf16(bf16(fp32 sum in rank order) + residual)" + ) + _gate_norm_one_shot(norm, step_norm, where) + + +# ---- one call of the entry, with its inputs and its check ----------------- + + +class _Call: + """One entry call on one workspace: owns its buffers, so it can be refilled and replayed.""" + + def __init__(self, rows: int, max_rows: int) -> None: + self.rows, self.max_rows = rows, max_rows + self.ws, self.mirror = WORKSPACES[max_rows] + dev = "cuda" + self.partial = torch.empty(rows, HIDDEN, dtype=torch.bfloat16, device=dev) + self.out = torch.empty(rows, HIDDEN, dtype=torch.bfloat16, device=dev) + self.residual = torch.empty(rows, HIDDEN, dtype=torch.bfloat16, device=dev) + self.weight = torch.empty(HIDDEN, dtype=torch.bfloat16, device=dev) + self._inputs: Optional[tuple] = None + + def fill(self, seed: int, scale: float = 1.0) -> None: + """Fresh inputs for `seed`; NaN in the output, so an unwritten element fails.""" + partials = _partials(self.rows, seed, scale) + residual, weight = _residual(self.rows, seed, scale), _weight(seed) + self.partial.copy_(partials[RANK]) + self.residual.copy_(residual) + self.weight.copy_(weight) + self.out.fill_(float("nan")) + self._inputs = (partials, residual, weight) + + def launch(self) -> None: + assert op(self.partial, self.residual, self.weight, self.out, self.ws) is self.out + + def check(self, where: str, report: bool = False) -> None: + """The slot state and the gate on this rank, agreed; then bit-identity across ranks.""" + where = f"{where}: {self.rows} rows on the {self.max_rows}-row workspace" + with _rank_local(where): + self.mirror.check(where) + _gate(self.rows, self.out, self.residual, self._inputs, self.ws, where, report) + _assert_same_on_every_rank(self.out, where) + _assert_same_on_every_rank(self.residual, where) + + +def _count_entry_calls() -> None: + """What `_validating.certifying` does under pytest, for this rank body. + + The entry's own `__call__` (with `is_valid` interposed first, as + `validating` does) and its own `reference` are wrapped to count, so every + cell can assert that both ran for it rather than a local copy. + """ + cls = type(op) + call, ref = cls.__dict__["__call__"], cls.__dict__["reference"] + + def counted_call(self, *args, **kwargs): + SEEN["__call__"] += 1 + self.is_valid(*args, **kwargs) + return call(self, *args, **kwargs) + + def counted_reference(self, *args, **kwargs): + SEEN["reference"] += 1 + return ref(self, *args, **kwargs) + + cls.__call__ = counted_call + cls.reference = counted_reference + + +def _drove(i: int, before: Dict[str, int], calls: int, refs: int, driven: Dict[int, int]) -> None: + """Record that cell `i` ran through the entry's own `__call__` and `reference`.""" + got = (SEEN["__call__"] - before["__call__"], SEEN["reference"] - before["reference"]) + assert got == (calls, refs), ( + f"cell {op.CELLS[i].why!r}: the entry's __call__ / reference ran {got} times, " + f"expected {(calls, refs)}" + ) + driven[i] = driven.get(i, 0) + 1 + + +def _mine() -> List[int]: + """The cells this job drives: those at its world size.""" + return [i for i, c in enumerate(op.CELLS) if c.spec["world_size"] == WORLD] + + +# ---- checks --------------------------------------------------------------- + + +def check_every_cell_names_this_configuration() -> None: + """The constants this matrix builds tensors from are the ones every cell states. + + Every cell is at one of the two jobs' world sizes, and this job's cells + state the topology it runs on: `nodes` (1 when absent) distinct hosts. + """ + with _rank_local("the cells' configuration"): + assert _mine(), f"no cell is at world size {WORLD}" + for cell in op.CELLS: + spec = cell.spec + assert spec["world_size"] in (WORLD_SIZE, MPI_WORLD_SIZE), (cell.why, spec) + assert spec["hidden"] == HIDDEN and spec["eps"] == EPS, (cell.why, spec) + assert 1 <= spec["rows"] <= spec["max_rows"], (cell.why, spec) + if spec.get("graph"): + assert spec["rows"] <= TWOSHOT_ABOVE, (cell.why, "a decode graph cell is one-shot") + for i in _mine(): + cell = op.CELLS[i] + nodes = cell.spec.get("nodes", 1) + assert nodes == NODES, (cell.why, f"this job spans {NODES} node(s)") + + +def check_the_workspaces_agreed_and_launched_nothing() -> None: + """Every rank agreed the same multicast decision, and building launched no kernel.""" + for max_rows, (ws, mirror) in WORKSPACES.items(): + flags: List[Any] = [None] * WORLD + dist.all_gather_object(flags, ws.multicast_base != 0) + with _rank_local(f"the {max_rows}-row workspace"): + assert len(set(flags)) == 1, f"ranks disagree on multicast: {flags}" + assert ws.max_rows == max_rows and ws.hidden == HIDDEN and ws.world == WORLD + assert ws.eps == EPS + mirror.check(f"after building the {max_rows}-row workspace") + if RANK == 0: + print( + f"[rank 0] {max_rows}-row workspace: multicast publish " + f"{'on' if flags[0] else 'off'}", + flush=True, + ) + + +def _eager(i: int, rnd: int, driven: Dict[int, int]) -> None: + cell = op.CELLS[i] + rows, max_rows = cell.spec["rows"], cell.spec["max_rows"] + where = f"cell {cell.why!r}, round {rnd}" + call = _Call(rows, max_rows) + call.fill(_seed(cell.why, rnd)) + torch.cuda.synchronize() + dist.barrier() + before = dict(SEEN) + call.launch() + call.mirror.launched(rows) + call.check(where, report=rnd == 0) + with _rank_local(where): + _drove(i, before, calls=1, refs=_refs_per_check(rows), driven=driven) + dist.barrier() + + +class _Step: + """A decode cell: one step's `STEP_LAUNCHES` launches at its row count, captured once. + + As in the target's forward, one residual buffer is carried through every + launch in place, each launch with its own partial, output and norm weight + (each layer's norms differ), and nothing between the launches: launch + `k + 1` reads the residual launch `k` wrote. On a one-shot row count each + residual is exact, so the chained reference stays exact to the last launch. + """ + + def __init__(self, i: int) -> None: + self.i = i + cell = op.CELLS[i] + self.rows, self.max_rows = cell.spec["rows"], cell.spec["max_rows"] + self.ws, self.mirror = WORKSPACES[self.max_rows] + shape, bf16 = (self.rows, HIDDEN), torch.bfloat16 + self.residual = torch.empty(shape, dtype=bf16, device="cuda") + self.partials = torch.empty((STEP_LAUNCHES, *shape), dtype=bf16, device="cuda") + self.outs = torch.empty((STEP_LAUNCHES, *shape), dtype=bf16, device="cuda") + self.weights = torch.empty((STEP_LAUNCHES, HIDDEN), dtype=bf16, device="cuda") + self._inputs: Optional[tuple] = None + self.fill(_seed(cell.why, "capture")) + torch.cuda.synchronize() + dist.barrier() + before = SEEN["__call__"] + self.graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(self.graph, capture_error_mode="thread_local"): + for k in range(STEP_LAUNCHES): + out = self.outs[k] + assert op(self.partials[k], self.residual, self.weights[k], out, self.ws) is out + with _rank_local(f"capturing {cell.why!r}"): + captured = SEEN["__call__"] - before + assert captured == STEP_LAUNCHES, captured + dist.barrier() + + def fill(self, seed: int) -> None: + """Fresh inputs for `seed`; NaN in the outputs, so an unwritten element fails.""" + partials = [_partials(self.rows, _seed(seed, k)) for k in range(STEP_LAUNCHES)] + residual = _residual(self.rows, seed) + weights = [_weight(_seed(seed, k)) for k in range(STEP_LAUNCHES)] + self.partials.copy_(torch.stack([p[RANK] for p in partials])) + self.residual.copy_(residual) + self.weights.copy_(torch.stack(weights)) + self.outs.fill_(float("nan")) + self._inputs = (partials, residual, weights) + + def replay(self, rnd: int, driven: Dict[int, int]) -> None: + cell = op.CELLS[self.i] + where = f"cell {cell.why!r}, replay {rnd}" + self.fill(_seed(cell.why, rnd)) + torch.cuda.synchronize() + dist.barrier() + before = dict(SEEN) + self.graph.replay() + for _ in range(STEP_LAUNCHES): + self.mirror.launched(self.rows) + with _rank_local(where): + self.mirror.check(where) + partials, residual, weights = self._inputs + differ = 0 + for k in range(STEP_LAUNCHES): + ref_norm, residual = op.reference( + partials[k], residual, weights[k], self.outs[k], self.ws + ) + differ += int((self.outs[k] != ref_norm).sum()) + _gate_norm_one_shot(self.outs[k], ref_norm, f"{where}, launch {k}") + assert torch.equal(self.residual, residual), ( + f"{where}: the residual carried through {STEP_LAUNCHES} launches is not exact" + ) + if RANK == 0: + print( + f"[rank 0] {where}: {differ} norm elements of " + f"{STEP_LAUNCHES * self.rows * HIDDEN} differ from the reference, " + f"the carried residual exact", + flush=True, + ) + _assert_same_on_every_rank(self.outs, where) + _assert_same_on_every_rank(self.residual, where) + # A replay runs the kernels the capture recorded and no Python: the + # entry's `__call__` ran (and its guard) at capture, counted there. + with _rank_local(where): + _drove(self.i, before, calls=0, refs=STEP_LAUNCHES, driven=driven) + dist.barrier() + + +def check_every_cell_on_its_workspace() -> None: + """Drive every cell of this job's world size, through the entry's own `__call__` and `reference`. + + Eager cells of a workspace without graph cells run in order and then in + reverse, so a small call follows the largest one and has to clear the slot + it filled. A workspace with graph cells captures each once, then replays + each `GRAPH_ROUNDS` times, each replay after an eager cell's call (a + different one per round): a decode step after a prefill chunk, sharing one + workspace, as the target serves. The slot state is checked after every + call and every replay. + """ + driven: Dict[int, int] = {} + for max_rows, (_, mirror) in WORKSPACES.items(): + mine = [i for i in _mine() if op.CELLS[i].spec["max_rows"] == max_rows] + eager = [i for i in mine if not op.CELLS[i].spec.get("graph")] + graphs = [_Step(i) for i in mine if op.CELLS[i].spec.get("graph")] + with _rank_local(f"the {max_rows}-row workspace's graphs"): + mirror.check(f"after capturing {len(graphs)} graphs: capturing launches nothing") + if not graphs: + for rnd, order in enumerate((eager, eager[::-1])): + for i in order: + _eager(i, rnd, driven) + continue + with _rank_local(f"the {max_rows}-row workspace's rounds"): + assert eager, "graph cells are replayed between eager cells" + assert len(graphs) % 3, "the replays of a graph would all start on one slot" + for rnd in range(GRAPH_ROUNDS): + for k, g in enumerate(graphs): + _eager(eager[(k + rnd) % len(eager)], rnd, driven) + g.replay(rnd, driven) + for g in graphs: + g.graph.reset() + with _rank_local("every cell driven"): + undriven = [op.CELLS[i].why for i in _mine() if i not in driven] + assert not undriven, f"cells never driven: {undriven}" + if RANK == 0: + print(f"[rank 0] cells driven: {driven}", flush=True) + + +# ---- peer-buffer offsets past 2^31 ----------------------------------------- + +#: Bitwise bf16 -0.0: the in-band "not yet arrived" mark a cleared vector holds. +_NOT_ARRIVED = -32768 +#: Bitwise bf16 1.0: a stale vector that reads as arrived. +_STALE = 0x3F80 + + +def _probe_call(rows: int, max_rows: int, *seed: Any) -> "_Call": + """One call outside any cell, on the workspace's current slot, gated like a cell's.""" + call = _Call(rows, max_rows) + call.fill(_seed("offsets", *seed)) + torch.cuda.synchronize() + dist.barrier() + call.launch() + call.mirror.launched(rows) + call.check(f"offsets probe {seed}") + dist.barrier() + return call + + +def check_peer_buffer_offsets_past_2_31_land_in_place() -> None: + """At 32768 rows a peer buffer passes 2^31 elements; read the buffer where it does. + + Each rank's buffer is `[3 slots, world regions, max_rows x 6144]` bf16. At + max_rows 32768 that is 2,415,919,104 elements (4.5 GiB) at world 4 and + 4,831,838,208 (9 GiB, past 2^32 elements) at world 8, and slot 2's last + region starts at element (2 x world + world - 1) x 201,326,592: + 2,214,592,512 at world 4, past 2^31 elements and 2^32 bytes; + 4,630,511,616 at world 8, past 2^32 elements and 2^33 bytes. A one-shot + launch publishes the rank's partial into region of every + peer's buffer, and every launch clears the next slot's regions; two-shot + moves data through regions 0 and 1 only (on slot 2 below 2^31 at world 4, + from element 3,221,225,472 at world 8) but its clear covers every region. + A 32-bit offset there would wrap to an address outside the buffer. The + cells reach slot 2 as well, gated on their outputs; this reads the buffer + itself: + + - a one-shot call at 3072 rows (the most one-shot publishes) on slot 2: + every peer's region of this rank's buffer holds that peer's partial bit + for bit (-0.0 is published as +0.0), the last rank's furthest out; + - a one-row call on slot 0, then one on slot 1, which clears slot 2: + those regions read -0.0 again; + - a 32768-row two-shot call on slot 2, gated; then every region of slot 2 + overwritten with a ready-looking 1.0, out to the buffer's last element; + a one-row call on slot 0 and a two-shot call on slot 1, whose clear + covers slot 2's regions to 32768 rows: every element of slot 2 reads + -0.0; + - a one-shot call on slot 2 after that, gated: no stale vector was read. + """ + probed = [] + for max_rows, (ws, mirror) in WORKSPACES.items(): + numel = max_rows * HIDDEN + if 3 * WORLD * numel <= 2**31: + continue + bits = ws.buffer.view(torch.int16) + last = (2 * WORLD + WORLD - 1) * numel + with _rank_local(f"the {max_rows}-row workspace's buffer"): + assert tuple(bits.shape) == (3, WORLD, numel), bits.shape + assert last > 2**31, "slot 2's last region starts below 2^31" + peers = [s for s in range(WORLD) if s != RANK] + + while mirror.slot != 2: + _probe_call(1, max_rows, "to slot 2", mirror.slot) + rows = TWOSHOT_ABOVE + partials = _probe_call(rows, max_rows, "one-shot publish")._inputs[0] + with _rank_local(f"{max_rows}-row workspace, slot 2, one-shot publish"): + for s in peers: + want = partials[s].view(torch.int16).reshape(-1) + want = torch.where(want == _NOT_ARRIVED, torch.zeros_like(want), want) + assert torch.equal(bits[2, s, : rows * HIDDEN], want), ( + f"{max_rows}-row workspace, slot 2, region {s} (from element " + f"{(2 * WORLD + s) * numel}): rank {s}'s partial is not where it was " + f"published" + ) + _probe_call(1, max_rows, "one-shot clear", 0) + _probe_call(1, max_rows, "one-shot clear", 1) + torch.cuda.synchronize() + with _rank_local(f"{max_rows}-row workspace, slot 2, one-shot clear"): + for s in peers: + left = int((bits[2, s, : rows * HIDDEN] != _NOT_ARRIVED).sum()) + assert left == 0, ( + f"{max_rows}-row workspace, slot 2, region {s}: {left} published " + f"elements not cleared by the launch on slot 1" + ) + assert mirror.slot == 2, mirror.slot + + _probe_call(max_rows, max_rows, "two-shot, whole workspace") + bits[2].fill_(_STALE) + torch.cuda.synchronize() + with _rank_local(f"{max_rows}-row workspace, slot 2 overwritten"): + assert int((bits[2] == _STALE).sum()) == WORLD * numel + dist.barrier() + _probe_call(1, max_rows, "two-shot clear", 0) + _probe_call(TWOSHOT_ABOVE + 1, max_rows, "two-shot clear", 1) + torch.cuda.synchronize() + with _rank_local(f"{max_rows}-row workspace, slot 2, two-shot clear"): + left = int((bits[2] != _NOT_ARRIVED).sum()) + assert left == 0, ( + f"{max_rows}-row workspace: {left} of slot 2's {WORLD * numel} elements, last " + f"element {3 * WORLD * numel - 1}, not cleared by the two-shot launch on slot 1" + ) + _probe_call(16, max_rows, "slot 2 after the clear") + probed.append(max_rows) + if RANK == 0: + print( + f"[rank 0] {max_rows}-row workspace: rank {WORLD - 1}'s one-shot publish on " + f"slot 2 from element {last} in place and cleared; a two-shot call on slot 2 " + f"(regions 0 and 1 from element {2 * WORLD * numel}) gated; slot 2 cleared to " + f"element {3 * WORLD * numel - 1}", + flush=True, + ) + with _rank_local("offsets probed"): + assert probed, "no workspace passes 2^31 elements: the 32768-row cells are gone" + + +def check_the_workspace_eps_reaches_the_kernel() -> None: + """`eps` is compiled into the kernels at build; rows whose mean square is near it show which. + + At the cells' payload (mean square about 2) eps 1e-5 moves the norm by + 2.5e-6 relative, below any bf16 step, so no cell can tell 1e-5 from + another eps. Here the payload is scaled by `_SMALL`, putting the mean + square at 2e-6: measured, the kernel's norm then sits 129 to 500 ulp from + a reference at eps 1e-6, 1e-4 or 0, against 3 ulp at 1e-5. The gate passes + at the workspace's eps and must fail at 1e-6, one-shot and two-shot. + """ + wrong = types.SimpleNamespace(eps=1e-6) + for max_rows, (ws, mirror) in WORKSPACES.items(): + for rows in (16, max_rows): + where = f"eps: {rows} rows on the {max_rows}-row workspace" + call = _Call(rows, max_rows) + call.fill(_seed("eps", max_rows, rows), scale=_SMALL) + torch.cuda.synchronize() + dist.barrier() + call.launch() + mirror.launched(rows) + with _rank_local(where): + mirror.check(where) + _gate(rows, call.out, call.residual, call._inputs, ws, where) + try: + _gate(rows, call.out, call.residual, call._inputs, wrong, where) + except AssertionError: + pass + else: + raise AssertionError(f"{where}: a reference at eps 1e-6 passes the gate too") + dist.barrier() + + +def check_guards_refuse_a_row_count_above_the_budget() -> None: + """Above `max_rows` the kernel would write past every peer's buffer. + + The guard raises before anything launches, on every rank alike, so the + group stays in step. + """ + for max_rows, (_, mirror) in WORKSPACES.items(): + big = _Call(max_rows + 1, max_rows) + with _rank_local(f"{max_rows + 1} rows on the {max_rows}-row workspace"): + try: + with validating(op): + big.launch() + except AssertionError as exc: + assert "budget" in str(exc), exc + else: + raise AssertionError(f"{max_rows + 1} rows accepted by a {max_rows}-row workspace") + mirror.check(f"after the refused call on the {max_rows}-row workspace") + dist.barrier() + + +CHECKS = ( + check_every_cell_names_this_configuration, + check_the_workspaces_agreed_and_launched_nothing, + check_every_cell_on_its_workspace, + check_peer_buffer_offsets_past_2_31_land_in_place, + check_the_workspace_eps_reaches_the_kernel, + check_guards_refuse_a_row_count_above_the_budget, +) + + +def _run_one_rank(mpi: bool) -> int: + """Body of one rank: run every check; a failing rank exits nonzero. + + Spawned (`mpi` False): the launcher kills the siblings of a failing rank. + Under MPI: the failure was agreed, so every rank fails the same check and + exits nonzero, and the launcher reports it; a rank wedged past + `MPI_DEADLINE_S` is ended by faulthandler. + """ + global RANK, WORLD, NODES, COMM, TWOSHOT_ABOVE, op, validating, assert_within_ulp + if mpi: + faulthandler.dump_traceback_later(MPI_DEADLINE_S, exit=True) + # A node's environment can hold a rendezvous address of its own (container hooks + # derive MASTER_ADDR/MASTER_PORT/RANK/WORLD_SIZE per node from the Slurm job), and a + # launcher need not forward one node's to the other: ranks that kept it waited on a + # store their siblings never joined. Every rank drops it, so all of them take + # `ensure_process_group`'s MPI case, rank 0's address broadcast over COMM_WORLD. + for key in ("MASTER_ADDR", "MASTER_PORT", "RANK", "WORLD_SIZE", "LOCAL_RANK"): + os.environ.pop(key, None) + from mpi4py import MPI + + COMM = MPI.COMM_WORLD + rank, world = COMM.Get_rank(), COMM.Get_size() + local = int(os.environ["OMPI_COMM_WORLD_LOCAL_RANK"]) + hosts = COMM.allgather(socket.gethostname()) + NODES = len(set(hosts)) + else: + os.environ.setdefault("TLLM_DISABLE_MPI", "1") + rank = local = int(os.environ["RANK"]) + world = int(os.environ["WORLD_SIZE"]) + hosts = [socket.gethostname()] * world + torch.cuda.set_device(local) + + from tensorrt_llm._torch.modeling_v2.models.mistral_large4.kernels import _lamport + from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.comm.cute_dsl_allreduce_add_rmsnorm import ( # noqa: E501 + cute_dsl_allreduce_add_rmsnorm, + ) + + # This file runs as a rank process, not under pytest, so the repo's + # `__extra_import_path__` is not in play: reach the shared helper the + # plain way (tests/unittest/_torch/modeling_v2). Confined to these rank + # processes. + sys.path.insert(0, str(Path(__file__).resolve().parents[4])) + from _validating import assert_within_ulp as _within + from _validating import validating as _validating_cm + + RANK, WORLD = rank, world + want = MPI_WORLD_SIZE if mpi else WORLD_SIZE + assert WORLD == want, f"this job drives the world-{want} cells, got {WORLD} ranks" + if RANK == 0: + dev = torch.cuda.current_device() + print( + f"[rank 0] {torch.cuda.get_device_name(dev)} " + f"sm_{''.join(map(str, torch.cuda.get_device_capability(dev)))}; " + f"{WORLD} ranks on {NODES} node(s): {hosts}", + flush=True, + ) + # Spawned: the group comes up from the address the launcher exported. + # Under MPI: no address is exported, so rank 0's is broadcast over + # MPI.COMM_WORLD, as the engine's ranks bring it up. + _lamport.ensure_process_group(rank, world) + assert dist.is_initialized() and dist.get_world_size() == world + op = cute_dsl_allreduce_add_rmsnorm + TWOSHOT_ABOVE = _lamport.TWOSHOT_ABOVE + validating, assert_within_ulp = _validating_cm, _within + for max_rows in sorted({op.CELLS[i].spec["max_rows"] for i in _mine()}): + t0 = time.monotonic() + free = torch.cuda.mem_get_info()[0] + ws = _lamport.build_workspace(HIDDEN, EPS, max_rows, rank, world) + WORKSPACES[max_rows] = (ws, _Mirror(ws)) + torch.cuda.synchronize() + if RANK == 0: + print( + f"[rank 0] built the {max_rows}-row workspace in {time.monotonic() - t0:.1f}s, " + f"{(free - torch.cuda.mem_get_info()[0]) / 2**30:.2f} GiB of device memory " + f"taken; buffer {tuple(ws.buffer.shape)}", + flush=True, + ) + _count_entry_calls() + + for check in CHECKS: + t0 = time.monotonic() + try: + check() + except BaseException: + print(f"[rank {RANK}] FAILED {check.__name__}", flush=True) + traceback.print_exc() + sys.stdout.flush() + sys.stderr.flush() + # Exit hard rather than return: a rank that leaves a collective + # early wedges every sibling in it. The spawned launcher kills the + # rest the moment it sees this nonzero exit; under MPI every rank + # agreed the failure and exits here too. + os._exit(13) + if RANK == 0: + print(f"[rank 0] {check.__name__}: {time.monotonic() - t0:.1f}s", flush=True) + dist.barrier() + torch.cuda.synchronize() + print( + f"[rank {RANK}] {len(CHECKS)} checks passed; peak allocated " + f"{torch.cuda.max_memory_allocated() / 2**30:.2f} GiB", + flush=True, + ) + sys.stdout.flush() + if mpi: + faulthandler.cancel_dump_traceback_later() + # mpirun fails a rank that exits without finalizing. + dist.destroy_process_group() + COMM.Barrier() + MPI.Finalize() + # Exit without tearing the symmetric memory down: these are throwaway + # processes their launcher owns. + os._exit(0) + + +def _spawn_ranks() -> None: + """Start one worker process per claimed device; no external launcher. + + The workers find each other through the address and port exported here. + Each child gets its own session; on the first nonzero exit or on the + deadline, every session is killed, which is what keeps one wedged + collective from outliving the run. + """ + visible = os.environ.get("CUDA_VISIBLE_DEVICES") + assert visible, ( + "set CUDA_VISIBLE_DEVICES to the devices this run owns, " + "e.g. export CUDA_VISIBLE_DEVICES=0,1,2,3" + ) + world_size = len([d for d in visible.split(",") if d.strip()]) + assert world_size == WORLD_SIZE, ( + f"CUDA_VISIBLE_DEVICES names {world_size} device(s); this job needs {WORLD_SIZE}" + ) + with socket.socket() as s: + s.bind(("127.0.0.1", 0)) + port = s.getsockname()[1] + env = dict( + os.environ, + MASTER_ADDR="127.0.0.1", + MASTER_PORT=str(port), + WORLD_SIZE=str(world_size), + TLLM_DISABLE_MPI="1", + ) + # Drop a launching srun step's PMIx/PMI/OMPI namespace from the children: + # inherited, it registers them as participants of that step, and the first + # hard exit trips srun's errhandler, which kills the whole step before + # anything is reported. These ranks rendezvous through torch.distributed. + env = {k: v for k, v in env.items() if not k.startswith(("PMIX_", "PMI_", "OMPI_", "SLURM_"))} + print(f"[launcher] spawning {world_size} ranks, rendezvous on :{port}", flush=True) + children = [ + subprocess.Popen( + [sys.executable, str(Path(__file__).resolve()), _WORKER_FLAG], + env=dict(env, RANK=str(r)), + start_new_session=True, + ) + for r in range(world_size) + ] + + def _kill_all() -> None: + for p in children: + try: + os.killpg(p.pid, signal.SIGKILL) + except ProcessLookupError: + pass + for p in children: + p.wait() + + deadline = time.monotonic() + DEADLINE_S + live = set(range(world_size)) + failure = None + while live and failure is None: + if time.monotonic() > deadline: + failure = f"did not finish in {DEADLINE_S}s (wedged)" + break + for r in sorted(live): + code = children[r].poll() + if code is None: + continue + live.discard(r) + if code != 0: + failure = f"rank {r} exited {code}" + break + time.sleep(0.5) + if failure is not None: + _kill_all() + raise AssertionError(f"the {world_size}-rank run failed: {failure}") + + +if __name__ == "__main__": + if _MPI_WORKER_FLAG in sys.argv: + sys.exit(_run_one_rank(mpi=True)) + if _WORKER_FLAG in sys.argv: + sys.exit(_run_one_rank(mpi=False)) + _spawn_ranks() + print("OK") diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/comm/test_modeling_v2_cute_dsl_allreduce_add_rmsnorm.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/comm/test_modeling_v2_cute_dsl_allreduce_add_rmsnorm.py new file mode 100644 index 000000000000..6fb82eccf4a7 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/comm/test_modeling_v2_cute_dsl_allreduce_add_rmsnorm.py @@ -0,0 +1,103 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Collected entry point for the `cute_dsl_allreduce_add_rmsnorm` entry. + +The matrix is `_cute_dsl_allreduce_add_rmsnorm_op_matrix.py` beside this file, +which drives the entry's `CELLS` in two jobs, one test each: + +- the world-4 cells: its own 4-rank launcher over 4 devices of one node, + which `_rank_job` starts in a fresh interpreter and whose exit code it turns + into an assertion; +- the world-8 cells (Mistral Large 4 NVFP4 tep8 and tp8): 8 ranks over two nodes + of 4 devices, one rank per device, started by an MPI launcher as the + engine's ranks are. A process cannot place ranks on another node by itself, + so this test takes that launcher from `MODELING_V2_MPI_LAUNCH` -- a command + prefix that starts the command after it once per rank over the job's two + nodes, four per node -- and skips without it, as on the one 4-GPU node + pre-merge runs this file on. On a two-node compute session whose node-0 + scheduler holds both nodes for a job, that is + `bash /scripts/mn/mn_mpirun.sh 4`. +""" + +import os +import shlex +import signal +import subprocess +import sys +from pathlib import Path + +import pytest +import torch + +__extra_import_path__ = ["../../../../comm"] +import _rank_job # noqa: E402 — needs the path declared above + +_HERE = Path(__file__).resolve().parent +_MATRIX = _HERE / "_cute_dsl_allreduce_add_rmsnorm_op_matrix.py" +#: Names the launcher that starts the world-8 cells' ranks. +_LAUNCH_ENV = "MODELING_V2_MPI_LAUNCH" +#: Headroom over the ranks' own deadline, so their faulthandler dump wins the race. +_LAUNCHER_GRACE_S = 300 +# The engine's ranks run with MPI on; and the launcher must not join an enclosing +# MPI job or Slurm step. (Each rank drops any rendezvous address itself: a node's +# own environment can hold one this process never sees.) +_DROPPED = ("TLLM_DISABLE_MPI",) +_DROPPED_PREFIXES = ("PMIX_", "PMI_", "OMPI_", "SLURM_", "PRTE_") + + +# Spawns 4 rank processes over the visible devices. Under xdist several +# workers would fight for the same GPUs, so this must run alone, like the +# other collective entries. +def _sm103_devices() -> int: + """How many of the devices the job would claim are sm_103.""" + if not torch.cuda.is_available(): + return 0 + claimed = range(min(torch.cuda.device_count(), _rank_job.WORLD_SIZE)) + return sum(torch.cuda.get_device_capability(i) == (10, 3) for i in claimed) + + +@pytest.mark.no_xdist +def test_cute_dsl_allreduce_add_rmsnorm_op_matrix() -> None: + if _sm103_devices() < _rank_job.WORLD_SIZE: + pytest.skip(f"the Lamport all-reduce entry runs on {_rank_job.WORLD_SIZE} sm_103 GPUs") + _rank_job.run("cute_dsl_allreduce_add_rmsnorm", directory=_HERE) + + +@pytest.mark.no_xdist +def test_cute_dsl_allreduce_add_rmsnorm_world8_mpi_ranks() -> None: + """The world-8 cells: the matrix's MPI job, under the environment's launcher.""" + launch = os.environ.get(_LAUNCH_ENV, "").strip() + if not launch: + pytest.skip( + f"the world-8 cells need 8 ranks over two nodes, which this process cannot " + f"start; set {_LAUNCH_ENV} to a launcher that starts them" + ) + constants = _rank_job._load_launcher_constants(_MATRIX) + # tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/comm -> repo root. + repo_root = _HERE.parents[7] + assert (repo_root / "tensorrt_llm").is_dir(), f"expected the repo root at {repo_root}" + env = { + k: v + for k, v in os.environ.items() + if k not in _DROPPED and not k.startswith(_DROPPED_PREFIXES) + } + env["PYTHONPATH"] = os.pathsep.join(p for p in (str(repo_root), env.get("PYTHONPATH", "")) if p) + command = [*shlex.split(launch), sys.executable, str(_MATRIX), "--mpi-rank-worker"] + print(f"[launcher] {shlex.join(command)}", flush=True) + # The launcher's own session, so a wedged job is ended as a whole: mpirun + # takes SIGTERM down to every rank it started, on every node. + job = subprocess.Popen(command, env=env, start_new_session=True) + try: + code = job.wait(timeout=constants.MPI_DEADLINE_S + _LAUNCHER_GRACE_S) + except subprocess.TimeoutExpired: + os.killpg(job.pid, signal.SIGTERM) + try: + job.wait(timeout=60) + except subprocess.TimeoutExpired: + os.killpg(job.pid, signal.SIGKILL) + job.wait() + raise AssertionError( + f"the world-8 job did not finish in " + f"{constants.MPI_DEADLINE_S + _LAUNCHER_GRACE_S}s (wedged)" + ) from None + assert code == 0, f"the world-8 job exited {code}: {shlex.join(command)}" diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/comm/test_modeling_v2_lamport_process_group.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/comm/test_modeling_v2_lamport_process_group.py new file mode 100644 index 000000000000..34fc0b31d044 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/comm/test_modeling_v2_lamport_process_group.py @@ -0,0 +1,111 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""CPU regressions for the Lamport process group's cross-node rendezvous.""" + +import os +import socket +import sys +from types import SimpleNamespace +from unittest.mock import MagicMock, Mock + +import pytest +import torch +import torch.distributed as dist + +from tensorrt_llm._torch.disaggregation.native import utils as native_utils +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.kernels import _lamport + +pytestmark = pytest.mark.cpu_only + + +@pytest.fixture +def process_group(monkeypatch): + for name in ("MASTER_ADDR", "MASTER_PORT", "RANK", "WORLD_SIZE"): + monkeypatch.setenv(name, "") + monkeypatch.setattr(dist, "is_initialized", lambda: False) + monkeypatch.setattr(torch.cuda, "current_device", lambda: 2) + initialize = Mock() + monkeypatch.setattr(dist, "init_process_group", initialize) + return initialize + + +def _mpi_comm(monkeypatch, rank, address=None): + comm = Mock() + comm.Get_size.return_value = 8 + comm.Get_rank.return_value = rank + comm.bcast.side_effect = lambda value, root: value if rank == root else address + monkeypatch.setitem( + sys.modules, "mpi4py", SimpleNamespace(MPI=SimpleNamespace(COMM_WORLD=comm)) + ) + return comm + + +def test_mpi_root_broadcasts_routable_address_for_loopback_hostname(monkeypatch, process_group): + comm = _mpi_comm(monkeypatch, rank=0) + monkeypatch.setattr(socket, "gethostname", lambda: "rank-zero") + monkeypatch.setattr(socket, "gethostbyname", lambda name: "127.0.1.1") + listener = MagicMock() + listener.__enter__.return_value = listener + listener.getsockname.return_value = ("0.0.0.0", 23456) + route = MagicMock() + route.__enter__.return_value = route + route.getsockname.return_value = ("10.20.30.40", 54321) + monkeypatch.setattr( + socket, + "socket", + lambda family=socket.AF_INET, kind=socket.SOCK_STREAM: ( + route if kind == socket.SOCK_DGRAM else listener + ), + ) + + _lamport.ensure_process_group(rank=0, world=8) + + listener.bind.assert_called_once_with(("", 0)) + route.connect.assert_called_once_with(("10.255.255.255", 1)) + comm.bcast.assert_called_once_with(("10.20.30.40", 23456), root=0) + assert os.environ["MASTER_ADDR"] == "10.20.30.40" + assert os.environ["MASTER_PORT"] == "23456" + process_group.assert_called_once_with( + backend="nccl", rank=0, world_size=8, device_id=torch.device("cuda", 2) + ) + + +def test_explicit_rendezvous_environment_skips_discovery(monkeypatch, process_group): + for name, value in { + "MASTER_ADDR": "explicit-master.example", + "MASTER_PORT": "34567", + "RANK": "3", + "WORLD_SIZE": "8", + }.items(): + monkeypatch.setenv(name, value) + monkeypatch.setitem(sys.modules, "mpi4py", None) + discover = Mock(side_effect=AssertionError("Explicit rendezvous must not discover an address")) + monkeypatch.setattr(native_utils, "get_local_ip", discover) + + _lamport.ensure_process_group(rank=3, world=8) + + discover.assert_not_called() + assert os.environ["MASTER_ADDR"] == "explicit-master.example" + assert os.environ["MASTER_PORT"] == "34567" + process_group.assert_called_once_with( + backend="nccl", rank=3, world_size=8, device_id=torch.device("cuda", 2) + ) + + +def test_mpi_nonroot_uses_broadcast_address(monkeypatch, process_group): + comm = _mpi_comm(monkeypatch, rank=5, address=("10.20.30.40", 23456)) + discover = Mock(side_effect=AssertionError("Only rank zero chooses the rendezvous address")) + monkeypatch.setattr(native_utils, "get_local_ip", discover) + open_socket = Mock(side_effect=AssertionError("Only rank zero chooses the rendezvous port")) + monkeypatch.setattr(socket, "socket", open_socket) + + _lamport.ensure_process_group(rank=5, world=8) + + discover.assert_not_called() + open_socket.assert_not_called() + comm.bcast.assert_called_once_with(None, root=0) + assert os.environ["MASTER_ADDR"] == "10.20.30.40" + assert os.environ["MASTER_PORT"] == "23456" + process_group.assert_called_once_with( + backend="nccl", rank=5, world_size=8, device_id=torch.device("cuda", 2) + ) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/gemm/__init__.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/gemm/__init__.py new file mode 100644 index 000000000000..52a7a9daf028 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/gemm/__init__.py @@ -0,0 +1,2 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/gemm/test_modeling_v2_cute_dsl_fp8_block_scale_linear.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/gemm/test_modeling_v2_cute_dsl_fp8_block_scale_linear.py new file mode 100644 index 000000000000..edca5eac4ae7 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/gemm/test_modeling_v2_cute_dsl_fp8_block_scale_linear.py @@ -0,0 +1,51 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Local VR200 coverage of the TEP/DEP/TP K128 FP8 projections at four and eight ranks.""" + +import pytest +import torch + +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.gemm.cute_dsl_fp8_block_scale_linear import ( # noqa: E501 + cute_dsl_fp8_block_scale_linear as op, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.gemm.cute_dsl_fp8_block_scale_linear import ( # noqa: E501 + prepare_weight_scale, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.gemm.fp8_prequantized_swap_ab_gemm import ( # noqa: E501 + fp8_prequantized_swap_ab_gemm as packed_op, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.quantization.fp8_quantize_1x128_packed_ue8m0 import ( # noqa: E501 + fp8_quantize_1x128_packed_ue8m0 as quantize, +) +from tensorrt_llm.quantization.utils.fp8_utils import get_col_major_tma_aligned_packed_tensor + +__extra_import_path__ = ["../../../.."] +from _validating import assert_within_ulp, certifying # noqa: E402,F401 + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="requires a VR200/SM107 device", +) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[cell.why for cell in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest fixture + certifying(op) + m, n, k = (cell.spec[name] for name in ("m", "n", "k")) + generator = torch.Generator(device="cuda").manual_seed(cell.seed_base()) + input = torch.randn(m, k, device="cuda", dtype=torch.bfloat16, generator=generator) + weight = torch.randn(n, k, device="cuda", generator=generator).to(torch.float8_e4m3fn) + exponent = torch.randint(-6, 4, (n // 128, k // 128), device="cuda", generator=generator) + scale = torch.ldexp(torch.ones_like(exponent, dtype=torch.float32), exponent) + packed_scale = get_col_major_tma_aligned_packed_tensor(scale.repeat_interleave(128, dim=0)) + prepared = prepare_weight_scale(packed_scale, k) + actual = op(input, weight, prepared) + reference = op.reference(input, weight, prepared) + codes, input_scale = quantize.reference(input) + packed_reference = packed_op.reference( + codes, input_scale, weight, packed_scale, torch.bfloat16, True + ) + torch.testing.assert_close(reference, packed_reference, rtol=0, atol=0) + # FP8 products are exact in FP32; accumulation order can straddle a BF16 + # rounding boundary. Use the packed GEMM's four-element/two-RMS ULP band. + assert_within_ulp(actual, reference, element_ulp=4, rms_ulp=2) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/gemm/test_modeling_v2_fp8_prequantized_swap_ab_gemm.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/gemm/test_modeling_v2_fp8_prequantized_swap_ab_gemm.py new file mode 100644 index 000000000000..d49ab95d0e22 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/gemm/test_modeling_v2_fp8_prequantized_swap_ab_gemm.py @@ -0,0 +1,414 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the fp8_prequantized_swap_ab_gemm entry.""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.gemm.fp8_prequantized_swap_ab_gemm import ( # noqa: E501 + fp8_prequantized_swap_ab_gemm as op, +) +from tensorrt_llm.quantization.utils.fp8_utils import ( + resmooth_to_fp8_e8m0, + transform_sf_into_required_layout, +) + +__extra_import_path__ = ["../../../.."] +from _validating import ( # noqa: E402,F401 — needs the path declared above + ULP, + assert_within_ulp, + certifying, +) + +assert torch.cuda.is_available(), "fp8_prequantized_swap_ab_gemm requires a CUDA device" + +#: The band, in bf16 ulp of the row's largest magnitude; derived in `_compare`. +ELEMENT_ULP, RMS_ULP = 4.0, 2.0 + +#: Fills the gaps of an MN-major scale's storage: every byte 0xFF, a UE8M0 NaN, so a kernel +#: that read a gap for a valid row would put NaN in that row. +_GAP = -1 + +#: Rows per slice of a sweep's band: an fp32 slice is at most 4096 x 49152 x 4 B = 0.75 GiB. +_BAND_ROWS = 4096 + + +def _compare(out: torch.Tensor, ref: torch.Tensor) -> None: + """One GEMM's worth of accumulation-order difference, in the output's own ulp. + + Kernel and reference consume bit-identical operands, and every product is exact in + fp32 on both sides: a dequantized operand is an e4m3 code (4 significant bits) times a + power of two, so a product has at most 8. What differs is how the 6144 (8192, 24576, + 3072, 2048, 1536, 1024, 768) products are summed in fp32: DeepGEMM's block-scaled + tensor-core MMA accumulating in tensor memory, against cuBLAS SGEMM with TF32 off. The + two fp32 sums agree to within about 3e-5 of the row's scale (a float32 output measured at + most 92 fp32 ulp off at K 6144, 102 at K 8192, 245 at K 24576, 50 at K 3072, 27 at + K 1024, 20 at K 768), and each side then rounds once to + bf16. Where the two sums straddle a bf16 rounding boundary they land one bf16 spacing + apart, which is between 1 and 2 ulp of the row's scale as `ULP` counts it, so both gates + are against the row's scale + (`assert_within_ulp`): 4 ulp element-wise, twice one spacing; 2 ulp of relative RMS, + every element one spacing off. + + Measured worst case over every m in 1..9216 at the sixteen N x K, on the eager cells' own + operands: 1.39 to 1.94 ulp element-wise and 0.0095 to 0.062 RMS; over the 32768-row calls + of the windowed cells, 1.64 to 1.98 and 0.010 to 0.057. Wrong operands land + far outside, at M 1024: activation scale bytes reversed in each int32, 480-768 ulp + element-wise and 303-308 RMS; the weight scale shifted one 128-row block, 371-491 and + 284-298; one K block's activation scale doubled, 131-305 and 15.6-123; the weight's two K + halves swapped, 524-659 and 362 (at 2048x6144, 6144x1536, 6144x6144, 8192x6144, + 6144x8192, 49152x6144, 6144x24576 and the five tep8 shards; bytes reversed not at K 768, + whose six blocks fill no second int32). At the five tep8 shards also: adjacent K blocks' + activation scales swapped, 657-746 and 304-309; the last K block's activation scale + doubled, 179-277 and 31-91. + `test_the_band_refuses_misplaced_scales` pins three of them and + `test_the_band_refuses_k768s_last_block_scale_doubled` one at K 768. + """ + assert_within_ulp(out, ref, element_ulp=ELEMENT_ULP, rms_ulp=RMS_ULP) + + +def _ue8m0_exponent(x: torch.Tensor) -> torch.Tensor: + """Biased exponent of the smallest power of two >= x, for positive normal fp32 x. + + Read off the bits rather than through `log2`: `torch.log2` on CUDA is inexact at exact + powers of two, and `ceil` then moves the scale up a binade. + """ + bits = x.float().contiguous().view(torch.int32) + return ((bits >> 23) & 0xFF) + ((bits & 0x7FFFFF) != 0).to(torch.int32) + + +def _pow2(exponent: torch.Tensor) -> torch.Tensor: + return (exponent.to(torch.int32) << 23).view(torch.float32) + + +def _activation(m: int, k: int, gen: torch.Generator) -> tuple[torch.Tensor, torch.Tensor]: + """bf16 rows quantized the way `fp8_quantize_1x128_packed_ue8m0(x, False)` quantizes them. + + Per 1x128 block: the fp32 amax, floored at 1e-10; the scale 2^ceil(log2(amax / 448)); + the codes e4m3(x / scale). The codes and the unpacked scales are bitwise that op's at + K 6144, 2048 and 1536 and eleven M from 1 to 9216, 1..5, 17 and 9215 among them, and at + K 8192 and 24576 and thirteen M from 1 to 32768, and at K 768, 1024, 3072 and 6144 and + fourteen M from 1 to 32768 (measured; the op is not called here), and `_pack` gives its + strides; there, the GEMM on the op's own (codes, scale) was bitwise the GEMM on these. + Returns the codes and the [M, K/128] biased exponents; `_pack` lays + the exponents out. Block magnitudes span 2^-4..2^4, so neighbouring blocks carry + different scales and a scale applied to the wrong block shows. + """ + x = torch.randn(m, k, generator=gen, device="cuda") + spread = torch.randint(-4, 5, (m, k // 128, 1), generator=gen, device="cuda") + x = (x.view(m, k // 128, 128) * torch.exp2(spread.float())).to(torch.bfloat16).float() + exponent = _ue8m0_exponent(x.abs().amax(dim=2).clamp(min=1e-10) / 448.0) + codes = (x / _pow2(exponent)[..., None]).to(torch.float8_e4m3fn).view(m, k) + return codes, exponent + + +def _pack(exponent: torch.Tensor, empty: int = _GAP & 0xFF) -> torch.Tensor: + """[M, K/128] exponents -> int32 [M, ceil(K/512)] with strides (1, align(M, 4)). + + The quantizer's layout and allocation: the storage ends at row M of the last column, + and every gap between columns holds `_GAP`. Where K/128 is not a multiple of four (K 768, + six blocks), the last int32 of each row has empty byte slots past the last block; they + hold `empty`, by default 0xFF like the gaps (the quantizer writes 0 there). + """ + m, blocks = exponent.shape + cols = -(-blocks // 4) + m_aligned = (m + 3) // 4 * 4 + storage = torch.full(((cols - 1) * m_aligned + m,), _GAP, dtype=torch.int32, device="cuda") + packed = storage.as_strided((m, cols), (1, m_aligned)) + padded = torch.full((m, cols * 4), empty, dtype=torch.uint8, device="cuda") + padded[:, :blocks] = exponent + packed.copy_(padded.view(torch.int32)) + return packed + + +def _weight(n: int, k: int, gen: torch.Generator) -> tuple[torch.Tensor, torch.Tensor]: + """A checkpoint-like FP8 weight, prepared the way the target's loader prepares it. + + The checkpoint holds e4m3 [N, K] and `.weight_scale`, one fp32 power-of-two scale per + 128 x 128 block shaped [N/128, 1, K/128, 1], which the loader views [N/128, K/128]; the + load path is then `FP8BlockScalesLinearMethod.transform_weights`' DeepGEMM branch, + `resmooth_to_fp8_e8m0` and `transform_sf_into_required_layout(recipe=(1, 128, 128), + is_sfa=False)`. Block magnitudes span 2^-3..2^4 and the draw is cut at 5 sigma, so a + block's scale is 2^(spread - 12) and the scales cover 2^-15..2^-8, the range the + checkpoint's 404 block scales span; asserted. + """ + w = torch.randn(n, k, generator=gen, device="cuda").clamp(-5.0, 5.0) * 0.02 + spread = torch.randint(-3, 5, (n // 128, 1, k // 128, 1), generator=gen, device="cuda") + blocks = w.view(n // 128, 128, k // 128, 128) * torch.exp2(spread.float()) + amax = blocks.abs().amax(dim=(1, 3), keepdim=True).clamp(min=1e-4) + checkpoint_scale = _pow2(_ue8m0_exponent(amax / 448.0)) + codes = (blocks / checkpoint_scale).to(torch.float8_e4m3fn).view(n, k) + codes, scale = resmooth_to_fp8_e8m0(codes, checkpoint_scale.view(n // 128, k // 128)) + drawn = _ue8m0_exponent(scale) - 127 + assert (drawn.min().item(), drawn.max().item()) == (-15, -8), "block scales off range" + weight_scale = transform_sf_into_required_layout( + scale, mn=n, k=k, recipe=(1, 128, 128), is_sfa=False + ) + assert weight_scale.dtype == torch.int32 and weight_scale.stride() == (1, n) + return codes, weight_scale + + +def _flags(spec) -> tuple[torch.dtype, bool]: + return spec["output_dtype"], spec["disable_ue8m0_cast"] + + +def _rows(spec) -> list[int]: + """The m a sweep drives: every m of `m_range`, or, with `m_window` w, three in each w-row + window [w j + 1, w j + w] of the range: its two edges and the interior row + w j + 2 + j % (w - 2), so that across windows the rows take every residue mod w. + """ + lo, hi = spec["m_range"] + w = spec.get("m_window") + if w is None: + return list(range(lo, hi + 1)) + assert (lo - 1) % w == 0 and hi % w == 0, "a windowed range is whole windows" + return [ + m + for j in range((lo - 1) // w, hi // w) + for m in (w * j + 1, w * j + 2 + j % (w - 2), w * j + w) + ] + + +def _sweep(spec, seed: int) -> None: + """Each m of `_rows`, its own call, on the leading m rows of one operand set. + + The call at the top of the range is gated against the reference, and every other m has + to return bitwise the leading rows of that call. Row i of the product reads only row i + of the activation and of its scale, so a correct kernel does that whatever tile layout + it picks for m, and it carries the gate to every m exactly: the per-m gates below are + `assert_within_ulp`'s two over the leading m rows, from running maxima and sums. Each + m's scale is packed for m rows, strides (1, align(m, 4)), as the quantizer returns it. + + The band is taken `_BAND_ROWS` rows at a time, `_compare` on each slice (its RMS gate per + slice is at least as strict as over the whole call), so the fp32 and fp64 copies stay a + slice's size at N 49152. The bitwise check compares int64 words and reads the verdicts + once, at the end. + """ + lo, hi = spec["m_range"] + rows = _rows(spec) + assert rows[-1] == hi + gen = torch.Generator(device="cuda").manual_seed(seed) + weight, weight_scale = _weight(spec["n"], spec["k"], gen) + codes, exponent = _activation(hi, spec["k"], gen) + scale = _pack(exponent) + full = op(codes, scale, weight, weight_scale, *_flags(spec)) + assert full.shape == (hi, spec["n"]) and full.dtype is spec["output_dtype"] + ref = op.reference(codes, scale, weight, weight_scale, *_flags(spec)) + + element = torch.empty(hi, dtype=torch.float32, device="cuda") + err, mag = (torch.empty(hi, dtype=torch.float64, device="cuda") for _ in range(2)) + for i in range(0, hi, _BAND_ROWS): + out_i, ref_i = full[i : i + _BAND_ROWS], ref[i : i + _BAND_ROWS] + _compare(out_i, ref_i) + diff, ref_i = out_i.float() - ref_i.float(), ref_i.float() + element[i : i + _BAND_ROWS] = diff.abs().amax(dim=1) / ref_i.abs().amax(dim=1).clamp_min( + 1e-9 + ) + err[i : i + _BAND_ROWS] = diff.double().pow(2).sum(dim=1) + mag[i : i + _BAND_ROWS] = ref_i.double().pow(2).sum(dim=1) + del ref, diff, ref_i + ulp = ULP[torch.bfloat16] + element = element.cummax(0).values + rms = (err.cumsum(0) / mag.cumsum(0).clamp_min(1e-18)).sqrt() + worst = int(element[lo - 1 : hi].argmax()) + lo + assert element[worst - 1] <= ELEMENT_ULP * ulp, ( + f"m={worst}: {element[worst - 1].item() / ulp:.3f} ulp element-wise exceeds {ELEMENT_ULP}" + ) + worst = int(rms[lo - 1 : hi].argmax()) + lo + assert rms[worst - 1] <= RMS_ULP * ulp, ( + f"m={worst}: relative RMS {rms[worst - 1].item() / ulp:.3f} ulp exceeds {RMS_ULP}" + ) + + words = full.view(torch.int64) + differs = torch.empty(len(rows), dtype=torch.bool, device="cuda") + for i, m in enumerate(rows): + out = op(codes[:m], _pack(exponent[:m]), weight, weight_scale, *_flags(spec)) + assert out.shape == (m, spec["n"]) + differs[i] = torch.ne(out.view(torch.int64), words[:m]).any() + differ = [rows[i] for i in differs.nonzero().flatten().tolist()] + assert not differ, ( + f"m={differ[:8]}{'...' if len(differ) > 8 else ''}: rows differ from the leading rows " + f"of the {hi}-row call; the kernel is no longer batch-invariant, and these m were not " + "gated against the reference" + ) + + +def _replays(spec, seed: int) -> None: + """Each decode batch captured in its own CUDA graph, then replayed on new operands. + + The new codes and scales are written into the captured inputs -- the scale through its + logical elements, its gaps untouched -- as into an engine's static buffers. The replay + has to equal an eager call on the same operands bit for bit (a graph that froze + something of the capture would differ), and is gated against the reference. Each + capture follows one call on the capture stream, as the engine's warm-up does: DeepGEMM + compiles a tile layout on its first call, which a capture cannot contain. + """ + gen = torch.Generator(device="cuda").manual_seed(seed) + k = spec["k"] + weight, weight_scale = _weight(spec["n"], k, gen) + stream = torch.cuda.Stream() + for m in spec["graph_m"]: + codes, exponent = _activation(m, k, gen) + a_in, s_in = codes.clone(), _pack(exponent) + stream.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(stream): + op(a_in, s_in, weight, weight_scale, *_flags(spec)) + torch.cuda.current_stream().wait_stream(stream) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=stream): + out = op(a_in, s_in, weight, weight_scale, *_flags(spec)) + codes, exponent = _activation(m, k, gen) + a_in.copy_(codes) + s_in.copy_(_pack(exponent)) + graph.replay() + eager = op(codes, _pack(exponent), weight, weight_scale, *_flags(spec)) + assert torch.equal(out, eager), f"m={m}: graph replay differs from eager" + _compare(out, op.reference(codes, s_in, weight, weight_scale, *_flags(spec))) + + +def _points(spec, seed: int) -> None: + """A few row counts at the cell's flag: gated, and bitwise the other flag's result.""" + gen = torch.Generator(device="cuda").manual_seed(seed) + weight, weight_scale = _weight(spec["n"], spec["k"], gen) + output_dtype, flag = _flags(spec) + for m in spec["m_points"]: + codes, exponent = _activation(m, spec["k"], gen) + scale = _pack(exponent) + out = op(codes, scale, weight, weight_scale, output_dtype, flag) + other = op(codes, scale, weight, weight_scale, output_dtype, not flag) + assert torch.equal(out, other), f"m={m}: disable_ue8m0_cast={flag} differs from {not flag}" + _compare(out, op.reference(codes, scale, weight, weight_scale, output_dtype, flag)) + + +def _cell_params(): + """`op.CELLS`, a cell whose spec says `large` under pytest's `large` marker: pre-merge + deselects it, and the stage that runs the rest drives it.""" + return [ + pytest.param(c, marks=pytest.mark.large) if c.spec.get("large") else c for c in op.CELLS + ] + + +@pytest.mark.parametrize("cell", _cell_params(), ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 + certifying(op) + seed = zlib.crc32(cell.why.encode()) % 2**31 + if "m_range" in cell.spec: + _sweep(cell.spec, seed) + elif "graph_m" in cell.spec: + _replays(cell.spec, seed) + else: + _points(cell.spec, seed) + + +def _operands(m=64, n=2048, k=6144, seed=11): + gen = torch.Generator(device="cuda").manual_seed(seed) + weight, weight_scale = _weight(n, k, gen) + codes, exponent = _activation(m, k, gen) + return codes, exponent, weight, weight_scale + + +def _rolled_rows(weight_scale: torch.Tensor) -> torch.Tensor: + """The weight scale shifted one 128-row block, back in its (1, N) layout.""" + return weight_scale.roll(128, 0).t().contiguous().t() + + +@pytest.mark.parametrize( + "wrong", + ["activation scale bytes reversed", "weight scale one block off", "one K block doubled"], +) +def test_the_band_refuses_misplaced_scales(wrong) -> None: + """The gate sees a scale applied to the wrong block, not only a wrong product.""" + codes, exponent, weight, weight_scale = _operands(m=1024) + ref = op.reference(codes, _pack(exponent), weight, weight_scale, torch.bfloat16, True) + scale = _pack(exponent) + if wrong == "activation scale bytes reversed": + scale = _pack(exponent.view(1024, -1, 4).flip(-1).reshape(1024, -1)) + elif wrong == "weight scale one block off": + weight_scale = _rolled_rows(weight_scale) + else: + scale = _pack(torch.cat([exponent[:, :1] + 1, exponent[:, 1:]], dim=1)) + out = op(codes, scale, weight, weight_scale, torch.bfloat16, True) + with pytest.raises(AssertionError): + _compare(out, ref) + + +def test_the_band_refuses_k768s_last_block_scale_doubled() -> None: + """At K 768 the last K block's scale shares a half-empty int32 with block 4's only; the + gate sees that one scale doubled.""" + codes, exponent, weight, weight_scale = _operands(m=1024, n=6144, k=768) + ref = op.reference(codes, _pack(exponent), weight, weight_scale, torch.bfloat16, True) + doubled = _pack(torch.cat([exponent[:, :5], exponent[:, 5:] + 1], dim=1)) + out = op(codes, doubled, weight, weight_scale, torch.bfloat16, True) + with pytest.raises(AssertionError): + _compare(out, ref) + + +@pytest.mark.parametrize("m", [1, 301, 4097]) +def test_k768s_empty_scale_slots_are_never_read(m) -> None: + """At K 768 (six K blocks, two int32 per scale row) bytes 2 and 3 of each row's second + int32 are empty. The quantizer and the weight transform write 0 there; 0xFF there, in + the activation's scale or the weight's, gives bitwise the same output, while 0xFF in a + live slot (block 5's) turns the rows into NaN. + """ + codes, exponent, weight, weight_scale = _operands(m=m, n=6144, k=768) + assert weight_scale.shape == (6144, 2) and bool((weight_scale[:, 1] >> 16 == 0).all()) + filled = weight_scale.clone() + filled[:, 1] |= -65536 # 0xFFFF0000: the two empty bytes + assert filled.stride() == weight_scale.stride() + out = op(codes, _pack(exponent, empty=0), weight, weight_scale, torch.bfloat16, True) + assert not out.isnan().any() + for a_empty, w_scale in ((0xFF, weight_scale), (0, filled), (0xFF, filled)): + again = op(codes, _pack(exponent, empty=a_empty), weight, w_scale, torch.bfloat16, True) + assert torch.equal(again, out), f"activation empty 0x{a_empty:02X}, weight empty set" + live = _pack(torch.cat([exponent[:, :5], torch.full_like(exponent[:, 5:], 255)], dim=1)) + assert op(codes, live, weight, weight_scale, torch.bfloat16, True).isnan().any(dim=1).all() + + +def test_a_row_major_activation_scale_is_restrided_inside_the_call() -> None: + """Any strides are accepted: the runner copies the scale into (1, align(M, 4)) first.""" + codes, exponent, weight, weight_scale = _operands(m=301) + scale = _pack(exponent) + row_major = scale.contiguous() + assert row_major.stride() != scale.stride() + assert torch.equal( + op(codes, row_major, weight, weight_scale, torch.bfloat16, True), + op(codes, scale, weight, weight_scale, torch.bfloat16, True), + ) + + +def test_a_row_strided_activation_reads_its_stride() -> None: + """Codes that are a column slice of a wider buffer: the TMA descriptor takes the stride.""" + codes, exponent, weight, weight_scale = _operands(m=301) + wide = torch.zeros(301, codes.shape[1] + 128, dtype=codes.dtype, device="cuda") + wide[:, : codes.shape[1]] = codes + view = wide[:, : codes.shape[1]] + assert not view.is_contiguous() + assert torch.equal( + op(view, _pack(exponent), weight, weight_scale, torch.bfloat16, True), + op(codes, _pack(exponent), weight, weight_scale, torch.bfloat16, True), + ) + + +def test_the_r128c4_scale_layout_is_refused() -> None: + """`use_r128c4_layout=True`, the quantizer's default, returns a 1-D uint8 scale of + align(M, 128) * align(K/32, 4) bytes; the runner's re-stride raises on it. + """ + codes, _, weight, weight_scale = _operands(m=16) + r128c4 = torch.zeros(128 * 192, dtype=torch.uint8, device="cuda") + with pytest.raises(RuntimeError, match=r"dimensionality of sizes \(1\) must match"): + op(codes, r128c4, weight, weight_scale, torch.bfloat16, True) + + +def test_an_fp32_activation_scale_is_refused_at_the_call_sites_flag() -> None: + """With disable_ue8m0_cast=True DeepGEMM takes int32 scales only.""" + codes, exponent, weight, weight_scale = _operands(m=16) + with pytest.raises(RuntimeError, match="Unsupported architecture or scaling factor types"): + op(codes, _pow2(exponent), weight, weight_scale, torch.bfloat16, True) + + +def test_a_weight_scale_that_is_not_mn_major_is_refused() -> None: + codes, exponent, weight, weight_scale = _operands(m=16) + with pytest.raises(RuntimeError, match=r"sf\.stride\(-2\) == 1 or mn == 1"): + op(codes, _pack(exponent), weight, weight_scale.contiguous(), torch.bfloat16, True) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/moe/__init__.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/moe/__init__.py new file mode 100644 index 000000000000..52a7a9daf028 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/moe/__init__.py @@ -0,0 +1,2 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/moe/test_modeling_v2_renorm_moe_routing_op.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/moe/test_modeling_v2_renorm_moe_routing_op.py new file mode 100644 index 000000000000..0d0f4251e104 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/moe/test_modeling_v2_renorm_moe_routing_op.py @@ -0,0 +1,535 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the renorm_moe_routing_op entry. + +The entry owns the contract. This file turns a cell into logits and holds the +claims no cell carries: the return order, the tie-break against `torch.topk`, +the sibling op one flag away, the sweep of the old certified surface, the edge +behaviour, and the guards. The Mistral Large 4 cells add three spec keys: +`graph_tokens` (each batch captured in a CUDA graph and replayed), +`tokens_range` (every row count of the range, each its own call) and +`rank_tokens` (the dep4 gathered layout: each rank's live rows zero-padded to +`tokens / len(rank_tokens)`). The dep8 cell drives `graph_tokens` at the +decode graphs' 8-rank gathered batches. +""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.moe.renorm_moe_routing_op import ( + renorm_moe_routing_op as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402,F401 — path above; fixture by name + +assert torch.cuda.is_available(), "renorm_moe_routing_op requires a CUDA device" + + +def _compare( + out: tuple[torch.Tensor, torch.Tensor], + ref: tuple[torch.Tensor, torch.Tensor], +) -> None: + """Ids equal and weights bit-exact. + + The reference is not an approximation of the kernel: over 2040 + configurations, not one id and not one weight bit differed; at 128 / top-4 + on fp32 logits, none differed at any T in 1..131072 nor at 9216, 32768 or + 131072 tokens for logits of sigma 1e-3 to 100, in either output dtype, nor + on the looping path past the 8192-block cap (T 32769..131073). A one-ulp + `assert_close` runs first only so a drift reports a readable diff. + The nearest wrong answers sit 0.4476 away (un-renormalized softmax, + `1/topk`), and a `torch.topk` reference differs on a third of the rows at + 512 bf16 experts. At 128 / top-4, fp32 logits ranked after narrowing to + bf16 move the ids of 566 of 9216 rows and 28394 of 36864 bf16 weights. + """ + ids, weights = out + ref_ids, ref_weights = ref + assert ids.dtype == ref_ids.dtype == torch.int32, (ids.dtype, ref_ids.dtype) + assert weights.dtype == ref_weights.dtype, (weights.dtype, ref_weights.dtype) + assert ids.shape == ref_ids.shape == weights.shape, (ids.shape, ref_ids.shape) + torch.testing.assert_close(ids, ref_ids, rtol=0, atol=0) + eps = torch.finfo(weights.dtype).eps + torch.testing.assert_close(weights, ref_weights, rtol=eps, atol=0.0) + mismatched = int((weights != ref_weights).sum()) + assert mismatched == 0, f"{mismatched} of {weights.numel()} weights are not bit-exact" + + +DEV = "cuda" + + +def _logits(tokens, num_experts, logits_dtype, dist="randn", magnitude=1.0, seed=0): + g = torch.Generator(device=DEV).manual_seed(seed) + if dist == "coarse": + x = torch.randint(0, 3, (tokens, num_experts), device=DEV, generator=g).float() + else: + x = torch.randn(tokens, num_experts, device=DEV, generator=g) * magnitude + return x.to(logits_dtype) + + +def _check(logits, topk, output_dtype=None): + """Drive the op and gate it through the entry's own reference and `_compare`.""" + ids, weights = op(logits, topk, output_dtype) + assert ids.is_contiguous() and weights.is_contiguous() + assert ids.device == weights.device == logits.device + _compare((ids, weights), op.reference(logits, topk, output_dtype)) + return ids, weights + + +def _bits(weights: torch.Tensor) -> torch.Tensor: + """The weights' bit patterns, so `!=` is bitwise (NaN and -0.0 included).""" + return weights.view(torch.int16 if weights.element_size() == 2 else torch.int32) + + +def _cell_logits(s, seed, tokens=None): + return _logits( + s["tokens"] if tokens is None else tokens, + s["num_experts"], + s["logits_dtype"], + s["dist"], + s.get("magnitude", 1.0), + seed=seed, + ) + + +def _words(x: torch.Tensor) -> torch.Tensor: + """`x`'s bits as int64 words where a row is whole words (at top-4: the ids + two words a row, the bf16 weights one), else as `_bits`.""" + return x.view(torch.int64) if x.shape[-1] * x.element_size() % 8 == 0 else _bits(x) + + +_SWEEP_GROUP = 16 + + +def _sweep(s, seed) -> None: + """Every T of `tokens_range`, each its own call, on the leading T rows of one set. + + The reference is taken once, at the top of the range, and sliced: a + token's routing reads only its own row, so the leading T rows of the + reference are the reference for a T-row call. The gate is `_compare`'s, + ids equal and weights bitwise, read as int64 words: the outputs of + `_SWEEP_GROUP` consecutive T are concatenated and compared with the + matching reference rows at once, the verdict flagged on the device, so + the sweep pays one synchronization and few compare launches. A flag per T + cost 1.87 s over 1..32768 and 7.03 s over 32769..131072; a flag per group + of 16, 0.88 s and 4.55 s, with the op alone 4.18 s over 32769..131072. + One flipped weight bit in one T of a group is seen. Each T of the first + failing group is re-driven through `_compare` for its diff. + """ + lo, hi = s["tokens_range"] + logits = _cell_logits(s, seed, tokens=hi) + ref_ids, ref_weights = op.reference(logits, s["topk"], s["output_dtype"]) + ref_id_words, ref_weight_words = _words(ref_ids), _words(ref_weights) + row_counts = range(lo, hi + 1) + groups = [row_counts[i : i + _SWEEP_GROUP] for i in range(0, len(row_counts), _SWEEP_GROUP)] + bad = torch.zeros(len(groups), dtype=torch.bool, device=DEV) + for g, group in enumerate(groups): + outs = [op(logits[:t], s["topk"], s["output_dtype"]) for t in group] + for t, (ids, weights) in zip(group, outs): + assert ids.shape == weights.shape == (t, s["topk"]) + assert weights.dtype is s["output_dtype"] + torch.logical_or( + torch.ne( + torch.cat([_words(ids) for ids, _ in outs]), + torch.cat([ref_id_words[:t] for t in group]), + ).any(), + torch.ne( + torch.cat([_words(weights) for _, weights in outs]), + torch.cat([ref_weight_words[:t] for t in group]), + ).any(), + out=bad[g], + ) + failing = torch.nonzero(bad).flatten() + if failing.numel(): + for t in groups[int(failing[0])]: + out = op(logits[:t], s["topk"], s["output_dtype"]) + _compare(out, (ref_ids[:t], ref_weights[:t])) + assert failing.numel() == 0, ( + f"{failing.numel()} groups of {_SWEEP_GROUP} row counts differ from the reference; " + f"first T={groups[int(failing[0])][0]}..{groups[int(failing[0])][-1]}" + ) + + +def _replays(s, seed) -> None: + """Each captured batch in its own CUDA graph, replayed on new logits. + + The replay has to equal an eager call on the same logits bit for bit -- a + graph that froze something of the capture would differ -- and is gated + against the reference like any other output. Each capture is preceded by + one call on the capture stream, as an engine's warm-up is. + """ + stream = torch.cuda.Stream() + for t in s["graph_tokens"]: + static = _cell_logits(s, seed + t, tokens=t) + stream.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(stream): + op(static, s["topk"], s["output_dtype"]) + torch.cuda.current_stream().wait_stream(stream) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=stream): + ids, weights = op(static, s["topk"], s["output_dtype"]) + for replay in range(2): + fresh = _cell_logits(s, seed + 1000 * (replay + 1) + t, tokens=t) + static.copy_(fresh) + graph.replay() + eager_ids, eager_weights = op(fresh, s["topk"], s["output_dtype"]) + assert torch.equal(ids, eager_ids) and torch.equal( + _bits(weights), _bits(eager_weights) + ), f"T={t}: graph replay {replay} differs from eager" + _compare((ids, weights), op.reference(fresh, s["topk"], s["output_dtype"])) + + +def _padded(s, seed) -> None: + """The dep4 gathered layout: rank r's `rank_tokens[r]` live rows, then + all-+0.0 rows up to T_max = tokens / ranks, the ranks' blocks in order. + + +0.0 in every column is what a zero hidden row gives through the router + GEMM. The gate is `_compare` against the reference, and every padding row + must be the exact 128-way tie's answer: ids 0..topk-1 at weight 1/topk. + """ + t_max = s["tokens"] // len(s["rank_tokens"]) + logits = torch.zeros(s["tokens"], s["num_experts"], dtype=s["logits_dtype"], device=DEV) + padding = torch.ones(s["tokens"], dtype=torch.bool, device=DEV) + for rank, count in enumerate(s["rank_tokens"]): + logits[rank * t_max : rank * t_max + count] = _cell_logits(s, seed + rank, tokens=count) + padding[rank * t_max : rank * t_max + count] = False + ids, weights = _check(logits, s["topk"], s["output_dtype"]) + assert int(padding.sum()) == s["tokens"] - sum(s["rank_tokens"]) + tie_ids = torch.arange(s["topk"], dtype=torch.int32, device=DEV) + assert bool((ids[padding] == tie_ids).all()), "a padding row is not routed to experts 0..topk-1" + assert bool((weights[padding] == 1.0 / s["topk"]).all()), "a padding weight is not 1/topk" + + +# ── the certified cells ─────────────────────────────────────────────────── + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying): # noqa: F811 — pytest resolves the fixture by this name + certifying(op) + s = cell.spec + seed = zlib.crc32(cell.why.encode()) + if "tokens_range" in s: + _sweep(s, seed) + return + if "graph_tokens" in s: + _replays(s, seed) + return + if "rank_tokens" in s: + _padded(s, seed) + return + logits = _cell_logits(s, seed) + ids, weights = _check(logits, s["topk"], s["output_dtype"]) + assert weights.dtype == s["output_dtype"] + # renormalized: every row sums to 1 + torch.testing.assert_close( + weights.float().sum(-1), + torch.ones(s["tokens"], device=DEV), + rtol=8e-3 if s["output_dtype"] is torch.bfloat16 else 1e-6, + atol=0.0, + ) + + +def test_old_certified_surface(): + """The rest of the surface the entry was certified over, through the entry's + own reference and gate: the token column in both output dtypes, every input + dtype against every output dtype, and the num_experts / topk sweep across + the kernel's power-of-two instantiation boundaries.""" + for tokens in (1, 2, 7, 8, 64, 512, 2048, 4096, 8192): + logits = _logits(tokens, 512, torch.bfloat16, seed=tokens) + for out_dtype in (torch.float32, torch.bfloat16): + _check(logits, 10, out_dtype) + base = _logits(256, 512, torch.float32, magnitude=2.0, seed=8) + for in_dtype in (torch.bfloat16, torch.float16, torch.float32): + for out_dtype in (torch.float32, torch.bfloat16, None): + _check(base.to(in_dtype), 10, out_dtype) + for num_experts in (1, 2, 8, 31, 32, 33, 64, 96, 128, 129, 256, 384, 511, 512): + _check(_logits(64, num_experts, torch.bfloat16, seed=num_experts), min(10, num_experts)) + logits = _logits(256, 512, torch.bfloat16, seed=9) + for topk in (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 16): # topk 0: test_degenerate_shapes + _check(logits, topk, torch.bfloat16) + + +# ── the traps ───────────────────────────────────────────────────────────── + + +def test_return_order_is_ids_then_weights(): + logits = _logits(256, 512, torch.bfloat16, seed=1) + first, second = op(logits, 10, torch.bfloat16) + assert first.dtype == torch.int32 and second.dtype == torch.bfloat16 + top1 = torch.sort(logits, dim=-1, descending=True, stable=True).indices[:, 0] + assert torch.equal(first[:, 0], top1.to(torch.int32)) + + +def test_equivalent_to_hf_norm_topk_prob(): + """Softmax over all experts, top-k, divide by the selected sum: the same + operator up to rounding (5.96e-08 measured); skipping the renormalization + lands three orders of magnitude outside that.""" + for dtype in (torch.bfloat16, torch.float32): + logits = _logits(1024, 512, dtype, magnitude=2.0, seed=2) + _, weights = op(logits, 10, torch.float32) + hf_values, _ = torch.topk(torch.softmax(logits.float(), dim=-1), 10, dim=-1) + hf_weights = hf_values / hf_values.sum(-1, keepdim=True) + # ids can differ on tie rows; the sorted weight multiset cannot + torch.testing.assert_close( + weights.sort(-1, descending=True).values, + hf_weights.sort(-1, descending=True).values, + rtol=1e-5, + atol=1e-7, + ) + assert (weights - hf_values).abs().max().item() > 1e-2 + + +def test_top4_of_128_matches_the_softmax_over_all_experts(): + """Mistral Large 4's routing as its reference model states it -- an fp32 + softmax over all 128 experts, the top 4 probabilities, divided by their + sum -- against the op's top-4-then-softmax, at the target's values. + + Both exponentiate the same differences from the row's largest logit, so + the two differ only in how the division is split: measured at most + 2.4e-07 relative in fp32 (about 2 ulp; the gate allows 8), and the bf16 + weights bitwise equal on these 9216 rows (the gate allows one bf16 + rounding flip). The same probabilities left un-renormalized sit more than + 1e-2 away. + """ + logits = _logits(9216, 128, torch.float32, magnitude=3.0, seed=14) + probs, prob_ids = torch.topk(torch.softmax(logits, dim=-1), 4, dim=-1) + renormalized = probs / probs.sum(-1, keepdim=True) + ids, weights = op(logits, 4, torch.float32) + assert torch.equal(ids, prob_ids.to(torch.int32)) + torch.testing.assert_close(weights, renormalized, rtol=8 * 2.0**-23, atol=0.0) + assert (weights - probs).abs().max().item() > 1e-2 + bf16_ids, bf16_weights = op(logits, 4, torch.bfloat16) + assert torch.equal(bf16_ids, ids) + torch.testing.assert_close( + bf16_weights.float(), renormalized.to(torch.bfloat16).float(), rtol=2.0**-8, atol=0.0 + ) + + +def test_distinct_from_default_moe_routing_op(): + """The sibling one flag away: same ids, weights that do not sum to 1.""" + logits = _logits(512, 512, torch.bfloat16, seed=3) + ids, weights = op(logits, 10, torch.float32) + other_ids, other_weights = torch.ops.trtllm.default_moe_routing_op(logits, 10, torch.float32) + assert torch.equal(ids, other_ids) + row_sums = other_weights.sum(-1) + assert float(row_sums.max()) < 0.99 + assert (weights - other_weights).abs().max().item() > 1e-2 + torch.testing.assert_close( + other_weights / row_sums.unsqueeze(-1), weights, rtol=1e-6, atol=1e-7 + ) + + +def test_ties_break_toward_lower_expert_index(): + """Opposite to torch.topk, and common at 512 bf16 experts.""" + logits = torch.zeros(4, 64, dtype=torch.bfloat16, device=DEV) + logits[0, :] = 1.0 + logits[1, 3] = logits[1, 7] = 5.0 + logits[2, ::2] = 2.0 + logits[3, 0] = logits[3, 63] = 1.0 + ids, _ = _check(logits, 4, torch.float32) + expected = [[0, 1, 2, 3], [3, 7, 0, 1], [0, 2, 4, 6], [0, 63, 1, 2]] + assert ids.tolist() == expected + + dense = _logits(4096, 512, torch.bfloat16, seed=4) + ids, _ = _check(dense, 10, torch.float32) + torch_topk_ids = torch.topk(dense, 10, dim=-1).indices.to(torch.int32) + disagreeing = int((ids != torch_topk_ids).any(-1).sum()) + selected = torch.gather(dense, 1, ids.long()) + tied = int((selected[:, :-1] == selected[:, 1:]).any(-1).sum()) + print(f" {tied} of 4096 rows tie inside the top-10; torch.topk disagrees on {disagreeing}") + assert disagreeing > 0 and tied > dense.shape[0] // 4 + for tokens, num_experts, topk in ((512, 128, 8), (256, 32, 4)): + _check(_logits(tokens, num_experts, torch.bfloat16, "coarse", seed=5), topk, torch.float32) + + +def test_sign_bits_rank_by_their_bits(): + """Where the op and `reference` part: the kernel orders logits by their + bit pattern, torch's sort by their value. + + +0.0 ranks above -0.0, where torch holds the two zeros equal and keeps the + lower index first; the weights agree, since both zeros exponentiate to 1. + A NaN with its sign bit set ranks below -inf, so the op skips it and + answers the rest of the row as if it held -1e30, where torch ranks it + first and answers NaN. + """ + logits = torch.full((2, 128), -0.0, dtype=torch.float32, device=DEV) + logits[0, 9] = 0.0 + logits[1] = 0.0 + ids, weights = op(logits, 4, torch.bfloat16) + ref_ids, ref_weights = op.reference(logits, 4, torch.bfloat16) + assert ids.tolist() == [[9, 0, 1, 2], [0, 1, 2, 3]] + assert ref_ids.tolist() == [[0, 1, 2, 3], [0, 1, 2, 3]] + assert torch.equal(weights, ref_weights) and bool((weights == 0.25).all()) + + row = _logits(1, 128, torch.float32, magnitude=3.0, seed=18) + masked = row.clone() + masked[0, 5] = -1e30 + row.view(torch.int32)[0, 5] = -1 # 0xffffffff: a NaN with its sign bit set + assert bool(torch.isnan(row[0, 5])) + ids, weights = op(row, 4, torch.bfloat16) + masked_ids, masked_weights = op(masked, 4, torch.bfloat16) + assert torch.equal(ids, masked_ids) and torch.equal(weights, masked_weights) + assert 5 not in ids.tolist()[0] and bool(torch.isfinite(weights).all()) + ref_ids, ref_weights = op.reference(row, 4, torch.bfloat16) + assert int(ref_ids[0, 0]) == 5 and bool(torch.isnan(ref_weights).all()) + + +def test_slot_order_and_weight_monotonicity(): + logits = _logits(512, 512, torch.bfloat16, seed=6) + ids, weights = _check(logits, 10, torch.float32) + selected = torch.gather(logits.float(), 1, ids.long()) + assert bool((selected[:, :-1] >= selected[:, 1:]).all()) + assert bool((weights[:, :-1] >= weights[:, 1:]).all()) + + +def test_output_dtype_defaults_to_fp32_and_casts_after_the_softmax(): + logits = _logits(1024, 512, torch.float32, magnitude=3.0, seed=7) + default_ids, default_weights = op(logits, 10) + assert default_weights.dtype == torch.float32 + explicit_ids, explicit_weights = op(logits, 10, torch.float32) + assert torch.equal(default_ids, explicit_ids) and torch.equal(default_weights, explicit_weights) + _, bf16_weights = _check(logits, 10, torch.bfloat16) + order = torch.sort(logits, dim=-1, descending=True, stable=True).indices[:, :10] + narrowed = torch.gather(logits, 1, order).to(torch.bfloat16).float() + e = torch.exp(narrowed - narrowed.max(-1, keepdim=True).values) + cast_first = (e / e.sum(-1, keepdim=True)).to(torch.bfloat16) + differing = int((bf16_weights != cast_first).sum()) + print(f" cast-before-softmax disagrees on {differing} of {bf16_weights.numel()}") + assert differing > bf16_weights.numel() // 10 + + +# ── edges ───────────────────────────────────────────────────────────────── + + +def test_degenerate_shapes(): + ids, weights = op(_logits(8, 512, torch.bfloat16), 0, torch.float32) + assert ids.shape == (8, 0) and weights.shape == (8, 0) + ids, weights = op(torch.empty(0, 512, dtype=torch.bfloat16, device=DEV), 10, torch.float32) + assert ids.shape == (0, 10) and weights.shape == (0, 10) + assert ids.dtype == torch.int32 and weights.dtype == torch.float32 + + +def test_input_not_mutated_and_deterministic(): + logits = _logits(4096, 512, torch.bfloat16, seed=10) + snapshot = logits.clone() + ids_a, weights_a = op(logits, 10, torch.bfloat16) + ids_b, weights_b = op(logits, 10, torch.bfloat16) + assert torch.equal(logits, snapshot) + assert torch.equal(ids_a, ids_b) and torch.equal(weights_a, weights_b) + assert weights_a.data_ptr() not in (weights_b.data_ptr(), logits.data_ptr()) + stream = torch.cuda.Stream() + with torch.cuda.stream(stream): + ids_c, weights_c = op(logits, 10, torch.bfloat16) + stream.synchronize() + assert torch.equal(ids_a, ids_c) and torch.equal(weights_a, weights_c) + + +def test_non_finite_logits_poison_the_whole_row(): + logits = _logits(3, 512, torch.float32) + logits[0, 5], logits[1, 7] = float("inf"), float("nan") + logits[2, :] = float("-inf") + ids, weights = op(logits, 10, torch.float32) + assert bool(torch.isnan(weights).all()) + assert int(ids[0, 0]) == 5 and int(ids[1, 0]) == 7 + # -inf is a soft mask: excluded while topk fits in the finite experts... + masked = _logits(4, 512, torch.float32, seed=1) + masked[:, :8] = float("-inf") + masked_ids, masked_weights = op(masked, 10, torch.float32) + assert bool(torch.isfinite(masked_weights).all()) and int(masked_ids.min()) >= 8 + # ...and at weight exactly 0 once it does not + starved = _logits(4, 512, torch.float32, seed=2) + starved[:, 7:] = float("-inf") + starved_ids, starved_weights = op(starved, 10, torch.float32) + assert bool(torch.isfinite(starved_weights).all()) + assert bool((starved_weights[:, 7:] == 0).all()) and bool((starved_weights[:, :7] > 0).all()) + assert int(starved_ids.max()) < 512 + + +def test_garbage_rows_keep_their_ids_in_range_and_to_themselves(): + """A padded 16-row batch whose last rows hold anything at all. + + Whatever a padding row holds -- random bit patterns, NaN, -inf -- its ids + are distinct experts in [0, 128), so an expert runner downstream never + indexes outside its table, and the live rows are bitwise what they are + without the padding. + """ + live = _logits(13, 128, torch.float32, magnitude=3.0, seed=15) + g = torch.Generator(device=DEV).manual_seed(16) + raw = torch.randint(-(2**31), 2**31, (128,), generator=g, device=DEV, dtype=torch.int64) + batch = torch.cat( + [ + live, + raw.to(torch.int32).view(torch.float32).unsqueeze(0), + torch.full((1, 128), float("nan"), device=DEV), + torch.full((1, 128), float("-inf"), device=DEV), + ] + ) + ids, weights = op(batch, 4, torch.bfloat16) + assert int(ids.min()) >= 0 and int(ids.max()) < 128 + assert bool((ids.sort(-1).values.diff(dim=-1) > 0).all()), "a row repeats an expert" + live_ids, live_weights = _check(live, 4, torch.bfloat16) + assert torch.equal(ids[:13], live_ids) and torch.equal(_bits(weights[:13]), _bits(live_weights)) + + +def test_a_row_window_is_read_at_its_offset(): + """A contiguous run of rows inside a larger buffer: a nonzero storage + offset, which `data_ptr()` carries, and the guard admits.""" + logits = _logits(64, 128, torch.float32, magnitude=3.0, seed=17) + window = logits[17:40] + assert window.is_contiguous() and window.storage_offset() == 17 * 128 + with validating(op): + _check(window, 4, torch.bfloat16) + + +def test_op_rejects_unsupported_domains(): + """Loud on the op's own; none of these needs a guard.""" + logits = _logits(8, 512, torch.bfloat16) + cases = [ + ("1D router_logits", lambda: op(logits[0], 6)), + ("3D router_logits", lambda: op(logits.unsqueeze(0), 6)), + ("fp64 router_logits", lambda: op(logits.double(), 6)), + ("int32 router_logits", lambda: op(logits.int(), 6)), + ("fp16 output_dtype", lambda: op(logits, 6, torch.float16)), + ("fp16 output_dtype, fp16 logits", lambda: op(logits.half(), 6, torch.float16)), + ("fp64 output_dtype", lambda: op(logits, 6, torch.float64)), + ("int32 output_dtype", lambda: op(logits, 6, torch.int32)), + ("topk 17", lambda: op(logits, 17)), + ("negative topk", lambda: op(logits, -1)), + ("num_experts 513", lambda: op(_logits(8, 513, torch.bfloat16), 6)), + ("cpu router_logits", lambda: op(logits.cpu(), 6)), + ] + for tag, call in cases: + with pytest.raises((RuntimeError, ValueError, NotImplementedError)): + call() + + +# ── the guards ──────────────────────────────────────────────────────────── + + +def test_unguarded_hazards_are_silent(): + """What `is_valid` exists for: the raw op answers both of these.""" + wide = _logits(8, 1024, torch.bfloat16, seed=12) + sliced = wide[:, :512] + dense = wide.reshape(-1)[: 8 * 512].reshape(8, 512).contiguous() + strided_ids, _ = op(sliced, 10, torch.float32) + assert torch.equal(strided_ids, op(dense, 10, torch.float32)[0]), ( + "the strided read is no longer a dense reinterpretation" + ) + assert not torch.equal(strided_ids, op.reference(sliced, 10, torch.float32)[0]) + narrow = _logits(4, 8, torch.bfloat16, seed=13) + padded_ids, padded_weights = op(narrow, 12, torch.float32) + assert bool((padded_ids[:, 8:] == 0).all()) and bool((padded_weights[:, 8:] == 0).all()) + + +def test_guards_refuse(): + wide = _logits(8, 1024, torch.bfloat16, seed=12) + with validating(op): + with pytest.raises(AssertionError, match="contiguous"): + op(wide[:, :512], 10) + with pytest.raises(AssertionError, match="contiguous"): + op(wide.t(), 6) + with pytest.raises(AssertionError, match="exceeds num_experts"): + op(_logits(4, 8, torch.bfloat16), 12) + _check(wide[:, :512].contiguous(), 10, torch.float32) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/norm/__init__.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/norm/__init__.py new file mode 100644 index 000000000000..52a7a9daf028 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/norm/__init__.py @@ -0,0 +1,2 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/norm/test_modeling_v2_layer_norm.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/norm/test_modeling_v2_layer_norm.py new file mode 100644 index 000000000000..3bf265fb8abb --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/norm/test_modeling_v2_layer_norm.py @@ -0,0 +1,448 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the layer_norm entry. + +The entry owns what is certified: `CELLS` is the list and `reference` is what +the kernel is gated against. This file owns what a cell cannot carry -- +turning a spec into real tensors, and the band -- plus the measurements the +`note` states. + +Weight and bias are drawn over the call site's own checkpoint tensors' +ranges. At the indexer site (44 q_input_norm tensors) the affine is near unit, +and the input carries the spread. +""" + +import zlib + +import pytest +import torch +import torch.nn.functional as F + +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.norm.layer_norm import ( + layer_norm as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402,F401 + +assert torch.cuda.is_available(), "layer_norm requires a CUDA device" + +DEVICE = torch.device("cuda") +HIDDEN = 6144 +EPS = 1e-6 + +#: Over layers.*.attention.indexer.q_input_norm.{weight,bias}, all 44 indexer layers (bf16). +Q_WEIGHT_LO, Q_WEIGHT_HI = 0.890625, 1.140625 +Q_BIAS_LO, Q_BIAS_HI = -0.08935547, 0.08935547 +#: The same 44 tensors on the NVFP4 checkpoint, another training snapshot: the +#: weights sit inside the range above, the bias reaches one bf16 step past it. +#: Its attention_norm weights (the hidden-state model below) measured the same +#: range and percentiles as the FP8 checkpoint's. +NVFP4_Q_WEIGHT_LO, NVFP4_Q_WEIGHT_HI = 0.89453125, 1.1328125 +NVFP4_Q_BIAS_LO, NVFP4_Q_BIAS_HI = -0.08740234, 0.08984375 + +#: The indexer's input is the layer's attention-normed hidden state: an +#: RMS-normalized row times attention_norm's weight, whose magnitudes over the +#: checkpoint run from about 0.01 to 2.4 on nearly every column and reach 14.3 +#: on a handful. Row r is further scaled by _ROW_SCALES[r % 4], so its +#: variance is about 1e-6, 1, 1e-2 or 1e-4 times the hidden state's own (which +#: is 1.9-3.1 here). At the hidden state's own variance eps moves the result +#: by about a millionth: a reference at eps 1e-5, 1e-7 or 0 was measured inside +#: the gate over 9216 such rows. At 1e-6 of it eps decides the answer. Row 0 +#: is the small one, so a one-row call sees eps too. +_ROW_SCALES = (1e-3, 1.0, 1e-1, 1e-2) + +#: Significand bits (the implicit one included) and the exponent of the +#: smallest subnormal, per output dtype. +_FORMAT = {torch.bfloat16: (8, -133), torch.float16: (11, -24), torch.float32: (24, -149)} + + +def _gen(seed: int) -> torch.Generator: + return torch.Generator(device=DEVICE).manual_seed(seed) + + +def _affine(gen, width=HIDDEN, dtype=torch.bfloat16, checkpoint="fp8"): + if checkpoint == "nvfp4": + (w_lo, w_hi) = (NVFP4_Q_WEIGHT_LO, NVFP4_Q_WEIGHT_HI) + (b_lo, b_hi) = (NVFP4_Q_BIAS_LO, NVFP4_Q_BIAS_HI) + else: + (w_lo, w_hi), (b_lo, b_hi) = (Q_WEIGHT_LO, Q_WEIGHT_HI), (Q_BIAS_LO, Q_BIAS_HI) + weight = torch.rand(width, generator=gen, device=DEVICE) * (w_hi - w_lo) + w_lo + bias = torch.rand(width, generator=gen, device=DEVICE) * (b_hi - b_lo) + b_lo + return weight.to(dtype), bias.to(dtype) + + +def _hidden_rows(rows, width, gen): + """Rows shaped like the attention-normed hidden state, row r scaled by `_ROW_SCALES[r % 4]`.""" + columns = torch.rand(width, generator=gen, device=DEVICE) * (2.5 - 0.01) + 0.01 + columns[torch.randperm(width, generator=gen, device=DEVICE)[:8]] = 14.3125 + scale = torch.tensor(_ROW_SCALES, device=DEVICE).repeat((rows + 3) // 4)[:rows, None] + return torch.randn(rows, width, generator=gen, device=DEVICE) * columns * scale + + +def _build(spec, seed, rows=None): + """One cell's operands, at `rows` rows (the cell's own by default).""" + gen = _gen(seed) + shape, dtype = spec["shape"], spec["dtype"] + if rows is not None: + shape = (rows, *shape[1:]) + assert spec["site"] == "indexer_q", spec["site"] + x = _hidden_rows(shape[0], shape[-1], gen).to(dtype) + weight, bias = _affine( + gen, width=shape[-1], dtype=dtype, checkpoint=spec.get("checkpoint", "fp8") + ) + return x, weight, bias, spec.get("eps", EPS) + + +def _spacing(t): + """The gap from each element of `t` to the next representable value of its dtype, as fp64.""" + bits, tiny = _FORMAT[t.dtype] + _, exponent = torch.frexp(t.double()) + exponent = torch.where( + t == 0, torch.full_like(exponent, tiny), (exponent - bits).clamp_min(tiny) + ) + return torch.ldexp(torch.ones_like(exponent, dtype=torch.float64), exponent) + + +def _terms(x, weight, bias, eps): + """Per element, the magnitudes the kernel's fp32 arithmetic rounds (fp64, from the inputs).""" + xd = x.double() + mean = xd.mean(dim=-1, keepdim=True) + rstd = torch.rsqrt((xd - mean).square().mean(dim=-1, keepdim=True) + eps) + terms = ((xd - mean).abs() + xd.abs().mean(dim=-1, keepdim=True)) * rstd * weight.double().abs() + return terms + bias.double().abs() if bias is not None else terms + + +def _compare(out, ref, x, weight, bias, eps): + """Within one output step of the reference plus 2^-19 of the rounded terms, and rarely off. + + Kernel and reference evaluate the same expression and round it once: the + kernel in fp32, the reference in fp64. They differ by the kernel's fp32 + evaluation error plus at most one step of the output dtype (both round to + nearest, so to the same value or to neighbours). That fp32 error is bounded + by the magnitudes it rounds: the row mean is a sum of N values and carries + about log2(N) = 13 roundings of mean|x| into every `x - mean`; the + variance's sum about as many more, halved through the sqrt; the + approximate sqrt and division of rstd a few; and the closing multiply-add + one of `|x_hat * w| + |b|`. That is about 25 roundings of 2^-24, so 2^-19 + (32 of them) of S = (|x - mean| + mean|x|) * rstd * |w| + |b| bounds it. + Where `x_hat * w` and `b` cancel, the result is far below S and can sit + many output steps from the reference inside that allowance: on the sweep + cell's 9216 x 6144 input, 21 elements sit more than one step off (up to + 26; 92 in another draw), each a result within 2e-6 of zero, off by at most + 1.2e-8 and using under 1% of its allowance. Elsewhere the allowance is one + step. The fp32 error itself measured at most 2.1 x 2^-24 of S there. On + the 32768 x 6144 cell's input (the NVFP4 checkpoint's ranges): 117 more + than one step off (up to 965 steps), each within 1.9e-6 of zero, off by at + most 1.5e-8 and using at most 1.1% of its allowance; the fp32 error (the + op on fp32 operands against the exact value) at most 4.3 x 2^-24 of S. + An element one step off reads just under one allowance by construction, + `spacing / (spacing + 2^-19 S)`: the worst such reading there was 0.9997, + the same as `F.layer_norm`'s. + + An element differs from the correctly rounded value only when a rounding + boundary falls inside the fp32 error, about 2^-24 / 2^-p of the time for a + p-bit significand: measured 1.6e-5 to 3.0e-5 at bf16. The count allows 64 + times that, 2^(p - 18): 2^-10 at bf16 and 2^-7 at fp16. At fp32 the step + is the error's own size and the count does not apply. + + What it rejects, measured over the indexer's 9216 x 6144 sweep rows: eps + 1e-5, 1e-7 or 0 for 1e-6 lands 12,000-32,000 allowances off on the + small-variance rows (7,000-21,000 in a single row); a variance + over N - 1 puts 1.8% of the elements off; rounding toward zero 50%; + `x_hat` rounded to bf16 before the affine 27% and 177 allowances off; the + weight or bias shifted one column, a dropped bias or an RMS norm 89-98% + and 16,000-128,000 allowances off. Each holds in a single 6144-wide row. + On the 32768-row cell's output, rows past 9215 read wrapped at 9216 or one + row early, or rows from 16384 left at zero, fall outside the gate in every + such row (up to 4.4e6, 7.3e6 and 255 allowances off). + """ + assert out.dtype == ref.dtype, (out.dtype, ref.dtype) + assert out.shape == ref.shape, (out.shape, ref.shape) + allowance = _spacing(ref) + 2.0**-19 * _terms(x, weight, bias, eps) + excess = ((out.double() - ref.double()).abs() / allowance).max().item() + assert excess <= 1.0, f"{excess:.4g} allowances from the reference; the gate is 1" + bits, _ = _FORMAT[out.dtype] + if bits < 24: + off = int((out != ref).sum()) + assert off * 2 ** (18 - bits) <= out.numel(), ( + f"{off} of {out.numel()} elements differ from the correctly rounded value; " + f"at most 2^{bits - 18} of them may" + ) + + +def _check(x, weight, bias, eps): + """One call, gated; the operands are left as they were.""" + before = [t.clone() for t in (x, weight, bias) if t is not None] + out = op(x, weight, bias, eps) + assert out.shape == x.shape and out.dtype == x.dtype + assert out.is_contiguous() and out.data_ptr() != x.data_ptr() + after = [t for t in (x, weight, bias) if t is not None] + assert all(torch.equal(a, b) for a, b in zip(before, after)), "inputs must not be mutated" + assert torch.equal(out, op(x, weight, bias, eps)), "two launches must agree bitwise" + _compare(out, op.reference(x, weight, bias, eps), x, weight, bias, eps) + return out + + +def _words(t): + """`t`'s bits as int64 words over whole contiguous rows, for the sweep's compare.""" + assert t.is_contiguous() and (t.shape[-1] * t.element_size()) % 8 == 0 + return t.view(torch.int64) + + +def _sweep(spec, seed): + """Every row count of `sweep_rows`, each its own call on the leading rows of one input. + + The call at the top of the range is gated against the reference; each + smaller one has to equal the leading rows of that result bit for bit. The + kernel computes each row in its own program, so a row count that changed + any row's answer would be a grid or specialization defect, not rounding. + + Each call is compared as whole contiguous rows of int64 words (the same + bits, four bf16 to a word) and reduced with `any()`, which is still exact: + over 1..32768 on sm_103 the sweep took 6.0 s this way, against 18.6 s + with bf16 `!=` and `.sum()` and 3.2 s for the op calls alone. Each + verdict stays on the device, so the sweep synchronizes once; the first + failing row count is re-run to count its elements. + """ + lo, hi = spec["sweep_rows"] + assert spec["shape"][0] == hi, "a sweep's shape is its largest call" + x, weight, bias, eps = _build(spec, seed) + full = _check(x, weight, bias, eps) + full_words = _words(full) + differs = [ + torch.ne(_words(op(x[:rows], weight, bias, eps)), full_words[:rows]).any() + for rows in range(lo, hi + 1) + ] + failing = torch.nonzero(torch.stack(differs)).flatten() + if failing.numel(): + rows = lo + int(failing[0]) + count = int((op(x[:rows], weight, bias, eps) != full[:rows]).sum()) + raise AssertionError( + f"rows={rows}: {count} elements differ from the leading rows of the {hi}-row call" + ) + + +def _replays(spec, seed): + """Each row count of `graph_rows` in its own CUDA graph, replayed on new data. + + Captured and replayed under `inference_mode`, as the engine runs its + decode graphs, after one warm-up call on the capture stream, as the + engine's warm-up does: that call is where the Triton kernel compiles. The + replay has to equal an eager call on the same data bit for bit -- a graph + that froze anything of its capture would differ -- and is gated against + the reference like any other output. + """ + stream = torch.cuda.Stream() + with torch.inference_mode(): + for rows in spec["graph_rows"]: + x, weight, bias, eps = _build(spec, seed + rows, rows=rows) + stream.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(stream): + op(x, weight, bias, eps) + torch.cuda.current_stream().wait_stream(stream) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=stream): + out = op(x, weight, bias, eps) + x.copy_(_build(spec, seed + rows + 1, rows=rows)[0]) + graph.replay() + eager = op(x, weight, bias, eps) + assert torch.equal(out, eager), f"rows={rows}: graph replay differs from eager" + _compare(out, op.reference(x, weight, bias, eps), x, weight, bias, eps) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying): # noqa: F811 + certifying(op) + seed = zlib.crc32(cell.why.encode()) % 2**31 + if "sweep_rows" in cell.spec: + _sweep(cell.spec, seed) + elif "graph_rows" in cell.spec: + _replays(cell.spec, seed) + else: + _check(*_build(cell.spec, seed)) + + +def test_the_gate_sees_eps(): + """The scaled rows are what make eps a value the cells certify, not a passenger.""" + for rows in (1, 16): + x, weight, bias, eps = _build( + dict(shape=(rows, HIDDEN), dtype=torch.bfloat16, site="indexer_q"), seed=11 + rows + ) + out = op(x, weight, bias, eps) + _compare(out, op.reference(x, weight, bias, eps), x, weight, bias, eps) + for wrong in (1e-5, 1e-7, 0.0): + with pytest.raises(AssertionError, match="allowances from the reference"): + _compare(out, op.reference(x, weight, bias, wrong), x, weight, bias, eps) + + +def _fp64(x, weight, bias, eps, ddof=0): + xd = x.double() + mean = xd.mean(dim=-1, keepdim=True) + var = (xd - mean).square().sum(dim=-1, keepdim=True) / (xd.shape[-1] - ddof) + out = (xd - mean) / torch.sqrt(var + eps) * weight.double() + return out + bias.double() if bias is not None else out + + +def _toward_zero(value): + """fp64 -> bf16 rounding toward zero instead of to nearest.""" + bits = value.float().contiguous().view(torch.int32) & ~0xFFFF + return bits.view(torch.float32).to(torch.bfloat16) + + +def test_wrong_variants_land_outside_the_gate(): + """The cells' gate separates this computation from its neighbours, at the indexer's values.""" + spec = dict(shape=(256, HIDDEN), dtype=torch.bfloat16, site="indexer_q") + x, weight, bias, eps = _build(spec, seed=3) + + xd, wd, bd = x.double(), weight.double(), bias.double() + mean = xd.mean(dim=-1, keepdim=True) + rstd = torch.rsqrt((xd - mean).square().mean(dim=-1, keepdim=True) + eps) + wrong = { + "rmsnorm": xd * torch.rsqrt(xd.square().mean(dim=-1, keepdim=True) + eps) * wd + bd, + "bias_dropped": _fp64(x, weight, None, eps), + "weight_as_delta": (xd - mean) * rstd * (wd + 1.0) + bd, + "weight_shifted": _fp64(x, torch.roll(weight, 1), bias, eps), + "variance_over_n_minus_1": _fp64(x, weight, bias, eps, ddof=1), + "x_hat_rounded_to_bf16": ((xd - mean) * rstd).to(torch.bfloat16).double() * wd + bd, + } + wrong = {name: value.to(torch.bfloat16) for name, value in wrong.items()} + wrong["rounded_toward_zero"] = _toward_zero(_fp64(x, weight, bias, eps)) + + out = op(x, weight, bias, eps) + _compare(out, op.reference(x, weight, bias, eps), x, weight, bias, eps) + for name, value in wrong.items(): + with pytest.raises(AssertionError): + _compare(value, op.reference(x, weight, bias, eps), x, weight, bias, eps) + + +def test_the_reference_models_layer_norm_is_inside_the_same_gate(): + """The reference model computes this step as torch's `F.layer_norm` on the + bf16 operands (fp32 inside, one rounding). It sits as close to the exact + value as this op does, so the two are interchangeable at this gate.""" + x, weight, bias, eps = _build( + dict(shape=(1024, HIDDEN), dtype=torch.bfloat16, site="indexer_q"), seed=21 + ) + ref = op.reference(x, weight, bias, eps) + _compare(op(x, weight, bias, eps), ref, x, weight, bias, eps) + _compare(F.layer_norm(x, (HIDDEN,), weight, bias, eps), ref, x, weight, bias, eps) + + +def test_eps_is_inside_the_square_root(): + """Only observable where `var` is comparable to `eps`: a near-constant fp32 + row. The kernel must sit orders of magnitude nearer `1/sqrt(var + eps)` + than `1/(sqrt(var) + eps)`.""" + gen = _gen(17) + x = torch.full((4, HIDDEN), 1.0, device=DEVICE, dtype=torch.float32) + x[:, ::2] += 1e-3 + weight, bias = _affine(gen, dtype=torch.float32) + + out = op(x, weight, bias, EPS).double() + xd, wd, bd = x.double(), weight.double(), bias.double() + mean = xd.mean(dim=-1, keepdim=True) + var = (xd - mean).square().mean(dim=-1, keepdim=True) + inside = (xd - mean) / torch.sqrt(var + EPS) * wd + bd + outside = (xd - mean) / (torch.sqrt(var) + EPS) * wd + bd + to_inside = (out - inside).abs().max().item() + to_outside = (out - outside).abs().max().item() + assert to_outside > 100.0 * to_inside, (to_inside, to_outside) + + +def test_constant_rows_return_the_bias_exactly(): + """Zero variance: `x - mean` is exactly 0, so the row is `bias` bit for bit + (a zero row included, the likeliest content of a padded graph row).""" + gen = _gen(31) + weight, bias = _affine(gen) + for value in (2.5, 0.0): + x = torch.full((4, HIDDEN), value, device=DEVICE, dtype=torch.bfloat16) + out = op(x, weight, bias, EPS) + assert torch.isfinite(out.float()).all() + assert torch.equal(out, bias.expand_as(out).contiguous()), value + + +def test_runs_unchanged_under_inference_mode(): + gen = _gen(55) + x = torch.randn(256, HIDDEN, generator=gen, device=DEVICE).to(torch.bfloat16) + weight, bias = _affine(gen) + eager = op(x, weight, bias, EPS) + with torch.inference_mode(): + inferred = op(x, weight, bias, EPS).clone() + assert torch.equal(inferred, eager) + + +def test_exactly_one_kernel_is_launched(): + """The atomicity claim: one `_layer_norm_fwd_1pass_kernel`, no repacking copy.""" + from torch.profiler import ProfilerActivity, profile + + gen = _gen(66) + x = torch.randn(1024, HIDDEN, generator=gen, device=DEVICE).to(torch.bfloat16) + weight, bias = _affine(gen) + op(x, weight, bias, EPS) # warm the Triton JIT + torch.cuda.synchronize() + + with profile(activities=[ProfilerActivity.CUDA]) as prof: + op(x, weight, bias, EPS) + torch.cuda.synchronize() + + launched = [ + event.name + for event in prof.events() + if str(event.device_type) == "DeviceType.CUDA" and event.device_time_total > 0 + ] + assert launched == ["_layer_norm_fwd_1pass_kernel"], launched + + +def test_capturable_in_a_cuda_graph(): + gen = _gen(77) + x = torch.randn(512, HIDDEN, generator=gen, device=DEVICE).to(torch.bfloat16) + weight, bias = _affine(gen) + static_x = x.clone() + op(static_x, weight, bias, EPS) # compile before capture + torch.cuda.synchronize() + + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + static_out = op(static_x, weight, bias, EPS) + + static_x.copy_(x) + graph.replay() + torch.cuda.synchronize() + assert torch.equal(static_out, op(x, weight, bias, EPS)) + + +def test_rejects_a_weight_whose_length_is_not_the_normalized_width(): + """Loud: `_layer_norm_fwd` asserts the shapes, with no message.""" + gen = _gen(88) + x = torch.randn(8, HIDDEN, generator=gen, device=DEVICE).to(torch.bfloat16) + weight, bias = _affine(gen) + with pytest.raises(AssertionError): + op(x, weight[:-1].contiguous(), bias, EPS) + with pytest.raises(AssertionError): + op(x, weight, bias[:-1].contiguous(), EPS) + + +def test_rejects_a_feature_dim_that_does_not_fit_the_fused_kernel(): + width = 65536 + x = torch.randn(2, width, device=DEVICE).to(torch.bfloat16) + weight = torch.ones(width, device=DEVICE, dtype=torch.bfloat16) + bias = torch.zeros(width, device=DEVICE, dtype=torch.bfloat16) + with pytest.raises(RuntimeError, match="doesn't support feature dim"): + op(x, weight, bias, EPS) + + +def test_accepts_an_fp8_activation_and_returns_fp8(): + """Measured, not certified: `empty_like(x)` makes the output e4m3, the + arithmetic is still fp32, and the store lands within one e4m3 ulp (2^-3 + relative, 2^-9 smallest subnormal) of the exact value.""" + gen = _gen(111) + x = torch.randn(64, HIDDEN, generator=gen, device=DEVICE).to(torch.bfloat16) + weight, bias = _affine(gen) + x8 = x.to(torch.float8_e4m3fn) + + out = op(x8, weight, bias, EPS) + + assert out.dtype == torch.float8_e4m3fn + assert torch.isfinite(out.float()).all() + expected = _fp64(x8.float(), weight, bias, EPS) + deviation = (out.float().double() - expected).abs() + assert (deviation <= 2.0**-3 * expected.abs() + 2.0**-9).all(), deviation.max().item() diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/quantization/__init__.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/quantization/__init__.py new file mode 100644 index 000000000000..52a7a9daf028 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/quantization/__init__.py @@ -0,0 +1,2 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/quantization/test_modeling_v2_fp8_quantize_1x128_packed_ue8m0.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/quantization/test_modeling_v2_fp8_quantize_1x128_packed_ue8m0.py new file mode 100644 index 000000000000..75d9c502865c --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/quantization/test_modeling_v2_fp8_quantize_1x128_packed_ue8m0.py @@ -0,0 +1,380 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the fp8_quantize_1x128_packed_ue8m0 entry. + +The entry owns the contract. This file turns a cell into an activation, +gates the op against the entry's reference bit for bit -- the codes, and the scale words +read through the scale's own strides -- and holds what a cell cannot carry: the layout the +GEMM reads, row independence up to max_num_tokens (32768), every exponent boundary, the +edge inputs, the loud refusals the entry's `note` records, and K 768's half-empty last +scale word written over a dirty allocation. +""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.quantization.fp8_quantize_1x128_packed_ue8m0 import ( # noqa: E501 + E4M3_MAX, + UE8M0_BIAS, + block_exponents, + scale_leading_dim, +) +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.quantization.fp8_quantize_1x128_packed_ue8m0 import ( # noqa: E501 + fp8_quantize_1x128_packed_ue8m0 as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402,F401 — needs the path declared above + +assert torch.cuda.is_available(), "fp8_quantize_1x128_packed_ue8m0 requires a CUDA device" + +DEV = "cuda" + +#: max_num_tokens: the largest eager step the NVFP4 target takes (chunked prefill off). +MAX_TOKENS = 32768 + +#: The values planted on a block's exponent boundary: 7 * 2**sh is exactly 448 * 2**(sh - 6), +#: so amax / 448 is a power of two there; the other two are the bf16 neighbours of 7. +_ON_BOUNDARY, _ABOVE, _BELOW = 7.0, 7.03125, 6.96875 + + +def _activation(m: int, k: int, seed: int) -> torch.Tensor: + """A bf16 `[m, k]` activation with about 60 exponents and planted exponent boundaries. + + Rows of N(0, 1), each 128-block scaled by its own 2**sh, sh in [-30, 30]. One element + of every block but each fourth is planted on the block's exponent boundary (exactly, + one bf16 step above, one below): where amax / 448 is a power of two, a scale that + rounds up from one ulp high takes the next exponent, and these blocks are where it + would show. + """ + g = torch.Generator(device=DEV).manual_seed(seed) + nb = k // 128 + x = torch.randn(m, nb, 128, device=DEV, generator=g).to(torch.bfloat16).float() + kind = torch.arange(m * nb, device=DEV).reshape(m, nb) % 4 + sign = torch.randint(0, 2, (m, nb), device=DEV, generator=g).float() * 2 - 1 + pos = torch.randint(0, 128, (m, nb, 1), device=DEV, generator=g) + plant = torch.tensor([0.0, _ON_BOUNDARY, _ABOVE, _BELOW], device=DEV)[kind] * sign + old = x.gather(2, pos).squeeze(2) + x.scatter_(2, pos, torch.where(kind > 0, plant, old).unsqueeze(2)) + sh = torch.randint(-30, 31, (m, nb, 1), device=DEV, generator=g).float() + return (x * torch.exp2(sh)).reshape(m, k).to(torch.bfloat16) + + +def _bytes(scale: torch.Tensor, nb: int) -> torch.Tensor: + """The UE8M0 bytes `[M, nb]` of a packed scale, read through its strides.""" + return scale.contiguous().view(torch.uint8).reshape(scale.shape[0], -1)[:, :nb].long() + + +def _compare(out: tuple[torch.Tensor, torch.Tensor], ref: tuple[torch.Tensor, torch.Tensor]): + """Bit for bit, with no band to derive. + + The op and the reference divide by the same power of two, an exact operation, and + round the quotient once to e4m3; the exponent is an integer. A difference is therefore + a different exponent or a different rounding, never accumulation order. Codes compare + as bytes (a NaN code included); the scale compares word for word through both + tensors' strides, which is every exponent byte. + """ + codes, scale = out + ref_codes, ref_scale = ref + assert codes.dtype == ref_codes.dtype == torch.float8_e4m3fn, (codes.dtype, ref_codes.dtype) + assert codes.shape == ref_codes.shape and codes.is_contiguous(), (codes.shape, codes.stride()) + assert scale.dtype == ref_scale.dtype == torch.int32, (scale.dtype, ref_scale.dtype) + assert scale.shape == ref_scale.shape, (scale.shape, ref_scale.shape) + assert scale.stride() == ref_scale.stride(), (scale.stride(), ref_scale.stride()) + bad = codes.view(torch.uint8) != ref_codes.view(torch.uint8) + assert not bool(bad.any()), ( + f"{int(bad.sum())} of {bad.numel()} codes differ, first at {bad.nonzero()[0].tolist()}" + ) + nb = (codes.shape[1] + 127) // 128 + got, want = _bytes(scale, nb), _bytes(ref_scale, nb) + wrong = got != want + assert not bool(wrong.any()), ( + f"{int(wrong.sum())} exponent bytes differ, first at {wrong.nonzero()[0].tolist()}: " + f"{got[wrong][0].item()} vs {want[wrong][0].item()}" + ) + assert torch.equal(scale, ref_scale), "a byte slot past the last block is not zero" + + +def _boundary_blocks(x: torch.Tensor) -> int: + """How many blocks have amax / 448 exactly a power of two.""" + m, k = x.shape + amax = x.float().abs().reshape(m, k // 128, 128).amax(dim=2).double() / E4M3_MAX + return int((torch.frexp(amax)[0] == 0.5).sum()) + + +# ── the certified cells ─────────────────────────────────────────────────── + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying): # noqa: F811 — pytest resolves the fixture by this name + certifying(op) + m, k = cell.spec["m"], cell.spec["k"] + seed = zlib.crc32(cell.why.encode()) + x = _activation(m, k, seed) + if not cell.spec["graph"]: + out = op(x) + _compare(out, op.reference(x)) + else: + side = torch.cuda.Stream() + side.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(side): + op(x) + torch.cuda.current_stream().wait_stream(side) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + out = op(x) + for step in (1, 2): + x.copy_(_activation(m, k, seed + step)) + graph.replay() + torch.cuda.synchronize() + _compare(out, op.reference(x)) + eager = op(x) + assert torch.equal(out[0].view(torch.uint8), eager[0].view(torch.uint8)) + assert torch.equal(out[1], eager[1]) and out[1].stride() == eager[1].stride() + exps = block_exponents(x) + on = _boundary_blocks(x) + assert on > 0, "the cell's input never reaches an exponent boundary" + print(f" {cell.why[:44]}: {exps.unique().numel()} exponents, {on} blocks on a boundary") + + +# ── the layout the GEMM reads ───────────────────────────────────────────── + + +@pytest.mark.parametrize("k", [2048, 8192, 24576, 1024, 3072, 768]) +@pytest.mark.parametrize("m", [1, 2, 3, 4, 5, 6, 7, 16, 17, 9215, 9216, 16385, 32767, 32768]) +def test_scale_layout_strides_storage_and_padding(m, k): + """The scale is MN-major int32 with column stride align(M, 4), read through its strides. + + Its storage ends at the last real row, so only a strided read is in bounds; the pad + slots inside it are zero; the codes view a 4-row-padded allocation. K 8192 and 24576 + are the 16- and 48-column scales dep4's wo and dense layer-0 down read; K 1024, 3072 + and 768 the 2-, 6- and 2-column scales of the tep8 and tp8 wo, dense layer-0 down and + shared-expert down (768's second column holds two blocks). + """ + codes, scale = op(_activation(m, k, seed=m)) + lead, cols = scale_leading_dim(m), -(-k // 512) + assert scale.shape == (m, cols) and scale.stride() == (1, lead), (scale.shape, scale.stride()) + words = scale.untyped_storage().nbytes() // 4 + assert words == (cols - 1) * lead + m, words + flat = torch.as_strided(scale, (words,), (1,), 0) + pads = [flat[c * lead + r].item() for c in range(cols - 1) for r in range(m, lead)] + assert all(p == 0 for p in pads), pads + if m % 4: + with pytest.raises(RuntimeError, match="out of bounds for storage"): + torch.as_strided(scale, (cols * lead,), (1,), 0) + assert codes.stride() == (k, 1) and codes.untyped_storage().nbytes() == lead * k + + +@pytest.mark.parametrize("k", [6144, 2048, 1536, 8192, 24576, 1024, 3072, 768]) +def test_every_eager_row_count_is_a_prefix_of_the_largest(k): + """Rows are independent, so the certified max_num_tokens covers every eager row count. + + Each M up to max_num_tokens returns, bit for bit, the first M rows of the 32768-row call. + """ + x = _activation(MAX_TOKENS, k, seed=k) + full_codes, full_scale = op(x) + _compare((full_codes, full_scale), op.reference(x)) + rows = (1, 2, 3, 5, 7, 8, 9, 15, 16, 17, 31, 33, 127, 129, 1023, 1025, 4095, 4097, 8191) + for m in rows + (9215, 9216, 9217, 16383, 16384, 16385, 24577, 32765, 32766, 32767): + codes, scale = op(x[:m]) + assert torch.equal(codes.view(torch.uint8), full_codes[:m].view(torch.uint8)), m + assert torch.equal(scale, full_scale[:m]), m + + +@pytest.mark.parametrize("m", [1, 5, 4097, 32767]) +def test_k768_empty_byte_slots_are_written_over_a_dirty_allocation(m): + """K 768 is six blocks in two scale words: the op writes 0 into bytes 2 and 3 of the second. + + The op's two allocations (the 4-row-padded codes, then the `[2, align(M, 4)]` int32 + scale) are first made, filled with 0xFF and freed in the same sizes and order, so the + caching allocator hands the op those blocks back. The control: the code rows past M, + which the kernel never writes, still read 0xFF afterwards. So the empty byte slots and + the column-0 pad slots read 0 because the kernel wrote them, not because the memory was + fresh. + """ + k = 768 + x = _activation(m, k, seed=m) + lead = scale_leading_dim(m) + codes_block = torch.full((lead * k,), 0xFF, dtype=torch.uint8, device=DEV) + scale_block = torch.full((2 * lead * 4,), 0xFF, dtype=torch.uint8, device=DEV) + where = (codes_block.data_ptr(), scale_block.data_ptr()) + del scale_block, codes_block + codes, scale = op(x) + assert (codes.data_ptr(), scale.data_ptr()) == where, "the op's outputs left the 0xFF blocks" + unwritten = torch.as_strided(codes.view(torch.uint8), (lead * k,), (1,), 0)[m * k :] + assert unwritten.numel() == (lead - m) * k > 0 and bool((unwritten == 0xFF).all()) + _compare((codes, scale), op.reference(x)) + slots = scale.contiguous().view(torch.uint8).reshape(m, 8) + assert not bool(slots[:, 6:].any()), "an empty byte slot of the second word is not 0" + assert bool(slots[:, :6].ne(0).all()), "a live block's byte is 0" + flat = torch.as_strided(scale, (lead + m,), (1,), 0) + assert not bool(flat[m:lead].any()), "a pad slot of the first column is not 0" + + +# ── the exponent ────────────────────────────────────────────────────────── + + +def test_every_exponent_boundary(): + """Every exponent from the floor's 2**-42 to the largest a bf16 needs, 2**119. + + Each on its boundary amax = 448 * 2**e (byte e + 127, the block max coded 448), one + bf16 step above (byte e + 128, coded 224) and one below (byte e + 127, coded 448 after + rounding). + """ + exps = torch.arange(-42, 120, device=DEV) + nb = exps.numel() + on = torch.ldexp(torch.full((nb,), 7.0, device=DEV), exps + 6).to(torch.bfloat16) + above = (on.view(torch.int16) + 1).view(torch.bfloat16) + below = (on.view(torch.int16) - 1).view(torch.bfloat16) + rows = torch.stack([on, above, below]) + x = torch.zeros(3, nb, 128, dtype=torch.bfloat16, device=DEV) + x[:, :, 7] = rows + x[:, :, 100] = -rows * 0.5 + x = x.reshape(3, nb * 128) + codes, scale = op(x) + _compare((codes, scale), op.reference(x)) + got = _bytes(scale, nb) + assert torch.equal(got[0], exps + UE8M0_BIAS) + assert torch.equal(got[1], exps + UE8M0_BIAS + 1) + assert torch.equal(got[2], exps + UE8M0_BIAS) + lead = codes.reshape(3, nb, 128)[:, :, 7].float() + assert bool((lead[0] == 448).all() and (lead[1] == 224).all() and (lead[2] == 448).all()) + + +def test_edge_blocks(): + """One block per edge input, pinned to its measured bytes and codes. + + Each also equals the reference: zero, NaN, an all-NaN block, +-inf, -0.0, the largest + bf16, and values under the amax floor. + """ + blocks = torch.zeros(9, 128, dtype=torch.bfloat16, device=DEV) + blocks[1, 0], blocks[1, 1], blocks[1, 2] = float("nan"), 1.0, 3.0 + blocks[2] = float("nan") + blocks[3, 0], blocks[3, 1], blocks[3, 2] = float("inf"), 1.0, -2.0 + blocks[4, 0], blocks[4, 1] = float("-inf"), 1.0 + blocks[5, 0], blocks[5, 1] = -0.0, 1.0 + blocks[6, 0], blocks[6, 1] = 3.3895313892515355e38, 1.0 + blocks[7, 0], blocks[7, 1] = 1e-12, 5e-13 + blocks[8, 0], blocks[8, 1] = 9.2e-41, -1e-39 + x = blocks.reshape(1, -1).contiguous() + codes, scale = op(x) + _compare((codes, scale), op.reference(x)) + b = _bytes(scale, 9)[0].tolist() + c = codes.view(torch.uint8).reshape(9, 128)[:, :3].tolist() + assert b == [85, 120, 85, 254, 254, 119, 247, 85, 85], b + assert c[0] == [0x00, 0x00, 0x00], c[0] # zero block, scale 2**-42 + assert c[1] == [0x7F, 0x70, 0x7C], c[1] # NaN in place; 1.0 and 3.0 at 2**-7 + assert c[2] == [0x7F, 0x7F, 0x7F], c[2] # all NaN: floor scale, NaN codes + assert c[3] == [0x7E, 0x00, 0x80], c[3] # inf -> 448, finite -> +-0 + assert c[4][:2] == [0xFE, 0x00], c[4] # -inf -> -448 + assert c[5][:2] == [0x80, 0x78], c[5] # -0.0 keeps its sign + assert c[6][:2] == [0x78, 0x00], c[6] # bf16 max / 2**120 = 255 -> 256 + assert c[7][:2] == [0x49, 0x41], c[7] # under the floor: 1e-12 * 2**42 -> 4.5 + assert c[8][:2] == [0x00, 0x80], c[8] # bf16 subnormals -> +-0 + + +def test_discrimination_against_wrong_readings(): + """Each plausible wrong reading of the recipe or the layout gets some bytes wrong. + + Scored against one activation as the fraction of codes or exponent bytes it gets + wrong. The bitwise gate rejects any nonzero fraction; these show the cells' inputs + make each one nonzero. + """ + m, k = 64, 6144 + x = _activation(m, k, seed=7) + codes, scale = op(x) + nb = k // 128 + got = _bytes(scale, nb) + amax = x.float().abs().reshape(m, nb, 128).amax(dim=2).double().clamp(min=1e-10) / E4M3_MAX + exact = block_exponents(x) + + def byte_miss(e): + return (e + UE8M0_BIAS != got).float().mean().item() + + def code_miss(s): + q = (x.double() / s.repeat_interleave(128, dim=1)).float().clamp(-E4M3_MAX, E4M3_MAX) + wrong = q.to(torch.float8_e4m3fn).view(torch.uint8) != codes.view(torch.uint8) + return wrong.float().mean().item() + + def word_bytes(words): + return words.contiguous().view(torch.uint8).reshape(m, -1).long() + + big_endian = word_bytes(scale).reshape(m, -1, 4).flip(-1).reshape(m, -1) + dense_rows = word_bytes(torch.as_strided(scale, (m, k // 512), (k // 512, 1))) + rows = { + "exact recipe (codes)": code_miss(torch.ldexp(torch.ones_like(amax), exact)), + "exact recipe (bytes)": byte_miss(exact), + "floor(log2) exponent": byte_miss(torch.floor(torch.log2(amax)).long()), + "nearest exponent": byte_miss(torch.round(torch.log2(amax)).long()), + "exponent one high on boundaries": byte_miss( + torch.where(torch.frexp(amax)[0] == 0.5, exact + 1, exact) + ), + "plain amax / 448 scale (codes)": code_miss(amax), + "bytes big-endian in the word": (big_endian != got).float().mean().item(), + "scale storage read as dense rows": (dense_rows != got).float().mean().item(), + } + for name, miss in rows.items(): + print(f" {name:34s} wrong {miss:.4f}") + assert rows.pop("exact recipe (codes)") == 0.0 and rows.pop("exact recipe (bytes)") == 0.0 + for name, miss in rows.items(): + assert miss > 0.0, name + + +# ── behaviour around the call ───────────────────────────────────────────── + + +def test_registered_fake_matches_eager(): + from torch._subclasses.fake_tensor import FakeTensorMode + + shapes = ((1, 6144), (3, 2048), (17, 1536), (9216, 6144), (32767, 1536), (32768, 6144)) + dep4 = ((5, 8192), (32768, 8192), (3, 24576), (32767, 24576)) + tep8 = ((1, 1024), (32767, 1024), (5, 3072), (32768, 3072), (3, 768), (32768, 768)) + for m, k in shapes + dep4 + tep8: + e_codes, e_scale = op(torch.zeros(m, k, dtype=torch.bfloat16, device=DEV)) + eager = [(tuple(t.shape), t.stride(), t.dtype) for t in (e_codes, e_scale)] + with FakeTensorMode(): + f_codes, f_scale = op(torch.empty(m, k, dtype=torch.bfloat16, device=DEV)) + fake = [(tuple(t.shape), t.stride(), t.dtype) for t in (f_codes, f_scale)] + assert fake == eager, (m, k, fake, eager) + + +def test_input_untouched_deterministic_and_stream_independent(): + x = _activation(129, 6144, seed=3) + before = x.clone() + c1, s1 = op(x) + c2, s2 = op(x) + alt = torch.cuda.Stream() + alt.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(alt): + c3, s3 = op(x) + alt.synchronize() + assert torch.equal(x, before) + for c, s in ((c2, s2), (c3, s3)): + assert torch.equal(c.view(torch.uint8), c1.view(torch.uint8)) and torch.equal(s, s1) + + +def test_refusals_are_the_ops_own(): + """Every precondition the note lists is enforced loudly by the op. + + That is why the entry has no `is_valid`: restating one would only swap its error for a + worse one. + """ + bf = dict(dtype=torch.bfloat16, device=DEV) + cases = [ + ("Input matrix dtype must be BF16", lambda: op(torch.randn(4, 128, device=DEV))), + ( + "Input matrix dtype must be BF16", + lambda: op(torch.randn(4, 128, dtype=torch.float16, device=DEV)), + ), + ("input must be a matrix", lambda: op(torch.randn(2, 4, 128, **bf))), + ("must be a multiple of 16, but got 6152", lambda: op(torch.randn(4, 6152, **bf))), + ("self must be contiguous", lambda: op(torch.randn(8, 12288, **bf)[:, :6144])), + ("not registered with any CUDA device", lambda: op(torch.empty(0, 6144, **bf))), + ] + for message, call in cases: + with pytest.raises(RuntimeError, match=message): + call() + with pytest.raises(NotImplementedError): + op(torch.randn(4, 128, dtype=torch.bfloat16)) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/vision/__init__.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/vision/__init__.py new file mode 100644 index 000000000000..52a7a9daf028 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/vision/__init__.py @@ -0,0 +1,2 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/vision/test_modeling_v2_pixtral_rope_2d.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/vision/test_modeling_v2_pixtral_rope_2d.py new file mode 100644 index 000000000000..0c37e019a304 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/ops/vision/test_modeling_v2_pixtral_rope_2d.py @@ -0,0 +1,352 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the Mistral Large 4 pixtral_rope_2d entry. + +The cells drive the vision tower's image grids (one patch, a small image, the +largest 110x110 image, two images per call up to the envelope's two max-size +images) at the tower's head geometry, in its packed-qkv layout. The rest of +the file pins what a cell cannot show: which rotation this is (the +discrimination test) and the inputs `is_valid` refuses because the op takes +them and answers wrongly. +""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch.modeling_v2.models.mistral_large4.ops.vision.pixtral_rope_2d import ( # noqa: E501 + pixtral_rope_2d as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402,F401 + +assert torch.cuda.is_available(), "pixtral_rope_2d requires a CUDA device" + +HEADS = 16 +HEAD_DIM = 128 +THETA = 10000.0 + +#: Explicit mantissa bits of each output dtype: one ulp at |v| in [2^e, 2^(e+1)) +#: is 2^(e - bits). +_MANTISSA_BITS = {torch.bfloat16: 7, torch.float32: 23} + +# The band, per element, against the fp64 closed form rounded once to the +# output dtype (`op.reference`). Writing y for the exact rotated value, y' for +# the op's fp32 value before its final rounding, and r for the magnitude of +# the input pair the element belongs to (a rotation preserves it): +# +# |out - ref| <= |out - y'| + |y' - y| + |y - ref| +# <= ulp(out) / 2 + r * delta + ulp(ref) / 2 +# +# (rounding to nearest never leaves the binade downward, so the half ulp at the +# rounded value bounds each rounding). delta is the fp32 math's own error, +# relative to r: +# - the frequency `1 / theta ** (2i / D)`: fp32 pow then fp32 reciprocal, +# budgeted at 2^-21 relative (four ulp at the bottom of a binade; measured +# 7.2e-8 = 2^-23.7 on sm_103 against fp64); +# - the angle `fp32(position) * f`: one more rounding, 2^-24 relative; the +# angle error is the position times both, at most p_max * (2^-21 + 2^-24) +# radians (measured 6.3e-6 rad over positions 0..109); +# - cos and sin of the fp32 angle: 2 ulp each of a value <= 1, 2^-22 each; +# - the complex multiply of the fp32 pair (exact from bf16) by the rotor: two +# products and a sum, three roundings of magnitude <= r, 3 * 2^-24. +# The bound is position-dependent through p_max, the largest row or column of +# the cell. Measured against it: see the printed ratio per cell. +_FREQ_REL = 2.0**-21 +_ANGLE_ROUND = 2.0**-24 +_COS_SIN = 2 * 2.0**-22 +_ROTATE = 3 * 2.0**-24 + + +def _delta(p_max: int) -> float: + return p_max * (_FREQ_REL + _ANGLE_ROUND) + _COS_SIN + _ROTATE + + +def _seed(text: str) -> int: + return zlib.crc32(text.encode()) + + +def _grids(grid) -> list[tuple[int, int]]: + """A cell's `grid`: one image's (rows, cols), or a tuple of them.""" + if isinstance(grid[0], int): + return [tuple(grid)] + return [tuple(g) for g in grid] + + +def _positions(grids: list[tuple[int, int]]) -> torch.Tensor: + """int64 [N, 2] (row, col), each image row-major from (0, 0), concatenated.""" + per_image = [ + torch.stack( + torch.meshgrid(torch.arange(h), torch.arange(w), indexing="ij"), dim=-1 + ).reshape(-1, 2) + for h, w in grids + ] + return torch.cat(per_image).to("cuda") + + +def _qk(n: int, heads: int, head_dim: int, dtype: torch.dtype, layout: str, seed: int): + """q and k as the tower holds them: head-reshaped column chunks of one + [N, 3 * heads * head_dim] projection ("packed_qkv"), or two contiguous + tensors ("separate").""" + g = torch.Generator(device="cuda").manual_seed(seed) + if layout == "packed_qkv": + qkv = (1.5 * torch.randn(n, 3 * heads * head_dim, generator=g, device="cuda")).to(dtype) + q, k, _ = qkv.chunk(3, dim=-1) + return q.reshape(n, heads, head_dim), k.reshape(n, heads, head_dim) + assert layout == "separate", layout + q = (1.5 * torch.randn(n, heads, head_dim, generator=g, device="cuda")).to(dtype) + k = (1.5 * torch.randn(n, heads, head_dim, generator=g, device="cuda")).to(dtype) + return q, k + + +def _build(spec, seed: int): + positions = _positions(_grids(spec["grid"])) + q, k = _qk( + positions.shape[0], spec["heads"], spec["head_dim"], spec["dtype"], spec["layout"], seed + ) + return q, k, positions + + +def _ulp(v: torch.Tensor, dtype: torch.dtype) -> torch.Tensor: + """fp64 spacing of `dtype` at each value of `v` (0 at 0).""" + _, exponent = torch.frexp(v.double()) + spacing = torch.ldexp( + torch.ones_like(v, dtype=torch.float64), exponent - 1 - _MANTISSA_BITS[dtype] + ) + return torch.where(v == 0, torch.zeros_like(spacing), spacing) + + +def _pair_magnitude(x: torch.Tensor) -> torch.Tensor: + """|(x[2j], x[2j+1])| in fp64, repeated onto both elements of the pair.""" + pairs = x.double().reshape(*x.shape[:-1], -1, 2) + return pairs.norm(dim=-1).repeat_interleave(2, dim=-1) + + +def _bound(x: torch.Tensor, out: torch.Tensor, ref: torch.Tensor, p_max: int) -> torch.Tensor: + return ( + 0.5 * _ulp(out, out.dtype) + 0.5 * _ulp(ref, out.dtype) + _pair_magnitude(x) * _delta(p_max) + ) + + +def _ratio(x, out, candidate, p_max) -> float: + """Largest |out - candidate| over the band, both sides in out's dtype.""" + bound = _bound(x, out, candidate, p_max) + err = (out.double() - candidate.double()).abs() + assert bool((err[bound == 0] == 0).all()), "a zero pair must come out zero" + return (err / bound.clamp_min(torch.finfo(torch.float64).tiny)).max().item() + + +def _budget_share(x, out, ref, p_max) -> float: + """Largest error beyond the two final roundings, over r * delta.""" + rounding = 0.5 * _ulp(out, out.dtype) + 0.5 * _ulp(ref, out.dtype) + beyond = ((out.double() - ref.double()).abs() - rounding).clamp_min(0.0) + budget = _pair_magnitude(x) * _delta(p_max) + return (beyond / budget.clamp_min(torch.finfo(torch.float64).tiny)).max().item() + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying): # noqa: F811 + certifying(op) + spec = cell.spec + q, k, positions = _build(spec, seed=_seed(cell.why)) + before = (q.clone(), k.clone(), positions.clone()) + + q_out, k_out = op(q, k, positions, spec["theta"]) + + for x, out in ((q, q_out), (k, k_out)): + assert out.shape == x.shape and out.dtype == x.dtype and out.device == x.device + assert out.is_contiguous() + assert out.untyped_storage().data_ptr() != x.untyped_storage().data_ptr() + for was, now in zip(before, (q, k, positions)): + assert torch.equal(was, now), "the entry must not mutate its inputs" + + q_ref, k_ref = op.reference(q, k, positions, spec["theta"]) + p_max = int(positions.max().item()) + for name, x, out, ref in (("q", q, q_out, q_ref), ("k", k, k_out, k_ref)): + ratio = _ratio(x, out, ref, p_max) + differ = int((out != ref).sum().item()) + # A ratio just under 1 at bf16 is a one-ulp rounding split (y' and y on + # either side of a rounding boundary: |out - ref| is one ulp, the band + # one ulp plus r * delta). What the fp32 math itself spends is the + # error left after the two roundings, as a share of r * delta. + spent = _budget_share(x, out, ref, p_max) + print( + f"\n {cell.why} {name}: max |out-ref| / band {ratio:.3f}, fp32 budget spent " + f"{spent:.3f}, {differ} of {out.numel()} elements not bitwise the closed form" + ) + assert ratio <= 1.0, (name, ratio) + + # The patch at (0, 0) of every image has angle 0: the rotor is exactly 1. + origin = (positions == 0).all(dim=-1) + assert torch.equal(q_out[origin], q[origin]) and torch.equal(k_out[origin], k[origin]) + + +def _closed_form(x, positions, theta, *, pairing="gptj", angles=None): + """A 2-D rotation in fp64 (not rounded) with the pairing and the angle table + as parameters, for the variants the discrimination test needs.""" + pos = positions.double() + angle = angles(pos[:, 0:1], pos[:, 1:2], theta).unsqueeze(-2) + cos, sin = torch.cos(angle), torch.sin(angle) + x64 = x.double() + half = x.shape[-1] // 2 + if pairing == "gptj": + even, odd = x64[..., 0::2], x64[..., 1::2] + out = torch.empty_like(x64) + out[..., 0::2] = even * cos - odd * sin + out[..., 1::2] = even * sin + odd * cos + return out + lo, hi = x64[..., :half], x64[..., half:] # NeoX half-split: (i, i + D/2) + return torch.cat((lo * cos - hi * sin, lo * sin + hi * cos), dim=-1) + + +def _freq(index, theta): + return theta ** (-2.0 * index / HEAD_DIM) + + +def test_identity_is_the_pixtral_2d_rotation() -> None: + """Which rotation the entry computes, at a non-square 40x60 image: every + plausible wrong variant lands far outside the band, at bf16 (the cells' + dtype) and at fp32 I/O, where the band is the fp32 math's own budget and + the distance is not capped by the output's rounding; the closed form and + the reference implementation's host-built table land inside it.""" + positions = _positions([(40, 60)]) + q, _ = _qk(positions.shape[0], HEADS, HEAD_DIM, torch.bfloat16, "packed_qkv", seed=41) + q32 = q.float() + out, _ = op(q, q, positions, THETA) + out32, _ = op(q32, q32, positions, THETA) + p_max = int(positions.max().item()) + + def distances(candidate64): + """(bf16 ratio, fraction of bf16 elements outside, fp32 ratio).""" + cand = candidate64.to(torch.bfloat16) + bound = _bound(q, out, cand, p_max) + outside = ((out.double() - cand.double()).abs() > bound).double().mean().item() + return ( + _ratio(q, out, cand, p_max), + outside, + _ratio(q32, out32, candidate64.to(torch.float32), p_max), + ) + + dev = q.device + j = torch.arange(HEAD_DIM // 4, dtype=torch.float64, device=dev) + even, odd = _freq(2 * j, THETA), _freq(2 * j + 1, THETA) + lo, hi = _freq(j, THETA), _freq(j + HEAD_DIM // 4, THETA) + every = _freq(torch.arange(HEAD_DIM // 2, dtype=torch.float64, device=dev), THETA) + pixtral = dict(angles=lambda r, c, t: torch.cat((r * even, c * odd), -1)) + + r16, outside, r32 = distances(_closed_form(q, positions, THETA, **pixtral)) + print(f"\n closed form (correct): bf16 {r16:.3f}x, fp32 {r32:.3f}x band") + assert r16 <= 1.0 and r32 <= 1.0 and outside == 0.0 + + wrong = { + "half-split (NeoX) pairing": dict(pixtral, pairing="neox"), + "row and column swapped": dict(angles=lambda r, c, t: torch.cat((c * even, r * odd), -1)), + "frequency halves, not interleaved": dict( + angles=lambda r, c, t: torch.cat((r * lo, c * hi), -1) + ), + "the same frequencies on both axes": dict( + angles=lambda r, c, t: torch.cat((r * even, c * even), -1) + ), + "axes alternating pair by pair": dict( + angles=lambda r, c, t: torch.stack((r * even, c * odd), -1).flatten(-2) + ), + "rotation reversed": dict(angles=lambda r, c, t: -torch.cat((r * even, c * odd), -1)), + "1-D raster index, standard rope": dict(angles=lambda r, c, t: (r * 60 + c) * every), + "theta 1e6": dict( + angles=lambda r, c, t: torch.cat((r * _freq(2 * j, 1e6), c * _freq(2 * j + 1, 1e6)), -1) + ), + "positions off by one": dict( + angles=lambda r, c, t: torch.cat(((r + 1) * even, (c + 1) * odd), -1) + ), + } + for name, variant in wrong.items(): + r16, outside, r32 = distances(_closed_form(q, positions, THETA, **variant)) + print( + f" {name:36s} bf16 {r16:8.1f}x band ({outside:6.1%} of elements outside), " + f"fp32 {r32:10.1f}x band" + ) + assert r16 > 50.0 and r32 > 1000.0, (name, r16, r32) + + # fp32 math, not the input dtype's: the same rotation with a bf16 rotor + # (cos/sin rounded to bf16, as an implementation in the input dtype has it) + # leaves the fp32 band. Its error is up to half a bf16 ulp of cos/sin, + # 2^-9, against a band of about 3.2e-5 r at p_max 59 (about 60x), so the + # margin asked of it is 10x rather than the structural variants' 1000x. + angle = torch.cat((positions[:, 0:1].double() * even, positions[:, 1:2].double() * odd), -1) + rotor = torch.complex(angle.cos().bfloat16().float(), angle.sin().bfloat16().float()) + bf16_rotor = torch.view_as_real( + torch.view_as_complex(q32.reshape(*q32.shape[:-1], -1, 2)) * rotor.unsqueeze(-2) + ).flatten(-2) + times = _ratio(q32, out32, bf16_rotor, p_max) + print(f" {'bf16 rotor':36s} fp32 {times:10.1f}x band") + assert times > 10.0, times + + # A correct alternative: the reference implementation's recipe, which builds + # the frequency vector and the cos/sin table in fp32 on the host and moves + # it to the device. Inside the band; not what the entry does (it builds + # them on the device: a few fp32 ulp apart in the table). + host_f = 1.0 / (THETA ** (torch.arange(0, HEAD_DIM, 2).float() / HEAD_DIM)) + pos_h = positions.cpu() + host_angle = torch.cat( + (pos_h[:, 0:1].float() * host_f[0::2], pos_h[:, 1:2].float() * host_f[1::2]), -1 + ) + host_rotor = torch.polar(torch.ones_like(host_angle), host_angle).to(dev).unsqueeze(-2) + for name, x, mine in (("bf16", q, out), ("fp32", q32, out32)): + alt = ( + torch.view_as_real( + torch.view_as_complex(x.float().reshape(*x.shape[:-1], -1, 2)) * host_rotor + ) + .flatten(-2) + .to(x.dtype) + ) + ratio = _ratio(x, mine, alt, p_max) + differ = int((alt != mine).sum().item()) + print( + f" {'host-built table (correct) ' + name:36s} {ratio:8.3f}x band, " + f"{differ} of {mine.numel()} elements differ from the entry" + ) + assert ratio <= 1.0, (name, ratio) + + +# Guards: each input below is first shown, unguarded, to come back without an +# error and wrong; then `is_valid` refuses it. Every operand is allocated at the +# size the call reads, so the unguarded calls touch only memory the test owns. + + +def _small(): + positions = _positions([(2, 4)]) + q, k = _qk(positions.shape[0], HEADS, HEAD_DIM, torch.bfloat16, "separate", seed=44) + return q, k, positions + + +def test_guard_positions_must_cover_every_token() -> None: + q, k, positions = _small() + q_out, _ = op(q, k, positions[:1], THETA) + assert q_out.shape == q.shape, "accepted: one row of positions broadcasts" + first_patch_everywhere, _ = op(q, k, positions[:1].expand_as(positions), THETA) + assert torch.equal(q_out, first_patch_everywhere) + assert not torch.equal(q_out, op(q, k, positions, THETA)[0]) + with validating(op), pytest.raises(AssertionError, match="positions must be"): + op(q, k, positions[:1], THETA) + + +def test_guard_k_must_carry_the_tokens_of_q() -> None: + q, k, positions = _small() + _, k_out = op(q, k[:1], positions, THETA) + assert k_out.shape == q.shape and k_out.shape != k[:1].shape, ( + "accepted: a one-token k broadcasts to every token's angles" + ) + with validating(op), pytest.raises(AssertionError, match="same tokens"): + op(q, k[:1], positions, THETA) + + +def test_guard_q_and_k_must_be_three_dimensional() -> None: + q, k, positions = _small() + flat_q, flat_k = q[:, 0], k[:, 0] + q_out, _ = op(flat_q, flat_k, positions, THETA) + n = positions.shape[0] + assert q_out.shape == (n, n, HEAD_DIM), "accepted: [N, D] broadcasts to [N, N, D]" + with validating(op), pytest.raises(AssertionError, match=r"\[tokens, heads, head_dim\]"): + op(flat_q, flat_k, positions, THETA) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/test_collapsed_image_tokens.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/test_collapsed_image_tokens.py new file mode 100644 index 000000000000..ea15817f8950 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/test_collapsed_image_tokens.py @@ -0,0 +1,323 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Chat token ids with one [IMG] per image expand to the ids trtllm-serve renders. + +A frontend that renders Mistral-Large-4 chats itself, without resizing images, puts +one [IMG] where each image goes and hands the engine those token ids with the decoded +images. `MistralNativeInputProcessor` expands each [IMG] into the image's grid -- an +[IMG] per merged tile, [IMG_BREAK] after every row but the last, [IMG_END] -- from the +image as mistral-common resizes it. + +These tests render the collapsed ids independently of that expansion: mistral-common's +own chat encoder with an image encoder that emits a single [IMG] per image. The +expansion must reproduce trtllm-serve's native chat path exactly -- token ids, pixels +and image sizes -- for one image, several images of different sizes, text interleaved +with images, image-only turns, multi-turn chats with earlier images, tools with +reasoning effort, an image larger than the encoder's maximum size, and JPEG / RGBA PNG / +palette PNG inputs. The images reach the processor as decoded PIL images in RGB, +transparency composited over white, while trtllm-serve loads the same bytes itself. + +CPU only. Needs MISTRAL_LARGE4_CHECKPOINT_DIR (the checkpoint's tekken.json and +params.json) and OpenCV, which mistral-common's image resize uses. +""" + +from __future__ import annotations + +import asyncio +import base64 +import contextlib +import os +from io import BytesIO + +import numpy as np +import pytest +import torch +from PIL import Image + +pytestmark = pytest.mark.cpu_only + + +@pytest.fixture(scope="module") +def native_processor(): + if not os.environ.get("MISTRAL_LARGE4_CHECKPOINT_DIR"): + pytest.skip("needs the checkpoint's tokenizer and configuration") + pytest.importorskip("cv2") + from tensorrt_llm._torch.modeling_v2.models.mistral_large4.routing import serves_images + from tensorrt_llm._torch.models.checkpoints.mistral.config_loader import MistralConfigLoader + from tensorrt_llm._torch.models.modeling_mistral import MistralNativeInputProcessor + + path = os.environ["MISTRAL_LARGE4_CHECKPOINT_DIR"] + config = MistralConfigLoader().load_pretrained_config(path) + if not serves_images(config): + pytest.skip(f"{path}: this checkpoint's target serves text only") + return MistralNativeInputProcessor(path, config, None) + + +def _noise(seed: int, height: int, width: int, mode: str = "RGB") -> Image.Image: + gen = np.random.default_rng(seed) + channels = {"RGB": 3, "RGBA": 4}[mode] + pixels = gen.integers(0, 256, size=(height, width, channels), dtype=np.uint8) + return Image.fromarray(pixels, mode) + + +def _encoded(image: Image.Image, fmt: str) -> bytes: + buffer = BytesIO() + image.save(buffer, fmt) + return buffer.getvalue() + + +def _png(seed: int, height: int, width: int) -> bytes: + return _encoded(_noise(seed, height, width), "PNG") + + +def _data_url(data: bytes) -> str: + kind = "jpeg" if data[:2] == b"\xff\xd8" else "png" + return f"data:image/{kind};base64," + base64.b64encode(data).decode() + + +def _worker_image(data: bytes) -> Image.Image: + """The image as a frontend's worker decodes it before handing it to the engine: RGB, with + transparency composited over white as trtllm-serve's media loader does.""" + from tensorrt_llm.inputs.media_io import convert_image_mode + + return convert_image_mode(Image.open(BytesIO(data), formats=["JPEG", "PNG", "WEBP"]), "RGB") + + +def _image_part(data: bytes) -> dict: + return {"type": "image_url", "image_url": {"url": _data_url(data)}} + + +def _text(text: str) -> dict: + return {"type": "text", "text": text} + + +@contextlib.contextmanager +def _one_img_per_image(tokenizer): + """mistral-common's chat encoder, with every image encoded as a single [IMG].""" + from mistral_common.tokens.tokenizers.image import ImageEncoding + + instruct = tokenizer.mistral.instruct_tokenizer + image_encoder = instruct.image_encoder + image_token = image_encoder.special_ids.img + + def one_token(chunk): + return ImageEncoding(tokens=[image_token], image=np.zeros((3, 1, 1), np.float32)) + + instruct.image_encoder = one_token + try: + yield image_token + finally: + instruct.image_encoder = image_encoder + + +def _collapsed_ids(tokenizer, messages, tools, reasoning_effort): + from mistral_common.protocol.instruct.request import ChatCompletionRequest + + with _one_img_per_image(tokenizer): + request = ChatCompletionRequest.from_openai( + messages=messages, tools=tools, reasoning_effort=reasoning_effort + ) + return tokenizer.mistral.encode_chat_completion(request).tokens + + +def _serve_ids(native_processor, messages, tools, reasoning_effort): + """Token ids and images of trtllm-serve's native chat path for `messages`.""" + from tensorrt_llm.inputs.utils import async_apply_chat_template_with_mm + from tensorrt_llm.serve.chat_utils import parse_chat_messages_coroutines + + conversation, media, counts, _ = parse_chat_messages_coroutines( + messages, native_processor.config + ) + kwargs = {"reasoning_effort": reasoning_effort} if reasoning_effort else None + ids, (mm_data, _) = asyncio.run( + async_apply_chat_template_with_mm( + media, + model_type="mistral_large_4", + tokenizer=native_processor.tokenizer, + processor=None, + conversation=conversation, + add_generation_prompt=True, + mm_placeholder_counts=counts, + tools=tools, + chat_template_kwargs=kwargs, + ) + ) + return ids, mm_data + + +_TOOLS = [ + { + "type": "function", + "function": { + "name": "describe_region", + "description": "Describe a region of an image.", + "parameters": { + "type": "object", + "properties": {"image": {"type": "integer"}, "region": {"type": "string"}}, + "required": ["image", "region"], + }, + }, + } +] + + +def _cases(): + one = _png(1, 280, 420) + small = _png(2, 56, 84) + tall = _png(3, 900, 300) + eight = [_png(10 + i, 56 + 37 * i, 420 - 29 * i) for i in range(8)] + large = _png(4, 2300, 1700) + jpeg = _encoded(_noise(5, 333, 517), "JPEG") + rgba = _encoded(_noise(6, 200, 300, "RGBA"), "PNG") + palette = _encoded(_noise(7, 150, 240).convert("P"), "PNG") + return { + "single": ( + [{"role": "user", "content": [_text("Describe this image."), _image_part(one)]}], + None, + None, + ), + "image_only": ([{"role": "user", "content": [_image_part(small)]}], None, None), + "interleaved": ( + [ + {"role": "system", "content": "Compare images in order."}, + { + "role": "user", + "content": [ + _text("First "), + _image_part(small), + _text(" then "), + _image_part(tall), + _text("; which is taller?"), + ], + }, + ], + None, + None, + ), + "eight_images": ( + [ + { + "role": "user", + "content": [_text("Count the images.")] + [_image_part(i) for i in eight], + } + ], + None, + None, + ), + "multi_turn": ( + [ + {"role": "user", "content": [_image_part(one), _text("What is this?")]}, + {"role": "assistant", "content": "Random noise."}, + { + "role": "user", + "content": [_text("And this one, next to the first?"), _image_part(tall)], + }, + ], + None, + None, + ), + "tools_reasoning": ( + [ + {"role": "system", "content": "Use tools when helpful."}, + {"role": "user", "content": [_text("Describe the top half."), _image_part(one)]}, + ], + _TOOLS, + "high", + ), + "larger_than_encoder": ( + [{"role": "user", "content": [_text("Describe."), _image_part(large)]}], + None, + None, + ), + "jpeg_rgba_palette": ( + [ + { + "role": "user", + "content": [ + _image_part(jpeg), + _image_part(rgba), + _image_part(palette), + _text("Which image has the most colors?"), + ], + } + ], + None, + "low", + ), + } + + +_CASES = _cases() + + +def _image_bytes(messages): + images = [] + for message in messages: + content = message.get("content") + for part in content if isinstance(content, list) else []: + if part["type"] == "image_url": + url = part["image_url"]["url"] + images.append(base64.b64decode(url.split(",", 1)[1])) + return images + + +@pytest.mark.parametrize("name", list(_CASES)) +def test_collapsed_ids_expand_to_the_served_chat(native_processor, name): + messages, tools, reasoning_effort = _CASES[name] + images = _image_bytes(messages) + proc = native_processor.processor + served, served_mm = _serve_ids(native_processor, messages, tools, reasoning_effort) + collapsed = _collapsed_ids(native_processor.tokenizer, messages, tools, reasoning_effort) + + assert collapsed.count(proc.image_token_id) == len(images) + assert proc.image_break_id not in collapsed and proc.image_end_id not in collapsed + assert len(native_processor._image_blocks(served)) == len(images) + + expanded, extra = native_processor( + { + "prompt_token_ids": collapsed, + "multi_modal_data": {"image": [_worker_image(data) for data in images]}, + }, + None, + ) + assert expanded == served + + _, served_extra = native_processor( + {"prompt_token_ids": served, "multi_modal_data": served_mm}, None + ) + payload = extra["multimodal_data"]["image"] + reference = served_extra["multimodal_data"]["image"] + assert payload["image_sizes"] == reference["image_sizes"] + assert torch.equal(payload["pixel_values"], reference["pixel_values"]) + + +def test_collapsed_ids_hash_and_place_images_as_the_served_chat(native_processor): + """KV-cache reuse keys images by content and position: both inputs must agree.""" + from tensorrt_llm.inputs.registry import create_input_processor_with_hash + + wrapped = create_input_processor_with_hash(native_processor) + messages, tools, reasoning_effort = _CASES["interleaved"] + images = [_worker_image(data) for data in _image_bytes(messages)] + served, _ = _serve_ids(native_processor, messages, tools, reasoning_effort) + collapsed = _collapsed_ids(native_processor.tokenizer, messages, tools, reasoning_effort) + + def process(ids): + out_ids, extra = wrapped( + {"prompt_token_ids": ids, "multi_modal_data": {"image": images}}, None + ) + mm = extra["multimodal_input"] + return out_ids, mm.multimodal_hashes, mm.multimodal_positions, mm.multimodal_lengths + + assert process(collapsed) == process(served) + + +def test_collapsed_ids_must_match_the_images(native_processor): + messages, tools, reasoning_effort = _CASES["interleaved"] + images = [_worker_image(data) for data in _image_bytes(messages)] + collapsed = _collapsed_ids(native_processor.tokenizer, messages, tools, reasoning_effort) + with pytest.raises(ValueError, match="placeholder count"): + native_processor( + {"prompt_token_ids": collapsed, "multi_modal_data": {"image": images[:1]}}, None + ) + with pytest.raises(ValueError, match="require image data"): + native_processor({"prompt_token_ids": collapsed}, None) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/test_multimodal_processor.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/test_multimodal_processor.py new file mode 100644 index 000000000000..e41e21d0c467 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/test_multimodal_processor.py @@ -0,0 +1,419 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The mistral-native image processor takes an image in every form the media loaders +hand one over. + +`checkpoint_format: mistral` builds `MistralNativeInputProcessor`, whose +`MistralCommonImageProcessor` encodes each image for mistral-common. The loaders +deliver a PIL image (`"pil"`), a float `(C, H, W)` tensor in [0, 1] (`"pt"`, the +default of `default_multimodal_input_loader` and of the boot gate) or a uint8 +`(H, W, C)` array (`"np"`); before `to_pil_image` the processor took PIL only and a +tensor raised `AttributeError: 'Tensor' object has no attribute 'convert'`. + +trtllm-serve resolves a chat request's image placeholder through the +placeholder registry under the config class's model_type, which for the +Mistral Large 4 composite config is "mistral_large_4"; with the metadata registered only +under "mistral_common" every image chat request was refused with +`TypeError: Unknown modality: image`. + +The conversion and placeholder tests run on CPU. The end-to-end test runs the +processor on the boot image when MISTRAL_LARGE4_CHECKPOINT_DIR names the checkpoint +(its tekken.json) and OpenCV is installed (mistral-common's image encoder needs +it): every format gives the same token ids and pixels, and the geometry the +Mistral Large 4 tower and its splice expect. +""" + +from __future__ import annotations + +import asyncio +import base64 +import os +from io import BytesIO +from pathlib import Path + +import numpy as np +import pytest +import torch +from PIL import Image +from torchvision.transforms import ToTensor + +from tensorrt_llm._torch.models.modeling_mistral import MistralCommonImageProcessor + +pytestmark = pytest.mark.cpu_only + +_to_pil = MistralCommonImageProcessor.to_pil_image + +#: The boot gate's image, from the tree under test. +_BOOT_IMAGE = ( + Path(__file__).resolve().parents[5] / "integration" / "test_input_files" / "merlion.png" +) + + +def _rgb(seed: int = 0, size=(37, 23)) -> Image.Image: + gen = np.random.default_rng(seed) + return Image.fromarray(gen.integers(0, 256, size=(*size, 3), dtype=np.uint8), "RGB") + + +def _every_level() -> Image.Image: + """Each of the 256 8-bit levels in every channel, so every level must come back.""" + levels = np.arange(256, dtype=np.uint8).reshape(16, 16) + return Image.fromarray(np.stack([levels, levels.T, 255 - levels], -1), "RGB") + + +def _pt(image: Image.Image) -> torch.Tensor: + """The `"pt"` format is `ToTensor`: uint8 / 255 in fp32.""" + tensor = ToTensor()(image) + assert tensor.dtype == torch.float32 and tensor.shape == (3, image.height, image.width) + return tensor + + +def _uint8_chw(image: Image.Image, channels: int = 3) -> torch.Tensor: + return torch.from_numpy(np.asarray(image)[..., :channels]).permute(2, 0, 1).contiguous() + + +@pytest.mark.parametrize( + "image,form,mode", + [ + (_rgb(), lambda image: image, "RGB"), + (_every_level(), _pt, "RGB"), + (_rgb(1), _pt, "RGB"), + (_rgb(2), np.asarray, "RGB"), + (_rgb(2), _uint8_chw, "RGB"), + (_rgb(2), lambda image: _uint8_chw(image, channels=1), "L"), + ], + ids=["pil", "pt_every_level", "pt", "uint8_hwc_array", "uint8_chw_tensor", "uint8_gray"], +) +def test_to_pil_image_takes_every_loader_form(image, form, mode): + """A PIL image passes through; a `"pt"` tensor comes back at its exact 8-bit + levels; a uint8 `(H, W, C)` array or `(C, H, W)` tensor comes back as it is, + with one channel as mode "L".""" + given = form(image) + back = _to_pil(given) + if isinstance(given, Image.Image): + assert back is given + assert back.mode == mode + if mode == "RGB": + assert back.size == image.size + assert np.array_equal(np.asarray(back), np.asarray(image)) + + +def test_an_unknown_type_is_refused(): + with pytest.raises(TypeError, match="unsupported image type"): + _to_pil([1, 2, 3]) + + +def test_the_server_places_an_image_for_the_mistral_large4_config(): + """The chat parsing trtllm-serve runs on a request with an image, on the + config the mistral loader builds for a Mistral Large 4 checkpoint.""" + from tensorrt_llm._torch.configs.mistral_large4 import MistralLarge4Config + from tensorrt_llm.inputs.model_config import resolve_top_level_model_type + from tensorrt_llm.inputs.registry import MULTIMODAL_PLACEHOLDER_REGISTRY + from tensorrt_llm.serve.chat_utils import parse_chat_messages_coroutines + + config = MistralLarge4Config() + assert resolve_top_level_model_type(config) == "mistral_large_4" + from tensorrt_llm.inputs.content_format import ContentFormat + + assert ( + MULTIMODAL_PLACEHOLDER_REGISTRY.get_content_format("mistral_large_4") + == ContentFormat.OPENAI + ) + + buffer = BytesIO() + _rgb(5).save(buffer, "PNG") + url = "data:image/png;base64," + base64.b64encode(buffer.getvalue()).decode() + prompt = "Describe this image in one sentence." + messages = [ + { + "role": "user", + "content": [ + {"type": "text", "text": prompt}, + {"type": "image_url", "image_url": {"url": url}}, + ], + } + ] + conversation, media, counts, _ = parse_chat_messages_coroutines(messages, config) + mm_data, _ = asyncio.run(media) + assert counts == [{"[IMG]": 1}] + # The native renderer receives the ordered content parts and resolved images. + assert conversation[0]["content"] == prompt + assert len(mm_data["image"]) == 1 + + +@pytest.mark.skipif( + not os.environ.get("MISTRAL_LARGE4_CHECKPOINT_DIR"), + reason="MISTRAL_LARGE4_CHECKPOINT_DIR names no Mistral Large 4 checkpoint", +) +def test_every_loader_format_encodes_the_boot_image_alike(): + pytest.importorskip("cv2") + from tensorrt_llm._torch.models.checkpoints.mistral.tokenizer import MistralTokenizer + from tensorrt_llm.inputs.utils import load_image + + tokenizer = MistralTokenizer.from_pretrained(os.environ["MISTRAL_LARGE4_CHECKPOINT_DIR"]) + processor = MistralCommonImageProcessor(tokenizer=tokenizer, dtype=torch.bfloat16) + prompt = "Describe this image in one sentence." + pil = load_image(str(_BOOT_IMAGE), format="pil") + images = { + "pil": pil, + "pt": load_image(str(_BOOT_IMAGE), format="pt"), + "np": np.asarray(pil), + } + results = {fmt: processor(text=prompt, images=[image]) for fmt, image in images.items()} + ids = results["pil"]["input_ids"] + pixels = results["pil"]["pixel_values"] + for fmt in ("pt", "np"): + assert torch.equal(results[fmt]["input_ids"], ids), fmt + assert torch.equal(results[fmt]["pixel_values"], pixels), fmt + # 1192 x 874 -> 1204 x 896 pixels: a 64 x 86 patch grid, 32 x 43 merged tiles, + # one [IMG] per tile, [IMG_BREAK] after each tile row but the last, [IMG_END]. + assert tuple(pixels.shape) == (1, 3, 896, 1204) + assert results["pil"]["image_sizes"].tolist() == [[896, 1204]] + flat = ids[0] + assert int((flat == processor.image_token_id).sum()) == 32 * 43 + assert int((flat == processor.image_break_token_id).sum()) == 31 + assert int((flat == processor.image_end_token_id).sum()) == 1 + assert int(flat[0]) == 1, "BOS" + + +@pytest.mark.skipif( + not os.environ.get("MISTRAL_LARGE4_CHECKPOINT_DIR"), reason="needs checkpoint tokenizer" +) +def test_two_different_image_sizes_preserve_each_grid(): + pytest.importorskip("cv2") + from tensorrt_llm._torch.models.checkpoints.mistral.tokenizer import MistralTokenizer + + tokenizer = MistralTokenizer.from_pretrained(os.environ["MISTRAL_LARGE4_CHECKPOINT_DIR"]) + processor = MistralCommonImageProcessor(tokenizer, torch.bfloat16) + images = [_rgb(12, (224, 336)), _rgb(13, (448, 280))] + singles = [processor("Describe.", images=[image]) for image in images] + together = processor("Describe.", images=images) + assert together["image_sizes"].tolist() == [one["image_sizes"][0].tolist() for one in singles] + for i, one in enumerate(singles): + h, w = one["image_sizes"][0].tolist() + assert torch.equal(together["pixel_values"][i, :, :h, :w], one["pixel_values"][0]) + expected = sum(int((one["input_ids"] == processor.image_token_id).sum()) for one in singles) + assert int((together["input_ids"] == processor.image_token_id).sum()) == expected + + +@pytest.mark.skipif( + not os.environ.get("MISTRAL_LARGE4_CHECKPOINT_DIR"), reason="needs checkpoint tokenizer" +) +@pytest.mark.parametrize("name", ["merlion.png", "excel_table_test.jpg"]) +def test_pixels_are_what_mistral_common_makes_of_the_loaded_image(name): + """Nothing lossy lies between the loaded image and mistral-common's resize and normalize.""" + pytest.importorskip("cv2") + from mistral_common.protocol.instruct.chunk import ImageChunk + + from tensorrt_llm._torch.models.checkpoints.mistral.tokenizer import MistralTokenizer + + tokenizer = MistralTokenizer.from_pretrained(os.environ["MISTRAL_LARGE4_CHECKPOINT_DIR"]) + processor = MistralCommonImageProcessor(tokenizer, torch.float32) + image = Image.open(_BOOT_IMAGE.parent / name).convert("RGB") + expected = torch.as_tensor(processor.image_processor(ImageChunk(image=image)).image) + pixels = processor("", images=[image])["pixel_values"][0] + assert torch.equal(pixels, expected.to(pixels.dtype)) + + +@pytest.fixture(scope="module") +def native_processor(): + if not os.environ.get("MISTRAL_LARGE4_CHECKPOINT_DIR"): + pytest.skip("needs checkpoint tokenizer") + from tensorrt_llm._torch.modeling_v2.models.mistral_large4.routing import serves_images + from tensorrt_llm._torch.models.checkpoints.mistral.config_loader import MistralConfigLoader + from tensorrt_llm._torch.models.modeling_mistral import MistralNativeInputProcessor + + path = os.environ["MISTRAL_LARGE4_CHECKPOINT_DIR"] + # The checkpoint's own configuration: its vision encoder and its NVFP4 export are what make + # the processor take images. + config = MistralConfigLoader().load_pretrained_config(path) + if not serves_images(config): + pytest.skip(f"{path}: this checkpoint's target serves text only") + return MistralNativeInputProcessor(path, config, None) + + +def _image_url(image): + buffer = BytesIO() + image.save(buffer, "PNG") + return "data:image/png;base64," + base64.b64encode(buffer.getvalue()).decode() + + +@pytest.mark.parametrize("multi_turn", [False, True]) +def test_native_chat_keeps_roles_image_order_and_control_tokens(native_processor, multi_turn): + from tensorrt_llm.inputs.utils import async_apply_chat_template_with_mm + from tensorrt_llm.serve.chat_utils import parse_chat_messages_coroutines + + images = [_rgb(21, (56, 84)), _rgb(22, (112, 56))] + messages = [ + {"role": "system", "content": "Describe images in order."}, + { + "role": "user", + "content": [ + {"type": "text", "text": "First "}, + {"type": "image_url", "image_url": {"url": _image_url(images[0])}}, + {"type": "text", "text": " then "}, + {"type": "image_url", "image_url": {"url": _image_url(images[1])}}, + {"type": "text", "text": "; compare them."}, + ], + }, + ] + if multi_turn: + messages += [ + {"role": "assistant", "content": "Two images."}, + { + "role": "user", + "content": [ + {"type": "text", "text": "Now compare the first to "}, + {"type": "image_url", "image_url": {"url": _image_url(images[0])}}, + {"type": "text", "text": " again."}, + ], + }, + ] + expected = native_processor.tokenizer.apply_chat_template( + messages, tokenize=True, return_dict=False, add_generation_prompt=True + ) + conversation, media, counts, _ = parse_chat_messages_coroutines( + messages, native_processor.config + ) + ids, (mm_data, _) = asyncio.run( + async_apply_chat_template_with_mm( + media, + model_type="mistral_large_4", + tokenizer=native_processor.tokenizer, + processor=None, + conversation=conversation, + add_generation_prompt=True, + mm_placeholder_counts=counts, + ) + ) + assert ids == expected + actual_ids, extra = native_processor( + {"prompt_token_ids": ids, "multi_modal_data": mm_data}, None + ) + assert actual_ids == expected + assert ids[0] == 1 + wanted_images = images + images[:1] if multi_turn else images + payload = extra["multimodal_data"]["image"] + for index, image in enumerate(wanted_images): + single = native_processor.processor("", images=[image]) + h, w = single["image_sizes"][0].tolist() + assert payload["image_sizes"][index] == [h, w] + assert torch.equal(payload["pixel_values"][index, :, :h, :w], single["pixel_values"][0]) + + +def test_raw_text_interleaving_matches_native_chat(native_processor): + images = [_rgb(30, (56, 84)), _rgb(31, (112, 56))] + prompt = "Before [IMG] between [IMG] after" + ids, extra = native_processor({"prompt": prompt, "multi_modal_data": {"image": images}}, None) + expected = native_processor.tokenizer.apply_chat_template( + [ + { + "role": "user", + "content": [ + {"type": "text", "text": "Before "}, + {"type": "image_url", "image_url": {"url": _image_url(images[0])}}, + {"type": "text", "text": " between "}, + {"type": "image_url", "image_url": {"url": _image_url(images[1])}}, + {"type": "text", "text": " after"}, + ], + } + ], + tokenize=True, + return_dict=False, + ) + assert ids == expected + assert len(extra["multimodal_data"]["image"]["image_sizes"]) == 2 + with pytest.raises(ValueError, match="placeholder count"): + native_processor({"prompt": prompt, "multi_modal_data": {"image": images[:1]}}, None) + + +def test_token_ids_expand_placeholders_without_reencoding(native_processor): + images = [_rgb(40, (56, 84)), _rgb(41, (112, 56))] + ids = [1, 1000, 10, 2000, 10, 3000] + actual, _ = native_processor( + {"prompt_token_ids": ids, "multi_modal_data": {"image": images}}, None + ) + blocks = native_processor._image_blocks(actual) + assert actual[: blocks[0][0]] == [1, 1000] + assert actual[blocks[0][1] : blocks[1][0]] == [2000] + assert actual[blocks[1][1] :] == [3000] + assert actual.count(10) == (2 * 3) + (4 * 2) + again, _ = native_processor( + {"prompt_token_ids": actual, "multi_modal_data": {"image": images}}, None + ) + assert actual == again + for invalid in (actual[:-2], actual + [13], [1, 10], [1, 12]): + with pytest.raises(ValueError, match="[Ii]mage"): + native_processor( + {"prompt_token_ids": invalid, "multi_modal_data": {"image": images}}, None + ) + with pytest.raises(ValueError, match="sizes and order"): + native_processor( + {"prompt_token_ids": actual, "multi_modal_data": {"image": list(reversed(images))}}, + None, + ) + + +def test_text_only_native_token_ids_are_unchanged(native_processor): + ids = native_processor.tokenizer.apply_chat_template( + [ + {"role": "user", "content": "Hello"}, + {"role": "assistant", "content": "Hi"}, + {"role": "user", "content": "Continue"}, + ], + tokenize=True, + return_dict=False, + ) + assert native_processor({"prompt_token_ids": ids}, None) == (ids, None) + with pytest.raises(ValueError, match="require image data"): + native_processor({"prompt_token_ids": [1, 10, 13]}, None) + + +def test_default_loader_returns_native_token_ids(native_processor): + from tensorrt_llm.inputs.utils import default_multimodal_input_loader + + images = [_rgb(50, (56, 84)), _rgb(51, (112, 56))] + inputs = default_multimodal_input_loader( + tokenizer=native_processor.tokenizer, + model_dir=native_processor.model_path, + model_type="mistral_large_4", + modality="multiple_image", + prompts=["Compare."], + media=[_image_url(image) for image in images], + ) + assert len(inputs) == 1 and "prompt_token_ids" in inputs[0] + ids, extra = native_processor(inputs[0], None) + assert ids == inputs[0]["prompt_token_ids"] + assert extra["multimodal_data"]["image"]["image_sizes"] == [[56, 84], [112, 56]] + + +def test_image_hashes_and_positions_are_independent_of_companions(native_processor): + from tensorrt_llm.inputs.multimodal import get_multimodal_item_metadata + from tensorrt_llm.inputs.registry import create_input_processor_with_hash + + wrapped = create_input_processor_with_hash(native_processor) + a, b, c = [_rgb(seed, (56, 84)) for seed in (61, 62, 63)] + + def process(images, prompt): + ids, extra = wrapped({"prompt": prompt, "multi_modal_data": {"image": images}}, None) + assert native_processor.multimodal_hashing_supported is True + mm = extra["multimodal_input"] + blocks = native_processor._image_blocks(ids) + assert mm.multimodal_positions == [begin for begin, _ in blocks] + assert mm.multimodal_lengths == [end - begin for begin, end in blocks] + # The wrapper records each image's embedding length in the item metadata. + metadata = get_multimodal_item_metadata(extra["multimodal_data"]) + assert list(metadata.output_embedding_lengths) == [6] * len(images) + return ids, mm.multimodal_hashes + + alone, ha = process([a], "Color [IMG]?") + warm, ha_warm = process([a], "Color [IMG]?") + different, hb = process([b], "Color [IMG]?") + ab, hab = process([a, b], "First [IMG], then [IMG].") + ac, hac = process([a, c], "First [IMG], then [IMG].") + ba, hba = process([b, a], "First [IMG], then [IMG].") + assert alone == warm == different + assert ha == ha_warm and ha != hb + assert ab == ac == ba + assert hab[0] == hac[0] == hba[1] == ha[0] + assert hab[1] == hba[0] == hb[0] + assert hac[1] != hab[1] diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/test_multimodal_vision.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/test_multimodal_vision.py new file mode 100644 index 000000000000..90c3423da719 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/test_multimodal_vision.py @@ -0,0 +1,602 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The Mistral Large 4 target's image path, pinned against independent implementations. + +* One vision-tower layer's attention -- the table's `[wq ; wk ; wv]` operand, the + 2-D RoPE entry, the no-KV-cache attention call and `wo`, exactly as the tower runs + them -- on the boot image's patch grid, against transformers' Pixtral vision + attention fed the same weights converted to its layout. The checkpoint stores each + head's q/k rows as interleaved (GPT-J) rotary pairs; transformers rotates halves, + so its q/k projections are the published mistral -> HF permutation of those rows + (each head's rows `(i, 2j + t)` -> `(i, t * D/2 + j)`). The tower must read the + checkpoint's rows as published: a control that applies that permutation to the + tower's own operand (what the stock Pixtral path's weight mapper does before its + half-split rotary) lands far outside the band. +* The table's q/k/v row reads the three checkpoint tensors in that order and + concatenates them bit for bit: no permutation on the load path. +* The splice: across the context chunks of a request (chunked prefill, or a resume + from a reused prefix that ends inside the image) the rows placed at the [IMG] + positions are, chunk by chunk, the whole image's rows in order; [IMG_BREAK], + [IMG_END] and text keep their token embeddings; the tower runs once per request. + +The boot image is `tests/integration/test_input_files/merlion.png` (1192 x 874) as +the engine's mistral-native processor delivers it: 896 x 1204 pixels, a 64 x 86 +patch grid (5504 patch rows), 32 x 43 [IMG] tokens. + +The attention tests use the checkpoint's vision layer 0 when MISTRAL_LARGE4_CHECKPOINT_DIR +names the checkpoint, else seeded random weights at the same shapes. They need a GPU; +the splice test runs on CPU. +""" + +from __future__ import annotations + +import importlib +import json +import os +from pathlib import Path +from types import SimpleNamespace + +import pytest +import torch +from torch import nn + +from tensorrt_llm._torch.pyexecutor.engine.runners.common import prepare_multimodal_indices +from tensorrt_llm.inputs.multimodal import MultimodalParams, MultimodalRuntimeData + +__extra_import_path__ = ["../.."] +from _precision import true_fp32_matmul # noqa: E402 + + +@pytest.fixture(params=[103, 107], ids=["sm103", "sm107"]) +def target_sm(request): + return request.param + + +@pytest.fixture(autouse=True, params=["tep", "tp", "dep"]) +def target(request, target_sm): + global modeling, weights + if torch.cuda.is_available(): + major, minor = torch.cuda.get_device_capability() + if 10 * major + minor != target_sm: + pytest.skip(f"requires SM{target_sm}") + module = ( + "tensorrt_llm._torch.modeling_v2.models.mistral_large4." + f"mistral_large_4_nvfp4__sm_{target_sm}__{request.param}" + ) + modeling = importlib.import_module(module + ".modeling") + weights = importlib.import_module(module + ".weights") + + +#: params.json `vision_encoder`: hidden 2048 over 16 heads of 128, rope_theta 1e4, +#: image_size 1540, patch 14. +HIDDEN = 2048 +HEADS = 16 +HEAD_DIM = 128 +THETA = 10000.0 +IMAGE_SIZE = 1540 +PATCH = 14 + +#: The boot image's patch grid, rows x cols. +BOOT_GRID = (64, 86) + +#: The prompt tokens of an image span (tekken.json special ids). +IMG, IMG_BREAK, IMG_END = 10, 12, 13 + +_CKPT_ENV = "MISTRAL_LARGE4_CHECKPOINT_DIR" +_LAYER0 = "vision_encoder.transformer.layers.0." + +needs_gpu = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs a GPU") + + +# --- weights --------------------------------------------------------------------- + + +def _checkpoint_layer0() -> dict | None: + """Vision layer 0's attention weights and attention norm, as published, or None.""" + root = os.environ.get(_CKPT_ENV) + if not root or not Path(root, "consolidated.safetensors.index.json").is_file(): + return None + from safetensors import safe_open + + index = json.loads(Path(root, "consolidated.safetensors.index.json").read_text()) + names = ("attention.wq", "attention.wk", "attention.wv", "attention.wo", "attention_norm") + out = {} + for name in names: + key = f"{_LAYER0}{name}.weight" + with safe_open(str(Path(root, index["weight_map"][key])), framework="pt") as f: + out[name] = f.get_tensor(key) + return out + + +def _random_layer0() -> dict: + gen = torch.Generator().manual_seed(2026) + std = HIDDEN**-0.5 + + def mat(): + return (torch.randn(HIDDEN, HIDDEN, generator=gen) * std).to(torch.bfloat16) + + return { + "attention.wq": mat(), + "attention.wk": mat(), + "attention.wv": mat(), + "attention.wo": mat(), + "attention_norm": torch.ones(HIDDEN, dtype=torch.bfloat16), + } + + +@pytest.fixture(scope="module") +def layer0(): + found = _checkpoint_layer0() + return (found, "checkpoint") if found is not None else (_random_layer0(), "random") + + +def _to_hf_layout(w: torch.Tensor) -> torch.Tensor: + """The mistral -> HF conversion of a q or k projection: within each head, the + interleaved rotary pairs `(2j, 2j + 1)` become the halves `(j, j + D/2)`.""" + return w.view(HEADS, HEAD_DIM // 2, 2, HIDDEN).transpose(1, 2).reshape(HEADS * HEAD_DIM, HIDDEN) + + +def _qkv_row(): + """The table's vision q/k/v row.""" + rows = [w for w in weights.MODEL_WEIGHTS.WEIGHTS if w.tag == "v" and w.name == "qkv"] + assert len(rows) == 1, rows + return rows[0] + + +# --- the two attention implementations -------------------------------------------- + + +def _grid_positions(grids, device) -> torch.Tensor: + """(row, col) of every patch, each image row-major from (0, 0).""" + return torch.cat( + [ + torch.stack( + torch.meshgrid( + torch.arange(h, device=device), torch.arange(w, device=device), indexing="ij" + ), + -1, + ).reshape(-1, 2) + for h, w in grids + ] + ) + + +def _tower_attention(x, wq, wk, wv, wo, grids) -> torch.Tensor: + """The tower's own attention call, on the operand the table builds.""" + qkv = torch.cat((wq, wk, wv), dim=0) + block = modeling._VisionBlock( + attn_norm=None, qkv=qkv.t(), wo=wo.t(), ffn_norm=None, w13=None, w2=None + ) + seg = [h * w for h, w in grids] + workspace = torch.empty(0, dtype=torch.int8, device=x.device) + step = modeling._vision_step(seg, x.device, workspace) + out = modeling._vision_attention( + x, block, _grid_positions(grids, x.device), step, HEADS, HEAD_DIM, THETA + ) + torch.cuda.synchronize() + return out + + +def _hf_attention(x, wq, wk, wv, wo, grids, dtype) -> torch.Tensor: + """transformers' Pixtral vision attention, its rotary embedding, its flat + position ids and its block mask, on the HF-layout weights, in `dtype`.""" + from transformers.models.pixtral.configuration_pixtral import PixtralVisionConfig + from transformers.models.pixtral.modeling_pixtral import ( + PixtralAttention, + PixtralRotaryEmbedding, + generate_block_attention_mask, + position_ids_in_meshgrid, + ) + + cfg = PixtralVisionConfig( + hidden_size=HIDDEN, + intermediate_size=6144, + num_hidden_layers=1, + num_attention_heads=HEADS, + image_size=IMAGE_SIZE, + patch_size=PATCH, + rope_parameters={"rope_type": "default", "rope_theta": THETA}, + ) + attn = PixtralAttention(cfg).to(device=x.device, dtype=dtype).eval() + with torch.no_grad(): + attn.q_proj.weight.copy_(_to_hf_layout(wq)) + attn.k_proj.weight.copy_(_to_hf_layout(wk)) + attn.v_proj.weight.copy_(wv) + attn.o_proj.weight.copy_(wo) + rope = PixtralRotaryEmbedding(cfg).to(x.device) + patch_list = [torch.empty(1, h, w) for h, w in grids] + position_ids = position_ids_in_meshgrid(patch_list, max_width=IMAGE_SIZE // PATCH).to(x.device) + hidden = x.to(dtype)[None] + cos, sin = rope(hidden, position_ids) + mask = None + if len(grids) > 1: + mask = generate_block_attention_mask([h * w for h, w in grids], hidden) + with torch.no_grad(): + out, _ = attn(hidden, attention_mask=mask, position_embeddings=(cos, sin)) + torch.cuda.synchronize() + return out[0] + + +def _errors(out: torch.Tensor, truth: torch.Tensor) -> tuple[float, float]: + """(max |error|, RMS error), relative to the truth's RMS.""" + diff = out.double() - truth.double() + scale = truth.double().pow(2).mean().sqrt().item() + return diff.abs().max().item() / scale, diff.pow(2).mean().sqrt().item() / scale + + +# The band: the tower's error from the fp32 answer may be at most this many times +# transformers' own bf16 error from it (both measures). Measured on sm_103 (GB300) +# with the checkpoint's layer 0: 0.97-0.98 of transformers' bf16 error, at the boot +# grid alone and beside a second image. +_BAND = 1.25 +# The permuted-weights control must miss the fp32 answer by more than this many +# times transformers' bf16 RMS error. Measured with the checkpoint's layer 0: 86x +# and 108x. +_CONTROL_MIN = 10.0 + + +@needs_gpu +@pytest.mark.parametrize( + "grids", + [(BOOT_GRID,), (BOOT_GRID, (32, 32))], + ids=["boot_grid", "boot_grid_and_32x32"], +) +def test_vision_attention_matches_transformers_pixtral(layer0, grids): + w, source = layer0 + dev = torch.device("cuda") + wq, wk, wv, wo = (w[f"attention.{n}"].to(dev) for n in ("wq", "wk", "wv", "wo")) + n = sum(h * c for h, c in grids) + gen = torch.Generator(device=dev).manual_seed(7) + x = torch.randn(n, HIDDEN, device=dev, generator=gen) + # The block's attention input is RMS-normalized and scaled by attention_norm. + x = x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + 1e-5) + x = (x * w["attention_norm"].to(dev).float()).to(torch.bfloat16) + + with true_fp32_matmul(): + truth = _hf_attention( + x.float(), *(t.float() for t in (wq, wk, wv, wo)), grids, torch.float32 + ) + ref_bf16 = _hf_attention(x, wq, wk, wv, wo, grids, torch.bfloat16) + out = _tower_attention(x, wq, wk, wv, wo, grids) + assert out.shape == (n, HIDDEN) and out.dtype == torch.bfloat16 + + ref_max, ref_rms = _errors(ref_bf16, truth) + got_max, got_rms = _errors(out, truth) + print( + f"[{source} weights, grids {grids}] relative to the fp32 answer: transformers bf16 " + f"max {ref_max:.3e} rms {ref_rms:.3e}; tower max {got_max:.3e} rms {got_rms:.3e}" + ) + assert got_max <= _BAND * ref_max, (got_max, ref_max) + assert got_rms <= _BAND * ref_rms, (got_rms, ref_rms) + + # The control: the stock path's permutation applied to the tower's own q/k rows. + wrong = _tower_attention(x, _to_hf_layout(wq), _to_hf_layout(wk), wv, wo, grids) + _, wrong_rms = _errors(wrong, truth) + print(f"[{source} weights] permuted-q/k control rms {wrong_rms:.3e}") + assert wrong_rms > _CONTROL_MIN * ref_rms, (wrong_rms, ref_rms) + + +@pytest.mark.cpu_only +def test_the_qkv_row_reads_the_checkpoint_rows_as_published(): + row = _qkv_row() + d = SimpleNamespace(v_layers=30, v_hidden=HIDDEN) + assert row.layer_indices(d) == list(range(30)) + sources = row.sources(d, 7, "vision_encoder.transformer.layers.7") + prefix = "vision_encoder.transformer.layers.7.attention." + assert [key for key, _ in sources] == [prefix + name + ".weight" for name in ("wq", "wk", "wv")] + assert row.transform is None + assert getattr(row, "shard", None) is None + gen = torch.Generator().manual_seed(3) + parts = tuple(torch.randn(HIDDEN, HIDDEN, generator=gen).to(torch.bfloat16) for _ in range(3)) + result = torch.empty(3 * HIDDEN, HIDDEN, dtype=torch.bfloat16) + for (_, index), part in zip(sources, parts): + result[index] = part + assert torch.equal(result, torch.cat(parts, 0)) + + +# --- the splice -------------------------------------------------------------------- + + +class _StubTower: + """The tower's contract, with known rows: request r's image row k is the vector + `1000 * (r + 1) + k`, in the text width. Counts its calls per request.""" + + def __init__(self, hidden: int) -> None: + self.hidden = hidden + self.calls: list[int] = [] + + def forward(self, params: list[MultimodalParams]) -> torch.Tensor: + out = [] + for p in params: + r = p.multimodal_data["image"]["request"] + n = p.multimodal_data["image"]["rows"] + self.calls.append(r) + out.append( + (1000.0 * (r + 1) + torch.arange(n, dtype=torch.float32))[:, None].expand( + n, self.hidden + ) + ) + return torch.cat(out).to(torch.bfloat16) + + +def _prompt(text_before: int, rows: int, cols: int, text_after: int) -> list[int]: + """BOS, text, an image of `rows` x `cols` [IMG] with [IMG_BREAK] after each row + but the last, which ends with [IMG_END], then text.""" + ids = [1] + [100 + t for t in range(text_before)] + for r in range(rows): + ids += [IMG] * cols + [IMG_END if r == rows - 1 else IMG_BREAK] + return ids + [200 + t for t in range(text_after)] + + +@pytest.mark.cpu_only +def test_the_splice_places_each_chunks_image_rows(): + hidden, vocab = 8, 512 + embed = torch.randn(vocab, hidden).to(torch.bfloat16) + tower = _StubTower(hidden) + model = SimpleNamespace( + config=SimpleNamespace(vocab_size=vocab), + multimodal_token_ids=torch.tensor([IMG], dtype=torch.int32), + ) + # Two image requests and a text-only one between them. + prompts = {0: _prompt(5, 4, 6, 3), 1: [1] + list(range(300, 340)), 2: _prompt(2, 3, 5, 4)} + image_rows = {0: 4 * 6, 2: 3 * 5} + data = { + r: { + "image": {"request": r, "rows": image_rows[r]}, + "multimodal_embed_mask_cumsum": (torch.tensor(prompts[r]) == IMG).cumsum( + 0, dtype=torch.int64 + ), + } + for r in image_rows + } + # Each step's window of each request. Request 0 resumes from a reused prefix + # that ends inside its image (position 9), then chunks; request 2 is chunked + # from 0, its second chunk wholly inside the image. + steps = [ + {0: (9, 20), 1: (0, 20), 2: (0, 7)}, + {0: (20, 29), 1: (20, 41), 2: (7, 15)}, + {0: (29, len(prompts[0])), 2: (15, len(prompts[2]))}, + ] + placed = {r: [] for r in image_rows} + for windows in steps: + ids, params = [], [] + spans = {} + for r, (start, end) in windows.items(): + spans[r] = (len(ids), len(ids) + end - start) + if r in data: + runtime = MultimodalRuntimeData( + past_seen_token_num=start, + chunk_end_pos=end, + embed_mask_cumsum=data[r]["multimodal_embed_mask_cumsum"], + ) + params.append( + MultimodalParams( + multimodal_data=data[r], + multimodal_runtime=runtime, + input_ids_start_offset=len(ids), + ) + ) + ids += prompts[r][start:end] + _, mm_idx = prepare_multimodal_indices(ids, model=model) + input_ids = torch.tensor(ids, dtype=torch.int32) + rows = modeling._image_rows(tower, params) + h = modeling._splice_image_rows(embed, input_ids, rows, mm_idx) + plain = nn.functional.embedding(input_ids, embed) + is_img = input_ids == IMG + # Every non-[IMG] position, the specials and text included, keeps its embedding. + assert torch.equal(h[~is_img], plain[~is_img]) + for r, (lo, hi) in spans.items(): + if r in placed: + placed[r].append(h[lo:hi][is_img[lo:hi]]) + + for r, chunks in placed.items(): + got = torch.cat(chunks) + start = steps[0][r][0] + skipped = int((torch.tensor(prompts[r][:start]) == IMG).sum()) + whole = tower.forward( + [MultimodalParams(multimodal_data={"image": {"request": r, "rows": image_rows[r]}})] + ) + assert torch.equal(got, whole[skipped:]), f"request {r}: rows out of order" + # Once per request across all its chunks (the last two calls are this test's own). + assert sorted(tower.calls[:-2]) == [0, 2] + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("window", ["before-image", "after-image"]) +def test_a_step_without_image_rows_keeps_the_token_path(window): + """A chunk that covers none of its request's [IMG] tokens splices nothing.""" + tower = _StubTower(4) + ids = _prompt(3, 2, 2, 6) + cumsum = (torch.tensor(ids) == IMG).cumsum(0, dtype=torch.int64) + start, end = (0, ids.index(IMG)) if window == "before-image" else (len(ids) - 6, len(ids)) + param = MultimodalParams( + multimodal_data={ + "image": {"request": 0, "rows": 4}, + "multimodal_embed_mask_cumsum": cumsum, + }, + multimodal_runtime=MultimodalRuntimeData( + past_seen_token_num=start, chunk_end_pos=end, embed_mask_cumsum=cumsum + ), + ) + assert modeling._image_rows(tower, [param]) is None + assert tower.calls == [] + assert "multimodal_embedding" not in param.multimodal_data + + +@pytest.mark.cpu_only +def test_image_encoding_waits_for_the_first_image_chunk(): + tower = _StubTower(4) + ids = _prompt(3, 2, 2, 6) + cumsum = (torch.tensor(ids) == IMG).cumsum(0, dtype=torch.int64) + data = {"image": {"request": 0, "rows": 4}, "multimodal_embed_mask_cumsum": cumsum} + image_start = ids.index(IMG) + for start, end, expected_calls in [(0, image_start, []), (image_start, len(ids), [0])]: + param = MultimodalParams( + multimodal_data=data, + multimodal_runtime=MultimodalRuntimeData( + past_seen_token_num=start, chunk_end_pos=end, embed_mask_cumsum=cumsum + ), + ) + rows = modeling._image_rows(tower, [param]) + assert tower.calls == expected_calls + if expected_calls: + expected = (1000.0 + torch.arange(4))[:, None].expand(4, 4).to(torch.bfloat16) + assert torch.equal(rows, expected) + else: + assert rows is None + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("cached_position", [0, 1, 2]) +def test_image_prefix_hit_is_skipped_beside_partial_and_fresh_requests(cached_position): + tower = _StubTower(4) + ids = _prompt(3, 2, 2, 6) + cumsum = (torch.tensor(ids) == IMG).cumsum(0, dtype=torch.int64) + + def request(index, start): + return MultimodalParams( + multimodal_data={ + "image": {"request": index, "rows": 4}, + "multimodal_embed_mask_cumsum": cumsum, + }, + multimodal_runtime=MultimodalRuntimeData( + past_seen_token_num=start, chunk_end_pos=len(ids), embed_mask_cumsum=cumsum + ), + ) + + cached = request(0, len(ids) - 6) + partial = request(1, ids.index(IMG) + 1) + fresh = request(2, 0) + params = [partial, fresh] + params.insert(cached_position, cached) + expected = torch.cat([2000.0 + torch.arange(1, 4), 3000.0 + torch.arange(4)]) + expected = expected[:, None].expand(7, 4).to(torch.bfloat16) + assert torch.equal(modeling._image_rows(tower, params), expected) + assert tower.calls == [1, 2] + assert "multimodal_embedding" not in cached.multimodal_data + # The active requests keep their request-local embeddings for subsequent chunks. + assert torch.equal(modeling._image_rows(tower, params), expected) + assert tower.calls == [1, 2] + + +@pytest.mark.cpu_only +def test_image_rows_without_runtime_keep_the_full_embedding_path(): + tower = _StubTower(4) + param = MultimodalParams(multimodal_data={"image": {"request": 0, "rows": 4}}) + expected = (1000.0 + torch.arange(4))[:, None].expand(4, 4).to(torch.bfloat16) + assert torch.equal(modeling._image_rows(tower, [param]), expected) + assert tower.calls == [0] + + +@pytest.mark.cpu_only +def test_the_splice_refuses_a_count_mismatch(): + embed = torch.randn(32, 4).to(torch.bfloat16) + rows = torch.zeros(3, 4, dtype=torch.bfloat16) + with pytest.raises(ValueError, match="image rows"): + modeling._splice_image_rows( + embed, torch.tensor([1, IMG, IMG]), rows, torch.tensor([1, 2], dtype=torch.int64) + ) + + +@pytest.mark.cpu_only +@pytest.mark.parametrize( + "sizes", + [ + None, + [], + [[56, 84]], + [[0, 84], [56, 84]], + [[112, 84], [56, 84]], + [[55, 84], [56, 84]], + [[56], [56, 84]], + ], +) +def test_vision_requires_true_image_geometry(sizes): + param = MultimodalParams( + multimodal_data={"image": {"pixel_values": torch.zeros(2, 3, 56, 84), "image_sizes": sizes}} + ) + with pytest.raises(ValueError, match="[Ii]mage"): + modeling.VisionTower._images(SimpleNamespace(patch=14, merge=2), [param]) + + +@pytest.mark.cpu_only +def test_vision_slices_padding_by_each_true_size(): + pixels = torch.rand(2, 3, 112, 84) + param = MultimodalParams( + multimodal_data={"image": {"pixel_values": pixels, "image_sizes": [[56, 84], [112, 56]]}} + ) + images = modeling.VisionTower._images(SimpleNamespace(patch=14, merge=2), [param]) + assert [image[1:] for image in images] == [(4, 6), (8, 4)] + assert torch.equal(images[0][0], pixels[0, :, :56, :84]) + assert torch.equal(images[1][0], pixels[1, :, :112, :56]) + + +@needs_gpu +def test_full_tower_images_are_independent(target): + """Full checkpoint vision outputs retain image identity beside other images.""" + from PIL import Image + from safetensors import safe_open + + from tensorrt_llm._torch.models.checkpoints.mistral.config_loader import adapt_config_dict + from tensorrt_llm._torch.models.checkpoints.mistral.tokenizer import MistralTokenizer + from tensorrt_llm._torch.models.modeling_mistral import MistralCommonImageProcessor + from tensorrt_llm.mapping import Mapping + + checkpoint = os.environ.get(_CKPT_ENV) + if not checkpoint: + pytest.skip("Requires MISTRAL_LARGE4_CHECKPOINT_DIR for full vision weights") + root = Path(checkpoint) + config = adapt_config_dict(json.loads((root / "params.json").read_text())) + core = SimpleNamespace( + model_config=SimpleNamespace( + pretrained_config=config, + torch_dtype=torch.bfloat16, + mapping=Mapping(world_size=4, tp_size=4, moe_ep_size=4), + ), + w={}, + ) + table = weights.MODEL_WEIGHTS + dimensions = table.dims(core) + index = json.loads((root / "consolidated.safetensors.index.json").read_text())["weight_map"] + with torch.inference_mode(): + for entry in table.WEIGHTS: + if entry.tag != "v" and not entry.name.startswith("vis_"): + continue + for key, layer, prefix in table._targets(entry, dimensions): + destination = torch.empty( + entry.shape(dimensions), dtype=entry.dtype or dimensions.dtype, device="cuda" + ) + for name, selection in entry.sources(dimensions, layer, prefix): + with safe_open(str(root / index[name]), framework="pt") as shard: + value = shard.get_tensor(name) + if entry.transform: + value = entry.transform(value, core) + destination[selection if selection is not None else ...].copy_(value) + core.w[key] = destination + tower = modeling.VisionTower(core) + processor = MistralCommonImageProcessor( + MistralTokenizer.from_pretrained(checkpoint), torch.bfloat16 + ) + red = Image.new("RGB", (224, 168), "red") + blue = Image.new("RGB", (168, 280), "blue") + green = Image.new("RGB", (336, 112), "green") + + def encode(images): + processed = processor(text="", images=images) + params = [MultimodalParams(multimodal_data={"image": processed})] + output = tower.forward(params) + assert output.shape[0] == sum( + height * width // (PATCH * 2) ** 2 + for height, width in processed["image_sizes"].tolist() + ) + return output + + singles = [encode([image]) for image in (red, blue, green)] + for order in ((0, 1), (1, 0), (0, 2), (0, 1, 1)): + images = (red, blue, green) + actual = encode([images[index] for index in order]) + expected = torch.cat([singles[index] for index in order]) + relative_rms = ( + actual.float() - expected.float() + ).square().mean().sqrt() / expected.float().square().mean().sqrt() + assert relative_rms < 0.005, (order, relative_rms.item()) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/test_native_chat_template.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/test_native_chat_template.py new file mode 100644 index 000000000000..e10ba6eb9621 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/test_native_chat_template.py @@ -0,0 +1,499 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""A mistral_large_4 chat request renders to the token ids mistral-common gives the raw +OpenAI messages. + +mistral-common, not a Jinja template, renders the chat. The serving rules: + +- the top-level `reasoning_effort` selects the reasoning mode; a `reasoning_effort` + in `chat_template_kwargs` is accepted too and must agree with it; +- a request `chat_template` is rejected; `documents` and the `chat_template_kwargs` + keys the encoder does not read are ignored; +- `add_generation_prompt` and `continue_final_message` must agree with the final + message; +- tools reach the encoder without a null `strict`, and each text part of a content + list reaches it as its own chunk; +- a `json_object` or named `json_schema` response format is stated in the prompt's + model settings; +- a named `tool_choice`, whose prefix is text, is a bad request on rendered token ids. + +The request-rule tests run on CPU without a checkpoint. The rendering tests need +MISTRAL_LARGE4_CHECKPOINT_DIR (its tekken.json) and compare the server's rendering steps +with mistral-common encoding the raw request messages. +""" + +from __future__ import annotations + +import asyncio +import base64 +import copy +import os +from io import BytesIO +from types import SimpleNamespace +from unittest.mock import patch + +import numpy as np +import pytest +from PIL import Image + +from tensorrt_llm.inputs import utils as inputs_utils +from tensorrt_llm.inputs.utils import apply_chat_template, async_apply_chat_template_with_mm +from tensorrt_llm.serve.chat_utils import parse_chat_messages_coroutines +from tensorrt_llm.serve.openai_protocol import ChatCompletionRequest +from tensorrt_llm.serve.serving_extensions import apply_model_chat_extensions + +pytestmark = pytest.mark.cpu_only + +_WEATHER_TOOL = { + "type": "function", + "function": { + "name": "get_weather", + "description": "Weather for a city.", + "parameters": { + "type": "object", + "properties": {"city": {"type": "string"}}, + "required": ["city"], + }, + }, +} + + +class _RecordingTokenizer: + """Stands in for the native tokenizer and records each chat mistral-common is asked to encode.""" + + def __init__(self): + self.calls = [] + + def encode(self, tokenizer, messages, image_sizes, **kwargs): + assert tokenizer is self + self.calls.append((messages, kwargs)) + return [1] + + +def _render(tokenizer, messages, *, add_generation_prompt=True, **kwargs): + with patch.object(inputs_utils, "_encode_native_mistral_chat", tokenizer.encode): + return apply_chat_template( + model_type="mistral_large_4", + tokenizer=tokenizer, + processor=None, + conversation=messages, + add_generation_prompt=add_generation_prompt, + mm_placeholder_counts=[{} for _ in messages], + **kwargs, + ) + + +def _request(**fields): + fields.setdefault("messages", [{"role": "user", "content": "Hi."}]) + return ChatCompletionRequest(model="m", **fields) + + +def test_only_encoder_kwargs_reach_the_renderer(): + tokenizer = _RecordingTokenizer() + _render( + tokenizer, + [{"role": "user", "content": "Hi."}], + chat_template_kwargs={"reasoning_effort": "high", "enable_thinking": False, "foo": 1}, + documents=[{"title": "t", "text": "x"}], + ) + ((_, kwargs),) = tokenizer.calls + assert kwargs == { + "tools": None, + "continue_final_message": False, + "reasoning_effort": "high", + "response_format": None, + } + + +def test_unset_reasoning_effort_is_left_to_the_checkpoint(): + tokenizer = _RecordingTokenizer() + _render(tokenizer, [{"role": "user", "content": "Hi."}]) + ((_, kwargs),) = tokenizer.calls + assert kwargs["reasoning_effort"] is None + + +def test_continue_final_message_needs_a_final_assistant_message(): + tokenizer = _RecordingTokenizer() + messages = [{"role": "user", "content": "Finish:"}, {"role": "assistant", "content": "The"}] + _render( + tokenizer, + messages, + add_generation_prompt=False, + chat_template_kwargs={"continue_final_message": True}, + ) + assert tokenizer.calls[0][1]["continue_final_message"] is True + + +def test_top_level_continue_final_message_reaches_native_renderer(): + request = _request( + messages=[{"role": "user", "content": "Hi"}, {"role": "assistant", "content": "Hello"}], + add_generation_prompt=False, + continue_final_message=True, + ) + tokenizer = _RecordingTokenizer() + _render( + tokenizer, + request.messages, + add_generation_prompt=request.add_generation_prompt, + chat_template_kwargs=request.chat_template_kwargs, + ) + assert tokenizer.calls[0][1]["continue_final_message"] is True + + +def test_top_level_continue_final_message_rejects_generation_prompt(): + with pytest.raises(ValueError, match="Cannot set both"): + _request(continue_final_message=True) + + +@pytest.mark.parametrize("stream", [False, True]) +@pytest.mark.parametrize("limit_field", ["max_tokens", "max_completion_tokens"]) +def test_chat_rejects_oversized_output_before_generation(tmp_path, stream, limit_field): + from test_common.serve.app_factory import AggConfig, isolated_serve_env + + with isolated_serve_env(tmp_path) as env: + server = env.build_openai_server(AggConfig(model_name="m")) + server.model_config = SimpleNamespace(model_type="mistral_large_4") + server.generator.args.max_seq_len = 64 + response = env.client(server.app).post( + "/v1/chat/completions", + json={ + "model": "m", + "messages": [{"role": "user", "content": "Hi"}], + "stream": stream, + limit_field: 999999999, + }, + ) + assert response.status_code == 400, response.text + assert "exceeds the maximum context length of 64 tokens" in response.text + + +@pytest.mark.parametrize( + "last_role,add_generation_prompt,continue_final_message,match", + [ + ("assistant", True, False, "last message is from the assistant"), + ("assistant", True, True, "Cannot set both"), + ("user", False, True, "last message is not from the assistant"), + ], +) +def test_generation_flags_must_agree_with_the_final_message( + last_role, add_generation_prompt, continue_final_message, match +): + messages = [{"role": "user", "content": "Hi."}] + if last_role == "assistant": + messages.append({"role": "assistant", "content": "Hello"}) + tokenizer = _RecordingTokenizer() + with pytest.raises(ValueError, match=match): + _render( + tokenizer, + messages, + add_generation_prompt=add_generation_prompt, + chat_template_kwargs={"continue_final_message": continue_final_message}, + ) + assert tokenizer.calls == [] + + +def test_null_strict_is_dropped_from_tools(): + tools = [ + {"type": "function", "function": {"name": "a", "description": None, "strict": None}}, + {"type": "function", "function": {"name": "b", "strict": True}}, + ] + original = copy.deepcopy(tools) + tokenizer = _RecordingTokenizer() + _render(tokenizer, [{"role": "user", "content": "Hi."}], tools=tools) + assert tokenizer.calls[0][1]["tools"] == [ + {"type": "function", "function": {"name": "a", "description": None}}, + {"type": "function", "function": {"name": "b", "strict": True}}, + ] + assert tools == original + + +@pytest.mark.parametrize("effort", ["low", "high"]) +def test_top_level_reasoning_effort_reaches_the_renderer(effort): + request = _request(reasoning_effort=effort, chat_template_kwargs={"x": 1}) + apply_model_chat_extensions(request, "mistral_large_4") + assert request.chat_template_kwargs == {"x": 1, "reasoning_effort": effort} + + +def test_an_unset_reasoning_effort_derives_nothing(): + """The protocol's own default is not a mistral-common effort and must not leak in.""" + request = _request() + apply_model_chat_extensions(request, "mistral_large_4") + assert request.chat_template_kwargs is None + + +def test_reasoning_effort_in_kwargs_is_kept_and_must_agree(): + request = _request(chat_template_kwargs={"reasoning_effort": "low", "x": 1}) + apply_model_chat_extensions(request, "mistral_large_4") + assert request.chat_template_kwargs == {"reasoning_effort": "low", "x": 1} + + request = _request(reasoning_effort="high", chat_template_kwargs={"reasoning_effort": "high"}) + apply_model_chat_extensions(request, "mistral_large_4") + assert request.chat_template_kwargs == {"reasoning_effort": "high"} + + with pytest.raises(ValueError, match="conflicts"): + apply_model_chat_extensions( + _request(reasoning_effort="high", chat_template_kwargs={"reasoning_effort": "low"}), + "mistral_large_4", + ) + + +def test_the_renderer_judges_the_reasoning_effort_name(): + """The extension passes an effort on unchecked; mistral-common rejects it at rendering.""" + request = _request(chat_template_kwargs={"reasoning_effort": "turbo"}) + apply_model_chat_extensions(request, "mistral_large_4") + assert request.chat_template_kwargs == {"reasoning_effort": "turbo"} + + +def test_a_request_chat_template_is_rejected(): + with pytest.raises(ValueError, match="chat_template is not supported"): + apply_model_chat_extensions(_request(chat_template="{{ messages }}"), "mistral_large_4") + + +_CITY_FORMAT = { + "type": "json_schema", + "json_schema": { + "name": "city", + "schema": { + "type": "object", + "properties": {"city": {"type": "string"}}, + "required": ["city"], + }, + "strict": True, + }, +} + + +@pytest.mark.parametrize( + "response_format,passed", + [(_CITY_FORMAT, True), ({"type": "json_object"}, True), ({"type": "text"}, False)], +) +def test_a_response_format_reaches_the_renderer(response_format, passed): + request = _request( + response_format=response_format, + chat_template_kwargs={"response_format": {"type": "json_object"}}, + ) + apply_model_chat_extensions(request, "mistral_large_4") + expected = request.response_format.model_dump(exclude_none=True) if passed else None + assert (request.chat_template_kwargs or {}).get("response_format") == expected + + +def test_a_named_tool_choice_on_token_ids_is_a_bad_request(tmp_path, monkeypatch): + from test_common.serve.app_factory import AggConfig, isolated_serve_env + + from tensorrt_llm.serve.render import chat as render_chat_module + + async def render_token_ids(**kwargs): + return [1, 2, 3] + + monkeypatch.setattr(render_chat_module, "async_apply_chat_template", render_token_ids) + + class _Config: + model_type = "mistral_large_4" + vocab_size = 1024 + + with isolated_serve_env(tmp_path) as env: + server = env.build_openai_server(AggConfig(model_name="m", tool_parser="qwen3")) + server.model_config = _Config() + server.tokenizer = SimpleNamespace(tokenizer=SimpleNamespace(vocab_size=_Config.vocab_size)) + server.generator.args.guided_decoding_backend = "xgrammar" + response = env.client(server.app).post( + "/v1/chat/completions", + json={ + "model": "m", + "messages": [{"role": "user", "content": "Weather in Paris?"}], + "tools": [_WEATHER_TOOL], + "tool_choice": {"type": "function", "function": {"name": "get_weather"}}, + "max_tokens": 4, + }, + ) + assert response.status_code == 400, response.text + assert "renders token ids" in response.text + + +def _config(model_type): + return type("Config", (), {"model_type": model_type})() + + +@pytest.mark.parametrize("model_type,kept", [("mistral_large_4", True), ("llama", False)]) +def test_text_part_lists_keep_their_parts(model_type, kept): + messages = [ + { + "role": "system", + "content": [ + {"type": "text", "text": "Rule one."}, + {"type": "text", "text": "Rule two."}, + ], + }, + {"role": "user", "content": "One string."}, + ] + conversation, media, _, _ = parse_chat_messages_coroutines(messages, _config(model_type)) + asyncio.run(media) + assert conversation[0]["content"] == "Rule one.\nRule two." + assert conversation[0].get("content_parts") == (["Rule one.", "Rule two."] if kept else None) + assert "content_parts" not in conversation[1] + + +@pytest.fixture(scope="module") +def tokenizer(): + if not os.environ.get("MISTRAL_LARGE4_CHECKPOINT_DIR"): + pytest.skip("needs checkpoint tokenizer") + from tensorrt_llm._torch.models.checkpoints.mistral.tokenizer import MistralTokenizer + + return MistralTokenizer.from_pretrained(os.environ["MISTRAL_LARGE4_CHECKPOINT_DIR"]) + + +def _server_ids(tokenizer, **fields): + """The openai_chat rendering steps for one mistral_large_4 request.""" + from tensorrt_llm._torch.configs.mistral_large4 import MistralLarge4Config + + request = _request(**copy.deepcopy(fields)) + apply_model_chat_extensions(request, "mistral_large_4") + conversation, media, counts, _ = parse_chat_messages_coroutines( + request.messages, MistralLarge4Config(dtype="bfloat16", vocab_size=131072) + ) + tools = None if request.tools is None else [tool.model_dump() for tool in request.tools] + ids, _ = asyncio.run( + async_apply_chat_template_with_mm( + media, + model_type="mistral_large_4", + tokenizer=tokenizer, + processor=None, + conversation=conversation, + add_generation_prompt=request.add_generation_prompt, + mm_placeholder_counts=counts, + tools=tools, + documents=request.documents, + chat_template=request.chat_template, + chat_template_kwargs=request.chat_template_kwargs or {}, + ) + ) + return ids + + +def _raw_ids(tokenizer, messages, **kwargs): + """mistral-common encoding the raw request messages.""" + return tokenizer.apply_chat_template(messages, tokenize=True, return_dict=False, **kwargs) + + +def _image_url(seed, size): + pixels = np.random.default_rng(seed).integers(0, 256, size=(*size, 3), dtype=np.uint8) + buffer = BytesIO() + Image.fromarray(pixels, "RGB").save(buffer, "PNG") + return "data:image/png;base64," + base64.b64encode(buffer.getvalue()).decode() + + +def test_text_part_lists_render_part_by_part(tokenizer): + messages = [ + {"role": "system", "content": [{"type": "text", "text": "Rule one."}] * 2}, + { + "role": "user", + "content": [{"type": "text", "text": "First part."}, {"type": "text", "text": "Next."}], + }, + ] + ids = _server_ids(tokenizer, messages=messages) + assert ids == _raw_ids(tokenizer, messages) + joined = [ + {"role": m["role"], "content": "\n".join(p["text"] for p in m["content"])} for m in messages + ] + assert ids != _raw_ids(tokenizer, joined) + + +def test_tools_render_as_sent(tokenizer): + messages = [{"role": "user", "content": "Weather in Paris?"}] + ids = _server_ids(tokenizer, messages=messages, tools=[_WEATHER_TOOL]) + assert ids == _raw_ids(tokenizer, messages, tools=[_WEATHER_TOOL]) + assert ids != _raw_ids(tokenizer, messages) + + +def test_top_level_reasoning_effort_renders(tokenizer): + messages = [{"role": "user", "content": "Think."}] + high = _server_ids(tokenizer, messages=messages, reasoning_effort="high") + assert high == _raw_ids(tokenizer, messages, reasoning_effort="high") + assert high != _raw_ids(tokenizer, messages, reasoning_effort="low") + assert _server_ids(tokenizer, messages=messages) == _raw_ids(tokenizer, messages) + + +@pytest.mark.parametrize( + "fields", + [ + {"reasoning_effort": "max"}, + {"chat_template_kwargs": {"reasoning_effort": "minimal"}}, + {"chat_template_kwargs": {"reasoning_effort": "turbo"}}, + ], + ids=["undeclared", "undeclared_in_kwargs", "unknown_in_kwargs"], +) +def test_a_reasoning_effort_the_checkpoint_does_not_declare_is_an_invalid_request( + tokenizer, fields +): + # openai_chat returns HTTP 400 for a ValueError raised while rendering. + with pytest.raises(ValueError, match="reasoning_effort"): + _server_ids(tokenizer, messages=[{"role": "user", "content": "Think."}], **fields) + + +def test_ignored_fields_leave_the_tokens_unchanged(tokenizer): + messages = [{"role": "user", "content": "Summarize."}] + ids = _server_ids( + tokenizer, + messages=messages, + documents=[{"title": "t", "text": "Some text."}], + chat_template_kwargs={"enable_thinking": True}, + ) + assert ids == _raw_ids(tokenizer, messages) + + +def test_a_single_image_turn_renders_as_before(tokenizer): + """The frozen MMMU prompts: one user turn of text and images, at reasoning effort low.""" + content = [ + {"type": "text", "text": "Question: "}, + {"type": "image_url", "image_url": {"url": _image_url(1, (48, 64))}}, + {"type": "text", "text": " Options: A, B."}, + ] + messages = [{"role": "user", "content": content}] + ids = _server_ids( + tokenizer, messages=messages, chat_template_kwargs={"reasoning_effort": "low"} + ) + assert ids == _raw_ids(tokenizer, messages, return_tensors=None, reasoning_effort="low") + + +def _direct_ids(tokenizer, messages, **kwargs): + """mistral-common encoding the raw request messages with request-level settings.""" + from mistral_common.protocol.instruct.request import ChatCompletionRequest + + request = ChatCompletionRequest.from_openai(messages=messages, **kwargs) + return tokenizer.mistral.encode_chat_completion(request).tokens + + +def test_a_response_format_renders_in_the_model_settings(tokenizer): + from mistral_common.protocol.instruct.request import JsonSchema, ResponseFormat, ResponseFormats + + messages = [{"role": "user", "content": "Give a city as JSON."}] + named = _CITY_FORMAT["json_schema"] + schema = ResponseFormat( + type=ResponseFormats.json_schema, + json_schema=JsonSchema(name=named["name"], schema=named["schema"], strict=True), + ) + ids = _server_ids(tokenizer, messages=messages, response_format=_CITY_FORMAT) + assert ids == _direct_ids(tokenizer, messages, response_format=schema) + assert ids != _raw_ids(tokenizer, messages) + assert _server_ids( + tokenizer, messages=messages, response_format={"type": "json_object"} + ) == _direct_ids( + tokenizer, messages, response_format=ResponseFormat(type=ResponseFormats.json_object) + ) + text = _server_ids(tokenizer, messages=messages, response_format={"type": "text"}) + assert text == _raw_ids(tokenizer, messages) + + +def test_a_response_format_keeps_the_image_grid(tokenizer): + from mistral_common.protocol.instruct.request import ResponseFormat, ResponseFormats + + content = [ + {"type": "text", "text": "Describe as JSON: "}, + {"type": "image_url", "image_url": {"url": _image_url(2, (48, 64))}}, + ] + messages = [{"role": "user", "content": content}] + assert _direct_ids(tokenizer, messages) == _raw_ids(tokenizer, messages) + ids = _server_ids(tokenizer, messages=messages, response_format={"type": "json_object"}) + assert ids == _direct_ids( + tokenizer, messages, response_format=ResponseFormat(type=ResponseFormats.json_object) + ) diff --git a/tests/unittest/_torch/modeling_v2/models/mistral_large4/test_native_dspark.py b/tests/unittest/_torch/modeling_v2/models/mistral_large4/test_native_dspark.py new file mode 100644 index 000000000000..5dddfd905638 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/mistral_large4/test_native_dspark.py @@ -0,0 +1,491 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Native DSpark math independently of weights, CUDA kernels and runtime sampling.""" + +import importlib +import json +import struct +from types import MethodType, SimpleNamespace +from unittest.mock import Mock + +import pytest +import torch + +pytestmark = pytest.mark.cpu_only + + +@pytest.fixture(scope="module") +def native_checkpoint(): + from utils.llm_data import llm_models_root + + root = llm_models_root() + path = None if root is None else root / "Mistral-Large-4-1T-A52B-NVFP4" + if path is None or not (path / "params.json").is_file(): + pytest.skip("the NVFP4 checkpoint is not under LLM_MODELS_ROOT") + index = json.loads((path / "consolidated.safetensors.index.json").read_text())["weight_map"] + keys = {key for key in index if key.startswith("mtp_dspark.")} + headers = {} + for shard in {index[key] for key in keys}: + with (path / shard).open("rb") as stream: + size = struct.unpack("