Repository navigation
[AgentX] MiniMax-M3.1-Flash GB300 Dynamo TRT: add disaggregated points / [AgentX] MiniMax-M3.1-Flash GB300 Dynamo TRT:新增分离式运行点 - #5066
Conversation
…ints Add seven NVFP4 disaggregated AgentX points for minimaxm31flash-fp4-gb300-dynamo-trt-agentic-dspark-disagg on the TensorRT-LLM feat-m31-flash-perf2-gb-clean image with Dynamo 1.7.0.dev20261028: TP4/TP8 1P3D and 1P4D points plus DEP2 prefill with DEP8/DEP16 decode. GPU and host CPU power collection are enabled. Co-authored-by: Pietro Cicotti <5833013+pcicotti@users.noreply.github.com>
|
Thanks for the contribution!
中文感谢你的贡献!
|
| speculative_model: /model/dspark | ||
| cache_transceiver_config: | ||
| backend: NIXL | ||
| transceiver_runtime: PYTHON | ||
| max_tokens_in_buffer: 524288 | ||
| kv_transfer_timeout_ms: 600000 | ||
| kv_cache_bounce_size_mb: 1024 | ||
| agent_bounce_buffer_enable: true | ||
| agent_bounce_params: | ||
| max_average_descriptor_size: 4MB | ||
| min_descriptor_count: '1' | ||
| enable_autotuner: true | ||
| enable_tokenization_cache: true | ||
| stream_interval: 20 | ||
| reasoning_parser: minimax_m3 | ||
| print_iter_log: false | ||
| enable_iter_perf_stats: true | ||
| iter_perf_stats_interval: 20 | ||
| return_perf_metrics: true | ||
| backend: pytorch | ||
| gpus_per_node: 4 | ||
| env_overrides: | ||
| TRTLLM_DISAGG_ROLE: context | ||
| decode: | ||
| env: | ||
| TLLM_LOG_LEVEL: INFO | ||
| TRTLLM_SERVER_DISABLE_GC: '1' | ||
| TRTLLM_WORKER_DISABLE_GC: '1' | ||
| TRTLLM_ENABLE_PDL: '1' | ||
| NCCL_GRAPH_MIXING_SUPPORT: '0' | ||
| PYTHONNOUSERSITE: '1' | ||
| TQDM_DISABLE: '1' | ||
| HF_HUB_OFFLINE: '1' | ||
| TRANSFORMERS_OFFLINE: '1' | ||
| PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True | ||
| TRTLLM_KV_LAYERWISE_ONBOARD: '0' | ||
| TRTLLM_KV_ONBOARD_CTAS: '12' | ||
| TRTLLM_M31_NO_MIX: '1' | ||
| TRTLLM_M31_NO_MIX_DECODE_INTERVAL: '1' | ||
| TRTLLM_KV_TRANSFER_NUM_THREADS: '4' | ||
| PYTHONDONTWRITEBYTECODE: '1' | ||
| DYN_ROUTER_MODE: least-loaded | ||
| DYN_REASONING_PARSER: minimax_m3 | ||
| DYN_TOOL_CALL_PARSER: minimax_m3 | ||
| DYN_CUSTOM_JINJA_TEMPLATE: /model/chat_template.jinja | ||
| args: | ||
| max_num_tokens: 16384 | ||
| disable_overlap_scheduler: false | ||
| cuda_graph_config: | ||
| enable_padding: true | ||
| trust_remote_code: true | ||
| max_seq_len: 1048576 | ||
| attn_backend: TRTLLM | ||
| enable_chunked_prefill: true | ||
| allreduce_strategy: AUTO | ||
| moe_config: | ||
| backend: TRTLLM | ||
| use_low_precision_moe_combine: false | ||
| sparse_attention_config: | ||
| algorithm: minimax_m3 | ||
| implementation: msa | ||
| sparse_disable_index_value: true | ||
| fuse_qkv_index_projection: true | ||
| kv_cache_config: | ||
| dtype: nvfp4 | ||
| tokens_per_block: 128 | ||
| use_kv_cache_manager_v2: true | ||
| free_gpu_memory_fraction: 0.94 | ||
| iteration_stats_interval: 20 | ||
| speculative_config: | ||
| decoding_type: DSpark | ||
| max_draft_len: 7 | ||
| block_size: 7 | ||
| use_rejection_sampling: true | ||
| advanced_sampling_mode: no_topk | ||
| attention_backend: TRTLLM | ||
| speculative_model: /model/dspark |
There was a problem hiding this comment.
🟡 (optional) All seven new points will publish their draft model as the container path /model/dspark. Every other MiniMax-M3.1-Flash DSpark result, and the golden curve, records native. TensorRT-LLM's speculative_model: /model/dspark reaches speculation_facts (infx/srt_slurm/spec_identity.py:48-54), which maps only Hugging Face cache paths to a repo and returns any other string unchanged. Fix: make speculation_facts report native for a draft loaded from inside the served checkpoint mount (/model/...). It must still map HF-cache paths to org/name, because the musespark TRT recipe relies on that.
Why this was flagged
For each row, bind_multinode calls bind_speculation (infx/srt_slurm/workload.py:479-496). That reads the decode role's speculative_config through spec_parameters (infx/srt_slurm/synthetic_acceptance.py:125-132), which returns model: /model/dspark. speculation_facts (spec_identity.py:51-54) checks only HF_CACHE_REPO, so it returns /model/dspark, and the binder writes SPEC_DRAFT_MODEL=/model/dspark into benchmark.env. speculation_fields (infx/results/metadata.py:45-48) then publishes it as spec_draft_model in every agentic result. The draft is the checkpoint's bundled DSpark head, and the docstring says such a draft should be reported as native. So these rows are the first to publish a container-local path in that field. Any comparison that groups MiniMax DSpark results by draft model will treat them as a different draft. The fingerprint excludes SPEC_DRAFT_MODEL (infx/matrix/fingerprint.py:22), and compare ignores draft, so no check catches this.
Verification: Both roles set speculative_model: /model/dspark (disagg-variants.yaml:79 and :155). speculation_facts (infx/srt_slurm/spec_identity.py:48-54) rewrites only paths that match HF_CACHE_REPO and otherwise returns model or "native", so the string /model/dspark passes through unchanged.
@Ankur-singh TTFT/E2E needs improving |
|
@Oseltamivir whats wrong? I don't see any problem with either TTFT/E2EL. Can you please clarify? |
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=43388252410 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=43388252410 |
|
/use 43388252410 [by Claude Code] |
|
As a PR reviewer and CODEOWNER, I have reviewed this and have: Checklist items
Additional detail section:
Signed: [by Claude Code] |
|


Summary
Add seven disaggregated AgentX points for MiniMax-M3.1-Flash NVFP4 on GB300 with Dynamo and TensorRT-LLM (
minimaxm31flash-fp4-gb300-dynamo-trt-agentic-dspark-disagg), using thefeat-m31-flash-perf2-gb-clean-1.4.0a0.dev202611061956TensorRT-LLM image and Dynamo1.7.0.dev20261028. The points extend the TRT disaggregated curve from TP4 low-latency layouts up to DEP2 prefill with DEP8/DEP16 decode:All points use DSpark draft length 7 with golden-curve synthetic acceptance, NVFP4 KV cache, native DRAM KV offload and NIXL transfer; the indexer KV cache is FP8 at 1P3D c36 and NVFP4 elsewhere. Every TensorRT-LLM worker sets
stream_interval: 20, iteration and KV-cache statistics at interval 20, and publishes metrics. GPU and host CPU power collection are enabled on every point.Overlap with #4751: #4751 adds the same
…-dspark-disaggconfig key anddisagg-variants.yamlon the earlierfeat-m31-flash-perf1image. This PR replaces that disaggregated half; #4751 should keep only its…-dspark-aggkey before either PR merges, otherwise the two conflict.Risks:
TRTLLM_M31_SM103_TP2_TARGET=1. Effective DSpark draft precision on that path still has to be confirmed from the sweep logs.AI model disclosure
claude-opus-5-5andclaude-sonnet-5-5(Claude Code).claude-sonnet-5-5did the initial config inspection;claude-opus-5-5ported the config bundle, enabled power collection, wrote the changelog entry, ran the local checks and drafted this description. No delegated agents contributed.PR checklist and change type
Type of Change
Checklist
python -m infx.matrix.changed --base origin/mainlists affected config keys, I have appended one new entry covering them to the physical end ofinferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entriesinferencex-e2e/perf-changelog.yamlentry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs):full-sweep-fail-fast(recommended),full-sweep-enabled, ornon-canary-full-sweep-enabled. Optional modifiersall-evals,evals-only, andagentx-fastrequire a primary label; the last two block reuse while applied./use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.中文
为 GB300 上的 MiniMax-M3.1-Flash NVFP4 新增七个 Dynamo + TensorRT-LLM 分离式 AgentX 运行点(
minimaxm31flash-fp4-gb300-dynamo-trt-agentic-dspark-disagg),使用 TensorRT-LLM 镜像feat-m31-flash-perf2-gb-clean-1.4.0a0.dev202611061956和 Dynamo1.7.0.dev20261028。覆盖范围从 TP4 低延迟布局延伸到 DEP2 prefill 搭配 DEP8/DEP16 decode:1P3D(TP4 / 3 × TP4)并发 36、56、76;1P4D(TP4 / 4 × TP8)并发 60;4P1D(4 × DEP2 / DEP8)并发 320;5P1D(5 × DEP2 / DEP16)并发 300;7P1D(7 × DEP2 / DEP8)并发 480。所有运行点均使用 DSpark 草稿长度 7 及 golden-curve synthetic acceptance、NVFP4 KV cache、原生 DRAM KV offload 和 NIXL 传输;indexer KV cache 在 1P3D c36 为 FP8,其余运行点为 NVFP4。每个 TensorRT-LLM worker 设置
stream_interval: 20,迭代统计与 KV cache 统计间隔为 20,并发布 metrics。所有运行点均开启 GPU 及主机 CPU 功耗采集。与 #4751 的重叠: #4751 基于较早的
feat-m31-flash-perf1镜像,新增了相同的…-dspark-disaggconfig key 和disagg-variants.yaml。本 PR 取代其分离式部分;#4751 应在任一 PR 合并前只保留…-dspark-aggkey,否则两者会冲突。风险:
TRTLLM_M31_SM103_TP2_TARGET=1,该路径下 DSpark 草稿模型的实际精度仍需通过 sweep 日志确认。AI 模型使用说明:
claude-sonnet-5-5(Claude Code)完成初步配置检查;claude-opus-5-5(Claude Code)移植配置、开启功耗采集、编写 changelog 条目、运行本地检查并起草本说明。没有委派其他 agent。变更类型:配置变更。检查清单仅在上方填写。