Repository navigation
Add MiniMax M3.1 Flash GB300 Dynamo TRT AgentX configurations / 新增 MiniMax M3.1 Flash GB300 Dynamo TRT AgentX 配置 - #4751
RohitNagraj wants to merge 8 commits into
Conversation
新增 MiniMax M3.1 Flash GB300 TRT AgentX 配置。
|
Thanks for the contribution!
中文感谢你的贡献!
|
为 GB300 TensorRT-LLM AgentX 配置启用功耗采集,并补充缓存数据类型说明。
为 GB300 TRT AgentX 更新日志填写拉取请求链接。
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=42086507846 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=42086507846 |
| gpus_per_node: 4 | ||
| engine: | ||
| type: trtllm | ||
| served_model_name: MiniMaxAI/MiniMax-M3.1-Flash |
There was a problem hiding this comment.
add
numa_cpu_bind: true
numa_memory_bind: preferred
| env: | ||
| ETCD_LEASE_TTL: '120' | ||
| DYN_ROUTER_ACTIVE_REQUEST_EXPIRY_SECS: '3600' | ||
| DYN_TOKENIZER_CACHE_BYTES: '8589934592' |
There was a problem hiding this comment.
add
DYN_TOKENIZER: fastokens
| TRTLLM_M31_NO_MIX_DECODE_INTERVAL: '1' | ||
| TRTLLM_KV_TRANSFER_NUM_THREADS: '4' | ||
| DYN_ROUTER_MODE: kv | ||
| DYN_ROUTER_SESSION_AFFINITY_TTL_SECS: '14400' |
There was a problem hiding this comment.
change
DYN_ROUTER_MODE: kv
DYN_ROUTER_SESSION_AFFINITY_TTL_SECS: '14400'
on decode workers to
DYN_ROUTER_MODE: least-loaded
please leave prefill worker's session affinity settings untouched.
| TRTLLM_KV_ONBOARD_CTAS: '12' | ||
| TRTLLM_KV_TRANSFER_NUM_THREADS: '4' | ||
| DYN_ROUTER_MODE: kv | ||
| DYN_ROUTER_SESSION_AFFINITY_TTL_SECS: '14400' |
There was a problem hiding this comment.
add
DYN_ENGINE_CONV_AFFINITY: '1'
on prefill workers.
合并 main 分支以更新 MiniMax AgentX 配置。
调整 MiniMax AgentX 路由、分词器、NUMA 绑定及遥测配置。
合并 main 分支并解决 MiniMax GB300 配置冲突。
Add 1P4D TP4 at concurrency 40, 36 and 32, and configure 1P3D TP4 at 32, 28 and 24. 新增并行度为 40、36 和 32 的 1P4D TP4 配置,并将 1P3D TP4 的并行度设为 32、28 和 24。
更新 GB300 AgentX 配置矩阵。
|
/use 42086507846 |
|
@RohitNagraj staged run 42086507846. This run remains available across future |
|
As a PR reviewer and CODEOWNER, I have reviewed this and have: Checklist items
Additional detail section:
Pareto (tput/GPU vs P90 E2EL): 7 of 10 points are on the frontier. On tput/GPU vs mean interactivity, all 10 are.
Edited by Signed: |
❌❌❌ REJECTED ❌❌❌@kedarpotdar-nv Blocking: the new ❌ Check 18 (Recipe, image, and sweep consistency): FAIL — (f2) The changelog entry leaves out a quantization change. The TP2 agg workers (c22, c32, c44) and the TP2 decode workers in 1P2D c32 and 1P3D c36 set
Passed and not applicable checks✅ Check 0 (CODEOWNER): PASS — ✅ Check 1 (Sweep and evals on an in-PR commit): PASS — Run 42086507846 ran on the pinned head ✅ Check 2 (Eval accuracy): PASS — All 9 gsm8k rows ran on ➖ Check 3 (Upstream recipe): N/A — Both affected configs are ✅ Check 4 (Reuse command): PASS — ✅ Check 5 (Checklist template): PASS — All 20 items of the current ✅ Check 6 (Upstream images and engine-first): PASS — (a) does not apply because the framework is TRT-LLM. (b) GB300 MiniMax-M3.1-Flash ✅ Check 7 (Deprecated models and scenarios): PASS — ✅ Check 8 (Architecture hacks and all-reduce): PASS — There are no architecture overrides; ✅ Check 9 (Spec-decode chat template): PASS — AgentX traffic goes through the Dynamo chat frontend, with ✅ Check 10 (Engine patches): PASS — There are no patches, setup scripts or source edits. The only install is the published Dynamo wheel ✅ Check 11 (Golden AL): PASS — The c22 job log shows ➖ Check 12 (Append-only): N/A — The new changelog entry has no ✅ Check 14 (Pareto coverage): PASS — The single AgentX curve (minimaxm31flash, agentic-coding, gb300_dynamo-trt, fp4, P90 E2EL vs tput/GPU, run 42086507846 on ✅ Check 15 (PR description): PASS — The body's checked claims (10 benchmark and 9 eval jobs, GSM8K 96.0–97.0%, Dynamo ➖ Check 16 (NVIDIA overview): N/A — Only TensorRT-LLM ( ✅ Check 17 (Metrics publication): PASS — Dynamo Assessed commit: |
| allreduce_strategy: AUTO | ||
| moe_config: | ||
| backend: TRTLLM | ||
| use_low_precision_moe_combine: true |
Add aggregated and disaggregated GB300 AgentX configurations for
MiniMaxAI/MiniMax-M3.1-Flashusing Dynamo, TensorRT-LLM, DSpark, native DRAM KV offload, and NIXL transfer. Configure iteration and KV-cache statistics for each TensorRT-LLM worker, with iteration logging disabled.Validation from the pinned sweep:
1.7.0.dev20261028.TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS=3.04(golden AL 4.04 minus 1) on every speculating role. Eval-only jobs ran without it.MiniMaxAI/MiniMax-M3.1-Flash(dspark/), loaded withdtype='auto'. There is no submission-side quantization, dtype override or checkpoint substitution. The shipped attention and MLP projections are MXFP8 (1x32 blocks), the other draft weights are BF16, and the draft KV cache is BF16. TP4 workers log the MXFP8 drafter GEMV on 20 projections. On TP2 workers,TRTLLM_M31_SM103_TP2_TARGET=1turns off the MXFP8 drafter GEMV; the effective drafter projection precision on that path is not yet verified.AI model disclosure
PR checklist and change type
Type of Change
Checklist
python -m infx.matrix.changed --base origin/mainlists affected config keys, I have appended one new entry covering them to the physical end ofinferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entriesinferencex-e2e/perf-changelog.yamlentry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs):full-sweep-fail-fast(recommended),full-sweep-enabled, ornon-canary-full-sweep-enabled. Optional modifiersall-evals,evals-only, andagentx-fastrequire a primary label; the last two block reuse while applied.OWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.中文
新增
MiniMaxAI/MiniMax-M3.1-Flash在 GB300 上的聚合式和分离式 AgentX 配置,使用 Dynamo、TensorRT-LLM、DSpark、原生 DRAM KV 卸载及 NIXL 传输。为每个 TensorRT-LLM worker 配置迭代统计和 KV 缓存统计,并关闭逐迭代日志。基于固定的 sweep的验证结果:
1.7.0.dev20261028一起运行。TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS=3.04(golden AL 4.04 减 1)。eval-only 任务未设置该变量。MiniMaxAI/MiniMax-M3.1-Flash自带的dspark/,以dtype='auto'加载,没有提交侧量化、dtype 覆盖或检查点替换。出厂的 attention 与 MLP 投影为 MXFP8(1x32 块),其余草稿权重为 BF16,草稿 KV 缓存为 BF16。TP4 worker 日志显示 MXFP8 drafter GEMV 作用于 20 个投影。在 TP2 worker 上,TRTLLM_M31_SM103_TP2_TARGET=1关闭了 MXFP8 drafter GEMV,该路径下草稿投影的实际精度尚未验证。AI 使用说明:使用 GPT-6 准备配置、recipe 和 PR 草稿;运行环境未提供可确认的完整版本标识。Devin(Cognition AI)根据固定 sweep 的产物更新了本描述中的验证结果。