…#9641)
<!-- .github/pull_request_template.md -->
## 📌 Description
Follow up merged #5489 by replacing Prims-TS MLA decode's
`use_kv_bounds` / per-query `kv_bounds` interface with context-parallel
topology and global request lengths.
- `plan()` specializes `cp_world` and `cp_interleave_size`.
- `run()` receives `cp_rank` and graph-live global `seq_lens_kv`,
including the current query tokens. Planned `max_kv_len` and block
tables remain rank-local physical capacity.
- Both 1-CTA and 2-CTA kernels derive rank-local lengths and causal
prefixes directly. Packed queries use actual `qo_indptr` differences;
rank changes do not enter the compilation key.
- Preserve CP=1 behavior, empty/inactive-row contracts, split reduction
and LSE. The optional neutral writer uses a flattened grid without a
65,535-request grid-Y limit.
- Validate split-KV cluster size against the CUDA device’s launch limit
as well as occupancy. On B200 an occupancy query can report active
clusters for 17 CTAs, even though only 16 can launch. Over-limit
automatic splits use the separate reducer.
- Update validation, trace schemas/examples, documentation and the
existing focused regression file. Seven GPU cases remain; no large new
matrix is introduced.
The branch is based on upstream main. CP documentation updates are
confined to the PrimTS READMEs; `docs/development/backend_notes.md` is
unchanged. No new context-kernel or standard-attention changes are
introduced by this PR.
### Retained integration performance evidence
Same MLA implementation validated in the TRT-LLM integration, with full
FP8 KV cache, page size 128, 96 heads, latent/RoPE dimensions 512/64.
H-D1/H-D8: local batch 1/8, Q=8, CP8 rank 0, local history 512 (global
total 4104). D-D8/D-D16: local batch 8/16, Q=6, CP1, history 4096.
Values are **previous PrimTS → updated PrimTS microseconds**, using the
conservative order of balanced AB/BA runs and predefined confirmations.
These are retained integration-snapshot measurements, not a new GPU run
on this freshly main-based PR head.
| Scope | Case | Rubin SM107 | B200 SM100 | B300 SM103 |
|---|---|---|---|---|
| Raw generation | H-D1 | 14.240 → 13.752 | 20.480 → 20.480 | 20.448 →
18.432 |
| Raw generation | H-D8 | 18.000 → 17.936 | 16.368 → 16.384 | 16.352 →
16.368 |
| Raw generation | D-D8 | 26.656 → 26.624 | 36.832 → 36.864 | 34.784 →
34.816 |
| Raw generation | D-D16 | 30.552 → 30.584 | 65.504 → 65.536 | 65.792 →
66.261 |
| Attention backend | H-D1 | 17.680 → 17.536 | 24.576 → 24.608 | 24.448
→ 22.560 |
| Attention backend | H-D8 | 23.360 → 23.840 | 24.576 → 24.472 | 22.496
→ 22.496 |
| Attention backend | D-D8 | 30.464 → 30.472 | 40.960 → 40.944 | 38.896
→ 38.896 |
| Attention backend | D-D16 | 36.984 → 36.992 | 73.712 → 73.712 | 73.744
→ 73.752 |
50 warmups and 100 CUDA-event samples per process, CUDA graph replay,
2×L2 flush outside event intervals, clocks unlocked;
startup/JIT/reference work excluded. Backend timing includes
metadata/preprocessing/cache work, raw generation does not. TRT-LLM
still prepares compatibility bounds for other providers, so no
eliminated producer cost is claimed. Rubin H-D8 backend has a valid
+2.055% regression, narrowly above the predefined 2% cap; other Rubin
generation scopes and all Blackwell scopes pass. These are rank-local
component results, not CP communication, full-model throughput or TTFT.
The consumer also contains a separate native single-`is_lse` merge
cleanup and dispatch-policy update; those are not FlashInfer changes.
## 🔍 Related Issues
- Follow-up to merged #5489.
- Consumer integration:
NVIDIA/TensorRT-LLM#19395
- Existing experimental PrimTS tracking context: #4829, #5059, #5082.
## 🚀 Pull Request Checklist
Thank you for contributing to FlashInfer! Before we review your pull
request, please make sure the following items are complete.
### ✅ Pre-commit Checks
- [x] I have installed `pre-commit` by running `pip install pre-commit`
(or used your preferred method).
- [ ] I have installed the hooks with `pre-commit install`.
- [ ] I have run the hooks manually with `pre-commit run --all-files`
and fixed any reported issues.
> Changed-file pre-commit checks passed, including mypy and Ruff. The
existing checkout disables automatic hooks; hooks were run manually. A
full-repository hook run is not claimed.
## 🧪 Tests
- Cluster-limit regression: the existing device-local occupancy test
passed. Three B200 numerical checks covered the 17-CTA fallback on both
devices and a legal 16-CTA cluster; both previously failing TRT-LLM TP2
MTP tests passed.
- [x] Tests have been added or updated as needed.
- [ ] All tests are passing (`unittest`, etc.).
- [ ] This PR requires **full-scope** CI, not targeted tests (expensive;
justify below).
Reason: <!-- e.g. cross-cutting change to shared infrastructure -->
Suggested CI test targets (optional; paths under `tests/`,
whitespace-separated — a starting
point for `@flashinfer-bot run <targets>` / `/bot run <targets>`,
reviewers may adjust): `tests/attention/test_attention_ts_mla_cp.py
tests/attention/test_attention_ts_mla_decode.py
tests/trace/test_fi_trace_template_consistency.py`
Current-head local results:
- Focused CPU contract checks in `test_attention_ts_mla_decode.py`: **9
passed**.
- Trace consistency: **1,240 passed, 2 skipped**.
- CP GPU regression collection: **7 cases**, not executed again during
publication.
- Earlier matching-implementation GPU validation: seven CP kernel cases
plus strict two-GPU Helix/MTP replay on Rubin; retained TRT-LLM
focused/native/backend checks and paired generation measurements on
Rubin, B200 and B300. These historical results are not represented as a
fresh full-suite CI run.
- No FlashInfer CI bot command was issued; CI is left to the maintainer.
## 🔬 Experimental Track
<!-- Only for PRs submitted under the experimental policy
(CONTRIBUTING.md → "Experimental APIs and Backends").
Leave this section untouched for normal PRs. -->
- [x] This PR updates existing **experimental** PrimTS APIs marked with
`@flashinfer_experimental_api`. Tracking context: #5489, #4829, #5059,
#5082; a separate graduation-tracking issue is not claimed.
- [ ] The tracking issue names an owner, the reason for the experimental
path, and a graduation plan with a target release.
- [ ] Core changes are limited to a thin entry point (signature, shared
validation, feature-gate check, backend selection, handoff).
- [ ] Tests live in `tests/experimental/` and were validated on the
intended hardware; a runnable example is included.
- [x] Nothing is registered in `flashinfer/aot.py`, and no experimental
backend is reachable from `backend="auto"` without
`FLASHINFER_ALLOW_EXPERIMENTAL_AUTO_BACKENDS=1`. (Calling an
`@flashinfer_experimental_api` or naming a backend explicitly is itself
the opt-in and needs no environment variable.)
- [x] **Test scope declared below.** The experimental CI lane runs
exactly these targets, so keep them as narrow as the change allows.
<!-- Required for experimental PRs. Replace the commented lines below
with your targets.
Do not delete the fence or change its `experimental-tests` tag — the
experimental-track
watcher reads it verbatim to decide which targets to ask CI for. -->
```experimental-tests
tests/experimental/test_experimental_api.py
```
As in #5489, this scope covers shared experimental opt-in behavior, not
kernel correctness. Existing PrimTS implementation/tests remain under
`flashinfer/attention/prims_ts/` and `tests/attention/`; the targeted
attention tests above cover the kernels. Relocation/graduation is
outside this follow-up.
## Reviewer Notes
Please focus on global-to-local prefix arithmetic, packed-query
alignment, split-reduction empty rows and graph-live metadata. This was
developed and validated with Codex assistance. No NVIDIA-specific
notices were added to FlashInfer files.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **New Features**
* Added context-parallel support for paged MLA decode, with configurable
rank and interleave settings. Global KV lengths are mapped to rank-local
cache prefixes.
* Empty rows on ranks with no locally visible KV tokens now return zero
output and negative-infinity LSE by default; this behavior can be
disabled.
* **Bug Fixes**
* Improved handling of supported cluster sizes when selecting attention
launch configurations.
* **Documentation**
* Updated MLA decode guidance and trace examples for context-parallel
settings, local cache capacity, and empty-row behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
Description
Extend
PrimsTSFmhawith MLA prefill, linear MTP and Helix integration, and automatic MLA-generation selection on SM100/SM103/SM107. Kimi K3 full-attention layers can select PrimTS for context and decode with BF16 or full FP8 target KV cache.Dense PrimTS is included in the default registry before
CuteDslMlaFmha, with_is_perf_favorable()enabling generation-only MLA on SM100, SM103 and SM107. Context, mixed batches and standard attention retain their existing default restrictions. ExplicitTLLM_FMHA_LIBS=prims_tsor+prims_tsrequests bypass only the performance heuristic, never availability or request-correctness checks; explicit removal remains supported. Selection is fixed at attention-instance construction. KDA execution is unchanged.Current head:
1a2421546659a5c7cffa4592e2860fac7423d2d1. This publication adds three separately reviewable follow-up commits onfb0aee5fbb6c1d0622c9fa785402cef9d9988615; it does not rebase the consumer onto a new main snapshot.Previous rebase checkpoint:
61e2b9662c6786f7da32019678675b08e3acc55d, rebased onto TRT-LLM main8070e5d78aab872abf71297ae06b1cc6af2d7151. All 35 feature commits and all 58 changed-file entries are retained. The range audit has 34 identical patches and one intentional B300 test-list merge: retain both upstream CSA2 isolation and the PR's dedicated PrimTS backend test entry. There were no code-file conflicts or rebase-fallout commits.is_lseflag covers each accumulation; a format change between partitions raises an error.cp_world/cp_interleave_size, runtimecp_rank, and graph-live globalseq_lens_kv. Cache tables/capacity remain rank-local. Compatibility metadata needed by other providers is retained.latent_cache=None. Aggregated Helix remains development-only behindTRTLLM_HELIX_AGG_EXPERIMENTAL=1, not production-qualified support.functools.cache, while retaining workspace-allocation and capacity-bound plan invalidation.Review order
fb0aee5fbb.912d34b39d. The temporary FlashInfer source branch starts at canonical2ffd3f8b2c, carries merged [TRTLLM-1316] refactor: Remove unnecessary pipeline parallelism logic from postProcessRequest #5489 as one source commit, then adds the CP follow-up from [#9602][feat] AutoDeploy: Support TRTLLM Sampler #9641. The canonical QToken update is preserved; it is not part of the new MLA API change.16cadafcae. Oneis_lseflag, explicit format-change rejection, preserved legacy default, and consolidation of the existing merge tests.1a24215466. Extend the generation-only MLA performance policy to SM100/SM103/SM107 and update the existing CPU policy matrix (10→11 cases).Older review and qualification records below remain historical. No temporary benchmark/build/diagnostic artifacts are included in these commits.
Performance results and default-selection rationale
The tables below all compare the native provider → new PrimTS. They do not compare two PrimTS revisions. Times are microseconds; negative change means lower latency / faster, positive change means higher latency / slower.
The native configuration excludes dense PrimTS and retains the CuTe DSL + TRTLLM-Gen routing policy. In these cases, the actual generation provider was
CuteDslMlaFmha; context used the admitted TRTLLM-Gen fallback. These are not separate measurements of TRTLLM-Gen generation.MLA generation backend — the default-dispatch criterion
Backend-forward timing includes metadata production, preprocessing, cache handling and attention. This is the relevant boundary for choosing the FMHA provider.
Cells show native → new PrimTS µs (PrimTS latency change versus native).
Why enable PrimTS MLA generation on SM100/SM103/SM107
Accordingly,
_is_perf_favorable()enables generation-only MLA on SM100, SM103 and SM107, and the registry places PrimTS before CuTe DSL MLA. This is a dispatch heuristic informed by the measured backend costs, not a claim that every supported shape or dtype is faster.All availability and request-correctness guards remain in force. Explicit user selection/removal is preserved; explicit PrimTS selection bypasses only the performance heuristic. Context, mixed batches and standard attention are not newly enabled by default.
Raw MLA generation — diagnostic, not the dispatch criterion
These intervals exclude backend metadata/preprocessing/cache work. Cells again show native → new PrimTS µs (change versus native).
The +11.11% B200 H-D1 result is a raw-generation slowdown, not a backend slowdown. Its backend result is the +0.13% tie in the first table.
Context and cached-context merge — remain outside default enablement
These are backend-scope timings, native → new PrimTS µs (change versus native). Blank cells mean not measured.
PrimTS context is slower in these Rubin cases, so the generation enablement must not be generalized to default context selection.
Fixed K3 cases and evidence limits
Each displayed native control is one native process versus the initial-AB PrimTS process, with 50 warmups and 100 CUDA-event samples per process. They are descriptive native-vs-PrimTS comparisons, not balanced repeated native A/B experiments. Generation uses graph replay; context uses eager execution. A 2×L2 flush is outside each event interval; clocks are not locked. Compilation, startup, references and instrumentation are outside timing. Numerical checks, full pre/post checks and terminal cleanup passed.
The four FP8/MTP generation shapes do not establish optimality for every MLA model, dtype, query width or batch size. These are rank-local component results, not full CP communication, full-model accuracy, throughput or TTFT measurements.
Native builds match the measured functional snapshot. The later default-policy-only edit passed 11 source-isolated policy/delegation checks; normal local TRT pytest collection remains blocked by an existing libth_common.so/PyTorch ABI mismatch. Publication also carries the already-canonical QToken update; MLA kernel/context sources remain byte-identical to the validated functional implementation. No fresh whole-published-head GPU or CI result is claimed. B300 metadata-query timeouts changed from 30 to 120 seconds outside the clocks; inputs, references, tolerances and timer code were unchanged. Compatibility bounds for other providers are still prepared.
Separate regression check: previous PrimTS versus new PrimTS — not the native baseline
The same completed studies also performed balanced AB/BA comparisons between the earlier PrimTS snapshot and the CP/global + single-
is_lsefunctional snapshot. Those comparisons answer a different question from the native tables above:These source-regression checks completed 90 Rubin, 72 B200 and 64 B300 point processes, including rule-required confirmations. B300 H-D1 did not trigger additional confirmation; no coverage was dropped. The practical 2% threshold is not a statistical-significance claim.
Vendored source
d76f94bb6ae3bc1730689a1a080a79913dcc3606.trtllm-prims-ts-dev-20261107at31fe2dbd6b65cca758c1318f177b315be0c33778(promoted after TRT-LLM [https://nvbugs/6959297][fix] Restore PrimTS FMHA and MLA task API compatibility #21151). The temporary branch adds three commits:9cc9333bf(merged [TRTLLM-1316] refactor: Remove unnecessary pipeline parallelism logic from postProcessRequest #5489 aggregate),5129a714e(CP follow-up), andd76f94bb6(device cluster-size guard).1a9f361db34977801e4a676eb54b0379527479b9. The CP follow-up is FlashInfer #9641.ContextTaskto retain the task API compatibility from [https://nvbugs/6959297][fix] Restore PrimTS FMHA and MLA task API compatibility #21151.cp_rank=0. Offline and local-source vendor checks pass. The canonical branch was not rewritten.Test Coverage
Follow-up publication — 2026-11-18
Helix caller opt-out and CI diagnosis — 2026-11-16
25b5c49f81b0e060e94544f91cc8a8728638eba4plans dense MLA withzero_empty_rows=Falseat runtime and during warmup. Helix already neutralizes empty O/statistics through NCCL reformatting, FIFO-v1 sanitization or the FIFO-v2 sender before combination. Per-token MTP bounds and Q1 local-length masking remain intact. This removes one redundant launch from bounded MLA; no measured speedup is claimed. Non-Helix decode and context are unaffected.True; source heads and TRT's dev pin946b484d2cd7918dafbbd28493b087c690e067f1are unchanged. No FlashInfer source update was required by the CI diagnosis.test_mamba_cache_manager.py; advisory Pyrefly is unavailable. No new GPU allocation, e2e or performance experiment was run. TRT-LLM CI is requested separately with/bot run --disable-fail-fast; no CI pass is claimed here.FlashInfer CPU CI job 551865695 in pipeline82543944 completed all tests: 10,971 passed, 20 skipped, 0 failed, with
suite_complete=true. The general archive upload succeeded; the GitLab runner helper was then Killed while uploading the gzip JUnit report, and the job became failed. This explains why the bot could not extract a test failure.This is a report-upload failure, not a PrimTS test failure. The trace does not establish the reason for the kill; a memory-limit diagnosis needs runner evidence. Per maintainer direction, the FlashInfer retry is left to CI ownership. No source patch, waiver or CI retry was made for this event.
Perkz review follow-up — 2026-11-16
zero_empty_rows=Trueto dense MLA planning and the one-shot API. Disabling it omits the neutral initialization launch, not prefix masking; empty-row contents are unspecified. The effective flag selects a compiled launch variant but does not change workspace sizing. TRT-LLM runtime and startup planning now disable the extra initialization launch and rely on the existing Helix zero-row sanitization before combination.772e460928025d9f2c3a8717d5c3f5889aaabe57and pinned dev946b484d2cd7918dafbbd28493b087c690e067f1carry the same two follow-up patches. TRT-LLM head053e80782bc230eef4d9df24cdf665a7344fb66eimports them. Its compatibility patch was regenerated only to re-anchor context; every added/removed patch line remains identical. Canonical dev is unchanged.BatchAttentionon local SM86: cooperative-grid launch limit, then inherited CUDA error. No unrelated fix, skip or tolerance change was added. TRT-LLM all-file lint also retains the pre-existing main F811 duplicate import intest_mamba_cache_manager.py; changed-file checks pass. Advisory Pyrefly is unavailable.Latest review fix and main rebase — 2026-11-16
+in explicit PrimTS selection consistently with the registry. Reuse the existing positive-delta regression case; it failed before the fix and passes afterward._run_preparedspy/call-count assertion per maintainer direction; [None][perf] Precompile PrimTS sparse MLA in decoder warmup and reuse decode views #22274 owns removal of that private implementation path. Public validated/unvalidated output parity and reload checks remain.KVCacheLayerSpecimport in unchangedtest_mamba_cache_manager.py:89(also imported at 33); other required hooks passed. Advisory Pyrefly is unavailable locally. No unrelated lint fix was added.Earlier review follow-up — 2026-11-16 (before this rebase)
3874b546b646d2283d44cc463f549ef7cf0ac4a1; pinned FlashInfer dev source:95546a07288ce7beaf3f88f7cffda090eed68313onprims-ts-mla-prefill--flashinfer. The canonical branch and compatibility patch are unchanged.61e2b9662c67CI wheel with the current Python sources; all 95 native ELF files remain byte-identical.The earlier
61e2b9662c67head passed CI pipeline 70292. That is not a final-head CI result for these follow-up commits. Older validation snapshots below remain historical.Earlier update validation
confcutdirto the worktree resolved the runner issue without changing tests or production code.CI was re-requested for
61e2b9662cusing exactly/bot run --disable-fail-fast: invocation. This is a request, not a passing CI result.Earlier qualification — prior snapshots
CI failure #69865 had a confirmed warmup-order expectation error on both CPU architectures. The B300 aggregate hit its 90-minute budget after 5,808 inner records with zero inner assertion failures; the backend file accounted for 895 seconds of recorded test time, including substantial coverage added by this PR. It now has its own 45-minute entry and is excluded from the unchanged 90-minute aggregate. No test, parameter, reference or tolerance is removed by this scheduling change.
The H100 stage also had checkpoint filesystem errors (Err108/Err5) and a worker-readiness timeout; the latter has no worker-level traceback and is not claimed fixed. No unrelated inference change or waiver is added.
The earlier CI requests and helper/pipeline results in this subsection predate the current head; they are retained as historical evidence, not the status of the new invocation above.
Retained coverage and consolidation
TIMEOUT (90)on the DEP16 entry means 90 minutes, not seconds.Remaining qualification
92adb50c83) and FlashInfer performance branch (9dacdaa1b). These are historical snapshots, not current-head A/B controls differing only by that optimization.Historical validation and performance — older snapshots, not current-head qualification
Earlier B200 host-sizing measurements compared trusted uncached MLA preparation (5.46–5.63 us) with the former per-layer cache (2.40–2.55 us). Standard decode baseline/trusted preparation was 23.50–24.50 / 17.85–18.21 us. These measurements do not benchmark the current shared
functools.cachewrappers.An earlier warmup comparison with already-loaded kernels measured 33.170→9.133 ms median across seven fresh warmer objects, with sample SD 0.122/0.036 ms. Fresh-process warm-disk single samples were 1.984/2.000 s and do not establish a startup speedup.
At the earlier
f118184d6source checkpoint, 22 B200 kernel cases and seven TRT native anchors passed. Those results, and the older adapter counts below, are historical—not matching-native validation ofcad287ef61.Review cleanup —
a30df992e1FmhaManager. The private helper lives inprims_ts.py; the fixed-query MLA support check raises on ragged multi-token input rather than falling back. No forward-time scan is added. This is a cache-miss sanity check; callers must preserve fixed-width Q on cache hits.manager.py, sharedutils.py, andtest_fmha_manager.pynow match the PR base. Removed the requested header from the node2/gpu8 YAML and a stale test asserting the deleted BF16 V-copy reservation.Earlier main-rebase validation (
6df6036df4)06609107acto6df6036df4on maina1c22fc7a5. All 58 changed files remain. The package-version-removal commit is already in main, so 21 commits become 20 without losing functionality or performance work.python -Pgave a complete 567-pass rerun with exit 0, with the guard still enabled. No product-code or test change was needed; reruns are not additional unique coverage.Previous source-refresh validation
Fresh B200 native build completed with CUDA13.4, Torch2.14.0a0, CUTLASS DSL4.8.0 and FlashInfer0.7.2. The local out-of-tree attribution scanner warned about its working directory; the wheel is a validation artifact, not a release build.
The matching-wheel 522-case selection first reported 514 passed / 8 failed / 0 skipped. All failures were the prepared sparse MLA ABI adaptation and its old-schema fixtures. After appending the two optional slots and updating the fixtures, all 8 affected checks passed, including graph replay, one/two-pool paths and persistent-kernel reload. This is a full run plus a targeted post-fix rerun, not a second full 522-case execution.
All selected K3 full-attention component cases (BF16/FP8 KV, cached/chunked context, TP-local/DP head geometry), FP8 linear MTP, expanded geometries, FP8 scales, and dense graph replay passed. This is component qualification, not a new full-model or multi-GPU K3 run.
Native library/module origins and changed-source hashes were verified against the installed wheel. Final committed files exactly match the post-fix qualification manifest. Full pre-commit checks and remote Pyrefly passed; vendor integrity checks passed against local and published source.
FlashInfer's full selected source run reported 525 passed / 8 hardware-policy skips / 2 failed. The rebase-induced device-scale reducer compiler error was fixed; 22 targeted rerun cases passed. The remaining BF16 Helix boundary tolerance failure reproduces identically on both old source branches (
a4b6e9028and64273c074): 2/884736 elements, greatest absolute error0.0006412482 versus atol0.0005. Its test, reference and tolerances remain unchanged. The 22 cases overlap the full run and are not additional unique coverage.Before/after patch-range and changed-file audits preserve the previous integration and performance work. The old-dev-only sink helper is superseded by the refreshed implementation and covered by strict sink-scale regressions.
Test consolidation preserves distinct geometry/dtype/layout/topology contracts and the required K3 CI case. TRT native layout cases: 48→20; cache integration: 36→18; independent capacity boundaries: 144→18. FlashInfer affected-file collection: 2391→2373, including new device-scale/compact-split regressions. No proportional CI speedup is claimed.
No new end-to-end or performance experiment was run for this rebase. FlashInfer CI remains user-managed; no bot command is issued there.
Multi-GPU CI approval must follow the normal maintainer policy.
Historical performance and accuracy — unchanged, not refreshed-head qualification
Main-based prepared-context comparison on Rubin
After the correctness gates passed, 36 fresh-process receipts completed three ABBA blocks per case. Baseline
e2203f939e6c95475d7ec609fadfb916c04a2becand candidate005d91701b95e88a3495da84e6d384efccc606d7contain the same original-scale correctness change. Their production delta is only full-tile dense LDRED plus request-local dense KV domains. Both use actual runtime Q/K maxima.All cases use packed dense FP8 E4M3 attention, H96, Dqk192/Dv128, and BF16 output. The one-request cases use Q/K lengths 1920/1920 and Q/K capacities 2048/2048. The two-request case uses Q lengths
[1664,256], K lengths[0,3840], and capacities Q2048/K4096. All eager, graph, and post-timing numerical checks passed; input identities were unchanged. An independent audit recomputed all 360 raw CUDA-event samples, receipt medians, and nine block summaries. The means of paired block ratios are reported, with sample SD across three blocks.This is one-allocation prepared-wrapper graph replay, not a CuTeDSL/trtllm-gen comparison or a new K3 end-to-end result. The large empty-partition gain does not establish how frequently that geometry occurs in serving. Statistics ON/OFF also changes precision policy, so their absolute-time difference is not an isolated statistics-store cost.
Functionality-only Kimi K3 qualification on Rubin
26e9b3b33cplus FlashInferd9f0146; published source-equivalent functionality commitdf3434b1d), not every current-main combination.Controlled DEP16 performance experiment on Rubin
The performance experiment used FP8 target KV/P128, TP16/EP16 with attention DP, DSpark5 (six target query tokens per verification step), chunked prefill, overlap scheduling, and CUDA graphs. Both policies used the same BF16 KDA state and engine prefill path; drafter KV remained BF16/P32. The fixed workload was 4,096 input / 256 output tokens at concurrency 128 and 256, seed 42, temperature 0, and ignore-EOS. This is an aggregated-serving comparison, not the original disaggregated AgentX workload.
Three allocation-bounded ABBA blocks produced 24 accepted measured windows. All measurements were retained; sealed-source/artifact and compilation-log audits passed. Observed native compilation warnings were confined to startup; no matching compilation markers occurred in accepted measured windows. This does not prove zero unlogged JIT.
This supports near throughput parity in this workload, not a robust speedup, statistical equivalence, or noninferiority claim. Policy TPS columns are arithmetic means; percentage changes are means of paired block ratios. The experiment used the pre-rebase TRT
be927b7015runtime and the source-equivalent FlashInfer30e7ea65/76e822974kernels. It is not a latest-main native performance result or a causal before/after comparison across different allocation series.Accuracy and scope limits
PR Checklist
GitHub Bot Help
To see available CI commands, comment
/bot help.Dev Engineer Review
prims-ts-mla-prefill--flashinferat commitf118184d6d88332a83d46ea198717e2c5b76ed6a. Later vendor validation reported the vendor metadata and source attribution as valid for the checked revision. That validation did not approve the code. Promotion still requires the source PR to merge. Vendor comments also report a blocked canonical-branch promotion and a required branch migration. Confirm the pinned source and its digests before merge.mlaKernels.cuaccepts QK-nope/V pairs(64,128),(128,128), and(192,256)for non-absorption FP8 quantization, with RoPE dimension 64.mlaPreprocessOp.cppaccepts(128,128),(192,128), and(256,256). The lists differ. Confirm which pairs each path accepts and that the paths agree on shared geometry.mlaChunkedPrefill.cuaccepts head size 256 only for non-floattypes.floatrejects other sizes. A compile-time assertion limits in-place softmax-stat writes to 32 vectors per head.phased.py: MLA context attention now receives full K and V tensors, not a slice limited to the context query span. Non-MLA paths keep the existing slice logic.prims_ts.py:get_paged_kv_policy_unsupported_reasongainsallow_linear_mla, andget_attention_feature_unsupported_reasongainsallow_helix. Both are keyword-only with defaults, so existing callers remain compatible.SUPPORTED_MLA_SMSnow includes 107.validate=Falsecalls that meet the stated conditions. Review stale-pointer and shape-change risk in this cache.PrimsTSDenseMlaWarmupallocates per-device decode workspaces sized to the largest pending profile. It raises when planning runs during CUDA graph capture. Check peak memory at largemax_batch_sizesettings.precompile_onlyandprobe_cachedmodes. These raiseKernelBusyandKernelNotCached. The patch also moves several attention APIs to the standardflashinfer_apidecorator without trace dispatchers. Verify that tracing consumers do not depend on the removed dispatchers.TRTLLM_HELIX_AGG_EXPERIMENTAL=1.#70068reported failure. The current-head full-model Helix test is not reported as executed. Current-head DEP16/Helix8 full-checkpoint accuracy and the paired Rubin performance comparison are pending. An earlier BF16 Helix page-boundary tolerance failure is not resolved.QA Engineer Review
Test files changed:
tests/unittest/others/test_kimi3_ci_selection.py,tests/unittest/_torch/attention/test_mla_context_merge.py,tests/unittest/_torch/attention/test_prims_ts_mla_capacity.py,tests/unittest/_torch/attention/test_prims_ts_mla_context_geometry.py,tests/unittest/_torch/attention/test_prims_ts_mla_warmup.py,tests/unittest/_torch/attention/hw_agnostic/test_prims_ts_sparse_mla_prepared.py.tests/integration/defs/accuracy/test_kimi3.py,tests/integration/defs/accuracy/references/gsm8k.yaml,tests/test_common/grouped_test_utils.py,tests/unittest/_torch/attention/multi_gpu/test_mla_helix_context.py,tests/unittest/_torch/attention/sparse/deepseek_v4/test_prims_ts_sparse_mla_fmha.py,tests/unittest/_torch/attention/test_cute_dsl_mla.py,tests/unittest/_torch/attention/test_prims_ts_attention_backend.py,tests/unittest/_torch/attention/test_prims_ts_fmha.py,tests/unittest/_torch/compilation/test_phase_selective_forward.py.Test-list changes:
l0_cpu.yml: addsunittest/others/test_kimi3_ci_selection.py.l0_gb300_multi_nodes_node2_gpu8.yml: addsTestKimiK3DSpark.test_prims_ts_fp8_tep8.l0_gb300_multi_nodes_node4_gpu16.yml: addsTestKimiK3.test_prims_tswith a 90-second timeout.l0_b300.yml: removestest_prims_ts_attention_backend.pyfrom the B300 attention aggregate. That file is listed separately with a 45-minute timeout. The aggregate keeps its 90-minute timeout.Covered behavior:
(128,128),(192,128), and(256,256)with BF16 and FP8 KV.prims_tsselection on workers, with environment restore.Gaps:
ci: full pre-merge approvedlabel.Coverage verdict: needs follow-up.
Per-File QA Perspective
3rdparty/vendor_patches/flashinfer-prims-ts.patch: Verify precompile and probe modes, theKernelBusyandKernelNotCachederrors, and the 256-entry prepared cache. Confirm the 1024-entry sparse plan limit.3rdparty/vendor_sources.lock.yaml: Confirm the pinned commit and digests match the vendored snapshot.cpp/tensorrt_llm/kernels/mlaChunkedPrefill.cu: Verify merge results for head sizes 128 and 256 and rejection of unsupported sizes forfloat. Covered bytest_mla_context_merge.py.cpp/tensorrt_llm/kernels/mlaKernels.cu: Verify FP8 quantize with zero and nonzerototal_kv_len, and rejection of unlisted QK/V pairs. Confirm the pair list matches the preprocess op.cpp/tensorrt_llm/thop/mlaPreprocessOp.cpp: NewmlaContextPreprocessop. Verify rejection of INT8 and FP4 cache modes, alignment checks, and aliasing rules for Q/K outputs. Covered by the prims_ts attention backend tests.tensorrt_llm/_torch/attention/backends/fmha/phased.py: Verify MLA context receives full K and V tensors. Non-MLA context still receives the sliced span.tensorrt_llm/_torch/attention/backends/fmha/prims_ts.py: Verify MLA context and generation dispatch, linear MTP acceptance, SM 107 support, and the new keyword-only flags. Check that standard generation remains limited to one token.tensorrt_llm/_torch/attention/backends/fmha/prims_ts_mla_warmup.py: Verify warmup runs only for eligible profiles. Verify that CUDA graph capture raises. Check decode workspace memory at high batch sizes.tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/warmup.py: Verify that the dense MLA warmup runs after CUDA graph capture with the configured limits.tests/integration/defs/accuracy/references/gsm8k.yaml: Adds FP8 KV references (96.5) for Kimi-K3. Verify the reference matches the observed run.tests/integration/defs/accuracy/test_kimi3.py: New 16-GPUtest_prims_tsand FP8 or Helix runner paths. Confirm the listed entries run and report results.tests/integration/test_lists/test-db/l0_cpu.yml: Adds the Kimi K3 CI selection unit test to the CPU list.tests/integration/test_lists/test-db/l0_b300.yml: Moves the PrimTS attention backend test out of the B300 aggregate. Confirm the separate 45-minute entry runs.tests/integration/test_lists/test-db/l0_gb300_multi_nodes_node2_gpu8.yml: Adds the FP8 TEP8 PrimTS test to the GB300 GPU8 list.tests/integration/test_lists/test-db/l0_gb300_multi_nodes_node4_gpu16.yml: Adds the 16-GPU PrimTS test with a 90-second timeout. Confirm the timeout is sufficient.tests/test_common/grouped_test_utils.py: Test-only helpers for worker environment snapshot and restore. Confirm that restore runs on failure paths.tests/unittest/_torch/attention/hw_agnostic/test_prims_ts_sparse_mla_prepared.py: CPU-only checks of prepared versus full launch arguments. Confirm the test runs in the hardware-agnostic list.tests/unittest/_torch/attention/multi_gpu/test_mla_helix_context.py: Covers speculative query widths, Helix owner-only KV, and FP8 two-GPU runs. Multi-GPU execution depends on the CI label.tests/unittest/_torch/attention/sparse/deepseek_v4/test_prims_ts_sparse_mla_fmha.py: Verifies exact equality between full andvalidate=Falselaunches.tests/unittest/_torch/attention/test_cute_dsl_mla.py: Expects SM 107 128-token pages to be non-exclusive to CuTe DSL. Verify the expectation matches the runtime.tests/unittest/_torch/attention/test_mla_context_merge.py: CUDA-gated merge tests. Verify the unsupported-width error text matches the implementation.tests/unittest/_torch/attention/test_prims_ts_attention_backend.py: Covers MLA preprocessing, FP8 quantize, and Kimi K3 cached and chunked attention. Verify the removed skip markers do not expose unsupported SMs.tests/unittest/_torch/attention/test_prims_ts_fmha.py: Broad support-check and wrapper coverage. Verify the SM 107 CUTLASS 13.3 expectation in CI.tests/unittest/_torch/attention/test_prims_ts_mla_capacity.py: CPU-only AST-loaded checks of MLA capacity floors. Verify the loader matches the currentprims_ts.pylayout.tests/unittest/_torch/attention/test_prims_ts_mla_context_geometry.py: SM 100, 103, and 107 geometry test against an FP64 reference. Verify NaN handling for empty historical partitions.tests/unittest/_torch/attention/test_prims_ts_mla_warmup.py: CPU-only warmup planner tests. Verify retry and deduplication behavior.tests/unittest/_torch/compilation/test_phase_selective_forward.py: Verifies the dense MLA warmup call with compiled forward enabled.tests/unittest/others/test_kimi3_ci_selection.py: CPU-only check of strictprims_tsworker selection and environment restore. Verify the restore after constructor and evaluation failures.