Skip to content

[None][feat] Add PrimTS MLA prefill, MTP/Helix integration, and Rubin support - #19395

Merged
yuxianq merged 49 commits into
NVIDIA:mainfrom
yuxianq:feat/prims-ts-mla-prefill
Nov 21, 2026
Merged

yuxianq merged 49 commits into
NVIDIA:mainfrom
yuxianq:feat/prims-ts-mla-prefill

Conversation

@yuxianq

@yuxianq yuxianq commented Sep 18, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Extend PrimsTSFmha with MLA prefill, linear MTP and Helix integration, and automatic MLA-generation selection on SM100/SM103/SM107. Kimi K3 full-attention layers can select PrimTS for context and decode with BF16 or full FP8 target KV cache.

Dense PrimTS is included in the default registry before CuteDslMlaFmha, with _is_perf_favorable() enabling generation-only MLA on SM100, SM103 and SM107. Context, mixed batches and standard attention retain their existing default restrictions. Explicit TLLM_FMHA_LIBS=prims_ts or +prims_ts requests bypass only the performance heuristic, never availability or request-correctness checks; explicit removal remains supported. Selection is fixed at attention-instance construction. KDA execution is unchanged.

Current head: 1a2421546659a5c7cffa4592e2860fac7423d2d1. This publication adds three separately reviewable follow-up commits on fb0aee5fbb6c1d0622c9fa785402cef9d9988615; it does not rebase the consumer onto a new main snapshot.

Previous rebase checkpoint: 61e2b9662c6786f7da32019678675b08e3acc55d, rebased onto TRT-LLM main 8070e5d78aab872abf71297ae06b1cc6af2d7151. All 35 feature commits and all 58 changed-file entries are retained. The range audit has 34 identical patches and one intentional B300 test-list merge: retain both upstream CSA2 isolation and the PR's dedicated PrimTS backend test entry. There were no code-file conflicts or rebase-fallout commits.

  • Reuse native MLA RoPE/cache updates and FP8 preparation for fresh, cached-prefix and chunked context.
  • Support full QK/V dimensions (128,128), (192,128), (256,256) with the existing latent512/RoPE64 integration. Native quantization's non-RoPE widths 64/128/192 become full QK widths 128/192/256 after adding RoPE64; the geometry contracts agree.
  • Pass aligned strided FP16/BF16 V directly, removing its packing copy and PrimTS-specific reservation. FP8 conversion buffers remain.
  • Use compact base-2 LSE for PrimTS partition/Helix merging while preserving native providers' legacy max/sum path. A single is_lse flag covers each accumulation; a format change between partitions raises an error.
  • Replace PrimTS's per-query bounds input with compile-time cp_world/cp_interleave_size, runtime cp_rank, and graph-live global seq_lens_kv. Cache tables/capacity remain rank-local. Compatibility metadata needed by other providers is retained.
  • Admit owner-prepared Helix context only after shared MLA has applied RoPE/cache ownership, with local-slot metadata and latent_cache=None. Aggregated Helix remains development-only behind TRTLLM_HELIX_AGG_EXPERIMENTAL=1, not production-qualified support.
  • Warm selected MLA profiles through public PrimTS planners using shared temporary per-device workspace. Cache standard/MLA decode workspace-size queries with TRT-LLM-local functools.cache, while retaining workspace-allocation and capacity-bound plan invalidation.

Review order

  1. Existing MLA prefill/LSE integration through fb0aee5fbb.
  2. CP/global metadata integration and vendor pin: 912d34b39d. The temporary FlashInfer source branch starts at canonical 2ffd3f8b2c, carries merged [TRTLLM-1316] refactor: Remove unnecessary pipeline parallelism logic from postProcessRequest #5489 as one source commit, then adds the CP follow-up from [#9602][feat] AutoDeploy: Support TRTLLM Sampler #9641. The canonical QToken update is preserved; it is not part of the new MLA API change.
  3. Single accumulation format: 16cadafcae. One is_lse flag, explicit format-change rejection, preserved legacy default, and consolidation of the existing merge tests.
  4. Blackwell default dispatch: 1a24215466. Extend the generation-only MLA performance policy to SM100/SM103/SM107 and update the existing CPU policy matrix (10→11 cases).

Older review and qualification records below remain historical. No temporary benchmark/build/diagnostic artifacts are included in these commits.

Performance results and default-selection rationale

The tables below all compare the native provider → new PrimTS. They do not compare two PrimTS revisions. Times are microseconds; negative change means lower latency / faster, positive change means higher latency / slower.

The native configuration excludes dense PrimTS and retains the CuTe DSL + TRTLLM-Gen routing policy. In these cases, the actual generation provider was CuteDslMlaFmha; context used the admitted TRTLLM-Gen fallback. These are not separate measurements of TRTLLM-Gen generation.

MLA generation backend — the default-dispatch criterion

Backend-forward timing includes metadata production, preprocessing, cache handling and attention. This is the relevant boundary for choosing the FMHA provider.

Cells show native → new PrimTS µs (PrimTS latency change versus native).

Case Rubin / SM107 B200 / SM100 B300 / SM103
H-D1 21.568 → 17.536 (−18.69%) 24.576 → 24.608 (+0.13%, essentially tied) 24.544 → 22.560 (−8.08%)
H-D8 49.008 → 23.840 (−51.36%) 49.152 → 24.448 (−50.26%) 47.104 → 22.496 (−52.24%)
D-D8 45.408 → 30.512 (−32.80%) 51.184 → 40.928 (−20.04%) 49.120 → 38.912 (−20.78%)
D-D16 48.096 → 36.896 (−23.29%) 81.920 → 73.696 (−10.04%) 79.856 → 73.760 (−7.63%)

Why enable PrimTS MLA generation on SM100/SM103/SM107

  • SM107 / Rubin: PrimTS lowers backend latency in all four measured generation cases, by 18.69%–51.36%.
  • SM100 / B200: H-D1 is effectively tied: the difference is only +0.032 µs / +0.13%. The other three backend cases improve by 10.04%–50.26%.
  • SM103 / B300: PrimTS lowers backend latency in all four cases, by 7.63%–52.24%.
  • Use backend latency, not the isolated raw call, for dispatch. B200 H-D1's raw PrimTS call is slower, but that difference is not exposed as a meaningful backend-forward regression. The raw timings below are diagnostic; subtraction of separately measured scopes does not identify which preparation operation accounts for the offset.

Accordingly, _is_perf_favorable() enables generation-only MLA on SM100, SM103 and SM107, and the registry places PrimTS before CuTe DSL MLA. This is a dispatch heuristic informed by the measured backend costs, not a claim that every supported shape or dtype is faster.

All availability and request-correctness guards remain in force. Explicit user selection/removal is preserved; explicit PrimTS selection bypasses only the performance heuristic. Context, mixed batches and standard attention are not newly enabled by default.

Raw MLA generation — diagnostic, not the dispatch criterion

These intervals exclude backend metadata/preprocessing/cache work. Cells again show native → new PrimTS µs (change versus native).

Case Rubin / SM107 B200 / SM100 B300 / SM103
H-D1 14.992 → 13.632 (−9.07%) 18.432 → 20.480 (+11.11%, raw call only) 18.400 → 18.432 (+0.17%, essentially tied)
H-D8 40.928 → 17.920 (−56.22%) 40.960 → 16.384 (−60.00%) 38.912 → 16.352 (−57.98%)
D-D8 39.632 → 26.464 (−33.23%) 47.072 → 36.864 (−21.69%) 45.024 → 34.816 (−22.67%)
D-D16 40.864 → 30.688 (−24.90%) 73.696 → 65.536 (−11.07%) 71.680 → 65.696 (−8.35%)

The +11.11% B200 H-D1 result is a raw-generation slowdown, not a backend slowdown. Its backend result is the +0.13% tie in the first table.

Context and cached-context merge — remain outside default enablement

These are backend-scope timings, native → new PrimTS µs (change versus native). Blank cells mean not measured.

Case Rubin / SM107 B200 / SM100 B300 / SM103
H-C 57.632 → 64.384 (+11.72%)
D-Cmerge 308.944 → 353.264 (+14.35%)

PrimTS context is slower in these Rubin cases, so the generation enablement must not be generalized to default context selection.

Fixed K3 cases and evidence limits

  • Full FP8 E4M3 KV cache, page size 128, 96 generation heads, latent/RoPE dimensions 512/64, BF16 output, seed 1729.
  • H-D1/H-D8: local batch 1/8, Q=8 (MTP7 + target), CP8 rank 0, local history 512; global total 4104 maps to visible local prefixes 513–520.
  • D-D8/D-D16: local batch 8/16, Q=6 (MTP5 + target), CP1, history 4096.
  • H-C: expanded rank-local context, Q4096/H12, no LSE export. D-Cmerge: Q2048/H96, cached 2048 + fresh 2048, with actual partition merge.

Each displayed native control is one native process versus the initial-AB PrimTS process, with 50 warmups and 100 CUDA-event samples per process. They are descriptive native-vs-PrimTS comparisons, not balanced repeated native A/B experiments. Generation uses graph replay; context uses eager execution. A 2×L2 flush is outside each event interval; clocks are not locked. Compilation, startup, references and instrumentation are outside timing. Numerical checks, full pre/post checks and terminal cleanup passed.

The four FP8/MTP generation shapes do not establish optimality for every MLA model, dtype, query width or batch size. These are rank-local component results, not full CP communication, full-model accuracy, throughput or TTFT measurements.

Native builds match the measured functional snapshot. The later default-policy-only edit passed 11 source-isolated policy/delegation checks; normal local TRT pytest collection remains blocked by an existing libth_common.so/PyTorch ABI mismatch. Publication also carries the already-canonical QToken update; MLA kernel/context sources remain byte-identical to the validated functional implementation. No fresh whole-published-head GPU or CI result is claimed. B300 metadata-query timeouts changed from 30 to 120 seconds outside the clocks; inputs, references, tolerances and timer code were unchanged. Compatibility bounds for other providers are still prepared.

Separate regression check: previous PrimTS versus new PrimTS — not the native baseline

The same completed studies also performed balanced AB/BA comparisons between the earlier PrimTS snapshot and the CP/global + single-is_lse functional snapshot. Those comparisons answer a different question from the native tables above:

  • Rubin: 9/10 scopes pass the predefined 2% source-regression cap. H-D8 backend has a valid +2.055% increase (23.360 → 23.840 µs); it is still 51.36% faster than its native control.
  • B200: all 8 scopes pass; changes range from −0.423% to +0.130%.
  • B300: all 8 scopes pass; H-D1 improves 7.723% backend / 9.859% raw versus the older PrimTS implementation. Other changes are at most +0.713%.

These source-regression checks completed 90 Rubin, 72 B200 and 64 B300 point processes, including rule-required confirmations. B300 H-D1 did not trigger additional confirmation; no coverage was dropped. The practical 2% threshold is not a statistical-significance claim.

Vendored source

schema_version: 1
vendors:
  flashinfer-prims-ts:
    upstream_prs:
      - https://github.com/flashinfer-ai/flashinfer/pull/5489
      - https://github.com/flashinfer-ai/flashinfer/pull/9641

Test Coverage

Follow-up publication — 2026-11-18

  • Retained matching-functional-snapshot validation: 18 Rubin, 17 B200, 15 B300 focused checks, with zero failures/errors/skips; includes legacy/single-LSE merging, format-change rejection and context geometries. These are not full-model E2E results.
  • Completed and independently verified component studies above: Rubin90/B20072/B30064 point processes, required confirmations and all full postchecks. Every task-owned GPU allocation is released.
  • Current default-policy CPU matrix:11 source-isolated cases pass (the two newly enabled architectures, unsupported SM90, preserved kernel-support rejection, non-MLA/context/mixed exclusions and explicit selection). The normal local pytest import fails on a pre-existing native/PyTorch ABI mismatch; that is not reported as a pytest pass.
  • Current FlashInfer upstream branch:9 focused CPU checks pass; trace consistency1,240 pass/2 skip;7 CP GPU cases collect. Temporary dev branch:9 focused CPU checks and1,198 trace checks pass/2 skip;3 canonical QToken host checks pass.
  • Changed-file hooks and vendor offline/local-source verification pass. Pyrefly is unavailable locally, so the advisory wrapper's success is not represented as a completed type check.
  • No new full-model runs or manually triggered CI were performed during publication. FlashInfer CI remains maintainer-triggered as requested.

Helix caller opt-out and CI diagnosis — 2026-11-16

  • TRT-LLM 25b5c49f81b0e060e94544f91cc8a8728638eba4 plans dense MLA with zero_empty_rows=False at runtime and during warmup. Helix already neutralizes empty O/statistics through NCCL reformatting, FIFO-v1 sanitization or the FIFO-v2 sender before combination. Per-token MTP bounds and Q1 local-length masking remain intact. This removes one redundant launch from bounded MLA; no measured speedup is claimed. Non-Helix decode and context are unaffected.
  • The standalone FlashInfer default stays True; source heads and TRT's dev pin 946b484d2cd7918dafbbd28493b087c690e067f1 are unchanged. No FlashInfer source update was required by the CI diagnosis.
  • Reused coverage: 37 focused CPU checks passed (adapter/planning/cache/warmup plus existing Helix mask/sanitization checks). Selected current modules match the isolated dependency prefix; its native dependencies remain the older CI70292 wheel, not full new-main native qualification. The existing two-GPU strict MTP7 regression now poisons empty raw O/statistics and compares raw graph parity only on nonempty rows; its final full-output/reference tolerance is unchanged. That updated multi-GPU test has not been rerun locally. Prior B200 opt-out kernel results remain historical.
  • Changed-file hooks pass. The all-file sweep retains the existing main F811 duplicate import in test_mamba_cache_manager.py; advisory Pyrefly is unavailable. No new GPU allocation, e2e or performance experiment was run. TRT-LLM CI is requested separately with /bot run --disable-fail-fast; no CI pass is claimed here.

FlashInfer CPU CI job 551865695 in pipeline82543944 completed all tests: 10,971 passed, 20 skipped, 0 failed, with suite_complete=true. The general archive upload succeeded; the GitLab runner helper was then Killed while uploading the gzip JUnit report, and the job became failed. This explains why the bot could not extract a test failure.

This is a report-upload failure, not a PrimTS test failure. The trace does not establish the reason for the kill; a memory-limit diagnosis needs runner evidence. Per maintainer direction, the FlashInfer retry is left to CI ownership. No source patch, waiver or CI retry was made for this event.

Perkz review follow-up — 2026-11-16

  • Shared causal/dense score-mask and maximum-reduction loop, preserving distinct visibility bounds, tile coordinates and the full-tile fast path.
  • Added zero_empty_rows=True to dense MLA planning and the one-shot API. Disabling it omits the neutral initialization launch, not prefix masking; empty-row contents are unspecified. The effective flag selects a compiled launch variant but does not change workspace sizing. TRT-LLM runtime and startup planning now disable the extra initialization launch and rely on the existing Helix zero-row sanitization before combination.
  • Clarified that the existing base-2 LSE API remains independent of the natural-log maximum/sum export; both can coexist. Reused existing tests, adding opt-out controls to the 1CTA Q1 and 2CTA Q8 graph cases without adding test definitions or a broad matrix.
  • Upstream 772e460928025d9f2c3a8717d5c3f5889aaabe57 and pinned dev 946b484d2cd7918dafbbd28493b087c690e067f1 carry the same two follow-up patches. TRT-LLM head 053e80782bc230eef4d9df24cdf665a7344fb66e imports them. Its compatibility patch was regenerated only to re-anchor context; every added/removed patch line remains identical. Canonical dev is unchanged.
  • Actual B200 / CUDA 13.4 / CuTe DSL 4.8 validation: 11 passed, 0 failed/skipped, including bounds, empty rows, graph opt-out, dense-domain replay, chunk merging and simultaneous stats/LSE. Source hashes checked before/after; the allocation was released.
  • Host-only checks with CUDA hidden: upstream 1,271 passed / 2 skipped, pinned dev 1,248 passed / 2 skipped. All changed-file FlashInfer hooks passed. TRT-LLM validated 30 focused CPU checks (16 initial passes plus 14 passes after correcting an existing profile assertion); changed modules are byte-identical to the tested isolated prefix, but dependencies/native files remain the older CI70292 runtime, not full new-main native qualification.
  • The broader local trace run was not CPU-only: 2,133 passed / 341 skipped / 2 failed. Both failures were in unchanged persistent BatchAttention on local SM86: cooperative-grid launch limit, then inherited CUDA error. No unrelated fix, skip or tolerance change was added. TRT-LLM all-file lint also retains the pre-existing main F811 duplicate import in test_mamba_cache_manager.py; changed-file checks pass. Advisory Pyrefly is unavailable.
  • No new end-to-end/performance experiment or CI command was issued for these follow-ups. Historical performance tables retain their original source/hardware scope.

Latest review fix and main rebase — 2026-11-16

  • Normalize whitespace after + in explicit PrimTS selection consistently with the registry. Reuse the existing positive-delta regression case; it failed before the fix and passes afterward.
  • Remove the sparse reload test's _run_prepared spy/call-count assertion per maintainer direction; [None][perf] Precompile PrimTS sparse MLA in decoder warmup and reuse decode views #22274 owns removal of that private implementation path. Public validated/unvalidated output parity and reload checks remain.
  • Retain all 38 pre-rebase commits: 32 patch-identical and six intentional test-only conflict adaptations, plus one comment/format cleanup commit. Preserve main's shared Helix helper/single-GPU test split, CPU-directory collection and disjoint B300 batches; retain the PR's MTP7 helper behavior, supported-SM guard and separate 45-minute backend coverage. No production-code or vendor-pin conflict.
  • Post-rebase local checks: 22 policy/registry cases, 6 Kimi worker-environment cases and 10 standalone warmup cases passed. The policy/registry modules match the rebased source byte-for-byte but run against isolated pre-rebase dependencies; this is not a matching full-tree native build. The other 16 cases require no TRT native import. AST checks preserve the 3 relocated helper implementations and existing strict MTP7 test; syntax checks pass for all 50 changed Python files.
  • PR-file Ruff and commit hooks passed. All-file hooks found one upstream F811 duplicate KVCacheLayerSpec import in unchanged test_mamba_cache_manager.py:89 (also imported at 33); other required hooks passed. Advisory Pyrefly is unavailable locally. No unrelated lint fix was added.
  • The rebased two-GPU Helix/native integration still needs CI validation; no new GPU, e2e or performance experiment was run. Historical performance tables and their snapshot identities below are unchanged.

Earlier review follow-up — 2026-11-16 (before this rebase)

  • Pre-rebase follow-up head: 3874b546b646d2283d44cc463f549ef7cf0ac4a1; pinned FlashInfer dev source: 95546a07288ce7beaf3f88f7cffda090eed68313 on prims-ts-mla-prefill--flashinfer. The canonical branch and compatibility patch are unchanged.
  • Plan-only tensor-scale placeholders avoid retaining profiling workspaces. CuTe DSL's page-128 coverage bypass no longer depends on an earlier PrimTS provider. Existing tests now check current capture-call bindings, prepared sparse launches, supported SMs, and fixture module cleanup.
  • Shared statistics-budget/helper-documentation fixes are synchronized from FlashInfer [TRTLLM-1316] refactor: Remove unnecessary pipeline parallelism logic from postProcessRequest #5489. Its trace-template identity fix also resolves that PR's CPU trace-apply regression; the trace module is outside TRT-LLM's vendored selection but is included on the pinned FlashInfer branch.
  • Local checks: 88 PrimTS CPU cases, 84 CuTe DSL adapter cases, 34 final targeted/registry cases, and 10 standalone warmup cases passed. These runs overlap; they are not a count of distinct tests. The warmup fixture restores its module namespace exactly. Adapter checks use the exact 61e2b9662c67 CI wheel with the current Python sources; all 95 native ELF files remain byte-identical.
  • Mandatory all-file pre-commit checks and offline/local-source vendor verification pass. Advisory Pyrefly is unavailable locally. No new GPU/e2e/performance experiment was run for this follow-up; previously reported performance and GPU evidence remain historical.
  • The Helix acceptance-length baseline is deferred at maintainer request. The sparse resolver-cache issue is handled separately by [None][perf] Precompile PrimTS sparse MLA in decoder warmup and reuse decode views #22274; no duplicate fix/test is added here.

The earlier 61e2b9662c67 head passed CI pipeline 70292. That is not a final-head CI result for these follow-up commits. Older validation snapshots below remain historical.

Earlier update validation

  • Post-rebase: all 35 commits / 58 changed-file entries retained; 34 identical patches and the single intentional B300 YAML resolution. Both CSA2 and PrimTS backend groups remain excluded from the aggregate and present exactly once in dedicated entries; the 90-/45-minute budgets are retained.
  • 50 standalone host-contract tests passed on the rebased source with GPUs hidden (capacity, public-planner warmup and K3 worker selection). Initial collection attempts hit an unrelated stale home-directory entry; bounding pytest's confcutdir to the worktree resolved the runner issue without changing tests or production code.
  • 22 focused registry/dispatch tests passed for the final policy before rebase, with explicit new-code origins and separate before/after original-runtime audits. All four policy/test files are byte-identical after rebase. Existing tests were extended; no new GPU test matrix was added.
  • Syntax checks passed for all 48 changed Python files, and required pre-commit/vendor/version/test-list checks passed. The advisory Pyrefly hook could not run because Pyrefly is not installed; this is not a type-check pass.
  • Only the test-list YAML conflicted, so no new native build, GPU test, performance experiment or E2E run was performed for this rebase. Earlier GPU evidence remains scoped to its recorded snapshots.

CI was re-requested for 61e2b9662c using exactly /bot run --disable-fail-fast: invocation. This is a request, not a passing CI result.

Earlier qualification — prior snapshots

Scope Result Qualification limit
Rebase audit 31 commits retained; 27 identical patches, four generated-vendor metadata/header adaptations Lock and patch regenerated with the vendor tool; no code-body conflict resolution
Standalone capacity/warmup/K3 worker-selection checks 50 passed, GPUs hidden Host contracts, not native attention numerics
Actual phase-selective CPU regressions Original two failures reproduced; all 8 CPU cases passed after the assertion fix Real x86 CI wheel from build #13122 / pipeline #70055; tested product sources verified byte-identical; no collection stubs
B300 scheduling Collection partition 5,894 = 5,795 + 99, no duplicates or omissions; test-list validation passed Collection-only evidence, not a new GPU execution result
Repository checks Cumulative pre-commit, syntax and vendor verification passed Five advisory branch/loop Pyrefly warnings on the rebased range; the two-file CI fix has zero diagnostics
Paired FlashInfer cleanup 21 host + 13 B300 GPU cases passed, no GPU skips Upstream #5489 test additions reduced 7,453 → 1,465 lines; not TRT-LLM full-model qualification

CI failure #69865 had a confirmed warmup-order expectation error on both CPU architectures. The B300 aggregate hit its 90-minute budget after 5,808 inner records with zero inner assertion failures; the backend file accounted for 895 seconds of recorded test time, including substantial coverage added by this PR. It now has its own 45-minute entry and is excluded from the unchanged 90-minute aggregate. No test, parameter, reference or tolerance is removed by this scheduling change.

The H100 stage also had checkpoint filesystem errors (Err108/Err5) and a worker-readiness timeout; the latter has no worker-level traceback and is not claimed fixed. No unrelated inference change or waiver is added.

The earlier CI requests and helper/pipeline results in this subsection predate the current head; they are retained as historical evidence, not the status of the new invocation above.

Retained coverage and consolidation

  • Keep the FP8 K3 DEP16 test for reuse/chunking/graphs and TEP8 DSpark7 test for speculative decode/acceptance length. TIMEOUT (90) on the DEP16 entry means 90 minutes, not seconds.
  • Keep CP2 Helix/MTP7 ownership/replay, numerical geometry/merge, FP8-scale, sparse-launch ABI and CUDA-graph regressions. The full-model CP8 Helix test remains development/QA coverage without a new CI-list entry; it is not reported as executed on this head.
  • The prior focused consolidation reduced adapter/capacity/warmup/native-layout/phase-orchestration coverage from 386→373 defined cases, not the whole PR/CI suite. The new dispatch policy extends one existing case to ten CPU cases (+9), covering default eligibility, explicit selection and unchanged correctness rejection; no additional GPU cases or CI entries are introduced.
  • Native layout cases are 20→12, an unchanged-body subset retaining all six FP8 geometry/input-dtype specializations, strided non-FP8 alias checks, and compact/empty controls.
  • Route 76 new adapter + 44 capacity/warmup cases to CPU CI; unchanged base tests keep their prior scheduling. No tolerances, references or required full-model coverage were weakened. No CI-time speedup is claimed from counts alone.

Remaining qualification

  • Current-head full-checkpoint DEP16/Helix8 accuracy and end-to-end performance have not been rerun. The completed rank-local kernel/backend comparisons above do not establish current-head full-model throughput, TTFT, output equivalence or deployment suitability.
  • Earlier qualification recorded a BF16 Helix page-boundary tolerance failure; this update neither waives it nor claims it fixed.
  • The shape-specific SM107 single-split override remains outside this feature PR on the held TRT performance branch (92adb50c83) and FlashInfer performance branch (9dacdaa1b). These are historical snapshots, not current-head A/B controls differing only by that optimization.
Historical validation and performance — older snapshots, not current-head qualification

Earlier B200 host-sizing measurements compared trusted uncached MLA preparation (5.46–5.63 us) with the former per-layer cache (2.40–2.55 us). Standard decode baseline/trusted preparation was 23.50–24.50 / 17.85–18.21 us. These measurements do not benchmark the current shared functools.cache wrappers.

An earlier warmup comparison with already-loaded kernels measured 33.170→9.133 ms median across seven fresh warmer objects, with sample SD 0.122/0.036 ms. Fresh-process warm-disk single samples were 1.984/2.000 s and do not establish a startup speedup.

At the earlier f118184d6 source checkpoint, 22 B200 kernel cases and seven TRT native anchors passed. Those results, and the older adapter counts below, are historical—not matching-native validation of cad287ef61.

Review cleanup — a30df992e1

  • Query uniformity is no longer a dispatch-cache field or a per-selection scan in FmhaManager. The private helper lives in prims_ts.py; the fixed-query MLA support check raises on ragged multi-token input rather than falling back. No forward-time scan is added. This is a cache-miss sanity check; callers must preserve fixed-width Q on cache hits.
  • manager.py, shared utils.py, and test_fmha_manager.py now match the PR base. Removed the requested header from the node2/gpu8 YAML and a stale test asserting the deleted BF16 V-copy reservation.
  • Reviewed all 19 previously changed test files. Keep meaningful dtype/layout/geometry, graph, merge, sparse-ABI, capacity and warmup regressions. Consolidate context-wrapper cases 48→8, decode-wrapper cases 28→14, workspace-argument cases 12→4, and native preprocessing integration 18→6. Retained parameter combinations are subsets of the previous matrices; no tolerances, samples or accuracy gates were weakened.
  • Additional full-model K3 CI cases: 3→2. Keep FP8 DEP16 cache reuse/chunking/graphs and FP8 TEP8 DSpark7. Drop the additional BF16 DEP16 full-model run; BF16 coverage remains in native and real-module tests.
  • Validation at that checkpoint: 457 host-contract tests passed; both full-model CI selectors collect. The same component selection used for the previous B200 qualification now collects 465 instead of 567 cases. These counts are not the entire CI suite and do not imply a proportional CI speedup.
  • Reused the matching native build; source module origins were verified. Actual strict file-list Pyrefly, pre-commit and formatting checks passed. Retained GPU cases are unchanged subsets of the prior passing B200 run; no new GPU/full-model/performance experiment was run for this cleanup. FlashInfer pin/code is unchanged. No new CI bot retry is posted.

Earlier main-rebase validation (6df6036df4)

  • Rebased 06609107ac to 6df6036df4 on main a1c22fc7a5. All 58 changed files remain. The package-version-removal commit is already in main, so 21 commits become 20 without losing functionality or performance work.
  • Resolved overlaps in PrimTS admission/workspace logic and tests: retain standard FP8 preprocessing/device scales, MLA context/MTP/Helix, and Rubin MLA/P128 routing. The old Blackwell-only test restriction is redundant with this PR's existing SM107 support.
  • FlashInfer source pin, compatibility patch and vendored tree are unchanged byte-for-byte from the previous PR head.
  • Fresh matching native B200 build passed. The complete 567-case selection passed with 0 failed / 0 skipped, covering standard FP8 and native GQA, expanded MLA geometries/strided V, K3 cached/chunked context and FP8 MTP5/MTP7, scale/device-metadata graph replay, prepared sparse launches, page-size routing, and phase warmup. This is component validation, not a new full-model or multi-GPU run.
  • The first run passed all 567 test assertions but exited 4 because pytest's import-path guard rejected the temporary runner directory automatically added by Python. Correcting only the local launcher to python -P gave a complete 567-pass rerun with exit 0, with the guard still enabled. No product-code or test change was needed; reruns are not additional unique coverage.
  • Full pre-commit, syntax/undefined-name checks, remote Pyrefly, and vendor integrity checks passed. Installed native/Python origins and all 58 changed-file hashes match the tested source. The prior test consolidation is retained; no new parameter grids or CI entries were added by this rebase.

Previous source-refresh validation

  • Fresh B200 native build completed with CUDA13.4, Torch2.14.0a0, CUTLASS DSL4.8.0 and FlashInfer0.7.2. The local out-of-tree attribution scanner warned about its working directory; the wheel is a validation artifact, not a release build.

  • The matching-wheel 522-case selection first reported 514 passed / 8 failed / 0 skipped. All failures were the prepared sparse MLA ABI adaptation and its old-schema fixtures. After appending the two optional slots and updating the fixtures, all 8 affected checks passed, including graph replay, one/two-pool paths and persistent-kernel reload. This is a full run plus a targeted post-fix rerun, not a second full 522-case execution.

  • All selected K3 full-attention component cases (BF16/FP8 KV, cached/chunked context, TP-local/DP head geometry), FP8 linear MTP, expanded geometries, FP8 scales, and dense graph replay passed. This is component qualification, not a new full-model or multi-GPU K3 run.

  • Native library/module origins and changed-source hashes were verified against the installed wheel. Final committed files exactly match the post-fix qualification manifest. Full pre-commit checks and remote Pyrefly passed; vendor integrity checks passed against local and published source.

  • FlashInfer's full selected source run reported 525 passed / 8 hardware-policy skips / 2 failed. The rebase-induced device-scale reducer compiler error was fixed; 22 targeted rerun cases passed. The remaining BF16 Helix boundary tolerance failure reproduces identically on both old source branches (a4b6e9028 and 64273c074): 2/884736 elements, greatest absolute error0.0006412482 versus atol0.0005. Its test, reference and tolerances remain unchanged. The 22 cases overlap the full run and are not additional unique coverage.

  • Before/after patch-range and changed-file audits preserve the previous integration and performance work. The old-dev-only sink helper is superseded by the refreshed implementation and covered by strict sink-scale regressions.

  • Test consolidation preserves distinct geometry/dtype/layout/topology contracts and the required K3 CI case. TRT native layout cases: 48→20; cache integration: 36→18; independent capacity boundaries: 144→18. FlashInfer affected-file collection: 2391→2373, including new device-scale/compact-split regressions. No proportional CI speedup is claimed.

  • No new end-to-end or performance experiment was run for this rebase. FlashInfer CI remains user-managed; no bot command is issued there.

  • Multi-GPU CI approval must follow the normal maintainer policy.

Historical performance and accuracy — unchanged, not refreshed-head qualification

Main-based prepared-context comparison on Rubin

After the correctness gates passed, 36 fresh-process receipts completed three ABBA blocks per case. Baseline e2203f939e6c95475d7ec609fadfb916c04a2bec and candidate 005d91701b95e88a3495da84e6d384efccc606d7 contain the same original-scale correctness change. Their production delta is only full-tile dense LDRED plus request-local dense KV domains. Both use actual runtime Q/K maxima.

Synthetic case Baseline graph latency Optimized graph latency Mean paired change Sample SD
One request, statistics ON 100.212 us 91.859 us -8.335% 0.089 pp
Two ragged/empty requests, statistics ON 376.195 us 102.620 us -72.721% 0.018 pp
One request, statistics OFF 73.045 us 72.087 us -1.312% 0.040 pp

All cases use packed dense FP8 E4M3 attention, H96, Dqk192/Dv128, and BF16 output. The one-request cases use Q/K lengths 1920/1920 and Q/K capacities 2048/2048. The two-request case uses Q lengths [1664,256], K lengths [0,3840], and capacities Q2048/K4096. All eager, graph, and post-timing numerical checks passed; input identities were unchanged. An independent audit recomputed all 360 raw CUDA-event samples, receipt medians, and nine block summaries. The means of paired block ratios are reported, with sample SD across three blocks.

This is one-allocation prepared-wrapper graph replay, not a CuTeDSL/trtllm-gen comparison or a new K3 end-to-end result. The large empty-partition gain does not establish how frequently that geometry occurs in serving. Statistics ON/OFF also changes precision policy, so their absolute-time difference is not an isolated statistics-store cost.

Functionality-only Kimi K3 qualification on Rubin

  • 365 checks passed, including real K3 MLA module coverage with MTP5/MTP7 and backend/selection contracts.
  • A 1,319-question GSM8K run scored 96.058% with strict PrimTS, FP8 target KV/P64, TP8/EP8, attention DP disabled, DSpark7, CUDA graphs, and FP32 KDA state. No forced KDA-engine override was used in this functionality run.
  • This validates the functionality-only snapshot (26e9b3b33c plus FlashInfer d9f0146; published source-equivalent functionality commit df3434b1d), not every current-main combination.

Controlled DEP16 performance experiment on Rubin

The performance experiment used FP8 target KV/P128, TP16/EP16 with attention DP, DSpark5 (six target query tokens per verification step), chunked prefill, overlap scheduling, and CUDA graphs. Both policies used the same BF16 KDA state and engine prefill path; drafter KV remained BF16/P32. The fixed workload was 4,096 input / 256 output tokens at concurrency 128 and 256, seed 42, temperature 0, and ignore-EOS. This is an aggregated-serving comparison, not the original disaggregated AgentX workload.

Three allocation-bounded ABBA blocks produced 24 accepted measured windows. All measurements were retained; sealed-source/artifact and compilation-log audits passed. Observed native compilation warnings were confined to startup; no matching compilation markers occurred in accepted measured windows. This does not prove zero unlogged JIT.

Concurrency CuTeDSL + trtllm-gen output tokens/s Optimized PrimTS output tokens/s Mean paired-block change Sample SD
128 3,077.84 3,144.82 +2.18% 2.57 percentage points
256 4,671.57 4,696.30 +0.56% 2.03 percentage points

This supports near throughput parity in this workload, not a robust speedup, statistical equivalence, or noninferiority claim. Policy TPS columns are arithmetic means; percentage changes are means of paired block ratios. The experiment used the pre-rebase TRT be927b7015 runtime and the source-equivalent FlashInfer 30e7ea65/76e822974 kernels. It is not a latest-main native performance result or a causal before/after comparison across different allocation series.

Accuracy and scope limits

  • For the DEP16 performance configuration, native GSM8K scored 96.5125% (1,273/1,319) and PrimTS 95.8302% (1,264/1,319). Both passed the reused 93.2975% gate, but there is no dedicated FP8+DSpark reference row. The nine-answer net gap and exact-text differences remain; output equivalence and deployment suitability are not established.
  • Historical CP8 prepared-attention plus real FIFO2 component measurements showed approximately 1.09–1.10x graph speedup for their static-capacity/single-split composition. They are not full-model Helix results or a rerun of the final kernel bundle.
  • That historical performance study excluded strict 8-GPU aggregated Helix. Current code has a development-only experimental opt-in; current-head full-model Helix validation remains pending.
  • The benchmark's shared KDA-engine choice does not fix the separate Small-BH NaN issue, and no KDA behavior change is shipped here.
  • Latest-head CI and human review remain required before merge.

PR Checklist

  • Description explains the implementation, source attribution, validation scope and limitations.
  • Changes follow coding guidelines and DCO requirements; tests cover the new paths.
  • No new public LLM configuration fields or model identifiers are added.
  • Vendor/dependency CODEOWNER and normal review/CI requirements remain applicable.
  • Current evidence is separated from historical component, full-model and performance results.
  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see available CI commands, comment /bot help.

Dev Engineer Review

  • Historical vendor pin and attribution: Before this follow-up, the FlashInfer vendor entry tracked branch prims-ts-mla-prefill--flashinfer at commit f118184d6d88332a83d46ea198717e2c5b76ed6a. Later vendor validation reported the vendor metadata and source attribution as valid for the checked revision. That validation did not approve the code. Promotion still requires the source PR to merge. Vendor comments also report a blocked canonical-branch promotion and a required branch migration. Confirm the pinned source and its digests before merge.
  • MLA context FP8 quantize pairs: mlaKernels.cu accepts QK-nope/V pairs (64,128), (128,128), and (192,256) for non-absorption FP8 quantization, with RoPE dimension 64. mlaPreprocessOp.cpp accepts (128,128), (192,128), and (256,256). The lists differ. Confirm which pairs each path accepts and that the paths agree on shared geometry.
  • Merge kernel head size: mlaChunkedPrefill.cu accepts head size 256 only for non-float types. float rejects other sizes. A compile-time assertion limits in-place softmax-stat writes to 32 vectors per head.
  • Behavior change in phased.py: MLA context attention now receives full K and V tensors, not a slice limited to the context query span. Non-MLA paths keep the existing slice logic.
  • Signature changes in prims_ts.py: get_paged_kv_policy_unsupported_reason gains allow_linear_mla, and get_attention_feature_unsupported_reason gains allow_helix. Both are keyword-only with defaults, so existing callers remain compatible. SUPPORTED_MLA_SMS now includes 107.
  • Prepared sparse MLA launch: The wrapper caches up to 256 prepared argument sets and clears the cache on replan. Reuse occurs only for validate=False calls that meet the stated conditions. Review stale-pointer and shape-change risk in this cache.
  • Warmup memory and graph capture: PrimsTSDenseMlaWarmup allocates per-device decode workspaces sized to the largest pending profile. It raises when planning runs during CUDA graph capture. Check peak memory at large max_batch_size settings.
  • Compile cache modes: The vendor patch adds precompile_only and probe_cached modes. These raise KernelBusy and KernelNotCached. The patch also moves several attention APIs to the standard flashinfer_api decorator without trace dispatchers. Verify that tracing consumers do not depend on the removed dispatchers.
  • Helix: Aggregated Helix stays development-only behind TRTLLM_HELIX_AGG_EXPERIMENTAL=1.
  • Scope expansion: Supported MLA SMs now include 107. The MLA tests drop the separate SM100/SM103 skip markers and rely on the file-level SM100/SM103/SM107 condition.
  • Open validation gaps: The final-head CI run has no passing result. Pipeline #70068 reported failure. The current-head full-model Helix test is not reported as executed. Current-head DEP16/Helix8 full-checkpoint accuracy and the paired Rubin performance comparison are pending. An earlier BF16 Helix page-boundary tolerance failure is not resolved.

QA Engineer Review

Test files changed:

  • Added: tests/unittest/others/test_kimi3_ci_selection.py, tests/unittest/_torch/attention/test_mla_context_merge.py, tests/unittest/_torch/attention/test_prims_ts_mla_capacity.py, tests/unittest/_torch/attention/test_prims_ts_mla_context_geometry.py, tests/unittest/_torch/attention/test_prims_ts_mla_warmup.py, tests/unittest/_torch/attention/hw_agnostic/test_prims_ts_sparse_mla_prepared.py.
  • Modified: tests/integration/defs/accuracy/test_kimi3.py, tests/integration/defs/accuracy/references/gsm8k.yaml, tests/test_common/grouped_test_utils.py, tests/unittest/_torch/attention/multi_gpu/test_mla_helix_context.py, tests/unittest/_torch/attention/sparse/deepseek_v4/test_prims_ts_sparse_mla_fmha.py, tests/unittest/_torch/attention/test_cute_dsl_mla.py, tests/unittest/_torch/attention/test_prims_ts_attention_backend.py, tests/unittest/_torch/attention/test_prims_ts_fmha.py, tests/unittest/_torch/compilation/test_phase_selective_forward.py.
  • Removed: none reported.

Test-list changes:

  • l0_cpu.yml: adds unittest/others/test_kimi3_ci_selection.py.
  • l0_gb300_multi_nodes_node2_gpu8.yml: adds TestKimiK3DSpark.test_prims_ts_fp8_tep8.
  • l0_gb300_multi_nodes_node4_gpu16.yml: adds TestKimiK3.test_prims_ts with a 90-second timeout.
  • l0_b300.yml: removes test_prims_ts_attention_backend.py from the B300 attention aggregate. That file is listed separately with a 45-minute timeout. The aggregate keeps its 90-minute timeout.

Covered behavior:

  • Kimi K3 PrimTS accuracy with FP8 KV cache, TEP8, and aggregated Helix CP8.
  • MLA context geometry for (128,128), (192,128), and (256,256) with BF16 and FP8 KV.
  • Merge-chunked-attention ragged in-place behavior and unsupported-width errors.
  • Warmup planning, capacity bucketing, and capture-time gates.
  • Strict prims_ts selection on workers, with environment restore.
  • Prepared sparse MLA launch equivalence.
  • Helix owner-only KV bounds and CUDA graph replay.
  • MLA support checks, including SM 107 with CUTLASS 13.3.

Gaps:

  • No passing final-head CI result is reported.
  • Current-head full-checkpoint DEP16/Helix8 accuracy is pending.
  • The GB300 node-level runs are listed, but their results are not reported.
  • Multi-GPU tests need the ci: full pre-merge approved label.
  • The BF16 Helix page-boundary tolerance failure is not resolved.
  • Native attention numerical validation is not reported as a new GPU result.

Coverage verdict: needs follow-up.

Per-File QA Perspective

  • 3rdparty/vendor_patches/flashinfer-prims-ts.patch: Verify precompile and probe modes, the KernelBusy and KernelNotCached errors, and the 256-entry prepared cache. Confirm the 1024-entry sparse plan limit.
  • 3rdparty/vendor_sources.lock.yaml: Confirm the pinned commit and digests match the vendored snapshot.
  • cpp/tensorrt_llm/kernels/mlaChunkedPrefill.cu: Verify merge results for head sizes 128 and 256 and rejection of unsupported sizes for float. Covered by test_mla_context_merge.py.
  • cpp/tensorrt_llm/kernels/mlaKernels.cu: Verify FP8 quantize with zero and nonzero total_kv_len, and rejection of unlisted QK/V pairs. Confirm the pair list matches the preprocess op.
  • cpp/tensorrt_llm/thop/mlaPreprocessOp.cpp: New mlaContextPreprocess op. Verify rejection of INT8 and FP4 cache modes, alignment checks, and aliasing rules for Q/K outputs. Covered by the prims_ts attention backend tests.
  • tensorrt_llm/_torch/attention/backends/fmha/phased.py: Verify MLA context receives full K and V tensors. Non-MLA context still receives the sliced span.
  • tensorrt_llm/_torch/attention/backends/fmha/prims_ts.py: Verify MLA context and generation dispatch, linear MTP acceptance, SM 107 support, and the new keyword-only flags. Check that standard generation remains limited to one token.
  • tensorrt_llm/_torch/attention/backends/fmha/prims_ts_mla_warmup.py: Verify warmup runs only for eligible profiles. Verify that CUDA graph capture raises. Check decode workspace memory at high batch sizes.
  • tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/warmup.py: Verify that the dense MLA warmup runs after CUDA graph capture with the configured limits.
  • tests/integration/defs/accuracy/references/gsm8k.yaml: Adds FP8 KV references (96.5) for Kimi-K3. Verify the reference matches the observed run.
  • tests/integration/defs/accuracy/test_kimi3.py: New 16-GPU test_prims_ts and FP8 or Helix runner paths. Confirm the listed entries run and report results.
  • tests/integration/test_lists/test-db/l0_cpu.yml: Adds the Kimi K3 CI selection unit test to the CPU list.
  • tests/integration/test_lists/test-db/l0_b300.yml: Moves the PrimTS attention backend test out of the B300 aggregate. Confirm the separate 45-minute entry runs.
  • tests/integration/test_lists/test-db/l0_gb300_multi_nodes_node2_gpu8.yml: Adds the FP8 TEP8 PrimTS test to the GB300 GPU8 list.
  • tests/integration/test_lists/test-db/l0_gb300_multi_nodes_node4_gpu16.yml: Adds the 16-GPU PrimTS test with a 90-second timeout. Confirm the timeout is sufficient.
  • tests/test_common/grouped_test_utils.py: Test-only helpers for worker environment snapshot and restore. Confirm that restore runs on failure paths.
  • tests/unittest/_torch/attention/hw_agnostic/test_prims_ts_sparse_mla_prepared.py: CPU-only checks of prepared versus full launch arguments. Confirm the test runs in the hardware-agnostic list.
  • tests/unittest/_torch/attention/multi_gpu/test_mla_helix_context.py: Covers speculative query widths, Helix owner-only KV, and FP8 two-GPU runs. Multi-GPU execution depends on the CI label.
  • tests/unittest/_torch/attention/sparse/deepseek_v4/test_prims_ts_sparse_mla_fmha.py: Verifies exact equality between full and validate=False launches.
  • tests/unittest/_torch/attention/test_cute_dsl_mla.py: Expects SM 107 128-token pages to be non-exclusive to CuTe DSL. Verify the expectation matches the runtime.
  • tests/unittest/_torch/attention/test_mla_context_merge.py: CUDA-gated merge tests. Verify the unsupported-width error text matches the implementation.
  • tests/unittest/_torch/attention/test_prims_ts_attention_backend.py: Covers MLA preprocessing, FP8 quantize, and Kimi K3 cached and chunked attention. Verify the removed skip markers do not expose unsupported SMs.
  • tests/unittest/_torch/attention/test_prims_ts_fmha.py: Broad support-check and wrapper coverage. Verify the SM 107 CUTLASS 13.3 expectation in CI.
  • tests/unittest/_torch/attention/test_prims_ts_mla_capacity.py: CPU-only AST-loaded checks of MLA capacity floors. Verify the loader matches the current prims_ts.py layout.
  • tests/unittest/_torch/attention/test_prims_ts_mla_context_geometry.py: SM 100, 103, and 107 geometry test against an FP64 reference. Verify NaN handling for empty historical partitions.
  • tests/unittest/_torch/attention/test_prims_ts_mla_warmup.py: CPU-only warmup planner tests. Verify retry and deduplication behavior.
  • tests/unittest/_torch/compilation/test_phase_selective_forward.py: Verifies the dense MLA warmup call with compiled forward enabled.
  • tests/unittest/others/test_kimi3_ci_selection.py: CPU-only check of strict prims_ts worker selection and environment restore. Verify the restore after constructor and evaluation failures.

@yuxianq

yuxianq commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator Author

@yuxianq — vendor promotion needs author action:

The pinned temporary source commit e377847b798c8203eec83802ea81794bf07303f1 does not contain the latest canonical branch tip ae3851ab5f958ba22f154059df5b38321f6369d2.

Fix: Fetch https://github.com/yuxianq/flashinfer.git / trtllm-prims-ts-dev-20261012 and rebase the temporary source branch prims-ts-mla-prefill--flashinfer onto ae3851ab5f958ba22f154059df5b38321f6369d2. Push it, then use scripts/vendor/manage.py to update the consumer lock and vendored snapshot to the rebased SHA. The consumer lock's old pin alone is not a sufficient base; updating only the PR metadata or temporary branch without its lock pin will not fix this.

The bot will recheck automatically on a later poll.

@yuxianq
yuxianq force-pushed the feat/prims-ts-mla-prefill branch from 47a02ee to d575c3f Compare October 9, 2026 08:17
@yuxianq

yuxianq commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #76928 [ run ] triggered by Bot. Commit: d575c3f Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #76928 [ run ] completed with state SUCCESS. Commit: d575c3f
/LLM/main/L0_MergeRequest_PR pipeline #63420 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@yuxianq

yuxianq commented Oct 13, 2026

Copy link
Copy Markdown
Collaborator Author

@yuxianq — the PrimTS upstream refresh #20155 has merged. Please update this source-update PR before merging it.

The new canonical source branch is yuxianq/flashinfer:trtllm-prims-ts-dev-20261012, initially pinned at ae312f7cc1723b06fda5168105ade0f328ee47d8.

  1. Rebase your temporary FlashInfer branch onto the new canonical branch; retain only changes not already included upstream, resolving any API differences.
  2. Rebase this TRT-LLM PR onto latest main, including the refreshed compatibility patch and integration changes.
  3. Keep vendors.flashinfer-prims-ts in 3rdparty/vendor_sources.lock.yaml pointing to your temporary source branch, update its immutable commit, and regenerate the selected vendored files and digests. Do not restore the old canonical branch/pin while resolving conflicts.
  4. Update the PR description's vendor-promotion block if the paired upstream PRs or explicit commit resolutions change after rebasing.
  5. Verify the vendored snapshot and rerun the relevant tests and CI.

The publishing bot now targets the new dated canonical branch. After this source-update PR merges, its lock-only promotion will target that branch. The old trtllm-prims-ts-dev branch is intentionally unchanged.

@yuxianq
yuxianq force-pushed the feat/prims-ts-mla-prefill branch from 8298aba to 72a6cea Compare October 13, 2026 09:32
@yuxianq

yuxianq commented Oct 13, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #77492 [ run ] triggered by Bot. Commit: 72a6cea Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #77492 [ run ] completed with state SUCCESS. Commit: 72a6cea
/LLM/main/L0_MergeRequest_PR pipeline #63880 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@yuxianq

yuxianq commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator Author

Vendor validation: flashinfer-prims-ts

@yuxianq — Vendor promotion needs author action:

The pinned temporary source commit e377847b798c8203eec83802ea81794bf07303f1 does not contain the latest canonical branch tip ae3851ab5f958ba22f154059df5b38321f6369d2.

Fix: Fetch https://github.com/yuxianq/flashinfer.git / trtllm-prims-ts-dev-20261012 and rebase the temporary source branch prims-ts-mla-prefill--flashinfer onto its latest tip (observed ae3851ab5f958ba22f154059df5b38321f6369d2). Push it, then use scripts/vendor/manage.py to update the consumer lock and vendored snapshot to the rebased SHA. The consumer lock's old pin alone is not a sufficient base; updating only the PR metadata or temporary branch without its lock pin will not fix this.

The bot will recheck automatically on a later poll.

Checked PR revision: 72a6ceab79ef. This is a snapshot; newer results are posted as new comments.

Checked source: https://github.com/yuxianq/flashinfer.git / prims-ts-mla-prefill--flashinfer, pin e377847b798c8203eec83802ea81794bf07303f1.

Supersedes previous vendor validation.

@yuxianq

yuxianq commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator Author

Vendor validation: flashinfer-prims-ts

@yuxianq — Vendor promotion needs maintainer action:

Waiting for the preceding vendor promotion: the base lock still points to repository https://github.com/heyuhhh/flashinfer.git, branch trtllm-prims-ts-dev.

Fix: A maintainer must finish the preceding lock-only promotion back to yuxianq/flashinfer / trtllm-prims-ts-dev-20261012. After it merges, the author should rebase this consumer PR onto the latest base branch and rebase its temporary source branch onto the new canonical pin, then update the lock/vendor snapshot. No metadata edit can unblock the preceding promotion.

The bot will recheck automatically on a later poll.

Checked PR revision: 72a6ceab79ef. This is a snapshot; newer results are posted as new comments.

Supersedes previous vendor validation.

@yuxianq

yuxianq commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator Author

Vendor validation: flashinfer-prims-ts

@yuxianq — Vendor promotion needs author action:

Source update is not a fast-forward.

Fix: Fetch canonical repository https://github.com/yuxianq/flashinfer.git, branch trtllm-prims-ts-dev-20261012, and rebase the temporary source branch prims-ts-mla-prefill--flashinfer onto its latest tip without merge commits, not just the consumer lock's pin 0e0761ef5ef10cb0762cc1b890a5c1a329bb49bf. Then use scripts/vendor/manage.py to update 3rdparty/vendor_sources.lock.yaml and the vendored snapshot to the rebased SHA, and push the source branch and consumer PR. Editing metadata alone will not fix this.

The bot will recheck automatically on a later poll.

Checked PR revision: 72a6ceab79ef. This is a snapshot; newer results are posted as new comments.

Checked source: https://github.com/yuxianq/flashinfer.git / prims-ts-mla-prefill--flashinfer, pin e377847b798c8203eec83802ea81794bf07303f1.

Supersedes previous vendor validation.

@yuxianq

yuxianq commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator Author

Vendor validation: flashinfer-prims-ts

@yuxianq — Vendor promotion needs maintainer action:

Waiting for the preceding vendor promotion: the base lock still points to repository https://github.com/yunruis/flashinfer.git, branch trtllm-prims-ts-dev-20261023.

Fix: A maintainer must finish the preceding lock-only promotion back to yuxianq/flashinfer / trtllm-prims-ts-dev-20261012. After it merges, the author should rebase this consumer PR onto the latest base branch and rebase its temporary source branch onto the new canonical pin, then update the lock/vendor snapshot. No metadata edit can unblock the preceding promotion.

The bot will recheck automatically on a later poll.

Checked PR revision: 72a6ceab79ef. This is a snapshot; newer results are posted as new comments.

Supersedes previous vendor validation.

@yuxianq

yuxianq commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator Author

Vendor validation: flashinfer-prims-ts

@yuxianq — Vendor promotion needs author action:

Source update is not a fast-forward.

Fix: Fetch canonical repository https://github.com/yuxianq/flashinfer.git, branch trtllm-prims-ts-dev-20261012, and rebase the temporary source branch prims-ts-mla-prefill--flashinfer onto its latest tip without merge commits, not just the consumer lock's pin aa709061230a58858375c513056e7733fa74294a. Then use scripts/vendor/manage.py to update 3rdparty/vendor_sources.lock.yaml and the vendored snapshot to the rebased SHA, and push the source branch and consumer PR. Editing metadata alone will not fix this.

The bot will recheck automatically on a later poll.

Checked PR revision: 72a6ceab79ef. This is a snapshot; newer results are posted as new comments.

Checked source: https://github.com/yuxianq/flashinfer.git / prims-ts-mla-prefill--flashinfer, pin e377847b798c8203eec83802ea81794bf07303f1.

Supersedes previous vendor validation.

@yuxianq
yuxianq force-pushed the feat/prims-ts-mla-prefill branch from 72a6cea to d829f61 Compare October 26, 2026 10:46
@yuxianq

yuxianq commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@yuxianq

yuxianq commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator Author

Vendor validation: flashinfer-prims-ts

@yuxianq — Vendor promotion metadata and source attribution are valid. This is not a code-review approval. Promotion still requires the source PR to merge.

No exact match was found for the following commits; similarity matching accepted them:

  • 9b4e7c9357a3d865dea0e19fbc38916c55dd3de5: upstream PR #5489, similarity 92.06% — accepted (threshold 90.00%; similar-edits).

Scores compare normalized patch lines in lock-selected files, not semantic equivalence. These changes retain retain-until-verified refresh policy.

Checked PR revision: d829f61b3628. This is a snapshot; newer results are posted as new comments.

Checked source: https://github.com/yuxianq/flashinfer.git / prims-ts-mla-prefill--flashinfer, pin 75961f74dac2c438cd7bce3dd135d527e78e49ec.

Supersedes previous vendor validation.

@github-actions

Copy link
Copy Markdown

👎 Promotion blocked, new vulnerability found

Vulnerability report

Component Vulnerability Description Severity
py-filelock CVE-2026-112906 filelock is a platform-independent file lock for Python. From 4.1.0 until 4.2.1, SoftFileLock versions from 4.1.0 can reuse a stale observation when an attacker pre-dates and removes its own marker, at least two lock-user processes are waiting, coarse or matching modification times preserve the observation, and an honest contender creates a fresh marker on the reused inode, causing that live marker to be removed as stale. This issue is fixed in version 4.2.1. MEDIUM
py-filelock CVE-2026-112914 filelock is a platform-independent file lock for Python. From 4.1.0 until 4.2.1, SoftFileLock can restore a stale marker by name after moving it aside, allowing another local user with rename access in a non-sticky lock directory to swap in a symbolic link and, on a runtime that follows the link and a same-filesystem target, make the lock user hard-link one of its files into the lock directory. This issue is fixed in version 4.2.1. MEDIUM
py-filelock CVE-2026-112937 filelock is a platform-independent file lock for Python. From 4.1.0 until 4.2.1, ReadWriteLock can remove a randomized .filelock-* private-hard-link directory during cleanup, allowing another local user with rename access in a non-sticky parent to observe a live name or forge a dead-owner sweep name and rename a sibling directory onto it for deletion. This issue is fixed in version 4.2.1. MEDIUM
py-filelock CVE-2026-112940 filelock is a platform-independent file lock for Python. From 3.20.4 until 4.2.1, ReadWriteLock let SQLite reopen the database and side files by name; on POSIX systems an attacker with rename access in a non-sticky parent can swap those names to cause overwrite or unlink operations, while a writable adjacent hot journal is additionally required for attacker-chosen content overwrites. This issue is fixed in version 4.2.1. HIGH

@yuxianq
yuxianq force-pushed the feat/prims-ts-mla-prefill branch from d829f61 to 0c80a0c Compare November 1, 2026 10:58
@yuxianq

yuxianq commented Nov 1, 2026

Copy link
Copy Markdown
Collaborator Author

Vendor validation: flashinfer-prims-ts

@yuxianq — Vendor promotion needs author action:

Upstream attribution is unresolved for these source commits:

  • df3434b1dc23df96527e202aa43ae233ed8641ee: no exact match; best similarities: upstream PR #5489: 12.95%; no PR meets the 90.00% similarity threshold
  • 06e9cc5c084a0f5791ae4989a9bda4a7a595eaca: no exact match; best similarities: upstream PR #5489: 2.63%; no PR meets the 90.00% similarity threshold
  • 502353da9e0f22d8eb1d772ecebd830d08eafbfe: no exact match; best similarities: upstream PR #5489: 6.59%; no PR meets the 90.00% similarity threshold
  • 9e26388ea95e5b40c924ffd2c6acef575cb5a8b1: no exact match; best similarities: upstream PR #5489: 7.84%; no PR meets the 90.00% similarity threshold
  • 76e8229748606d0e51f27f84fe19335cbe605ae9: no exact match; best similarities: upstream PR #5489: 5.45%; no PR meets the 90.00% similarity threshold

No exact match was found for the following commits; similarity matching accepted them:

  • 9b4e7c9357a3d865dea0e19fbc38916c55dd3de5: upstream PR #5489, similarity 92.06% — accepted (threshold 90.00%; similar-edits).

Scores compare normalized patch lines in lock-selected files, not semantic equivalence. These changes retain retain-until-verified refresh policy.

Fix: In the PR description, add missing PRs to vendors.flashinfer-prims-ts.upstream_prs, remove incorrectly listed PRs, or correct/rebase the source pin and update the vendored snapshot. For an intentional adaptation or ambiguous ownership, merge a targeted resolutions entry into the existing vendor metadata as shown below. Replace the SHA and URL and give a specific reason; preserve other entries. No complete commit_map is required. A resolution records an author assertion, not verified equivalence; it is retained during refresh until separately verified.

vendors:
  flashinfer-prims-ts:
    upstream_prs:
      - https://github.com/flashinfer-ai/flashinfer/pull/123
    resolutions:
      FULL_40_CHARACTER_SOURCE_SHA:
        upstream_pr: https://github.com/flashinfer-ai/flashinfer/pull/123
        reason: Explain the downstream adaptation or why this PR owns the change.

The bot will recheck automatically on a later poll.

Checked PR revision: 0c80a0cfc644. This is a snapshot; newer results are posted as new comments. Observed dev/upstream tip SHAs may be stale: tip movement alone does not post a new comment when the result is unchanged. Fetch the live branch before rebasing or choosing a new pin.

Checked source: https://github.com/yuxianq/flashinfer.git / prims-ts-mla-prefill--flashinfer, pin 76e8229748606d0e51f27f84fe19335cbe605ae9.

Supersedes previous vendor validation.

@yuxianq yuxianq changed the title [None][feat] Add PrimTS MLA prefill and Kimi K3 coverage [None][feat] Add PrimTS MLA prefill, MTP, and Rubin support Nov 1, 2026
@yuxianq

yuxianq commented Nov 1, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --extra-stage "GB300-8_GPUs-2_Nodes-PyTorch-SingleNvlinkDomain-Post-Merge-1"

@yuxianq

yuxianq commented Nov 1, 2026

Copy link
Copy Markdown
Collaborator Author

Vendor validation: flashinfer-prims-ts

@yuxianq — Vendor promotion needs author action:

Upstream attribution is unresolved for these source commits:

  • df3434b1dc23df96527e202aa43ae233ed8641ee: no exact match; best similarities: upstream PR #5489: 12.95%, upstream PR #6774: 6.24%; no PR meets the 90.00% similarity threshold
  • 06e9cc5c084a0f5791ae4989a9bda4a7a595eaca: no exact match; best similarities: upstream PR #6774: 3.92%, upstream PR #5489: 2.63%; no PR meets the 90.00% similarity threshold
  • 9e26388ea95e5b40c924ffd2c6acef575cb5a8b1: no exact match; best similarities: upstream PR #6774: 10.39%, upstream PR #5489: 7.84%; no PR meets the 90.00% similarity threshold
  • 76e8229748606d0e51f27f84fe19335cbe605ae9: no exact match; best similarities: upstream PR #6774: 9.41%, upstream PR #5489: 5.45%; no PR meets the 90.00% similarity threshold

No exact match was found for the following commits; similarity matching accepted them:

  • 9b4e7c9357a3d865dea0e19fbc38916c55dd3de5: upstream PR #5489, similarity 92.06% — accepted (threshold 90.00%; similar-edits).

Scores compare normalized patch lines in lock-selected files, not semantic equivalence. These changes retain retain-until-verified refresh policy.

Fix: In the PR description, add missing PRs to vendors.flashinfer-prims-ts.upstream_prs, remove incorrectly listed PRs, or correct/rebase the source pin and update the vendored snapshot. For an intentional adaptation or ambiguous ownership, merge a targeted resolutions entry into the existing vendor metadata as shown below. Replace the SHA and URL and give a specific reason; preserve other entries. No complete commit_map is required. A resolution records an author assertion, not verified equivalence; it is retained during refresh until separately verified.

vendors:
  flashinfer-prims-ts:
    upstream_prs:
      - https://github.com/flashinfer-ai/flashinfer/pull/123
    resolutions:
      FULL_40_CHARACTER_SOURCE_SHA:
        upstream_pr: https://github.com/flashinfer-ai/flashinfer/pull/123
        reason: Explain the downstream adaptation or why this PR owns the change.

The bot will recheck automatically on a later poll.

Checked PR revision: 0c80a0cfc644. This is a snapshot; newer results are posted as new comments. Observed dev/upstream tip SHAs may be stale: tip movement alone does not post a new comment when the result is unchanged. Fetch the live branch before rebasing or choosing a new pin.

Checked source: https://github.com/yuxianq/flashinfer.git / prims-ts-mla-prefill--flashinfer, pin 76e8229748606d0e51f27f84fe19335cbe605ae9.

Supersedes previous vendor validation.

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #81839 [ run ] triggered by Bot. Commit: 0c80a0c Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #81839 [ run ] completed with state SUCCESS. Commit: 0c80a0c
/LLM/main/L0_MergeRequest_PR pipeline #67491 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
…ime contracts

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
@yuxianq
yuxianq force-pushed the feat/prims-ts-mla-prefill branch from 7ff3d34 to cb08340 Compare November 19, 2026 03:12
@yuxianq

yuxianq commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@yuxianq

yuxianq commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator Author

Vendor validation: flashinfer-prims-ts

@yuxianq — Vendor promotion metadata and source attribution are valid. This is not a code-review approval. Promotion still requires the source PR to merge.

No exact match was found for the following commits; similarity matching accepted them:

  • 9cc9333bfe89eb4d5e9d9dd5e87e0bcb275e8b27: upstream PR #5489, similarity 99.90% — accepted (threshold 90.00%; similar-upstream-aggregate).

Scores compare normalized patch lines in lock-selected files, not semantic equivalence. These changes retain retain-until-verified refresh policy.

Checked PR revision: cb08340507f0. This is a snapshot; newer results are posted as new comments. Observed dev/upstream tip SHAs may be stale: tip movement alone does not post a new comment when the result is unchanged. Fetch the live branch before rebasing or choosing a new pin.

Checked source: https://github.com/yuxianq/flashinfer.git / prims-ts-fmha-lib--cp-dev-task-api-20261119, pin 5129a714e7f3e1c45a7127425fe5173b1f1f9e87.

Supersedes previous vendor validation.

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #85932 [ run ] triggered by Bot. Commit: cb08340 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #85932 [ run ] completed with state SUCCESS. Commit: cb08340
/LLM/main/L0_MergeRequest_PR pipeline #71161 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@QiJune QiJune left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yuxianq

yuxianq commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86072 [ run ] triggered by Bot. Commit: cb08340 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86072 [ run ] completed with state SUCCESS. Commit: cb08340
/LLM/main/L0_MergeRequest_PR pipeline #71285 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
@yuxianq
yuxianq requested a review from a team as a code owner November 20, 2026 01:50
@yuxianq

yuxianq commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator Author

Vendor validation: flashinfer-prims-ts

@yuxianq — Vendor promotion metadata and source attribution are valid. This is not a code-review approval. Promotion still requires the source PR to merge.

No exact match was found for the following commits; similarity matching accepted them:

  • 9cc9333bfe89eb4d5e9d9dd5e87e0bcb275e8b27: upstream PR #5489, similarity 99.90% — accepted (threshold 90.00%; similar-upstream-aggregate).

Scores compare normalized patch lines in lock-selected files, not semantic equivalence. These changes retain retain-until-verified refresh policy.

Checked PR revision: d3e08ba8e64a. This is a snapshot; newer results are posted as new comments. Observed dev/upstream tip SHAs may be stale: tip movement alone does not post a new comment when the result is unchanged. Fetch the live branch before rebasing or choosing a new pin.

Checked source: https://github.com/yuxianq/flashinfer.git / prims-ts-fmha-lib--cp-dev-task-api-20261119, pin d76f94bb6ae3bc1730689a1a080a79913dcc3606.

Supersedes previous vendor validation.

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
@yuxianq

yuxianq commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86239 [ run ] triggered by Bot. Commit: f2300c7 Link to invocation

qsang-nv pushed a commit to flashinfer-ai/flashinfer that referenced this pull request Nov 20, 2026
…#9641)

<!-- .github/pull_request_template.md -->

## 📌 Description

Follow up merged #5489 by replacing Prims-TS MLA decode's
`use_kv_bounds` / per-query `kv_bounds` interface with context-parallel
topology and global request lengths.

- `plan()` specializes `cp_world` and `cp_interleave_size`.
- `run()` receives `cp_rank` and graph-live global `seq_lens_kv`,
including the current query tokens. Planned `max_kv_len` and block
tables remain rank-local physical capacity.
- Both 1-CTA and 2-CTA kernels derive rank-local lengths and causal
prefixes directly. Packed queries use actual `qo_indptr` differences;
rank changes do not enter the compilation key.
- Preserve CP=1 behavior, empty/inactive-row contracts, split reduction
and LSE. The optional neutral writer uses a flattened grid without a
65,535-request grid-Y limit.
- Validate split-KV cluster size against the CUDA device’s launch limit
as well as occupancy. On B200 an occupancy query can report active
clusters for 17 CTAs, even though only 16 can launch. Over-limit
automatic splits use the separate reducer.
- Update validation, trace schemas/examples, documentation and the
existing focused regression file. Seven GPU cases remain; no large new
matrix is introduced.

The branch is based on upstream main. CP documentation updates are
confined to the PrimTS READMEs; `docs/development/backend_notes.md` is
unchanged. No new context-kernel or standard-attention changes are
introduced by this PR.

### Retained integration performance evidence

Same MLA implementation validated in the TRT-LLM integration, with full
FP8 KV cache, page size 128, 96 heads, latent/RoPE dimensions 512/64.
H-D1/H-D8: local batch 1/8, Q=8, CP8 rank 0, local history 512 (global
total 4104). D-D8/D-D16: local batch 8/16, Q=6, CP1, history 4096.

Values are **previous PrimTS → updated PrimTS microseconds**, using the
conservative order of balanced AB/BA runs and predefined confirmations.
These are retained integration-snapshot measurements, not a new GPU run
on this freshly main-based PR head.

| Scope | Case | Rubin SM107 | B200 SM100 | B300 SM103 |
|---|---|---|---|---|
| Raw generation | H-D1 | 14.240 → 13.752 | 20.480 → 20.480 | 20.448 →
18.432 |
| Raw generation | H-D8 | 18.000 → 17.936 | 16.368 → 16.384 | 16.352 →
16.368 |
| Raw generation | D-D8 | 26.656 → 26.624 | 36.832 → 36.864 | 34.784 →
34.816 |
| Raw generation | D-D16 | 30.552 → 30.584 | 65.504 → 65.536 | 65.792 →
66.261 |
| Attention backend | H-D1 | 17.680 → 17.536 | 24.576 → 24.608 | 24.448
→ 22.560 |
| Attention backend | H-D8 | 23.360 → 23.840 | 24.576 → 24.472 | 22.496
→ 22.496 |
| Attention backend | D-D8 | 30.464 → 30.472 | 40.960 → 40.944 | 38.896
→ 38.896 |
| Attention backend | D-D16 | 36.984 → 36.992 | 73.712 → 73.712 | 73.744
→ 73.752 |

50 warmups and 100 CUDA-event samples per process, CUDA graph replay,
2×L2 flush outside event intervals, clocks unlocked;
startup/JIT/reference work excluded. Backend timing includes
metadata/preprocessing/cache work, raw generation does not. TRT-LLM
still prepares compatibility bounds for other providers, so no
eliminated producer cost is claimed. Rubin H-D8 backend has a valid
+2.055% regression, narrowly above the predefined 2% cap; other Rubin
generation scopes and all Blackwell scopes pass. These are rank-local
component results, not CP communication, full-model throughput or TTFT.

The consumer also contains a separate native single-`is_lse` merge
cleanup and dispatch-policy update; those are not FlashInfer changes.

## 🔍 Related Issues

- Follow-up to merged #5489.
- Consumer integration:
NVIDIA/TensorRT-LLM#19395
- Existing experimental PrimTS tracking context: #4829, #5059, #5082.

## 🚀 Pull Request Checklist

Thank you for contributing to FlashInfer! Before we review your pull
request, please make sure the following items are complete.

### ✅ Pre-commit Checks

- [x] I have installed `pre-commit` by running `pip install pre-commit`
(or used your preferred method).
- [ ] I have installed the hooks with `pre-commit install`.
- [ ] I have run the hooks manually with `pre-commit run --all-files`
and fixed any reported issues.

> Changed-file pre-commit checks passed, including mypy and Ruff. The
existing checkout disables automatic hooks; hooks were run manually. A
full-repository hook run is not claimed.

## 🧪 Tests

- Cluster-limit regression: the existing device-local occupancy test
passed. Three B200 numerical checks covered the 17-CTA fallback on both
devices and a legal 16-CTA cluster; both previously failing TRT-LLM TP2
MTP tests passed.

- [x] Tests have been added or updated as needed.
- [ ] All tests are passing (`unittest`, etc.).
- [ ] This PR requires **full-scope** CI, not targeted tests (expensive;
justify below).
Reason: <!-- e.g. cross-cutting change to shared infrastructure -->

Suggested CI test targets (optional; paths under `tests/`,
whitespace-separated — a starting
point for `@flashinfer-bot run <targets>` / `/bot run <targets>`,
reviewers may adjust): `tests/attention/test_attention_ts_mla_cp.py
tests/attention/test_attention_ts_mla_decode.py
tests/trace/test_fi_trace_template_consistency.py`

Current-head local results:
- Focused CPU contract checks in `test_attention_ts_mla_decode.py`: **9
passed**.
- Trace consistency: **1,240 passed, 2 skipped**.
- CP GPU regression collection: **7 cases**, not executed again during
publication.
- Earlier matching-implementation GPU validation: seven CP kernel cases
plus strict two-GPU Helix/MTP replay on Rubin; retained TRT-LLM
focused/native/backend checks and paired generation measurements on
Rubin, B200 and B300. These historical results are not represented as a
fresh full-suite CI run.
- No FlashInfer CI bot command was issued; CI is left to the maintainer.

## 🔬 Experimental Track

<!-- Only for PRs submitted under the experimental policy
(CONTRIBUTING.md → "Experimental APIs and Backends").
     Leave this section untouched for normal PRs. -->

- [x] This PR updates existing **experimental** PrimTS APIs marked with
`@flashinfer_experimental_api`. Tracking context: #5489, #4829, #5059,
#5082; a separate graduation-tracking issue is not claimed.
- [ ] The tracking issue names an owner, the reason for the experimental
path, and a graduation plan with a target release.
- [ ] Core changes are limited to a thin entry point (signature, shared
validation, feature-gate check, backend selection, handoff).
- [ ] Tests live in `tests/experimental/` and were validated on the
intended hardware; a runnable example is included.
- [x] Nothing is registered in `flashinfer/aot.py`, and no experimental
backend is reachable from `backend="auto"` without
`FLASHINFER_ALLOW_EXPERIMENTAL_AUTO_BACKENDS=1`. (Calling an
`@flashinfer_experimental_api` or naming a backend explicitly is itself
the opt-in and needs no environment variable.)
- [x] **Test scope declared below.** The experimental CI lane runs
exactly these targets, so keep them as narrow as the change allows.

<!-- Required for experimental PRs. Replace the commented lines below
with your targets.
Do not delete the fence or change its `experimental-tests` tag — the
experimental-track
watcher reads it verbatim to decide which targets to ask CI for. -->

```experimental-tests
tests/experimental/test_experimental_api.py
```

As in #5489, this scope covers shared experimental opt-in behavior, not
kernel correctness. Existing PrimTS implementation/tests remain under
`flashinfer/attention/prims_ts/` and `tests/attention/`; the targeted
attention tests above cover the kernels. Relocation/graduation is
outside this follow-up.

## Reviewer Notes

Please focus on global-to-local prefix arithmetic, packed-query
alignment, split-reduction empty rows and graph-live metadata. This was
developed and validated with Codex assistance. No NVIDIA-specific
notices were added to FlashInfer files.



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Added context-parallel support for paged MLA decode, with configurable
rank and interleave settings. Global KV lengths are mapped to rank-local
cache prefixes.
* Empty rows on ranks with no locally visible KV tokens now return zero
output and negative-infinity LSE by default; this behavior can be
disabled.
* **Bug Fixes**
* Improved handling of supported cluster sizes when selecting attention
launch configurations.
* **Documentation**
* Updated MLA decode guidance and trace examples for context-parallel
settings, local cache capacity, and empty-row behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86239 [ run ] completed with state SUCCESS. Commit: f2300c7
/LLM/main/L0_MergeRequest_PR pipeline #71444 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

CI Report

Link to invocation

@yuxianq
yuxianq merged commit 464b458 into NVIDIA:main Nov 21, 2026
9 checks passed
yuxianq added a commit that referenced this pull request Nov 21, 2026
----- BEGIN VENDOR PROMOTION V1 -----
{
  "attribution": {
    "5129a714e7f3e1c45a7127425fe5173b1f1f9e87": {
      "method": "same-edits",
      "upstream_base": "a7d9e18658d3954527675a3d3a9e7b2ec75b2eae",
      "upstream_head": "f276b1561a484fecca055cee1a35e4d8901f992b"
    },
    "9cc9333bfe89eb4d5e9d9dd5e87e0bcb275e8b27": {
      "comparison": {
        "source_base": "31fe2dbd6b65cca758c1318f177b315be0c33778",
        "source_head": "9cc9333bfe89eb4d5e9d9dd5e87e0bcb275e8b27",
        "upstream_base": "1bdba872568a87391f64ffb91910e52aeff9479d",
        "upstream_head": "9ff50c4e324b56869670b70ea64dcf525bc94b85"
      },
      "method": "similar-upstream-aggregate",
      "refresh_policy": "retain-until-verified",
      "similarity": 0.9989658738366081,
      "similarity_metric": "sequence-matcher-lines-v1",
      "similarity_threshold": 0.9,
      "upstream_base": "1bdba872568a87391f64ffb91910e52aeff9479d",
      "upstream_head": "9ff50c4e324b56869670b70ea64dcf525bc94b85",
      "upstream_pr": 5489
    },
    "d76f94bb6ae3bc1730689a1a080a79913dcc3606": {
      "method": "same-edits",
      "upstream_base": "a7d9e18658d3954527675a3d3a9e7b2ec75b2eae",
      "upstream_head": "f276b1561a484fecca055cee1a35e4d8901f992b"
    }
  },
  "attribution_scope": {
    "include": [
      "**/*.py"
    ],
    "source": "flashinfer/attention/prims_ts"
  },
  "canonical_branch": "trtllm-prims-ts-dev-20261107",
  "canonical_url": "https://github.com/yuxianq/flashinfer.git",
  "changes": [
    {
      "commits": [
        "9cc9333bfe89eb4d5e9d9dd5e87e0bcb275e8b27"
      ],
      "upstream_base": "1bdba872568a87391f64ffb91910e52aeff9479d",
      "upstream_head": "9ff50c4e324b56869670b70ea64dcf525bc94b85",
      "upstream_merge": "1a9f361db34977801e4a676eb54b0379527479b9",
      "upstream_pr": "flashinfer-ai/flashinfer#5489"
    },
    {
      "commits": [
        "5129a714e7f3e1c45a7127425fe5173b1f1f9e87",
        "d76f94bb6ae3bc1730689a1a080a79913dcc3606"
      ],
      "upstream_base": "a7d9e18658d3954527675a3d3a9e7b2ec75b2eae",
      "upstream_head": "f276b1561a484fecca055cee1a35e4d8901f992b",
      "upstream_merge": "0cf9d70d8db1dbc74dc34a4d69c33b60602414ac",
      "upstream_pr": "flashinfer-ai/flashinfer#9641"
    }
  ],
  "consumer_base": "main",
  "consumer_merge": "464b458a34696ee42a3206cb5ba4d231f3ac0a0f",
  "consumer_pr": "#19395",
  "contributor_metadata": {
    "metadata_digest": "f6066e75990b21c7bc5fef79f79df588eabdc27341a03382b4c6d086a311c85a",
    "pr_author": "yuxianq",
    "source_head": "f2300c7b0a971b0d99e57fdc736d51e5ee47704a"
  },
  "kind": "promotion",
  "previous": "31fe2dbd6b65cca758c1318f177b315be0c33778",
  "promoted": "d76f94bb6ae3bc1730689a1a080a79913dcc3606",
  "source_branch": "prims-ts-fmha-lib--cp-dev-task-api-20261119",
  "source_url": "https://github.com/yuxianq/flashinfer.git",
  "upstream_base": "911d6a9d62308b6c8a5249e8dc4aabc7518020f1",
  "upstream_branch": "main",
  "upstream_observed_main": "ceb7af388d41ae3532f27de3750375ef7efc29aa",
  "upstream_repository": "flashinfer-ai/flashinfer",
  "vendor": "flashinfer-prims-ts",
  "version": 1
}
----- END VENDOR PROMOTION V1 -----

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants