Skip to content

[None][perf] Precompile PrimTS sparse MLA in decoder warmup and reuse decode views - #22274

Open
yuxianq wants to merge 22 commits into
NVIDIA:mainfrom
yuxianq:cute-dsl-precompile-api--trtllm
Open

yuxianq wants to merge 22 commits into
NVIDIA:mainfrom
yuxianq:cute-dsl-precompile-api--trtllm

Conversation

@yuxianq

@yuxianq yuxianq commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

Dev Engineer Review

  • Silent warmup fallback: run_precompile_workers now returns None instead of bool. In non-strict mode, worker errors log a warning and return. Warmup continues with uncompiled kernels, and the caller cannot detect this. The first forward may still compile.
  • Prepared-launch path: The vendor patch adds a bounded prepared-launch cache for eligible native BF16/E4M3 cases. run(validate=False) skips per-call validation. The caller must meet trusted-input conditions. The code does not check violations at runtime. The cache bound is not stated.
  • Vendor pin: 3rdparty/vendor_sources.lock.yaml tracks branch cute-dsl-precompile-api--prims-ts-vendor at commit be630d2bef243b7475ff352bfdc6974512ef57f9. The objectives name the same commit. The source work is not merged upstream, so promotion depends on the source PR merging.
  • Worker lifecycle: Each rank starts a fresh process pool. The pool exits before KV-cache sizing. Timeouts and parent exit kill the worker process group. Cleanup correctness depends on this process-group handling.
  • Cache checks: With parallel compilation enabled, each warmup pass checks disk-cache membership for each distinct kernel. Repeated-pass cost is not measured.
  • Scope: The diff changes 21 files (+1532/-1936). The supplied material describes 15 of them.
  • CI: Pipelines #71066, #71185, and #71286 reported failures. Failing tests are not identified. Pipeline #86382 was aborted. Current-head CI status is not available.
  • Timings: Full-model timings were measured on pre-rebase revisions. Post-rebase timings are not available.

QA Engineer Review

  • Test files changed:
    • test_prims_ts_sparse_mla_fmha.py (+396/-464): workspace-size cache reset, warmup profile selection, row-count coverage, largest-workspace-first ordering, cache-miss-only worker dispatch, strict and non-strict worker failures, timeout cleanup. Two prepared-launch tests were removed.
    • test_phase_selective_forward.py (+4): checks that the "prims_ts" event follows the FMHA prewarm event.
    • test_warmup.py (+58): CPU-only test of _prewarm_prims_ts_sparse_mla ordering, skip conditions, and barriers.
    • test_dspark_model_engine_runtime.py (+8/-5): passes max_total_draft_tokens=5 and checks metadata construction.
  • Named tests (author-reported): Commit c7b6964da9 added a process-group timeout-cleanup test and a warmup test for a distinct exact-row plan that reuses the compiled kernel. The author reports both passed on B200.
  • Test-db or QA list membership: Not established from supplied material.
  • Coverage gap: No supplied test covers the validate=False prepared-launch path.
  • Author-reported checks: B200 and B300 cold-cache compilation, JIT-disabled loading, worker cleanup, and full-model output and graph-replay checks. These are not independently verified.
  • Coverage verdict: needs follow-up. Add prepared-launch coverage, confirm test-list membership, and identify the failing pipeline tests.

Per-File QA Perspective

  • 3rdparty/vendor_patches/flashinfer-prims-ts.patch: Adds precompile_only, probe_cached, KernelBusy, and KernelNotCached. Verify that warm-up fails when JIT or the CuTe DSL cache is disabled.
  • 3rdparty/vendor_sources.lock.yaml: Points the entry to the new branch and commit. Verify the digests match the intended pin.
  • _prims_ts_worker.py (new): Standalone worker that imports PrimTS from a supplied sys.path. Verify process-group cleanup when the parent exits.
  • prims_ts_precompile.py: Starts workers only for two or more misses. Verify timeout cleanup and non-strict warning behavior.
  • prims_ts_sparse_mla.py: Adds warmup_prims_ts_sparse_mla and removes the per-forward variant scan. Verify warmup duration and plan loading in the serving process.
  • prims_ts_dsa_sparse_mla.py: Marks a profile warmed only when variants exist. Verify that empty candidate sets leave profiles unwarmed.
  • sparse/deepseek_v4/module.py: Moves epilogue-fusion checks into supports_dsv4_epilogue_fusion. Verify fusion selection is unchanged.
  • pyexecutor/engine/runners/decoder/warmup.py: Adds a PrimTS prewarm phase with TP or TP×CP synchronization. Verify it skips context-only and SWA-limited caches.
  • _cute_dsl_compat.py (new): Adds JitSpecCuteDsl and build_and_load_cute_dsl_kernel. Verify fallback to the in-memory kernel when disk persistence fails.
  • pyexecutor/engine/metadata.py and runners/decoder/metadata.py: Pass max_total_draft_tokens only to TrtllmAttentionMetadata subclasses. Verify other backends are unaffected.
  • Test files: See QA Engineer Review.

Description

Move DeepSeek-V4 PrimTS sparse MLA compilation from the first eligible forward into decoder warmup. Each warmup pass discovers reachable specializations with plan(compile=False) and checks each distinct kernel with is_kernel_cached() when parallel compilation is enabled.

  • Two or more missing variants go to a fresh per-rank process pool. The serving rank handles a single miss locally. A complete disk cache starts zero workers.
  • The standalone worker imports the vendored kernels under their canonical module names, so the serving rank can reload their exports. The pool exits before KV-cache sizing during the profiling pass.
  • The serving rank prepares its own plans and shared workspace for each pass under inference mode. Forward uses the public run(validate=False) path with independent input-view caches.
  • The immutable FlashInfer vendor pin is c4a601c01341c0a2b6f5623e9f64adb917c27be8 on the fork branch cute-dsl-precompile-api--prims-ts-vendor-20261121. It includes the current TensorRT-LLM vendor baseline d76f94bb6 and the reviewed PrimTS changes. Upstream tracks compile caching and deferred planning in FlashInfer #9395 and sparse MLA run-time view reuse in FlashInfer #9767. With the pinned FlashInfer 0.7.3 runtime, TensorRT-LLM supplies the per-kernel JIT locking in its local compatibility module. FlashInfer cache PR · Original TRT-LLM implementation

The vendor topic is based on the canonical source promoted by TensorRT-LLM #22723 after #19395 merged. All nine commits that touch vendored source match the declared upstream PRs by normalized edits. The local cache adapter can be removed once the pinned FlashInfer release includes #9395's cache API and locking. #22659 also updates this pin. Whichever integration lands second should replay its source changes on the promoted tip and retain deferred planning, membership probing, and the dense CP launch ABI.

Sparse warmup prepares all eligible local PrimTS profiles whenever the library is enabled, regardless of its position in the FMHA list. Dispatch uses the actual forward inputs. With JIT disabled, prewarm loads available artifacts and defers each missing variant to a forward that selects it. Eager generation derives its kernel profile through the same library method. The supervisor reports worker output on failure and terminates surviving descendants on both timeout and abnormal exit. Positive infinity allows an unlimited worker wait. DSA gets its top-k from runtime indices and attempts its optional scan on the first eligible eager forward. In non-strict mode a failed representative leaves subsequent batches free to plan their own rows.

The configured draft limit is applied before the first forward. Workers check that their artifact was published and the supervisor terminates its process group if the rank dies. On B300, two cold workers published both variants in 48.14 s. A separate JIT-disabled process loaded both in 0.49 s with zero workers. In a separate rank-kill check, the supervisor and pool descendants stopped within 0.70 s.

schema_version: 1
vendors:
  flashinfer-prims-ts:
    upstream_prs:
      - https://github.com/flashinfer-ai/flashinfer/pull/9395
      - https://github.com/flashinfer-ai/flashinfer/pull/9767
      - https://github.com/flashinfer-ai/flashinfer/pull/9087
      - https://github.com/flashinfer-ai/flashinfer/pull/9446

Downstream performance

DeepSeek-V4-Flash ran on four B200s with TP4/EP4, attention data parallelism, and 256 input and output tokens. Warmup times below are rank-0 wall time. The measurements predate the rebase. Each row group compares revisions from the same allocation.

PrimTS cache and comparison Mode First warmup First PrimTS phase
Cold, main-base c9c07e04 → standalone pool e9e140d7 Eager 418.2 → 195.1 s Included in attention warmup on main
Warm, main-base c9c07e04 → cache check 715858bac0 Eager 42.50 → 42.95 s 5.10 s with the cache check
Same warm comparison CUDA graph 48.00 → 47.30 s 5.15 s with the cache check

On a cold PrimTS cache, grouping all 12 specializations in one pool saved 223.1 s in the first warmup (one process per version). In the warm comparison, the earlier standalone pool without the cache check took 20.2 s eager / 20.0 s graph in its first PrimTS phase on the same node (one reference process per mode). Querying disk membership saved about 15 s in that phase. Main and the cache-check version each ran in two fresh processes per mode.

The final implementation checks the cache again when the executor binds its serving resources. We isolated this change on a separate exclusive four-B200 node with the same FlashInfer code, comparing 715858bac0 against its flag-removal patch, subsequently committed as 3fcfb3f572.

Final pass, warm disk cache First-pass-only policy Check every pass (this PR)
Eager PrimTS phase / total final warmup 0.85 / 5.65 s 3.15 / 6.45 s
CUDA-graph PrimTS phase / total final warmup 2.60 / 16.35 s 3.55 / 16.65 s

Both versions launched zero workers with all 12 exports available and four workers enabled per rank. The additional 12 checks took 1.35 s in a separate two-pass probe. The full PrimTS phase also includes local planning and rank synchronization. If two new variants first appear on the final pass, the new policy can compile them in parallel.

Steady-state time per output token
Execution Main-base → cache check, ms/token First-pass-only → every pass, ms/token
Eager C8 116.728 → 117.878 113.246 → 113.072
Eager C128 121.362 → 121.814 117.265 → 117.218
CUDA graph C8 7.755 → 7.842 7.346 → 7.345
CUDA graph C128 11.913 → 12.454 11.685 → 11.697

Columns are separate experiments. Each used before/after/after/before process order, two warmup batches and three measured batches per concurrency in each process. In the main-base comparison, eager paired differences changed sign with run order. Graph C128 averaged 4.5% slower and included one 14.33 ms/token candidate batch. In the isolated flag comparison the largest absolute difference was 0.174 ms/token. Outputs, graph replay and KV capacity matched within each mode. Weight-file page cache and weight-loading times varied across fresh processes.

Sparse MLA wrapper microbenchmark

The earlier public run() comparison used TRT-LLM main-base c9c07e04 and feature revision 9d8394d2 with FlashInfer 81f96025f. It included the warmed prepared dispatch on main. The benchmark used BF16, 64 heads, fixed KV pools, and one process per source.

Local rows Extra Persistent inputs Replaced inputs, prior tensors alive One-use inputs, prior tensors released
2 0 16.120 → 13.152 16.314 → 17.212 16.513 → 15.278
2 32 15.960 → 14.125 16.451 → 18.342 16.853 → 16.659
2 512 18.112 → 16.101 18.532 → 20.605 18.939 → 18.621
32 0 16.314 → 13.243 16.772 → 17.667 16.913 → 15.559
32 32 16.152 → 14.117 17.408 → 19.555 17.265 → 17.029
32 512 18.071 → 15.999 18.888 → 20.628 19.280 → 18.711

Values are host microseconds per call. Persistent inputs improved 11.1–18.8% and one-use inputs improved 1.1–8.0%. Replacing inputs while keeping the prior tensors alive cost 0.90–2.15 µs more per call (5.3–12.3%). Output/LSE hashes and native arguments matched in all 18 cases. The fixture used 768 physical cache rows and fixed per-profile pools.

In a separate B200 wrapper experiment on FlashInfer #9767 (5880ee354), bypassing only the query, index and output view caches saved 0.956 µs per host call on average with ephemeral adapter inputs. It cost 0.082–0.356 µs per call with persistent inputs across six profiles. Two fresh processes exercised both orders. Numerical results and all 17 launch arguments matched. We retained the general cache. These timings isolate the wrapper and exclude model-level TPOT.

Test Coverage

On 002a9767e9, four targeted B200 regression cases and two cold-cache dispatch checks passed. With JIT disabled and an empty PrimTS cache, prewarm completed in both orders. The fallback-first forward matched its reference, and the PrimTS-first forward raised MissingJITCacheError for the requested kernel. A mixed-cache case verified that warmup continued loading available variants after a miss.

On 29257e701d after rebasing onto 4af2d530, all 64 targeted B200 tests passed with zero skips. The run used stock FlashInfer 0.7.3, CuTe DSL 4.8.0 and a TensorRT-LLM CI wheel built from the same C++ tree. Coverage includes the vendored-import regression, all ten sampler cases that failed in Jenkins #71596, four public trusted-launch ABI configurations, dense MLA graph replay, sparse warmup followed by zero new kernel compilations, worker cleanup, TP/CP failure propagation, and DSA exact-row fallback.

Two additional single-B200 Helix tests passed against the vendored provider. The page-boundary case exercised logical CP ranks 0, 1 and 2 and compared merged output and LSE with the reference. The CP8 case exercised nonzero-rank CUDA graph replay. This validates the launch ABI on one device. A third additional test passed for packed sparse partial-head BF16 graph replay. Total on the rebased revision: 67 passed, zero skipped.

The vendor tree verifies against its published SHA. The cumulative pre-commit checks passed. The advisory type checker reported the existing gen_offset warning in the unchanged FP8 path.

  • Fresh B200 processes verified full hits, two misses, one miss, stale source metadata and fresh reload with JIT disabled. Queries did no building or loading. The serving process loaded all 12 variants, and only missing variants went to workers.
  • The final eight full-model processes passed output, graph replay, zero-worker and capacity checks. Separate two-pass tests exercised newly encountered final-pass variants and preserved native arguments and numerical results.
  • On vendor revision d346fb95e1, the packed dense launch-bounds and sparse caller-workspace/cache-membership tests both passed on B200. Six B200 cases passed on the separate FlashInfer PR tree, including the latest upstream KV-bounds ABI.
  • After rebasing onto e466dcb2, both affected Qwen3.8-Flash-Next FP8 test files collected on B300 (120 cases). Four snapshot and cached-replay cases passed, including graph replay.
  • Post-rebase checks include the vendor lock against the published source, changed-file hooks, and comparison with the three-way merge of the benchmarked patch. The full-model timings above were taken on the listed pre-rebase revisions.

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

@yuxianq

yuxianq commented Nov 14, 2026

Copy link
Copy Markdown
Collaborator Author

Vendor validation: flashinfer-prims-ts

@yuxianq — Vendor promotion needs maintainer action:

The base lock now uses refresh branch trtllm-prims-ts-dev-20261107, but this monitor still publishes to trtllm-prims-ts-dev-20261012.

Fix: A maintainer must stop this monitor and migrate its workdir with --canonical-branch trtllm-prims-ts-dev-20261107 --migrate-canonical-branch-from trtllm-prims-ts-dev-20261012. Verify a dry run before resuming publication. Do not promote back to the old branch or change contributor metadata to bypass this cutover.

The bot will recheck automatically on a later poll.

Checked PR revision: 8309df42328a. This is a snapshot; newer results are posted as new comments. Observed dev/upstream tip SHAs may be stale: tip movement alone does not post a new comment when the result is unchanged. Fetch the live branch before rebasing or choosing a new pin.

@yuxianq

yuxianq commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

Vendor validation: flashinfer-prims-ts

@yuxianq — Vendor promotion needs author action:

Upstream attribution is unresolved for these source commits:

Fix: In the PR description, add missing PRs to vendors.flashinfer-prims-ts.upstream_prs, remove incorrectly listed PRs, or correct/rebase the source pin and update the vendored snapshot. For an intentional adaptation or ambiguous ownership, merge a targeted resolutions entry into the existing vendor metadata as shown below. Replace the SHA and URL and give a specific reason; preserve other entries. No complete commit_map is required. A resolution records an author assertion, not verified equivalence; it is retained during refresh until separately verified.

vendors:
  flashinfer-prims-ts:
    upstream_prs:
      - https://github.com/flashinfer-ai/flashinfer/pull/123
    resolutions:
      FULL_40_CHARACTER_SOURCE_SHA:
        upstream_pr: https://github.com/flashinfer-ai/flashinfer/pull/123
        reason: Explain the downstream adaptation or why this PR owns the change.

The bot will recheck automatically on a later poll.

Checked PR revision: 17ec695e32a6. This is a snapshot; newer results are posted as new comments. Observed dev/upstream tip SHAs may be stale: tip movement alone does not post a new comment when the result is unchanged. Fetch the live branch before rebasing or choosing a new pin.

Checked source: https://github.com/yuxianq/flashinfer.git / cute-dsl-precompile-api--prims-ts-upstream, pin 62372666f04cad0156b6c30321597c287d49512a.

Supersedes previous vendor validation.

@yuxianq

yuxianq commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

Vendor validation: flashinfer-prims-ts

@yuxianq — Vendor promotion needs author action:

Upstream attribution is unresolved for these source commits:

Fix: In the PR description, add missing PRs to vendors.flashinfer-prims-ts.upstream_prs, remove incorrectly listed PRs, or correct/rebase the source pin and update the vendored snapshot. For an intentional adaptation or ambiguous ownership, merge a targeted resolutions entry into the existing vendor metadata as shown below. Replace the SHA and URL and give a specific reason; preserve other entries. No complete commit_map is required. A resolution records an author assertion, not verified equivalence; it is retained during refresh until separately verified.

vendors:
  flashinfer-prims-ts:
    upstream_prs:
      - https://github.com/flashinfer-ai/flashinfer/pull/123
    resolutions:
      FULL_40_CHARACTER_SOURCE_SHA:
        upstream_pr: https://github.com/flashinfer-ai/flashinfer/pull/123
        reason: Explain the downstream adaptation or why this PR owns the change.

The bot will recheck automatically on a later poll.

Checked PR revision: 9d8394d2e673. This is a snapshot; newer results are posted as new comments. Observed dev/upstream tip SHAs may be stale: tip movement alone does not post a new comment when the result is unchanged. Fetch the live branch before rebasing or choosing a new pin.

Checked source: https://github.com/yuxianq/flashinfer.git / cute-dsl-precompile-api--prims-ts-upstream, pin 81f96025f4659da1c2ce42a9fa51d4d282453963.

Supersedes previous vendor validation.

@yuxianq

yuxianq commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Vendor validation: flashinfer-prims-ts

@yuxianq — Vendor promotion needs maintainer action:

Waiting for the preceding vendor promotion: the base lock still points to repository https://github.com/yuxianq/flashinfer.git, branch flashinfer-9222-prims-ts--source.

Fix: A maintainer must finish the preceding lock-only promotion back to yuxianq/flashinfer / trtllm-prims-ts-dev-20261107. After it merges, the author should rebase this consumer PR onto the latest base branch and rebase its temporary source branch onto the new canonical pin, then update the lock/vendor snapshot. No metadata edit can unblock the preceding promotion.

The bot will recheck automatically on a later poll.

Checked PR revision: 9d8394d2e673. This is a snapshot; newer results are posted as new comments. Observed dev/upstream tip SHAs may be stale: tip movement alone does not post a new comment when the result is unchanged. Fetch the live branch before rebasing or choosing a new pin.

Supersedes previous vendor validation.

@yuxianq

yuxianq commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Vendor validation: flashinfer-prims-ts

@yuxianq — Vendor promotion needs author action:

Source update is not a fast-forward.

Fix: Fetch canonical repository https://github.com/yuxianq/flashinfer.git, branch trtllm-prims-ts-dev-20261107, and rebase the temporary source branch cute-dsl-precompile-api--prims-ts-upstream onto its latest tip without merge commits, not just the consumer lock's pin 2ffd3f8b2c2d163237c95bdf40e7a065a99cd544. Then use scripts/vendor/manage.py to update 3rdparty/vendor_sources.lock.yaml and the vendored snapshot to the rebased SHA, and push the source branch and consumer PR. Editing metadata alone will not fix this.

The bot will recheck automatically on a later poll.

Checked PR revision: 9d8394d2e673. This is a snapshot; newer results are posted as new comments. Observed dev/upstream tip SHAs may be stale: tip movement alone does not post a new comment when the result is unchanged. Fetch the live branch before rebasing or choosing a new pin.

Checked source: https://github.com/yuxianq/flashinfer.git / cute-dsl-precompile-api--prims-ts-upstream, pin 81f96025f4659da1c2ce42a9fa51d4d282453963.

Supersedes previous vendor validation.

@yuxianq
yuxianq force-pushed the cute-dsl-precompile-api--trtllm branch from 9d8394d to 562d15e Compare November 18, 2026 19:09
@yuxianq yuxianq changed the title [None][refactor] Move PrimTS sparse MLA precompilation into warmup [None][perf] Precompile missing PrimTS sparse MLA kernels during decoder warmup Nov 18, 2026
@yuxianq

yuxianq commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@yuxianq

yuxianq commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator Author

Vendor validation: flashinfer-prims-ts

@yuxianq — Vendor promotion needs author action:

Upstream attribution is unresolved for these source commits:

No exact match was found for the following commits; similarity matching accepted them:

  • d346fb95e145a37cc5f3cc24711ee8e81b147d33: upstream PR #9395, similarity 97.09% — accepted (threshold 90.00%; similar-edits).

Scores compare normalized patch lines in lock-selected files, not semantic equivalence. These changes retain retain-until-verified refresh policy.

Fix: In the PR description, add missing PRs to vendors.flashinfer-prims-ts.upstream_prs, remove incorrectly listed PRs, or correct/rebase the source pin and update the vendored snapshot. For an intentional adaptation or ambiguous ownership, merge a targeted resolutions entry into the existing vendor metadata as shown below. Replace the SHA and URL and give a specific reason; preserve other entries. No complete commit_map is required. A resolution records an author assertion, not verified equivalence; it is retained during refresh until separately verified.

vendors:
  flashinfer-prims-ts:
    upstream_prs:
      - https://github.com/flashinfer-ai/flashinfer/pull/123
    resolutions:
      FULL_40_CHARACTER_SOURCE_SHA:
        upstream_pr: https://github.com/flashinfer-ai/flashinfer/pull/123
        reason: Explain the downstream adaptation or why this PR owns the change.

The bot will recheck automatically on a later poll.

Checked PR revision: 562d15e7aa0f. This is a snapshot; newer results are posted as new comments. Observed dev/upstream tip SHAs may be stale: tip movement alone does not post a new comment when the result is unchanged. Fetch the live branch before rebasing or choosing a new pin.

Checked source: https://github.com/yuxianq/flashinfer.git / cute-dsl-precompile-api--prims-ts-vendor, pin d346fb95e145a37cc5f3cc24711ee8e81b147d33.

Supersedes previous vendor validation.

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #85822 [ run ] triggered by Bot. Commit: 562d15e Link to invocation

@yuxianq

yuxianq commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator Author

Vendor validation: flashinfer-prims-ts

@yuxianq — Vendor promotion metadata and source attribution are valid. This is not a code-review approval. Promotion still requires the source PR to merge.

No exact match was found for the following commits; similarity matching accepted them:

  • d346fb95e145a37cc5f3cc24711ee8e81b147d33: upstream PR #9395, similarity 97.09% — accepted (threshold 90.00%; similar-edits).

Scores compare normalized patch lines in lock-selected files, not semantic equivalence. These changes retain retain-until-verified refresh policy.

Checked PR revision: 562d15e7aa0f. This is a snapshot; newer results are posted as new comments. Observed dev/upstream tip SHAs may be stale: tip movement alone does not post a new comment when the result is unchanged. Fetch the live branch before rebasing or choosing a new pin.

Checked source: https://github.com/yuxianq/flashinfer.git / cute-dsl-precompile-api--prims-ts-vendor, pin d346fb95e145a37cc5f3cc24711ee8e81b147d33.

Supersedes previous vendor validation.

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #85822 [ run ] completed with state SUCCESS. Commit: 562d15e
/LLM/main/L0_MergeRequest_PR pipeline #71066 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@yuxianq

yuxianq commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator Author

Vendor validation: flashinfer-prims-ts

@yuxianq — Vendor promotion needs maintainer action:

Waiting for the preceding vendor promotion: the base lock still points to repository https://github.com/yuxianq/flashinfer.git, branch nvbug-6959297-cutlass-dsl-compat--flashinfer.

Fix: A maintainer must finish the preceding lock-only promotion back to yuxianq/flashinfer / trtllm-prims-ts-dev-20261107. After it merges, the author should rebase this consumer PR onto the latest base branch and rebase its temporary source branch onto the new canonical pin, then update the lock/vendor snapshot. No metadata edit can unblock the preceding promotion.

The bot will recheck automatically on a later poll.

Checked PR revision: 562d15e7aa0f. This is a snapshot; newer results are posted as new comments. Observed dev/upstream tip SHAs may be stale: tip movement alone does not post a new comment when the result is unchanged. Fetch the live branch before rebasing or choosing a new pin.

Supersedes previous vendor validation.

@yuxianq

yuxianq commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator Author

Vendor validation: flashinfer-prims-ts

@yuxianq — Vendor promotion needs author action:

Source update is not a fast-forward.

Fix: Fetch canonical repository https://github.com/yuxianq/flashinfer.git, branch trtllm-prims-ts-dev-20261107, and rebase the temporary source branch cute-dsl-precompile-api--prims-ts-vendor onto its latest tip without merge commits, not just the consumer lock's pin 31fe2dbd6b65cca758c1318f177b315be0c33778. Then use scripts/vendor/manage.py to update 3rdparty/vendor_sources.lock.yaml and the vendored snapshot to the rebased SHA, and push the source branch and consumer PR. Editing metadata alone will not fix this.

The bot will recheck automatically on a later poll.

Checked PR revision: 562d15e7aa0f. This is a snapshot; newer results are posted as new comments. Observed dev/upstream tip SHAs may be stale: tip movement alone does not post a new comment when the result is unchanged. Fetch the live branch before rebasing or choosing a new pin.

Checked source: https://github.com/yuxianq/flashinfer.git / cute-dsl-precompile-api--prims-ts-vendor, pin d346fb95e145a37cc5f3cc24711ee8e81b147d33.

Supersedes previous vendor validation.

@yuxianq
yuxianq force-pushed the cute-dsl-precompile-api--trtllm branch from 562d15e to 632b0b9 Compare November 19, 2026 03:25
@yuxianq

yuxianq commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator Author

Vendor validation: flashinfer-prims-ts

@yuxianq — Vendor promotion metadata and source attribution are valid. This is not a code-review approval. Promotion still requires the source PR to merge.

No exact match was found for the following commits; similarity matching accepted them:

  • b116138c9c1da75f5520cd8b4b5a7ebe1ca4551b: upstream PR #9395, similarity 97.09% — accepted (threshold 90.00%; similar-edits).

Scores compare normalized patch lines in lock-selected files, not semantic equivalence. These changes retain retain-until-verified refresh policy.

Checked PR revision: 632b0b95239a. This is a snapshot; newer results are posted as new comments. Observed dev/upstream tip SHAs may be stale: tip movement alone does not post a new comment when the result is unchanged. Fetch the live branch before rebasing or choosing a new pin.

Checked source: https://github.com/yuxianq/flashinfer.git / cute-dsl-precompile-api--prims-ts-vendor, pin b116138c9c1da75f5520cd8b4b5a7ebe1ca4551b.

Supersedes previous vendor validation.

@yuxianq

yuxianq commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #85959 [ run ] triggered by Bot. Commit: 632b0b9 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #85959 [ run ] completed with state SUCCESS. Commit: 632b0b9
/LLM/main/L0_MergeRequest_PR pipeline #71185 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@yuxianq

yuxianq commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86073 [ run ] triggered by Bot. Commit: 2bda8f1 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86073 [ run ] completed with state SUCCESS. Commit: 2bda8f1
/LLM/main/L0_MergeRequest_PR pipeline #71286 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@yuxianq
yuxianq marked this pull request as ready for review November 20, 2026 01:09
@yuxianq
yuxianq requested review from a team as code owners November 20, 2026 01:09
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
…er runtime

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
…fixes

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
…contracts

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
…rmup

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
@yuxianq
yuxianq force-pushed the cute-dsl-precompile-api--trtllm branch from ef1c7d7 to 29257e7 Compare November 21, 2026 02:12
@yuxianq
yuxianq requested review from a team as code owners November 21, 2026 02:12
@yuxianq yuxianq changed the title [None][perf] Precompile missing PrimTS sparse MLA kernels during decoder warmup [None][perf] Precompile PrimTS sparse MLA in decoder warmup and reuse decode views Nov 21, 2026
@yuxianq

yuxianq commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@yuxianq

yuxianq commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator Author

I rebased after #22723 merged. The vendor topic starts at the promoted source for #19395 and preserves its dense and Helix ABI, including the runtime cp_rank argument. The ported host test checks all 17 positional arguments on three trusted public launches for both pool and workspace configurations. The dense phase remains after profile discovery. The sparse phase needs eager metadata before the first forward. #22659 also updates the source pin. The later integration should replay its changes on the promoted tip.

The worker and cache-adapter changes retain FlashInfer 0.7.3 compatibility. I added the adapter to the attention and OSS compliance owners and documented its removal condition in the PR description. The release notes now give the measured four-B200 cold first-warmup result with its configuration. The developer guide remains outside the change as requested during review. The PR title and description now identify both sparse warmup and view reuse, with separate wrapper measurements.

The upstream view-cache experiment saved 0.956 µs per host call for ephemeral inputs but cost 0.082–0.356 µs for persistent inputs. I kept the general cache. No end-to-end speedup is inferred from this comparison. The linked inline replies cover the worker timeout, pipeline ownership, error propagation and tests.

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86493 [ run ] triggered by Bot. Commit: 29257e7 Link to invocation

@yuxianq

yuxianq commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator Author

Vendor validation: flashinfer-prims-ts

@yuxianq — Vendor promotion metadata and source attribution are valid. This is not a code-review approval. Promotion still requires the source PR to merge.

Checked PR revision: 29257e701d2d. This is a snapshot; newer results are posted as new comments. Observed dev/upstream tip SHAs may be stale: tip movement alone does not post a new comment when the result is unchanged. Fetch the live branch before rebasing or choosing a new pin.

Checked source: https://github.com/yuxianq/flashinfer.git / cute-dsl-precompile-api--prims-ts-vendor-20261121, pin c4a601c01341c0a2b6f5623e9f64adb917c27be8.

Supersedes previous vendor validation.

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
@yuxianq

yuxianq commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86497 [ run ] triggered by Bot. Commit: e2945a7 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86493 [ run ] completed with state ABORTED. Commit: 29257e7

Link to invocation

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
@yuxianq

yuxianq commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86508 [ run ] triggered by Bot. Commit: 002a976 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86497 [ run ] completed with state ABORTED. Commit: e2945a7

Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86508 [ run ] completed with state SUCCESS. Commit: 002a976
/LLM/main/L0_MergeRequest_PR pipeline #71695 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

CI Report

Link to invocation

@trtllm-agent-review trtllm-agent-review left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up check: the Blockers of the earlier review are resolved.

  • [Blocker] (B2: Crash, hang, race or leak) fixed: In prims_ts_sparse_mla.py:929-938, warmup now skips any MLA layer with no parameters (line 933; remove_weights clears them on other-stage layers) and calls warmup_profile only for a PrimsTSSparseMlaFmha library, so the o_a_proj read at :442 runs only on local layers. The new test at test_prims_ts_sparse_mla_fmha.py:1147-1153 strips a real MLA layer with remove_weights and asserts that it yields no profile; the file runs in B200 L0 (l0_b200.yml:153). (prims_ts_sparse_mla.py:890)
  • [Blocker] (B2: Crash, hang, race or leak) fixed: The rank now sends its PID to the supervisor (prims_ts_precompile.py:166). In _prims_ts_worker.py:68-80 a watchdog thread kills the supervisor's whole process group with SIGKILL once getppid() changes, and on timeout or abnormal supervisor exit the rank cleans up the group with SIGTERM then SIGKILL (prims_ts_precompile.py:125-138, :184-194). A local check that SIGKILLed a fake rank while two real spawn workers were blocked left no process in the group after about 0.9 s; only a few 32-byte spawn semaphores remained. (prims_ts_precompile.py:108)
  • [Blocker] (B5: User-visible switch behavior) fixed: communicate(timeout=...) is gone: _worker_timeout (prims_ts_precompile.py:102-116) maps inf to None and turns invalid or non-positive values into a warning plus the 1800 s default. _wait_for_pool (:125-138) polls with sleep(min(0.05, remaining)) or blocks in os.waitid, so no OverflowError can occur. The bounded-wait test covers inf and 2147484 (test_prims_ts_sparse_mla_fmha.py:1629-1630, :1662). An isolated run against real child processes returned normally for inf, 2147484 and 1e300. (prims_ts_precompile.py:100)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants