Repository navigation
Conversation
Vendor validation:
|
Vendor validation:
|
Vendor validation:
|
Vendor validation:
|
Vendor validation:
|
9d8394d to
562d15e
Compare
|
/bot run --disable-fail-fast |
Vendor validation:
|
|
PR_Github #85822 [ run ] triggered by Bot. Commit: |
Vendor validation:
|
|
PR_Github #85822 [ run ] completed with state
|
Vendor validation:
|
Vendor validation:
|
562d15e to
632b0b9
Compare
Vendor validation:
|
|
/bot run --disable-fail-fast |
|
PR_Github #85959 [ run ] triggered by Bot. Commit: |
|
PR_Github #85959 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #86073 [ run ] triggered by Bot. Commit: |
|
PR_Github #86073 [ run ] completed with state
|
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
…er runtime Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
…fixes Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
…contracts Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
…rmup Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
ef1c7d7 to
29257e7
Compare
|
/bot run --disable-fail-fast |
|
I rebased after #22723 merged. The vendor topic starts at the promoted source for #19395 and preserves its dense and Helix ABI, including the runtime cp_rank argument. The ported host test checks all 17 positional arguments on three trusted public launches for both pool and workspace configurations. The dense phase remains after profile discovery. The sparse phase needs eager metadata before the first forward. #22659 also updates the source pin. The later integration should replay its changes on the promoted tip. The worker and cache-adapter changes retain FlashInfer 0.7.3 compatibility. I added the adapter to the attention and OSS compliance owners and documented its removal condition in the PR description. The release notes now give the measured four-B200 cold first-warmup result with its configuration. The developer guide remains outside the change as requested during review. The PR title and description now identify both sparse warmup and view reuse, with separate wrapper measurements. The upstream view-cache experiment saved 0.956 µs per host call for ephemeral inputs but cost 0.082–0.356 µs for persistent inputs. I kept the general cache. No end-to-end speedup is inferred from this comparison. The linked inline replies cover the worker timeout, pipeline ownership, error propagation and tests. |
|
PR_Github #86493 [ run ] triggered by Bot. Commit: |
Vendor validation:
|
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
|
/bot run --disable-fail-fast |
|
PR_Github #86497 [ run ] triggered by Bot. Commit: |
|
PR_Github #86493 [ run ] completed with state |
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
|
/bot run --disable-fail-fast |
|
PR_Github #86508 [ run ] triggered by Bot. Commit: |
|
PR_Github #86497 [ run ] completed with state |
|
PR_Github #86508 [ run ] completed with state |
trtllm-agent-review
left a comment
There was a problem hiding this comment.
Follow-up check: the Blockers of the earlier review are resolved.
- [Blocker] (B2: Crash, hang, race or leak) fixed: In prims_ts_sparse_mla.py:929-938, warmup now skips any MLA layer with no parameters (line 933; remove_weights clears them on other-stage layers) and calls warmup_profile only for a PrimsTSSparseMlaFmha library, so the o_a_proj read at :442 runs only on local layers. The new test at test_prims_ts_sparse_mla_fmha.py:1147-1153 strips a real MLA layer with remove_weights and asserts that it yields no profile; the file runs in B200 L0 (l0_b200.yml:153). (
prims_ts_sparse_mla.py:890) - [Blocker] (B2: Crash, hang, race or leak) fixed: The rank now sends its PID to the supervisor (prims_ts_precompile.py:166). In _prims_ts_worker.py:68-80 a watchdog thread kills the supervisor's whole process group with SIGKILL once getppid() changes, and on timeout or abnormal supervisor exit the rank cleans up the group with SIGTERM then SIGKILL (prims_ts_precompile.py:125-138, :184-194). A local check that SIGKILLed a fake rank while two real spawn workers were blocked left no process in the group after about 0.9 s; only a few 32-byte spawn semaphores remained. (
prims_ts_precompile.py:108) - [Blocker] (B5: User-visible switch behavior) fixed: communicate(timeout=...) is gone: _worker_timeout (prims_ts_precompile.py:102-116) maps inf to None and turns invalid or non-positive values into a warning plus the 1800 s default. _wait_for_pool (:125-138) polls with sleep(min(0.05, remaining)) or blocks in os.waitid, so no OverflowError can occur. The bounded-wait test covers inf and 2147484 (test_prims_ts_sparse_mla_fmha.py:1629-1630, :1662). An isolated run against real child processes returned normally for inf, 2147484 and 1e300. (
prims_ts_precompile.py:100)
Dev Engineer Review
run_precompile_workersnow returnsNoneinstead ofbool. In non-strict mode, worker errors log a warning and return. Warmup continues with uncompiled kernels, and the caller cannot detect this. The first forward may still compile.run(validate=False)skips per-call validation. The caller must meet trusted-input conditions. The code does not check violations at runtime. The cache bound is not stated.3rdparty/vendor_sources.lock.yamltracks branchcute-dsl-precompile-api--prims-ts-vendorat commitbe630d2bef243b7475ff352bfdc6974512ef57f9. The objectives name the same commit. The source work is not merged upstream, so promotion depends on the source PR merging.#71066,#71185, and#71286reported failures. Failing tests are not identified. Pipeline#86382was aborted. Current-head CI status is not available.QA Engineer Review
test_prims_ts_sparse_mla_fmha.py(+396/-464): workspace-size cache reset, warmup profile selection, row-count coverage, largest-workspace-first ordering, cache-miss-only worker dispatch, strict and non-strict worker failures, timeout cleanup. Two prepared-launch tests were removed.test_phase_selective_forward.py(+4): checks that the"prims_ts"event follows the FMHA prewarm event.test_warmup.py(+58): CPU-only test of_prewarm_prims_ts_sparse_mlaordering, skip conditions, and barriers.test_dspark_model_engine_runtime.py(+8/-5): passesmax_total_draft_tokens=5and checks metadata construction.c7b6964da9added a process-group timeout-cleanup test and a warmup test for a distinct exact-row plan that reuses the compiled kernel. The author reports both passed on B200.validate=Falseprepared-launch path.Per-File QA Perspective
3rdparty/vendor_patches/flashinfer-prims-ts.patch: Addsprecompile_only,probe_cached,KernelBusy, andKernelNotCached. Verify that warm-up fails when JIT or the CuTe DSL cache is disabled.3rdparty/vendor_sources.lock.yaml: Points the entry to the new branch and commit. Verify the digests match the intended pin._prims_ts_worker.py(new): Standalone worker that imports PrimTS from a suppliedsys.path. Verify process-group cleanup when the parent exits.prims_ts_precompile.py: Starts workers only for two or more misses. Verify timeout cleanup and non-strict warning behavior.prims_ts_sparse_mla.py: Addswarmup_prims_ts_sparse_mlaand removes the per-forward variant scan. Verify warmup duration and plan loading in the serving process.prims_ts_dsa_sparse_mla.py: Marks a profile warmed only when variants exist. Verify that empty candidate sets leave profiles unwarmed.sparse/deepseek_v4/module.py: Moves epilogue-fusion checks intosupports_dsv4_epilogue_fusion. Verify fusion selection is unchanged.pyexecutor/engine/runners/decoder/warmup.py: Adds a PrimTS prewarm phase with TP or TP×CP synchronization. Verify it skips context-only and SWA-limited caches._cute_dsl_compat.py(new): AddsJitSpecCuteDslandbuild_and_load_cute_dsl_kernel. Verify fallback to the in-memory kernel when disk persistence fails.pyexecutor/engine/metadata.pyandrunners/decoder/metadata.py: Passmax_total_draft_tokensonly toTrtllmAttentionMetadatasubclasses. Verify other backends are unaffected.Description
Move DeepSeek-V4 PrimTS sparse MLA compilation from the first eligible forward into decoder warmup. Each warmup pass discovers reachable specializations with
plan(compile=False)and checks each distinct kernel withis_kernel_cached()when parallel compilation is enabled.run(validate=False)path with independent input-view caches.c4a601c01341c0a2b6f5623e9f64adb917c27be8on the fork branchcute-dsl-precompile-api--prims-ts-vendor-20261121. It includes the current TensorRT-LLM vendor baselined76f94bb6and the reviewed PrimTS changes. Upstream tracks compile caching and deferred planning in FlashInfer #9395 and sparse MLA run-time view reuse in FlashInfer #9767. With the pinned FlashInfer 0.7.3 runtime, TensorRT-LLM supplies the per-kernel JIT locking in its local compatibility module. FlashInfer cache PR · Original TRT-LLM implementationThe vendor topic is based on the canonical source promoted by TensorRT-LLM #22723 after #19395 merged. All nine commits that touch vendored source match the declared upstream PRs by normalized edits. The local cache adapter can be removed once the pinned FlashInfer release includes #9395's cache API and locking. #22659 also updates this pin. Whichever integration lands second should replay its source changes on the promoted tip and retain deferred planning, membership probing, and the dense CP launch ABI.
Sparse warmup prepares all eligible local PrimTS profiles whenever the library is enabled, regardless of its position in the FMHA list. Dispatch uses the actual forward inputs. With JIT disabled, prewarm loads available artifacts and defers each missing variant to a forward that selects it. Eager generation derives its kernel profile through the same library method. The supervisor reports worker output on failure and terminates surviving descendants on both timeout and abnormal exit. Positive infinity allows an unlimited worker wait. DSA gets its top-k from runtime indices and attempts its optional scan on the first eligible eager forward. In non-strict mode a failed representative leaves subsequent batches free to plan their own rows.
The configured draft limit is applied before the first forward. Workers check that their artifact was published and the supervisor terminates its process group if the rank dies. On B300, two cold workers published both variants in 48.14 s. A separate JIT-disabled process loaded both in 0.49 s with zero workers. In a separate rank-kill check, the supervisor and pool descendants stopped within 0.70 s.
Downstream performance
DeepSeek-V4-Flash ran on four B200s with TP4/EP4, attention data parallelism, and 256 input and output tokens. Warmup times below are rank-0 wall time. The measurements predate the rebase. Each row group compares revisions from the same allocation.
c9c07e04→ standalone poole9e140d7c9c07e04→ cache check715858bac0On a cold PrimTS cache, grouping all 12 specializations in one pool saved 223.1 s in the first warmup (one process per version). In the warm comparison, the earlier standalone pool without the cache check took 20.2 s eager / 20.0 s graph in its first PrimTS phase on the same node (one reference process per mode). Querying disk membership saved about 15 s in that phase. Main and the cache-check version each ran in two fresh processes per mode.
The final implementation checks the cache again when the executor binds its serving resources. We isolated this change on a separate exclusive four-B200 node with the same FlashInfer code, comparing
715858bac0against its flag-removal patch, subsequently committed as3fcfb3f572.Both versions launched zero workers with all 12 exports available and four workers enabled per rank. The additional 12 checks took 1.35 s in a separate two-pass probe. The full PrimTS phase also includes local planning and rank synchronization. If two new variants first appear on the final pass, the new policy can compile them in parallel.
Steady-state time per output token
Columns are separate experiments. Each used before/after/after/before process order, two warmup batches and three measured batches per concurrency in each process. In the main-base comparison, eager paired differences changed sign with run order. Graph C128 averaged 4.5% slower and included one 14.33 ms/token candidate batch. In the isolated flag comparison the largest absolute difference was 0.174 ms/token. Outputs, graph replay and KV capacity matched within each mode. Weight-file page cache and weight-loading times varied across fresh processes.
Sparse MLA wrapper microbenchmark
The earlier public
run()comparison used TRT-LLM main-basec9c07e04and feature revision9d8394d2with FlashInfer81f96025f. It included the warmed prepared dispatch on main. The benchmark used BF16, 64 heads, fixed KV pools, and one process per source.Values are host microseconds per call. Persistent inputs improved 11.1–18.8% and one-use inputs improved 1.1–8.0%. Replacing inputs while keeping the prior tensors alive cost 0.90–2.15 µs more per call (5.3–12.3%). Output/LSE hashes and native arguments matched in all 18 cases. The fixture used 768 physical cache rows and fixed per-profile pools.
In a separate B200 wrapper experiment on FlashInfer #9767 (
5880ee354), bypassing only the query, index and output view caches saved 0.956 µs per host call on average with ephemeral adapter inputs. It cost 0.082–0.356 µs per call with persistent inputs across six profiles. Two fresh processes exercised both orders. Numerical results and all 17 launch arguments matched. We retained the general cache. These timings isolate the wrapper and exclude model-level TPOT.Test Coverage
On
002a9767e9, four targeted B200 regression cases and two cold-cache dispatch checks passed. With JIT disabled and an empty PrimTS cache, prewarm completed in both orders. The fallback-first forward matched its reference, and the PrimTS-first forward raisedMissingJITCacheErrorfor the requested kernel. A mixed-cache case verified that warmup continued loading available variants after a miss.On
29257e701dafter rebasing onto4af2d530, all 64 targeted B200 tests passed with zero skips. The run used stock FlashInfer 0.7.3, CuTe DSL 4.8.0 and a TensorRT-LLM CI wheel built from the same C++ tree. Coverage includes the vendored-import regression, all ten sampler cases that failed in Jenkins #71596, four public trusted-launch ABI configurations, dense MLA graph replay, sparse warmup followed by zero new kernel compilations, worker cleanup, TP/CP failure propagation, and DSA exact-row fallback.Two additional single-B200 Helix tests passed against the vendored provider. The page-boundary case exercised logical CP ranks 0, 1 and 2 and compared merged output and LSE with the reference. The CP8 case exercised nonzero-rank CUDA graph replay. This validates the launch ABI on one device. A third additional test passed for packed sparse partial-head BF16 graph replay. Total on the rebased revision: 67 passed, zero skipped.
The vendor tree verifies against its published SHA. The cumulative pre-commit checks passed. The advisory type checker reported the existing
gen_offsetwarning in the unchanged FP8 path.d346fb95e1, the packed dense launch-bounds and sparse caller-workspace/cache-membership tests both passed on B200. Six B200 cases passed on the separate FlashInfer PR tree, including the latest upstream KV-bounds ABI.e466dcb2, both affected Qwen3.8-Flash-Next FP8 test files collected on B300 (120 cases). Four snapshot and cached-replay cases passed, including graph replay.PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.