Skip to content

[None][fix] Wait for the Prim-TS MLA decode correction-record load before releasing its stage - #21987

Draft
bastianhagedorn wants to merge 4 commits into
NVIDIA:mainfrom
bastianhagedorn:fix/prims-ts-mla-decode-corr-load-wait-vendor
Draft

bastianhagedorn wants to merge 4 commits into
NVIDIA:mainfrom
bastianhagedorn:fix/prims-ts-mla-decode-corr-load-wait-vendor

Conversation

@bastianhagedorn

@bastianhagedorn bastianhagedorn commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

@coderabbitai summary

Description

tensorrt_llm/_torch/attention/backends/prims_ts/kernels/mla_decode/throughput_2cta/resources.py, TmemCorrResource.load_corr. The correction warps of the Prim-TS 2-CTA MLA decode kernel read the 4-word correction record of the current stage (row_sum, row_max, correction_factor, no_correction) from tensor memory with one prims.tcgen05_ld(..., num=4). TmemCorrResource is a 2-stage ring written by the softmax warps (store_corr: tcgen05_st after the runtime's producer acquire) and consumed by the correction warps; the task runtime releases the stage (an mbarrier.arrive on the stage's empty barrier) as soon as load_corr returns, and the loaded words are consumed only afterwards: rescale_o votes on no_correction and multiplies O by correction_factor, epilogue_store reads row_sum / row_max from final_row_stats.

The function issued prims.tcgen05_wait(kind=prims.Tcgen05Wait.LOAD) before the load. tcgen05.ld writes its destination registers asynchronously; the PTX ISA (tcgen05 instructions, "Asynchronous operations", section 9.7.18.6.4.5 in the 9.x documents) says that the load completes only at a tcgen05.wait::ld in the issuing thread and that a register dependency orders only the dependent instruction, "not any other form of ordering". The arrive that releases the stage has no register dependency on the loaded values, so nothing orders it after the load: a softmax warp parked on that barrier may acquire the stage and overwrite the same four columns while the load is still reading them. If that happened, tile t's O rescale would use tile t+2's correction_factor / no_correction (a skipped or misapplied rescale), and with a single KV tile the only record feeds the epilogue's normalisation. The wait that was there is not useless, it covers the previous iteration's O loads, which is why it stays, just on the other side of the load.

Change: one line, prims.tcgen05_wait(kind=prims.Tcgen05Wait.LOAD) moved from before the record load to right after it, with a short comment. This is what the 1-CTA kernel of the same backend already does after each of its loads (throughput_latency_1cta/resources/tmem_corr.py), and the same fix as #21849 for the correction-record load of the Blackwell MLA decode kernels (mla_decode_fp16.py / mla_decode_fp8.py). No other behaviour changes.

Vendored source

tensorrt_llm/_torch/attention/backends/prims_ts is a vendored copy of FlashInfer's flashinfer/attention/prims_ts (lock entry flashinfer-prims-ts in 3rdparty/vendor_sources.lock.yaml). Following 3rdparty/vendor-sources.md, the change was exported with scripts/vendor/manage.py export to a temporary branch of my FlashInfer fork at the locked commit (bastianhagedorn/flashinfer branch trtllm-prims-ts-mla-decode-corr-load-wait-20261110, commit 8846717c, one commit on top of the canonical trtllm-prims-ts-dev-20261107 tip 911d6a9d, which is the live tip of that branch) and the lock is pinned to that commit (manage.py pin); the compatibility patch is unchanged and manage.py check passes offline and against the source checkout. The paired upstream PR is flashinfer-ai/flashinfer#9319. After #21824 replaced the compatibility patch at the same canonical commit, main was merged into this branch (6f9432d), main's patch kept, and the entry re-pinned to the same fork commit 8846717c with manage.py pin; manage.py check passes again offline and against the source checkout. After main promoted the pin to the canonical tip 2ffd3f8b (#22319), main was merged again (ba1ad01) and the source branch replaced by trtllm-prims-ts-mla-decode-corr-load-wait-20261117 (commit 35f376c3): the same fix commit on top of the #21852 source branch (c5cf5f72, flashinfer-ai/flashinfer#9224), which is a fast-forward of the canonical tip, so this snapshot stacks on #21852 and will be re-pinned once that PR merges. Both manage.py check modes pass. After the next promotion moved the canonical tip to 31fe2dbd (#21151, CUTLASS DSL 4.9 compatibility), the same was done again (1489958): source branch trtllm-prims-ts-mla-decode-corr-load-wait-20261119 (commit 2c0e6c27) on top of the #21852 source branch at 56b92a6d, main's compatibility patch kept, both manage.py check modes pass.

schema_version: 1
vendors:
  flashinfer-prims-ts:
    upstream_prs:
      - https://github.com/flashinfer-ai/flashinfer/pull/9224
      - https://github.com/flashinfer-ai/flashinfer/pull/9319

Hardware verification

The FlashInfer kernel (byte-identical to this vendored file at the site) was run on a B200 (compute capability 10.0) with the repository's own test inputs and fp32 reference check: batch 64, 128 heads, latent 512, rope 64, page 64, KV lengths 4111..8192 (also 256..455, 1024..2023 and 8192 everywhere), bf16, 2-CTA clusters, non-persistent and CLC persistent schedules. The compiled stock kernel shows the gap: LDTM.x4 of the record, then the releasing SYNCS.ARRIVE with an empty wait mask, then the first consumer (ISETP on no_correction, the FMUL2s by correction_factor) waiting the load's scoreboard after the arrive. With the fix the arrive itself waits the scoreboard, no instruction is added (same SASS size), and the output is bitwise identical to the unpatched kernel over 2000 launches. A sentinel experiment (per-tile marker written into correction_factor by the softmax warps, the factor actually used recorded by the correction warps, extra TMEM loads queued in front of the record load, 10-100 us stalls between the release and the first use) read 211.7 M records over 11 800 launches of the unpatched kernel with 0 stale records: in this kernel the refill of a released stage is ordered behind the consumer's first use through the O and P pipelines (the softmax warp acquires P before it stores the record, P is released by the MMA warp after PV, which waits on O, and O is released by the correction warps after they have used the record), at least 2.2 us behind in every measurement. So the race is not observable with the shipped pipeline depths; it would be with a deeper O or P ring or a per-N O pipeline. The fix makes the ordering architectural at no measurable cost.

Test Coverage

The kernel is exercised by tests/unittest/_torch/attention/test_prims_ts_attention_backend.py (test_prims_ts_deepseek_v3_lite_mla_generation runs MLA decode through the Prim-TS backend; the 2-CTA throughput policy is the default selection for dense decode shapes). No new test: memory-model conformance, no observable failure to test for with the shipped pipeline depths (see above). The change is limited to the position of one existing prims.tcgen05_wait call.

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

In the vendored Prim-TS 2-CTA MLA decode kernel
(tensorrt_llm/_torch/attention/backends/prims_ts/kernels/mla_decode/
throughput_2cta/resources.py), TmemCorrResource.load_corr issued
tcgen05.wait::ld before the tcgen05.ld of the 4-word correction record
(row_sum, row_max, correction_factor, no_correction). The task runtime
releases the correction stage to the softmax warps as soon as the task
body returns, and the loaded words are consumed only afterwards
(rescale_o, epilogue_store). tcgen05.ld completes asynchronously and is
ordered only by tcgen05.wait::ld or a register dependency, so the
release could let a softmax warp overwrite the stage while the load is
still reading it.

Move the wait to right after the load, as the 1-CTA kernel of this
backend already does (throughput_latency_1cta/resources/tmem_corr.py).
Verified on a B200: the compiled SASS now has the releasing SYNCS.ARRIVE
wait on the LDTM.x4 scoreboard, no instruction is added, and the output
is bitwise identical to the previous build.

Vendored source: the change was exported with scripts/vendor/manage.py
export to a temporary branch of the FlashInfer fork at the locked commit
911d6a9d (trtllm-prims-ts-dev-20261107) and the lock is pinned to that
commit; the compatibility patch is unchanged and manage.py check passes.

Signed-off-by: Bastian Hagedorn <bhagedorn@nvidia.com>
@yuxianq

yuxianq commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator

Vendor validation: flashinfer-prims-ts

@bastianhagedorn — Vendor promotion needs maintainer action:

The base lock now uses refresh branch trtllm-prims-ts-dev-20261107, but this monitor still publishes to trtllm-prims-ts-dev-20261012.

Fix: A maintainer must stop this monitor and migrate its workdir with --canonical-branch trtllm-prims-ts-dev-20261107 --migrate-canonical-branch-from trtllm-prims-ts-dev-20261012. Verify a dry run before resuming publication. Do not promote back to the old branch or change contributor metadata to bypass this cutover.

The bot will recheck automatically on a later poll.

Checked PR revision: cb9927309da5. This is a snapshot; newer results are posted as new comments. Observed dev/upstream tip SHAs may be stale: tip movement alone does not post a new comment when the result is unchanged. Fetch the live branch before rebasing or choosing a new pin.

…er-prims-ts

main replaced the flashinfer-prims-ts compatibility patch at the same
canonical commit 911d6a9d (patch_digest 5e19fc30), which conflicted with the
lock entry of this branch. Keep main's patch, re-pin to the same fork commit
8846717c9 (trtllm-prims-ts-mla-decode-corr-load-wait-20261110) with
scripts/vendor/manage.py pin; the destination digest is now 1132289c.
manage.py check passes offline and against the source checkout.

Signed-off-by: Bastian Hagedorn <bhagedorn@nvidia.com>
@yuxianq

yuxianq commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

Vendor validation: flashinfer-prims-ts

@bastianhagedorn — Vendor promotion metadata and source attribution are valid. This is not a code-review approval. Promotion still requires the source PR to merge.

Checked PR revision: 6f9432dae8a5. This is a snapshot; newer results are posted as new comments. Observed dev/upstream tip SHAs may be stale: tip movement alone does not post a new comment when the result is unchanged. Fetch the live branch before rebasing or choosing a new pin.

Checked source: https://github.com/bastianhagedorn/flashinfer.git / trtllm-prims-ts-mla-decode-corr-load-wait-20261110, pin 8846717c966bdd53653b218ee563711a4235291a.

Supersedes previous vendor validation.

@yuxianq

yuxianq commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Vendor validation: flashinfer-prims-ts

@bastianhagedorn — Vendor promotion needs maintainer action:

Waiting for the preceding vendor promotion: the base lock still points to repository https://github.com/yuxianq/flashinfer.git, branch flashinfer-9222-prims-ts--source.

Fix: A maintainer must finish the preceding lock-only promotion back to yuxianq/flashinfer / trtllm-prims-ts-dev-20261107. After it merges, the author should rebase this consumer PR onto the latest base branch and rebase its temporary source branch onto the new canonical pin, then update the lock/vendor snapshot. No metadata edit can unblock the preceding promotion.

The bot will recheck automatically on a later poll.

Checked PR revision: 6f9432dae8a5. This is a snapshot; newer results are posted as new comments. Observed dev/upstream tip SHAs may be stale: tip movement alone does not post a new comment when the result is unchanged. Fetch the live branch before rebasing or choosing a new pin.

Supersedes previous vendor validation.

@yuxianq

yuxianq commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Vendor validation: flashinfer-prims-ts

@bastianhagedorn — Vendor promotion needs author action:

Source update is not a fast-forward.

Fix: Fetch canonical repository https://github.com/yuxianq/flashinfer.git, branch trtllm-prims-ts-dev-20261107, and rebase the temporary source branch trtllm-prims-ts-mla-decode-corr-load-wait-20261110 onto its latest tip without merge commits, not just the consumer lock's pin 2ffd3f8b2c2d163237c95bdf40e7a065a99cd544. Then use scripts/vendor/manage.py to update 3rdparty/vendor_sources.lock.yaml and the vendored snapshot to the rebased SHA, and push the source branch and consumer PR. Editing metadata alone will not fix this.

The bot will recheck automatically on a later poll.

Checked PR revision: 6f9432dae8a5. This is a snapshot; newer results are posted as new comments. Observed dev/upstream tip SHAs may be stale: tip movement alone does not post a new comment when the result is unchanged. Fetch the live branch before rebasing or choosing a new pin.

Checked source: https://github.com/bastianhagedorn/flashinfer.git / trtllm-prims-ts-mla-decode-corr-load-wait-20261110, pin 8846717c966bdd53653b218ee563711a4235291a.

Supersedes previous vendor validation.

@yuxianq yuxianq left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Vendor validation result: needs author action.

Latest vendor validation and step-by-step fix.

…s onto a rebased source branch

Main promoted the flashinfer-prims-ts pin to trtllm-prims-ts-dev-20261107 at
2ffd3f8b. The temporary source branch is replaced by
trtllm-prims-ts-mla-decode-corr-load-wait-20261117 (35f376c3), which stacks the
correction-record load wait on top of the NVIDIA#21852 source branch at c5cf5f72, itself
a fast-forward of the canonical tip. The lock and the vendored snapshot are
updated with scripts/vendor/manage.py pin; manage.py check passes offline and
against the source checkout.

Signed-off-by: Bastian Hagedorn <bhagedorn@nvidia.com>
@yuxianq
yuxianq dismissed their stale review November 17, 2026 09:51

Automated vendor metadata issue resolved; code review remains required.

@yuxianq

yuxianq commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Vendor validation: flashinfer-prims-ts

@bastianhagedorn — Vendor promotion metadata and source attribution are valid. This is not a code-review approval. Promotion still requires the source PR to merge.

Checked PR revision: ba1ad014bafc. This is a snapshot; newer results are posted as new comments. Observed dev/upstream tip SHAs may be stale: tip movement alone does not post a new comment when the result is unchanged. Fetch the live branch before rebasing or choosing a new pin.

Checked source: https://github.com/bastianhagedorn/flashinfer.git / trtllm-prims-ts-mla-decode-corr-load-wait-20261117, pin 35f376c341c749ae0efdef4d5ea34af429cff38e.

Supersedes previous vendor validation.

@yuxianq

yuxianq commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

Vendor validation: flashinfer-prims-ts

@bastianhagedorn — Vendor promotion needs maintainer action:

Waiting for the preceding vendor promotion: the base lock still points to repository https://github.com/yuxianq/flashinfer.git, branch nvbug-6959297-cutlass-dsl-compat--flashinfer.

Fix: A maintainer must finish the preceding lock-only promotion back to yuxianq/flashinfer / trtllm-prims-ts-dev-20261107. After it merges, the author should rebase this consumer PR onto the latest base branch and rebase its temporary source branch onto the new canonical pin, then update the lock/vendor snapshot. No metadata edit can unblock the preceding promotion.

The bot will recheck automatically on a later poll.

Checked PR revision: ba1ad014bafc. This is a snapshot; newer results are posted as new comments. Observed dev/upstream tip SHAs may be stale: tip movement alone does not post a new comment when the result is unchanged. Fetch the live branch before rebasing or choosing a new pin.

Supersedes previous vendor validation.

@yuxianq

yuxianq commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

Vendor validation: flashinfer-prims-ts

@bastianhagedorn — Vendor promotion needs author action:

Source update is not a fast-forward.

Fix: Fetch canonical repository https://github.com/yuxianq/flashinfer.git, branch trtllm-prims-ts-dev-20261107, and rebase the temporary source branch trtllm-prims-ts-mla-decode-corr-load-wait-20261117 onto its latest tip without merge commits, not just the consumer lock's pin 31fe2dbd6b65cca758c1318f177b315be0c33778. Then use scripts/vendor/manage.py to update 3rdparty/vendor_sources.lock.yaml and the vendored snapshot to the rebased SHA, and push the source branch and consumer PR. Editing metadata alone will not fix this.

The bot will recheck automatically on a later poll.

Checked PR revision: ba1ad014bafc. This is a snapshot; newer results are posted as new comments. Observed dev/upstream tip SHAs may be stale: tip movement alone does not post a new comment when the result is unchanged. Fetch the live branch before rebasing or choosing a new pin.

Checked source: https://github.com/bastianhagedorn/flashinfer.git / trtllm-prims-ts-mla-decode-corr-load-wait-20261117, pin 35f376c341c749ae0efdef4d5ea34af429cff38e.

Supersedes previous vendor validation.

@yuxianq yuxianq left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Vendor validation result: needs author action.

Latest vendor validation and step-by-step fix.

…s onto a rebased source branch

Main promoted the flashinfer-prims-ts pin to trtllm-prims-ts-dev-20261107 at
31fe2dbd (CUTLASS DSL 4.9 compatibility). The temporary source branch is replaced
by trtllm-prims-ts-mla-decode-corr-load-wait-20261119 (2c0e6c27), which stacks the
correction-record load wait on top of the NVIDIA#21852 source branch at 56b92a6d, itself
a fast-forward of the canonical tip. The lock and the vendored snapshot are updated
with scripts/vendor/manage.py pin; manage.py check passes offline and against the
source checkout.

Signed-off-by: Bastian Hagedorn <bhagedorn@nvidia.com>
@yuxianq
yuxianq dismissed their stale review November 19, 2026 04:55

Automated vendor metadata issue resolved; code review remains required.

@yuxianq

yuxianq commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

Vendor validation: flashinfer-prims-ts

@bastianhagedorn — Vendor promotion metadata and source attribution are valid. This is not a code-review approval. Promotion still requires the source PR to merge.

Checked PR revision: 148995884ad1. This is a snapshot; newer results are posted as new comments. Observed dev/upstream tip SHAs may be stale: tip movement alone does not post a new comment when the result is unchanged. Fetch the live branch before rebasing or choosing a new pin.

Checked source: https://github.com/bastianhagedorn/flashinfer.git / trtllm-prims-ts-mla-decode-corr-load-wait-20261119, pin 2c0e6c27d10dda66ac1104dacfb35b070addebf9.

Supersedes previous vendor validation.

@yuxianq

yuxianq commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator

Vendor validation: flashinfer-prims-ts

@bastianhagedorn — Vendor promotion needs maintainer action:

Waiting for the preceding vendor promotion: the base lock still points to repository https://github.com/yuxianq/flashinfer.git, branch prims-ts-fmha-lib--cp-dev-task-api-20261119.

Fix: A maintainer must finish the preceding lock-only promotion back to yuxianq/flashinfer / trtllm-prims-ts-dev-20261107. After it merges, the author should rebase this consumer PR onto the latest base branch and rebase its temporary source branch onto the new canonical pin, then update the lock/vendor snapshot. No metadata edit can unblock the preceding promotion.

The bot will recheck automatically on a later poll.

Checked PR revision: 148995884ad1. This is a snapshot; newer results are posted as new comments. Observed dev/upstream tip SHAs may be stale: tip movement alone does not post a new comment when the result is unchanged. Fetch the live branch before rebasing or choosing a new pin.

Supersedes previous vendor validation.

@yuxianq

yuxianq commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator

Vendor validation: flashinfer-prims-ts

@bastianhagedorn — Vendor promotion needs author action:

Source update is not a fast-forward.

Fix: Fetch canonical repository https://github.com/yuxianq/flashinfer.git, branch trtllm-prims-ts-dev-20261107, and rebase the temporary source branch trtllm-prims-ts-mla-decode-corr-load-wait-20261119 onto its latest tip without merge commits, not just the consumer lock's pin d76f94bb6ae3bc1730689a1a080a79913dcc3606. Then use scripts/vendor/manage.py to update 3rdparty/vendor_sources.lock.yaml and the vendored snapshot to the rebased SHA, and push the source branch and consumer PR. Editing metadata alone will not fix this.

The bot will recheck automatically on a later poll.

Checked PR revision: 148995884ad1. This is a snapshot; newer results are posted as new comments. Observed dev/upstream tip SHAs may be stale: tip movement alone does not post a new comment when the result is unchanged. Fetch the live branch before rebasing or choosing a new pin.

Checked source: https://github.com/bastianhagedorn/flashinfer.git / trtllm-prims-ts-mla-decode-corr-load-wait-20261119, pin 2c0e6c27d10dda66ac1104dacfb35b070addebf9.

Supersedes previous vendor validation.

@yuxianq yuxianq left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Vendor validation result: needs author action.

Latest vendor validation and step-by-step fix.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants