Repository navigation
[None][fix] Fence TMEM load before early stage release in STG GEMM epilogue - #21988
Draft
bastianhagedorn wants to merge 1 commit into
Draft
bastianhagedorn wants to merge 1 commit into
bastianhagedorn wants to merge 1 commit into
Conversation
…ilogue In the direct-store epilogue helper `epilogue()` of tensorrt_llm/_torch/kernels/utils/gemm/sm100.py the epilogue warps hand the accumulator stage back to the MMA warp at the last subtile with an elected `consumer_release` (an mbarrier arrive) right after the `tcgen05.ld` copy and before the loaded registers are read (`tTR_rAcc.load()`, scale, convert, store). There is no `tcgen05.wait::ld` in the kernel, and the MMA warp's `producer_acquire` waits on exactly that arrive and then issues the next tile's `tcgen05.mma` into the same stage. The PTX ISA makes a `tcgen05.ld` complete only at a `tcgen05.wait::ld` in the issuing thread; a register dependency orders the consuming instruction, not a later mbarrier arrive. The compiled code shows the arrive issued with no scoreboard dependency on the load and the first convert waiting on the load after it. Insert `cute.arch.fence_view_async_tmem_load()` (`tcgen05.wait::ld`) before the release, the same ordering the helper's TMA-store sibling and the DSL's own epilogue helper already use. The wait is on registers the warp consumes a few instructions later anyway, so no instruction is added; the scoreboard wait moves onto the arrive. Verified: the patched helper compiles in the `use_tma_store=False` path of kernels/rubin/dense_gemm_persistent.py (sm_107a) and the pre-LLVM IR now has the wait between the load and the arrive. On a B200 the stock and the fenced Blackwell build of the same helper shape produce bitwise identical output. Signed-off-by: Bastian Hagedorn <bhagedorn@nvidia.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@coderabbitai summary
Description
What. In the direct-store (STG) epilogue helper
epilogue()oftensorrt_llm/_torch/kernels/utils/gemm/sm100.pythe four epilogue warps load each accumulator subtile from tensor memory into registers and hand the accumulator stage back to the MMA warp at the last subtile, right after the load and before the loaded registers are read:The
cute.copywith thetcgen05.ldatom writes its destination registers asynchronously.consumer_releaseis anmbarrier.arriveon the stage's empty barrier; the MMA warp waits on exactly that barrier inproducer_acquireand then issues the next tile'stcgen05.mmainto the same stage. There is notcgen05.wait::ldanywhere in the compiled kernel, so the stage the in-flight load is still reading is handed to its next writer before the load is known to be complete. The exposed data is the last subtile of every tile (32 fp32 per thread in the Rubin fp8 GEMM configuration, the whole 128x128 tile whenepi_tile == cta_tile).Why it is a bug. The PTX ISA (tcgen05 instructions, "Asynchronous operations", section 9.7.18.6.4.5 in the 9.x documents) states that a
tcgen05.ldcompletes only at atcgen05.wait::ldin the issuing thread, and that an intra-thread register dependency is respected but "does not imply any other form of ordering". The arrive has no register dependency on the load, so neither rule orders it after the load. The compiled code shows exactly that:LDTMwrites its scoreboard, the electedSYNCS.ARRIVEon the empty barrier issues with an empty wait mask a few instructions later, and the firstF2FPconvert after the arrive is the first instruction that waits on the load.Where it is reachable.
kernels/rubin/dense_gemm_persistent.pyis the caller of this copy of the helper (itsuse_tma_store=Falsepath; the other dense GEMMs in the tree call the DSL's owncutlass.utils.gemm.sm100.epilogue). The kernel's default isuse_tma_store=True, but the fp8 per-tensor Rubin GEMM autotuner (CuteDSLFp8RubinPerTensorGemmRunnerincute_dsl_custom_ops.py) tries bothuse_tma_storevalues, so the direct-store form is a shipped tactic. In that kernel nothing but the empty-barrier wake-up stands between the release and the MMA warp's nexttcgen05.mmainto the stage. The symptom would be silent: the last subtile of a tile converted from accumulator values the next tile's first MMA has started to overwrite.Change. One line:
cute.arch.fence_view_async_tmem_load()(tcgen05.wait::ld) after the TMEM load and before the release. This is the same ordering fix as in the DSL's own epilogue helper (whose overlapping-accumulator path already fences before the release), and the idiom of TensorRT-LLM #21849 (MLA decode correction stage) and #21855 (visual_gen FMHA S release). The wait is on registers the warp reads a few instructions later anyway (tTR_rAcc.load()and the converts), so no instruction is added; the scoreboard wait moves from the first convert onto the instruction before the arrive. The TMA-store helperepilogue_tma_storein the same file releases after its named barriers and is unchanged.Hardware verification
A static review found the gap. The direct-store epilogue of this shape was then run on a B200 (compute capability 10.0) through a Blackwell persistent GEMM caller with
use_tma_store=False(bf16 in and out, mnkl 4096 x 8192 x 512 x 1, 2048 tiles on 148 CTAs, 2 accumulator stages of 128 columns, dynamic persistent scheduler), 100 launches per variant againsttorch.bmm, withglobaltimerprobes around the release and the MMA warp's re-acquire of the stage, an MMA-warp variant that stores a sentinel into the just-released stage immediately after itsproducer_acquire, and a 1 us to 100 us stall of the epilogue warps between the release and the first read.LDTM.x128followed by@P SYNCS.ARRIVEwith no scoreboard dependency on the load, the firstF2FPafter the arrive waiting on it.ELECTbefore the arrive now carries the wait on the load's scoreboard.The Rubin fp8 GEMM that calls this copy compiles with the change (sm_107a,
use_tma_store=False, raster N) and its IR has thetcgen05.wait loadbetween the load and the arrive; it was not run on hardware here. So the race is not observable in the Blackwell caller measured; in a caller whose MMA warp acquires the stage as soon as the barrier flips, nothing in the ISA keeps the window closed. This is a conformance fix at no measurable cost.Test Coverage
No new test: this is a memory-model conformance fix with no observable failure to test for on current hardware (see above). The helper is exercised by the Rubin fp8 GEMM unit tests that go through
cute_dsl_fp8_per_tensor_gemm_rubin(tests/unittest/_torch/kernels/parallel/test_cute_dsl_quant_dense_rubin.py, with the autotuner trying both store modes) and by the kernel's own harness (python -m tensorrt_llm._torch.kernels.rubin.dense_gemm_persistent ...without--use_tma_store, reference check againsttorch).PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.