From cd63b40ce989a292d5dd7207d1ffd703aff2048a Mon Sep 17 00:00:00 2001 From: Bastian Hagedorn Date: Tue, 10 Nov 2026 08:51:57 +0100 Subject: [PATCH] [None][fix] Fence TMEM load before early stage release in STG GEMM epilogue In the direct-store epilogue helper `epilogue()` of tensorrt_llm/_torch/kernels/utils/gemm/sm100.py the epilogue warps hand the accumulator stage back to the MMA warp at the last subtile with an elected `consumer_release` (an mbarrier arrive) right after the `tcgen05.ld` copy and before the loaded registers are read (`tTR_rAcc.load()`, scale, convert, store). There is no `tcgen05.wait::ld` in the kernel, and the MMA warp's `producer_acquire` waits on exactly that arrive and then issues the next tile's `tcgen05.mma` into the same stage. The PTX ISA makes a `tcgen05.ld` complete only at a `tcgen05.wait::ld` in the issuing thread; a register dependency orders the consuming instruction, not a later mbarrier arrive. The compiled code shows the arrive issued with no scoreboard dependency on the load and the first convert waiting on the load after it. Insert `cute.arch.fence_view_async_tmem_load()` (`tcgen05.wait::ld`) before the release, the same ordering the helper's TMA-store sibling and the DSL's own epilogue helper already use. The wait is on registers the warp consumes a few instructions later anyway, so no instruction is added; the scoreboard wait moves onto the arrive. Verified: the patched helper compiles in the `use_tma_store=False` path of kernels/rubin/dense_gemm_persistent.py (sm_107a) and the pre-LLVM IR now has the wait between the load and the arrive. On a B200 the stock and the fenced Blackwell build of the same helper shape produce bitwise identical output. Signed-off-by: Bastian Hagedorn --- tensorrt_llm/_torch/kernels/utils/gemm/sm100.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tensorrt_llm/_torch/kernels/utils/gemm/sm100.py b/tensorrt_llm/_torch/kernels/utils/gemm/sm100.py index 8f4e953ddfb8..f74ff4ba1192 100755 --- a/tensorrt_llm/_torch/kernels/utils/gemm/sm100.py +++ b/tensorrt_llm/_torch/kernels/utils/gemm/sm100.py @@ -543,6 +543,8 @@ def epilogue( # tTR_tAcc_mn = tTR_tAcc[(None, None, None, subtile_idx)] cute.copy(tiled_copy_t2r, tTR_tAcc_mn, tTR_rAcc) + # Wait for the TMEM load to complete before the stage can be released to the MMA warp + cute.arch.fence_view_async_tmem_load() # Async arrive accumulator buffer empty # Release early for perf if subtile_idx == subtile_cnt - 1: