Skip to content

[None][fix] Fence TMEM load before early stage release in STG GEMM epilogue - #21988

Draft
bastianhagedorn wants to merge 1 commit into
NVIDIA:mainfrom
bastianhagedorn:fix/gemm-sm100-stg-epilogue-tmem-load-wait
Draft

bastianhagedorn wants to merge 1 commit into
NVIDIA:mainfrom
bastianhagedorn:fix/gemm-sm100-stg-epilogue-tmem-load-wait

Conversation

@bastianhagedorn

Copy link
Copy Markdown
Contributor

@coderabbitai summary

Description

What. In the direct-store (STG) epilogue helper epilogue() of tensorrt_llm/_torch/kernels/utils/gemm/sm100.py the four epilogue warps load each accumulator subtile from tensor memory into registers and hand the accumulator stage back to the MMA warp at the last subtile, right after the load and before the loaded registers are read:

cute.copy(tiled_copy_t2r, tTR_tAcc_mn, tTR_rAcc)
# Async arrive accumulator buffer empty
# Release early for perf
if subtile_idx == subtile_cnt - 1:
    with cute.arch.elect_one():
        acc_pipeline.consumer_release(acc_consumer_state)
    acc_consumer_state.advance()

acc_vec = tTR_rAcc.load()
...
acc_vec = epilogue_op(acc_vec.to(gemm_kernel.c_dtype))

The cute.copy with the tcgen05.ld atom writes its destination registers asynchronously. consumer_release is an mbarrier.arrive on the stage's empty barrier; the MMA warp waits on exactly that barrier in producer_acquire and then issues the next tile's tcgen05.mma into the same stage. There is no tcgen05.wait::ld anywhere in the compiled kernel, so the stage the in-flight load is still reading is handed to its next writer before the load is known to be complete. The exposed data is the last subtile of every tile (32 fp32 per thread in the Rubin fp8 GEMM configuration, the whole 128x128 tile when epi_tile == cta_tile).

Why it is a bug. The PTX ISA (tcgen05 instructions, "Asynchronous operations", section 9.7.18.6.4.5 in the 9.x documents) states that a tcgen05.ld completes only at a tcgen05.wait::ld in the issuing thread, and that an intra-thread register dependency is respected but "does not imply any other form of ordering". The arrive has no register dependency on the load, so neither rule orders it after the load. The compiled code shows exactly that: LDTM writes its scoreboard, the elected SYNCS.ARRIVE on the empty barrier issues with an empty wait mask a few instructions later, and the first F2FP convert after the arrive is the first instruction that waits on the load.

Where it is reachable. kernels/rubin/dense_gemm_persistent.py is the caller of this copy of the helper (its use_tma_store=False path; the other dense GEMMs in the tree call the DSL's own cutlass.utils.gemm.sm100.epilogue). The kernel's default is use_tma_store=True, but the fp8 per-tensor Rubin GEMM autotuner (CuteDSLFp8RubinPerTensorGemmRunner in cute_dsl_custom_ops.py) tries both use_tma_store values, so the direct-store form is a shipped tactic. In that kernel nothing but the empty-barrier wake-up stands between the release and the MMA warp's next tcgen05.mma into the stage. The symptom would be silent: the last subtile of a tile converted from accumulator values the next tile's first MMA has started to overwrite.

Change. One line: cute.arch.fence_view_async_tmem_load() (tcgen05.wait::ld) after the TMEM load and before the release. This is the same ordering fix as in the DSL's own epilogue helper (whose overlapping-accumulator path already fences before the release), and the idiom of TensorRT-LLM #21849 (MLA decode correction stage) and #21855 (visual_gen FMHA S release). The wait is on registers the warp reads a few instructions later anyway (tTR_rAcc.load() and the converts), so no instruction is added; the scoreboard wait moves from the first convert onto the instruction before the arrive. The TMA-store helper epilogue_tma_store in the same file releases after its named barriers and is unchanged.

Hardware verification

A static review found the gap. The direct-store epilogue of this shape was then run on a B200 (compute capability 10.0) through a Blackwell persistent GEMM caller with use_tma_store=False (bf16 in and out, mnkl 4096 x 8192 x 512 x 1, 2048 tiles on 148 CTAs, 2 accumulator stages of 128 columns, dynamic persistent scheduler), 100 launches per variant against torch.bmm, with globaltimer probes around the release and the MMA warp's re-acquire of the stage, an MMA-warp variant that stores a sentinel into the just-released stage immediately after its producer_acquire, and a 1 us to 100 us stall of the epilogue warps between the release and the first read.

  • Unpatched: 0/100 launches wrong, 0 sentinel hits in 175,200 probed reads per run. In this caller the MMA warp re-acquires the released stage only 21-24 us after the release (median), because its single-stage dynamic scheduler pipeline is released by the epilogue warps at the end of their tile; the stall moves the acquire by the same amount, so the window is structurally closed there. SASS: LDTM.x128 followed by @P SYNCS.ARRIVE with no scoreboard dependency on the load, the first F2FP after the arrive waiting on it.
  • Patched: 0/100 wrong, output bitwise identical to the unpatched run, same launch time within noise. SASS: no added instruction; the ELECT before the arrive now carries the wait on the load's scoreboard.

The Rubin fp8 GEMM that calls this copy compiles with the change (sm_107a, use_tma_store=False, raster N) and its IR has the tcgen05.wait load between the load and the arrive; it was not run on hardware here. So the race is not observable in the Blackwell caller measured; in a caller whose MMA warp acquires the stage as soon as the barrier flips, nothing in the ISA keeps the window closed. This is a conformance fix at no measurable cost.

Test Coverage

No new test: this is a memory-model conformance fix with no observable failure to test for on current hardware (see above). The helper is exercised by the Rubin fp8 GEMM unit tests that go through cute_dsl_fp8_per_tensor_gemm_rubin (tests/unittest/_torch/kernels/parallel/test_cute_dsl_quant_dense_rubin.py, with the autotuner trying both store modes) and by the kernel's own harness (python -m tensorrt_llm._torch.kernels.rubin.dense_gemm_persistent ... without --use_tma_store, reference check against torch).

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

…ilogue

In the direct-store epilogue helper `epilogue()` of
tensorrt_llm/_torch/kernels/utils/gemm/sm100.py the epilogue warps hand the
accumulator stage back to the MMA warp at the last subtile with an elected
`consumer_release` (an mbarrier arrive) right after the `tcgen05.ld` copy and
before the loaded registers are read (`tTR_rAcc.load()`, scale, convert,
store). There is no `tcgen05.wait::ld` in the kernel, and the MMA warp's
`producer_acquire` waits on exactly that arrive and then issues the next
tile's `tcgen05.mma` into the same stage.

The PTX ISA makes a `tcgen05.ld` complete only at a `tcgen05.wait::ld` in the
issuing thread; a register dependency orders the consuming instruction, not a
later mbarrier arrive. The compiled code shows the arrive issued with no
scoreboard dependency on the load and the first convert waiting on the load
after it.

Insert `cute.arch.fence_view_async_tmem_load()` (`tcgen05.wait::ld`) before
the release, the same ordering the helper's TMA-store sibling and the DSL's
own epilogue helper already use. The wait is on registers the warp consumes a
few instructions later anyway, so no instruction is added; the scoreboard wait
moves onto the arrive.

Verified: the patched helper compiles in the `use_tma_store=False` path of
kernels/rubin/dense_gemm_persistent.py (sm_107a) and the pre-LLVM IR now has
the wait between the load and the arrive. On a B200 the stock and the fenced
Blackwell build of the same helper shape produce bitwise identical output.

Signed-off-by: Bastian Hagedorn <bhagedorn@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant