Skip to content

[None][fix] Order TMEM dealloc and split-K mailbox reads in the GLM-5.2 high-throughput split-K GEMM - #22714

Draft
bastianhagedorn wants to merge 1 commit into
NVIDIA:mainfrom
bastianhagedorn:fix/glm-ht-sm107-splitk-sync
Draft

bastianhagedorn wants to merge 1 commit into
NVIDIA:mainfrom
bastianhagedorn:fix/glm-ht-sm107-splitk-sync

Conversation

@bastianhagedorn

Copy link
Copy Markdown
Contributor

Description

Two ordering defects in glm_5_2_nvfp4__sm_107__high_throughput/kernels/gemm/splitk.py (SplitKDenseGemmKernel, all tactics).

1. TMEM deallocation before the epilogue's loads are done. bar_tmem_alloc is initialised with 160 arrivals per phase: the MMA warp's 32 plus the 128 epilogue threads'. The MMA warp arrives after alloc_tmem, runs the K loop, arrives a second time, waits for parity 1 and deallocates. The epilogue threads arrive, wait for phase 0 (the published TMEM base), load the accumulator with tcgen05.ld, fence the load (fence_view_async_tmem_load) and arrive again. If the MMA warp's second arrival lands while phase 0 is still open (the epilogue threads have not all arrived yet), the parity-1 wait names the phase before phase 0 and returns at once: per the PTX ISA (mbarrier.test_wait / mbarrier.try_wait) the parity operand selects either the current phase or the immediately preceding one, and the preceding phase of a freshly initialised barrier counts as completed. dealloc_tmem then runs while the epilogue warps' tcgen05.ld may still be in flight on those columns. The MMA warp now waits for phase 0 before its second arrival, so the parity-1 wait can only name the phase that the epilogue's post-load arrivals complete. Same pattern and fix as #21849 (TGV GEMM) and flashinfer-ai/flashinfer#8787.

2. CTA-scope acquire on the split-K mailbox. The owner's epilogue threads wait for bar_reduce with cute.arch.mbarrier_wait, which acquires at CTA scope, and then read mailbox[...]. The barrier is completed by the peer CTAs' st.async.shared::cluster.mbarrier::complete_tx::bytes (_store_shared_remote_v4), whose complete-tx the PTX ISA defines as a release at cluster scope. Under the PTX memory consistency model a release and an acquire synchronize only when each specifies a scope that includes the thread executing the other, so the mailbox reads are not ordered after the peers' stores. The wait now uses the file's own _wait_parity_cluster helper (mbarrier.try_wait.parity.acquire.cluster.shared::cta.b64), which the kernel already uses for bar_mailbox_free in the mailbox_in_stages configurations. Same defect and fix: NVIDIA/TensorRT-LLM #21843, #21846, flashinfer-ai/flashinfer#7443, #8853.

Hardware verification

The patched kernel was compiled for sm_107a at the shipped tactics (splitk_default_tactic m8 n256 k6144, the HnormB draft path, and the LamportPublish o_proj path) and the generated IR shows the added phase-0 wait before the MMA warp's second arrival and the cluster-scope acquire at the mailbox wait. Not run on hardware this round. Neither defect has produced an observed wrong result: the MMA warp's second arrival follows the whole K loop, which in practice ends long after the epilogue's first arrivals, and payload and complete-tx travel together on the DSMEM path. The ISA promises neither, and both fixes are one wait each with no measurable cost (the phase-0 wait is already satisfied when reached; the cluster acquire is the same instruction the kernel already issues on the bar_mailbox_free path).

Test Coverage

Existing model tests of the glm_5_2_nvfp4__sm_107__high_throughput package exercise the kernel through ops/gemm/low_m.py and ops/attention/projection.py; there is no dedicated unit test for splitk.py. No new test: ordering fixes with no deterministic failure to test for.

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
  • Test cases are provided for new code paths (see test instructions)
  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
  • Any new dependencies have been scanned for license and vulnerabilities
  • CODEOWNERS updated if ownership changes

….2 high-throughput split-K GEMM

Two ordering defects in splitk.py. (1) bar_tmem_alloc takes the MMA
warp's 32 arrivals plus the 128 epilogue arrivals per phase. The MMA
warp arrived after alloc_tmem, ran the K loop, arrived a second time and
waited for parity 1 before dealloc_tmem. When the second arrival lands
while phase 0 is still open, the parity-1 wait names the phase before
phase 0, returns at once, and the dealloc races the epilogue warps'
tcgen05.ld of the accumulator. The MMA warp now waits for phase 0 before
its second arrival. (2) The owner's epilogue threads waited for
bar_reduce with cute.arch.mbarrier_wait, a CTA-scope acquire, while the
barrier is completed by the peers' st.async complete-tx, a release at
cluster scope, so the mailbox reads after the wait were not ordered
after the peers' stores under the PTX memory consistency model. The wait
now uses the file's _wait_parity_cluster helper, which acquires at
cluster scope.

Signed-off-by: Bastian Hagedorn <bhagedorn@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant