Repository navigation
[None][fix] Order TMEM dealloc and split-K mailbox reads in the GLM-5.2 high-throughput split-K GEMM - #22714
Draft
bastianhagedorn wants to merge 1 commit into
Draft
[None][fix] Order TMEM dealloc and split-K mailbox reads in the GLM-5.2 high-throughput split-K GEMM#22714bastianhagedorn wants to merge 1 commit into
bastianhagedorn wants to merge 1 commit into
Conversation
….2 high-throughput split-K GEMM Two ordering defects in splitk.py. (1) bar_tmem_alloc takes the MMA warp's 32 arrivals plus the 128 epilogue arrivals per phase. The MMA warp arrived after alloc_tmem, ran the K loop, arrived a second time and waited for parity 1 before dealloc_tmem. When the second arrival lands while phase 0 is still open, the parity-1 wait names the phase before phase 0, returns at once, and the dealloc races the epilogue warps' tcgen05.ld of the accumulator. The MMA warp now waits for phase 0 before its second arrival. (2) The owner's epilogue threads waited for bar_reduce with cute.arch.mbarrier_wait, a CTA-scope acquire, while the barrier is completed by the peers' st.async complete-tx, a release at cluster scope, so the mailbox reads after the wait were not ordered after the peers' stores under the PTX memory consistency model. The wait now uses the file's _wait_parity_cluster helper, which acquires at cluster scope. Signed-off-by: Bastian Hagedorn <bhagedorn@nvidia.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Two ordering defects in
glm_5_2_nvfp4__sm_107__high_throughput/kernels/gemm/splitk.py(SplitKDenseGemmKernel, all tactics).1. TMEM deallocation before the epilogue's loads are done.
bar_tmem_allocis initialised with 160 arrivals per phase: the MMA warp's 32 plus the 128 epilogue threads'. The MMA warp arrives afteralloc_tmem, runs the K loop, arrives a second time, waits for parity 1 and deallocates. The epilogue threads arrive, wait for phase 0 (the published TMEM base), load the accumulator withtcgen05.ld, fence the load (fence_view_async_tmem_load) and arrive again. If the MMA warp's second arrival lands while phase 0 is still open (the epilogue threads have not all arrived yet), the parity-1 wait names the phase before phase 0 and returns at once: per the PTX ISA (mbarrier.test_wait/mbarrier.try_wait) the parity operand selects either the current phase or the immediately preceding one, and the preceding phase of a freshly initialised barrier counts as completed.dealloc_tmemthen runs while the epilogue warps'tcgen05.ldmay still be in flight on those columns. The MMA warp now waits for phase 0 before its second arrival, so the parity-1 wait can only name the phase that the epilogue's post-load arrivals complete. Same pattern and fix as #21849 (TGV GEMM) and flashinfer-ai/flashinfer#8787.2. CTA-scope acquire on the split-K mailbox. The owner's epilogue threads wait for
bar_reducewithcute.arch.mbarrier_wait, which acquires at CTA scope, and then readmailbox[...]. The barrier is completed by the peer CTAs'st.async.shared::cluster.mbarrier::complete_tx::bytes(_store_shared_remote_v4), whose complete-tx the PTX ISA defines as a release at cluster scope. Under the PTX memory consistency model a release and an acquire synchronize only when each specifies a scope that includes the thread executing the other, so the mailbox reads are not ordered after the peers' stores. The wait now uses the file's own_wait_parity_clusterhelper (mbarrier.try_wait.parity.acquire.cluster.shared::cta.b64), which the kernel already uses forbar_mailbox_freein themailbox_in_stagesconfigurations. Same defect and fix: NVIDIA/TensorRT-LLM #21843, #21846, flashinfer-ai/flashinfer#7443, #8853.Hardware verification
The patched kernel was compiled for sm_107a at the shipped tactics (
splitk_default_tacticm8 n256 k6144, the HnormB draft path, and the LamportPublish o_proj path) and the generated IR shows the added phase-0 wait before the MMA warp's second arrival and the cluster-scope acquire at the mailbox wait. Not run on hardware this round. Neither defect has produced an observed wrong result: the MMA warp's second arrival follows the whole K loop, which in practice ends long after the epilogue's first arrivals, and payload and complete-tx travel together on the DSMEM path. The ISA promises neither, and both fixes are one wait each with no measurable cost (the phase-0 wait is already satisfied when reached; the cluster acquire is the same instruction the kernel already issues on thebar_mailbox_freepath).Test Coverage
Existing model tests of the
glm_5_2_nvfp4__sm_107__high_throughputpackage exercise the kernel throughops/gemm/low_m.pyandops/attention/projection.py; there is no dedicated unit test forsplitk.py. No new test: ordering fixes with no deterministic failure to test for.PR Checklist
Please review the following before submitting your PR:
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.