Repository navigation
knn: wait for the TMEM load before releasing the accumulator stage (Blackwell CuTe DSL kernels) - #33
Draft
bastianhagedorn wants to merge 1 commit into
Conversation
In blackwell_impl.py the four warps copy the distance tile from tensor memory to registers (cute.copy with tiled_copy_t2r, i.e. tcgen05.ld) and release the single accumulator stage (acc_pipeline.consumer_release) in the very next instruction, before any loaded register is read. Warp 0 then acquires the stage and issues the next database tile's MMA into the same columns, and only after that do the warps read the fragment for the top-K. tcgen05.ld is asynchronous: the PTX ISA (tcgen05.ld / tcgen05.wait) orders it only through tcgen05.wait::ld or a register dependency, and the mbarrier arrive has no dependency on the loaded values, so the refill MMA may overwrite the tile while the load is still reading it. The comment saying the copy had already drained the accumulator to registers was not true at the ISA level. Add cute.arch.fence_view_async_tmem_load() (tcgen05.wait::ld) between the copy and the release, in BlackwellKnnBuild and in the same loop of BlackwellKnnSearchTC, and correct the comment. On a B200 the SASS shows the arrive as the next instruction after LDTM.x64 with no scoreboard dependency on it; with warps 1-3 stalled the refill MMA was committed before their first use of the fragment in every probed tile, but the load had completed by then and no result was wrong in 900 launches. The fence adds no instruction and removes the dependency on that latency; the output is bitwise identical. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
In
flashlib/primitives/knn/cutedsl/blackwell_impl.pythe warps release the single tensor-memory accumulator stage in the instruction after thetcgen05.ldthat reads it, before any loaded register is used, and warp 0 then issues the next database tile's MMA into the same columns. This adds thetcgen05.wait::ld(cute.arch.fence_view_async_tmem_load()) the PTX ISA requires between the load and the release, inBlackwellKnnBuildand in the same loop ofBlackwellKnnSearchTC, and corrects the comment that said the copy had already drained the accumulator. No functional or performance change; the fence adds no instruction.Details
Main loop over database tiles: all four warps
acc_pipeline.consumer_wait,cute.copy(tiled_copy_t2r, tTR_tAcc[...], tTR_rAcc)(atcgen05.ldof the 64-column distance tile),acc_pipeline.consumer_release(anmbarrier.arriveon the empty barrier),cute.arch.barrier(); then warp 0producer_acquires the stage, which waits exactly that empty barrier, setsACCUMULATE = Falseand runs the next tile'scute.gemminto the same accumulator; and only then does every warp readfrag = tTR_rAcc.load()for the top-K.tcgen05.ldis asynchronous. The PTX ISA (tcgen05.ld/tcgen05.wait) orders its completion only throughtcgen05.wait::ldor a register dependency on the destination registers; the arrive has neither and the CTA barrier orders threads, not the asynchronous load. The documented hand-off istcgen05.ld,tcgen05.wait::ld, then the arrive. Without it the next tile's MMA may overwrite the 64 columns while the loads are still reading them, and the top-K would consume a mix of two tiles' distances. The exposed values are the whole fragment (64 fp32 per thread).The fix is one line between the copy and the release:
The same ordering fix was merged for the same release-before-wait shape in NVIDIA/TensorRT-LLM#21849, flashinfer-ai/flashinfer#8854 and sgl-project/sglang#48616.
Observed behaviour
B200,
BlackwellKnnBuild(k=32, num_splits=1)on 8192 x 128 bf16 (64 CTAs, 128 database tiles each): the SASS of the shipped kernel has the releasingSYNCS.ARRIVEas the very next instruction afterLDTM.x64, with no scoreboard dependency on the load (the first use of the registers does wait on it); with the fence the arrive carries that dependency and the instruction count is unchanged. Unperturbed, warp 0 commits the refill MMA about 0.6 us after the release; with warps 1-3 stalled for 1-100 us the refill was committed before their first use of the fragment in up to 812 800 of 812 800 probed tiles per run, yet the load had completed first every time and no result was wrong in 900 launches with or without the fence. The ordering is wrong at the ISA level and the window is below the load latency on this part; the fence removes the dependency on that latency at no cost. Both kernels compile with the change; the compiled IR differs frommainonly by the addedtcgen05.wait::ld.🤖 Generated with Claude Code