Skip to content

knn: wait for the TMEM load before releasing the accumulator stage (Blackwell CuTe DSL kernels) - #33

Draft
bastianhagedorn wants to merge 1 commit into
FlashML-org:mainfrom
bastianhagedorn:fix/knn-blackwell-tmem-load-wait
Draft

bastianhagedorn wants to merge 1 commit into
FlashML-org:mainfrom
bastianhagedorn:fix/knn-blackwell-tmem-load-wait

Conversation

@bastianhagedorn

Copy link
Copy Markdown

Summary

In flashlib/primitives/knn/cutedsl/blackwell_impl.py the warps release the single tensor-memory accumulator stage in the instruction after the tcgen05.ld that reads it, before any loaded register is used, and warp 0 then issues the next database tile's MMA into the same columns. This adds the tcgen05.wait::ld (cute.arch.fence_view_async_tmem_load()) the PTX ISA requires between the load and the release, in BlackwellKnnBuild and in the same loop of BlackwellKnnSearchTC, and corrects the comment that said the copy had already drained the accumulator. No functional or performance change; the fence adds no instruction.

Details

Main loop over database tiles: all four warps acc_pipeline.consumer_wait, cute.copy(tiled_copy_t2r, tTR_tAcc[...], tTR_rAcc) (a tcgen05.ld of the 64-column distance tile), acc_pipeline.consumer_release (an mbarrier.arrive on the empty barrier), cute.arch.barrier(); then warp 0 producer_acquires the stage, which waits exactly that empty barrier, sets ACCUMULATE = False and runs the next tile's cute.gemm into the same accumulator; and only then does every warp read frag = tTR_rAcc.load() for the top-K.

tcgen05.ld is asynchronous. The PTX ISA (tcgen05.ld / tcgen05.wait) orders its completion only through tcgen05.wait::ld or a register dependency on the destination registers; the arrive has neither and the CTA barrier orders threads, not the asynchronous load. The documented hand-off is tcgen05.ld, tcgen05.wait::ld, then the arrive. Without it the next tile's MMA may overwrite the 64 columns while the loads are still reading them, and the top-K would consume a mix of two tiles' distances. The exposed values are the whole fragment (64 fp32 per thread).

The fix is one line between the copy and the release:

cute.copy(tiled_copy_t2r, tTR_tAcc[(None, None, None, 0, 0)], tTR_rAcc)
cute.arch.fence_view_async_tmem_load()
acc_pipeline.consumer_release(acc_cons)

The same ordering fix was merged for the same release-before-wait shape in NVIDIA/TensorRT-LLM#21849, flashinfer-ai/flashinfer#8854 and sgl-project/sglang#48616.

Observed behaviour

B200, BlackwellKnnBuild(k=32, num_splits=1) on 8192 x 128 bf16 (64 CTAs, 128 database tiles each): the SASS of the shipped kernel has the releasing SYNCS.ARRIVE as the very next instruction after LDTM.x64, with no scoreboard dependency on the load (the first use of the registers does wait on it); with the fence the arrive carries that dependency and the instruction count is unchanged. Unperturbed, warp 0 commits the refill MMA about 0.6 us after the release; with warps 1-3 stalled for 1-100 us the refill was committed before their first use of the fragment in up to 812 800 of 812 800 probed tiles per run, yet the load had completed first every time and no result was wrong in 900 launches with or without the fence. The ordering is wrong at the ISA level and the window is below the load latency on this part; the fence removes the dependency on that latency at no cost. Both kernels compile with the change; the compiled IR differs from main only by the added tcgen05.wait::ld.

🤖 Generated with Claude Code

In blackwell_impl.py the four warps copy the distance tile from tensor
memory to registers (cute.copy with tiled_copy_t2r, i.e. tcgen05.ld) and
release the single accumulator stage (acc_pipeline.consumer_release) in
the very next instruction, before any loaded register is read. Warp 0
then acquires the stage and issues the next database tile's MMA into the
same columns, and only after that do the warps read the fragment for the
top-K. tcgen05.ld is asynchronous: the PTX ISA (tcgen05.ld /
tcgen05.wait) orders it only through tcgen05.wait::ld or a register
dependency, and the mbarrier arrive has no dependency on the loaded
values, so the refill MMA may overwrite the tile while the load is still
reading it. The comment saying the copy had already drained the
accumulator to registers was not true at the ISA level.

Add cute.arch.fence_view_async_tmem_load() (tcgen05.wait::ld) between
the copy and the release, in BlackwellKnnBuild and in the same loop of
BlackwellKnnSearchTC, and correct the comment.

On a B200 the SASS shows the arrive as the next instruction after
LDTM.x64 with no scoreboard dependency on it; with warps 1-3 stalled the
refill MMA was committed before their first use of the fragment in every
probed tile, but the load had completed by then and no result was wrong
in 900 launches. The fence adds no instruction and removes the
dependency on that latency; the output is bitwise identical.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant