cuda: AMD RDNA4 Q1_0/Q2_0 — HIP-path vec_dots (+37%/+16% decode), opt-in quant dedup, opt-in hipBLASLt prefill routes - #116
Conversation
…in hipBLASLt prefill routes
HIP-guarded vec_dot_q2_0_q8_1 (dynamic-selector __byte_perm lowers poorly on
AMD): Bonsai-27B Q2_0 decode 34.52 -> 47.20 t/s on gfx1201, PPL byte-identical.
Opt-in GGML_HIP_DEDUP_MMVQ_QUANT skips redundant sibling activation quantize
launches (+2-3% decode here, more under spec-decode verify). Opt-in
GGML_HIP_{Q1_0,Q2_0}_HIPBLASLT_PREFILL routes large-M matmuls to hipBLASLt
int8 GEMM with a self-tuning algo cache and buffer-free weight-cache
invalidation: pp1024 +37.5% (Q2_0) / +17.8% (Q1_0) warm. hipBLASLt is an
optional dependency; without it (Windows HIP) and on CUDA builds the routes
compile to inert stubs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ode)
Same treatment as the Q2_0 HIP path: bit-spread raw code bits c in {0,1} into
{0,1}-byte dp4a operands and fold dot(s,u) = 2*dot(c,u) - sum(u) through the
q8_1 stored sum, replacing the select-chain fallback in unpack_q1_0_bytes.
Bonsai-27B Q1_0 tg128 on 1x R9700: 54.53 -> 63.03 t/s (64.93 with
GGML_HIP_DEDUP_MMVQ_QUANT=1). test-backend-ops MUL_MAT: all pass. CUDA path
untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The-Monk
changed the title
|
@The-Monk Thanks for the chnages, will do a more careful review this week. By the way Q1_0 is fully upstreamed to llama.cpp so migth be worth also trying to upstream it there, so check whetehr they have added better AMD support since we merged things. Happy to merge these here but also worst trying to upstream to main llama.cpp as well since most things are already there too (with the caveat of Q2_0 being slightly different there). |
There was a problem hiding this comment.
Pull request overview
Adds RDNA4 optimizations for Q1_0/Q2_0 inference in the shared CUDA/HIP backend.
Changes:
- Optimizes HIP vector-dot decoding.
- Adds optional activation-quantization deduplication.
- Adds optional hipBLASLt prefill routes, tuning, and weight caches.
Policy note: The PR description declares Claude Code generation, conflicting with AGENTS.md:36-40, which prohibits AI-written PR descriptions.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
ggml/src/ggml-hip/CMakeLists.txt |
Adds optional hipBLASLt discovery. |
ggml/src/ggml-cuda/vecdotq.cuh |
Adds HIP/MUSA low-bit vector-dot paths. |
ggml/src/ggml-cuda/mul_mat_q2_0_hipblaslt.cuh |
Declares the Q2_0 prefill route. |
ggml/src/ggml-cuda/mul_mat_q2_0_hipblaslt.cu |
Implements Q2_0 hipBLASLt prefill. |
ggml/src/ggml-cuda/mul_mat_q1_0_hipblaslt.cuh |
Declares the Q1_0 prefill route. |
ggml/src/ggml-cuda/mul_mat_q1_0_hipblaslt.cu |
Implements Q1_0 hipBLASLt prefill. |
ggml/src/ggml-cuda/mmvq.cu |
Adds activation-quantization deduplication. |
ggml/src/ggml-cuda/hipblaslt_wcache.cuh |
Declares cache invalidation APIs. |
ggml/src/ggml-cuda/hipblaslt_wcache.cu |
Implements invalidator registration. |
ggml/src/ggml-cuda/ggml-cuda.cu |
Integrates routes and cache lifecycle. |
ggml/src/ggml-cuda/common.cuh |
Adds FP8 conversion and cache state. |
Suppressed comments (2)
ggml/src/ggml-cuda/mul_mat_q1_0_hipblaslt.cu:415
- The claim that all uses share one stream is false: the backend dispatches concurrent branches and devices independently. Both cache builders publish the entry immediately after enqueueing requantization, so another stream can hit and read it before initialization completes. Publish a completion event and wait on hits, or synchronize before insertion as the existing Hopper cache does at
mmq-hopper-q1.cu:149-153.
// Returns cached int8 weight (building it on first miss if within budget), or
// nullptr -> caller must requant on-the-fly. Build + all uses share the stream,
// so the one-time requant is correctly ordered before any GEMM that reads it.
ggml/src/ggml-cuda/mul_mat_q2_0_hipblaslt.cu:425
- The cache entry is published immediately after requantization is enqueued. A concurrent branch or request on another stream can hit this entry and launch GEMM before the converted weights are initialized. Publish a completion event and wait on hits, or synchronize before insertion as the existing Hopper cache does at
mmq-hopper-q1.cu:149-153.
g_wcache_bytes += need;
auto res = g_wcache.emplace(key, c);
return &res.first->second;
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| const int qs0 = bq2_0->qs[offset + 0] | (bq2_0->qs[offset + 1] << 8) | | ||
| (bq2_0->qs[offset + 2] << 16) | (bq2_0->qs[offset + 3] << 24); | ||
| const int qs1 = bq2_0->qs[offset + 4] | (bq2_0->qs[offset + 5] << 8) | | ||
| (bq2_0->qs[offset + 6] << 16) | (bq2_0->qs[offset + 7] << 24); |
| const bool dedup_hit = dedup_quant && | ||
| ctx.mmvq_quant_cache_tensor == src1 && ctx.mmvq_quant_cache_buf; |
| int man = (int) (ax * 512.0f + 0.5f); | ||
| if (man > 7) { | ||
| man = 7; | ||
| } |
| if (src0->type != GGML_TYPE_Q1_0) return false; | ||
| if (src1->type != GGML_TYPE_F32 || dst->type != GGML_TYPE_F32) return false; | ||
| if (src0->ne[2] != 1 || src0->ne[3] != 1) return false; | ||
| if (src1->ne[2] != 1 || src1->ne[3] != 1) return false; | ||
| if (src0->ne[0] != src1->ne[0] || src0->ne[0] % Q1K != 0) return false; |
| if (src0->type != GGML_TYPE_Q2_0) return false; | ||
| if (src1->type != GGML_TYPE_F32 || dst->type != GGML_TYPE_F32) return false; | ||
| if (src0->ne[2] != 1 || src0->ne[3] != 1) return false; | ||
| if (src1->ne[2] != 1 || src1->ne[3] != 1) return false; | ||
| if (src0->ne[0] != src1->ne[0] || src0->ne[0] % Q2K != 0) return false; |
| std::map<const void *, cached_w> g_wcache_i8; | ||
| std::map<const void *, cached_w_f8> g_wcache_f8; |
| // cached (requant paid once), the rest fall back to on-the-fly pool requant. | ||
| // hipMalloc failure also falls back -- never OOM-crash (the Stage-3 lesson). | ||
| struct cached_w { int8_t * q8 = nullptr; float * wscale = nullptr; size_t bytes = 0; }; | ||
| std::map<const void *, cached_w> g_wcache; |
| // ---- activation int8/e4m3 (per-token) + 4-byte accumulator, from the pool ---- | ||
| ggml_cuda_pool_alloc<int8_t> x8 (ctx.pool(), (size_t)K * M); | ||
| ggml_cuda_pool_alloc<float> asc (ctx.pool(), (size_t)M); | ||
| ggml_cuda_pool_alloc<int32_t> acc (ctx.pool(), (size_t)N * M); // i32 (int8) or reinterpreted f32 (fp8) |
Review hardening (Copilot flag on the converted-weight caches): entries are published under the mutex right after the requant kernel is launched, so ordering vs the GEMM that reads them was guaranteed only for same-stream consumers. Record a build_done event on the build stream and make any consumer on a different stream hipStreamWaitEvent on it before using the entry (falling back to on-the-fly requant if the wait cannot be issued). Events are destroyed with their entries in the invalidators. No behavior change on the current single-compute-stream-per-device backend (the wait never fires); the previously documented invariant is now enforced. Verified: build clean, both routes smoke-tested on 1x R9700 (Q1_0 pp1024 1500 t/s, Q2_0 1487 t/s, build+hit paths exercised). CCA (Claude Code Augmented)
Five fixes from the 2026-08-11 review pass: - vecdotq: pack the Q2_0 HIP-path qs bytes as unsigned. A byte >= 128 shifted by 24 overflows the promoted int (UB); build qs0/qs1 as uint32_t so optimized HIP/MUSA builds cannot miscompile the high byte. - common: fix fp32->E4M3 rounding at the subnormal/normal boundary. A subnormal mantissa that rounds to 8 is exactly 2^-6, the minimum normal (encoding 0x08); clamping it to mantissa 7 returned a non-nearest value. - mmvq: make the opt-in activation-quant dedup cache stream-safe. Concurrent graph regions fork sibling matmuls onto separate streams; a hit could consume the cached q8_1 before the populating quantize (enqueued on the miss stream) completed. Record an event after the populating quantize and make hits on any other stream wait on it (same treatment the hipblaslt wcache got in 65cd5bd). The event is destroyed with the context. - hipblaslt: key both weight caches by (device, addr, N, K) instead of the raw device address, which multi-GPU runs and reshaped views can alias. Invalidation matches on address across devices (a false positive only costs a redundant re-requant). - hipblaslt: give the Q1_0 route the Q2_0 route's VRAM preflights: the in-cache headroom check before hipMalloc and the op-level transient guard (weight-miss + activation + accumulator + workspace) so a near-full card falls back to mmq instead of OOM-aborting in the pool. Verified on 1x R9700 (Bonsai-27B): test-backend-ops MUL_MAT passes plain and with dedup+batch; PPL wiki.test 20 chunks identical with dedup on/off (11.5603); int8/fp8 prefill routes run with the expected uplift over the same-build no-route baseline (Q2_0 797->1178 t/s pp1024 warm, Q1_0 1022->1169) and sane PPL (int8 11.7646, fp8 11.6337). CCA (Claude Code Augmented)
… Q2_0 prefill)
Two findings from PC-sampling mul_mat_q on gfx1201, both replacing
unpack sequences on the HIP/MUSA path with borrow-free SWAR byte maps.
Outputs are bit-identical by construction in both cases -- prefill
perplexity reproduces exactly (Q1_0 11.6466, Q2_0 10.1465, wikitext
ub-default ch4) -- so the changes are pure performance.
1. unpack_q1_0_bytes HIP fallback: the per-bit `(bit ? 1 : -1)`
ternaries compile to v_cmp_eq + v_cndmask chains that serialize on
VCC (PC-sampled at 9.5% of mul_mat_q), and their live ranges push
the mmq_x=128 tile to the 256-VGPR ISA cap with scratch spills in
the hot loop (76B/lane; 188B on the need_check variant). Replaced
with bit-spread to {0,1} bytes (3 fused lshl_or + and) then a
carry-free byte map 1 -> 0x01 / 0 -> 0xFF via
((spread << 1) + 0x7F7F7F7F) ^ 0x80808080. On an equivalent tree
this also eliminated the spills outright (vgpr 256 -> 239, scratch
-> 0); the measured alternative of capping mmq_x at the spill-free
96 tile costs 8-9% pp, so the SWAR keeps the large tile AND drops
the spills.
2. load_tiles_q2_0's 4x __byte_perm crumb unpack: correct on HIP (PPL
verified), but HIP's __byte_perm emulates PRMT's nibble-selector
convention with a runtime control-word conversion before
v_perm_b32, so each call costs several VALU ops here rather than
one instruction. The SWAR equivalent (2-bit spread then c-1 per
byte, carry-free) produces the same linear crumbs-minus-one
ordering the perm/unshuffle pair computes, HIP/MUSA-guarded; the
CUDA path keeps the perms.
Measured (Bonsai-27B, 1x Radeon AI PRO R9700 gfx1201, ROCm 7.14,
same-session A/B on this branch):
- Q1_0 pp2048 1252.5 -> 1327.6 t/s (+6.0%, r=5); tg128 unchanged
- Q2_0 pp2048 1036.2 -> 1310.0 t/s (+26.4%, r=3); tg128 unchanged
- prefill PPL bit-exact both quants (gates above)
Note for CUDA reviewers: the pre-existing unguarded __byte_perm q2
unpack feeds raw crumb pairs as PRMT selector nibbles; the pair
(2,2) produces selector value 10, which on NVIDIA engages PRMT's
sign-replicate mode rather than a table lookup. Worth a quick check
on CUDA hardware that qx/qy are as intended there; the HIP path no
longer depends on it either way.
CCA (Claude Code Augmented)
|
Before the careful review starts: pushed one more commit ( 1. 2. The q2 Measured (Bonsai-27B, 1× Radeon AI PRO R9700 gfx1201, ROCm 7.14, same-session A/B on this branch):
The "our fork" column is the roc10 tree these techniques were developed on, same R9700/ROCm 7.14 — with this commit the branch's prefill is now at parity with it. The remaining decode gap (62 vs 68 Q1_0, 47 vs 52 Q2_0) comes from launch-geometry and kernel-fusion tuning on that tree; those constants are kernel-implementation-specific and don't transfer by copy (we measured nwarps=8 cratering this branch's kernel to 8 t/s in the original PR round), so porting them properly is a future round rather than part of this one. One question for anyone with NVIDIA hardware handy: the pre-existing (unguarded) q2 perm unpack feeds raw crumb pairs as PRMT selector nibbles. The ternary pair Full run provenance (stack/firmware/clock telemetry) captured as bench cards; happy to attach if useful. The same two techniques are on their way upstream in generic form for mainline's ternary types (ggml-org#27127). |
|
Follow-up on the upstream-to-mainline suggestion — we took the SWAR findings to mainline master and measured, and the case there turns out to be much bigger than on this fork. Mainline baseline on AMD is running at less than half of what the types can do. On gfx1201 (Radeon AI PRO R9700, ROCm 7.14), ggml-org/llama.cpp master with the 27B Q1_0 model:
Root cause is the same The fix is the same ~60-line, four-site SWAR patch as Our recommendation: this should go upstream under PrismML's name. You authored and upstreamed the types; the kernels are your architecture, and a doubling of AMD decode for them lands best coming from the type owners. Take the branch or the diff as-is, restructure freely, no attribution needed beyond whatever's natural. One heads-up for whoever submits: mainline's CONTRIBUTING.md AI policy requires the submitting author's own comprehensive manual review, an explicit disclosure of how AI was used (these kernels were developed and A/B-tested with AI assistance on our hardware), and a personally-written PR description. If you'd rather we submit it ourselves with you reviewing, that works too — but our preference is that it ships under the flag of the people who built the types. |
|
Completing the mainline table with Q2_0 — we requantized the 27B ternary model to mainline's group-64 Q2_0 (via F16; near-lossless for ternary values, PPL 10.1383 vs 10.1465 on this fork's g128) and ran the same A/B on
So the branch is fully characterized on both types: four bit-exact gates, |
|
One more dataset for the upstream decision, since we hold the AMD hardware in this collaboration: we measured the full ternary scale-granularity curve on Bonsai-27B — including mainline's TQ2_0, which is byte-for-byte the "g256" point — so the format question for mainline distribution can be decided on data rather than taste. Everything below is from our instrumented rig (1× Radeon AI PRO R9700 gfx1201, ROCm 7.14; methodology notes at the end). Quality vs scale granularity (same model content, requantized per format)
Exception-tensor accounting, since it matters: the g64 and TQ2_0 files carry q4_K/q6_K token-embd/output (mainline's rule), the g128 and g1024 files carry the fork's q2_0 ones. The decisive cell is therefore TQ2_0 vs g64 at identical exceptions: −0.84% for TQ2_0 — the advantage is the format/quantizer itself. (ch4 quick-pass numbers told the same ordering but sit inside their ±0.95 SE; ch20 is the number to trust. CPU-vs-GPU evaluation cross-checked on g128: within ±0.01.) Why very coarse groups lose — and why g256 doesn't. We analyzed the shipped g128 GGUF's scale structure directly (per-tensor, per-row block statistics over all 498 ternary tensors):
Amax-based coarse scales zero out survivors in low-magnitude groups (any group whose local scale sits under ~half the merged amax loses its ±1s to rounding). Merging two g128 groups (→g256) stays inside the tolerable band; merging eight (→g1024) crosses it and costs +6.4% even with an MSE-optimal magnitude second pass (d = mean |w| over survivors, codes preserved) — the within-row variance is trained signal. We built the g1024 type end-to-end (CPU+HIP kernels, two-pass quantizer) so that number is a measurement, not an extrapolation; it also ran 1.3% slower than g128 at bs=1 because the byte savings don't survive untuned launch geometry. The quality plateau runs from g64 through g256 and falls off a cliff before g1024 — and TQ2_0 sits at the cheap end of the plateau. AMD performance by format (same GPU, post-SWAR kernels everywhere)
Roofline context for the decode column: our measured pure-read ceiling on this card is 640.0 GB/s (custom streaming-kernel calibration, reproduced ×2 — we use measured, not datasheet), and the g128 decode kernel runs at ~87% of it with exact per-model byte accounting from GGUF metadata — i.e., ternary decode on RDNA4 is memory-bound, which is why bytes-per-weight dominates every row of that table and why sub-percent bpw differences show up in tg. What this means for the upstream/distribution question
Methodology: PPLs are wikitext-2, 4×512 chunks, prefill-path unless marked; quality comparisons use requants of the identical g128 source through F16 (lossless for ternary values); TQ2_0's PPL is CPU-evaluated (no GPU path exists) with a same-config CPU-vs-GPU calibration run on g128 agreeing within ±0.01 PPL. Decode A/Bs are same-session interleaved with VRAM-drain guards; perf numbers are r=3–5 llama-bench with reported ±. |
|
Follow-up from the AMD measurement series — this one is a capability proposal rather than a kernel finding. Proposal: ship a nextn/MTP head for Bonsai-27B. We validated MTP self-speculation on AMD (llama.cpp A Bonsai-distilled head (1 transformer layer + eh_proj/norms, ~400 MB bf16/fp8, trained against Bonsai hidden states) should recover the ~50%+ accept that makes speculation pay on fast ternary targets; projected ~1.5× effective decode on top of all the kernel wins, on every GPU vendor — spec decode multiplies whatever the platform baseline is. We can supply the gguf graft tooling (drop-in file format already worked out) and same-day AMD benchmarks for any candidate head: |
|
Update — measured result worth flagging: the stock Qwen3.6 MTP head is already net-positive on Bonsai-27B Q2_0, no training needed.
(RDNA4 R9700, temp 0 — greedy verify is exact-match, so output is bit-identical to plain greedy decode.) The split is informative: Q2_0 preserves the ternary master faithfully and the head recognizes it; Q1_0s binarization drifts the distribution and accept drops below breakeven on a 67 t/s target. Which sharpens the earlier proposal: a Bonsai-distilled head should lift Q1_0 into the same ~50% accept band (projected ~67 → 85+), and likely push Q2_0 further still. Meanwhile the Q2_0 +21% is available to your users today — the graft is 15 tensors (blk.64.* from any Qwen3.6-27B MTP gguf) + block_count 65 + nextn_predict_layers=1; happy to share the script. |
…xact) The existing fast path requires VNNI (Alder Lake / Zen 4+, 2021+); every older AVX2 CPU (Haswell 2013 through Zen 3) fell to the scalar loop. This adds an #elif AVX2 tier: pshufb-replicate + nibble-LUT crumb decode, maddubs with the dot(c-1,u) = dot(c,u) - sum(u) fold. Bonsai-8B Q2_0, dual E5-2699v4 (AVX2, no VNNI), t=11: scalar 1.81 t/s -> 5.90 t/s (3.26x), PPL 10.4508 bit-exact both ways. CCA (Claude Code Augmented)
|
Pushed one more commit to the branch: AVX2 tier for the Q2_0 CPU vec_dot. The current fast path is gated on VNNI ( Measured (Bonsai-8B Q2_0, dual Xeon E5-2699 v4 = AVX2-only, 11 threads): scalar 1.81 t/s → 5.90 t/s (3.26×), PPL bit-exact (10.4508 both ways, 2-chunk gate). One observation on the existing VNNI tier while we were in there (not changed — we have no VNNI hardware to validate on): it horizontal-sums both dpbusd results to scalar every 32 elements ( Thread-scaling note for CPU users: on the 2-socket test box, t=11 beat t=44 (6.0 vs 4.5 t/s) — per-op sync dominates at high thread counts on many-core machines; worth a mention in docs. |
What this is
The AMD side of the Q1_0/Q2_0 story. This fork's HIP release bundles currently run Prism's binary/ternary formats through kernels tuned for NVIDIA; on RDNA4 hardware that leaves a lot on the table. This PR contributes three self-contained pieces from our RDNA4 fork (The-Monk/llama.cpp
roc8, the kernel base of The Rock8), re-measured on this tree on 2× Radeon AI PRO R9700 (gfx1201, ROCm/TheRock 7.14).All numbers below: Bonsai-27B (
qwen35hybrid), 1 GPU,llama-bench -r 5(tg) /-r 3(pp).1. HIP-path
vec_dotfor Q2_0 (+37%) and Q1_0 (+16%), on by defaultThe existing Q2_0 vec_dot extracts symbols with dynamic-selector
__byte_permchains, which are built around NVIDIA'sPRMT; on HIP they lower poorly and the decode GEMV pays for it end-to-end. This PR adds an AMD path using the same split this file already uses forunpack_q1_0_bytes: bit-spread the raw 2-bit codes into bytes and use the identitydot(s,u) = dot(c,u) − sum(u), applying the offset once via the q8_1 stored sum — plain shift/mask +dp4a, no per-code arithmetic. The CUDA path is untouched.The second commit gives Q1_0 the same treatment with the binary form of the identity (
s = 2c − 1⇒dot(s,u) = 2·dot(c,u) − sum(u)), replacing the select-chain fallback inunpack_q1_0_bytesthat materialized ±1 bytes on a VALU-bound kernel.Perplexity is identical (wikitext-2, 20 chunks:
11.5603 ± 0.47269both builds, Q2_0), and binary now decodes faster than ternary, as its byte ratio says it should.2. Opt-in activation-quant dedup (
GGML_HIP_DEDUP_MMVQ_QUANT), default OFFSibling mmvq matmuls that read the same activation tensor (wq/wk/wv, ffn_gate/ffn_up, wqkv/wqkv_gate) each launch a
quantize_row_q8_1that recomputes byte-identical output. With the flag set, the quantization is computed once per activation tensor and reused; the cache lives on the backend context, is keyed by tensor pointer, and is reset at everygraph_compute, so entries cannot survive a graph rebuild.GGML_HIP_DEDUP_MMVQ_QUANT_BATCHadditionally widens it tone11>1(spec-decode verify batches).Measured here: Q2_0 48.20 (+2.1% on top of #1), Q1_0 64.93 (+3.0% on top of the new Q1_0 dot). On our fork the same lever is worth considerably more under MTP self-speculative decode, where the verify pass multiplies the redundancy — that's why the
_BATCHvariant exists. Lossless: the cached bytes are the bytes the skipped launch would have produced; PPL run with both flags on is byte-identical to stock.3. Opt-in Q1_0/Q2_0 hipBLASLt prefill routes, default OFF, optional dependency
Large-M (prefill) matmuls routed through hipBLASLt int8 GEMM (dequantize once → cached int8 weights + per-channel scale; activations int8 per call), with a self-tuning per-shape algorithm cache — gfx1201 has no hipBLASLt cost model and the heuristic picks badly. Includes the weight-cache invalidation registry (
hipblaslt_wcache): caches are keyed on device addresses, so the backend drops entries when it frees a buffer — without this, multi-model runs and server model swaps can silently serve stale weights from a reused allocation.Knobs per format (
<FMT>∈Q1_0,Q2_0):GGML_HIP_<FMT>_HIPBLASLT_PREFILL=1(enable),_MTHRESH=<n>(default 384 — deliberately conservative because the crossover is model-shape-dependent; override per model),_TUNE_CACHE=<path>,_NOTUNE=1,_WCACHE_MB=<n>,_FP8=1(dequantize to e4m3 instead of int8).Build impact:
find_package(hipblaslt QUIET)— optional. Not found (e.g. Windows HIP, where hipBLASLt doesn't exist) → routes compile as inert stubs and the release bundles are unaffected. CUDA builds compile the same stubs (__HIP_PLATFORM_AMD__-guarded).What we tried and deliberately did NOT include
Our fork runs Q2_0 with
nwarps=8/rows_per_block=3on RDNA4. Transplanting those constants onto this tree's kernel was a disaster (nwarps=8 → 8.2 t/s) to neutral (full sweep: rpb 2/3/4 × nwarps 2/4/8 all ≤ +0.7% over stock). Launch-geometry constants do not transfer between kernel implementations; this tree's geometry is already at its optimum, so no tuning changes are included.Testing
test-backend-ops test -o MUL_MAT: all pass, both with default env and with both dedup flags set.-DCMAKE_HIP_ARCHITECTURES=gfx1201.Reproduce:
cmake -B build -DGGML_HIP=ON -DCMAKE_HIP_ARCHITECTURES=gfx1201 -DCMAKE_BUILD_TYPE=Release cmake --build build -j --target llama-bench test-backend-ops ./build/bin/llama-bench -m Ternary-Bonsai-27B-Q2_0.gguf -ngl 999 -p 0 -n 128 -r 5 GGML_HIP_DEDUP_MMVQ_QUANT=1 ./build/bin/llama-bench -m ... GGML_HIP_Q2_0_HIPBLASLT_PREFILL=1 GGML_HIP_Q2_0_HIPBLASLT_TUNE_CACHE=/tmp/t.bin \ ./build/bin/llama-bench -m ... -p 1024 -n 0 # run twice; first run pays the tuning costStanding offer
The dual-R9700 box these numbers come from is available for validating HIP release candidates or future AMD-touching PRs — happy to run your release CI artifacts against real gfx1201 before they ship.
🤖 Generated with Claude Code