alexreinking/ggml app by alexreinking · Pull Request #9417 · halide/Halide

alexreinking · 2026-09-01T16:48:12Z

Fixes #

Breaking changes

List any breaking changes here. If there are none, you can remove this section.
Common breaking changes include:

Changes to any existing APIs in the Halide:: namespace, but not the
Halide::Internal:: namespace.
Anything else that may cause existing Halide code to fail to compile or change
output.

These do not necessarily disqualify a PR from being merged, but they should at
least be tagged with the release_notes label.

Checklist

Tests added or updated (not required for docs, CI config, or typo fixes)
Documentation updated (if public API changed)
Python bindings updated (if public API changed)
Benchmarks are included here if the change is intended to affect performance.
Commits include AI attribution where applicable (see Code of Conduct)

codecov · 2026-09-02T07:53:26Z

Codecov Report

❌ Patch coverage is 49.82206% with 141 lines in your changes missing coverage. Please review.
✅ Project coverage is 69.65%. Comparing base (e9123a1) to head (34fb2b9).

Files with missing lines	Patch %	Lines
src/Generator.cpp	0.00%	43 Missing and 1 partial ⚠️
src/Approximation.h	15.21%	39 Missing ⚠️
src/Func.cpp	57.50%	21 Missing and 13 partials ⚠️
src/Pipeline.cpp	66.66%	10 Missing and 4 partials ⚠️
src/Approximation.cpp	85.50%	5 Missing and 5 partials ⚠️

Additional details and impacted files

@@                      Coverage Diff                      @@
##           alexreinking/struct-types    #9417      +/-   ##
=============================================================
- Coverage                      69.73%   69.65%   -0.08%     
=============================================================
  Files                            262      264       +2     
  Lines                          79988    80304     +316     
  Branches                       19528    19580      +52     
=============================================================
+ Hits                           55778    55939     +161     
- Misses                         18296    18419     +123     
- Partials                        5914     5946      +32

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

hoist_invariants() looks for a single loop-invariant factor of the whole increment, so a sum of terms with different factors -- a*g(r) + b*h(r) -- had nothing it could do: flattening the multiply chain sees the sum as one opaque leaf, and the directive threw. Give each term its own accumulator instead, all advanced by one loop over the same reduction domain, with the write-back applying each factor once. Terms sharing a factor stay in one accumulator, so an increment that was already hoistable is unaffected. This is the TODO that was sitting on extract_factor(). Each term's accumulator is its own single-valued Func, not a value of one Tuple-valued intermediate, so it can be scheduled -- or severed by Pipeline::compute_offline() -- independently of the others. An update definition that did not split keeps the shape it had: a single intermediate carrying every value of the original reduction, whose values may reference each other's. hoist_invariants() therefore returns a std::vector<Func>: one entry per accumulator, or a single entry for a definition that did not split. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Reaching the terms of a *product* of sums needs the increment multiplied out first, and whether that pays is not something to guess at: it depends on what the terms turn out to contain. Splitting s * (r + 1) into two accumulators is a pessimization, while splitting (d*q(r) + m) * (e*p(r)) into d*e*sum(q*p) + m*e*sum(p) turns one float reduction into two integer ones. So multiplying out is its own schedule directive, Stage::distribute(), rather than a heuristic inside hoist_invariants(). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

compute_offline() severs a set of Funcs from a Pipeline's computation, rewriting every call to them (and anything only they depended on) into calls to fresh or caller-supplied ImageParams of matching type and dimensionality. This turns work that would otherwise be recomputed on every realize() -- e.g. a static weight quantizer's encode() step -- into ordinary input data supplied once, from wherever the returned `offline` Pipeline was realized or compiled. The rewrite is eager and destructive, like rfactor()/approximate_by(): by the time the call returns, none of the Pipeline's outputs depend on the severed Funcs for their computation, only for the shape the returned ImageParams must satisfy. Two overloads: one mints fresh ImageParams, one binds each severed Func to a caller-supplied ImageParam (e.g. a Generator's own Input). v1 requires each severed Func to be single-valued. Verified by test/correctness/compute_offline.cpp. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

approximate_by() eagerly and destructively replaces every call to a Func inside a set of consumers with a call to the round trip decode(encode(f)) -- a lossy, quantified Func-to-Func transform where decode(encode(f)) reproduces f's signature. The substitution happens immediately, like rfactor(), rather than being deferred to lowering the way Func::in() is. The transform is expressed as an Approximation: a bidirectional encode()/decode() pair (src/Approximation.{h,cpp}), with combinators -- Compose, Apply, Permute, Choose, Identity, and the TrustedInverse escape hatch -- for building up a codec from smaller pieces. See doc/ApproximationDesign.md for the design rationale. Also adds the Generator plumbing that lets a Generator wire the two halves of an approximate_by()/compute_offline() split into real ports: GeneratorBase::add_input(const ImageParam &) and add_output(const Func &), backed by adopt() helpers on GeneratorInputBase/GeneratorOutputBase. Verified by test/correctness/approximate_by.cpp. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Ports apps/ggml onto the mature quantized-kernels substrate: a from-scratch Halide reimplementation of GGML's quantize / dequantize / vec-dot / repack kernels for the full format catalog (Q4_0..Q8_K, K-quants, IQ*, TQ*, FP4, f16/bf16), benchmarked against GGML's own kernels via kernel-bench. The codecs are built as composable Approximation combinators (quant_components.h) and spliced into pipelines with Func::approximate_by() / Pipeline::compute_offline(). Wiring: add_app(ggml) + a "ggml" dependency and apps/vcpkg/ports/ggml overlay port (ggml v0.15.3, CPU static). A minimal apps/ggml/vcpkg.json + vcpkg-configuration.json make the standalone build the app's CMakeLists already advertises actually resolve ggml (the integrated apps/ build still uses the root manifest). The app was written against the alpha rfactor-hoisting API, which was renamed and split on the mature branch; the generators are migrated accordingly: - inline_calls({...}) -> .update().eager_inline({...}) - rfactor(..., RFactorOptions::HoistInvariantFactor) -> rfactor(...) + eager_inline(...).hoist_invariants() + change_type(...) Known deviation: the SDOT vec-dot schedules (q4_0/q8_0/q5_0/iq4_nl/mxfp4) are temporarily set to the correct Float schedule. The mature hoist_invariants() cannot lift the per-block scale out of the reduction when the dequant is built through approximate_by()'s round-trip replacement (a single eager_inline() leaves the scale behind decode-chain Func boundaries), unlike test/correctness/struct_type_dot_product.cpp which inlines direct dequantizer Funcs. See the TODOs in symmetric_vec_dot_generator.cpp. Verified: all 28 codec round-trip tests pass, and kernel-bench --all reports zero correctness mismatches vs GGML across quantize/dequantize/vec_dot/repack. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Adopts mature Type::Struct/field()/pack_struct() for the symmetric quantize/dequantize codecs (Q4_0/Q8_0/Q1_0), replacing the hand-rolled byte-layout engine for those formats: - StructBlockLayout (quant_components.h): a struct-typed leaf that reads the block's scale straight out of a typed `d` field (subsuming Fp16Pack's manual lo|(hi<<8) reassembly + reinterpret) and hands the `qs` bytes to the same code_pack the byte path uses. It replaces StructPack + Permute + the scale's Fp16Pack; the compiler owns the field offsets and the total byte size (block_type.bytes()), so no hand-summed block_bytes. - make_symmetric_block_scheme() gains an opt-in struct_layout flag; the symmetric *codec* generator sets it (vec_dot/repack stay on the byte path for now). SchemeAndBytes carries the block's Type::Struct. - codec_generator_base.h declares a 1-D Type::Struct ImageParam/Output when the scheme is structured (vs the 2-D (byte, blk) UInt(8) buffer), and compute_offline() binds/severs it as an ordinary struct-typed Func. - ggml_quants.cpp: the Q4_0/Q8_0/Q1_0 wrappers hand GGML's raw bytes to the struct kernel via a 1-D halide_buffer_t whose type is {halide_type_struct, 8, reserved=block_bytes} -- Type::to_abi()'s exact ABI form, constructed with Halide::Runtime::Buffer<void,1>. Validated: q4_0/q8_0/q1_0 roundtrip tests pass and kernel-bench --all reports zero mismatches vs GGML -- the struct-typed buffer flows through approximate_by/compute_offline and the GGML byte-buffer ABI boundary unchanged. Deferred: converting the vec_dot/repack operands (which would also eliminate the re-derived block widths in symmetric_vec_dot_generator.cpp), and rolling out to the affine (min), split-code (Q5_0), K-quant, and IQ layouts. The old StructPack/FieldSpec/make_block_layout engine stays for those. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The SDOT integer-dot schedule was downgraded to a plain Float reduction in Phase 2 because hoist_invariants() could not find the per-block scale as a distributable factor: a single eager_inline() of approximate_by()'s .replacement only peels the outermost relayout wrapper, leaving the scale*codes product buried inside the dequantizer Func the Approximation combinators build. Fix: in the SDOT branch, after rfactor() preserves the block index, fold the *entire* decode chain of both operands into the per-block partial's update. eager_inline() no-ops on any Func not currently directly called and flattens exposed calls left to right, so inlining the whole set of inlinable decode handles -- one pass per possible chain level -- flattens it regardless of build order, leaving (codes*scale)*(codes*scale) with the scales as loop-invariant leaves. hoist_invariants() then lifts them and change_type(Int(32)) retypes the scale-free inner dot, which CodeGen_ARM matches to SDOT. Re-enables SDOT for Symmetric (Q4_0/Q8_0), Symmetric5Bit (Q5_0 -- its CombineBits code reconstruction is all inside the r.x-dependent codes leaf, so the scale stays a top-level factor), and the single-scale codebook families (IQ4_NL/MXFP4). Q1_0 stays Float: its 128-wide 1-bit block trips change_type(Int(32))'s overflow proof. Verified: kernel-bench --all reports zero mismatches vs GGML and all 28 roundtrip tests pass. q4_0 vec_dot now runs at 1.48x ggml-cpu (was 0.07x on the Float schedule) and q5_0 at 2.24x -- both beating GGML's hand-written kernels. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Factors the vec_dot SDOT schedule into a shared sdot_partial() helper (sdot_schedule.h): rfactor the block-index reduction, flatten every operand's decode chain (deep eager_inline so hoist_invariants() sees each per-block scale as a top-level factor), hoist the scales, and change_type(Int(32)) the scale-free inner dot. vec_dot now calls it instead of open-coding the sequence. Applies the same schedule to the repack matmul generators for the simple single-scale weight families (Q4_0/Q8_0/IQ4_NL/MXFP4): in gemv the weight scale depends only on (block, column) and the activation scale only on the block, so both hoist out of the within-block reduction, same as gemm. K-quant weights carry two-level (super/sub-block) scales that aren't a single per-block- invariant factor, so they keep the default schedule. Verified: kernel-bench --repack reports zero mismatches vs GGML. The simple families go from the default schedule (~0.06-0.20x) to ~0.6x of GGML's hand-written interleaved kernels for both gemv and gemm -- an efficient, SDOT-based schedule is now writable for repack, matching the vec_dot path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Rolls the Type::Struct weight blocks from the codec path into the vec_dot generator for the symmetric weight families (Q4_0/Q8_0/Q1_0): - VecDotSpec carries an optional Type per operand; the base generator declares a 1-D Type::Struct ImageParam (block index only) when set, vs the 2-D (byte, blk) UInt(8) buffer otherwise, and indexes the reduction's block RDom off the right dimension. compute_offline() binds/severs the struct-typed operand as an ordinary Func. - The symmetric weight uses make_symmetric_block_scheme(struct_layout=true) and reports its block Type. The activation stays on the byte path for now (Q8_0/ Q8_1 are shared across many weight formats and the Reblock relayout is byte-based), so operands are struct-typed independently. - The Q4_0/Q8_0/Q1_0 vec_dot wrappers wrap GGML's raw weight bytes in the same 1-D struct halide_buffer_t helper the codecs use. SDOT survives the struct decode unchanged: the base header's deep inline flattens StructBlockLayout's field()-based dequantizer the same way it flattens the byte path's, so hoist_invariants() still lifts the per-block scale. Verified: kernel-bench --vecdot reports zero mismatches; q4_0 vec_dot runs at 1.55x ggml-cpu (struct-typed weight, unchanged from the byte-path SDOT result). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…s SDOT) LinearDequant dequantized as cast<float>(cast<int32_t>(codes)) * scale. The redundant int32 detour broke change_type(Int(32))'s SDOT rewrite pattern (cast<f32>(int8) * cast<f32>(int8)), so the repack gemv/gemm and codebook (IQ4_NL/MXFP4) inner products fell back to a float16 SIMD multiply instead of an int8 dot -- confirmed by `-e stmt`: widening_mul(float16x32, ...) vs vec_dot's widening_mul(int8x32, int8x32). Casting the int8 codes straight to float restores the SDOT match. Repack q4_0 gemv ~doubles (8.5 -> 14 GFLOP/s), IQ4_NL vec_dot 1188 -> 837 ns; 0 mismatches, 28/28 roundtrip tests still pass. Note on the remaining repack gap: this is the best Halide can do here. ggml-cpu matches the machine's `matmul_int8` (i8mm/SMMLA) instruction for its repack kernels (~166 GFLOP/s on q4_0 4x4), but Halide's CodeGen_ARM has no SMMLA path -- only SDOT (CodeGen_ARM.cpp:793-801). So a Halide repack tops out at SDOT throughput (~12x below ggml-cpu's SMMLA matmul), regardless of schedule; closing it would require i8mm support in the Halide backend. (vec_dot, a pure dot product, has no such gap -- ggml-cpu uses SDOT there too, and Halide matches/beats it.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The SDOT vec_dot schedule collapsed each block's integer dot to a scalar and accumulated it into a single float, which cost a horizontal reduce per block and left the whole kernel bound by one nb-deep multiply-add chain. Restructure it into ggml's shape: - Take the accumulator's vector lanes from the within-block reduction, so the sdot's four Int(32) lanes survive into the float accumulator and no block pays for a horizontal reduce. Lanes have to come from r.x, not r.y: blocks are interleaved {scale, codes} records, so a lane per block would gather both the codes and the scales. - Cut r.x so one sdot covers 16 int8s, with the chunks run serially into the same register. Reducing straight to 4 lanes instead makes Halide lower the wide reduce as two independent sdots plus an addp. - Interleave four blocks into independent accumulators. Widening the vector does not help here -- every lane of one accumulator advances on every block, so only interleaving blocks shortens the chain. - Give the main reduction a block count divisible by the interleave and sweep the remainder in a second update. A predicated tail is not a local cost: it makes the per-block sdot a dynamic-extent allocation that has to be zeroed and accumulated through memory, roughly doubling the cost of every block. - Reduce the lanes-by-blocks accumulators vectorially rather than as a serial chain of scalar adds. vec_dot is called once per output element of a matvec, so the argument marshalling is not amortized over anything; at these row lengths building three Halide::Runtime::Buffers cost about as much as the dot product. Fill a halide_buffer_t in place in the vec_dot wrappers instead, and build these kernels with no_asserts/no_bounds_query. Also pin the byte-path block stride, which was costing a serial pointer-add chain per block, and add KERNEL_BENCH_N to the harness so per-call overhead can be separated from per-block cost. vec_dot vs ggml-cpu on M3 Max, n=4096: q4_0 0.52x -> 0.97x, q8_0 0.61x -> 0.96x, q5_0 0.38x -> 0.51x. All 28 roundtrip tests pass and kernel-bench --all reports no mismatches. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The affine formats decode to d*code + m, so the per-block product (d*code + m) * (d_act*act) has no single scale to hoist and the generator left them on the plain float reduction -- an unscheduled, fully scalar loop over every element, redoing the nibble extraction and both fp16 scale loads per element. Multiplying the product out with the new Stage::distribute() gives d*d_act * sum(code*act) + m*d_act * sum(act) and hoist_invariants() puts each term in its own accumulator. Both bodies are integer, so both reach SDOT: the first as the ordinary code-by- activation dot, the second as a dot with a vector of ones. That is ggml's own decomposition of these formats, except that ggml reads sum(act) from the s field block_q8_1 stores at quantize time rather than recomputing it, which is worth about four SIMD ops per block. vec_dot vs ggml-cpu on M3 Max, n=4096: q4_1 0.03x -> 0.80x (3843 ns -> 134 ns), q5_1 0.04x -> 0.50x (4059 ns -> 289 ns). All 28 roundtrip tests pass, kernel-bench --all reports no mismatches, and the odd-block tail is correct for block counts that are not a multiple of the interleave. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

… sdot_partial() sdot_partial() was written against hoist_invariants()'s old single-Func return, so it retyped the intermediate and handed back one Func. Now that hoist_invariants() gives each term of a split reduction its own single-valued accumulator, thread that through: sdot_partial() returns a std::vector<Func> -- one Int(32) part per term -- and vec_dot_generator_base.h carries the vector through instead of narrowing it back down to one Func. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Where q4_0/q8_0/q4_1 landed and why, the build/measure workflow (including the traps: core-type noise, the generator's env var not invalidating ninja, WITH_TESTS=OFF), the design decisions behind distribute() and per-term accumulator Funcs, and the measured numbers for the stored-block-sum route that is still outstanding. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The affine vec_dots decompose (d*code + m)*(d_act*act) into d*d_act*sum(code*act) + m*d_act*sum(act). ggml does not recompute the second sum -- it reads the `s` field block_q8_1 stores at quantize time. Do the same: leave the activation decode un-inlined through distribute() and the first hoist so the offset term's accumulator is exactly the whole-block activation sum, change_type it to Float(16), and compute_offline it to a new third Input bound to a zero-copy fp16 view of the stored `s`. The product term re-inlines the activation's full decode chain and re-hoists to recover the scale-free Int(32) SDOT. q4_1 0.80x -> 0.97x (110.2 ns), q5_1 0.50x -> 0.64x. This matched the lane-split+sever projection, so the harder route (per-term rfactors via distribute-into-update-definitions) is unnecessary. All 28 roundtrips pass, kernel-bench --all clean, odd tails correct. Format knowledge lives with the codec: SchemeAndBytes::has_block_sums, set by make_symmetric_byte_sum_block_scheme, carried to VecDotSpec::act_has_block_sums (guarded a_nat == wbs). The GGML_PER_BLOCK_PROBE variant-A branch is kept for the symmetric formats. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The q5_0/q5_1 gap over q4_x is the per-element high bit unpacked from the qh field. PlanarBitPack's 1-bit decode emitted (qh[kk/8] >> (kk%8)) & 1, which lowers to a per-lane variable shift plus a transpose_vector to broadcast the qh bytes across the sdot lanes (~24 NEON ops/block). ggml avoids this with a byte->8-byte memory LUT (table_b2b). Mirror it with a compile-time Buffer<uint8_t> b2b(bit, byte) embedded in the binary. The table read is a contiguous 8-byte load only when the qh byte is a scalar and the 8 bit positions are the vector lanes, so materialize the reconstructed codes per block (combine_bits_code compute_at the block loop, kk split (byte, pos): pos vectorizes the load, byte unrolls to a scalar index). Inlined into the sdot the byte varies per lane -> a 16-wide gather (0.12x), so materialization is required. Two enablers: sdot_partial() gained a keep_out list to hold the codes leaf out of its deep inline (can_be_inlined() ignores compute level, so a schedule alone doesn't stop eager_inline); and the odd-block tail gets its own decode chain (a second Wt/Vec + approximate_by bound to the same inputs) so the main reduction's compute_at can fuse without the tail -- which also stops Halide hoisting the codes buffer to whole-row. q5_0 0.51->0.63x, q5_1 0.64->0.68x; q5_K on the float path 8615->6730 ns from the shared decode. Transpose is gone (no uzp2/dup.8b). All 28 roundtrips pass, kernel-bench --all clean, odd tails correct. The residual gap to 0.95x is the per-block codes store/reload, which needs a CodeGen_ARM change or ggml's hand-scheduled load balance -- see PERF_NOTES. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…x, 0.68->0.78x) Two fixes on top of the qh LUT: - q5_0_vec_dot / q5_1_vec_dot were missing FEATURES no_asserts no_bounds_query (every other tuned vec_dot has it), so they paid the assert/bounds-query prologue -- ~11 ns of startup on a ~170 ns call. Worth ~0.04x on its own. - The materialized reconstructed codes were round-tripping a per-block stack buffer. store_in(MemoryType::Register), with kk split into 16-code units (one sdot chunk = two qh bytes x 8 positions) so the table-load store width matches the sdot read width, keeps them in the vector register file (stmt: "in Register", str drops to 2). q5_0 0.63->0.69x (166.6 ns), q5_1 0.68->0.78x (170.3 ns). All 28 roundtrips pass, kernel-bench --all clean (24/24), odd tails correct. The residual gap to 0.95x is the reconstruction op count (ggml's table folds the +16/-16 into one sub) and qh load traffic -- see PERF_NOTES. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

q5_0_vec_dot still built three Halide::Runtime::Buffer objects per call (~13 ns of marshalling on a ~160 ns kernel) while q4_0/q8_0/q5_1 fill a halide_buffer_t in place via StackBuffer. Migrate it. q5_0 0.69->0.73x. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

alexreinking force-pushed the alexreinking/ggml-app branch from e836bfa to b5b072a Compare September 1, 2026 17:16

alexreinking force-pushed the alexreinking/ggml-app branch 2 times, most recently from ccd899e to b07cef9 Compare September 1, 2026 18:51

alexreinking force-pushed the alexreinking/ggml-app branch from b07cef9 to 34fb2b9 Compare September 1, 2026 19:06

alexreinking and others added 22 commits September 3, 2026 13:10

Improve q5_0/q5_1 performance (note: breaks contracts)

81e3255

Checkpoint

74ba337

Checkpoint

e918168

Manual cleanup

f94a1e2

alexreinking force-pushed the alexreinking/ggml-app branch from 34fb2b9 to f94a1e2 Compare September 3, 2026 17:10

Navigation Menu

Uh oh!

alexreinking/ggml app - #9417

alexreinking/ggml app#9417
alexreinking wants to merge 22 commits into
alexreinking/struct-typesfrom
alexreinking/ggml-app

alexreinking commented Sep 1, 2026

Uh oh!

codecov Bot commented Sep 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

alexreinking commented Sep 1, 2026

Breaking changes

Checklist

Uh oh!

codecov Bot commented Sep 2, 2026

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant