Generalize COUNT_REL_TABLE metadata counting to packed extend chains - #918
Merged
Conversation
adsharma
force-pushed
the
packed_extend_chains
branch
from
September 5, 2026 22:47
6af77b3 to
db56ce3
Compare
COUNT(*) over a long chain of extends (LSQB q1: 7 hops) is slow: the
planner materializes 100M+ intermediate tuples and hash joins each hop
(~5.7s on ldbc_snb_sf1). The COUNT_REL_TABLE rewrite handles only a
single hop.
Add a CountRelTableOptimizer rewrite that detects a pure path of >= 2
extends under a keyless COUNT(*) aggregate — the hops may be connected
by node-ID-only inner hash joins (packed INL join plans) — and replaces
the whole subtree with a count-only operator, LogicalCountExtendChain /
CountExtendChain.
Execution propagates per-node int64 count vectors hop by hop instead of
materializing tuples:
c_0[v] = 1 for every source node
c_{j+1}[w] = sum over visible edges (v -> w) of c_j[v]
result = sum over last-hop edges of c_{N-1}[v]
Each hop scans the rel table's CSR with the direction keyed by the hop's
from-node, so the per-destination accumulation is a pure gather+add over
the neighbor column. Only one int64 per node offset is carried between
hops; neighbor IDs are consumed immediately and never materialized, and
no hash joins are performed.
Correctness gates (rewrite skipped otherwise):
- read-only transaction (uncommitted node inserts are not enumerated)
- no filters; scans carrying property predicates or primary-key scans
restrict the node set and disqualify the rewrite
- extends not BOTH direction, single-entry storage-backed rel groups,
single-table nodes, path graph (all node degrees <= 2, connected)
- the subtree contains only row-count-invariant operators (projections,
node-ID-only inner hash joins, scans)
Versioned deletions, in-memory CSR groups and local (uncommitted) rel
data are handled by reusing the standard rel-table scan machinery
(ReqTableScanState), including multi-parent packed batches via
packedChildOffsets.
LSQB q1 on ldbc_snb_sf1 (7 hops, count = 179,510,748):
before: 5.6-5.8s executing
after: 0.11-0.17s executing (~35-50x faster)
Fixes from CI:
- Skip the extend-chain rewrite for non-native storage (arrow://, icebug-disk)
node and rel tables: the fast path iterates the native node-group grid and
CSR, which arrow-backed tables do not have (returned 0 for 2-hop counts).
- OptimizerTest.JoinHint: RETURN COUNT(*) in q4 is now legitimately rewritten
into a count-only plan with no joins; use RETURN COUNT(e1.date) so the test
still exercises the hinted join order.
adsharma
force-pushed
the
packed_extend_chains
branch
from
September 6, 2026 01:37
db56ce3 to
9cbe6e4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
`COUNT(*)` over a long chain of extends (LSQB q1: 7 hops, packed path extend enabled) materializes 100M+ intermediate tuples and hash joins each hop:
The existing
COUNT_REL_TABLErewrite handles only a single hop (metadata count). Multi-hop chains get no benefit: every intermediate hop materializes neighbor IDs into data chunks and feeds hash joins, only for the result to be aggregated away.Change
Add a
CountRelTableOptimizerrewrite (tryRewriteExtendChainCount) that detects a pure path of >= 2 extends under a keylessCOUNT(*)aggregate — hops may be connected by node-ID-only inner hash joins (packed INL join plans) — and replaces the whole subtree with a new count-only operator:LogicalCountExtendChain→CountExtendChain(physical).Execution propagates per-node int64 count vectors hop by hop instead of materializing tuples:
int64per node offset is carried between hops; neighbor IDs are read from the nbr column and consumed immediately — never materialized, no hash joins.RelTableScanState), including multi-parent packed batches viapackedChildOffsets.Correctness gates (rewrite skipped otherwise)
BOTHdirection, single-entry storage-backed rel groups, single-table nodesCOUNT_REL_TABLErewriteResults (ldbc_snb_sf1, 7-hop chain, count = 179,510,748 — identical before/after)
enable_packed_path_extend=trueenable_packed_path_extend=false~35–50x faster. Chain-prefix cross-checks (2–7 hops, reversed chain) all produce identical counts vs the pre-change engine; filtered queries correctly fall back to the regular plan and match.
Testing
COUNT_EXTEND_CHAINas a single-threaded source; filtered / GROUP BY queries unchangedtest/test_files/lsqb/lsqb_queries.test) to be exercised in CI