Skip to content

Generalize COUNT_REL_TABLE metadata counting to packed extend chains - #918

Merged
adsharma merged 1 commit into
mainfrom
packed_extend_chains
Sep 6, 2026
Merged

Generalize COUNT_REL_TABLE metadata counting to packed extend chains#918
adsharma merged 1 commit into
mainfrom
packed_extend_chains

Conversation

@adsharma

@adsharma adsharma commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Problem

`COUNT(*)` over a long chain of extends (LSQB q1: 7 hops, packed path extend enabled) materializes 100M+ intermediate tuples and hash joins each hop:

Time: 14.50ms (compiling), 5645.25ms (executing)

The existing COUNT_REL_TABLE rewrite handles only a single hop (metadata count). Multi-hop chains get no benefit: every intermediate hop materializes neighbor IDs into data chunks and feeds hash joins, only for the result to be aggregated away.

Change

Add a CountRelTableOptimizer rewrite (tryRewriteExtendChainCount) that detects a pure path of >= 2 extends under a keyless COUNT(*) aggregate — hops may be connected by node-ID-only inner hash joins (packed INL join plans) — and replaces the whole subtree with a new count-only operator: LogicalCountExtendChainCountExtendChain (physical).

Execution propagates per-node int64 count vectors hop by hop instead of materializing tuples:

c_0[v]      = 1 for every source node
c_{j+1}[w]  = sum over visible edges (v -> w) of rel table j of c_j[v]
result      = sum over last-hop edges of c_{N-1}[v]
  • Each hop scans the rel table CSR with the direction keyed by the hop's from-node (FWD if from-node is the rel's src side, BWD if dst side), so the per-destination accumulation is a pure gather+add over the neighbor column.
  • Only one int64 per node offset is carried between hops; neighbor IDs are read from the nbr column and consumed immediately — never materialized, no hash joins.
  • Versioned deletions, in-memory CSR groups and local (uncommitted) rel data are handled by reusing the standard rel-table scan machinery (RelTableScanState), including multi-parent packed batches via packedChildOffsets.

Correctness gates (rewrite skipped otherwise)

  • read-only transaction (uncommitted node inserts are not enumerated)
  • no filters; scans carrying property predicates or primary-key scans restrict the node set and disqualify the rewrite
  • extends not BOTH direction, single-entry storage-backed rel groups, single-table nodes
  • the hop graph is a simple path (all degrees <= 2, connected, edges == nodes - 1)
  • the subtree contains only row-count-invariant operators (projections, node-ID-only inner hash joins, scans); single-hop chains keep the cheaper existing COUNT_REL_TABLE rewrite

Results (ldbc_snb_sf1, 7-hop chain, count = 179,510,748 — identical before/after)

config before after
enable_packed_path_extend=true 5.65–5.84s 0.11–0.17s
enable_packed_path_extend=false 5.60–5.69s 0.11–0.11s

~35–50x faster. Chain-prefix cross-checks (2–7 hops, reversed chain) all produce identical counts vs the pre-change engine; filtered queries correctly fall back to the regular plan and match.

Testing

  • Manual cross-validation against the pre-change binary on LDBC SNB sf1 (see above)
  • Plan shows COUNT_EXTEND_CHAIN as a single-threaded source; filtered / GROUP BY queries unchanged
  • LSQB test suite (test/test_files/lsqb/lsqb_queries.test) to be exercised in CI

@adsharma
adsharma force-pushed the packed_extend_chains branch from 6af77b3 to db56ce3 Compare September 5, 2026 22:47
COUNT(*) over a long chain of extends (LSQB q1: 7 hops) is slow: the
planner materializes 100M+ intermediate tuples and hash joins each hop
(~5.7s on ldbc_snb_sf1). The COUNT_REL_TABLE rewrite handles only a
single hop.

Add a CountRelTableOptimizer rewrite that detects a pure path of >= 2
extends under a keyless COUNT(*) aggregate — the hops may be connected
by node-ID-only inner hash joins (packed INL join plans) — and replaces
the whole subtree with a count-only operator, LogicalCountExtendChain /
CountExtendChain.

Execution propagates per-node int64 count vectors hop by hop instead of
materializing tuples:

  c_0[v] = 1 for every source node
  c_{j+1}[w] = sum over visible edges (v -> w) of c_j[v]
  result     = sum over last-hop edges of c_{N-1}[v]

Each hop scans the rel table's CSR with the direction keyed by the hop's
from-node, so the per-destination accumulation is a pure gather+add over
the neighbor column. Only one int64 per node offset is carried between
hops; neighbor IDs are consumed immediately and never materialized, and
no hash joins are performed.

Correctness gates (rewrite skipped otherwise):
- read-only transaction (uncommitted node inserts are not enumerated)
- no filters; scans carrying property predicates or primary-key scans
  restrict the node set and disqualify the rewrite
- extends not BOTH direction, single-entry storage-backed rel groups,
  single-table nodes, path graph (all node degrees <= 2, connected)
- the subtree contains only row-count-invariant operators (projections,
  node-ID-only inner hash joins, scans)

Versioned deletions, in-memory CSR groups and local (uncommitted) rel
data are handled by reusing the standard rel-table scan machinery
(ReqTableScanState), including multi-parent packed batches via
packedChildOffsets.

LSQB q1 on ldbc_snb_sf1 (7 hops, count = 179,510,748):
  before: 5.6-5.8s executing
  after:  0.11-0.17s executing (~35-50x faster)

Fixes from CI:
- Skip the extend-chain rewrite for non-native storage (arrow://, icebug-disk)
  node and rel tables: the fast path iterates the native node-group grid and
  CSR, which arrow-backed tables do not have (returned 0 for 2-hop counts).
- OptimizerTest.JoinHint: RETURN COUNT(*) in q4 is now legitimately rewritten
  into a count-only plan with no joins; use RETURN COUNT(e1.date) so the test
  still exercises the hinted join order.
@adsharma
adsharma force-pushed the packed_extend_chains branch from db56ce3 to 9cbe6e4 Compare September 6, 2026 01:37
@adsharma
adsharma merged commit 95e1f69 into main Sep 6, 2026
4 checks passed
@adsharma
adsharma deleted the packed_extend_chains branch September 6, 2026 19:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant