Skip to content

perf(table): keep clustered row ID ranges compact in index - #9501

Open
Abhisheklearn12 wants to merge 1 commit into
lance-format:mainfrom
Abhisheklearn12:perf/compact-rowid-ranges-index
Open

Abhisheklearn12 wants to merge 1 commit into
lance-format:mainfrom
Abhisheklearn12:perf/compact-rowid-ranges-index

Conversation

@Abhisheklearn12

Copy link
Copy Markdown

Summary

PR #9311 introduced compact Ranges row ID segments, but merged RowIdIndex construction expands them into per-row pairs even when there are no deletions. Keep the compact segment and map its positions to a contiguous address range.

Tests cover gaps, shifted physical offsets, overlapping fragment spans, and batched lookups. The deletion-aware path and on-disk format are unchanged.

Fixes #9487.

Benchmark

BuildIndexClusteredRanges uses 100 fragments, 500,000 live IDs across 1,000,000 ID slots, and empty deletion vectors. Fixture construction is outside the timed loop. Both variants used the same benchmark and release-with-debug profile with 30 Criterion samples. The baseline differs only by removal of the new Ranges match arm.

Index construction path Estimate Criterion interval
Existing per-row expansion 4.335 ms 4.327 to 4.344 ms
Compact Ranges path 26.44 µs 26.15 to 26.74 µs

This is about 164× faster for this synthetic index-build case. Dataset opening, storage I/O, and query latency were not measured.

Validation

  • cargo fmt --all
  • cargo clippy --all --tests --benches -- -D warnings
  • cargo test -p lance-table rowids:: (131 passed)

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Gate recommendation: approve.

This removes the per-row expansion identified in #9487 for compact ranges without deletions. The position-to-address mapping preserves lookups across gaps and shifted offsets, while overlapping chunks retain the existing merge path. The broader BTree replacement has separate prerequisites, so this scoped improvement can land independently.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 23, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

K-approved Latest Gatekeeper recommendation permits acceptance. performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf(table): avoid expanding compact row ID ranges when building RowIdIndex

1 participant