Skip to content

fix(core): bound Context point-read memory and defer payload loading - #243

Merged
beinan merged 1 commit into
lance-format:mainfrom
beinan:codex/context-point-reads
Sep 10, 2026
Merged

fix(core): bound Context point-read memory and defer payload loading#243
beinan merged 1 commit into
lance-format:mainfrom
beinan:codex/context-point-reads

Conversation

@beinan

@beinan beinan commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator

Summary

Context.get(...) and partial updates currently list the entire context before locating a record; updates also materialize every record's payload. Replace these reads with filtered LSM scans that process identity and lifecycle fields in chunks of at most 256 candidates, retain one selected identity, and load only that record's requested payload after visibility and uniqueness checks finish.

  • Resolve supersession outside the candidate set using only id and supersedes_id, sharing one WAL snapshot across all scans. Preserve lifecycle visibility, lazy payload defaults, deterministic external-id lookup, ambiguous-update errors, and legacy schemas.
  • Avoid cloning the original record and its payload when constructing an update.
  • Add regression coverage for WAL/base reads, 519-record histories spanning chunks, cross-chunk ambiguity and ordering, quoted identifiers, and unreadable metadata on unselected records.

The 256-row bound applies to candidate bookkeeping and supersession predicates, not total process memory: Arrow batches, Lance internals, and the selected record's payload still consume memory. Large histories require a supersession scan per candidate chunk; scans without suitable indexes are not constant-time.

Testing

  • cargo fmt --all -- --check — passed.
  • cargo clippy --workspace --all-targets -- -D warnings — passed.
  • cargo test -p lance-context-core --lib store::tests:: -- --skip validate_uniqueness_against_large_store — 142 passed, 3 benchmark tests ignored.
  • cargo test --workspace --all-targets — passed, including the large-store uniqueness test.
  • GitHub Actions — all 10 checks passed, including Rust tests/coverage, Python 3.11 tests, three-platform wheel builds, and lint/style checks. The additional local Python setup was stopped after GitHub CI passed; no local Python test result is claimed.

@beinan
beinan merged commit 8fe27b2 into lance-format:main Sep 10, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant