Skip to content

Exclude live MemWAL generations from explicitly checked-out snapshots #249

Description

@beinan

StorageBase::checkout pins only the base dataset. Subsequent reads still discover the latest WAL manifests without consulting pinned_version, so a historical read can include later writes.

Reproduction (confirmed for ContextStore and RolloutStore):

  1. Add "old", merge it into base, and save that base version.
  2. Add "future" and make its WAL generation visible without merging.
  3. Check out the saved base version and list rows.

Expected: the pinned base snapshot returns only "old".
Actual: it returns ["old", "future"].

This reproduction uses a concrete already-merged base version; it does not assume that each rollout add advances the base version.

Relevant code: crates/lance-context-core/src/store_base.rs::checkout, wal_shard_snapshots, and LSM read construction.

Please define pinned reads as a stable historical view and apply that rule across list, point/blob reads, and observations. Background maintenance should not silently unpin an explicitly selected version. Add coverage for ContextStore and RolloutStore.

Found while reviewing main at e292518; fixes will be based on refreshed main (currently 8fe27b2).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions