Skip to content

fix: bound MemWAL merge memory with ROLLOUT_MERGE_MAX_BYTES - #242

Merged
beinan merged 1 commit into
lance-format:mainfrom
beinan:codex/memwal-merge-max-bytes
Sep 10, 2026
Merged

fix: bound MemWAL merge memory with ROLLOUT_MERGE_MAX_BYTES#242
beinan merged 1 commit into
lance-format:mainfrom
beinan:codex/memwal-merge-max-bytes

Conversation

@beinan

@beinan beinan commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator

Summary

MemWAL generations have variable sizes, so limiting a merge to a fixed generation count can still buffer several GiB of inline blobs. Add ROLLOUT_MERGE_MAX_BYTES / --rollout-merge-max-bytes, defaulting to 1 GiB, alongside the existing generation cap.

  • Accumulate aligned batches' get_array_memory_size() before deduplication, and stop after the generation that reaches the byte budget. Either cap ends the pass; remaining generations stay pending for subsequent passes.
  • Always finish a whole generation, including an oversized first generation, so merges keep making progress without partially draining a generation. The buffer can exceed the budget by up to one generation; scan, deduplication, and commit allocations require additional memory headroom.
  • Wire the budget through store options and server creation/reopen paths. Setting a cap to 0 disables only that cap. Document the defaults and generation-boundary behavior.
  • Add regression coverage for both cap orderings, independently disabled caps, oversized generations, blob integrity, and pending-generation directory cleanup.

Testing

  • cargo test -p lance-context-core -p lance-context-server --all-targets — 306 passed, 0 failed, 3 existing tests ignored; includes all 8 WAL merge cleanup regression tests.
  • cargo clippy --workspace --all-targets -- -D warnings — passed.
  • cargo clippy --manifest-path crates/lance-context/Cargo.toml --all-targets -- -D warnings — passed.
  • cargo fmt --all -- --check — passed.

@beinan
beinan merged commit e292518 into lance-format:main Sep 10, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant