feat: add cell-unchanged preconditions to commit transactions - #9118
feat: add cell-unchanged preconditions to commit transactions#9118lichuang wants to merge 1 commit into
Conversation
A transaction may now declare that specific cells (fields × row addresses) must be unchanged since its read version. The commit loop evaluates the declarations against the pinned read-version state on every attempt, before rebase, and rejects with PrerequisiteFailed wrapped in RetryableCommitConflict instead of rebasing over changed data. The precondition travels on the Transaction proto so the lost-response recovery in verify_commit_outcome keeps working, and the retryable error lets execute_with_retry recompute and retry. Closes lance-format#8976.
|
Important Format specification voteThis PR modifies the Lance format specification, so it requires 3 binding +1 votes from PMC members (excluding the proposer) and a minimum 72-hour voting period, weekends excluded, before it can merge. Vote by approving this PR (+1) or requesting changes (−1, a veto). See the voting process. Status: ❌ Blocked — 0 of 3 required approvals
Updated automatically by the format-spec vote gate, which re-checks every 15 minutes — just voted? Re-check now (press Run workflow; leave the input blank to re-check every open format PR). A PMC member may apply the |
|
pr #9121 must be merged before this pr. |
There was a problem hiding this comment.
❌ Gate recommendation: request changes.
The author’s prerequisite update is verified: #9121 is the standalone format-vote change containing the protobuf contract, transaction specification, and compile-only edits. Merge #9121 first, then rebase this branch so this PR is implementation-only. Until that dependency lands, this PR cannot safely precede the durable format contract.
Defines the Transaction.Precondition message: a declared set of cells (fields × physical row addresses) that must be unchanged relative to the transaction's read_version for the commit to land. Rows are encoded as per-fragment RowSelection entries (whole-fragment flag or portable Roaring bitmap of offsets). The spec pins the composition semantics (conjunctive), absent/empty behavior, invalid encoding handling, and the conservative meaning of "unchanged" — declared fields must still exist with unchanged data type and nullability, and over-rejection is permitted while under-rejection is forbidden. This is the format-vote half of lance-format#8976; it carries only the proto change, the docs/src/format specification, and compile-required edits. The builder and commit-loop implementation lands in a follow-up PR (lance-format#9118).
Closes #8976.
Summary
A read-compute-write task (e.g. publishing derived column
y = f(x)) can now declare that the cells it read must be unchanged at commit time. The commit loop evaluates the declarations against the pinned read-version state on every attempt and rejects the commit — instead of rebasing over the changed data — when a concurrent transaction modified any protected cell.Design
Transactionand intransaction.proto.verify_commit_outcomedetects a lost-response landing by comparing the committed transaction read back from the manifest against the in-memory one; anything not in the proto breaks that equality and a landed commit would be misclassified, recomputed, and republished (the double-publish Support safe concurrent publication of derived columns in the same fragment #9043 forbids). This is also what distinguishes this from feat: commit preconditions judged on the manifest a commit lands on #8801'sCommitPrecondition, which covers state re-derivable from the latest manifest — a declared read set only exists at staging time.check_txn. Conflict compatibility and precondition validity are orthogonal: a concurrentDeletecan be absorbable by rebase while still invalidating the cells a staged value was computed from. Re-checking against the same original dependencies on every attempt gives Support safe concurrent publication of derived columns in the same fragment #9043's "concurrent winner re-checks against the original dependencies" for free.conflict_resolver.rsis untouched, keeping the Update: action translation + conflict resolution #6898 differential-oracle constraint.PrerequisiteFailednested insideRetryableCommitConflict, soexecute_with_retry(delete/update/merge_insert) recomputes and retries against the latest version automatically.Overwrite(cell-level protection is meaningless when the dataset is replaced, and overwrite'sread_version = 0leaves no aligned baseline); the strict-overwrite path incommit_transactiondefends in depth for non-builder callers.Known limitations (for reviewers)
Operation::Updatecould enable field-level precision viafields_modified). Safe direction; required for Support safe concurrent publication of derived columns in the same fragment #9043's field-disjoint composition to stop rejecting each other.execute_batch(append-only today) does not carry preconditions.Phase 2 (#9043, field-disjoint
Update × Updatecomposition) lands in v1 per the maintainer's confirmation below, and ports to the V2 actions from #6898 after cutover.