Skip to content

feat: add cell-unchanged preconditions to commit transactions - #9118

Open
lichuang wants to merge 1 commit into
lance-format:mainfrom
lichuang:issue-8976
Open

feat: add cell-unchanged preconditions to commit transactions#9118
lichuang wants to merge 1 commit into
lance-format:mainfrom
lichuang:issue-8976

Conversation

@lichuang

Copy link
Copy Markdown
Contributor

Closes #8976.

Summary

A read-compute-write task (e.g. publishing derived column y = f(x)) can now declare that the cells it read must be unchanged at commit time. The commit loop evaluates the declarations against the pinned read-version state on every attempt and rejects the commit — instead of rebasing over the changed data — when a concurrent transaction modified any protected cell.

CommitBuilder::new(dataset)
    .with_cells_unchanged([x_field_id], rows_read)
    .execute(transaction)
    .await?

Design

  • The precondition travels on Transaction and in transaction.proto. verify_commit_outcome detects a lost-response landing by comparing the committed transaction read back from the manifest against the in-memory one; anything not in the proto breaks that equality and a landed commit would be misclassified, recomputed, and republished (the double-publish Support safe concurrent publication of derived columns in the same fragment #9043 forbids). This is also what distinguishes this from feat: commit preconditions judged on the manifest a commit lands on #8801's CommitPrecondition, which covers state re-derivable from the latest manifest — a declared read set only exists at staging time.
  • Evaluation runs in the commit loop against the pinned read-version state, before rebase, and is not gated on check_txn. Conflict compatibility and precondition validity are orthogonal: a concurrent Delete can be absorbable by rebase while still invalidating the cells a staged value was computed from. Re-checking against the same original dependencies on every attempt gives Support safe concurrent publication of derived columns in the same fragment #9043's "concurrent winner re-checks against the original dependencies" for free. conflict_resolver.rs is untouched, keeping the Update: action translation + conflict resolution #6898 differential-oracle constraint.
  • Violation reports PrerequisiteFailed nested inside RetryableCommitConflict, so execute_with_retry (delete/update/merge_insert) recomputes and retries against the latest version automatically.
  • Guards: preconditions are rejected at build time for detached commits and for Overwrite (cell-level protection is meaningless when the dataset is replaced, and overwrite's read_version = 0 leaves no aligned baseline); the strict-overwrite path in commit_transaction defends in depth for non-builder callers.

Known limitations (for reviewers)

  • Fragment-granular detection for base data: any data-file change on a protected fragment violates, including field-disjoint column updates (Operation::Update could enable field-level precision via fields_modified). Safe direction; required for Support safe concurrent publication of derived columns in the same fragment #9043's field-disjoint composition to stop rejecting each other.
  • Deletion-file identity is fragment-granular: a concurrent delete of non-protected rows in the same fragment also violates.
  • Overlay narrowing outside the protected region over-rejects: matching is by full content equality, so an overlay narrowed only in unrelated rows is treated as changed.
  • Stable row-id preconditions are left for a future change, per the issue.
  • execute_batch (append-only today) does not carry preconditions.

Phase 2 (#9043, field-disjoint Update × Update composition) lands in v1 per the maintainer's confirmation below, and ports to the V2 actions from #6898 after cutover.

A transaction may now declare that specific cells (fields × row
addresses) must be unchanged since its read version. The commit loop
evaluates the declarations against the pinned read-version state on
every attempt, before rebase, and rejects with PrerequisiteFailed
wrapped in RetryableCommitConflict instead of rebasing over changed
data.

The precondition travels on the Transaction proto so the
lost-response recovery in verify_commit_outcome keeps working, and the
retryable error lets execute_with_retry recompute and retry.

Closes lance-format#8976.
@github-actions github-actions Bot added A-format On-disk format: protos and format spec docs format-change A change to the format spec, which requires a vote. Remove if minor (e.g. fixing typo). enhancement New feature or request labels Sep 10, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Important

Format specification vote

This PR modifies the Lance format specification, so it requires 3 binding +1 votes from PMC members (excluding the proposer) and a minimum 72-hour voting period, weekends excluded, before it can merge. Vote by approving this PR (+1) or requesting changes (−1, a veto). See the voting process.

Status: ❌ Blocked — 0 of 3 required approvals

Approvals (this commit) none (0/3)
Vetoes none
Voting period ends Tue 2026-09-15 04:10 UTC (Mon 21:10 PDT)

Updated automatically by the format-spec vote gate, which re-checks every 15 minutes — just voted? Re-check now (press Run workflow; leave the input blank to re-check every open format PR). A PMC member may apply the format-waived label to waive the vote for a trivial edit (typo, wording, formatting).

lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Sep 10, 2026
@lichuang

Copy link
Copy Markdown
Contributor Author

pr #9121 must be merged before this pr.

@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Sep 10, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: request changes.

The author’s prerequisite update is verified: #9121 is the standalone format-vote change containing the protobuf contract, transaction specification, and compile-only edits. Merge #9121 first, then rebase this branch so this PR is implementation-only. Until that dependency lands, this PR cannot safely precede the durable format contract.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Sep 10, 2026
lichuang added a commit to lichuang/lance that referenced this pull request Sep 10, 2026
Defines the Transaction.Precondition message: a declared set of cells
(fields × physical row addresses) that must be unchanged relative to the
transaction's read_version for the commit to land. Rows are encoded as
per-fragment RowSelection entries (whole-fragment flag or portable
Roaring bitmap of offsets). The spec pins the composition semantics
(conjunctive), absent/empty behavior, invalid encoding handling, and the
conservative meaning of "unchanged" — declared fields must still exist
with unchanged data type and nullability, and over-rejection is
permitted while under-rejection is forbidden.

This is the format-vote half of lance-format#8976; it carries only the proto change,
the docs/src/format specification, and compile-required edits. The
builder and commit-loop implementation lands in a follow-up PR (lance-format#9118).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-format On-disk format: protos and format spec docs enhancement New feature or request format-change A change to the format spec, which requires a vote. Remove if minor (e.g. fixing typo). K-changes Latest Gatekeeper recommendation requests changes.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Commit precondition on cells not changing.

1 participant