diff --git a/docs/src/format/table/transaction.md b/docs/src/format/table/transaction.md index c88c3170988..48b599c7b9d 100644 --- a/docs/src/format/table/transaction.md +++ b/docs/src/format/table/transaction.md @@ -466,6 +466,29 @@ The following operations are retryable conflicts with DataReplacement: A concurrent Delete or Update that only adds a deletion vector to a target fragment (without removing it) is compatible: the positional column file stays aligned and the rebase preserves the deletion vector. +#### Replaced Offsets + +A replacement rewrites a whole column file, but its values may differ from the previous file on only some rows: a partial +refresh, or a fill of rows that were null. `replaced_offset_bitmaps` lets the writer say which. It maps a replaced fragment's +id to the physical row offsets whose values changed, encoded as a portable-serialized Roaring bitmap. + +When stable row ids are enabled, committing a replacement stamps `last_updated_at_version` (see +[Row ID Lineage](row_id_lineage.md)) on each replaced fragment: + +- Absent, or absent for a fragment, or an empty bitmap: every row of that fragment takes the new version. This is what every + writer before this field produced. +- Present for a fragment that already carries a `last_updated_at_version` sequence: only the listed offsets take the new + version; the other rows keep theirs. +- Present for a fragment with no sequence to keep: every row takes the new version, since there is nothing to preserve. + +A writer must list every row whose value changed, including a row written as null that previously held a value. Listing +rows that did not change is allowed and only over-stamps. Consumers must reject as invalid a transaction whose keys name a +fragment that is not in `replacements`, whose bitmaps are not valid portable Roaring bitmaps, or whose offsets reach the +fragment's physical row count. + +The field is additive: absent decodes as a full stamp, and readers that predate it ignore it, so no existing transaction +changes meaning. + ### DataOverlay Attaches [overlay files](data_overlay_file.md) to fragments, supplying new values diff --git a/protos/transaction.proto b/protos/transaction.proto index 6e01831b4af..c260fc84947 100644 --- a/protos/transaction.proto +++ b/protos/transaction.proto @@ -365,6 +365,14 @@ message Transaction { // An operation that replaces the data in a region of the table with new data. message DataReplacement { repeated DataReplacementGroup replacements = 1; + /* Per replaced fragment, the physical row offsets whose values changed, + * as portable RoaringBitmap bytes. Drives row-level last_updated_at_version + * stamping: only the listed offsets take the new version, the rest keep + * theirs. Absent, or absent for a fragment, stamps every row of it. + * Keys must name fragments in `replacements`; offsets must be below the + * fragment's physical row count. + */ + map replaced_offset_bitmaps = 2; } /* Overlay files to append to a single fragment, in order (the last entry is diff --git a/rust/lance-table/src/transaction/proto.rs b/rust/lance-table/src/transaction/proto.rs index c51c8f16719..b318a1fbf94 100644 --- a/rust/lance-table/src/transaction/proto.rs +++ b/rust/lance-table/src/transaction/proto.rs @@ -373,7 +373,7 @@ impl TryFrom for Transaction { } } Some(pb::transaction::Operation::DataReplacement( - pb::transaction::DataReplacement { replacements }, + pb::transaction::DataReplacement { replacements, .. }, )) => Operation::DataReplacement { replacements: replacements .into_iter() @@ -679,6 +679,7 @@ impl From<&Transaction> for pb::Transaction { .iter() .map(pb::transaction::DataReplacementGroup::from) .collect(), + replaced_offset_bitmaps: Default::default(), }) } Operation::DataOverlay { groups } => {