Skip to content

Support nested values and deletion vectors in batch Diff - #944

Merged
JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/native-diff-nested-values
Sep 24, 2026
Merged

JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/native-diff-nested-values

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

What changed

  • Compare ROW, ARRAY, MAP, MULTISET, and VECTOR values recursively in batch incremental Diff, including nested nulls, NaNs, and signed zero. Keep primary-key ordering restricted to scalar types.
  • Plan deletion-vector index manifests independently for the before and after snapshots and apply each file's bitmap through the existing key-value reader. This supports both materialized DV scans and DV merge-on-read.
  • Add end-to-end read/write coverage for nested values, vectors, projections, audit-log row kinds, historical DVs, and missing DV files.

This follows Java IncrementalDiffSplitRead: compare complete logical values and read each side with its own deletion files.

Verification

  • cargo test --locked -p paimon --test incremental_diff_extended_test --features fulltext,vortex (6 passed)
  • cargo test --locked -p paimon --test incremental_batch_scan_test --features fulltext,vortex (39 passed)
  • cargo test --locked -p paimon --lib table::table_read::tests::test_diff --features fulltext,vortex (4 passed)
  • cargo clippy --locked -p paimon --all-targets --features fulltext,vortex -- -D warnings
  • cargo fmt --all -- --check

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Reviewed ab209d3; no blocking issues found.

Checked recursive ROW/ARRAY/MAP/MULTISET/VECTOR value equality, null-parent handling, NaN/signed-zero semantics, full-value comparison under projection, and independent before/after deletion-vector resolution for both materialized DV and merge-on-read scans.

Local validation (default features):

  • cargo test --locked -p paimon --lib: 3113 passed, 6 ignored.
  • incremental_batch_scan_test: 39 passed.
  • incremental_diff_extended_test: 6 passed.
  • Two additional temporary regression tests passed: deeply nested ROW/ARRAY/MAP values with null/empty distinctions and key-only projection; snapshot-specific DVs across small read batches with key filtering and audit-log assertions in both DV modes.

CI has no reported failures at the time of review; some jobs are still running. Optional fulltext/vortex feature builds were not rerun locally.

@JingsongLi
JingsongLi merged commit dd6c95a into apache:main Sep 24, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants