Skip to content

Support Avro writes, defaults, and custom partition locations for native Format Tables - #950

Merged
JingsongLi merged 4 commits into
apache:mainfrom
JingsongLi:codex/native-format-table-parity
Sep 25, 2026
Merged

JingsongLi merged 4 commits into
apache:mainfrom
JingsongLi:codex/native-format-table-parity

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

Summary

Extend native Format Table read and write behavior to match the Java implementation:

  • Read catalog-managed partitions at their registered custom locations. Canonicalize filesystem identities and reject traversal, table-directory overlap, and overlapping partition ownership before listing files.
  • Write Avro object container files with Java-compatible nullable, logical, and nested schemas. Stream completed Avro blocks to storage, honor avro.codec and file.block-size, and expose Avro through the existing format writer factory.
  • Apply schema column defaults after input nullability validation and before partition routing or file projection, matching Java's DefaultValueRow order.
  • Treat a resolved catalog partition root as a complete partition regardless of its filesystem path depth, so hidden staging directories are excluded.

Verification

  • cargo test -p paimon --lib format_ --quiet (94 passed)
  • cargo test -p paimon-datafusion --test rest_format_table_scan --quiet (12 passed)
  • cargo test -p paimon-datafusion --test rest_format_table_dml --quiet (9 passed)
  • cargo test -p paimon --lib file_block_size --quiet (1 passed)
  • cargo clippy -p paimon --lib -- -D warnings
  • cargo fmt --all -- --check
  • git diff --check

The commit adds 2,054 lines, including behavioral tests for Avro round trips, default values, and catalog location failure paths.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Re-reviewed with a focus on Java parity. Verified integer default truncation and overflow handling against 1,356 executed Java cases, rechecked ARRAY and TIMESTAMP defaults through Java Avro readback, and ran the Rust unit suite plus 35 focused DataFusion regression tests. The previous findings are addressed; no blocking issues found. The CI check is green; the remaining CI jobs should complete before merging.

@JingsongLi
JingsongLi merged commit b2f1eb3 into apache:main Sep 25, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants