[Parquet] Implement num distinct values for parquet writer - #10654
Merged
Conversation
Rich-T-kid
force-pushed
the
rich-t-kid/parquet-write-nvd
branch
from
August 12, 2026 13:09
cda6e4a to
5111139
Compare
Rich-T-kid
commented
Aug 12, 2026
Rich-T-kid
force-pushed
the
rich-t-kid/parquet-write-nvd
branch
from
August 12, 2026 13:16
5111139 to
5676cca
Compare
etseidl
changed the title
Rich-T-kid
commented
Aug 14, 2026
Rich-T-kid
force-pushed
the
rich-t-kid/parquet-write-nvd
branch
from
August 14, 2026 14:27
48c40aa to
d1fa404
Compare
Contributor
Author
|
@etseidl could you please take a look at this when you get a chance |
Contributor
|
Yes, sorry @Rich-T-kid, still catching up from some time off 😅 |
Contributor
Author
@etseidl no worries! sorry about the ping |
2 tasks
etseidl
approved these changes
Aug 18, 2026
etseidl
left a comment
etseidl
left a comment
Contributor
There was a problem hiding this comment.
Thanks @Rich-T-kid, I think this looks good.
Am I right in thinking this will only work for the arrow writer (i.e. the row-based API will not populate the distinct count)? If so, we might want to note that in the docs.
Contributor
And feel free to ping me any time, it's no bother...I'm scatterbrained 😅 |
Rich-T-kid
force-pushed
the
rich-t-kid/parquet-write-nvd
branch
from
August 18, 2026 22:39
d1fa404 to
4db4725
Compare
Rich-T-kid
force-pushed
the
rich-t-kid/parquet-write-nvd
branch
from
August 18, 2026 22:41
4db4725 to
62d6619
Compare
Contributor
Author
|
Contributor
|
I'll give this a day before merging to allow others to take a look if they wish. Thanks again! |
Contributor
|
Hearing no objections, the PR is merged. Thanks @Rich-T-kid |
This was referenced Aug 23, 2026
Rich-T-kid
added a commit
to Rich-T-kid/arrow-rs
that referenced
this pull request
Aug 26, 2026
) # Which issue does this PR close? - Closes apache#8608. - Closes apache#10650 # Rationale for this change see apache#8608, apache#10650 & apache/datafusion#24114 # What changes are included in this PR? Adds `set_write_row_group_number_distinct_values` to WriterProperties, which when enabled causes the ArrowWriter to track the exact number of distinct non-null values per column across the full row group and write it into the column chunk statistics footer as `distinct_count`. Tracking is implemented by hashing each non-null value using XxHash64 into a per-column HashSet<u64> that persists across batch writes and is finalized in `close()`. **The flag defaults to false so there is no impact on existing writers.** # Are these changes tested? yes, see test. # Are there any user-facing changes? yes, users will not be able to write `distinct_count` meta data to parquet files.
Rich-T-kid
added a commit
to Rich-T-kid/arrow-rs
that referenced
this pull request
Aug 28, 2026
) # Which issue does this PR close? - Closes apache#8608. - Closes apache#10650 # Rationale for this change see apache#8608, apache#10650 & apache/datafusion#24114 # What changes are included in this PR? Adds `set_write_row_group_number_distinct_values` to WriterProperties, which when enabled causes the ArrowWriter to track the exact number of distinct non-null values per column across the full row group and write it into the column chunk statistics footer as `distinct_count`. Tracking is implemented by hashing each non-null value using XxHash64 into a per-column HashSet<u64> that persists across batch writes and is finalized in `close()`. **The flag defaults to false so there is no impact on existing writers.** # Are these changes tested? yes, see test. # Are there any user-facing changes? yes, users will not be able to write `distinct_count` meta data to parquet files.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
see #8608, #10650 & apache/datafusion#24114
What changes are included in this PR?
Adds
set_write_row_group_number_distinct_valuesto WriterProperties, which when enabled causes the ArrowWriter to track the exact number of distinct non-null values per column across the full row group and write it into the column chunk statistics footer asdistinct_count. Tracking is implemented by hashing each non-null value using XxHash64 into a per-column HashSet that persists across batch writes and is finalized inclose(). The flag defaults to false so there is no impact on existing writers.Are these changes tested?
yes, see test.
Are there any user-facing changes?
yes, users will not be able to write
distinct_countmeta data to parquet files.