Add UnionArray

[connortsui20]

Rationale for this change

Tracking issue: #8769

This is the array layer of the Union stack. The compute integration has been split into the stacked follow-up #8884.

What changes are included in this PR?

• Adds the canonical sparse UnionArray encoding with nullable u8 type IDs and row-aligned children.
• Adds construction and invariant validation, typed accessors, scalar access, top-level validity, serialization, and session registration.
• Registers UnionArray as the canonical representation for Union dtypes.
• Leaves filter, take, slice, mask, cast, aggregate, constant/chunked canonicalization, and compression support to Add UnionArray compute functions #8884.

I think that this is a mostly straightforward implementation. The one implementation thing I will shout out is that we do not validate that the entire type IDs array has valid type IDs. If we come across an invalid type ID, we just panic, and that is similar behavior to DictArray validation.

What APIs are changed? Are there any user-facing changes?

Adds the public Union, UnionArray, UnionArrayExt, and UnionDataParts APIs.

[codspeed-hq]

Merging this PR will regress 1 benchmark

⚡ 2 improved benchmarks
❌ 1 regressed benchmark
✅ 1832 untouched benchmarks
🆕 12 new benchmarks
⏩ 46 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	decode_varbin[(1000, 8)]	26.2 µs	34.3 µs	-23.53%
⚡	Simulation	copy_nullable[65536]	1,033.2 µs	579.1 µs	+78.42%
⚡	Simulation	copy_non_nullable[65536]	913.5 µs	799.4 µs	+14.28%
🆕	Simulation	take_chunked_fsl_random[16, 64]	N/A	358 µs	N/A
🆕	Simulation	take_chunked_fsl_random[32, 64]	N/A	515.9 µs	N/A
🆕	Simulation	take_chunked_fsl_random[8, 64]	N/A	275 µs	N/A
🆕	Simulation	take_chunked_fsl_sorted[16, 64]	N/A	354.6 µs	N/A
🆕	Simulation	take_chunked_fsl_sorted[32, 64]	N/A	514.8 µs	N/A
🆕	Simulation	take_chunked_fsl_sorted[8, 64]	N/A	269.3 µs	N/A
🆕	Simulation	take_fsl_chunked_elements_random[16, 64]	N/A	273.7 µs	N/A
🆕	Simulation	take_fsl_chunked_elements_random[32, 64]	N/A	436.2 µs	N/A
🆕	Simulation	take_fsl_chunked_elements_random[8, 64]	N/A	197.7 µs	N/A
🆕	Simulation	take_fsl_chunked_elements_sorted[16, 64]	N/A	268.6 µs	N/A
🆕	Simulation	take_fsl_chunked_elements_sorted[32, 64]	N/A	437.1 µs	N/A
🆕	Simulation	take_fsl_chunked_elements_sorted[8, 64]	N/A	188.1 µs	N/A

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing ct/union-array (772b738) with develop (cbaa3d7)²}

Footnotes

1. 46 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

2. No successful run was found on develop (69f5bc5) during the generation of this report, so cbaa3d7 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c5b0a79a5f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[github-actions]

Polar Signals Profiling Results

Latest Run

Status	Commit	Job	Attempt	Link
🟢 Done	772b738		1	Explore Profiling Data

Previous Runs (13)

Status	Commit	Job	Attempt	Link
🟢 Done	dd86118		1	Explore Profiling Data
🟢 Done	9c9debf		1	Explore Profiling Data
🟢 Done	cd92358		1	Explore Profiling Data
🟢 Done	3233233		1	Explore Profiling Data
🟢 Done	96923e7		1	Explore Profiling Data
🟢 Done	3feabed		1	Explore Profiling Data
🟢 Done	9e8367e		1	Explore Profiling Data
🟢 Done	66e354f		1	Explore Profiling Data
🟢 Done	cfa9d17		1	Explore Profiling Data
🟢 Done	e2186b0		1	Explore Profiling Data
🟢 Done	9238470		1	Explore Profiling Data
🟢 Done	c33685a		1	Explore Profiling Data
🟢 Done	c5b0a79		1	Explore Profiling Data

---

Powered by Polar Signals Cloud

[github-actions]

Benchmarks: Vortex queries 📖

Verdict: No clear signal (low confidence)
Attributed Vortex impact: +1.5%
Engines: DataFusion No clear signal (+4.5%, low confidence) · DuckDB No clear signal (-1.5%, low confidence)
Vortex (geomean): 0.992x ➖
Parquet (geomean): 0.985x ➖
Shifts: Parquet (control) -1.5% · Median polish -0.6%

How to read Verdict and Engines

• Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
• Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
• Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.

---

datafusion / vortex-file-compressed (1.012x ➖, 0↑ 0↓)

name	PR 772b738 (ns)	base 69f5bc5 (ns)	ratio (PR/base)
vortex_q00/datafusion:vortex-file-compressed	9475319	9640865	0.98
vortex_q01/datafusion:vortex-file-compressed	6343540	6087569	1.04

datafusion / parquet (0.968x ➖, 0↑ 0↓)

name	PR 772b738 (ns)	base 69f5bc5 (ns)	ratio (PR/base)
vortex_q00/datafusion:parquet	19961123	20804028	0.96
vortex_q01/datafusion:parquet	4531412	4638739	0.98

duckdb / vortex-file-compressed (0.986x ➖, 0↑ 0↓)

name	PR 772b738 (ns)	base 69f5bc5 (ns)	ratio (PR/base)
vortex_q00/duckdb:vortex-file-compressed	9917885	9980322	0.99
vortex_q01/duckdb:vortex-file-compressed	5771728	5899636	0.98

duckdb / parquet (1.001x ➖, 0↑ 0↓)

name	PR 772b738 (ns)	base 69f5bc5 (ns)	ratio (PR/base)
vortex_q00/duckdb:parquet	23313281	23225257	1.00
vortex_q01/duckdb:parquet	9351650	9364034	1.00

---

No file size changes detected.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.