Parquet Java ALP Implementation

[vinooganesh]

cc @julienledem @alamb @emkornfield @prtkgaur

Rationale for this change

Reworks the ALP encoding implementation to address emkornfield's architectural feedback on PR #3390. The original buffered all values in memory and decoded eagerly. This makes the writer incremental (encode per-vector as values arrive) and the reader lazy (decode on demand), matching how other Parquet encodings work.

Builds on Julien Le Dem's original implementation (#3390). File structure, integration points, core math, and interop test infrastructure all come from his work. The rework focused on the internal writer/reader plumbing.

What changes are included in this PR?

Architecture (addressing review feedback):

• Incremental writer. Values buffer in a fixed-size vector, each full vector encodes and flushes immediately.
• Lazy reader. Vectors decode on first access via offset array, skip() is O(1).
• Interleaved page layout so each vector is self-contained.
• Extracted AlpValuesReader abstract base class for shared decode logic (float/double readers only implement decodeBody).
• Preset caching. Full parameter search for first 8 vectors, top 5 combos cached for the rest.

Spec compliance:

• Fixed packed data size formula to ceil(n * bitWidth / 8) (reusing BytesUtils.paddedByteCountFromBits).
• Signed frame-of-reference range in the size estimator (a prior unsigned max - min overstated the span on mixed-sign vectors).
• Reads little-endian through ByteBuffer's typed getters on a LITTLE_ENDIAN-ordered buffer.
• Uses parquet-encoding's BytePacker instead of custom bit-packing.
• Capped max vector size at 32768 to prevent uint16 overflow in num_exceptions.
• bitWidth bounds checks in the readers (> 32 float / > 64 double throw).

Configuration:

• withAlpEncoding(...) and withAlpVectorSize(...) on ParquetProperties.Builder and the Hadoop ParquetWriter.Builder, globally or per-column.
• Threaded through DefaultV1ValuesWriterFactory and DefaultV2ValuesWriterFactory so per-column overrides work.
• Vector size defaults to 1024; validated against AlpConstants min/max bounds eagerly at builder time.
• The enabled flag and vector size are bundled into a single per-column AlpConfig inside ParquetProperties (one ColumnProperty<AlpConfig> rather than two); public setters/getters are unchanged.

Reader null tolerance (bug fix):

• AlpValuesReader was asserting num_elements == page.valuesCount, which fails on optional columns with nulls (num_elements is the encoded non-null count, valuesCount is the page row count including nulls). Relaxed to num_elements <= valuesCount.

Integration:

• Wired ALP into both DefaultV1ValuesWriterFactory and DefaultV2ValuesWriterFactory as a fallback data-page encoding for FLOAT/DOUBLE.

Are these changes tested?

Yes, extensively. The full parquet-column module (840 tests) and all downstream non-ALP modules (arrow/avro/protobuf/thrift/variant/cli, 908 tests) pass; the parquet-hadoop ALP interop tests pass. Coverage spans:

Correctness & spec:

• Encoder/decoder tests that construct ALP page bytes directly per the spec and feed them to the reader without going through the writer — catches bugs where writer and reader agree with each other but disagree with the spec.
• Bit-packing round-trip across every width (int 1–32, long 1–64), top bit exercised.
• Full-bit-space fuzz (double & float): random raw bit patterns — every NaN payload, subnormals, ±0, ±Inf, chaotic mixed magnitudes — with strict raw-bit round-trip (verifies NaN payloads are preserved exactly).
• Exceptions (NaN/Inf/−0.0, one/all-exception vectors), nulls (all-null and partial-null pages), every partial-vector remainder, skip across vector boundaries.
• Extreme frame-of-reference widths: values engineered to force 63-bit (non-overflow) and 64-bit (signed subtraction overflows → modular reconstruction) FOR deltas, plus 32-bit for float — all lossless with zero exceptions.
• Preset-cache correctness under distribution shift within a row group.

Integration (production paths):

• Dictionary → ALP fallback: dictionary enabled (the real default) with overflow forcing fallback to ALP mid-column — the path all prior tests skipped by disabling the dictionary.
• ALP under Snappy / Gzip / Zstd compression.
• Statistics correctness on ALP columns: NaN excluded from min/max, null_count correct.
• ALP on a repeated (nested) double field with varying repetition/definition levels.

Robustness & scale:

• Reader rejects malformed/truncated/corrupt pages cleanly (no crash, OOM, hang, or OOB).
• Deterministic allocator-leak test: a counting ByteBufferAllocator verifies all off-heap buffers are released across 200 write/reset page cycles.
• Large-scale round-trips (2M values; 500k rows across many row groups through the full pipeline).

Cross-language verification

The Arrow C++ ALP decoder (apache/arrow#48345) reads every Java-written fixture bit-exact against the canonical _expect.csv truth tables. Local verification covers the full {V1, V2} × {vs1024, vs4096} matrix plus the corner-case and extreme-value columns: >1.5M values, 0 mismatches. Six representative fixtures are submitted as a stacked PR (prtkgaur/parquet-testing#1) toward apache/parquet-testing#100 for the other-language readers to verify in CI.

Test fixtures for #100: generateAlpFixturesAtMultipleVectorSizes re-encodes the source datasets as Java ALP across page version / vector size / dataset, each verified bit-exact; generateAndVerifyCornerCaseFixture writes a small synthetic file whose columns each hit a specific corner case (no/one/all exceptions, NaN/Inf/−0.0, constant, differing exponents, nulls, wide and extreme FOR ranges) with a _expect.csv sidecar emitted from the construction recipe.

Are there any user-facing changes?

• Users can enable ALP encoding for FLOAT and DOUBLE columns via ParquetProperties.withAlpEncoding() (or the Hadoop ParquetWriter.Builder), globally or per-column.
• Users can configure the ALP vector size via withAlpVectorSize(int), also globally or per-column. Default is 1024.

Known limitation: ALP is not yet in the parquet-format spec

ALP is not yet in the parquet-format Thrift spec (apache/parquet-format#533, PR #548). As a temporary bridge, parquet-format-structures/src/main/perl/patch-alp-encoding.pl runs at build time to inject ALP(10) into the generated Encoding enum, and TestParquetMetadataConverter skips ALP in its enum round-trip. Both are clearly marked to be removed once parquet-format ships ALP and parquet-java bumps its dependency. The ALP = 10 value is provisional: if the spec assigns a different value, files written now would need regeneration. Flagging this explicitly so it can be weighed in the vote.

[prtkgaur]

The code organization looks good to me and the code follows the spec. I looked for areas of any extra buffer allocations which might impact performance and I think it is optimally written.

I think we should add a few benchmarks and publish numbers from them.

Thanks for working on this Vinoo!

[prtkgaur]

Wanted to make sure we have the following testing.

For the cross compatibility testing are we making sure that we write both V1 and V2 pages and the implementation in other language is able to read it.

[wgtmac]

Sorry for a late review. I haven't reviewed tests yet but the production code looks pretty solid. I've left some comments. Please let me know what you think.

[wgtmac]

We should probably remove these lines by either releasing a new version of parquet-format, or after merging #3709

[vinooganesh]

Your timing on this was good. #3709 merged a couple of days after you left the comment, so parquet.thrift is now inlined in the repo and I was able to take the approach you suggested.

In 229e0d9, ALP = 10 is declared directly in the Encoding enum, and the perl script and the exec-maven-plugin execution that ran it are both gone. I verified the generated Encoding.java still carries ALP(10) after BYTE_STREAM_SPLIT(9) with no patch step involved.

There is one wrinkle I want your opinion on. dev/update-parquet-thrift.sh overwrites parquet.thrift wholesale from upstream, so the next time anyone runs it the ALP entry will disappear silently. It also rewrites the parquet-format.version sidecar, so a note there would not survive either. For now I have put a clearly marked notice in the enum itself saying it is a local addition that needs re-applying until ALP is accepted into parquet-format, on the grounds that at least the loss shows up in a diff. If you would prefer this handled a different way, such as a guard in the update script, I am glad to change it.

[wgtmac]

This int arithmetic can overflow for a forged numElements. The result can be negative or trigger a huge allocation.

[vinooganesh]

Thanks for catching this, it's a good find. You're right that numElements is only bounded by valuesCount, and since a forged file controls that value it can go all the way up to Integer.MAX_VALUE. Two separate things went wrong from there. The + vectorSize - 1 overflowed into a negative count, and even when it didn't overflow, a large count drove a big allocation before anything had been validated.

While fixing it I found the allocation is genuinely reachable rather than theoretical. MultiBufferInputStream.slice() calls ByteBuffer.allocate(length) before it checks for EOF, so a small forged page really can ask for hundreds of megabytes.

This is fixed in 338fb99. numVectors is computed in long now and checked against the bytes actually present before anything gets allocated. Since every vector needs a 4 byte offset entry plus at least its ALP and FOR headers, that gives a firm ceiling on how many vectors the remaining bytes could possibly describe, and anything above it throws a ParquetDecodingException that says how far short the page is.

[wgtmac]

These offsets are trusted without validating the first entry, ordering, or bounds. Decode also ignores the next offset, so malformed bytes can silently produce the wrong vector.

[vinooganesh]

You're right on all three counts, thank you. Fixed in 338fb99.

initFromPage now validates the whole offset array before any vector is decoded. The first offset has to equal the offset array size, offsets have to increase by at least one vector's worth of fixed headers, and each one has to start early enough to leave room for those headers before the body ends.

The observation about decode ignoring the next offset was the more important half of this, since that's the case that quietly returns wrong values instead of failing. decodeVector now takes each vector's end position from the following offset, or from the end of the body for the last vector, and rejects a packed body or exception block that runs past it.

[wgtmac]

This addition can overflow. A large skip can pass the check and make pageValueIndex negative.

[vinooganesh]

Good catch, thank you. Fixed in 338fb99 by comparing against the remaining count with n > totalCount - pageValueIndex rather than adding to pageValueIndex, so a large n can no longer overflow past the check and leave the reader at a negative index. I added testSkipRejectsOverflowingCount to cover it.

[wgtmac]

With the default 20K-row page limit, this cadence yields only about two samples per page, while eight are required. The preset cache never builds on normal pages.

[vinooganesh]

Thank you for working through the arithmetic on this one, it turned out to be the most valuable comment in the review. I checked it and the numbers come out exactly as you described. At the default 20k row page limit with a 1024 vector size a page holds roughly 19 vectors, and rowgroupSampleJump works out to 15, so each page contributes about two samples. Because reset() cleared sampledParams and vectorsProcessed at every page boundary, the count never got near SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP, buildPresetCache() never ran, and findBest*ParamsWithPresets was effectively dead code. Every vector was paying for a full parameter search.

Fixed in eb0e594. The sampler is meant to operate over a rowgroup rather than a page, so the sampling state now survives reset() and samples accumulate across pages the way the design intended. I added testPresetCacheBuildsAcrossPageBoundaries, which fails without the change.

One consequence I want to flag, since it isn't obvious. Now that the preset path is genuinely reachable, encoded output can shift slightly, because the search narrows to the top MAX_PRESET_COMBINATIONS pairs instead of trying everything. That is the intended ALP sampling behavior and it matches the C++ implementation. Round tripping is still exact, since the chosen pair is recorded in each vector's header and exceptions cover anything it cannot represent.

[wgtmac]

Why the writer implementations are nested static class but the readers are in the separate files? Should we make them consistent?

[vinooganesh]

Fair question, and the honest answer is that I was not making a deliberate choice here so much as following the layout that was already in the codebase. ByteStreamSplit is structured exactly the same way: ByteStreamSplitValuesWriter nests all five typed writers as public static classes, while its readers live in their own files as ByteStreamSplitValuesReaderForDouble, ByteStreamSplitValuesReaderForFloat and so on. ALP mirrors that, with the two writers nested and AlpValuesReaderForDouble and AlpValuesReaderForFloat separate.

So I agree the asymmetry is odd on its own terms, but it is the existing convention rather than something new, and matching the neighbouring encoding seemed more useful than being internally tidy in a way nothing else in the package is.

That said I do not feel strongly about it. If you would rather ALP be consistent within itself, splitting the two writers into their own files is a small and low risk change and I am happy to do it. It would just leave ALP looking different from ByteStreamSplit instead.

[wgtmac]

This line is not aligned on my editor.

[vinooganesh]

Thanks for spotting that. The cause is that the cell contains 4B × numVectors, and the HTML entity is seven characters in the source but renders as a single glyph, so the box could never line up in both your editor and the rendered javadoc at the same time.

Fixed in 78c1d18 by using the literal character and padding the cell to the right width, so every row is 73 characters in both views. The same diagram had been copied into AlpValuesReader with the same problem, so I fixed it there too.

[wgtmac]

This does not follow other interop tests in the repo to download it from parquet-testing repo.

[vinooganesh]

You're right, and I would like to move this over to the standard harness. The blocker is that the ALP fixture files are not in parquet-testing yet. They are proposed in apache/parquet-testing#100, which is still open.

Once that merges I will convert this to use InterOpTester.GetInterOpFile with a pinned changeset, the same as TestInterOpReadByteStreamSplit and the other interop tests. Until then the test reads from a local directory and skips when it is not present, which does mean it is effectively a no-op in CI, so I understand it is not carrying its weight right now.

If you have any influence over getting parquet-testing #100 looked at, that would help a lot. Cross language read back is the part of this work I am least able to demonstrate on my own, so it matters for the vote more than for this test alone.

[wgtmac]

vectorsProcessed resets for every page. With the default page size, we only collect about two samples, but need eight to build the cache. So cachedPresets is never used in the normal path.

[vinooganesh]

This is the same underlying problem as your comment on the sampling cadence, and it's fixed in eb0e594. I've written up the details in that thread rather than repeating them here. Thanks for flagging it from both angles, it made the problem much easier to pin down.

[wgtmac]

Why not directly use withAlp(AlpConfig)? It looks a little bit over-complicated with the current API, especially buildAlp().

[vinooganesh]

You're right that this was more complicated than it needed to be, thanks for pushing on it. The builder was keeping the enabled flag and the vector size as two separate ColumnProperty scaffolds, merging them into an AlpConfig in buildAlp(), and then splitting them apart again in the copy constructor. Nothing actually needed that round trip.

Fixed in 99ae547. There is now a single ColumnProperty, with withAlp(AlpConfig) and withAlp(columnPath, AlpConfig) as the primary API. I kept withAlpEncoding and withAlpVectorSize as thin read-modify-write conveniences on top, since they match the ergonomics of withByteStreamSplitEncoding and it means no existing caller had to change, including ParquetWriter.Builder. buildAlp and the copy constructor split are both gone.

I also moved the vector size validation into the AlpConfig constructor, which felt like the right home for it since an invalid size should be rejected however the config was built. That had the nice side effect of removing the last reference to AlpConstants from ParquetProperties, which made your visibility comment straightforward to act on.

One thing I want to flag for you. To let the convenience setters modify an existing config rather than replace it, I added getDefaultValue and getValue to ColumnProperty.Builder. That class is package private so it is not public API, but it is a shared file rather than something ALP specific, so please say if you would rather I found another way around it.