fix(encoding): prevent bits_per_value=0 in FullZipLayout for FSL with AllNull child

[westonpace]

If we had a fixed-size-list array where each list was non-null but consisted only of null elements (e.g. FSL<2> = [[NULL, NULL], [NULL, NULL], [NULL, NULL]]) then we would store it with bits_per_values=0 and the resulting file would be unreadable.

This PR fixes the issue. Arguably we could do it more efficiently if we turn this into a special case (we currently still store the validity bytes which is redundant information) but this is probably a unique enough case to not worry about too much yet.

Details of fix with four coordinated changes in value.rs:

1. fsl_to_encoding: instead of returning constant(None) early for AllNull children, fall through to the inner-encoding match, set has_validity=true, and emit fsl(dim, has_validity=true, constant(None)). This tells the decoder that validity bytes are present per row.

2. nullable_per_value_fsl AllNull arm: add an all-zero validity buffer to validity_iters and increment bytes_per_row by cum_dim.div_ceil(8). bytes_per_row is now >= 1, so bits_per_value > 0 in the layout.

3. ValueDecompressor::from_fsl: handle Compression::Constant as a terminal case. Only the validity bytes count toward bits_per_value; bits_per_item=0 signals the all-null path to unzip_decompress.

4. unzip_decompress: when bits_per_item==0 produce DataBlock::AllNull for the inner block instead of FixedWidth{bits_per_value:0}. AllNull::into_arrow creates a properly-typed all-null array for any element type.

Backward compatibility: existing files whose AllNull-child FSL used constant(None) directly as the inner encoding still decode via ConstantDecompressor unchanged; only new files take the new path.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The current-reader round trip is fixed, but stable writers must keep emitting value-encoding shapes understood by the latest released stable reader. Preserve the v2.1 forward-compatibility contract—for example, retain a FixedSizeList → Flat terminal with masked leaf-width payload bytes, or restrict the new encoding composition to an unstable file version.

[lance-gatekeeper]

This makes stable v2.1 writers emit FixedSizeList(has_validity=true) → Constant(None). Lance v11.0.0 is the latest stable release and resolves stable to v2.1, but its ValueDecompressor::from_fsl accepts only nested FixedSizeList and a terminal Flat; this terminal reaches _ => unreachable!(). Files written through this new path therefore panic in that released reader. The same-version round-trip test cannot detect this forward-compatibility break. Please keep stable writes on an encoding shape v11 understands (for example, a Flat terminal with masked payload bytes at the actual leaf width), or version-gate this composition to an unstable format.

Reproducer against v11.0.0

#[test]
fn accepts_nested_constant_fsl_value_encoding() {
    let encoding = super::ProtobufUtils21::fsl(
        4,
        true,
        super::ProtobufUtils21::constant(None),
    );
    let Compression::FixedSizeList(fsl) = encoding.compression.unwrap() else {
        panic!("expected fixed-size-list encoding");
    };

    let _ = ValueDecompressor::from_fsl(fsl.as_ref());
}

cargo test -p lance-encoding accepts_nested_constant_fsl_value_encoding -- --nocapture

Expected: the released stable reader accepts the stable v2.1 encoding. Observed: internal error: entered unreachable code in ValueDecompressor::from_fsl.

[westonpace]

The old writer would have crashed without this change. We are not making things worse in terms of compatibility.

[lance-gatekeeper]

The old writer failure is the bug being fixed, but it does not make the new stable encoding forward-compatible: this patch makes a v2.1 file readable by the current reader while the released stable v11 reader still panics on its nested Constant terminal. The repository contract requires every new stable-format write to remain readable by the released stable reader, even when the old writer could not successfully produce that edge case, so this finding remains.

[westonpace]

I accept this as a known risk. Masked payload bytes would be inefficient and this is a fairly niche corner case already.

[westonpace]

Actually, I guess I'll try and fix it.

[lance-gatekeeper]

The follow-up commit only permits zero-width pages in the current FullZip scheduler; it does not change fsl_to_encoding, which still emits stable v2.1 FixedSizeList → Constant. The released v11 reader therefore still reaches its unsupported terminal, so the compatibility finding remains.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The follow-up broadens the current reader to accept zero-width FullZip pages, but it does not address the stable-format compatibility blocker: v2.1 writers still emit a FixedSizeList → Constant terminal that the released v11 reader cannot decode. Keep stable output on a v11-readable shape (such as FixedSizeList → Flat with masked leaf-width payload bytes), or restrict this composition to an unstable format version.