feat: Load Parquet page indexes for external row selections by haohuaijin · Pull Request #25460 · apache/datafusion

haohuaijin · 2026-09-18T02:26:16Z

Which issue does this PR close?

Closes Support Parquet page index loading for external row selections #25454.

Rationale for this change

External row selections can benefit from offset indexes without a page-pruning predicate, or when row-group statistics fully match the scan predicate.

For example, an external index can select rows for a service while the Parquet scan checks only a time range. A row group entirely within that range is fully matched for the predicate, but the external selection can still exclude most rows.

Offset indexes allow the reader to locate relevant pages by row position. Column-index statistics are unnecessary for this path because the selection already identifies the rows to read.

What changes are included in this PR?

Explicitly honor enable_page_index before either loading path.
Load indexes when a surviving row group has a partial external selection and an offset index on a projected Parquet leaf column.
Detect partial selections using a second alternating run instead of counting all selected and skipped rows. Bitmap selections use a streaming iterator without materializing selectors.
Resolve the projection with the existing read-plan logic, handling nested fields and excluding virtual columns.
Keep the existing predicate-based fallback, including its handling of predicate columns outside the output projection.

The projection check controls whether loading is triggered; it does not restrict index I/O to projected columns. Read-plan caching, decoder refactoring, and arrow-rs changes are outside this PR.

What is the testing strategy for this PR?

Unit tests cover offset-only metadata, missing indexes, fully matched and skipped row groups, selector/bitmap representations, empty/uniform selections, and indexes only on unprojected columns. They also verify the predicate-based fallback.

A V1 Parquet test selects the final 100 of 10,000 rows and checks exact output values and reduced bytes_scanned, both without a predicate and with a fully matching predicate.

The following checks passed, including all 11 page-index tests:

cargo test -p datafusion-datasource-parquet --lib page_index
cargo clippy -p datafusion-datasource-parquet --all-targets --all-features -- -D warnings
cargo fmt --all -- --check

The repository-wide ./dev/rust_lint.sh could not proceed because the local Python environment is missing PyYAML. The targeted checks above passed.

Are there any user-facing changes?

External partial row selections can use offset indexes to skip unselected pages without a useful page-pruning predicate. Query results and public APIs are unchanged, and disabling page indexes still disables this loading path.

codecov-commenter · 2026-09-18T02:51:03Z

Codecov Report

❌ Patch coverage is 98.82629% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.46%. Comparing base (d20936c) to head (3f4da1b).
⚠️ Report is 47 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/datasource-parquet/src/opener/mod.rs	98.80%	3 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #25460      +/-   ##
==========================================
+ Coverage   82.38%   82.46%   +0.07%     
==========================================
  Files        1138     1140       +2     
  Lines      434491   436818    +2327     
  Branches   434491   436818    +2327     
==========================================
+ Hits       357969   360210    +2241     
+ Misses      54875    54849      -26     
- Partials    21647    21759     +112

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

jayzhan211

Thanks @haohuaijin , here are some suggestions

haohuaijin · 2026-09-19T09:42:10Z

Thanks for your reviews @jayzhan211 , i applied the suggestions in f5f1b7f

jayzhan211

Thanks @haohuaijin 🚀

alamb · 2026-09-22T20:52:15Z

run benchmark clickbench clickbench_partitioned

alamb

Thanks @haohuaijin and @jayzhan211 -- I an a little worried about the overhead of this change as it seems to make an entire read plan and then discard it -- if it makes a read plan, shouldn't we be using that if possible?

adriangbot · 2026-09-22T20:53:23Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5783971488-2613-hpz5v 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing fix/parquet-row-selection-page-index (d104ad0) to d20936c (merge-base) diff

Run configuration

run benchmark clickbench

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-09-22T20:54:38Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5783971488-2614-ljptx 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing fix/parquet-row-selection-page-index (d104ad0) to d20936c (merge-base) diff

Run configuration

run benchmark clickbench_partitioned

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-09-22T20:59:32Z

Benchmark for this request failed before finishing (Kubernetes reason: BackoffLimitExceeded).

Benchmarks requested: clickbench

Runner log (last 40 lines)

h2o_medium_window_sorted_parquet: Window Top-N over a declared-sorted h2o input, medium dataset (1e8 rows), source file format is parquet
h2o_big_window_sorted_parquet:    Window Top-N over a declared-sorted h2o input, large dataset (1e9 rows),  source file format is parquet
h2o_small_parquet:              h2oai benchmark with small dataset (1e7 rows) for groupby,  file format is parquet
h2o_medium_parquet:             h2oai benchmark with medium dataset (1e8 rows) for groupby, file format is parquet
h2o_big_parquet:                h2oai benchmark with large dataset (1e9 rows) for groupby,  file format is parquet
h2o_small_join_parquet:         h2oai benchmark with small dataset (1e7 rows) for join,  file format is parquet
h2o_medium_join_parquet:        h2oai benchmark with medium dataset (1e8 rows) for join, file format is parquet
h2o_big_join_parquet:           h2oai benchmark with large dataset (1e9 rows) for join,  file format is parquet
h2o_small_window_parquet:       Extended h2oai benchmark with small dataset (1e7 rows) for window,  file format is parquet
h2o_medium_window_parquet:      Extended h2oai benchmark with medium dataset (1e8 rows) for window, file format is parquet
h2o_big_window_parquet:         Extended h2oai benchmark with large dataset (1e9 rows) for window,  file format is parquet

# Join Order Benchmark (IMDB)
imdb:                   Join Order Benchmark (JOB) using the IMDB dataset converted to parquet

# Micro-Benchmarks (specific operators and features)
cancellation:           How long cancelling a query takes
asof_join:              ASOF join workloads varying size, ordering, grouping, match direction, and payload width
nlj:                    Benchmark for simple nested loop joins, testing various join scenarios
hj:                     Benchmark for simple hash joins, testing various join scenarios
smj:                    Benchmark for simple sort merge joins, testing various join scenarios
dict:                   Benchmark for dictionary-encoded group-by scenarios
array_agg_distinct:     1000K-group, two-row-per-group array_agg(DISTINCT) benchmark
compile_profile:        Compile and execute TPC-H across selected Cargo profiles, reporting timing and binary size


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Supported Configuration (Environment Variables)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
DATA_DIR            directory to store datasets
CARGO_COMMAND       command that runs the benchmark binary
DATAFUSION_DIR      directory to use (default /workspace/datafusion-base)
RESULTS_NAME        folder where the benchmark files are stored
PREFER_HASH_JOIN    Prefer hash join algorithm (default true)
SIMULATE_LATENCY    Simulate object store latency to mimic S3 (default false)
DATAFUSION_*        Set the given datafusion configuration


stderr:

Kubernetes message

Job has reached the specified backoff limit

File an issue against this benchmark runner

adriangbot · 2026-09-22T21:05:29Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing fix/parquet-row-selection-page-index (d104ad0) to d20936c (merge-base) diff

Run configuration

run benchmark clickbench_partitioned

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

Comparing HEAD and fix_parquet-row-selection-page-index
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃       HEAD ┃ fix_parquet-row-selection-page-index ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │    1.25 ms │                              1.25 ms │     no change │
│ QQuery 1  │   11.66 ms │                             11.54 ms │     no change │
│ QQuery 2  │   36.62 ms │                             36.82 ms │     no change │
│ QQuery 3  │   31.46 ms │                             31.15 ms │     no change │
│ QQuery 4  │  253.60 ms │                            249.45 ms │     no change │
│ QQuery 5  │  280.46 ms │                            294.17 ms │     no change │
│ QQuery 6  │    1.30 ms │                              1.30 ms │     no change │
│ QQuery 7  │   13.20 ms │                             12.97 ms │     no change │
│ QQuery 8  │  355.90 ms │                            344.87 ms │     no change │
│ QQuery 9  │  498.36 ms │                            485.45 ms │     no change │
│ QQuery 10 │   66.56 ms │                             64.47 ms │     no change │
│ QQuery 11 │   76.28 ms │                             75.41 ms │     no change │
│ QQuery 12 │  272.09 ms │                            280.19 ms │     no change │
│ QQuery 13 │  389.51 ms │                            376.43 ms │     no change │
│ QQuery 14 │  291.11 ms │                            290.63 ms │     no change │
│ QQuery 15 │  300.84 ms │                            302.57 ms │     no change │
│ QQuery 16 │  637.02 ms │                            649.14 ms │     no change │
│ QQuery 17 │  643.18 ms │                            641.01 ms │     no change │
│ QQuery 18 │ 1357.78 ms │                           1328.42 ms │     no change │
│ QQuery 19 │   28.62 ms │                             28.38 ms │     no change │
│ QQuery 20 │  520.44 ms │                            522.71 ms │     no change │
│ QQuery 21 │  515.96 ms │                            512.36 ms │     no change │
│ QQuery 22 │  994.76 ms │                            995.77 ms │     no change │
│ QQuery 23 │ 3120.85 ms │                           3126.45 ms │     no change │
│ QQuery 24 │   40.87 ms │                             40.53 ms │     no change │
│ QQuery 25 │  105.10 ms │                            106.14 ms │     no change │
│ QQuery 26 │   41.82 ms │                             41.65 ms │     no change │
│ QQuery 27 │  516.40 ms │                            518.42 ms │     no change │
│ QQuery 28 │ 2935.47 ms │                           2964.21 ms │     no change │
│ QQuery 29 │   42.41 ms │                             42.39 ms │     no change │
│ QQuery 30 │  320.21 ms │                            310.27 ms │     no change │
│ QQuery 31 │  285.07 ms │                            286.06 ms │     no change │
│ QQuery 32 │  990.44 ms │                           1029.26 ms │     no change │
│ QQuery 33 │ 1526.03 ms │                           1554.76 ms │     no change │
│ QQuery 34 │ 1538.19 ms │                           1612.87 ms │     no change │
│ QQuery 35 │  321.30 ms │                            312.28 ms │     no change │
│ QQuery 36 │   69.05 ms │                             71.97 ms │     no change │
│ QQuery 37 │   38.84 ms │                             35.86 ms │ +1.08x faster │
│ QQuery 38 │   39.82 ms │                             40.92 ms │     no change │
│ QQuery 39 │  142.34 ms │                            157.45 ms │  1.11x slower │
│ QQuery 40 │   14.99 ms │                             15.57 ms │     no change │
│ QQuery 41 │   14.70 ms │                             15.04 ms │     no change │
│ QQuery 42 │   13.49 ms │                             13.90 ms │     no change │
└───────────┴────────────┴──────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                   ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                                   │ 19695.34ms │
│ Total Time (fix_parquet-row-selection-page-index)   │ 19832.43ms │
│ Average Time (HEAD)                                 │   458.03ms │
│ Average Time (fix_parquet-row-selection-page-index) │   461.22ms │
│ Queries Faster                                      │          1 │
│ Queries Slower                                      │          1 │
│ Queries with No Change                              │         41 │
│ Queries with Failure                                │          0 │
└─────────────────────────────────────────────────────┴────────────┘

Distribution per query (min / mean ±stddev / max):

Comparing HEAD and fix_parquet-row-selection-page-index
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃                                  HEAD ┃  fix_parquet-row-selection-page-index ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │          1.25 / 4.15 ±5.64 / 15.44 ms │          1.25 / 4.04 ±5.48 / 15.00 ms │     no change │
│ QQuery 1  │        11.66 / 11.97 ±0.24 / 12.31 ms │        11.54 / 11.78 ±0.16 / 12.01 ms │     no change │
│ QQuery 2  │        36.62 / 37.01 ±0.38 / 37.74 ms │        36.82 / 37.10 ±0.28 / 37.50 ms │     no change │
│ QQuery 3  │        31.46 / 32.10 ±0.66 / 33.27 ms │        31.15 / 31.85 ±0.57 / 32.69 ms │     no change │
│ QQuery 4  │     253.60 / 259.38 ±3.92 / 265.42 ms │     249.45 / 261.03 ±8.68 / 270.32 ms │     no change │
│ QQuery 5  │     280.46 / 288.43 ±5.31 / 293.82 ms │     294.17 / 295.40 ±0.96 / 296.53 ms │     no change │
│ QQuery 6  │           1.30 / 1.45 ±0.24 / 1.92 ms │           1.30 / 1.46 ±0.24 / 1.93 ms │     no change │
│ QQuery 7  │        13.20 / 13.26 ±0.04 / 13.32 ms │        12.97 / 13.25 ±0.23 / 13.56 ms │     no change │
│ QQuery 8  │     355.90 / 363.28 ±8.44 / 379.70 ms │     344.87 / 350.83 ±3.66 / 356.21 ms │     no change │
│ QQuery 9  │     498.36 / 508.80 ±9.20 / 520.36 ms │    485.45 / 505.92 ±12.31 / 521.34 ms │     no change │
│ QQuery 10 │        66.56 / 70.93 ±6.57 / 83.94 ms │        64.47 / 67.46 ±3.63 / 74.43 ms │     no change │
│ QQuery 11 │        76.28 / 77.98 ±2.21 / 82.30 ms │        75.41 / 80.19 ±4.49 / 88.52 ms │     no change │
│ QQuery 12 │     272.09 / 281.20 ±8.89 / 292.53 ms │    280.19 / 292.41 ±12.99 / 316.30 ms │     no change │
│ QQuery 13 │    389.51 / 401.62 ±10.55 / 416.31 ms │     376.43 / 390.73 ±8.84 / 402.38 ms │     no change │
│ QQuery 14 │     291.11 / 303.03 ±8.30 / 312.45 ms │     290.63 / 300.71 ±8.26 / 315.60 ms │     no change │
│ QQuery 15 │    300.84 / 315.36 ±11.62 / 327.00 ms │     302.57 / 313.60 ±9.06 / 325.30 ms │     no change │
│ QQuery 16 │    637.02 / 658.57 ±20.00 / 695.43 ms │    649.14 / 664.28 ±11.94 / 679.07 ms │     no change │
│ QQuery 17 │    643.18 / 671.36 ±19.24 / 701.77 ms │    641.01 / 676.61 ±24.08 / 711.74 ms │     no change │
│ QQuery 18 │ 1357.78 / 1380.33 ±20.04 / 1410.82 ms │ 1328.42 / 1376.83 ±30.26 / 1419.91 ms │     no change │
│ QQuery 19 │        28.62 / 29.33 ±0.63 / 30.48 ms │       28.38 / 41.39 ±10.50 / 51.66 ms │  1.41x slower │
│ QQuery 20 │    520.44 / 530.27 ±11.76 / 552.04 ms │     522.71 / 530.74 ±4.50 / 536.06 ms │     no change │
│ QQuery 21 │     515.96 / 521.94 ±3.15 / 524.70 ms │    512.36 / 526.78 ±10.08 / 538.95 ms │     no change │
│ QQuery 22 │  994.76 / 1023.62 ±19.52 / 1049.00 ms │  995.77 / 1017.37 ±11.62 / 1028.09 ms │     no change │
│ QQuery 23 │ 3120.85 / 3175.17 ±46.24 / 3256.32 ms │ 3126.45 / 3164.41 ±31.81 / 3209.71 ms │     no change │
│ QQuery 24 │        40.87 / 42.77 ±2.16 / 46.98 ms │        40.53 / 43.18 ±3.32 / 49.67 ms │     no change │
│ QQuery 25 │     105.10 / 107.16 ±1.60 / 109.49 ms │     106.14 / 114.10 ±8.26 / 127.89 ms │  1.06x slower │
│ QQuery 26 │        41.82 / 45.94 ±5.92 / 57.71 ms │        41.65 / 42.41 ±0.51 / 43.01 ms │ +1.08x faster │
│ QQuery 27 │     516.40 / 524.62 ±5.81 / 534.49 ms │     518.42 / 527.27 ±7.29 / 540.16 ms │     no change │
│ QQuery 28 │ 2935.47 / 2969.69 ±27.20 / 3011.78 ms │ 2964.21 / 2988.50 ±16.67 / 3015.40 ms │     no change │
│ QQuery 29 │        42.41 / 43.56 ±1.19 / 45.67 ms │       42.39 / 56.50 ±20.63 / 97.07 ms │  1.30x slower │
│ QQuery 30 │    320.21 / 333.96 ±10.03 / 350.76 ms │    310.27 / 328.37 ±11.76 / 341.96 ms │     no change │
│ QQuery 31 │     285.07 / 298.26 ±7.88 / 308.57 ms │    286.06 / 302.42 ±11.51 / 321.21 ms │     no change │
│ QQuery 32 │  990.44 / 1020.28 ±17.77 / 1042.25 ms │ 1029.26 / 1073.08 ±41.63 / 1149.12 ms │  1.05x slower │
│ QQuery 33 │ 1526.03 / 1567.83 ±25.84 / 1589.76 ms │ 1554.76 / 1593.43 ±25.42 / 1624.42 ms │     no change │
│ QQuery 34 │ 1538.19 / 1604.75 ±52.39 / 1666.10 ms │ 1612.87 / 1640.29 ±23.95 / 1668.06 ms │     no change │
│ QQuery 35 │     321.30 / 329.46 ±5.70 / 337.50 ms │    312.28 / 356.95 ±34.56 / 408.10 ms │  1.08x slower │
│ QQuery 36 │        69.05 / 76.58 ±5.28 / 83.90 ms │        71.97 / 81.39 ±7.58 / 91.35 ms │  1.06x slower │
│ QQuery 37 │        38.84 / 45.26 ±5.97 / 55.51 ms │        35.86 / 38.71 ±1.62 / 40.50 ms │ +1.17x faster │
│ QQuery 38 │        39.82 / 43.29 ±2.50 / 47.47 ms │        40.92 / 48.55 ±6.36 / 60.09 ms │  1.12x slower │
│ QQuery 39 │    142.34 / 159.09 ±13.21 / 179.87 ms │     157.45 / 167.60 ±7.73 / 180.11 ms │  1.05x slower │
│ QQuery 40 │        14.99 / 16.81 ±2.80 / 22.38 ms │        15.57 / 16.72 ±1.67 / 20.05 ms │     no change │
│ QQuery 41 │        14.70 / 18.00 ±5.42 / 28.81 ms │        15.04 / 15.11 ±0.10 / 15.31 ms │ +1.19x faster │
│ QQuery 42 │        13.49 / 15.83 ±3.95 / 23.70 ms │        13.90 / 14.15 ±0.23 / 14.57 ms │ +1.12x faster │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                   ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                                   │ 20223.73ms │
│ Total Time (fix_parquet-row-selection-page-index)   │ 20404.92ms │
│ Average Time (HEAD)                                 │   470.32ms │
│ Average Time (fix_parquet-row-selection-page-index) │   474.53ms │
│ Queries Faster                                      │          4 │
│ Queries Slower                                      │          8 │
│ Queries with No Change                              │         31 │
│ Queries with Failure                                │          0 │
└─────────────────────────────────────────────────────┴────────────┘

Resource Usage

clickbench_partitioned — base (merge-base)

Metric	Value
Wall time	105.0s
Peak memory	10.7 GiB
Avg memory	4.1 GiB
CPU user	1035.1s
CPU sys	75.8s
Peak spill	0 B

clickbench_partitioned — branch

Metric	Value
Wall time	105.0s
Peak memory	11.6 GiB
Avg memory	4.5 GiB
CPU user	1038.3s
CPU sys	78.3s
Peak spill	0 B

File an issue against this benchmark runner

haohuaijin · 2026-09-23T04:00:30Z

run benchmark clickbench clickbench_partitioned

adriangbot · 2026-09-23T04:04:01Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5788767202-2621-dhm6j 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing fix/parquet-row-selection-page-index (3f4da1b) to d20936c (merge-base) diff

Run configuration

run benchmark clickbench

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-09-23T04:04:09Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5788767202-2622-lktb5 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing fix/parquet-row-selection-page-index (3f4da1b) to d20936c (merge-base) diff

Run configuration

run benchmark clickbench_partitioned

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-09-23T04:08:27Z

Benchmark for this request failed before finishing (Kubernetes reason: BackoffLimitExceeded).

Benchmarks requested: clickbench

Runner log (last 40 lines)

h2o_medium_window_sorted_parquet: Window Top-N over a declared-sorted h2o input, medium dataset (1e8 rows), source file format is parquet
h2o_big_window_sorted_parquet:    Window Top-N over a declared-sorted h2o input, large dataset (1e9 rows),  source file format is parquet
h2o_small_parquet:              h2oai benchmark with small dataset (1e7 rows) for groupby,  file format is parquet
h2o_medium_parquet:             h2oai benchmark with medium dataset (1e8 rows) for groupby, file format is parquet
h2o_big_parquet:                h2oai benchmark with large dataset (1e9 rows) for groupby,  file format is parquet
h2o_small_join_parquet:         h2oai benchmark with small dataset (1e7 rows) for join,  file format is parquet
h2o_medium_join_parquet:        h2oai benchmark with medium dataset (1e8 rows) for join, file format is parquet
h2o_big_join_parquet:           h2oai benchmark with large dataset (1e9 rows) for join,  file format is parquet
h2o_small_window_parquet:       Extended h2oai benchmark with small dataset (1e7 rows) for window,  file format is parquet
h2o_medium_window_parquet:      Extended h2oai benchmark with medium dataset (1e8 rows) for window, file format is parquet
h2o_big_window_parquet:         Extended h2oai benchmark with large dataset (1e9 rows) for window,  file format is parquet

# Join Order Benchmark (IMDB)
imdb:                   Join Order Benchmark (JOB) using the IMDB dataset converted to parquet

# Micro-Benchmarks (specific operators and features)
cancellation:           How long cancelling a query takes
asof_join:              ASOF join workloads varying size, ordering, grouping, match direction, and payload width
nlj:                    Benchmark for simple nested loop joins, testing various join scenarios
hj:                     Benchmark for simple hash joins, testing various join scenarios
smj:                    Benchmark for simple sort merge joins, testing various join scenarios
dict:                   Benchmark for dictionary-encoded group-by scenarios
array_agg_distinct:     1000K-group, two-row-per-group array_agg(DISTINCT) benchmark
compile_profile:        Compile and execute TPC-H across selected Cargo profiles, reporting timing and binary size


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Supported Configuration (Environment Variables)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
DATA_DIR            directory to store datasets
CARGO_COMMAND       command that runs the benchmark binary
DATAFUSION_DIR      directory to use (default /workspace/datafusion-base)
RESULTS_NAME        folder where the benchmark files are stored
PREFER_HASH_JOIN    Prefer hash join algorithm (default true)
SIMULATE_LATENCY    Simulate object store latency to mimic S3 (default false)
DATAFUSION_*        Set the given datafusion configuration


stderr:

Kubernetes message

Job has reached the specified backoff limit

File an issue against this benchmark runner

adriangbot · 2026-09-23T04:13:37Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing fix/parquet-row-selection-page-index (3f4da1b) to d20936c (merge-base) diff

Run configuration

run benchmark clickbench_partitioned

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

Comparing HEAD and fix_parquet-row-selection-page-index
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query     ┃       HEAD ┃ fix_parquet-row-selection-page-index ┃       Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 0  │    1.23 ms │                              1.26 ms │    no change │
│ QQuery 1  │   11.62 ms │                             11.57 ms │    no change │
│ QQuery 2  │   36.62 ms │                             36.85 ms │    no change │
│ QQuery 3  │   30.94 ms │                             31.13 ms │    no change │
│ QQuery 4  │  236.47 ms │                            236.96 ms │    no change │
│ QQuery 5  │  270.70 ms │                            274.08 ms │    no change │
│ QQuery 6  │    1.30 ms │                              1.30 ms │    no change │
│ QQuery 7  │   12.81 ms │                             13.09 ms │    no change │
│ QQuery 8  │  326.26 ms │                            338.57 ms │    no change │
│ QQuery 9  │  464.23 ms │                            470.99 ms │    no change │
│ QQuery 10 │   63.93 ms │                             64.46 ms │    no change │
│ QQuery 11 │   75.14 ms │                             76.22 ms │    no change │
│ QQuery 12 │  259.44 ms │                            257.04 ms │    no change │
│ QQuery 13 │  354.24 ms │                            356.14 ms │    no change │
│ QQuery 14 │  276.08 ms │                            276.57 ms │    no change │
│ QQuery 15 │  284.28 ms │                            285.54 ms │    no change │
│ QQuery 16 │  612.25 ms │                            620.51 ms │    no change │
│ QQuery 17 │  612.51 ms │                            625.92 ms │    no change │
│ QQuery 18 │ 1257.35 ms │                           1274.56 ms │    no change │
│ QQuery 19 │   27.57 ms │                             27.41 ms │    no change │
│ QQuery 20 │  514.17 ms │                            520.64 ms │    no change │
│ QQuery 21 │  507.34 ms │                            509.61 ms │    no change │
│ QQuery 22 │  974.21 ms │                            971.09 ms │    no change │
│ QQuery 23 │ 3029.66 ms │                           3045.77 ms │    no change │
│ QQuery 24 │   39.88 ms │                             39.85 ms │    no change │
│ QQuery 25 │  103.94 ms │                            104.15 ms │    no change │
│ QQuery 26 │   40.34 ms │                             40.17 ms │    no change │
│ QQuery 27 │  508.25 ms │                            513.30 ms │    no change │
│ QQuery 28 │ 2879.56 ms │                           2881.67 ms │    no change │
│ QQuery 29 │   41.55 ms │                             41.47 ms │    no change │
│ QQuery 30 │  300.79 ms │                            302.83 ms │    no change │
│ QQuery 31 │  278.48 ms │                            274.66 ms │    no change │
│ QQuery 32 │  920.57 ms │                            940.76 ms │    no change │
│ QQuery 33 │ 1432.06 ms │                           1449.50 ms │    no change │
│ QQuery 34 │ 1518.60 ms │                           1451.11 ms │    no change │
│ QQuery 35 │  295.72 ms │                            292.07 ms │    no change │
│ QQuery 36 │   66.44 ms │                             66.50 ms │    no change │
│ QQuery 37 │   34.83 ms │                             35.16 ms │    no change │
│ QQuery 38 │   39.87 ms │                             42.35 ms │ 1.06x slower │
│ QQuery 39 │  133.63 ms │                            148.39 ms │ 1.11x slower │
│ QQuery 40 │   13.81 ms │                             13.98 ms │    no change │
│ QQuery 41 │   13.57 ms │                             13.55 ms │    no change │
│ QQuery 42 │   13.19 ms │                             13.45 ms │    no change │
└───────────┴────────────┴──────────────────────────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                   ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                                   │ 18915.44ms │
│ Total Time (fix_parquet-row-selection-page-index)   │ 18992.21ms │
│ Average Time (HEAD)                                 │   439.89ms │
│ Average Time (fix_parquet-row-selection-page-index) │   441.68ms │
│ Queries Faster                                      │          0 │
│ Queries Slower                                      │          2 │
│ Queries with No Change                              │         41 │
│ Queries with Failure                                │          0 │
└─────────────────────────────────────────────────────┴────────────┘

Distribution per query (min / mean ±stddev / max):

Comparing HEAD and fix_parquet-row-selection-page-index
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃                                  HEAD ┃  fix_parquet-row-selection-page-index ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │          1.23 / 4.00 ±5.47 / 14.95 ms │          1.26 / 4.00 ±5.41 / 14.83 ms │     no change │
│ QQuery 1  │        11.62 / 11.73 ±0.08 / 11.83 ms │        11.57 / 11.89 ±0.21 / 12.20 ms │     no change │
│ QQuery 2  │        36.62 / 37.05 ±0.30 / 37.50 ms │        36.85 / 37.09 ±0.21 / 37.46 ms │     no change │
│ QQuery 3  │        30.94 / 31.82 ±0.77 / 33.07 ms │        31.13 / 31.55 ±0.52 / 32.55 ms │     no change │
│ QQuery 4  │     236.47 / 239.88 ±3.42 / 245.64 ms │     236.96 / 241.43 ±3.52 / 245.67 ms │     no change │
│ QQuery 5  │     270.70 / 272.86 ±2.24 / 276.71 ms │     274.08 / 277.96 ±3.19 / 283.56 ms │     no change │
│ QQuery 6  │           1.30 / 1.46 ±0.22 / 1.89 ms │           1.30 / 1.46 ±0.24 / 1.94 ms │     no change │
│ QQuery 7  │        12.81 / 13.01 ±0.16 / 13.28 ms │        13.09 / 13.26 ±0.14 / 13.50 ms │     no change │
│ QQuery 8  │     326.26 / 329.53 ±2.79 / 334.44 ms │     338.57 / 339.50 ±1.27 / 341.97 ms │     no change │
│ QQuery 9  │     464.23 / 471.44 ±4.17 / 475.17 ms │    470.99 / 484.43 ±12.28 / 506.17 ms │     no change │
│ QQuery 10 │        63.93 / 64.60 ±0.41 / 65.19 ms │        64.46 / 68.13 ±4.58 / 77.05 ms │  1.05x slower │
│ QQuery 11 │        75.14 / 75.76 ±0.39 / 76.37 ms │        76.22 / 77.24 ±1.06 / 78.78 ms │     no change │
│ QQuery 12 │     259.44 / 265.47 ±3.32 / 268.89 ms │     257.04 / 271.10 ±9.65 / 284.56 ms │     no change │
│ QQuery 13 │     354.24 / 369.11 ±9.56 / 380.99 ms │    356.14 / 370.53 ±15.20 / 399.20 ms │     no change │
│ QQuery 14 │     276.08 / 280.12 ±4.00 / 286.93 ms │     276.57 / 289.05 ±9.35 / 302.97 ms │     no change │
│ QQuery 15 │     284.28 / 292.92 ±7.88 / 306.47 ms │     285.54 / 290.63 ±5.73 / 300.86 ms │     no change │
│ QQuery 16 │    612.25 / 622.21 ±10.83 / 643.22 ms │     620.51 / 629.19 ±6.27 / 637.66 ms │     no change │
│ QQuery 17 │     612.51 / 621.94 ±8.03 / 632.42 ms │     625.92 / 636.35 ±5.42 / 641.35 ms │     no change │
│ QQuery 18 │ 1257.35 / 1277.66 ±13.44 / 1290.15 ms │ 1274.56 / 1305.21 ±27.49 / 1347.94 ms │     no change │
│ QQuery 19 │        27.57 / 32.15 ±7.00 / 45.80 ms │        27.41 / 27.79 ±0.28 / 28.17 ms │ +1.16x faster │
│ QQuery 20 │    514.17 / 527.77 ±11.35 / 543.09 ms │    520.64 / 536.57 ±17.98 / 567.95 ms │     no change │
│ QQuery 21 │     507.34 / 512.02 ±4.61 / 520.66 ms │     509.61 / 519.81 ±7.75 / 532.36 ms │     no change │
│ QQuery 22 │     974.21 / 982.58 ±5.60 / 987.84 ms │   971.09 / 984.99 ±14.03 / 1008.27 ms │     no change │
│ QQuery 23 │ 3029.66 / 3059.16 ±20.03 / 3081.45 ms │ 3045.77 / 3077.73 ±25.92 / 3106.60 ms │     no change │
│ QQuery 24 │        39.88 / 40.34 ±0.43 / 41.12 ms │        39.85 / 46.39 ±9.07 / 64.31 ms │  1.15x slower │
│ QQuery 25 │     103.94 / 110.07 ±4.98 / 117.39 ms │     104.15 / 106.24 ±3.10 / 112.38 ms │     no change │
│ QQuery 26 │        40.34 / 42.10 ±2.82 / 47.72 ms │        40.17 / 45.32 ±7.30 / 59.69 ms │  1.08x slower │
│ QQuery 27 │     508.25 / 516.79 ±6.28 / 525.62 ms │     513.30 / 517.14 ±4.67 / 525.03 ms │     no change │
│ QQuery 28 │ 2879.56 / 2907.41 ±16.49 / 2925.04 ms │ 2881.67 / 2942.21 ±37.60 / 2988.05 ms │     no change │
│ QQuery 29 │        41.55 / 45.61 ±8.00 / 61.61 ms │        41.47 / 41.95 ±0.48 / 42.81 ms │ +1.09x faster │
│ QQuery 30 │    300.79 / 316.49 ±15.31 / 344.86 ms │     302.83 / 310.73 ±8.08 / 323.39 ms │     no change │
│ QQuery 31 │     278.48 / 285.31 ±6.20 / 295.15 ms │    274.66 / 296.44 ±24.94 / 343.73 ms │     no change │
│ QQuery 32 │   920.57 / 962.33 ±33.97 / 1015.08 ms │   940.76 / 985.04 ±33.62 / 1034.74 ms │     no change │
│ QQuery 33 │ 1432.06 / 1472.50 ±39.09 / 1543.43 ms │ 1449.50 / 1491.01 ±32.04 / 1536.56 ms │     no change │
│ QQuery 34 │ 1518.60 / 1546.68 ±21.11 / 1572.34 ms │ 1451.11 / 1495.81 ±27.12 / 1524.19 ms │     no change │
│ QQuery 35 │    295.72 / 311.23 ±20.58 / 351.82 ms │    292.07 / 319.46 ±35.37 / 388.64 ms │     no change │
│ QQuery 36 │        66.44 / 67.86 ±1.33 / 70.18 ms │        66.50 / 68.51 ±1.58 / 70.89 ms │     no change │
│ QQuery 37 │        34.83 / 40.65 ±5.61 / 49.95 ms │        35.16 / 36.16 ±1.19 / 38.32 ms │ +1.12x faster │
│ QQuery 38 │        39.87 / 43.59 ±2.22 / 46.64 ms │        42.35 / 43.83 ±0.93 / 44.85 ms │     no change │
│ QQuery 39 │     133.63 / 147.28 ±7.21 / 153.03 ms │     148.39 / 157.05 ±4.87 / 161.33 ms │  1.07x slower │
│ QQuery 40 │        13.81 / 16.69 ±4.73 / 26.12 ms │        13.98 / 15.98 ±3.37 / 22.71 ms │     no change │
│ QQuery 41 │        13.57 / 13.90 ±0.22 / 14.17 ms │        13.55 / 15.78 ±3.80 / 23.35 ms │  1.14x slower │
│ QQuery 42 │        13.19 / 14.66 ±2.51 / 19.67 ms │        13.45 / 16.40 ±4.00 / 23.78 ms │  1.12x slower │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                   ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                                   │ 19297.71ms │
│ Total Time (fix_parquet-row-selection-page-index)   │ 19478.36ms │
│ Average Time (HEAD)                                 │   448.78ms │
│ Average Time (fix_parquet-row-selection-page-index) │   452.99ms │
│ Queries Faster                                      │          3 │
│ Queries Slower                                      │          6 │
│ Queries with No Change                              │         34 │
│ Queries with Failure                                │          0 │
└─────────────────────────────────────────────────────┴────────────┘

Resource Usage

clickbench_partitioned — base (merge-base)

Metric	Value
Wall time	100.0s
Peak memory	10.8 GiB
Avg memory	4.3 GiB
CPU user	989.3s
CPU sys	67.4s
Peak spill	0 B

clickbench_partitioned — branch

Metric	Value
Wall time	100.0s
Peak memory	12.4 GiB
Avg memory	4.3 GiB
CPU user	993.2s
CPU sys	69.0s
Peak spill	0 B

File an issue against this benchmark runner

alamb

Thanks @haohuaijin

alamb · 2026-09-23T11:17:12Z

 struct RowGroupsPrunedParquetOpen {
    prepared: FiltersPreparedParquetOpen,
    row_groups: RowGroupAccessPlanFilter,
+    /// Built lazily for external-selection index checks and reused by the stream.


alamb · 2026-09-23T11:18:37Z

-            &prepared.output_schema,
-            prepared.virtual_state.as_deref(),
-        )?;
+        // Reuse plans built for the external-selection index check. Other


this is great

alamb · 2026-09-23T11:19:22Z

+            decoder_read_plans: None,
        };
-        open.should_load_page_index()
+        open.should_load_page_index().unwrap()


how do we know this won't panic? Maybe this should also return Result?

this is a helper function in test mods, maybe it is ok to panic or should we change it to return a Result?

panic is fine for test

jayzhan211 · 2026-09-23T12:12:50Z

Thanks @haohuaijin and @alamb 🚀

haohuaijin added 2 commits September 17, 2026 23:11

fix: load Parquet page indexes for external row selections

0190dc5

update

602349a

github-actions Bot added the datasource Changes to the datasource crate label Sep 18, 2026

jayzhan211 reviewed Sep 19, 2026

View reviewed changes

Comment thread datafusion/datasource-parquet/src/opener/mod.rs Outdated

Comment thread datafusion/datasource-parquet/src/opener/mod.rs

Comment thread datafusion/datasource-parquet/src/opener/mod.rs Outdated

haohuaijin added 2 commits September 19, 2026 17:09

Merge branch 'main' into fix/parquet-row-selection-page-index

36afade

update

f5f1b7f

haohuaijin force-pushed the fix/parquet-row-selection-page-index branch from a3185dc to f5f1b7f Compare September 19, 2026 09:40

github-actions Bot removed execution Related to the execution crate proto Related to proto crate functions Changes to functions implementation ffi Changes to the ffi crate physical-plan Changes to the physical-plan crate spark labels Sep 19, 2026

jayzhan211 approved these changes Sep 19, 2026

View reviewed changes

Merge branch 'main' into fix/parquet-row-selection-page-index

d104ad0

alamb reviewed Sep 22, 2026

View reviewed changes

Comment thread datafusion/datasource-parquet/src/opener/mod.rs

perf: reuse Parquet decoder plans for page index checks

3f4da1b

haohuaijin requested a review from alamb September 23, 2026 08:31

alamb approved these changes Sep 23, 2026

View reviewed changes

jayzhan211 added this pull request to the merge queue Sep 23, 2026

github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Sep 23, 2026

alamb added this pull request to the merge queue Sep 23, 2026

Merged via the queue into apache:main with commit d939014 Sep 23, 2026
41 checks passed

haohuaijin deleted the fix/parquet-row-selection-page-index branch September 24, 2026 02:15

Conversation

haohuaijin commented Sep 18, 2026 • edited by alamb Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

What is the testing strategy for this PR?

Are there any user-facing changes?

Uh oh!

codecov-commenter commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jayzhan211 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

haohuaijin commented Sep 19, 2026

Uh oh!

jayzhan211 left a comment

Choose a reason for hiding this comment

Uh oh!

alamb commented Sep 22, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

adriangbot commented Sep 22, 2026

Uh oh!

adriangbot commented Sep 22, 2026

Uh oh!

adriangbot commented Sep 22, 2026

Uh oh!

adriangbot commented Sep 22, 2026

Uh oh!

haohuaijin commented Sep 23, 2026

Uh oh!

adriangbot commented Sep 23, 2026

Uh oh!

adriangbot commented Sep 23, 2026

Uh oh!

adriangbot commented Sep 23, 2026

Uh oh!

adriangbot commented Sep 23, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

haohuaijin Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jayzhan211 Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

jayzhan211 commented Sep 23, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

haohuaijin commented Sep 18, 2026 •

edited by alamb

Loading

codecov-commenter commented Sep 18, 2026 •

edited

Loading

haohuaijin Sep 23, 2026 •

edited

Loading