perf(io): batch scheduled object store range reads

[Xuanwo]

Lance currently passes one range at a time to ObjectStore::get_ranges, discarding OpenDAL's per-reader HF XET resolution cache after each scheduled read. Submit bounded batches through both schedulers so cloud reads can share a backend reader. Local and custom readers retain independent scheduling by default.

Reserve concurrency per range and memory for the batch's total bytes, cap batches at the bridge's eight concurrent reads, and disable additional OpenDAL gap coalescing after Lance has budgeted the reads. Refund only admitted tasks when closing the standard queue, and preserve result order and length validation across batch failures and cancellation.

Integration coverage exercises ScanScheduler::open_file through the real OpenDAL adapter and checks shared readers, exact fetched ranges, and returned bytes. HF resolution reuse requires the existing hf_download_mode=xet setting and is limited to a batch; the default remains HTTP.

Live HF benchmark

Baseline 71c49d90eb220b0b71d7ca77d83cd8fa5c1cfb91 (includes #9123) versus candidate 25916efac287968ea49d84df405fbabadb68ba4c, on the same EC2 c7i.8xlarge in us-east-2. Three alternating pairs per workload/mode, identical dependencies and release-no-lto builds, fresh processes and dataset sessions, XET disk chunk cache disabled. Timings cover native Lance operations, excluding dataset opening, Python dispatch, and output packing.

Workload	Mode	Baseline median	Candidate median
First 32 video ranges, 200 KiB each	HTTP	1.020 s	0.959 s
First 32 video ranges, 200 KiB each	XET	2.504 s	2.382 s
64 scattered rows, 33 non-video/image columns	HTTP	3.908 s	2.988 s
64 scattered rows, 33 non-video/image columns	XET	49.462 s	45.582 s

Data: lance-format/lerobot-bench/droid_1.0.1-lance-compacted/videos.lance version 399 and droid_1.0.1-lance/frames.lance version 86. The video ranges match the reported example. The scalar workload uses a fixed representative projection; the original report did not specify its exact projection or seed. All paired and cross-mode outputs match, and dataset identities remained unchanged.

XET scalar latency improved by 5.1–8.9% in each pair (7.8% lower median). HTTP paired results have mixed direction, so these medians do not establish a stable HTTP speedup. The video example retains separate blob objects and shows no request-reduction benefit. This is a limited optimization, not a complete fix for HF throttling; sustained multiworker training was not tested.

[lance-gatekeeper]

✅ Gate recommendation: approve.

This batches scheduled cloud ranges through one OpenDAL reader while preserving per-range concurrency accounting, bounded bytes, result ordering, and length validation. The bounded reader-scoped approach is preferable to the operator-wide cache closed in apache/opendal#8242, which could return stale or mixed file versions across its TTL.

[Xuanwo]

didn't work