Support Qdrant FineWeb-10B, PubMed-MV, and Coyo-VE datasets #869

Summary

Add the three vector-search datasets announced by Qdrant to VectorDBBench through the Hugging Face dataset-source and DatasetManager infrastructure introduced in #868:

• Qdrant/Coyo-VE
• Qdrant/PubMed-MV
• Qdrant/FineWeb-10B

They should use the existing public interface: a normal Performance case with the selected dataset name. A new case type should not be introduced.

Announcement: https://www.reddit.com/r/vectordatabase/comments/1w5j3zz/new_massive_and_diverse_vector_datasets_opened_to/

Dataset characteristics

Difficulties to resolve

1. Source IDs versus database IDs

All three datasets expose string document identifiers, while regular VectorDBBench vector clients commonly use integer primary keys. The integration needs a generic, deterministic ID translation mechanism shared by corpus insertion and ground-truth loading.

Requirements:

• Do not keep a corpus-wide mapping in memory for billion-scale datasets.
• Generate any mapping or sidecar before insertion timing starts.
• Keep mappings deterministic across runs and revisions.
• Coyo hit_ids join directly to corpus id.
• FineWeb corpus IDs use <urn:uuid:...>, while ground truth contains bare UUIDs, so canonicalization is required.
• Verify whether PubMed hit_ids can safely map through the corpus PMID column, including uniqueness and actual value format.

2. Parquet column projection and remote access

The current Hugging Face reader downloads complete matching files, and the Parquet iterator reads every column. This is especially problematic for PubMed because dense, sparse, and very large nested multivector columns coexist in the same shards.

The manager should read only the configured ID, vector, query, and ground-truth columns. For PubMed and FineWeb, investigate revision-pinned hf:///range-based Parquet reads or another resumable projected-access mechanism so a dense test does not download unrelated columns or an entire multi-terabyte repository.

Dataset preparation and downloads must remain outside measured insertion time.

3. Combined query and ground-truth artifacts

Coyo and PubMed store query vectors and ground truth in the same Parquet files. The role-based schema should allow one physical file to satisfy both query and ground-truth selectors without duplicating the download. Add fixtures covering this layout.

4. Vector modality and scoring support

The current vector runner assumes one dense vector per record. Full coverage requires explicit capability-gated workloads:

• Dense ANN search: suitable for the first implementation.
• Sparse vector search: needed for PubMed and FineWeb.
• Token-level multivector search with MaxSim: needed for PubMed.
• Hybrid/fusion and multistage reranking: needed to exercise PubMed's intended comparisons.

Unsupported database/workload combinations should fail during validation with a clear explanation rather than during insertion or search.

5. Filtered retrieval

FineWeb publishes both text/domain filters and structured filters over language score, dates, and crawl dumps. Registered Parquet datasets currently support only unfiltered performance cases. Supporting these queries requires a portable filter/payload schema, database-specific translation, filtered ground-truth selection, and capability validation.

6. FineWeb query vectors and licensing

FineWeb does not redistribute the MS MARCO query text or derived embeddings. Its scripts recover the queries and regenerate embeddings, but the dataset card notes that float32 regeneration can differ from the bfloat16 inputs used for the published ground truth and can affect results near the top-1000 boundary.

Before enabling FineWeb recall reporting, decide whether to:

• wait for an upstream reproducible query-vector artifact;
• regenerate vectors during dataset preparation and explicitly record the numerical limitation; or
• support corpus-only ingestion/index measurements until exact query compatibility is available.

The result metadata and documentation must also distinguish the ODC-BY corpus/ground-truth terms from the separate non-commercial MS MARCO query terms.

7. Scale-aware verification

Coyo is approximately 135 GB and is suitable for the first real end-to-end validation. Full PubMed and FineWeb downloads are not appropriate for regular CI or a single development smoke test.

Use layered verification:

1. Unit fixtures for selectors, schemas, combined query/GT files, ID translation, and provenance.
2. Small revision-pinned slices for download, insertion, search, and recall smoke tests.
3. A full Coyo dense run on an appropriately sized environment.
4. Manifest/schema/projected-read validation for PubMed and FineWeb before scheduling full-scale runs.

Proposed implementation sequence

Phase 1: generic dense Parquet support

• Add projected corpus/query/ground-truth column reads.
• Add deterministic, disk-backed source-ID translation and source-ID canonicalization hooks.
• Test a combined query-and-ground-truth Parquet file.
• Register Coyo-VE dense cosine under the existing Performance case.
• Record repository, immutable revision, vector modality, metric, model, dimensions, ID mapping, and license provenance in results.

Phase 2: PubMed dense

• Prove that dense-only reading does not fetch or materialize the nested multivector column.
• Validate PubMed corpus/ground-truth ID mapping.
• Register and smoke-test PubMed dense cosine search.

Phase 3: additional modalities and FineWeb

• Design capability-gated sparse-vector workloads.
• Design multivector/MaxSim and hybrid/multistage workloads.
• Add a portable filtered-retrieval schema for FineWeb.
• Resolve FineWeb query-vector reproducibility and licensing behavior.
• Define resumable, distributed-friendly full-scale execution for PubMed and FineWeb.

Acceptance criteria

• Each enabled dataset is selectable through the existing Performance case and dataset-name interface.
• Dataset revisions are immutable and recorded in exported results.
• Download/preparation time is excluded from insertion timing.
• Corpus vectors remain batch-streamed; queries and selected ground truth may be held in memory when bounded.
• Recall uses the published exact ground truth and rejects incompatible query vectors or ID mappings.
• Unsupported modalities, filters, or database clients fail during preflight validation with actionable messages.
• Tests cover schema failures, missing files, selector expansion, query/GT alignment, ID translation, and maximum supported k.