Add Hugging Face dataset support

[jamesgao-jpg]

Summary

• add a reusable Hugging Face dataset reader with revision-pinned exact-file and wildcard resolution
• separate the vector dataset contract from storage format through abstract DatasetManager, ParquetDatasetManager, and Hdf5DatasetManager implementations
• register the 24 advertised VIBE datasets, including ID, OOD, cosine, L2, normalized, and inner-product workloads
• register the VDBBench multimodal embedding datasets at 1M, 10M, and 100M scale
• expose every registered vector dataset through the source-independent Performance case and --dataset-name
• preserve dataset-specific load and optimize timeout budgets when using the generic Performance case
• read HDF5 artifacts directly: queries and ground truth stay in memory while corpus vectors stream through one single-open iterator per load worker
• resolve Parquet corpus, query, and ground-truth roles independently; corpus shards stream in deterministic order and query shards are concatenated before GT validation
• preserve optional dataset source, revision, schema, and resolved-file provenance in exported results
• add a complete Hugging Face dataset release note while keeping README discovery concise

Public interface

vectordbbench milvushnsw \
  --case-type Performance \
  --dataset-name glove-200-cosine \
  --k 100

vectordbbench milvushnsw \
  --case-type Performance \
  --dataset-name multimodal-embedding-10m \
  --k 100

Dataset source and file format are internal registration details. There is no VIBE-, Hugging-Face-, HDF5-, or Parquet-specific case type or public dataset flag.

Architecture change

Before: Original VDBBench

CaseType
   |
   v
+-----------------------+
| Case                  |
| chooses its dataset   |
+-----------+-----------+
            |
            v
+-----------------------+       Configured source
| DatasetManager        | <----- S3 or Aliyun OSS
|                       |
| - downloads files     |
| - reads Parquet       |
| - streams Parquet     |
| - loads query / GT    |
+-----------+-----------+
            |
            v
+-----------------------+
| Common runner input   |
| - corpus batches      |
| - query vectors       |
| - ground truth        |
+-----------+-----------+
            |
            v
    Performance runner


DatasetManager was both:
- the common runner-facing interface
- the concrete Parquet implementation

After: Source and format are independent

Performance + dataset_name
            |
            v
+--------------------------+
| Dataset Registry         |
|                          |
| - source metadata        |
| - storage format         |
| - artifact roles         |
| - dataset metadata       |
| - lifecycle timeouts      |
+------------+-------------+
             |
       +-----+------+
       |            |
       v            v
Source selection   Format selection
       |            |
       |            v
       |    +----------------------+
       |    | DatasetManager (ABC) |
       |    +----------+-----------+
       |               |
       |       +-------+-------+
       |       |               |
       |   ParquetDataset  Hdf5Dataset
       |      Manager         Manager
       |       |               |
       v       |               |
actual_source = dataset.preferred_source or configured_source
       |
       v
+--------------------------+
| DatasetReader            |
+------------+-------------+
             |
      +------+------+
      |             |
 HuggingFace     S3 / OSS
    Reader        Readers
      |             |
      +------+------+
             |
      resolved file paths
             |
             v
   Selected format manager
             |
             v
+--------------------------+
| Common runner input      |
| - corpus batches         |
| - query vectors          |
| - ground truth           |
+------------+-------------+
             |
             v
     Performance runner


The runner contract stays unchanged.
Source acquisition and storage parsing are separate abstractions.

Hugging Face Parquet roles

Each registered Parquet dataset declares its corpus, query, and ground-truth selectors plus column mappings and expected query/GT sizes. HuggingFaceReader acquires exact files with hf_hub_download and repositories containing wildcard selectors with snapshot_download(..., allow_patterns=...), following the Hugging Face download API.

ParquetDatasetManager.prepare() then:

1. resolves selectors to concrete cached paths before insertion timing starts;
2. sorts corpus and query files deterministically;
3. keeps corpus paths shard-streamed;
4. concatenates query shards in filename order;
5. validates query count, query/GT ID alignment, and GT width.

Existing S3/OSS datasets continue to use their exact artifact names.

Registered VDBBench multimodal datasets

Dataset name	Repository	Corpus selector	Query selector	Ground truth
multimodal-embedding-1m	VDBBench/multimodal-embedding-1M	train.parquet	test.parquet	neighbors.parquet
multimodal-embedding-10m	VDBBench/multimodal-embedding-10M	data/train-*.parquet	data/test-*.parquet	data/neighbors.parquet
multimodal-embedding-100m	VDBBench/multimodal-embedding-100M	train/shard-*/*.parquet	test/*.parquet	neighbors/neighbors.parquet

All three registrations use 4096-dimensional normalized vectors, inner-product search, 10,000 queries, and GT width 100. Repository revisions are pinned.

Compatibility

• existing built-in and custom Parquet datasets use ParquetDatasetManager
• Hugging Face can supply either Parquet or HDF5 files
• VIBE uses the generic HDF5 manager and the same in-memory runner contract as Parquet datasets
• existing fixed performance case types remain available
• registered versions of legacy datasets retain the fixed cases’ load and optimize timeout budgets
• exact-file Hugging Face downloads retain the previous hf_hub_download path
• existing S3/OSS and LAION artifact selection behavior remains unchanged
• result metadata remains optional, and old result files retain their previous serialized shape

Dependencies

• h5py for native HDF5 reads
• huggingface_hub for revision-pinned dataset downloads and caching

No additional dependency was introduced for the Parquet registrations.

Verification

Development checks

• focused Level 1 dataset, registry, CLI/UI, result-compatibility, HDF5, and Parquet tests: 40 passed
• timeout regressions: 14 passed; the complete affected HDF5/Parquet reader files: 28 passed
• deterministic Parquet fixtures cover wildcard expansion, shard ordering, multi-file queries, GT validation, and streamed insertion batches
• direct HDF5 fixtures cover Euclidean, normalized/cosine, and inner-product datasets
• verified one HDF5 open across multiple insertion batches and pickle-safe manager transfer
• verified old-result round trips do not gain new null provenance keys
• Ruff checks and git diff --check passed for the modified implementation
• release-note catalogs match all 24 HDF5 and 3 Parquet registrations; documentation paths and external links were checked
• pinned Hugging Face manifest audit resolved:
  • 1M: 1 corpus file, 1 query file, 1 GT file
  • 10M: 20 corpus files, 1 query file, 1 GT file
  • 100M: 16,824 corpus files, 2 query files, 1 GT file

Milvus functional smoke

Tested at 2ca947f9 with the Performance case against a standalone Milvus deployment. Each case loaded the full corpus with four workers and batch size 1,000, built HNSW with M=16 and efConstruction=200, and ran serial Recall@100 with efSearch=200. Concurrent search and filtering were intentionally disabled.

Dataset	Format	Inserted	Total load	Serial P99	Recall@100
yi-128-ip	HDF5	187,843	65.11s	2.6ms	0.2399
yahoo-minilm-384-normalized	HDF5	677,305	266.78s	3.1ms	0.9692
glove-200-cosine	HDF5	1,192,514	359.04s	3.0ms	0.6851
multimodal-embedding-1m	Parquet	1,000,000	1,771.53s	3.5ms	0.9890

• all four full-corpus runs completed normally and produced result JSONs
• all four embedded milvus/v1 run contexts passed validation
• VIBE HDF5 and VDBBench Parquet both completed Hugging Face acquisition, streamed corpus insertion, query/GT loading, and ground-truth evaluation
• multimodal-embedding-1m streamed 1M × 4,096 training vectors without client OOM and evaluated all 10,000 queries
• the fixed HNSW parameters were selected for integration verification, not recall tuning or cross-dataset performance comparison
• the functional smoke was performed at 2ca947f9; later changes are scoped to registered-result naming, CLI linting, documentation, and dataset timeout registration, and the smoke run has not been repeated at the current head

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: jamesgao-jpg
To complete the pull request process, please assign xuanyang-cn after the PR has been reviewed.
You can assign the PR to them by writing /assign @xuanyang-cn in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jamesgao-jpg]

Reviewed the effective diff at 2ca947f. The focused dataset, HDF5, Parquet, registry, and compatibility tests passed (40 passed, 6 deselected). Two findings below: a reproduced cross-dataset result collision and the current CI lint failure. No live backend benchmark was run.

[XuanYang-cn]

One finding on timeout defaults in the generic Performance case.