feat(index): accept a vector index a table is too small to train

[xuanyu-z]

Blocked on #9151. That PR proposes target_num_partitions on
VectorIndexDetails; this one stores the same value in runtime_hints, which
is specified for settings that do not affect index structure. Merge #9151
first, then this PR moves onto the typed field. See the last item under
For review.

Also needs the stale A-format and format-change labels removed: this PR
modifies no protos/** or docs/src/format/** path, so format-spec-vote is
red on a label left over from a proto edit that was reverted.

The problem

Creating a vector index on a table with too few rows fails with an error, and so
does maintaining one on a table that later shrinks. create_index(train=False)
reaches a todo!() for vector indices, and build_empty_vector_index returns
not_supported_source, so deferred training never worked for them even though it
works for scalar and FTS.

The shrink half is the worse one. An IVF-PQ index whose table falls below the
PQ minimum makes optimize_indices() fail outright, so index maintenance is
stuck until someone drops and recreates the index.

What this does

Implements #4034 for vector indices, and keeps the empty-index contract #3940
set for scalar and FTS:

• Too few vectors to train: the index is created with its settings and no
  data file, covering no rows. The next optimize_indices() with enough data
  trains it for real. This applies on creation and when a table shrinks, so
  maintenance is never blocked.
• Only PQ has a real minimum (256 vectors at 8 bits, 2^num_bits in
  general). RQ trains from two values and flat storage needs none, so those
  build normally on a small table.
• Vectors, not rows. A column of nulls has rows but nothing to train on. A
  multivector row holds a list, so it counts for its whole list — 100 rows of 10
  vectors give 1,000. The count stops as soon as it reaches the number needed.
• Too many partitions requested: train the number the data supports,
  min(requested, vectors / sample_rate), and log it. A partition's centroid is
  what a search compares against to choose partitions, so one built from a
  handful of vectors prunes nothing and still costs compression accuracy. The
  per-partition number is the IVF sample rate — the vectors IVF training fits one
  centroid on, 256 by default, and where KMeans starts warning. It is not the
  codebook size, which happens to be 256 as well at 8 bits.

A query skips an index that covers nothing and reads the table instead;
fast_search returns nothing, since it is limited to what the index covers.
Statistics report zero rows indexed and everything still to do.

Main pieces

• should_train_index (create.rs) — the single decision: train now, or record
  the settings and train later. A build handed an explicit list of fragments is
  exempt, because the pieces of one index must agree on their partition count.
• count_trainable_vectors (vector.rs) — one definition of "how many vectors
  are there", used by that decision and by the partition cap.
• supported_num_partitions (vector.rs) — the cap above.
• optimize_indices (append.rs) — trains an index that is still only settings,
  reading them from the index metadata. It decides that before opening the index
  file, since there is no file yet.
• The rebuild check accepts an index covering nothing only when the caller
  established beforehand that the column cannot train and the rebuild wrote no
  file, so rows lost for any other reason still fail loudly.

Tests

Lifecycle, so the index survives what happens to the data:

• delete every row, update every row — index still there, still there after
  reopening
• 1024 rows, delete 800, optimize — the case in
  Top-level indices part 2: empty vector indices #4034 (comment)
• create too small, add data, optimize — trains and covers every fragment
• retraining one that is still only settings does nothing rather than failing

Thresholds:

• empty table; 100 rows against a 256-code table
• 1000 rows holding 100 vectors — degrades, because blanks do not count
• 100 rows of 10 vectors — trains, because the lists do count
• 10 rows of 2 vectors — degrades
• IVF_FLAT and IVF_RQ on 100 rows — train, having no codebook to build

Partition counts:

• 300 vectors, 1000 requested → 1
• 1000 vectors, 8 requested → 3; 3000 vectors, 8 requested → 8
• 3000 rows holding 300 vectors, 8 requested → 1
• 100 rows of 10 vectors, 8 requested → 3
• 300 vectors with a 4-bit codebook, 8 requested → 1, so the cap does not
  follow the codebook size
• 300 vectors at a sample rate of 64, 8 requested → 4, so it does follow the
  rate
• created as IVF-8 on 100 rows, grown to 3100, optimized → 8, the count it
  asked for

Queries and statistics: a normal search, fast_search, and index_statistics
against an index that covers nothing.

Maintenance over an index that covers nothing:

• compaction leaves it alone — it has no file to open, and no rewrite group
  intersects an empty coverage bitmap
• a repeated append-mode optimize stays a no-op while the table is still too
  small, so a caller optimizing on a schedule keeps getting the commit it
  expects

The partition count and the runtime hints round-trip through
VectorIndexDetails for all eight IVF combinations, and an auto-sized build is
checked to record no count.

Each new test was checked by breaking the code under it and confirming it fails.
cargo nextest run -p lance --lib — 3450 pass, 3 skipped, none failing.
cargo fmt --all --check, cargo clippy -p lance --lib --tests -D warnings and
ci/check_proto_comments.py are clean.

For review

• num_partitions becomes a maximum, not an exact request. On a small table
  you get fewer partitions than you asked for, logged rather than returned as an
  error, and this applies to create_index as well as optimize_indices().
  Top-level indices part 2: empty vector indices #4034 asks for it; worth confirming you want it on creation too. The code is
  one match arm in supported_num_partitions, so narrowing it to "only reduce
  when the build would otherwise fail" is a small change.

  Fifteen existing tests asserted the count they requested on fixtures too small
  to support it. Rather than lower those assertions, each test about index
  metadata was given the vectors its request needs, so it still asserts the exact
  count. Tests that measure search quality kept their original data and assert
  the count it supports — changing data under a recall threshold is how those
  become flaky.

• The two 256s are kept apart. The training floor is the codebook size,
  2^num_bits, matching the error PQ raises. The per-partition number for the
  cap is the IVF sample rate, which is configurable and independent of the
  quantizer: a build declaring 64 samples per centroid supports four partitions
  on 300 vectors where the default supports one.

  The cap applies only where a codebook makes thin partitions a bad trade. An
  index that stores vectors uncompressed has no precision to lose, so two
  partitions over 400 vectors still give some pruning for free, and KMeans
  needing a vector per centroid stays the only limit there.

• The requested num_partitions survives a deferral. Without it an index
  created as IVF-8 on a small table stays IVF-1 however far the table grows,
  which contradicts the part of Top-level indices part 2: empty vector indices #4034 that returns the requested count once the
  data supports it. An auto-sized build records nothing and keeps auto-sizing;
  the count recorded is the one requested, with the cap re-derived from the data
  present at rebuild time.

  This currently rides in runtime_hints under lance.ivf.num_partitions,
  which is the wrong home: that field is specified as build preferences that do
  not affect index structure, and a partition count is exactly index structure.
  feat(format): record the requested IVF partition count on a vector index #9151 proposes a typed target_num_partitions on VectorIndexDetails
  instead, and this PR moves onto it once that vote passes.

Finishes the direction of #7211 (@burlacio), whose creation, statistics and
detection changes this supersedes; happy to rebase onto it instead if preferred.

Fixes: #4034

[github-actions]

Important

Format specification vote

This PR modifies the Lance format specification, so it requires 3 binding +1 votes from PMC members (excluding the proposer) and a minimum 72-hour voting period, weekends excluded, before it can merge. Vote by approving this PR (+1) or requesting changes (−1, a veto). See the voting process.

Status: ❌ Blocked — 0 of 3 required approvals

Approvals (this commit)	none (0/3)
Vetoes	none
Voting period	ends Wed 2026-09-16 02:30 UTC (Tue 19:30 PDT)

_{Updated automatically by the format-spec vote gate, which re-checks every 15 minutes — just voted? Re-check now (press Run workflow; leave the input blank to re-check every open format PR). A PMC member may apply the format-waived label to waive the vote for a trivial edit (typo, wording, formatting).}

[lance-gatekeeper]

✅ Gate recommendation: approve.

The new revision adds focused coverage for repeated append-mode optimization and compaction while an index remains definition-only. Production behavior is unchanged, and both lifecycle scenarios preserve the previously verified empty-coverage contract.

[xuanyu-z]

this one has no format change, wrong label