feat(index): support MinHash LSH scalar index

[zhangyue19921010]

Please refer to #8820 for the full discussion

A minhashlsh scalar index returns the rows whose token shingles have the highest estimated Jaccard similarity to a query text.

Text is tokenized with the full text search tokenizer, shingled, hashed and permuted into a 16-bit MinHash signature that LSH banding groups into candidates. A segment stores signatures.lance (one signature per document) and bands.lance (fixed-width (band_key, doc_id) rows with a page table); every signature parameter lives in
MinHashLshIndexDetails, whose tokenizer reuses InvertedIndexDetails. Queries read one page per band, refine the candidates by signature under the prefilter, merge segments by distance, and score rows the index does not cover on the fly unless the scan uses fast_search. Builds spill sorted runs and merge them by partition, so memory does not grow with the table; merges, updates and remaps rebuild from stored signatures.

Exposed as Scanner::minhash_search in Rust, MinHashQuery as nearest in Python and IndexType::MinHashLsh in Java.

[github-actions]

Important

Format specification vote

This PR modifies the Lance format specification, so it requires 3 binding +1 votes from PMC members (excluding the proposer) and a minimum 72-hour voting period, weekends excluded, before it can merge. Vote by approving this PR (+1) or requesting changes (−1, a veto). See the voting process.

Status: ❌ Blocked — 0 of 3 required approvals

Approvals (this commit)	none (0/3)
Vetoes	none
Voting period	ends Tue 2026-09-15 02:32 UTC (Mon 19:32 PDT)

_{Updated automatically by the format-spec vote gate, which re-checks every 15 minutes — just voted? Re-check now (press Run workflow; leave the input blank to re-check every open format PR). A PMC member may apply the format-waived label to waive the vote for a trivial edit (typo, wording, formatting).}

[lance-gatekeeper]

❌ Gate recommendation: request changes.

MinHash/LSH is a suitable native near-duplicate-search direction, but its persisted contract must land first through the independently scoped format PR #9076. Merge that prerequisite, then rebase this implementation so the duplicate format changes disappear; the implementation can then be assessed against the accepted contract.

[lance-gatekeeper]

The MinHashLshIndexDetails proto and associated build-script change are byte-for-byte the blobs already isolated in #9076, but this PR does not include that prerequisite’s normative format document. The persisted-format process requires the contract and docs to land in their dedicated PR before implementation. Merging this branch first would establish the durable contract without those docs and collapse the required review sequence. Please merge #9076, then rebase this PR so these format changes come from the base.