Skip to content

Add --drop_vector_index to update collection - #171

Open
jfrancoa wants to merge 17 commits into
mainfrom
jose/alter-schema-drop-index
Open

jfrancoa wants to merge 17 commits into
mainfrom
jose/alter-schema-drop-index

Conversation

@jfrancoa

@jfrancoa jfrancoa commented Jul 23, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Adds weaviate-cli update collection --drop_vector_index <name> to drop the index of a
named vector — the Weaviate 1.39 experimental endpoint
(DELETE /v1/schema/{Class}/vectors/{name}/index), exposed by the Python client in
weaviate/weaviate-python-client#1991 as collection.config.delete_vector_index(). The drop
is destructive and applied asynchronously (a 200 means accepted, not done). The vector can
be re-created afterwards (a fresh, empty index via config.add_vector()) once the drop
finalizes — verified live.

Closes #170

What changed

  • update.py / defaults.py — --drop_vector_index <name>, mutually exclusive with
    --vector_index; warns when the server is older than v1.39.0.
  • collection_manager.py
    • The drop runs after config.update(). config.update() reads the whole schema and
      PUTs it back, so dropping first would send a vectorIndexType: "none" vector back to the
      server. Ordering is asserted by a unit test.
    • get collection listing fix: a dropped vector's vector_index_config is None; the
      old listing called .vector_index_type() on it and crashed. It now renders none and
      shows every distinct index type across named vectors (e.g. hnsw, none).
    • Re-drop is allowed: re-issuing the drop on an already-none vector returns 200 on the
      server — a no-op while cleanup runs, and a fresh cleanup task if the previous one FAILED
      (the only operator recovery path for a stalled drop). The CLI reports the re-trigger
      instead of blocking it.
  • create data auto-skips dropped vectors — after a drop, the server rejects any object
    carrying the dropped vector, so --randomize ingestion failed every batch. Generated
    objects now exclude any named vector whose index is dropped (vectorIndexType: "none"),
    with an informational note (suppressed under --json). Ingestion keeps working; re-ingest
    after re-creating the index.
  • Unit tests — drop happy path, --vector_index conflict, unknown/absent vector, listing
    (none / hnsw, none), re-trigger, and the create data skip (one / all / none / --json).
    336 pass.
  • Operating skill docs updated (drop semantics incl. re-creation; create data auto-skip).

⚠️ Temporary dependency pin (needs follow-up before release)

requirements-dev.txt and setup.cfg pin weaviate-client to the commit on the
python-client PR #1991 branch
(which adds delete_vector_index() and VectorIndexType.NONE). That PR is not yet merged.
Before a weaviate-cli release this must be reverted to a released weaviate-client>=X. A git
URL cannot be published to PyPI, so this should not merge to a release tag as-is.

Test plan

  • make lint
  • make test (336 passed)
  • Manual, live 3-node 1.39 cluster: drop accepted; sibling search / reads / clean writes
    stay 100% available during cleanup; dropped-vector search rejected immediately; hnsw, none
    shown mid-drop; re-drop of a none vector accepted; --vector_index conflict rejected;
    vector re-created after finalize via config.add_vector() and searchable again; create data skips a dropped vector and ingests successfully (note shown; --json clean).

🤖 Generated with Claude Code

Adds `weaviate-cli update collection --drop_vector_index <name>` to drop the
index of a named vector (Weaviate 1.39 experimental endpoint, exposed by
weaviate-python-client#1991 as collection.config.delete_vector_index()).

- defaults.py: UpdateCollectionDefaults.drop_vector_index
- update.py: --drop_vector_index option, mutually exclusive with --vector_index
- collection_manager.py:
  - drop runs AFTER config.update() (which reads-and-writes the whole schema),
    so a "none"-marked vector is never PUT back to the server
  - get collection listing renders a dropped vector as `none` and no longer
    crashes on a null vector_index_config; shows all distinct index types
    (e.g. `hnsw, none`)
  - re-dropping an already-"none" vector is allowed (server returns 200): it is
    a no-op while cleanup runs and re-enqueues a fresh task if the previous one
    FAILED -- the only operator recovery path for a stalled drop
- version warning when the server is older than v1.39.0
- unit tests: happy path, validation, listing, and re-trigger
- requirements-dev.txt / setup.cfg: temporary pin to the client PR #1991 commit
  (revert to a released weaviate-client before release)
- operating skill docs

Closes #170

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@orca-security-eu orca-security-eu Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Orca Security Scan Summary

Status Check Issues by priority
Passed Passed Secrets high 0   medium 0   low 0   info 0 View in Orca

@jfrancoa
jfrancoa changed the base branch from master to main July 23, 2026 13:14
A dropped vector index CAN be re-created (confirmed with the developer and
verified live via config.add_vector() once the drop finalizes). Correct the
docs/help/messages that wrongly said "cannot be re-created" / "irreversible":

- update.py help, collection_manager success + conflict messages
- operating skill (SKILL.md, collections.md): re-creation is allowed after the
  drop finalizes (fresh, empty index); "irreversible" removed

create data: automatically skip named vectors whose index was dropped
(vectorIndexType "none", reported as vector_index_config == None). The server
rejects any object carrying a dropped vector, so every batch was failing after
a drop. Now those vectors are excluded from generated objects (with an
informational note, suppressed under --json) and ingestion keeps working.

- data_manager.__ingest_data: filter dropped vectors, derive vectorizer from a
  live one, note skipped vectors
- data.md: document the auto-skip
- unit tests: skip one, skip all, skip none, --json suppression

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@orca-security-eu orca-security-eu Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Orca Security Scan Summary

Status Check Issues by priority
Passed Passed Infrastructure as Code high 0   medium 0   low 0   info 0 View in Orca
Passed Passed SAST high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Secrets high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Vulnerabilities high 0   medium 0   low 0   info 0 View in Orca

jfrancoa and others added 8 commits August 6, 2026 10:42
Advances the temporary weaviate-python-client pin to the current head of
PR #1991. Since our previous pin (2dbfaedf) the PR added two parser fixes we
need: b92dbb71 replaced an assert in the dropped-vector schema parser with an
explicit SchemaValidationError (-O safe), and 9d7b9163 fixes a KeyError
'vectorizer' crash when parsing a legacy (single-vector) collection whose
vector index was dropped -- which otherwise breaks `get collection` and
`delete collection --all`. No weaviate-cli code change is required; the fix is
entirely in the client parser. Lint clean, 336 unit tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`create collection --named_vector_name` now accepts a comma-separated list,
creating one named vector per name (all sharing the chosen --vectorizer and
--vector_index). A single name keeps the previous behavior, so existing
callers are unaffected. Duplicate or empty names are rejected. This makes it
possible to create collections with more than one named vector, e.g. to test
dropping one named vector's index while keeping others.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`update collection --add_vector NAME` adds a named vector via the client's
config.add_vector(), so a vector whose index was dropped can be re-created
from the CLI (and then re-ingested). --add_vector_vectorizer picks the
vectorizer (none/contextionary/transformers/model2vec, default none =
self-provided) and --add_vector_index_type picks the index, including
quantized variants (hnsw_pq, hnsw_sq, hnsw_bq, hnsw_rq, hfresh, flat_bq,
hnsw_acorn); pq/sq honor --training_limit. Mutually exclusive with
--drop_vector_index and --vector_index. Unit tests cover the added config,
quantizers, and the validation guards.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`query data --search_type keyword` passed `return_objects=True` to the
python-client's `query.bm25()`, which does not accept that argument (bm25
returns objects by default). Against a current client this raised
`_BM25QueryExecutor.bm25() got an unexpected keyword argument 'return_objects'`,
breaking every keyword query. Remove the kwarg. The other search types
(fetch/vector/hybrid) never passed it.

Add an integration test that runs all four `query data` search types against a
real client, so a client query-signature drift like this is caught in CI (the
unit tests mock the client and cannot catch an unexpected-kwarg mismatch).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Pin both setup.cfg and requirements-dev.txt to weaviate-python-client commit
4abce144, the current head of PR #1991, validated by exploratory testing of the
drop-vector-index feature against Weaviate 1.39.2. The branch was rebased onto
main v4.23.0; 4abce144 also changes `delete_vector_index()` to return None
(the CLI ignores the return). Pinned to an exact SHA, not the branch ref, so CI
installs the validated commit rather than a drifting branch head.

Regression-checked: 349 unit tests pass, lint clean, and a live smoke of
create/get/update/drop/add_vector/query/backup/restore/tenants against the
1.39.2 cluster.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`update collection --add_vector_index_type` only offered 9 index types while
`create collection --vector_index` offers 18 -- the dynamic variants and
hnsw_multivector were missing. Both build a fresh index (Configure.VectorIndex.*),
so share one source of truth instead of two drifting lists:

- Add a VECTOR_INDEX_TYPES constant in defaults.py; both commands' click.Choice
  now use it.
- Extract CollectionManager._build_vector_index_map() from create_collection and
  reuse it for --add_vector, so the two commands always support the same types.
  --add_vector validates against VECTOR_INDEX_TYPES and passes the built config
  through the shared helper.

Verified live against 1.39.2: adding vectors with dynamic, dynamic_hnsw_pq and
dynamic_flat_bq_hnsw_sq now succeeds, and create collection still builds every
type through the shared helper. 350 unit tests pass, lint clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Newer weaviate-python-client (PR #1991 rounds 5-7) parses a dropped vector's
vector_index_config as a _VectorIndexConfigNone object instead of None, so
`vector_index_config is None` no longer identifies a dropped vector. Two checks
relied on it:

- data_manager create-data auto-skip kept the dropped vector, so `create data
  --randomize` would send it and the server would reject every batch.
- collection_manager __check_drop_target reported a re-drop as not-already-dropped,
  giving the wrong success message on the idempotent re-trigger path.

Add utils.vector_index_dropped() treating a vector as dropped when its config is
None (older clients) or its vector_index_type() == "none" (newer clients), and use
it in both places. Works against the pinned client (4abce144, returns None) and the
latest branch head (returns _VectorIndexConfigNone), verified against both.

355 unit tests pass (5 added for the new representation); lint clean.
The alter-schema-drop-vector-index work (weaviate-python-client #1991) merged to
main (commit decbbd23). Track main until a released weaviate-client ships it, then
swap to a released `weaviate-client>=X` before the next weaviate-cli release.

Verified against main: CLI imports resolve, dropped vectors parse as
_VectorIndexConfigNone and vector_index_dropped() detects them, 355 unit tests pass,
lint clean.
@jfrancoa

Copy link
Copy Markdown
Collaborator Author

@copilot review

Copilot AI and others added 7 commits September 16, 2026 10:49
Co-authored-by: jfrancoa <23482278+jfrancoa@users.noreply.github.com>
Co-authored-by: jfrancoa <23482278+jfrancoa@users.noreply.github.com>
Co-authored-by: jfrancoa <23482278+jfrancoa@users.noreply.github.com>
Co-authored-by: jfrancoa <23482278+jfrancoa@users.noreply.github.com>
Co-authored-by: jfrancoa <23482278+jfrancoa@users.noreply.github.com>
Co-authored-by: jfrancoa <23482278+jfrancoa@users.noreply.github.com>
Co-authored-by: jfrancoa <23482278+jfrancoa@users.noreply.github.com>

Copilot AI commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

@copilot review

Addressed in 41c9196.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add --drop_vector_index to weaviate-cli update collection

2 participants