Conversation
Adds `weaviate-cli update collection --drop_vector_index <name>` to drop the
index of a named vector (Weaviate 1.39 experimental endpoint, exposed by
weaviate-python-client#1991 as collection.config.delete_vector_index()).
- defaults.py: UpdateCollectionDefaults.drop_vector_index
- update.py: --drop_vector_index option, mutually exclusive with --vector_index
- collection_manager.py:
- drop runs AFTER config.update() (which reads-and-writes the whole schema),
so a "none"-marked vector is never PUT back to the server
- get collection listing renders a dropped vector as `none` and no longer
crashes on a null vector_index_config; shows all distinct index types
(e.g. `hnsw, none`)
- re-dropping an already-"none" vector is allowed (server returns 200): it is
a no-op while cleanup runs and re-enqueues a fresh task if the previous one
FAILED -- the only operator recovery path for a stalled drop
- version warning when the server is older than v1.39.0
- unit tests: happy path, validation, listing, and re-trigger
- requirements-dev.txt / setup.cfg: temporary pin to the client PR #1991 commit
(revert to a released weaviate-client before release)
- operating skill docs
Closes #170
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
7 tasks
There was a problem hiding this comment.
Orca Security Scan Summary
| Status | Check | Issues by priority | |
|---|---|---|---|
| Secrets | View in Orca |
A dropped vector index CAN be re-created (confirmed with the developer and verified live via config.add_vector() once the drop finalizes). Correct the docs/help/messages that wrongly said "cannot be re-created" / "irreversible": - update.py help, collection_manager success + conflict messages - operating skill (SKILL.md, collections.md): re-creation is allowed after the drop finalizes (fresh, empty index); "irreversible" removed create data: automatically skip named vectors whose index was dropped (vectorIndexType "none", reported as vector_index_config == None). The server rejects any object carrying a dropped vector, so every batch was failing after a drop. Now those vectors are excluded from generated objects (with an informational note, suppressed under --json) and ingestion keeps working. - data_manager.__ingest_data: filter dropped vectors, derive vectorizer from a live one, note skipped vectors - data.md: document the auto-skip - unit tests: skip one, skip all, skip none, --json suppression Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Orca Security Scan Summary
| Status | Check | Issues by priority | |
|---|---|---|---|
| Infrastructure as Code | View in Orca | ||
| SAST | View in Orca | ||
| Secrets | View in Orca | ||
| Vulnerabilities | View in Orca |
Advances the temporary weaviate-python-client pin to the current head of PR #1991. Since our previous pin (2dbfaedf) the PR added two parser fixes we need: b92dbb71 replaced an assert in the dropped-vector schema parser with an explicit SchemaValidationError (-O safe), and 9d7b9163 fixes a KeyError 'vectorizer' crash when parsing a legacy (single-vector) collection whose vector index was dropped -- which otherwise breaks `get collection` and `delete collection --all`. No weaviate-cli code change is required; the fix is entirely in the client parser. Lint clean, 336 unit tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`create collection --named_vector_name` now accepts a comma-separated list, creating one named vector per name (all sharing the chosen --vectorizer and --vector_index). A single name keeps the previous behavior, so existing callers are unaffected. Duplicate or empty names are rejected. This makes it possible to create collections with more than one named vector, e.g. to test dropping one named vector's index while keeping others. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`update collection --add_vector NAME` adds a named vector via the client's config.add_vector(), so a vector whose index was dropped can be re-created from the CLI (and then re-ingested). --add_vector_vectorizer picks the vectorizer (none/contextionary/transformers/model2vec, default none = self-provided) and --add_vector_index_type picks the index, including quantized variants (hnsw_pq, hnsw_sq, hnsw_bq, hnsw_rq, hfresh, flat_bq, hnsw_acorn); pq/sq honor --training_limit. Mutually exclusive with --drop_vector_index and --vector_index. Unit tests cover the added config, quantizers, and the validation guards. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`query data --search_type keyword` passed `return_objects=True` to the python-client's `query.bm25()`, which does not accept that argument (bm25 returns objects by default). Against a current client this raised `_BM25QueryExecutor.bm25() got an unexpected keyword argument 'return_objects'`, breaking every keyword query. Remove the kwarg. The other search types (fetch/vector/hybrid) never passed it. Add an integration test that runs all four `query data` search types against a real client, so a client query-signature drift like this is caught in CI (the unit tests mock the client and cannot catch an unexpected-kwarg mismatch). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Pin both setup.cfg and requirements-dev.txt to weaviate-python-client commit 4abce144, the current head of PR #1991, validated by exploratory testing of the drop-vector-index feature against Weaviate 1.39.2. The branch was rebased onto main v4.23.0; 4abce144 also changes `delete_vector_index()` to return None (the CLI ignores the return). Pinned to an exact SHA, not the branch ref, so CI installs the validated commit rather than a drifting branch head. Regression-checked: 349 unit tests pass, lint clean, and a live smoke of create/get/update/drop/add_vector/query/backup/restore/tenants against the 1.39.2 cluster. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`update collection --add_vector_index_type` only offered 9 index types while `create collection --vector_index` offers 18 -- the dynamic variants and hnsw_multivector were missing. Both build a fresh index (Configure.VectorIndex.*), so share one source of truth instead of two drifting lists: - Add a VECTOR_INDEX_TYPES constant in defaults.py; both commands' click.Choice now use it. - Extract CollectionManager._build_vector_index_map() from create_collection and reuse it for --add_vector, so the two commands always support the same types. --add_vector validates against VECTOR_INDEX_TYPES and passes the built config through the shared helper. Verified live against 1.39.2: adding vectors with dynamic, dynamic_hnsw_pq and dynamic_flat_bq_hnsw_sq now succeeds, and create collection still builds every type through the shared helper. 350 unit tests pass, lint clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Newer weaviate-python-client (PR #1991 rounds 5-7) parses a dropped vector's vector_index_config as a _VectorIndexConfigNone object instead of None, so `vector_index_config is None` no longer identifies a dropped vector. Two checks relied on it: - data_manager create-data auto-skip kept the dropped vector, so `create data --randomize` would send it and the server would reject every batch. - collection_manager __check_drop_target reported a re-drop as not-already-dropped, giving the wrong success message on the idempotent re-trigger path. Add utils.vector_index_dropped() treating a vector as dropped when its config is None (older clients) or its vector_index_type() == "none" (newer clients), and use it in both places. Works against the pinned client (4abce144, returns None) and the latest branch head (returns _VectorIndexConfigNone), verified against both. 355 unit tests pass (5 added for the new representation); lint clean.
The alter-schema-drop-vector-index work (weaviate-python-client #1991) merged to main (commit decbbd23). Track main until a released weaviate-client ships it, then swap to a released `weaviate-client>=X` before the next weaviate-cli release. Verified against main: CLI imports resolve, dropped vectors parse as _VectorIndexConfigNone and vector_index_dropped() detects them, 355 unit tests pass, lint clean.
Collaborator
Author
|
@copilot review |
Co-authored-by: jfrancoa <23482278+jfrancoa@users.noreply.github.com>
Co-authored-by: jfrancoa <23482278+jfrancoa@users.noreply.github.com>
Co-authored-by: jfrancoa <23482278+jfrancoa@users.noreply.github.com>
Co-authored-by: jfrancoa <23482278+jfrancoa@users.noreply.github.com>
Co-authored-by: jfrancoa <23482278+jfrancoa@users.noreply.github.com>
Co-authored-by: jfrancoa <23482278+jfrancoa@users.noreply.github.com>
Co-authored-by: jfrancoa <23482278+jfrancoa@users.noreply.github.com>
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
weaviate-cli update collection --drop_vector_index <name>to drop the index of anamed vector — the Weaviate 1.39 experimental endpoint
(
DELETE /v1/schema/{Class}/vectors/{name}/index), exposed by the Python client inweaviate/weaviate-python-client#1991 as
collection.config.delete_vector_index(). The dropis destructive and applied asynchronously (a 200 means accepted, not done). The vector can
be re-created afterwards (a fresh, empty index via
config.add_vector()) once the dropfinalizes — verified live.
Closes #170
What changed
update.py/defaults.py—--drop_vector_index <name>, mutually exclusive with--vector_index; warns when the server is older than v1.39.0.collection_manager.pyconfig.update().config.update()reads the whole schema andPUTs it back, so dropping first would send a
vectorIndexType: "none"vector back to theserver. Ordering is asserted by a unit test.
get collectionlisting fix: a dropped vector'svector_index_configisNone; theold listing called
.vector_index_type()on it and crashed. It now rendersnoneandshows every distinct index type across named vectors (e.g.
hnsw, none).nonevector returns 200 on theserver — a no-op while cleanup runs, and a fresh cleanup task if the previous one FAILED
(the only operator recovery path for a stalled drop). The CLI reports the re-trigger
instead of blocking it.
create dataauto-skips dropped vectors — after a drop, the server rejects any objectcarrying the dropped vector, so
--randomizeingestion failed every batch. Generatedobjects now exclude any named vector whose index is dropped (
vectorIndexType: "none"),with an informational note (suppressed under
--json). Ingestion keeps working; re-ingestafter re-creating the index.
--vector_indexconflict, unknown/absent vector, listing(
none/hnsw, none), re-trigger, and thecreate dataskip (one / all / none /--json).336 pass.
create dataauto-skip).requirements-dev.txtandsetup.cfgpinweaviate-clientto the commit on thepython-client PR #1991 branch
(which adds
delete_vector_index()andVectorIndexType.NONE). That PR is not yet merged.Before a weaviate-cli release this must be reverted to a released
weaviate-client>=X. A gitURL cannot be published to PyPI, so this should not merge to a release tag as-is.
Test plan
make lintmake test(336 passed)stay 100% available during cleanup; dropped-vector search rejected immediately;
hnsw, noneshown mid-drop; re-drop of a
nonevector accepted;--vector_indexconflict rejected;vector re-created after finalize via
config.add_vector()and searchable again;create dataskips a dropped vector and ingests successfully (note shown;--jsonclean).🤖 Generated with Claude Code