Apply node table update before updating indexes (fixes HNSW SET-on-NULL segfault, #896) - #900
Merged
Merged
Conversation
adsharma
force-pushed
the
fix-896-update-before-index
branch
2 times, most recently
from
September 2, 2026 17:47
e4dac98 to
7f61361
Compare
adsharma
force-pushed
the
fix-896-update-before-index
branch
from
September 2, 2026 18:34
7f61361 to
4906e9f
Compare
Indexes have conflicting needs when a column they are built on is updated: FTS delete_ re-tokenizes the OLD document from the node table (must run before the write), while the HNSW index re-scans the updated node's embedding during re-insertion (must run after the write). Previously all indexes ran before the write, so HNSW observed a stale embedding: when the previous value was NULL it dereferenced a null EmbeddingHandle and segfaulted in the distance function, potentially leaving the WAL unreplayable. Run "before" indexes first (all indexes except HNSW, preserving existing behavior), then apply the row update to the node table or local storage, then run the HNSW index update. The ordering is keyed on the serialized index type name to avoid extending the extension-facing Index API. The isLoaded() orphaned-holder guard is preserved in both loops. Fixes #896.
adsharma
force-pushed
the
fix-896-update-before-index
branch
from
September 2, 2026 19:27
4906e9f to
b01b1da
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #896.
Problem
SET n.embedding = $vecon a node whoseembeddingis NULL, while an HNSW vector index exists on the column, segfaults (SIGSEGV) nondeterministically and can leave the WAL unreplayable (Corrupted wal file. Read out invalid WAL record type.on next open). Reported with a self-contained repro and stacks in #896 (0.20.1, macOS arm64).Root cause
NodeTable::update()calledindex->update()before writing the new value into the node table. HNSW index maintenance (OnDiskHNSWIndex::update->commitInsert->insertToLayer->createRels->shrinkForNode) scans the node table to read the updated node's embedding. With the old ordering it read the stale value — NULL when the row had no embedding at index build — producing a nullEmbeddingHandlewhosegetPtr()was passed to the distance function (simsimd_cos_f32_neon), i.e. an out-of-bounds read that segfaults depending on memory layout. This also explains the issue's observed shape-dependence: re-SET of a row that already had a vector and CREATE of a row carrying its vector never hit the NULL path.Changes
Index kinds have conflicting needs when their column is updated:
delete_re-tokenizes the old document from the node table → must run before the row write,NodeTable::update()is now two-phase: run "before" indexes, apply the row update to the node table (local storage or node group), then run "after" indexes. The ordering is keyed on the serialized index type name ("HNSW") to avoid extending the extension-facingIndexAPI, so this PR needs no coordinated extension change. The existingisLoaded()orphaned-holder guard is preserved in both loops.Paired extension change (already merged, submodule bumped here): LadybugDB/extensions#78 — guards
shrinkForNode()against a NULL scanned embedding and adds regression tests.Testing
show_loaded_extensions, which requires duckdb/httpfs extensions not built in this environment.mu 30 / ml 60 / cosine / efc 200): 50 SETs on NULL rows +CHECKPOINT+ reopen, 5/5 runs clean, WAL replays without corruption.