fix(index)!: do not route json_extract predicates to JSON indices - #9101
Open
westonpace wants to merge 1 commit into
Open
fix(index)!: do not route json_extract predicates to JSON indices#9101westonpace wants to merge 1 commit into
westonpace wants to merge 1 commit into
Conversation
A JSON-path index stores the value at `path` decoded into a native Arrow
type: `{"val": "click"}` trains a Utf8 btree whose key is `click`, and
`{"val": 9}` trains an Int64 btree whose key is `9`. But `json_extract`
evaluates to *serialized* JSON text — `"click"` with quotes, `9` as text —
so `JsonQueryParser::is_valid_reference` was handing the index literals in
a representation its keys never use, and reporting a hardcoded `Utf8` for
the reference regardless of what the index was trained as.
Three ways that went wrong, all silent:
* `json_extract(v, 'val') = '"click"'` searched for a quoted key and
matched nothing, where the unindexed scan matched.
* `json_extract(v, 'val') > '"m"'` returned every row: quoting is not
order-preserving (`ab` < `ab!` but `"ab"` > `"ab!"`), so a decoded-key
btree cannot answer a text-ordered range, and its page min/max pruning
is unsound for one.
* A numeric path took a `Utf8` literal into an `Int64` btree, whose page
scan selects its comparator from the query's type on the stated
invariant that the two always agree, and panicked.
Decline these predicates so they fall back to a full scan, which is what
the unindexed plan already computes correctly. Only the typed accessors
(`json_get_int`/`_float`/`_bool`/`_string`) decode the path value, so only
they match the stored keys and stay routable. `json_get` leaves the
accepted set too, but it never had a type arm and so never routed.
This costs `json_extract` filters their index acceleration; no correct
behavior is lost. Restoring it needs the literal transcoded into the
index's own representation for equality, and the trained type recovered via
`training_data_type()` so the reported reference type stops being a guess —
both follow-ups, neither requiring a format change.
Fixes lance-format#8806
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RSJs3zhGP8GogcDqXvtQ8v
![lance-gatekeeper[bot]](https://avatars.githubusercontent.com/in/4433141?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Declining json_extract at the JSON-index reference boundary restores indexed/unindexed result parity without changing persisted index data. The fallback covers both string and numeric equality and ranges.
The accepted trade-off is that existing json_extract filters no longer receive JSON-index acceleration and can become full scans on large datasets. Typed JSONPath extraction is the natural route to recover that performance without conflating serialized JSON text with native index keys; no additional change is requested here.
lance-gatekeeper
Bot
added
K-approved
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A JSON-path index stores the value at
pathdecoded into a native Arrow type:{"val": "click"}trains a Utf8 btree whose key isclick, and{"val": 9}trains an Int64 btree whose key is9. Butjson_extractevaluates to serialized JSON text —"click"with quotes,9as text.This caused some problems we could potentially fix:
json_extract(v, 'val') = '"click"'searched for a quoted key and matched nothing, where the unindexed scan matched.Utf8literal into anInt64btree, whose page scan selects its comparator from the query's type on the stated invariant that the two always agree, and panicked.But it also meant some subtle issues we could not fix:
json_extract(v, 'val') > '"m"'returned every row: quoting is not order-preserving (ab<ab!but"ab">"ab!"), so a decoded-key btree cannot answer a text-ordered range, and its page min/max pruning is unsound for one.This PR changes the index to decline these predicates so they fall back to a full scan. It's unfortunate, but I think we'd need a specialized string-based index if we wanted to fully support
json_extract. If there is sufficient demand then we can investigate this approach.Fixes #8806