fix(fts): honor custom ignore_pattern (and tokenizer) when normalizing queries - #82
Merged
Merged
Conversation
…g queries (#910)
CREATE_FTS_INDEX accepts an ignore_pattern option, but the rewritten
_CREATE_FTS_INDEX call only forwarded stemmer and stopWords, so the
catalog entry that QUERY_FTS_INDEX reads kept the default pattern. Any
token the custom pattern preserved (digits, hyphens, ...) could never be
matched by an exact query, e.g.:
CREATE (:T {content:'Audi A4 Avant 2022'});
CALL CREATE_FTS_INDEX('T','idx',['content'], stemmer:='none',
ignore_pattern:='[^[:alnum:]-]+');
CALL QUERY_FTS_INDEX('T','idx','A4') RETURN *; -- [] before, now matches
- forward ignore_pattern, tokenizer and jieba_dict_dir to
_CREATE_FTS_INDEX so the persisted FTSConfig matches the index that
was actually built (tokenizer/jieba_dict_dir were dropped too, which
made jieba indexes tokenize queries with the simple tokenizer)
- FTSUtils::normalizeQuery: optionally protect '*'/'?' wildcards from
the ignore pattern by normalizing in segments between wildcard
characters. The previous trick of deriving a query pattern by deleting
'*'/'?' from the pattern string only worked for enumerated character
classes; with e.g. [^[:alnum:]-]+ a wildcard query would lose its
wildcards. The query path opts in; document normalization is unchanged
- jieba query tokenization: skip whitespace-only tokens that CutForSearch
emits between words (surfaced now that the query side actually uses the
configured tokenizer; they broke conjunctive queries)
- regression test ignore_pattern.test using the existing fts-emails
dataset (file names carry digits/hyphens), covering exact queries,
wildcard queries and index reload
Fixes #910
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes LadybugDB/ladybug#910
Problem
CREATE_FTS_INDEXaccepts anignore_patternoption, but the rewrittenCALL _CREATE_FTS_INDEX(...)only forwardedstemmerandstopWords. The index itself was built with the custom pattern (via the TOKENIZE macro), but the catalog entry thatQUERY_FTS_INDEXreads kept the default pattern — so any token the custom pattern was meant to preserve (digits, hyphens, …) could never be matched by an exact query:tokenizerandjieba_dict_dirwere dropped the same way, which additionally made jieba indexes tokenize queries with the simple tokenizer.Changes
create_fts_index.cpp: forwardignore_pattern,tokenizerandjieba_dict_dirto_CREATE_FTS_INDEXso the persistedFTSConfigmatches the index that was actually built.fts_utils.cpp(normalizeQuery): new optional wildcard protection — the query is normalized in segments between*/?characters so wildcards survive even when the pattern would match them (e.g. a negated class like[^[:alnum:]-]+). The previous trick of deriving a query pattern by deleting*/?from the pattern string only worked for enumerated character classes. Only the query path opts in; document-content normalization is unchanged (insert/update/delete still strip wildcards, matching the TOKENIZE macro).fts_utils.cpp(tokenizeString): skip whitespace-only tokens that jiebaCutForSearchemits between words. This surfaced once the query side actually used the configured tokenizer — e.g. the conjunctive query深度学习 发展produced a" "term that no document contains.fts_config.cpp: the customignore_patternis now stored as-is for queries (ignorePatternQuery = ignorePattern) since wildcards are protected during normalization.fts/test/test_files/ignore_pattern.testreusing the existingfts-emailsdataset (file names likeallen-p/_sent_mail/102.carry digits/hyphens), covering exact queries, wildcard queries, and query behavior after index reload. No dataset changes needed.Test plan