Skip to content

fix(fts): honor custom ignore_pattern (and tokenizer) when normalizing queries - #82

Merged
adsharma merged 1 commit into
mainfrom
fix/fts-ignore-pattern-query-910
Sep 6, 2026
Merged

fix(fts): honor custom ignore_pattern (and tokenizer) when normalizing queries#82
adsharma merged 1 commit into
mainfrom
fix/fts-ignore-pattern-query-910

Conversation

@adsharma

@adsharma adsharma commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Fixes LadybugDB/ladybug#910

Problem

CREATE_FTS_INDEX accepts an ignore_pattern option, but the rewritten CALL _CREATE_FTS_INDEX(...) only forwarded stemmer and stopWords. The index itself was built with the custom pattern (via the TOKENIZE macro), but the catalog entry that QUERY_FTS_INDEX reads kept the default pattern — so any token the custom pattern was meant to preserve (digits, hyphens, …) could never be matched by an exact query:

CALL CREATE_FTS_INDEX('T', 'idx', ['content'], stemmer := 'none', ignore_pattern := '[^[:alnum:]-]+');
CALL QUERY_FTS_INDEX('T', 'idx', 'A4') RETURN *;    -- [] before, now matches
CALL QUERY_FTS_INDEX('T', 'idx', 't-roc') RETURN *; -- [] before, now matches

tokenizer and jieba_dict_dir were dropped the same way, which additionally made jieba indexes tokenize queries with the simple tokenizer.

Changes

  • create_fts_index.cpp: forward ignore_pattern, tokenizer and jieba_dict_dir to _CREATE_FTS_INDEX so the persisted FTSConfig matches the index that was actually built.
  • fts_utils.cpp (normalizeQuery): new optional wildcard protection — the query is normalized in segments between */? characters so wildcards survive even when the pattern would match them (e.g. a negated class like [^[:alnum:]-]+). The previous trick of deriving a query pattern by deleting */? from the pattern string only worked for enumerated character classes. Only the query path opts in; document-content normalization is unchanged (insert/update/delete still strip wildcards, matching the TOKENIZE macro).
  • fts_utils.cpp (tokenizeString): skip whitespace-only tokens that jieba CutForSearch emits between words. This surfaced once the query side actually used the configured tokenizer — e.g. the conjunctive query 深度学习 发展 produced a " " term that no document contains.
  • fts_config.cpp: the custom ignore_pattern is now stored as-is for queries (ignorePatternQuery = ignorePattern) since wildcards are protected during normalization.
  • New regression test fts/test/test_files/ignore_pattern.test reusing the existing fts-emails dataset (file names like allen-p/_sent_mail/102. carry digits/hyphens), covering exact queries, wildcard queries, and query behavior after index reload. No dataset changes needed.

Test plan

  • Local: full FTS e2e suite passes (69 tests), including the new regression test
  • CI

…g queries (#910)

CREATE_FTS_INDEX accepts an ignore_pattern option, but the rewritten
_CREATE_FTS_INDEX call only forwarded stemmer and stopWords, so the
catalog entry that QUERY_FTS_INDEX reads kept the default pattern. Any
token the custom pattern preserved (digits, hyphens, ...) could never be
matched by an exact query, e.g.:

    CREATE (:T {content:'Audi A4 Avant 2022'});
    CALL CREATE_FTS_INDEX('T','idx',['content'], stemmer:='none',
        ignore_pattern:='[^[:alnum:]-]+');
    CALL QUERY_FTS_INDEX('T','idx','A4') RETURN *;  -- [] before, now matches

- forward ignore_pattern, tokenizer and jieba_dict_dir to
  _CREATE_FTS_INDEX so the persisted FTSConfig matches the index that
  was actually built (tokenizer/jieba_dict_dir were dropped too, which
  made jieba indexes tokenize queries with the simple tokenizer)
- FTSUtils::normalizeQuery: optionally protect '*'/'?' wildcards from
  the ignore pattern by normalizing in segments between wildcard
  characters. The previous trick of deriving a query pattern by deleting
  '*'/'?' from the pattern string only worked for enumerated character
  classes; with e.g. [^[:alnum:]-]+ a wildcard query would lose its
  wildcards. The query path opts in; document normalization is unchanged
- jieba query tokenization: skip whitespace-only tokens that CutForSearch
  emits between words (surfaced now that the query side actually uses the
  configured tokenizer; they broke conjunctive queries)
- regression test ignore_pattern.test using the existing fts-emails
  dataset (file names carry digits/hyphens), covering exact queries,
  wildcard queries and index reload

Fixes #910
@adsharma
adsharma merged commit 62aa925 into main Sep 6, 2026
2 checks passed
@adsharma
adsharma deleted the fix/fts-ignore-pattern-query-910 branch September 6, 2026 21:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: FTS ignore_pattern is applied when building the index but not when normalising the query

1 participant