[Yueqing Zhang (BoarQing)]

Summary

PreTokenizerWithRegEx::MatchWithSTLRegEx (the std::regex fallback for Split patterns without a hardcoded matcher) desynced its UTF-8 and UTF-32 copies of the text whenever a multi-byte character came before a match. A later match could then be missing from the UTF-32 copy, so m_text.substr(npos) threw invalid string_view position and encoding failed.

Cause

regex_search runs on the UTF-8 copy (m_utf8_text), but both copies were advanced by m_text.find(res), a code-point index into the UTF-32 copy (m_text):

auto pos = m_text.find(res);                                    // code points
m_utf8_text = m_utf8_text.substr(pos + match[0].str().size());  // used as bytes

With café\nnext\n中文\nend and the pattern (?:\r?\n)+(?!\r?\n), the first \n is at byte 5 but code point 4. From then on the two copies start at different places. After the third match the UTF-8 copy has been cut mid-character and still contains a \n that m_text no longer has.

Fix

Advance the UTF-8 copy by the match's byte position and length (match.position(0), match.length(0)). Advance the UTF-32 copy by the number of code points before the match, counted as the non-continuation bytes before match.position(0).

How it was found

The tokenizer of the Laguna-XS 2.1 model begins its pre-tokenizer with Split on (?:\r?\n)+(?!\r?\n). That pattern takes the fallback path, so encoding any prompt with a non-ASCII character (for example ’) followed by a newline and more text threw. Hugging Face tokenizers 0.23.2 encodes the same text without error.

Test

• New OrtxTokenizerTest.FallbackRegexAfterMultiByteChar in test/pp_api_test/test_tokenizer_impl.cc. Without the fix it fails with invalid string_view position; with the fix it passes.
• pp_api_test on Windows x64 (MSVC 19.44, RelWithDebInfo): 223 tests passed.
• With the same change on the 99fec2db pin used by onnxruntime-genai 0.17, OGA encodes two real Laguna prompts (121 and 1973 tokens) that threw before. The token IDs match Hugging Face tokenizers exactly, apart from the BOS token that OGA leaves out by default (add_special_tokens=false).

Not in this PR

Split steps inside a Sequence are always treated as behavior=Isolated (SplitIsolated). Laguna's first Split uses MergedWithNext. This didn't change the token IDs on the prompts above, but it is a separate gap.

AI assistance: Cursor helped find the cause and write the fix and test. I reviewed the change and ran the tests above.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

🟡 Changes recommended

Zero-length fallback matches can empty the cursor before callers access its next character.

1 open issue

• Zero-length fallback match consumes prefix and causes invalid access · New

What changed in this PR

Fixes UTF-8 and UTF-32 cursor synchronization in the tokenizer’s std::regex fallback.

Changes:

• Advances each text representation using its corresponding offset units.
• Adds a regression test for newline matches after multibyte characters.

File	Description
test/​pp_api_test/​test_tokenizer_impl.cc	Tests fallback splitting after multibyte characters.
operators/​tokenizer/​bpe_utils.hpp	Separates byte offsets from code-point offsets.

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

The focused fix addresses the offset error and prior feedback, with regression coverage and no unresolved findings.

No open issues 🎉

1 resolved since last review

• Zero-length fallback match consumes prefix and causes invalid access

_{Review effort: Balanced}

[Yueqing Zhang (BoarQing)]

Sayan Shaw (@sayanshaw24) Hi, can u approve and merge this?