Repository navigation
Off-by-one buffer overrun in external scanner serialize() with deep indentation and an open string #350
Description
tree_sitter_python_external_scanner_serialize writes the indent stack in a loop guarded by size < TREE_SITTER_SERIALIZATION_BUFFER_SIZE, then writes two bytes per iteration. When the header before the loop has odd length (2 + delimiter_count, so one open string delimiter), size can reach 1023. The last iteration then writes one byte past the 1024-byte buffer and returns 1025, and the runtime aborts:
Assertion failed: (length <= 1024), function ts_parser__external_scanner_serialize, file parser.c, line 409.
tree-sitter-python/src/scanner.c
Lines 382 to 387 in 26855ea
Repro (tree-sitter-python 0.25.0, tree-sitter 0.27.0 runtime, macOS arm64; src/scanner.c is byte-identical on master at 26855ea): 600 lines of if 1:, each indented one space more than the last, then 'a' as the body of the innermost one.
python3 -c "print(''.join(' ' * i + 'if 1:\n' for i in range(600)) + ' ' * 600 + \"'a'\")" > deep.pyThe same shape at 400 levels parses.
Suggested fix: test for the bytes about to be written:
for (; iter < scanner->indents.size && size + 2 <= TREE_SITTER_SERIALIZATION_BUFFER_SIZE; ++iter) {
Activity
daiki44 commentedon Oct 29, 2026
Confirming this independently from a downstream project (Onvilo, a macOS app that embeds tree-sitter-python v0.25.0 / commit 293fdc0 as a static C target).
if 1:(one more space each) andx = "a"as the innermost body: the runtime'sts_assert(length <= 1024)fires and the process exits. In our build (tree-sitter 0.25.10 runtime) the byte written atbuffer[1024]lands on the low byte ofTSParser'sstackpointer, so the assert is the lucky outcome.size + 1 < TREE_SITTER_SERIALIZATION_BUFFER_SIZE, equivalent to thesize + 2 <=form suggested above) fixes it; we carry that patch locally until fix(scanner): don't write past the serialization buffer #351 lands.Happy to see #351 merged.