gh-149183: avoid MemoryError in tokenize#149445
Conversation
Documentation build overview
65 files changed ·
|
StanFromIreland
changed the title
| size tracking to underflow. Treat as a tokenizer error rather | ||
| than passing a negative length (cast to huge size_t) to malloc or | ||
| PyUnicode_DecodeUTF8. */ | ||
| return -1; |
There was a problem hiding this comment.
Can we set tok->done or raise a tokenizer syntax error before returning here? Otherwise this reports unknown tokenization error and compile() gets unknown parsing error.
There was a problem hiding this comment.
Good point, I'll look into that. Thanks.
| size tracking to underflow. Treat as a tokenizer error rather | ||
| than passing a negative length (cast to huge size_t) to malloc or | ||
| PyUnicode_DecodeUTF8. */ | ||
| return -1; |
There was a problem hiding this comment.
We return -1 here without setting tok->done or an exception, so the user ends up with the generic unknown tokenization error (and unknown parsing error from the compiler) instead of something descriptive. Should we set tok->done = E_TOKEN or call _PyTokenizer_syntaxerror so the message points at the actual problem?
There was a problem hiding this comment.
(I believe this is the same suggestion as your other comment, I'll try to do it the way you suggest 🙇)
There was a problem hiding this comment.
Okay so I did exactly what you suggested -- tok->done = E_TOKEN :)
|
A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated. Once you have made the requested changes, please leave a comment on this pull request containing the phrase |
|
Sorry in advance, I had to rebase on |
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
KowalskiThomas
force-pushed
the
kowalski/fix-avoid-memoryerror-in-tokenize
branch
from
9203a8e to
1937551
Compare
3 participants
Fixes #149183
MemoryErroron invalid input #149183