Repository navigation
fix(scorer): score_cats crashes when the predicted doc has no cats - #14075
Open
davrens101 wants to merge 1 commit into
Open
davrens101 wants to merge 1 commit into
davrens101 wants to merge 1 commit into
Conversation
1AbdullahButt
left a comment
There was a problem hiding this comment.
Looks correct. I reproduced the crash and ran the fixed function.
What I checked:
- The crash is real on the current release. On spaCy 3.8.18,
Scorer.score_catswith gold{"POSITIVE": 1.0, "NEGATIVE": 0.0}, predicted{}andmulti_label=FalseraisesValueError: too many values to unpack (expected 2). Iterating the dict gives its string keys,it[1]is the second character, and the unpack fails on the whole label name. - With
.items()the result is what the new test expects. I took the source ofscore_catsfrom 3.8.18, made only this one-line change, and ran it on the same example. I gotPOSITIVEandNEGATIVEboth atp/r/f = 0.0, andcats_macro_f == 0.0. Thefngoes toPOSITIVEbecause its gold score1.0 > 0. - The other branches are unchanged. With both sides set (
pred {"POSITIVE": 0.9, ...}), macro F is still0.5. A gold with every score at0.0and nothing predicted adds nofn, because of thegold_score > 0guard. - The fixed line now matches the two other
max(..., key=lambda it: it[1])calls in the same block (scorer.py:526-527,:538).
How I tested: python:3.12-slim in Docker with spaCy 3.8.18 from PyPI and spacy.blank("en") docs, running the patched function rather than this branch. I didn't run the repo's test suite.
Reviewed with help from Claude Code.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix
Scorer.score_catscrashing when the predicted doc has no cats but the gold doc does.In the single-label branch, when
pred_catsis empty andgold_catsis non-empty, the code didmax(gold_cats, key=lambda it: it[1]), which iterates dict keys (strings) and tries to unpack each key into two values, raisingValueError: too many values to unpack. Changed tomax(gold_cats.items(), key=...)to match the other branches.Adds
test_score_cats_missing_predicted_catsasserting the highest-scoring gold label counts as a false negative andcats_macro_f == 0.0.AI assistance disclosure
This PR was prepared with assistance from an AI coding assistant. The fix and test were verified directly (buggy form crashes with ValueError, fixed form returns the correct label/score).
Checklist: