Skip to content

fix(scorer): score_cats crashes when the predicted doc has no cats - #14075

Open
davrens101 wants to merge 1 commit into
explosion:masterfrom
davrens101:fix/scorer-score-cats-missing-labels-round18
Open

davrens101 wants to merge 1 commit into
explosion:masterfrom
davrens101:fix/scorer-score-cats-missing-labels-round18

Conversation

@davrens101

@davrens101 davrens101 commented Nov 12, 2026 •

Copy link
Copy Markdown

Fix Scorer.score_cats crashing when the predicted doc has no cats but the gold doc does.

In the single-label branch, when pred_cats is empty and gold_cats is non-empty, the code did max(gold_cats, key=lambda it: it[1]), which iterates dict keys (strings) and tries to unpack each key into two values, raising ValueError: too many values to unpack. Changed to max(gold_cats.items(), key=...) to match the other branches.

Adds test_score_cats_missing_predicted_cats asserting the highest-scoring gold label counts as a false negative and cats_macro_f == 0.0.

AI assistance disclosure

This PR was prepared with assistance from an AI coding assistant. The fix and test were verified directly (buggy form crashes with ValueError, fixed form returns the correct label/score).


Checklist:

  • I confirm that I have the right to submit this contribution under the project's MIT license.
  • I ran the tests, and all new and existing tests passed.
  • My changes don't require a change to the documentation

@1AbdullahButt 1AbdullahButt left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks correct. I reproduced the crash and ran the fixed function.

What I checked:

  • The crash is real on the current release. On spaCy 3.8.18, Scorer.score_cats with gold {"POSITIVE": 1.0, "NEGATIVE": 0.0}, predicted {} and multi_label=False raises ValueError: too many values to unpack (expected 2). Iterating the dict gives its string keys, it[1] is the second character, and the unpack fails on the whole label name.
  • With .items() the result is what the new test expects. I took the source of score_cats from 3.8.18, made only this one-line change, and ran it on the same example. I got POSITIVE and NEGATIVE both at p/r/f = 0.0, and cats_macro_f == 0.0. The fn goes to POSITIVE because its gold score 1.0 > 0.
  • The other branches are unchanged. With both sides set (pred {"POSITIVE": 0.9, ...}), macro F is still 0.5. A gold with every score at 0.0 and nothing predicted adds no fn, because of the gold_score > 0 guard.
  • The fixed line now matches the two other max(..., key=lambda it: it[1]) calls in the same block (scorer.py:526-527, :538).

How I tested: python:3.12-slim in Docker with spaCy 3.8.18 from PyPI and spacy.blank("en") docs, running the patched function rather than this branch. I didn't run the repo's test suite.

Reviewed with help from Claude Code.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants