Skip to content

Fix comprehensiveness/sufficiency crash on code-sequence inputs - #1284

Merged
jhnwu3 merged 1 commit into
sunlabuiuc:masterfrom
solarsys:fix/ablation-integer-inputs
Oct 11, 2026
Merged

jhnwu3 merged 1 commit into
sunlabuiuc:masterfrom
solarsys:fix/ablation-integer-inputs

Conversation

@solarsys

@solarsys solarsys commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator

Why

ComprehensivenessMetric and SufficiencyMetric (and evaluate_attribution / Evaluator) crash for any model whose inputs are plain code-sequence tensors: MLP, Transformer, RETAIN and so on.

RuntimeError: Expected tensor for argument #1 'indices' to have one of the following scalar types: Long, Int; but got torch.FloatTensor instead (while checking arguments for embedding)

RemovalBasedMetric._apply_ablation handles StageNet's (time, values) tuples with an integer-safe path, but plain tensors are ablated with x * (1 - mask). The float mask turns the code ids into floats, so the embedding lookup fails. The "mean" strategy has the same problem, and averaging code ids would be meaningless even if it ran.

@fbonc proposed a fix for this in #1170, which was auto-closed as stale. This PR revives it. Credit to them for the diagnosis.

What

  • pyhealth/metrics/interpretability/base.py:
    • New helper _ablate_codes sets the removed codes to the padding id 0 with torch.where, so the dtype is kept. It also keeps the existing guard: a sample whose codes would all be removed keeps its first code.
    • Both the tuple path and the plain-tensor path now use the helper for integer inputs, whatever ablation_strategy is set to.
    • Float inputs behave exactly as before.
  • docs/api/metrics/pyhealth.metrics.interpretability.rst: explains how integer inputs are ablated.
  • examples/interpretability/comprehensiveness_code_sequences.py: trains an MLP on synthetic code sequences and scores Integrated Gradients. It runs in seconds on CPU with no downloads. It prints comprehensiveness 0.987 and sufficiency -0.006, because the label is driven by a single code.

Tests

  • New tests/core/test_interp_ablation_codes.py (5 tests):
    • the helper's dtype and padding behaviour;
    • the keep-one-code guard;
    • float inputs still follow the strategy;
    • evaluate_attribution with MLP and Transformer under all three strategies gives finite scores.
  • The end-to-end tests crash with the error above on master and pass with this change.
  • Existing interpretability tests pass: test_interp_metrics, test_interpret_ensemble, test_ig_gim, test_stagenet, test_stagenet_mha, test_shap, test_lime (178 tests, OK).
  • tools/check_pr_rules.py passes.

🤖 Generated with Claude Code

RemovalBasedMetric._apply_ablation multiplied plain integer code-id
tensors by a float mask, which turned them into floats, so nn.Embedding
raised and comprehensiveness/sufficiency crashed for every model with
sequence inputs (MLP, Transformer, RETAIN, ...). Only StageNet's tuple
inputs took the integer-safe path.

Integer inputs are now ablated by setting the removed codes to the
padding id 0 with torch.where, keeping the dtype, for every ablation
strategy. One helper serves both the tuple and plain-tensor paths, and
keeps the existing guard that leaves one code per sample. Float inputs
are unchanged.

Revives the fix proposed in sunlabuiuc#1170 by @fbonc, which was closed as stale.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@solarsys
solarsys requested a review from jhnwu3 October 9, 2026 17:15

@jhnwu3 jhnwu3 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@jhnwu3
jhnwu3 merged commit 9463aae into sunlabuiuc:master Oct 11, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants