You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Benchmarking & evaluation harness for perturbation prediction (baseline-aware) #1035
As we add perturbation-prediction models, we need a standardized, honest way to evaluate them.
There is strong evidence in the field that trivial baselines (control mean, additive model) often match or beat elaborate models, so baselines must be first-class, not an afterthought. PerturbationComparison is a start but is limited.
What's missing
Standard train/test splits for perturbation prediction: held-out perturbations, held-out combinations, held-out cell types.
Built-in baselines: control mean, additive/linear, nearest-perturbation (reuse DistanceSpace.nearest_perturbations).
Standard metrics: per-DEG delta / logFC correlation, E-distance to ground truth, direction (sign) accuracy, top-k DEG overlap.
A simple leaderboard-style summary over models × metrics × splits.
Proposal / API
A pertpy.tools evaluator that takes ground-truth and predicted AnnData (as returned by the prediction models) and returns a tidy results DataFrame, reusing the existing Distance metrics.
Why it matters
Gives users — and reviewers — an honest, reproducible answer to "is this model actually better than doing nothing clever?".
This is exactly the rigor scverse is trusted for, and it is the natural companion to a broader prediction module.
Related
Prior discussion in #173 (benchmarking tool speed) is about runtime, not predictive accuracy — this is complementary.
Activity