Skip to content

Hosted storyboard history: represent incomplete fixture coverage separately from executed passes #8416

Open

Description

@bokelley

Observed behavior

A full reporting run against a native catalog without the optional feature returns an explicit coverage_gaps[].reason: fixture_unsatisfied with zero authored reporting steps. The SDK's raw overall_passed boolean can still be true because no executed assertion failed. Addie's existing history writer stores that boolean and counts without a separate incomplete-coverage state; the manual matrix's clean count similarly means no failed rows, not tested feature support.

#8336 fixes Addie's visible report to show Incomplete — coverage gap and Not applicable for this run, while preserving actual SDK results, existing history fields and floor accounting. Training CLI now prints the real coverage-gap detail. No missing native fixture is treated as proof of nonsupport or as an executed reporting pass.

Follow-up: give hosted run history/aggregate presentation a reviewed representation of incomplete fixture coverage, retaining original failure evidence and distinguishing a completed no-match from discovery/schema/protocol errors. Preserve immutable historical records; do not backfill invented support, globally skip mandatory failures or silently lower grading floors. A failed selected implementation must stay failed even when another scope is unexercised.

This host accounting/presentation follow-up is separate from catalog selection and the per-invocation reporting adapter; no evaluator/registry policy implementation or release is assigned by this issue.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions