Calibrate quality study claims and simplify appendix

[pmcelhaney]

Summary

This is pull request 1 of 3 in the quality-study improvement stack.

• remove duplicated five-year comparison tables from the study appendix so the article owns the headline analysis
• reframe the study as observational and remove causal or overall-quality claims the evidence cannot support
• identify branch coverage precisely and label the mixed 2025 local/Coveralls measurements as non-comparable
• clarify matched-window boundaries, right truncation, source limitations, and evidence provenance

Verification

• npm --prefix site run test:audit
• npm --prefix site run verify:audit
• npm --prefix site run build
• git diff --check

Manual acceptance tests

• Confirm the article headline and abstract describe an observational study rather than a causal result.
• Confirm the five-year comparison tables appear in the article and are not duplicated in the appendix.
• Confirm the 2025 coverage pair is visibly labeled non-comparable.
• Confirm the methods and candidate pages state the limits of the available evidence.
• Confirm the published data identifies analysis v2.1 and branch coverage.

Repository learning check

• Learning found: No
• Guidance updated: No
• Updated file(s): N/A
• Rationale: This stage applies the repository's existing audit guidance and did not reveal a new reusable rule.

[Copilot]

🟡 Changes recommended

The published manifest and supplementary appendix still contain “introduced” wording that contradicts the now-explicit same-window (report + first affected release) rule, which can mislead readers about what was measured.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR recalibrates the 2026 quality study to better match what the underlying evidence supports (observational, measure-specific conclusions) while simplifying the appendix and tightening the manifest/verifier around coverage comparability and analysis provenance.

Changes:

• Reframes defect/coverage claims and “same-window” rules throughout the article and methodology, emphasizing observational limits and right-truncation.
• Removes duplicated matched-window delivery/coverage tables from the supplementary appendix, keeping the article as the canonical presentation.
• Adds an analysis block to the published evidence manifest (versioning + coverage comparability metadata) and extends the verifier/tests to assert branch-coverage semantics.

File summaries

File	Description
site/src/pages/quality/2026/methodology.astro	Updates methodological framing, outcomes wording, and limitations (same-window + truncation).
site/src/pages/quality/2026/index.astro	Rewrites headline/abstract/discussion/conclusion to avoid causal/overall-quality claims; labels mixed 2025 coverage sources as non-comparable.
site/src/pages/quality/2026/context.astro	Removes duplicated tables and refocuses the appendix on supporting detail.
site/src/pages/quality/2026/candidates.astro	Clarifies scope as the record-level 2025–2026 ledger and adjusts dataset language.
site/src/data/quality-audit-evidence.json	Adds analysis provenance/coverage metadata; updates outcome language for branch coverage.
site/src/data/quality-2026.ts	Introduces coverageComparisonStatus to drive table labeling for comparability.
site/scripts/verify-quality-audit.mjs	Adjusts verifier diagnostics messaging to reflect same-window introduced-case logic.
site/scripts/quality-audit.test.mjs	Adds tests asserting branch coverage + 2025 mixed-source labeling in the manifest.
site/scripts/quality-audit-lib.mjs	Enforces new manifest invariants for branch coverage and 2025 comparability labeling.

Review details

Suppressed comments (2)

site/src/pages/quality/2026/context.astro:39

• This appendix still defines an “introduced” defect using only the first affected release being inside the window, but the updated study rule is same-window (both the external report and first affected release must be inside the window). Align this wording with the study rule to avoid contradicting the article/methodology and the verifier logic.

      These complete public GitHub issue and pull-request censuses use the same
      exact window. A reported defect counts as introduced only when the first
      confirmed affected public release also fell within that year’s window.

site/src/data/quality-audit-evidence.json:69

• The retrospectiveComparison.introducedDefectRule description still reads like a first-affected-release-only “within-window introductions” rule, but the implemented introduced-case selection is same-window (reportedAt and firstAffectedPublishedAt both inside the window). Update this manifest text to match the verifier logic so readers don’t infer a different benchmark denominator.

    "retrospectiveComparison": {
      "constructedAfterObservation": true,

• Files reviewed: 9/9 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.