Skip to content

[roadmap] Range and list line-anchors are silently dropped by the review validator #83

Description

@azevedo

Problem

skills/ba-review/SKILL.md Bullet-anchor grammar requires line to be a single positive integer.
Reviewers routinely emit line ranges (:4-6, :20-37) and comma lists (:21,27) instead.
When the consolidation step enforces the grammar literally, every such finding is dropped as
dropped_no_fileline — silently, with only a counter to show for it.

Observed on a dry-run of the new comment-quality-reviewer
(docs/reviews/2026-08-09-221623-staged/):

⚠ Dropped: 10 findings (no file:line)
  comment-quality-reviewer: 6  (:4-6, :10-13, :8-9, :15-18, :32-34, :1-14)
  complexity-reviewer:      4  (:20-37, :17-18, :15-37, :21,27)

The six lost from comment-quality were its highest-value findings — a High/100 return-shape
enumeration, Medium/75 type-restating params, a Medium/50 missing caller contract, and a Medium/50
standard-mechanics comment. What rendered was the tail. Worse, the consolidated summary.md then
carried a Looks Good entry praising the exact block another reviewer had condemned at High/100 —
because only the surviving verdict made it through.

Why it went unnoticed until now

The "validator" is prompt-implemented, not code — the consolidating model reads the grammar and
applies it. So enforcement is stochastic, and that is the actual bug surface:

Run Range/list anchors present Reported dropped
2026-08-02-224403 6 (complexity 4, error-handling 2) 1
2026-08-04-112242 2 (error-handling 2) 0 — "No records dropped"
2026-08-09-221623 10 10

Range anchors have been emitted in every persisted run on record. Earlier runs tolerated or quietly
normalised them; the 2026-08-09 run enforced the letter of the grammar. So this is not a new
regression — it is a long-standing silent-loss channel that only became visible when a reviewer that
quotes multi-line blocks made the volume large enough to notice.

Why comment-quality-reviewer surfaced it

Structural, not accidental. The other seven anchor to a single statement. A doc comment is a
multi-line block, so quoting one invites a span. It will keep hitting this hardest.

Already mitigated in that one agent (v0.45.0): its ## Principles now require one positive
integer and name the failure. The other seven and the shared grammar are untouched.

Options

  1. Tighten the shared grammar + the Task dispatch template so every reviewer is told
    "single integer, never a range" once. Fixes all eight; touches a consumer file.
  2. Make the consolidation step normalise :4-6 to :4 instead of dropping. Recovers findings
    from reviewers that ignore the rule, and removes the stochastic-enforcement gap.
  3. Both — normalise defensively, instruct explicitly.

Item 2 is the one that closes the silent-loss channel: instruction alone still depends on eight
agents complying every run.

Not in scope of the port

docs/plans/2026-08-09-feat-comment-quality-reviewer-builtin-plan.md explicitly excluded
dispatch-template edits, so only the single-agent mitigation shipped with v0.45.0.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    cluster:review-quality/ba:review & review-plan qualityreadyClear starting point — can build now

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions