Skip to content

Prepare AutoProver reports for Solana - #241

Open
ericeil wants to merge 4 commits into
masterfrom
eric/report-abandonment
Open

ericeil wants to merge 4 commits into
masterfrom
eric/report-abandonment

Conversation

@ericeil

@ericeil ericeil commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Third in a series splitting long-lived Solana/CVLR work into independently reviewable pieces (after #239 and #240). AutoProver cannot run the Solana Prover yet; this is shared report work that has to be in place before a Solana backend can produce a report anyone can trust.

Two different layers are involved, and they read more easily apart:

  • Getting verdicts back from the Prover — how a finished job's per-rule results are fetched. An integration defect, and a bug regardless of what consumes the results.
  • What the report document records — two places the report's own vocabulary was narrower than what the pipeline already knew.

composer/spec/source/report_prover.py is the adapter between those layers, which is the only reason one file appears in both halves below.

Getting verdicts back from the Prover

prover_output_utility extracts a job id from /output/<user>/<job>/… or /job/<job> and recognizes no other shape. The Solana Prover's report link — the one certoraSolanaProver prints and the CLI records — is /jobStatus/<userId>/<jobHash>?anonymousKey=…, so the extraction raises.

The report's verdict fetch is deliberately best-effort: any failure means "no verdicts for this run". Together those two facts make a completed Solana job come back empty. On the development branch, a run whose own output.json reported SUCCESS for all seven rules produced a report in which all seven were UNKNOWN, with no source lines and no durations — a false negative in the deliverable, arrived at silently, from a job that was green.

POU's entry points accept a bare job id as well as a URL, so job_input extracts the id and hands that over. Every shape POU already parses passes through untouched, which the tests pin — including the local emv- path its offline mode wants, because fixing the Solana link must not break the EVM one.

Two things to be clear about: the real fix belongs upstream in POU, and nothing on master reaches this path, since the shape is the Solana Prover's. It is here rather than in a PR of its own because it lives in the same adapter as the type change below.

What the report document records

A component that gave up now says why

When AutoProver stops on a component, it usually knows exactly what stopped it — often naming the prover error and the setting that would have unblocked it. The report recorded only that the component produced nothing: the reason existed on both paths (the author's own account, or an exception's text) and was discarded at the boundary, because the input type for "produced nothing" was None and a None has nowhere to put it.

Abandoned(reason) is that variant, and GaveUpComponent.reason is where it lands.

This is the one change here that affects AutoProver today, on EVM runs as much as any other, and the reason is often the most useful line in the document.

The verdict hook no longer names one backend

make_prover_fetcher reads nothing but a run link, but was typed VerdictFetcher[GeneratedCVL], so any other backend wanting the same roll-up had to reach past the factory for the module-private _fetch and re-implement the run_link is None check around it. It is VerdictFetcher[ReportableResult] now, and _fetch becomes public as fetch_verdicts. The test drives the returned fetcher over a non-CVL Formalized, on both the verdict path and the run_link is None one.

Review notes

🤖 Generated with Claude Code

ericeil and others added 2 commits September 16, 2026 15:19
…ined

Three places the report layer was losing something it had been handed.

A component that produced nothing reached `collect` as `None`. The reason was known — a
`GaveUp` carries the author's own account of what stopped it, and a crash carries its
exception text — and both were discarded at the boundary because the input type had nowhere
to put them. `Abandoned(reason)` is that variant, `GaveUpComponent.reason` is where it
lands, and `_abandonment` in the pipeline picks the better of the two sources. This is the
one change here with an immediate effect on every run: a give-up in the report now says why.

`BuildEnvironment` is the second. A backend that compiles the project under verification
can be told not to confine those builds, which makes every verdict it then produces a
development result rather than a production one — and the only record of that was stderr on
whatever machine ran it. `ConfinedBuilds | UnconfinedBuilds` is a union rather than a flag
because the confined case names its mechanism and the other carries nothing, and the field
is optional rather than defaulted because absent is a third state: an EVM backend compiles
nothing of the project and has no build to have confined, and a report written before
schema 3.3 cannot speak to it either. The renderer states only what it was told, and marks
the unconfined case in the header. `Formalizer.build_environment()` is the hook that
supplies it; every backend on master inherits the `None`.

Third, `make_prover_fetcher` was annotated at `GeneratedCVL` while reading nothing but
`run_link`, so a backend wanting the same POU roll-up had to reach past it for the private
function. It is typed at `ReportableResult` now and `fetch_verdicts` is public. Alongside
it, `job_input` handles a job link shape POU's URL parser does not know, by handing over the
job id POU also accepts. That half has no caller on master — the shape is the Solana
Prover's — but it belongs with the fetcher rather than alone, and the failure it prevents is
silent: POU raises, the best-effort fetch swallows it, and every rule of a green run is
reported UNKNOWN.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review pass over c95deaa and e6f9e2e.

Comments describe the code as it is; how it got there belongs here, in the
commit. Eight comments narrated the edit instead of the code -- the CVLR run
that motivated GaveUpComponent.reason, the backend that reached past
make_prover_fetcher, the twelve verdicts re-composed on the code the draft gate
replaced, and five shorter "it used to be None" asides. The claim each one was
making about the code survives; the history is dropped.

"stderr on a machine nobody kept is not a record" appeared three times. It is a
line, not information, and the sentences around it already carry the point.
render.py's build_environment comment restated schema.py's verbatim; trimmed to
the template-side fact.

Two comments the drafted/did_read rename missed: ui_harness_natspec named
validator `did_rough_draft_read` and the read it no longer requires, and
test_autoprove_report's module docstring still described a gap as `None`.

test_the_fetcher_serves_any_reportable_result asserted only callable(fetcher),
so its docstring claimed more than it checked. It now runs the fetcher over a
non-CVL Formalized, both the verdict path and the run_link is None one.

Four completion-rejection messages still told the agent to read the draft and
call read_rough_draft. They fire on `not drafted`, which a write now satisfies,
so they were instructing a rejected agent to make exactly the extra round-trip
e6f9e2e removed. Every .j2 prompt was updated there; these four were not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ericeil ericeil changed the title Report: what a component gave up on, and how its builds were confined Make the report say what the run already knew Sep 16, 2026
@ericeil ericeil changed the title Make the report say what the run already knew Keep Solana prover results and give-up reasons in the AutoProver report Sep 16, 2026
@ericeil ericeil changed the title Keep Solana prover results and give-up reasons in the AutoProver report Prepare AutoProver reports for Solana Sep 16, 2026
ericeil and others added 2 commits September 16, 2026 16:14
Recording whether a build ran confined was carried here from the Solana work on the
assumption that a reader of the report would want the caveat. It does not earn its place:
nothing consumes it, no backend on master can set it, and a field that only ever renders
absent is a schema commitment bought with nothing.

`schema_version` therefore moves one step rather than two: 3.2 is `GaveUpComponent.reason`
and nothing else.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
152e254 reached six files this PR otherwise does not change: the four
completion-rejection messages still asking for a read_rough_draft, the natspec
tape's comment about the validator that replaced, and a docstring trim in the
rough-draft tool tests. All of it is correct and none of it is about the report,
so a reviewer here would be reading it cold.

Restored to master's text. The rough-draft message fix is worth landing where it
is the subject.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ericeil
ericeil marked this pull request as ready for review September 16, 2026 23:21
@ericeil
ericeil requested a review from jtoman September 16, 2026 23:21
ericeil added a commit that referenced this pull request Sep 18, 2026
All three are out of draft. S1 (#239) and S4 (#241) are awaiting a first read;
S3 (#240) has changes requested, with six of its eight threads outdated by the
commits that answered them and two still live as questions.

S3's sizes moved most: the ProverOptions rework came out of that review, so the
PR is 28 files where the plan recorded 16. S4's moved the other way — the
build-environment field was dropped before review, taking report/render.py and
the report template out of its file set.

Two path lists were stale beyond the counts. S1 reaches composer/foundry/entry.py
and spec/source/autoprove_common.py, and S3 reaches the latter too — three lines
apart in the same ProverBackend construction, which is a merge to pay rather than
a hunk to pick, so the appendix no longer claims every path appears once.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ericeil added a commit that referenced this pull request Sep 18, 2026
#239 merged as 27f2fe9 and #240 as 568ba02, half an hour apart on 2026-09-17.
S3 answered its changes-requested review by absorbing the ProverOptions rework,
which is most of why it nearly doubled between opening and merging.

That leaves S4 as wave 1's only open PR, and it inherits both: S1's ecosystem
parameter sits a few lines from the give-up boundary S4 retypes in
pipeline/core.py, and S3's cex_dump envelope is in a test file S4 otherwise
rewrites. merge-tree reports no textual conflict, but #241 has not been gated
against the new head — the caution about those files is rewritten to say so
rather than to describe an ordering that is now settled.

The autoprove_common.py collision between S1 and S3 resolved itself in the merge
order. The cli.py cwd bug fix is still unlanded and still belongs on its own.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant