test(appraisal): candidate vectors for appraisal.policy_ref resolution

[opento-suggestions]

Candidate conformance vectors for appraisal.policy_ref resolution — answering the acceptance on agentrust-io/trace-spec#66 (2026-08-18). The boundary originates from #63: policy_ref can be format-checked but not reproduced, because the record carries a digest for the enforcement policy (policy.bundle_hash) and none for the appraisal policy that produced the verdict.

Seven vectors under tests/vectors/appraisal-resolution/, one defect each, every record identical except appraisal.policy_ref:

#	Vector	Boundary	Expected
01	no-binding-declared	accept	pass
02	resolved-and-matches	accept	pass
03	digest-mismatch, one byte apart	contradicted	reject
04	digest-mismatch, other object	contradicted	reject
05	referent unreachable	unresolvable	deferred
06	digest algorithm uncomputable	unresolvable	deferred
07	binding bound to another URI's object	contradicted	reject

Standalone by design. No module, no schema change, no dependency, no edits to existing files (+1611/−0). The set runs green on main with no verifier present, because none exists for this field — which is the gap. The tests grade the set's internal consistency: every digest is SHA-256 over the exact bytes of sibling files under policies/, recomputable from the directory alone, and a deterministic generator is held to byte-reproduction by a self-contained in-repo test (per the principle merged in #66: a guard that needs another repository checked out is a guard that gets skipped).

What it deliberately does not decide. candidate_binding is marked CANDIDATE: in every vector and lives in the vector's context, never in record — appraisal is additionalProperties: false, and proposing a field is an editorial decision. Vectors 05 and 06 assert only must_not: "affirming", deferring the unresolvable-case outcome to agentrust-io/trace-spec#190 rather than coining one; deferred is fixture bookkeeping, not a candidate appraisal.status value, and a completeness test fails if any vector reuses a status value as an outcome. Both shortfalls — no verifier exercised, unresolvable outcome unnamed — are pinned in KNOWN_SHORTFALLS so they cannot drift silently.

Built to the adequacy criteria in agentrust-io/trace-spec#186: the must-accept pair (01, 02) makes the set fail an unconditionally-rejecting verifier, and each boundary carries two independent vectors split by a named defect.

Two reviewable choices, flagged rather than buried: the scoped .gitattributes pinning eol=lf for this directory is load-bearing (with core.autocrlf=true, checkout rewrites LF→CRLF and every digest in the set breaks on a clean clone) — but whether it belongs here, at the root, or elsewhere is a repository-convention call. And the candidate binding's field name is a placeholder shape for review, not a proposal.

Verification on main at c725bbb: 246 passed, 5 xpassed (+45 over base, nothing displaced); 88 in the level0 or negative gate with all 45 new tests collecting; ruff and mypy clean on the new files.

Refs: #63, agentrust-io/trace-spec#66, agentrust-io/trace-spec#190

[lywinged]

This is well built, and I checked rather than assumed. Flipping one byte in a policies/ file, changing 05's outcome to a status value, and deleting 04 each turn the suite red, on test_the_declared_resolution_matches_the_bytes_on_disk, on test_no_vector_proposes_an_appraisal_status_value, and on test_the_generator_produces_nothing_the_set_does_not_carry respectively. The four policy digests recompute from the directory alone. The set also genuinely meets the criteria it cites: 01 and 02 expecting pass against 03, 04 and 07 expecting reject makes it fail both unconditional implementations, which canonicalization-boundary over in trace-spec does not manage.

What is missing is the one you recorded yourself, no_verifier_exercised, and it is worth saying plainly that on this field it cannot be closed. appraisal.policy_ref is a bare URI. The record states nothing about the referent, so there is nothing to compare a resolved document against, and there will not be until a field lands, which is normative and not yours or mine to decide. Putting candidate_binding in context is the right way to write that down, and it is also the reason no verifier can run these.

One field over, both halves are already merged.

policy is top-level required. Inside it bundle_hash is required, "SHA-256 or SHA-384 digest of the policy bundle in force at execution time", and beside it sits policy_uri, optional, "format": "uri", described in the schema as "URI to the policy bundle for verification". Where both are present the record carries a referent and a digest of that referent, both fields already merged, with no candidate field needed and nothing for the schema editorial process to decide. Nothing resolves the pair. In this repository policy_uri is zero occurrences under src/, and bundle_hash appears only in tr_pol.py, where TR-POL-001 regex-matches the digest's shape and reports policy.bundle_hash has valid digest format. In trace-spec, setting aside the packaged schema copies, both appear only in the adapters that write them and in the dataclass fields that declare them. Nobody fetches the bundle and recomputes anything.

So what is left, concretely:

• A resolver, caller-supplied, defaulting to absent so the check skips offline, and skipping equally where a record carries no policy_uri, since that field is optional. verify_record in trace-spec already takes revocation: RevocationStore | None, a Container[str] | Callable[[str], bool]. A Callable[[str], bytes] is the same shape, and your policies/ directory is already the fixture side of it.
• TR-POL-003. tr_pol.py stops at 002 today, and check() takes only the record, so where a resolver reaches a module is the one real design question in this.
• The registration this repo gives a new code, which as far as I can see no test enforces, so it is easy to miss: docs/error-codes.md, docs/modules/tr-pol.md, docs/modules.md, and the per-level failure lists in docs/levels.md.
• Your seven boundaries, unchanged: 02 resolves and matches, 03 minimal mutation, 04 different object, 05 unreachable, 06 uncomputable algorithm, 07 bound to another referent.

On 05 and 06, the distinction you are deferring already exists here, one layer down. Status.UNVERIFIED is in result.py, held apart from SKIP so that, in its own comment, callers can never mistake an unverified record for a benign omission, and TR-SIG-005 emits it today. Whether TR-POL-003 can reuse it is a real question rather than a given: that same comment scopes it to cryptographic verification and makes it a failure at any level that requires signatures, and docs/levels.md runs TR-POL at every level, so an unreachable policy bundle inheriting it would fail a signed record for a reason that has nothing to do with its signature. Either way this does not wait on agentrust-io/trace-spec#190, because a finding status is not appraisal.status, and that issue is about the latter.

[imran-siddique]

The set itself is built the way I want vectors built here. Digests recomputable from the directory alone, a generator held to byte-reproduction by an in-repo test rather than one that needs another repository checked out, the must-accept pair so the set fails an unconditionally-rejecting verifier as well as an unconditionally-accepting one, and both shortfalls pinned in KNOWN_SHORTFALLS instead of left to be noticed later. Keeping candidate_binding in context rather than in record was right: appraisal is additionalProperties: false and proposing a field is an editorial decision, not something a fixture gets to make. The scoped .gitattributes is load-bearing and belongs where you put it; with core.autocrlf=true a clean clone would rewrite LF to CRLF and every digest in the set would break.

I am holding the merge, and it is because of where the vectors point rather than how they are made.

@lywinged is right, and I checked the claims rather than taking them. appraisal.policy_ref is a bare URI: the record says nothing about the referent, so there is nothing to compare a resolved document against, and there will not be until a normative field lands. That is why the set runs green on main with no verifier, and it is not a gap you or I can close from here.

One field over it is already closeable with nothing merged that is not merged already. policy.bundle_hash is required and is a digest of the bundle in force at execution time. policy_uri sits beside it, optional, format: uri, described in the schema as the URI to the policy bundle for verification. A record carrying both carries a referent and a digest of that referent, which is exactly the pair your seven boundaries are about. Nothing resolves it: in this repository policy_uri appears only in schemas/trace-claim.json and nowhere under src/, and tr_pol.py stops at TR-POL-002 where TR-POL-001 regex-matches the digest's shape and reports it valid without ever fetching anything.

So what I would like, and it is your seven boundaries essentially unchanged:

• 02 resolves and matches, 03 minimal mutation one byte apart, 04 a different object, 05 unreachable, 06 uncomputable digest algorithm, 07 bound to another referent, and 01 the no-binding-declared accept case, which becomes "no policy_uri present" and stays a must-accept because the field is optional.
• TR-POL-003, with a caller-supplied resolver defaulting to absent so the check skips offline and skips equally where a record carries no policy_uri. verify_record in trace-spec already takes revocation: RevocationStore | None as a Container[str] | Callable[[str], bool]; a Callable[[str], bytes] is the same shape, and your policies/ directory is already the fixture side of it.
• The registration a new code needs here: docs/error-codes.md, docs/modules/tr-pol.md, docs/modules.md, and the per-level failure lists in docs/levels.md. No test enforces that today, which is exactly why it is easy to miss.

The one real design question is the seam, and it is worth deciding before the code rather than after: check() currently takes only the record, so how a resolver reaches a module is a new shape for this repo. Propose one in the PR and I will settle it there.

On 05 and 06: @lywinged is also right that this does not have to wait on trace-spec#190. That issue is about appraisal.status, and a finding status is not the same thing. Status.UNVERIFIED already exists in result.py, deliberately held apart from SKIP so an unverified record cannot be read as a benign omission. Whether TR-POL-003 may reuse it is a genuine question rather than a given, because its own comment scopes it to cryptographic verification and docs/levels.md runs TR-POL at every level, so an unreachable policy bundle inheriting it would fail a signed record for a reason unrelated to its signature. Raise it in the PR and I will decide it.

None of this is wasted work. The boundaries, the generator, the mutation discipline and the digest recomputation all carry over. What changes is the field they are aimed at, from one that cannot be verified until a spec decision lands to one that can be verified today.

Thank you both, and @lywinged specifically for verifying the set by mutation before arguing with it.

[opento-suggestions]

We're in agreement, and thank you both for verifying before responding — @lywinged's mutation checks and the schema walk are exactly the review I was hoping for with this set. The layering note on agentrust-io/trace-spec#190 settles what waits where: everything below is finding-status territory, one layer down, so none of it waits on that issue. Both reserved questions follow; decisions yours.

The seam.

Proposing the smallest one that matches the precedent you both cited. The runner's entry point gains policy_resolver: Callable[[str], bytes] | None = None, and runner.py passes it explicitly to tr_pol.check(trace, policy_resolver=...) — which is not a new shape for the runner so much as a fourth instance of its established one: tr_env.check(trace, max_age_seconds=...), tr_sig.check(trace, record, fmt, level), and tr_rte.check(trace, level) all receive hardwired per-module extras today, one of them already keyword-form. What is genuinely new is the kind of extra — a callable that performs I/O rather than a configuration value — which is why it defaults to None and the module SKIPs without it. No signature inspection, no context object; if a second capability arrives it is a second explicit keyword, and a context dataclass is the consolidation move when there are three. The resolver's contract is deliberately thin: return the bundle bytes, or raise — any raise is "unreachable," and the distinction between failure modes stays out of the contract until something needs it. No resolver supplied → TR-POL-003 SKIPs, so offline verification stays first-class; no policy_uri on the record → SKIP equally, since the field is optional — which keeps 01 the must-accept exactly as restated. The resolver is caller-supplied and never derived from the record, for the same reason a bundle cannot assert its own acceptable staleness.

One thing turned up while grounding this against the schema.

policy_uri's format: "uri" executes nothing in this repository — no format checker is passed at any of the jsonschema.validate call sites — so a record carrying an unparseable policy_uri is schema-valid today. TR-POL-003 therefore checks well-formedness itself before invoking the resolver, and a malformed policy_uri fails rather than reading as unreachable: a reference the record got wrong and a referent the network lost are different defects, and only the second is weather. The set gains one reject vector for it.

---

The status question — both options, one recommendation.

Unreachable-with-a-resolver is "check attempted, could not complete, not a defect of the record," which is Status.UNVERIFIED's shape exactly; the blockers are its comment's cryptographic scoping and the blanket level >= 1 failure rule, with TR-POL running at every level.

Option A, recommended: generalize the contract. The comment widens from cryptographic verification to "the check could not be executed against evidence the record cites," and level-failure becomes per-code rather than blanket. Concretely that is a two-site code change, not a docs edit: the blanket rule is real code, in cli.py's defense-in-depth tally and its mirror in report.py, both counting unverified findings at level >= 1 with no awareness of which code emitted them. Per-code means those tallies consult the same per-level registration any new code lands in anyway — docs/levels.md's failure lists, backed by a table the code reads. TR-SIG-005 is unchanged in effect: signatures are required at level >= 1, so its UNVERIFIED keeps failing there. TR-POL-003's UNVERIFIED fails at no level today, because no level requires bundle resolution, and surfaces as advisory. Two merged precedents argue for this shape: receipt_unverified in trace-spec's verification doc — unverified, not invalid, surfaced with an advisory rather than a failure — and the depth rule that evidence which does not resolve downgrades honestly rather than failing records for the weather.

Option B: keep UNVERIFIED scoped to cryptographic verification and add a sibling member for evidence-execution failure, per-code registered the same way. The cost is a second held-apart-from-SKIP value in shared result.py; the benefit is TR-SIG's documented contract untouched.

I would take A. @imran-siddique — your call on both, as offered.

---

A vector boundary migrated into the schema, and that vector should transform. The old 06 — a declared digest algorithm the verifier cannot compute — cannot exist in a schema-valid record now that the digest under test is policy.bundle_hash: the admitted algorithms are sha256 and sha384, so the uncomputable case is rejected a layer earlier, and it belongs to that layer. Proposing 06 becomes the sha384 resolve-and-match accept case. Retargeted, the set otherwise carries only sha256 digests, and a set that never exercises sha384 would pass a verifier that hardcodes sha256 — the same shape as a corpus that stayed green because every record was ASCII. That, with the malformed-reference vector above, makes three accepts against four rejects and one unreachable, with the must-accept criterion met from both directions.

Registration. TR-POL-003 lands with all four rows: docs/error-codes.md, docs/modules/tr-pol.md, docs/modules.md, and the per-level failure lists in docs/levels.md. Since nothing enforces that today, happy to include a small test in this PR that fails when a code exists under src/ without its documentation rows — or to leave that for its own change if the scope here should stay tight.

Sequence, unless you'd rather otherwise: settle the seam and the status question here, then the retargeted set and TR-POL-003 push to this branch and the PR description updates to match. Let me know! 🦏