test(conformance): adequacy criteria for a vector set, applied to every set here - #186
Merged
imran-siddique merged 1 commit intoAug 20, 2026
Merged
Conversation
…ry set here A conformance vector set is a claim that a non-implementing verifier will fail it. Nothing checked that claim. agentrust-io#169 and agentrust-io#170 were both found by asking it of a set rather than of a vector, and both were sets that were passing. Four criteria, each from a defect on a real set rather than from first principles: - a set must fail both unconditional implementations, accept-everything and reject-everything, or it pins nothing - each boundary needs more than one vector, since a single vector cannot distinguish a check that reads the head of a list from one that reads all of it - every set on disk is measured here or named with the test that measures it - shortfalls are recorded exactly, so they cannot widen unnoticed and the entry is deleted when someone closes the gap Applied to every set in this repository. `build-provenance-depth` carries a margin at every boundary. `canonicalization-boundary`, which I wrote, expects acceptance in every vector and so cannot tell a conformant verifier from one that accepts unconditionally; that is recorded rather than skipped, and the record is asserted so it cannot grow. `action-receipts` is named as covered by test_vector_completeness.py rather than graded twice. The completeness guard is on the instrument itself for a reason. SETS is a hand-maintained list of what gets graded, which is the defect these criteria exist to catch, and the one place it would otherwise be invisible: a set added later would simply not be graded and nothing would fail. Adding an unlisted set directory turns the guard red, as does a stale entry, as does naming a test that does not exist. Each was checked by making the change and watching the specific test fail. Signed-off-by: lywinged <louie.lunz@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
|
❔ Contributor Check: UNKNOWN
Automated check by AgenTrust Contributor Check. |
lywinged
added a commit
to lywinged/trace-spec
that referenced
this pull request
Aug 21, 2026
… merged agentrust-io#186 added criteria that every vector set on disk is measured against, and test_every_vector_set_on_disk_is_measured_somewhere fails for a set in neither SETS nor MEASURED_ELSEWHERE. This branch was opened three days before those criteria landed, so merging upstream leaves `delegation-link` as the one set nothing grades, and it is the only failure in the merged tree. Registered in SETS rather than named in MEASURED_ELSEWHERE, because the set holds up when it is actually graded rather than only pointed at: delegation-link: 23 vectors, 3 accepting, 10 distinct failure codes No shortfall on either criterion decidable from the fixtures. It is not satisfiable by an implementation that answers "accept" to everything or one that answers "reject" to everything, and every one of the ten failure codes is carried by exactly two vectors, which is the margin agentrust-io#124 asks for. Boundaries are counted by failure code, the default. adequacy.py says that assumption is the set's to justify: here the codes are the unit, because tests/delegation_margins.json records the per-code margin and tests/test_delegation_completeness.py holds each rule to being load-bearing for both of its vectors, deleting the rule from the registry rather than matching source text. The criteria adequacy.py leaves to each set, a rule nothing pins and a weakness shared across a boundary's vectors, are implemented there too, by rebuilding the registry without an entry and by substituting shortcut checks that read only the first link or the first hop. 605 passed, 1 skipped. Verified by removing the SETS entry again, which fails test_every_vector_set_on_disk_is_measured_somewhere on its own. Signed-off-by: Louielunz <48041247+lywinged@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A conformance vector set is a claim: a verifier that does not implement these rules
will fail this set. Nothing in the repository checked that claim, and it is not implied
by the vectors passing.
#169 and #170 both came out of asking it. Both were sets that were green.
build-provenance-depthreported the same verdict at every depth, so averifier that resolved one link and assumed the rest matched passed it.
canonicalization-boundaryseparated its closest form by a single vector,and that vector put the divergence at the top level, where the obvious implementation
already sorts.
Neither was found by reading a vector. Both were found by asking what a set fails to
distinguish.
The criteria
Four, each from a defect on a real set rather than from first principles:
What it says about this repository today
Applied to every set, by the same loader, with the results recorded where they fall.
canonicalization-boundaryis mine. Every vector in it expects acceptance, so theset cannot tell a conformant verifier from one that accepts unconditionally. That second
implementation is a real failure mode, not a hypothetical, so it is a gap and not a
design. It is recorded in
KNOWN_ONE_DIRECTIONALwith the record asserted, so it cannotwiden quietly, and the entry is deleted when the set gains a record signed over a non-JCS
form that a conformant verifier must reject.
I mention this first because a standard that only ever measures other people's work is
advocacy. The only shortfall this currently records is on a set I wrote.
The guard on the instrument itself
SETSis a hand-maintained list of what gets graded, which is the third criterion'sdefect, sitting in the one place it would otherwise be invisible: a set added later would
simply not be graded, and nothing would fail to say so.
test_every_vector_set_on_disk_is_measured_somewherecomparesSETSandMEASURED_ELSEWHEREagainst what is actually inexamples/, in both directions, since astale entry hides a renamed set as effectively as a missing one.
Rather than assert these fail for the right reason, I broke each and recorded which test
went red:
test_every_vector_set_on_disk_is_measured_somewhereMEASURED_ELSEWHEREnames a test file that does not existtest_the_test_named_as_measuring_a_set_elsewhere_existsWhat this costs, said before it happens rather than after
A future PR that adds a thin vector set will go red here, and the failure will name
the boundary and the vector rather than a line number. That is the intended behaviour and
it is a policy consequence of merging a test, so it should be a decision rather than a
surprise. The escape hatch is deliberate and visible: record the shortfall in
KNOWN_THINwith its exact extent, which documents the gap instead of hiding it, and theentry is deleted when someone writes the second vector.
Scope
Tests and an informative document. No normative text, no RFC 2119 keyword, no schema
change, so no sponsor is needed under
GOVERNANCE.md. Same class as #169, #171 and #175.485 passed, 1 skipped;ruff check src tests scriptsclean;mypyclean. Branched offmainat697e20a.The criteria are also usable outside this repository, since they read
expectedandnothing else. If they are more useful in trace-tests than here, say so and I will move
them.