Skip to content

test(conformance): adequacy criteria for a vector set, applied to every set here - #186

Merged
imran-siddique merged 1 commit into
agentrust-io:mainfrom
lywinged:test/vector-set-adequacy
Aug 20, 2026
Merged

test(conformance): adequacy criteria for a vector set, applied to every set here#186
imran-siddique merged 1 commit into
agentrust-io:mainfrom
lywinged:test/vector-set-adequacy

Conversation

@lywinged

Copy link
Copy Markdown
Collaborator

A conformance vector set is a claim: a verifier that does not implement these rules
will fail this set.
Nothing in the repository checked that claim, and it is not implied
by the vectors passing.

#169 and #170 both came out of asking it. Both were sets that were green.

Neither was found by reading a vector. Both were found by asking what a set fails to
distinguish.

The criteria

Four, each from a defect on a real set rather than from first principles:

Criterion The defect behind it
A set must fail both unconditional implementations a set of all-rejections is passed by a verifier that rejects everything
Every boundary needs more than one vector one vector cannot separate a check that reads the head of a list from one that reads all of it
Every set on disk is measured, or named with the test that measures it a hand-maintained list of what gets graded silently stops being complete
Shortfalls are recorded exactly a gap that is skipped can widen; a gap asserted to its exact extent cannot

What it says about this repository today

Applied to every set, by the same loader, with the results recorded where they fall.

build-provenance-depth      margin at every boundary, nothing recorded
canonicalization-boundary   one-directional: every vector expects acceptance
action-receipts             covered by tests/test_vector_completeness.py

canonicalization-boundary is mine. Every vector in it expects acceptance, so the
set cannot tell a conformant verifier from one that accepts unconditionally. That second
implementation is a real failure mode, not a hypothetical, so it is a gap and not a
design. It is recorded in KNOWN_ONE_DIRECTIONAL with the record asserted, so it cannot
widen quietly, and the entry is deleted when the set gains a record signed over a non-JCS
form that a conformant verifier must reject.

I mention this first because a standard that only ever measures other people's work is
advocacy. The only shortfall this currently records is on a set I wrote.

The guard on the instrument itself

SETS is a hand-maintained list of what gets graded, which is the third criterion's
defect, sitting in the one place it would otherwise be invisible: a set added later would
simply not be graded, and nothing would fail to say so.

test_every_vector_set_on_disk_is_measured_somewhere compares SETS and
MEASURED_ELSEWHERE against what is actually in examples/, in both directions, since a
stale entry hides a renamed set as effectively as a missing one.

Rather than assert these fail for the right reason, I broke each and recorded which test
went red:

Mutation Test that caught it
an unlisted set directory appears test_every_vector_set_on_disk_is_measured_somewhere
an entry names a set that is not on disk the same
the directory scan stops finding anything the same
MEASURED_ELSEWHERE names a test file that does not exist test_the_test_named_as_measuring_a_set_elsewhere_exists

What this costs, said before it happens rather than after

A future PR that adds a thin vector set will go red here, and the failure will name
the boundary and the vector rather than a line number. That is the intended behaviour and
it is a policy consequence of merging a test, so it should be a decision rather than a
surprise. The escape hatch is deliberate and visible: record the shortfall in
KNOWN_THIN with its exact extent, which documents the gap instead of hiding it, and the
entry is deleted when someone writes the second vector.

Scope

Tests and an informative document. No normative text, no RFC 2119 keyword, no schema
change, so no sponsor is needed under GOVERNANCE.md. Same class as #169, #171 and #175.

485 passed, 1 skipped; ruff check src tests scripts clean; mypy clean. Branched off
main at 697e20a.

The criteria are also usable outside this repository, since they read expected and
nothing else. If they are more useful in trace-tests than here, say so and I will move
them.

…ry set here

A conformance vector set is a claim that a non-implementing verifier will fail it.
Nothing checked that claim. agentrust-io#169 and agentrust-io#170 were both found by asking it of a set
rather than of a vector, and both were sets that were passing.

Four criteria, each from a defect on a real set rather than from first principles:

- a set must fail both unconditional implementations, accept-everything and
  reject-everything, or it pins nothing
- each boundary needs more than one vector, since a single vector cannot
  distinguish a check that reads the head of a list from one that reads all of it
- every set on disk is measured here or named with the test that measures it
- shortfalls are recorded exactly, so they cannot widen unnoticed and the entry is
  deleted when someone closes the gap

Applied to every set in this repository. `build-provenance-depth` carries a margin
at every boundary. `canonicalization-boundary`, which I wrote, expects acceptance in
every vector and so cannot tell a conformant verifier from one that accepts
unconditionally; that is recorded rather than skipped, and the record is asserted so
it cannot grow. `action-receipts` is named as covered by test_vector_completeness.py
rather than graded twice.

The completeness guard is on the instrument itself for a reason. SETS is a
hand-maintained list of what gets graded, which is the defect these criteria exist
to catch, and the one place it would otherwise be invisible: a set added later would
simply not be graded and nothing would fail. Adding an unlisted set directory turns
the guard red, as does a stale entry, as does naming a test that does not exist.
Each was checked by making the change and watching the specific test fail.

Signed-off-by: lywinged <louie.lunz@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Contributor Check: UNKNOWN

Check Result
Profile UNKNOWN
Credential LOW
Overall UNKNOWN

Automated check by AgenTrust Contributor Check.

@github-actions github-actions Bot added the needs-review:UNKNOWN Contributor check flagged UNKNOWN risk label Aug 18, 2026
@imran-siddique
imran-siddique merged commit 1f76106 into agentrust-io:main Aug 20, 2026
6 checks passed
lywinged added a commit to lywinged/trace-spec that referenced this pull request Aug 21, 2026
… merged

agentrust-io#186 added criteria that every vector set on disk is measured against, and
test_every_vector_set_on_disk_is_measured_somewhere fails for a set in neither
SETS nor MEASURED_ELSEWHERE. This branch was opened three days before those
criteria landed, so merging upstream leaves `delegation-link` as the one set
nothing grades, and it is the only failure in the merged tree.

Registered in SETS rather than named in MEASURED_ELSEWHERE, because the set
holds up when it is actually graded rather than only pointed at:

    delegation-link: 23 vectors, 3 accepting, 10 distinct failure codes

No shortfall on either criterion decidable from the fixtures. It is not
satisfiable by an implementation that answers "accept" to everything or one
that answers "reject" to everything, and every one of the ten failure codes is
carried by exactly two vectors, which is the margin agentrust-io#124 asks for.

Boundaries are counted by failure code, the default. adequacy.py says that
assumption is the set's to justify: here the codes are the unit, because
tests/delegation_margins.json records the per-code margin and
tests/test_delegation_completeness.py holds each rule to being load-bearing for
both of its vectors, deleting the rule from the registry rather than matching
source text. The criteria adequacy.py leaves to each set, a rule nothing pins
and a weakness shared across a boundary's vectors, are implemented there too,
by rebuilding the registry without an entry and by substituting shortcut checks
that read only the first link or the first hop.

605 passed, 1 skipped. Verified by removing the SETS entry again, which fails
test_every_vector_set_on_disk_is_measured_somewhere on its own.

Signed-off-by: Louielunz <48041247+lywinged@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-review:UNKNOWN Contributor check flagged UNKNOWN risk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants