Skip to content
3 changes: 2 additions & 1 deletion docs/error-codes.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,9 @@ All TRACE test failures emit a structured error code of the form `TR-<MODULE>-<N

| Code | Description | How to fix |
|------|-------------|------------|
| TR-POL-001 | `policy.bundle_hash` is not a valid `sha256:` digest | Compute `sha256:` + hex digest of your Cedar policy bundle bytes |
| TR-POL-001 | `policy.bundle_hash` is not a valid `sha256:` or `sha384:` digest | Compute `sha256:` + 64 hex chars, or `sha384:` + 96 hex chars, over your policy bundle bytes. Both are accepted by the schema and by the module |
| TR-POL-002 | `policy.enforcement_mode` is not `enforce`, `advisory`, `silent`, or `declared` | Replace `"strict"` or `"monitor"` with one of the four accepted values; `"declared"` is the honest value for a producer that binds a policy without evaluating it |
| TR-POL-003 | `policy.policy_uri` is not an absolute URI, or the bundle it resolves to does not have the digest `policy.bundle_hash` declares. Unverified when a resolver was supplied and the bundle could not be read; skipped when no `policy_uri` is present or no resolver was supplied | Point `policy_uri` at the bundle whose bytes hash to `bundle_hash`. A record that cites a bundle it cannot be checked against is reported as unverified rather than passed |

## TR-TXN — Transcript

Expand Down
25 changes: 25 additions & 0 deletions docs/levels.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ Level 0 records are signed with a software key. The `runtime.platform` must be `
- `policy.enforcement_mode` is `"strict"` or `"monitor"` — TR-POL-002
- `cnf.jwk` missing, of an unsupported key type, or carrying private key material (`d`) — TR-SIG-004
- Signature does not verify against `cnf.jwk` — TR-SIG-005
- `policy.policy_uri` is not an absolute URI, or the bundle it resolves to does not have the digest `policy.bundle_hash` declares — TR-POL-003. The malformed case is reported with or without a resolver: a reference the record got wrong needs no network to detect

---

Expand Down Expand Up @@ -145,6 +146,30 @@ Level 2 adds tool transcript and transparency anchor requirements. The `transpar
- `tool_transcript.call_count` negative or not an integer — TR-TXN-002
- `transparency` is absent or empty, is not a string, or is not an `https://` URI with a host — TR-ANC-001
- no anchor receipt was supplied, the receipt is malformed, or its inclusion proof does not reproduce the committed `merkle_root` — TR-ANC-002. A record cannot reach Level 2 on a URI alone: pass the receipt with `--receipt`
- `policy.policy_uri` is present, a resolver was supplied with `--policy-dir`, and the bundle could not be read — TR-POL-003. Unverified rather than failed, and unverified fails the run from Level 2

---

## Unverified findings

An unverified finding means **the check could not be executed against the
evidence the record cites**. It is held apart from a skip so that a consumer can
never read it as a benign omission, and it is not a pass.

Whether it fails the run is per code rather than one rule over all of them.
Different checks lose their evidence for different reasons, and the level at
which that stops being tolerable is a property of the check.

| Code | UNVERIFIED fails the run from level |
|------|-------------------------------------|
| TR-SIG-005 | 1 |
| TR-POL-003 | 2 |

A code absent from this table fails from Level 1. That default is deliberate: a
new code nobody registered still fails closed, and the registration guard turns
red rather than the run turning quietly permissive. The table is
`UNVERIFIED_FAILS_FROM_LEVEL` in `src/trace_tests/modules/unverified.py`, and
`tests/test_docs_match_the_modules.py` fails if the two disagree.

---

Expand Down
2 changes: 1 addition & 1 deletion docs/modules.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ The TRACE conformance suite is divided into seven modules. Each module maps to a
| [Envelope](modules/tr-env.md) | TR-ENV | §3.2 | `eat_profile` URI, `iat` validity, `subject` form, presence of `cnf.jwk.kty` |
| [Signature](modules/tr-sig.md) | TR-SIG | §3.2.1 | Private key leak detection, key type support, and the Ed25519 signature verification outcome |
| [Runtime](modules/tr-rte.md) | TR-RTE | §3.1 | TEE platform enum, measurement format, RIM URI scheme |
| [Policy](modules/tr-pol.md) | TR-POL | §3.1 | Policy bundle hash format, enforcement mode values |
| [Policy](modules/tr-pol.md) | TR-POL | §3.1 | Policy bundle hash format, enforcement mode values, and whether the bundle at `policy_uri` has the declared digest |
| [Transcript](modules/tr-txn.md) | TR-TXN | §3.1 | Tool-call transcript hash binding |
| [Transparency](modules/tr-anc.md) | TR-ANC | §3.2 | SCITT receipt URI form (TR-ANC-001), and offline replay of the inclusion proof against the committed Merkle root when a receipt is supplied (TR-ANC-002). The URI itself is never resolved |
| [Provenance](modules/tr-sca.md) | TR-SCA | §3.1 | SLSA provenance level and digest format |
24 changes: 23 additions & 1 deletion docs/modules/tr-pol.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,27 @@ Tests Cedar policy bundle binding.

| Test ID | Description | Positive Case | Negative Case |
|---------|-------------|---------------|---------------|
| TR-POL-001 | `policy.bundle_hash` is a valid `sha256:` digest | `sha256:` followed by 64 hex chars | missing, wrong prefix, wrong length |
| TR-POL-001 | `policy.bundle_hash` is a valid `sha256:` or `sha384:` digest | `sha256:` + 64 hex chars, `sha384:` + 96 hex chars | missing, wrong prefix, wrong length |
| TR-POL-002 | `policy.enforcement_mode` is `enforce`, `advisory`, `silent`, or `declared` | `enforce` | `strict`, `monitor`, absent |
| TR-POL-003 | `policy.policy_uri` is an absolute URI, and the bundle it resolves to has the digest `policy.bundle_hash` declares | absent `policy_uri`; or a resolved bundle whose digest matches | relative reference, whitespace in the URI, resolved bundle with a different digest |

## Resolving the bundle

TR-POL-003 needs somewhere to fetch the bundle from. The resolver is supplied
by the caller and never derived from the record — a record that named its own
resolver could name one that agrees with it. From the CLI that is `--policy-dir
DIR`, where `DIR/resolutions.json` maps each `policy_uri` to a relative path
inside `DIR`:

```
trace-tests verify --record record.json --policy-dir ./bundles
```

Without it the resolution part of the check skips, so offline verification
stays a first-class use rather than a degraded one.

**A malformed `policy_uri` is reported with or without a resolver.** A
reference the record got wrong is a defect in the record, visible with no
network at all, exactly like the digest shape TR-POL-001 tests. A referent that
could not be fetched is not: that is reported as unverified, and only when a
resolver was supplied and failed.
22 changes: 22 additions & 0 deletions docs/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,28 @@ trace-tests verify --record sample-record.json --level 2 --expected-nonce "$VERI

The sample fixture passes Level 0. Levels 1 and 2 will fail on runtime attestation and transparency fields — that is expected. See [Trust Levels](levels.md) for what each level requires.

## Resolving the policy bundle

If your record carries `policy.policy_uri`, TR-POL-003 can fetch the bundle and
check that it has the digest `policy.bundle_hash` declares. Point `--policy-dir`
at a directory holding a `resolutions.json` that maps each URI to a relative
path inside it:

```bash
trace-tests verify --record sample-record.json --level 0 --policy-dir ./bundles
```

```text
{
"https://policy.example.org/bundles/agent-v1.json": "agent-v1.json"
}
```

Without `--policy-dir` the resolution part of the check skips, so verifying
offline costs you nothing. A `policy_uri` that is malformed rather than
unreachable is still reported either way: that is a defect in the record, not a
fetch that failed.

## Exit codes

| Code | Meaning |
Expand Down
119 changes: 112 additions & 7 deletions src/trace_tests/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,9 @@
import json
import importlib.metadata
import pathlib
import re
import sys
from collections.abc import Callable
from typing import Any

import click
Expand All @@ -15,6 +17,7 @@
from trace_tests import report as report_mod
from trace_tests.loader import LoadError, load_record
from trace_tests.modules.tr_env import DEFAULT_MAX_AGE_SECONDS
from trace_tests.modules.unverified import unverified_fails
from trace_tests.result import Status
from trace_tests.runner import run

Expand Down Expand Up @@ -47,6 +50,7 @@ def _print_report(path: str, fmt: str, level: int, results: dict[str, list[Any]]
skips = 0
passes = 0
unverified = 0
unverified_failing = 0

for module, findings in results.items():
for f in findings:
Expand All @@ -58,21 +62,25 @@ def _print_report(path: str, fmt: str, level: int, results: dict[str, list[Any]]
passes += 1
elif f.unverified():
unverified += 1
if unverified_fails(f.code, level):
unverified_failing += 1
else:
skips += 1

# Defense in depth: unverified findings must fail the run at any level that
# requires signatures, even if a module forgot to emit a hard FAIL.
if unverified and level >= 1:
failures += unverified
# Defense in depth: an unverified finding must fail the run from the level
# its code is registered at, even if a module forgot to emit a hard FAIL.
# The level is per-code rather than blanket; an unregistered code fails
# from level 1, which is what the blanket rule did for all of them.
failures += unverified_failing

total = passes + failures + skips + (unverified if level == 0 else 0)
total = passes + failures + skips + (unverified - unverified_failing)
click.echo("")
if failures == 0:
if unverified:
click.echo(
f"Result: PASS ({total} checks, {skips} skipped, {unverified} UNVERIFIED "
f"-- record is NOT cryptographically verified)"
f"-- {unverified} check(s) could not be executed against the evidence "
f"this record cites)"
)
else:
click.echo(f"Result: PASS ({total} checks, {skips} skipped)")
Expand Down Expand Up @@ -104,6 +112,71 @@ def _load_receipt(path: str | None) -> dict | None:
sys.exit(2)
return data


def _load_policy_resolver(policy_dir: str | None) -> Callable[[str], bytes] | None:
"""Build a policy-bundle resolver from DIR/resolutions.json, or None when not supplied.

The manifest is checked for *form* only: an object, string to string,
relative paths, no parent traversal. Whether a mapped file is actually
there is deliberately not checked here. Existence is a resolve-time fact,
and a manifest that refused to load because one bundle had gone missing
would be the manifest-level version of treating a lost referent as a wrong
reference, which is the confusion TR-POL-003 exists to keep apart. A
missing file surfaces as an unverified finding for the record that cites
it, and leaves every other record in the run readable.
"""
if policy_dir is None:
return None
root = pathlib.Path(policy_dir)
manifest_path = root / "resolutions.json"
try:
with open(manifest_path, encoding="utf-8") as fh:
data = json.load(fh)
except OSError as exc:
click.echo(f"Error: cannot read policy manifest {manifest_path}: {exc}", err=True)
sys.exit(2)
except json.JSONDecodeError as exc:
click.echo(f"Error: policy manifest {manifest_path} is not valid JSON: {exc}", err=True)
sys.exit(2)
if not isinstance(data, dict):
click.echo(
f"Error: policy manifest {manifest_path} must be a JSON object, "
f"got {type(data).__name__}",
err=True,
)
sys.exit(2)
for uri, rel in data.items():
if not isinstance(rel, str):
click.echo(
f"Error: policy manifest {manifest_path} maps {uri!r} to "
f"{type(rel).__name__}, expected a relative path string",
err=True,
)
sys.exit(2)
if _is_unsafe_relative(rel):
click.echo(
f"Error: policy manifest {manifest_path} maps {uri!r} to {rel!r}; "
"entries must be relative paths inside the directory, with no "
"parent traversal",
err=True,
)
sys.exit(2)

def _resolve(uri: str) -> bytes:
# A URI the manifest does not hold raises, exactly as a fetch would;
# so does a mapped file that is not there. Both reach TR-POL-003 as
# unverified, and the message carries which happened.
return (root / data[uri]).read_bytes()

return _resolve


def _is_unsafe_relative(rel: str) -> bool:
"""True when *rel* escapes the manifest's own directory, or tries to."""
if not rel or rel.startswith(("/", "\\")) or re.match(r"^[A-Za-z]:", rel):
return True
return ".." in pathlib.PurePosixPath(rel.replace("\\", "/")).parts

@click.group()
@click.version_option(__version__)
def main() -> None:
Expand Down Expand Up @@ -142,7 +215,25 @@ def main() -> None:
"says where the anchor lives, the receipt is what proves the record is in it."
),
)
def verify(record: str, level: int, max_age: int, expected_nonce: str | None, receipt: str | None) -> None:
@click.option(
"--policy-dir",
"policy_dir",
default=None,
type=click.Path(),
help=(
"Directory holding resolutions.json, a map from policy_uri to a relative "
"path inside it. Supplying it lets TR-POL-003 resolve policy.policy_uri and "
"compare the bundle against policy.bundle_hash; without it that check skips."
),
)
def verify(
record: str,
level: int,
max_age: int,
expected_nonce: str | None,
receipt: str | None,
policy_dir: str | None,
) -> None:
"""Verify a TRACE trust record against the conformance suite."""
try:
data, fmt = load_record(record)
Expand All @@ -151,6 +242,7 @@ def verify(record: str, level: int, max_age: int, expected_nonce: str | None, re
sys.exit(2)

receipt_data = _load_receipt(receipt)
policy_resolver = _load_policy_resolver(policy_dir)

results = run(
data,
Expand All @@ -159,6 +251,7 @@ def verify(record: str, level: int, max_age: int, expected_nonce: str | None, re
max_age_seconds=max_age,
expected_nonce=expected_nonce,
receipt=receipt_data,
policy_resolver=policy_resolver,
)
exit_code = _print_report(record, fmt, level, results)
sys.exit(exit_code)
Expand Down Expand Up @@ -204,6 +297,15 @@ def verify(record: str, level: int, max_age: int, expected_nonce: str | None, re
type=click.Path(),
help="Path to the anchor receipt (JSON). Required for TR-ANC-002 at Level 2.",
)
@click.option(
"--policy-dir",
"policy_dir",
default=None,
type=click.Path(),
help="Directory holding resolutions.json, a map from policy_uri to a relative "
"path inside it. Required for TR-POL-003 to resolve the bundle; without it "
"that check skips.",
)
def report(
record: str,
max_level: int,
Expand All @@ -214,6 +316,7 @@ def report(
fail_under: int | None,
expected_nonce: str | None,
receipt: str | None,
policy_dir: str | None,
) -> None:
"""Produce a conformance report you can hand to someone else.

Expand All @@ -228,6 +331,7 @@ def report(
sys.exit(2)

receipt_data = _load_receipt(receipt)
policy_resolver = _load_policy_resolver(policy_dir)

results_by_level = {
level: run(
Expand All @@ -237,6 +341,7 @@ def report(
max_age_seconds=max_age,
expected_nonce=expected_nonce,
receipt=receipt_data,
policy_resolver=policy_resolver,
)
for level in range(max_level + 1)
}
Expand Down
Loading
Loading