Skip to content

test(intent-bridge): the signature-tampering test sometimes does not tamper - #185

Merged
imran-siddique merged 1 commit into
agentrust-io:mainfrom
lywinged:fix/flaky-signature-tamper
Aug 20, 2026
Merged

test(intent-bridge): the signature-tampering test sometimes does not tamper#185
imran-siddique merged 1 commit into
agentrust-io:mainfrom
lywinged:fix/flaky-signature-tamper

Conversation

@lywinged

Copy link
Copy Markdown
Collaborator

test_tampering_is_rejected[signature] substitutes a fixed "A" for the first
character of the signature:

tampered["signature"] = "A" + tampered["signature"][1:]

When the signature already starts with "A", that substitution changes nothing. The
bridge verifies, pytest.raises(IntentBridgeError) gets no exception, and the test
fails having never tampered with anything.

That is one signature in 64, and it is how I found it: a single red run in an otherwise
green suite, with a failure message that reads as a signature verification which should
have failed.

Measurement

Calling the test body 3000 times per arm, counting runs in which the tampered bridge
still verified:

before: 45/3000
after :  0/3000

45/3000 is 1.5%, against the 1/64 = 1.56% the first base64url character predicts.

Harness
import sys, copy; sys.path.insert(0, "tests")
from test_intent_bridge import _fixture, key_to_jwk, verify_bridge, IntentBridgeError

def body(tamper):
    bridge, key, declaration, intent, args, tool_call, transcript = _fixture()
    t = copy.deepcopy(bridge)
    t["signature"] = tamper(t["signature"])
    try:
        verify_bridge(t, {**key_to_jwk(key), "kid": "key-7"}, declaration=declaration,
                      pic_intent_digest=intent, pic_args_digest=args,
                      tool_call=tool_call, transcript=transcript, now=150)
    except IntentBridgeError:
        return True
    return False

before = lambda s: "A" + s[1:]
after = lambda s: ("B" if s[0] == "A" else "A") + s[1:]
for name, fn in (("before", before), ("after", after)):
    print(name, sum(0 if body(fn) else 1 for _ in range(3000)))

Why this is worth a commit rather than a rerun

The visible cost is a red CI run whose cause cannot be read off the failure message. The
part I would flag is the other 63 runs in 64. On those the test passes, and it passes
having done exactly what it does now, so nothing was lost on them. But the same shape in
a test whose mutation is silently absorbed rather than caught would pass every time
while pinning nothing, and there would be no red run to notice.

I looked for the pattern elsewhere in the repository and this is the only instance:

$ grep -rnE '= "[A-Za-z0-9_-]" \+' tests/ src/
tests/test_intent_bridge.py:65

The change

Substitute a character conditional on what is already there, which is the smallest form
that always mutates. The authorization arm of the same parametrize is unaffected: it
sets a field to a different value outright, so it cannot be a no-op.

Branched off main at 697e20a. 478 passed, 1 skipped; ruff check src tests scripts
clean.

…tart with

test_tampering_is_rejected[signature] substitutes a fixed "A" for the first
character of the signature. Whenever the signature already starts with "A" the
substitution changes nothing, the bridge verifies, and the test fails having
never tampered with anything.

That is one signature in 64. Measured over 3000 runs of the test body: 45 runs
in which the tampered bridge still verified before this change, 0 after.

The failure is not a flake in the sense of an unreliable environment. It is a
test that sometimes does not perform the mutation it is named for, so on the
other 63 runs in 64 it passes without having established anything different from
what it establishes now. Fixing it removes a red CI run whose cause is invisible
from the failure message, which reads as a signature verification that should
have failed.

Substituting a character conditional on what is already there is the smallest
change that always mutates. The authorization arm of the same parametrize is
unaffected: it sets a field to a different value outright.

Signed-off-by: lywinged <louie.lunz@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Contributor Check: UNKNOWN

Check Result
Profile UNKNOWN
Credential LOW
Overall UNKNOWN

Automated check by AgenTrust Contributor Check.

@github-actions github-actions Bot added the needs-review:UNKNOWN Contributor check flagged UNKNOWN risk label Aug 18, 2026
@imran-siddique
imran-siddique merged commit f51e1f7 into agentrust-io:main Aug 20, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-review:UNKNOWN Contributor check flagged UNKNOWN risk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants