test(intent-bridge): the signature-tampering test sometimes does not tamper - #185
Merged
imran-siddique merged 1 commit intoAug 20, 2026
Merged
Conversation
…tart with test_tampering_is_rejected[signature] substitutes a fixed "A" for the first character of the signature. Whenever the signature already starts with "A" the substitution changes nothing, the bridge verifies, and the test fails having never tampered with anything. That is one signature in 64. Measured over 3000 runs of the test body: 45 runs in which the tampered bridge still verified before this change, 0 after. The failure is not a flake in the sense of an unreliable environment. It is a test that sometimes does not perform the mutation it is named for, so on the other 63 runs in 64 it passes without having established anything different from what it establishes now. Fixing it removes a red CI run whose cause is invisible from the failure message, which reads as a signature verification that should have failed. Substituting a character conditional on what is already there is the smallest change that always mutates. The authorization arm of the same parametrize is unaffected: it sets a field to a different value outright. Signed-off-by: lywinged <louie.lunz@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
|
❔ Contributor Check: UNKNOWN
Automated check by AgenTrust Contributor Check. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test_tampering_is_rejected[signature]substitutes a fixed"A"for the firstcharacter of the signature:
When the signature already starts with
"A", that substitution changes nothing. Thebridge verifies,
pytest.raises(IntentBridgeError)gets no exception, and the testfails having never tampered with anything.
That is one signature in 64, and it is how I found it: a single red run in an otherwise
green suite, with a failure message that reads as a signature verification which should
have failed.
Measurement
Calling the test body 3000 times per arm, counting runs in which the tampered bridge
still verified:
45/3000 is 1.5%, against the 1/64 = 1.56% the first base64url character predicts.
Harness
Why this is worth a commit rather than a rerun
The visible cost is a red CI run whose cause cannot be read off the failure message. The
part I would flag is the other 63 runs in 64. On those the test passes, and it passes
having done exactly what it does now, so nothing was lost on them. But the same shape in
a test whose mutation is silently absorbed rather than caught would pass every time
while pinning nothing, and there would be no red run to notice.
I looked for the pattern elsewhere in the repository and this is the only instance:
The change
Substitute a character conditional on what is already there, which is the smallest form
that always mutates. The
authorizationarm of the sameparametrizeis unaffected: itsets a field to a different value outright, so it cannot be a no-op.
Branched off
mainat697e20a.478 passed, 1 skipped;ruff check src tests scriptsclean.