Skip to content

feat: add cancellation and progress notification conformance scenarios - #450

Open
elang2 wants to merge 2 commits into
modelcontextprotocol:mainfrom
elang2:feat/cancellation-progress-scenarios
Open

feat: add cancellation and progress notification conformance scenarios#450
elang2 wants to merge 2 commits into
modelcontextprotocol:mainfrom
elang2:feat/cancellation-progress-scenarios

Conversation

@elang2

@elang2 elang2 commented Aug 9, 2026

Copy link
Copy Markdown

Summary

Adds two new server conformance scenarios covering utilities that previously had no test coverage:

Cancellation (notifications/cancelled) — 3 checks

  • Server stability after cancellation of unknown request IDs
  • Cancellation of in-progress requests (timing-based evidence)
  • Rapid burst cancellation tolerance (SHOULD-level)

Progress notifications (notifications/progress) — 4 checks

  • Progress value MUST increase with every notification (strict monotonicity)
  • Token matching between request _meta and notifications
  • No spurious notifications when progressToken is absent
  • Cessation of notifications after request completion

Also adds test_tool_slow and test_tool_fast to the everything-server for cancellation testing, and fixes test_tool_with_progress to not send notifications when no progressToken is provided.

Both scenarios are registered in pendingClientScenariosList until the everything-server tools are merged upstream.

Closes #433, closes #434

Add two new server conformance test scenarios covering utilities that
previously had no test coverage:

Cancellation (notifications/cancelled):
- Server stability after cancellation of unknown request IDs
- Cancellation of in-progress requests without server degradation
- Rapid burst cancellation tolerance

Progress notifications (notifications/progress):
- Non-decreasing progress values with consistent totals
- Token matching between request _meta and notifications
- No spurious notifications when progressToken is absent
- Cessation of notifications after request completion

Everything-server additions:
- test_tool_slow: configurable sleep duration for cancellation testing
- test_tool_fast: immediate response for health-check verification
- Fixed test_tool_with_progress to not send notifications when no
  progressToken is provided (was using ?? 0 fallback)

Both scenarios are registered in pendingClientScenariosList until the
everything-server tools are merged upstream.

Tested against everything-server (spec 2025-06-18):
  cancellation: 4/4 checks passed (24 wire messages validated)
  progress-notifications: 5/5 checks passed (29 wire messages validated)

Closes modelcontextprotocol#433, closes modelcontextprotocol#434
@elang2

elang2 commented Aug 9, 2026

Copy link
Copy Markdown
Author

@joy7758 if you have time, would love your thoughts on this one too. This is the cancellation/progress work I mentioned in #435.

@joy7758

joy7758 commented Aug 10, 2026

Copy link
Copy Markdown

Thanks for sharing the replacement PR. I reviewed current head 35452176e5b144a93c2f57a2f637fa0eb6c3e6b2. The token/no-token and post-completion progress checks are promising, but I found several spec-backing and observability gaps before the results can support conformance claims:

  1. The in-progress cancellation check can succeed without proving that the target request was cancelled. cancellation.ts:L147-L202 assumes the tool-call request ID is 1. If that assumption is wrong, the server may legitimately ignore the notification as an unknown request. The result still becomes SUCCESS whenever the later health check responds; cancelledEarly and timedOut are recorded but never asserted. The scenario should capture the actual request ID and distinguish cancellation evidence from post-notification server health.

  2. Unknown-ID and burst behavior is being given stronger normative language than the spec. cancellation.ts:L1-L7 states a rapid-burst MUST, while the cancellation specification says cancellation senders MUST reference previously issued, believed-active requests; receivers MAY ignore unknown IDs, and invalid cancellations SHOULD be ignored. The nonexistent-ID probes can be useful robustness tests, but the burst case should not be presented as an independent protocol MUST or scored as one without a normative citation.

  3. The progress comparison is weaker than the specification. progress-notifications.ts:L128-L135 fails only when the value decreases. The 2025-06-18 progress specification says the value MUST increase with every notification, so equal consecutive values currently pass incorrectly.

  4. The total warnings are not backed by the cited page. progress-notifications.ts:L137-L165 warns when total decreases or progress exceeds total. I cannot find either requirement in the cited 2025-06-18 text. Those observations can remain in details, but should not change conformance status without a normative source.

  5. The reported evidence count needs reconciliation. The PR body reports cancellation 4/4 and progress 5/5; the current source emits three cancellation check IDs and four progress check IDs. Please attach the exact command/output for the current head, along with the contribution guide's specific failing case and a pinned real-SDK run, so the evidence matches the reviewed revision.

This is a request to narrow claims and strengthen the test oracle, not a protocol-change proposal.

- Cancellation: use timing evidence to distinguish actual cancellation from
  server ignoring unknown ID; INFO when healthy but not cancelled early
- Cancellation: soften burst check to WARNING (spec says MAY ignore)
- Cancellation: align docblock and description with spec MAY/SHOULD language
- Progress: change monotonicity check from < to <= (spec says MUST increase)
- Progress: move total-decrease and exceeds-total to observations only
  (no normative backing in current spec text)
@elang2

elang2 commented Aug 14, 2026

Copy link
Copy Markdown
Author

Thanks for the detailed review. Pushed a fix addressing all five points.

For (1), the SDK doesn't expose request IDs so I kept the assumption but now use timing as the actual assertion. SUCCESS requires the request to terminate early, otherwise it reports INFO acknowledging the server MAY ignore.

For (2), softened to match spec language. Burst stability is now WARNING (SHOULD-level), not FAILURE.

For (3), fixed. Comparison is now <= so equal values correctly fail.

For (4), total-decrease and exceeds-total moved to details.observations only. No longer affects conformance status.

For (5), PR body updated to 3 cancellation / 4 progress. Will attach a pinned run once the everything-server tools land.

@elang2

elang2 commented Aug 17, 2026

Copy link
Copy Markdown
Author

All five points from @joy7758 addressed and pushed. Burst is WARNING not FAILURE, progress uses <=, total warnings are non-scoring, check counts updated.

Maintainer review welcome when someone has time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add conformance scenario: progress notification edge cases Add conformance scenario: notifications/cancelled

2 participants