agent: MCP Events follow-ups — panic exit, delivery by tag, permanent refusals, runtime restore

[jaredLunde]

What

Follow-ups to the merged MCP Events work (#133), rebased on main after #134, #135, #138, #139, #140, #141, #142 and #143. Every known gap is fixed, including item 5's former known limit (now item 7) and every re-audit finding (N1–N7). Each fix has a test that fails before it, either on main or with the fix reverted in place. The exceptions are item 6, which only adds tests (see its row), and N7's parity tests, which pin behaviour rather than fix it.

Original items

#	gap	fix	proving test	fails before
1	A panic inside serve hung in runtime drop, on the parked stdin reader, before ExitSweep ran.	main catches the unwind, abandons the runtime's threads, sweeps the stdio servers and exits 101.	mcp_stdio_transport::a_panic_inside_serve_exits_promptly_and_still_sweeps	yes: hung 15 s, orphan left
1b	A panic in one daemon session	Contained: that session's clients get an error frame, it ends, and the daemon keeps serving the others.	mcp_stdio_transport::a_panic_in_a_daemon_session_ends_that_session_only	yes (no error frame)
1c	…and if that session was the MCP Events session, configured subscriptions stopped until restart	The events session restarts automatically, with backoff (1 s doubling to 60 s, reset after 10 min up).	mcp_events_routing::the_events_session_is_restarted_after_a_panic	yes
2	A mid-run compaction could summarize a steered batch away, and the transcript-text match then re-injected it.	Delivery is recorded by tag at the moment the model receives the batch (SteeringMessage::tag, reported in AgentEvent::Steered after the checkpoint), not re-derived from the transcript.	mcp_events_durability::a_steered_batch_summarized_away_by_a_mid_run_compaction_is_not_injected_again, which also asserts an empty pending log and durable done records; agent-core unit a_steered_message_reports_its_tag_when_it_reaches_the_model	yes
3	A permanently refused configured subscription retried every 60 s forever and kept its session alive.	Permanent refusals are classified. They are reported as refused (as a frame, and in mcp_events_list.unestablished), retried hourly (…_REFUSED_RETRY_MS), and not counted as live.	mcp_events_routing::a_permanently_refused_configured_subscription_is_reported_and_does_not_keep_the_session	yes
4	Runtime subscriptions did not survive a restart.	Persisted, and restored at session start. A daemon starts the sessions that hold them, and their webhook tokens are held.	mcp_events_receiver::a_runtime_subscription_is_restored_after_a_restart_without_its_client	yes
5	Message-size cap	Extends #134's single cap (mcp_stdio::max_message_bytes, BEYOND_AI_AGENT_MCP_MAX_MESSAGE_BYTES, still one env var) to every path it did not cover, listed below.	mcp_message_cap::* (6 tests), plus mcp_wire units a_json_body_over_the_cap_answers_the_request_with_an_error and any_request_is_held_to_the_cap	yes: every path below except stdio (already #134's)
6	Nested request during events/*	Tests only: over stdio (poll and stream), in a daemon with a bystander session, and over direct HTTP (see F4)	mcp_events_nested::* (6 tests, incl. N3)	the rmcp-path tests pass on main by design; all fail with EventsPeer::track_request disabled

Item 5 covers these paths:

• Events wire, unary JSON body: an_over_cap_unary_json_answer_on_the_events_wire_is_refused.
• Events wire, unary SSE answer: an_over_cap_unary_sse_answer_on_the_events_wire_is_refused.
• Events wire, events/stream event: never read whole: skipped and reported (item 7). The wire's own 4 MiB / 1 MiB caps are gone.
• mcp_wire::HttpClient: every POST, not only skills/* (the skills-only 32 MiB constant is gone). A request's over-cap answer becomes its JSON-RPC error. Test: an_over_limit_message_from_an_http_server_fails_its_call_and_the_connection_survives.
• The GET stream: capped through rmcp's own SSE-event limit.

Item 7: an over-cap push event is skipped, not reconnected into. This was item 5's known limit. The event's bounded head is read structurally (mcp_stdio::oversized_stand_in, the same top-level member walk as #134's scan_head) for its routing, cursor and id, and the rest is skipped unread. A small $oversized stand-in takes its place, on both transports (the stdio pump, and the events SSE reader). The subscription keeps that cursor (or else the next heartbeat's), records an oversized gap, which reaches the client as a gap frame and the model as a notice that an event was dropped unread, and carries on. Proved with the blob fixture: a giant push event followed by a normal one; the normal event is delivered and the gap reported, over HTTP and over stdio (mcp_message_cap::an_over_cap_push_event_over_{http,stdio}_is_skipped_and_reported_and_the_next_is_delivered).

Audit fixes (at 229c29e)

#	finding	fix	proving test	fails with the fix reverted
F1	A runtime subscription to a removed server pinned its session forever.	An unknown or removed server is a permanent refusal. A restore refused for good forgets the spec. Runtime subscriptions expire after their session's last client command (…_RUNTIME_TTL_MS, 7 d). Boot restoration is bounded (…_MAX_RESTORED_SESSIONS, 32).	mcp_events_receiver::a_runtime_subscription_to_a_removed_server_is_forgotten_not_resurrected (renamed server, 1 s idle timeout, two restarts); state units runtime_subscriptions_expire_without_a_client, client_activity_is_recorded_only_while_it_matters	yes
F2	The item-2 test did not kill its mutants.	The redundant received-list is removed, so delivery rests on the tag alone. The test now asserts an empty pending log and durable done records.	as item 2	yes (removing receipts.received fails it)
F3	A terminating batch returned before Steered was emitted.	Steered is now emitted before the terminate return.	agent-core a_steer_folded_into_a_terminating_batch_is_still_reported	yes
F4	Over direct HTTP, server→client requests were silently ignored.	They are answered with an error at once (unary SSE and events/stream). Over rmcp they are attributed by track_call.	mcp_events_nested::over_direct_http_a_nested_request_during_an_events_{poll,stream}_is_refused_not_ignored	yes, both
F5	-32012 was classified as permanent.	Retried with credentials re-resolved, on a short backoff (≤5 s), and refused only after 5 in a row.	mcp_events_poll::a_forbidden_subscribe_is_retried_and_comes_up_once_credentials_work, …forbidden_over_and_over_is_eventually_refused	yes
F6	The fixture's blob tool was unused.	It is now used by the message-cap tests.	—	—

Re-audit fixes (at f86d197)

Each "fails with the fix reverted" was checked by reverting the fix in place and running the named test.

#	finding	fix	proving test	fails with the fix reverted
N1	A runtime subscription made by a single client command was resurrected on every boot, because it had no client-activity record and None was trusted forever.	set_runtime records client activity when it creates one, and a state with runtime subscriptions but no record is not restored.	mcp_events_receiver::a_runtime_subscription_made_by_one_command_expires_with_no_client (subscribe, disconnect, crash, restart with TTL 1 ms: no resubscribe, session not started)	yes. The creation record alone is also load-bearing: without it, a_runtime_subscription_is_restored_after_a_restart_without_its_client fails.
N2	The refusal POST had no Accept, so a streamable-HTTP server answers 406 and keeps waiting.	It sends the same Accept as every events POST, and the fixture now 406s an answer without one.	mcp_events_nested::over_direct_http_* (both)	yes
N3	Refusals on an events/stream spawned a task per request, without limit.	They are answered inline, one at a time, each under a 5 s bound.	mcp_events_nested::a_flood_of_server_requests_on_an_events_stream_is_answered_one_at_a_time (20 requests; the fixture reports at most 1 answer in flight)	yes
N4	The boot cap kept the first sessions alphabetically.	It ranks by most recent client activity.	mcp_events::tests::the_boot_cap_keeps_the_most_recently_used_sessions	yes
N5	A forgotten runtime subscription left its webhook token and secret in the snapshot.	TTL expiry, permanent refusal on restore, and a server ending it all forget_sub: spec, cursor and callback.	mcp_events_receiver::a_runtime_subscription_to_a_removed_server_is_forgotten_not_resurrected (asserts no "webhook" or whsec_ remains)	yes
N6	A panicked session holding runtime subscriptions was not restarted.	It restarts like the events session, with per-session backoff (1 s doubling to 60 s, reset after 10 min up).	mcp_events_receiver::a_session_with_runtime_subscriptions_is_restarted_after_a_panic	yes
N7	ordinary_limit serialized the whole message to read its id, and the move of every POST off rmcp's client was not documented or tested for parity.	It reads the id off the request. ARCHITECTURE.md states that every streamable-HTTP POST goes through HttpClient::post_bounded; rmcp's own post_message is never called. That still holds after #138 and #140: OAuthHttp → ViewCappedHttp → HttpClient. #140's HttpClient { oauth } flag, which routed only an OAuth server's POSTs through post_bounded, now selects nothing and is removed, along with OAUTH_MAX_MESSAGE_BYTES; the one per-message cap applies.	differential units against rmcp's reqwest client: mcp_wire::an_ordinary_request_is_sent_as_rmcps_own_client_would (bearer, Mcp-Session-Id, Accept, protocol version, custom headers, body) and …_answered_as_rmcps_own_client_would (401 AuthRequired, 403 InsufficientScope, 404-on-session SessionExpired, 202, JSON with session id, SSE) , which also pins the one deliberate divergence: a 401 with no challenge and a JSON-RPC error body is AuthRequired here, while rmcp's own client returns an ordinary error response (#140's rule, now for every server)	parity pins: dropping the session header or the bearer fails them
N2b	Found while rebasing after #138, which made every direct events/* POST carry the server's current OAuth token and refresh on 401. The refusal still sent the token from dial time, so after a refresh it spent a 401 and the server waited.	The refusal goes through the same Conn::send, without Mcp-Method because an answer has no method.	mcp_events::wire::a_refused_server_request_is_answered_with_the_current_token_and_refreshed_on_401	yes

N2 interop note. The Python SDK servers (mcp 2.3.0) used by check.py are stateless. They answer 400 ("Body must be a single JSON-RPC request or notification object") to any POSTed JSON-RPC response, with or without Accept, so they never raise server→client requests and the 406 cannot be reproduced against them. The fixture enforces Accept the way the SDK's stateful transport does, so a regression fails over_direct_http_*.

Design notes

• Delivery by tag.

  • SteeringMessage gains tag, and AgentEvent::Steered gains tags, which is not serialized.
  • Steered is emitted after the checkpoint on every path, including a terminating batch.
  • serve tags each MCP Events steer with its batch. Receipts::received records the batch as delivered when the tag is reported, and commits that at once.
  • The exactly-once exceptions are now three: a failed or aborted run before the model received the batch; a crash between the checkpoint and the done record; that record failing to be written.

• Permanent refusals:

  • an unknown server;
  • an event the server doesn't offer;
  • -32601, -32602, -32011;
  • -32014 other than schema_changed;
  • no compatible delivery mode.

  -32012 (forbidden) is auth-retry instead. Transient failures keep the 60 s-capped backoff and still keep the session alive.

• Runtime restore.

  • A successful mcp_events_subscribe stores its spec (PersistedSub::runtime), and the session re-subscribes it on start.
  • The TTL runs from the session's last client command; creating a runtime subscription counts as one. A state with no record at all is not restored.
  • At boot, a daemon starts at most 32 sessions holding specs still within their TTL, choosing the most recently used.
  • A spec is forgotten on explicit unsubscribe, on the server ending the subscription (unless re-discovery succeeds), on a permanent refusal at restore, and after the TTL with no client. Forgetting removes the cursor and the webhook token and secret too.

• Panics.

  • A main-future panic exits promptly, with the sweep.
  • A daemon session panic is contained. That is sound because sessions share only state behind plain replace-the-value locks, and MCP server processes belong to the process.
  • The events session, and any session holding runtime subscriptions, is restarted on per-session backoff. Any other panicked session restarts when a client comes back.
  • The test hook is a debug-build-only __test_panic command, gated on BEYOND_AI_AGENT_TEST_PANICS.

• One POST path. Every streamable-HTTP POST goes through mcp_wire::HttpClient::post_bounded, so rmcp's own post_message is unused. post_bounded applies the message cap, mcp_stdio::rescue (rust-sdk#1197), and rmcp's status handling. The differential tests above pin it to rmcp's client on request headers and response shapes. agent: refresh a rejected MCP OAuth token mid-session and retry once #138's OAuthHttp wraps it as the outermost layer. Since agent: MCP OAuth hardening — any 401 by status, RFC 6749 refusals definitive, pinned reload rule #140, any 401 is AuthRequired for every server, so mcp_oauth always sees the status; this is one of two pinned divergences from rmcp; the other, from agent: pin the OAuth server routing end to end; rmcp parity for non-JSON-RPC successes #142, is that a non-JSON-RPC success answering a request is an error (a_json_success_that_is_not_json_rpc_is_accepted_for_a_notification_but_fails_a_request). agent: pin the OAuth server routing end to end; rmcp parity for non-JSON-RPC successes #142's routing tests and docs now describe post_bounded as every server's path.

Checks

• Tests: every tests/mcp_*.rs suite passes (39 binaries, incl. agent: refresh a rejected MCP OAuth token mid-session and retry once #138's mcp_oauth_refresh), as do serve_reaper, serve_http, serve_drain, run_signal_handling, run_cli_flags and agent: kernel-locked manifest writes off the runtime, LISTEN_PID required, one frame deadline #139's serve_harness_deadlines (its frame-deadline lint). The mcp_events suites read stdout through agent: file_lock module, unreachable deadline-free readers, LISTEN_PID operator note #141's guarded Frames::new(&mut child, …), and agent: file_lock module, unreachable deadline-free readers, LISTEN_PID operator note #141's widened lint passes.
• Lib tests: agent and agent-core pass: 2760 tests across all 48 binaries in this sweep (incl. agent: pin the OAuth server routing end to end; rmcp parity for non-JSON-RPC successes #142's mcp_oauth_routing), 0 failed, after the rebase onto agent: file_lock module, unreachable deadline-free readers, LISTEN_PID operator note #141.
• Lint and format: cargo clippy (agent with code-mode, and the whole workspace) is clean with -D warnings. cargo fmt --check and dprint check are clean.
• Interop: check.py passes 12/12 over HTTP and 12/12 over stdio.
• Docs: both ARCHITECTURE.md files are updated.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JimHGjsfk2Ktm5GxyZJKKk