fix(observability): make the journal, metrics and one-shot error injection report what happened

[jpr5]

This PR aligns the request journal and Prometheus metrics with the response actually delivered, and makes one-shot error injection safe across concurrent requests. A handler crash amends its own request entry; a destroyed socket is recorded as interrupted rather than as a delivered error. Compatible requests reserve a queued error before asynchronous chaos processing, and release it if it is never served.

Behavior

Journal and route errors

• Route catches share error handling for logging, CORS, error envelopes, and journaling. Entries are attributed by server request identity through AsyncLocalStorage and Journal.onAdd, so concurrent requests sharing an x-request-id cannot amend each other's entries.
• A crash after a completed response preserves the delivered status and adds response.error. A crash during a response marks it interrupted; SSE crashes emit their error frame and destroy the socket. Proxy interruption reasons distinguish a client disconnect from recorder destruction.
• Files, batches, fine-tuning, and ElevenLabs voice catches retain service attribution. Files and voice journal after successful response writes, avoiding phantom entries when writes throw; files errors include their envelopes, and voice entries carry service: "elevenlabs-voice".
• Body-size enforcement destroys the request socket. Those requests now log at warn and journal status: 0, interrupted: true, and interruptReason: "request body exceeded size limit", without attempting a response on the dead transport. Metrics count destroyed. Deliverable malformed JSON still receives and journals 400. Client aborts similarly record an interruption with reason client aborted.
• Files rejection warnings escape control characters and Unicode line separators and cap the complete warning to 1,024 characters before the logger prefix. Request-controlled purpose/cursor text cannot forge another log line or generate request-sized warnings. Response and journal envelopes retain their existing contents and limits.

One-shot errors and chaos

• LLMock.nextRequestError and POST /__aimock/error share the existing endpoint-compatibility rule and queue. Selection reserves the one-shot before awaiting chaos in chat, embeddings, and all four ElevenLabs voice routes, including mixed chat/embedding concurrency. Exactly one request receives the injected error.
• Unserved reservations return to the queue after chaos or cancellation. Queue clearing/reset/hot reload advances a generation so an older request cannot restore an error into a reset queue. OpenRouter fallback cannot consume an error while discarding its candidate; queued errors appear as responseKind: "error" in the control API.
• Chaos source labels use one provider-aware proxy rule, including per-request strict mode. A strict malformed response is applied locally and labeled internal; unrelated upstream configuration cannot label a locally answered request proxy.
• Chaos journaling follows writes, preserves committed status for disconnects, and records null request bodies for fal gates where appropriate. All five fixture chaos fields and server defaults use the shared validation table, rejecting invalid rates, non-finite values, negative zero, strings, and fractional latency consistently.

Metrics and library semantics

• Metrics count both completed and destroyed responses once, and retain malformed-Host requests under {unknown}. Labels follow route shape independently of response status, preserving routed fixture-miss 404s while bounding unknown paths.
• Dynamic Gemini/Vertex actions, fal paths, music paths, and files/batches depths use bounded labels. Control and mount labels take precedence over provider patterns, matching dispatch order, including mounts beneath /fal, /v1/files, and /v1/music. Mounts added after startup are read at record time; overlapping mounts retain registration order. Bare /fal shares the router's path rule.
• Journal.getAll returns nothing for finite nonpositive limits, floors fractions, and treats non-finite values as unlimited. Journal test-id filtering shares the full-header resolver used for request scoping.
• clearRequests() preserves fixture match counts; sibling sequencing compares toolResultContains and context. Fixture debug descriptions render predicates and nested objects more readably.

Local red-green proof

The following reproductions exercised real local HTTP/socket surfaces before and after production edits at base 142f7bdb87894714b67541801edf6f209a99b18a. No external LLM APIs were used. Skipped counts below come from explicit test-name filters.

A01 — concurrent one-shot selection

pnpm exec vitest run src/__tests__/one-shot-concurrency.test.ts

Red: 11 failed | 10 passed; concurrent embeddings, four voice routes, and mixed chat/embedding requests returned duplicate [500,500] instead of [200,500]. Green: 21 passed. The same suite covers cancellation, chaos release, and generation-reset controls on the changed HTTP routes.

A02 — oversized request transport and journal agreement

pnpm exec vitest run src/__tests__/server.test.ts -t 'oversized body|deliverable malformed'

Red, exit 1: 4 failed | 2 passed | 78 skipped (84). Chat, responses, embeddings, and batches sockets returned no HTTP response, while their journals claimed 400. Green, exit 0: 6 passed | 78 skipped (84). Each oversized request has one interrupted status-0 entry and one destroyed metric; both ordinary malformed-JSON controls retain delivered/journaled 400.

A03 — bounded, escaped rejection warnings

The same real HTTP/stderr probe ran before and after, changing only capture filenames:

pnpm exec tsx /Users/jpr5/.local/share/copilotkit/cr/aimock-pr466/fix-A03/probe.mts > /Users/jpr5/.local/share/copilotkit/cr/aimock-pr466/fix-A03/red.stdout 2> /Users/jpr5/.local/share/copilotkit/cr/aimock-pr466/fix-A03/red.stderr
pnpm exec tsx /Users/jpr5/.local/share/copilotkit/cr/aimock-pr466/fix-A03/probe.mts > /Users/jpr5/.local/share/copilotkit/cr/aimock-pr466/fix-A03/green.stdout 2> /Users/jpr5/.local/share/copilotkit/cr/aimock-pr466/fix-A03/green.stderr

Both measurement runs exit 0. Red: 6 physical log lines; maximum warning 200,146 characters. Green: 5 physical log lines; maximum 1,033 characters including the logger's 9-character prefix. Injected newlines become literal escaped text within one rejection line. Response/journal summaries are unchanged, including the existing journal body cap for the large envelope.

Regression assertions before the edit:

pnpm exec vitest run src/__tests__/files-api.test.ts -t 'bounds and escapes'

Red: 4 failed | 179 skipped (183); oversized warnings measured 200,146 and 8,137 characters against a 1,033-character limit. After the edit, the full file ran:

pnpm exec vitest run src/__tests__/files-api.test.ts

Green: 183 passed, including ordinary readable-warning coverage.

A04 — mount labels before provider labels

pnpm exec vitest run src/__tests__/metrics.test.ts -t 'mounted service metric precedence'

Red, exit 1: 4 failed | 2 passed | 86 skipped. Real mounts under /fal/custom, /v1/files/custom, and /v1/music/custom answered 200 but lost their mount labels; for example, /fal/{other} incorrectly counted all four requests. Green, exit 0: 6 passed | 86 skipped. Live scrapes show mount-root count 1, agent-card count 1, bounded other count 2, with control precedence, registration order, late mounts, and provider boundary controls preserved.

Scope and validation

Endpoint compatibility is unchanged: files, batches, and fine-tuning are not newly eligible for injected errors. Existing multipart parser edge cases, malformed control-API input handling, Azure label rules, and the bare control-prefix label discrepancy remain outside this change.

Independent replay at integrated commit 0ebfb27d35360b66b3abb2016e5e9eb089d3a3c9 passed:

pnpm exec vitest run src/__tests__/one-shot-concurrency.test.ts src/__tests__/server.test.ts src/__tests__/metrics.test.ts -t 'one-shot errors across awaited serving selection|oversized body|deliverable malformed|mounted service metric precedence' --reporter=verbose

Result: 3 files passed; 33 passed | 164 skipped (197), exit 0. Both standalone real HTTP probes also exited 0: three concurrent embedding pairs each delivered one 500 and one 200; oversized sockets matched interrupted journals; all three colliding mount namespaces retained bounded mount labels. Independent warning measurements confirmed five warnings, maximum length 1,033, escaped newlines, and no raw control characters or unbounded payload.

Final local pre-push checks passed in order: pnpm run format:check, pnpm run lint, pnpm run typecheck, pnpm run test, and pnpm run build, all exit 0. The full suite passed 6,818 tests across 209 files, with zero skips. The checked source tree is da47a2e702da57a2c74378ab89deb57281c8dbde; the five-commit reorganization at d487e940e1359a6e927b5aa08b3cce8e696fb868 preserves that exact tree. All five rewritten commit messages pass commitlint, and the working tree is clean. All 29 CI checks passed on d487e940e1359a6e927b5aa08b3cce8e696fb868. The Docker check passed on its single retry after the first Linux/arm64 build stalled inside tsdown; the retry used the unchanged commit.

[pkg-pr-new]

Open in StackBlitz

npm i https://pkg.pr.new/@copilotkit/aimock@466

commit: d487e94