Skip to content

fix(blocks): return text payloads and correct provider completion reasons - #467

Merged
jpr5 merged 4 commits into
mainfrom
blitz/blocks-only-outcomes/integration
Sep 17, 2026
Merged

jpr5 merged 4 commits into
mainfrom
blitz/blocks-only-outcomes/integration

Conversation

@jpr5

@jpr5 jpr5 commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Text-only blocks fixtures currently end with a tool-call terminal even though no tools were emitted. Nonstreaming OpenAI chat also drops block content and tools entirely. This change derives block-path defaults from the resolved output: text finishes normally, tool-containing blocks retain tool-call terminals, and nonstreaming OpenAI chat returns the authored block payload.

The shared outcome feeds both streaming and nonstreaming paths for OpenAI chat, Claude messages, Gemini, Gemini Interactions, Cohere, Bedrock invoke, and Bedrock Converse. Existing supported finish overrides, legacy fallback for absent/empty blocks, block order, and provider-supported tool IDs remain intact. Nonempty blocks remain authoritative over conflicting legacy fields. Interactions retains its existing ignored finish-override behavior.

Local real red–green proof

Baseline: b1ba430658c0fac66dcf24464e7d44178553e477.
Verified candidate: 5c3d27461a038b7e6aa055f295ab3782f1991966 (tree f70d3873d3d35717437ef35c3b12e4f28eacfa23).

The same frozen probe exercised 96 actual HTTP responses from locally running LLMock: eight protocols including unchanged Ollama controls, both response modes, and text/tool/mixed/legacy/override/conflicting-legacy fixtures. JSON, SSE, NDJSON, and AWS EventStream responses were decoded from the wire; no fake responses or external LLM calls were used.

Commands run before production changes, from the integration checkout at the baseline:

pnpm build
node /Users/jpr5/.local/share/copilotkit/cr/aimock-section1/repro/probe.mjs /Users/jpr5/.local/state/copilotkit/worktrees/aimock/section1-r1-integration /Users/jpr5/.local/share/copilotkit/cr/aimock-section1/repro/red.json
node /Users/jpr5/.local/share/copilotkit/cr/aimock-section1/repro/check.mjs /Users/jpr5/.local/share/copilotkit/cr/aimock-section1/repro/red.json
# RESULT 63 passed; 33 failed; 96 real HTTP cases
# checker exit 1

Commands run independently after integration, from that checkout at the candidate:

pnpm build
node /Users/jpr5/.local/share/copilotkit/cr/aimock-section1/repro/probe.mjs /Users/jpr5/.local/state/copilotkit/worktrees/aimock/section1-r1-integration /Users/jpr5/.local/share/copilotkit/cr/aimock-section1/integrated-value/green.json
node /Users/jpr5/.local/share/copilotkit/cr/aimock-section1/repro/check.mjs /Users/jpr5/.local/share/copilotkit/cr/aimock-section1/integrated-value/green.json
# RESULT 96 passed; 0 failed; 96 real HTTP cases
# checker exit 0
shasum -a 256 -c /Users/jpr5/.local/share/copilotkit/cr/aimock-section1/repro/probe-sha256.txt
# probe.mjs: OK; check.mjs: OK

Build and probe commands exited 0 in both runs. The exact 33 failing cells flipped; all 63 passing controls stayed passing. Probe and checker were byte-identical between runs, verified against these SHA-256 hashes:

probe.mjs  012a6508d5ff6c18154199661d6523b0151e1c26dfaf3f58fbe06352e31f8ca9
check.mjs  968d495a16bb50965ec91bdbb66b865e70930cb19eb174dab6614b781ab0b89e

Actual text-only wire comparisons:

Surface Before After
OpenAI nonstream chat content="", finish_reason="tool_calls" content="BLOCK_TEXT", finish_reason="stop"
Claude streaming messages stop_reason="tool_use" stop_reason="end_turn"
Gemini nonstream generateContent finishReason="FUNCTION_CALL" finishReason="STOP"
Cohere streaming chat finish_reason="TOOL_CALL" finish_reason="COMPLETE"
Bedrock nonstream invoke stop_reason="tool_use" stop_reason="end_turn"

The standalone probe paths above are local evidence artifacts, not repository files. Full responses and cell comparison are retained under ~/.local/share/copilotkit/cr/aimock-section1/{repro,integrated-value}/. Repository-contained HTTP regressions are reproducible from the candidate checkout:

pnpm exec vitest run src/__tests__/fixture-blocks-outcome-openai.test.ts src/__tests__/fixture-blocks-outcome-claude-gemini.test.ts src/__tests__/fixture-blocks-outcome-bedrock.test.ts src/__tests__/fixture-blocks-outcome-cohere-interactions.test.ts
pnpm exec vitest run src/__tests__/fixture-blocks-*.test.ts src/__tests__/blocks-fixture-tolerance.test.ts
# Integrated broader command: 225 tests passed, 20 files passed, no skips

Each new suite was observed failing against its pre-fix implementation before passing after the fix. Controls cover empty text, both mixed orders, trailing empty text, tool arguments/IDs, overrides, legacy fallback, and payload plus terminal together.

Review and validation

All 17 valid review reports were adjudicated with zero mandatory findings. The independent promotion audit recommended zero promotions; no source changes or further confirmation round were required. Separate maintenance items are recorded in the review follow-ups.

Final pre-push checks passed in order, all exit 0:

pnpm run format:check
pnpm run lint
pnpm run typecheck
pnpm run test --maxWorkers=1 --no-file-parallelism
pnpm run build

The full suite passed 6,974 tests in 213 files, with zero skips or unhandled errors. The initial default-concurrency run had four timing failures and two worker RPC timeouts while host load exceeded 46. All four failed cases passed once in isolation with unchanged assertions and timeouts; the full gate then restarted with one worker to reduce contention. No source fixes, timeout changes, or test exclusions were used.

The four commits already separate the shared OpenAI change and provider adaptations by concern; all commit messages pass commitlint. The checked HEAD and tree above remain unchanged and clean. All 30 CI checks passed on 5c3d27461a038b7e6aa055f295ab3782f1991966, including the supported Node test matrix, Python tests, static quality, package exports, and Docker build.

@pkg-pr-new

pkg-pr-new Bot commented Sep 17, 2026

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/@copilotkit/aimock@467

commit: 5c3d274

@jpr5
jpr5 merged commit daeb05d into main Sep 17, 2026
30 checks passed
@jpr5
jpr5 deleted the blitz/blocks-only-outcomes/integration branch September 17, 2026 19:46
tylerslaton added a commit that referenced this pull request Sep 22, 2026
## [1.43.0] - 2026-09-22

> **BREAKING** — `aimock -h` is `--help`, not `--host`: `aimock -h 0.0.0.0` exits 1 with `Error:
> Unexpected argument '0.0.0.0'. This command does not take positional arguments`. Migration:
> `--host <string>` (long form only). The `llmock` bin (the Docker ENTRYPOINT) keeps `-h, --host`
> (#453).

### Added

- OpenAI GPT-Live mock, record and offline replay on `GET /v1/live/sessions`, with `onLive` fixtures
  (#468)
- Live sessions enforce message/audio/queue/lifetime limits; recordings sanitize credentials (#468)
- Live drift canaries for lifecycle, audio/transcript, delegation and usage, attributed to
  `openai-live` (#468)
- Chaos `rateLimitRate` / `--chaos-ratelimit`: deterministic 429 with `Retry-After` (#449)
- Chaos `latencyMs` / `--chaos-latency` now actually delays responses on every path (#449)
- OpenAI Files API mock — byte-exact uploads, create-purpose enum, CORS on faults (#445)
- OpenAI fine-tuning jobs mock — deterministic lifecycle, events, cursor pages (#447)
- OpenAI Batches API mock — create/list/retrieve/cancel with real output files (#446)
- `X-Request-Id` echoed or minted on every response; `?requestId=` filters the journal (#450)
- `aimock validate` lints fixture files or directories offline, failing on broken files (#453)
- ElevenLabs Voice Design record/replay — design, save-as-voice, and voice slot management (#452)
- ElevenLabs Voice Design provenance block and strict-mode 503 coverage (#454)

### Changed

- **BREAKING:** `aimock -h` is `--help`, matching `aimock convert -h` and `aimock validate -h`; the
  host override is `--host` only. The `llmock` bin (the Docker entrypoint) keeps its own `-h,
  --host` — see the note above (#453)
- `aimock --config ""`, `--port ""` and `--host ""` are usage errors naming the option (#453)
- Realtime `OpenAI-Beta: realtime=v1` now returns the real sunset rejection, not a session (#461)
- `POST /v1/images/variations` now replays the real removal 404; OpenAI deleted it (#462)
- `ChaosAction` gains `"rateLimit"` — an exhaustive switch over it needs a case (#449)
- `applyChaosAsync()` returns `false | "handled" | "unwritable"` instead of a bare `boolean` (#449)
- Journal `headers` now ALWAYS carry `x-request-id` — exact `toEqual` asserts break (#450)

### Deprecated

- Synchronous `applyChaos()` warns once per process; it skips chaos latency. Use `await
  applyChaosAsync(...)` (#449)

### Fixed

- Reasoning-first chat streams with content and tool calls carry the assistant role in the first
  chunk (#470)
- Journals and metrics reflect delivered vs interrupted responses; one-shot errors never
  double-deliver (#466)
- Nonstreaming OpenAI chat returns block text and tools; text-only blocks finish normally on every
  provider (#467)
- AG-UI drift reads generated 1.0 schemas; `AGUIRunStartedEvent` gains optional `protocolVersion`
  (#469)
- Drift reports list `unverifiedSurfaces`; offline Bedrock/Vertex checks no longer skip silently
  (#460)
- Drift docs match actual coverage; ElevenLabs drift separates vendor observations from fixture
  checks (#465)
- Chaos no longer treats committed headers as a dead response; only status-writing actions are
  skipped (#449)
- Chaos skip logs name what happened; `aimock_chaos_triggered_total` counts only written responses
  (#449)
- A request whose chaos latency was cancelled by client hang-up is no longer served or journalled
  (#449)
- Moderations echoes the request's `model`; default is now `omni-moderation-latest` (#459)
- Image endpoints default to `gpt-image-1` — `dall-e-2`/`dall-e-3` were removed (#459)
- AG-UI record/proxy forwards the caller's headers and raw body upstream; `Accept` is forced to SSE
  (#455)
- The AG-UI recorder refuses to write a fixture from a non-stream or empty 2xx upstream reply (#455)
- A MINTED `x-request-id` is no longer forwarded upstream in record/proxy mode (#450)
MikeRyanDev added a commit that referenced this pull request Sep 22, 2026
Cuts the accumulated `[Unreleased]` work. **MINOR (1.43.0)**, kept at a
minor bump despite the
`-h` BREAKING banner (#453): the pre-2.0 precedent (1.14.2 shipped a
BREAKING note as a patch)
applies, and the version was agreed ahead of this PR. Do not re-version
to 2.0.0 at review.

**Prepared, not merged.** `publish-release.yml` fires on push-to-main,
so merging this
publishes to npm, tags `v1.43.0`, force-moves `v1`, cuts the GitHub
Release, dispatches the
Docker build, posts to Slack, and runs the PyPI job gated on
`_version.py`. Merge when you
want it live.

## What ships


> **BREAKING** — `aimock -h` is `--help`, not `--host`: `aimock -h
0.0.0.0` exits 1 with `Error: Unexpected argument '0.0.0.0'. This
command does not take positional arguments`. Migration: `--host
<string>` (long form only). The `llmock` bin (the Docker ENTRYPOINT)
keeps `-h, --host` (#453).

### Added

- OpenAI GPT-Live mock, record and offline replay on `GET
/v1/live/sessions`, with `onLive` fixtures (#468)
- Live sessions enforce message/audio/queue/lifetime limits; recordings
sanitize credentials (#468)
- Live drift canaries for lifecycle, audio/transcript, delegation and
usage, attributed to `openai-live` (#468)
- Chaos `rateLimitRate` / `--chaos-ratelimit`: deterministic 429 with
`Retry-After` (#449)
- Chaos `latencyMs` / `--chaos-latency` now actually delays responses on
every path (#449)
- OpenAI Files API mock — byte-exact uploads, create-purpose enum, CORS
on faults (#445)
- OpenAI fine-tuning jobs mock — deterministic lifecycle, events, cursor
pages (#447)
- OpenAI Batches API mock — create/list/retrieve/cancel with real output
files (#446)
- `X-Request-Id` echoed or minted on every response; `?requestId=`
filters the journal (#450)
- `aimock validate` lints fixture files or directories offline, failing
on broken files (#453)
- ElevenLabs Voice Design record/replay — design, save-as-voice, and
voice slot management (#452)
- ElevenLabs Voice Design provenance block and strict-mode 503 coverage
(#454)

### Changed

- **BREAKING:** `aimock -h` is `--help`, matching `aimock convert -h`
and `aimock validate -h`; the host override is `--host` only. The
`llmock` bin (the Docker entrypoint) keeps its own `-h, --host` — see
the note above (#453)
- `aimock --config ""`, `--port ""` and `--host ""` are usage errors
naming the option (#453)
- Realtime `OpenAI-Beta: realtime=v1` now returns the real sunset
rejection, not a session (#461)
- `POST /v1/images/variations` now replays the real removal 404; OpenAI
deleted it (#462)
- `ChaosAction` gains `"rateLimit"` — an exhaustive switch over it needs
a case (#449)
- `applyChaosAsync()` returns `false | "handled" | "unwritable"` instead
of a bare `boolean` (#449)
- Journal `headers` now ALWAYS carry `x-request-id` — exact `toEqual`
asserts break (#450)

### Deprecated

- Synchronous `applyChaos()` warns once per process; it skips chaos
latency. Use `await applyChaosAsync(...)` (#449)

### Fixed

- Reasoning-first chat streams with content and tool calls carry the
assistant role in the first chunk (#470)
- Journals and metrics reflect delivered vs interrupted responses;
one-shot errors never double-deliver (#466)
- Nonstreaming OpenAI chat returns block text and tools; text-only
blocks finish normally on every provider (#467)
- AG-UI drift reads generated 1.0 schemas; `AGUIRunStartedEvent` gains
optional `protocolVersion` (#469)
- Drift reports list `unverifiedSurfaces`; offline Bedrock/Vertex checks
no longer skip silently (#460)
- Drift docs match actual coverage; ElevenLabs drift separates vendor
observations from fixture checks (#465)
- Chaos no longer treats committed headers as a dead response; only
status-writing actions are skipped (#449)
- Chaos skip logs name what happened; `aimock_chaos_triggered_total`
counts only written responses (#449)
- A request whose chaos latency was cancelled by client hang-up is no
longer served or journalled (#449)
- Moderations echoes the request's `model`; default is now
`omni-moderation-latest` (#459)
- Image endpoints default to `gpt-image-1` — `dall-e-2`/`dall-e-3` were
removed (#459)
- AG-UI record/proxy forwards the caller's headers and raw body
upstream; `Accept` is forced to SSE (#455)
- The AG-UI recorder refuses to write a fixture from a non-stream or
empty 2xx upstream reply (#455)
- A MINTED `x-request-id` is no longer forwarded upstream in
record/proxy mode (#450)

The 35 `[Unreleased]` bullets were condensed to 34 one-liners, most ≤100
chars, because
this text is fed verbatim into the GitHub Release body and summarized
into `#oss-alerts`.
Every `(#N)` reference is preserved; the three un-numbered GPT-Live
bullets now cite #468.
Detail lives in the PRs.

## Behaviour changes to read before you merge

- **#453** — `aimock -h` is now `--help`. `aimock -h 0.0.0.0` exits 1.
Migrate to `--host 0.0.0.0`.
The `llmock` bin (Docker `ENTRYPOINT`) is unchanged and still accepts
`-h, --host`.
`--config ""`, `--port ""`, `--host ""` are now usage errors instead of
"not given".
- **#450** — journal `headers` always carry `x-request-id`; exact
`toEqual` asserts on headers break.
- **#449** — `ChaosAction` gains `"rateLimit"` (exhaustive switches need
a case);
`applyChaosAsync()` returns `false | "handled" | "unwritable"`
(truthiness unchanged, explicit
`boolean` bindings need updating); sync `applyChaos()` warns once per
process.
- **#459 / #461 / #462** — defaults track upstream: images default to
`gpt-image-1`, moderations
to `omni-moderation-latest`; `OpenAI-Beta: realtime=v1` returns the real
sunset rejection;
  `POST /v1/images/variations` returns the real removal 404.
- **#466 / #467** — journal and metrics now count interrupted responses
and text-only block
fixtures finish with a normal terminal reason; suites asserting the old
counts or
`finish_reason: "tool_calls"` on text-only blocks will see different
values.

## Version surfaces

Enumerated with `git grep -F 1.42.0`, not from a list. Seven carriers
bumped:

| Surface | Field |
| --- | --- |
| `package.json` | `version` |
| `charts/aimock/Chart.yaml` | `appVersion` |
| `.claude-plugin/plugin.json` | `version` |
| `.claude-plugin/marketplace.json` | `plugins[0].source.version`
(`^1.43.0`) |
| `docs/index.html` | the `aimock v…` banner |
| `packages/aimock-pytest/src/aimock_pytest/_version.py` |
`AIMOCK_VERSION` |
| `packages/aimock-pytest/README.md` | the `--aimock-version` default |

Deliberately untouched: `packages/aimock-pytest/pyproject.toml` (own
PyPI cadence, stays
`0.5.3`), `charts/aimock/Chart.yaml` `version: 0.1.0` (the chart's own
version), the
historical `1.41.0` mentions in `publish-release.yml` and
`npm-publish-verify-workflow.test.ts`, and CHANGELOG history.

`_version.py` is bumped because #466 and #450 change `/__aimock/journal`
and one-shot
`/__aimock/error` delivery — routes the pytest client calls — same rule
as 1.42.0.

## 1.42.1

npm `latest` is currently **1.42.1**, cut from
`maintenance/v1.42.1-lgts-mcp` (#471) and never
merged to main, so main's CHANGELOG had no `[1.42.1]` entry. Its fix
(#470) is on main and ships
here. This PR adds the one-line `## [1.42.1] - 2026-09-18` history entry
so every published
version appears in the file; every prior patch release (1.37.1–1.37.4)
already did.

## README coverage

The Features list had no mention of the OpenAI Files API (#445),
fine-tuning jobs (#447) or
Batches (#446), all shipped in this release — the same gap #443 closed
for the control API.
Added one **OpenAI platform APIs** bullet under Multimedia APIs linking
the Files and fine-tuning
docs pages; Batches has no docs page yet so it is described inline.
`aimock validate`, `--host`,
chaos latency/rate-limit, `X-Request-Id` and GPT-Live were already
covered. The "11 providers
across 22 API surfaces" headline was left as-is: that line enumerates
providers, and re-counting
it is a docs decision rather than a release step.

## npm description sync

Ran the workflow's inline extractor against this branch's README (after
the edit); output is
byte-identical to `package.json.description`, so the on-merge sync is a
no-op.

## Gates

Exit codes captured by redirect to separate files, never through a pipe.
Fresh worktree,
`pnpm install --frozen-lockfile`.

| Gate | Exit |
| --- | --- |
| `pnpm format:check` | 0 |
| `pnpm lint` | 0 |
| `pnpm typecheck` | 0 |
| `pnpm build` | 0 |
| `pnpm test:exports` | 0 |
| `pnpm test` | 0 — 225 files passed; 7341 tests passed |
| `npx commitlint --from origin/main --to HEAD` | 0 (1 cosmetic
`footer-leading-blank` warning) |

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant