feat(integrations): add shared harness contract module to core by miguelg719 · Pull Request #2746 · browserbase/stagehand

miguelg719 · 2026-08-16T16:43:19Z

Stacked on #2743. Part 1 of the harness consolidation stack (see plan).

What

Adds @browserbasehq/stagehand-integrations/harness — the shared contract module for harness adapters:

Contract stays in evals (reshaped after review): AgentMount, AgentRunToolSpec, and the run-tool constants remain in packages/evals/core/contracts/tool.ts; the harness module carries only shared utilities and seam types. Integrations never imports from evals — dependency direction is evals → integrations.
New seam types for the upcoming adapter packages: StartedSurface, HarnessTask, HarnessLogger (deliberately no getLogs), HarnessAdapterError.
sanitizeErrorMessage dedupe: the codexCodeBridge copy was a 2-rule subset of the facade stdio-server's 5 rules (drifted). Single merged superset in harness/redact.ts; both callers rewired. Only behavior change in the PR: bridge redaction widens to the superset.
buildAllowlistedEnv dedupe: one copy replaces 4 identical ones (vercel-ai, mastra, claude-code, codex examples). All four verified identical before merging.

Verification

turbo build/typecheck/test:unit across the affected graph (26 tasks) + full unit suite + lint/fmt ✅
New core/tests/harness.test.ts: per-rule redaction assertions (both source rule sets survive the merge), env allowlist behavior, run-tool constant derivation
Example tests unchanged except import paths (assertions identical)
codex exec review second-opinion pass: no source findings

Summary by cubic

Centralizes harness utilities in @browserbasehq/stagehand-integrations/harness to share env allowlisting and error redaction across adapters. The codex bridge switches from a 2-rule subset to the 5-rule superset redaction, so more secrets are redacted; no other behavior changes.

Keeps the agent mount contract in packages/evals/core/contracts/tool.ts; @browserbasehq/stagehand-integrations/harness exports sanitizeErrorMessage, buildAllowlistedEnv, HarnessLogger, and HarnessAdapterError only.
Replaces four duplicate buildAllowlistedEnv copies in vercel-ai, mastra, claude-code, and codex; tests/examples now import from @browserbasehq/stagehand-integrations/harness.
De-duplicates sanitizeErrorMessage; both the facade stdio server and the codex bridge import the shared superset (core uses explicit .js ESM specifiers).
Adds packages/integrations/core/tests/harness.test.ts for redaction and env allowlist; expands packages/evals/.gitignore to exclude run artifacts.

^{Written for commit 948c4c3. Summary will update on new commits.}

changeset-bot · 2026-08-16T16:43:22Z

⚠️ No Changeset found

Latest commit: 948c4c3

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

cubic-dev-ai

All reported issues were addressed across 21 files

Architecture diagram

sequenceDiagram
    participant Evals as Evals Package
    participant Harness as Integrations Harness
    participant Bridge as Codex Code Bridge
    participant Facade as Facade Stdio Server
    participant Adapters as Adapter Clients (Vercel-AI, Mastra, Claude-Code, Codex)

    Note over Evals,Harness: Contract Module Boundary
    Evals->>Harness: Import AgentMount, AgentRunToolSpec, AGENT_RUN_TOOL_*
    Harness-->>Evals: Re-exported contracts (evals depends on integrations)

    Note over Bridge,Facade: Shared Redaction Utility
    Bridge->>Harness: sanitizeErrorMessage(message)
    Facade->>Harness: sanitizeErrorMessage(message)
    Harness-->>Bridge: Redacted message (5-rule superset)
    Harness-->>Facade: Redacted message (5-rule superset)

    Note over Adapters,Harness: Shared Environment Allowlist
    Adapters->>Harness: buildAllowlistedEnv()
    Harness->>Harness: Filter env (STAGEHAND_*, BROWSERBASE_*)
    Harness-->>Adapters: Allowlisted env object

    Note over Evals,Harness: New Seam Types (future adapters)
    Harness->>Harness: StartedSurface, HarnessTask, HarnessLogger, HarnessAdapterError

_{Reply with feedback, questions, or to request a fix.

Re-trigger cubic}

cubic P2 on #2746 — core uses explicit .js relative ESM specifiers.

cubic-dev-ai

All reported issues were addressed across 4 files (changes from recent commits).

_{Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic}

miguelg719 · 2026-08-16T23:55:11Z

Reshaped after review feedback (see latest commit): the AgentMount/AgentRunToolSpec contract returned to evals/core/contracts/tool.ts verbatim — it abstracts over evals' tool-surface registry and doesn't belong in integrations. The shared harness module now holds only what thin SDK adapters genuinely need: merged sanitizeErrorMessage, buildAllowlistedEnv, HarnessLogger, HarnessAdapterError. Net diff for the module rework: +51/−79.

cubic P2 on #2746 — core uses explicit .js relative ESM specifiers.

Adds @browserbasehq/stagehand-integrations/harness with: - the harness mount contract (AgentMount, AgentRunToolSpec, AGENT_RUN_TOOL_* constants) moved verbatim from evals core/contracts/tool.ts, which now re-exports them - StartedSurface / HarnessTask / HarnessLogger / HarnessAdapterError, the narrow seam types for upcoming harness adapter packages - sanitizeErrorMessage: single merged copy (the codexCodeBridge variant was a 2-rule subset of the stdio-server's 5; both callers now share the superset) - buildAllowlistedEnv: single copy replacing 4 identical ones (vercel-ai, mastra, claude-code, codex examples) No behavior change except bridge redaction widening to the superset.

cubic P2 on #2746 — core uses explicit .js relative ESM specifiers.

… utilities The AgentMount/AgentRunToolSpec contract abstracts over evals' tool surface registry — an evals concern, not an integrations one. It moves back to evals/core/contracts/tool.ts verbatim. The shared harness module keeps only what thin SDK adapters genuinely need: the merged sanitizeErrorMessage, buildAllowlistedEnv, HarnessLogger, and HarnessAdapterError. Also ignore evals run artifacts (.trajectories, rubric cache) — generated output that was dirtying trees and fmt scans.

…ls (#2748) Stacked on #2746 (← #2743). Part 2 of the harness consolidation stack. **Reshaped after review feedback: thin session layer only.** ## What New thin package `@browserbasehq/stagehand-integrations-claude-agent-sdk`: `loadClaudeAgentSdk` + `runClaudeAgentSession` — the `query()` streaming loop (abort control, message logging, token usage, status/stop-reason, max-turns classification) with explicit options and **no `EVAL_*` env reads**. This is the code the evals `claude_code` runner and the claude-code facade example were both maintaining; both now call the package. **Deliberately NOT extracted** (returned to evals verbatim after the first cut over-reached): the tool-surface mount machinery — `AgentMount` handling, the in-process `run` MCP tool over live handles, permission-gate derivation. That code abstracts over evals' surface registry and has no integrations consumer. Net diff +661/−515 — the positive remainder is package scaffolding (package.json/tsconfig/tsdown/vitest/turbo/CI globs); the logic itself is a move. ## Verification - Full gates (build/typecheck/test:unit/lint/fmt) ✅; session tests moved with the code - Connected smoke (`b:webvoyager --harness claude_code --tool stagehand_code -e browserbase`) re-run on the reshaped code — results in PR comment - The example keeps its original security posture (stagehand-tools-only `canUseTool`, deny everything else) ## Review updates (2026-08-29) - **Structural max-turns detection**: `resolveClaudeCodeStatus` now checks the SDK's `subtype: "error_max_turns"` on the result message (required for SDK 0.3.224 from #2743); the prose regex applies only to thrown iteration errors, so agent text mentioning "turn limit" can no longer misclassify a successful run. - **No default tool grants**: `allowedTools` defaults to `[]` instead of `WebFetch`+`WebSearch`; both existing callers already pass explicit lists. - The claude-code example resolves the facade stdio-server inside `main()` again so a missing build routes through `handleFailure`. --------- Co-authored-by: Miguel Gonzalez <miguel@browserbase.com>

Stacked on #2748. Part 3 of the harness consolidation stack. **Reshaped after review feedback: thin session layer only.** ## What New thin package `@browserbasehq/stagehand-integrations-codex-sdk`: `loadCodexSdk` + `runCodexSession` — the `startThread`/`runStreamed` event loop (thread config, tool-step budget with abort + listener detach, usage/stop-reason capture) with explicit options and no `EVAL_*` env reads. Evals' codex runner and the codex facade example both call it. **Deliberately NOT extracted** (stays in evals): the loopback code bridge and mount machinery — they expose evals' in-process tool surfaces to codex and have no integrations consumer. `codexCodeBridge.ts` is back in evals unchanged (still using the shared redaction from #2746). The example keeps its posture (read-only sandbox, on-failure approvals, harness-tuned default model) and now exits non-zero on failed sessions instead of printing nothing and exiting 0. Net diff +674/−420 — scaffolding plus the session module; logic is a move. ## Verification - Full gates ✅; session tests moved with the code - Connected smoke re-run on the reshaped code — results in PR comment; step-budget semantics previously verified identical to main (50 counted command executions) ## Review updates (2026-08-29) - **Fail-closed sandbox**: `validateCodexSandboxMode` falls back to `read-only` for unset/unknown modes (was `workspace-write`). - **Budget stops are `max_turns`**: tool-step budget exhaustion now reports `status: "max_turns"` (matching claude-agent-sdk) instead of `sdk_error`. - **Absent-vs-zero usage**: `cached_input_tokens`/`reasoning_output_tokens` stay absent when the SDK never reported them; the evals runner forwards them conditionally so trajectories don't record synthetic zeros. - Default tool-step budget is 100 (session + evals fallback) — codex budgets per tool call while claude budgets per turn. - Codex example: server resolution moved inside `buildCodexConfig` (fails through `handleFailure`), non-completed runs surface the agent's last message, and the `test` script builds the codex-sdk package it imports. --------- Co-authored-by: Alyssa Keimach <7604716+akeimach@users.noreply.github.com> Co-authored-by: Miguel Gonzalez <miguel@browserbase.com>

miguelg719 mentioned this pull request Aug 16, 2026

chore(evals): converge harness SDK pins onto the workspace catalog #2743

Merged

cubic-dev-ai Bot reviewed Aug 16, 2026

View reviewed changes

Comment thread packages/integrations/core/src/facade/stdio-server.ts Outdated

Comment thread packages/integrations/core/src/harness/index.ts

miguelg719 added a commit that referenced this pull request Aug 16, 2026

fix(integrations): use .js specifier for harness redact import

b0ed077

cubic P2 on #2746 — core uses explicit .js relative ESM specifiers.

miguelg719 mentioned this pull request Aug 16, 2026

feat(integrations): extract Claude Agent SDK harness adapter from evals #2748

Merged

cubic-dev-ai Bot reviewed Aug 16, 2026

View reviewed changes

Comment thread packages/integrations/core/tests/harness.test.ts

miguelg719 mentioned this pull request Aug 16, 2026

feat(integrations): extract Codex SDK harness adapter from evals #2749

Merged

miguelg719 added a commit that referenced this pull request Aug 22, 2026

fix(integrations): use .js specifier for harness redact import

c3c310e

cubic P2 on #2746 — core uses explicit .js relative ESM specifiers.

miguelg719 force-pushed the harness/contract branch from 9c7eaa0 to fcb5d07 Compare August 22, 2026 16:07

akeimach reviewed Aug 28, 2026

View reviewed changes

Comment thread packages/integrations/core/src/harness/contract.ts

akeimach approved these changes Aug 28, 2026

View reviewed changes

miguelg719 force-pushed the harness/contract branch from fcb5d07 to fd7ea81 Compare August 30, 2026 05:14

miguelg719 added 3 commits August 29, 2026 22:22

fix(integrations): use .js specifier for harness redact import

34dca12

cubic P2 on #2746 — core uses explicit .js relative ESM specifiers.

miguelg719 force-pushed the harness/contract branch from fd7ea81 to 948c4c3 Compare August 30, 2026 05:24

miguelg719 merged commit 26c4aa8 into main Aug 30, 2026
58 of 73 checks passed

Navigation Menu

Uh oh!

feat(integrations): add shared harness contract module to core - #2746

feat(integrations): add shared harness contract module to core#2746
miguelg719 merged 3 commits into
evals/harness-sdk-catalogfrom
harness/contract

miguelg719 commented Aug 16, 2026 •

edited

Loading

Uh oh!

changeset-bot Bot commented Aug 16, 2026 •

edited

Loading

Uh oh!

cubic-dev-ai Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

miguelg719 commented Aug 16, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

miguelg719 commented Aug 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Verification

Summary by cubic

Uh oh!

changeset-bot Bot commented Aug 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ No Changeset found

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

miguelg719 commented Aug 16, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

miguelg719 commented Aug 16, 2026 •

edited

Loading

changeset-bot Bot commented Aug 16, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading