fix: move the swe-bench-pro seed agent to chat completions

[shehabyasser-scale]

Follow-up to #64, answering @varunursekar's question there. Stacked on
fix/swebp-agent-context, so the diff shown is only the migration. Re-target to
main with gh pr edit --base main if we would rather supersede #64 than stack.

Is it a sidecar bug? No

The gateway is a deliberate provider-agnostic passthrough:

Provider-agnostic passthrough: forward any inference endpoint (responses,
chat/completions, messages, embeddings, ...) to the upstream litellm proxy,
which decides what it supports.

previous_response_id appears exactly once in vero, in _RequestAttributor,
purely to stamp a thread_id on log records. That class is documented as
"best-effort by construction ... it can never affect proxying". The field is
forwarded untouched to a proxy fronting fireworks_ai/deepseek-v4-flash, and
Fireworks has no Responses API response store.

We could give the sidecar one. It would not help. To honour the pointer we would
reconstruct the transcript and send it upstream in full anyway, because the
provider cannot resolve it: identical bytes on the wire, identical token cost.
The only thing that changes is that the gateway becomes stateful, so a store bug
corrupts conversations across every trial at once instead of in one agent.

Why chat completions, beyond style

The only thing the Responses API gives us over chat completions for these models
is reasoning passthrough across turns, and we already throw it away. The agent
asks for reasoning: {"effort": "high"} on every turn, then rebuilds a
conversation of assistant text plus function_call/function_call_output pairs
with no reasoning items in it. We were paying for statefulness and getting a
stateless conversation regardless.

Chat completions has no field a proxy can accept and silently ignore, so the
failure mode stops existing rather than being worked around.

What changed

• _TOOL_SCHEMAS holds the six schemas verbatim; TOOLS wraps them in the
  nested {"type": "function", "function": {...}} shape.
• _responses_create -> _completion_create on chat.completions. Retry and
  backoff untouched.
• instructions= becomes a system message, input becomes messages,
  max_output_tokens becomes max_tokens, reasoning: {effort} becomes
  reasoning_effort; usage reads prompt/completion tokens.
• A turn is now one assistant message carrying both its text and its
  tool_calls, followed by role: "tool" results. Chat completions rejects a
  tool result whose tool_call_id was not declared by the assistant message
  before it, so unlike Responses-API items these cannot be split. The existing
  whole-turn trimming invariant already guaranteed this and now protects both
  directions.

The gateway keeps forwarding the field, because rejecting it would be wrong: a
gateway fronting real OpenAI honours it, and _RequestAttributor threads those
chains. It now warns once per process instead, so a delegated conversation is one
grep away rather than invisible. Non-null only, since SDKs serialise the key as
null when unused. Verified against the last grid run: of 65,262 logged requests,
zero request bodies ever carried the field.

Tests

suite	result
seed agent	7 passed (was 5)
test_v05_harbor_inference	18 passed
full vero suite	456 passed, 15 skipped, 2 failed

New coverage: the nested tool schema, and reasoning_effort reaching only
reasoning models (deepseek-v4-flash matches none, so target behaviour is
unchanged). The resend and whole-turn-trim tests are rewritten for the chat shape
and now assert the call/result pairing chat completions enforces.

The 2 suite failures are test_v05_docker_sandbox and
test_v05_harbor_isolation_container, both from a corrupted local Docker content
store (blob sha256:... input/output error on every docker run), not this
change.

Not yet measured

Unit tests pin the wire shape; only a real run confirms deepseek-v4-flash
behaves the same through chat completions. Blocked on the same Docker corruption.
Worth an A/B on the 66-case sample before this is trusted for headline
numbers.

🤖 Generated with Claude Code

Greptile Summary

This PR migrates the swe-bench-pro seed agent from the OpenAI Responses API to Chat Completions, eliminating the previous_response_id delegation bug that caused 0.0000 scores on all 66 sampled cases. The gateway gains a one-shot process-level warning (not a rejection) when previous_response_id is still forwarded, so future regressions are one grep away rather than invisible.

• Agent migration (agent.py): tool schemas are split into _TOOL_SCHEMAS (flat, Responses-API shape) and TOOLS (nested {"type":"function","function":{...}} for Chat Completions); instructions/input/max_output_tokens/reasoning.effort become system message/messages/max_tokens/reasoning_effort; usage fields map to prompt_tokens/completion_tokens; each turn is one assistant message carrying both text and tool_calls followed by role:"tool" results.
• Gateway addition (inference.py): a closure-captured set latch (_stateful_warned) fires a logger.warning once per app instance when a request body carries a non-null previous_response_id, without touching proxying behaviour.
• Test expansion (test_agent.py, test_v05_harbor_inference.py): helpers refactored for Chat Completions shape; two new test cases cover the nested tool schema and reasoning_effort gating; the resend/trim tests are rewritten to assert the call/result tool_call_id pairing Chat Completions enforces.

Confidence Score: 5/5

The migration is straightforward and well-guarded: the if not calls: break path prevents empty tool_calls arrays from ever entering history, the call/result pairing invariant is preserved, and the gateway change is purely additive (warn-only, no proxying change).

All Chat Completions field mappings are correct (messages, max_tokens, reasoning_effort, prompt_tokens, tool_calls shape). The early-break on empty calls ensures the assistant message always carries at least one tool call when written to history. The gateway latch is safe under asyncio's single-threaded model. Test coverage is expanded rather than reduced, covering the new wire shape, tool-schema nesting, reasoning_effort gating, and call/result pairing.

Files Needing Attention: No files require special attention. The only caveat is the live A/B validation against deepseek-v4-flash that the author flags as blocked on Docker corruption.

Important Files Changed

Filename	Overview
harness-engineering-bench/swe-bench-pro/baseline/target/src/swebench_pro_agent/agent.py	Full migration from Responses API to Chat Completions: tool schema wrapping, message format, usage fields, and per-turn call/result pairing are all correct; the if not calls: break guard prevents empty tool_calls arrays from ever reaching the history.
harness-engineering-bench/swe-bench-pro/baseline/target/tests/test_agent.py	Tests fully rewritten for Chat Completions shape; two new cases cover the nested tool schema and reasoning_effort gating; call/result pairing invariants asserted.
vero/src/vero/gateway/inference.py	One-shot warning latch added for previous_response_id; uses a closure-captured set to avoid nonlocal, fires only on non-null values, leaves proxying untouched.
vero/tests/test_v05_harbor_inference.py	New test correctly verifies the one-shot warning fires exactly once for non-null previous_response_id, stays quiet for null, and leaves all three requests proxied successfully.

Sequence Diagram

sequenceDiagram
    participant Agent as SweBenchProAgent
    participant CC as chat.completions
    participant GW as VeRO Gateway
    participant LM as litellm/upstream

    Note over Agent: Turn 1
    Agent->>CC: "create(messages=[system,user], tools=TOOLS)"
    CC->>GW: POST chat/completions
    GW->>LM: forward
    LM-->>GW: choices[0].message
    GW-->>CC: response
    CC-->>Agent: message with tool_calls
    Agent->>Agent: append turn_items to blocks

    Note over Agent: Turn 2
    Agent->>CC: "create(messages=[system,user,block1,...])"
    CC->>GW: POST with full conversation
    GW->>LM: forward
    LM-->>GW: submit call
    GW-->>CC: response
    CC-->>Agent: submit - break

    Note over GW: previous_response_id guard
    GW->>GW: if non-null and not warned - log warning once

Loading

_{Reviews (1): Last reviewed commit: "fix: move the swe-bench-pro seed agent t..." | Re-trigger Greptile}