Anthropic Claude model on Vertex fails when served as an A2aAgent on Agent Engine (serving instance killed mid-request → 400 on message:send); Gemini works

[BrianAlderman]

Existing issue check

Closest are #3861 (AdkApp↔A2aAgent comms, Gemini + Sequential agent — closed stale, not resolved) and #3020 (Vertex "model not found" — closed). Neither covers this Claude-specific A2A serving failure; searches for Anthropic/Claude + A2A return nothing.

Describe the bug

An ADK LlmAgent whose model is an Anthropic Claude model on Vertex (via google.adk.models.anthropic_llm.Claude / AnthropicLlm), deployed to Vertex AI Agent Engine as a managed A2aAgent (vertexai.preview.reasoning_engines.A2aAgent + agent_engines.create), fails to serve over A2A. On an a2a/v1/message:send call the serving instance is killed mid-request (clean SIGTERM + instance replacement) and the caller receives HTTP 400.

The same agent/topology with a Gemini model serves cleanly over the identical A2A path. The failure tracks the model, not tools/MCP.

Environment

• google-adk: 2.3.0 (2.4.0 changelog shows no related fix)
• a2a-sdk: 0.3.26 ([http-server] extra installed)
• google-cloud-aiplatform: 1.159.x
• Runtime: Vertex AI Agent Engine (managed A2aAgent)
• Model: Anthropic Claude on Vertex (publishers/anthropic/models/…), region global
• Client: ADK RemoteA2aAgent resolving the agent card (…/a2a/v1/card)

To reproduce

1. Build a minimal LlmAgent whose model is an Anthropic Claude-on-Vertex model (ADK Claude(...)), with one trivial tool or none.
2. Wrap it as A2aAgent(agent_card=…, agent_executor_builder=…) and deploy via agent_engines.create(agent_engine=A2aAgent(...), requirements=[…, "a2a-sdk[http-server]"], …).
3. Wait for the engine to stabilize; confirm the card at …/a2a/v1/card (HTTP 200).
4. Send one message via a2a/v1/message:send (e.g. through ADK RemoteA2aAgent).

Observed behavior

• The task is created, then the serving instance logs a clean SIGTERM and a replacement container starts mid-request (~10–45s in), around/before the model turn.
• The client's message:send returns HTTP 400; upstream this surfaces as an empty result.
• It fails even on a no-tool ping — before any tool call, no MCP involved.
• The Claude call itself works: …:rawPredict returns 200 in the logs. So inference is fine — the break is in the native A2A serving / response-conversion path, not the model call.

Isolation

Agent	Model	Tools/MCP	A2A serve
control	Gemini	none	PASS (clean reply, no SIGTERM)
minimal	Claude (Sonnet)	one trivial tool, no MCP	FAIL (SIGTERM → 400)

The only variable that tracks the failure is the model (Claude vs Gemini). A heavier Claude agent (many tools + MCP) fails the same way, and hours after deploy (ruling out a fresh-deploy confound).

Expected behavior

A Claude-on-Vertex LlmAgent served as an A2aAgent should handle message:send and return a response, as the Gemini equivalent does.

Hypothesis / notes

• Likely the Claude event/part structure (e.g. thinking blocks or streaming aggregation) differs from Gemini in a way the A2A message/event converter (google/adk/a2a/converters/…) or the A2aAgentExecutor response path doesn't handle, taking down the serving worker.
• Untested cheap idea: disable Claude "thinking" (thinking_budget=0) so it emits plain text/tool_use like Gemini.
• Happy to provide a fully self-contained minimal repro + engine logs on request.

[surajksharma07]

@BrianAlderman — the control/minimal matrix makes this much easier to reason about.

One thing worth trying: the Anthropic client has no explicit request timeout (SDK default is 600s), so a slow or stalled Claude call might be what's tripping Agent Engine's own liveness check.
Could you set an explicit shorter timeout (~55s) on the client and see if the instance stays up?

Separately, if you can pull the Cloud Logging entry for the killed instance around the SIGTERM (exit reason, any stack trace, memory at the time), that'd tell us for sure what's actually happening.

[BrianAlderman]

@surajksharma07 — thanks, both good asks. I re-ran a clean, isolated repro today on the exact stack from the report (google-adk 2.3.0, a2a-sdk 0.3.26 [http-server], google-cloud-aiplatform 1.159.0; Vertex AI Agent Engine, model region global, engines in us-central1). Answers to both below.

Setup

Four identical minimal A2aAgents — same topology, no tools, no MCP, no I/O — deployed via agent_engines.create(agent_engine=A2aAgent(...)). The only variable changed between them is the model / model config. Each was driven with a single trivial message:send ("hello - are you alive?") via ADK RemoteA2aAgent.

Agent	Model / config	message:send result
control	Gemini 3.5 Flash	PASS — real reply, no SIGTERM
baseline	Claude Sonnet	FAIL — HTTP 400, empty result
no-thinking	Claude Sonnet, thinking_budget=0	FAIL — HTTP 400, empty result
timeout	Claude Sonnet, explicit 55s AsyncAnthropicVertex timeout	FAIL — HTTP 400, empty result

The Gemini control on the byte-for-byte identical A2A path returns a clean reply. The only thing that tracks the failure is the model.

(1) Explicit 55s client timeout — does not help

I subclassed ADK's Claude to pass timeout=55.0 to AsyncAnthropicVertex (default is indeed unset → SDK's 600s). The instance is still killed and message:send still returns 400. The kill happens ~9s into the request — nowhere near 55s (or 600s). So the client timeout isn't the mechanism: the serving instance is being replaced by the platform well before any Anthropic-client deadline could fire.

(2) The killed-instance log around the SIGTERM

Cleaned-up serving-container log for the baseline Claude engine across one message:send (worker pid in brackets; I stripped the repeated [EXPERIMENTAL] A2A warnings):

14:34:49.443 [21] INFO: 169.254.169.126 - "GET /api/a2a/v1/card?history_length= HTTP/1.1" 200 OK
14:34:49.844 [20] convert_a2a_request_to_agent_run_request  (a2a_agent_executor.py:211)
14:34:49.847 [20] convert_a2a_part_to_genai_part             (request_converter.py:106)
14:34:49.847 [20] AgentRunRequest(...)                       (request_converter.py:111)
      ── worker [20] emits nothing further for ~13s: no model call (no :rawPredict),
         no LLM response, no ERROR, no Python traceback ──
14:35:02.097      Entrypoint: Starting up services...        ← replacement container boots
14:35:03.041 [1]  INFO: Uvicorn running on http://0.0.0.0:8080
14:35:04.758 [1]  INFO: Received SIGTERM, exiting.           ← old instance killed
14:35:04.759 [1]  INFO: Terminated child process [14] … [22]

Reading of this:

• Exit reason: a clean SIGTERM (graceful uvicorn shutdown), not a Python crash and not an OOM/SIGKILL — there's no traceback, no 137, no OOM line. This is the platform replacing an unhealthy instance, i.e. the request wedges the worker, the container stops answering Agent Engine's health/liveness probe, and it gets SIGTERM'd and swapped. The client's in-flight message:send then returns HTTP 400.
• Where it wedges: the incoming request is converted by ADK's native A2A request path (convert_a2a_request_to_agent_run_request → convert_a2a_part_to_genai_part → AgentRunRequest), and then the worker goes silent — before any model turn completes (I never see :rawPredict or an LLM response for the Claude variants, whereas the Gemini control logs its turn and replies). So the break is in the native A2A serving path for the Claude model, consistent with the original report.
• The timeout variant shows the same shape — GET /card 200 → silence → Received SIGTERM, exiting. ~9s later → replacement.

The thinking_budget=0 result also rules out Claude thinking blocks as the trigger (it fails identically), which is a useful elimination since the wedge happens on the request conversion side anyway.

Summary

• Neither a short client timeout nor disabling thinking prevents it.
• The kill is a platform-initiated instance replacement (clean SIGTERM), not an app-level exception — so there's no stack trace to hand over; the informative part is that it's a graceful platform kill after a silent worker wedge in the native A2A request path, with the same agent on Gemini serving fine.

Happy to share the fully self-contained 4-variant repro package if useful.

[surajksharma07]

@BrianAlderman The timeout ask was a dead end, and your log confirms it clearly — kill at ~9s rules out both the 55s and the SDK's 600s default, so it's not a slow or stalled Claude call.
More importantly, there's no rawPredict (or equivalent) logged at all before the silence, and no exception either (both executor implementations catch broadly and would log a TS_FAILED, not go quiet) — so this is a genuine hang between AgentRunRequest being built and the model ever getting dispatched, not a crash and not response latency.

One lead worth checking against your own data: ADK's Claude client is a cached_property, built lazily on first access per agent instance. The installed Anthropic SDK offloads credential resolution to a thread rather than blocking the loop directly, but it's still a one-time cold path per instance (cached after that).
Does the wedge only happen on the very first message to a fresh instance, or on every message?
That'd tell us whether this is a one-off cold-start stall versus something recurring per request, without needing a new deploy.

Still can't reproduce Agent Engine's own container kill locally, so I'd rather have that one answer than guess at a fix from here!

[BrianAlderman]

@surajksharma07 Your cold-path lead was right, and it also explains the container kill. Ran two things on the exact stack (adk 2.3.0 / a2a-sdk 0.3.26 / aiplatform 1.159.0), same minimal no-tool agent.

First-vs-every: I sent 4 sequential message:send calls to one baseline (non-prewarmed) Claude engine — all 4 return 400. Per the container logs, each one is GET /card 200 → the A2A request converter runs → the worker goes silent → a clean SIGTERM + instance replacement (~1 SIGTERM per request), no :rawPredict, no LLM response, no traceback. So it's not "first message only": each wedge replaces the instance, so every message lands on another fresh worker that pays the cold path again and wedges again. It reads as a one-time-per-instance cold-start stall that recurs because the instance never survives to get warm.

Pre-warm test (the decisive one): I deployed an identical agent that, at set_up(), exercises the model's cached Anthropic client once — in a dedicated thread (its own loop), off the request event loop — before taking traffic:

def build_executor(**_):
    prewarm_model(root_agent.model)   # threaded, one-shot messages.create(max_tokens=4)
    return build_executor_for(root_agent, app_name=APP_NAME)

Result: 3/3 message:send succeed, each with a real reply, Received response from Claude logged per turn, and 0 SIGTERM. The pre-warm cost ~48s of startup (launch → Application startup complete, vs ~1s for the non-prewarmed baseline), i.e. the cold path is real and substantial — it just needs to happen off the serving loop.

Variant (same agent, only this differs)	message:send	model call logged	SIGTERM
baseline Claude	4/4 → 400	none	1 per request
Claude, client pre-warmed off-loop at set_up	3/3 → PASS	yes	none
(earlier) 55s client timeout	→ 400	none	kill at ~9s
(earlier) thinking_budget=0	→ 400	none	1
(earlier) Gemini control	PASS	yes	none

Takeaway: it's exactly the lazy first-access of Claude._anthropic_client you flagged — but the reason it kills the container (rather than just being slow) is that that cold path runs on the request event loop, and blocks it long enough that Agent Engine's liveness probe fails → SIGTERM before the model is ever dispatched. Your note that the SDK offloads credential resolution to a thread still holds; something else on that first-access path (client construction / google.auth default / first httpx setup) is still blocking the loop. Building/warming the Claude client off the event loop before first serve avoids it entirely.

So a plausible ADK-side fix would be to initialize _anthropic_client (and force its first credential/connection setup) off the request loop — e.g. await asyncio.to_thread(...) on first use, or eager async init at agent/runner startup — rather than lazily on the first in-request access.

Happy to share the full repro package.