idle_timeout kills long in-flight turns despite active turn_liveness

[rmichelena]

Summary

During a long in-flight ACP turn, buzz-acp can emit turn_liveness observer frames every ~10s (so Desktop activity shows the turn as alive) while simultaneously counting down BUZZ_ACP_IDLE_TIMEOUT based only on silence on the agent subprocess stdout. When stdout goes silent longer than idle_timeout, the harness kills the turn, respawns the agent subprocess (new ACP session — prior session context orphaned), and surfaces a red activity error: idle_timeout: Agent session timed out due to inactivity.

This produces contradictory signals: Desktop sees liveness pings; the harness treats the turn as dead.

Expected behavior

While a turn is in-flight and turn_liveness is actively emitting (or equivalently: while read_until_response_with_idle_timeout is waiting on a prompt response), idle_timeout should not fire solely because stdout is silent — OR turn_liveness should not imply the turn is healthy if stdout silence will kill it.

At minimum, document clearly that:

• BUZZ_ACP_IDLE_TIMEOUT resets only on valid JSON lines on agent stdout (acp_read)
• turn_liveness / kind 24200 frames do not reset idle
• buzz messages send / channel publish does not reset idle

Actual behavior (production incident, 2026-08-25)

Harness: buzz-acp → omp acp (Globis agent, VPS bots)
Config: BUZZ_ACP_IDLE_TIMEOUT=120 (inherited from an earlier OpenClaw anti-churn tuning; default in code is 900s)
Symptom: ~30 min visible work in Desktop activity; OMP session map showed active turn at 341k context on openrouter/stealth/ox-alpha, then sudden respawn — old session ctx 0, new session with fresh context. Agent had not yet published to channel.

Journal (bots, buzz-globis):

Aug 25 20:24:48 bots buzz-globis[2036161]: WARN buzz_acp::acp: idle timeout (120s) — no agent activity
Aug 25 20:24:48 bots buzz-globis[2036161]: WARN buzz_acp: agent_returned — respawning agent=0 outcome="idle_timeout"
Aug 25 20:24:53 bots buzz-globis[2036161]: INFO buzz_acp: respawn complete agent=0

Mitigation applied: raised BUZZ_ACP_IDLE_TIMEOUT to 3600 (< default MAX_TURN_DURATION 7200). Structural fix still needed.

Code pointers

Location	Behavior
crates/buzz-acp/src/config.rs	BUZZ_ACP_IDLE_TIMEOUT default 900s; resets on agent stdout activity
crates/buzz-acp/src/acp.rs	read_until_response_with_idle_timeout: idle deadline reset on each valid JSON stdout line (acp_read); fires IdleTimeout on silence
crates/buzz-acp/src/pool.rs	run_turn_liveness: emits turn_liveness every BUZZ_ACP_TURN_LIVENESS_SECS (default 10) — does not extend idle deadline
crates/buzz-acp/src/lib.rs	On PromptOutcome::Timeout(Idle): respawn agent, emit "Agent session timed out due to inactivity"; batch is requeued but ACP session state is lost

Proposed fix

1. While a prompt turn is in-flight (waiting in read_until_response_with_idle_timeout) and liveness task is running, suspend idle_timeout — or extend idle deadline on each turn_liveness tick (since the harness itself knows the turn is active).

2. Alternatively: stop emitting turn_liveness if stdout silence would soon trigger idle kill (honest signal to Desktop).

3. Document IDLE_TIMEOUT semantics in crates/buzz-acp/README.md (currently mentions 620s default in README vs 900s in code — worth aligning).

Environment

• buzz-acp from block/buzz main (~Aug 2025)
• Agent adapter: OMP (omp acp) — long tool runs can go silent on ACP stdout for extended periods
• Relay: self-hosted, BUZZ_ACP_RELAY_OBSERVER=true

Workaround

Set BUZZ_ACP_IDLE_TIMEOUT high enough for longest expected stdout silence (must remain < BUZZ_ACP_MAX_TURN_DURATION). Not a substitute for aligned liveness/idle semantics.

[rmichelena]

Additional verification (@Bumble, Desktop side)

Confirms harness-side analysis and adds the Desktop contradiction mechanism:

Signal	What counts as "alive"
Idle guard (acp.rs:1563)	Only acp_read — valid JSON line on agent stdout
Desktop activity (activeAgentTurnsStore.ts:427-439)	acp_read ∪ acp_write ∪ turn_liveness

turn_liveness uses the same recordActivity() path as stream activity — comment in Desktop: "same refresh-only path as stream activity". But turn_liveness is a harness ticker (pool.rs:4173-4212) that never consults the subprocess, so it cannot reset idle.

Secondary: blank lines and malformed JSON continue before idle reset (acp.rs:1536-1557) — stdout bytes without valid JSON still count as silence.

Source doc trap: lib.rs:4184 says IdleTimeout returns agent to pool; lib.rs:4096 respawns on Timeout(Idle) — the latter path actually runs (pool.rs:2755).

Not duplicate of #6066 (wants hung-agent detection — inverse problem). Related: #4860, #5183.

Proposed fix direction (Bumble): Prefer Desktop distinguishing liveness ping vs agent output ("alive but no stdout for N min") rather than having turn_liveness suppress idle guard (would make idle_timeout dead letter, inverse of #6066).

Production journal evidence (Globis, BUZZ_ACP_IDLE_TIMEOUT=120):

Aug 25 20:24:48 WARN buzz_acp::acp: idle timeout (120s) — no agent activity
Aug 25 20:24:48 WARN buzz_acp: agent_returned — respawning agent=0 outcome="idle_timeout"
Aug 25 20:24:53 INFO buzz_acp: respawn complete agent=0

[rmichelena]

Floor: idle_timeout must exceed shell tool max (600s)

@Bumble verified the default 900s is designed against buzz-dev-mcp shell cap:

• config.rs:18-27: "900s gives 300s of breathing room above the 600s max shell timeout"
• buzz-dev-mcp/src/shell.rs:17: MAX_TIMEOUT_MS = 600_000 (10 min per shell invocation)

Rule: any harness with shell/MCP tools should set BUZZ_ACP_IDLE_TIMEOUT > 600. Below that, the guard kills legitimate long tool runs (cargo build, test suites, git push with hooks) — not "hung agent" detection.

Globis at 120s was 5× below floor — a substantial task could not complete once any single tool exceeded 2 minutes of ACP stdout silence.

3600 is valid mitigation (< default max_turn 7200) but the invariant + rationale should be documented so values aren't copied without context.

[rmichelena]

Correction (follow-up): the 600s figure is buzz-dev-mcp only (shell.rs), not a universal floor. OMP/OpenClaw/Cursor bring their own tools with their own caps — idle should exceed that harness's single-tool max (+ margin). Without measurement, use ≥900s (upstream default). Workaround 3600 remains valid; the rule wording in ops docs updated accordingly.

[rmichelena]

Measured harness tool caps (OpenClaw, bots VPS 2026-08-25):

Harness	Single-tool max	Recommended idle
buzz-dev-mcp (buzz-agent/sprig)	600s	900s (upstream default)
OMP (omp bash tool)	3600s (default 300s; async jobs clamped)	5400s (1800s margin, < max_turn 7200)
OpenClaw	LLM stalls (historical fix: 120s for ~5s cancel_drain pre-fix)	5400s on bots (next natural restart)

Note: idle=3600 exactly equals OMP bash max — zero margin; 5400 avoids racing a full-length async job.

[rmichelena]

Ops pairing (Bumble): raising idle alone is insufficient — max_turn default 7200 competes with OMP tools up to 3600s. When hard cap fires during long tool silence (recently_active=false, window 60s), batch is dead-lettered (no retry) — worse than idle timeout which requeues.

Recommended OMP/OpenClaw bots: BUZZ_ACP_IDLE_TIMEOUT=5400 + BUZZ_ACP_MAX_TURN_DURATION=14400 so idle remains the primary guard and max_turn is a true backstop.

[rmichelena]

Ops trio (@Bumble): when raising max_turn, also consider BUZZ_ACP_AGENTS (default 1). One slot processes one channel turn at a time — a long productive turn (JSON keeps idle from firing) monopolizes all channels for up to max_turn. With dedup=queue messages wait (not dropped); cross-channel latency is the cost.

Globis multi-channel recommendation: idle 5400 + max_turn 14400 + agents=2.