Skip to content

fix(stt): notice a half-open websocket in the openai, elevenlabs and inworld streams - #7913

Open
Rehansanjay wants to merge 2 commits into
livekit:mainfrom
Rehansanjay:fix/stt-halfopen-heartbeat
Open

Rehansanjay wants to merge 2 commits into
livekit:mainfrom
Rehansanjay:fix/stt-halfopen-heartbeat

Conversation

@Rehansanjay

Copy link
Copy Markdown
Contributor

Fixes #7900. The measurement and the half-open probe in that issue are @DEVJHAWAR11's — this is just the three-plugin fix it asks for. If you were already working on it, say so and I'll close this.

The bug

aiohttp defaults heartbeat to None. A websocket that dies without a FIN/RST — the provider vanishing, or a NAT in between dropping the flow — therefore leaves the receive side parked on ws.receive() forever. Nothing raises, so the stream never reconnects and stt.FallbackAdapter never switches to the next provider. On a call, the agent stops hearing the user and nothing anywhere reports an error.

Deepgram, Soniox, Telnyx, xAI and Simplismart each got this fix already (#7206, #7357, #7359, #7364, #7369). These three were missed. The issue measured all four against a server that accepts the upgrade and then answers nothing, pings included:

plugin time to notice
openai STT (realtime) still waiting after 75s
elevenlabs STT still waiting after 75s
inworld STT still waiting after 75s
deepgram STT (fixed in #7206) APIConnectionError after 45.4s

The fix

heartbeat=30.0 on the three ws_connect calls, matching the value the already-fixed plugins use. An unanswered ping makes aiohttp close the socket, which surfaces in each plugin's receive loop and takes the existing reconnect / failover path — so nothing else needed changing.

A plain read timeout is not available here: a quiet user is the normal state for STT, as #5717 points out. That leaves the ping/pong heartbeat as the signal.

Tests

One per plugin, asserting the socket is opened with the heartbeat, following the pattern already in tests/test_plugin_soniox_stt.py::test_socket_is_opened_with_a_heartbeat:

  • tests/test_plugin_openai_websocket_urls.py — extends the fake session already there to record kwargs
  • tests/test_plugin_elevenlabs_stt.py — uses the file's own _new_stream
  • tests/test_plugin_inworld_stt.py — new file; inworld had no STT test module

Mutation-verified. Removing the heartbeat from each plugin, with everything else unchanged, fails that plugin's test:

E  AssertionError: assert None == 30.0
E   +  where {'headers': {'Authorization': 'Basic test-key'}}.get
E  AssertionError: assert None == 30.0
E   +  where {'headers': {'xi-api-key': 'test-key'}}.get
E  AssertionError: assert None == 30.0

116 tests pass across the three files.

Deliberately not in this PR

#7900 also makes the case that 45s is a long time to be deaf before failover, and suggests a shorter window. I agree, but shortening it means changing the five plugins that already shipped 30.0 plus these three, and picking a number is a policy call rather than a bug fix — livekit-plugins-microsoft-ai already uses 10.0 and livekit-plugins-meta uses a module constant, so there is no single existing convention to follow. Happy to send that separately if you have a preferred value.

…inworld streams

aiohttp defaults `heartbeat` to None. A websocket that dies without a FIN/RST --
the provider vanishing, or a NAT in between dropping the flow -- therefore
leaves the receive side parked on `ws.receive()` forever. Nothing raises, so the
stream never reconnects and `stt.FallbackAdapter` never switches to the next
provider: on a call the agent simply stops hearing the user, with no error
anywhere.

Deepgram, Soniox, Telnyx, xAI and Simplismart each got this fix already (livekit#7206,
livekit#7357, livekit#7359, livekit#7364, livekit#7369). These three were missed. livekit#7900 measured all four:
the fixed plugins raise after about 45s, while openai, elevenlabs and inworld
were still waiting after 75s.

A read timeout is not available here, since a quiet user is the normal state for
STT (livekit#5717), which leaves the ping/pong heartbeat as the signal. 30s matches
what the already-fixed plugins use.

One test per plugin asserts the socket is opened with the heartbeat, following
the pattern in tests/test_plugin_soniox_stt.py. livekit#7900 also makes the case that
45s is a long time to go deaf before failover; shortening the window everywhere
is a separate change from closing the gap in three plugins.

Fixes livekit#7900
devin-ai-integration[bot]

This comment was marked as resolved.

… quietly

The heartbeat added in the previous commit closes a dead socket, but on its own
that is only half the fix: aiohttp delivers the failure as an ERROR frame rather
than as a close frame, and all three plugins stepped over it.

openai and elevenlabs let it fall through to their unexpected-message-type
branch, which logs and continues, so the receive loop simply ended. inworld
logged it and returned. In each case `_run` then saw a receive task that had
completed normally, broke out of its loop instead of reconnecting, and returned
without raising -- and `stt.SpeechStream` only reports a failure when `_run`
raises. So the socket was correctly detected as dead and the stream still died
silently, with no error for a FallbackAdapter to react to.

All three already raise on an unexpected *close*. An ERROR frame now takes the
same path, keeping the cause from `ws.exception()`, which is where it only
survives. Intentional shutdown is unaffected: the existing `closing_ws` guards
still return, as they do for a close.

This is the pairing livekit-plugins-soniox already has, and livekit#7900's expected
behaviour: a streaming STT whose socket goes silent should raise a retryable
APIConnectionError so the stream reconnects or the FallbackAdapter moves on.

Raised by the automated review on the PR, which was right about inworld; the
same hole was in the other two in a different shape.
@Rehansanjay

Copy link
Copy Markdown
Contributor Author

This flag is right, and it is right about more than inworld — the heartbeat on its own was half a fix. Pushed the other half.

aiohttp delivers a heartbeat failure as an ERROR frame, not a close frame, and all three plugins stepped over it:

  • inworld logged it and returned.
  • openai and elevenlabs let it fall through to their unexpected-message-type branch, which logs and continues, so the receive loop just ended.

All three _run loops then saw a receive task that completed normally, hit if wait_reconnect_task not in done: break, and returned without raising — and stt.SpeechStream only reports a failure when _run raises. So the socket was correctly detected as dead and the stream still died quietly, which is exactly the symptom #7900 is about. The heartbeat alone moved the failure from "never noticed" to "noticed and discarded".

All three already raise on an unexpected close, so an ERROR frame now takes the same path and keeps the cause from ws.exception(), which is the only place it survives. The existing closing_ws guards still return on intentional shutdown, as they do for a close. This is the pairing livekit-plugins-soniox already has — its own comment there says treating ERROR as an unexpected message type "stepped over it and discarded the reason", which is precisely what these three were still doing.

Test added for inworld: an ERROR frame through _recv_messages_task must raise APIConnectionError. Mutation-verified — restoring the return gives DID NOT RAISE APIConnectionError.

One honest limitation: openai's and elevenlabs' receive loops are closures inside _run, so they are not reachable by the test patterns in this repo the way inworld's method is. Their change mirrors the close branch immediately above it in the same function. If you would rather have those covered too, the way to do it is to lift each receive loop to a method, and I will if you want it — it is a bigger diff than the fix.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Streaming STT: openai, elevenlabs and inworld never notice a silently dropped websocket, and the fixed plugins take ~45s

1 participant