Repository navigation
fix(stt): notice a half-open websocket in the openai, elevenlabs and inworld streams - #7913
Rehansanjay wants to merge 2 commits into
Conversation
…inworld streams aiohttp defaults `heartbeat` to None. A websocket that dies without a FIN/RST -- the provider vanishing, or a NAT in between dropping the flow -- therefore leaves the receive side parked on `ws.receive()` forever. Nothing raises, so the stream never reconnects and `stt.FallbackAdapter` never switches to the next provider: on a call the agent simply stops hearing the user, with no error anywhere. Deepgram, Soniox, Telnyx, xAI and Simplismart each got this fix already (livekit#7206, livekit#7357, livekit#7359, livekit#7364, livekit#7369). These three were missed. livekit#7900 measured all four: the fixed plugins raise after about 45s, while openai, elevenlabs and inworld were still waiting after 75s. A read timeout is not available here, since a quiet user is the normal state for STT (livekit#5717), which leaves the ping/pong heartbeat as the signal. 30s matches what the already-fixed plugins use. One test per plugin asserts the socket is opened with the heartbeat, following the pattern in tests/test_plugin_soniox_stt.py. livekit#7900 also makes the case that 45s is a long time to go deaf before failover; shortening the window everywhere is a separate change from closing the gap in three plugins. Fixes livekit#7900
… quietly The heartbeat added in the previous commit closes a dead socket, but on its own that is only half the fix: aiohttp delivers the failure as an ERROR frame rather than as a close frame, and all three plugins stepped over it. openai and elevenlabs let it fall through to their unexpected-message-type branch, which logs and continues, so the receive loop simply ended. inworld logged it and returned. In each case `_run` then saw a receive task that had completed normally, broke out of its loop instead of reconnecting, and returned without raising -- and `stt.SpeechStream` only reports a failure when `_run` raises. So the socket was correctly detected as dead and the stream still died silently, with no error for a FallbackAdapter to react to. All three already raise on an unexpected *close*. An ERROR frame now takes the same path, keeping the cause from `ws.exception()`, which is where it only survives. Intentional shutdown is unaffected: the existing `closing_ws` guards still return, as they do for a close. This is the pairing livekit-plugins-soniox already has, and livekit#7900's expected behaviour: a streaming STT whose socket goes silent should raise a retryable APIConnectionError so the stream reconnects or the FallbackAdapter moves on. Raised by the automated review on the PR, which was right about inworld; the same hole was in the other two in a different shape.
|
This flag is right, and it is right about more than inworld — the heartbeat on its own was half a fix. Pushed the other half. aiohttp delivers a heartbeat failure as an ERROR frame, not a close frame, and all three plugins stepped over it:
All three All three already raise on an unexpected close, so an ERROR frame now takes the same path and keeps the cause from Test added for inworld: an ERROR frame through One honest limitation: openai's and elevenlabs' receive loops are closures inside |
Fixes #7900. The measurement and the half-open probe in that issue are @DEVJHAWAR11's — this is just the three-plugin fix it asks for. If you were already working on it, say so and I'll close this.
The bug
aiohttpdefaultsheartbeattoNone. A websocket that dies without a FIN/RST — the provider vanishing, or a NAT in between dropping the flow — therefore leaves the receive side parked onws.receive()forever. Nothing raises, so the stream never reconnects andstt.FallbackAdapternever switches to the next provider. On a call, the agent stops hearing the user and nothing anywhere reports an error.Deepgram, Soniox, Telnyx, xAI and Simplismart each got this fix already (#7206, #7357, #7359, #7364, #7369). These three were missed. The issue measured all four against a server that accepts the upgrade and then answers nothing, pings included:
APIConnectionErrorafter 45.4sThe fix
heartbeat=30.0on the threews_connectcalls, matching the value the already-fixed plugins use. An unanswered ping makes aiohttp close the socket, which surfaces in each plugin's receive loop and takes the existing reconnect / failover path — so nothing else needed changing.A plain read timeout is not available here: a quiet user is the normal state for STT, as #5717 points out. That leaves the ping/pong heartbeat as the signal.
Tests
One per plugin, asserting the socket is opened with the heartbeat, following the pattern already in
tests/test_plugin_soniox_stt.py::test_socket_is_opened_with_a_heartbeat:tests/test_plugin_openai_websocket_urls.py— extends the fake session already there to record kwargstests/test_plugin_elevenlabs_stt.py— uses the file's own_new_streamtests/test_plugin_inworld_stt.py— new file; inworld had no STT test moduleMutation-verified. Removing the heartbeat from each plugin, with everything else unchanged, fails that plugin's test:
116 tests pass across the three files.
Deliberately not in this PR
#7900 also makes the case that 45s is a long time to be deaf before failover, and suggests a shorter window. I agree, but shortening it means changing the five plugins that already shipped
30.0plus these three, and picking a number is a policy call rather than a bug fix —livekit-plugins-microsoft-aialready uses10.0andlivekit-plugins-metauses a module constant, so there is no single existing convention to follow. Happy to send that separately if you have a preferred value.