Repository navigation
fix(stt): detect half-open websockets in OpenAI, ElevenLabs and Inworld STT - #7912
Open
ninglinLiu wants to merge 3 commits into
Open
ninglinLiu wants to merge 3 commits into
ninglinLiu wants to merge 3 commits into
Conversation
…rld STT plugins Add heartbeat=30.0 to ws_connect and handle WSMsgType.ERROR by raising retryable APIConnectionError in OpenAI realtime, ElevenLabs, and Inworld STT plugins. Without heartbeat, a silently dropped socket (half-open TCP connection, no FIN/RST) is never noticed: recv_task parks on ws.receive() forever and the reconnect loop, which is exception-driven, never runs. The heartbeat closes the socket when a ping goes unanswered, and that surfaces as WSMsgType.ERROR. ws.exception() is the only place the reason survives, so we chain it to the APIConnectionError. Add regression tests for each plugin proving they notice a silent websocket via heartbeat and raise retryable APIConnectionError within bounded time. Fixes livekit#7900 Co-authored-by: NingLin Liu <x07514475479@gmail.com>
Run ruff format on 4 files per upstream CI requirements: - livekit-plugins-elevenlabs/livekit/plugins/elevenlabs/stt.py - tests/test_plugin_openai_stt_halfopen.py - tests/test_plugin_elevenlabs_stt_halfopen.py - tests/test_plugin_inworld_stt_halfopen.py Restructured the Inworld test's _stream function to keep the type: ignore[arg-type] comment on the http_session argument line where mypy will apply it, while satisfying ruff format requirements. Co-authored-by: NingLin Liu <x07514475479@gmail.com>
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
|
Ran the probe from #7900 against this branch, and it works:
So it now matches the other fixed plugins at about 45s. I'm happy to follow up with the shared helper and a shorter heartbeat across all the STT plugins if that's the direction you want, so it can land separately from this one. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
If a streaming STT websocket goes half-open (the provider, or a NAT in between, disappears without a FIN/RST), the OpenAI realtime, ElevenLabs and Inworld STT streams wait on
ws.receive()forever. Nothing raises, so the stream never reconnects andstt.FallbackAdapternever fails over; the agent just stops hearing the user (#7900).Deepgram, Soniox, Telnyx, xAI and Simplismart were already fixed the same way (#7206, #7357, #7359, #7364, #7369).
Fix
Same pattern as those merged fixes, applied to the three remaining plugins:
heartbeat=30.0tows_connectinopenai/stt.py,elevenlabs/stt.pyandinworld/stt.py. aiohttp defaultsheartbeattoNone, so without it an unanswered socket is never noticed.WSMsgType.ERRORin the receive loop as a retryableAPIConnectionError, chained fromws.exception(). That is where aiohttp reports the missed pong. It already returns quietly when we are closing the socket ourselves. Before this change, OpenAI/ElevenLabs logged it as an "unexpected message type" and continued, and Inworld logged it and returned without triggering a reconnect.I kept the change inline per plugin, the same way as the existing fixes, and kept the 30s interval they use. The issue asks whether a shared helper in
livekit.agents.utilsand/or a shorter interval would be better. I'm happy to rework this PR either way if you prefer.Tests
New unit tests (one file per plugin,
pytest.mark.unit) using a fake aiohttp session/socket:tests/test_plugin_openai_stt_halfopen.pytests/test_plugin_elevenlabs_stt_halfopen.pytests/test_plugin_inworld_stt_halfopen.pyEach checks that the socket is opened with
heartbeat=30.0, and that a heartbeat-timeoutERRORframe ends the receive loop and leads to a reconnect without spinning.Fixes #7900