Bug Report: ElevenLabs TTS retries NOT handled correctly

[zaheerabbas-prodigal]

Bug Description

The retry of elevenlabs is not properly handled in the cases of websocket closure or API connection timeouts.

When the websocket closes unexpectedly - the next retries fail too because of:

1. The retry logic runs again but doesn’t actually resend the text. The input channel was already drained on the first attempt, and the plugin’s per-attempt state (tokenizer stream, context id, alignment buffers) isn’t reset. So the second attempt connects but sends nothing, produces no audio, and the code raises “no audio frames were pushed for text …” instead of recovering
2. Code reference - https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/tts/tts.py#L405C1-L408C1. The pushed text is checked again in retried attempts even though there was no text sent to 11labs after websocket was reconnected.

When the API connection timesout - the next retries fail too because of:

1. The input/context state doesn't get rebuilt in the next try, nothing new gets sent on the next attempts. So every attempt just waits on waiter, hits the 10s inactivity timeout, raises, retries, and eventually marks the session unrecoverable.
2. Code refence - https://github.com/livekit/agents/blob/main/livekit-plugins/livekit-plugins-elevenlabs/livekit/plugins/elevenlabs/tts.py#L591C1-L593C1. The connection is timedout again without pushing any new text after the retry as it doesn't reset the state.

Expected Behavior

Both the websocket disconnection and API connection timeout should be handled correctly

Reproduction Steps

1. Don't have exact code to reproduce. Have attached logs for both scenarios in the context section

Operating System

Docker, k8s

Models Used

11labs, turbo-v2.5

Package Versions

livekit==1.0.16
livekit-agents==1.2.17
livekit-api==1.0.7
livekit-blingfire==1.0.0
livekit-plugins-anthropic==1.2.17
livekit-plugins-cartesia==1.2.17
livekit-plugins-deepgram==1.2.17
livekit-plugins-elevenlabs==1.2.17
livekit-plugins-google==1.2.17
livekit-plugins-noise-cancellation==0.2.5
livekit-plugins-silero==1.2.17
livekit-plugins-turn-detector==1.2.17
livekit-protocol==1.0.4

Session/Room/Call IDs

Elevenlabs Webscoket issue:

1. Room ID - RM_9S8ptchQf5tp

Elevenlabs API connection timeout:

1. Room ID - RM_uNMshkain7MV

Don't think these will help but just sharing. The additional context has required logs to debug further.

Proposed Solution

Additional Context

Logs for websocket disconnection issue:

{"levelname": "WARNING", "name": "livekit.plugins.elevenlabs", "process": 41718, "event": null, "message": "websocket closed unexpectedly", "pathname": "/usr/local/lib/python3.11/site-packages/livekit/plugins/elevenlabs/tts.py", "lineno": 622, "level": "WARNING", "timestamp": "2025-12-01T16:38:51.656777+00:00"}
{"message": "failed to synthesize speech, retrying in 0.1s\nTraceback (most recent call last):\n File "/usr/local/lib/python3.11/site-packages/livekit/agents/tts/tts.py", line 383, in _main_task\n await self._run(output_emitter)\n File "/usr/local/lib/python3.11/site-packages/livekit/plugins/elevenlabs/tts.py", line 440, in run\n raise e\n File "/usr/local/lib/python3.11/site-packages/livekit/plugins/elevenlabs/tts.py", line 435, in run\n await waiter\nlivekit.agents.exceptions.APIStatusError: connection closed (status_code=-1, request_id=None, body=None, retryable=True)", "level": "WARNING", "name": "livekit.agents", "tts": "livekit.plugins.elevenlabs.tts.TTS", "attempt": 1, "streamed": true, "pid": 41718, "job_id": "AJ_dcqWqSSxiYUW", "timestamp": "2025-12-01T16:38:51.659237+00:00"}
{"message": "failed to synthesize speech, retrying in 2.0s\nTraceback (most recent call last):\n File "/usr/local/lib/python3.11/site-packages/livekit/agents/tts/tts.py", line 394, in main_task\n raise APIError(f"no audio frames were pushed for text: {self.pushed_text}")\nlivekit.agents.exceptions.APIError: no audio frames were pushed for text: I'm sorry, I didn't catch that. Could you please provide your **REDACTED***. (body=None, retryable=True)", "level": "WARNING", "name": "livekit.agents", "tts": "livekit.plugins.elevenlabs.tts.TTS", "attempt": 2, "streamed": true, "pid": 41718, "job_id": "AJ_dcqWqSSxiYUW", "timestamp": "2025-12-01T16:38:51.843954+00:00"}
{"message": "failed to synthesize speech, retrying in 2.0s\nTraceback (most recent call last):\n File "/usr/local/lib/python3.11/site-packages/livekit/agents/tts/tts.py", line 394, in main_task\n raise APIError(f"no audio frames were pushed for text: {self.pushed_text}")\nlivekit.agents.exceptions.APIError: no audio frames were pushed for text:I'm sorry, I didn't catch that. Could you please provide your **REDACTED***. (body=None, retryable=True)", "level": "WARNING", "name": "livekit.agents", "tts": "livekit.plugins.elevenlabs.tts.TTS", "attempt": 3, "streamed": true, "pid": 41718, "job_id": "AJ_dcqWqSSxiYUW", "timestamp": "2025-12-01T16:38:53.870520+00:00"}
{"message": "Error in _tts_inference_task\nTraceback (most recent call last):\n File "/usr/local/lib/python3.11/site-packages/livekit/agents/tts/tts.py", line 556, in anext\n val = await self._event_aiter.anext()\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nStopAsyncIteration\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/usr/local/lib/python3.11/site-packages/livekit/agents/utils/log.py", line 16, in async_fn_logs\n return await fn(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/site-packages/opentelemetry/util/_decorator.py", line 71, in async_wrapper\n return await func(*args, **kwargs) # type: ignore\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/site-packages/livekit/agents/voice/generation.py", line 222, in _tts_inference_task\n async for audio_frame in tts_node:\n File "/usr/local/lib/python3.11/site-packages/livekit/agents/voice/agent.py", line 432, in tts_node\n async for ev in stream:\n File "/usr/local/lib/python3.11/site-packages/livekit/agents/tts/tts.py", line 559, in anext\n raise exc # noqa: B904\n ^^^^^^^^^\n File "/usr/local/lib/python3.11/site-packages/opentelemetry/util/_decorator.py", line 71, in async_wrapper\n return await func(*args, **kwargs) # type: ignore\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/site-packages/livekit/agents/tts/tts.py", line 394, in _main_task\n raise APIError(f"no audio frames were pushed for text: {self._pushed_text}")\nlivekit.agents._exceptions.APIError: no audio frames were pushed for text: I'm sorry, I didn't catch that. Could you please provide your **REDACTED***. (body=None, retryable=True)", "level": "ERROR", "name": "livekit.agents", "pid": 41718, "job_id": "AJ_dcqWqSSxiYUW", "timestamp": "2025-12-01T16:38:55.903670+00:00"}

Logs for API Timeout issue:

{"message": "failed to synthesize speech, retrying in 0.1s\nTraceback (most recent call last):\n File "/usr/local/lib/python3.11/site-packages/livekit/plugins/elevenlabs/tts.py", line 435, in _run\n await waiter\nlivekit.agents._exceptions.APITimeoutError: 11labs tts timed out after 10.0 seconds (body=None, retryable=True)\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "/usr/local/lib/python3.11/site-packages/livekit/agents/tts/tts.py", line 383, in _main_task\n await self._run(output_emitter)\n File "/usr/local/lib/python3.11/site-packages/livekit/plugins/elevenlabs/tts.py", line 441, in _run\n raise APIStatusError("Could not synthesize") from e\nlivekit.agents.exceptions.APIStatusError: Could not synthesize (status_code=-1, request_id=None, body=None, retryable=True)", "level": "WARNING", "name": "livekit.agents", "tts": "livekit.plugins.elevenlabs.tts.TTS", "attempt": 1, "streamed": true, "pid": 30844, "job_id": "AJ_5Q9Q9Sgvq5H3", "timestamp": "2025-12-01T15:54:36.712546+00:00"}
{"message": "failed to synthesize speech, retrying in 2.0s\nTraceback (most recent call last):\n File "/usr/local/lib/python3.11/site-packages/livekit/plugins/elevenlabs/tts.py", line 435, in _run\n await waiter\nlivekit.agents._exceptions.APITimeoutError: 11labs tts timed out after 10.0 seconds (body=None, retryable=True)\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "/usr/local/lib/python3.11/site-packages/livekit/agents/tts/tts.py", line 383, in _main_task\n await self._run(output_emitter)\n File "/usr/local/lib/python3.11/site-packages/livekit/plugins/elevenlabs/tts.py", line 441, in _run\n raise APIStatusError("Could not synthesize") from e\nlivekit.agents.exceptions.APIStatusError: Could not synthesize (status_code=-1, request_id=None, body=None, retryable=True)", "level": "WARNING", "name": "livekit.agents", "tts": "livekit.plugins.elevenlabs.tts.TTS", "attempt": 2, "streamed": true, "pid": 30844, "job_id": "AJ_5Q9Q9Sgvq5H3", "timestamp": "2025-12-01T15:54:46.815912+00:00"}
{"message": "failed to synthesize speech, retrying in 2.0s\nTraceback (most recent call last):\n File "/usr/local/lib/python3.11/site-packages/livekit/plugins/elevenlabs/tts.py", line 435, in _run\n await waiter\nlivekit.agents._exceptions.APITimeoutError: 11labs tts timed out after 10.0 seconds (body=None, retryable=True)\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "/usr/local/lib/python3.11/site-packages/livekit/agents/tts/tts.py", line 383, in _main_task\n await self._run(output_emitter)\n File "/usr/local/lib/python3.11/site-packages/livekit/plugins/elevenlabs/tts.py", line 441, in run\n raise APIStatusError("Could not synthesize") from e\nlivekit.agents.exceptions.APIStatusError: Could not synthesize (status_code=-1, request_id=None, body=None, retryable=True)", "level": "WARNING", "name": "livekit.agents", "tts": "livekit.plugins.elevenlabs.tts.TTS", "attempt": 3, "streamed": true, "pid": 30844, "job_id": "AJ_5Q9Q9Sgvq5H3", "timestamp": "2025-12-01T15:54:58.818542+00:00"}
{"message": "Error in _tts_inference_task\nTraceback (most recent call last):\n File "/usr/local/lib/python3.11/site-packages/livekit/plugins/elevenlabs/tts.py", line 435, in _run\n await waiter\nlivekit.agents._exceptions.APITimeoutError: 11labs tts timed out after 10.0 seconds (body=None, retryable=True)\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "/usr/local/lib/python3.11/site-packages/livekit/agents/utils/log.py", line 16, in async_fn_logs\n return await fn(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/site-packages/opentelemetry/util/_decorator.py", line 71, in async_wrapper\n return await func(*args, **kwargs) # type: ignore\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/site-packages/livekit/agents/voice/generation.py", line 222, in _tts_inference_task\n async for audio_frame in tts_node:\n File "/usr/local/lib/python3.11/site-packages/livekit/agents/voice/agent.py", line 432, in tts_node\n async for ev in stream:\n File "/usr/local/lib/python3.11/site-packages/livekit/agents/tts/tts.py", line 559, in anext\n raise exc # noqa: B904\n ^^^^^^^^^\n File "/usr/local/lib/python3.11/site-packages/opentelemetry/util/_decorator.py", line 71, in async_wrapper\n return await func(*args, **kwargs) # type: ignore\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/site-packages/livekit/agents/tts/tts.py", line 383, in _main_task\n await self._run(output_emitter)\n File "/usr/local/lib/python3.11/site-packages/livekit/plugins/elevenlabs/tts.py", line 441, in _run\n raise APIStatusError("Could not synthesize") from e\nlivekit.agents._exceptions.APIStatusError: Could not synthesize (status_code=-1, request_id=None, body=None, retryable=True)", "level": "ERROR", "name": "livekit.agents", "pid": 30844, "job_id": "AJ_5Q9Q9Sgvq5H3", "timestamp": "2025-12-01T15:55:10.822993+00:00"}

Screenshots and Recordings

No response