speaches: openai.APIConnectionError out of the box

[Splizard]

Prints a series of connection errors, unsure what is going wrong here.

harbor.speaches  | Traceback (most recent call last):
harbor.speaches  |   File "/home/ubuntu/speaches/.venv/lib/python3.12/site-packages/gradio/queueing.py", line 625, in process_events
harbor.speaches  |     response = await route_utils.call_process_api(
harbor.speaches  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
harbor.speaches  |   File "/home/ubuntu/speaches/.venv/lib/python3.12/site-packages/gradio/route_utils.py", line 322, in call_process_api
harbor.speaches  |     output = await app.get_blocks().process_api(
harbor.speaches  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
harbor.speaches  |   File "/home/ubuntu/speaches/.venv/lib/python3.12/site-packages/gradio/blocks.py", line 2042, in process_api
harbor.speaches  |     result = await self.call_function(
harbor.speaches  |              ^^^^^^^^^^^^^^^^^^^^^^^^^
harbor.speaches  |   File "/home/ubuntu/speaches/.venv/lib/python3.12/site-packages/gradio/blocks.py", line 1587, in call_function
harbor.speaches  |     prediction = await fn(*processed_input)
harbor.speaches  |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
harbor.speaches  |   File "/home/ubuntu/speaches/.venv/lib/python3.12/site-packages/gradio/utils.py", line 850, in async_wrapper
harbor.speaches  |     response = await f(*args, **kwargs)
harbor.speaches  |                ^^^^^^^^^^^^^^^^^^^^^^^^
harbor.speaches  |   File "/home/ubuntu/speaches/src/speaches/ui/tabs/tts.py", line 43, in handle_audio_speech
harbor.speaches  |     res = await openai_client.audio.speech.create(
harbor.speaches  |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
harbor.speaches  |   File "/home/ubuntu/speaches/.venv/lib/python3.12/site-packages/openai/resources/audio/speech.py", line 180, in create
harbor.speaches  |     return await self._post(
harbor.speaches  |            ^^^^^^^^^^^^^^^^^
harbor.speaches  |   File "/home/ubuntu/speaches/.venv/lib/python3.12/site-packages/openai/_base_client.py", line 1856, in post
harbor.speaches  |     return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
harbor.speaches  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
harbor.speaches  |   File "/home/ubuntu/speaches/.venv/lib/python3.12/site-packages/openai/_base_client.py", line 1550, in request
harbor.speaches  |     return await self._request(
harbor.speaches  |            ^^^^^^^^^^^^^^^^^^^^
harbor.speaches  |   File "/home/ubuntu/speaches/.venv/lib/python3.12/site-packages/openai/_base_client.py", line 1613, in _request
harbor.speaches  |     return await self._retry_request(
harbor.speaches  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
harbor.speaches  |   File "/home/ubuntu/speaches/.venv/lib/python3.12/site-packages/openai/_base_client.py", line 1683, in _retry_request
harbor.speaches  |     return await self._request(
harbor.speaches  |            ^^^^^^^^^^^^^^^^^^^^
harbor.speaches  |   File "/home/ubuntu/speaches/.venv/lib/python3.12/site-packages/openai/_base_client.py", line 1613, in _request
harbor.speaches  |     return await self._retry_request(
harbor.speaches  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
harbor.speaches  |   File "/home/ubuntu/speaches/.venv/lib/python3.12/site-packages/openai/_base_client.py", line 1683, in _retry_request
harbor.speaches  |     return await self._request(
harbor.speaches  |            ^^^^^^^^^^^^^^^^^^^^
harbor.speaches  |   File "/home/ubuntu/speaches/.venv/lib/python3.12/site-packages/openai/_base_client.py", line 1623, in _request
harbor.speaches  |     raise APIConnectionError(request=request) from err
harbor.speaches  | openai.APIConnectionError: Connection error.

this occurs both after going to the web dashboard, as well as when used via API from webui.

[ackalker]

harbor seems to be applying two drop-in files with fixes to the speaches container, but these seem to be related to an older version of speaches, development has moved on since, including changes to the project's directory structure. @av Could you please have a look at this, and perhaps tell us which version of speaches you used for testing?

In the meantime, I would suggest using both tts and stt services (based on faster-whisper-server) with harbor. Although they're marked as deprecated and superceded by speaches, at least they work well together with webui: I can use them for 'voice calls' (turn by turn speaking and listening) without any issues.

[ackalker]

I've found a way to get speaches working (I might try and make a PR if @av doesn't beat me to it):

• Stop all harbor services, just to be sure: harbor down.
• In compose.speaches.yml, comment out both lines with the following content:

      - ./speaches/hf_utils.py:/home/ubuntu/speaches/speaches/hf_utils.py
      - ./speaches/kokoro_utils.py:/home/ubuntu/speaches/speaches/kokoro_utils.py

• Restart harbor with speaches: harbor up speaches (or harbor up if you have added speaches to your profile defaults).
• Download the models you plan to use (contrary to what I remember reading in the documentation, speaches doesn't download any models automatically.) Note: the speaches service must be running for this to work.

# `harbor url speaches` doesn't seem to work, or I don't know how to use it correctly,
# so we need to construct the URL manually

# Used by `speaches-cli`
export SPEACHES_BASE_URL=http://localhost:$(harbor config get speaches.host.port)

# Choose which models to use for automatic speech recognition (speech-to-text)
# and for text-to-speech
uvx speaches-cli repository ls --task=automatic-speech-recognition
uvx speaches-cli repository ls --task=text-to-speech

# Download the STT and TTS models (replace with your choices from above)
uvx speaches-cli model download Systran/faster-whisper-large-v3
uvx speaches-cli model download speaches-ai/Kokoro-82M-v1.0-ONNX

# Configure `harbor` to use the downloaded models (replace with your choices from above)
harbor speaches stt_model Systran/faster-whisper-large-v3
harbor speaches tts_model speaches-ai/Kokoro-82M-v1.0-ONNX
unset SPEACHES_BASE_URL

• Restart harbor for the configuration changes to be picked up.

harbor restart speaches

I've tested this setup with harbor linked from its cloned Git repository, all other settings at their default values, and it seems to work well.

[realnikolaj]

Details

I've found a way to get `speaches` working (I might try and make a PR if [@av](https://github.com/av) doesn't beat me to it):
* Stop all `harbor` services, just to be sure: `harbor down`.
* In `compose.speaches.yml`, comment out both lines with the following content:

- ./speaches/hf_utils.py:/home/ubuntu/speaches/speaches/hf_utils.py
- ./speaches/kokoro_utils.py:/home/ubuntu/speaches/speaches/kokoro_utils.py
* Restart `harbor` with `speaches`: `harbor up speaches` (or `harbor up` if you have added `speaches` to your profile defaults).
* Download the models you plan to use
(contrary to what I remember reading in the documentation, `speaches` doesn't download any models automatically.)
Note: the `speaches` service _must_ be running for this to work.

# `harbor url speaches` doesn't seem to work, or I don't know how to use it correctly,
# so we need to construct the URL manually

# Used by `speaches-cli`
export SPEACHES_BASE_URL=http://localhost:$(harbor config get speaches.host.port)

# Choose which models to use for automatic speech recognition (speech-to-text)
# and for text-to-speech
uvx speaches-cli repository ls --task=automatic-speech-recognition
uvx speaches-cli repository ls --task=text-to-speech >
# Download the STT and TTS models (replace with your choices from above)

.. uvx speaches-cli model download Systran/faster-whisper-large-v3 uvx speaches-cli model download speaches-ai/Kokoro-82M-v1.0-ONNX

Configure harbor to use the downloaded models (replace with your choices from above)

harbor speaches stt_model Systran/faster-whisper-large-v3 harbor speaches tts_model speaches-ai/Kokoro-82M-v1.0-ONNX unset SPEACHES_BASE_URL

• Restart harbor for the configuration changes to be picked up. harbor restart speaches I've tested this setup with harbor linked from its cloned Git repository, all other settings at their default values, and it seems to work well.

@ackalker this also works for me, but did you actually experience the error reported in this issue?

I've experienced it and also found that at least the "kokoro_utils.py" drop in is no longer needed, I've also commented out the hf_utils.py and it appears to work with the native speaches cli except for the fact that models downloaded using the built-in speaches-cli gets the wrong permissions so I have to do chown -R 1000:1000 <path-to-file-with-incorrect-owner:group>*

But for this topic what did it for me was using the .harbor/speaches/.env and setting (my current version**) these:

/ .harbor/speaches/.env

CHAT_COMPLETION_BASE_URL=http://ollama:11434/v1
CHAT_COMPLETION_API_KEY=<openai-1st-key>
LOG_LEVEL=DEBUG
LOOPBACK_HOST_URL=http://speaches:8000
ALLOW_ORIGINS=["*"]

**The ALLOW_ORIGINS one is what I think did it for me but below I also provide my harbor.speaches variables:

:> h config ls | grep -i speaches
OPENAI_URLS                    http://ollama:11434/v1;http://speaches:8000/v1
OPENAI_KEYS                    <openai-1st-key>;<openai-2nd-key>
SPEACHES_HOST_PORT             34331
SPEACHES_VERSION               latest
SPEACHES_STT_MODEL             Systran/faster-whisper-large-v2
SPEACHES_TTS_MODEL             speaches-ai/Kokoro-82M-v1.0-ONNX
SPEACHES_TTS_VOICE             zf_xiaobei
SPEACHES_API_KEY               <openai-2nd-key>

Note that I'm using Systran/faster-whisper-large-v2 because it acutally "works" with Chinese, but that specific Kokoro above actually doesn't and Kokoro is there because I'm still testing more stuff, and my point is just that if anyone cares v3 of faster-whisper-large does not support Chinese.

*I'm using something like this command to find dirs with incorrect owner:group: exa -l ~/data/.cache/huggingface/hub/ --group --numeric --recurse --level 5 --only-dirs | grep -v '1000 1000'

[av]

Thanks to OP for the report and for the suggested workarounds!

Fixing this in v0.3.19 (to be released very soon)

• Removed Harbor's patches as no longer needed
• New version no longer has auto-download behavior (was a side-product in a previous one too), so added a custom entrypoint that'll download pre-configured STT and TTS models to keep this convenient behavior
• New version has issues with Systran's distill whisper large v3, it's not listed at /v1/models after download due to some reason, so had to change the default
• New version adresses Kokoro versions differently, so had to change the default as well

Since defaults have changed, you might need to manually update speaches.tts_model and speaches.stt_model after update.

[av]

Ok, fixes are out in v0.3.19

To update config to the new defaults (after an update itself)

harbor config set speaches.tts_model $(harbor config get --env-file ./profiles/default.env speaches.tts_model)
harbor config set speaches.stt_model $(harbor config get --env-file ./profiles/default.env speaches.stt_model)