faster-whisper-server
faster-whisper-server copied to clipboard
fedirz
Docker compose deployment should not rebuild the container + Documentation bugs
According to the docs, one shall download two compose files (base compose + platform specific like cpu or cuda). The compose files themselves are set such that they rebuild the image from scratch https://github.com/speaches-ai/speaches/blob/d954451e1e06d39061f225a49ad617d2ba4d378b/compose.yaml#L5
I am currently using the cuda (none cdi setup)
This process has two problems:
- The
Dockerfile, that is required for this to work, has not been downloaded (according to the docs). Building with the suggested setup therefore fails - One should not require to (re)build the images in the first place, since this project already supplies images via CI/ghcr.
Since I manage my docker stack manually, I have therefore written a strapped down compose service entry (alongside to my existing ones), that tries to build a container from the images supplied via ghcr without any image rebuilding.
services
# Some of my other services before here...
# ...
# Speaches
speaches:
image: ghcr.io/speaches-ai/speaches:latest-cuda
ports:
- 8000:8000
restart: always
hostname: speaches
env_file: speaches.env
environment:
- NVIDIA_VISIBLE_DEVICES=all # GPU Support
- NVIDIA_DRIVER_CAPABILITIES=compute,video,utility # GPU Support
runtime: nvidia # GPU Support
security_opt:
- no-new-privileges
volumes:
- "/mnt/cache/ai/speaches:/home/ubuntu/.cache:rw"
This in principle should now download the image from ghcr and spawn a container. And it indeed does.
However, it is missing crucial dependencies in the container like uvicorn (and even python was not found or in $PATH).
NOTE: The details below are now obsolete, since my compose file had an error, that is now rectified.
2025-02-16T21:38:39.334777+01:00 Debian Docker: ai_speaches_1[1193]: ==========
2025-02-16T21:38:39.334926+01:00 Debian Docker: ai_speaches_1[1193]: == CUDA ==
2025-02-16T21:38:39.335648+01:00 Debian Docker: ai_speaches_1[1193]: ==========
2025-02-16T21:38:39.337736+01:00 Debian Docker: ai_speaches_1[1193]: CUDA Version 12.6.3
2025-02-16T21:38:39.338738+01:00 Debian Docker: ai_speaches_1[1193]: Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2025-02-16T21:38:39.339949+01:00 Debian Docker: ai_speaches_1[1193]: This container image and its contents are governed by the NVIDIA Deep Learning Container License.
2025-02-16T21:38:39.340056+01:00 Debian Docker: ai_speaches_1[1193]: By pulling and using the container, you accept the terms and conditions of this license:
2025-02-16T21:38:39.340105+01:00 Debian Docker: ai_speaches_1[1193]: https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license
2025-02-16T21:38:39.340155+01:00 Debian Docker: ai_speaches_1[1193]: A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.
2025-02-16T21:38:39.350702+01:00 Debian Docker: ai_speaches_1[1193]: /opt/nvidia/nvidia_entrypoint.sh: line 67: exec: uvicorn: not found
My conclusion
- The compose stack in its current form should not rely on rebuilding the image
The image supllied via ghcr is lacking dependencies in order to run on its own- And even if rebuilding the image is intended (which I consider none-ideal practice), the documentation is missing the point, that one also needs the Dockerfile in order to build a container from the compose files
I rectified my compose file. Previously I had
volumes:
- "/mnt/cache/ai/speaches:/home/ubuntu/:rw"
that was overwriting speaches runtime. I modified it to
volumes:
- "/mnt/cache/ai/speaches:/home/ubuntu/.cache:rw"
and now my container runs. I removed the parts in OP regarding that point.
However the compose files should still be updated so that the container image is not rebuild when deployed with the way as the documentation suggests.
@systemofapwne Would you be able to share a sanitized version of you .env file? Also how did you download a STT model as required on their website? uvx and jq are not part of this docker image.
export SPEACHES_BASE_URL="http://localhost:8000"
# Listing all available STT models
uvx speaches-cli registry ls --task automatic-speech-recognition | jq '.data | [].id'
# Downloading a Systran/faster-distil-whisper-small.en model
uvx speaches-cli model download Systran/faster-distil-whisper-small.en
# Check that the model has been installed
uvx speaches-cli model ls --task text-to-speech | jq '.data | map(select(.id == "Systran/faster-distil-whisper-small.en"))'
@systemofapwne Would you be able to share a sanitized version of you .env file? Also how did you download a STT model as required on their website? uvx and jq are not part of this docker image.
export SPEACHES_BASE_URL="http://localhost:8000" # Listing all available STT models uvx speaches-cli registry ls --task automatic-speech-recognition | jq '.data | [].id' # Downloading a Systran/faster-distil-whisper-small.en model uvx speaches-cli model download Systran/faster-distil-whisper-small.en # Check that the model has been installed uvx speaches-cli model ls --task text-to-speech | jq '.data | map(select(.id == "Systran/faster-distil-whisper-small.en"))'
My env file just has some none working entries
HF_TOKEN=<REDACTED>
WHISPER__TTL=0
WHISPER__INFERENCE_DEVICE=cuda
WHISPER__MODEL=TiMoMuc/whisper-large-v3-german-ct2
WHISPER__BEAM=20
WHISPER__LANG=de
As it turns out, custom models are not really supported yet. I even tried to install them from within the docker container by grabbing them via huggingface-cli (oy by manually copying them there, when previously fetched by huggingface-cli externally - I do not remember). But the container then does not respec the model.
I guess, we have to wait until custom models are fully supported. See also #352
And even if rebuilding the image is intended (which I consider none-ideal practice), the documentation is missing the point, that one also needs the Dockerfile in order to build a container from the compose files
I've updated the docs
The compose stack in its current form should not rely on rebuilding the image
I don't think this is true (or maybe I've addressed this at some point). Please re-open if you can still reproduce the issue
See Below
❯ docker system prune -af
Deleted Images:
untagged: ghcr.io/speaches-ai/speaches:latest-cuda
deleted: sha256:6359cfc2055e2aa0f5d935220dc7174b0ed7d49691af978d4d09722374330b03
deleted: sha256:b3c40c14f179a63d55dca62715f9f9c74c9f8cd83bfa8ca1fda82953c6c1658c
deleted: sha256:9a8a8204dadc43a8577bfe3d2c08b64affcab1842e749472249b70c51eb23306
deleted: sha256:e3366dd687552c0533285c7066bb8e937a5aaa2ff3a0c1c7f1305b3310399895
deleted: sha256:65401d81f667b0e36df6548ea9ff5abb51a827c31be259a7b94387dacc4641a4
deleted: sha256:56107d16fc0a7ee1872abf310f47065fd1c7174c47d155b0f046a2ba20615fed
deleted: sha256:8424fb7da3516286de4775edd3109ec45069baf151f7fdf73eac4d8169405e1e
deleted: sha256:150ae9de14cf67d192ab855eb94172865ba70f5b4b2e832ffed3e8f1df6307f5
deleted: sha256:6c3d13f3dea605afc511eaa3a29de67b5f3a9e944eeb8aa9f5b30d5363bac7b6
deleted: sha256:cd4a561b57e13e0c955bc782d6b0a52d32c977247df80b5de3bf2a2b6d161990
deleted: sha256:ec4e1d2c222df5b9978b003d3ae65099a184d1a28eff75e0378d909c11ebf94d
deleted: sha256:e6a341e968989f0958a7e10a771ed4947d73881754e9adf202d2c2f70d8f68d5
deleted: sha256:0d0d08c411e1876e5d51c8b405cbec8cea03fd5732ac98a37d9fa0ff772ec386
deleted: sha256:5399c540fe27607f12b5d2b0a4f8e98afc90ebd83f54fea6bdd045c4f04e98c9
deleted: sha256:8b4fbb887555e83a4fe35ebc6cdda1e582f26ed212670b79226c755037b7bee3
deleted: sha256:392b8dec166357d09928cc176ca500a9ed5b4ad4026eebefe4003b376e89bf48
deleted: sha256:00e85adade6d308a0cd29f14de1c6526fed359c30f34805c1621e54e8244ab8f
deleted: sha256:293d873571652439e1ee0bd335123731067661cead32bdf99465d9b83201e8c9
deleted: sha256:a9f51e9b8b0c543418d1165ef2253ad20bd8ff8b167558f1bf0a4c233f9da29b
deleted: sha256:88fd6e1a473fdca5977cc4238bbe7f9538d8e71a576dfd11da174f654b7f01b0
Total reclaimed space: 2.593GB
~/code/speaches-test
❯ ls -la
~/code/speaches-test
❯ curl --silent --remote-name https://raw.githubusercontent.com/speaches-ai/speaches/master/compose.yaml
curl --silent --remote-name https://raw.githubusercontent.com/speaches-ai/speaches/master/compose.cuda.yaml
export COMPOSE_FILE=compose.cuda.yaml
~/code/speaches-test
❯ docker compose up --detach
[+] Running 15/19
⠇ speaches [⣿⣿⣿⣿⣿⡀⣿⣿⣿⣀⣷⣿⣿⣿⣿⣿⣿⣿] 911.4MB / 2.592GB Pulling 18.9s
✔ 88fd6e1a473f Download complete 0.4s
✔ e6a341e96898 Download complete 0.2s
✔ 5399c540fe27 Download complete 8.0s
✔ 8424fb7da351 Pull complete 0.4s
✔ cd4a561b57e1 Download complete 0.4s
⠹ 6c3d13f3dea6 Downloading [======> ] 198.2MB/1.435GB 17.2s
✔ 8b4fbb887555 Download complete 0.4s
✔ 56107d16fc0a Pull complete 9.7s
✔ ec4e1d2c222d Download complete 0.4s
⠹ 0d0d08c411e1 Downloading [=============> ] 155.6MB/575.9MB 17.2s
⠹ 00e85adade6d Downloading [============================================> ] 194MB/215.7MB 17.2s
✔ 150ae9de14cf Pull complete 0.4s
✔ a9f51e9b8b0c Download complete 13.3s
✔ e3366dd68755 Pull complete 7.0s
✔ 4f4fb700ef54 Already exists 0.0s
✔ 392b8dec1663 Download complete 6.2s
✔ 65401d81f667 Pull complete 7.1s
✔ 293d87357165 Download complete 0.5s
```
</details>
