moq icon indicating copy to clipboard operation
moq copied to clipboard

Improve STT

Open kixelated opened this issue 4 months ago • 1 comments

We're using whisper-base right now, however the performance and quality could be improved: https://github.com/kixelated/moq/blob/13c266c143e3a17fceb18e5bfe077ba4215d6023/js/hang/src/publish/audio/captions-worker.ts#L168

Based on the leaderboards, Parakeet is both faster and more accurate than Whisper. However it's also a lot larger and needs quite a bit of massaging, as it doesn't work natively with transformers.js even when converted to ONNX. Here's a thread: https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2/discussions/9#681a7e1208d65e102119f8a8

Moonshine is also an option but tbh it seems like a sidegrade to Whisper.

kixelated avatar Aug 25 '25 17:08 kixelated

Also there is parakeet.js but based on the OPTIMIZATION_PLAN, it needs some work.

kixelated avatar Aug 25 '25 17:08 kixelated