speech-assistant

Table of Contents

• Welcome
• How to Use
  • Steps
• Configurations
• Notes and Considerations
• Contributions
• Acknowledgements

Welcome

Welcome to Speech-Assistant!

This is a project to implement a working desktop application on both Linux and Windows that provides a real-time, offline speech-to-text dictation program. It uses the distil-whisper models from HuggingFace which offers an accurate transcription of speech that is fully structured, complete with proper punctuation and syntax. Distil-whisper is based on OpenAI's Whisper model that is 6 times faster, 49% smaller, and performs within 1% word error rate on speech it has never seen before. The research is based on this repo.

The speech-to-text assistant writes down spoken words directly to the keyboard cursor. To use it is easy, hold down a hotkey combination of Windows key (Super) and Shift to begin, and let go to end the recording. Your speech will be transcribed (or translated) in real time and the transcription will be typed in for you at the keyboard cursor. I made this program to enhance efficiency and add quality of life to the experience of PC users. Additionally, there isn't an accurate speech-to-text model used for transcription on Linux, however, do check out nerd-dictation for the implementation of speech-to-text for the vosk models.

https://github.com/Mohamad-Hussein/speech-assistant/assets/115669425/57975e07-af13-4582-95df-ae8c8e049bfe

Here is a longer demo of speech-assistant of a previous and slower version of the program.

How to Use

You can get started on any operating system you would like. The program was tested in Pop-os (Ubuntu 22.04), Windows 10 and 11. Here is Anaconda's installation instructions. If you are on Windows make sure to have access to the conda command using the Anaconda cmd terminal, or to source it directly. Nvidia and AMD have different packages needed to run Pytorch, please follow as appropriate to ensure smooth compatibility.

Steps

1. Navigate to the speech-assistant repo using the terminal (using the Anaconda CMD on Windows).

2. Install dependencies. Please use the command for your corresponding GPU brand and operating system. Depending on your internet connection, this will take ~5-15 minutes (Type y and press enter when asked to download packages).

   • Nvidia GPU:

     conda env create -f env-cuda.yml
     

   • AMD GPU or any CPU Integrated Graphics on Windows:

     conda env create -f env-general-win.yml
     

   • AMD GPU on Linux:

     conda env create -f env-amd-linux.yml
     

3. Activate the conda environment.

   conda activate speech-assistant
   

4. Start running the program.

   python main.py
   

5. The program is now ready to use!

Configurations

The program will download the distil-whisper/distil-small.en model by default and cache it locally in a folder named 'model'. The model consumes ~600 MB of GPU memory, and to improve accuracy, you could choose a bigger model. You could change models in the Options menu. The available model choices are shown below.

Please note that the distil models are currently English only, except for whisper-large, which supports transcription capabilities of multiple languages.

Notes and Considerations

• You can translate your speech to English in real-time using Whisper-Large by going to options and checking Translate to English
• Users with dedicated graphics cards will have a better experience running the big models.
• Make sure to locate your primary sound input device!
• There is a problem with using PowerShell, use cmd, and activate the conda environment.
• Installing with requirement.txt, package ffmpeg will be missing on model inference. This module can be downloaded with Anaconda with conda install ffmpeg -c pytorch.
• For transcribing on Windows you can use its built-in dictation service with left windows + h. However, the whisper models can be useful for formatting expressive punctuation, and the implementation allows for private and quick dictation.

Future contributions

Future features

• Add choice of direct connection to ChatGPT API, local LLM, or AutoGPT
• Add text-to-speech capability for the assistant
• Add sequential inference, for transcription as you talk

Needed features

• Add optimizations suggested by HuggingFace (added BetterTransformer)
• Add the option to use whatever key bind of the user's choosing (GUI)
• Add GUI
• Make Dockerfile for containers

Acknowledgements

Distil-whisper paper:

• Title: Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling
• Authors: Sanchit Gandhi, Patrick von Platen, Alexander M. Rush
• Year: 2023
• Link: ArXiv

Sound effect: soundsforyou on pixabay