Voice assistant example - the "command" tool

[ggerganov]

trafficstars

There seems to be significant interest for a voice assistant application of Whisper, similar to "Ok, Google", "Hey Siri", "Alexa", etc. The existing stream tool is not very applicable for this use case, because the voice assistant commands are usually short (i.e. play some music, turn on the TV, kill all humans, feed the baby, etc), while stream expects a continuous stream of speech.

Therefore, implement a basic command-line tool called command that does the following:

• Upon start, asks the person to say a "key phrase". The phrase should be an average sentence that normally takes 2-3 seconds to pronounce. We want to have enough "training" data of the person's voice
• If the transcribed text matches the expected phrase, then we "remember" this audio and use it later. Else, we ask to say it again until we succeed
• We start listening continuously for voice activity using my VAD detector that I implemented for talk.wasm - I think it works very well given it's simplicity
• When we detect speech, we prepend the recorded key-phrase to the last 2-3 seconds of the live audio and transcribe
• The result should be: [key phrase][command], so by knowing the key phrase we can extract only the [command]

This should work in Web and Raspberry Pi and thanks to the VAD, it will be energy efficient. Should be a good starting example for creating a voice assistant.

[ggerganov]

This is now fully functional:

https://user-images.githubusercontent.com/1991296/204038393-2f846eae-c255-4099-a76d-5735c25c49da.mp4

Code is in examples/command

Web version: examples/command.wasm

[StuartIanNaylor]

@ggerganov I am missing a lib or the usual of doing something wrong :)

rock@rock-5b:~/whisper.cpp$ make command                                                                                                                                       g++ -I. -I./examples -O3 -std=c++11 -pthread examples/command/command.cpp ggml.o                                                                                                whisper.o -o command `sdl2-config --cflags --libs`
examples/command/command.cpp:136:10: error: 'mutex' in namespace 'std' does not                                                                                                name a type
  136 |     std::mutex m_mutex;
      |          ^~~~~
examples/command/command.cpp:21:1: note: 'std::mutex' is defined in header '<mut                                                                                               ex>'; did you forget to '#include <mutex>'?
   20 | #include <regex>
  +++ |+#include <mutex>
   21 |
examples/command/command.cpp: In member function 'bool audio_async::clear()':
examples/command/command.cpp:256:14: error: 'lock_guard' is not a member of 'std                                                                                               '
  256 |         std::lock_guard<std::mutex> lock(m_mutex);
      |              ^~~~~~~~~~
examples/command/command.cpp:256:14: note: 'std::lock_guard' is defined in heade                                                                                               r '<mutex>'; did you forget to '#include <mutex>'?
examples/command/command.cpp:256:30: error: 'mutex' is not a member of 'std'
  256 |         std::lock_guard<std::mutex> lock(m_mutex);
      |                              ^~~~~
examples/command/command.cpp:256:30: note: 'std::mutex' is defined in header '<m                                                                                               utex>'; did you forget to '#include <mutex>'?
examples/command/command.cpp:256:42: error: 'm_mutex' was not declared in this s                                                                                               cope; did you mean 'SDL_mutex'?
  256 |         std::lock_guard<std::mutex> lock(m_mutex);
      |                                          ^~~~~~~
      |                                          SDL_mutex
examples/command/command.cpp:256:37: error: 'lock' was not declared in this scop                                                                                               e; did you mean 'clock'?
  256 |         std::lock_guard<std::mutex> lock(m_mutex);
      |                                     ^~~~
      |                                     clock
examples/command/command.cpp: In member function 'void audio_async::callback(uin                                                                                               t8_t*, int)':
examples/command/command.cpp:279:14: error: 'lock_guard' is not a member of 'std                                                                                               '
  279 |         std::lock_guard<std::mutex> lock(m_mutex);
      |              ^~~~~~~~~~
examples/command/command.cpp:279:14: note: 'std::lock_guard' is defined in heade                                                                                               r '<mutex>'; did you forget to '#include <mutex>'?
examples/command/command.cpp:279:30: error: 'mutex' is not a member of 'std'
  279 |         std::lock_guard<std::mutex> lock(m_mutex);
      |                              ^~~~~
examples/command/command.cpp:279:30: note: 'std::mutex' is defined in header '<m                                                                                               utex>'; did you forget to '#include <mutex>'?
examples/command/command.cpp:279:42: error: 'm_mutex' was not declared in this s                                                                                               cope; did you mean 'SDL_mutex'?
  279 |         std::lock_guard<std::mutex> lock(m_mutex);
      |                                          ^~~~~~~
      |                                          SDL_mutex
examples/command/command.cpp:279:37: error: 'lock' was not declared in this scop                                                                                               e; did you mean 'clock'?
  279 |         std::lock_guard<std::mutex> lock(m_mutex);
      |                                     ^~~~
      |                                     clock
examples/command/command.cpp: In member function 'void audio_async::get(int, std                                                                                               ::vector<float>&)':
examples/command/command.cpp:312:14: error: 'lock_guard' is not a member of 'std                                                                                               '
  312 |         std::lock_guard<std::mutex> lock(m_mutex);
      |              ^~~~~~~~~~
examples/command/command.cpp:312:14: note: 'std::lock_guard' is defined in heade                                                                                               r '<mutex>'; did you forget to '#include <mutex>'?
examples/command/command.cpp:312:30: error: 'mutex' is not a member of 'std'
  312 |         std::lock_guard<std::mutex> lock(m_mutex);
      |                              ^~~~~
examples/command/command.cpp:312:30: note: 'std::mutex' is defined in header '<m                                                                                               utex>'; did you forget to '#include <mutex>'?
examples/command/command.cpp:312:42: error: 'm_mutex' was not declared in this s                                                                                               cope; did you mean 'SDL_mutex'?
  312 |         std::lock_guard<std::mutex> lock(m_mutex);
      |                                          ^~~~~~~
      |                                          SDL_mutex
examples/command/command.cpp:312:37: error: 'lock' was not declared in this scop                                                                                               e; did you mean 'clock'?
  312 |         std::lock_guard<std::mutex> lock(m_mutex);
      |                                     ^~~~
      |                                     clock
make: *** [Makefile:153: command] Error 1

[ggerganov]

I think you haven't updated to latest master: git pull

[StuartIanNaylor]

Yep I must be fresh on your heels as it was a new install this morning and thought it was latest apols.

Works great