Trigger of second voice assistant does not work

[HarvsG]

If homeassistant-satellite detects the wakeword of the non-favourite voice assistant the --debug output stops and the device becomes unresponsive:

pi@redpi:~/homeassistant-satellite $ script/run --host 192.168.0.30 --token eyJredactedTA4Nzg4fQ.CM-43hmuImaukgtrsw2eG6clobjG9kg22mvNOwr94zQ --done-sound sounds/done.wav --vad webrtcvad --debug --mic-device plughw:CARD=Device --debug-recording-dir records --noise-suppression 2 --auto-gain 25 --volume-multiplier 6
DEBUG:__main__:Namespace(host='192.168.0.30', token='eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiI0NDA3ZTM4NjNkZmQ0NmY2YTM4MjE0NDJjZmEzMWI4YyIsImlhdCI6MTY5NzE0ODc4OCwiZXhwIjoyMDEyNTA4Nzg4fQ.CM-43hmuImaukgtrsw2eG6clobjG9kg22mvNOwr94zQ', port=8123, api_path='/api', pipeline=None, protocol='http', mic_device='plughw:CARD=Device', snd_device=None, mic_command=None, snd_command=None, snd_command_sample_rate=22050, awake_sound=None, done_sound='sounds/done.wav', volume=1.0, vad='webrtcvad', vad_model=PosixPath('/home/pi/homeassistant-satellite/homeassistant_satellite/models/silero_vad.onnx'), vad_threshold=0.5, vad_trigger_level=3, vad_buffer_chunks=40, wake_buffer_seconds=0, noise_suppression=2, auto_gain=25, volume_multiplier=6.0, udp_mic=None, udp_snd=None, udp_snd_sample_rate=22050, debug_recording_dir='records', debug=True)
DEBUG:__main__:Waiting for speech
DEBUG:__main__:Using webrtc audio processing
DEBUG:root:Microphone command: ['arecord', '-D', 'plughw:CARD=Device', '-r', '16000', '-c', '1', '-f', 'S16_LE', '-t', 'raw']
Recording raw data 'stdin' : Signed 16 bit Little Endian, Rate 16000 Hz, Mono
DEBUG:__main__:Speech detected
DEBUG:homeassistant_satellite.remote:{'type': 'auth_required', 'ha_version': '2023.10.2'}
DEBUG:homeassistant_satellite.remote:{'type': 'auth_ok', 'ha_version': '2023.10.2'}
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'result', 'success': True, 'result': None}
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'event', 'event': {'type': 'run-start', 'data': {'pipeline': '01gzv00ms6vap6j6fa7cn6t2s4', 'language': 'en', 'runner_data': {'stt_binary_handler_id': 1, 'timeout': 300}}, 'timestamp': '2023-10-13T13:02:40.765878+00:00'}}
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'event', 'event': {'type': 'wake_word-start', 'data': {'entity_id': 'wake_word.openwakeword', 'metadata': {'format': 'wav', 'codec': 'pcm', 'bit_rate': 16, 'sample_rate': 16000, 'channel': 1}, 'timeout': 3}, 'timestamp': '2023-10-13T13:02:40.766173+00:00'}}
DEBUG:__main__:wake_word-start {'entity_id': 'wake_word.openwakeword', 'metadata': {'format': 'wav', 'codec': 'pcm', 'bit_rate': 16, 'sample_rate': 16000, 'channel': 1}, 'timeout': 3}
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'event', 'event': {'type': 'wake_word-end', 'data': {'wake_word_output': {'wake_word_id': 'hey_rhasspy_v0.1', 'timestamp': 1570}}, 'timestamp': '2023-10-13T13:02:42.746383+00:00'}}
DEBUG:__main__:wake_word-end {'wake_word_output': {'wake_word_id': 'hey_rhasspy_v0.1', 'timestamp': 1570}}
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'event', 'event': {'type': 'stt-start', 'data': {'engine': 'cloud', 'metadata': {'language': 'en-GB', 'format': 'wav', 'codec': 'pcm', 'bit_rate': 16, 'sample_rate': 16000, 'channel': 1}}, 'timestamp': '2023-10-13T13:02:42.746907+00:00'}}
DEBUG:__main__:stt-start {'engine': 'cloud', 'metadata': {'language': 'en-GB', 'format': 'wav', 'codec': 'pcm', 'bit_rate': 16, 'sample_rate': 16000, 'channel': 1}}
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'event', 'event': {'type': 'stt-vad-start', 'data': {'timestamp': 1890}, 'timestamp': '2023-10-13T13:02:43.269117+00:00'}}
DEBUG:__main__:stt-vad-start {'timestamp': 1890}
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'event', 'event': {'type': 'stt-vad-end', 'data': {'timestamp': 2555}, 'timestamp': '2023-10-13T13:02:44.627470+00:00'}}
DEBUG:__main__:stt-vad-end {'timestamp': 2555}
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'event', 'event': {'type': 'stt-end', 'data': {'stt_output': {'text': 'Turn off the study lights.'}}, 'timestamp': '2023-10-13T13:02:44.786910+00:00'}}
DEBUG:__main__:stt-end {'stt_output': {'text': 'Turn off the study lights.'}}
DEBUG:root:play ffmpeg: ['ffmpeg', '-i', 'sounds/done.wav', '-f', 'wav', '-ar', '22050', '-ac', '1', '-filter:a', 'volume=1.0', '-']
DEBUG:root:play: ['aplay', '-r', '22050', '-c', '1', '-f', 'S16_LE', '-t', 'raw']
Playing raw data 'stdin' : Signed 16 bit Little Endian, Rate 22050 Hz, Mono
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'event', 'event': {'type': 'intent-start', 'data': {'engine': 'homeassistant', 'language': 'en', 'intent_input': 'Turn off the study lights.', 'conversation_id': None, 'device_id': None}, 'timestamp': '2023-10-13T13:02:44.787097+00:00'}}
DEBUG:__main__:intent-start {'engine': 'homeassistant', 'language': 'en', 'intent_input': 'Turn off the study lights.', 'conversation_id': None, 'device_id': None}
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'event', 'event': {'type': 'intent-end', 'data': {'intent_output': {'response': {'speech': {'plain': {'speech': 'Turned off lights', 'extra_data': None}}, 'card': {}, 'language': 'en', 'response_type': 'action_done', 'data': {'targets': [], 'success': [{'name': 'Study', 'type': 'area', 'id': 'study'}, {'name': 'Study Light', 'type': 'entity', 'id': 'light.study_light'}, {'name': 'Study Bulb 4', 'type': 'entity', 'id': 'light.study_bulb_4'}, {'name': 'Study Bulb 3', 'type': 'entity', 'id': 'light.study_bulb_3'}, {'name': 'Study Bulb 2', 'type': 'entity', 'id': 'light.study_bulb_2'}, {'name': 'Study Bulb 1', 'type': 'entity', 'id': 'light.study_bulb_1'}, {'name': 'Study Lamp', 'type': 'entity', 'id': 'light.study_lamp'}], 'failed': []}}, 'conversation_id': None}}, 'timestamp': '2023-10-13T13:02:45.073797+00:00'}}
DEBUG:__main__:intent-end {'intent_output': {'response': {'speech': {'plain': {'speech': 'Turned off lights', 'extra_data': None}}, 'card': {}, 'language': 'en', 'response_type': 'action_done', 'data': {'targets': [], 'success': [{'name': 'Study', 'type': 'area', 'id': 'study'}, {'name': 'Study Light', 'type': 'entity', 'id': 'light.study_light'}, {'name': 'Study Bulb 4', 'type': 'entity', 'id': 'light.study_bulb_4'}, {'name': 'Study Bulb 3', 'type': 'entity', 'id': 'light.study_bulb_3'}, {'name': 'Study Bulb 2', 'type': 'entity', 'id': 'light.study_bulb_2'}, {'name': 'Study Bulb 1', 'type': 'entity', 'id': 'light.study_bulb_1'}, {'name': 'Study Lamp', 'type': 'entity', 'id': 'light.study_lamp'}], 'failed': []}}, 'conversation_id': None}}
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'event', 'event': {'type': 'tts-start', 'data': {'engine': 'cloud', 'language': 'en-GB', 'voice': 'MaisieNeural', 'tts_input': 'Turned off lights'}, 'timestamp': '2023-10-13T13:02:45.074084+00:00'}}
DEBUG:__main__:tts-start {'engine': 'cloud', 'language': 'en-GB', 'voice': 'MaisieNeural', 'tts_input': 'Turned off lights'}
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'event', 'event': {'type': 'tts-end', 'data': {'tts_output': {'media_id': 'media-source://tts/cloud?message=Turned+off+lights&language=en-GB&voice=MaisieNeural', 'url': '/api/tts_proxy/3ab0fdf873d5e839d27be59ad331f26dc357d73f_en-gb_5230df2865_cloud.audiooutput.mp3', 'mime_type': 'audio/mpeg'}}, 'timestamp': '2023-10-13T13:02:45.075294+00:00'}}
DEBUG:__main__:tts-end {'tts_output': {'media_id': 'media-source://tts/cloud?message=Turned+off+lights&language=en-GB&voice=MaisieNeural', 'url': '/api/tts_proxy/3ab0fdf873d5e839d27be59ad331f26dc357d73f_en-gb_5230df2865_cloud.audiooutput.mp3', 'mime_type': 'audio/mpeg'}}
DEBUG:root:play ffmpeg: ['ffmpeg', '-i', 'http://192.168.0.30:8123/api/tts_proxy/3ab0fdf873d5e839d27be59ad331f26dc357d73f_en-gb_5230df2865_cloud.audiooutput.mp3', '-f', 'wav', '-ar', '22050', '-ac', '1', '-filter:a', 'volume=1.0', '-']
DEBUG:root:play: ['aplay', '-r', '22050', '-c', '1', '-f', 'S16_LE', '-t', 'raw']
Playing raw data 'stdin' : Signed 16 bit Little Endian, Rate 22050 Hz, Mono
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'event', 'event': {'type': 'run-end', 'data': None, 'timestamp': '2023-10-13T13:02:45.075435+00:00'}}
DEBUG:__main__:run-end None
DEBUG:homeassistant_satellite.remote:Pipeline finished
DEBUG:__main__:Waiting for speech
DEBUG:__main__:Speech detected
DEBUG:homeassistant_satellite.remote:{'type': 'auth_required', 'ha_version': '2023.10.2'}
DEBUG:homeassistant_satellite.remote:{'type': 'auth_ok', 'ha_version': '2023.10.2'}
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'result', 'success': True, 'result': None}
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'event', 'event': {'type': 'run-start', 'data': {'pipeline': '01gzv00ms6vap6j6fa7cn6t2s4', 'language': 'en', 'runner_data': {'stt_binary_handler_id': 1, 'timeout': 300}}, 'timestamp': '2023-10-13T13:02:49.766683+00:00'}}
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'event', 'event': {'type': 'wake_word-start', 'data': {'entity_id': 'wake_word.openwakeword', 'metadata': {'format': 'wav', 'codec': 'pcm', 'bit_rate': 16, 'sample_rate': 16000, 'channel': 1}, 'timeout': 3}, 'timestamp': '2023-10-13T13:02:49.767102+00:00'}}
DEBUG:__main__:wake_word-start {'entity_id': 'wake_word.openwakeword', 'metadata': {'format': 'wav', 'codec': 'pcm', 'bit_rate': 16, 'sample_rate': 16000, 'channel': 1}, 'timeout': 3}

Stream suddenly stops their

OpenWakeWord debug log shows lots of triggers:

DEBUG:root:Triggered hey_rhasspy_v0.1 (client=5826969686261)
DEBUG:wyoming_openwakeword.handler:Client disconnected: 5826969686261
DEBUG:wyoming_openwakeword.handler:Client connected: 5835970430964
DEBUG:wyoming_openwakeword.handler:Receiving audio from client: 5835970430964
DEBUG:root:Triggered hey_jarvis_v0.1 (client=5835970430964)
DEBUG:root:Triggered hey_jarvis_v0.1 (client=5835970430964)
DEBUG:root:Triggered hey_jarvis_v0.1 (client=5835970430964)
DEBUG:root:Triggered hey_jarvis_v0.1 (client=5835970430964)
DEBUG:root:Triggered hey_jarvis_v0.1 (client=5835970430964)
DEBUG:root:Triggered hey_jarvis_v0.1 (client=5835970430964)
DEBUG:root:Triggered hey_jarvis_v0.1 (client=5835970430964)
DEBUG:root:Triggered hey_jarvis_v0.1 (client=5835970430964)
DEBUG:root:Triggered hey_jarvis_v0.1 (client=5835970430964)
DEBUG:wyoming_openwakeword.handler:Client connected: 5855276917884
DEBUG:wyoming_openwakeword.handler:Sent info to client: 5855276917884
DEBUG:wyoming_openwakeword.handler:Client disconnected: 5855276917884
DEBUG:wyoming_openwakeword.handler:Client connected: 5887450564782
DEBUG:wyoming_openwakeword.handler:Sent info to client: 5887450564782
DEBUG:wyoming_openwakeword.handler:Client disconnected: 5887450564782
DEBUG:wyoming_openwakeword.handler:Client connected: 5919591716748
DEBUG:wyoming_openwakeword.handler:Sent info to client: 5919591716748
DEBUG:wyoming_openwakeword.handler:Client disconnected: 5919591716748
DEBUG:wyoming_openwakeword.handler:Client connected: 5951717174131
DEBUG:wyoming_openwakeword.handler:Sent info to client: 5951717174131
DEBUG:wyoming_openwakeword.handler:Client disconnected: 5951717174131
DEBUG:wyoming_openwakeword.handler:Client connected: 5983853725401
DEBUG:wyoming_openwakeword.handler:Sent info to client: 5983853725401
DEBUG:wyoming_openwakeword.handler:Client disconnected: 5983853725401
DEBUG:wyoming_openwakeword.handler:Client connected: 6016004259331
DEBUG:wyoming_openwakeword.handler:Sent info to client: 6016004259331
DEBUG:wyoming_openwakeword.handler:Client disconnected: 6016004259331
DEBUG:wyoming_openwakeword.handler:Client connected: 6048179180758
DEBUG:wyoming_openwakeword.handler:Sent info to client: 6048179180758
DEBUG:wyoming_openwakeword.handler:Client disconnected: 6048179180758

[HarvsG]

OK, so the lock up seems to have stopped. However the issue remains, when I speak the wake-word of the non-starred assistant the satalite reports no wake word detected:

DEBUG:__main__:Waiting for speech
DEBUG:__main__:Speech detected
DEBUG:homeassistant_satellite.remote:{'type': 'auth_required', 'ha_version': '2023.10.2'}
DEBUG:homeassistant_satellite.remote:{'type': 'auth_ok', 'ha_version': '2023.10.2'}
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'result', 'success': True, 'result': None}
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'event', 'event': {'type': 'run-start', 'data': {'pipeline': '01h4nxbvp9dnh23seeqm1abg7v', 'language': 'en', 'runner_data': {'stt_binary_handler_id': 1, 'timeout': 300}}, 'timestamp': '2023-10-13T13:12:02.625599+00:00'}}
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'event', 'event': {'type': 'wake_word-start', 'data': {'entity_id': 'wake_word.openwakeword', 'metadata': {'format': 'wav', 'codec': 'pcm', 'bit_rate': 16, 'sample_rate': 16000, 'channel': 1}, 'timeout': 3}, 'timestamp': '2023-10-13T13:12:02.625924+00:00'}}
DEBUG:__main__:wake_word-start {'entity_id': 'wake_word.openwakeword', 'metadata': {'format': 'wav', 'codec': 'pcm', 'bit_rate': 16, 'sample_rate': 16000, 'channel': 1}, 'timeout': 3}
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'event', 'event': {'type': 'error', 'data': {'code': 'wake-word-timeout', 'message': 'Wake word was not detected'}, 'timestamp': '2023-10-13T13:12:07.494267+00:00'}}
DEBUG:__main__:error {'code': 'wake-word-timeout', 'message': 'Wake word was not detected'}
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'event', 'event': {'type': 'run-end', 'data': None, 'timestamp': '2023-10-13T13:12:07.494436+00:00'}}
DEBUG:__main__:run-end None
DEBUG:homeassistant_satellite.remote:Pipeline finished
DEBUG:__main__:Waiting for speech

But open wake word reports it is detected:

DEBUG:wyoming_openwakeword.handler:Receiving audio from client: 6377838000350
DEBUG:wyoming_openwakeword.handler:Client disconnected: 6377838000350
DEBUG:wyoming_openwakeword.handler:Client connected: 6388829661544
DEBUG:wyoming_openwakeword.handler:Receiving audio from client: 6388829661544
DEBUG:root:Triggered hey_rhasspy_v0.1 (client=6388829661544)
DEBUG:root:Triggered hey_rhasspy_v0.1 (client=6388829661544)
DEBUG:root:Triggered hey_rhasspy_v0.1 (client=6388829661544)

[synesthesiam]

homeassistant-satellite only runs one pipeline at the moment, which you can change with --pipeline <NAME>

The pipeline must be chosen ahead of time because multiple pipelines can share the same wake word. To activate multiple assistants from the same satellite, there would need to be some architectural changes in homeassistant-satellite (without just running two copies of the program).

[Bram-diederik]

I tried to run two copies for two diffrent activation words to make a dutch and english. It did not work.

[joie96]

You can run two copies when you use pulseaudio and passthrough the pulseaudio socket like so:

version: "3.8" services: homeassistant-satellite: container_name: homeassistant-satellite image: "synesthesiam/homeassistant-satellite_arm64:latest" restart: unless-stopped network_mode: host volumes: - ./sounds:/sounds - ./debug_dir:/debug_dir - /var/run/pulse/native:/var/run/pulse/native group_add: - audio environment: - PULSE_SERVER=unix:/var/run/pulse/native command: - --pulseaudio
- --host ...

[bogorad]

To activate multiple assistants from the same satellite

Wouldn't the easiest interim solution be to stream the same recorded audio snippet to two two/three/whatever pipelines and let HA decide? This shouldn't require a serious change in architecture?