openhab-webui icon indicating copy to clipboard operation
openhab-webui copied to clipboard

[WIP] [UI] Add voice dialog

Open GiviMAD opened this issue 1 year ago • 1 comments

This is a WIP PR for the issue https://github.com/openhab/openhab-webui/issues/2275

The PR uses these audio WebSocket which is still in review https://github.com/openhab/openhab-core/pull/4032 so you need to use the org.openhab.core.io.websocket and org.openhab.core.voice bundles from there if you want to test it.

It is built on top of the webpack 5 upgrade PR https://github.com/openhab/openhab-webui/pull/2267, because on webpack 4 you need to use loaders and I didn't want to generate more work to do in a future upgrade. The actual PR changes are in the last commit.

Changes added:

Screenshot 2024-01-26 at 17 12 13

I have added that panel at the end of the "help & about" page (it should only be displayed if the necessary APIs are available), it contains the following options:

  • "Enable Voice Dialog": When enabled the dialog execution will start after the first user event on the page (needed to start the microphone stream).
  • "Connection ID": And auto generated code that is used to register the audio components. Ex: the id of the registered sink will look like "pcm::ui-52-68::sink".
  • "Selected Keyword": Rustpotter config, choose rustpotter model file. At the moment the file is selected using an input type file and persisted using an IndexedDB instance.
  • "Keyword Threshold": Rustpotter config, threshold in range 0 - 1.
  • "Keyword Min Detections": Rustpotter config, min number of positive scores to emit the detection.
  • "Listening Item": Dialog configuration, Item to toggle while listening the command.
  • "Location Item": Dialog configuration, used by the Standard language processor to improve the Item selection.

Modifying any of the options triggers a page reload.

The dialog will use the voice services configured at "Settings / Voice". Once the dialog is running it can be triggered using the voice if the rustpotter model was provided, or using the listenAndAnswer action. For example from the oh console by running voice listenandanswer --source pcm::ui-52-68::source --sink pcm::ui-52-68::sink.

For now I have placed some console warning in the events emitted about the dialog state.

I wanted to open the PR in case you can give me some feedback about the integration of the feature or in case someone wants to dig into the implementation.

Best regards.

GiviMAD avatar Jan 26 '24 17:01 GiviMAD