idear

A general purpose voice user interface for the IntelliJ Platform, inspired by Tavis Rudd. Possible use cases: visually impaired and RSI users. Originally developed as part of a JetBrains hackathon, it is now a community-supported project. For background information, check out this presentation.

Usage

To get started, press the button in the toolbar, then speak a command, e.g. "Hi, IDEA!" Idear supports a simple grammar. For a complete list of commands, please refer to the wiki. Click the button once more to deactivate.

Building

For Linux or Mac OS users:

git clone https://github.com/OpenASR/idear && cd idear && ./gradlew runIde

For Windows users:

git clone https://github.com/OpenASR/idear & cd idear & gradlew.bat runIde

Recognition works with most popular microphones (preferably 16kHz, 16-bit). For best results, minimize background noise.

Contributing

Contributors who have IntelliJ IDEA installed can simply open the project. Otherwise, run the following command from the project's root directory:

./gradlew runIde -PluginDev

Architecture

Idear is implemented using the IntelliJ Platform SDK. For more information about the plugin architecture, please refer to the wiki page.

Plugin Actions

plugin.xml defines <action>s:

VoiceRecordControllerAction

This action is invoked when the user clicks on the button in the toolbar. This simply tells ASRService to activate or standby. When the ASRService is active, the ASRSystem, by default ASRControlLoop (see below).

ExecuteActionFromPredefinedText

A debugging aid to use one of the ActionRecognizer extension classes configured in plugin.xml to generate an ActionCallInfo which is then runInEditor().

ExecuteVoiceCommandAction

Similar to ExecuteActionFromPredefinedText but uses the Idear.VoiceCommand.Text data attached to the invoking AnActionEvent.

WhereAmIAction

IDEA Actions

There are many Actions (classes which extend AnAction) provided by IDEA:

• ExternalSystemActions
• LangActions
• PlatformActions
• VcsActions

ASRControlLoop

When ASRControlLoop detects an utterance, it invokes PatternBasedNlpProvider.processUtterance() which typically calls invokeAction() and/or one or more of the methods of IDEService

Programming By Voice

• Interactive IDE Voice Control
• Using Python to Code by Voice
• How a Blind Developer uses Visual Studio

Maintainers

• Breandan Considine
• Nicholas Albion