Added ability for braille to show what NVDA is speaking

[Emil-18]

Link to issue number:

fixes #15898

Summary of the issue:

Both in NVDA core, as well as some add-ons, have features that work with speech but don't work with braille

Description of user facing changes

Braille only users will be able to read NVDA speech in braille, thus get access to a lot more features then they would otherwise

Description of development approach

A new braille mode has been added, called "speech emulation". In this mode, NVDA will send all text that has been spoken since the last time speech was canceled, to the braille display. If the speech was canceled do to the user scrolling the braille display back or fourth, the speech output mode will treat it as if the speech was not canceled, so the user can read the text without erasing it accidentally. If braille.handler.rawText is grater then 100000 characters, it will not be sent, because in my experience, NVDA will freeze when that much text is sent to the braille display at once. The braille settings panel has been rearranged so that all settings not aplickable to speech emulation is grouped together. A new combo box has been added to the braille settings panel, and a new script with the gesture NVDA+alt+t have been added, that allows the user to change braille mode.

Testing strategy:

tested with a Focus braille display. To test that it ignored speech being canceled when scrolling the braille display back or fourth, I turned off "Speech interrupt for typed characters", then opened the run dialog, scrolled the braille display to the end of the text while in speech output mode with the "Interrupt speech while scrolling" setting turned on, and then typed some text. I then scrolled the braille display back and fourth, and continued typing text while doing this to ensure that it worked as it should. Everything worked as expected

Known issues with pull request:

None so far

Code Review Checklist:

• [x] Documentation:
  • Change log entry
  • User Documentation
  • Developer / Technical Documentation
  • Context sensitive help for GUI changes
• [x] Testing:
  • Unit tests
  • System (end to end) tests
  • Manual testing
• [x] UX of all users considered:
  • Speech
  • Braille
  • Low Vision
  • Different web browsers
  • Localization in other languages / culture than English
• [x] API is compatible with existing add-ons.
• [x] Security precautions taken.

[AppVeyorBot]

• Build (for testing PR): https://ci.appveyor.com/api/buildjobs/scw6s7i6bv722rp9/artifacts/output/
• CI timing (mins): INIT 0.0, INSTALL_START 1.0, INSTALL_END 1.0, BUILD_START 0.0, FINISH_END 25.2

See test results for failed build of commit 0af568a7f4

[XLTechie]

Is this something that enough users will want to toggle on the fly, that it warrants assigning a key by default?

[lukaszgo1]

Both in NVDA core, as well as some add-ons, have features that work with speech but don't work with braille

Would you be able to provide some examples. Especially for core features if they don't work well in Braille it is useful to have an issue to track the problem.

[burmancomp]

I tested and noticed that something about new setting should be added to configSpec.py to get nvda from source running.

If braille is tethered to review, changing setting from settings dialog braille category from "speech emulation" to "follow cursors" causes attribute error and displaying braille is stopped until changed to what braille is tethered to.

I did not get nvda+alt+t to work with laptop layout when I used caps lock as nvda key although it was shown in braille category of gestures but this is something what is not problem of this pr.

[AppVeyorBot]

• Build (for testing PR): https://ci.appveyor.com/api/buildjobs/x0au4lamabrahdg7/artifacts/output/
• CI timing (mins): INIT 0.0, INSTALL_START 1.0, INSTALL_END 1.1, BUILD_START 0.0, FINISH_END 24.6

See test results for failed build of commit 5b9cc125d5

[Emil-18]

@lukaszgo1 As of the latest alpha, braille does not show:

• The object under the mouse when mouse tracking is enabled.
• Formatting information when it is turned on in document formatting settings.
• How many list items it is in a list on the web when entered, in a situation where speech would report list with x items.
• How many rows and colums a web/word table have when entered.
• Information about objects that are higher up in the object hierarchy, such as lists or groupings, in legacy edge UIA browse mode documents when browse mode is turned on. Same for open/libre office documents.
• On screen text on web pages that you can review with the review cursor commands in focus mode that isn't in edit boxes. I know that you aren't actualy supposed to review this text, but it is still information that a speech user can get and a braille only user can't. It is impossible to use the f9 selection mode with braille when the current review position is in an object that don't has useful text according to the braille.NVDAObjectHasUsefulText function. When ui.message is called in a fiew milliseconds after a previous call, something I have seen a lot of add-ons do, it is impossible to read what was sent to the braille display in the first call, as this text is overwritten by the text in the second. When ui.message is called inside a thread, and a braille display is connected, and show messages is set to use timeout, it gives the following error ERROR - stderr (19:11:34.275) - Thread-2 (test) (10760): C++ assertion "wxThread::IsMain()" failed at ....\src\common\timerimpl.cpp(57) in wxTimerImpl::Start(): timer can only be started from the main thread resulting in the rest of the function never executing.

[Emil-18]

@XLTechie I don't know.

[XLTechie]

Actually, given your response to @lukaszgo1, and the probable use cases you site, I am thinking that this probably does warrant a shortcut key out of the box.

[AppVeyorBot]

• Build (for testing PR): https://ci.appveyor.com/api/buildjobs/b9k9rv0xwhf2r2h8/artifacts/output/
• CI timing (mins): INIT 0.0, INSTALL_START 1.5, INSTALL_END 1.3, BUILD_START 0.0, FINISH_END 24.4

See test results for failed build of commit 91b257cc34

[AppVeyorBot]

• PASS: Translation comments check.
• FAIL: Unit tests. See test results for more information.
• FAIL: Lint check. See test results for more information.
• Build execution time has reached the maximum allowed time for your plan (60 minutes).

See test results for failed build of commit cfe4f6cb25

[burmancomp]

* The object under the mouse when mouse tracking is enabled.

I cannot confirm this at least as whole. If I set braille tethered to review and move mouse in desktop content of braille line changes. Is this feature which is partially working?

* On screen text on web pages that you can review with the review cursor commands in focus mode that isn't in edit boxes. 

If using object navigation commands like move navigator object up/down/right/left in object structure braille line content seems to update at least when braille is tethered to review. Is this also partially working feature or are we speaking about different matter?

I commented just issues I tested quickly. It does not mean that those features are currently working as they should.

In general, and this is not against @Emil-18 or any other developers, I feel that there is still quite a lot of to do with braille output.

[Emil-18]

I cannot confirm this at least as whole. If I set braille tethered to review and move mouse in desktop content of braille line changes. Is this feature which is partially working?

@burmancomp Are you sure you don't have review cursor follow mouse turned on?

If using object navigation commands like move navigator object up/down/right/left in object structure braille line content seems to update at least when braille is tethered to review. Is this also partially working feature or are we speaking about different matter?

The object navigation works fine, I am talking about when moving the review cursor with numbpad 1/2/3/etc. The text that is reviewed then, may differ from the objects name/description/... And is therefor currently inaccessible to braille only users.

In general, and this is not against @Emil-18 or any other developers, I feel that there is still quite a lot of to do with braille output.

I think the same.

[AppVeyorBot]

• Build (for testing PR): https://ci.appveyor.com/api/buildjobs/livas561fvjrplva/artifacts/output/
• CI timing (mins): INIT 0.0, INSTALL_START 1.2, INSTALL_END 1.1, BUILD_START 0.0, FINISH_END 24.0

See test results for failed build of commit ba863960e5

[AppVeyorBot]

• PASS: Translation comments check.
• FAIL: Unit tests. See test results for more information.
• FAIL: Lint check. See test results for more information.
• Build execution time has reached the maximum allowed time for your plan (60 minutes).

See test results for failed build of commit da49e3dee2

[AppVeyorBot]

• PASS: Translation comments check.
• FAIL: Unit tests. See test results for more information.
• PASS: Lint check.
• FAIL: System tests (tags: installer NVDA). See test results for more information.
• Build (for testing PR): https://ci.appveyor.com/api/buildjobs/avt7nxdwt3sfi1il/artifacts/output/nvda_snapshot_pr15956-30402,a53dc433.exe
• CI timing (mins): INIT 0.0, INSTALL_START 1.0, INSTALL_END 1.0, BUILD_START 0.0, BUILD_END 26.8, TESTSETUP_START 0.0, TESTSETUP_END 0.3, TEST_START 0.0, TEST_END 8.5, FINISH_END 0.1

See test results for failed build of commit a53dc433e2

[burmancomp]

I cannot confirm this at least as whole. If I set braille tethered to review and move mouse in desktop content of braille line changes. Is this feature which is partially working?

@burmancomp Are you sure you don't have review cursor follow mouse turned on?

I have review cursor following mouse and braille tethered to review. Could you clarify problem?

If using object navigation commands like move navigator object up/down/right/left in object structure braille line content seems to update at least when braille is tethered to review. Is this also partially working feature or are we speaking about different matter?

The object navigation works fine, I am talking about when moving the review cursor with numbpad 1/2/3/etc. The text that is reviewed then, may differ from the objects name/description/... And is therefor currently inaccessible to braille only users.

I cannot know with braille where review cursor is, and this seems to be case outside of edit boxes, and this can be seen as issue, because with speech cursor position can be detected. But are there also problems that braille shows different thing from speech?

[Emil-18]

@burmancomp Braille should show, as a flash message, what is under the mouse in the same way as speech when the mouse moves and the review cursor don't follow the mouse. In this way, the user can know where the mouse landed when moving the mouse to the review cursor. In some situations, the name of an object in a web browser may not be the same as the visual text on the screen. When using the review cursor to review text as described above, the visual text is reviewed instead of the name. Because the braille display only shows the objects name/value/... the visual text is inaccessible to a braille only user.

[Emil-18]

@seanbudd Ok. I thought I fixed all the linting issues, because it says that all checks have passed?

[seanbudd]

The lint check can only pick up issues that can be automatically picked up. Using double quotes is standard, but there is occasional exceptions to this e.g. when using a string that contains double quotes like: 'she said: "example"'.

Sometimes new line issues do not get picked up as the linter misses the additional context of lines above/below.

[josephsl]

Hi, a tip: do linting from local copy if possible before committing (run runlint.bat master and that should tell you what to do after). Also, if you don’t mind, can we call this something other than “speech emulation” to help users and translators better understand what’s up? Thanks, and good luck.

[Emil-18]

@josephsl Ok. Do you think it should be called "speech transcription" instead?

[josephsl]

Hi,

While "emulation" might work, I think a different term would be better to describe the overall feature (see below as to why I'm advising this). Perhaps using terms like "presentation" or "shows spoken messages" might be a better fit (Sean and others, any toughts?).

I bring up user interface/experience messages for a number of reasons:

1. Think about different audiences: the term "speech emulation" might work for folks who know what the feature is (especially people who are coming from different screen readers). At the same time, we have people who do not have screen reader experience other than NVDA or may have heard of other screen readers out there. A message that takes different audiences into account works more effectively in that you are actively thinking about people who will teach and use your feature in vrious ways.
2. Helps translators: most translators do not speak English as their first language (I myself speak Korean as my first language and English as second). Even if the translator knows English and can speak it well, they may not know nuances of culture that is at the heart of languages we speak. The word "emulation" could be understood differently based on languages and cultures people are used to, and so is "presentation" or other suggestions people may propose. This is one of the reasons why NVDA's user interface/experience messages are a bit neutral in tone when heard/read/seen/understood by English speakers to help translators better.
3. Ease of documentation: I'm well known in this community for being both enthusiastic and strict about user guide and other documentation materials. While the user guide can explain what "emulation" means, I'm a strong believer of making feature names easy to remember and document (also connects with item 2 because the user guide is one of the important documents translated into multiple languages).

I might as well document a few tips for pull request writers and post it as a GitHub discussion, but to offer one more advice: whenever folks work on pull requests that will involve adding or changing user interface/experience messages, I advise writing a "tentative" user guide in your mind. That way you can mentally run through your proposed feature and ask yourself if the message you are about to say (via code) is understandable to different audiences (for people familiar with old assistive tech, I consider the older BrailleNote Classic/mPower/PK/Apex user guide to be the gold standard and Jonathan Mosen's tutorials to be a gold standard on what to do when producing audio instructions; while I understand that my own Welcome to NVDA tutorial set became a gold standard for NVDA audio tutorials, I was heavily influenced by Jonathan).

Back to the PR: I see big potential with this pull request (I'm sold after doing a brief test at the source code level).

Thanks.

[ruifontes]

For me, "Braille spoken messages" is the better.

[Emil-18]

What about calling it "speech output"?

[josephsl]

Hi,

Speech output: that could work. Another option is to say different things when changing settings depending on how the user is changing it. For example, NVDA could say:

• Braille mode: follows cursors/speech output (if changing from braille settings interface)
• Braile follows cursors/Braille shows (or displays) speech output when pressing Alt+NVDA+T

A precedent exists: tether braille setting where NVDA will say different things based on how you are changing this setting.

Thanks.

[CyrilleB79]

I prefer the following, both for the GUI and for the script output: Braille Mode:

• Follow cursor
• Display speech output

All the possible values begin with a verb, what denotes an action.

If possible, I'd avoid "braille" to be used as a verb (e.g. in "Braille spoken message") for ease of translation. In French at least, there is no short equivalent verb corresponding to the English verb "to braille" and we need to translate it as a more complex expression, e.g. "to display in braille". To be confirmed or not by other translators though.

[AppVeyorBot]

• PASS: Translation comments check.
• PASS: Unit tests.
• FAIL: Lint check. See test results for more information.
• PASS: System tests (tags: installer NVDA).
• Build (for testing PR): https://ci.appveyor.com/api/buildjobs/4nintn6c9fu021te/artifacts/output/nvda_snapshot_pr15956-30442,98f8e8d3.exe
• CI timing (mins): INIT 0.0, INSTALL_START 0.9, INSTALL_END 1.0, BUILD_START 0.0, BUILD_END 25.4, TESTSETUP_START 0.0, TESTSETUP_END 0.3, TEST_START 0.0, TEST_END 16.5, FINISH_END 0.1

See test results for failed build of commit 98f8e8d3ff

[AppVeyorBot]

• PASS: Translation comments check.
• FAIL: Unit tests. See test results for more information.
• FAIL: Lint check. See test results for more information.
• FAIL: System tests (tags: installer NVDA). See test results for more information.
• Build (for testing PR): https://ci.appveyor.com/api/buildjobs/eef5p73c31hi8exv/artifacts/output/nvda_snapshot_pr15956-30485,a6fb32c0.exe
• CI timing (mins): INIT 0.0, INSTALL_START 0.9, INSTALL_END 1.0, BUILD_START 0.0, BUILD_END 25.4, TESTSETUP_START 0.0, TESTSETUP_END 0.4, TEST_START 0.0, TEST_END 4.5, FINISH_END 0.1

See test results for failed build of commit a6fb32c012

[Emil-18]

@seanbudd Speech is automaticly canceled almost constantly. Plus if all regions in the _regions list together contains over 100000 characters, _regions will not be appended to anymore unless it is cleared. I did it in this way because to my knowledge, there are no way to check if speech has actualy stopped or is just done speaking the current speech sequence when speech mode is set to off.

[AppVeyorBot]

• PASS: Translation comments check.
• PASS: Unit tests.
• FAIL: Lint check. See test results for more information.
• PASS: System tests (tags: installer NVDA).
• Build (for testing PR): https://ci.appveyor.com/api/buildjobs/re1xxtq4423eln19/artifacts/output/nvda_snapshot_pr15956-30493,f51a4217.exe
• CI timing (mins): INIT 0.0, INSTALL_START 0.9, INSTALL_END 1.0, BUILD_START 0.0, BUILD_END 24.9, TESTSETUP_START 0.0, TESTSETUP_END 0.3, TEST_START 0.0, TEST_END 16.4, FINISH_END 0.1

See test results for failed build of commit f51a421719