[Spec] VT Sequence for Screen Reader Control

[carlos-zamora]

This spec outlines...

• Windows Terminal and ConHost's implementation for the VT sequence introduced in Control Screen Reader from Applications (#18) · Issues · terminal-wg / specifications · GitLab.
• potential command-line applications/partners that would benefit from using this VT sequence and how they could leverage it
• necessary work on screen readers to take advantage of this feature (if any)

References

• Control Screen Reader from Applications (#18) · Issues · terminal-wg / specifications · GitLab
• #13666

[carlos-zamora]

@Tyriar since you wrote the spec in the Terminal WG repo, I'd love to get some feedback and thoughts here. 😊

@j4james you're generally pretty on top of what's going on in the VT sequence world. Curious if you have any thoughts here too.

@codeofdusk I'd also love your feedback since you're a screen reader user and you've contributed to NVDA.

Thanks all.

[codeofdusk]

As I've said before one-on-one, this will be quite tricky to get right. This is a very large change/project.

A few notes:

• I don't like the thought of disabling textChange events in response to new text from this feature. If users are using a UIA client that hasn't adopted this protocol, disabling events might have unexpected results.
  • In particular, if a screen reader (or by extension the user) hasn't opted into receiving events in this way, nothing should change.
• This will require significant buy-in from not just terminal, but also various command-line app and screen reader manufacturers.
  • Check whether ncurses would accept a patch to implement this protocol.
  • JAWS, Narrator, and NVDA cover the Windows screen reader market, but there are various other solutions on other systems (including console-only screen readers such as TDSR and Speakup in the Linux kernel, plus "audio-first" interfaces such as Emacspeak that provide shell access).

CC @josephsl, @leonardder, @michaelDCurran, @tspivey, @tvraman.

[zadjii-msft]

People don't care about a11y enough to add this to programs, shell scripts, etc. which is sad but true. The fact that the issue only got a single comment several months later kind of proved this imo.

That might be more of an indictment of terminal-wg moreso than demonstrative that people don't care about this issue. There's fundamentally no good way of doing anything like this currently, so I doubt anyone's bothered even thinking about it all that much.

The synchronized updates idea is a great alternative plan here. Perhaps we should work with the winget (@denelon), pwsh, gh folks to explore how using those sequences would modify the screen reader output. Where would they start/end the frame, and what would the resulting screen reader output be like?

[lhecker]

• @textshell's alternative proposals in https://gitlab.freedesktop.org/terminal-wg/specifications/-/issues/18#note_329814
  • Implementing synchronized updates and never update screen reader content during a sync update

Let's say I implement autocompletion for my shell and when you press tab on robocopy.exe it replaces it with Robocopy.exe (since that's the proper capitalization of the executable). If we diff the viewport in a trivial way, this would just read out "R". Or if I tab on roboc and get Robocopy.exe it then would read out "R" and "copy.exe", since those are the only parts that got changed. So, clearly, we need to announce entire words at least, but technically my shell didn't write "Robocopy" either - it wrote "Robocopy.exe", since that's the full application name. But how do we parse such "words"? Should we modify UAX#29 to recognize "." as being part of a word? How would such a diffing algorithm look like? Is it even feasible in the first place?

[Tyriar]

That might be more of an indictment of terminal-wg moreso than demonstrative that people don't care about this issue

I think you're being a little too optimistic here, during my years of work on terminals the only interest or mention of the topic has come from Microsoft people. But sure, some people will want to support it for sure, whether they actually do when prioritized against their backlog is another thing.

If we diff the viewport in a trivial way, this would just read out "R". Or if I tab on roboc and get Robocopy.exe it then would read out "R" and "copy.exe", since those are the only parts that got changed.

@lhecker a diff like that wouldn't make sense, the changed range is R to .exe, so you would read out Robocopy.exe.

[lhecker]

@lhecker a diff like that wouldn't make sense, the changed range is R to .exe, so you would read out Robocopy.exe.

Right, that addresses the latter example. What about the first one? And I really only intended them as examples. I'm sure there's way more examples one could contrive. For instance another one I had in mind was: What about applications that want to move some text left/right/up/down by a bunch of columns/lines? Or "scroll" some content themselves? Would the terminal have to cleverly identify identical pieces of "words" while also detecting their (positional) relationship with each other, so that false positives are suppressed?

[Tyriar]

Right, that addresses the latter example. What about the first one?

The first may be the same because the whole prompt likely got re-printed. I would want to tie the a11y improvements like this into our shell integration support so we would also know when we're in a prompt and where the start/end/rprompt/continuations are.

What about applications that want to move some text left/right/up/down by a bunch of columns/lines?

That's in the bucket of things that feel out of scope to me. We won't be able to make everything in the terminal accessible, I was aiming for better prompt interactions and better natural ltr/top to bottom text flow.

[carlos-zamora]

• I don't like the thought of disabling textChange events in response to new text from this feature. If users are using a UIA client that hasn't adopted this protocol, disabling events might have unexpected results.
• This will require significant buy-in from not just terminal, but also various command-line app and screen reader manufacturers.

@codeofdusk I'm a bit confused from the stuff above. The idea with this proposal is that no changes would be required on the screen reader side because they're handling textChanged events and notifications appropriately (NVDA has a special case here, but we'll get back to that in a minute). So theoretically, if this spec was implemented and an app (i.e. winget) started sending out these VT sequences, the terminal would just send less textChanged events or notifications with different payloads. So, the attached screen reader wouldn't know the difference and read out the notification as usual.

NVDA, of course, is special in that it's currently ignoring notifications unless a setting is enabled. If we gave the UIA notifications a different ID however, could NVDA whitelist that class of notifications?

[carlos-zamora]

Not sure what you mean by this, this proposal is essentially alt text, no?

Sort of? I see the similarity to alt text, but I think we should not embed the sequence into the buffer. This sequence should be limited to new output and the resulting notifications entirely. So on a resize or when in scan mode, whatever special text was notified out shouldn't be found.

#13666 is a standard example I can think of (gh pr create). It's not taking up the entire screen, it's just rewriting the prompt entirely because that's (presumably) the easiest way to rewrite the prompt. There's other lightweight examples that could benefit from this (i.e. PowerShell suggestions). Sure we're not solving the problem for heavy-weight apps like midnight commander, but (1) now it's on the app to decide if it's worth making it accessible and (2) I don't know if it's worth lumping those scenarios into this scope since other accessible options exist (like, idk if the screen reader community would even want to use it if there's non-cli apps that already meets their needs.)

[carlos-zamora]

Chatted with @Tyriar today. Here's some takeaways from that meeting:

Main Benefits of this Approach

• We need a way for apps to be more in control of making their content more accessible.

Main Concerns

• Adoption of this seems difficult. CLI apps don't really want to think about how to be more accessible. It'd be best to give them a more fine-tuned solution.
• Yes, the above solution is versatile, but that also means it's a bit hacky because the developer has to think about how to make their workflow more accessible, vs simply saying "I'm in a 'select an option' workflow, I'll use the guidelines designed for that".

Other Proposals

We discussed a few ideas that could be more fine-tuned. They're not mutually exclusive. Also note, these are very lightweight specs. They definitely need some fleshing out, but I'll do that when I add them to the actual spec (at some point).

Idea 1: Decorative Tags

• A VT sequence marking a range of text as purely decorative.
• Example:
  • App displays progress bar as such: [===___] 50%
• Solution:
  • The region of text that looks like this [===___] would be marked as decorative and ignored by the screen reader. This would be accomplished by having the terminal not append that to the output text in a UIA notification event.
  • The 50% displayed at the end would still be read out as normal text, no additional work would be required.

Idea 2: Semantic Embedding

• Inspired by HTML, Accessible Rich Internet Applications (ARIA) provide hints to the screen reader. We could use a combination of HTML-like tags and ARIA labels to make workflows more clear.
• Example:
  • gh pr create outputs a prompt with multiple choices (see this comment for what it looks like)
• Solution:
  • add an option tag for each option presented
  • the selected option has an additional tag (maybe through an ARIA-like label?)
  • Terminal knows to read "X selected"

Idea 3: Flag to know if a screen reader is active

• An environment variable or flag that lets CLI apps know if a screen reader is active.
• NOTE: PowerShell does this to disable PSReadLine. We should definitely check out how they do that.
• Example:
  • gh pr create could benefit from adding numbers before each option to make it more clear that they are distinct options.
• Solution:
  • gh.exe checks this flag. If a screen reader is active, change the layout of gh pr create to add numbers to the front.
• This could also be used for more complex scenarios. Consider PowerShell's MenuComplete option. Each suggestion is displayed in a grid, which means that multiple options are on a single line. This makes it very difficult for screen reader users to identify distinct options. If PowerShell could know when a screen reader is attached, they could force a different behavior.

[j4james]

An environment variable or flag that lets CLI apps know if a screen reader is active.

Ideally you want this to work over remote connections too, and the terminal assumedly wouldn't be able to set an environment variable in that case. So my recommendation would be using one of the standard VT reporting sequences for this.

For scenario 1, if we were using something like DECSCA, an app could tell if the new "decorative" attribute was supported by sending a DECRQSS query. And the spec could require that a terminal not respond successfully to that query unless there is actually an active screen reader.

If the solution is mode based, the app can query whether the mode is supported with a DECRQM query. And again the spec could require the presence of a screen reader.

For a more general way to query whether a screen reader is present, regardless of support for any particular functionality, we could define a new DSR query. For a similar use case, look at the way the printer status is reported for DSR-PP: https://www.vt100.net/docs/vt510-rm/DSR-PP

And as a last restore, if we just wanted a way for terminals to indicate that they support this screen reader spec in general, we could define a new feature number that is reported in the DA response. It would be preferable if we could avoid this though.

[github-actions[bot]]

@check-spelling-bot Report

:red_circle: Please review

See the :open_file_folder: files view or the :scroll:action log for details.

Unrecognized words (6)

dispatchees finsh imm teminal ther versionf

Previously acknowledged words that are now absent

aabbcc abbcc DECCARA DECERA DECFRA DECRARA DECSACE DECSERA Imm Qaabbcc Qxxxxxxxxxxxxxxx Tdd xwwyzz xxyyzz ZYXWVU ZYXWVUTd :arrow_right:

To accept :heavy_check_mark: these unrecognized words as correct and remove the previously acknowledged and now absent words, run the following commands

... in a clone of the [email protected]:microsoft/terminal.git repository on the dev/cazamor/spec/a11y-vt-seq branch (:information_source: how do I use this?):

curl -s -S -L 'https://raw.githubusercontent.com/check-spelling/check-spelling/v0.0.21/apply.pl' |
perl - 'https://github.com/microsoft/terminal/actions/runs/3568352485/attempts/1'

:pencil2: Contributor please read this

By default the command suggestion will generate a file named based on your commit. That's generally ok as long as you add the file to your commit. Someone can reorganize it later.

:warning: The command is written for posix shells. If it doesn't work for you, you can manually add (one word per line) / remove items to expect.txt and the excludes.txt files.

If the listed items are:

• ... misspelled, then please correct them instead of using the command.
• ... names, please add them to .github/actions/spelling/allow/names.txt.
• ... APIs, you can add them to a file in .github/actions/spelling/allow/.
• ... just things you're using, please add them to an appropriate file in .github/actions/spelling/expect/.
• ... tokens you only need in one place and shouldn't generally be used, you can add an item in an appropriate file in .github/actions/spelling/patterns/.

See the README.md in each directory for more information.

:microscope: You can test your commits without appending to a PR by creating a new branch with that extra change and pushing it to your fork. The check-spelling action will run in response to your push -- it doesn't require an open pull request. By using such a branch, you can limit the number of typos your peers see you make. :wink:

If the flagged items are :exploding_head: false positives

If items relate to a ...

• binary file (or some other file you wouldn't want to check at all).

  Please add a file path to the excludes.txt file matching the containing file.

  File paths are Perl 5 Regular Expressions - you can test yours before committing to verify it will match your files.

  ^ refers to the file's path from the root of the repository, so ^README\.md$ would exclude README.md (on whichever branch you're using).

• well-formed pattern.

  If you can write a pattern that would match it, try adding it to the patterns.txt file.

  Patterns are Perl 5 Regular Expressions - you can test yours before committing to verify it will match your lines.

  Note that patterns can't match multiline strings.

[github-actions[bot]]

@check-spelling-bot Report

:red_circle: Please review

See the :open_file_folder: files view or the :scroll:action log for details.

Unrecognized words (2)

dispatchees ther

Previously acknowledged words that are now absent

Hirots :arrow_right:

To accept :heavy_check_mark: these unrecognized words as correct and remove the previously acknowledged and now absent words, run the following commands

... in a clone of the [email protected]:microsoft/terminal.git repository on the dev/cazamor/spec/a11y-vt-seq branch (:information_source: how do I use this?):

curl -s -S -L 'https://raw.githubusercontent.com/check-spelling/check-spelling/v0.0.21/apply.pl' |
perl - 'https://github.com/microsoft/terminal/actions/runs/3708027252/attempts/1'

:pencil2: Contributor please read this

By default the command suggestion will generate a file named based on your commit. That's generally ok as long as you add the file to your commit. Someone can reorganize it later.

:warning: The command is written for posix shells. If it doesn't work for you, you can manually add (one word per line) / remove items to expect.txt and the excludes.txt files.

If the listed items are:

• ... misspelled, then please correct them instead of using the command.
• ... names, please add them to .github/actions/spelling/allow/names.txt.
• ... APIs, you can add them to a file in .github/actions/spelling/allow/.
• ... just things you're using, please add them to an appropriate file in .github/actions/spelling/expect/.
• ... tokens you only need in one place and shouldn't generally be used, you can add an item in an appropriate file in .github/actions/spelling/patterns/.

See the README.md in each directory for more information.

:microscope: You can test your commits without appending to a PR by creating a new branch with that extra change and pushing it to your fork. The check-spelling action will run in response to your push -- it doesn't require an open pull request. By using such a branch, you can limit the number of typos your peers see you make. :wink:

If the flagged items are :exploding_head: false positives

If items relate to a ...

• binary file (or some other file you wouldn't want to check at all).

  Please add a file path to the excludes.txt file matching the containing file.

  File paths are Perl 5 Regular Expressions - you can test yours before committing to verify it will match your files.

  ^ refers to the file's path from the root of the repository, so ^README\.md$ would exclude README.md (on whichever branch you're using).

• well-formed pattern.

  If you can write a pattern that would match it, try adding it to the patterns.txt file.

  Patterns are Perl 5 Regular Expressions - you can test yours before committing to verify it will match your lines.

  Note that patterns can't match multiline strings.

[github-actions[bot]]

@check-spelling-bot Report

:red_circle: Please review

See the :open_file_folder: files view or the :scroll:action log for details.

Unrecognized words (6)

colsize Hostettler IAccessible Koblinger rowsize takeaways

Previously acknowledged words that are now absent

Hirots :arrow_right:

To accept :heavy_check_mark: these unrecognized words as correct and remove the previously acknowledged and now absent words, run the following commands

... in a clone of the [email protected]:microsoft/terminal.git repository on the dev/cazamor/spec/a11y-vt-seq branch (:information_source: how do I use this?):

curl -s -S -L 'https://raw.githubusercontent.com/check-spelling/check-spelling/v0.0.21/apply.pl' |
perl - 'https://github.com/microsoft/terminal/actions/runs/3753453952/attempts/1'

Errors (1)

See the :open_file_folder: files view or the :scroll:action log for details.

:x: Errors	Count
:x: forbidden-pattern	1

See :x: Event descriptions for more information.

:pencil2: Contributor please read this

By default the command suggestion will generate a file named based on your commit. That's generally ok as long as you add the file to your commit. Someone can reorganize it later.

:warning: The command is written for posix shells. If it doesn't work for you, you can manually add (one word per line) / remove items to expect.txt and the excludes.txt files.

If the listed items are:

• ... misspelled, then please correct them instead of using the command.
• ... names, please add them to .github/actions/spelling/allow/names.txt.
• ... APIs, you can add them to a file in .github/actions/spelling/allow/.
• ... just things you're using, please add them to an appropriate file in .github/actions/spelling/expect/.
• ... tokens you only need in one place and shouldn't generally be used, you can add an item in an appropriate file in .github/actions/spelling/patterns/.

See the README.md in each directory for more information.

:microscope: You can test your commits without appending to a PR by creating a new branch with that extra change and pushing it to your fork. The check-spelling action will run in response to your push -- it doesn't require an open pull request. By using such a branch, you can limit the number of typos your peers see you make. :wink:

If the flagged items are :exploding_head: false positives

If items relate to a ...

• binary file (or some other file you wouldn't want to check at all).

  Please add a file path to the excludes.txt file matching the containing file.

  File paths are Perl 5 Regular Expressions - you can test yours before committing to verify it will match your files.

  ^ refers to the file's path from the root of the repository, so ^README\.md$ would exclude README.md (on whichever branch you're using).

• well-formed pattern.

  If you can write a pattern that would match it, try adding it to the patterns.txt file.

  Patterns are Perl 5 Regular Expressions - you can test yours before committing to verify it will match your lines.

  Note that patterns can't match multiline strings.