nvda
nvda copied to clipboard
Windows Terminal: Leverage new UI Automation notifications support
Is your feature request related to a problem? Please describe.
With microsoft/terminal#12358, wt
now sends UIA notifications containing new text written to the terminal. To prevent conflict with our existing LiveText
implementation, NVDA PR #13261 ignores these notifications. However, replacing LiveText
with UIA notifications eliminates NVDA's need to diff the terminal (and possibly the need to listen to textChange
events), increasing performance and stability. See related issue #11002 (which is significantly improved when switching to UIA notifications).
Describe the solution you'd like
Consider rethinking how NVDA interacts with Windows Terminal:
- We no longer need to diff, so don't inherit from
LiveText
and instead create a new, separate class for these terminals supporting UIA notifications. - If we don't react to
textChange
any more (i.e. stop following theEnhancedTermTypedCharSupport
approach), consider a new way to detect when typed characters aren't being echoed to the screen (i.e. don't speak typed passwords).
Describe alternatives you've considered
Make no modifications.
CC @carlos-zamora.
I guess this is currently blocked by the fact that we can't yet distinguish between a terminal with and without notifications at the time of initializing the overlay classes.
I guess this is currently blocked by the fact that we can't yet distinguish between a terminal with and without notifications at the time of initializing the overlay classes.
Per official word from the Windows Terminal team (CCing @DHowett), NVDA should always assume that Windows Terminal supports notifications, since all versions that don't are now unsupported. I think we can now consider this unblocked.
I'm officially out of time to work on this issue (I start work on Microsoft Accessibility Insights later today), but I'll leave a few notes here if someone picks it up. Happy to review PRs and provide guidance where allowed by Microsoft policies.
- Since all active versions of Windows Terminal (
wt.exe
) now support notifications,NVDAObjects.UIA.winConsoleUIA.WinTerminalUIA
should not inherit fromNVDAObjects.behaviors.EnhancedTermTypedCharSupport
, because we explicitly don't wantLiveText
behaviour any more (we don't need to diff, unlike every other terminal in NVDA).- During development, maybe create a new
WinTerminalUIAWithNotifications
overlay class that doesn't inherit fromEnhancedTermTypedCharSupport
and choose which class to use inNVDAObjects.UIA.findOverlayClasses
via a feature flag. -
NVDAObjects.UIA.UIA
seems like a suitable new base class here. -
NVDAObjects.UIA.winConsoleUIA.WinConsoleUIA
(theconhost.exe
/Windows Console overlay class) should continue to haveLiveText
, since there are no plans to add notifications support at this time. (wt
will be the default terminal on Windows in a future update, so I don't know how much more workconhost
will get).
- During development, maybe create a new
- Since
WinTerminalUIA
no longer inherits fromTerminal
, you'll need to manually set the object's role (refer toNVDAObjects.behaviors.Terminal
). - The
WinTerminalUIA.event_UIA_notification
override that blocks notification events will need to be removed (or not added to the new class, depending on the development approach). - Since we don't need to diff, there's no need for NVDA to register for or receive
event_textChange
in terminals. Processing the number of text change events sent bywt
andconhost
is a huge bottleneck (probably the biggest contributer to performance issues in terminal today), so not registering for or receiving text changes should seriously improve performance/lockups when a lot of text is written to the terminal. I added global support for this inUIAHandler
by mappingUIAHandler.UIA.UIA_Text_TextChangedEventId
totextChanged
inUIAHandler.UIAEventIdsToNVDAEventNames
on Windows 10 (motivated by UIA console), maybe it can be made conditional/disabled somehow? - You'll need to figure out how to determine if typed characters are passwords/actually are printed to the screen without relying on text changes and a queue (i.e. my
EnhancedTermTypedCharSupport
approach).- If
config.conf["terminals"]["speakPasswords"]
isTrue
, all typed characters should always be spoken (i.e. don't do anything special). - If
config.conf["terminals"]["speakPasswords"]
isFalse
, typed characters that don't appear on-screen should be filtered out (i.e. never reported by NVDA/event_typedCharacter
should be thrown out).
- If
According to https://github.com/microsoft/terminal/issues/13601#issuecomment-1217144689, all WT versions that are currently supported have UIA notifications implemented. Therefore it is safe to drop the older approach for Windows Terminal, at least for NVDA 2022.4 and newer.