jcsteh
Action feedback bottlenecks in Mac OS
Reproduction steps:
- Create 20 tracks, then position yourself somewhere central in that list of tracks. The bug doesn't hinge on a specific number of tracks, we just need some room to manoeuvre in the next step.
- Now create one more track, and straight after hitting Enter, attempt to navigate quickly through the list of tracks in either direction. When the track name dialog closes, VoiceOver will start to read the window title etc, and action feedback from OSARA gets queued up.
- Now try hitting Control to silence VoiceOver midstream. Expected behaviour would be for those buffered messages to be cleared, but if you hit UpArrow or DownArrow again now, you should find that VO resumes speaking the queued up feedback and there's no way to skip it.
- Repeat the same test with any other dialog being dismissed, and you should get the same result.
@mulcahy1000 says that this is due to how VoiceOver handles a mix of new window events and announcement events (the latter event type being how OSARA provides action feedback). He hasn't been able to find a way to get announcement events to override new window events. Some time ago he experimented with using the system voice instead. That code can be found on his fork in a branch called MacTTS if it's of any interest, though he was quick to point out that it's not merge-ready.
If there's any other info I can chase down to be helpful, gimme a shout.
Thanks for the great summary Scott.
I would only add that I experimented with controlling VoiceOver directly from its apparent AppleScript interface, which indeed does work from AppleScript, but I couldn't figure out how to get it working from c++. Apparently there's a way, but I Haven't mucked about enough with Mac OS programming to know what it is.
Perhaps, we could fire a fake new window event or something to get VoiceOver to shut up. Not sure...
Yeah; reading Scott's description, a fake new window event is the first thought that popped into my head. The question is when we would fire it. We certainly don't want to fire it on every announcement. Perhaps we could fire it if an announcement occurs within a certain time (half a second? A few seconds?) after a dialog closes. That then raises the question of how we detect a dialog closing on Mac, since we can't use win events like we do on Windows.
I've just had a chat with a buddy who's an iOS developer. She's encountered this problem, and thinks that setting a timer for say half a second, firing setneedsdisplay when the timer expires then passing VoiceOver a string as an announce event will clear the backlog. She warned me that posting a blank announcement caused VO to crash in iOS, but says she hasn't tried it recently, so that might've been fixed in the meantime or might not even be an issue on Mac OS.
Most of that is Dutch to me, but hope it's useful. There's an open invitation for me to hit her up again if we need clarification.
One thing I've noticed in perusing the code, just because you hav. NSAccessibilityPostNotificationWithUserInfo and using NSAccessibilityPriorityHigh, that doesn't actually have any effect on the text spoken by VoiceOver. A screenreader, VoiceOVer in this case has to act on that message; stopping speech if necessary and then speaking that new announcement. According to this apple dev page.
An example of using this key is VoiceOver which decides whether to speak an announcement immediately or after the current speech has finished.
I'm not exactly sure this does anything. I think you have to explicitly tell voiceover to stop. I've experimented with putting it on NSAccessibilityPriorityLow and its still the same behavior across Osara. Unfortunately, the window title etc that gets read after leaving dialogues probably has priority over anything else. I believe that we would have to implement NSSpeechSynthesizer and use stopSpeech(). Although determining when to stop speech would still be an issue in this case. I was a bit surprised to see that Osara wasn't using NSSpeechSynthesizer in the first place. At this point that might be a bit of a bear to implement. although I don't mind working towards that. I was going to start implementing just something similar in some of my stuff.
NSSpeechSynthesizer documentation
I believe there's a build of OSARA somewhere (I've never seen the code) that does use NSSpeechSynthesizer. The problem with that is that it doesn't interact with VO in any way. For example, that's not going to output to braille, it's not going to respect your VO settings, it won't shut up when you tell VO to shut up, it'll potentially talk over the top of VO, etc. IMO, the way OSARA is doing it now is the correct way to do it, but unfortunately, we're constrained somewhat by bugs in VO.
I actually think the right way to do this might be to use VO’s AppleScript interface. This is how FloTools works via Keyboard Maestro, and it doesn’t seem to have these issues.
Unfortunately, a few hours of fiddling about a year ago didn’t produce anything useful. The auto-generated AppleScript interfaces from VO didn’t seem to include all the methods being used by real AppleScripts. It’s likely that I just don’t understand how this all works. :)
Yes, I threw together some code that uses the Mac system voice for OSARA output so I could play with it and see if it was an improvement, and I decided it wasn’t. :) Just different issues.
Code’s not fit for public consumption, but is in the MacTTS branch on my fork.
This came up in a conversation with Chi Kim on mastodon. Dropping relevant bits here in case they turn out to be useful:
It only happens when you close a window. VO reads the top thing on the window (project name reaper version, license) when you close a window. I can manually have VO read the same thing and just press up/down arrow and it immediately interrupts. It might be notification not being posted to the right element when a window disappears. Just speculation of course. Also, the bug about the buffer not being cleared described in step 3 seems fixed. If I interrupt the speech with Control and press up/down, it starts reading the tracks correctly. It does not resume the previous speech.
We use [NSApp keyWindow] to post the announcement notification. I believe that's the window that accepts input, so I guess it is possible that this isn't the window we should be using in the case of a foreground window change. We previously used mainWindow, but that didn't work when REAPER was full screen; see 5a5548dc. So, that raises the question: if this isn't the right window for us to use, what should we use?
Thanks for sharing these notes. I can confirm this behaviour. The interesting thing is that closing a window in an app like apple Mail does not make VoiceOver read the last bits of information as in Reaper. But I have certainly experienced this in other software packages. Numbers being one of them. This also explains a few things that we talked about on a "I am perplexed, what's next" meeting a few months back, where I tried to reproduce the issues somebody had on Mac OS, and could not reproduce it at all. So something is fixed. If I can test any build, please let me know. I try to follow OSARA dvelopment quite close, but cannot always respond the same day as a mention, and no programming skills at all!
