VoiceOver for macOS Changes Requested: "Trigger an alert" (Alert Example, Test 1, V24.04.26) for Candidate Report

[cookiecrook]

Description of Behavior

Several of the subtests are written in a way that may lead inexperienced tester to assume VoiceOver is not conveying the alert role, but it is…

For example, several results are incorrectly noted as:

MAY: Convey role 'alert' … Unsupported

In this case, I would change the existing test expectation to something like:

In the case of audible testing, "Convey role alert, either through spoken role description "alert", or through the alert sound icon played simultaneously behind the spoken label."

And add a new test case for deaf-blind braille users (where audio cannot be a valid form of informative conveyance):

In the case of braille testing, Convey role alert, either through brailled role description "alert", or through braille emphasis.

• Note 1: Currently in VoiceOver, alert emphasis is conveyed dots using dots 1–8 (⣿) rendered on either side of the alert text, but this is an interface implementation detail that is subject to change and may be different between platforms or screen readers.
• Note 2: Test VoiceOver's braille rendering either through either an external refreshable display or using the VoiceOver braille caption panel.

Both of the above can be RFC-2119 MUST requirements with default settings...

I'm hopeful that will prevent inexperienced testers from incorrectly marking these as "unsupported."

Thanks.

Test Setup

• Test File: test-03-triggerAlert-voiceover_macos.collected.html
• AT: VoiceOver for macOS

[css-meeting-bot]

The ARIA-AT Community Group just discussed Request for change to alert test plan.

The full IRC log of that discussion

<jugglinmike> Topic: Request for change to alert test plan
<jugglinmike> github: https://github.com/w3c/aria-at/issues/1316
<jugglinmike> Matt_King: This is feedback from James Craig at Apple
<jugglinmike> Matt_King: I wasn't able to test this prior to today's meeting, so we'll make the assumption that we can reproduce the behavior that James is reportng
<jugglinmike> Matt_King: It involves an earcon for the alert role
<jugglinmike> Matt_King: This is similar to test cases for JAWS and NVDA where we have a mode-switching test, and the mode switch is conveyed only via sound (with the default settings, anyway)
<jugglinmike> Matt_King: Both JAWS and NVDA have settings to convey that via speech (And thus capture it in a response), but we are not asking the tester to change their configuration for this specific test
<jugglinmike> Matt_King: We could treat this similar to how we test mode-switching tests. Those are currently not "bot testable" due to the sound. They could be made bot-testable if we changed the default settings. But that is a bit of a gap in our automatic verdict assignment capabilities
<jugglinmike> Matt_King: We can set aside that issue for now, but I want to focus on how we handle feedback for this one specific test
<jugglinmike> james: The assertion was classified as "MAY" due to feedback from Vispero and others
<jugglinmike> james: So, if we say that VoiceOver fails this assertion, that does not actually reduce their score
<jugglinmike> Matt_King: They are aligned with the assertion priority
<jugglinmike> james: They are saying that it could be changed...
<jugglinmike> Matt_King: I didn't read that as suggesting that they want the priority increased in order to approve it
<jugglinmike> Matt_King: We made a decision that's actually contrary to the intent of the ARIA specification and is more aligned to the real-world practice where "alert" is misused. We hypothesize that misuse is most of the time
<jugglinmike> Matt_King: It's not a great rationalization, to be honest, but it is a practical one that matches JAWS and NVDA's design
<jugglinmike> Matt_King: It was intended to do exactly what VoiceOver is doing--to call your attention
<jugglinmike> James: I feel like that is part of the problem. If a web app wants to draw your attention, it shouldn't be the responsibility of the screen reader.
<jugglinmike> Matt_King: Yeah, that's true. It could be a browser responsibility. That is, practically speaking, a valid approach and something that I'm almost motivated to raise an ARIA issue for
<jugglinmike> James: There becomes an issue for me when too much is placed on the screen reader. Then, web authors have a lot less control. And it implies that the only people who would benefit from sounds are screen reader users
<jugglinmike> james: But if the app implements its own sound, then it doesn't have a way to turn off VoiceOver's sound
<jugglinmike> Matt_King: Right
<jugglinmike> Matt_King: Tabling that for now, in the same way that JAWS and NVDA make unique sounds when their mode switches, should we be marking this assertion as "supported"?
<jugglinmike> james: The wording is out of date with our current practices
<jugglinmike> Matt_King: agreed, but we can address that separately
<jugglinmike> Matt_King: The VoiceOver help does include a feature that allows you to hear every sound, and it describes the meaning of the sound. We can validate that it is the appropriate sound
<jugglinmike> Joe_Humbert: I don't remember hearing a sound. I probably would have left a note regarding a sound
<jugglinmike> james: They could be alluding to a sound that was not present when the testing was conducted.
<jugglinmike> james: It's on them to be explicit about what they're pointing out. It's not clear what the sound is and when it was added.
<jugglinmike> james: We should verify those details
<jugglinmike> james: Does VoiceOver enable audio ducking by default?
<jugglinmike> IsaDC: Yes, it does
<jugglinmike> james: So it could be the case that the sound is subtle, and it was ducked
<jugglinmike> Joe_Humbert: So they're saying that this is a new sound or that it is a sound that people have to enable specific settings to hear the sound
<jugglinmike> Joe_Humbert: If that's the case, that seems pretty extreme. If we, as professionals, don't know about this, then I can't say that even power users would know to do that
<jugglinmike> Matt_King: I'm testing this, now. It does make a sound that I do recognize as distinct from other VoiceOver sounds. I do think that it's making an "alert" sound in this situation
<jugglinmike> Matt_King: I've heard this sound before. It's different from the sound you get if macOS is prompting you in the background for a password or something. It's definitely more subtle than that one
<jugglinmike> Matt_King: I am using macOS 15.6.1 and whatever version of Safari came with that (perhaps version 18)
<jugglinmike> Joe_Humbert: I just did the same thing, and I did not hear anything
<jugglinmike> Joe_Humbert: And I'm on 15.7.1
<jugglinmike> Joe_Humbert: When I trigger an alert on the APG example page, I see it visibly open up, but I hear no sound effects, and VoiceOver says nothing
<jugglinmike> Joe_Humbert: This is the "alert" example
<jugglinmike> Matt_King: The last time, I used "VO + space-bar" to trigger it
<jugglinmike> Joe_Humbert: I used "enter"
<jugglinmike> Matt_King: Relative to the volume of the voice, the sound is not nearly as subtle as the "activation" sound. It's present, but it's underwhelming
<jugglinmike> Matt_King: I just tried it another way, and it's very consistent for me. Though you have to reload the page if you want to trigger it a second time
<jugglinmike> james: My goodness
<jugglinmike> Matt_King: I don't know if that is a Safari-specific thing. You should be able to trigger the alert many times in a row
<jugglinmike> Matt_King: Doesn't the alert disappear visualy
<jugglinmike> Matt_King: This isn't the "click", this is a dissonant chord sound
<jugglinmike> james: The sound I'm hearing is probably the activation sound--two clicks with slightly different pitches
<jugglinmike> s/james/Joe_Humbert/
<jugglinmike> Joe_Humbert: I'm on Safari 26
<jugglinmike> Matt_King: I want to return to the hypothetical: let's say that everybody was getting the same result that I and James Craig observe. A distinct sound is played. For people who get that experience, should we say that the assertion "may" convey...
<jugglinmike> james: Do we say that the sound played by NVDA satisfies the assertion?
<jugglinmike> Matt_King: We do
<jugglinmike> james: So the answer to me is that, if VoiceOver conveys the role via a sound, then it should pass the test
<jugglinmike> james: However, I can understand the objection in the group here today. Because it does explicitly mention "inexperienced testers", but the people here today have the most experience, and they are not observing it
<jugglinmike> Matt_King: Could you comment with your experience and to share your macOS and Safari versions when you do?
<jugglinmike> IsaDC: I can do that
<jugglinmike> IsaDC: I can test the braille behavior, as well
<jugglinmike> Matt_King: That would be great. Thank you
<jugglinmike> Matt_King: I have another sort of related question for us. If this were to change in the future (and they didn't play the sound), then a bot would not be able to detect that change. I can think of a couple ways of approaching this. In this particular case, since the non-audio case would not be detected by the bot (since the bot doesn't receive braille instructions, either), I think we need a way to designate that some tests always requi
<jugglinmike> re a human tester
<jugglinmike> james: I think so, too
<jugglinmike> james: They are kind of pointing out that we are mainly testing speech, and that the project is not currently taking non-auditory feedback into account
<jugglinmike> james: There is a world in which the project addresses that in a truly holistic way. That's a huge lift, so I think in the mean time, we could have a flag that designates some tests as being "untestable by a bot"/"always needs human verification"
<jugglinmike> Matt_King: When we do automated testing, those assertions would need to be left as not set
<jugglinmike> James: Would they be marked only for VoiceOver, though?
<jugglinmike> Matt_King: Yes. The flag would need to be set at the command-assertion level. That becomes more difficult--not insurmountable, but a little bit
<jugglinmike> Matt_King: We could set it at the assertion level or at the command-assertion level
<jugglinmike> Matt_King: In both cases, I think it would have to be a new column in the CSV
<jugglinmike> james: I think this is another assertion exception
<jugglinmike> Joe_Humbert: I did re-test quickly, and I did experience. The problem I found is that it plays it almost concurrently with the activation sound, so unless you are specifically trying to find it, then you will miss it
<jugglinmike> james: And that somewhat goes against the intent of the ARIA specification design on this
<jugglinmike> james: But regarding the CSV, the format could go after the ID of the assertion
<jugglinmike> Matt_King: Like a whole separate word
<jugglinmike> Matt_King: I'll raise an issue for this. It can hopefully make our bot reporting more accurate over time
<jugglinmike> Zakim, end the meeting

[jha11y]

After retesting on MacOS 15.7.1 with Safari 18.6 and on MacOS 15.7.1 and Safari 26.0.1 using the ARIA APG example (https://www.w3.org/WAI/ARIA/apg/patterns/alert/examples/alert/), I am able to hear the "Alert role" sound.

But this "Alert role" sound seems (to my hearing capability) to be played either simultaneously with the "activation click" sound or immediately after the "activation click" sound making it hard to discern the individual sounds. I was only able to discern the sound after hearing it once using headphones at maximum volume.

[cookiecrook]

In VoiceOver usability testing, most daily VO users can reliably hear up to at-least three simultaneously layered sound icons (many can distinguish more), so the two "alert+focus" sound icon behavior you describe is by design. The multiple sound icon behavior is intended to richly convey a dense interface without the tedium of higher speech verbosity...