VoiceOver does not work when more info is opened for an element

[zacwest]

Checklist

• [X] I have updated to the latest available Home Assistant version.
• [X] I have cleared the cache of my browser.
• [X] I have tried a different browser to see if it is related to my browser.

Describe the issue you are experiencing

When opening a more info panel, none of the elements within are exposed to VoiceOver and it is not possible to dismiss the panel. Note in the STR below there is some nuance here that can make it work for the moment.

Describe the behavior you expected

Focus is moved to the more info panel and it is interactable.

Steps to reproduce the issue

1. Locate an element whose activation shows a more info panel.
2. Activate the element.
3. Note the more info panel is not exposed at all.
4. Activate the element again (as focus didn't move).
5. Note the more info panel is now exposed if you manually enumerate/focus on the elements within.

I would expect when the activation occurs it also moves focus.

What version of Home Assistant Core has the issue?

dev, frontend fb55ab197f77bc33429401dcbf97ce722898bf15

What was the last working version of Home Assistant Core?

No response

In which browser are you experiencing the issue with?

No response

Which operating system are you using to run this browser?

No response

State of relevant entities

No response

Problem-relevant frontend configuration

No response

Javascript errors shown in your browser console/inspector

No response

Additional information

Refs #8178.

[steverep]

@zacwest, are you referring to modal dialogs (e.g. for an entity's settings)? If so, this is a duplicate of #10754. It's not specific to iOS and I'm currently working on it.

If not, please be more specific about an exact button I should check out.

[zacwest]

I believe they are separate issues. In this issue, opening the modal not only doesn't move focus but the elements on the screen aren't interactable at all, even if you manually try and move the VoiceOver focus over to them, unless you trigger the 'open' a second time.

[steverep]

Yep, you're right. I just tested a quick fix. The focus part only is the duplicate. The need to click twice is specific to mobile.

And clicking twice isn't always a work around. Something fishier is going on. For example, turn on VoiceOver and try to open an entity's settings dialog from the entities table...

[digitaldarragh]

I don't know if this is of any benefit to you. But in modal dialogues, setting the tabindex to -1 has often helped me to set and retain focus within the contents of that modal dialogue. Also, please note that I don't have problems with these dialogues in Windows. In fact, they work really well. I'm sure you have seen the example on W3 schools relating to modal dialgues. It's quite complete.

[steverep]

I am trying to work on this bug and just wanted to report some findings:

• The issue title should be changed. This affects all dialogs and not just more-info, although behavior varies.
• The mwc-dialog works fine when loaded statically (see demo page), so this is something about the HA implementation.
• Dialogs follow one of 2 behaviors:
  • As described above, where focus seems to remain on the element that opened the dialog, and simulating a second click then will jump focus to the dialog.
  • Focus does not stay there and there's no way to get it inside the dialog. The work around here is to turn on "screen recognition" (experimental feature recently added to VoiceOver to help make inaccessible stuff accessible).
• Issue seems to be timing related. For example, I get different behavior for the area dialog by:
  • Merging in change by @bramkragten earlier changing paper-input to ha-text-field
  • New area versus existing area (same dialog code)
  • Awaiting update complete promise for ha-dialog

Code seems to be stuck inside the update cycle for the dialogs, but I haven't figured out where and why yet. If the first behavior occurs, the second click clears the glitch for some reason.

[steverep]

Bad news is I've ruled out a bunch of stuff but still haven't figured this one out.

Good news is that there's a consistent work-around. When VoiceOver is enabled, issue a double click to any button that opens a dialog (i.e. a quadruple tap after the button is selected). That seems to consistently open the dialog and set focus per #10754 which I'm also working on.

Can someone with access to android test this with Talkback enabled please and report the behavior? I don't have any android devices at the moment.

[steverep]

I had some more time to debug this one and traced the root cause to historically buggy implementation of aria-modal in Safari on iOS. Patching mwc-dialog to omit that attribute would be a temporary fix. Home Assistant's method for showing dialogs seems to exacerbate the webkit issue, but I haven't figured out why yet.

[github-actions[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[steverep]

Still very much an issue

[github-actions[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[steverep]

Definitely still an issue

[steverep]

My comment above was indeed correct - this is a bug in webkit with aria-modal.

I finally worked out a good patch for this, only to notice that very recent versions of webkit have now fixed it, probably with one of these bugs:

• VoiceOver highlight does not move to Shadow DOM button when using aria-modal and role
• WebKit does not break AX modality when focus is explicitly moved outside the modal

I cannot reproduce the issue now on iOS 16.2, but definitely still occurs on 15.6.1. @zacwest can you corroborate this or narrow the version further? I'm not even sure if this is iOS specific or also occurs on Mac.

I think I'll still create a PR for the patch to support devices no longer getting updates.

[zacwest]

Yeah this seems fixed for me in newer iOS. It's not super pleasing where the focus lands in the modal, though -- seems like it's the first text element inside, but it reads the previous screen's value while it's selected. Moving forward or back among the elements does read the right thing though.

[steverep]

Where the focus lands initially is what I've been working on as part of #10754, which got a little broken for more-info dialogs when they were merged with entity settings. It should always land on the current tab now. I need to fix that.

Do we have any analytics on what versions of iOS users are on? Wondering if I should bother with a fix now.

[zacwest]

Generally industry practice is to focus on latest versions for accessibility so it may not be worth worrying too much. Our opt-in rate for analytics on iOS is 24% and shows about 87% on iOS 16+.

[steverep]

Is there a separate opt-in for iOS? I don't see it in the app settings.

Given the high percentage though, I think we can mark this resolved.

[zacwest]

Yeah it's an iOS-wide setting called "Share with App Developers." The app itself doesn't have analytics.