david-vanderson
Some Accessibility bugs
Sorry for the super generic issue title, not really sure what I should title this since these are just issues I encountered while playing with the demos. This is quite rambly, sorry!
First, there's a slight delay/lag in the DirectX 11 demos. Not entirely sure why but this seems go vanish with SDL3. Godot had this same issue; not sure how they fixed it.
Another issue Godot had that your running into is edit boxes not really acting like edit boxes. As in, I can focus them, but I can't navigate them with the arrow keys, or read the line with the up/down arrows.
Another is that the menu bar (the file/edit menus) are focusable. I don't know if you have to do something different to make them act more like menu bars; I wonder if we could potentially simulate OS behavior here? I don't really know how best to do that or if AccessKit would even allow us to do that.
Lastly, the tree view widget is... Kinda broken. If you launch NVDA and run the sdl3-standalone app, you'll notice a few things after going into the demo window and selecting the reordering/tree button:
- Tree view widgets appear in tab order. Usually you want the tree view itself to be in tab order but not the individual nodes. Ideally, a tree view is navigable with the arrow keys: up and down navigate among either the nodes at the current level or, if the node has children and is expanded, descends to the next level automatically, and so on all the way down. Left and right arrows collapse and expand nodes: if you are on a node and it is collapsed, right arrow expands the node but does not descend into it, and left arrow anywhere in a list of nodes which has a parent node at the top jumps to that parent node first, and another press of the left arrow collapses it. If your already on the parent node, one press of the key collapses the node.
- Tree view items aren't read right now; NVDA just says "tree view item". (It does announce the level your at though!) I was able to sort-of navigate the tree with object nav, but it felt awkward since I also saw the graphics/images and such.
Tabbing also felt a bit weird. Like, I would tab to the end of the window, and hitting tab again would bring me to the actual window itself and nothing would be focused again. I'm not sure if we want this behavior or not. Part of me wants it to be like this (after all, that could be where you might want to pass all keyboard input through to the application), like a keyboard area of sorts. But part of it doesn't feel very natural and definitely gives "this isn't a native UI" vibes. Which isn't a bad thing, by any means. Sometimes, you want that.
I'm not really sure if elements not appearing as part of groups when they are supposed to be grouped (or I would assume they would usually be that way) is just not that being implemented or if that's intentional. Some other thoughts as I go through the app:
- In the basic widgets demo, in the "Radio Buttons" grouping, each radio button is in tab order. Usually when something is in a group, the group is in tab order, and it's children can be navigated via the arrows.
- When I navigate the options in the combo boxes, the choices force my focus in the combo box while it is expanded, such that I can't tab out of it without first hitting escape. This may be a nitpick and not too big of a problem.
- The sliders increment is a bit too small. You might want to change this to a float/double spin button/box; sliders for screen readers are normalized to whole numbers only, and a change that is only a fractional change won't be announced until the integer portion changes. That can lead to people thinking that something is broken. This kind of behavior (misbehavior?) is not uncommon though.
- The icon browser has some strangely named buttons, but I honestly am uncertain how exactly you could make a browser of icons accessible. You could be as descriptive as you want but words can only communicate so much.
- The "multi-line button" was a bit confusing at first but that's okay.
- You have some automatic label associations already working which is awesome. For example, if I tab to the text entry edit box, NVDA appropriately says "Text Entry edit" instead of just "Edit". Whatever your doing with that layout, your doing it perfectly fine.
- Multi-line edit boxes, like single-line ones, aren't recognized as such; in the text entry demo, if I tab to the multi-line one, NVDA just says it's an edit but doesn't say "Edit multi line" like it usually might. The similar edit box problems occur with this as it does with other edit boxes.
- The plots demo is inaccessible but this is another demo/widget that I don't think can really be made all that accessible unless you want to, Idk, present it as a table maybe? I've seen online apps make graphs and such accessible (e.g. Desmos) but I think they had to use ARIA for that. AccessKit does give you ARIA-like functionality so maybe that might work, if we can make plots/charts/graphs focusable and then give ARIA notifications that way?
- The Menus / Focus demo's tabs are also focusable; was that intentional or were all the tabs supposed to be a part of a larger tab control?
- The direct dialogs don't auto-announce the text in them. I'll be honest: I don't know how windows does this with functions like MessageBox. If I use the NVDA log viewer (NVDA + F1) to inspect the IAccessible information, it doesn't really tell me much. I know it's possible to do, I just don't recall how off the top of my head.
- Grids also have their data items focusable.
This is really great. Thank you so much.
Some comments / questions below:
-
Re: Edit boxes. Yeah that is a known issue. We need to implement the 'text run' role to tell AccessKit where each word/letter starts and ends etc for this to work. It is something we'll be looking into soon.
-
Regarding the menu bar focusing and "making them act more like menu bars". Could you expand on that point a little? It's not clear to me how the menus are behaving differently.
-
Tree view widget - Likely due to an issue we had to revert recently. Hopefully will have a fix for that soon.
-
Tabbing - When you get to the end of the tab order, DVUI focuses nothing before focusing the first widget again. This is not something common on Windows applications, but it does happen for some of them. Is it a big issue or just a minor annoyance?
-
Groups - Yeah, this one is tricky just because of the way DVUI works. A surrounding box needs to be manually created with a role of group. I've not gone through and done this for everything in the demo. But we also don't really have a 'radio group' widget. There is no visual indication they are in a group in the demo.
-
Tab order for groups. That is interesting. I'm not sure we can do that because of the above. But will think about it.
-
Re: Icon browser. This is using the "fallback" labels. If a label is not specified, it basically uses the icon name, which comes from the filename in this case. So, it's a case of having some label is better than no label in this particular situation.
-
Plots. I wasn't really sure what to do with them. We can try out a few things, but it's probably a low priority right now. I think in the interim, we just encourage people to give them a descriptive label at least.
-
Re: Focus for the Tabs in the demo tabs. Could you expand on this point as well? You can focus those "tabs" using the tab key, and select them with space etc. So, they are focusable, in a similar way to menus.
-
When you say the dialogs don't auto-announce, you mean that is something that happens as soon as the dialog opens, without you having to do anything else to read the screen? If so, I think I know how to do that.
-
With the grid focus. What is the expected behaviour here? I think I did that on purpose so you could tab to each entry and have it read. Is there a better way to present the grid? TBH the grids were a bit tricky because we don't require people to create rows and columns as such. Hopefully grids at least make some sort of sense. Did the sorting and selection seem OK? That also was a bit of a compromise.
-
Re: Comboboxes: I think that's just how they work at the moment. I'll leave that one for DV.
I tried to collect your points below as TODOs. Hopefully I didn't miss anything?
Accessibility TODO
- [ ] Investigate Direct X delay / lag. This is the only one not using the subclassing adapter.
- [ ] Focus issues with menu bar / items?
- [ ] Implement text run for edit boxes
- [ ] Re-implement multi_line edit role for multi-line edit boxes.
- [ ] Re-test tree widget after fixes to prevent tree_item role being propagated to implementing box widgets.
- [ ] Should we have an "unfocused tab index" when accessibility is enabled?
- [x] Investigate any improvements to groups and navigation within a group?
- [ ] Investigate issues with dialogs not announcing text. Maybe we need to use nodeSetLive to force this?
- [ ] Investigate grid cell focus issue
- [ ] Make plot widget more accessible. E.g. by providing data points as values.
- [ ] Make sure tabs are children of a 'tab widget' role.
Really appreciate this.
This is really great. Thank you so much.
No problem!
Some comments / questions below:
1. Re: Edit boxes. Yeah that is a known issue. We need to implement the 'text run' role to tell AccessKit where each word/letter starts and ends etc for this to work. It is something we'll be looking into soon.
Sweet!
2. Regarding the menu bar focusing and "making them act more like menu bars". Could you expand on that point a little? It's not clear to me how the menus are behaving differently.
Sorry, didn't mean to be unclear.
In a traditional GUI application, the menu bar is different from ordinary widgets in tab order. Pressing alt (or whatever platform equivalent of alt you have available) typically brings it up and makes it the focused widget; pressing it again makes it vanish. Menus in menu bars typically aren't in tab order of a typical dialog/window. Having menu bars function this way (where alt/meta) summons/banishes them also enables pretty much automatic accelerator keys, where, say, alt-f brings up the file menu, alt-e the edit menu, etc.
3. Tree view widget - Likely due to an issue we had to revert recently. Hopefully will have a fix for that soon.
Oh very nice! Can't wait to test it out!
4. Tabbing - When you get to the end of the tab order, DVUI focuses nothing before focusing the first widget again. This is not something common on Windows applications, but it does happen for some of them. Is it a big issue or just a minor annoyance?
I'd categorize it more as a minor annoyance than anything else. I think the "best" thing to do here would be to make it togglable, and change behavior depending on, say, this is a dialog or an ordinary window. If it's a dialog, this behavior feels a bit counterintuitive, but for a window not so much.
5. Groups - Yeah, this one is tricky just because of the way DVUI works. A surrounding box needs to be manually created with a role of group. I've not gone through and done this for everything in the demo. But we also don't really have a 'radio group' widget. There is no visual indication they are in a group in the demo.
Oh really? Interesting. NVDA thinks they're grouped, and reports several other groupings throughout other widget demos. Usually a "grouping" is just a group box, and you typically set the group box to be non-focusable and the widgets within as focusable. Then you just let the screen reader do it's thing. I think that's all you have to do and the screen reader will take care of auto-focusing.
6. Tab order for groups. That is interesting. I'm not sure we can do that because of the above. But will think about it.
Again, a group is just a group box technically. At least I'm pretty sure.
7. Re: Icon browser. This is using the "fallback" labels. If a label is not specified, it basically uses the icon name, which comes from the filename in this case. So, it's a case of having some label is better than no label in this particular situation.
Understandable. As I said, I don't really know what else you could possibly do here, given that icons are just images.
8. Plots. I wasn't really sure what to do with them. We can try out a few things, but it's probably a low priority right now. I think in the interim, we just encourage people to give them a descriptive label at least.
Yeah I'm honestly completely uncertain how we'd tackle these. I don't think AccessKit even has a role for them, so if we did do something we'd have to fake a bunch of stuff. Plots are one of these widgets where forcing a specific accessibility role is a good idea, provided you actually implement what the screen reader expects for that role.
9. Re: Focus for the Tabs in the demo tabs. Could you expand on this point as well? You can focus those "tabs" using the tab key, and select them with space etc. So, they are focusable, in a similar way to menus.
What I meant to ask is: are they supposed to be a part of a larger tab control? Usually a tab is a collection of items under a tab control (for example, in Windows Forms you have TabControl Control and in QT you have QTabWidget). Tabs usually aren't something you focus with the tab key directly, but are instead grouped together. That way, when the group is focused, you can hit left/right arrows to move among tabs. Right now, you can't do this, because each individual tab is considered a separate widget in the accessibility tree.
10. When you say the dialogs don't auto-announce, you mean that is something that happens as soon as the dialog opens, without you having to do anything else to read the screen? If so, I think I know how to do that.
Yup, exactly. If you want to see what I mean, open any windows message box with NVDA active.
11. With the grid focus. What is the expected behaviour here? I think I did that on purpose so you could tab to each entry and have it read. Is there a better way to present the grid? TBH the grids were a bit tricky because we don't require people to create rows and columns as such. Hopefully grids at least make some sort of sense. Did the sorting and selection seem OK? That also was a bit of a compromise.
I'm... Honestly not certain how people present this ordinarily. I... Think... That grids are another widget where the arrows are supposed to direct things, not tab. I didn't really test grids too much so couldn't evaluate the sorting behavior, but if you like I can definitely dig deeper into them.
12. Re: Comboboxes: I think that's just how they work at the moment. I'll leave that one for DV.
Sounds good. Like I said, I'm not going to make a huge fuss over it, it's just a bit unusual.
I tried to collect your points below as TODOs. Hopefully I didn't miss anything?
Accessibility TODO
* [ ] Investigate Direct X delay / lag. This is the only one not using the subclassing adapter.[ ] Focus issues with menu bar / items?[ ] Implement text run for edit boxes[ ] Re-implement multi_line edit role for multi-line edit boxes.[ ] Re-test tree widget after fixes to prevent tree_item role being propagated to implementing box widgets.[ ] Should we have an "unfocused tab index" when accessibility is enabled?[ ] Investigate any improvements to groups and navigation within a group?[ ] Investigate issues with dialogs not announcing text. Maybe we need to use nodeSetLive to force this?[ ] Investigate grid cell focus issue[ ] Make plot widget more accessible. E.g. by providing dara points as values.Really appreciate this.
Yeah, I think you got everything. And I'm happy to help! This is one of the UI toolkits that is doing this from the beginning instead of bolting it on as an afterthought, so I really hope it takes off. I'm thinking that in future I could propose a C API too because I'd love to use this in non-Zig applications. I can't wait to see where this goes!
@ethindp Thank you so much for the careful and detailed feedback!
@phatchman thanks for going through it and making the checklist!
I will need to go through this more carefully, but one big thing seems to be our handling of focus in naturally grouped places (menus, tabs, tree, grid, radio buttons). Currently we treat everything as a big bag of widgets, but it sounds like we need more structure here if possible. This has come up a bit in the menu stuff before. Will think about this and probably split it out into a separate issue.
@ethindp I was thinking those other groups you saw might have been expanders? (or accordions as they are sometimes called) The StructUI demo uses them a lot.
Because AccessKit doesn't support expand/collapse yet, the best I could do was renamed the expander's button based on the expanded/collapsed state.
I'm not sure but it's very possible. I didn't know AccessKit had that limitation atm, good to know.
@david-vanderson FYI, I'm taking a look at the text_run enhancement.
It doesn't look as bad as I first thought as the character sizes and positions are already calculated for me in render.renderText().
@david-vanderson One issue I'd like your thoughts on. For word lengths, I need to work out where the word boundaries are. It turns out this is not trivial, e.g. see the word iterator in ziglyph.
Your thoughts on pulling in an external dependency just for this?
In the accesskit examples, the word length is the word plus any trailing spaces or punctuation or newlines. So, the only other thing I can think of is to break words on whitespace (and add the whitespace to the word length) using the equivalent of isWsegspace from ziglyph, which is much more manageable than the word iterator implementation above. But won't be correct for languages that don't use whitespace between words. (I assume this would be some of the character based languages). I'm guessing we don't really support those properly yet anyway if they aren't left to right?
So, I'm thinking to go for the simpler break on whitespace implementation and see how far that gets us.
IMO we should go for the word iterator approach just because it means this is probably something we won't have to touch again after that's said and done. Just my $0.02 :)
Also, as a related note, Ziglyph appears to have been replaced with zg, which might be a lot smaller to include.
So, I'm thinking to go for the simpler break on whitespace implementation and see how far that gets us.
This sounds like a good strategy for now. I've been hesitant to pull in a bunch of unicode stuff, so am seeing how far we can get without it. TextLayout does word select on "double click", and also has to move the cursor by words when people are using ctrl-cmd and left/right. Right now it has:
/// This is used for word selection - 2 clicks and ctrl+left/right - everything
/// here is not a word, and everything else is.
pub const word_breaks = " \n!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~";
So a reasonable thing to do would be to use that same division (at least for now). Sound good?
(I assume this would be some of the character based languages). I'm guessing we don't really support those properly yet anyway if they aren't left to right?
Also, as a related note, Ziglyph appears to have been replaced with zg, which might be a lot smaller to include.
Good points. The longer we can get away without diving too far into unicode the more robust the libraries will be once we do (at least I hope). I'm sure we will have to eventually, but I think it makes sense to have our accessibility stuff match the rest of our text rendering.
Fair enough. I saw zg and thought that that may address the Unicode concerns since it's a very modular library from the looks of things, so if all we need is word segmentation, we need only include that module. Same thing for code points/grapheme clustering.
Fair enough. I saw zg and thought that that may address the Unicode concerns since it's a very modular library from the looks of things, so if all we need is word segmentation, we need only include that module. Same thing for code points/grapheme clustering.
I think you are correct, and likely we'll go that route at some point, but maybe not as part of this. Goodness knows we need more language and emoji support in the text rendering.
So a reasonable thing to do would be to use that same division (at least for now). Sound good?
Yep, makes perfect sense. I forgot about the double-click text select.
Any update on any of these?
The big recent change was adding focusGroup stuff, which should have fixed focus navigation for radio buttons, menus, trees, and grids.
We also got some better keyboard navigation for trees in terms of expanding/contracting and moving from children up to parents.
Not sure if we've made much progress otherwise. I assume doing text runs in textLayouts and textEntries is the next largest priority here?
I think so, I knew about the first couple. But yeah, text runs and such is the next step.
I think so, I knew about the first couple. But yeah, text runs and such is the next step.
Good - thanks for pinging on this!
Hoping to get back to this list soon.
I did have text run working for single line text entry / areas, but there was some issue with the multi-line ones that I hadn't figured out yet.