Accessibility API

[clayote]

There's an issue #2820 to add screenreader/UI Automation support to Kivy, but it got closed because no one was up to it.

Discussion on IRC suggests that what's needed to get started is an API for widgets to implement that can then be provided by whatever the target platform does for screenreading. That would split the work up so we can work on one platform at a time and eventually support them all.

To design that API, we need someone knowledgeable about how screenreaders work on all target platforms to decide on a subset of functionality that they all share, and which all Kivy widgets ought to support.

[tyrylu]

Hello. I unfortunately do not know the exact specifics of the Apple's accessibility APIs, however as a visually impaired developer myself, my guess at the at least somewhat common accessibility related properties is:

• Role - button, checkbox, etc.
• Name - a short name of the element for accessibility purposes
• States - somewhat role dependend, e. g. checked, readonly, editable etc.

As a basic inspiration you could look at the Firefox's accessibility tree, they are using (or at least it seems) a cross platform internal tree at least under Windows and Linux. And yes, it will have to be a tree, every accessibility API i know of expects an object tree.

[tshirtman]

Thanks for this info, so i guess as a first step, Widget could have a property (maybe DictProperty or an AliasProperty to a method to build the updated info if that depends on state) to store specific informations about this widget's accessibility informations, and it would be up to each more specific widget to add relevant information to it? so we could just traverse the widget tree that already exists and give this information to reader providers.

There are some things that kivy can't know for sure unless the widgets provide the info, for example, current visibillity of a widget is not easy to compute, as some widgets can cover the screen, while being totally invisible to the user, and not preventing interactions to widgets behind it, so a widget should declare its "visible" zone in some way, so kivy can compute its impact on widgets behind it. It's easy for Button/checkbox/Popup and others, but the more specific cases will require custom code (Image is a difficult one, it could be fully transparent and won't prevent any interaction by default, but the screen reader should certainly know there is something there, i assume each "image" use in the program would require an "alt" tag, and the screen reader would simply ignore these with an empty value).

In case this whole traverse/update state dance is costly and slows down existing programs, we could have a kivy setting to enable/disable the whole thing, but that would certainly reduce the number of people checking their widgets have good accessibility.

[tyrylu]

Someting along the lines of it, yes. Also, in reaction to a widget event (e. g. state/property change), the appropriate accessibility events should be raised so a screen reader can notify the user in real time. You are right that if you make the accessibility engine optional, you can be certain that some developers will just ignore it.

[tyrylu]

And, you probably should not think of a screen reader interactions in terms of mouse actions, just to be on the same page.

[clayote]

How should we represent the widgets' boundaries in the tree? (pos, size) might be sufficient for most widgets, but if it's a Scatter or something like that, we might need an arbitrary polygon. Can screen readers handle those?

[clayote]

Ah -- perhaps tyrylu meant that we shouldn't represent boundaries within this tree at all, instead implementing some other API that takes an element in that tree and changes its state, without touch events being involved at all. Then, if the user is in fact using a pointing device, we can represent what it's doing by changing the state of a widget to indicate that the mouse is above it. But some users will be navigating with arrow keys and Tab and such. This API would need to either simulate touch with those, or add tab-browsing functionality to all the widgets.

[tyrylu]

Yes, correct.

[matham]

I haven't had a chance to read this carefully, but I think we can maybe take inspiration from FocusBehavior. That provides a mechanism for widgets to opt in to the behavior as well as a method to iterate those opted-in widgets.

For this problem, by inheriting from the accessibility behavior, the dev could provide more info about the widget, other than the visual representation. Such as whether it's visible, a description, a way to increase zoom or change font or whatever we come up with as common things that the widget should support. Then we can have a global module that when activated walks this behavior tree, like tab does for FocusBehavior and interacts with it.

But we would need to know the kind of things that such widgets need to support.

On Sat, Sep 8, 2018, 6:00 PM Lukáš Tyrychtr [email protected] wrote:

Yes, correct.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/kivy/kivy/issues/5836#issuecomment-419675602, or mute the thread https://github.com/notifications/unsubscribe-auth/ABkW_vlXzRf3bSPzboXDDCrmmho5J6Ivks5uZD30gaJpZM4U-WCd .

[tyrylu]

For the supported operations, you could look at IAccessible2: http://accessibility.linuxfoundation.org/a11yspecs/ia2/docs/html/

[clayote]

I'm not sure whether it's better to support such accessibility as we can without AccessibleBehavior, or not to, and just focus on making it maximally convenient for developers to mix it in.

[tyrylu]

Well, if you make such thing optional, you can be sure that many developers will just ignore it - time/management priorities, not thinking that it might be useful etc.

[clayote]

OK, so the accessibility module should still work when the app does not support tabbing. I guess that means it needs to detect when a widget is being hovered over by, like, checking collide_point recursively on the whole widget tree when you hold the mouse still for a second. Not for any widget with AccessibleBehavior, though.

Some widgets already give us the information we need for operations like IAccessible2. Offhand:

• The basic IAccessibleComponent interface can be easily derived from any widget with a foreground, background, and a parent that can transform positions to_local
• ScrollView is always scrollable; RecycleView knows when it's full enough to need scrolling; Scatter can be scrollable, if it's got that type of transformation enabled
• text or hint_text on Button and the like provides a description
• GridLayout is a table
• The text formatting modules, like kivy.uix.rst, support hypertext; TextInput is a document

It's already basically possible to use apps designed for mice without a mouse using the joycursor module, but it assumes you have a joystick attached. You can get around that with an external program like joy2key, but maybe we should fall back to some keybindings if no joypad is detected.

[mwcampbell]

Would it help the project if I prototyped a module that made Kivy accessible on Windows using UI Automation, using whatever information is already available on the widget objects? This would be single-platform and hacky, but I think it would help us get a better idea of what changes need to be made in the widgets to make this work well.

[matham]

I can't speak for the people who need this feature, but what you suggests seems like a good fit for garden.

[ParadoxiKat]

Is there any further info on this work? As a blind developer I would love love love to get this moving and contribute however I can. I want to develop android apps and really want to use Kivy but the lack of accessibility is what's stopping me. Using it on windows would be nice too -- but at least there I have alternatives to Kivy.

[matham]

There is a proposal here about a API that could potentially be used for Accessibility that could use some insight from people familiar with the topic. Whether an Accessibility API would work as described in the proposal: https://github.com/kivy/kivy/pull/6724#issuecomment-709865689.

[FFY00]

I am also interested in this, and would be happy to throw in some work if that made this progress faster. Let me know if there is anything I can contribute.

[xayhewalo]

It looks like there were plans to make Kivy more accessible in the past I have no idea what recent conversations have been, but I thought the above link was relevant to this thread

[matham]

I took a quick look around and it seems implementing accessibility is actually quite complex and would require proper design with an API. There are multiple frameworks (On Windows MSAA and IAccessible, with the newest API Microsoft UI Automation; on linux IAccessible2 is used, but ATK/AT-SPI seems the more common API; on Mac there's NSAccessibility; not sure what iOS or android do).

• GTK and gnome rely on the ATK/AT-SPI interface, but they seem to be considering changing it up.
• Qt supports UI Automation on Windows, macOS Accessibility on macOS, and AT-SPI via DBus on Unix/X11.
• wxWidgets uses MSAA on Windows and relies GTK for its linux support. However, it's linux support has issues because it doesn't implement the ATK interface itself.
• Mozilla has a bunch of info about accessibility. But as they say there's no cross-platform API so they implement ATK and MSAA. On OSX they seem to rely on the community to implement some (not sure what) interface.

Some more background links.

All this shows that each platform has their own interface and if we need cross platform support we'd need to have some kind of intermediate interface. And even if we only implement one interface, it'd still need to be properly implemented using a client/server architecture to communicate with the screen readers etc.

So it seems like it would take a significant amount of resources to implement this. Although perhaps just getting started may not be as hard.

[clayote]

It's not clear when it'll be ready, but we may be able to use AccessKit for our accessibility API.

[clayote]

Looks like AccessKit is ready! Here's the Python package for it.

[clayote]

Let's move to #8547