Android: Position in Collection not supported by Flatlist, SectionList, VirtualizedList, or ScrollView

[amarlette]

Description

Collections on Android have a CollectionInfo property which defines the size of the entire collection, and whether those elements are in rows, columns, or both. This information will be announced as part of the "in list" and "out of list" announcements, as well as when the list scrolls. Currently React Native does not support any way to define this information, and also doesn't support the list/grid/pager roles required to surface it.

Position in Collection not from collection Not supported by:

• [ ] Flatlist #30973
• [ ] SectionList #30974
• [ ] VirtualizedList #30975
• [ ] ScrollView #30976

Since this behavior depends on both the collection info being present and the component having the proper accessibility role set, this task is dependent on #30839.

React Native version:

v0.63

Expected Behavior

On focus of a list component, screen readers should announce "In list, N items". Upon scroll of the list via the next page/previous page gestures, it should announce "Showing items X to Y of Z".

Android Details

Collection-type views on android (lists, grids, and pagers) all have a CollectionInfo object attached to their AccessibilityNodeInfo. This CollectionInfo object contains the total count of the items in the collection, how many rows the collection has, how many columns it has, and whether its hierarchical. Each item in the collection (ie. every direct child element) has a CollectionItemInfo object on its AccessibilityNodeInfo. This object defines the row/column index, the row/column span, and whether the item is selected and whether it is a heading.

https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeInfo.CollectionInfo https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeInfo.CollectionItemInfo

Right now, some of this information might be being added to these list components, depending on what native Android components they are being mapped to. But it's currently not being surfaced to the user if that is the case, due to the components not having the ability to set (or a hard coded value set) of the proper accessibilityRole.

[blavalla]

@kacieb, @lunaleaps, @nadiia, this one may require some API to be added to set this information, for example <View accessibilityCollectionInfo={{count: 10, rows: 1, columns, 2}} />.

Or alternatively, if you don't want a user to be able to set the collection info directly (as iOS has no counterpart here), it should be automatically set based on the number of child elements passed into these lists. Either approach will work, as long as the accessibility system can parse out the collection info.

You may be able to re-purpose the accessibilityValue prop, which has a "min", "max", and "current" value, but it wouldn't be able to work for grid-like structures (although we don't have any core grid component currently to worry about).

[kacieb]

Potentially a prop like this could be exposed as a native prop on ScrollView.js.

Maybe an automatic count could be accomplished in the native Android React Native ScrollView component. What does "count" mean in this context? Number of children? Number of items in the list? Number of focusable items in the list? Is it possible for us to automatically understand this? I would think with an infinitely available number of structures, it might be hard to automatically determine what counts as a list item, so we might have to rely on this information getting passed in by the user.

[kacieb]

This is an issue I think would be great to see a proposal for before anyone begins working on it, since there are likely multiple ways to solve this.

[blavalla]

What does "count" mean in this context?

Count should be the number of child elements, not the number of descendants. Technically speaking, Talkback treats all children of collections as focusable elements, as this is one if its weird edge cases in its focusability logic.

[kacieb]

Count should be the number of child elements

That's great, I think that means this can be done automatically in the native ScrollView (I have to double check, but I know there's a way to get children, so I'd think we can count them in native) so that the user doesn't have to think about it.

Something worth noting for implementing this - ScrollView.js is it is internally structured (this is pseudocode):

<ScrollViewNativeView>
   <ScrollViewContentView>
      // actual user content here
   </...>
</...>

So we'll want to make sure the count is the number of children in the content view.

[intergalacticspacehighway]

My initial implementation.

https://user-images.githubusercontent.com/23293248/111256309-d4ba3700-863e-11eb-830f-9f749ca8ceb2.mp4

@amarlette can you help me with Upon scroll of the list via the next page/previous page gestures, it should announce "Showing items X to Y of Z". Is this similar to the 2 finger scroll gesture?

I think passing it externally from JS might make it easier to handle with Virtualized lists. @blavalla @kacieb Let me know what you think!

[blavalla]

@intergalacticspacehighway, the "next/previous page gestures" the the ones that trigger scrolling by an entire page at a time. With Talkback's default settings, these will be swipe-right-then-left without lifting your finger for "next page" and swipe-left-then-right without lifting your finger for "previous page". In Talkback's settings these are called "Scroll forward" and "Scroll backward", and can be remapped to something a little easier to trigger if desired (like swipe up or swipe down for example). Let me know if you need any help on how to use these, or remap them on your device.

When these gestures are triggered, since it moves an entire page at once, it should announce where on the page it landed via an announcement like "Showing items 2 to 8 of 15". This same announcement intentionally does not occur for two-finger scrolling, as this is meant to be used to scroll more slowly and you'd be hearing these announcements constantly as each new item entered/exited the screen.

[blavalla]

@intergalacticspacehighway, in your implementation is it possible for the accessibilityCollectionItemInfo to have defaults that work without a using having to specify? As in, the row or column index could be automatically set to be the elements index among its siblings, the row/column span could default to "1", etc. It seems like 90% of the time the defaults here will be correct, so having to specify it each time may be kind of a pain.

@kacieb, what are your thoughts on the API shown here? This is information that is only really used by Android and not iOS, so I want to make sure that it's clear at the API level (or at least well documented) that this is the case.

[intergalacticspacehighway]

Thanks @blavalla I can confirm that collectionInfo and collectionItemInfo are being set appropriately. But I am not sure due to some reason, the talkback doesn't announce "Showing items x to y of z" on "next/previous page gesture". Other list announcements like "out of list" and "in list, N items" are announced. I haven't added any roles to children, the parent view is just set to list => android.widget.ListView. I'll dig deeper, but If you have any brief ideas, it'd be helpful. Do you think collection items might need a role?

[blavalla]

@intergalacticspacehighway, the elements with CollectionItemInfo shouldn't need any particular roles applied to them for it to work.

One thing worth looking in to is whether or not the children of <ScrollView> on the JS side actually become direct children of the ReactScrollView on the java side. If there is some wrapper element around each one for example, the CollectionItemInfo would need to be on the wrapper.

Talkback's actual logic here is (psuedo-code):

// on scroll event
if (element.role == 'list') {
  if(element.isVisible && event.fromIndex >= 0 && event.itemCount > 0) {
    // output text "Showing items {event.fromIndex + 1} to {event.toIndex + 1} of {event.itemCount}."
  }
}

So what is likely happening is that the accessibility event is not populating the fromIndex, toIndex, or itemCount properties properly. If this is mappjng to a RecyclerView in Android, this happens automatically, but I am not sure what React's scroll view actually does on the native Android side.

Here's an example of how the RecyclerView handles this: fromIndex and toIndex are set in the linear layout manager here:

https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:recyclerview/recyclerview/src/main/java/androidx/recyclerview/widget/LinearLayoutManager.java;l=245-246?q=setFromIndex&ss=androidx%2Fplatform%2Fframeworks%2Fsupport

itemCount is set on the RecyclerView itself here: https://cs.android.com/android/platform/superproject/+/master:frameworks/base/core/java/com/android/internal/widget/RecyclerView.java;l=9410;drc=master

My guess as to what is happening here is that the AccessibilityDelegate set by React Native is overriding any default accessibility delegate that is on the native scroll view class (which is how they work), so we'll need to add the event.fromIndex event.toIndex and event.itemCount directly in the ReactAccessibilityDelegate's onInitializeAccessibilityEvent method.

Looking at that class I see that this is already being done for items with the "accessibilityValue" property, in order to get it to announce increments for when this is used for a slider, so I think doing something similar here would work.

https://github.com/facebook/react-native/blob/aad423d59e99a76cfb10b466bd13463aa4926b80/ReactAndroid/src/main/java/com/facebook/react/uimanager/ReactAccessibilityDelegate.java#L283

Let me know if any of that didn't make sense, or you want any additional clarification!

One thing to note here, is that since it seems that we need to implement the event ourselves, we aren't really beholden to matching Android's existing API with the concept of a "collection info" and "collection item info". All these do in the end is populate the AccessibilityEvent, so it seems like we might be able to simplify this a bit. Once you get it functioning I'm sure you'll see if there are opportunities for streamlining :)

Thanks for all the help here!

[intergalacticspacehighway]

@blavalla Sorry to reply so late. Thanks for such a thorough explanation. This has really helped me to come up with a solution that won't require any additional API from the user side.

Testing it with FlatList

https://user-images.githubusercontent.com/23293248/120919855-63d46980-c6d9-11eb-9e48-dde96187d86c.mp4

I would like to point out some observations.

Flatlist, Section or Virtualized list provides JS based virtualisation on top of ScrollView. This makes it difficult to derive the total number of child views on native side. (child mount/unmount is controlled by JS).

During my test, there are 2 announcements.

1. In list, x items <- This is controlled by accessibilityCollectionInfo and role=list
2. Showing x of y items <- This requires to set setFromIndex, setToIndex and setItemCount in onInitializeAccessibilityEvent as you pointed out.

1st is simple and can be solved by introducing accessibilityCollectionInfo as shown here

Solving 2nd has multiple approaches

I have added 2 APIs in View accessibilityCollectionInfo and accessibilityCollectionItemInfo

setFromIndex and setToIndex requires to measure visibility of a View which is provided by Flat/Virtualized list in onViewableItemsChanged but can have some performance impact as it does calculation on JS side. So, I have kept it in native side

The above is a draft PR. Would like to hear your thoughts before proceeding further. Thanks!

[kacieb]

Thanks so much for working on this! The example above seems like a great proof of concept to me. I'll leave it to @blavalla to verify the accessibility behavior!

Questions from me - how would this work with nested lists?

ex.

<FlatList >
  <FlatList ... />
  <FlatList ... />
</FlatList>

Additionally, how would this work if FlatList numColumns is greater than 1? Ex. in MultiColumnExample.js where we can have multiple columns.

[intergalacticspacehighway]

@kacieb Thanks! For multicolumns, I am working on the grid support. Tricky part is that the numOfColumns is only known to FlatList and the child Views where we need to set the column index using accessibilityCollectionItem are on VirtualizedList. So, we need some way to communicate from FlatList's renderer to VirtualizedList's renderer. I hacked it by passing numOfColumns to VirtualizedList. I'll think of a better approach soon. Probably this weekend 😅

https://user-images.githubusercontent.com/23293248/121405917-36a6e600-c97b-11eb-93df-ab690d7800cd.mp4

For nested, I suppose it won't be an issue. Focusing on any item should make announcement respective to that list. Still I haven't tested with an example. Will add an example and verify! Thanks!

P.S. There's an initial announcement error in the above video

[blavalla]

@kacieb Thanks! For multicolumns, I am working on the grid support. Tricky part is that the numOfColumns is only known to FlatList and the child Views where we need to set the column index using accessibilityCollectionItem are on VirtualizedList. So, we need some way to communicate from FlatList's renderer to VirtualizedList's renderer. I hacked it by passing numOfColumns to VirtualizedList. I'll think of a better approach soon. Probably this weekend 😅

Record_2021-06-09-23-20-36_516912a46cba986d39e6e95b9623a5e8.mp4 For nested, I suppose it won't be an issue. Focusing on any item should make announcement respective to that list. Still I haven't tested with an example. Will add an example and verify! Thanks!

P.S. There's an initial announcement error in the above video

Wow, this multi-column support looks great!

I wonder if it would be possible to support this same interaction on iOS with something like UIAccessibilityContainerDataTable (https://developer.apple.com/documentation/uikit/uiaccessibilitycontainerdatatable?language=objc), which is how VoiceOver on iOS handles multi-column tables and has similar announcements to Talkback here.

I am not very familiar with our iOS React Native codebase, and don't know whether that is remotely feasible, but if you happen to also be an iOS expert @intergalacticspacehighway, and want to make this x-platform, I'm sure we can find someone on the RN side to review it :)

Thanks again for all the great work here!

[intergalacticspacehighway]

@blavalla I am not an expert by any means, but will surely check 😂. I have updated the PR to include grid and list support, also moved it to ready for review. @kacieb I have added the nested FlatList example also verified on my end.

Let me know of any changes. Thanks!

[kriti18singh]

@amarlette Is this specific to android? I don't see a 'list' role on the RN Accessibility page.

[ExplorerSunil]

This sound exciting. We are also trying to create an accessible app using react native, but not having the list role and users unable to hear list roles and item numberings sound very non-native behavior for talkback users. Would really love to see this in production of react native.

Whats the status here? would be glad to help in it any way possible.

[ExplorerSunil]

hey guys, @amarlette and @intergalacticspacehighway whats the status here?

[fabOnReact]

This issue will be solved with my pr https://github.com/facebook/react-native/pull/33180

[fabOnReact]

TalkBack support for ScrollView accessibility announcements (list and grid) #33180 initially landed on 20th April 2022. Java API relanded with commit a7bb9664009. JavaScript API relanded with commit cbf53bcaf0c, but reverted with commit c2169c776e271ea.

[github-actions[bot]]

This issue is stale because it has been open 180 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 7 days with no activity.