Android: using AccessibilityNodeInfo#addAction to announce Expandable/Collapsible State

[fabOnReact]

Summary

Expandable and Collapsible are unique in the Android Accessibility API, in that they are not represented as properties on the View or AccessibilityNodeInfo, but are only represented as AccessibilityActions on the AccessibilityNodeInfo. This means that Talkback determines whether or not a node is "expandable" or "collapsible", or potentially even both, by looking at the list of AccessibilityActions attached to the AccessibilityNodeInfo.

When setting the accessibilityState's expandable property, it should correlate to adding an action of either AccessibilityNodeInfoCompat.ACTION_EXPAND or AccessibilityNodeInfoCompat.ACTION_COLLAPSE on the AccessibilityNodeInfo. This work should be done in the ReactAccessibilityDelegate class's

Currently, this feature is being "faked" by appending to the contentDescription in the BaseViewManager class. This should be removed when this feature is implemented properly.

fixes https://github.com/facebook/react-native/issues/30841

Changelog

[Android] [Fixed] - using AccessibilityNodeInfo#addAction to announce Expandable/Collapsible State

Test Plan

• On some components, the state expanded/collapsed is properly announced on focus, on some it is not.
• On some components only the expanded/collapsed state is announced, and not other component text.
• Upon change, state change is not always announced.
• The accessibilityState's "expanded" field does not seem to work on all element types (for example, it has no effect on 's).
• using accessibilityActions it is possible to add an action for expand/collapse, but these are treated as custom actions and must have their own label defined, rather than using Androids built in expand/collapse actions, which Talkback has predefined labels for.

https://snack.expo.io/0YOQfXFBi

Paper https://github.com/facebook/react-native/pull/34353#issuecomment-1217425302 Fabric https://github.com/facebook/react-native/pull/34353#issuecomment-1217781734

Announcing state with custom actions on Fabric (FAIL). The issue is not a regression from this PR, as documented in https://github.com/facebook/react-native/pull/34353#issuecomment-1207744977. It will be fixed in a separate PR.

[analysis-bot]

Platform	Engine	Arch	Size (bytes)	Diff
android	hermes	arm64-v8a	7,637,392	-509
android	hermes	armeabi-v7a	7,049,372	-519
android	hermes	x86	7,939,356	-516
android	hermes	x86_64	7,911,371	-524
android	jsc	arm64-v8a	9,513,631	-64
android	jsc	armeabi-v7a	8,288,904	-90
android	jsc	x86	9,453,169	-82
android	jsc	x86_64	10,044,414	-75

Base commit: a789ead5459b772a1ae9b17fce6565d4717554f8 Branch: main

[analysis-bot]

Platform	Engine	Arch	Size (bytes)	Diff
ios	-	universal	n/a	--

Base commit: a789ead5459b772a1ae9b17fce6565d4717554f8 Branch: main

[fabOnReact]

This issue will be fixed in a separate PR, it is tracked in a separate task as not a regression introduced in this PR, neither documented or reported in the original issue.

Main Branch - onAccessibilityAction not called on Fabric renderer

on Paper renderer, onAccessibilityAction is correctly called. It is not called on Fabric.

https://user-images.githubusercontent.com/24992535/183425752-d0cb78a9-5333-48df-ae8b-cc95afe73866.mp4

The callback is triggered here

https://github.com/facebook/react-native/blob/28e9c3b87e709dce3ffec208ff5228d07eaad512/ReactAndroid/src/main/java/com/facebook/react/uimanager/ReactAccessibilityDelegate.java#L366

https://github.com/facebook/react-native/blob/28e9c3b87e709dce3ffec208ff5228d07eaad512/Libraries/NativeComponent/BaseViewConfig.android.js#L108

• [ ] Move back in the commit history. Understand when the onAccessibilityAction stopped to work on Fabric

[fabOnReact]

PR Branch - using accessibilityActions it is possible to add an action for expand/collapse, but these are treated as custom actions and must have their own label defined, rather than using Android built-in expand/collapse actions, which Talkback has predefined labels for.

https://user-images.githubusercontent.com/24992535/183358727-aaa3d1b8-c872-4ed0-8a45-906ce3850d53.mp4

https://user-images.githubusercontent.com/24992535/183359244-9ff11682-a625-48f5-810b-d198aa173cce.mp4

solved with https://github.com/facebook/react-native/pull/34353/commits/d58a2dbe88ee63bc55dd6c908770c98000b573d4

[fabOnReact]

extended/collapsed is announced on state change on Paper (PASS) (enable audio)

https://user-images.githubusercontent.com/24992535/185028855-17791b9d-a586-420e-9c10-29673ba224e2.mp4

announce the visible text on Paper (PASS) (enable audio)

the same result as with fabric see the corresponding example in https://github.com/facebook/react-native/pull/34353#issuecomment-1217425302

announcing state with custom actions on Paper (PASS) (enable audio)

https://user-images.githubusercontent.com/24992535/185032425-987edaf9-4f99-4283-8ce3-d1f29c874641.mp4

[fabOnReact]

extended/collapsed is announced on state change on Fabric (enable audio) (PASS)

https://user-images.githubusercontent.com/24992535/185082749-0cb2ad4d-59de-4338-a22c-b3563d0a98b8.mp4

announce the visible text on Fabric (enable audio) (Pass)

https://user-images.githubusercontent.com/24992535/185088735-4d559be8-7183-444f-b140-7ebc262128ee.mp4

announcing state with custom actions on Fabric (FAIL). The issue is with main branch https://github.com/facebook/react-native/pull/34353#issuecomment-1207744977, will be fixed in a different PR.

[fabOnReact]

Button which keeps control of extended/collapsed state in JavaScript with onAccessibilityAction, accessibilityActions and accessibiltyState (Paper)

https://user-images.githubusercontent.com/24992535/188535897-b24609ec-d134-4987-ba83-138984403e0d.mp4

[fabOnReact]

TouchableWithoutFeedback keeps control of extended/collapsed state in Android Widget (Paper)

https://user-images.githubusercontent.com/24992535/188540174-5bc19b96-da72-43de-ad7c-e9dbebd1b68b.mp4

Just for note, the initial available action is collapse. The video was recorded after interacting with the component.

[fabOnReact]

TouchableWithoutFeedback keeps control of extended/collapsed state in Android Widget (Fabric)

https://user-images.githubusercontent.com/24992535/188542275-9ea6e531-0a84-48a3-a2a5-f1d8aa90d033.mp4

[fabOnReact]

TouchableOpacity announces visible text and triggers expanded/collapsed with onPress and accessiblity menu (Fabric)

https://user-images.githubusercontent.com/24992535/188542974-83e23b4c-6c99-4594-b30c-622bc8a36170.mp4

[facebook-github-bot]

@blavalla has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[react-native-bot]

This pull request was successfully merged by @fabriziobertoglio1987 in 082a033fbbe7d7094af78bafc3b2048194a02bd5.

^{When will my fix make it into a release? | Upcoming Releases}