Semantic matching basics

[d4vidi]

Describe your idea

This is an issue meant to specifically hold the idea mentioned here: https://github.com/wix/Detox/issues/4145#issuecomment-1689920814

We want to allow for a more intuitive matching API that would allow for a middle-ground solution between the "traditional" matching techniques (e.g. test ID / label based) and the fully blown Detox pilot. Meaning:

• by.type('list') // or horizontal-list / vertical-list, or all of those
• by.type('image') // or icon
• by.type('input-field')

etc.

---

• TODO: This kind of an API really raises the need for an easier disambiguation. This ideas could be introduced further down the road based on more discussions in the original issue (#4145)
• Also related: #4686

[noomorph]

Proposal about possible semantic UI types

Here is my attempt to outline semantic UI types relevant for mobile E2E testing (e.g. with Detox). Below you can find primary names, common aliases, source references (ARIA, HIG, Material, Open UI), and typical mappings to React Native, Android, iOS components, and accessibility roles.

@markdevocht @d4vidi

---

📄 Static Content

• text Aliases: label, static-text Sources: ARIA text, iOS UILabel, Android TextView RN: Text, RCTText, ReactTextView a11y: accessibilityRole="text", StaticText in XCUI

• image Aliases: image-view, icon Sources: ARIA img, iOS UIImageView, Android ImageView RN: Image, RCTImageView, ReactImageView a11y: accessibilityRole="image", Image in XCUI

---

✍️ Input Controls

• text-field Aliases: input, textbox Sources: ARIA textbox, HIG UITextField, Android EditText RN: TextInput, RCTTextInput, ReactEditText a11y: accessibilityRole="textbox", TextField in XCUI

• slider Aliases: seek-bar, range-slider Sources: ARIA slider, UISlider, Android SeekBar RN: Slider, RCTSlider, ReactSlider a11y: accessibilityRole="adjustable", Adjustable in XCUI

• stepper Aliases: incrementor Sources: HIG UIStepper, Android +/- Button pattern RN: custom composed buttons a11y: grouped buttons with accessibilityLabel or adjustable

• date-picker Aliases: time-picker, wheel-picker Sources: HIG UIDatePicker, Android DatePickerDialog RN: @react-native-community/datetimepicker a11y: role varies, use accessibilityLabel, XCUI shows Date Picker

• search-field Aliases: search-box Sources: ARIA searchbox, iOS UISearchBar, Android SearchView RN: TextInput with accessibilityRole="search" a11y: SearchField in XCUI

---

🔘 Selection Controls

• button Aliases: action-button, submit Sources: ARIA button, iOS UIButton, Android Button RN: Button, TouchableOpacity, RCTButton a11y: accessibilityRole="button", Button in XCUI

• link Aliases: hyperlink Sources: ARIA link RN: Text with onPress, accessibilityRole="link" a11y: Link in XCUI

• switch Aliases: toggle, on-off Sources: ARIA switch, iOS UISwitch, Android SwitchCompat RN: Switch, RCTSwitch, ReactSwitch a11y: accessibilityRole="switch", Switch in XCUI

• checkbox Aliases: check-box Sources: ARIA checkbox, Android CheckBox, custom iOS RN: @react-native-community/checkbox or custom a11y: accessibilityRole="checkbox", CheckBox in XCUI

• radio-button Aliases: option-button Sources: ARIA radio, Android RadioButton, iOS custom or segmented RN: RadioButton, SegmentedControlIOS a11y: accessibilityRole="radio", RadioButton in XCUI

• picker Aliases: selector, spinner, dropdown Sources: ARIA combobox, iOS UIPickerView, Android Spinner RN: Picker, RCTPicker, ReactPickerManager a11y: accessibilityRole="menu", Picker in XCUI

---

🧩 Containers & Collections

• scrollview Aliases: scroll-container Sources: iOS UIScrollView, Android ScrollView RN: ScrollView, FlatList, SectionList a11y: adjustable, or no role

• list Aliases: table-view, recycler-view Sources: ARIA list, iOS UITableView, Android RecyclerView RN: FlatList, SectionList, RCTScrollView a11y: accessibilityRole="list", List in XCUI

• grid Aliases: collection-view Sources: ARIA grid, iOS UICollectionView, Android GridLayout RN: custom FlatList with numColumns > 1 a11y: accessibilityRole="grid", CollectionView in XCUI

• card Aliases: tile, panel Sources: Material Design, Open UI RN: View with border/shadow, accessible via label a11y: grouped container, use accessibilityHint

• form Aliases: form-group Sources: ARIA form RN: View with grouped TextInputs a11y: accessibilityRole="form"

• header Aliases: section-header, title Sources: ARIA heading, iOS trait, Android TextView styled RN: Text with accessibilityRole="header" a11y: Header in XCUI

• toolbar Aliases: nav-bar, top-app-bar Sources: ARIA toolbar, iOS UINavigationBar, Android Toolbar RN: react-navigation header, or custom a11y: accessibilityRole="toolbar"

---

🧭 Navigation

• tab Aliases: tab-item, tab-bar-button Sources: ARIA tab, iOS UITabBarItem, Android TabLayout RN: react-navigation TabNavigator a11y: accessibilityRole="tab", Tab in XCUI

• menu Aliases: dropdown-menu, overflow-menu, popup-menu Sources: ARIA menu, Android PopupMenu, iOS UIMenu RN: custom modal list, or react-native-popup-menu a11y: accessibilityRole="menu", Menu in XCUI

• drawer Aliases: side-menu, navigation-drawer Sources: Material Design DrawerLayout, iOS custom RN: react-navigation DrawerNavigator a11y: accessibilityRole="menu", Drawer in XCUI

---

📢 Feedback & Dialogs

• dialog Aliases: modal, alert-dialog Sources: ARIA dialog, iOS UIAlertController, Android AlertDialog RN: Modal, react-native-dialog a11y: accessibilityRole="dialog", Dialog in XCUI

• alert Aliases: toast, banner, snackbar Sources: ARIA alert, Android Toast, iOS Banner, Snackbar RN: ToastAndroid, react-native-snackbar a11y: accessibilityRole="alert", or Announcement in XCUI

• progress-indicator Aliases: spinner, progress-bar Sources: ARIA progressbar, iOS UIActivityIndicatorView, Android ProgressBar RN: ActivityIndicator, ProgressBarAndroid, RCTActivityIndicatorView a11y: accessibilityRole="progressbar", Progress Indicator in XCUI

---

🧷 Minor Elements

• tooltip Aliases: helper-text, hint Sources: ARIA tooltip, iOS accessibilityHint, Android contentDescription RN: accessibilityHint prop a11y: Hint in XCUI (no direct element)

• invisible-overlay Aliases: hit-area, tap-region Sources: gesture design patterns RN: View, Pressable with opacity: 0, often used for custom gestures a11y: may need accessible={false} or explicit labeling

[markdevocht]

Need to check the Android inheritance is found on custom components.

[markdevocht]

Here's the breakdown for the Android class name search.

The fun isOfClassName(className: String): Matcher<View> in ViewMatchers.kt uses private class IsAssignableFromMatcher(private val clazz: Class<*>) : TypeSafeMatcher<View>() which in turn calls clazz.isAssignableFrom(view.javaClass) which works as follows:

The Class.isAssignableFrom() method in Java/Kotlin returns true if:

• The class is exactly the same class
• The class is a superclass of the target class
• The class is a superinterface implemented by the target class

So this means that when searching for a 'list' in the semantic matching, any custom component inherited from the described components ['android.widget.ListView', 'androidx.recyclerview.widget.RecyclerView', 'com.facebook.react.views.scroll.ReactScrollView'] will be matched.

[d4vidi]

Here's the breakdown for the Android class name search.

The fun isOfClassName(className: String): Matcher<View> in ViewMatchers.kt uses private class IsAssignableFromMatcher(private val clazz: Class<*>) : TypeSafeMatcher<View>() which in turn calls clazz.isAssignableFrom(view.javaClass) which works as follows:

The Class.isAssignableFrom() method in Java/Kotlin returns true if:

* The class is exactly the same class

* The class is a superclass of the target class

* The class is a superinterface implemented by the target class

So this means that when searching for a 'list' in the semantic matching, any custom component inherited from the described components ['android.widget.ListView', 'androidx.recyclerview.widget.RecyclerView', 'com.facebook.react.views.scroll.ReactScrollView'] will be matched.

thanks for looking into it!