w3c
Review of existing tests' ability to be automated
Background
In the ARIA-AT Automation Design Doc, we discussed that the existing ARIA AT tests may need some changes to allow for automation without ambiguity in the test instructions. This issue reviews the existing tests and suggests changes we think are necessary for automation. These changes could make the tests clearer for manual testers as well.
View these suggestions as a starting point for discussion; they are not set in stone.
alert
Test instructions look OK wrt automation.
breadcrumb
Test instructions look OK wrt automation.
checkbox
PR: https://github.com/w3c/aria-at/pull/447 (WIP)
- [x] Should split “navigate forwards” and “navigate backwards” to separate tests.
- [x] For tests that navigate forwards, insert a link before the checkbox and set focus to it with the setup script. Similar to checkbox-tri-state.
- [x] For tests that navigate backwards, insert a link after the checkbox and set focus to it with the setup script. Similar to checkbox-tri-state.
- [x] Remove “Note: it should be in the unchecked state.” from the instructions. This is covered in the success criteria.
checkbox-tri-state
- [ ] Remove “Note: it should be in the partially checked state.” from the instructions. This is covered in the success criteria.
- [ ] Should use command sequences: "You may need to press certain commands (such as Down Arrow) multiple times"
combobox-autocomplete-both-updated
- [ ] Should use command sequences: "You may need to press certain commands (such as Down Arrow) multiple times"
combobox-select-only
- [ ] Should use command sequences: "You may need to press certain commands (such as Down Arrow) multiple times"
command-button
Test instructions look OK wrt automation.
disclosure-faq
- [ ] Remove “Note: it should be in the collapsed state.” from the instructions. This is covered in the success criteria.
- [ ] Should use command sequences: “you may need to press Down Arrow multiple times”
disclosure-navigation
- [ ] Should use command sequences: "you may need to press certain commands (such as Down Arrow) multiple times"
horizontal-slider
This test is currently skipped in automation because key press instructions are ambiguous.
menu-button-actions
Test instructions look OK wrt automation.
menu-button-actions-active-descendant
Test instructions look OK wrt automation.
menubar-editor
- [ ] Split “Up Arrow / Down Arrow” into separate tests, with a link before or after that has initial focus.
minimal-data-grid
- [ ] Uses command sequences ("T followed by Down Arrow"), but not fully? "you may need to press certain commands (such as Down Arrow) multiple times"
modal-dialog
Test instructions look OK wrt automation.
radiogroup-aria-activedescendant
- [ ] Should use command sequences: “you may need to press Down Arrow multiple times”
radiogroup-roving-tabindex
- [ ] Should use command sequences: “you may need to press Down Arrow multiple times”
rating-slider
This test is currently skipped in automation because key press instructions are ambiguous.
tabs-manual-activation
- [ ] Should use command sequences: "You may need to press certain commands (such as Down Arrow) multiple times"
toggle-button
Test instructions look OK wrt automation.
vertical-temperature-slider
This test is currently skipped in automation because key press instructions are ambiguous.
“Insure NVDA is in browse mode by pressing Escape. Note: This command has no effect if NVDA is already in browse mode.”
See https://github.com/w3c/aria-at/issues/382.
I've updated the OP with my findings for the current state of the tests.
The main thing is that some of the tests aren't using command sequences yet.
menubar-editor also uses the old "Up Arrow / Down Arrow" pattern in one test.