Extend testdriver.js and wptrunner to test AAMs (platform accessibility APIs)
Closes https://github.com/web-platform-tests/interop-accessibility/issues/210
This PR adds an extension to testdriver.js in order to test accessibility API, as defined by the ARIA, Core-AAM and HTML-AAM specifications. The RFC to extend testdriver.js in this way can be found here. The decision to open a PR with these changes was made at TPAC 2024.
The change in this CL:
- Tests in the core-aam directory: These tests are marked as tentative because some work may need to be done on the design of the API endpoint, and these tests are only run when supplied with
--enable-accessibility-api. So on regular runs of wpt.fyi, they will fail, for now, as you can see with: - Changes to wptrunner:
a. The option
--enable-accessibility-apihas been added. This option will install the appropriate python requirements for accessibility API tests and turn on accessibility for Firefox and Chromium browsers. If this option is not supplied, tests may fail withAccessibility API testing not enabled. The primary reason for the flag is that on linux, there are system requirements for the python requirements, so./wpt runwill fail in environments where the system requirements are not installed. For now I think it's better to only run the tests intentionally/behind a flag. a. The product name ("chrome", "firefox", etc) is passed back to the executor level. This is necessary because accessibility APIs are exposed externally to the browser, and we need to find the browser in the accessibility API by name. Ideally, this will be done by PID (to allow for parallelization), but that requires some changes to chromedriver and safaridriver. executorplatformaccessibility.pyandtools/wptrunner/wptrunner/executors/platformaccessibility: these executors run tests against platform specific accessibility APIs, so there is an executor for Linux, Windows, and macOS. The Windows and macOS ones are proof of concepts and will need more work. The Linux one is fully featured and can run all of the tests added in this PR.- Changes to the docker system installations and updating the docker image, to test the linux runs of these tests. You can see the tests pass in the linux environment for Chrome and Firefox with the commit below: Switch CLI arg --enable-accessibility-api to true by default. Here are passing tests in the CI: https://wpt.fyi/results/core-aam?label=pr_head&max-count=1&pr=53733
To run tests on Linux:
On Debian distros:
apt install libatspi2.0-dev libcairo2-dev libgirepository1.0-dev
# Chrome
./wpt run --enable-accessibility-api --log-mach-level debug --log-mach - chrome core-aam/role/blockquote.tentative.html
# Chromium
./wpt run --enable-accessibility-api --binary <chromiumbinary> --force-accessibility --log-mach-level debug --log-mach - chromium core-aam/role/blockquote.tentative.html
# Firefox
./wpt run --enable-accessibility-api --no-headless firefox core-aam/role/blockquote.tentative.html
Notes about macOS and Windows
On Windows and macOS, even with the following dependencies, these tests will fail with the message "Fail: Test not implemented".
Thanks to @alice for the macOS code and thanks @jcsteh, the code to get IA2 interface was very inspired by the firefox test infrastructure: https://searchfox.org/mozilla-central/source/accessible/tests/browser/windows/a11y_setup.py
Running the tests on macOS
To run against Firefox, you need to turn on VoiceOver the screen reader. To run against Chrome, you need to include --no-headless.
# To run against Firefox, you need accessibility turned on. You can run on the Orca screen reader or `export GNOME_ACCESSIBILITY=1` and use `--no-headless`.
I wonder if we can make the name --force-renderer-accessibility more generic (e.g. --enable-accessibility or --force-accessibility) and implement it for various browsers as required? For Firefox, we can use the pref accessibility.force_disabled=-1 (I know, it's horrible, I'm sorry). I'm not quite sure where we'd wire that up, but I see that we do set extra prefs here, so we have the capability to do that.
I wonder if we can make the name
--force-renderer-accessibilitymore generic (e.g.--enable-accessibilityor--force-accessibility) and implement it for various browsers as required? For Firefox, we can use the prefaccessibility.force_disabled=-1(I know, it's horrible, I'm sorry). I'm not quite sure where we'd wire that up, but I see that we do set extra prefs here, so we have the capability to do that.
Amazing, thanks @jcsteh! I didn't know about that Firefox option, I always used the gnome accessibility trick. I changed the CLI arg to --force-accessibility and it works with both chrome and firefox now, the PR description has been updated.
The failures are unrelated, see: https://github.com/web-platform-tests/wpt/issues/54302
Ready for review :)
@gsnedders, @jgraham, @cookiecrook or @zcorpan, perhaps a couple of you could take a look?
@spectranaut Do you think it could resolve the conflicts please? That would help with the review as it goes through CI.