owncloud
Automated checks for accessibility rules and standards
Automated Checks for Accessibility Rules
Accessibility issues like these (but not all) could have an automated checks:
- https://github.com/owncloud/web/issues/5397
- https://github.com/owncloud/web/issues/5380
- https://github.com/owncloud/web/issues/10722
+accessibility standard rules violations
Why automate?
- Checking a11y rules is easy and quicker than manual
- Checks can be brought to the CI pipeline
- Early accessibility assessment
Why not depend on automated tests ONLY?
- Automated tests can only check if certain rules have been applied to certain elements or not
- Cannot guarantee the accessibility of a complete feature
Available Tools
Manual
- Google Lighthouse (available on chrome browser)
- Accessibility Insights
- ...
Automation
- axe-core (Most tools and libraries use this under the hood)
- e2e @axe-core/playwright
- component vue-axe@next
POC
Reports
| Lighthouse | Accessibility Insights |
|---|---|
Automated (axe-core):
{
"description": "Ensure buttons have discernible text",
"help": "Buttons must have discernible text",
"helpUrl": "https://dequeuniversity.com/rules/axe/4.10/button-name?application=playwright",
"id": "button-name",
"impact": "critical",
"failureSummary": "Fix any of the following:
Element does not have inner text that is visible to screen readers
aria-label attribute does not exist or is empty
aria-labelledby attribute does not exist, references elements that do not exist or references elements that are empty
Element has no title attribute
Element does not have an implicit (wrapped) <label>
Element does not have an explicit <label>
Element's default semantics were not overridden with role=\"none\" or role=\"presentation\"",
"html": "<button data-v-babade49=\"\" type=\"button\" class=\"oc-button oc-rounded oc-button-m oc-button-justify-content-center oc-button-gap-m oc-button-passive oc-button-passive-raw oc-ml-xs\" id=\"users-filter-confirm\">",
"none": [],
"target": ["#users-filter-confirm"],
"tags": [
"cat.name-role-value",
"wcag2a",
"wcag412",
"section508",
"section508.22.a",
"TTv5",
"TT6.a",
"EN-301-549",
"EN-9.4.1.2",
"ACT"
]
}
Proposal
Add automated accessibility checks:
- [ ] Rules checks - in ONE of the following forms
- Component level
- Separate e2e (Playwright)
- Integrate with existing e2e (Playwright - Cucumber)
- [ ] Keyboard only navigation - e2e (?)
- [ ] Rules checks - in ONE of the following forms
- Component level
- Separate e2e (Playwright)
- Integrate with existing e2e (Playwright - Cucumber)
@LukasHirt Could you recommend/suggest at which level we should have the automated accessibility checks? e2e would be more closer one to the end user experience. Also, if we use the existing playwright-cucumber setup (checking page accessibility using gherkin), it would be very quick to get started.
Could you recommend/suggest at which level we should have the automated accessibility checks?
How would it look like if it would be implemented as part of e2e tests? Would it go through every page one-by-one? Would that mean we need to handle setup of different scenarios to cover all the screens?
Generally speaking I am a bit hesitant to have this as part of e2e tests. Having to spin up the whole setup to check a11y roles, etc. feels like an overkill. Since this feels more like a "linter" (please correct me if I am wrong here) it looks more to me like a test that should be run outside of playwright.
Could you give me please some better understanding what's the difference in effort between these two approaches? I guess if you say that having this as part of e2e tests would be 10x faster to develop as POC than having it at the component level, I would say go for it. If the difference wouldn't be that drastic I would personally prefer trying the component level first. But that's my 2 cents. If you still believe that playwright for this is the way to go, I'm not going to block you.
How would it look like if it would be implemented as part of e2e tests? Would it go through every page one-by-one? Would that mean we need to handle setup of different scenarios to cover all the screens?
yeah exactly. e2e means accessing web via url, navigating to the page and do the accessibility checks. that means it tests the accessibility of integrated components.
As far as I can tell, with the component level, effort wise, we probably need to add a lot tests as there are large number of components and there will involve mocking and tests with state changes. This can do the element-level isolated checks.
But with e2e and since we already have working setup for e2e, we just have to implement one or two test steps and write an accessibility feature then run the test using the existing e2e test run command. Also, checking the whole page or certain part of the page would be closer to how/what the end users will experience. And if we integrate to existing e2e tests, maybe we can also have some scenarios or journey for keyboard only navigations later on.
I would say, it would be easier to get started with e2e test because of the existing setup. But with the component level, we might get lost in the large volume of available components in different packages. :sweat_smile:
As far as I can tell, with the component level, effort wise, we probably need to add a lot tests as there are large number of components and there will involve mocking and tests with state changes. This can do the element-level isolated checks.
I see. I misunderstood than how this would work. I was imagining something alongside of a scan of codebase and just checking whether the roles are applied in the code. Then yes, POC with e2e tests seems more reasonable to me as well.
I was imagining something alongside of a scan of codebase and just checking whether the roles are applied in the code
This is also valid but would be good to start with rendered html. Thanks Lukas, we will add checks for some important pages first.
We have implemented an accessibility test for the basic functionalities for file view (personal space, without sidebar) as an example/proof of concept for what aspects of accessibility can be tested in an automatic fashion using the axe-core plugin.
Motivation for this is making sure that oCIS web is compliant with the WCAG accessibility standards in light of European Accessibility Act (EAA) being implemented in June 2025
Next steps
- [ ] decide which accessibility guidelines should be included (currently we scan for WACA & best practice violations, see list of all available tags offered by axe-core)
- [ ] decide where to report findings and processes to make sure that the issues will get fixed
- [ ] consider if/which of the following aspects should be covered with similar accessibility tests
- [ ] login page
- [ ] app header functionalities (search, notifications, account page, app switcher, etc)
- [ ] left navigation sidebar of files view
- [ ] side bar with sub menus of files view
- [ ] shares (only functionalities that differ from personal files view)
- [ ] spaces? (it doesn’t seem to have any unique UI elements that aren’t used in other views)
- [ ] deleted files (trashbin)
- [ ] refactor page object if a lot more tests get implemented, create accessibility utility function
preliminary findings of usability issues (TBD where to document them, see open point above) some of them where detected through the tests implemented in https://github.com/owncloud/web/pull/12155, others (issues about login) through further spot checks (proper test scenario not yet implemented)
issues with table view
``` { id: 'button-name', impact: 'critical', tags: [ 'cat.name-role-value', 'wcag2a', 'wcag412', 'section508', 'section508.22.a', 'TTv5', 'TT6.a', 'EN-301-549', 'EN-9.4.1.2', 'ACT' ], description: 'Ensure buttons have discernible text', help: 'Buttons must have discernible text', helpUrl: 'https://dequeuniversity.com/rules/axe/4.10/button-name?application=playwright', nodes: [ { any: [Array], all: [], none: [], impact: 'critical', html: 'issues with login
``` { id: 'autocomplete-valid', impact: 'serious', tags: [ 'cat.forms', 'wcag21aa', 'wcag135', 'EN-301-549', 'EN-9.1.3.5', 'ACT' ], description: 'Ensure the autocomplete attribute is correct and suitable for the form field', help: 'autocomplete attribute must be used correctly', helpUrl: 'https://dequeuniversity.com/rules/axe/4.10/autocomplete-valid?application=playwright', nodes: [ { any: [], all: [Array], none: [], impact: 'serious', html: '', target: [Array], failureSummary: 'Fix all of the following:\n' + ' the autocomplete attribute is incorrectly formatted' }, { any: [], all: [Array], none: [], impact: 'serious', html: '', target: [Array], failureSummary: 'Fix all of the following:\n' + ' the autocomplete attribute is incorrectly formatted' } ] } ```issue with share side panel
``` { id: 'color-contrast', impact: 'serious', tags: [ 'cat.color', 'wcag2aa', 'wcag143', 'TTv5', 'TT13.c', 'EN-301-549', 'EN-9.1.4.3', 'ACT' ], description: 'Ensure the contrast between foreground and background colors meets WCAG 2 AA minimum contrast ratio thresholds', help: 'Elements must meet minimum color contrast ratio thresholds', helpUrl: 'https://dequeuniversity.com/rules/axe/4.10/color-contrast?application=playwright', nodes: [ { any: [Array], all: [], none: [], impact: 'serious', html: 'Add link', target: [Array], failureSummary: 'Fix any of the following:\n' + ' Element has insufficient color contrast of 4.19 (foreground color: #4a76ac, background color: #edf3fa, font size: 10.5pt (14px), font weight: normal). Expected contrast ratio of 4.5:1' } ] } ```issue with breadcrumbs
``` { id: 'aria-hidden-focus', impact: 'serious', tags: [ 'cat.name-role-value', 'wcag2a', 'wcag412', 'TTv5', 'TT6.a', 'EN-301-549', 'EN-9.4.1.2' ], description: 'Ensure aria-hidden elements are not focusable nor contain focusable elements', help: 'ARIA hidden element must not be focusable or contain focusable elements', helpUrl: 'https://dequeuniversity.com/rules/axe/4.10/aria-hidden-focus?application=playwright', nodes: [ { any: [], all: [Array], none: [], impact: 'serious', html: '
decide where to report findings and processes to make sure that the issues will get fixed
Maybe we could dedicate some time in sprints to fixing the current issues and after getting into a clean state, we could make it a requirement to have green pipeline on A11Y before merging? As long as we want to deliver accessible app, we should not merge things that break it.
@LukasHirt the proof of concept test (#12155) doesn't cover all key functionalities of the end user UI and some of the findings I documented above resulted from spot checks and experimental exploration of what we can do with the axe-core plugin. I expect that it would take about 2 days to implement basic accessibility test coverage of all key aspects of the parts of the UI, that are targeted to end users.