Axe plugin

[matejchalk]

Metric

100+ web accessibility metrics provided by Axe (listed here).

User story

EAA compliance is likely to be a growing concern in the coming years for companies based in Europe. From June 25, 2025, all new digital products and services in the EU market must meet accessibility standards defined by the EAA. For services that were already in operation before that date, the law allows a transition period until June 30, 2030. The requirements are closely aligned with the Web Content Accessibility Guidelines (WCAG).

This is expected to increase demand for accessibility testing tools. Therefore, building an A11Y solution into Code PushUp would make it more appealing for many potential customers.

Accessibility is somewhat covered by existing plugins:

• Lighthouse measures an Accessibility category with ~50 pass/fail audits (uses axe-core)
• ESLint can be configured to check source code for A11Y anti-patterns using plugins like eslint-plugin-jsx-a11y or @angular-eslint/eslint-plugin-template

However, neither seems to be as comprehensive as Axe.

Dependencies

• axe-core
• @axe-core/cli (optional)
• @axe-core/puppeteer (optional)
• @axe-core/playwright (optional)

Acceptance criteria

• [ ] plugin executes axe-core with user-provided configuration
  • [ ] multiple URLs can be provided
  • [ ] custom scripts may be provided, which serve to programmatically reach certain application states
• [ ] plugin parse JSON report from AXE and convert results to Code PushUp data models
  • [ ] Axe rules should map to (binary) audits 1-to-1
  • [ ] violations should be reported as issues with a given severity level
  • [ ] at least one accessibility group should be provided, to simplify configuration
• [ ] plugin E2E tests run against local example app
• [ ] plugin is (manually) tested against some application which requires a login (e.g., Code PushUp Portal)

Implementation details

The simplest way to run Axe is via its CLI. E.g.:

npx @axe-core/cli https://mdn.github.io/learning-area/accessibility/html/bad-form.html --save=axe.json

However, we need to consider that for many modern applications, it won't be sufficient to provide a URL, as many features won't be reachable without a user login or some other interaction. Therefore, we should provide users with the option to provide their own test scripts to reach the application states they're interested in. These scripts could be written using one of the testing tools Axe supports (e.g., Playwright, Puppeteer, WebdriverIO).

[matejchalk]

I've encountered a few timeout errors since the plugin was merged in #1141(e.g., here and here):

  TimeoutError: page.goto: Timeout 30000ms exceeded.
  Call log:
    - navigating to "https://github.com/code-pushup/cli?tab=readme-ov-file#code-pushup-cli/", waiting until "networkidle"

So far, I've been able to "solve" it by retrying the failed job.

In my experience from E2E tests, it's often a good idea to first "warm up" the server in a setup script, to prevent flakiness due to a slow initial response (cold starts are pretty common). I guess we could have the user provide their own setupScript for this purpose in #1135. Or we could make it the default plugin behaviour, that we always ping the URL first. What do you think? 🤔