18f.gsa.gov icon indicating copy to clipboard operation
18f.gsa.gov copied to clipboard

Prevent pa11y from silently failing in CI

Open echappen opened this issue 1 year ago • 0 comments

Feature suggestion

When pa11y scans a URL and returns a scanning error, the CI job fails. But when pa11y fails to scan any urls and returns a generic Failed to run error, the CI job still succeeds.

I propose failing the entire CI job in this scenario, in order to prevent silent failures.

User story

As a developer, I want to know if my build is failing so I can prevent a11y bugs from entering production.

Other considerations and design implementation suggestions

Note that existing behavior for pa11y job in CI (before and after the recent pa11y bug) is to SIGHUP regardless if the URL scans succeed. I'm not sure if this behavior was intended or correct, but a SIGHUP may be causing the job to succeed in all scenarios.

echappen avatar Oct 25 '23 14:10 echappen