cypress-axe
cypress-axe copied to clipboard
component-driven
Waiting arbitrary time for the checks to work
Related: #22, vercel/next.js#7945
Full repo: here (note: some of the packages there are not yet published, but the test can run without them)
I have a simple test:
import formatAxeLog from "../helpers/formatAxeLog";
describe("Home page", () => {
beforeEach(() => {
cy.visit("/");
});
it("Has no detectable a11y violations on load", () => {
cy.injectAxe();
// cy.wait(500);
cy.checkA11y(null, null, formatAxeLog);
});
});
When I run the tests multiple times they sometimes fail (due to vercel/next.js#7945, as they should), sometimes pass. The only "reliable" way is just to wait. Am I doing something wrong, or this is a bug?
Can we make it wait the same timeout amount that cypress waits for before failing an assertion?
I'm having the same issue with an Angular app <a href="somwhere">{{ 'some.text' | translate }}</a>
translations are fetched so they sometimes take a bit longer to appear
I ran into a similar issue. I was surprised I was getting no wcag2aa violations in my site, so I removed alt text in some of my images to test the tests, so to speak. I still had no violations.
Once I ran my test with a cy.wait(1000), I started seeing serious and critical impacts. So, the problem seems to be that Axe is running before the images (and the page) are fully loaded.
It seems like there could be a built-in fix for this, or at least a recipe on how to work around the problem.
FYI, here is my code:
cy.visit(page);
cy.injectAxe();
cy.wait(1000);
cy.checkA11y(null, a11yOptions, terminalLog, true);
I've to the conclusion that we should wait for things that we are asserting against.
So in case of images (or translations), the test should
it('waits for content', () => {
cy.intercept('/path/image.webp).as('image');
cy.visit('/');
cy.injectAxe();
// test and interact with other stuff that doesn't need waiting
cy.wait('@image');
cy.checkA11y();
});
Any possible solutions without cy.wait(1000); ? I want to avoid that workaround.
Any possible solutions without
cy.wait(1000);? I want to avoid that workaround.
Instead of wait, you can try and get a DOM element that you know will render once the page is ready for the tests to run.
Given a markup like
<html>
...
<body>
<main data-test="main-region">
This loads when the page is ready for testing.
</main>
</body>
</html>
you could write a test like this
it('has no detectable a11y violations on load', () => {
cy.visit('/')
cy.get('[data-test=main-region]')
cy.injectAxe()
cy.checkA11y()
})
☝️ this could be a false positive scenario.
cy.injectAxe() takes time (~a few ms sometimes) to load axe-core into window, which might explain why you don't get any violations reported when calling cy.checkA11y() right after cy.injectAxe().
To confirm (or refute) this claim, try adding cy.wait(1000); between the two commands and see if you get any violations.
Any possible solutions without
cy.wait(1000);? I want to avoid that workaround.
I fixed this by using @mcha-dev's solution, by waiting for my API call:
cy.intercept('POST', Cypress.env('api_server'), (req) => {
.
.
.
}).as('apiLoadCall');
cy.wait('@apiLoadCall').then(() => {
cy.injectAxe();
cy.checkA11y();
});
This solution waits for my fake API call (I load .JSON fixtures to simulate my backend data) to finish and then executes a11y check on the selected website.
@mcha-dev made an excellent suggestion:
Can we make it wait the same timeout amount that cypress waits for before failing an assertion?
For a long time, I assumed that this was already the case. After all, Cypress was built with retry-ability in mind. I assumed that this package would also retry failing assertions as well but over time I have come to learn that this doesn't seem to be the case. Failures due to the lack of retrying in cy.checkA11y() are a big source of flake in our test suite and it feels like some simple automatic retrying of any failed checks would alleviate a lot of burden involved in using this package.
To add a use-case, I'm trying to checkA11y before and after a button is enabled and the button has a transition set for the bg/color changes. I'm getting random errors depending on where in the transition the check catches the button. I don't see a way to wait for the transition to end aside from the [bad practice] cy.wait. Having the a11y checks follow the default assertion timeout would work wonders.
Feature Suggestion
It would be lovely if checkA11y could accept a selector (or an array of selectors) that must be in the DOM before the check runs. e.g:
cy.checkA11y({
waitFor: ['h1', '.mat-card'], 👈
exclude: ['.firebase-emulator-warning']
})
I'm also facing the same issue, tried resolving with cy.wait, however this is not an elegant solution at all. I can't rely on API Calls, because there are no API calls when I click on an element. I could make the test more stable by adding cy.injectAxe(); right before cy.checkA11y( ( previously I've had it in beforeEach block )
as cy.injectAxe(); takes some time to be loaded ( as was stated in earlier comments ) it gives some time for the image to load. this is disgusting solution of course and still very flaky So, if anyone could share how you resolved it in a more elegant way, that would be great.
