axe-core timeout with mock timer in JS unit tests

[mohanraj-r]

Product: axe-core

Expectation: axe-core does not timeout when timer is mocked in JS unit tests

Actual: When mock timers are used with axe (e.g. jest.useFakeTimers()), tests fail with error "Timeout - Async callback was not invoked within the 5000ms timeout specified by jest.setTimeout.Timeout"

Motivation: Mocking timers are common in JS unit tests. Getting axe-core to work as expected even when mock timers are being used would be great. The timeout issue occurs even with other third-party mock timer libs such as @sinonjs/fake-timers - so it is not a problem just with Jest.

axe-core version: 4.2.3

For Tooling issues:
- Node version: v14.17.1
- Platform:  OSX 10.15.7

Something about mocking timers results in axe timeout - not sure why ?

[straker]

Thanks for the issue. Could you provide an example of a test using axe and fake timers? Axe doesn't use too many timeouts so I'm not sure if timers would help make axe mock a run.

[mohanraj-r]

Here is a minimal example @straker

describe('demo axe timeout with mock timer', () => {
    it('should not timeout when using mock timer', async () => {
        jest.useFakeTimers(); // Commenting this line out will make the test pass
        await axe.run();
    });
});

Running the above test with Jest results in timeout error.

  demo axe timeout with mock timer
    ✕ should not timeout when using mock timer (5022 ms)

  ● demo axe timeout with mock timer › should not timeout when using mock timer

    : Timeout - Async callback was not invoked within the 5000 ms timeout specified by jest.setTimeout.Timeout - Async callback was not invoked within the 5000 ms timeout specified by jest.setTimeout.Error:

      39 |
      40 | describe('demo axe timeout with mock timer', () => {
    > 41 |     it('should not timeout when using mock timer', async () => {
         |     ^
      42 |         jest.useFakeTimers();
      43 |         await axe.run();
      44 |     });

      at new Spec (../../node_modules/jest-jasmine2/build/jasmine/Spec.js:116:22)
      at Suite.<anonymous> (__tests__/jest.test.js:41:5)
      at Object.<anonymous> (__tests__/jest.test.js:40:1)

Test Suites: 1 failed, 1 total

[straker]

Out of curiosity, does jest.runAllTimers(); help at all?

[mohanraj-r]

Tried jest.runAllTimers(); - it doesn't help, still times out.

[WilcoFiers]

@mohanraj-r any guess as to what is causing the issue? If you're up for it, we'd appreciate a pull request. Axe-core needs setTimeout for a few things. I'm not sure if there is a way around it, but if you have an idea, we'd be open to it.

[carloscasal]

I wonder if people are still encountering this issue. Been more than a year now. 👀

[compulim]

I am still seeing it with fake timers.

Looks like we need a way to let axe-core use/remember the original setTimeout before fake timer is installed.

[compulim]

I think axe-core should add an option to axe.configure() and allow passing the real setTimeout in case the clock is faked/polluted.

AFAIK, axe-core use setTimeout for checking things across frames (including the main frame). When a fake timer (jest, sinonjs, or lolex) is installed globally, axe-core will use the globally polluted setTimeout and it would fail all checks.

The alternative is to ask the app developer not to pollute the global setTimeout. I have done this in one of my repos. This is very tricky and very limited because:

• All clock-related functions need to be unpolluted, including setTimeout, setInterval, and the whole Date class
  • For Date class, think about the code calling Date.now() and new Date().getTime(), both should return 0
  • Date class could be used everywhere in the app
• Third party packages used in the app may still call the global setTimeout(), setInterval(), Date.now(), and Date.prototype.getXXX()
  • Unpolluting fake clock functions will only apply to 1P, not 3P packages until they adopted this approach
  • Mixing fake/real clock could fail the original test scenarios (imagine 1P is using a date of 0, while the 3P is on a date of "now")
  • Then, app developers will need to ask all 3P packages to use a different set of clock functions, this is close to impossible for now

So, it is much easier for axe-core to use a setTimeout passed from the axe.configure().

[clyncha]

Was this ever solved? I'm seeing the same issue even if I advance my timer before using axe()