[BUG] Empty extra video with axe playwright

[apis3445]

System info

• Playwright Version: 1.34
• Operating System: All
• Browser: All

Source code

• [x ] I provided exact source code that allows reproducing the issue locally.

Link to the GitHub repository with the repro

https://github.com/apis3445/playwright-sample

Test file (self-contained)

https://github.com/apis3445/playwright-sample/blob/master/tests/accessibility.spec.ts

Steps

• [Run the test]
• [See the HTML report]

Expected

[Describe expected behavior]

I am not sure why is adding an empty video after the right video when I am adding the axe playwright step to check accessibility

[Describe actual behavior]

I am adding some accessibility checks with playwright and @axe-core/[email protected] but is adding a extra empty video

[dgozman]

@apis3445 That's probably because you have multiple pages in a single test. For example, I see { page } being used in beforeEach() and test(), although it looks like they are intended to be different pages. More broadly, saving page from beforeEach() into a variable is probably not what you wanted, because it will be closed after each test, and using it in afterAll() seems like an error.

[apis3445]

I refactor the code to remove beforeEach, but it's the same as adding an empty video for each page (HomePage, EditroPage,SettingsPage)

[mxschmitt]

You are launching Chromium manually and connecting to a third-party service via connect() which we don't support. We recommend to specify the options in the connectOptions in the playwright.config.ts via:

    {
      name: 'webkit',
      use: {
        ...devices['Desktop Safari'],
        connectOptions: {
          wsEndpoint: `wss://cdp.lambdatest.com/playwright?capabilities=${encodeURIComponent(JSON.stringify(capabilities))`
        }
      },
    },

(for all your projects)

and never do chromium.launch inside your config, then it should all work.

[apis3445]

In the tests I am not using that connection. If I can't use chromium launch how I can login with forms (not api) and store session?

[apis3445]

I remove the global setup, and it's the same; the report is adding an extra empty video.

[mxschmitt]

Your reproduction is fairly complicated and this issue grew over time. Can you help us with a smaller repro by any chance?

[apis3445]

It is not complicated this is the smaller repo and smaller test

[apis3445]

I don't understand why is complicated

[mxschmitt]

It has a lot of files, overrides fixtures, has LambdaTest code in it which is not used. And if I try to clone it I can't run it, since it throws errors. So please remove all the unnecessary stuff and then we can take a look again, thanks for you understanding!

[apis3445]

What errors I can clone and run without any errors

[apis3445]

The test should fail and it's very hard create a repo for all the bugs for this new version of playwright, so I added one repo for all current bugs that I have open. The test always should fail but will show only one video not one extra empty video This is the test for this bug https://github.com/apis3445/playwright-errors/blob/main/tests/accessibility.spec.ts

[mxschmitt]

Thank you appreciate!

So I added a breakpoint at the page creation and found out, that AxeBuilder is creating the second page, so it seems not an issue from Playwright:

    at BrowserContext.newPage (/Users/maxschmitt/Developer/tmp/playwright-errors/node_modules/@playwright/test/node_modules/playwright-core/lib/client/browserContext.js:224:17)
    at AxeBuilder.finishRun (/Users/maxschmitt/Developer/tmp/playwright-errors/node_modules/@axe-core/playwright/dist/index.js:335:37)
    at AxeBuilder.analyze (/Users/maxschmitt/Developer/tmp/playwright-errors/node_modules/@axe-core/playwright/dist/index.js:250:25)
    at /Users/maxschmitt/Developer/tmp/playwright-errors/utils/accessibility/accessibilityHelper.ts:65:46
    at /Users/maxschmitt/Developer/tmp/playwright-errors/node_modules/@playwright/test/lib/common/testType.js:190:14

I would strongly recommend to change your artifact config btw, from: trace: 'retain-on-failure', video: 'retain-on-failure', screenshot: 'only-on-failure', to: trace: 'on-first-retry', video: 'off', screenshot: 'off', The reasoning behind that is that tracing is more more powerful for debugging. Having it only activated for a test which is only failing, makes your test execute much faster. For the video the same story, but there we tend to generally step away from video based debugging to trace based debugging. Same for screenshots.

I hope this answers your question, thanks!

[apis3445]

This is not the answer I can't change to on-first-retry. Retry is very slow for test so in general I don't use retry option for failure tests

[mxschmitt]

Sounds like we have nothing to act on here in this issue, since Playwright is just recording all the videos from the pages which are getting opened. I recommend filing an issue on Axe side, that they maybe can get rid of this extra page somehow.

Closing by that, feel free to re-file for further questions or bug reports.

[apis3445]

It's not a new page it's the same page

[ltsuda]

Thank you appreciate!

So I added a breakpoint at the page creation and found out, that AxeBuilder is creating the second page, so it seems not an issue from Playwright:

    at BrowserContext.newPage (/Users/maxschmitt/Developer/tmp/playwright-errors/node_modules/@playwright/test/node_modules/playwright-core/lib/client/browserContext.js:224:17)
    at AxeBuilder.finishRun (/Users/maxschmitt/Developer/tmp/playwright-errors/node_modules/@axe-core/playwright/dist/index.js:335:37)
    at AxeBuilder.analyze (/Users/maxschmitt/Developer/tmp/playwright-errors/node_modules/@axe-core/playwright/dist/index.js:250:25)
    at /Users/maxschmitt/Developer/tmp/playwright-errors/utils/accessibility/accessibilityHelper.ts:65:46
    at /Users/maxschmitt/Developer/tmp/playwright-errors/node_modules/@playwright/test/lib/common/testType.js:190:14

I would strongly recommend to change your artifact config btw, from: trace: 'retain-on-failure', video: 'retain-on-failure', screenshot: 'only-on-failure', to: trace: 'on-first-retry', video: 'off', screenshot: 'off', The reasoning behind that is that tracing is more more powerful for debugging. Having it only activated for a test which is only failing, makes your test execute much faster. For the video the same story, but there we tend to generally step away from video based debugging to trace based debugging. Same for screenshots.

I hope this answers your question, thanks!

Following @mxschmitt comment here, see the same/similar issue from Axe's package.

https://github.com/dequelabs/axe-core-npm/issues/707

The problem with you code is probably when you can this line here that creates a new empty page from Axe, the same as described in the link above.

https://github.com/apis3445/playwright-errors/blob/main/utils/accessibility/accessibilityHelper.ts#L65

Here is the same code that user calls in their code.

const fromAxe = await new AxeBuilder(driver)
                .options({
                    resultTypes: ['violations']
                })
                .setLegacyMode()
                .analyze();

I hope you can get some help but the issue seems to be with the Axe team.

[mxschmitt]

Thats the code pointer to where they create a new temporary page for every analyze() call: https://github.com/dequelabs/axe-core-npm/blob/48e222e87da6f5ee79a81f61ab2e4cc4cc3e6ae6/packages/playwright/src/index.ts#L277