[BUG] Tests do not respect timeout

[pbiggar]

Context:

• Playwright Version: 1.26.1
• Operating System: Linux
• Node.js version: 14.20.1
• Browser: Chromium

Playwright config publicly available

Code Snippet

// Config file for integration tests
import { PlaywrightTestConfig } from "@playwright/test";

const config: PlaywrightTestConfig = {
  testDir: ".",
  testMatch: "tests.ts",
  expect: {
    timeout: 5000,
  },
  // If nothing is printed for 10m, then CircleCI times us out, which means we don't
  // find out which test hung. So timeout here first so that it tells us what hung.
  globalTimeout: 600000,
  // in ms. 30000 = 30s
  timeout: 30000,
  use: {
    actionTimeout: 5000,
    headless: true,
    trace: "retain-on-failure",
    screenshot: "off",
    video: "on",
  },
  reporter: [
    ["list"],
    ["json", { outputFile: "rundir/test_results/integration_tests.json" }],
    ["junit", { outputFile: "rundir/test_results/integration_tests.xml" }],
  ],
};

export default config;

Describe the bug

Sometimes when I run my playwright tests, some of the tests hang despite a timeout being set.

For example:

https://app.circleci.com/pipelines/github/darklang/dark/4283/workflows/59ac7d0e-36d7-44a9-8ebf-e947dd9ee268/jobs/44444/parallel-runs/0/steps/0-109

In this CI run, the globalTimeout kicks in after 10 minutes, but the individual test timeout did not kick in after 30s.

(Note the overall run time of these tests is typically less than 5m, and before we added the globaltimeout we'd see the same thing, except circleci would timeout due to no output for 10m.)

[dgozman]

@pbiggar I looked at the report, and it has more than 60 tests, with an average test time of ~10seconds. This amounts for more than 10 minutes, so globalTimeout kicks in. Seems like working as expected.

[pbiggar]

Those tests are split across 8 workers though.

[pbiggar]

Here's a run where the test doesn't hang, taking a total of a little over 2m. That's how long they usually take when no test hangs. https://app.circleci.com/pipelines/github/darklang/dark/4284/workflows/cf16dadb-516c-4222-b960-c223b9146f93/jobs/44482

[dgozman]

@pbiggar Indeed, it seems like everything stalled after the last test finished. Could you run with DEBUG=pw:test environment variable? Perhaps that could give us some insight.

Also, are there any instructions on how I can run this? Does this ever reproduce locally, not on Circle CI? It would be ideal for us to have a repro so that we can debug the issue.

[pbiggar]

I'll try with DEBUG shortly.

I have only ever seen this on CI, despite significant efforts. If you clone the repo at https://github.com/darklang/dark, you should be able to run it on CircleCI by enabling the project in CircleCI and then just pushing to your branch. CircleCI has an SSH feature where you can connect to an instance, so that might be a good way to reproduce.

[pbiggar]

Looks like I got one to hang here: https://app.circleci.com/pipelines/github/darklang/dark/4299/workflows/81480f92-be82-40e7-badc-3c86442be11d/jobs/44706, with DEBUG enabled.

[dgozman]

@pbiggar Thank you for the logs run. Unfortunately, logs don't show anything suspicious. Since we cannot reproduce, let me add more logging so that we can better debug issues like that.

[pbiggar]

Thanks for looking. Let me know when the debugg statements are done and I can try with the new code.

Alternatively, it should be really easy to reproduce this as this code is all source available. Instructions here, and happy to help if I can.

[jweb89]

We're having the exact same issue in CircleCI. Hangs on last test and ends up timing out but can't reproduce locally.

[pbiggar]

This has actually happened more and more for us too.

[ltsuda]

I'm sorry if this is not related to the issue, but I see that in you project there's playwright and playwright/test packages. A long shot here, is it possible to be something with this?

https://github.com/darklang/dark/blob/main/integration-tests/package.json

[dgozman]

We still need a reliable repro to be able to pin-point the issue. The fact that it happens mostly (only?) on CircleCI is concerning. Note that we are still working on more comprehensive logging that could theoretically help, but there is no guarantee.

[nicolae536]

This also happens to me in Github actions, unfortunately the pattern seems Random and I cannot give you access to my repository since it's owned by the company. If you want we can do a call to show you the actions where it failed, I can also provide you the reports since they are stored in artefacts just write me if you need them.

[nicolae536]

I understood the problem 😶‍🌫️ It seems my Failed Tests were not failed were actually Flaky and I have retry attempts set to 2 and it succeeds on retry. and then the pipeline naturally succeds.

[margarytaD]

Looks like I have the same issue with CircleCI. Started discussion here https://github.com/microsoft/playwright/discussions/24323 . Is there any workaround for it 🙏 ?

[Rohanpandey268]

I am having the same issue in Gitlab runner machine, where the config timeout is ignored by test. I have a config timeout of 5 mins in config file(timeout: 5 * 60 * 1000, actionTimeout: 30 * 1000,). But test get hanged for hours and it gets over when job timeout. In that case even artifacts does not get attached, so i am not able to figure out. Locally, it works fine "@playwright/test": "1.38.0"

[VinayKumarBM]

@Rohanpandey268 Even I'm facing similar issue, any breakthrough or workaround for this issue ?

[Rohanpandey268]

@Rohanpandey268 Even I'm facing similar issue, any breakthrough or workaround for this issue ?

@VinayKumarBM For me the curprit was waitForAngular function used in my code which does not have a time constraint. Putting it in a promise race condition solved it for me.