[Bug]: reportSlowTests setting doesn't have effect

[mindtraveller]

Version

1.54.1

Steps to reproduce

Set reportSlowTests object in the config: e.g.

reportSlowTests: {
  threshold: 2, <- this is ms according to docs
  max: 10,
}

Run test tests which take more time to run than set threshold value. Observe that in the test report there are no info about slow tests. Test reporters verified: line, html.

Expected behavior

The tests with execution time more than set threshold should be reported.

Actual behavior

No tests reported

Additional context

No response

Environment

System:
    OS: macOS 15.6.1
    CPU: (14) x64 Apple M4 Pro
    Memory: 929.38 MB / 48.00 GB
  Binaries:
    Node: 22.9.0 - /Users/vladimirpenyazkov/.asdf/installs/nodejs/22.9.0/bin/node
    Yarn: 3.6.1 - /Users/vladimirpenyazkov/.asdf/installs/nodejs/22.9.0/.npm/bin/yarn
    npm: 10.8.3 - /Users/vladimirpenyazkov/.asdf/installs/nodejs/22.9.0/bin/npm
  Languages:
    Bash: 5.2.26 - /usr/local/bin/bash
  npmPackages:
    @playwright/test: 1.54.1 => 1.54.1

[Skn0tt]

Hi! We have good test coverage on this, and you're the first one to report the bug. It's most likely related to your specific setup. If you think it's not, could you provide me with a minimal reproduction case of this, as per the issue template?

[mindtraveller]

Hey, sure, I will create a playground to reproduce the issue

Best Regards, Vladimir Penyazkov

On Mon, Oct 6, 2025, 19:17 Simon Knott @.***> wrote:

Skn0tt left a comment (microsoft/playwright#37724) https://github.com/microsoft/playwright/issues/37724#issuecomment-3372582004

Hi! We have good test coverage on this, and you're the first one to report the bug. It's most likely related to your specific setup. If you think it's not, could you provide me with a minimal reproduction case of this, as per the issue template?

— Reply to this email directly, view it on GitHub https://github.com/microsoft/playwright/issues/37724#issuecomment-3372582004, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACVVTNEFIXNNDP75CVSSVA33WKIYRAVCNFSM6AAAAACIM5A6DOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTGNZSGU4DEMBQGQ . You are receiving this because you authored the thread.Message ID: @.***>

[mindtraveller]

Here is an example to showcase the issue.

I used npm init playwright@latest to setup the project, so it doesn't have any extra functionality. I adjusted the list of reporters to include list as by default there was html only. And added reportSlowTests setting with 100ms threshold, which should match all available tests.

The result - no tests reported as slow ones in console nor in html report:

Attaching zip with the project: pw-example.zip. Install deps, then use npx playwright test to run tests and observe the results.

[Skn0tt]

Thanks, I can reproduce! That's indeed surprising. I dug up the code, and it seems to be related to files being executed across multiple workers:

https://github.com/microsoft/playwright/blob/9de91b78af11efc2ab05ca1a1a76df100a06396b/packages/playwright/src/reporters/base.ts#L234-L236

I'll discuss this with the rest of the team, but my hunch is that we have this in place so we don't double-count worker setup. Apparently it also leads to false negatives.

Could you elaborate on the context where you discovered this? Was this on a real, big test suite?

[Skn0tt]

Looking at the docs again, it looks like it tricked both of us. This is not about tests, but about test files. I'll see what we can do.

[mindtraveller]

It's our production setup, currently we have 5 shards and 4 workers in each of them. The amount of tests close to 3k. I wanted to bring reporting of slow running tests, that's how I found out that the setting doesn't work.

[Skn0tt]

I'll try make it do what it says on the tin, and otherwise we'll deprecate it. It doesn't seem to be doing something useful in its current form.

[lanej0]

I think this is the same issue that I'm running into. After some debugging, I've found that:

• It only seems to be happening when blobs are merged from sharded execution. If you aren't running your test suite sharded, there's no issue.
• During test execution, the values are being picked up correctly (I output them in GH actions in that step). It's only during merge that they're being corrupted (in my case, it's the threshold and max values). When I output the values during that step, they've been reset to 300000 and 5 (from 7000 and 50 that I have set in my playwright.config.js)

[Skn0tt]

I tried changing it, then deprecating it, all is nontrivial. I'll turn this into "P3" to collect some upvotes. If you're reading this and stumbled over it, please thumbs-up the issue. If a lot thumbs-ups are collected, i'll take another look.