[Bug]: Test duration is incorrect with beforeAll and afterAll

[TomW-Skyline]

Version

1.43.1

Steps to reproduce

To gather performance data of a web application, we are using the duration of Playwright tests. The problem is that the time of the beforeAll method is added to the first test in the file, and the time of the afterAll method is added to the duration of the last test.

Because of this, the duration of the first and last test is not correct in the report. We are not interested in how long it took to setup the test, and to cleanup afterwards.

Expected behavior

The duration of the beforeAll/beforeEach and afterAll/afterEach methods is not added to the duration of the first and last test.

Actual behavior

The duration of the first and last test is not correct.

Additional context

No response

Environment

System:
    OS: Windows 11 10.0.22631
    CPU: (8) x64 Intel(R) Core(TM) i7-9700 CPU @ 3.00GHz
    Memory: 6.99 GB / 31.78 GB
  Binaries:
    Node: 20.11.1 - C:\Program Files\nodejs\node.EXE
    npm: 10.5.0 - C:\Program Files\nodejs\npm.CMD
  IDEs:
    VSCode: 1.95.3 - C:\Program Files\Microsoft VS Code\bin\code.CMD
  npmPackages:
    @playwright/test: ^1.43.1 => 1.49.0

[Skn0tt]

Hi Tom! What report are you referring to? Is it the HTML report? Do you use a custom reporeter?

[TomW-Skyline]

Hi @Skn0tt, I was referring to the standard json reporter, but also the junit XML report does it. Was thinking to create my own custom reporter, but already goes wrong during the collection of timings.

[yury-s]

Can you share example project where this happens?

I ran the following code with html, json and junit reporters:

import { test, expect } from '@playwright/test';

test.beforeAll(async () => {
  await new Promise(f => setTimeout(f, 2000));
});

test.afterAll(async () => {
  await new Promise(f => setTimeout(f, 2000));
});

test('test', async ({ }) => {
});

And all of them only include test self time.

In junit report only the top level <testsuites> contains the total time:

<testsuites id="" name="" tests="1" failures="0" skipped="0" errors="0" time="4.323623">
<testsuite name="my-test.spec.ts" timestamp="2024-12-09T18:48:10.273Z" hostname="chromium" tests="1" failures="0" skipped="0" time="0.002" errors="0">
<testcase name="test" classname="my-test.spec.ts" time="0.002">

[TomW-Skyline]

Apparently it only happens with beforeEach and afterEach.

[yury-s]

beforeEach and afterEach are considered a part of the test (as there is a separate call for each test) and their execution time is included into the test duration. This is by design.

[yury-s]

Closing per the response above, feel free to open a new issue if it doesn't work.