[BUG] Test mistakenly identified as flaky

[agoldis]

trafficstars

System info

• Playwright Version: v1.39.0
• Operating System: All
• Browser: All
• Other info:

Source code

• [x] I provided the exact source code that allows reproducing the issue locally.

Test file (self-contained)

import { expect, test } from "@playwright/test";

test.describe.configure({ mode: "serial", retries: 3 });

test("A", async ({ page }, { retry }) => {
  if (retry === 0 || retry === 2) {
    throw new Error("oh!");
  }
  if (retry === 1 || retry === 3) {
    expect(true).toBe(true);
  }
});

test("B", async ({ page }, { retry }) => {
  throw new Error("oh!");
});

Steps

• Run the test npx playwright test
• See both tests identified as flaky
• CLI exit status is 0

1.39 produces the following outcome for attempts:

• Test A: failed, passed, failed, passed
• Test B: skipped, failed, skipped, failed

Expected

• Test B should not be flaky, it should be failed. All the non-skipped attempts do not match the expected status - i.e. the test never produces the expected passed status.
• CLI exit status should not be 0 to mark

Actual

• Both tests are identified as flaky
• CLI exit status is 0 and the CI passes

[403-html]

hmm same, so bump, any news? After bump to 1.40.1 seems like it's the same case

[Manvel]

In our company we also has been affected by this (potentially related to latest serial mode switch).

Unless missing something, it seems like https://github.com/microsoft/playwright/pull/26385 was targeted towards fixing https://github.com/microsoft/playwright/issues/17652#issuecomment-1260552486 which seem to have introduced current edge case regression with serial mode.

On the other hand I read that serial mode usage is not recommended(for a different reason though).

I would rather expect that all tests that are not specifically marked as skipped(test.skip) to pass at least once before being marked as Flaky.

(I see current ticket is marked as feature, but potentially might-be regression)

@aslushnikov @dgozman - can you please confirm if this is expected behavior of how playwright tests suppose to run?

[scottwalter-nice]

We have a similar issue. When retry is set to > 0 its marking the test a flaky and not a failure. Even though it failed on the retry:

[scottfwalter]

@mxschmitt is this considered to be a bug? Has there been any prioritization for it?