[v3.0.5]Allure-Report not marking tests as flaky (allure-playwright)

[PatatN7]

Describe the bug When a tests failed within the last 5 test runs, the test is not marked as flaky. None of the conditions specified in the documentations mark a test as flaky (https://allurereport.org/docs/test-stability/)

To Reproduce Steps to reproduce the behavior:

1. Run a test and make sure it passes
2. Generate allure report
3. Clean out allure-results folder
4. Run a test and make sure it fails
5. Copy history folder from allure-report to allure-results
6. Generate allure report
7. Clean out allure-results folder
8. Run a test and make sure it passes
9. Copy history folder from allure-report to allure-results
10. Generate allure report
11. Repeat steps 7-10 until you have a total of 5 test runs

Expected behavior Expected the test that failed in the last 5 test runs to be marked as flaky.

Screenshots This is what my trend looks like where multiple tests failed within the last 5 test runs

But none of the tests were marked as flaky

Desktop :

• OS: Windows 11
• Browser: Chrome
• Version: 131.0.6778.86

Additional context The tests are also set to retry 3 times when failed

After the tests have completed, it does show in the CLI that there are flaky tests, but it is not shown in the allure-report.

My playwright configuration is also set to only allure-playwright as the reporter

Command used to run tests: npx playwright test JobLogin.spec.ts -g "Job login and logout" Command used to generate reports: ** allure generate --clean**

[clergyman]

Hello @PatatN7 , thank you for detailed report and insightful screenshots Please refer to Allure Report Test Stability documentation

In order for the flaky bomb to show up, ALL of the following conditions should be met Current result is FAILED or BROKEN AND There's at least one PASS in the last 5 runs AND There's at least one FAILED in the last 5 runs

In other words, it should be failed or broken and there has to be 2 transitions between passing and failing within the last 5 runs, and looking at the results from your example, the results do not match the pattern - they are passed and don't have 2 transitions

What you are seeing in the console is probably a built-in playwright evaluation of flakiness, that is not passed to allure results.

We are planning on improvement the flakiness detection in Allure3, please refer to this issue and please feel free to express your vision in the comments, for example, allure could adopt the flag from the framework, or detect flakiness by user-defined patterns (which is already one of the options in demand)

I am going to close this issue as working by design some time soon