Passed tests marked as failed under specific circumstances

[JessefSpecialisterren]

Describe the bug When test retries are enabled and a beforeEach hook fails once in an early test and 1 + retries times in a later test, all tests from the early test to the later test are marked as Broken/Defect: This causes incorrect failure counts in the allure reports of our nightly Cypress runs.

To Reproduce

1. Clone https://github.com/JessefSpecialisterren/cypress-allure-bug and open a terminal in the cloned repo
2. npm ci
3. npx cypress run
4. allure serve (note: allure2 is not included in the repo, so install it manually if it isn't already on your system)
5. Compare the tests marked as Broken/Defect in the allure report and the actually failed tests in Cypress's command line output

Expected behavior In the allure report, test E should be marked as Broken/Defect, and all the other tests should be marked as Passed.

Actual behavior In the allure report, tests B, C, D and E are all marked as Broken/Defect.

Environment

• Cypress version: 10.3.0
• OS: Windows 10
• cypress-allure-plugin version: 2.28.0
• allure2 version: 2.18.1

Additional context Thank you so much for developing and maintaining this plugin, it's by far the best reporting solution we've been able to find for Cypress projects that cannot use Dashboard!

[kunalashar25]

The same issue happens without retries as well.

Case 4 failed

but the report shows cases 2 to 5 are broken.

code

describe('Allure issue when beforeEach hook fails', () => {
    before('To pritn message', () => {
        console.log("I'm before");
    });

    after('To pritn message', () => {
        console.log("I'm after");
    });

    beforeEach('Will fail after few iterations', () => {
        const numbers = [50, 30, 1, 20];
        const num = numbers[Math.floor(Math.random() * numbers.length)];
        expect(num).to.be.greaterThan(10);
    });

    it('case 1', () => {
        console.log('case 1');
    });
    it('case 2', () => {
        console.log('case 2');
    });
    it('case 3', () => {
        console.log('case 3');
    });
    it('case 4', () => {
        console.log('case 4');
    });
    it('case 5', () => {
        console.log('case 5');
    });
});

[Shelex]

Fix published in v2.30.0. Please reopen in case you can still reproduce it.

[JessefSpecialisterren]

@Shelex Thank you so much for your bug-fixing effort on this one! Unfortunately, while test cases are no longer incorrectly marked as failed, the fix appears to have introduced a new issue where a failed test is incorrectly marked as skipped. I opened a new issue for it under #158