[allure-codeceptjs][3.2.0]: Reports with test cases that failed on Before Hook are generated incorrectly

[Ommovich]

Describe the bug Test cases, that fails on Before Hook are not added to the report, if there is no test cases running after failed suite.

If there is at least one more test case after that which failed on Before Hook, failed case will be added to the report.

This strange behavior is especially noticeable if two test cases fails the Before hook in a row. In that case only first test case are added to the report

If there three test cases that failed on Before Hook, then only first two of them will be added to report. If four - then three of them will be in report etc.

To Reproduce

1. You need to create two Suites, both with broken Before hook. It doesn't matter in one file or in several.

import expect from 'expect';

Feature('First suite').tag('@debug');

Before(() => {
  expect(1 + 1).toBe(3);
});

Scenario('First scenario', () => {
  expect(1 + 1).toBe(2);
});

Feature('Second suite').tag('@debug');

Before(() => {
  expect(1 + 1).toBe(3);
});

Scenario('Second scenario', () => {
  expect(1 + 1).toBe(2);
});

2. Run all suites. In our case we use --grep to run all "debug" tests yarn report:remove ; codeceptjs run --tests --grep '^(?!.*@ignore)(?=.*@debug)' --debug --debug-brk --verbose
3. After run ended serve Allure report In result you will see only first failed test suite with first test case.

Expected behavior All test cases that failed on Before hook should be added to report.

Screenshots Test execution log. Please note that an error from the reporter appears after the start of the second suite

Allure report with only one of failed suites.

Desktop (please complete the following information):

• OS: macOS 13.3.1 arm64 Apple M1 Pro
• Browser: chrome 134.0.6998.166
• nodeInfo: 20.10.0
• [email protected]

Additional info There is a suggestion that this is happening because some Mocha events are not being emitted if test fails on Before hook.

[eripley]

Hey guys! We’re really struggling with this issue, and it’s causing us a lot of pain. We have no idea how to fix it.

We believe the problem originates from the allure-js-commons package (since the reporter modules are implemented there), but the source code is quite unclear, and despite our attempts, we haven’t been able to resolve the issue on our own.

We’d really appreciate your help!

[eripley]

For any additional data please call and we will provide it asap.

[baev]

Thanks for reporting this issue — we’ll take a look!

That said, given the way fixtures are implemented in CodeceptJS, it might be tricky to address this immediately. We're currently focused on revisiting the Pytest and PHP integrations, so we'll be circling back to JavaScript-related issues in a couple of weeks. Stay tuned!