When mocha runs in parallel mode, the failed test cases appear yellow instead of red

[jamesmortensen]

Describe the bug

When we run tests, any test which fails due to an AssertionError is reported in Allure in red, as a failing test. Any other thrown Error is reported as a broken test, in yellow.

While this functionality works as expected in sequential mode, failed tests are reported as broken tests (in yellow) when run in parallel.

To Reproduce Steps to reproduce the behavior:

1. Clone this repo: https://github.com/jamesmortensen/allure-js-reporter-issues#when-run-in-parallel-mode-with-hooks-hookerror-is-not-a-function and follow the instructions to run the tests and generate the report.

The example has us run tests in sequential mode as well as parallel mode. The README also has screenshots which show the issue.

Expected behavior

• For AssertionErrors, results should appear in the report in RED. In parallel mode, they're reported in YELLOW.

Screenshots

• Good (shows red): https://github.com/jamesmortensen/allure-js-reporter-issues/blob/master/screenshots/sequential-mode-report.png
• Bad (shows yellow): https://github.com/jamesmortensen/allure-js-reporter-issues/blob/master/screenshots/parallel-mode-report.png

Environment (please complete the following information):

| Test framework | [email protected] | [email protected] | Allure adaptor | [email protected] | [email protected] | Generate report using | [email protected] |

Additional context

Desktop (please complete the following information):

• OS: macOS 12.x
• Browser [Chrome, Yandex]
• Version [latest]

[MASQA]

@jamesmortensen , could you share structure your framework and runner? I`m interested you use data annotation or no, write some custom runner such as from here https://github.com/allure-framework/allure-js/issues/245#issuecomment-1176254785 or use this runner https://github.com/allure-framework/allure-js/blob/master/packages/allure-mocha/test/runner.ts ? I ask because as I see you actively use framework with parallel mode, but when I execute test runs, all my test run in sequentially. I dont understand where I am wrong I use so config for runner

const mocha = new Mocha({
  timeout: 16000,
  reporter: "mocha-multi-reporters",
  reporterOptions: {
    reporterEnabled: "allure-mocha",
    parallel: true,
    jobs: 4,
    allureMochaReporterOptions: {
      resultsDir: path.resolve(__dirname, "./../allure-results"),
    },
  },
});

I have tried some combinations of runs with different values jobs, with and without data annotation and always parallel mode is not work

[jamesmortensen]

Hi @MASQA sorry for taking so long to get back to you. Because for the longest time Allure wouldn't work with mocha in parallel mode using mocha's parallel feature, we used the script referenced in https://github.com/allure-framework/allure-js/issues/245#issuecomment-1176254785:

npx mocha parallel-launcher.js

This spawns several mocha processes to run in parallel but without using mocha's parallelism features. We still use this methodology in some tests we run daily.

The issue I'm reporting here is that the allure report shows tests in yellow instead of red with mocha's default parallelism feature:

npx mocha -p test/**.test.js

We don't use any annotations. It's a JavaScript project.

I'll also say we're not running mocha programmatically. Because our tests are written in ESM, Mocha can't load reporters because it tries to "require" them, but it needs to "import" them. Instead, the wrapper script deals with this by using the execa module to spawn mocha processes from the shell:

testFiles.map(testFile =>
            limit(async () => {
                const subProcess = execa('mocha', [testFile]);

In short, we achieve parallelism using the script referenced https://github.com/allure-framework/allure-js/issues/245#issuecomment-1176254785. Hope this helps!

[jamesmortensen]

I opened a pull request in mocha to fix the inconsistent reporting of failures in Allure. If they don't accept the change, then the next step is to hack the error type from the error stack trace, in the allure-mocha codebase, and add that to the deserialized error object. See the linked pull request for details. https://github.com/mochajs/mocha/pull/4995

[delatrie]

Allure-mocha now handles the parallel mode differently. We don't rely on how Mocha serializes its events anymore. Instead, we emit the results directly from the worker processes, which have access to error.name.

As of [email protected], the issue is fixed. Feel free to open a new issue if it doesn't work as expected.