do we have any way to combine multiple test runs report into one report (when running test in parallel)

[govindn]

What are you trying to achieve? Suppose I am running three test case in parallel mode(i.e. codeceptjs run-multiple parallel) so three report gets generated each have details on one test case each,

But I want one report which has information of all three test run just like test run in normal mode(i.e without parallel mode).

What do you get instead? three separate reports generated with details of one test case in each report

[SeriousM]

merge reports is the magic word. it's not supported as far as I know

[adamgruber]

@SeriousM is correct. This is something that's been on my radar for a while and held up mostly by lack of time to work on it.

[SeriousM]

It would be awesome to have it because cypress.io is now generating a single report for every spec (which are a lot) and how I end up with mochawesome_001.html to mochawesome_130.html

This is really bad and cypress.io team is not gonna fix it. What data in the json file should be joined to generate a single report?

[adamgruber]

I'm not really familiar with cypress.io and how it runs the tests. Seems strange that it generates one report for each spec. Is it running each in a separate process? If so I wonder if there's even a way at the reporter level to aggregate everything at the end. You may still need to post-process all the reports through the mochawesome-report-generator CLI.

Each report JSON should have a top level suite. These would need to be combined into one. But the overall stats would be off.

[SeriousM]

Each report JSON should have a top level suite. These would need to be combined into one. But the overall stats would be off.

Can I just omit the overall stats then and just do something like this:

{
  "suites": {
    "suites": [
      ...<all tests here>...
    ]
  }
}

[SeriousM]

ok, I can combine the json files as described. of course my stats are off (maybe I can calculate them later) but the report is ok.

mochawesome-report.zip

I noticed that missing json properties create validation errors. That's ok, but the latest version of mochawesome creates an incompatible json: Invalid value undefined supplied to /suites/suites/0/tests/0/timedOut: Boolean. Adding it let marge work again.

Btw: is there a documentation what all the json properties mean? "beforeHooks", "afterHooks", "pending", etc... when is what filled with and do I have to care?

[adamgruber]

Most of the JSON properties come directly from mocha.

the latest version of mochawesome creates an incompatible json

Is this with cypress.io? Can you provide a repo to reproduce this? I put the validation in place to avoid bad data when generating the report but it has caused some headaches. I may need to revisit how the JSON is validated and consider loosening the requirements.

[SeriousM]

Sure, here is the setup I'm using. I guess the content of the package.json is the important aspect: https://gitlab.com/seriousm/CypressDockerTester

https://gitlab.com/seriousm/CypressDockerTester/blob/master/cypress-package.json

[fsmaia]

Just inserting a suggestion here too: we could implement a [file] wildcard in reporterFilename to be replaced with current spec, then we would have predictable filenames at least.

I think the implementation is easy: just insert a regex replace from [file] to current report data filename in .getFilename(). We would need to send reportData parameter from .prepare(), to .getOptions() and then .getFilename().

If someone thinks this can be helpful, I can give it a try and open a PR if it works.

[mjhea0]

I have mochawesome working here -> https://github.com/testdrivenio/cypress-mochawesome-s3

Essentially-

1. cypress runs
2. each spec generates a new report
3. a custom script combines each mochawesome json file and calculates the stats
4. html report is then generated

[izydorekjakub]

I noticed that missing json properties create validation errors. That's ok, but the latest version of mochawesome creates an incompatible json: Invalid value undefined supplied to /suites/suites/0/tests/0/timedOut: Boolean. Adding it let marge work again.

@SeriousM Have you managed to solve this problem?

[izydorekjakub]

I noticed that missing json properties create validation errors. That's ok, but the latest version of mochawesome creates an incompatible json: Invalid value undefined supplied to /suites/suites/0/tests/0/timedOut: Boolean. Adding it let marge work again.

@SeriousM Have you managed to solve this problem?

OK i got it. @mjhea0 do the job.

[mjhea0]

what job? @izydorekjakub

[izydorekjakub]

I have mochawesome working here -> https://github.com/testdrivenio/cypress-mochawesome-s3

Essentially-

1. cypress runs
2. each spec generates a new report
3. a custom script combines each mochawesome json file and calculates the stats
4. html report is then generated

It solves all my problems @mjhea0

[stephenchan9]

I have mochawesome working here -> https://github.com/testdrivenio/cypress-mochawesome-s3

Essentially-

1. cypress runs
2. each spec generates a new report
3. a custom script combines each mochawesome json file and calculates the stats
4. html report is then generated

I am getting this issue when running tests that are in a "pending" state. Any possible fix for this? Great job on the code so far.

Invalid value "pending" supplied to /suites/suites/0/tests/0/state: TestState

[milankrstin]

@SeriousM

I noticed that missing json properties create validation errors. That's ok, but the latest version of mochawesome creates an incompatible json: Invalid value undefined supplied to /suites/suites/0/tests/0/timedOut: Boolean. Adding it let marge work again.

When I fixed this and ran marge command I always got 2 separate files. For example: I have report1.json and report2.json When I run: marge report1.json report2.json Result will be: report1.html and report2.html. Merge is not executed. Did you have the same issue?

[SeriousM]

Currently we just generate one report per fixture. Sorry that I can't help you any further but we decided to stop investigate this issue any longer as it costs more than it gives us.

[mjhea0]

@stephenchan9 - Yes, this has been fixed.

[sandeepbantode]

@govindn Do you found solution for original issue ? merge reports when run in parallel

[0xc0d3r]

@adamgruber Any update on the original issue?