Add an option to output a single report

[SeriousM]

Current behavior:

As described extensively in #1824, specs run in isolation and generate multiple reports. This is not a big issue for junit reports as they can be combined but for mochawesome html reports as you get N not combinable html files. In addition to this, the documentation is not updated and thus new users have a hard time to figure out why N specs are executed but only one report with the latest spec is generated.

Desired behavior:

Cypress should have an option to call the generator only once after all specs are executed. Reporters would work as expected again.

Versions

Cypress: 3.0.1

[mjhea0]

I have mochawesome working here -> https://github.com/testdrivenio/cypress-mochawesome-s3

Essentially-

1. cypress runs
2. each spec generates a new report
3. a custom script combines each mochawesome json file and calculates the stats
4. html report is then generated

Would you like me to incorporate this into cypress?

[SeriousM]

The script in question is https://github.com/testdrivenio/cypress-mochawesome-s3/blob/master/scripts/combine.js I would love to have it integrated into mochawesome! I just saw that you're already in the loop: https://github.com/adamgruber/mochawesome-report-generator/issues/97

[Karolis92]

Don't think that this should be a part of cypress or should be somehow universal for all reporters. Because anyone should be able to use it with any mocha reporter they want (of course after recent parallelization updates that's not possible with all reporters...)

[brian-mann]

We've talked about this as a team and its possible for us to modify Cypress to enable it to "work" when running on a single machine by capturing and silencing the events that happen when a new spec starts, and only forwarding the ones "in the middle" in between the start, end, and root suite events.

That would enable those running on a single machine to receive a single, discrete report from any mocha reporter.

For those running on parallelization, we can simply write an external service for that (since there's no other way) and parallelization is only available through our service anyway, so it would not create any kind of exclusivity.

The only bad news is that this isn't a super high priority - but we have/are in the middle of hiring and our bandwidth should be expanding relatively soon-ish.

[SeriousM]

That would enable those running on a single machine to receive a single, discrete report from any mocha reporter.

that sounds very good!

The only bad news is that this isn't a super high priority

please keep us updated about this issue, thank you!

[ryparker]

I would appreciate this for our reporting tool. We use cypress to send a report to our test case management tool testrails.io . One of our testrails suite could cover 100's of tests for one page of our webapp, however in cypress we have broken this suite into several spec files in order to not have one massive spec file. Our problem with cypress is when we run multiple spec files for one testrails suite. Currently the way we send a report is to watch for the mocha runner's end event at the end of every spec file run. The result is multiple testrails suite reports each with only a portion of test results for the testrails suit.

[izydorekjakub]

We also have exactly the same problem as @ryparker team. Please resolve this problem.

[Phillipe-Bojorquez]

I just ran into this issue myself. I'm attempting to implement the solution above but I may just make a script that combines the Json post suite run and deletes the files that were individually combined. Seems like the easiest option

[JesterXL]

@Phillipe-Bojorquez Lemme know what you come up with, I just did this myself for work for #1824 and will post what I have there.

[Antontelesh]

To partially solve this issue I have released mochawesome-merge.

I use it to merge all json reports together and then feed the output to mochawesome-report-generator.

[bahmutov]

Anton, mind making a pull request with a link to your tool to cypress-documentation plugins page?

Sent from my iPhone

On Dec 22, 2018, at 15:54, Anton Telesh [email protected] wrote:

To partially solve this issue I have released mochawesome-merge.

I use it to merge all json reports together and then feed the output to mochawesome-report-generator.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[Antontelesh]

Will do

[badgeek]

{
  "reporter": "json",
  "reporterOptions": {
    "mochaFile": "cypress/results/results-[hash].json"
  }
}

how to change json report filename?

[Antontelesh]

Anton, mind making a pull request with a link to your tool to cypress-documentation plugins page?

@bahmutov Done here

[badgeek]

Anton, mind making a pull request with a link to your tool to cypress-documentation plugins page?

@bahmutov Done here

can i use mochawesome-merge as reporter?

[badgeek]

Anton, mind making a pull request with a link to your tool to cypress-documentation plugins page?

@bahmutov Done here

can i use mochawesome-merge as reporter?

nevermind, found the instruction on mochawesome-merge

[EmmanuelDemey]

@Antontelesh Do we have the same kind of merge tool for jUnit reports ?

[alioguzhan]

@EmmanuelDemey and for others looking for merging multiple JUnit reports, this looks promising -> https://www.npmjs.com/package/junit-report-merger

[vicusbass]

How about TestRail integration with cypress-testrail-reporter? It creates a test run for each spec

[ajeetd]

We'd also like to see this at Tesults (https://www.tesults.com) - currently results reporting is problematic for Cypress specifically due to this limitation.

[LVCarnevalli]

More information in link: https://github.com/cypress-io/cypress/issues/3654

[mariamadamyan]

How about TestRail integration with cypress-testrail-reporter? It creates a test run for each spec Hi, in a case if you haven't found solution yet. FYI: https://www.npmjs.com/package/cypress-testrail-accumulative-reporter

[wasiqkhan786]

can we use the mochawesome for xml as well?

[poponuts]

Is there a way we can merge multiple junit-[hash].xml results? I saw the issue 1824 and it was closed and pointed to this open issue where mochawesome-merge is being used. That's fine for mochawesome but for bitbucket pipeline which uses ../test-reports/junit.xml to display the no. of tests on the pipeline (and not mochawesome), I need to merge multiple junit-[hash].xml file that it has created (since using it without [hash] just overwrites everything and passes the last run test). If someone can do a mochawesome-merge library similar to junit, that would be great.

[LucyQA94]

Hi. We use Cypress with mochawesome reporter, multireporters. We generate a jsons files and xml files and after ran tests we merge the jsons to a single json file and after we merge a html report, and xml files merge to a single file with all information of the tests. We need a feature in Cypress and mochaawesome reporter have an option to create a single report html and several xml (because if you put overwrite only show the last test in the report and i think is bad). We need no use jsons because context and screenshots doesnt appear and have several issues / is dificult to generates html with all the reports. Please, implement a solution to generate a single HTML, single JSON and single XML file with all the tests and the configuration of Cypress/mochawesome should different for each case. Configuration should individual to html / json / xml for enhance the settings in the framework.