Output JUnit XML test metadata for CircleCI

[sophiebits]

@benbraou has claimed this issue. Please give them a chance to work on this.

---

Here's a screenshot of a CircleCI build for another (private) project I'm a part of:

It clearly highlights the parts of the build that failed.

In contrast, see one of our builds: https://circleci.com/gh/facebook/react/8402. In this one the ./scripts/circleci/upload_build.sh step failed but you need to scroll through the output a significant amount to see that. This is in part because we have a custom entry point for all our CI steps (scripts/circleci/test_entry_point.sh) to make it easier for us to parallelize some build steps.

I'd like us to change that test_entry_point.sh script to output JUnit XML metadata for CircleCI as described in https://circleci.com/docs/2.0/collect-test-data/. Then I believe the CircleCI build results page will show exactly which step failed.

(Bonus: If we can show individual Jest tests that would be even cooler. But I think the most important part is breaking out jest vs flow vs prettier, etc. into separate chunks.)

[benbraou]

I would like to work on this issue! (this will be my first contribution)

[sophiebits]

@benbraou, it's yours!

[benbraou]

I am working on this issue this weekend

[benbraou]

Overview of changes

I will be updating this comment as subtasks of this issue get implemented.

Subtasks

• [x] Handle Jest JUnit reports generation
• b88d010
• jest-junit reporter is used to output JUnit report reports in /junit/jest-results.xml
• An example of CircleCI Build insight based on the generated JUnit report can be found here
• test_entry_point.sh is calling yarn test several times. So, a unit test failure will be duplicated in the build summary
• [x] Handle ESLint JUnit reports generation
• 59ad2d5
• For every ESLint run, a temporary JUnit report file is created. Then the temporary generated files are merged synchronously (but can be enhanced if needed to asynchronously) into one file reports/junit/eslint-results.xml. Finally, the temporary files are removed synchronously.
• Introduced a new file formatterUtils that groups utilities around ESLint junit formatting.
• ESLint script can either use the default stylish formatter or junit formatter.
• An example of react CircleCI Build insight based on the generated JUnit report can be found here. It shows 3 problems. :warning: The report is currently broken into multiple tests. An assessment of whether this is the better approach(vs single test will be done)
• [x] Handle flow JUnit reports generation
• 6c95a0d
• Flow result is treated as a single test that has a single test output
• Added junitReporting.js which exports the method writeReportAsSingleTest allowing to writing single test JUnit repot
• An example of react CircleCI Build insight based on the generated JUnit report can be found here.
• [x] Handle prettier JUnit reports generation
• 90b58cb
• An example of react CircleCI Build insight error when prettier returns warnings is here.
• [x] Handle check_license.sh JUnit report generation
• 613e0ae
• refactoring + new shell script write_junit_report.sh to create simple report
• An example of react CircleCI Build insight error when PATENT keyword is used here.
• [x] Handle check_modules.sh JUnit report generation
• af523fe
• An example of react CircleCI Build insight error whenprovidesModule keyword is used here.
• [x] test_print_warnings.sh
• Nothing to be done as this script is simply used to extract all messages from `warning()
• [x] Handle track_stats JUnit.sh report generation
• 9584236
• Also includes some refactoring (JUnit mode is now an environment variable, one entry point to write JUnit reports)
• [x] Handle build.sh JUnit.sh report generation
• [x] Handle upload_build.sh JUnit report generation
• 9f77625
• An example of react CircleCI Build insight error whencurl the upload fails can be found here.
• [x] Squashing commits + fix upload failure
• eec8248
• [x] Added JUnit report generation for coveralls
• 399d67b

[benbraou]

@sophiebits I would like to check with you my proposal for Flow JUnit report generation before I start the implementation.

Flow only supports JSON output. So what I propose when flow script is run is:

1. In normal dev mode, the normal output is logged in console
2. In case of circleci build, flow is run with --json option. The json output is provided to a transformer (that I will implement) that will generate the JUnit output.

That transformer could also be used to generate reports based on the return of the other build scripts ( upload_build.sh, test_print_warnings ...)

What do you think ?

[sophiebits]

I think for flow it is fine to treat it as a single "test" that has a single output and just use the formatted output. Not sure if there is much advantage to try to split flow into multiple "tests".

[benbraou]

I see. Thank you!

[benbraou]

A pull request has been submitted #12023

Summary

Currently, when the build fails, there is no way to know what happened other than going through the logs in CircleCI.

The goal of this PR is to output JUnit XML metadata for every build step. This allows CircleCI summary page to display exactly which step has failed.

What this PR does:

1. Handle Jest JUnit reports generation jest-junit reporter is used to output JUnit report reports in /junit/jest-results.xml :warning: test_entry_point.sh is calling yarn test several times. So, a unit test failure will be duplicated in the build summary
2. Handle ESLint JUnit reports generation ESLint script can either use the default stylish formatter or junit formatter. :warning: in JUnit mode, ESlint results are in XML format. Not only are they written to report fils, but also they are logged to console. This may be a noise to the build logs.

:warning: For the tasks described below, the package junit-report-builder has been used. Initially, I was writing the JUnit report on my own.

1. Handle flow JUnit reports generation Flow result is treated as a single test that has a single test output
2. Handle prettier JUnit reports generation
3. Handle check_license.sh JUnit report generation
4. Handle check_modules.sh JUnit report generation
5. test_print_warnings.sh Nothing to be done as this script is simply used to extract all messages from `warning()
6. Handle track_stats JUnit.sh report generation
7. Handle build.sh JUnit.sh report generation
8. Handle upload_build.sh JUnit report generation
9. Handle JUnit report generation for coveralls
10. Handle JUnit report generation for Danger

[pree-T]

May I work on it? Can someone help me?

[MananTaneja]

@bvaughn @sophiebits is it okay if I pick this up? Just starting out (first open source issue)

[koyuncuoglum95]

Hello @sophiebits , I'm here first time. Can I work on this ?

[Ank61]

Hello @sophiebits! Can I work on this?

[simi360]

Hi @sophiebits please let me know if the issue still exists? I would like to get working

[kassens]

We're moving off of CircleCI, so this no longer applies. Maybe there's something similar on GitHub Actions that would make sense, I don't know.