Number of tests (total, failed, etc.) vary wildly

[berndgoetz]

When running our tests, the reported numbers of test executions vary wildly. Here are some outcomes, based on the same code base, server and test client:

Tests run: 2622, Failures: 14, Errors: 0, Skipped: 8 Tests run: 2609, Failures: 4, Errors: 0, Skipped: 1 Tests run: 2302, Failures: 0, Errors: 404, Skipped: 1268 Tests run: 2623, Failures: 0, Errors: 0, Skipped: 0 Tests run: 2302, Failures: 0, Errors: 404, Skipped: 1268 Tests run: 2609, Failures: 4, Errors: 0, Skipped: 5

I haven't figured out the math here. Adding up the numbers don't match the overall number of tests. Any explanations? Any idea how to fix/improve this? Thanks.

[AlexeyBuzdin]

This appears to be a global issue with cucumber https://github.com/cucumber/cucumber-jvm/issues/577 and https://github.com/cucumber/cucumber-jvm/issues/263. Cucumber report is generated correctly though. Managed to create a workaround with TeamCity and Bamboo configurations. Will look into the issue further

[AlexeyBuzdin]

The surefire integration in Cucumber isn't that stable at all, so a possible solution is to create a stand-alone maven/gradle plugin and use cucumber.json as a source for reports. The .json itself is generated correctly and supported in Bamboo. The issue will be solved with https://github.com/ctco/cukes-rest/issues/45