mocha
mocha copied to clipboard
mochajs
🚀 Feature: Add option to exit with standard exit codes
Description
Mocha's exit code is the number of failed tests (up to 255). This is nonstandard and should be more like Unix/POSIX programs.
process.exitCode should be one of:
-
0for success -
1for failure (e.g., internal error, if any tests fail, no tests were run) -
2for failure due to invalid command line usage -
126if "bin/_mocha" not executable (handled by shell) -
128+nwherenis signal# (e.g.,SIGINT=2, so rc=130)
where:
-
0is EXIT_SUCCESS (POSIX) -
1is EXIT_FAILURE (POSIX) - rest are shell scripting maxims (e.g., sh, ksh, bash) (UNIX)
This should be explicitly added to the documentation.
Notes
Exit codes have a range from 0-255. An exit value greater than 255 returns an exit code modulo 256.
The 126 exit code can only happen on UNIX and would be provided by the shell itself.
Mocha's current scheme leads to ambiguous error codes.
For example, there is no means to distinguish between 130 test failures and interrupt by <Ctrl>-C.
It was mentioned that this would certain CI systems relied on this exit code abuse.
I fail to see how this could affect them; a test failure from above would still return 1, just not a count.
Related
#2445 #2438 (same problem, but fix only addressed clamping #failures)
the CI expects the count. I can’t recall which it is. there’s plenty of historical discussion around this.
I’d prefer to err on the side of not breaking stuff for existing consumers rather than causing headaches for potential consumers expecting POSIX codes, but that’s just me..
Couldn't find anything specific (outside references to this unknown CI).
Fail to understand the difference one way or another (CI-wise) to exiting with 1 vs. error count -- either way, an "error occurred" state should be triggered. By implication, the CI would do something different depending on the number of errors, which seems nonsensical.
One error is the same as a hundred -- it's a failure.
I traced this conceptually (exit code=#failures) all the way back to the initial commit (20110828). How and where Mocha invoked process.exit moved around from file to file, but found no references to CIs as the reason -- it was just always done this way.
I was ... surprised, to put it nicely, to learn that Mocha uses the number of failed tests for its status code. What a bizarre behavior. Fixing this to align to POSX conventions would be a breaking change, but a good one.
Renaming this issue to make it target adding an option for the new behavior, as written in #4989. We can file a separate followup issue to make the standard exit codes the default behavior in a later major version of Mocha.
Nevermind, lets get this in now. Would love some references for the:
1 for failure (e.g., internal error, if any tests fail, no tests were run) 2 for failure due to invalid command line usage 126 if "bin/_mocha" not executable (handled by shell) 128+n where n is signal# (e.g., SIGINT=2, so rc=130)
https://stackoverflow.com/questions/1101957/are-there-any-standard-exit-status-codes-in-linux http://tldp.org/LDP/abs/html/exitcodes.html (as linked previously in the initial description)
