save-cli
save-cli copied to clipboard
Support a specific return code (or a range thereof) as a test failure indicator
Currently, save-cli
can only fail a test if there're unmatched warnings (ExecutionResult.code
is preserved but never read).
Yet, we currently cannot differentiate between:
- the case when the number of warnings is zero (as expected), and
- the case when the tool under test hasn't even run.
For instance, KtLint may return 0 on success and 1 on failure (if warnings were found), but the return code willl be 127 if Java is not installed (POSIX), and we can't reliably detect that yet.
We need smth like
[general]
execCmd = "java -jar ..."
failIfExitCodeMatches = "2,4,127-255"
suiteName = "..."
description = "..."
tags = [ "..." ]
The above will, particularly, allow us to easily diagnose problems like:
- 127: command not found,
-
128+n: program was killed with signal
<n>
.
More details: Exit Codes With Special Meanings.