save-cli icon indicating copy to clipboard operation
save-cli copied to clipboard

Support a specific return code (or a range thereof) as a test failure indicator

Open 0x6675636b796f75676974687562 opened this issue 2 years ago • 0 comments

Currently, save-cli can only fail a test if there're unmatched warnings (ExecutionResult.code is preserved but never read).

Yet, we currently cannot differentiate between:

  • the case when the number of warnings is zero (as expected), and
  • the case when the tool under test hasn't even run.

For instance, KtLint may return 0 on success and 1 on failure (if warnings were found), but the return code willl be 127 if Java is not installed (POSIX), and we can't reliably detect that yet.

We need smth like

[general]
execCmd = "java -jar ..."
failIfExitCodeMatches = "2,4,127-255"
suiteName = "..."
description = "..."
tags = [ "..." ]

The above will, particularly, allow us to easily diagnose problems like:

  • 127: command not found,
  • 128+n: program was killed with signal <n>.

More details: Exit Codes With Special Meanings.