quint icon indicating copy to clipboard operation
quint copied to clipboard

CLI should produce different error codes

Open konnov opened this issue 1 year ago • 2 comments

Currently, if a test in cli-tests.md returns a non-zero exit code, we do not know, whether it was the expected behavior or not. For instance, if we expect the simulator to find a violation, and the tool is passed incorrect CLI arguments, the test is considered as passed, though it should not.

We should differentiate exit codes that are caused by different exit paths. For instance, all CLI parsing should return one exit code and quint run and quint test should return another error code.

konnov avatar Jul 20 '23 08:07 konnov

Just wanted to express interest in this, it would be helpful to be able to tell whether an invariant violation was found or not just from the exit code not only for testing, but also for user-side tooling :)

p-offtermatt avatar Nov 22 '23 09:11 p-offtermatt

I've added it on my stack

konnov avatar Nov 22 '23 10:11 konnov