goss
goss copied to clipboard
Separate exit code for test parsing errors?
Thanks for all the hard work on this utility, I've found it really easy to use.
I use a simple bash script to run tests depending on different facter facts on hosts. However, I'd like to be able to tell failing tests from synctactically incorrect tests, e.g.:
#============ The test which should fail:
lan1:~ # cat goss_incorrect.yaml
port:
tcp:22:
listening: false
lan1:~ # goss --gossfile goss_incorrect.yaml validate -f tap
Error: Could not parse resource Port:tcp:22
lan1:~ # echo $?
1
#============ The incorrect test:
lan1:~ # cat goss_failing.yaml
port:
tcp:22:
listening: false
lan1:~ # goss --gossfile goss_failing.yaml validate -f tap
1..1
not ok 1 - Port: tcp:22: listening: doesn't match, expect: [false] found: [true]
lan1:~ # echo $?
1
I believe having a separate exit code for YAML parsing/template rendering would make scripting easier.
I was looking for documentation/code that would explain the different possible error values... and this proves that's no specific policy.
I would love to have different error codes to process whether there was gossfile errors or validation errors. (wish I had any Go skill to suggest a PR)
Agree, this would be useful for scripting.
I was looking for documentation/code that would explain the different possible error values... and this proves that's no specific policy.
It is. E.g., FreeBSD documents it in sysexits man page, Linux has /usr/include/sysexits.h
too. In Go there is https://godoc.org/sny.no/os/sysexit.
So, for incorrect yaml the proper exit code would probably be EX_CONFIG
while failed test could still be 1.
I'm also affected by this. I maintain the degoss Ansible role (which installs, runs, and removes Goss and test cases on a machine), and I have to write some very interesting logic to catch these kinds of errors. Goss returns 0
as a return code when it fails to load or parse the Goss file.
My workaround is to set --format
to json
and if I'm unable to deserialize JSON, I assume that there's an error, regardless of return code.
I would say that this is also importnat for check over HTTP? As I can see goss returns proper result (HTTP BODY) with errors - but then returns 503 as HTTP return code. So is really hard to know if connection with goss failed or whether checks failed:
http http://goss.zone/healthz
HTTP/1.1 503 Service Unavailable # !!!! <- 503 - which is not correct - service IS working, checks fails.
Content-Length: 1289
Content-Type: text/plain; charset=utf-8
Date: Thu, 08 Aug 2019 10:31:29 GMT
GOSS CRITICAL - Count: 60, Failed: 9, Skipped: 0, Duration: 0.219s