Orange-OpenSource
Don't write everything to stderr which breaks CI pipelines and scripts
On the console the tests are completed successfully and I get a nice concise output as you see above. For whatever reason you are writing this output to stderr instead of stdout as you can validate here, see below:
This breaks the CI pipelines and makes your tool a burden in every CI pipeline and in scripting.
Hi @loxal
makes your tool a burden in every CI pipeline and in scripting.
If this is a burden, I've hard time understanding why you're using it. There are plenty of alternatives, consider this tool as a free tool given freely to the community so there is no need to be harsh.
That's said, Hurl is an HTTP client, one of the primary goal of the tool is to give you a HTTP response on standard output.
For instance, you can test this file:
GET https://foo.bin
HTTP 200
GET https://bar.bin
[Options]
output: -
HTTP 200
And get the response of https://bar.bin on standard output.
We've followed these guidelines:
- standard output : returned HTTP response
- standard error: error messages, progress, summary
- exit code: 0 if success, != 0 failure.
Standard output and error can use ANSI escape code (colored text) but usually color is disabled if standard stream are piped (you can use --color / --no-color). I don't explain why the Hurl messages are in red in your CI logs, maybe you've an explanation.
So, on CI, you can use Hurl exit code to know if the tests have succeeded or not. You can use HTML/JUnit/JSON/TAP reports (sometimes CI have facilities to display these kind or reports) for reporting also.
This was just a specific proposal to make it easier for your tool to be adopted. I am using and have to use many tools that are much more of a burden than HURL but I won't bother to report issues for those tools. With HURL I had the hope that this would be received as a constructive specific feedback.
I am not talking about the exit code here but that all output goes to stderr which reserved for errors as the name implies and not for --verbose or regular output. I've fixed TeamCity CI via formatStderrAsError = false but this makes my test pipeline tolerant towards errors from tools that report only errors to stderr.
After formatStderrAsError = false the errors are turned into warnings and the CI pipeline succeeds:
...but that's a bad workaround, probably worse than using unsafe {} where it's not necessary.
The tool is great in many other aspects and that's why I am using it. 👍
Ok noted we'll see what we can do to improve your use case with TeamCity CI.