hurl icon indicating copy to clipboard operation
hurl copied to clipboard

Strange behavior with exit code when using --fail-at-end option

Open lepapareil opened this issue 1 year ago • 0 comments

Summary

When using --fail-at-end, Hurl's exit code is always the one with the highest priority from the established failure exit codes list.

Steps to reproduce

  • create test-fail-at-end-exit-codes.hurl file with queries and assert, returning 3 distinct exit codes:
GET http://127.0.0.1:1234/does-not-exist # exit code 3

GET https://hurl.dev
HTTP 200 # exit code 0

GET https://hurl.dev
HTTP 302 # exit code 4
  • execute test-fail-at-end-exit-codes.hurl file
$ hurl --connect-timeout 1 --fail-at-end test-fail-at-end-exit-codes.hurl
error: HTTP connection
  --> test-fail-at-end-exit-codes.hurl:1:5
   |
 1 | GET http://127.0.0.1:1234/does-not-exist # exit code 3
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (7) Failed to connect to 127.0.0.1 port 1234 after 0 ms: Connection refused
   |

error: Assert status code
  --> test-fail-at-end-exit-codes.hurl:7:6
   |
 7 | HTTP 302 # exit code 4
   |      ^^^ actual value is <200>
   |

$ echo $?
3

Even if we invert the line that triggers exit code 3 with the line that triggers exit code 4, we get the same result, which is the exit code 3.

What is the expected correct behavior?

I don't want Hurl to exit with the last error code or with a pre-prioritized exit code, I expect to have the exact exit code of all failures, here we have 2 and 4 then:

$ echo $?
6

I expect Hurl to manage more exit codes, for example:

Value Description
0 success
1 Failed to parse command-line options.
2 Input File Parsing Error.
3 Runtime error (such as failure to connect to host).
4 Assert Error.
5 (2+3) Input File Parsing Error + Runtime error (such as failure to connect to host).
6 (2+4) Input File Parsing Error + Assert Error.
7 (3+4) Runtime error (such as failure to connect to host) + Assert Error.
9 (2+3+4) Input File Parsing Error + Runtime error (such as failure to connect to host) + Assert Error.

Execution context

  • Hurl Version: 2.0.1
  • Operating system and version: Ubuntu 20.04

lepapareil avatar Apr 14 '23 09:04 lepapareil