Option to return the status code of a failed command

[Scripter17]

For part of a thing I'm working on, I use Hyperfine to benchmark an HTTP server using Curl. If I mess up writing the tests the server can return a 413 "Content Too Large" error and, because Curl is being run with the --fail/-f flag, Curl returns an exit code of 22.

Afterwards the bash script I use for benchmarking checks if Hyperfine returned a non-zero exit code and print some details on what could've gone wrong and what to do to fix it.

It'd be nice if I could tell which exact exit code Curl returned and provide only the necessary solution.

[sharkdp]

You can use the --export-json option. Exit codes are present in the output.

[Scripter17]

I can't seem to get that to work. Even just hyperfine "exit 1" --export-json abc.json writes nothing. Having multiple commands where one errors results in that specific command not being written to the json. I am using v1.18.0, which as I write this is the most recent release.

I assume the issue is that the initial measurement (or the warmup if that's enabled) returns an error? Can't exactly write a meaningful mean time in that case.

To clarify, that is what happens when a test in my benchmarking script sends too big a payload to the server. It errors on the first try.

[sharkdp]

I can't seem to get that to work. Even just hyperfine "exit 1" --export-json abc.json writes nothing.

Oh, you also need:

  -i, --ignore-failure
          Ignore non-zero exit codes of the benchmarked programs.

As written in the error message:

▶ hyperfine "exit 17" --export-json abc.json Benchmark 1: exit 17 Error: Command terminated with non-zero exit code: 17. Use the '-i'/'--ignore-failure' option if you want to ignore this. Alternatively, use the '--show-output' option to debug what went wrong.

[Scripter17]

I can make that work since I'm already exporting the json and using jq to get a list of the mean times, but that would be very annoying for anyone who wants a portable script that doesn't depend on either.

[sharkdp]

I'm not really sure what you're asking for.

[Scripter17]

hyperfine "exit 4"
echo $? # 1 (current behavior)

hyperfine --return-error "exit 4"
echo $? # 4

[sharkdp]

I'm sorry, but this doesn't seem like a well designed feature. What if I run hyperfine --return-error "exit 3" "exit 4"? This can't really work. If you need detailed analysis of exit codes, please use --export-json and write a script to parse the output.