Reconsider failure predicate defaults and/or consider shortcuts for setting them

[eric846]

Should Nighthawk exit by default after a single 5xx? We were suprised to discover an benchmark had been exiting after less than the specified time because of errors, and it was hard to know what was happening. We found that a failure mode of our system under test is elevated 5xx rates, and it would be good to measure them.

The only reason I know of for exiting on the first error (which I think is the default failure predicate, but I could be wrong) is that if you misconfigure Nighthawk, you don't have to wait till the end of the requested duration to find out.

Spinning off offline discussion with dubious90:

• Potentially adding a shorthand for this kind of thing?
• Maybe something like "default_failure_predicate"?

[oschaaf]

Context: The reasoning behind default termination on the first error was to avoid a “whoops I killed my server” by accident. And to force attention to potentially abnormal results in case error codes and connection failures aren’t anticipated. And indeed also UX on misconfiguration for the CLI. The client should exit with an exit code to draw attention the abnormal termination. I agree shorthands to override this would be pretty useful.