Stop fuzzer after first failure

[drathier]

Fuzzers seem to continue running the testcase with new input even after the first failure. This makes print-based debugging hard, and wastes resources.

[rtfeldman]

This makes sense, but is nontrivial to implement because the individual test runs get flattened before execution. By the time the failure happens, you don't know which Test the failure applies to, so it's hard to abort the others.

They do know what their (fully nested) descriptions are, though, so this would presumably work:

1. Build a Dict of failure messages
2. Have each run consult the Dict to see if its description is present
3. If it's present, then don't bother running
4. If it's missing, then run - and add its description to the Dict if the run fails

This overhead probably wouldn't be noticeable because the Dict would be empty until the first failure.

[drathier]

Oh, are fuzz tests expanded to 100 unit tests before running?

[mgold]

the individual test runs get flattened before execution

I'm not sure this is true, otherwise the #127 bug wouldn't have happened, right? We do some deduplicating of results and it might even be simpler to stop after one failure. Unless seeing multiple failures is instructive?

The other problem with print-based debugging is shrinking. Pure functions mean we can call them as many times as necessary within a single test and no one should be able to tell. Debug.log breaks that.

[drathier]

Right, shrinking. Python Hypothesis has a note function which works like print only after the shrinking is done; otherwise it's a no-op. Maybe we could supply our own Debug.log function e.g. through fuzzPrint that gives one extra argument to the lambda? Supplying another argument is probably a fairly easy thing to do. If print-based debugging is something we want to be able to do, that is.

[mgold]

If print-based debugging is something we want to be able to do, that is.

I suspect it is not. If you have to use print statements to debug your tests, your tests are too big and complicated. And you certainly shouldn't pass fuzzPrint to the code under test.