elm-test
elm-test copied to clipboard
elm-community
Stop fuzz test after n distinct shrunken failures
I'm currently writing some fuzz tests for converting some relatively complex data to JSON and back.
When the tests find a problem then they take 1-2 minutes and spew over 1000 lines to the console. But I only needed to look at the last failure case to find and fix the one place that was causing the errors.
I think a nice feature would be the ability to specify an amount of failures after which the tests stop and just display those failing cases, either on the command line or as a part of FuzzOptions.
This is reasonable. I might look into it at some point in the near future if someone else hasn't already.
Clarifying question: are these 1000 lines all from one failing test, or from many? If the former, this sounds like a shrinking bug (which I may or may not be able to fix); we can add an option to stop after n unique shrinks. If the latter, sounds like a general test option to stop after n failed tests.
They were from one failing fuzz test, which included many test cases. The 1000 number was based on at least 10 lines per error and (last time I checked) 100 fuzz test cases per fuzz test.
Okay, seems reasonable. Might be worth deciding whether we stop testing, or just truncate the display when writing to the terminal. (If we ever have JSON output, might as well include them all and let the UI handle it.)
@mgold, tests failing can be slow. It's best to decide a cutoff point and deal with it there. E.g 3 failures then fail the test overall and stop running any more. It's not just a UI thing - it's important for CI tests to be as fast as possible
Makes sense, although that seems like it will cause problems when we go to parallelize tests.
I don't think it will cause problems then. We can just kill off any in-progress workers as soon as we hit the cutoff.
Might also be worth having timeouts around fuzzing and shrinking - like if a fuzzer is taking too long to generate values, then time out and fail the test with "this fuzzer is taking too long; you should revise it" and if shrinking takes too long, bail out and show the shrunken values we've hit so far.
I recently managed to run Node out of memory trying to shrink a list string case. 😄
<--- Last few GCs --->
127603 ms: Scavenge 1396.9 (1457.0) -> 1396.9 (1457.0) MB, 55.4 / 0 ms (+ 2.0 ms in 1 steps since last GC) [allocation failure] [incremental marking delaying mark-sweep].
128722 ms: Mark-sweep 1396.9 (1457.0) -> 1396.8 (1457.0) MB, 1119.0 / 0 ms (+ 3.1 ms in 2 steps since start of marking, biggest step 2.0 ms) [last resort gc].
129819 ms: Mark-sweep 1396.8 (1457.0) -> 1396.7 (1457.0) MB, 1097.1 / 0 ms [last resort gc].
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x3a4c38137399 <JS Object>
1: func [0x3a4c38104131 <undefined>:~11237] [pc=0x55a21421cca] (this=0x1576c136aee1 <JS Function wrapper (SharedFunctionInfo 0x3a808a440c61)>,list1=0x21331abe4ba1 <an Object with map 0x15b477ceaa21>,list2=0x45811cdee19 <an Object with map 0x15b477ceaa21>)
2: A2(aka A2) [0x3a4c38104131 <undefined>:~89] [pc=0x55a213f2f96] (this=0x3a4c38104131 <undefined>,fun=0x1576c136aee1 <JS Function wr...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
[1] 86601 abort elm-test FailingTests.elm