minithesis icon indicating copy to clipboard operation
minithesis copied to clipboard

Add benchmarks to the shrinking challenge

Open jackfirth opened this issue 4 years ago • 3 comments

I was looking at https://github.com/jlink/shrinking-challenge and noticed it has benchmarks for Hypothesis. I think it would be handy to have benchmarks for Minithesis too, so it's easy to see how big of a difference the complexity of Hypothesis's shrinker makes when compared to the simpler (but easier to port) Minithesis shrinker.

jackfirth avatar Feb 26 '21 10:02 jackfirth

Yeah, absolutely. I'm keen for this to exist, but am a bit short on spoons at the moment so haven't been prioritising shrinking challenge work.

DRMacIver avatar Feb 26 '21 11:02 DRMacIver

Ideally it would be possible to make this work by just reusing the code from the existing Hypothesis shrinking challenges, but I'm not sure if that's easier than maintaining a separate version of each.

DRMacIver avatar Feb 26 '21 11:02 DRMacIver

Separate versions sounds simplest to me, but I have very little knowledge about how that repository is set up so I'm just shooting in the dark.

jackfirth avatar Feb 27 '21 05:02 jackfirth