jlink
nestedlists challenge
Why is nestedlists a reasonable problem (for a uniform shrinking behavior across PBT tools) let alone a challenge?
(This is also related to #9.)
In my view it's not about uniformity as such but about:
What in the falsified sample produced by the tool gives me a hint about the root cause of the problem?
If the sample contains two lists instead of just one I may wonder if this is an important aspect of the problem. The same way I try to rid example tests of all noise that's not relevant for the scenario under test the ideal falsified sample contains only the relevant part.
Do you think it's not a challenge because it's too easy?
If the sample contains two lists instead of just one I may wonder if this is an important aspect of the problem.
Well, arguably this is an important aspect of this problem. The property which is being tested / falsified explicitly mentions sum of lengths, and addition is a binary operator in the minds of programmers, isn't it? (E.g. it's not at all clear to me whether returning [[0,0,0,0,0,0,0,0,0,0,0], []] rather than relying on the implicit +0 for the empty list, or some other combination of two lists is a better counter-example for this property.)
To see what I mean, consider changing the property from sum of lengths to product of lengths. What would you have expected to get in this case? Also [[0,0,0,0,0,0,0,0,0,0,0]] ?
Anyway, the main problem with this challenge is that it's actually not at all clear (to me, at least) what is a good enough (let alone best) hint about the root cause of the problem.
Additionally, the challenge seems completely artificial (construct a list of lists containing 0s).
What the discussion shows for me:
- In the general case there is no such thing as "the smallest falsifiable sample". We've known that all along but it's good to be reminded.
- Different tools handle "smallest" and "normalization" differently, which is a good thing. Maybe we should somehow link a challenge to a discussion of the challenge - linking to an issue on Github that's never being closed may be an option for that.
- This challenge is artificial and so are most small-scale textbook examples.
I'm happy with replacing this and other challenges. I'm less happy with just removing all artificial ones since I saw how they helped me to understand the problems and subtleties connected to shrinking and normalization. I'd rather people learn from not-so-good examples than do not learn at all.