qcheck icon indicating copy to clipboard operation
qcheck copied to clipboard

Consider having corner cases by default

Open vch9 opened this issue 3 years ago • 1 comments

For instance QCheck2.Genui64 that would be useful to have by default:

graft_corners ui64 [0L; 1L; 2L; Int64.max_int] ()

As we do not have a proper fuzzer, these cases are important and we can not trust the generation to cover theses. However, one answer can be: "they need to unit test the corner cases, then write pbt"

vch9 avatar Mar 11 '22 14:03 vch9

I can certainly follow you. In a sense, the folklore of "the tester needs to be mindful of testing corner cases" translates to "the PBT tester needs to be mindful of generating corner cases". A personal pet-peeve here is https://github.com/jmid/qc-ptrees where changing the int generator to the following made a difference between finding a bug and not finding it: https://github.com/jmid/qc-ptrees/blob/f3c01664e76bc3944f48e155cb410db8317baf06/qctest.ml#L84-L89 I admit this situation is not particularly satisfactory though.

That said, I'm very hesitant about making such a change as it makes generators stateful by default. This means that simple generator transformations such as:

utop # let g = QCheck2.Gen.small_int;;
val g : int QCheck2.Gen.t = <abstr>
utop # QCheck2.Gen.(generate1 ~rand:(Random.State.make [|123|]) (pair g g));;
- : int * int = (6, 3)
utop # QCheck2.Gen.(generate1 ~rand:(Random.State.make [|123|]) (pair g g));;
- : int * int = (6, 3)

will no longer hold in the default case:

utop # let g = QCheck2.Gen.small_int_corners ();;
val g : int QCheck2.Gen.t = <abstr>
utop # QCheck2.Gen.(generate1 ~rand:(Random.State.make [|123|]) (pair g g));;
- : int * int = (1, 0)
utop # QCheck2.Gen.(generate1 ~rand:(Random.State.make [|123|]) (pair g g));;
- : int * int = (4611686018427387903, 2)

This will hurt QCheck/QCheck2's ability to reproduce pseudo-random runs, as behaviour now depends on both the random seed and the internal state of all involved generators.

A less invasive option is compose a non-stateful int-generator like above. Here's an example of less trivial one - which I learned from @jlouis some years ago. This has a greater chance of producing corner cases (and can be further composed with Gen.oneof):

(* sign * pow(2, k) + [-3;3] *)
let int_gen =
    (Gen.map3
       (fun sign expo addition ->
         let n = 1 lsl expo in
         let n = if sign then -n else n in
         n+addition)
       Gen.bool (Gen.int_range 0 63) (Gen.int_range (-3) 3));;

jmid avatar Mar 18 '22 10:03 jmid