jflex
jflex copied to clipboard
jflex-de
Property-based unit tests
The current unit testing in JFlex is a bit thin, mostly because the cost/benefit ratio of writing those single test cases is fairly low.
We should use property-based unit tests, at least for the different set implementations in JFlex, as well as partitions and other mathematical objects that have nicely statable properties. These would automate the test object construction, and let us specify the contracts for those objects in a way that gains some actual assurance about implementation correctness.
juni-quickcheck (https://github.com/pholser/junit-quickcheck) looks like it would be usable for the job.
I'll have a look at how much work it would be to set up. @regisd, @sarowe do you have any opinions on this?
I discovered this kind of testing when I learned scala, and liked it, but never used it in real life.
I'm a bit uncertain about the randomness of things ; as in can this make tests flaky?
In theory the randomness could lead to a test that does not always fail, in practice that happens rarely, although it does happen. How often it happens depends on the number of tests it runs for each checked property (default is 100), and how well the input space is covered in that number of tests. So far, if something was wrong, it failed pretty much always, but it's not guaranteed.
When a test fails, it shows the inputs and random seeds for which it failed, so it can be reproduced. It's usually a good idea to then turn that failing instance into a unit test for reliably checking regressions. E.g. for IntCharSet, you'd take the parameters it shows, construct them manually in the old IntCharSetTest unit test, but call the IntCharSetProperties method that failed with it, i.e. you don't have to duplicate the actual test code. This will unit test will now always fail until the problem is fixed.
It seems to be working fine, and it's at least for me way easier to find good tests to write, so I'll continue working on this a bit and add tests for state sets and char classes.
#687 bring this to reasonably comprehensive tests for IntCharSet, StateSet, and CharClasses. I'll see if it make sense to do something on RegExp next, and if that is successful, on NFA and DFA.