dscheck icon indicating copy to clipboard operation
dscheck copied to clipboard

Add random tests generator

Open art-w opened this issue 1 year ago • 2 comments

An attempt at improving the test coverage:

  1. Randomly generate a little program that do atomic operations
  2. Run this program with a random scheduler to sample the possible values that the atomics can hold after execution
  3. Run it again with dscheck and check that it can find all of those outcomes (otherwise it means that dscheck wrongly skipped a branch)
  4. goto 1

Currently this is all a bit cheap! The program generator could produce more interesting logic. The random sampling is also super naïve since it favors "easy to find" outcomes, but that seems exhaustive enough at the moment (since dscheck rarely discovers more endings, the programs are too simple to hide anything).

I initially wrote this to test #3 but haven't found any bugs yet. I did confirm that it would have found the issues in the previous version of this PR :) What I did not expect is that it also produces broken programs on main, I thought this branch was super exhaustive and rock solid. I've extracted the buggy cases in a new testsuite if someone wants to take a look, it's a bit weird.

art-w avatar Feb 26 '23 21:02 art-w