coreblocks icon indicating copy to clipboard operation
coreblocks copied to clipboard

Run tests in loop

Open lekcyjna123 opened this issue 1 year ago • 2 comments

Currently each unit test is parameterized using two values important from running test case: seed and test_number. First set seed for random generator and second said us how many iteration we should do for each seed. When we want to increase coverage we have two ways:

  • test with different seed's
  • test with more iterations Both ways are orthogonal, the first is better if we want to find shorter sequence of iterations which gave us the same result as some other longer tests, and the second is better when we want to check how our core behaves in long run.

As for now if someone want to change seed he has to either edit manually test case or wrap it with parameterized_class to test more seed's in one run_tests.py command. It would be nice to have a possibility to specify in run_tests.py number of loops to execute. Then this script should automatically set for each unit test run a different seed and test with it.

If implemented correctly, this will have an another pros, that we won't have to remember about setting seed manually and this will remove some boilerplate code from tests.

lekcyjna123 avatar Jul 17 '23 17:07 lekcyjna123