ouroboros-network icon indicating copy to clipboard operation
ouroboros-network copied to clipboard

Unified adjustment of number of QuickCheck tests

Open amesgen opened this issue 2 years ago • 1 comments

In several places, we adapt the number of QuickCheck tests to run from the default (100), either to a higher value, e.g. for important combinatorial tests, or lower, e.g. because they otherwise take too long in CI.

The goal of this ticket is to make sure that all such occurences use tasty (via adjustOption + QuickCheckTests) to do so, instead of QuickCheck's withMaxSuccess, which is not adaptable via tasty's CLI flags.

amesgen avatar Jul 12 '22 12:07 amesgen

Some context: we used to have a way to run nightly tests, via BuildKite. But then IOG removed its BuildKite infra structure, so our Test.Util.Nightly stuff is no longer used.

  • We have a module module Test.Util.Nightly that defines a tasty "ingredient" for informing the individual tasty tests whether they're being run as normal or being run at night. This is what our BuildKite script used.

  • Looks like Networking has a nightly flag in ouroboros-network-testing/ouroboros-network-testing.cabal, so they approached that same question differently.

  • I could have sworn I saw that Clarke had another approach to Test.Util.Nightly that I had found after the fact, but I'm unable to find it now.

nfrisby avatar Jul 21 '22 15:07 nfrisby