Provide a way to check the mean of distribution of test cases' labels

[ghost]

cover, coverTable and checkCoverage provide ways to target some specific distribution of test cases, and fail the test if this target is not reached within some confidence interval. I would like to write a Property that expresses some requirement about the mean of the distribution of test cases. This property provides some context on what I am trying to achieve. In this particular case I would like to test the mean of retryCount property of the test cases to be 2.

[MaximilianAlgehed]

This sounds like a reasonable thing to want indeed. However, before we can discuss details of combinator design etc. I would be really grateful for a tiny example right here that makes it crystal clear what you want to achieve.

[ghost]

Definitely! As mentioned in the issue this need arose while working on some cryptographic protocol for building "small" certificates through sampling of an initial "large" set. Here is some code sketching what I would like to be able to write, glossing over the details of what type of numbers are to be used:

prop_meanRetryForLargeSetIs1 :: Property
prop_meanRetryForLargeSetIs1 = 
   forAll arbitrary $ \ items -> 
       let (proof, retries) = prove items
       in verify proof &
             label "retries" retries &
             mean "retries" 1 &
             checkCoverage 

The idea is that I should be able to attach some kind of label to test cases, then use mean to ensure QC computes the mean of the relevant labels and warns if the value is not equal to some parameter, with some error margin. checkCoverage could be used to make the test fail if the target is not reached, instead of a warning, like how cover works.

Is this helpful?

[MaximilianAlgehed]

Perfect, that's exactly what I was looking for.

The issue I foresee here is that if we want to fail a test the way we do with insufficient coverage we need to know what kind of distribution we are dealing with. In the case of boolean yes-no coverage that's easy - it's always a binomial distribution. That fact makes it possible to answer the question "is the coverage higher than X% with Y% confidence?" (Y defaults to something very close to 100% to make sure that things aren't flaky). With means this becomes much more tricky. Firstly, I don't think it would be sufficient to specify "I want the mean to be X" but rather "I want the mean to be X +- Z" and then to answer the question "is this true with a confidence of at least Y%" one would probably need to know more about the distribution (or assumed distribution?) of test cases.

[ghost]

Defining the expected distribution makes sense, of course. Perhaps that's something relatively easy to provide as an argument to the mean function? Or even it could default to normal distribution. Comparing an empirical distribution with a theoretical one seems pretty straightforward.