Venturecxx
Venturecxx copied to clipboard
Add mechanism to stats.py for testing the conjunction of multiple statistical statements
For example, reportKnownGaussian could use the K-S test to check the shape of the distribution, and also the Z test to check its mean. This might be desirable due to having greater power per sample to detect bugs that produce rare outliers, like #489 . (Which wasn't a Gaussian posterior, but never mind.)
The natural approach would be to define a combinator on TestResult objects that took the minimum p-value and appropriately combined the reports.
- In order to maintain honest assessments of statistical significance, the p-value would need to be increased from the minimum. Under the assumption of independence, the correction is not hard to derive: The probability that the min of k independent random variables distributed uniformly in (0,1) is no bigger than x is well-defined, and even fast to compute.
- Applying this correction would have the effect of (slightly?) reducing the power of any individual conjunct, but that's to be expected.
- I am somewhat worried whether this combinator will be associative. Should that concern give pause to its adoption?