Venturecxx icon indicating copy to clipboard operation
Venturecxx copied to clipboard

Add mechanism to stats.py for testing the conjunction of multiple statistical statements

Open axch opened this issue 9 years ago • 0 comments

For example, reportKnownGaussian could use the K-S test to check the shape of the distribution, and also the Z test to check its mean. This might be desirable due to having greater power per sample to detect bugs that produce rare outliers, like #489 . (Which wasn't a Gaussian posterior, but never mind.)

The natural approach would be to define a combinator on TestResult objects that took the minimum p-value and appropriately combined the reports.

  • In order to maintain honest assessments of statistical significance, the p-value would need to be increased from the minimum. Under the assumption of independence, the correction is not hard to derive: The probability that the min of k independent random variables distributed uniformly in (0,1) is no bigger than x is well-defined, and even fast to compute.
  • Applying this correction would have the effect of (slightly?) reducing the power of any individual conjunct, but that's to be expected.
  • I am somewhat worried whether this combinator will be associative. Should that concern give pause to its adoption?

axch avatar Apr 06 '16 20:04 axch