Significance and discovery test statistic

[kratsg]

Summary

Create a learn notebook that demonstrates the s/sqrt(b) approximation using the q0 test statistic like so:

>>> import pyhf
>>> 
>>> model = pyhf.simplemodels.uncorrelated_background([25], [2500], [2.5])
>>> pyhf.infer.test_statistics.q0(
...     0.0,
...     [2525] + model.config.auxdata,
...     model,
...     model.config.suggested_init(),
...     model.config.suggested_bounds(),
...     model.config.suggested_fixed(),
... )
array(0.24854737)
>>> _**0.5
0.49854525422391327
>>> 25 / (2500**0.5)
0.5

Additional Information

Equation 53 in https://arxiv.org/abs/1007.1727

Code of Conduct

• [X] I agree to follow the Code of Conduct

[alexander-held]

In this context it might also be nice to include the approximation via best-fit POI value divided by its uncertainty.

[kratsg]

include the approximation via best-fit POI value divided by its uncertainty.

what do you mean?

[alexander-held]

Assuming Gaussianity, the significance is (best-fit POI value) / (POI uncertainty), e.g. for mu = 1 +/- 0.5 this would translate to 2 sigma.

[kratsg]

Hrmm, isn't that generally seen more like the z-score, rather than significance? (e.g. the distinction between p-value and z-score...)