Matthew Feickert

Labeling a `v0.7.0` as if we're going to break APIs might as well do as much at once.

> are you sure you have enough toys to evaluate this (several 10k)? Here I'm using the calculator API in a strange way as only 1 experiment is being evaluated,...

> The last two numbers scale with the number of signal events, so these are something else. yeah. The fact that the calculators are going in opposite directions as the...

> At the moment this doesn't run any toys as far as I can tell, so I'm assuming this is a question about the calculators, and not about asymptotic vs...

Okay @kratsg has pointed out that the behavior of the calculator APIs is (known to be) not consistent across the asymptotic and toy based as asymptotic is returning p-values (so...

> well, it's consistent here, it's that the `calc.teststatistic(test_poi)` has a different meaning which is translated by the respective calculator's `calc.pvalues(teststat, ...)` call if that makes sense. The API is...

My main complaint at the moment is that while we make it clear that the asymptotic test stats are in `-^mu/sigma` space https://github.com/scikit-hep/pyhf/blob/9fd99be886349a90e927672e950cc233fad0916c/src/pyhf/infer/calculators.py#L85-L89 we don't make this clear again in...

This all came up as I was trying to take a stab at Issue #1712 and was trying to figure out to have things work for either calculator type.

I should note of course, that you can do something similar to like what @alexander-held is doing now in `cabinetry` with [`cabinetry.fit.significance`](https://cabinetry.readthedocs.io/en/stable/api.html#cabinetry.fit.significance) ```python import pyhf from scipy.stats import norm model...

Yeah, I think just stealing what `cabinetry` is doing and then also implementing Issue #1712 would be the way to go here.