Tutorial on coverage

[michaeldeistler]

Coverage is implemented in sbi, but we have no tutorial on it:

num_sbc_runs = 1_000  # choose a number of sbc runs, should be ~100s or ideally 1000
thetas = prior.sample((num_sbc_runs,))
xs = simulator(thetas)

# pass `reduce_fns=posterior.log_prob` for coverage.
ranks, dap_samples = run_sbc(
    thetas, xs, posterior, num_posterior_samples=num_posterior_samples, reduce_fns=posterior.log_prob
)

[gmoss13]

More context:

The SBC tutorial currently only discusses calibration of the 1D marginals of the posteriors. Simulation-based calibration can be done with any reducing function (Talts et al.). Using the posterior log probabilities as the reducing function is already implemented. This issue would involve adding an example of using this in the existing tutorial, and discussing when one might to use this as a reducing function instead of the marginals.

More info: run_sbc docstring and https://arxiv.org/abs/2210.04815 section 3.3. Essentially, we want to add expected coverage calculation to the tutorial.

[psteinb]

I am currently reviewing the sbc tutorial. Let me see, if I can help to add this.