sbi-dev
A "flexible interface" for diagnostic tools
In a couple of issues now, users have reported memory issues with SBC. I think the easiest way to overcome these issues is to introduce a whole new interface for SBC which might contain a couple of lines of code but which gives much more flexibility.
The other option is to just adding more and more kwargs to run_sbc, but I would really be in favor of just developing a new interface.
In the refactoring #1196 we already separated the sampling from the run_{sbc, tarp}. So in principle, it's possible to do things more low-level, e.g., first generate posterior samples on a test set, and pass them to sbc and tarp without having to pass a posterior for sampling anymore. But this is not documented well.
But overall, I agree: we should think about a more elegant interface for the diagnostics in general. First, something more high-level like sbi.train() -> sbi.build_posterior() -> sbi.diagnose() and something more low-level like you are suggesting.
