Moving from ParetoSmooth to PSIS+PosteriorStats

[sethaxen]

On Slack we were discussing replacing ParetoSmooth with PSIS+PosteriorStats. This issue compares the functions of the two sets of packages, which will hopefully make it easier to make the switch.

Function mappings

• ParetoSmooth.psis -> PSIS.psis
• ParetoSmooth.psis_loo/ParetoSmooth.loo -> PosteriorStats.loo
• ParetoSmooth.loo_compare -> PosteriorStats.compare

Note that the signatures and dimension orders of the arguments may differ from ParetoSmooth. However, PSIS+PosteriorStats use the same conventions as MCMCDiagnosticTools, so arguably their signatures are more consistent.

Struct mappings

• ParetoSmooth.ModelComparison -> PosteriorStats.ModelComparisonResult
• ParetoSmooth.Psis -> PSIS.PSISResult
• ParetoSmooth.PsisLoo -> PosteriorStats.PSISLOOResult

These are just included for completeness. Note that the fields of the structs, methods defined on them, and how they are shown in the terminal are not the same.

Functions that should be moved here

• ParetoSmooth.pointwise_loglikelihoods: The main use of this utility is to get log-likelihoods from a Turing model in the format required by ParetoSmooth.loo, so perhaps it should be moved here (or to DynamicPPL).

Note again that dimension ordering of the output would I think need to change.

Functions that should go away

• ParetoSmooth.loo_from_psis: this could be replaced with a simple overload PosteriorStats.loo(::PSIS.PSISResult) in PosteriorStats. A coming refactor to PosteriorStats might introduce this.
• ParetoSmooth.naive_lpd: PosteriorStats has the internal method PosteriorStats._lpd_pointwise for computing this, but even ParetoSmooth advises against its use.
• ParetoSmooth.psis_ess: I'm not certain where this ESS estimate comes from, as I can't find it described in the reference it cites. However, the loo package and recent versions of the PSIS paper give an ESS for the normalization term, which is implemented as PSIS.ess_is
• ParetoSmooth.ess_sup: While the one in PSIS.ess_is is a variance-based (i.e. l2-norm) ESS estimate, this implements one based on the linf-norm. This isn't mentioned in any of the LOO or PSIS papers, and since MCMCDiagnosticTools.ess is also variance-based, I think it is unnecessary to include this estimator. Note that varianced-based ESS estimates are closely tied to the concepts of estimator variance and MCSE, so they have a more straightforward interpretation.
• ParetoSmooth.relative_eff: in PosteriorStats.loo, leaving reff=None results in the relative ESS being computed internally, so no need to expose this function.

[ParadaCarleton]

Agreed with basically all of these comments/changes, or else I don't disagree strongly enough to object.

• ParetoSmooth.ess_sup: While the one in PSIS.ess_is is a variance-based (i.e. l2-norm) ESS estimate, this implements one based on the linf-norm. This isn't mentioned in any of the LOO or PSIS papers, and since MCMCDiagnosticTools.ess is also variance-based, I think it is unnecessary to include this estimator. Note that varianced-based ESS estimates are closely tied to the concepts of estimator variance and MCSE, so they have a more straightforward interpretation.

It's not mentioned in the PSIS paper, but I added it based on a few papers finding it was a more accurate measure than the quadratic ESS. You can find one here and another here, and you can find a lot more discussion on this metric if you go hunting through the citations for each.

You can think of this as the maximally pessimistic estimator of ESS, since out of any possible ESS metric based on the variation in the size of the weights, the one using the L-infinity norm will produce the smallest ESS. The paper I linked found it was the best predictor of error when the proposal and target distributions are very different (i.e. when we should be most concerned about the accuracy).

In any case, I don't think there's any issues with including it alongside the more-familiar L2-ESS.