posterior
posterior copied to clipboard
[Feature] Add relative_eff function
For some diagnostics, such as pareto-k diagnostics, we may want to calculate the relative efficiency (r_eff). The loo package has an implementation of this loo::relative_eff. The documentation states that it 'computes the MCMC effective sample size divided by the total sample size', but the output is not equal to ess_bulk(x)/ndraws(x) nor ess_basic(x)/ndraws(x), so perhaps the method in loo could be adapted and moved to posterior?
I now see that the recommendation for calculating r_eff is to use the split-chain ESS ("[...] R_{eff,MCMC} = S_{eff,MCMC}/S is the estimated relative efficiency of the MCMC draws, and the effective sample size S_{eff,MCMC} for r(θ) is computed using the split-chain effective sample size estimate method" from Appendix H in Vehtari et al, 2022).
~~It seems that loo:::ess_rfun is not using the split-chain estimation, so I guess this issue is actually about whether a relative_eff function which is based on the split-chain ESS would make sense to include in posterior (and then potentially used by loo). Would it just be ess_bulk(x)/ndraws(x)?~~ EDIT: The correct calculation is ess_basic(x)/ndraws(x).
As relative eff should be quantity specific, I don't think it makes sense to have a single function for it. Closing this for now