bayestestR icon indicating copy to clipboard operation
bayestestR copied to clipboard

Allow Multiple Values of the Log Marginal Likelihood

Open ajnafa opened this issue 1 year ago • 3 comments

Describe the solution you'd like

As I mentioned to @bwiernik and @mattansb on twitter yesterday, it appears the bayesfactor_.* functions only support a single value for the log marginal likelihood. As far as I can tell, the current behavior of the package is to throw a warning if less than 40,000 post-warmup samples are taken but as noted in Schad et al. (2022) and as I discuss in the context of Bayesian Model Averaging here this doesn't necessary guarantee repeated runs of the bridge sampling algorithm will produce stable results and a preferable approach where tractable is to calculate a distribution of estimates for the log marginal likelihood by passing the repetitions argument to bridgesampling::bridge_sampler.

I looked into implementing this for brmsfit objects in #573 since you can store the log marginal likelihood estimates internally in the model object using brms::add_criterion but it seems like all of the functions that inherent from bayesfactor_models expect a single estimate for the log ML of each model so for the time being it just takes the median of the stored values. Longer term though, it is probably worth considering adding support for repeated estimates of the log ML.

How could we do it?

As far as implementation is concerned, the first step would be to allow users to pass additional arguments to the call to bridgesampling::bridge_sampler through bayesfactor_models. After that things get a bit more complicated because the other bayesfactor.* functions need to be modified to handle a vector of values for each model. One way to do this that would integrate nicely into the current structure of the print functions might be to simply return the median and quantiles of the bayes factor estimates but I'd need to dig more into the structure of the package to figure out how feasible this is.

ajnafa avatar Oct 22 '22 19:10 ajnafa