loo
loo copied to clipboard
stan-dev
loo R package for approximate leave-one-out cross-validation (LOO-CV) and Pareto smoothed importance sampling (PSIS)
Currently, mcse_elpd_loo is computed by assuming the pointwise mcse's to be independent. As the same MC draws are used for each pointwise PSIS weights, the weights are correlated and this...
Fixes #236 1. Uses the recommended h-specific Pareto-k approach 2. Uses type specific h when computing Pareto-k 3. Documents the used h-specific Pareto-k approach in the doc 4. Uses `posterior::pareto_khat`,...
1. @sethaxen pointed out that E_loo h-specific khat diagnostic is using the old approach `sqrt(1 + h_theta^2) * r_theta`, while the Appendix E of 2022 version of PSIS paper recommends...
I just discovered `crps`! This is great, I was actually asking about how to compare probabilistic vs non-probabilistic predictions https://discourse.mc-stan.org/t/comparison-of-machine-learning-models-and-bayesian-models/30833 and this is exactly what I needed. I just noticed...
I think it would make sense to ask the user for only one argument with predictions with at least two columns and split the predictions array. (Also in #221). In...
I noticed that there might be an inconsistency in the Pareto k-values between different approaches for standard importance sampling (SIS): ```r library(loo) log_ratios
If the user already has a psis object then `loo_predictive_metric` and `loo_[s]crps` could use that object instead of rerunning psis internally (similar to what `E_loo` already does, except in this...
Currently `loo_moment_match` only works on `stanfit` objects (due to the reliance on the `log_prob` methods from `rstan`), but not `CmdStanFit` or just a matrix of draws (see e.g. #209). I...
Julia is generally a much faster language than R, especially when doing lots of computations. Given that JuliaCall lets you use Julia very easily from within R, I'm wondering if...
I am working on a model averaging problem with [very simple models](https://wlandau.github.io/historicalborrow/articles/methods.html#independent-model), and I am getting intermittently high Pareto k values even on simple well-behaved simulated datasets. I would like...
