model comparison

[Lukas231]

Hello,

I would like to ask you, I want to compare few models where predators and source data are same but trophic discrimination factors differ in each model, using either LOO or WAIC.

Compare models 1 and 2 using LOO

compare_models(x=list(jags.1, jags.2), loo=TRUE)

Compare models 1 and 2 using WAIC

compare_models(x=list(jags.1, jags.2), loo=FALSE)

I follow script on this website and everything is working well but I got lot of warning messages. Does it mean that model sensitivity is not high enough? the scipt (in txt) and source data (xlsx source_raw_v1.xlsx fracionalization_1.xlsx fracionalization_2.xlsx fracionalization_3.xlsx

script.txt predator.xlsx

) are uploaded here are some of warning messages

Warning messages: 1: Relative effective sample sizes ('r_eff' argument) not specified. For models fit with MCMC, the reported PSIS effective sample sizes and MCSE estimates will be over-optimistic. 2: Some Pareto k diagnostic values are slightly high. See help('pareto-k-diagnostic') for details.

3: Relative effective sample sizes ('r_eff' argument) not specified. For models fit with MCMC, the reported PSIS effective sample sizes and MCSE estimates will be over-optimistic. 4: Some Pareto k diagnostic values are slightly high. See help('pareto-k-diagnostic') for details.

5: Relative effective sample sizes ('r_eff' argument) not specified. For models fit with MCMC, the reported PSIS effective sample sizes and MCSE estimates will be over-optimistic. 6: Some Pareto k diagnostic values are too high. See help('pareto-k-diagnostic') for details.

7: 'loo::compare' is deprecated. Use 'loo_compare' instead. See help("Deprecated") 8: 'loo::compare' is deprecated. Use 'loo_compare' instead. See help("Deprecated") 9: 'loo::compare' is deprecated. Use 'loo_compare' instead. See help("Deprecated")

Thanks for any advice and help

Lukas

[brianstock]

Hi Lukas,

These are warnings, not errors. Both are related to changes in the loo package.

The first set occurs because I have not passed r_eff to loo: https://discourse.mc-stan.org/t/getting-warning-with-loo-package-regarding-r-eff/3937 "It’s just a warning not an error. elpd_loo and SE for elpd_loo will be same without r_eff, but diagnostic are more accurate if we take into account that draws are not independent. PSIS-LOO computation itself is invariant for permutation of draws, but to estimate the efficiency of MCMC draws we need to know chains and have draws in order."

The second set is because the loo authors changed from loo::compare to loo::loo_compare. This has been fixed in MixSIAR, so you must not be using the latest version. If you reinstall the latest version that warning should go away. remotes::install_github("brianstock/MixSIAR", dependencies=T)

[Lukas231]

Hi Brian,

Thanks a lot for your message. So, I reinstall MixSIAR and tried as you suggest. Most of warning messages disappeared but few of them remains (see below). But If I understand well, it could be given by changes in LOO package, so I can still use this approach, right?

Warning messages: 1: Relative effective sample sizes ('r_eff' argument) not specified. For models fit with MCMC, the reported PSIS effective sample sizes and MCSE estimates will be over-optimistic. 2: Some Pareto k diagnostic values are too high. See help('pareto-k-diagnostic') for details.

3: Relative effective sample sizes ('r_eff' argument) not specified. For models fit with MCMC, the reported PSIS effective sample sizes and MCSE estimates will be over-optimistic. 4: Relative effective sample sizes ('r_eff' argument) not specified. For models fit with MCMC, the reported PSIS effective sample sizes and MCSE estimates will be over-optimistic. 5: Some Pareto k diagnostic values are too high. See help('pareto-k-diagnostic') for details.