loo icon indicating copy to clipboard operation
loo copied to clipboard

Add support for Jacobian adjustment

Open avehtari opened this issue 1 year ago • 0 comments

Related to discussion in a Stan discourse post

ROS book and some Stan discourse answers show how to make Jacobian adjustment for elpd_loo values if different models use different transformations of the continuous target. I had tested that a simple approach of adjusting the pointwise values as

loo_2_with_jacobian <- loo_2
loo_2_with_jacobian$pointwise[,1] <- loo_2_with_jacobian$pointwise[,1] - log(2*sqrt(data$y))

works when comparing two models. In case of more than two models, the summary values in $estimates slot are used to sort the models, which causes problems if the order is different after Jacobian adjustment. A quick fix is to update the summary, too

loo_2_with_jacobian$estimates["elpd_loo",] <- loo:::table_of_estimates(loo_2_with_jacobian$pointwise[,"elpd_loo", drop=FALSE])

It would be better to make a function that would properly update all the slots, or probably even better to add jacobian argument to loo() function.

This update would not be big, doesn't cause backward compatibility issues, and would make the life of users easier

avehtari avatar Apr 05 '24 08:04 avehtari