stan-dev
add Continuous Ranked Probability Score
Continuous Ranked Probability Score could be useful alternative for log score, As it is comparing predictive distribution to the observed data, it's a bit more complicated to implement than point prediction loss/utility functions, and thus instead of relying people to make user defined one, we could add the support for this.
I know it is an old issue but it popped to my eyes when looking for implementation in the Stan pacakges. I can take a stab at this as I implemented CRPS for my own experiments. One thing to consider in the implementation is whether to compute CRPS using formula (20) or (21) from Gneiting and Raftery (2007). (21) is probably more straightforward especially when computing LOO-CRPS, as one can just call E_loo twice. One caveat I can think of is that interface of the function would require passing two matrices of predictive draws but I suppose it is not a big problem. Any thoughts?
I'm fine with either. I guess existing CRAN packages like scoringRules and scoringutils are not helpful as they don't do loo
Wanted to also add SCRPS to the issue https://arxiv.org/abs/1912.05642