scoringutils

scoringutils copied to clipboard

Improve calibration plotting

2

- Add Talagrand diagram (verification rank histogram) --> see Gneiting 2007. Somewhat analogous to PIT. - look at sample autocorrelation functions for PIT values - add marginal calibration plot

nikosbosse

Implement a binned Log Score for integer valued forecasts

1

nikosbosse

Add Diebold Mariano test

2

It would be cool to have an implementation of the Diebold-Mariano test to test whether the scores of two models are significantly different.

nikosbosse

Rework `interval_score` function so that it also works with quantiles, not only ranges

2

nikosbosse

E.g. IQR, SD

nikosbosse

i.e. add some code to score_continuous_integer.R ``` if ("coverage" %in% metrics) { } ```

nikosbosse

For example, in `pit_sample()` there is currently this: ``` if (missing("true_values") | missing("predictions")) { stop("`true_values` or `predictions` missing in function 'pit_sample()'") } check_not_null(true_values = true_values, predictions = predictions) ``` This...

nikosbosse

A simple function could be passed in `eval_forecasts` This could be especially useful for scoring point forecasts so that we don't have to implement all metrics on our own

nikosbosse

Add integration for scoring point forecasts

1

a lot of metrics are already present in the metrics package. It would be good to have a convenient to use wrapper that automatically scores point forecasts

nikosbosse

Make scoringutils paper a package Vignette

2

## Description This PR closes #784. The PR - makes some nonsense changes in .Rbuildignore and .Rinstignore and _pkgdown.yml - please ignore - makes a small update to the manuscript...

nikosbosse