scoringutils

scoringutils copied to clipboard

Should `wis()` warn about missing values?

3

Consider the following example: ``` df score(metrics = metrics_quantile(select = "wis")) ``` This just produces a normal output. `wis()` does have an argument `na.rm = TRUE`, so in theory all...

nikosbosse

Create a wrapper around log score to warn about its use for integer-valued forecasts

7

In the old version, we don't compute a log score for discrete sample-based forecasts. The reason for that is that the `scoringRules` implementation of the log score estimates a density,...

nikosbosse

`get_forecast_unit()` currently relies on an internal call to `get_protected_columns()` to identify which columns it should ignore. These are currently things like `quantile_level` and `sample_id`. This all works well within the...

nikosbosse

Add check to `validate_forecast()` that predictions must not decrease with quantile

3

**Update**: Current plan is to: - write a helper function to check that - enforce it in `bias_quantile()` as it's needed there - check + warn in `validate_forecast()`. --- This...

nikosbosse

Simplify "Getting Started" | Split into smaller chunks

7

I think the "Getting Started" vignette is well-written but quite long. May I suggest writing a simpler version of the current one and splitting the current one into smaller chunks,...

jamesmbaazam

Create vignette that showcases different ways to visualise forecasts and scores

1

It would be nice to give the users code that exemplifies how they can visualise their scores and forecasts. We should also add an updated version of `plot_score_table()` which is...

nikosbosse

In addition, should check that there aren't any function calls that aren't imported in a function. One example was ` dt

nikosbosse

Improve code coverage

2

We should aim to increase our code coverage and bring it back up. ~Specifically, we should double-check the codecov report from #583 - codecov complained that a few lines were...

nikosbosse

Currently, a few tests print messages and #583 adds more tests that do that. It would be nicer to have those tests not print anything ideally by - addressing the...

nikosbosse

Testing fails locally when done repeatedly

2

Result of repeated `devtools::test()` ```r `assert_input_interval(observed, lower, upper, 0.5)` did not throw the expected warning. ``` because the warning is only shown once by design.

sbfnk