scoringutils
scoringutils copied to clipboard
Avoid computation of p-values in `add_relative_skill()`
The main source of this request is the following example where we use add_relative_skill
to compute a relative absolute error, which generates a bunch of warnings from p-value computations which are not used or saved in the output:
library(scoringutils)
#> scoringutils 2.0.0 introduces major changes. We'd love your feedback!
#> <https://github.com/epiforecasts/scoringutils/issues>. To use the old version,
#> run: `remotes::install_github('epiforecasts/[email protected]')`
#> This message is displayed once per session.
forecast_quantile <- example_quantile |>
as_forecast_quantile(
forecast_unit = c(
"location", "forecast_date", "target_end_date", "target_type", "model", "horizon"
)
)
#> ℹ Some rows containing NA values may be removed. This is fine if not
#> unexpected.
scores <- forecast_quantile |>
score(metrics = get_metrics(forecast_quantile, "ae_median"))
scores_w_rel_skill <- scores |>
add_relative_skill(compare = "model", by = "location", metric = "ae_median")
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with ties
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with ties
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with ties
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with zeroes
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with ties
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with ties
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with ties
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with zeroes
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with ties
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with zeroes
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with ties
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with ties
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with ties
#> Warning in wilcox.test.default(values_x, values_y, paired = TRUE): cannot
#> compute exact p-value with zeroes
colnames(scores_w_rel_skill)
#> [1] "model" "location"
#> [3] "forecast_date" "target_end_date"
#> [5] "target_type" "horizon"
#> [7] "ae_median" "ae_median_relative_skill"
Created on 2025-01-03 with reprex v2.1.1
It would be nice to skip the computation of p-values in a call to add_relative_skill
. I think a solution could involve these changes:
- Add support to
compare_forecasts
fortest_type = NULL
, in which case no test is performed. To avoid changing inputs/outputs, it could setpval = NA_real_
in that case. - In
add_relative_skill
, when callingget_pairwise_comparisons
, passtest_type = NULL
.
This is related to discussion in #750, but I'm suggesting smaller changes here.