Argument for additional metrics in model_performance()

[bwiernik]

I'd like to be able to request additional performance metrics in model_performance(), such as AICC or confidence intervals for R2 or sigma. It would be nice if, in addition to the metrics argument, there were an additional_metrics argument that could take a named list of additional functions to run, each of which should return a vector a results that would be added as additional columns to the performance data frame.

For example:

m <- lm(formula = mpg ~ wt + qsec + am, data = mtcars)
performance(m, metrics = "common", additional_metrics = list(`R2 CI` = confint_R2, AICC = performance_aicc))

Or perhaps it would be cleaner to just have the one metrics argument and allow it to be either the current string vector or a named list of functions:

m <- lm(formula = mpg ~ wt + qsec + am, data = mtcars)
performance(m, metrics = "list(R2 = r2, `R2 CI` = confint_R2, AICC = performance_aicc, Sigma = get_sigma))

[DominiqueMakowski]

Or perhaps it would be cleaner to just have the one metrics argument and allow it to be either the current string vector or a named list of functions

I would prefer to have less arguments.

Regarding AICc, perhaps we should return it by default? Or even instead of the AIC, if that makes sense?

[bwiernik]

I think retaining first order AIC makes sense given its ubiquity. I’d really like there to be an option to add R2 CIs and n_obs() to the performance table to facilitate reporting (especially when teaching), so an option to add additional functions to metrics would be great.

[strengejacke]

You can already request AICc (https://easystats.github.io/performance/reference/model_performance.lm.html).

But we could indeed add the other ones, too.

[bwiernik]

Ah I missed the casing "AICC” is ignored.

[strengejacke]

Ah I missed the casing "AICC” is ignored.

That's indeed inconsistent, we should convert all input into upper or lower case inside the function, so case can mostly be ignored.

[strengejacke]

I just checked, and should already work:

library(performance)
m <- lm(formula = mpg ~ wt + qsec + am, data = mtcars)
performance(m, metrics = c("R2", "AICc"))
#> # Indices of model performance
#> 
#> AICc    |    R2
#> ---------------
#> 156.427 | 0.850

performance(m, metrics = c("R2", "AICC"))
#> # Indices of model performance
#> 
#> AICc    |    R2
#> ---------------
#> 156.427 | 0.850

performance(m, metrics = c("R2", "r2_adj", "aicc"))
#> # Indices of model performance
#> 
#> AICc    |    R2 | R2 (adj.)
#> ---------------------------
#> 156.427 | 0.850 |     0.834

performance(m, metrics = c("R2", "r2_adj", "aicc", "AiC"))
#> # Indices of model performance
#> 
#> AIC     |    AICc |    R2 | R2 (adj.)
#> -------------------------------------
#> 154.119 | 156.427 | 0.850 |     0.834

^{Created on 2021-04-05 by the reprex package (v2.0.0)}

Please re-open if you still find any issues!

[bwiernik]

Re-opening to discuss the broader question of adding additional functions.

For example, let's say I'd like to add n_obs() to performance, output, something like this would be nice:

library(performance)
m <- lm(formula = mpg ~ wt + qsec + am, data = mtcars)
performance(m, metrics = list(R2 = r2, Sigma = insight::get_sigma, N = insight::n_obs))

Within the performance function, the type of metrics could be checked. If its "character", the current behavior is used; if it's "list", it is taken as a list of functions.

I can put together a PR if there is interest.

[strengejacke]

Yeah, that sounds reasonable! Please open a PR, if you like. And we could indeed add n_obs() as further metric (N, nobs?) as well.