additonal adjusted R2 estimators

[SchmidtPaul]

Apparently, nowadays even Wikipedia points out the shortcomings of (adj.) R²:

Despite using unbiased estimators for the population variances of the error and the dependent variable, adjusted R2 is not an unbiased estimator of the population R2,[16] which results by using the population variances of the errors and the dependent variable instead of estimating them. Ingram Olkin and John W. Pratt derived the Minimum-variance unbiased estimator for the population R2,[17] which is known as Olkin-Pratt estimator. Comparisons of different approaches for adjusting R2 concluded that in most situations either an approximate version of the Olkin-Pratt estimator [16] or the exact Olkin-Pratt estimator [18] should be preferred over (Ezekiel) adjusted R2.

The German Wikipedia article even phrases it as (translated):

In practice, it is not recommended to use adj. R² for model selection, because the "penalty" for newly added explanatory variables seems too small. [...] For this reason, other criteria such as the AIC and the BIC have been developed for model selection, which also follow Occam's razor idea that a model should not be unnecessarily complex. [...] Numerous alternative estimators for R² in the population exist (see [51]). Of particular importance is the Olkin-Pratt estimator[52], since it is an unbiased estimator. In fact, it is the uniformly best unbiased estimator. Empirical comparisons of the different estimators logically conclude that in most cases the approximate[49] or the exact[50] Olkin-Pratt estimator should be used instead of the corrected coefficient of determination.

Luckily, Karch (2020) nicely compares the multiple measures and even published the {altR2} package which estimates several of them. Unfortunately, it appears to only work for lm() objects.

Question:

So my question is simply whether you have thought about implementing an Olkin-Pratt estimator to the {performance} package i.e. to model_performance() and if you have, I am genuinely curious why you decided against it.

[bwiernik]

Yeah, the ubiquity of the Fisher adjusted R-squared is that it is pretty good and was easier to compute before computers made the gamma function easy to estimate.

My go to reference for population and cross-validity R^2 estimators is Shieh 2008. They compared several for both quantities (including the Pratt estimator). It would be good to add an argument for type of adjusted R-squared, defaulting to the traditional adjusted R-squared. In addition to the named estimators, I suggest we also include "traditional", "population", and "cross-validity" as aliases for the Fisher and best-estimate population and CV R-squared per Shieh

[bwiernik]

See here for the 3 estimators we should include at minimum (traditional/Fisher, positive-part Pratt population, and Browne/positive-part Pratt cross-validity): https://github.com/bwiernik/configural/blob/47e917501b0026fbfa5c64de20b6585850a2b335/R/utility.R#L22

[bwiernik]

In general, I would recommend information criteria over any form of R-squared index for model selection, not least because they generalize to non-normal models in a way that R-squared does not

[DominiqueMakowski]

you have thought about implementing an Olkin-Pratt estimator

Besides the validity questions, do we have this estimator and, if not, is it straightforward to add?

[bwiernik]

I can implement it or something very similar easily

[SchmidtPaul]

Great, thanks! The implementation in {altR2} (permalink to GitHub code line) seems relatively straightforward to me.

[bwiernik]

Ah, the reference [49] there is the Shieh article I mentioned. In that, Shieh refers to this estimator as just the "Pratt" estimator.