parameters icon indicating copy to clipboard operation
parameters copied to clipboard

Erroneous warning in `.p_adjust`

Open bwiernik opened this issue 3 years ago • 1 comments

m <- lm(mpg ~ factor(cyl), data = mtcars)
parameters::parameters(m, adjust = "fdr")
Parameter   | Coefficient |   SE |          95% CI | t(29) |      p
-------------------------------------------------------------------
(Intercept) |       26.66 | 0.97 | [ 24.68, 28.65] | 27.44 | < .001
cyl [6]     |       -6.92 | 1.56 | [-10.11, -3.73] | -4.44 | < .001
cyl [8]     |      -11.56 | 1.30 | [-14.22, -8.91] | -8.90 | < .001

p-value adjustment method: Benjamini & Hochberg (1995)

Uncertainty intervals (equal-tailed) and p-values (two-tailed) computed using a Wald
  t-distribution approximation.
Warning message:
Could not apply fdr-adjustment to p-values. Either something went wrong, or the non-adjusted
  p-values were already very large. 

The warning isn't correct. It's just that the adjustment here isn't very large because the p values are very small and the number of p values in the set is also small.

This is caused by this check in .p_adjust:

if (isTRUE(all.equal(old_p_vals, params$p)) && !identical(p_adjust, 
    "none")) {
    if (verbose) {
        warning(insight::format_message(paste0("Could not apply ", 
            p_adjust, "-adjustment to p-values. Either something went wrong, or the non-adjusted p-values were already very large.")), 
            call. = FALSE)
    }
}

Do we need this check? Should we use identical() or all(old == new) instead, or are there specific computational issues we are trying to detect?

bwiernik avatar Jul 14 '22 21:07 bwiernik

Yes, I think we need this check, because there are some exceptions where adjustment doesn't work, and no error/warning is thrown (can't recall an example right now), that's why we have this check here. But we probably can replace by identical() or all(old == new) to be more exact.

strengejacke avatar Jul 16 '22 20:07 strengejacke