effectsize

effectsize copied to clipboard

The confidence intervals for the effect size of repeated measures calculated by the rm_d() function do not match the results of other existing functions. For example, the following two outputs have matching point estimates but not matching confidence intervals.

# generate data for repeated measurements
library(mvtnorm)
sigma <- matrix(c(1, 0.5, 0.5, 1), byrow = TRUE, ncol = 2) 　# variance-covariance matrix
mu <- c(1, 2)  # population mean vector
n <- 100       # sample size
set.seed(123)  # fixing random number of seeds
dat <- data.frame(mvtnorm::rmvnorm(n = n, mean = mu, sigma = sigma)) # random numbers following a multivariate normal distributio
colnames(dat) <- c("t1", "t2")  # naming variables
dat$diff <- dat$t2 - dat$t1     # difference score

# effect size : difference score variance d_z
library(effectsize)
effectsize::cohens_d(dat$diff, mu = 0) |> print(digits = 5)
#> Cohen's d |             95% CI
#> ------------------------------
#> 0.95874   | [0.71998, 1.19412]

effectsize::rm_d(Pair(t2, t1) ~ 1, data = dat, method = "z",adjust = F)|> print(digits = 5)
#> d (z)   |             95% CI
#> ----------------------------
#> 0.95874 | [0.72195, 1.19553]

The reason for the discrepancy is that rm_d() calculates confidence intervals by approximating the sample distribution with a normal distribution, while the other existing functions calculate confidence intervals using the non-central distribution method. I consider the non-central distribution method to be more accurate.

In addition, the results of the following functions should be consistent.

# Within-Subject Variance: d_rm
r <- cor(dat$t1, dat$t2) # Pearson's product-moment correlation coefficient
effectsize::cohens_d(dat$diff, mu = 0) * sqrt(2 * (1 - r))  # non-central distribution method
#>   Cohens_d       CI    CI_low  CI_high
#> 1 1.081488 1.071631 0.8121631 1.347004

effectsize::rm_d(Pair(t2, t1) ~ 1, data = dat, method = "rm", adjust = F)|> print(digits = 6) # normal approximation method
#> d (rm)   |               95% CI
#> -------------------------------
#> 1.081488 | [0.803159, 1.359817]

# Control Variance: d_b
effectsize::glass_delta(dat$t2, dat$t1, adjust = F)|> print(digits = 6) # non-central distribution method
#> Glass' delta |               95% CI
#> -----------------------------------
#> 1.134785     | [0.801919, 1.463171]

effectsize::rm_d(Pair(t2, t1) ~ 1, data = dat, method = "b", adjust = F)|> print(digits = 6) # normal approximation method
#> Becker's d |               95% CI
#> ---------------------------------
#> 1.134785   | [0.863465, 1.406106]

# Average Variance: d_av
effectsize::cohens_d(dat$t2, dat$t1, pooled_sd = F) |> print(digits = 6)  # non-central distribution method
#> Cohen's d |               95% CI
#> --------------------------------
#> 1.082732  | [0.784408, 1.617938]

effectsize::rm_d(Pair(t2, t1) ~ 1, data = dat, method = "av", adjust = F)|> print(digits = 6) # normal approximation method
#> d (av)   |               95% CI
#> -------------------------------
#> 1.082732 | [0.861387, 1.304076]

# All Variance: Just d
effectsize::cohens_d(dat$t2, dat$t1, pooled_sd = T)|> print(digits = 6) # non-central distribution method
#> Cohen's d |               95% CI
#> --------------------------------
#> 1.082732  | [0.784570, 1.378496]

effectsize::rm_d(Pair(t2, t1) ~ 1, data = dat, method = "d", adjust = F)|> print(digits = 6) # normal approximation method
#> Cohen's d |               95% CI
#> --------------------------------
#> 1.082732  | [0.835758, 1.329706]

I would like the confidence intervals to be calculated in the same method consistently within the package.