[Bug]: RM ANOVA: Wrong Confidence Intervals

[maximusmontis]

JASP Version

0.17.1

Commit ID

No response

JASP Module

ANOVA

What analysis are you seeing the problem on?

Repeated Measures ANOVA

What OS are you seeing the problem on?

Windows 11

Bug Description

Hi,

when calculating an RM ANOVA, I encountered a problem. Under "Descriptive Plots" you can set that confidence intervals are displayed. However, these confidence intervals do not match the confidence intervals that are output by SPSS. The confidence intervals in JASP are much smaller. I have attached screenshots of the JASP graph and SPSS graph. Both analyses are based on exactly the same data set. CI's were set to 95% in both cases.

What could be the reason for this?

Expected Behaviour

I would expect the Confidence Intervals to be similar to the ones given by SPSS.

Steps to Reproduce

1. Calculate any RM ANOVA.
2. Create a Descriptives Plot with 95% CI.

Log (if any)

No response

Final Checklist

• [X] I have included a screenshot showcasing the issue, if possible.
• [x] I have included a JASP file (zipped) or data file that causes the crash/bug, if applicable.
• [X] I have accurately described the bug, and steps to reproduce it.

[github-actions[bot]]

@maximusmontis, thanks for taking the time to create this issue. If possible (and applicable), please upload to the issue website (https://github.com/jasp-stats/jasp-issues/issues/2127) a screenshot showcasing the problem, and/or a compressed (zipped) .jasp file or the data file that causes the issue. If you would prefer not to make your data publicly available, you can send your file(s) directly to us, [email protected]

[JohnnyDoorn]

Hi @maximusmontis ,

We follow a special procedure for the RM ANOVA plots. This is from our helpfile: Display error bars: By selecting this option, error bars will be displayed in the plot. The error bars can either represent confidence intervals or standard errors. In order to get accurate confidence intervals and standard errors, the data are normalized by subtracting the appropriate participantʹs mean performance from each observation, and then adding the grand mean score to every observation. The variances of the resulting normalized values in each condition, and thus the size of the bars, no longer depend on the participant effects and are therefore a more accurate representation of the experimental manipulation. See Morey (2008) for a thorough discussion of this procedure.

We apply the procedure from Morey (2008), where some normalization is applied, so it is not as straightforward as sd / sqrt(n). See Morey, R. D. (2008) Confidence Intervals from Normalized Data: A correction to Cousineau (2005). Tutorial in Quantatitative Methods for Psychology, 4(2), 61-64.

I do think we should add a checkbox that makes this procedure optional, and more salient.

Kind regards Johnny

[maximusmontis]

Hi @JohnnyDoorn ,

thank you for the fast reply! I agree that it would be good to make this procedure optional.

[maximusmontis]

Hi @JohnnyDoorn,

unfortunately I think there is still some issure with the RM ANOVA confidence intervals. I understand that in version 0.19, Morey's correction has been made optional (with the checkbox "normalize error bars"). However, even with this disabled, the CI bars still deviate from the SPSS graph; they even get a bit smaller and therefore deviate further. Both JASP graphs suggest a notable difference in T3 (no overlap in the CIs), while SPSS does not. I would expect the non-normalized graph from JASP to be identical to the SPSS graph, or am I wrong?

JASP normalized:

JASP non-normalized

SPSS:

[tomtomme]

@maximusmontis Thx for the report. It might not be crucial here, but can you share the .jasp file by renaming to zip and drag and dropping it here?

[maximusmontis]

Hi @tomtomme,

sure, here's the file: RMANOVA.zip

[tomtomme]

for what its worth, I can confirm the JASP results.

@JohnnyDoorn Any idea whats going (wr)on(g) here?

[boutinb]

@JohnnyDoorn is on holidays, maybe @vandenman can have a look.

[maximusmontis]

Small addition: I just noticed that when conducting a Paired Samples T-Test and generating a Descriptives Plot, the checkbox "normalize error bars" does not change anything, the confidence interval bars stay exactly the same. Apparently, it is not an isolated ANOVA issue.

[JohnnyDoorn]

Hi @maximusmontis,

Thanks for your report. How did you create the graph in SPSS? It might not take into account the repeated measures structure (in the chart builder, if i remember correctly), which affects the standard deviations. In JASP, we first center the observations for each subject to account for the repeated measures structure, so maybe that is causing a discrepancy between JASP and SPSS? Only after this centering do we apply the Morey correction or not. I did just update the t-test behavior, since that one indeed did not account for the option being selected while it should (similar to how it behaves for the bar chart).

Cheers, Johnny

[maximusmontis]

Hi @JohnnyDoorn,

thank you for you answer. In SPSS, I used General Linear Model -> Repeated Measures -> defined the within-subject factor with 3 levels (measurement) and one between-subject factor (condition). For creating the plot, I used these settings:

[JohnnyDoorn]

Hi @maximusmontis,

I cannot find much information about how the current version of SPSS computes the error bars, but this paper from 2014 and this one from 2021 suggest that SPSS does not apply any correction procedures to the error bars, while we subtract each participant’s overall mean (across conditions or time points) from their individual scores, then add the grand mean of all scores. I think that explains the discrepancy. I do realize that JASP's option to normalize was only concerning the Morey correction component, and not the full normalization initially proposed by Cousineau. Based on this I will revise how JASP handles the normalization, with the checkbox on by default, but with the normalization (not just the Morey factor) optional when you are interested in the between subject effects of the RM ANOVA design. PR is here https://github.com/jasp-stats/jaspAnova/pull/385

Cheers, Johnny

[maximusmontis]

Hi @JohnnyDoorn,

I see, thank you very much for the information and the fix!

[maximusmontis]

Hi @JohnnyDoorn,

it appears this fix is not included in version 0.19.1, is that intended?

[JohnnyDoorn]

@maximusmontis unfortunately we decided to have a speedy hotfix, rather than incorporate statistics fixes that need testing, so it will only be in the version after this one (planned for October). If you already want a version with the fix, you could try one of the "development" nightlies from https://static.jasp-stats.org/Nightlies/

Cheers Johnny

[maximusmontis]

@JohnnyDoorn I see, thank you for the link!