A question on the multiple comparison of emmeans_test()

[neurothew]

First of all, thanks for the amazing package and the resource online, that helps a lot in conducting the analysis.

I am trying to fit a 2-way mixed anova, with a between-subject variable with 2 levels, and a within-subject variable with 32 levels. By using anova_test I found that there is significant interaction between the two variables.

Then, I proceed to do the post-hoc comparison with emmeans_test() like the following:

df %>% group_by(between-subject) %>% emmeans_test(dv ~ within-subject)

Everything seems ok until I notice that the p and p.adj is the same, no matter how I choose the p.adjust.method parameter. Wasnt I doing 32 comparisons by grouping the original data with the 32 within-subject variable? Why didnt the multiple comparison work there? Am I making any conceptual mistakes?

Thank you, any help would be greatly appreciated.

[jackbrookes]

This package does not make it clear that group_by followed by tests does not correct for p values across groups. Should it?

[Scott-S-Hannah]

I have also ran into this issue with the multiplicity adjustments. Using eemeans::contrast()you can use the combine argument to create one family of comparisons and adjust accordingly, but with group_by() %>% eemeans_test() these are treated as separate families of contrasts. I know that I would benefit from having an option to combine contrast families. I mostly use this function for custom significance plotting.

Many thanks, Scott