jasp-issues
jasp-issues copied to clipboard
jasp-stats
Output order - Coherence in line with the methodological flow
- Enhancement: Displaying assumptions checks and descriptives first in the output of a test
- Purpose: The behavior of the software should mimic a rigorous methodological flow
- Use-case:
Is your feature request related to a problem? Please describe.
Presently, JASP behaves similarly to SPSS and displays the assumptions checks after the results of the tests. This behavior might bias the scientific endeavour, because assumptions need always be checked first. Based on the results of these tests, the researcher might indeed change the test to use... I think that changing this behavior would be great. As a professor, I noticed that the way the software behaves give a strong message to the students. So if JASP was more in line with the methodological flow according to best practices, that would be great.
Describe the solution you'd like
When the user check the box "Descriptives" and "Assumption Checks", these results need to be displayed first in the output for every test.
Describe alternatives you've considered
Additional context
Statistical thinking, Methodological coherence, Methodological rigour.
Similarly, when ticked, perhaps the descriptive tables and plots ought to come first. But this warrants some discussion before doing this for all tests
+1 from me. We teach (hopefully) something like Anscombe-Quartett to emphasize the importance of graphs. Thus currently I teach to always start an analysis with flexplot. So +1 from me to always show a graph as default.
To enable an aditional checkbox as default to every analysis would probably be much easier to implement than to not show p-values as default or to change the order of all the tables and graphs in all analysis. Because many R analyses we use in JASP default to the table that show the p-value first (which is probably inherited from SPSS behaviour).
But it certainly is a bad default to show a t/F/p-value without context (t-tests, ANOVAS) and better to show at least the effect size first or alongside (correlation, regression) and only then the p-value. So all this would be solved by showing a graph first and would be in line with @chantelanuit 's request, because the graph can be used to check assumptions AND can show some descriptives at the same time.
I agree with the general sentiment but we need to discuss this more, because it would mean an overhaul of most of our analyses. It may also confuse the user. If the descriptives are obvious, then it could work -- for instance we already automatically show a scatterplot for the correlation test, and we could show a raincloud plot for the t-test. But I don't want to hide the ultimate inference people are looking for underneath a thick blanket of assumption checks.
I respectfully disagree with the idea that assumption checks might cloud data analysis in JASP. Presently, the user has no obligation to do these assumption checks. The user is requested to check the appropriate boxes to have the results displayed (note that some authors advise against some of these checks), so the interface is not cluttered. I do not suggest that these checks need to be mandatory displayed.
However, the display order of a) the assumptions checks, when the user requests them, and more importantly, b) the descriptive statistics is often a neglected part of statistical software's ergonomic. This might however have a real impact on the user learning how to proceed with data analysis (i.e. first read the results of the inferential test, then, have a look at the assumptions checks and descriptives... which is wrong - that strategy might inadvertently lead to data butchery).
My suggestion is to simply improve the display order to get aligned with the best practices:
The t test here is only an example. That reasoning applies to every inferential test.
+1 for @chantelanuit
And for always showing a plot as default.
@EJWagenmakers We do not "automatically show a scatterplot for the correlation test". There is no module I know of, that automatically shows a plot except flexplot. And to automatically check one plot per analysis as default would not mean to "overhaul of most of our analyses". That might be needed for the proposal of @chantelanuit.
Indeed, for the correlation, the scatterplot is essential and also needs to be displayed before the inferential test table. A curvilinear relation can sometimes result in a weak r-value although there is a strong nonlinear association between the variables. In that situation, the analyst must understand that his/her model is the wrong one and the interpretation of the correlation is not relevant.
Yes, and we have Anscombe's quartet in the JASP Data Library :-) I see that the scatterplot is only shown by default in the Bayesian correlation test, under pairwise plots. The issue is whether or not we want to show the entire correlation matrix by default. I could go either way on this. However, one guiding principle in JASP is gradual disclosure of results -- minimal output is presented at first, and we do this for almost all analyses. Another idea is to allow users to customize JASP; so if you tick the option "show correlation matrix" then JASP will do so for any future analysis.
The guiding principle is certainly great! I take back my stance of adding more outputs as default. And cusomization is also a good idea. This brings a third option to resolve this discussion:
- @chantelanuit proposes: changing the order in which the results are presented => big rework of modules needed
- @tomtomme proposes: just changing the default of showing a table first into showing a graph first (since the table stats might hide/obscure the information from the data. This is where flexplot and Tukey got it right. Show the graph first.)
- @EJWagenmakers proposes: No change to the modules, but a cusomizability options to save new defaults
Option 1 and 2 would certainly be more geared towards learners, since defaults send a strong message. Option 3 seems to be geared towards professionals and teachers that want to have it "their way" (like us...). So maybe we need all 3...
P.S. The question of showing the whole correlation matrix is none, since we already show it, just in numbers not via a plot. And even bayesian correlation does not show the graph by default - you still need to drag two variables over - only the checkbox is already marked, but it does nothing on its own.