scikit-posthocs icon indicating copy to clipboard operation
scikit-posthocs copied to clipboard

Alpha cannot be specified for critical difference diagram

Open j-adamczyk opened this issue 5 months ago • 1 comments

Describe the bug

I haven't seen any way to provide alpha (0.05 or 0.1) for plotting critical difference diagram of Nemenyi-Friedman test. I'm trying to follow Demsar's paper link and this tutorial (which sadly uses Orange package), and:

  • "The performance of two classifiers is significantly different if the corresponding average ranks differ by at least the critical difference"
  • "critical values q_α are based on the Studentized range statistic divided by √2", and examples of such values are provided in the paper
  • relevant scikit-posthocs code

So I get the p-values from the test. But looking at CD plotting function, we can see that it uses sign_array() function with constant arguments, which is alpha=0.05. So there is no way to plot CD diagram with any other alpha level. This should be configurable.

j-adamczyk avatar Jan 18 '24 08:01 j-adamczyk