rasbt
Independent t-test
Comparing the same algorithm but on different data sets, and deciding whether the difference in a chosen metric is statistically significant or just due to randomness could be useful.
It might be useful to add a 2-sample independent t-test in general, but I think using either (a) a 5x2cv paired t test or McNemar test is better for classifier comparisons
- http://rasbt.github.io/mlxtend/user_guide/evaluate/paired_ttest_5x2cv/
- http://rasbt.github.io/mlxtend/user_guide/evaluate/mcnemar/
EDIT: I see you are asking for the different-dataset scenario. Yes, then these two above cannot be used and either a permutation test of 2-sample independent t-test may be used.
Yes, I suggested it since I needed to use the independent version to compare the same classifier but on different data sets :)
niedz., 14 mar 2021, 16:57 użytkownik Sebastian Raschka < @.***> napisał:
It might be useful to at 2-sample independent t-test in general, but I think using either (a) a 5x2cv paired t test or McNemar test is better for classifier comparisons
http://rasbt.github.io/mlxtend/user_guide/evaluate/paired_ttest_5x2cv/
- http://rasbt.github.io/mlxtend/user_guide/evaluate/mcnemar/
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/rasbt/mlxtend/issues/807#issuecomment-798931391, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEGSVABKACEU2IP6SDJ7T63TDTMHZANCNFSM4Y7C2DCA .