mlxtend.evaluate for corrected resampled t-test from Nadeau's paper

[anbai106]

Hi,

Thanks for your contribution. I am interested to know if you have implemented the corrected resampled t-test from Nadeau's paper https://link.springer.com/article/10.1023%2FA%3A1024068626366. As you have implemented the resampled t-test and 5x2cv paired t test. That would be great if you include this evaluation procedure.

Actually, What I did for my CV is to split my data into 250 times with a stratifiedShuflleSplit, then for each iteration, for the training dataset (0.8), I did 10-fold to find the hyper parameters and then performance of my classier was measured using the testing dataset.

Any idea to use the corrected resampled t-test for my two classifiers????

Thanks in advance

[rasbt]

Hi, there,

no, I only implemented the standard procedure and haven't looked at this version from Nadeau's paper. Let's leave the issue open though since I want to read the paper some time and come back to it

[anbai106]

@rasbt Actually, I have implemented it just now, hahah. I think I highly recommend to use the corrected resampled t-test, because I compared the results of resampled t-test and corrected resampled t-test, the resample t-test gave much lower p-value, which means higher type 1 error than the corrected resampled t-test.

Best

[rasbt]

I highly recommend to use the corrected

not sure about the corrected one (haven't checked it out yet), but the I agree that the "classic" resampled t-test definitely shouldn't be used in practice. I think I mentioned that in the docs but I will double check.

In any case

[anbai106]

@rasbt Yes, I saw that you recommend to use the 52 cv paired t-test, but I think that this depends also the CV procedure, for example, in my case, I can not use the 52 cv paired t-test.

Best

[rasbt]

Yeah, good point!

[danpak94]

@anbai106 This is a bit late, but could you please show where you have implemented the corrected resampled t-test? It would be of great help for my research project. Thanks a lot in advance!

[rasbt]

On a side note, while the 5x2cv t-test has a lower false positive as the resampled t-test (not sure about a comparison with the corrected resampled t-test), Tom Dietterich (the author of the 5x2cv t-test paper) recommend the combined 5x2cv F-test by Ethem Alpaydin to me (https://www.mitpressjournals.org/doi/abs/10.1162/089976699300016007), which is superior.

[jensdebruijn]

@danpak94 See for a simple implementation https://gist.github.com/jensdebruijn/13e8eeda85eb8644ac2a4ac4c3b8e732

[rasbt]

Thanks a lot for sharing. I should add it some time