moabb icon indicating copy to clipboard operation
moabb copied to clipboard

Handled errors during validation via error_score parameter

Open jsosulski opened this issue 5 years ago • 5 comments
trafficstars

When using error_score=np.nan in sklearn, this results in np.nan value whenver there's an error during transformations or model fitting. For my personal use case I also would like values of np.nan when there's an error in the validation itself. I have a dataset for which (due to early stopping) sometimes the AUC cannot be calculated for 5-folds. In current moabb, this error is raised and stops the benchmark.

So far I only implemented it for WithinSessionEvaluation as I don't use the other ones.

jsosulski avatar Feb 06 '20 16:02 jsosulski

It is ok for me. You could rebase on master branch to merge these changes.

sylvchev avatar May 28 '20 15:05 sylvchev

For the sake of consistency, it could be good to update CrossSessionEvaluation and CrossSubjectEvaluation as well. Could you do that? I could help if needed.

sylvchev avatar May 28 '20 15:05 sylvchev

I think there was some issues with CrossSubjectEvaluation, but I can check again and update this PR if possible.

jsosulski avatar Jun 03 '20 11:06 jsosulski

My newest commit should contain the necessary changes, however, I have not yet tested this, as I currently do not use any of the other evaluations. I do not know when I will be able to check this on my ERP benchmark.

jsosulski avatar Jun 04 '20 10:06 jsosulski

Did you had the time to check your code? You could add som test to verify that it is correctly working.

sylvchev avatar Jul 13 '20 13:07 sylvchev