Creating a Global Benchmarking Pipeline and Results Page

[Div12345]

This sort of is a continuation to #6 to get the global comparisons of the different algorithms on all the datasets. This comment gives the one of the possibilities -

... at least one global ranking page, highlighting the top 3 algorithms for each category, and one page allowing to compare any of two algorithms.

There are a few questions like -

1. Which algorithms on which of the paradigms
2. Quirks specific to certain datasets and mitigating it for the comparison
3. Comparison metrics
4. Will Github Actions be able to handle the running of the benchmark or will it have to be outsourced to some other server to run?

I could try to work on this and would like to hear what are your thoughts on this and any other pointers or features you would like to have.. @sylvchev @jsosulski

[sylvchev]

This is linked with #175, as the best course is to use the MOABB script to launch the evaluation.

Regarding the specific points you raised:

1. The pipelines define on what kind of paradigm they could be applied, see this pipeline for example. The code should be update as MDM could be use for all paradigms
2. This is already taken into account in the dataset methods, that are specific for each one
3. The comparison metric are already defined in the paradigm and called by the evaluation classes. No need to change anything.
4. GH action won't be able to take the charge. We could use a private server (I could give you access to one from my lab) or rely on ebrains/neurohub ressources. As the code should be executed only once to get the number, I think a private server is enough for now.

[Div12345]

Hey @sylvchev , I was running the benchmarking but ran into an issue where the download is trying to happen despite having the datasets downloaded.. Can you please check your gitter, sent the detailed error there.

[Div12345]

Also linking https://github.com/braindecode/braindecode/issues/46 as a follow-up after the basic benchmarking is done.

[Div12345]

@sylvchev when you get the time, if you can check out the recent issues I raised that came from running the benchmark pipeline and possibly the Colab notebook itself which I shared on Gitter, I could work on resolving them. My Colab Pro RAM(25 GB) also crashes for cross-subject BNCI apart from the other things I mentioned in the issues.

[sylvchev]

Possible improvments for global benchmarking sprint:

• [x] progress bar like tqdm, see #240
• [x] make a link towards dataset page on Paper with code, when the dataset is download to invite people reporting their paper and score on this website
• [ ] Make a tutorial for reproducible research that explain the steps
• [ ] Add Connected Paper link in the doc
• [ ] Create a sanity check for dataset, to ensure that they are all correctly loaded and interpreted, as mentionned in #184 and https://github.com/NeuroTechX/moabb/issues/216 .

TODO @sylvchev

• solve https://github.com/NeuroTechX/moabb/issues/153
• solve https://github.com/NeuroTechX/moabb/issues/239

TODO @Div12345

• Open a PR for progress bar
• Make a small example for run.py and notebook, that use only one/two dataset from each paradigm and only two pipelines
• Use incremental approach, to add datasets and pipelines