moabb icon indicating copy to clipboard operation
moabb copied to clipboard

Save results immediately

Open PierreGtch opened this issue 1 year ago • 4 comments

Closes #419

Instead of waiting for the evaluation of a whole dataset to finish before saving the results, we could save them as soon as they are computed. Beneficial in case the evaluation fails in the middle.

Pending questions:

  • [ ] Can that create issues in case of parallel access to the hdf5 file?
  • [ ] What is the computational overhead due to accessing the hdf5 more often?

PierreGtch avatar Jul 11 '23 13:07 PierreGtch

@bruAristimunha the code change should be quite simple I think (see my commits) but I don't have time now to test the parallel case and benchmark in the standard case.

PierreGtch avatar Jul 11 '23 13:07 PierreGtch

Okay, but now we lost all the results in the tutorials with this modification. We will need to debug this issue more to get it all green.

bruAristimunha avatar Jul 11 '23 14:07 bruAristimunha

I wonder if you could focus a little more on this problem in the CI, and then we could create a more specific test to work on the parallelization and overhead access issue. @PierreGtch

bruAristimunha avatar Jul 11 '23 14:07 bruAristimunha

I see it fails while testing the benchmark function. By default, benchmark uses -1 jobs. I think it might already be an issue with the parallel access to the file...

When I tested in local, the tests/evaluations.py were working fine

PierreGtch avatar Jul 11 '23 15:07 PierreGtch