Automated tests for egs

[jonashaag]

🚀 Feature

Add automated tests for egs

Motivation

Currently these are entirely untested, which means any changes done to the run.sh scripts etc. must be tested manually. When making changes, testing all the egs is a heavy burden on the developer; some of them even have commercial licenses that not everyone may have access to.

It also means that refactoring egs code takes much more time than could be.

What you'd like

Add a "CI" mode to each eg:

• Run mini dataset preprocessing (maybe with autogenerated garbage source files so you do not need the actual dataset?)
• Run a mini epoch with a few batches
• Run eval

Automatically run this in CI for each eg.

Feedback welcome! It know it's a lot of work but we could easily split it into small steps.

[groadabike]

Hi, I am not familiar with the CI mode and what would imply in the egs, but:

Run mini dataset preprocessing (maybe with autogenerated garbage source files so you do not need the actual dataset?)

For the dataset preprocessing, it could be a simple dataset that generates random tensors based on some parameters like the number of channels, the number of targets, sample rate and the __getitem__ return (mixture and sources or mixture, sources and noise).

Run a mini epoch with a few batches

We can set the PL trainer parameter limit_train_batches and I believe it can be combined with a small number of max_epoch. Currently, several models in Asteroid have the train_percent_check parameter set to 1.0, but this parameter was deprecated in version 0.8.0 and will be removed in 0.10.0.

[mpariente]

Thanks for suggesting potential improvements, it's great ! There is also the fast_dev_run in lightning which would be useful. Let's go how this goes here #180 before trying to make CI on recipes, it might be much easier afterwards