[FEATURE REQUEST] Interrupted scan to resume where it left off

[Manofdesign]

Hello,

I wanted see if there was a way to resume a scan from where it left off if it was interrupted? If that feature does not exist, it would be really awesome. The scenario I have is an experiment has been running for a few days and made it about 25% in. Unfortunately it interrupted by something that was my fault and I am wishing I could pick up where it left off.

I am running the version pulled from the GitHub link and using the experimental feature to use specific train and validation sets.

[mikkokotila]

Thanks for suggesting this. It is indeed a useful feature, but will require some thinking and testing in terms of implementing it. There are three parts to this:

1. storing in the experiment output the permutation id of each result
2. storing separately the list of all permutation id
3. adding the parameter for turning on "recoverable" mode

1 and 3 are trivial, and maybe 2 could be handled in a way where the permutation id list is stored and then in the end of the experiment there is a cleanup process where it is deleted.

[mikkokotila]

Related with #207

[mikkokotila]

Possibly related with #447