on_exception hook event

[miguelvr]

🚀 Feature

It would be nice to have the ability to process failed experiments by applying an action over an exception.

Pitch

This would be useful to trigger edge cases that might happen such as interrupted training procedures and saving the current state of the model, for example.

Another thing that comes to mind is a hook that sends you a warning of a failed training experiment, via some webhook.

There are multiple scenarios where this would be useful, for multiple reasons.

I would be happy to help, as always :)

[vreis]

Sounds good to me!

This seems pretty simple to implement within ClassyTrainer, let me know if you need any pointers.