Easy training restarts

[lubbersnick]

@yingwaili requests that we do the training in such away that restarting the model is basically automated. Maybe we can have some function to write a restart script in the model directory or something. Or a function that restarts training and takes as input only the directory of the checkpoint.

Items of difficulty

• restarting databases is a bit fragile - may need to be customized somehow.