Adding features (but may break old models that you have)

[mtanana]

I've been working on some revamps for this project that make it much easier to restart models that have stopped from unexpected events (i.e. computer crashes, or you were preempted on some shared resources) . I think this changes might make it tricky to load any saved models you have right now.

So I'm going to hold off on the commit for a few weeks to make sure people are aware.

But be warned!

[kenkit]

did you finish adding the features. I'd like to see what's changed ?

[mtanana]

I did commit some of this work, and restarting models worked more or less. I think there still may be some issues with restarting models. (I think the main issue is that if you start in the middle of an epoch, the loss never seems to be right). But pre-training and restarting with a new dataset seemed to work. Let me know what you find. I might do a new commit sometime soon if you're using the framework.