Support model checkpoints from previous training runs

[cosmicBboy]

As a ML engineer, I want to be able to continue training a model from a previous model.remote_train run (with equivalent semantics for a local model.train call) so that I can pick up training from where I left off.