[Feature] Continue fine-tuning after the training has been interrupted

[hschaeufler]

Feature request as discussed with @awni in discussion 997.

I often had the problem that I had to interrupt the training and then had to start the training all over again. With --resume-adapter-file the training for an existing adapter can be continued. The problem is that the training is not continued where it stopped before. You can reduce the steps accordingly and shuffle the data set. The problem is that I use cosine_decay as learning-rate scheduler with certain values for warmup (309 steps), warmup-init (0) etc. This means that the training starts again at the initial values and do not follow the intended curve. It would be a cool feature if it were possible to take the number of steps already performed from the name of the adapter file or pass it along. MLX could then calculate the continuation point and continue the training with the appropriate learning rate and record from the training set.

[aperepel]

I would make it a more generic request. I don't seem to see an obvious way to resume training from a checkpoint (either interrupted or just requiring additional iterations). A number of online search results mention non-existent resume checkpoint flags.