continue training model

[tiger-bug]

Good morning,

First of all thank you for your work. I was wondering if there was a way to easily continue training from a checkpoint. Feel free to close if this question has already been answered, I just didn't see an easy way to do this. I have a dataset I'm training with but I'm getting new data and would like to continue training from the latest checkpoint instead of restarting the whole thing. Thank you!

[HuguesTHOMAS]

Hi @tiger-bug,

Actually it is not that easy to do. With a few simple modifications to the current implementation, you can restart a new training with weights from a given checkpoint as a starting point, and you could also choose the learning rate you restart from to be equal to the one you stopped at. But it would require a serious rework to actually resume a training that was stopped with everything behaving as if the training did not stop.

In you case I guess the first simple solution would be enough, and I am not sure but I think I did answer a question about loading pretrained weights for training somewhere in the issues. If you don't find it, and cant figure out how to do it, feel free to ask again here, and I will try to help you.

Best, Hugues

[tiger-bug]

Actually the first solution should be sufficient. I will look in the issues to see how to do this. Thank you! If I find a solution I will post it here if you don't mind. Maybe it will help somebody else, especially if they are using their own dataset.