can you/how can you continue training?

[aelbialy-tbox]

Hi guys,

Is it possible to continue training from where it left off? I know it is a silly question. I know that it saves checkpoints after each epoch for generating the audio. Is there a way carry on training after it has been shutdown?

Thank you! Starting to get alright results too

[chief7]

If you didn't mess with the code - it should continue training from the latest saved checkpoint.

[aelbialy-tbox]

When I type in python3 train.py and I print the current epoch it starts again from 1. Is there a way to check that it continues training?

[chief7]

As far as I remember, epochs aren't saved with the model. It's just a range that the code iterates over. You may want to check with tensorboard and see if your graphs start from scratch (they shouldn't)

[zuoxiang95]

@ahmed-tbox it would continue training from the latest saved checkpoint. you can see Supervisor in tensorflow . here is the guide.

[aelbialy-tbox]

Is it possible to continue training from another model that was run on a different machine? I used a machine that ran on 1 GPU with train.py and on the new machine I am using 2 GPUs with train_multi_gpus.py will that affect anything? What files do I need to transfer over if I need to?