How to resume checkpoint for continuous training?

[ithieund]

Hi @nlpyang , can you give me some suggestion on how to resume the previous checkpoint model for continuous training without having to train the model from beginning at different session?

Thank you very much.

[henrypham67]

use -train_from <model_path> arguement to continue where left

[SabrinaZhuangxx]

After I set this parameter it has no effect，I set the parameter -train_from ../models/model_step_4000.pt. it right？thank you

[henrypham67]

After I set this parameter it has no effect，I set the parameter -train_from ../models/model_step_4000.pt. it right？thank you

Could you provide full log?

[vedantgannu]

Same here. It just loads the data file and then exits

[vedantgannu]

[vedantgannu]

Nvm figured out the issue. Basically need to make sure your -train_steps value is higher than the checkpoint's train step