table-transformer icon indicating copy to clipboard operation
table-transformer copied to clipboard

Can I resume the training if I stop it?

Open linkstatic12 opened this issue 1 year ago • 2 comments

If i run my PC for the epoch 0 training, stop the training and then change the epoch value to 1 in the json and start the training again does it take the weights from the epoch 0?

linkstatic12 avatar Aug 11 '23 17:08 linkstatic12

Hi,

The model weights and current optimizer state is saved after every epoch. If epoch N completes and then training is interrupted after that, you can resume training at epoch N+1 by starting training with the flag --model_load_path /path/to/model.pth.

Let me know if that answers your question.

Best, Brandon

bsmock avatar Sep 01 '23 00:09 bsmock

@bsmock It did. I was training the fintable dataset but you guys already did it.

linkstatic12 avatar Sep 11 '23 13:09 linkstatic12