Gradio: training Pause/Resume support

[kagekiyo7]

With the Colab free plan, the GPU are stopped after a few hours of use. It will be available again after 12 hours. I would like a feature to pause training, save a temporary file to google drive, and resume in the middle after reboot the rvc.

[KillauraHacks]

+1 would be a cool feature to have like a pause / stop training button so I can try out my model without having to close out of the infer-web window and reopen it just to stop the training so I can inference

[Eipckz]

I +1 this too. Additionally, I would appreciate the inclusion of this feature in the standard build. I have a particular interest in periodically evaluating my voice models, approximately every 50 epochs. However, due to the excessive consumption of VRAM during training, my graphics card is unable to simultaneously handle both training and voice evaluation processes. Moreover, I have encountered difficulties restarting the training if I close the program. Therefore, I kindly request guidance on how to successfully restart the training process in such cases.

[Mshriver2]

please add pause and resume

[Frontesque]

This would be useful

[omensight]

Yes, It would be nice to have something like this.

[smfreeze]

Agreed

[omensight]

You can use an already trained model in future training, I guess this solves this problem, you can train 20-50 epochs at a time and use the G and D for a future training.

[AIhasArrived]

Hello, what is or where is this G or D @omensight ? How to resume a training??? THanks? Do you have to save states?

[KillauraHacks]

You can use an already trained model in future training, I guess this solves this problem, you can train 20-50 epochs at a time and use the G and D for a future training.

No it doesn't really solve the problem should add a way to pause and save the current g and d then resume training later with that inconvenient to have to wait for the next checkpoint to test

[AIhasArrived]

I dont even how to use a checkpoint to continue training.. also i dont what it means (g and d) (I am learning AI alone)

[github-actions[bot]]

This issue was closed because it has been inactive for 15 days since being marked as stale.