How to restart training?

[Askejm]

I want to train a bunch of stems. I figured I could do this with the --restart_every argument. So I set it to 58k secs but when it reaches that it just exits How would I make it restart by itself? I want it to make a bunch of different stems that are 1k kimg. Windows 10, rtx 3070

[woctezuma]

when it reaches that it just exits

https://github.com/autonomousvision/stylegan_xl/blob/223430d0adb5eabee9f7e38e0bce73fc4b1818f6/training/training_loop.py#L133

[Askejm]

Could you elaborate on what you mean with that?

[woctezuma]

The argument of --restart_every is the time interval in seconds to exit code. So this works as intended.

What is stopping you from starting the program again after it exited? Is it supposed to automatically restart?

https://github.com/autonomousvision/stylegan_xl/blob/223430d0adb5eabee9f7e38e0bce73fc4b1818f6/training/training_loop.py#L174-L177

Be careful, because it seems to resume from the last checkpoint when you manually restart the program.

Maybe just set the training duration another way, and start the training from scratch a bunch of times.

[Askejm]

Yeah I noticed it resumes. But would you know a way? I might just open a bunch of anaconda prompts and have them offset by 58k secs. I can't always manually restart it because I'm on holiday

[woctezuma]

I might just open a bunch of anaconda prompts and have them offset by 58k secs.

That is what I would do as well.