so-vits-svc-fork icon indicating copy to clipboard operation
so-vits-svc-fork copied to clipboard

Published 20 hours ago • verified publisher icon voicepaw

Training past max. epoch

Open Lordmau5 opened this issue 2 years ago • 2 comments

Describe the bug It appears that there is currently no hard-limit on the max. epoch for the training, meaning it will go past the default 10k epochs (or, 0-9999 rather) image

To Reproduce Train to the limit and let it train further

Additional context I'm not sure if there's a method to limit the max. epoch and have it end training at that point automatically, or if we need to handle that ourselves

Lordmau5 avatar Apr 16 '23 00:04 Lordmau5

Looking at the code there clearly is a max_epoch value set, but somehow it isn't considered by the trainer... I'll look into it a bit more

Lordmau5 avatar Apr 16 '23 09:04 Lordmau5

This actually seems to be related to overriding internal variables for the current epoch. Commenting those out and trying to train a model up to 5 epochs will make it stop after that perfectly fine. image

Lordmau5 avatar Apr 16 '23 16:04 Lordmau5

I've opened a discussion thread over on the Lightning repository as well, forgot to mention that. https://github.com/Lightning-AI/lightning/discussions/17396

I might dig more into the internals to see where it's checking against the max epochs and such

Lordmau5 avatar Apr 18 '23 21:04 Lordmau5

Resolved in 3.10.2

Lordmau5 avatar Apr 19 '23 14:04 Lordmau5