OneTrainer
OneTrainer copied to clipboard
Validation timesteps
Implementing the conclusions of this thread: https://github.com/Nerogar/OneTrainer/issues/772
Summarized:
- Validation on timestep 500 is not ideal, but hardcoded currently
- Choosing validation timesteps from a distribution is not good either, especially for small validation sets
- [X] let the user choose
- There is no meaningful way to average the loss of different timesteps
- [X] report separately to tensorboard
I find it helpful to validate on a high timestep that determines image composition, and something in between - but other best practices might evolve