OneTrainer icon indicating copy to clipboard operation
OneTrainer copied to clipboard

Validation timesteps

Open dxqb opened this issue 8 months ago • 0 comments

Implementing the conclusions of this thread: https://github.com/Nerogar/OneTrainer/issues/772

Summarized:

  • Validation on timestep 500 is not ideal, but hardcoded currently
  • Choosing validation timesteps from a distribution is not good either, especially for small validation sets
  • [X] let the user choose
  • There is no meaningful way to average the loss of different timesteps
  • [X] report separately to tensorboard

I find it helpful to validate on a high timestep that determines image composition, and something in between - but other best practices might evolve

grafik

grafik

dxqb avatar May 01 '25 10:05 dxqb