[Feat]: sampling limiter

[yggdrasil75]

Describe your use-case.

I prefer to use 1 epoch between samples when I have a lot of images (ie: finetune) but when I do few examples (lora) every epoch goes from 1 minute to 5 due to sample rate (dont know why its so slow, that might need a bug report. only 4 samples requested and on a 3090 - but the sampling takes 8 seconds per image, but with 45 seconds in between. is it reloading the model between each sample?)

What would you like to see as a solution?

I want to say "dont sample more often than every 10 minutes" so its every epoch but when an epoch is less than 10 minutes, it will skip it. either as an option to wait until time runs out and pause then sample, or wait until next cycle and generate.

Have you considered alternatives? List them here.

Instead, I could use a dedicated sampling device, ie: generate samples on my 1080 ti while the 3090 continues training