Very noisy training curves when trying to fine-tune with paired data, what am I doing wrong?

[roey1rg]

I'm trying to train Real-ESRGAN with 400 images. The training curves are very noisy, especially the perceptual loss (l_g_precep). I've tried to use various different learning rates. I'm using 4 GPUs with 5 images on each GPU

here is the config file I use:

[JensDA]

The logged losses are only averages of the mini batch of the one iteration at the time of logging. They are not averaged over the iterations between logs, so they are more like regular samples and thus are prone to noise.

I've implemented averaging in my own code. This is the standard logging: And this the averaged logging:

Note: I'm using a low batch size of 8 on a single GPU.

[roey1rg]

Thanks @JensDA, good point

[mindFlayer123]

@roey1rg where to find the log file ?

[HoliSimo]

Can you explain better the concept? Which is the script that you used to obtain that plot?

[Tsouuu]

I'm trying to train Real-ESRGAN with 400 images. The training curves are very noisy, especially the perceptual loss (l_g_precep). I've tried to use various different learning rates. I'm using 4 GPUs with 5 images on each GPU

here is the config file I use:

Can you explain how to obtain that plot? plz

[FengMu1995]

I'm trying to train Real-ESRGAN with 400 images. The training curves are very noisy, especially the perceptual loss (l_g_precep). I've tried to use various different learning rates. I'm using 4 GPUs with 5 images on each GPU

here is the config file I use:

My l_g_percep loss have been increasing gradually, do you know why it haven't decreased?