ganhacks
ganhacks copied to clipboard
soumith
G loss increase, what is this mean?
Hi, I am training a conditional GAN. At the beginning, both G and D loss decrease, but around 200 epoch, G loss start to increase from 1 to 3, and the image quality seems to stop improve.
Any ideas? Thank you in advance.
Ok this is for an unconditional boilerplate GAN. What I found for loss increase in G was that: a) it was accompanied by a decrease in D loss. Essentially G starts diverging. b) image quality improved subtle but it did.
I think the discriminator got too strong relative to the generator. Beyond this point, the generator finds it almost impossible to fool the discriminator, hence the increase in it's loss. I'm facing a similar problem.
I am facing similar problem while training infoGAN on svhn dataset. Any suggestion on how to overcome this?
I am also facing similar problem with Infogan on a different dataset. Any suggestions?
In my experience, when d loss decrease to a small value (0.1 to 0.2) and g loss increase to a high value (2 to 3), it means the training finish as generator cannot be further improved.
Bit if the d loss decrease to a small value in just few epochs, it means the training fail, and you may need to check the network architecture.
I have the same problem. When i train GAN, i expect that in the end of training(some infinite moment) G will always fool D. But in fact I am faced with the following problem: at the beginning of the process, G learns correctly - it learns to produce good images with nessesary conditions. But after some moment G starts to diverge. In the end, G could produce only random noise. Why this happens?
Probably, the problem is that the discriminator overfit. One of the reasons leading to this is following thing: discriminator may "notice" that images from true distribution is a matrix of numbers of the form n/255. So, adding gaussian noise to the input images may help to avoid the problem. It helps in my case.
Label switching has also helped for me.
Two updates of discriminator with real_label = 1, fake_label=0 and one update with real_label=0 and fake_label=1.
This is followed by one generator update with real_label = 1 and fake_label = 0.
@Howie-hxu and @EvgenyZamyatin : I saw that adding Gaussian noise in the discriminator helped in your case. I have few questions :
- What did you keep as mean and variance of the Gaussian noise
- Did you apply Gaussian noise in each layer of the discriminator ? Lets say, if we are using DCGAN architecture ?
- Do you apply the noise layer after the activation or before doing convolution?
- If suppose, I am using Tensorflow, how do you implement that?
Keenly waiting for your help !!! Thanks, Avisek
I have used the idea of instance noise described here. My experiment was to add the Gaussian noise only to the input tensor of the discriminator. It was zero mean and its standard deviation ranges from 0.1 to 0 (i.e. decaying with each mini batch iteration). This has improved the result much better for the MNIST dataset.
Hello. I am training CycleGAN and my loss looks like that attached picture. The discriminator loss decreases but the generator loss fluctuates. I do not quite understand the reasons. Are there anyone have any suggestions? Thanks
Adding noise to input seems to help. To be specific, i am implementing with tensorflow by adding：
input = input + tf.random_normal(shape=tf.shape(input), mean=0.0, stddev=0.1, dtype=tf.float32)
I agree here that by adding noise to the discriminator loss model function, it does improves your generator loss to decrease. @ahmed-fau suggested very good tips.
Hi, I tried what you guys did, adding gaussian noise to the input of the discriminator. It does improve the graph, but however the test images generated by the generator comes out as noise as well. (previously I have relatively ok images, but my generator loss fn was going up).
Thoughts?
Hi, I tried what you guys did, adding gaussian noise to the input of the discriminator. It does improve the graph, but however the test images generated by the generator comes out as noise as well. (previously I have relatively ok images, but my generator loss fn was going up).
Thoughts?
Did you also have decay of the noise after a while?
I am facing a similar problem while using WGAN-GP. The generator initially produces good results but seems to diverge after some time and the discriminator loss suddenly dips and becomes very powerful making the generator output random noise. What can be done instead of label smoothing since I am using WGAN?
@aradhyamathur could try adding a penalty loss term for the discriminator output magnitude, similarly to https://github.com/tkarras/progressive_growing_of_gans
This helps to prevent a training dynamic where the models engage in a "magnitudes race" and eventually lose any meaningful learning signals.
@phamnam95 That looks like typical cycleGAN loss. What is your batchsize? If it is one or 2, there will be lots of fluctuations in your objective function. Seen it before, looks pretty normal to me.
@LukasMosser My batch size is 1. After adding some more constraints such as the identity loss, the self-distance loss, and I also semi-supervised cyclegan by using pair images, I can get generator decreases but not fast, instead it decreases very slowly and seems that after 200 epochs, the trend is still decreasing. And the discriminator decreases until it reaches a certain of epochs, it starts fluctuate. What do you think will be good? What batch size do you think is appropriate?
Hi, I tried what you guys did, adding gaussian noise to the input of the discriminator. It does improve the graph, but however the test images generated by the generator comes out as noise as well. (previously I have relatively ok images, but my generator loss fn was going up).
Thoughts?
Hi, I have the same problem? Can you solve it?great thanks
@phamnam95 I think batch size = 1 is ok, I'm not really worried about the fluctuation it just means you'll have to pick one with appropriate generator loss and not one where it seemingly diverged.