Chapter 5: Understanding the cGAN's loss functions

[tobiasschweizer]

Hi @markhliu,

I am not sure I understand the implementation of the critic's loss function in chapter 05:

loss_critic=(-(torch.mean(critic_real) - torch.mean(critic_fake)) + 10 * gp)

In the book on page 107, the formula is: critic_value(fake) - critic_value(real) + weight * GradientPenalty.

The values of critic_value(fake) and critic_value(real) are inverted by -1 and the overall result in parenthesis has a negative sign. This is also the case for the generator's loss: loss_gen = -torch.mean(gen_fake).

I understand that unlike in chapter 04, the discriminator / critic does not produce a value between 0 and 1 from which we can calculate the loss (namely 1 - prediction for real images from the training dataset and simply prediction for those made by the generator).
Instead, the critic produces a higher value for images classified as "more real" and a lower (negative) value classified as "more fake".

Thanks a lot for your feedback.

[tobiasschweizer]

I found some possible explanations here: https://machinelearningmastery.com/how-to-implement-wasserstein-loss-for-generative-adversarial-networks/

In the case of the generator, a larger score from the critic will result in a smaller loss for the generator, encouraging the critic to output larger scores for fake images. For example, an average score of 10 becomes -10, an average score of 50 becomes -50, which is smaller, and so on.

I think this is consistent with: loss_gen = -torch.mean(gen_fake)

In the case of the critic, a larger score for real images results in a larger resulting loss for the critic, penalizing the model. This encourages the critic to output smaller scores for real images. For example, an average score of 20 for real images and 50 for fake images results in a loss of -30; an average score of 10 for real images and 50 for fake images results in a loss of -40, which is better, and so on.

In the formula used in the book, the resulting loss has a negative sign, so the statement would be inverted I think: loss_critic=-(torch.mean(critic_real) - torch.mean(critic_fake))

[markhliu]

Hi, @tobiasschweizer: Good question. It all starts with the critic() function, which is defined on page 107 as

critic=Critic(img_channels+2,features).to(device)

The critic takes in an image, and produces a score between -infinity and infinity: the critic's job is to give a real image a high score (ideally infinity) and a fake image a low score (ideally -infinity).

On page 111, you'll see that we define critic_real = critic(real).reshape(-1) critic_fake = critic(fake).reshape(-1)

that is, critic_real is the critic's score assigned to a real image, and critic_fake is the critic's score assigned to the fake image. By minimizing loss_critic=-(torch.mean(critic_real) - torch.mean(critic_fake)), the critic is minimizing critic_fake and maximizing critic_real.

For the generator, we define gen_fake = critic(fake).reshape(-1), which is the critic's score on a piece of fake image. The generator is minimizing loss_gen = -torch.mean(gen_fake), which is to say the generator is trying to maximize the score produced by the critic on the fake image.

Hope this helps.

BTW, all the formulas in the book related to our discussion here are correct.

[tobiasschweizer]

Hi @markhliu,

Thanks for your explanation.

If I may, let me summarize the critic's loss: The higher the score for the real image and the lower the score for the fake image, the bigger the resulting difference (a large positive number) becomes. The loss is this number with a negative sign, hence the bigger the difference between real and fake scores, the lower the loss.