Code Loss Formulation Vs in Paper

[Andrew-Brown1]

Hi,

Thanks for the awesome repo! I had a couple of questions about the implementation of the loss function.

In the paper you multiply the entire GAN loss by the adaptive weight, lambda. The adaptive loss is a function of the entire GAN loss (L_GAN). Two questions:

1. In the code, only the generator loss is multiplied by the adaptive weight (https://github.com/CompVis/taming-transformers/blob/9d17ea64b820f7633ea6b8823e1f78729447cb57/taming/modules/losses/vqperceptual.py#L107).

• For me, the adaptive weight gets fairly high and so "loss" is far higher than "d_loss". I think this is why some people are not seeing the discriminator loss decrease during training e.g. https://github.com/CompVis/taming-transformers/issues/44)

2. In the code, the adaptive weight, lambda, is only a function of the generator loss (whereas I thought the GAN loss was a function of both the generator and discriminator loss).

Could you offer any advice here?

Thank you