Any suggestion to modify the arch based on the gan training result?

[3zhang]

I'm training a photo swin_unet_2x model using gan. I use a cosine lr scheduler with init lr = 1e-5. After some tries I found that the discriminator loss fluctuated around 0.8 (the threshold for generator training to begin), so I increased discriminator lr to 5e-5. And this is the result.

After ~160 epochs the gen loss start to increase which trade off with the decrease of discr loss, which is not good. My guess is that maybe the gen model is undergoing some underfitting? So could you give me some suggestion to modify the arch to make the model more complex? Or should I try with a different arch?