vqvae reconstructed images are too blurry

[lelechen63]

trafficstars

Hi there, thanks for the excellent work. I am trying to use the vqvae to demonstrate face images. But the results are blurry. Is there any suggestion for this? For example, can I add perceptual loss or gan loss to regularize this. Below are the testing results.

[rosinality]

Yes, perceptual loss will be easy to try. But I think you can get quite nice results with MSE loss alone.

[SURABHI-GUPTA]

@lelechen63 the image you have attached above, is the top layer dataset and bottom is its reconstruction ? and also what are the steps to do that.

[kklemon]

The reconstructed images will generally have fewer details than the original one as VQ-VAE is essentially lossy compression. Possible options to mitigate this would be indeed for example perceptual losses. A GAN discriminator trained jointly with the model could also be considered as such.

Another simple approach that could help is to treat the pixel colours as discrete classes and replace MSE by cross entropy. MSE often times leads to such blurry results when used as reconstruction loss in image tasks due to the fact, that colours very close to the target will lead to a sufficiently low loss. CE should help here, as any slight deviation from the true colour will lead to a high loss, but I haven't tried that myself so far.