dSRVAE
dSRVAE copied to clipboard
Unsupervised Real Image Super-Resolution via Variational AutoEncoder in CVPR2020
dSRVAE (Generative Variational AutoEncoder for Real Image Super-Resolution)
By Zhi-Song Liu, Li-Wen Wang, Chu-Tak Li, Marie-Paule Cani and Wan-Chi Siu
This repo only provides simple testing codes, pretrained models and the network strategy demo.
We propose a joint image denoising and Super-Resolution model by using generative Variational AutoEncoder (dSRVAE)
We participate CVPRW NTIRE2020 Real Image Super-Resolution Challenge
Please check our paper
BibTex
@InProceedings{Liu2020dsrvae,
author = {Zhi-Song Liu, Wan-Chi Siu and Li-Wen Wang and Chu-Tak Li and Marie-Paule Cani and Yui-Lam Chan},
title = {Unsupervised Real Image Super-Resolution via Generative Variational AutoEncoder},
booktitle = {IEEE International Conference on Computer Vision and Pattern Recognition Workshop(CVPRW)},
month = {June},
year = {2020}
}
For proposed dSRVAE model, we claim the following points:
• First working on using Variational AutoEncoder for image denoising.
• Then the Super-Resolution Sub-Network (SRSN) is attached as a small overhead to the DAE which forms the proposed dSRVAE to output super-resolved images.
Dependencies
Python > 3.0
OpenCV library
Pytorch > 1.0
NVIDIA GPU + CUDA
pytorch-gan-metrics
Complete Architecture
The complete architecture is shown as follows,
Implementation
1. Quick testing
- Download pre-trained models from
https://drive.google.com/open?id=1SUZGE04vw5_yDYiw6PJ4sbHAOIEV6TJ7
and copy them to the folder "models"
- Copy your image to folder "Test" and run
$ python test.py
The SR images will be in folder "Result" 3. For self-ensemble, run
$ python test_enhance.py
- GAN feature evaluation
# download statistics.npz from http://bioinf.jku.at/research/ttur/
from pytorch_gan_metrics import get_inception_score, get_fid
images = ... # [N, 3, H, W] normalized to [0, 1]
IS, IS_std = get_inception_score(images) # Inception Score
FID = get_fid(images, 'path/to/statistics.npz') # Frechet Inception Distance
2. Testing for NTIRE 20202
s1. Testing images on NTIRE2020 Real World Super-Resolution Challenge - Track 1: Image Processing artifacts can be downloaded from the following link:
https://drive.google.com/open?id=10ZutE-0idGFW0KUyfZ5-2aVSiA-1qUCV
s2. Testing images on NTIRE2020 Real World Super-Resolution Challenge - Track 2: Smartphone Images can be downloaded from the following link:
https://drive.google.com/open?id=1_R4kRO_029g-HNAzPobo4-xwp86bMZLW
s3. Validation images on NTIRE2020 Real World Super-Resolution Challenge - Track 1 and Track 2 can be downloaded from the following link:
https://drive.google.com/open?id=1nKEJ4N2V-0NFicfJxm8AJqsjXoGMYjMp
3. Training
s1. Download the training images from NTIRE2020.
https://competitions.codalab.org/competitions/22220#learn_the_details
s2. Start training on Pytorch
- Train the Denoising VAE by running
$ python main_denoiser.py
- Train the super-resolution SRSN overhead by running
$ python main_GAN.py
Partial image visual comparison
1. Visualization comparison
Results on 4x image SR on Track 1 dataset
Reference
You may check our newly work on General image super-resolution using VAE
You may also check our work on Reference based face SR using VAE
You may also check our work on Reference based General image SR using VAE
Special thanks to the contributions of Jakub M. Tomczak for their VAE with a VampPrior on KL loss calculation.