vsrnet_pytorch
vsrnet_pytorch copied to clipboard
PyTorch implementation of paper "Video super-resolution with convolutional neural networks"
VSRNet
A PyTorch implementation of VSRNet [Video super-resolution with convolutional neural networks] (https://ieeexplore.ieee.org/abstract/document/7444187/)
Requirements
- Anaconda
- PyTorch
conda install pytorch torchvision -c soumith
conda install pytorch torchvision cuda80 -c soumith # install it if you have installed cuda 8.0
- PyTorchNet
pip install git+https://github.com/pytorch/tnt.git@master
- tqdm
pip install tqdm
- opencv
conda install -c conda-forge opencv
- tensorboard_logger
pip install tensorboard_logger
- h5py
conda install h5py
Datasets
Train, Val, Test Video Dataset
The train and val datasets are sampled from CDVL dataset. We choose this dataset because we want to extend single frame based SRCNN to multi frame based VSRNet.
Train dataset (uf_X, X=2,3,4) is composed of multiple h5 files:
- Data_CDVL_LR_uf_4_ps_72_fn_5_tpn_225000.h5: patches sampled from LR frames, 225243x5x(72/X)x(72/X)
- Data_CDVL_HR_uf_4_ps_72_fn_5_tpn_225000.h5: patches sampled from HR frames, 225243x5x72x72
- Data_CDVL_LR_Bic_uf_4_ps_72_fn_5_tpn_225000.h5: patches sampled from Matlab bicubic interpolation upscaled frames, 225243x5x72x72
- Data_CDVL_LR_Bic_MC_uf_4_ps_72_fn_5_tpn_225000.h5: patches sampled from Matlab bicubic interpolation upscaled and optical flow motion compensated frames, 225243x5x72x72
Val dataset (uf_X, X=2,3,4) is composed of multiple h5 files:
- Data_CDVL_LR_uf_4_ps_72_fn_5_tpn_45000.h5: patches sampled from LR frames, 45159x5x(72/X)x(72/X)
- Data_CDVL_HR_uf_4_ps_72_fn_5_tpn_45000.h5: patches sampled from HR frames, 45159x5x72x72
- Data_CDVL_LR_Bic_uf_4_ps_72_fn_5_tpn_45000.h5: patches sampled from Matlab bicubic interpolation upscaled frames, 45159x5x72x72
- Data_CDVL_LR_Bic_MC_uf_4_ps_72_fn_5_tpn_45000.h5: patches sampled from Matlab bicubic interpolation upscaled and optical flow motion compensated frames, 45159x5x72x72
Test dataset (uf_X, X=2,3,4) is composed of multiple h5 folders:
- LR: LR frames of a scene, (1920/X)x(1080/X)x5xframe_number
- HR: HR frames of a scene, 1920x1080x1xframe_number
- LR_Bic: Matlab Bicubic upscaled LR frames of a scene, 1920x1080x5xframe_number
- LR_Bic_MC: Matlab Bicubic upscaled and Celiu optical flow motion compensated LR frames of a scene, 1920x1080x5xframe_number
- LR_MC: Celiu optical flow motion compensated LR frames of a scene, (1920/X)x(1080/X)x5xframe_number
Download the pre-processed h5 files from Dropbox here or Baidu Yun here with code: apkw , and then setup the path in the codes.
If you want to prepare your own train/val/test h5 files, please check the Matlab scripts in the matlab_pre_processing folder. Notice that you would need to install Celiu optical flow to run the Matlab code.
Test Image Dataset
The test image dataset are sampled from | Set 5 | Bevilacqua et al. BMVC 2012 | Set 14 | Zeyde et al. LNCS 2010 | BSD 100 | Martin et al. ICCV 2001 | Sun-Hays 80 | Sun and Hays ICCP 2012 | Urban 100 | Huang et al. CVPR 2015. Download the preprocessed H5 files from here, and then setup the path in test_image.py file.
Usage
Train SRCNN
SRCNN need to be trained to initialize VSRNet
python train_SRCNN.py
optional arguments:
--upscale_factor super resolution upscale factor [default value is 4]
--num_epochs super resolution epochs number [default value is 800]
Train VSRNet
python train_VSRNet.py
optional arguments:
--upscale_factor super resolution upscale factor [default value is 4]
--num_epochs super resolution epochs number [default value is 400]
Test Image
python test_image_SRCNN.py
optional arguments:
--upscale_factor super resolution upscale factor [default value is 3]
--model_name super resolution model name [default value is epochs_SRCNN/model_epoch_800.pth]
The output high resolution images are on results
directory.
Test CDVL Video (preprocessed H5 files) using SRCNN
python test_video_CDVL_SRCNN.py
optional arguments:
--upscale_factor super resolution upscale factor [default value is 3]
--model_name SRCNN model name [default value is epochs_SRCNN/model_epoch_800.pth]
--is_real_time whether to save results into video file or not [default value is False]
--delay_time display delay time [default value is 1]
The output high resolution images are on results
directory.
Test CDVL Video (preprocessed H5 files) using VSRNet
python test_video_CDVL_VSRNet.py
optional arguments:
--upscale_factor super resolution upscale factor [default value is 3]
--vsrnet_model_name VSRNet model name [default value is epochs_SRCNN/model_epoch_800.pth]
--srcnn_model_name SRCNN model name [default value is epochs_SRCNN/model_epoch_800.pth]
--is_real_time whether to save results into video file or not [default value is False]
--delay_time display delay time [default value is 1]
The output high resolution images are on results
directory.