:sparkles: RFDA-Pytorch :sparkles:

Official Code for 'Recursive Fusion and Deformable Spatiotemporal Attention for Video Compression Artifact Reduction'

ACM Multimedia 2021 (ACMMM2021) Accepted Paper

Task: Video Quality Enhancement / Video Compression Artifact Reduction

The code will be gradually open source!

Open Source Scheduler

1 Release RF and DSTA core code within one month after camera ready [Done]

2 Release test code and models at five QP [Done]

3 Release train code (you know, in a mass ) [Done]

:e-mail: Contact :e-mail:

Feel free to contact me if you have any problems! [email protected]

1. Pre-request

1.1. Environment

• Ubuntu 20.04/18.04
• CUDA 10.1
• PyTorch 1.6
• Packages: tqdm, lmdb, pyyaml, opencv-python, scikit-image

Suppose that you have installed CUDA 10.1, then:

$ git clone --depth=1 https://github.com/zhaominyiz/RFDA-PyTorch 
$ cd RFDA-PyTorch/
$ conda create -n video python=3.7 -y
$ conda activate video
$ python -m pip install torch==1.6.0+cu101 torchvision==0.7.0+cu101 -f https://download.pytorch.org/whl/torch_stable.html
$ python -m pip install tqdm lmdb pyyaml opencv-python scikit-image

1.2. DCNv2

Build DCNv2.

$ cd ops/dcn/
$ bash build.sh

(Optional) Check if DCNv2 works.

$ python simple_check.py

The DCNv2 source files here is different from the open-sourced version due to incompatibility. [issue]

:fire: 2. Train :fire:

Build the dataset! check create_lmdb_mfqe_rf.py

Train RF only!

Train RFDA! check train_rf_ft.py!

:zap: 3. Test :zap:

3.1 Test MFQE 2.0 dataset

Please build the MFQE 2.0 dataset first (See Here), then run test_yuv_RF.py.

More instructions will coming soon!

$ python test_yuv_RF.py --opt_path config/****.yml

3.2 Test your own video clip

For yuv videos, you may refer to test_one_video_yuv_RF.py.

$ python test_one_video_yuv_RF.py --opt_path config/****.yml

For rgb videos, we will update new model and codes soon.

:seedling: 3.3 Pretrain models :seedling:

RFDAQP22,27,32,37,42(trained on YUV space): BaiduDisk (RFDA) GoogleDisk

BRFQP37(trained on YUV space): ToBeDone!

:beers: Results :beers:

Comparison with State of the Art Methods

Speed and parameter size comparison

:sparkling_heart: Visualization Video Demo :sparkling_heart:

https://user-images.githubusercontent.com/43022408/127981531-f98ce54c-7b9d-4e12-903b-9b4bb0baf1f5.mp4

:wink: Related Works :wink:

· Boosting the performance of video compression artifact reduction with reference frame proposals and frequency domain information [Paper]

· Non-local convlstm for video compression artifact reduction [Paper] [Code]

:satisfied: Citation :satisfied:

If you find this project is useful for your research, please cite:

@inproceedings{zhao2021recursive,
  title={Recursive Fusion and Deformable Spatiotemporal Attention for Video Compression Artifact Reduction},
  author={Zhao, Minyi and Xu, Yi and Zhou, Shuigeng},
  booktitle={Proceedings of the 29th ACM International Conference on Multimedia},
  pages={5646--5654},
  year={2021}
}

:thumbsup: Special Thanks :thumbsup:

Our framework is based on STDF-Pytoch. Thank RyanXingQL for his work!