Deep3D Stabilizer (CVPR 2021)

This repository contains the pytorch implementations of the Deep3D Stabilizer.

3D Video Stabilization with Depth Estimation by CNN-based Optimization
Yao-Chih Lee, Kuan-Wei Tseng, Yu-Ta Chen, Chien-Cheng Chen, Chu-Song Chen, Yi-Ping Hung
CVPR 2021 [Paper] [Project Page] [Video]

Contribution

• The first 3D-based CNN method for video stabilization without training data.
• Handle parallax effect more properly leveraging 3D motion model.
• Allow users to manipulate the stability of a video efficiently.

Setup

• Main program

  • Python3.5+ and Pytorch 1.4.0+
  • Other dependencies
    apt-get install ffmpeg
pip3 install opencv-python scipy tqdm path imageio scikit-image pypng

• PWC-Net for getting optical flow as preprocessing

  • Require Pytorch 0.2.0 with python2.7 & cuda 8 (please refer to the official PWC-Net and its issue)

Running

To test your own video to be stabilized, run the commands below. The stabilized video will be saved in outputs/test by default.

python3 geometry_optimizer.py [your-video-path] [--name default=test]
python3 rectify.py [your-video-path] [--name same-as-above] [--stability default=12]

Stability Manipulation

If you have run geometry_optimizer.py for the video, you may run rectify.py for the same video multiple times with different --stability to manipulate the stability efficiently.

Citation

@InProceedings{Lee_2021_CVPR
    author    = {Lee, Yao-Chih and Tseng, Kuan-Wei and Chen, Yu-Ta and Chen, Chien-Cheng and Chen, Chu-Song and Hung, Yi-Ping},
    title     = {3D Video Stabilization with Depth Estimation by CNN-based Optimization},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2021},
    pages     = {10621-10630}
}

License

The provided implementation is strictly for academic purposes only.

Acknowledgement

We thank the authors for releasing SC-SfMLearner, monodepth2, and PWC-Net.