FSRT: Facial Scene Representation Transformer for Face Reenactment (CVPR 2024)

Official GitHub repository for FSRT: Facial Scene Representation Transformer for Face Reenactment from Factorized Appearance, Head-pose, and Facial Expression Features (accepted to CVPR 2024) by Andre Rochow, Max Schwarz, and Sven Behnke.

[Paper]   [Project Page]

Training code will be released soon!

Example Animations

The animated sequences (bottom row) are generated by transferring the motion extracted from the driving video (left) to the person in the source image (top row).

VoxCeleb Dataset

The source images and driving videos are selected from the official VoxCeleb test set. We demonstrate both animation modes, absolute and relative motion transfer, supported by our method.

1. Absolute Motion Transfer

https://github.com/andrerochow/fsrt/assets/56793847/b56027f0-3e63-471c-bafe-edff21c6c090

2. Relative Motion Transfer

https://github.com/andrerochow/fsrt/assets/56793847/6f88911f-3572-4a3b-a17e-88994cdca22e

CelebA-HQ Dataset

Our model generalizes to source images from the CelebA-HQ dataset and driving videos from the official VoxCeleb2 test set.

https://github.com/andrerochow/fsrt/assets/56793847/209ea11e-7209-40bb-bc79-bde10ce87284

https://github.com/andrerochow/fsrt/assets/56793847/4a7f0e87-0157-4da2-883e-2366584f2779

Setup

Please complete the following steps.

Clone the repository:

git clone https://github.com/andrerochow/fsrt.git
cd fsrt

We recommend to create a new conda environment:

conda create -n fsrt python=3.9
conda activate fsrt

Dependencies

This code requires at least Python 3.9 and PyTorch.

1. Install PyTorch (>= 1.12.0)

2. Additional dependencies can be installed via:

   pip install -r requirements.txt
   

3. In case you want to animate with relative motion transfer and automatically find a best-matching frame, you need to install the face-alignment library:

   git clone https://github.com/1adrianb/face-alignment
   cd face-alignment
   pip install -r requirements.txt
   python setup.py install
   

Pretrained Checkpoints

Pretrained models can be found at google-drive.

The keypoint detector weights should be located at fsrt_checkpoints/kp_detector.pt. Note that all pretrained checkpoints are trained using the same keypoint detector weights.

Animation Demo

Animate with Relative Motion Transfer (Better ID Preservation):

python demo.py --checkpoint fsrt_checkpoints/vox256.pt --config runs/vox256/vox256.yaml  --source_image path/to/source --driving_video path/to/driving  --source_idx 0 --relative --adapt_scale --find_best_frame

Animate with Absolute Motion Transfer:

python demo.py --checkpoint fsrt_checkpoints/vox256.pt --config runs/vox256/vox256.yaml  --source_image path/to/source --driving_video path/to/driving --source_idx 0

Acknowledgement

Our FSRT implementation ist based on the PyTorch implementation of Scene Representation Transformer and First Order Motion Model for Image Animation.

BibTeX

@inproceedings{rochow2024fsrt,
  title={{FSRT}: Facial Scene Representation Transformer for Face Reenactment from Factorized Appearance, Head-pose, and Facial Expression Features},
  author={Rochow, Andre and Schwarz, Max and Behnke, Sven},
  booktitle={IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2024}
}