Unsupervised Learning of Object Landmarks via Self-Training Correspondence (NeurIPS2020)

Dimitrios Mallis, Enrique Sanchez, Matt Bell, Georgios Tzimiropoulos

This repository contains the training and evaluation code for our NeurIPS 2020 paper "Unsupervised Learning of Object Landmarks via Self-Training Correspondence". The sofware learns a deep landmark detector, directly from raw images of a specific object category, without requiring any manual annotations.

Data Preparation

CelebA

CelebA can be found here. Download the .zip file inside an empty directory and unzip. We provide precomputed bounding boxes and 68-point annotations (for evaluation only) in data/CelebA.

LS3D

We use 300W-LP database for training and LS3D-balanced for evaluation. Download the files in 2 seperate empty directories and unzip. We provide precomputed bounding boxes for 300W-LP in data/LS3D.

Human3.6

Human3.6, database is availiable here. From the availiable downloads we only requre video data, bounding boxes (MySegmentsMat-> ground_truth_bb) and 2D keypoints for evaluation (MyPoseFeatures-> D2_Positions) . For easier download you can use an automated tool like this.

We provide a python script to preprocess the video data. Before executing the script ensure that download database follows the following path structure:

     training/
	 └── subjects/
		  └── S1/
		      └── Video/
		      |	    └── *.mp4
		      │
		      ├── MyPoseFeatures/
		      │		└── D2_Positions/
		      │		      └── *.cfd
		      └── MySegmentsMat/
			        └── ground_truth_bb/
			   	          └── *.mat  

To create the database please run:

python PrePreprocessHuman.py --path_to_extract_dataset <pathToHuman3.6_database> --path_to_Human <path_to_Human>

< path_to_Human > is the directory where Human3.6 is downloaded. Frames, bounding boxes and 2D point annotations (for evaluation only) will be extracted in < pathToHuman3.6_database >.

Installation

You require a reasonable CUDA capable GPU. This project was developed using Linux.

Create a new conda environment and activate it:

conda create -n UnsuperLandmEnv python=3.8
conda activate UnsuperLandmEnv

Install pythorch and the faiss library:

conda install pytorch torchvision cudatoolkit=10.2 -c pytorch
conda install -c pytorch faiss-gpu cudatoolkit=10.2

Install other external dependencies using pip.

pip install -r requirements.txt 

Our method is bootstraped by Superpoint. Download weights for a pretrained Superpoint model from here.

Before code execution you have to update paths/main.yaml so it includes all the required paths. Edit the following entries in paths/main.yaml.:

CelebA_datapath: <pathToCelebA_database>/celeb/Img/img_align_celeba_hq/
300WLP_datapath: <pathTo300W_LP_database>/300W_LP/
LS3Dbalanced_datapath: <pathToLS3D-balanced_database>/LS3D-balanced/
Human_datapath: <pathToHuman3.6_database>/
path_to_superpoint_checkpoint: <pathToSuperPointCheckPoint>/superpoint_v1.pth

Training

To execute the first step of our method please run:

python Train_Firststep.py --dataset_name <dataset_name> --experiment_name <experiment_name>

Similarly, to execute the second step please run:

python Train_Secondstep.py --dataset_name <dataset_name> --experiment_name <experiment_name>

where < dataset_name > is in ["CelebA","LS3D", "Human3.6"] and < experiment_name > is a custom name you choose for each experiment. Please use the same experiment name for both the first and second step. The software will automatically initiate the second step with the groundtruth descovered in step one.

Testing

To evaluate the trained model simply execute:

python Test.py --dataset_name <dataset_name> --experiment_name <experiment_name>

The script will calculate cumulative forward and backward error curves. Will be stored in Logs/<experiment_name>/Logs/ .

Visualisations

We provide 3 different visualisations.

Keypoints (Step 1):

To inspect keypoint 2D locations learned from the detector head without without correspondance run:

python Visualise.py --visualisation Step1_Keypoints --dataset_name <dataset_name> --experiment_name <experiment_name> 

Clusters (Step 1):

To inspect examples of keypoints assigned to the same cluster run:

python Visualise.py --visualisation Step1_Clusters --dataset_name <dataset_name> --experiment_name <experiment_name> 

This will create a .jpg file per cluster.

Visual Results (Step 2):

For visual results run:

python Visualise.py --visualisation Step2 --dataset_name <dataset_name> --experiment_name <experiment_name> 

The software will automatically load checkpoints and pseudogroundtruth files for the assosiated <experiment_name> .

Pretrained Models

We provide also pretrained models. Can be used to execute the testing script and produce visual results.

Simply uncompress the .zip files inside Logs/.

Pretrained weights can be used for calculating forward and backward error curves as well as running visualisation code for visual results (step 2).

Citation

If you found this code useful please consider citing:

@inproceedings{unsupervLandm2020,
title={Unsupervised Learning of Object Landmarks via Self-Training Correspondence},
author={Mallis, Dimitrios and Sanchez, Enrique and Bell, Matt and Tzimiropoulos, Georgios},
booktitle={Conference on Neural Information Processing Systems (NeurIPS)},
year={2020}
}