pergamo
pergamo copied to clipboard
Code for PERGAMO, a two step approach to learn deformable models
PERGAMO
[Project website] [Dataset] [Video]
Abstract
Clothing plays a fundamental role in digital humans. Current approaches to animate 3D garments are mostly based on realistic physics simulation, however, they typically suffer from two main issues: high computational run-time cost, which hinders their development; and simulation-to-real gap, which impedes the synthesis of specific real-world cloth samples. To circumvent both issues we propose PERGAMO, a data-driven approach to learn a deformable model for 3D garments from monocular images. To this end, we first introduce a novel method to reconstruct the 3D geometry of garments from a single image, and use it to build a dataset of clothing from monocular videos. We use these 3D reconstructions to train a regression model that accurately predicts how the garment deforms as a function of the underlying body pose. We show that our method is capable of producing garment animations that match the real-world behaviour, and generalizes to unseen body motions extracted from motion capture dataset.
Install instructions
Python dependencies
IGL only supports and recommends the use of Anaconda. However, the environment can be set up using only pip
by installing the IGL bindings from source.
The general steps are as follows:
- Install PyTorch according to your system ( https://pytorch.org/get-started/locally/ )
- See the
requirements.txt
file to check the needed packages- This is usually done with
pip install -r requirements.txt
, but Anaconda may have a different way of doing things
- This is usually done with
- Install IGL bindings ( https://github.com/libigl/libigl-python-bindings )
- Install Kaolin ( https://kaolin.readthedocs.io/en/latest/notes/installation.html )
Models
- You can download the weights from
OneDrive
. Place the
weights
folder from OneDrive into thedata
folder of this repository. - PERGAMO needs SMPL. You can download it from SMPL. Rename the file from
basicmodel_neutral_lbs_10_207_0_v1.1.0.pkl
tosmpl_neutral.pkl
and save it underdata/smpl/
.
Running the project
To run the reconstruction, please check out run_recons.sh
.
To run the regression, there are 2 sets of 3 scripts. Please check out run_regression.sh
to see how it works.
Visualizing regression results
The output is generated under data
(test_sequence
for AMASS scripts, train/validation_sequence
for reconstructed
scripts).
To visualize using Blender, load the .obj
file with the option Geometry > Keep Vert Order
. Then, add a Mesh Cache
modifier to the loaded mesh. Change the type to PC2
and then load the .pc2
file adjacent to the .obj
.
Datasets
You can download a dataset from OneDrive .
Structure
Each data set has the following folder hierarchy:
DataDanXXXXX
├─ clips (video files)
| ├─ dan-X01.mp4
| ├─ dan-X02.mp4
| ├─ ...
├─ reconstruction_input
| ├─ dan-X01
| | ├─ dan-X01 (video frames)
| | ├─ dan-X01_expose
| | ├─ dan-X01_parsing
| | ├─ dan-X01_pifu
| | ├─ dan-X01_smpl
| ├─ dan-X02
| | ├─ ...
| ├─ ...
├─ reconstruction_output (reconstructed garment meshes)
| ├─ dan-X01
| ├─ dan-X02
| ├─ ...
├─ regressor_training_data
├─ train_sequences
| ├─ meshes (reconstructed garment meshes in Tpose)
| | ├─ dan-X01
| | ├─ dan-X02
| | ├─ ...
| ├─ poses (encoded poses using the SoftSMPL encoding)
| | ├─ dan-X01
| | ├─ dan-X02
| | ├─ ...
├─ validation_sequences (same structure as train)
├─ ...
For reconstruction
Datasets for the reconstruction script are made by processing each frame with:
- ExPose (output is SMPL-X, they need to be converted to SMPL too)
- PifuHD
- Self-Correction-Human-Parsing
The necessary files are provided in the reconstruction_input folder. We also provide reconstructed meshes for each dataset (reconstruction_input folder) and the same meshes in Tpose space (inside the meshes folder on regressor_training_data).
For training
Our regressors predict wrinkles (vertex displacements with respect to a template mesh) from SMPL poses encoding using the SoftSMPL encoding. We provide such encoded poses for the DataDanGrey dataset and also the scripts to generate such encoding from arbitrary SMPL paramteres.
For regression
You can use AMASS sequences by placing the .npz
files under data/test_sequence
.
Alternatively, you can run the regression on sequences of SMPL poses saved as .pkl
files. Check the set
of reconstructed
scripts.
Citation
@article {casado2022pergamo,
journal = {Computer Graphics Forum (Proc. of SCA), 2022},
title = {{PERGAMO}: Personalized 3D Garments from Monocular video},
author = {Casado-Elvira, Andrés and Comino Trinidad, Marc and Casas, Dan},
year = {2022}
}