Zechuan Zhang  Zongxin Yang  Yi Yang✉ 

ReLER, CCAI, Zhejiang University

^✉Corresponding Author

CVPR 2024

Figure 1. With just a single image, SIFU is capable of reconstructing a high-quality 3D clothed human model, making it well-suited for practical applications such as 3D printing and scene creation. At the heart of SIFU is a novel Side-view Conditioned Implicit Function, which is key to enhancing feature extraction and geometric precision. Furthermore, SIFU introduces a 3D Consistent Texture Refinement process, greatly improving texture quality and facilitating texture editing with the help of text-to-image diffusion models. Notably proficient in dealing with complex poses and loose clothing, SIFU stands out as an ideal solution for real-world applications.

News

• [2024/2/28] We release the code of geometry reconstruction, including test and inference.
• [2024/2/27] SIFU has been accepted by CVPR 2024! See you in Seattle!
• [2023/12/13] We release the paper on arXiv.
• [2023/12/10] We build the Project Page.

Installation

• Ubuntu 20 / 18
• CUDA=11.6 or 11.7 or 11.8, GPU Memory > 16GB
• Python = 3.8
• PyTorch = 1.13.0 (official Get Started)
• PyTorch3D (official INSTALL.md, recommend install-from-local-clone)

git clone https://github.com/River-Zhang/SIFU.git
sudo apt-get install libeigen3-dev ffmpeg
cd SIFU
conda env create -f environment.yaml
conda activate sifu
pip install -r requirements.txt

Please download the checkpoint (google drive) and place them in ./data/ckpt

Please follow ICON to download the extra data, such as HPS and SMPL. There may be missing files about SMPL, and you can download from here and put them in /data/smpl_related/smpl_data/.

Inference

python -m apps.infer -cfg ./configs/sifu.yaml -gpu 0 -in_dir ./examples -out_dir ./results -loop_smpl 100 -loop_cloth 200 -hps_type pixie

Testing

# 1. Register at http://icon.is.tue.mpg.de/ or https://cape.is.tue.mpg.de/
# 2. Download CAPE testset
bash fetch_cape.sh 

# evaluation
python -m apps.train -cfg ./configs/train/sifu.yaml -test

# TIP: the default "mcube_res" is 256 in apps/train.

Applications of SIFU

Scene Building

3D Printing

Texture Editing

Animation

In-the-wild Reconstruction