The first work aims to solve 3D Human Mesh Reconstruction task in perspective-distorted images.

🗓️ News:

🎆 2023.Nov.23, training code of Zolly is released, pretrained zolly weight will come soon.

🎆 2023.Aug.12, Zolly is selected as ICCV2023 oral, project page.

🎆 2023.Aug.7, the dataset link is released. The training code is coming soon.

🎆 2023.Jul.14, Zolly is accepted to ICCV2023, codes and data will come soon.

🎆 2023.Mar.27, arxiv link is released.

🚀 Run the code

🌏 Environments

• You should install MMHuman3D firstly.

You should install the needed relies as ffmpeg, torch, mmcv, pytorch3d following its tutorials.

• It is recommended that you install the stable version of MMHuman3D:

wget https://github.com/open-mmlab/mmhuman3d/archive/refs/tags/v0.9.0.tar.gz;
tar -xvf v0.9.0.tar.gz;
cd mmhuman3d-0.9.0;
pip install -e .

You can install pytorch3d from file if you find any difficulty. E.g. python3.8 + pytorch-1.13.1 + cuda-11.7 + pytorch3d-0.7.4

wget https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch3d/linux-64/pytorch3d-0.7.4-py38_cu117_pyt1131.tar.bz2;
pip install fvcore;
pip install iopath;
conda install --use-local pytorch3d-0.7.4-py38_cu117_pyt1131.tar.bz2;

• install this repo

cd Zolly;
pip install -e .

📁 Required Data and Files

You can download the files from onedrive.

This link contains:

• Dataset annotations: all have ground-truth focal length, translation and smpl parameters.

  • HuMMan (train, test_p1, test_p2, test_p3)
  • SPEC-MTP (test_p1, test_p2, test_p3)
  • PDHuman (train, test_p1, test_p2, test_p3, test_p4, test_p5)
  • 3DPW (train(has optimized neutral betas), test_p1, test_p2, test_p3)

• Dataset images.

  • HuMMan
  • SPEC-MTP
  • PDHuman
  • For other open sourced datasets, please downlad from their origin website.

• Pretrained backbone

  • hrnetw48_coco_pose.pth
  • resnet50_coco_pose.pth

• Others

  • smpl_uv_decomr.npz
  • mesh_downsampling.npz
  • J_regressor_h36m.npy

• SMPL skinning weights

  • Please find in SMPL official link.

👇 Arrange the files

root
  ├── body_models
  │   └── smpl
  |       ├── J_regressor_extra.npy
  |       ├── J_regressor_h36m.npy
  |       ├── mesh_downsampling.npz
  |       ├── SMPL_FEMALE.pkl
  |       ├── SMPL_MALE.pkl
  |       ├── smpl_mean_params.npz
  |       ├── SMPL_NEUTRAL.pkl
  |       └── smpl_uv_decomr.npz
  ├── cache
  ├── mmhuman_data
  │   ├── datasets                                                                                
  |   │   ├── coco                                                                                   
  |   │   ├── h36m                                              
  |   │   ├── humman                                            
  |   │   ├── lspet                                             
  |   │   ├── mpii                                              
  |   │   ├── mpi_inf_3dhp                                      
  |   │   ├── pdhuman                                                                                 
  |   │   ├── pw3d                                              
  |   │   └── spec_mtp  
  │   └── preprocessed_datasets
  |       ├── humman_test_p1.npz
  |       ├── humman_train.npz
  |       ├── pdhuman_test_p1.npz
  |       ├── pdhuman_train.npz
  |       ├── pw3d_train.npz
  |       ├── pw3d_train_transl.npz
  |       ├── spec_mtp.npz
  |       └── spec_mtp_p1.npz
  └── pretrain
     ├── hrnetw48_coco_pose.pth
     └── resnet50_coco_pose.pth

And change the root in zolly/configs/base.py

🚅 Train

sh train_bash.sh zolly/configs/zolly_r50.py $num_gpu$ --work-dir=$your_workdir$

E.g, you can use

sh train_bash.sh zolly/configs/zolly_r50.py 8 --work-dir=work_dirs/zolly

🚗 Test

sh test_bash.sh zolly/configs/zolly/zolly_r50.py $num_gpu$ --checkpoint=$your_ckpt$

Demo

• For Demo and pretrained checkpoint, Please wait for several days.

💻Add Your Own Algorithm

• Add your own network in zolly/models/heads, and add it to zolly/models/builder.py.
• Add your own trainer in zolly/models/architectures, and add it to zolly/models/architectures/builder.py.
• Add your own loss function in zolly/models/losses, and add it to zolly/models/losses/builder.py.
• Add your own config file in zolly/configs/, you can modify from zolly/configs/zolly_r50.py. And remember to change the root parameter in zolly/configs/base.py, where your files should be put.

🎓 Citation

If you find this project useful in your research, please consider cite:

@inproceedings{wangzolly,
  title={Zolly: Zoom Focal Length Correctly for Perspective-Distorted Human Mesh Reconstruction Supplementary Material},
  author={Wang, Wenjia and Ge, Yongtao and Mei, Haiyi and Cai, Zhongang and Sun, Qingping and Wang, Yanjun and Shen, Chunhua and Yang, Lei and Komura, Taku},
  booktitle={Proceedings of the IEEE International Conference on Computer Vision (ICCV)},
  year={2023}
}

😁 Acknowledge

Emojis are collected from gist:7360908.

Some of the codes are based on MMHuman3D, DecoMR.

📧 Contact

Feel free to contact me for other questions or cooperation: [email protected]