Neural Retargeting

Code for the paper "Kinematic Motion Retargeting via Neural Latent Optimization for Learning Sign Language"

Prerequisite

• PyTorch Tensors and Dynamic neural networks in Python with strong GPU acceleration
• pytorch_geometric Geometric Deep Learning Extension Library for PyTorch
• Kornia a differentiable computer vision library for PyTorch.
• HDF5 for Python The h5py package is a Pythonic interface to the HDF5 binary data format.

Dataset

The Chinese sign language dataset can be downloaded here.

Model

The pretrained model can be downloaded here.

Get Started

Training

CUDA_VISIBLE_DEVICES=0 python main.py --cfg './configs/train/yumi.yaml'

Inference

CUDA_VISIBLE_DEVICES=0 python inference.py --cfg './configs/inference/yumi.yaml'

Simulation Experiment

We build the simulation environment using pybullet, and the code is in this repository.

After inference is done, the motion retargeting results are stored in a h5 file. Then run the sample code here.

Real-World Experiment

Real-world experiments could be conducted on ABB's YuMi dual-arm collaborative robot equipped with Inspire-Robotics' dexterous hands.

We release the code in this repository, please follow the instructions.

Citation

If you find this project useful in your research, please cite this paper.

@article{zhang2022kinematic,
  title={Kinematic Motion Retargeting via Neural Latent Optimization for Learning Sign Language},
  author={Zhang, Haodong and Li, Weijie and Liu, Jiangpin and Chen, Zexi and Cui, Yuxiang and Wang, Yue and Xiong, Rong},
  journal={IEEE Robotics and Automation Letters},
  year={2022},
  publisher={IEEE}
}