3D Diffusion Policy Implementation

[ruoxianglee]

Hi, I am trying to use 3d diffusion policy (DP3) code in ManiSkill, but encountering some issue. I would appreciate your help with this matter.

The DP3 code has been added into baselines diffusion_policy_3d. I trained DP3 for PickCube task in ManiSkill environment. Druing training, it was observed that the loss can converge very fast to a small value. Unfortunately, The success rate during evaluation is very low. The highest success rate I got is 10%. It is close to 0 for most cases. I tried to find the potential reason but failed.

README file contains the detailed steps to train DP3 in ManiSkill.

To implement DP3 into ManiSkill environment, I did the following modifications:

• Convert the ManiSkill demonstrations from hdf5 format into zarr format (see code here), so the demonstrations can be directly used for training DP3 policy.
• ManiSkill dataset for pytorch datasetloader (see code here)
• ManiSkill environment wrapper and runner

The followings are the key settings:

• TaskID: PickCube-v1
• Observation mode: pointcloud (pointcloud cropping and downsampling are used)
• Controller: arm_pd_ee_delta_pose
• Simulation backend: cpu
• Number of demos: 10
• Observation
  • State: 29-D (including joint positions [7+2], velocities [7+2], goal position [3], end-effector pose [7], is_grasped [1])
  • Pointcloud: 6-D (xyz [3], rgb [3])
• Action: 7-D (translation [3], rotation [3], gripper open/close [1])