Issue with Using "replay_trajectory" in "pd_ee_pose" Control Mode

[TongZhangTHU]

Hi,

Thanks for your explanation in this issue

Following your advice, I've incorporated both "arm_pd_ee_pose" and "pd_ee_pose" into ManiSkill2/mani_skill2/agents/configs/panda/defaults.py. However, I've encountered an issue when using the mani_skill2.trajectory.replay_trajectory function in the "pd_ee_pose" control mode.

Specifically, an AssertionError is triggered at this line in replay_trajectory.py. I'm curious if this assertion is an intended part of the functionality, and if so, how we should properly utilize replay_trajectory with the "pd_ee_pose" control mode to avoid this error.

Any insights or further guidance you could provide on this matter would be greatly appreciated.

Thank you for your time and assistance.

[StoneT2000]

Hi, could you elaborate a bit more about what you are trying to do? What do you mean by incorporating those two modes? And can you post the full commands you ran + any code changes you made?

[TongZhangTHU]

@StoneT2000 Hi, I am trying to add a new control mode "pd_ee_pose" as https://github.com/haosulab/ManiSkill2/issues/122 suggests. Code changes in ManiSkill2/mani_skill2/agents/configs/panda/defaults.py include: (1) create a new controller configuration

import numpy as np
arm_pd_ee_pose = PDEEPoseControllerConfig(
    self.arm_joint_names,
    -10.0,
    10.0,
    2*np.pi,
    self.arm_stiffness,
    self.arm_damping,
    self.arm_force_limit,
    ee_link=self.ee_link_name,
    use_delta=False,
    use_target=False,
    normalize_action=False,
    frame='base', # such that the position & rotation actions are always with respect to the base frame, since ee frame can change as control is being executed
)

(2) add this controller config to the robot's controller_configs under the robot's defaults.py ManiSkill2/mani_skill2/agents/configs/panda/defaults.py

pd_ee_pose=dict(
    arm=arm_pd_ee_pose, gripper=gripper_pd_joint_pos
),

Then I run python -m mani_skill2.trajectory.replay_trajectory --traj-path $trajpath --save-traj -o pointcloud -c pd_ee_pose --num-procs 8 to generate demos. Then the AssertionError is triggered.

[StoneT2000]

Oh I see. I think you shouldn't need to set the frame to be in the base frame. The default arm_pd_ee_pose controller doesn't set frame='base' either. Try that and see how it goes

[TongZhangTHU]

@StoneT2000

Thanks for your reply.

I tried to set the frame='ee' and run the python -m mani_skill2.trajectory.replay_trajectory --traj-path $trajpath --save-traj -o pointcloud -c pd_ee_pose --num-procs 8 to generate demos again. But another AssertionError is triggered at https://github.com/haosulab/ManiSkill2/blob/main/mani_skill2/trajectory/replay_trajectory.py#Line73. It seems I have to use_delta and normalize_action?

Actually what I want is to get the absolute position & rotation(without normalization), do you have any suggestions?

[StoneT2000]

You shouldn't need use delta pose. I think you code is fine, but our replay trajectory tool isn't general enough. It seems the code is trying to use the wrong conversion function.

You ask the code to convert to your new controller, in order to you should modify the code at line 143 and use your own function to convert the actions correctly.

[TongZhangTHU]

@StoneT2000 Thanks for your reply. I am not so familiar with the replay trajectory tool. Do you have any suggestions on how to write a function to convert the actions with use_delta=False and normalize_action=False?

[StoneT2000]

Hi @TongZhangTHU sorry for the late reply. the general approach for converting actions is always to write code to somehow generate actions that move the robots joints/end-effector pose to the position the original actions would lead to and minimize the error. Our conversion tool uses an "original environment" using the original actions and the new environment that we want to generate new actions in our new action space with.

I will close the issue for now as the problem of converting action spaces is highly non-trivial and not something the maniskill team plans to support as the problem itself is a large research problem (which is highly interesting but we do not have a good general purpose solution just yet and it currently requires a lot of extra programming that is hard to maintain).