Rotation is off

[daniekpo]

Hi, the rotation is consistently off across objects when I execute the grasp pose on the robot. My eye-in-hand calibration matrix seems to be right because I can reconstruct a coherent scene from different camera positions. I can also move the robot to known targets (on a ChArUco board). When I visualize the grasp pose it looks straight with just a small tilt here and there but when I convert the pose from the camera frame to the robot base I get rotations that don't make sense.

If I manually set the rotation to say [0, -pi, 0] then the robot is able to reach the target object. Do I need to do some other alignment? I'm I correct that Anygrasp's predicted poses are in the camera frame and I just need to do a camera to world transform?

I saw this related issue about translation, but we have the opposite case where the translation is correct but the rotation is off. I double-checked that we're using the correct intrinsics.

[chenxi-wang]

Hi Daniel, I guess there's a transformation between your end effector and the gripper coordinate frame defined in graspnet. You could refer to https://graspnetapi.readthedocs.io/en/latest/grasp_format.html#d-grasp to transform the tcp.

[daniekpo]

Hi Daniel, I guess there's a transformation between your end effector and the gripper coordinate frame defined in graspnet. You could refer to https://graspnetapi.readthedocs.io/en/latest/grasp_format.html#d-grasp to transform the tcp.

Thanks for your reply @chenxi-wang! I actually didn't know there was a readthedocs site for Anygrasp, that's super helpful. Just to confirm, Anygrasp assumes that the gripper's x is pointing up, y is pointing right and z points into the scene as shown in this image from the docs. So correcting the coordinates to align with my end effector should fix the problem.

My second question concerns the code block right above the coordinate image. It seems to be isolated but important. What is the code block about?

[chenxi-wang]

Just to confirm, Anygrasp assumes that the gripper's x is pointing up, y is pointing right and z points into the scene as shown in this image from the docs.

Yes, your understanding is right.

My second question concerns the code block right above the coordinate image. It seems to be isolated but important. What is the code block about?

It does not matter. I think it is just a typo lol.

[daniekpo]

I tried using a simple transformation to swap the x and z axis but that failed. The translation that was accurate before wasn't accurate anymore. We're using the same setup as the experiments in the Anygrasp paper with a camera mounted on the robot arm and the image is taken from the top facing down. I'm wondering why the translation was accurate before if the frame needed to be reoriented.

[yiyang0207]

@daniekpo Hi, I've been facing the same issue. Did you figure out how to deal with it?

[daniekpo]

@daniekpo Hi, I've been facing the same issue. Did you figure out how to deal with it?

Hi @yiyang0207, sorry for the delayed response. The solution for my setup was just swapping the axes like this

# grasp_pose = get any grasp pose
# To fix Anygrasp's coordinate system
pose_align_T = np.array([
    [0, 0, 1, 0],
    [1, 0, 0, 0],
    [0, 1, 0, 0],
    [0, 0, 0, 1]
])
camera_aligned_pose = grasp_pose @ pose_align_T
# followed by the camera to the robot frame transform
# world_pose = cam2world @ camera_aligned_pose