stepjam
Check the extrinsic parameters!
Thanks for providing such a good benchmark. Recently, I tried to project the center point of the end gripper to various perspectives by using the intrinsic and extrinsic parameter matrix, and found that the projection of the left shoulder camera and the right shoulder camera is correct, but the projection of the front camera and the wrist camera is wrong. I have a look at the intrinsic and extrinsic parameters, and I think the translation of the extrinsic parametric matrix is no problem, so I wonder if there is a problem with the rotation matrix of the extrinsic parametric matrix. Thank you very much!
Here are my visualization results.
front camera：
left shoulder camera：
right shoulder camera：
wrist camera：
My projection matrix is calculated as follows: extrinsic = np.array(cam_extrinsics[view + '_camera_extrinsics']) C = np.expand_dims(extrinsic[:3, 3], 0).T R = extrinsic[:3, :3] R_inv = R.T # inverse of rot matrix is transpose R_inv_C = np.matmul(R_inv, C) extrinsic = np.concatenate((R_inv, -R_inv_C), -1) extrinsic = np.concatenate( [extrinsic, [np.array([0, 0, 0, 1])]])
K = np.array(cam_intrinsics[view + '_camera_intrinsics']) intrinsic = np.eye(4) intrinsic[:3, :3] = K intrinsics.append(intrinsic) extrinsics.append(extrinsic) lidar2img.append(intrinsic @ extrinsic)
The intrinsic and extrinsic parameter matrices I obtained are as follows: {'left_shoulder_camera_extrinsics': array([[ 1.73648179e-01, 8.92538846e-01, 4.16198105e-01, -1.74999714e-01], [ 9.84807789e-01, -1.57378674e-01, -7.33871460e-02, 2.00000003e-01], [-1.78813934e-07, 4.22618657e-01, -9.06307697e-01, 1.97999895e+00], [ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 1.00000000e+00]]), 'left_shoulder_camera_intrinsics': array([[-175.8385604, 0. , 64. ], [ 0. , -175.8385604, 64. ], [ 0. , 0. , 1. ]]), 'left_shoulder_camera_near': 0.009999999776482582, 'left_shoulder_camera_far': 3.200000047683716, 'right_shoulder_camera_extrinsics': array([[-1.73648357e-01, 8.92538846e-01, 4.16198105e-01, -1.74997091e-01], [ 9.84807789e-01, 1.57378793e-01, 7.33869076e-02, -2.00000003e-01], [-1.19209290e-07, 4.22618628e-01, -9.06307697e-01, 1.97999227e+00], [ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 1.00000000e+00]]), 'right_shoulder_camera_intrinsics': array([[-175.8385604, 0. , 64. ], [ 0. , -175.8385604, 64. ], [ 0. , 0. , 1. ]]), 'right_shoulder_camera_near': 0.009999999776482582, 'right_shoulder_camera_far': 3.200000047683716, 'overhead_camera_extrinsics': array([[-2.38418579e-07, 9.06307697e-01, 4.22618628e-01, -1.74997091e-01], [ 1.00000012e+00, 1.19209290e-07, -7.45058060e-08, 0.00000000e+00], [-1.63912773e-07, 4.22618628e-01, -9.06307697e-01, 2.42999387e+00], [ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 1.00000000e+00]]), 'overhead_camera_intrinsics': array([[-175.8385604, 0. , 64. ], [ 0. , -175.8385604, 64. ], [ 0. , 0. , 1. ]]), 'overhead_camera_near': 0.009999999776482582, 'overhead_camera_far': 3.299999952316284, 'front_camera_extrinsics': array([[ 1.19209290e-07, -4.22617942e-01, -9.06307936e-01, 1.34999919e+00], [-1.00000000e+00, -5.96046448e-07, 1.49011612e-07, 3.71546562e-08], [-5.66244125e-07, 9.06307936e-01, -4.22617912e-01, 1.57999933e+00], [ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 1.00000000e+00]]), 'front_camera_intrinsics': array([[-175.8385604, 0. , 64. ], [ 0. , -175.8385604, 64. ], [ 0. , 0. , 1. ]]), 'front_camera_near': 0.009999999776482582, 'front_camera_far': 4.5, 'wrist_camera_extrinsics': array([[-5.50746918e-05, -9.98163462e-01, 6.05788231e-02, 3.20599765e-01], [ 9.62913036e-05, -6.05788231e-02, -9.98163462e-01, 7.31974468e-02], [ 1.00000000e+00, -4.92036343e-05, 9.94205475e-05, 1.15771675e+00], [ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 1.00000000e+00]]), 'wrist_camera_intrinsics': array([[-110.85124795, 0. , 64. ], [ 0. , -110.85124795, 64. ], [ 0. , 0. , 1. ]]), 'wrist_camera_near': 0.009999999776482582, 'wrist_camera_far': 3.5}
