RTM3D
RTM3D copied to clipboard
maudzung
Implementation of the 3D Bounding Box Estimation (3.2 part in the paper)
It should be argmin instead of argmax in the formula (7)
This part can be solved by using the g2opy library. Welcome to contribute to the implementation of this part! Thanks!
I have trained your code on KITTI, but got a bad result when I test it, Do you have got a good result? Thanks!
Hi @DMing11 The results have been not good, but also not bad. The results for 3D bounding boxes will definitely be optimized by one more step that I mentioned above. Please do it if you can. Many thanks!
Hi @maudzung Can you release your pre-trained models? I want to try part 3.2 in paper , but I don't have a relatively good pre-trained models. Thanks!
scipy.optimize.minimize can help you with minimizing the reprojection error of keypoints.
Hi @SPengLiang Many thanks. Did you implement the part? Can you please provide the source code or contribute to this repo?
#================================= Here are some of my code in my previous works, and it works well. You can customize it to fit your code, such as corner order and optimizer. #=================================
from scipy.optimize import minimize
def opt(P2, kpts, pos, dim, Ry):
init_x, init_y, init_z = pos
init_h, init_w, init_l = dim
init_Ry = Ry
fx, fy = P2[0, 0], P2[1, 1]
cx, cy = P2[0, 2], P2[1, 2]
# normalize image plane
norm_u = (kpts[0] - cx) / fx
norm_v = (kpts[1] - cy) / fy
def f_rect(states):
x = states[0]
y = states[1]
z = states[2]
Ry = states[3]
h, w, l = init_h, init_w, init_l
# 8 corners. note that it should align with the keypoints order.
shift = np.array([[1 / 2, 1 / 2, -1 / 2, -1 / 2, 1 / 2, 1 / 2, -1 / 2, -1 / 2],
[0, 0, 0, 0, -1 , -1, -1, -1],
[1 / 2, -1 / 2, -1 / 2, 1 / 2, 1 / 2, -1 / 2, -1 / 2, 1 / 2],
[1, 1, 1, 1, 1, 1, 1, 1]])
c = np.cos(Ry)
s = np.sin(Ry)
rot_mat = np.array([[c, 0, s, 0],
[0, 1, 0, 0],
[-s, 0, c, 0],
[0, 0, 0, 1]])
lhw = np.array([[l], [h], [w], [1]])
lhw_t = np.tile(lhw, [1, 8])
xyz_shfit = lhw_t * shift
xyz_shfit = np.dot(rot_mat, xyz_shfit)
xyz = np.array([[x], [y], [z], [0]]) + xyz_shfit
xyz[:2, :] /= np.tile(xyz[2:3, :], [2, 1])
res_u = np.abs(xyz[0] - norm_u) ** 2
res_v = np.abs(xyz[1] - norm_v) ** 2
return np.sum(res_u) + np.sum(res_v)
res = minimize(f_rect, [init_x, init_y, init_z, init_Ry])
return res.x, res.success
@maudzung You're quite welcome. You probably need to change Ry to -Ry as the orientation I used may be different from KITTI's, and the order of keypoints in the optimize function should align with that of your predicted keypoints.
How can I go about implementing the code from @SPengLiang to help the 3D bounding boxes perform better? More specifically, where is formula(7) in the repo as of now? Does it exist, yet?
