matlab_cuboid_detect： frame_full_infos.Rot for ICL-NUIM dataset

[TINY-KE]

Copy from https://github.com/shichaoy/matlab_cuboid_detect/issues/2#issue-844632533

Thanks a lot for your work. In the process of running "detect_cuboid.m" on the ICL_NUIM dataset, I encountered this problem: The camera Rot need be input into "frame_full_infos.mat", so I get the camera pose from the groudtruth. However ,the y-axis of ICLNUIM world frame is vertically upward. Detect_cuboid.m ask the z-axis of the world coordinate system is upward. So I input [1,0,0;0,0,-1;0,1,0]*R as the frame_full_infos.Rot, where R is got from the groudtruth. But, there are still errors on generating VPs. I think frame_full_infos.Rot is still wrong. Would you be so kind to provide some instructions? Could you tell how to input the frame_full_infos.Rot when you run the ICL_NUIM dataset. Thanks in advance for your support!

[TINY-KE]

Recently, I run ‘matlab_cuboid_Detect’ on the TUM dataset. The input is the groundtruth value of camera pose . The result of generating 2D cuboid, looks find. But the result of 3D modeling is not good, which deviates from the groundturh model of objects. Is there anything wrong with my operation? Thanks in advance for your support! The result for tum_fr2_desk is visualized as follow: https://user-images.githubusercontent.com/71861442/116020720-763e9a80-a679-11eb-8a88-e28aad11d694.mp4

[marrblue]

Hi @TINY-KE, according to your video, It seems all objects are localized on the ground. Actually, cube detection module depends on a strong assumption that all objects shall be on the ground, and the absolute scale of objects' dimension and position are derivated from camera height from the ground. In the case that the object is put on the table, cuboid detection module can not obtain the accurate estimation of object's dimension and position, though it may look good in 2D image. Maybe you can test this module using image sequences which contain ground-object (like cars, chairs etal..)

[TINY-KE]

Thanks for your answer.
Actually, in the test shown in the viedo, I input camera pose relative to the desk plane. It can be understood that the desk plane is used as the ground plane , and the height input of the camera is relative to the desktop. And Only objects on the desktop are visualized. In addition, in the test of ICL-NUIM and tum_fr3_cabinet, the selected objects are on the ground, such as sofa, chair, potted plants, etc., but the final 3D modeling results are still not good. The figure below is for one frame of tum_ fr3_ cabinet. Only one cabinet exists on the ground in this dataset.

[benchun123]

Well, this object detection method is not perfect, it depends on the sample and score function. The sofa in ICL-NUIM, the object in desk dataset, the cabinet dataset, the detections may be not good in every frame. I think we can adjust the sample or score function, or some weights, ...