[New Models] How to visualize the provided sun rgbd 3d box

[peiwang062]

trafficstars

Model/Dataset/Scheduler description

Hi, I'm new to the 3d vision. I'm trying to visualize the provided demo. Everything is ok for kitti but when I use the code below to visualize sun rgbd example, there is an unexpected shift, does anyone can suggest where is the error for my code?

import mmcv
import numpy as np
from mmengine import load
import torch
from mmdet3d.visualization import Det3DLocalVisualizer
from mmdet3d.structures import DepthInstance3DBoxes

# Load SUN RGB-D sample info
info_file = load('./demo/data/sunrgbd/sunrgbd_000017_infos.pkl')
cam_K = np.array(info_file['data_list'][0]['images']['CAM0']['depth2img'])  # Intrinsic matrix
bboxes_3d = []
for instance in info_file['data_list'][0]['instances']:
    bboxes_3d.append(instance['bbox_3d'])


gt_bboxes_3d = np.array(bboxes_3d, dtype=np.float32)
gt_bboxes_3d = DepthInstance3DBoxes(gt_bboxes_3d)  # SUN RGB-D uses depth mode

input_meta = {'depth2img': cam_K}

# Visualizer
visualizer = Det3DLocalVisualizer()

# Load the image
img = mmcv.imread('./demo/data/sunrgbd/' + info_file['data_list'][0]['images']['CAM0']['img_path'])
img = mmcv.imconvert(img, 'bgr', 'rgb')
visualizer.set_image(img)

# Project 3D bboxes to image
visualizer.draw_proj_bboxes_3d(gt_bboxes_3d, input_meta)

# Save output
mmcv.imwrite(visualizer.get_image(), 'sunrgbd_demo_output.png')

Open source status

• [ ] The model implementation is available
• [ ] The model weights are available.

Provide useful links for the implementation

No response

[Lybrantss]

we have the same question,It seems that there is an issue with the depth2img matrix when I project it. Have you solved this question?

[2001Iam]

I also encountered the same problem when using BEVFusion. How can I solve it？

[peiwang062]

we have the same question,It seems that there is an issue with the depth2img matrix when I project it. Have you solved this question?

No, I'm still having trouble with it:(

[cattpku]

I think maybe you can update [0.5, 0.5, 0] in this line https://github.com/open-mmlab/mmdetection3d/blob/fe25f7a51d36e3702f961e198894580d83c4387b/mmdet3d/structures/bbox_3d/depth_box3d.py#L73 to [0.5,0.5,0.5], the visualisation should be working correctly.