[FEATURE]: 3D bounding box ground truth genereration

[HannahHaensen]

Thanks for this awesome repository and nice work to easily generate syn. datasets. Its a really great contribution!

Is your feature request related to a problem? Please describe. 3D object detection ground truth data in e.g. Kitti format

Describe alternatives you've considered I was wondering if the 6D pose part of blenderproc can be utlizied to generate 3d bounding boxes as well without additional posprocessing

[cornerfarmer]

Hey @HannahHaensen,

sorry for the delay. It is at the moment not possible to write 3D bounding boxes in the Kitti format via BlenderProc. However, you could easily add that functionality yourself in your script.

You can access the object aligned bounding box in world coordinates for every object via obj.get_bound_box(). Depending on what you need you would then eventually need to convert them to axis-aligned bounding boxes and then write them to file in your desired format.

Let me know if you need any further help!

[andrewyguo]

@cornerfarmer @themasterlink , I am willing to try to work on this. Are you currently working on a fix?

[themasterlink]

Go for it.

[andrewyguo]

@themasterlink, for this issue, what kind of change are you thinking of?

Should I add a new writer under BlenderProc/blenderproc/python/writer to write 3d bounding boxes?

[themasterlink]

Hey,

The question is how to best store this kind of data. The 3D bounding box is an object-based attribute, so it might be smart to use the semantic segmentation feature.

I suggest adding a new custom function to the semantic segmentation called cf_3d_bounding_box. We already have one custom function here for cf_basename:

https://github.com/DLR-RM/BlenderProc/blob/6d569c5a9ec941e3677b14273c6092093404396e/blenderproc/python/postprocessing/PostProcessingUtility.py#L376-L380

The only thing one would need to add would be a new member function to a MeshObject, which returns the desired 3d_bounding_box and store it in the variable value. Maybe we have two different ones: cf_3d_bounding_box_axis_aligned and cf_3d_bounding_box_world_aligned.

Then we only need to call:

# enable segmentation masks (per class and per instance)
bproc.renderer.enable_segmentation_output(map_by=["category_id", "instance", "name", "cf_3d_bounding_box_axis_aligned"])

What do you think?

PS: Of course, we should also add a visualizer for this. The issue is this would require storing the Kamera Intrinsics as well.

Best, Max

[cornerfarmer]

@andrewyguo can you please also elaborate a bit more what your application for the bounding boxes is in the end. Why do you need bounding boxes at all, aren't object poses already sufficient? Do you want to store the boxes framewise or once for all objects? And do you want to store them in a specific data format (e.g. kitti) or just in the usual hdf5?