multimodal model / pipeline

[AvivSham]

Hi All, Thank you for this wonderful repo! Do you support multimodal models (point clouds + RGB)? if not do you have a pipeline / dataloader we can use for such purpose?

Cheers, A

[sshaoshuai]

Hi A, Currently, we only have one multimodal model for KITTI dataset, and you can get some information from this config: https://github.com/open-mmlab/OpenPCDet/blob/master/tools/cfgs/kitti_models/voxel_rcnn_car_focal_multimodal.yaml.

The multimodal models with multiple cameras are still not supported yet, but PRs related to this are welcome.

[AvivSham]

Hi @sshaoshuai, thank you for your quick response. Does this model use RGB+point clouds as its input? by looking at the paper it seems it only uses point clouds as input modality. Do you refer to multiple cameras as multiple modalities?

[sshaoshuai]

If you run the model, you can find that it actually has the option to use the image like this: https://github.com/open-mmlab/OpenPCDet/blob/master/pcdet/models/backbones_3d/focal_sparse_conv/focal_sparse_conv.py#L203-L205

Do you refer to multiple cameras as multiple modalities? No, I mean to say the LiDAR + Multiple cameras, like the setting in NuScenes/Waymo.

[AvivSham]

And what about the pipeline? what do I need to change in my config file in order to train this model in multimodal setup?

[sshaoshuai]

This model is commited by their authors, and by default it is trained with multimodal (the training/inferene is same with other models).

[AvivSham]

I see. Can you please help me and write down which models receive point clouds / RGB as inputs?

[sshaoshuai]

For now, only the model I mentioned above uses point clouds+RGB as inputs.

[AvivSham]

Sorry if I was not clear, can you please state which of the supported model receive point clouds as input and which receive RGB as input?

Thank you very much for your help.

[yukang2017]

Hi,

The original Voxel R-CNN receives point clouds only as input. Its config is https://github.com/open-mmlab/OpenPCDet/blob/master/tools/cfgs/kitti_models/voxel_rcnn_car.yaml

In addition, we modify it to be multi-modal in this config, https://github.com/open-mmlab/OpenPCDet/blob/master/tools/cfgs/kitti_models/voxel_rcnn_car_focal_multimodal.yaml It receives both point clouds and images. This improvement is from this paper.

[AvivSham]

what about the other models you support in this repo?

[yukang2017]

You can easily change other config files by DATA_CONFIG and BACKBONE_3D in voxel_rcnn_car_focal_multimodal.yaml to make them supported by multi-modal inputs.

For example, you can change https://github.com/open-mmlab/OpenPCDet/blob/master/tools/cfgs/kitti_models/pv_rcnn.yaml by the DATA_CONFIG and BACKBONE_3D information in the voxel_rcnn_car_focal_multimodal.yaml

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.