How to evaluate camera-lidar fusion method on the test set?

[ICEORY]

Hi,

I'm working on camera-lidar fusion methods for semantic segmentation. Because SemanticKITTI only contains front-view images, I can only generate the segmentation results of front-view point clouds. I want to know whether it is available to evaluate the front-view point clouds on the test set.

Most of the exiting fusion-based methods only provide results on the processed KITTI data set of SqueezeSeg (Front-view). I think it will be better if SemanticKITTI can also provide a leaderboard for fusion-based methods.

Thanks!

[jbehley]

thanks for the suggestion. we could provide such metric only for the frontal part of the point cloud as an additional "branch". I will think about it...I currently cannot judge if people really are interested in this particular metric.

[ICEORY]

Thanks for your reply!

I think this metric is important for evaluating the performance of fusion-based and single-sensor methods. I also notice that other data sets released recently (e.g. nuscenes and kitti-360) have provided multi-camera data for point clouds. Maybe I should try these data sets first.

[jbehley]

Good idea. I leave this issue for now open and will consider it when I have more time to checkout if we can provide such "subevaluation" on the codalab platform as separate metric.