Data augmentation with segmentation map/queries

[DianCh]

Hi! I noticed that in your petrv2_BEVseg.py config file there's GlobalRotScaleTransImage which applies rotation and scaling to the bev space and correspondingly 3d ground truth boxes (which makes total sense for detection because boxes are also modified accordingly). However, the transformations are not applied on results['gt_map'] or results['maps'].

I'm confused here since when segmentation queries are generated using fixed locations in the bev space, so in my understanding either:

• the queries should first be rotated & scaled, or
• the gt map should be rotated & scaled.

I didn't see this in Petr3D_seg, PETRHeadseg, or GlobalRotScaleTransImage. Am I missing anything? If that's already done could you point me to where it is?

Thank you!

[vaesl]

Hi, thanks for your pointing out this problem! As you mentioned, the extrinsic parameters (3D PE in our paper) as well as the gt_maps should be transformed together when applying the BDA augmentation. Here, the detection and segmentation branches share the same BDA augmentation for convenice. In our practice, the detection performance will drop less than 0.5% mAP and the segmentation branch will improve the overall IoU by 1%. Currently, we are decoupling these two branches with only applying the BDA on detection branches. We will update it as soon as there is any new result.