error in performanece of fsdv2

[yrt1993]

trafficstars

I tried to train fsdv2 on waymo perception V1_4_0，and get the performance on validation set as follows. which is much lower than the performance in the paper

OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1: [mAP 0.749648] [mAPH 0.745259] OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2: [mAP 0.669701] [mAPH 0.66562] OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1: [mAP 0.80301] [mAPH 0.755212] OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2: [mAP 0.73128] [mAPH 0.685319] OBJECT_TYPE_TYPE_SIGN_LEVEL_1: [mAP 0] [mAPH 0] OBJECT_TYPE_TYPE_SIGN_LEVEL_2: [mAP 0] [mAPH 0] OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1: [mAP 0.744445] [mAPH 0.734285] OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2: [mAP 0.719278] [mAPH 0.709439]

This is the config I used fsdv2_waymo_full_2x.txt

[Abyssaledge]

It should not be so. I provide complete training logs in the URLs in the instructions. I suggest that first compare your log with the provided logs to debug.

[yrt1993]

the only difference between the two configs is the different backbone, I use 'VirtualVoxelMixer' as in the fsdv2_waymo_2x.py，but you used 'FSDV2SparseUNet', which is not defined in the project

[Abyssaledge]

What I mean is to compare the training log (not only the config) and see if there are any significant differences (e.g., losses and other printed items). Besides, could you get the normal results using the provided weights for inference?

[Abyssaledge]

Thanks for your feedback. It seems that the training goes well but there are errors during the inference. Have you made any modifications to this repo?

[Abyssaledge]

In addition, is your validation data complete? How many samples are found by the WOD evaluation bool (the printed number during evaluation)?

[yrt1993]

They have similar loss as follows:

When I used your trained model, the metris is as follows: OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1: [mAP 0.748531] [mAPH 0.744346] OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2: [mAP 0.669115] [mAPH 0.665236] OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1: [mAP 0.798826] [mAPH 0.750668] OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2: [mAP 0.72766] [mAPH 0.6814] OBJECT_TYPE_TYPE_SIGN_LEVEL_1: [mAP 0] [mAPH 0] OBJECT_TYPE_TYPE_SIGN_LEVEL_2: [mAP 0] [mAPH 0] OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1: [mAP 0.756294] [mAPH 0.745802] OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2: [mAP 0.730251] [mAPH 0.720095]

But when I used your results.bin to calculate the metric, it is OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1: [mAP 0.79783] [mAPH 0.793368] OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2: [mAP 0.714395] [mAPH 0.710258] OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1: [mAP 0.84771] [mAPH 0.796692] OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2: [mAP 0.774113] [mAPH 0.724897] OBJECT_TYPE_TYPE_SIGN_LEVEL_1: [mAP 0] [mAPH 0] OBJECT_TYPE_TYPE_SIGN_LEVEL_2: [mAP 0] [mAPH 0] OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1: [mAP 0.806924] [mAPH 0.795709] OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2: [mAP 0.779097] [mAPH 0.768243]

[yrt1993]

In addition, is your validation data complete? How many samples are found by the WOD evaluation bool (the printed number during evaluation)?

39984 examples. I only make some modifications in data processing scripts of tools.

[Abyssaledge]

You could try our pretrained model in nuScenes to see if Waymo data corrupts.

[JuiceLemonLemon]

@Abyssaledge Hello, good afternoon! I found FSDv2 for nuScense didn't use roi_head. nuScenes doesn't need roi_head?

Actually, I modified CTRL refine to nuScenes, but the accuracy dropped. I found CTRL refine module was based on roi_head, so should I change roi_head of CTRL on nuScenes ?

[Abyssaledge]

What do you mean by accuracy dropped? Worse than the base detector? FSDv2 does not adopt roi_head in nusc is because the metric in nusc is center-based instead of IoU-based, so the gain of proposal refinement in nusc is not that significant. For simplicity, we remove roi_head in nusc. If you would like to use roi_head in nusc, just follow the code in Waymo, there are no differences between them in implementation.

[Abyssaledge]

@JuiceLemonLemon

[JuiceLemonLemon]

What do you mean by accuracy dropped? Worse than the base detector? FSDv2 does not adopt roi_head in nusc is because the metric in nusc is center-based instead of IoU-based, so the gain of proposal refinement in nusc is not that significant. For simplicity, we remove roi_head in nusc. If you would like to use roi_head in nusc, just follow the code in Waymo, there are no differences between them in implementation.

@Abyssaledge Yes, the CTRL refined accuracy is worse than the base detector on nuScenes. I wonder if the roi_head is not suitable for nuScenes.

[Abyssaledge]

@yrt1993 We tested our checkpoint again, and the performance can be perfectly reproduced.

[yrt1993]

actually, I produced a wrong idx2timestamp.pkl and idx2contextname.pkl, which causes this error. I have got the true results, thank you

[Abyssaledge]

Thanks for the feedback. So, can you reproduce the performance with your re-trained models? @yrt1993

[yrt1993]

yeah, now the results is

OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1: [mAP 0.799273] [mAPH 0.794589] OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2: [mAP 0.715242] [mAPH 0.710883] OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1: [mAP 0.852103] [mAPH 0.801296] OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2: [mAP 0.778003] [mAPH 0.729059] OBJECT_TYPE_TYPE_SIGN_LEVEL_1: [mAP 0] [mAPH 0] OBJECT_TYPE_TYPE_SIGN_LEVEL_2: [mAP 0] [mAPH 0] OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1: [mAP 0.795506] [mAPH 0.784628] OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2: [mAP 0.768605] [mAPH 0.758071]

[Abyssaledge]

Thanks for your kind feedback. Glad to see that. Good luck with your exploration!