mmdetection3d icon indicating copy to clipboard operation
mmdetection3d copied to clipboard

PointPillars' training logs confusion

Open TuanTNG opened this issue 2 years ago • 3 comments

Hi MMDetection3D teams,

I am re-producing PointPillars config, this config. However, when I read its training logs, this logs, I found something wrong.

The total number of training samples of KITTI is 3712 as in this readme. When training PointPillars configs with repeating KITTI dataset by 2 times, samples_per_gpu=6 and 8 GPUs. It must result in around 155 (~=3712*2/8/6) iterations per one epoch. However, the training log shows 1200 iterations per one epoch.

Is there anything wrong?

I found that other config and its log which trained using 4 GPUs & samples_per_gpu=4, are correct (the log shows 450 (~= 3712*2/4/4) iterations per epoch).

Thank you so much.

TuanTNG avatar May 06 '22 06:05 TuanTNG

Yeah, I found the training logs for pp_kitti-3class and pp_kitti-car are little different from the corresponding configs.

Xiangxu-0103 avatar May 11 '22 08:05 Xiangxu-0103

The potential difference has been fixed in #1486. I found the training log for pp_3class is exactly trained with only one GPU, which causes the iteration 1200 per epoch.

Xiangxu-0103 avatar May 19 '22 07:05 Xiangxu-0103

@ZCMax Please have a look at this issue and update the log files if necessary.

Tai-Wang avatar May 29 '22 03:05 Tai-Wang