Validation set performance of the pretrained model

[Yibin122]

I evaluated the performance of your uploaded model on the val split (3769 images) and got the following results: car_detection_ground AP: 83.374908 82.937012 75.748917 car_detection_3d AP: 78.142258 72.837662 65.392242

which are much higher than reported in the paper.

So I wonder if that model was trained using all the training images (trainval) instead of the train split?

[zxduan-r]

I got similar result when validate using provided checkpoint. It is much higher than reported and almost the same performance as using lidar data.

[jmellafe]

I got similar results as well, probably they've trained the model with trainval as you suggest

[ccerhan]

I've conducted some test about training the model and the results are like this: I trained this model using KITTI train (from ImageSets) and validate on val and I got approximately ~ %10 3d detection performance like the paper. Then I trained using trainval and I got ~ %80 3d detection results on val.

[malianghui]

I evaluated the performance of your uploaded model on the val split (3769 images) and got the following results: car_detection_ground AP: 83.374908 82.937012 75.748917 car_detection_3d AP: 78.142258 72.837662 65.392242

which are much higher than reported in the paper.

So I wonder if that model was trained using all the training images (trainval) instead of the train split?

I evaluated the performance of your uploaded model on the val split (3769 images) and got the following results: car_detection_ground AP: 83.374908 82.937012 75.748917 car_detection_3d AP: 78.142258 72.837662 65.392242

which are much higher than reported in the paper.

So I wonder if that model was trained using all the training images (trainval) instead of the train split?

Hello: I test kitty dadaset by python tools/plain_train_net.py --eval-only --config-file "configs/smoke_gn_vector.yaml"

And the "smoke_gn_vector.yaml" is: MODEL: #WEIGHT: "catalog://ImageNetPretrained/DLA34" WEIGHT: "/home/mlhui/project/SMOKE/dla34-ba72cf86.pth" INPUT: FLIP_PROB_TRAIN: 0.5 SHIFT_SCALE_PROB_TRAIN: 0.3 DATASETS: DETECT_CLASSES: ("Car", "Cyclist", "Pedestrian") TRAIN: ("kitti_train",) #TEST: ("kitti_test",) TEST: ("kitti_train",) TRAIN_SPLIT: "trainval" #TEST_SPLIT: "test" TEST_SPLIT: "val" SOLVER: BASE_LR: 2.5e-4 STEPS: (10000, 18000) MAX_ITERATION: 25000 IMS_PER_BATCH: 32

But I got the error:

[2020-11-11 19:52:00,830] smoke.engine.inference INFO: Start evaluation on kitti_train dataset(3769 images). 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3769/3769 [02:39<00:00, 23.57it/s] [2020-11-11 19:54:40,734] smoke.engine.inference INFO: Total run time: 0:02:39.904022 (0.042426113425720655 s / img per device, on 1 devices) [2020-11-11 19:54:40,749] smoke.engine.inference INFO: Model inference time: 0:02:16.615843 (0.03624723886867897 s / img per device, on 1 devices) [2020-11-11 19:54:40,749] smoke.data.datasets.evaluation.kitti.kitti_eval INFO: performing kitti detection evaluation: [2020-11-11 19:54:41,392] smoke.data.datasets.evaluation.kitti.kitti_eval INFO: Evaluate on KITTI dataset sh: 1: pdfcrop: not found sh: 1: pdfcrop: not found sh: 1: pdfcrop: not found sh: 1: pdfcrop: not found sh: 1: pdfcrop: not found sh: 1: pdfcrop: not found sh: 1: pdfcrop: not found sh: 1: pdfcrop: not found sh: 1: pdfcrop: not found sh: 1: pdfcrop: not found sh: 1: pdfcrop: not found sh: 1: pdfcrop: not found [2020-11-11 19:54:45,985] smoke.data.datasets.evaluation.kitti.kitti_eval INFO: Thank you for participating in our evaluation! Loading detections... number of files for evaluation: 3769 done. save /home/mlhui/project/SMOKE/tools/logs/inference/kitti_train/plot/car_detection.txt car_detection AP: 0.000000 0.002056 0.002038 save /home/mlhui/project/SMOKE/tools/logs/inference/kitti_train/plot/car_orientation.txt car_orientation AP: 0.000000 0.000981 0.001078 save /home/mlhui/project/SMOKE/tools/logs/inference/kitti_train/plot/pedestrian_detection.txt pedestrian_detection AP: 0.000000 0.000000 0.000000 save /home/mlhui/project/SMOKE/tools/logs/inference/kitti_train/plot/pedestrian_orientation.txt pedestrian_orientation AP: 0.000000 0.000000 0.000000 save /home/mlhui/project/SMOKE/tools/logs/inference/kitti_train/plot/cyclist_detection.txt cyclist_detection AP: 0.000000 0.000000 0.000000 save /home/mlhui/project/SMOKE/tools/logs/inference/kitti_train/plot/cyclist_orientation.txt cyclist_orientation AP: 0.000000 0.000000 0.000000 save /home/mlhui/project/SMOKE/tools/logs/inference/kitti_train/plot/car_detection_ground.txt car_detection_ground AP: 0.000000 0.000000 0.000000 save /home/mlhui/project/SMOKE/tools/logs/inference/kitti_train/plot/pedestrian_detection_ground.txt pedestrian_detection_ground AP: 0.000000 0.000000 0.000000 save /home/mlhui/project/SMOKE/tools/logs/inference/kitti_train/plot/cyclist_detection_ground.txt cyclist_detection_ground AP: 0.000000 0.000000 0.000000 save /home/mlhui/project/SMOKE/tools/logs/inference/kitti_train/plot/car_detection_3d.txt car_detection_3d AP: 0.000000 0.000000 0.000000 save /home/mlhui/project/SMOKE/tools/logs/inference/kitti_train/plot/pedestrian_detection_3d.txt pedestrian_detection_3d AP: 0.000000 0.000000 0.000000 save /home/mlhui/project/SMOKE/tools/logs/inference/kitti_train/plot/cyclist_detection_3d.txt cyclist_detection_3d AP: 0.000000 0.000000 0.000000 Your evaluation results are available at: /home/mlhui/project/SMOKE/tools/logs/inference/kitti_train

can you tell me why? thx!

[chengwang96]

I've conducted some test about training the model and the results are like this: I trained this model using KITTI train (from ImageSets) and validate on val and I got approximately ~ %10 3d detection performance like the paper. Then I trained using trainval and I got ~ %80 3d detection results on val.

+1. Thus, the answer may be that the provided trained model is trained on train+val @dwt189. (Actually the KITTI challege provide no offical train/val split)