OpenPCDet
OpenPCDet copied to clipboard
open-mmlab
Evaluation results are higher than reported
Hi there,
Thank you for preparing such a helpful codebase. I am trying to train some model based on KITTI dataset. As some typical settings, I cut KITTI dataset into 3712 for training while the rest for validation. However, I found that the evaluation results on the val set are higher than the reported ones here. Then I also found that even if I directly evaluation the ckpt listed on the model zoo, the val results are higher than reported. I am pretty confused about this. Hope someone can help me out. Really appreciated!
For example, I tested the pretrained pointpillar and the pointpillar trained on my computer. Their evaluation results are:
Pretrained Pointpillar
INFO Car [email protected], 0.70, 0.70: bbox AP:95.1080, 90.3473, 89.6938 bev AP:90.3344, 89.1101, 88.0095 3d AP:89.3723, 82.6596, 78.1262 aos AP:95.08, 90.23, 89.50 Car [email protected], 0.70, 0.70: bbox AP:97.5711, 95.0270, 94.1727 bev AP:95.2422, 91.5837, 88.7973 3d AP:91.7844, 84.5064, 81.0590 aos AP:97.55, 94.88, 93.94 Car [email protected], 0.50, 0.50: bbox AP:95.1080, 90.3473, 89.6938 bev AP:95.1467, 94.4280, 89.9857 3d AP:95.1178, 90.4278, 89.8449 aos AP:95.08, 90.23, 89.50 Car [email protected], 0.50, 0.50: bbox AP:97.5711, 95.0270, 94.1727 bev AP:97.6159, 96.8839, 94.8833 3d AP:97.5960, 95.2965, 94.6297 aos AP:97.55, 94.88, 93.94 Pedestrian [email protected], 0.50, 0.50: bbox AP:67.7514, 61.9745, 59.2535 bev AP:65.6865, 59.6320, 55.8867 3d AP:60.1515, 55.1462, 51.3551 aos AP:52.70, 48.14, 45.57 Pedestrian [email protected], 0.50, 0.50: bbox AP:68.1002, 61.8393, 58.5606 bev AP:65.5855, 59.0975, 55.1025 3d AP:60.3197, 54.0456, 50.1707 aos AP:50.92, 45.91, 43.07 Pedestrian [email protected], 0.25, 0.25: bbox AP:67.7514, 61.9745, 59.2535 bev AP:74.7678, 70.4506, 66.8868 3d AP:74.6754, 70.1897, 66.6181 aos AP:52.70, 48.14, 45.57 Pedestrian [email protected], 0.25, 0.25: bbox AP:68.1002, 61.8393, 58.5606 bev AP:76.1727, 71.3265, 67.6182 3d AP:76.0753, 70.9777, 67.3507 aos AP:50.92, 45.91, 43.07 Cyclist [email protected], 0.50, 0.50: bbox AP:89.9440, 78.0718, 74.7254 bev AP:87.5247, 72.9536, 69.1656 3d AP:83.4859, 70.7782, 65.7442 aos AP:89.64, 77.22, 73.93 Cyclist [email protected], 0.50, 0.50: bbox AP:92.0630, 79.6339, 75.6316 bev AP:89.5171, 73.7873, 69.4227 3d AP:86.6606, 70.9245, 66.7010 aos AP:91.74, 78.77, 74.72 Cyclist [email protected], 0.25, 0.25: bbox AP:89.9440, 78.0718, 74.7254 bev AP:87.9951, 75.4965, 71.6342 3d AP:87.9951, 75.4965, 71.6326 aos AP:89.64, 77.22, 73.93 Cyclist [email protected], 0.25, 0.25: bbox AP:92.0630, 79.6339, 75.6316 bev AP:90.0781, 76.4402, 72.4053 3d AP:90.0781, 76.4397, 72.4034 aos AP:91.74, 78.77, 74.72
Pointpillar trained on my computer
INFO Car [email protected], 0.70, 0.70: bbox AP:96.4328, 90.3635, 89.8062 bev AP:96.0165, 89.5867, 88.6934 3d AP:89.1395, 83.4021, 78.4925 aos AP:96.42, 90.28, 89.67 Car [email protected], 0.70, 0.70: bbox AP:98.4911, 95.2949, 94.5840 bev AP:98.0588, 93.7431, 91.2211 3d AP:93.9512, 85.2162, 82.0333 aos AP:98.47, 95.19, 94.41 Car [email protected], 0.50, 0.50: bbox AP:96.4328, 90.3635, 89.8062 bev AP:96.5060, 94.5817, 89.9689 3d AP:96.4414, 94.4658, 89.8560 aos AP:96.42, 90.28, 89.67 Car [email protected], 0.50, 0.50: bbox AP:98.4911, 95.2949, 94.5840 bev AP:98.5844, 97.2259, 94.9088 3d AP:98.5500, 96.4737, 94.7115 aos AP:98.47, 95.19, 94.41 Pedestrian [email protected], 0.50, 0.50: bbox AP:67.4846, 60.8238, 58.7901 bev AP:64.6022, 57.7348, 55.3456 3d AP:60.2626, 54.5934, 50.8046 aos AP:57.49, 51.82, 49.87 Pedestrian [email protected], 0.50, 0.50: bbox AP:67.7127, 60.7845, 58.0962 bev AP:64.4125, 57.2149, 54.1941 3d AP:60.3154, 52.8538, 49.6810 aos AP:56.39, 50.35, 47.77 Pedestrian [email protected], 0.25, 0.25: bbox AP:67.4846, 60.8238, 58.7901 bev AP:75.0180, 68.8051, 65.8184 3d AP:74.9583, 68.5116, 65.5478 aos AP:57.49, 51.82, 49.87 Pedestrian [email protected], 0.25, 0.25: bbox AP:67.7127, 60.7845, 58.0962 bev AP:76.2086, 69.2759, 66.1454 3d AP:76.1365, 68.8249, 65.7820 aos AP:56.39, 50.35, 47.77 Cyclist [email protected], 0.50, 0.50: bbox AP:87.7309, 84.1966, 80.9509 bev AP:85.3966, 80.3185, 74.2402 3d AP:84.0493, 80.0770, 74.0290 aos AP:87.65, 83.95, 80.68 Cyclist [email protected], 0.50, 0.50: bbox AP:91.3088, 86.4835, 81.7229 bev AP:88.5775, 81.2581, 76.1044 3d AP:87.2568, 80.7347, 75.6731 aos AP:91.22, 86.20, 81.44 Cyclist [email protected], 0.25, 0.25: bbox AP:87.7309, 84.1966, 80.9509 bev AP:86.3137, 81.1739, 78.2467 3d AP:86.3131, 81.1739, 78.1395 aos AP:87.65, 83.95, 80.68 Cyclist [email protected], 0.25, 0.25: bbox AP:91.3088, 86.4835, 81.7229 bev AP:89.7548, 83.2075, 78.6361 3d AP:89.7546, 83.2070, 78.6063 aos AP:91.22, 86.20, 81.44
While the listed val results for pointpillar are
| training time | Car@R11 | Pedestrian@R11 | Cyclist@R11 | download | |
|---|---|---|---|---|---|
| PointPillar | ~1.2 hours | 77.28 | 52.29 | 62.68 | model-18M |
I did that also and the official testing results are much lower than the results I got and closer to the results shown here. So I am very confused why would this happen.
Hi Jihan,
Thank you for your help.
By checking the log file in the MMDetection3D repo., for PointPillar, I found that there are two different criteria for a single evaluation category. For example, for Moderate Car at 11position, it has Car_3D_AP11_moderate_loose": 89.99775 and Car_3D_AP11_moderate_strict": 77.62063. It seems that what I got is more similar to the results under 'loose', while 'strict' is shown here and there. So could you please confirm this? and how could I change the criteria to 'strict' for evaluation on the OpenPCDet repo. Thank you very much.
Zhengwei
Actually, I recently found that everything goes right on MMDetection3D. But I still got higher evaluation results on OpenPCDet and I don't know why. The dataset should be good cuz I have reasonable results on MMDetection3D. But mmdetection3d does not support Voxel RCNN and VoxelNet. So I really hope that this problem can be solved.
Sorry, I have never meet this issue before and have no idea about this issue.
Can you provide the full evaluation log by using our provided pretrained model? I think you need to get exactly the same mAP as we listed.
Sorry for the late reply. Please see the log file below. Current evaluation results are based on KITTI val split (3,769 samples).
(open-mmlab) cert@cert-OMEN-30L-Desktop-GT13-0xxx:~/Documents/Zhengwei/PCD-Detection/OpenPCDet-052-dev/tools$ python test.py --cfg_file cfgs/kitti_models/pointpillar.yaml --ckpt ../checkpoints/pointpillar_7728.pth
2022-06-13 13:27:12,309 INFO **********************Start logging**********************
2022-06-13 13:27:12,309 INFO CUDA_VISIBLE_DEVICES=ALL
2022-06-13 13:27:12,309 INFO cfg_file cfgs/kitti_models/pointpillar.yaml
2022-06-13 13:27:12,309 INFO batch_size 4
2022-06-13 13:27:12,309 INFO workers 4
2022-06-13 13:27:12,309 INFO extra_tag default
2022-06-13 13:27:12,309 INFO ckpt ../checkpoints/pointpillar_7728.pth
2022-06-13 13:27:12,309 INFO launcher none
2022-06-13 13:27:12,309 INFO tcp_port 18888
2022-06-13 13:27:12,309 INFO local_rank 0
2022-06-13 13:27:12,309 INFO set_cfgs None
2022-06-13 13:27:12,309 INFO max_waiting_mins 30
2022-06-13 13:27:12,309 INFO start_epoch 0
2022-06-13 13:27:12,309 INFO eval_tag default
2022-06-13 13:27:12,310 INFO eval_all False
2022-06-13 13:27:12,310 INFO ckpt_dir None
2022-06-13 13:27:12,310 INFO save_to_file False
2022-06-13 13:27:12,310 INFO cfg.ROOT_DIR: /home/cert/Documents/Zhengwei/PCD-Detection/OpenPCDet-052-dev
2022-06-13 13:27:12,310 INFO cfg.LOCAL_RANK: 0
2022-06-13 13:27:12,310 INFO cfg.CLASS_NAMES: ['Car', 'Pedestrian', 'Cyclist']
2022-06-13 13:27:12,310 INFO
cfg.DATA_CONFIG = edict()
2022-06-13 13:27:12,310 INFO cfg.DATA_CONFIG.DATASET: KittiDataset
2022-06-13 13:27:12,310 INFO cfg.DATA_CONFIG.DATA_PATH: ../data/kitti
2022-06-13 13:27:12,310 INFO cfg.DATA_CONFIG.POINT_CLOUD_RANGE: [0, -39.68, -3, 69.12, 39.68, 1]
2022-06-13 13:27:12,310 INFO
cfg.DATA_CONFIG.DATA_SPLIT = edict()
2022-06-13 13:27:12,310 INFO cfg.DATA_CONFIG.DATA_SPLIT.train: train
2022-06-13 13:27:12,310 INFO cfg.DATA_CONFIG.DATA_SPLIT.test: val
2022-06-13 13:27:12,310 INFO
cfg.DATA_CONFIG.INFO_PATH = edict()
2022-06-13 13:27:12,310 INFO cfg.DATA_CONFIG.INFO_PATH.train: ['kitti_infos_train.pkl']
2022-06-13 13:27:12,310 INFO cfg.DATA_CONFIG.INFO_PATH.test: ['kitti_infos_val.pkl']
2022-06-13 13:27:12,310 INFO cfg.DATA_CONFIG.GET_ITEM_LIST: ['points']
2022-06-13 13:27:12,310 INFO cfg.DATA_CONFIG.FOV_POINTS_ONLY: True
2022-06-13 13:27:12,310 INFO
cfg.DATA_CONFIG.DATA_AUGMENTOR = edict()
2022-06-13 13:27:12,310 INFO cfg.DATA_CONFIG.DATA_AUGMENTOR.DISABLE_AUG_LIST: ['placeholder']
2022-06-13 13:27:12,310 INFO cfg.DATA_CONFIG.DATA_AUGMENTOR.AUG_CONFIG_LIST: [{'NAME': 'gt_sampling', 'USE_ROAD_PLANE': False, 'DB_INFO_PATH': ['kitti_dbinfos_train.pkl'], 'PREPARE': {'filter_by_min_points': ['Car:5', 'Pedestrian:5', 'Cyclist:5'], 'filter_by_difficulty': [-1]}, 'SAMPLE_GROUPS': ['Car:15', 'Pedestrian:15', 'Cyclist:15'], 'NUM_POINT_FEATURES': 4, 'DATABASE_WITH_FAKELIDAR': False, 'REMOVE_EXTRA_WIDTH': [0.0, 0.0, 0.0], 'LIMIT_WHOLE_SCENE': False}, {'NAME': 'random_world_flip', 'ALONG_AXIS_LIST': ['x']}, {'NAME': 'random_world_rotation', 'WORLD_ROT_ANGLE': [-0.78539816, 0.78539816]}, {'NAME': 'random_world_scaling', 'WORLD_SCALE_RANGE': [0.95, 1.05]}]
2022-06-13 13:27:12,310 INFO
cfg.DATA_CONFIG.POINT_FEATURE_ENCODING = edict()
2022-06-13 13:27:12,310 INFO cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.encoding_type: absolute_coordinates_encoding
2022-06-13 13:27:12,310 INFO cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.used_feature_list: ['x', 'y', 'z', 'intensity']
2022-06-13 13:27:12,310 INFO cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.src_feature_list: ['x', 'y', 'z', 'intensity']
2022-06-13 13:27:12,310 INFO cfg.DATA_CONFIG.DATA_PROCESSOR: [{'NAME': 'mask_points_and_boxes_outside_range', 'REMOVE_OUTSIDE_BOXES': True}, {'NAME': 'shuffle_points', 'SHUFFLE_ENABLED': {'train': True, 'test': False}}, {'NAME': 'transform_points_to_voxels', 'VOXEL_SIZE': [0.16, 0.16, 4], 'MAX_POINTS_PER_VOXEL': 32, 'MAX_NUMBER_OF_VOXELS': {'train': 16000, 'test': 40000}}]
2022-06-13 13:27:12,310 INFO cfg.DATA_CONFIG._BASE_CONFIG_: cfgs/dataset_configs/kitti_dataset.yaml
2022-06-13 13:27:12,310 INFO
cfg.MODEL = edict()
2022-06-13 13:27:12,310 INFO cfg.MODEL.NAME: PointPillar
2022-06-13 13:27:12,310 INFO
cfg.MODEL.VFE = edict()
2022-06-13 13:27:12,310 INFO cfg.MODEL.VFE.NAME: PillarVFE
2022-06-13 13:27:12,310 INFO cfg.MODEL.VFE.WITH_DISTANCE: False
2022-06-13 13:27:12,310 INFO cfg.MODEL.VFE.USE_ABSLOTE_XYZ: True
2022-06-13 13:27:12,311 INFO cfg.MODEL.VFE.USE_NORM: True
2022-06-13 13:27:12,311 INFO cfg.MODEL.VFE.NUM_FILTERS: [64]
2022-06-13 13:27:12,311 INFO
cfg.MODEL.MAP_TO_BEV = edict()
2022-06-13 13:27:12,311 INFO cfg.MODEL.MAP_TO_BEV.NAME: PointPillarScatter
2022-06-13 13:27:12,311 INFO cfg.MODEL.MAP_TO_BEV.NUM_BEV_FEATURES: 64
2022-06-13 13:27:12,311 INFO
cfg.MODEL.BACKBONE_2D = edict()
2022-06-13 13:27:12,311 INFO cfg.MODEL.BACKBONE_2D.NAME: BaseBEVBackbone
2022-06-13 13:27:12,311 INFO cfg.MODEL.BACKBONE_2D.LAYER_NUMS: [3, 5, 5]
2022-06-13 13:27:12,311 INFO cfg.MODEL.BACKBONE_2D.LAYER_STRIDES: [2, 2, 2]
2022-06-13 13:27:12,311 INFO cfg.MODEL.BACKBONE_2D.NUM_FILTERS: [64, 128, 256]
2022-06-13 13:27:12,311 INFO cfg.MODEL.BACKBONE_2D.UPSAMPLE_STRIDES: [1, 2, 4]
2022-06-13 13:27:12,311 INFO cfg.MODEL.BACKBONE_2D.NUM_UPSAMPLE_FILTERS: [128, 128, 128]
2022-06-13 13:27:12,311 INFO
cfg.MODEL.DENSE_HEAD = edict()
2022-06-13 13:27:12,311 INFO cfg.MODEL.DENSE_HEAD.NAME: AnchorHeadSingle
2022-06-13 13:27:12,311 INFO cfg.MODEL.DENSE_HEAD.CLASS_AGNOSTIC: False
2022-06-13 13:27:12,311 INFO cfg.MODEL.DENSE_HEAD.USE_DIRECTION_CLASSIFIER: True
2022-06-13 13:27:12,311 INFO cfg.MODEL.DENSE_HEAD.DIR_OFFSET: 0.78539
2022-06-13 13:27:12,311 INFO cfg.MODEL.DENSE_HEAD.DIR_LIMIT_OFFSET: 0.0
2022-06-13 13:27:12,311 INFO cfg.MODEL.DENSE_HEAD.NUM_DIR_BINS: 2
2022-06-13 13:27:12,311 INFO cfg.MODEL.DENSE_HEAD.ANCHOR_GENERATOR_CONFIG: [{'class_name': 'Car', 'anchor_sizes': [[3.9, 1.6, 1.56]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [-1.78], 'align_center': False, 'feature_map_stride': 2, 'matched_threshold': 0.6, 'unmatched_threshold': 0.45}, {'class_name': 'Pedestrian', 'anchor_sizes': [[0.8, 0.6, 1.73]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [-0.6], 'align_center': False, 'feature_map_stride': 2, 'matched_threshold': 0.5, 'unmatched_threshold': 0.35}, {'class_name': 'Cyclist', 'anchor_sizes': [[1.76, 0.6, 1.73]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [-0.6], 'align_center': False, 'feature_map_stride': 2, 'matched_threshold': 0.5, 'unmatched_threshold': 0.35}]
2022-06-13 13:27:12,311 INFO
cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG = edict()
2022-06-13 13:27:12,311 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.NAME: AxisAlignedTargetAssigner
2022-06-13 13:27:12,311 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.POS_FRACTION: -1.0
2022-06-13 13:27:12,311 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.SAMPLE_SIZE: 512
2022-06-13 13:27:12,311 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.NORM_BY_NUM_EXAMPLES: False
2022-06-13 13:27:12,311 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.MATCH_HEIGHT: False
2022-06-13 13:27:12,311 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.BOX_CODER: ResidualCoder
2022-06-13 13:27:12,311 INFO
cfg.MODEL.DENSE_HEAD.LOSS_CONFIG = edict()
2022-06-13 13:27:12,311 INFO
cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS = edict()
2022-06-13 13:27:12,311 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.cls_weight: 1.0
2022-06-13 13:27:12,311 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.loc_weight: 2.0
2022-06-13 13:27:12,311 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.dir_weight: 0.2
2022-06-13 13:27:12,311 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.code_weights: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
2022-06-13 13:27:12,312 INFO
cfg.MODEL.POST_PROCESSING = edict()
2022-06-13 13:27:12,312 INFO cfg.MODEL.POST_PROCESSING.RECALL_THRESH_LIST: [0.3, 0.5, 0.7]
2022-06-13 13:27:12,312 INFO cfg.MODEL.POST_PROCESSING.SCORE_THRESH: 0.1
2022-06-13 13:27:12,312 INFO cfg.MODEL.POST_PROCESSING.OUTPUT_RAW_SCORE: False
2022-06-13 13:27:12,312 INFO cfg.MODEL.POST_PROCESSING.EVAL_METRIC: kitti
2022-06-13 13:27:12,312 INFO
cfg.MODEL.POST_PROCESSING.NMS_CONFIG = edict()
2022-06-13 13:27:12,312 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.MULTI_CLASSES_NMS: False
2022-06-13 13:27:12,312 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_TYPE: nms_gpu
2022-06-13 13:27:12,312 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_THRESH: 0.01
2022-06-13 13:27:12,312 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_PRE_MAXSIZE: 4096
2022-06-13 13:27:12,312 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_POST_MAXSIZE: 500
2022-06-13 13:27:12,312 INFO
cfg.OPTIMIZATION = edict()
2022-06-13 13:27:12,312 INFO cfg.OPTIMIZATION.BATCH_SIZE_PER_GPU: 4
2022-06-13 13:27:12,312 INFO cfg.OPTIMIZATION.NUM_EPOCHS: 80
2022-06-13 13:27:12,312 INFO cfg.OPTIMIZATION.OPTIMIZER: adam_onecycle
2022-06-13 13:27:12,312 INFO cfg.OPTIMIZATION.LR: 0.003
2022-06-13 13:27:12,312 INFO cfg.OPTIMIZATION.WEIGHT_DECAY: 0.01
2022-06-13 13:27:12,312 INFO cfg.OPTIMIZATION.MOMENTUM: 0.9
2022-06-13 13:27:12,312 INFO cfg.OPTIMIZATION.MOMS: [0.95, 0.85]
2022-06-13 13:27:12,312 INFO cfg.OPTIMIZATION.PCT_START: 0.4
2022-06-13 13:27:12,312 INFO cfg.OPTIMIZATION.DIV_FACTOR: 10
2022-06-13 13:27:12,312 INFO cfg.OPTIMIZATION.DECAY_STEP_LIST: [35, 45]
2022-06-13 13:27:12,312 INFO cfg.OPTIMIZATION.LR_DECAY: 0.1
2022-06-13 13:27:12,312 INFO cfg.OPTIMIZATION.LR_CLIP: 1e-07
2022-06-13 13:27:12,312 INFO cfg.OPTIMIZATION.LR_WARMUP: False
2022-06-13 13:27:12,312 INFO cfg.OPTIMIZATION.WARMUP_EPOCH: 1
2022-06-13 13:27:12,312 INFO cfg.OPTIMIZATION.GRAD_NORM_CLIP: 10
2022-06-13 13:27:12,312 INFO cfg.TAG: pointpillar
2022-06-13 13:27:12,312 INFO cfg.EXP_GROUP_PATH: kitti_models
2022-06-13 13:27:12,313 INFO Loading KITTI dataset
2022-06-13 13:27:12,404 INFO Total samples for KITTI dataset: 3769
2022-06-13 13:27:14,447 INFO ==> Loading parameters from checkpoint ../checkpoints/pointpillar_7728.pth to GPU
2022-06-13 13:27:14,472 INFO ==> Done (loaded 127/127)
2022-06-13 13:27:14,479 INFO *************** EPOCH 7728 EVALUATION *****************
eval: 100%|████████████████████████████████████████████████████████████████████████████| 943/943 [01:18<00:00, 12.06it/s, recall_0.3=(0, 16639) / 17666]
2022-06-13 13:28:32,698 INFO *************** Performance of EPOCH 7728 *****************
2022-06-13 13:28:32,698 INFO Generate label finished(sec_per_example: 0.0208 second).
2022-06-13 13:28:32,698 INFO recall_roi_0.3: 0.000000
2022-06-13 13:28:32,698 INFO recall_rcnn_0.3: 0.941866
2022-06-13 13:28:32,698 INFO recall_roi_0.5: 0.000000
2022-06-13 13:28:32,698 INFO recall_rcnn_0.5: 0.896921
2022-06-13 13:28:32,698 INFO recall_roi_0.7: 0.000000
2022-06-13 13:28:32,698 INFO recall_rcnn_0.7: 0.687762
2022-06-13 13:28:32,700 INFO Average predicted number of objects(3769 samples): 16.820
2022-06-13 13:28:52,616 INFO Car [email protected], 0.70, 0.70:
bbox AP:95.1066, 90.3473, 89.6948
bev AP:90.3344, 89.1101, 88.0080
3d AP:89.3723, 82.6615, 78.1272
aos AP:95.08, 90.23, 89.51
Car [email protected], 0.70, 0.70:
bbox AP:97.5707, 95.0272, 94.1732
bev AP:95.2419, 91.5831, 88.7980
3d AP:91.7842, 84.5068, 81.0629
aos AP:97.55, 94.88, 93.94
Car [email protected], 0.50, 0.50:
bbox AP:95.1066, 90.3473, 89.6948
bev AP:95.1454, 94.4243, 89.9849
3d AP:95.1165, 90.4278, 89.8449
aos AP:95.08, 90.23, 89.51
Car [email protected], 0.50, 0.50:
bbox AP:97.5707, 95.0272, 94.1732
bev AP:97.6155, 96.8824, 94.8830
3d AP:97.5957, 95.2959, 94.6293
aos AP:97.55, 94.88, 93.94
Pedestrian [email protected], 0.50, 0.50:
bbox AP:67.7352, 61.9361, 59.2172
bev AP:65.6865, 59.6377, 55.8745
3d AP:60.1476, 55.1442, 51.3606
aos AP:52.69, 48.12, 45.54
Pedestrian [email protected], 0.50, 0.50:
bbox AP:68.0785, 61.7656, 58.4901
bev AP:65.5930, 59.1030, 55.0963
3d AP:60.3141, 54.0443, 50.1702
aos AP:50.91, 45.85, 42.97
Pedestrian [email protected], 0.25, 0.25:
bbox AP:67.7352, 61.9361, 59.2172
bev AP:74.7629, 70.4506, 66.8877
3d AP:74.6705, 70.2080, 66.6142
aos AP:52.69, 48.12, 45.54
Pedestrian [email protected], 0.25, 0.25:
bbox AP:68.0785, 61.7656, 58.4901
bev AP:76.1737, 71.3237, 67.6229
3d AP:76.0762, 70.9840, 67.3524
aos AP:50.91, 45.85, 42.97
Cyclist [email protected], 0.50, 0.50:
bbox AP:89.9482, 78.0703, 74.7261
bev AP:87.5247, 72.9537, 69.1656
3d AP:83.4698, 70.7782, 65.6989
aos AP:89.65, 77.21, 73.93
Cyclist [email protected], 0.50, 0.50:
bbox AP:92.0641, 79.6335, 75.6346
bev AP:89.5253, 73.7474, 69.4267
3d AP:86.6561, 70.8815, 66.6887
aos AP:91.74, 78.76, 74.73
Cyclist [email protected], 0.25, 0.25:
bbox AP:89.9482, 78.0703, 74.7261
bev AP:87.9951, 75.4949, 71.6397
3d AP:87.9951, 75.4949, 71.6381
aos AP:89.65, 77.21, 73.93
Cyclist [email protected], 0.25, 0.25:
bbox AP:92.0641, 79.6335, 75.6346
bev AP:90.0865, 76.4009, 72.4089
3d AP:90.0865, 76.4004, 72.4065
aos AP:91.74, 78.76, 74.73
2022-06-13 13:28:52,620 INFO Result is save to /home/cert/Documents/Zhengwei/PCD-Detection/OpenPCDet-052-dev/output/kitti_models/pointpillar/default/eval/epoch_7728/val/default
2022-06-13 13:28:52,621 INFO ****************Evaluation done.*****************
Actually, I also tried PV-RCNN, SECOND by directly using the listed pretrained model and got the results all higher than what they should be... Here I also attached the env_info generated by mmdetection3d, since it provides a script and I use the same env for both OpenPCDet and MMdetection3D. Hope it can help and thank you very much!
(open-mmlab) cert@cert-OMEN-30L-Desktop-GT13-0xxx:~/Documents/Zhengwei/mmdetection3d-018-dev$ python mmdet3d/utils/collect_env.py
sys.platform: linux
Python: 3.7.13 (default, Mar 29 2022, 02:18:16) [GCC 7.5.0]
CUDA available: True
GPU 0: NVIDIA GeForce RTX 3090
CUDA_HOME: /usr/local/cuda-11.1
NVCC: Build cuda_11.1.TC455_06.29069683_0
GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
PyTorch: 1.9.0+cu111
PyTorch compiling details: PyTorch built with:
- GCC 7.3
- C++ Version: 201402
- Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v2.1.2 (Git Hash 98be7e8afa711dc9b66c8ff3504129cb82013cdb)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- NNPACK is enabled
- CPU capability usage: AVX2
- CUDA Runtime 11.1
- NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
- CuDNN 8.0.5
- Magma 2.5.2
- Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.9.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,
TorchVision: 0.10.0+cu111
OpenCV: 4.5.5
MMCV: 1.3.17
MMCV Compiler: GCC 7.3
MMCV CUDA Compiler: 11.1
MMDetection: 2.19.0
MMSegmentation: 0.20.0
MMDetection3D: 0.18.0+5edad06
@zwbai offtopic question: how do you generate this evaluation statistics? a tool or custom code ?
This issue is stale because it has been open for 30 days with no activity.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue was closed because it has been inactive for 14 days since being marked as stale.
@zwbai I am facing a similar issue https://github.com/open-mmlab/OpenPCDet/issues/1463 . But in my case, its lower than pre-trained results. Did you find the reason for the correlation?
