OpenPCDet icon indicating copy to clipboard operation
OpenPCDet copied to clipboard

Published 20 hours ago • verified publisher icon open-mmlab

Can not run the evaluation function successful

Open sjtuljw520 opened this issue 3 years ago • 4 comments

2022-09-22 09:36:49,204 INFO Total samples for Waymo dataset: 39987 2022-09-22 09:36:52,147 INFO ==> Loading parameters from checkpoint ../output/waymo_models/pv_rcnn_plusplus_resnet_d1/default/ckpt/checkpoint_epoch_30.pth to GPU 2022-09-22 09:36:52,486 INFO ==> Checkpoint trained from version: pcdet+0.5.2+0000000 2022-09-22 09:36:52,521 INFO ==> Done (loaded 507/507) 2022-09-22 09:36:52,539 INFO *************** EPOCH 30 EVALUATION ***************** eval: 100%|█| 9997/9997 [54:32<00:00, 3.05it/s, recall_0.3=(514650, 515247) / 1 2022-09-22 10:31:25,011 INFO *************** Performance of EPOCH 30 ***************** 2022-09-22 10:31:25,012 INFO Generate label finished(sec_per_example: 0.0818 second). 2022-09-22 10:31:25,012 INFO recall_roi_0.3: 0.285802 2022-09-22 10:31:25,012 INFO recall_rcnn_0.3: 0.286134 2022-09-22 10:31:25,012 INFO recall_roi_0.5: 0.244660 2022-09-22 10:31:25,012 INFO recall_rcnn_0.5: 0.248518 2022-09-22 10:31:25,012 INFO recall_roi_0.7: 0.122996 2022-09-22 10:31:25,012 INFO recall_rcnn_0.7: 0.135234 2022-09-22 10:31:25,041 INFO Average predicted number of objects(39987 samples): 15.276 2022-09-22 10:31:28.316264: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 Start the waymo evaluation... Number: (pd, 610836) VS. (gt, 1630510) Level 1: 1377451, Level2: 253059) WARNING:tensorflow:From /opt/anaconda3/lib/python3.8/contextlib.py:83: TensorFlowTestCase.test_session (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version. Instructions for updating: Use self.session() or self.cached_session() instead. 2022-09-22 10:31:33.802913: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set 2022-09-22 10:31:33.803183: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1 2022-09-22 10:31:33.805971: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Found device 0 with properties: pciBusID: 0000:41:00.0 name: GeForce RTX 3090 computeCapability: 8.6 coreClock: 1.695GHz coreCount: 82 deviceMemorySize: 23.70GiB deviceMemoryBandwidth: 871.81GiB/s 2022-09-22 10:31:33.806008: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 2022-09-22 10:31:33.819510: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11 2022-09-22 10:31:33.819556: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11 2022-09-22 10:31:33.825136: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10 2022-09-22 10:31:33.825918: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10 2022-09-22 10:31:33.826491: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.11 2022-09-22 10:31:33.829262: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11 2022-09-22 10:31:33.829413: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8 2022-09-22 10:31:33.831694: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1888] Adding visible gpu devices: 0 2022-09-22 10:31:33.836754: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set 2022-09-22 10:31:33.837813: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Found device 0 with properties: pciBusID: 0000:41:00.0 name: GeForce RTX 3090 computeCapability: 8.6 coreClock: 1.695GHz coreCount: 82 deviceMemorySize: 23.70GiB deviceMemoryBandwidth: 871.81GiB/s 2022-09-22 10:31:33.837856: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 2022-09-22 10:31:33.837884: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11 2022-09-22 10:31:33.837907: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11 2022-09-22 10:31:33.837927: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10 2022-09-22 10:31:33.837948: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10 2022-09-22 10:31:33.837968: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.11 2022-09-22 10:31:33.837992: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11 2022-09-22 10:31:33.838012: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8 2022-09-22 10:31:33.839832: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1888] Adding visible gpu devices: 0 2022-09-22 10:31:33.840110: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 2022-09-22 10:31:34.135476: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1287] Device interconnect StreamExecutor with strength 1 edge matrix: 2022-09-22 10:31:34.135556: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1293] 0 2022-09-22 10:31:34.135579: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1306] 0: N 2022-09-22 10:31:34.139247: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1390 MB memory) -> physical GPU (device: 0, name: GeForce RTX 3090, pci bus id: 0000:41:00.0, compute capability: 8.6) 2022-09-22 10:31:34.147956: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes) 2022-09-22 10:31:34.152004: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2595055000 Hz WARNING: Logging before InitGoogleLogging() is written to STDERR I0922 10:31:34.384816 3230355 detection_metrics_ops.cc:157] Computing detection metrics for 610836 predicted boxes. I0922 10:31:34.384877 3230355 detection_metrics_ops.cc:159] Parsing prediction [610836,7][610836] I0922 10:31:34.574628 3230355 detection_metrics_ops.cc:168] Parsing ground truth [1630510,9][1630510] 2022-09-22 10:31:34.575550: F waymo_open_dataset/metrics/ops/utils.cc:92] Incorrect number of box DOF 9 test.sh: line 2: 3215304 Aborted (core dumped) python test.py --cfg_file cfgs/waymo_models/pv_rcnn_plusplus_resnet_d1.yaml --batch_size 4 --ckpt ../output/waymo_models/pv_rcnn_plusplus_resnet_d1/default/ckpt/checkpoint_epoch_30.pth

sjtuljw520 avatar Sep 22 '22 02:09 sjtuljw520

Hi, you should use 7-dim boxes (without speed) for evaluation and you can change pred_bbox to pred_bbox[:, :7]. The error log "Incorrect number of box DOF 9" means your boxes is 9-dim (with speed).

Cedarch avatar Sep 22 '22 05:09 Cedarch

Hi, where can I change the pred_bbox to pred_bbox[:, :7]? @Cedarch

sjtuljw520 avatar Sep 22 '22 06:09 sjtuljw520

Line 64 and 71 in OpenPCDet/pcdet/datasets/waymo/waymo_eval.py

Cedarch avatar Sep 22 '22 06:09 Cedarch

Thank you. I have finished the evaluattion, but the results are very bad (model: PV-RCNN++ ResNet): OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/AP: 0.0436 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/APH: 0.0422 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/AP: 0.0373 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/APH: 0.0361 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/AP: 0.0626 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/APH: 0.0540 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/AP: 0.0520 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/APH: 0.0448 OBJECT_TYPE_TYPE_SIGN_LEVEL_1/AP: 0.0000 OBJECT_TYPE_TYPE_SIGN_LEVEL_1/APH: 0.0000 OBJECT_TYPE_TYPE_SIGN_LEVEL_2/AP: 0.0000 OBJECT_TYPE_TYPE_SIGN_LEVEL_2/APH: 0.0000 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/AP: 0.3464 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/APH: 0.3387 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/AP: 0.3337 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/APH: 0.3264

I am not clear why this happens. @Cedarch

sjtuljw520 avatar Sep 22 '22 07:09 sjtuljw520

This issue is stale because it has been open for 30 days with no activity.

github-actions[bot] avatar Oct 23 '22 02:10 github-actions[bot]

This issue was closed because it has been inactive for 14 days since being marked as stale.

github-actions[bot] avatar Nov 07 '22 02:11 github-actions[bot]