OpenPCDet
OpenPCDet copied to clipboard
open-mmlab
Issue with KITTI "test set" evaluation turning all 0 values
I've successfully trained and evaluated several models using "kitti_infos_train.pkl" and "kitti_infos_val.pkl" without any problems. However, when I attempted to test using "kitti_infos_test.pkl," all the returned recall values are showing as 0!
Below, I've provided my configuration and experiment details. I would greatly appreciate it if someone could assist me in troubleshooting this issue. Thank you.
Openpcdet version: pcdet+0.6.0
Testing model: PVRCNN
- bash scripts/dist_test.sh 2 --cfg_file cfgs/kitti_models/pv_rcnn.yaml --ckpt ../output/kitti_models/pv_rcnn/default/ckpt/checkpoint_epoch_78.pth
Evaluation result: return _VF.meshgrid(tensors, *kwargs) # type: ignore[attr-defined] 2024-04-06 17:25:30,613 INFO ==> Loading parameters from checkpoint ../output/kitti_models/pv_rcnn_09_final/default/ckpt/checkpoint_epoch_78.pth to GPU 2024-04-06 17:25:30,695 INFO ==> Checkpoint trained from version: pcdet+0.6.0+255db8f+pybd06a67 2024-04-06 17:25:30,721 INFO ==> Done (loaded 367/367) 2024-04-06 17:25:30,740 INFO *************** EPOCH 78 EVALUATION ***************** eval: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3759/3759 [07:27<00:00, 8.40it/s, recall_0.3=(0, 0) / 0] 2024-04-06 17:32:58,430 INFO *************** Performance of EPOCH 78 ***************** 2024-04-06 17:32:58,430 INFO Generate label finished(sec_per_example: 0.0595 second). 2024-04-06 17:32:58,430 INFO recall_roi_0.3: 0.000000 2024-04-06 17:32:58,430 INFO recall_rcnn_0.3: 0.000000 2024-04-06 17:32:58,430 INFO recall_roi_0.5: 0.000000 2024-04-06 17:32:58,430 INFO recall_rcnn_0.5: 0.000000 2024-04-06 17:32:58,430 INFO recall_roi_0.7: 0.000000 2024-04-06 17:32:58,430 INFO recall_rcnn_0.7: 0.000000 2024-04-06 17:32:58,441 INFO Average predicted number of objects(7518 samples): 8.798 2024-04-06 17:32:58,779 INFO None 2024-04-06 17:32:58,779 INFO Result is saved to /hdd_10tb/sina/Radial_MAE/output/kitti_models/pv_rcnn/default/eval/epoch_78/test/default 2024-04-06 17:32:58,779 INFO Evaluation done.
kitti_dataset.yaml:
DATASET: 'KittiDataset' DATA_PATH: '../data/kitti'
POINT_CLOUD_RANGE: [0, -40, -3, 70.4, 40, 1]
DATA_SPLIT: { 'train': train, 'test': test }
INFO_PATH: { 'train': [kitti_infos_train.pkl], 'test': [kitti_infos_test.pkl], }
GET_ITEM_LIST: ["points"] FOV_POINTS_ONLY: True
DATA_AUGMENTOR: DISABLE_AUG_LIST: ['placeholder'] AUG_CONFIG_LIST: - NAME: gt_sampling USE_ROAD_PLANE: True DB_INFO_PATH: - kitti_dbinfos_train.pkl PREPARE: { filter_by_min_points: ['Car:5', 'Pedestrian:5', 'Cyclist:5'], filter_by_difficulty: [-1], }
SAMPLE_GROUPS: ['Car:20','Pedestrian:15', 'Cyclist:15']
NUM_POINT_FEATURES: 4
DATABASE_WITH_FAKELIDAR: False
REMOVE_EXTRA_WIDTH: [0.0, 0.0, 0.0]
LIMIT_WHOLE_SCENE: True
- NAME: random_world_flip
ALONG_AXIS_LIST: ['x']
- NAME: random_world_rotation
WORLD_ROT_ANGLE: [-0.78539816, 0.78539816]
- NAME: random_world_scaling
WORLD_SCALE_RANGE: [0.95, 1.05]
POINT_FEATURE_ENCODING: { encoding_type: absolute_coordinates_encoding, used_feature_list: ['x', 'y', 'z', 'intensity'], src_feature_list: ['x', 'y', 'z', 'intensity'], }
DATA_PROCESSOR: - NAME: mask_points_and_boxes_outside_range REMOVE_OUTSIDE_BOXES: True
- NAME: shuffle_points
SHUFFLE_ENABLED: {
'train': True,
'test': False
}
- NAME: transform_points_to_voxels
VOXEL_SIZE: [0.05, 0.05, 0.1]
MAX_POINTS_PER_VOXEL: 5
MAX_NUMBER_OF_VOXELS: {
'train': 16000,
'test': 40000
}
pv_rcnn.yaml:
CLASS_NAMES: ['Car', 'Pedestrian', 'Cyclist']
DATA_CONFIG: BASE_CONFIG: cfgs/dataset_configs/kitti_dataset.yaml DATA_AUGMENTOR: DISABLE_AUG_LIST: ['placeholder'] AUG_CONFIG_LIST: - NAME: gt_sampling USE_ROAD_PLANE: False DB_INFO_PATH: - kitti_dbinfos_train.pkl PREPARE: { filter_by_min_points: ['Car:5', 'Pedestrian:5', 'Cyclist:5'], filter_by_difficulty: [-1], }
SAMPLE_GROUPS: ['Car:15','Pedestrian:10', 'Cyclist:10']
NUM_POINT_FEATURES: 4
DATABASE_WITH_FAKELIDAR: False
REMOVE_EXTRA_WIDTH: [0.0, 0.0, 0.0]
LIMIT_WHOLE_SCENE: False
- NAME: random_world_flip
ALONG_AXIS_LIST: ['x']
- NAME: random_world_rotation
WORLD_ROT_ANGLE: [-0.78539816, 0.78539816]
- NAME: random_world_scaling
WORLD_SCALE_RANGE: [0.95, 1.05]
MODEL: NAME: PVRCNN
VFE:
NAME: MeanVFE
BACKBONE_3D:
NAME: VoxelBackBone8x
MAP_TO_BEV:
NAME: HeightCompression
NUM_BEV_FEATURES: 256
BACKBONE_2D:
NAME: BaseBEVBackbone
LAYER_NUMS: [5, 5]
LAYER_STRIDES: [1, 2]
NUM_FILTERS: [128, 256]
UPSAMPLE_STRIDES: [1, 2]
NUM_UPSAMPLE_FILTERS: [256, 256]
DENSE_HEAD:
NAME: AnchorHeadSingle
CLASS_AGNOSTIC: False
USE_DIRECTION_CLASSIFIER: True
DIR_OFFSET: 0.78539
DIR_LIMIT_OFFSET: 0.0
NUM_DIR_BINS: 2
ANCHOR_GENERATOR_CONFIG: [
{
'class_name': 'Car',
'anchor_sizes': [[3.9, 1.6, 1.56]],
'anchor_rotations': [0, 1.57],
'anchor_bottom_heights': [-1.78],
'align_center': False,
'feature_map_stride': 8,
'matched_threshold': 0.6,
'unmatched_threshold': 0.45
},
{
'class_name': 'Pedestrian',
'anchor_sizes': [[0.8, 0.6, 1.73]],
'anchor_rotations': [0, 1.57],
'anchor_bottom_heights': [-0.6],
'align_center': False,
'feature_map_stride': 8,
'matched_threshold': 0.5,
'unmatched_threshold': 0.35
},
{
'class_name': 'Cyclist',
'anchor_sizes': [[1.76, 0.6, 1.73]],
'anchor_rotations': [0, 1.57],
'anchor_bottom_heights': [-0.6],
'align_center': False,
'feature_map_stride': 8,
'matched_threshold': 0.5,
'unmatched_threshold': 0.35
}
]
TARGET_ASSIGNER_CONFIG:
NAME: AxisAlignedTargetAssigner
POS_FRACTION: -1.0
SAMPLE_SIZE: 512
NORM_BY_NUM_EXAMPLES: False
MATCH_HEIGHT: False
BOX_CODER: ResidualCoder
LOSS_CONFIG:
LOSS_WEIGHTS: {
'cls_weight': 1.0,
'loc_weight': 2.0,
'dir_weight': 0.2,
'code_weights': [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
}
PFE:
NAME: VoxelSetAbstraction
POINT_SOURCE: raw_points
NUM_KEYPOINTS: 2048
NUM_OUTPUT_FEATURES: 128
SAMPLE_METHOD: FPS
FEATURES_SOURCE: ['bev', 'x_conv1', 'x_conv2', 'x_conv3', 'x_conv4', 'raw_points']
SA_LAYER:
raw_points:
MLPS: [[16, 16], [16, 16]]
POOL_RADIUS: [0.4, 0.8]
NSAMPLE: [16, 16]
x_conv1:
DOWNSAMPLE_FACTOR: 1
MLPS: [[16, 16], [16, 16]]
POOL_RADIUS: [0.4, 0.8]
NSAMPLE: [16, 16]
x_conv2:
DOWNSAMPLE_FACTOR: 2
MLPS: [[32, 32], [32, 32]]
POOL_RADIUS: [0.8, 1.2]
NSAMPLE: [16, 32]
x_conv3:
DOWNSAMPLE_FACTOR: 4
MLPS: [[64, 64], [64, 64]]
POOL_RADIUS: [1.2, 2.4]
NSAMPLE: [16, 32]
x_conv4:
DOWNSAMPLE_FACTOR: 8
MLPS: [[64, 64], [64, 64]]
POOL_RADIUS: [2.4, 4.8]
NSAMPLE: [16, 32]
POINT_HEAD:
NAME: PointHeadSimple
CLS_FC: [256, 256]
CLASS_AGNOSTIC: True
USE_POINT_FEATURES_BEFORE_FUSION: True
TARGET_CONFIG:
GT_EXTRA_WIDTH: [0.2, 0.2, 0.2]
LOSS_CONFIG:
LOSS_REG: smooth-l1
LOSS_WEIGHTS: {
'point_cls_weight': 1.0,
}
ROI_HEAD:
NAME: PVRCNNHead
CLASS_AGNOSTIC: True
SHARED_FC: [256, 256]
CLS_FC: [256, 256]
REG_FC: [256, 256]
DP_RATIO: 0.3
NMS_CONFIG:
TRAIN:
NMS_TYPE: nms_gpu
MULTI_CLASSES_NMS: False
NMS_PRE_MAXSIZE: 9000
NMS_POST_MAXSIZE: 512
NMS_THRESH: 0.8
TEST:
NMS_TYPE: nms_gpu
MULTI_CLASSES_NMS: False
NMS_PRE_MAXSIZE: 1024
NMS_POST_MAXSIZE: 100
NMS_THRESH: 0.7
ROI_GRID_POOL:
GRID_SIZE: 6
MLPS: [[64, 64], [64, 64]]
POOL_RADIUS: [0.8, 1.6]
NSAMPLE: [16, 16]
POOL_METHOD: max_pool
TARGET_CONFIG:
BOX_CODER: ResidualCoder
ROI_PER_IMAGE: 128
FG_RATIO: 0.5
SAMPLE_ROI_BY_EACH_CLASS: True
CLS_SCORE_TYPE: roi_iou
CLS_FG_THRESH: 0.75
CLS_BG_THRESH: 0.25
CLS_BG_THRESH_LO: 0.1
HARD_BG_RATIO: 0.8
REG_FG_THRESH: 0.55
LOSS_CONFIG:
CLS_LOSS: BinaryCrossEntropy
REG_LOSS: smooth-l1
CORNER_LOSS_REGULARIZATION: True
LOSS_WEIGHTS: {
'rcnn_cls_weight': 1.0,
'rcnn_reg_weight': 1.0,
'rcnn_corner_weight': 1.0,
'code_weights': [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
}
POST_PROCESSING:
RECALL_THRESH_LIST: [0.3, 0.5, 0.7]
SCORE_THRESH: 0.1
OUTPUT_RAW_SCORE: False
EVAL_METRIC: kitti
NMS_CONFIG:
MULTI_CLASSES_NMS: False
NMS_TYPE: nms_gpu
NMS_THRESH: 0.1
NMS_PRE_MAXSIZE: 4096
NMS_POST_MAXSIZE: 500
OPTIMIZATION: BATCH_SIZE_PER_GPU: 2 NUM_EPOCHS: 80
OPTIMIZER: adam_onecycle
LR: 0.01
WEIGHT_DECAY: 0.01
MOMENTUM: 0.9
MOMS: [0.95, 0.85]
PCT_START: 0.4
DIV_FACTOR: 10
DECAY_STEP_LIST: [35, 45]
LR_DECAY: 0.1
LR_CLIP: 0.0000001
LR_WARMUP: False
WARMUP_EPOCH: 1
GRAD_NORM_CLIP: 10
Environment: Package Version Editable project location
absl-py 0.15.0 addict 2.4.0 aiofiles 22.1.0 aiosqlite 0.19.0 anyio 3.7.1 appdirs 1.4.4 argon2-cffi 23.1.0 argon2-cffi-bindings 21.2.0 arrow 1.2.3 astunparse 1.6.3 attrs 23.2.0 Babel 2.14.0 backcall 0.2.0 backports.functools-lru-cache 2.0.0 beautifulsoup4 4.12.3 bleach 6.0.0 Bottleneck 1.3.5 cached-property 1.5.2 cachetools 5.3.2 ccimport 0.4.2 certifi 2024.2.2 cffi 1.15.1 charset-normalizer 3.3.2 click 8.1.7 colorama 0.4.6 comm 0.1.4 cumm-cu113 0.4.11 cycler 0.11.0 debugpy 1.6.3 decorator 5.1.1 defusedxml 0.7.1 deprecation 2.1.0 descartes 1.1.0 docker-pycreds 0.4.0 easydict 1.11 entrypoints 0.4 exceptiongroup 1.2.0 faiss-cpu 1.7.4 fastjsonschema 2.19.1 filelock 3.12.2 fire 0.5.0 flatbuffers 1.12 fonttools 4.38.0 fqdn 1.5.1 fvcore 0.1.5.post20210915 gast 0.4.0 gdown 4.4.0 gitdb 4.0.11 GitPython 3.1.41 google-auth 2.27.0 google-auth-oauthlib 0.4.6 google-pasta 0.2.0 grpcio 1.34.1 h5py 3.7.0 idna 3.6 imageio 2.31.2 importlib-metadata 6.7.0 importlib-resources 5.12.0 iopath 0.1.9 ipykernel 6.16.2 ipython 7.33.0 ipython-genutils 0.2.0 ipywidgets 8.1.2 isoduration 20.11.0 jedi 0.19.1 Jinja2 3.1.3 joblib 1.3.2 json5 0.9.14 jsonpointer 2.4 jsonschema 4.17.3 jupyter 1.0.0 jupyter_client 7.4.9 jupyter-console 6.6.3 jupyter_core 4.12.0 jupyter-events 0.5.0 jupyter_packaging 0.12.3 jupyter-server 1.24.0 jupyter_server_fileid 0.9.1 jupyter_server_ydoc 0.8.0 jupyter-ydoc 0.2.5 jupyterlab 3.6.7 jupyterlab-pygments 0.2.2 jupyterlab_server 2.24.0 jupyterlab_widgets 3.0.10 keras-nightly 2.5.0.dev2021032900 Keras-Preprocessing 1.1.2 kiwisolver 1.4.5 lark 1.1.9 llvmlite 0.39.1 Markdown 3.4.4 MarkupSafe 2.1.5 matplotlib 3.5.2 matplotlib-inline 0.1.3 mistune 3.0.2 motmetrics 1.4.0 nbclassic 1.0.0 nbclient 0.7.0 nbconvert 7.6.0 nbformat 5.8.0 nest_asyncio 1.6.0 networkx 2.6.3 ninja 1.11.1.1 notebook 6.5.6 notebook_shim 0.2.3 numba 0.56.4 numexpr 2.7.3 numpy 1.19.5 nuscenes-devkit 1.0.5 oauthlib 3.2.2 open3d 0.15.2 opencv-python 4.6.0.66 opt-einsum 3.3.0 packaging 23.2 pandas 1.3.5 pandocfilters 1.5.1 parso 0.8.3 patsy 0.5.6 pccm 0.4.11 pcdet 0.6.0+255db8f /hdd_10tb/sina/Radial_MAE pexpect 4.9.0 pickleshare 0.7.5 Pillow 9.2.0 pip 24.0 pkgutil_resolve_name 1.3.10 plotly 5.8.1 plotly-express 0.4.1 portalocker 2.6.0 prometheus-client 0.17.1 prompt-toolkit 3.0.42 protobuf 3.20.1 psutil 5.9.3 ptyprocess 0.7.0 pyasn1 0.5.1 pyasn1-modules 0.3.0 pybind11 2.11.1 pycparser 2.21 Pygments 2.17.2 pyparsing 3.1.1 pyquaternion 0.9.9 pyrsistent 0.19.3 PySocks 1.7.1 python-dateutil 2.8.2 python-json-logger 2.0.7 pytorch3d 0.6.2 pytz 2024.1 PyWavelets 1.3.0 PyYAML 6.0 pyzmq 24.0.1 qtconsole 5.4.4 QtPy 2.4.1 requests 2.31.0 requests-oauthlib 1.3.1 rfc3339-validator 0.1.4 rfc3986-validator 0.1.1 rsa 4.9 scikit-image 0.19.3 scikit-learn 1.0.2 scipy 1.5.3 Send2Trash 1.8.2 sentry-sdk 1.40.3 setproctitle 1.3.3 setuptools 62.3.3 shapely 2.0.3 SharedArray 3.1.0 six 1.15.0 smmap 5.0.1 sniffio 1.3.0 soupsieve 2.4.1 spconv-cu113 2.3.6 statsmodels 0.13.2 tabulate 0.9.0 tenacity 8.2.3 tensorboard 2.11.2 tensorboard-data-server 0.6.1 tensorboard-plugin-wit 1.8.1 tensorboardX 2.5.1 tensorflow 2.5.0 tensorflow-estimator 2.5.0 termcolor 1.1.0 terminado 0.17.1 threadpoolctl 3.1.0 tifffile 2021.11.2 timm 0.4.5 tinycss2 1.1.1 tomli 2.0.1 tomlkit 0.11.0 torch 1.10.2 torch-scatter 2.1.1 torchvision 0.11.3 tornado 6.2 tqdm 4.64.0 traitlets 5.9.0 transforms3d 0.3.1 typing_extensions 4.7.1 uri-template 1.3.0 urllib3 2.0.7 wandb 0.16.3 waymo-open-dataset-tf-2-5-0 1.4.1 wcwidth 0.2.10 webcolors 1.13 webencodings 0.5.1 websocket-client 1.6.1 Werkzeug 2.2.3 wheel 0.42.0 widgetsnbextension 4.0.10 wrapt 1.12.1 xmltodict 0.13.0 y-py 0.6.2 yacs 0.1.8 ypy-websocket 0.8.4 zipp 3.15.0
Hi, I have the same problem. Have you solved this problem?
Unfortunately not!
This issue is stale because it has been open for 30 days with no activity.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
It is very annoying that I saw the same problem in "issues" several times and the team keeps on closing the issues due to 30 days stale, so is there any solution?
I have the same problem during training the loss is shown to be 0.2, I wanted to see the test results and everything is 0.
Any help is much appreciated
This issue is stale because it has been open for 30 days with no activity.
This issue was closed because it has been inactive for 14 days since being marked as stale.