CenterPoint-KITTI icon indicating copy to clipboard operation
CenterPoint-KITTI copied to clipboard

RuntimeError: CUDA error: out of memory

Open BigPig117 opened this issue 2 years ago • 0 comments

hello, this is my error when i try to train the centerpoint.yaml

cfg.OPTIMIZATION = edict() 2022-04-07 11:02:52,498 INFO cfg.OPTIMIZATION.BATCH_SIZE_PER_GPU: 4 2022-04-07 11:02:52,498 INFO cfg.OPTIMIZATION.NUM_EPOCHS: 80 2022-04-07 11:02:52,498 INFO cfg.OPTIMIZATION.OPTIMIZER: adam_onecycle 2022-04-07 11:02:52,498 INFO cfg.OPTIMIZATION.LR: 0.003 2022-04-07 11:02:52,498 INFO cfg.OPTIMIZATION.WEIGHT_DECAY: 0.01 2022-04-07 11:02:52,498 INFO cfg.OPTIMIZATION.MOMENTUM: 0.9 2022-04-07 11:02:52,498 INFO cfg.OPTIMIZATION.MOMS: [0.95, 0.85] 2022-04-07 11:02:52,498 INFO cfg.OPTIMIZATION.PCT_START: 0.4 2022-04-07 11:02:52,498 INFO cfg.OPTIMIZATION.DIV_FACTOR: 10 2022-04-07 11:02:52,498 INFO cfg.OPTIMIZATION.DECAY_STEP_LIST: [35, 45] 2022-04-07 11:02:52,498 INFO cfg.OPTIMIZATION.LR_DECAY: 0.1 2022-04-07 11:02:52,498 INFO cfg.OPTIMIZATION.LR_CLIP: 1e-07 2022-04-07 11:02:52,498 INFO cfg.OPTIMIZATION.LR_WARMUP: False 2022-04-07 11:02:52,498 INFO cfg.OPTIMIZATION.WARMUP_EPOCH: 1 2022-04-07 11:02:52,498 INFO cfg.OPTIMIZATION.GRAD_NORM_CLIP: 10 2022-04-07 11:02:52,498 INFO cfg.TAG: centerpoint 2022-04-07 11:02:52,498 INFO cfg.EXP_GROUP_PATH: kitti_models 2022-04-07 11:02:52,570 INFO Database filter by min points Car: 14357 => 13532 2022-04-07 11:02:52,571 INFO Database filter by min points Pedestrian: 2207 => 2168 2022-04-07 11:02:52,571 INFO Database filter by min points Cyclist: 734 => 705 2022-04-07 11:02:52,582 INFO Database filter by difficulty Car: 13532 => 10759 2022-04-07 11:02:52,584 INFO Database filter by difficulty Pedestrian: 2168 => 2075 2022-04-07 11:02:52,585 INFO Database filter by difficulty Cyclist: 705 => 581 2022-04-07 11:02:52,588 INFO Loading KITTI dataset 2022-04-07 11:02:52,646 INFO Total samples for KITTI dataset: 3712 Traceback (most recent call last): File "train.py", line 202, in main() File "train.py", line 116, in main model = build_network(model_cfg=cfg.MODEL, num_class=len(cfg.CLASS_NAMES), dataset=train_set) File "../pcdet/models/init.py", line 18, in build_network model_cfg=model_cfg, num_class=num_class, dataset=dataset File "../pcdet/models/detectors/init.py", line 30, in build_detector model_cfg=model_cfg, num_class=num_class, dataset=dataset File "../pcdet/models/detectors/centerpoint.py", line 7, in init self.module_list = self.build_networks() File "../pcdet/models/detectors/detector3d_template.py", line 47, in build_networks model_info_dict=model_info_dict File "../pcdet/models/detectors/detector3d_template.py", line 136, in build_dense_head voxel_size=model_info_dict.get('voxel_size', False) File "../pcdet/models/dense_heads/center_head.py", line 66, in init [self.class_names.index(x) for x in cur_class_names if x in class_names] RuntimeError: CUDA error: out of memory

base on GTX3070 cuda11.1 pytorch1.8.2 , can u give me some suggestions

BigPig117 avatar Apr 07 '22 05:04 BigPig117