train error

[neineimeow]

When I train the code , i have shape error as follow:

[sky@gpu01 CornerNet]$ python train.py CornerNet loading all datasets... using 4 threads loading from cache file: ./cache/coco_trainval2014.pkl No cache file found... loading annotations into memory... Done (t=11.85s) creating index... index created! 118287it [00:40, 2923.89it/s] loading annotations into memory... Done (t=11.86s) creating index... index created! loading from cache file: ./cache/coco_trainval2014.pkl loading annotations into memory... Done (t=10.91s) creating index... index created! loading from cache file: ./cache/coco_trainval2014.pkl loading annotations into memory... Done (t=10.25s) creating index... index created! loading from cache file: ./cache/coco_trainval2014.pkl loading annotations into memory... Done (t=11.62s) creating index... index created! loading from cache file: ./cache/coco_minival2014.pkl No cache file found... loading annotations into memory... Done (t=0.32s) creating index... index created! 5000it [00:01, 2962.00it/s] loading annotations into memory... Done (t=0.31s) creating index... index created! system config... {'batch_size': 49, 'cache_dir': './cache', 'chunk_sizes': [4, 5, 5, 5, 5, 5, 5, 5, 5, 5], 'config_dir': './config', 'data_dir': './data', 'data_rng': <mtrand.RandomState object at 0x7f2085918c60>, 'dataset': 'MSCOCO', 'decay_rate': 10, 'display': 5, 'learning_rate': 0.00025, 'max_iter': 500000, 'nnet_rng': <mtrand.RandomState object at 0x7f2085918ca8>, 'opt_algo': 'adam', 'prefetch_size': 5, 'pretrain': None, 'result_dir': './results', 'sampling_function': 'kp_detection', 'snapshot': 5000, 'snapshot_name': 'CornerNet', 'stepsize': 450000, 'test_split': 'testdev', 'train_split': 'trainval', 'val_iter': 100, 'val_split': 'minival', 'weight_decay': False, 'weight_decay_rate': 1e-05, 'weight_decay_type': 'l2'} db config... {'ae_threshold': 0.5, 'border': 128, 'categories': 80, 'data_aug': True, 'gaussian_bump': True, 'gaussian_iou': 0.7, 'gaussian_radius': -1, 'input_size': [511, 511], 'lighting': True, 'max_per_image': 100, 'merge_bbox': False, 'nms_algorithm': 'exp_soft_nms', 'nms_kernel': 3, 'nms_threshold': 0.5, 'output_sizes': [[128, 128]], 'rand_color': True, 'rand_crop': True, 'rand_pushes': False, 'rand_samples': False, 'rand_scale_max': 1.4, 'rand_scale_min': 0.6, 'rand_scale_step': 0.1, 'rand_scales': array([0.6, 0.7, 0.8, 0.9, 1. , 1.1, 1.2, 1.3]), 'special_crop': False, 'test_scales': [1], 'top_k': 100, 'weight_exp': 8} len of db: 118287 start prefetching data... shuffling indices... start prefetching data... shuffling indices... Traceback (most recent call last): File "train.py", line 42, in prefetch_data data, ind = sample_data(db, ind, data_aug=data_aug) File "/home/sky/CornerNet/sample/coco.py", line 182, in sample_data return globals()[system_configs.sampling_function](db, k_ind, data_aug, debug) File "/home/sky/CornerNet/sample/coco.py", line 96, in kp_detection image, detections = random_crop(image, detections, rand_scales, input_size, border=border) File "/home/sky/CornerNet/sample/utils.py", line 57, in random_crop image_height, image_width = image.shape[0:2] AttributeError: 'NoneType' object has no attribute 'shape' Process Process-1: Traceback (most recent call last): File "/home/sky/anaconda3/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/home/sky/anaconda3/lib/python3.6/multiprocessing/process.py", line 93, in run self._target(*self._args, **self._kwargs) File "train.py", line 46, in prefetch_data raise e File "train.py", line 42, in prefetch_data data, ind = sample_data(db, ind, data_aug=data_aug) File "/home/sky/CornerNet/sample/coco.py", line 182, in sample_data return globals()[system_configs.sampling_function](db, k_ind, data_aug, debug) File "/home/sky/CornerNet/sample/coco.py", line 96, in kp_detection image, detections = random_crop(image, detections, rand_scales, input_size, border=border) File "/home/sky/CornerNet/sample/utils.py", line 57, in random_crop image_height, image_width = image.shape[0:2] AttributeError: 'NoneType' object has no attribute 'shape' start prefetching data... Traceback (most recent call last): File "train.py", line 42, in prefetch_data data, ind = sample_data(db, ind, data_aug=data_aug) File "/home/sky/CornerNet/sample/coco.py", line 182, in sample_data return globals()[system_configs.sampling_function](db, k_ind, data_aug, debug) File "/home/sky/CornerNet/sample/coco.py", line 96, in kp_detection image, detections = random_crop(image, detections, rand_scales, input_size, border=border) File "/home/sky/CornerNet/sample/utils.py", line 57, in random_crop image_height, image_width = image.shape[0:2] AttributeError: 'NoneType' object has no attribute 'shape' Process Process-2: Traceback (most recent call last): File "/home/sky/anaconda3/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/home/sky/anaconda3/lib/python3.6/multiprocessing/process.py", line 93, in run self._target(*self._args, **self._kwargs) File "train.py", line 46, in prefetch_data raise e File "train.py", line 42, in prefetch_data data, ind = sample_data(db, ind, data_aug=data_aug) File "/home/sky/CornerNet/sample/coco.py", line 182, in sample_data return globals()[system_configs.sampling_function](db, k_ind, data_aug, debug) File "/home/sky/CornerNet/sample/coco.py", line 96, in kp_detection image, detections = random_crop(image, detections, rand_scales, input_size, border=border) File "/home/sky/CornerNet/sample/utils.py", line 57, in random_crop image_height, image_width = image.shape[0:2] AttributeError: 'NoneType' object has no attribute 'shape' shuffling indices... start prefetching data... Traceback (most recent call last): File "train.py", line 42, in prefetch_data data, ind = sample_data(db, ind, data_aug=data_aug) File "/home/sky/CornerNet/sample/coco.py", line 182, in sample_data return globals()[system_configs.sampling_function](db, k_ind, data_aug, debug) File "/home/sky/CornerNet/sample/coco.py", line 96, in kp_detection image, detections = random_crop(image, detections, rand_scales, input_size, border=border) File "/home/sky/CornerNet/sample/utils.py", line 57, in random_crop image_height, image_width = image.shape[0:2] AttributeError: 'NoneType' object has no attribute 'shape' Process Process-3: Traceback (most recent call last): File "/home/sky/anaconda3/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/home/sky/anaconda3/lib/python3.6/multiprocessing/process.py", line 93, in run self._target(*self._args, **self._kwargs) File "train.py", line 46, in prefetch_data raise e File "train.py", line 42, in prefetch_data data, ind = sample_data(db, ind, data_aug=data_aug) File "/home/sky/CornerNet/sample/coco.py", line 182, in sample_data return globals()[system_configs.sampling_function](db, k_ind, data_aug, debug) File "/home/sky/CornerNet/sample/coco.py", line 96, in kp_detection image, detections = random_crop(image, detections, rand_scales, input_size, border=border) File "/home/sky/CornerNet/sample/utils.py", line 57, in random_crop image_height, image_width = image.shape[0:2] AttributeError: 'NoneType' object has no attribute 'shape' shuffling indices... Traceback (most recent call last): File "train.py", line 42, in prefetch_data data, ind = sample_data(db, ind, data_aug=data_aug) File "/home/sky/CornerNet/sample/coco.py", line 182, in sample_data return globals()[system_configs.sampling_function](db, k_ind, data_aug, debug) File "/home/sky/CornerNet/sample/coco.py", line 96, in kp_detection image, detections = random_crop(image, detections, rand_scales, input_size, border=border) File "/home/sky/CornerNet/sample/utils.py", line 57, in random_crop image_height, image_width = image.shape[0:2] AttributeError: 'NoneType' object has no attribute 'shape' Process Process-4: Traceback (most recent call last): File "/home/sky/anaconda3/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/home/sky/anaconda3/lib/python3.6/multiprocessing/process.py", line 93, in run self._target(*self._args, **self._kwargs) File "train.py", line 46, in prefetch_data raise e File "train.py", line 42, in prefetch_data data, ind = sample_data(db, ind, data_aug=data_aug) File "/home/sky/CornerNet/sample/coco.py", line 182, in sample_data return globals()[system_configs.sampling_function](db, k_ind, data_aug, debug) File "/home/sky/CornerNet/sample/coco.py", line 96, in kp_detection image, detections = random_crop(image, detections, rand_scales, input_size, border=border) File "/home/sky/CornerNet/sample/utils.py", line 57, in random_crop image_height, image_width = image.shape[0:2] AttributeError: 'NoneType' object has no attribute 'shape' building model... module_file: models.CornerNet start prefetching data... shuffling indices... total parameters: 201035212 setting learning rate to: 0.00025 training start... 0%| | 0/500000 [00:00<?, ?it/s]^CTraceback (most recent call last):

How can i solve it?Thanks a lot for your help !

[heilaw]

#21

[neineimeow]

sorry, when i change the torch Vision to 0.4.0, I have the Import error:

ImportError: /home/sky/.local/lib/python3.6/site-packages/cpools-0.0.0-py3.6-lin ux-x86_64.egg/top_pool.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN2at 6detail20DynamicCUDAInterface10set_deviceE

[heilaw]

Did you recompile the corner pooling layers after you switched to 0.4.0?

[neineimeow]

Yes!When i switched to 0.4.0,I recompiled the corner pooling layers.

[ray-lee-94]

Maybe you lost some images, you can copy the 2014_val images to the minival2014

[GalyaZalesskaya]

I had the same error. It is necessary to force recompile pooling layers after switching to the another torch version. When all files will be recreated. Recompile your pooling layers with python setup.py install --user --force