AerialDetection 训练出错

(A) ff@ff:~/DG/AerialDetection$ bash ./tools/dist_train.sh configs/DOTA/faster_rcnn_h-obb_r50_fpn_1x_dota.py 2 2020-08-07 11:15:06,651 - INFO - Distributed training: True 2020-08-07 11:15:06,978 - INFO - load model from: modelzoo://resnet50 /home/ff/anaconda3/envs/A/lib/python3.7/site-packages/mmcv/runner/checkpoint.py:145: UserWarning: The URL scheme of "modelzoo://" is deprecated, please use "torchvision://" instead warnings.warn('The URL scheme of "modelzoo://" is deprecated, please ' /home/ff/anaconda3/envs/A/lib/python3.7/site-packages/mmcv/runner/checkpoint.py:145: UserWarning: The URL scheme of "modelzoo://" is deprecated, please use "torchvision://" instead warnings.warn('The URL scheme of "modelzoo://" is deprecated, please ' 2020-08-07 11:15:09,074 - WARNING - The model and loaded state dict do not match exactly

unexpected key in source state_dict: fc.weight, fc.bias

loading annotations into memory... loading annotations into memory... Done (t=0.98s) creating index... Done (t=1.00s) creating index... index created! index created! Traceback (most recent call last): File "./tools/train.py", line 95, in main() File "./tools/train.py", line 91, in main logger=logger) File "/home/ff/DG/AerialDetection/mmdet/apis/train.py", line 59, in train_detector _dist_train(model, dataset, cfg, validate=validate) File "/home/ff/DG/AerialDetection/mmdet/apis/train.py", line 144, in _dist_train model = MMDistributedDataParallel(model.cuda()) File "/home/ff/anaconda3/envs/A/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 288, in init self._ddp_init_helper() File "/home/ff/anaconda3/envs/A/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 306, in _ddp_init_helper self._module_copies = replicate(self.module, self.device_ids, detach=True) File "/home/ff/anaconda3/envs/A/lib/python3.7/site-packages/torch/nn/parallel/replicate.py", line 97, in replicate param_copies = _broadcast_coalesced_reshape(params, devices, detach) File "/home/ff/anaconda3/envs/A/lib/python3.7/site-packages/torch/nn/parallel/replicate.py", line 76, in _broadcast_coalesced_reshape return comm.broadcast_coalesced(tensors, devices) File "/home/ff/anaconda3/envs/A/lib/python3.7/site-packages/torch/cuda/comm.py", line 39, in broadcast_coalesced return torch._C._broadcast_coalesced(tensors, devices, buffer_size) RuntimeError: all tensors must be on devices[0] 2020-08-07 11:15:12,288 - INFO - Start running, host: ff@ff, work_dir: /home/ff/DG/AerialDetection/work_dirs/faster_rcnn_h-obb_r50_fpn_1x_dota 2020-08-07 11:15:12,288 - INFO - workflow: [('train', 1)], max: 12 epochs

到这里卡住不动，我在单卡1060上可以运行，在双卡2070的机子上，就会出现此问题