RuntimeError: CUDA error: device-side assert triggered

[hexiangdong2017]

deeprob000@deeprob000:~/Workspace/Yolo_Nano$ bash train.sh Unable to init server: Could not connect: Connection refused Unable to init server: 无法连接： Connection refused

(train.py:10610): Gdk-CRITICAL **: 14:13:21.564: gdk_cursor_new_for_display: assertion 'GDK_IS_DISPLAY (display)' failed /home/deeprob000/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) /home/deeprob000/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) /home/deeprob000/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) /home/deeprob000/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) /home/deeprob000/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) /home/deeprob000/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) /home/deeprob000/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) /home/deeprob000/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) /home/deeprob000/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) /home/deeprob000/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) /home/deeprob000/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) /home/deeprob000/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) Namespace(augment=False, batch_size=8, checkpoint_interval=1, compute_map=False, data_config='config/coco_person.data', epochs=120, evaluation_interval=1, gradient_accumulations=1, img_size=416, lr=0.00025, lr_policy='cosine', max_epoch=120, mix_up=True, model_def='./config/yolo-nano_person.cfg', multiscale_training=True, n_cpu=8, none_mix_epoch=100, pretrained_weights=None, visual=True, warm_up_epoch=1, warm_up_lr=0.0) loading annotations into memory... Done (t=6.95s) creating index... index created! Traceback (most recent call last): File "train.py", line 138, in loss, outputs = model(imgs,targets=targets,img_scores=img_scores,gt_mix_index = gt_mix_index) File "/home/deeprob000/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in call result = self.forward(*input, **kwargs) File "/home/deeprob000/Workspace/Yolo_Nano/models/yolo_nano_helper.py", line 239, in forward layer_0_x, layer_loss = self.yolo0(x_scale_1,targets,img_dim,img_scores = img_scores,gt_mix_index=gt_mix_index) File "/home/deeprob000/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in call result = self.forward(*input, **kwargs) File "/home/deeprob000/Workspace/Yolo_Nano/models/darknet.py", line 200, in forward gt_mix_index = gt_mix_index File "/home/deeprob000/Workspace/Yolo_Nano/utils/utils.py", line 374, in build_targets iou_scores[b, best_n, gj, gi] = bbox_iou(pred_boxes[b, best_n, gj, gi], target_boxes, x1y1x2y2=False) File "/home/deeprob000/Workspace/Yolo_Nano/utils/utils.py", line 215, in bbox_iou inter_rect_y2 - inter_rect_y1 + 1, min=0 RuntimeError: CUDA error: device-side assert triggered