exit without any wrong information present!

[haishibei]

i set the "epoch = 50", but the program at sixth epoch, the program exit without any wrong information or warning information. The last information in the terminal is " [2020-06-28 16:54:26,232-rk0-train_siammask.py#284] Epoch: [6][16850/37500] lr: 0.005000 batch_time: 1.259789 (0.319867) data_time: 0.659337 (0.132381) rpn_cls_loss: 0.063641 (0.071975) rpn_loc_loss: 0.267109 (nan) rpn_mask_loss: 0.158198 (0.168048) siammask_loss: 6.079290 (nan) mask_iou_mean: 0.609639 (0.526512) mask_iou_at_5: 0.859155 (0.620579) mask_iou_at_7: 0.140845 (0.180679) INFO:global:Epoch: [6][16850/37500] lr: 0.005000 batch_time: 1.259789 (0.319867) data_time: 0.659337 (0.132381) rpn_cls_loss: 0.063641 (0.071975) rpn_loc_loss: 0.267109 (nan) rpn_mask_loss: 0.158198 (0.168048) siammask_loss: 6.079290 (nan) mask_iou_mean: 0.609639 (0.526512) mask_iou_at_5: 0.859155 (0.620579) mask_iou_at_7: 0.140845 (0.180679) [2020-06-28 16:54:28,292-rk0-log_helper.py# 97] Progress: 204350 / 1875000 [10%], Speed: 0.320 s/iter, ETA 6:04:26 (D:H:M)

INFO:global:Progress: 204350 / 1875000 [10%], Speed: 0.320 s/iter, ETA 6:04:26 (D:H:M)

[2020-06-28 16:54:45,369-rk0-train_siammask.py#284] Epoch: [6][16860/37500] lr: 0.005000 batch_time: 1.414664 (0.319940) data_time: 0.691262 (0.132429) rpn_cls_loss: 0.070713 (0.071974) rpn_loc_loss: 0.221511 (nan) rpn_mask_loss: 0.186103 (0.168048) siammask_loss: 7.036219 (nan) mask_iou_mean: 0.627533 (0.526513) mask_iou_at_5: 0.948718 (0.620586) mask_iou_at_7: 0.179487 (0.180682) INFO:global:Epoch: [6][16860/37500] lr: 0.005000 batch_time: 1.414664 (0.319940) data_time: 0.691262 (0.132429) rpn_cls_loss: 0.070713 (0.071974) rpn_loc_loss: 0.221511 (nan) rpn_mask_loss: 0.186103 (0.168048) siammask_loss: 7.036219 (nan) mask_iou_mean: 0.627533 (0.526513) mask_iou_at_5: 0.948718 (0.620586) mask_iou_at_7: 0.179487 (0.180682) [2020-06-28 16:54:45,374-rk0-log_helper.py# 97] Progress: 204360 / 1875000 [10%], Speed: 0.320 s/iter, ETA 6:04:28 (D:H:M)

INFO:global:Progress: 204360 / 1875000 [10%], Speed: 0.320 s/iter, ETA 6:04:28 (D:H:M) " then the program exit. why did it exit without wrong information present? Could anyone answer me ?

[lllllqqqq]

I met the same problem. Have you solved it?

[tyj1996]

@lllllqqqq @haishibei Hi, I met the same problem. can you tell me how you solved it? thanks.

[xiaofengBian]

Hi, I met the same problem. Someone can tell me how to solve it? thanks.