keras-faster-rcnn
keras-faster-rcnn copied to clipboard
yizt
训练过程中,训练loss一开始是有效的,然后变成nan,只训练了1000左右次nan后又变为正常值,很神奇，感觉内部有个bug。
第二个epoch：
@836304831 ,您好，训练日志应该类似如下，会打印rpn、rccn分类和回归每个loss值，这样方便定位具体哪个loss导致整个loss 变为NAN
2133/2505 [========================>.....] - ETA: 4:01 - loss: 0.7034 - rpn_bbox_loss: 0.2536 - rpn_class_loss: 0.0786 - rcnn_bbox_loss: 0.2086 - rcnn_class_loss: 0.1622 - regular_loss: 3.7586e-04 - gt_num: 3.0169 - positive_anchor_num: 23.7975 - negative_anchor_num: 232.2025 - rpn_miss_gt_num: 0.0000e+00 - rpn_gt_min_max_iou: 0.5119 - roi_num: 1877.3690 - positive_roi_num: 25.7164 - rcnn_miss_gt_num: 0.1763 - rcnn_miss_gt_num_as: 0.2745 - gt_min_max_iou: 0.7099 2134/2505 [========================>.....] - ETA: 4:00 - loss: 0.7034 - rpn_bbox_loss: 0.2536 - rpn_class_loss: 0.0786 - rcnn_bbox_loss: 0.2086 - rcnn_class_loss: 0.1622 - regular_loss: 3.7586e-04 - gt_num: 3.0159 - positive_anchor_num: 23.7868 - negative_anchor_num: 232.2132 - rpn_miss_gt_num: 0.0000e+00 - rpn_gt_min_max_iou: 0.5119 - roi_num: 1877.2306 - positive_roi_num: 25.7146 - rcnn_miss_gt_num: 0.1762 - rcnn_miss_gt_num_as: 0.2744 - gt_min_max_iou: 0.7100
因为是使用tf.keras实现的，Model并没有metrics_tensors属性（有一个保护的属性，但是用法跟原生的不一样），所以无法添加rpn_bbox_loss、rpn_class_loss现在正在找可替代的方法。等找到了替换的方法会同步出来。
测试发现，各个loss的大小为rcnn_class_loss > rcnn_bbox_loss >rpn_bbox_loss、rpn_class_loss: 3.8251 > regular_loss，其中rcnn_class_loss是最大的，总训练loss为nan也是因为rcnn_class_loss引起的，如下图：
鉴于loss的分析，可以把个loss的权重调整一下（默认全是1.0），测试ing
@836304831 您好，rcnn_class_loss使用的是tf.nn.softmax_cross_entropy_with_logits_v2；该函数本身是稳定的，出现NAN,应该是logits中有inf或(-inf); 这样情况一般不会出现，特别是有BN层情况下。 还有一个问题，loss怎么怎么这么大？我刚开始训练loss只有7点几，是否有哪些配置参数改动？
@yizt
- 一开始的loss是不大，在8.0左右。但是分别看后面的loss就不一样的，有时候rcnn_bbox_loss的loss为60左右，，rpm_class_loss有的也能达到20左右。很纳闷这个总的loss是怎么算法，感觉跟4个子loss不一致。
- 在配置方面，我把学习率变小了（改为0.0001），将优化器由sgd变成了adam（以为使用sgd也会出现nan的情况）；没有使用自定义的bn，而是使用keras自带的bn（就是没有用您实现的bn）。
@yizt
- 理论上来讲，Adam优化器比SGD更稳定，如果SGD训练过程中出现了nan的情况，改用Adam可能可以避免。
- 我本地使用SGD测试也出现了nan的情况，因为显卡的限制，我把batch_size设置为2。训练loss出现nan的情况较少，测试时可以多关注val_loss，其出现nan的情况相对频繁一点。
@836304831 batch_size过小，还是建议BN 固定住，keras自带的BN,只固定住了需要学习权重参数，不需要学习的均值和方差本身还是没有固定。我测试SGD没有出现NAN哦
@836304831 另外，使用Adam测试出现NAN是正则损失，说明优化器使得权重参数变得非常大
18/2505 [..............................] - ETA: 49:23 - loss: nan - rpn_bbox_loss: 2.9959 - rpn_class_loss: 1.0740 - rcnn_bbox_loss: 0.8404 - rcnn_class_loss: 1.5444 - regular_loss: nan - gt_num: 2.8889 - positive_anchor_num: 18.1111 - negative_anchor_num: 61.8889 - rpn_miss_gt_num: 0.0000e+00 - rpn_gt_min_max_iou: 0.6235 - roi_num: 1996.3056 - positive_roi_num: 22.7778 - negative_roi_num: 99.8889 - rcnn_miss_gt_num: 1.2222 - rcnn_miss_gt_num_as: 1.2778 - gt_min_max_iou: 0.5495
@yizt
- 我这边测试正则损失很小，不会出现nan的问题。
- batch_size过小这是个硬件条件的限制，应该如何将BN固定住呢。
- 使用sgd训练，有时候确实不会出现nan的问题，是偶然性的。
424/3425 [============================>.] - ETA: 0s - loss: nan - rpn_bbox_loss: 0.4054 - rpn_class_loss: 0.1190 - rcnn_bbox_loss: 0.2831 - rcnn_class_loss: nan - regular_loss: 13.5767 - gt_num: 2.2009 - positive_anchor_num: 27.8823 - negative_anchor_num: 52.1177 - rpn_miss_gt_num: 0.0558 - rpn_gt_min_max_iou: 0.6257 - roi_num: 1954.6650 - positive_roi_num: 12.7421 - negative_roi_num: 99.2915 - rcnn_miss_gt_num: 0.4933 - rcnn_miss_gt_num_as: 0.4965 - gt_min_max_iou: 0.7118 Epoch 00005: saving model to /home/multiai4/Ruihan/face_mask_detection/chick_point/resnet50.005.h5 3425/3425 [==============================] - 860s 251ms/step - loss: nan - rpn_bbox_loss: 0.4055 - rpn_class_loss: 0.1190 - rcnn_bbox_loss: 0.2832 - rcnn_class_loss: nan - regular_loss: 13.5767 - gt_num: 2.2006 - positive_anchor_num: 27.8952 - negative_anchor_num: 52.1048 - rpn_miss_gt_num: 0.0558 - rpn_gt_min_max_iou: 0.6257 - roi_num: 1954.6782 - positive_roi_num: 12.7425 - negative_roi_num: 99.2958 - rcnn_miss_gt_num: 0.4931 - rcnn_miss_gt_num_as: 0.4964 - gt_min_max_iou: 0.7118 - val_loss: 3.0300 - val_rpn_bbox_loss: 0.3178 - val_rpn_class_loss: 0.0826 - val_rcnn_bbox_loss: 0.3290 - val_rcnn_class_loss: 0.2525 - val_regular_loss: 13.5779 - val_gt_num: 2.2560 - val_positive_anchor_num: 28.9400 - val_negative_anchor_num: 51.0600 - val_rpn_miss_gt_num: 0.0000e+00 - val_rpn_gt_min_max_iou: 0.6211 - val_roi_num: 1967.0040 - val_positive_roi_num: 12.8520 - val_negative_roi_num: 99.1840 - val_rcnn_miss_gt_num: 0.4360 - val_rcnn_miss_gt_num_as: 0.4360 - val_gt_min_max_iou: 0.7233 Epoch 6/50 3424/3425 [============================>.] - ETA: 0s - loss: 2.9800 - rpn_bbox_loss: 0.3825 - rpn_class_loss: 0.1046 - rcnn_bbox_loss: 0.2658 - rcnn_class_loss: 0.1819 - regular_loss: 13.5798 - gt_num: 2.3233 - positive_anchor_num: 27.8420 - negative_anchor_num: 52.1580 - rpn_miss_gt_num: 0.0648 - rpn_gt_min_max_iou: 0.6247 - roi_num: 1954.1166 - positive_roi_num: 13.0400 - negative_roi_num: 98.5610 - rcnn_miss_gt_num: 0.5222 - rcnn_miss_gt_num_as: 0.5388 - gt_min_max_iou: 0.7156 Epoch 00006: saving model to /home/multiai4/Ruihan/face_mask_detection/chick_point/resnet50.006.h5 3425/3425 [==============================] - 860s 251ms/step - loss: 2.9798 - rpn_bbox_loss: 0.3825 - rpn_class_loss: 0.1046 - rcnn_bbox_loss: 0.2658 - rcnn_class_loss: 0.1819 - regular_loss: 13.5798 - gt_num: 2.3229 - positive_anchor_num: 27.8409 - negative_anchor_num: 52.1591 - rpn_miss_gt_num: 0.0648 - rpn_gt_min_max_iou: 0.6247 - roi_num: 1954.0271 - positive_roi_num: 13.0394 - negative_roi_num: 98.5664 - rcnn_miss_gt_num: 0.5220 - rcnn_miss_gt_num_as: 0.5387 - gt_min_max_iou: 0.7156 - val_loss: 2.9243 - val_rpn_bbox_loss: 0.2911 - val_rpn_class_loss: 0.0716 - val_rcnn_bbox_loss: 0.2965 - val_rcnn_class_loss: 0.2231 - val_regular_loss: 13.5820 - val_gt_num: 2.0840 - val_positive_anchor_num: 28.4080 - val_negative_anchor_num: 51.5920 - val_rpn_miss_gt_num: 0.0000e+00 - val_rpn_gt_min_max_iou: 0.6296 - val_roi_num: 1952.5400 - val_positive_roi_num: 12.9160 - val_negative_roi_num: 97.6160 - val_rcnn_miss_gt_num: 0.1960 - val_rcnn_miss_gt_num_as: 0.2160 - val_gt_min_max_iou: 0.7347 Epoch 7/50 3424/3425 [============================>.] - ETA: 0s - loss: nan - rpn_bbox_loss: 0.3598 - rpn_class_loss: 0.0952 - rcnn_bbox_loss: 0.2400 - rcnn_class_loss: nan - regular_loss: 13.5827 - gt_num: 2.1857 - positive_anchor_num: 28.1527 - negative_anchor_num: 51.8473 - rpn_miss_gt_num: 0.0181 - rpn_gt_min_max_iou: 0.6271 - roi_num: 1951.0493 - positive_roi_num: 13.2036 - negative_roi_num: 98.7141 - rcnn_miss_gt_num: 0.3992 - rcnn_miss_gt_num_as: 0.4062 - gt_min_max_iou: 0.7239 Epoch 00007: saving model to /home/multiai4/Ruihan/face_mask_detection/chick_point/resnet50.007.h5 3425/3425 [==============================] - 861s 251ms/step - loss: nan - rpn_bbox_loss: 0.3599 - rpn_class_loss: 0.0953 - rcnn_bbox_loss: 0.2401 - rcnn_class_loss: nan - regular_loss: 13.5827 - gt_num: 2.1880 - positive_anchor_num: 28.1521 - negative_anchor_num: 51.8479 - rpn_miss_gt_num: 0.0181 - rpn_gt_min_max_iou: 0.6271 - roi_num: 1951.0636 - positive_roi_num: 13.2050 - negative_roi_num: 98.7174 - rcnn_miss_gt_num: 0.4003 - rcnn_miss_gt_num_as: 0.4073 - gt_min_max_iou: 0.7237 - val_loss: 2.9758 - val_rpn_bbox_loss: 0.3278 - val_rpn_class_loss: 0.0852 - val_rcnn_bbox_loss: 0.3027 - val_rcnn_class_loss: 0.2240 - val_regular_loss: 13.5830 - val_gt_num: 2.6960 - val_positive_anchor_num: 30.6280 - val_negative_anchor_num: 49.3720 - val_rpn_miss_gt_num: 0.0000e+00 - val_rpn_gt_min_max_iou: 0.6076 - val_roi_num: 1945.3440 - val_positive_roi_num: 12.9000 - val_negative_roi_num: 98.0480 - val_rcnn_miss_gt_num: 0.5680 - val_rcnn_miss_gt_num_as: 0.6360 - val_gt_min_max_iou: 0.7228 Epoch 8/50 3424/3425 [============================>.] - ETA: 0s - loss: 2.8500 - rpn_bbox_loss: 0.3420 - rpn_class_loss: 0.0874 - rcnn_bbox_loss: 0.2274 - rcnn_class_loss: 0.1603 - regular_loss: 13.5824 - gt_num: 2.3090 - positive_anchor_num: 27.7447 - negative_anchor_num: 52.2553 - rpn_miss_gt_num: 0.0555 - rpn_gt_min_max_iou: 0.6277 - roi_num: 1955.5555 - positive_roi_num: 13.2921 - negative_roi_num: 97.7039 - rcnn_miss_gt_num: 0.4746 - rcnn_miss_gt_num_as: 0.4907 - gt_min_max_iou: 0.7267 Epoch 00008: saving model to /home/multiai4/Ruihan/face_mask_detection/chick_point/resnet50.008.h5 3425/3425 [==============================] - 861s 251ms/step - loss: 2.8501 - rpn_bbox_loss: 0.3421 - rpn_class_loss: 0.0874 - rcnn_bbox_loss: 0.2274 - rcnn_class_loss: 0.1603 - regular_loss: 13.5824 - gt_num: 2.3089 - positive_anchor_num: 27.7466 - negative_anchor_num: 52.2534 - rpn_miss_gt_num: 0.0555 - rpn_gt_min_max_iou: 0.6277 - roi_num: 1955.5685 - positive_roi_num: 13.2928 - negative_roi_num: 97.7080 - rcnn_miss_gt_num: 0.4745 - rcnn_miss_gt_num_as: 0.4905 - gt_min_max_iou: 0.7267 - val_loss: 2.8839 - val_rpn_bbox_loss: 0.2739 - val_rpn_class_loss: 0.0689 - val_rcnn_bbox_loss: 0.3056 - val_rcnn_class_loss: 0.2056 - val_regular_loss: 13.5816 - val_gt_num: 2.4520 - val_positive_anchor_num: 31.1880 - val_negative_anchor_num: 48.8120 - val_rpn_miss_gt_num: 0.0000e+00 - val_rpn_gt_min_max_iou: 0.6297 - val_roi_num: 1942.1040 - val_positive_roi_num: 13.2160 - val_negative_roi_num: 96.9640 - val_rcnn_miss_gt_num: 0.4440 - val_rcnn_miss_gt_num_as: 0.5000 - val_gt_min_max_iou: 0.7391 Epoch 9/50 3424/3425 [============================>.] - ETA: 0s - loss: 2.8159 - rpn_bbox_loss: 0.3334 - rpn_class_loss: 0.0832 - rcnn_bbox_loss: 0.2193 - rcnn_class_loss: 0.1531 - regular_loss: 13.5817 - gt_num: 2.3668 - positive_anchor_num: 27.9690 - negative_anchor_num: 52.0310 - rpn_miss_gt_num: 0.0637 - rpn_gt_min_max_iou: 0.6243 - roi_num: 1952.9930 - positive_roi_num: 13.4521 - negative_roi_num: 96.8981 - rcnn_miss_gt_num: 0.5038 - rcnn_miss_gt_num_as: 0.5204 - gt_min_max_iou: 0.7243 Epoch 00009: saving model to /home/multiai4/Ruihan/face_mask_detection/chick_point/resnet50.009.h5 3425/3425 [==============================] - 862s 252ms/step - loss: 2.8160 - rpn_bbox_loss: 0.3334 - rpn_class_loss: 0.0832 - rcnn_bbox_loss: 0.2194 - rcnn_class_loss: 0.1532 - regular_loss: 13.5817 - gt_num: 2.3664 - positive_anchor_num: 27.9623 - negative_anchor_num: 52.0377 - rpn_miss_gt_num: 0.0636 - rpn_gt_min_max_iou: 0.6243 - roi_num: 1953.0067 - positive_roi_num: 13.4508 - negative_roi_num: 96.8990 - rcnn_miss_gt_num: 0.5036 - rcnn_miss_gt_num_as: 0.5203 - gt_min_max_iou: 0.7244 - val_loss: 2.8223 - val_rpn_bbox_loss: 0.2689 - val_rpn_class_loss: 0.0610 - val_rcnn_bbox_loss: 0.2746 - val_rcnn_class_loss: 0.1941 - val_regular_loss: 13.5790 - val_gt_num: 2.1520 - val_positive_anchor_num: 27.9640 - val_negative_anchor_num: 52.0360 - val_rpn_miss_gt_num: 0.0000e+00 - val_rpn_gt_min_max_iou: 0.6301 - val_roi_num: 1952.4919 - val_positive_roi_num: 12.7440 - val_negative_roi_num: 96.0400 - val_rcnn_miss_gt_num: 0.3360 - val_rcnn_miss_gt_num_as: 0.3360 - val_gt_min_max_iou: 0.7418 Epoch 10/50 3424/3425 [============================>.] - ETA: 0s - loss: 2.7805 - rpn_bbox_loss: 0.3228 - rpn_class_loss: 0.0781 - rcnn_bbox_loss: 0.2072 - rcnn_class_loss: 0.1517 - regular_loss: 13.5780 - gt_num: 2.1834 - positive_anchor_num: 27.6487 - negative_anchor_num: 52.3513 - rpn_miss_gt_num: 0.0105 - rpn_gt_min_max_iou: 0.6271 - roi_num: 1948.9939 - positive_roi_num: 13.3113 - negative_roi_num: 96.0362 - rcnn_miss_gt_num: 0.3768 - rcnn_miss_gt_num_as: 0.3969 - gt_min_max_iou: 0.7318 Epoch 00010: saving model to /home/multiai4/Ruihan/face_mask_detection/chick_point/resnet50.010.h5 3425/3425 [==============================] - 862s 252ms/step - loss: 2.7804 - rpn_bbox_loss: 0.3228 - rpn_class_loss: 0.0781 - rcnn_bbox_loss: 0.2072 - rcnn_class_loss: 0.1517 - regular_loss: 13.5780 - gt_num: 2.1839 - positive_anchor_num: 27.6432 - negative_anchor_num: 52.3568 - rpn_miss_gt_num: 0.0105 - rpn_gt_min_max_iou: 0.6270 - roi_num: 1949.0088 - positive_roi_num: 13.3136 - negative_roi_num: 96.0394 - rcnn_miss_gt_num: 0.3766 - rcnn_miss_gt_num_as: 0.3968 - gt_min_max_iou: 0.7318 - val_loss: 2.7738 - val_rpn_bbox_loss: 0.2493 - val_rpn_class_loss: 0.0530 - val_rcnn_bbox_loss: 0.2747 - val_rcnn_class_loss: 0.1793 - val_regular_loss: 13.5764 - val_gt_num: 2.2200 - val_positive_anchor_num: 28.4960 - val_negative_anchor_num: 51.5040 - val_rpn_miss_gt_num: 0.0000e+00 - val_rpn_gt_min_max_iou: 0.6260 - val_roi_num: 1949.1080 - val_positive_roi_num: 12.9320 - val_negative_roi_num: 95.0160 - val_rcnn_miss_gt_num: 0.1960 - val_rcnn_miss_gt_num_as: 0.2720 - val_gt_min_max_iou: 0.7448 Epoch 11/50 3424/3425 [============================>.] - ETA: 0s - loss: 2.7581 - rpn_bbox_loss: 0.3149 - rpn_class_loss: 0.0747 - rcnn_bbox_loss: 0.2026 - rcnn_class_loss: 0.1515 - regular_loss: 13.5745 - gt_num: 2.3496 - positive_anchor_num: 27.7818 - negative_anchor_num: 52.2182 - rpn_miss_gt_num: 0.0561 - rpn_gt_min_max_iou: 0.6266 - roi_num: 1950.1661 - positive_roi_num: 13.5242 - negative_roi_num: 96.7024 - rcnn_miss_gt_num: 0.4761 - rcnn_miss_gt_num_as: 0.4974 - gt_min_max_iou: 0.7329 Epoch 00011: saving model to /home/multiai4/Ruihan/face_mask_detection/chick_point/resnet50.011.h5 3425/3425 [==============================] - 861s 251ms/step - loss: 2.7580 - rpn_bbox_loss: 0.3148 - rpn_class_loss: 0.0747 - rcnn_bbox_loss: 0.2026 - rcnn_class_loss: 0.1515 - regular_loss: 13.5745 - gt_num: 2.3492 - positive_anchor_num: 27.7874 - negative_anchor_num: 52.2126 - rpn_miss_gt_num: 0.0561 - rpn_gt_min_max_iou: 0.6266 - roi_num: 1950.1808 - positive_roi_num: 13.5241 - negative_roi_num: 96.7077 - rcnn_miss_gt_num: 0.4759 - rcnn_miss_gt_num_as: 0.4972 - gt_min_max_iou: 0.7329 - val_loss: nan - val_rpn_bbox_loss: 0.2578 - val_rpn_class_loss: 0.0467 - val_rcnn_bbox_loss: 0.2583 - val_rcnn_class_loss: nan - val_regular_loss: 13.5725 - val_gt_num: 2.0480 - val_positive_anchor_num: 27.7600 - val_negative_anchor_num: 52.2400 - val_rpn_miss_gt_num: 0.0000e+00 - val_rpn_gt_min_max_iou: 0.6217 - val_roi_num: 1945.6479 - val_positive_roi_num: 13.1320 - val_negative_roi_num: 94.1560 - val_rcnn_miss_gt_num: 0.1640 - val_rcnn_miss_gt_num_as: 0.1920 - val_gt_min_max_iou: 0.7389 Epoch 12/50 3424/3425 [============================>.] - ETA: 0s - loss: nan - rpn_bbox_loss: 0.3108 - rpn_class_loss: 0.0690 - rcnn_bbox_loss: 0.1924 - rcnn_class_loss: nan - regular_loss: 13.5685 - gt_num: 2.3291 - positive_anchor_num: 28.4924 - negative_anchor_num: 51.5076 - rpn_miss_gt_num: 0.0567 - rpn_gt_min_max_iou: 0.6251 - roi_num: 1948.5964 - positive_roi_num: 13.6469 - negative_roi_num: 96.0730 - rcnn_miss_gt_num: 0.4527 - rcnn_miss_gt_num_as: 0.4825 - gt_min_max_iou: 0.7356 Epoch 00012: saving model to /home/multiai4/Ruihan/face_mask_detection/chick_point/resnet50.012.h5 3425/3425 [==============================] - 861s 251ms/step - loss: nan - rpn_bbox_loss: 0.3108 - rpn_class_loss: 0.0690 - rcnn_bbox_loss: 0.1925 - rcnn_class_loss: nan - regular_loss: 13.5685 - gt_num: 2.3288 - positive_anchor_num: 28.4861 - negative_anchor_num: 51.5139 - rpn_miss_gt_num: 0.0566 - rpn_gt_min_max_iou: 0.6251 - roi_num: 1948.3971 - positive_roi_num: 13.6453 - negative_roi_num: 96.0753 - rcnn_miss_gt_num: 0.4526 - rcnn_miss_gt_num_as: 0.4823 - gt_min_max_iou: 0.7356 - val_loss: 2.7528 - val_rpn_bbox_loss: 0.2614 - val_rpn_class_loss: 0.0524 - val_rcnn_bbox_loss: 0.2542 - val_rcnn_class_loss: 0.1797 - val_regular_loss: 13.5635 - val_gt_num: 2.0680 - val_positive_anchor_num: 29.1960 - val_negative_anchor_num: 50.8040 - val_rpn_miss_gt_num: 0.0000e+00 - val_rpn_gt_min_max_iou: 0.6288 - val_roi_num: 1937.9041 - val_positive_roi_num: 12.8480 - val_negative_roi_num: 94.9080 - val_rcnn_miss_gt_num: 0.3320 - val_rcnn_miss_gt_num_as: 0.3360 - val_gt_min_max_iou: 0.7470 Epoch 13/50