the loss is too high

[781458112]

When I used the coin data set provided by the author, training Loss was large and it took about 700 epochs to converge. When I adopted my own data set, loss was decreasing but the value was still extremely large, and the mean AP test was not good either. Was this due to the problem of code? Again, due to my setup problem, I posted my training log 2020-12-21 18:53:43,643 train.py[line:613] INFO: Using device cuda 2020-12-21 18:53:46,409 train.py[line:327] INFO: Starting training: Epochs: 300 Batch size: 256 Subdivisions: 16 Learning rate: 0.001 Training size: 2550 Validation size: 284 Checkpoints: True Device: cuda Images size: 608 Optimizer: adam Dataset classes: 7 Train label path:train.txt Pretrained:

2020-12-21 19:00:10,276 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 19:00:10,746 train.py[line:453] INFO: Checkpoint 1 saved ! 2020-12-21 19:06:25,537 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 19:06:25,999 train.py[line:453] INFO: Checkpoint 2 saved ! 2020-12-21 19:06:32,426 train.py[line:412] DEBUG: Train step_320: loss : 252729.109375,loss xy : 498.7034912109375,loss wh : 105.59320068359375,loss obj : 251240.78125，loss cls : 884.0379638671875,loss l2 : 90165.8828125,lr : 1.6000000000000002e-10 2020-12-21 19:12:45,360 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 19:12:45,827 train.py[line:453] INFO: Checkpoint 3 saved ! 2020-12-21 19:19:01,821 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 19:19:02,260 train.py[line:453] INFO: Checkpoint 4 saved ! 2020-12-21 19:19:12,259 train.py[line:412] DEBUG: Train step_640: loss : 252508.515625,loss xy : 534.0221557617188,loss wh : 83.49874877929688,loss obj : 250934.515625，loss cls : 956.4920043945312,loss l2 : 90021.0078125,lr : 2.5600000000000003e-09 2020-12-21 19:25:30,167 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 19:25:30,649 train.py[line:453] INFO: Checkpoint 5 saved ! 2020-12-21 19:31:52,792 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 19:31:53,218 train.py[line:453] INFO: Checkpoint 6 saved ! 2020-12-21 19:32:06,866 train.py[line:412] DEBUG: Train step_960: loss : 252897.203125,loss xy : 658.8988037109375,loss wh : 111.93785095214844,loss obj : 250961.25，loss cls : 1165.1033935546875,loss l2 : 90156.15625,lr : 1.2959999999999998e-08 2020-12-21 19:38:26,652 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 19:38:27,196 train.py[line:453] INFO: Checkpoint 7 saved ! 2020-12-21 19:45:01,857 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 19:45:02,358 train.py[line:453] INFO: Checkpoint 8 saved ! 2020-12-21 19:45:19,420 train.py[line:412] DEBUG: Train step_1280: loss : 252770.0625,loss xy : 648.0030517578125,loss wh : 172.1038818359375,loss obj : 250801.296875，loss cls : 1148.6546630859375,loss l2 : 90154.96875,lr : 4.0960000000000004e-08 2020-12-21 19:51:24,508 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 19:51:24,966 train.py[line:453] INFO: Checkpoint 9 saved ! 2020-12-21 19:57:49,967 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 19:57:50,436 train.py[line:453] INFO: Checkpoint 10 saved ! 2020-12-21 19:58:10,980 train.py[line:412] DEBUG: Train step_1600: loss : 252799.34375,loss xy : 637.1931762695312,loss wh : 95.4561538696289,loss obj : 250943.5625，loss cls : 1123.1195068359375,loss l2 : 90100.5859375,lr : 1.0000000000000002e-07 2020-12-21 20:04:09,951 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 20:04:10,434 train.py[line:453] INFO: Checkpoint 11 saved ! 2020-12-21 20:10:34,079 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 20:10:34,512 train.py[line:453] INFO: Checkpoint 12 saved ! 2020-12-21 20:10:59,693 train.py[line:412] DEBUG: Train step_1920: loss : 252588.625,loss xy : 577.5651245117188,loss wh : 117.3830337524414,loss obj : 250857.09375，loss cls : 1036.5802001953125,loss l2 : 90063.796875,lr : 2.0735999999999997e-07 2020-12-21 20:17:02,569 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 20:17:03,054 train.py[line:453] INFO: Checkpoint 13 saved ! 2020-12-21 20:23:28,850 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 20:23:29,347 train.py[line:453] INFO: Checkpoint 14 saved ! 2020-12-21 20:23:58,951 train.py[line:412] DEBUG: Train step_2240: loss : 252862.359375,loss xy : 606.147705078125,loss wh : 110.18650817871094,loss obj : 251060.421875，loss cls : 1085.6075439453125,loss l2 : 90183.8671875,lr : 3.841600000000001e-07 2020-12-21 20:29:55,495 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 20:29:56,004 train.py[line:453] INFO: Checkpoint 15 saved ! 2020-12-21 20:36:25,509 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 20:36:26,046 train.py[line:453] INFO: Checkpoint 16 saved ! 2020-12-21 20:36:59,839 train.py[line:412] DEBUG: Train step_2560: loss : 252532.125,loss xy : 611.5769653320312,loss wh : 96.57943725585938,loss obj : 250745.625，loss cls : 1078.3394775390625,loss l2 : 89977.6875,lr : 6.553600000000001e-07 2020-12-21 20:42:53,465 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 20:42:53,926 train.py[line:453] INFO: Checkpoint 17 saved ! 2020-12-21 20:49:21,731 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 20:49:22,223 train.py[line:453] INFO: Checkpoint 18 saved ! 2020-12-21 20:49:58,111 train.py[line:412] DEBUG: Train step_2880: loss : 252587.328125,loss xy : 498.42999267578125,loss wh : 157.50436401367188,loss obj : 251031.359375，loss cls : 900.0358276367188,loss l2 : 90175.875,lr : 1.0497599999999998e-06 2020-12-21 20:55:46,772 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 20:55:47,240 train.py[line:453] INFO: Checkpoint 19 saved ! 2020-12-21 21:02:14,535 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 21:02:15,010 train.py[line:453] INFO: Checkpoint 20 saved ! 2020-12-21 21:02:54,770 train.py[line:412] DEBUG: Train step_3200: loss : 252588.578125,loss xy : 530.1231079101562,loss wh : 101.58930969238281,loss obj : 251008.015625，loss cls : 948.8486328125,loss l2 : 90078.7265625,lr : 1.6000000000000004e-06 2020-12-21 21:08:40,006 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 21:08:40,517 train.py[line:453] INFO: Checkpoint 21 saved ! 2020-12-21 21:15:20,853 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 21:15:21,411 train.py[line:453] INFO: Checkpoint 22 saved ! 2020-12-21 21:16:03,562 train.py[line:412] DEBUG: Train step_3520: loss : 252873.296875,loss xy : 728.3275146484375,loss wh : 127.37005615234375,loss obj : 250720.984375，loss cls : 1296.6158447265625,loss l2 : 90134.3359375,lr : 2.34256e-06 2020-12-21 21:21:57,488 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 21:21:58,097 train.py[line:453] INFO: Checkpoint 23 saved ! 2020-12-21 21:28:25,944 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 21:28:26,450 train.py[line:453] INFO: Checkpoint 24 saved ! 2020-12-21 21:29:12,483 train.py[line:412] DEBUG: Train step_3840: loss : 252581.578125,loss xy : 590.881591796875,loss wh : 94.3615493774414,loss obj : 250832.890625，loss cls : 1063.43017578125,loss l2 : 89993.4375,lr : 3.3177599999999996e-06 2020-12-21 21:34:51,552 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 21:34:52,014 train.py[line:453] INFO: Checkpoint 25 saved ! 2020-12-21 21:41:20,212 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 21:41:20,686 train.py[line:453] INFO: Checkpoint 26 saved ! 2020-12-21 21:42:10,247 train.py[line:412] DEBUG: Train step_4160: loss : 252721.0625,loss xy : 638.7736206054688,loss wh : 97.53147888183594,loss obj : 250821.90625，loss cls : 1162.84521484375,loss l2 : 90047.0078125,lr : 4.569760000000001e-06 2020-12-21 21:47:50,952 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 21:47:51,426 train.py[line:453] INFO: Checkpoint 27 saved ! 2020-12-21 21:54:28,413 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 21:54:28,902 train.py[line:453] INFO: Checkpoint 28 saved ! 2020-12-21 21:55:21,808 train.py[line:412] DEBUG: Train step_4480: loss : 252679.453125,loss xy : 488.63671875,loss wh : 77.91655731201172,loss obj : 251241.5625，loss cls : 871.3480834960938,loss l2 : 90109.3203125,lr : 6.146560000000002e-06 2020-12-21 22:00:57,848 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 22:00:58,428 train.py[line:453] INFO: Checkpoint 29 saved ! 2020-12-21 22:07:23,389 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 22:07:23,924 train.py[line:453] INFO: Checkpoint 30 saved ! 2020-12-21 22:08:20,654 train.py[line:412] DEBUG: Train step_4800: loss : 252897.984375,loss xy : 719.8353271484375,loss wh : 114.90853118896484,loss obj : 250790.90625，loss cls : 1272.3218994140625,loss l2 : 90146.1796875,lr : 8.1e-06 2020-12-21 22:13:56,623 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 22:13:57,131 train.py[line:453] INFO: Checkpoint 31 saved ! 2020-12-21 22:20:27,559 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 22:20:28,067 train.py[line:453] INFO: Checkpoint 32 saved ! 2020-12-21 22:21:29,094 train.py[line:412] DEBUG: Train step_5120: loss : 252398.703125,loss xy : 565.4014282226562,loss wh : 87.87928009033203,loss obj : 250745.3125，loss cls : 1000.1114501953125,loss l2 : 89990.7265625,lr : 1.0485760000000001e-05 2020-12-21 22:27:09,724 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 22:27:10,212 train.py[line:453] INFO: Checkpoint 33 saved ! 2020-12-21 22:33:39,353 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 22:33:39,811 train.py[line:453] INFO: Checkpoint 34 saved ! 2020-12-21 22:34:44,873 train.py[line:412] DEBUG: Train step_5440: loss : 252261.0625,loss xy : 539.5140991210938,loss wh : 83.43766021728516,loss obj : 250679.5，loss cls : 958.6079711914062,loss l2 : 89943.4296875,lr : 1.3363360000000005e-05 2020-12-21 22:40:19,051 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 22:40:19,520 train.py[line:453] INFO: Checkpoint 35 saved ! 2020-12-21 22:46:47,352 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 22:46:47,894 train.py[line:453] INFO: Checkpoint 36 saved ! 2020-12-21 22:47:55,185 train.py[line:412] DEBUG: Train step_5760: loss : 252137.640625,loss xy : 582.5276489257812,loss wh : 94.50183868408203,loss obj : 250413.140625，loss cls : 1047.4752197265625,loss l2 : 89819.9140625,lr : 1.6796159999999997e-05 2020-12-21 22:53:11,437 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 22:53:11,919 train.py[line:453] INFO: Checkpoint 37 saved ! 2020-12-21 22:59:38,531 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 22:59:39,093 train.py[line:453] INFO: Checkpoint 38 saved ! 2020-12-21 23:00:50,268 train.py[line:412] DEBUG: Train step_6080: loss : 252045.140625,loss xy : 461.7595520019531,loss wh : 75.82793426513672,loss obj : 250686.296875，loss cls : 821.248046875,loss l2 : 89881.609375,lr : 2.085136e-05 2020-12-21 23:06:05,139 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 23:06:05,651 train.py[line:453] INFO: Checkpoint 39 saved ! 2020-12-21 23:12:45,868 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 23:12:46,336 train.py[line:453] INFO: Checkpoint 40 saved ! 2020-12-21 23:14:03,010 train.py[line:412] DEBUG: Train step_6400: loss : 251965.890625,loss xy : 529.942138671875,loss wh : 104.5448226928711,loss obj : 250374.78125，loss cls : 956.6289672851562,loss l2 : 89816.515625,lr : 2.5600000000000006e-05 2020-12-21 23:19:16,928 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 23:19:17,429 train.py[line:453] INFO: Checkpoint 41 saved ! 2020-12-21 23:25:46,093 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 23:25:46,632 train.py[line:453] INFO: Checkpoint 42 saved ! 2020-12-21 23:27:05,941 train.py[line:412] DEBUG: Train step_6720: loss : 251346.5625,loss xy : 554.101806640625,loss wh : 81.06137084960938,loss obj : 249727.984375，loss cls : 983.4208984375,loss l2 : 89502.3984375,lr : 3.111696e-05 2020-12-21 23:32:23,682 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 23:32:24,188 train.py[line:453] INFO: Checkpoint 43 saved ! 2020-12-21 23:39:02,913 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 23:39:03,461 train.py[line:453] INFO: Checkpoint 44 saved ! 2020-12-21 23:40:25,903 train.py[line:412] DEBUG: Train step_7040: loss : 250574.203125,loss xy : 398.0867614746094,loss wh : 63.71342086791992,loss obj : 249392.65625，loss cls : 719.7471923828125,loss l2 : 89232.609375,lr : 3.748096e-05 2020-12-21 23:45:30,175 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 23:45:30,637 train.py[line:453] INFO: Checkpoint 45 saved ! 2020-12-21 23:51:55,006 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 23:51:55,475 train.py[line:453] INFO: Checkpoint 46 saved ! 2020-12-21 23:53:21,179 train.py[line:412] DEBUG: Train step_7360: loss : 250835.6875,loss xy : 620.6097412109375,loss wh : 79.64192962646484,loss obj : 249034.765625，loss cls : 1100.6773681640625,loss l2 : 89244.28125,lr : 4.477456000000001e-05 2020-12-21 23:58:45,558 train.py[line:448] INFO: Created checkpoint directory 2020-12-21 23:58:46,128 train.py[line:453] INFO: Checkpoint 47 saved ! 2020-12-22 00:05:06,305 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 00:05:06,780 train.py[line:453] INFO: Checkpoint 48 saved ! 2020-12-22 00:06:35,057 train.py[line:412] DEBUG: Train step_7680: loss : 250094.515625,loss xy : 596.906982421875,loss wh : 122.00531768798828,loss obj : 248320.953125，loss cls : 1054.6531982421875,loss l2 : 88938.109375,lr : 5.308415999999999e-05 2020-12-22 00:11:30,609 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 00:11:31,087 train.py[line:453] INFO: Checkpoint 49 saved ! 2020-12-22 00:17:54,327 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 00:17:54,764 train.py[line:453] INFO: Checkpoint 50 saved ! 2020-12-22 00:19:27,647 train.py[line:412] DEBUG: Train step_8000: loss : 249619.546875,loss xy : 612.5057373046875,loss wh : 93.42715454101562,loss obj : 247825.484375，loss cls : 1088.1214599609375,loss l2 : 88716.828125,lr : 6.25e-05 2020-12-22 00:24:19,515 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 00:24:19,950 train.py[line:453] INFO: Checkpoint 51 saved ! 2020-12-22 00:30:42,569 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 00:30:43,005 train.py[line:453] INFO: Checkpoint 52 saved ! 2020-12-22 00:32:19,095 train.py[line:412] DEBUG: Train step_8320: loss : 248464.734375,loss xy : 709.0609741210938,loss wh : 106.85743713378906,loss obj : 246379.25，loss cls : 1269.5614013671875,loss l2 : 88115.9296875,lr : 7.311616000000002e-05 2020-12-22 00:37:08,713 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 00:37:09,206 train.py[line:453] INFO: Checkpoint 53 saved ! 2020-12-22 00:43:43,028 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 00:43:43,537 train.py[line:453] INFO: Checkpoint 54 saved ! 2020-12-22 00:45:24,474 train.py[line:412] DEBUG: Train step_8640: loss : 247633.53125,loss xy : 509.00390625,loss wh : 110.3015365600586,loss obj : 246111.0625，loss cls : 903.1536865234375,loss l2 : 87867.2734375,lr : 8.503056000000002e-05 2020-12-22 00:50:14,922 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 00:50:15,480 train.py[line:453] INFO: Checkpoint 55 saved ! 2020-12-22 00:56:44,390 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 00:56:44,923 train.py[line:453] INFO: Checkpoint 56 saved ! 2020-12-22 00:58:28,334 train.py[line:412] DEBUG: Train step_8960: loss : 245841.40625,loss xy : 455.60125732421875,loss wh : 105.19612121582031,loss obj : 244466.125，loss cls : 814.4820556640625,loss l2 : 86984.171875,lr : 9.834496000000003e-05 2020-12-22 01:03:10,880 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 01:03:11,370 train.py[line:453] INFO: Checkpoint 57 saved ! 2020-12-22 01:09:39,343 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 01:09:39,809 train.py[line:453] INFO: Checkpoint 58 saved ! 2020-12-22 01:11:27,517 train.py[line:412] DEBUG: Train step_9280: loss : 243908.84375,loss xy : 448.4067077636719,loss wh : 86.17689514160156,loss obj : 242571.65625，loss cls : 802.6090087890625,loss l2 : 86033.515625,lr : 0.00011316495999999998 2020-12-22 01:16:09,146 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 01:16:09,645 train.py[line:453] INFO: Checkpoint 59 saved ! 2020-12-22 01:22:42,539 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 01:22:43,008 train.py[line:453] INFO: Checkpoint 60 saved ! 2020-12-22 01:24:33,679 train.py[line:412] DEBUG: Train step_9600: loss : 242058.421875,loss xy : 594.2020263671875,loss wh : 89.33061218261719,loss obj : 240333.375，loss cls : 1041.5126953125,loss l2 : 85085.4453125,lr : 0.0001296 2020-12-22 01:29:26,738 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 01:29:27,210 train.py[line:453] INFO: Checkpoint 61 saved ! 2020-12-22 01:35:54,430 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 01:35:54,961 train.py[line:453] INFO: Checkpoint 62 saved ! 2020-12-22 01:37:48,485 train.py[line:412] DEBUG: Train step_9920: loss : 240118.78125,loss xy : 404.4917297363281,loss wh : 71.1359634399414,loss obj : 238928.796875，loss cls : 714.352294921875,loss l2 : 84237.1796875,lr : 0.00014776335999999997 2020-12-22 01:42:24,263 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 01:42:24,774 train.py[line:453] INFO: Checkpoint 63 saved ! 2020-12-22 01:49:08,813 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 01:49:09,363 train.py[line:453] INFO: Checkpoint 64 saved ! 2020-12-22 01:51:10,288 train.py[line:412] DEBUG: Train step_10240: loss : 236912.921875,loss xy : 596.1909790039062,loss wh : 106.61688995361328,loss obj : 235157.15625，loss cls : 1052.9564208984375,loss l2 : 82643.5390625,lr : 0.00016777216000000002 2020-12-22 01:55:55,239 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 01:55:55,762 train.py[line:453] INFO: Checkpoint 65 saved ! 2020-12-22 02:02:20,718 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 02:02:21,179 train.py[line:453] INFO: Checkpoint 66 saved ! 2020-12-22 02:04:23,988 train.py[line:412] DEBUG: Train step_10560: loss : 233965.921875,loss xy : 489.2898864746094,loss wh : 101.1136245727539,loss obj : 232526.34375，loss cls : 849.1663818359375,loss l2 : 81265.4453125,lr : 0.00018974736000000004 2020-12-22 02:08:56,443 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 02:08:57,015 train.py[line:453] INFO: Checkpoint 67 saved ! 2020-12-22 02:15:19,076 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 02:15:19,578 train.py[line:453] INFO: Checkpoint 68 saved ! 2020-12-22 02:17:26,161 train.py[line:412] DEBUG: Train step_10880: loss : 232259.640625,loss xy : 503.7485046386719,loss wh : 116.04818725585938,loss obj : 230751.578125，loss cls : 888.2708740234375,loss l2 : 80495.0,lr : 0.0002138137600000001 2020-12-22 02:21:56,878 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 02:21:57,404 train.py[line:453] INFO: Checkpoint 69 saved ! 2020-12-22 02:28:23,229 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 02:28:23,727 train.py[line:453] INFO: Checkpoint 70 saved ! 2020-12-22 02:30:35,739 train.py[line:412] DEBUG: Train step_11200: loss : 229035.78125,loss xy : 571.6559448242188,loss wh : 103.7479476928711,loss obj : 227373.078125，loss cls : 987.296630859375,loss l2 : 78886.7421875,lr : 0.00024009999999999995 2020-12-22 02:34:52,554 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 02:34:53,063 train.py[line:453] INFO: Checkpoint 71 saved ! 2020-12-22 02:41:26,217 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 02:41:26,714 train.py[line:453] INFO: Checkpoint 72 saved ! 2020-12-22 02:43:39,590 train.py[line:412] DEBUG: Train step_11520: loss : 224452.21875,loss xy : 496.45184326171875,loss wh : 98.12674713134766,loss obj : 222983.796875，loss cls : 873.8387451171875,loss l2 : 76744.3046875,lr : 0.00026873855999999996 2020-12-22 02:47:53,188 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 02:47:53,675 train.py[line:453] INFO: Checkpoint 73 saved ! 2020-12-22 02:54:20,757 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 02:54:21,227 train.py[line:453] INFO: Checkpoint 74 saved ! 2020-12-22 02:56:37,427 train.py[line:412] DEBUG: Train step_11840: loss : 220901.546875,loss xy : 490.3206787109375,loss wh : 85.98580932617188,loss obj : 219467.65625，loss cls : 857.5802001953125,loss l2 : 75016.671875,lr : 0.00029986575999999996 2020-12-22 03:00:48,759 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 03:00:49,291 train.py[line:453] INFO: Checkpoint 75 saved ! 2020-12-22 03:07:16,260 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 03:07:16,768 train.py[line:453] INFO: Checkpoint 76 saved ! 2020-12-22 03:09:36,026 train.py[line:412] DEBUG: Train step_12160: loss : 218381.09375,loss xy : 473.66650390625,loss wh : 63.221073150634766,loss obj : 217026.484375，loss cls : 817.7235717773438,loss l2 : 73808.8671875,lr : 0.00033362176 2020-12-22 03:13:44,721 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 03:13:45,289 train.py[line:453] INFO: Checkpoint 77 saved ! 2020-12-22 03:20:17,682 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 03:20:18,146 train.py[line:453] INFO: Checkpoint 78 saved ! 2020-12-22 03:22:44,857 train.py[line:412] DEBUG: Train step_12480: loss : 215773.796875,loss xy : 558.5208740234375,loss wh : 148.06674194335938,loss obj : 214101.5，loss cls : 965.7032470703125,loss l2 : 72680.8984375,lr : 0.00037015056000000006 2020-12-22 03:26:45,801 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 03:26:46,263 train.py[line:453] INFO: Checkpoint 79 saved ! 2020-12-22 03:33:25,078 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 03:33:25,612 train.py[line:453] INFO: Checkpoint 80 saved ! 2020-12-22 03:35:55,136 train.py[line:412] DEBUG: Train step_12800: loss : 212135.9375,loss xy : 580.5303344726562,loss wh : 110.60124206542969,loss obj : 210438.859375，loss cls : 1005.9506225585938,loss l2 : 70913.9375,lr : 0.0004096000000000001 2020-12-22 03:39:54,511 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 03:39:55,008 train.py[line:453] INFO: Checkpoint 81 saved ! 2020-12-22 03:46:22,790 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 03:46:23,299 train.py[line:453] INFO: Checkpoint 82 saved ! 2020-12-22 03:48:54,616 train.py[line:412] DEBUG: Train step_13120: loss : 208099.015625,loss xy : 672.5530395507812,loss wh : 174.0306854248047,loss obj : 206095.984375，loss cls : 1156.4595947265625,loss l2 : 69067.7734375,lr : 0.0004521217599999999 2020-12-22 03:52:57,048 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 03:52:57,630 train.py[line:453] INFO: Checkpoint 83 saved ! 2020-12-22 03:59:20,876 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 03:59:21,308 train.py[line:453] INFO: Checkpoint 84 saved ! 2020-12-22 04:01:56,324 train.py[line:412] DEBUG: Train step_13440: loss : 205949.53125,loss xy : 657.1719360351562,loss wh : 144.53524780273438,loss obj : 204020.21875，loss cls : 1127.6158447265625,loss l2 : 68029.7890625,lr : 0.00049787136 2020-12-22 04:05:42,570 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 04:05:42,997 train.py[line:453] INFO: Checkpoint 85 saved ! 2020-12-22 04:12:06,068 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 04:12:06,501 train.py[line:453] INFO: Checkpoint 86 saved ! 2020-12-22 04:14:45,427 train.py[line:412] DEBUG: Train step_13760: loss : 199885.375,loss xy : 733.7962646484375,loss wh : 214.91200256347656,loss obj : 197688.125，loss cls : 1248.5400390625,loss l2 : 65266.890625,lr : 0.00054700816 2020-12-22 04:18:33,718 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 04:18:34,268 train.py[line:453] INFO: Checkpoint 87 saved ! 2020-12-22 04:25:04,147 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 04:25:04,661 train.py[line:453] INFO: Checkpoint 88 saved ! 2020-12-22 04:27:47,088 train.py[line:412] DEBUG: Train step_14080: loss : 194853.671875,loss xy : 583.1172485351562,loss wh : 97.31462860107422,loss obj : 193183.5，loss cls : 989.7272338867188,loss l2 : 62811.21484375,lr : 0.00059969536 2020-12-22 04:31:30,716 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 04:31:31,226 train.py[line:453] INFO: Checkpoint 89 saved ! 2020-12-22 04:38:00,673 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 04:38:01,185 train.py[line:453] INFO: Checkpoint 90 saved ! 2020-12-22 04:40:48,525 train.py[line:412] DEBUG: Train step_14400: loss : 190746.25,loss xy : 576.7059936523438,loss wh : 108.34374237060547,loss obj : 189075.5625，loss cls : 985.6478881835938,loss l2 : 60923.73828125,lr : 0.0006561000000000001 2020-12-22 04:44:32,856 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 04:44:33,387 train.py[line:453] INFO: Checkpoint 91 saved ! 2020-12-22 04:51:08,802 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 04:51:09,324 train.py[line:453] INFO: Checkpoint 92 saved ! 2020-12-22 04:53:57,421 train.py[line:412] DEBUG: Train step_14720: loss : 185067.390625,loss xy : 463.22845458984375,loss wh : 75.70800018310547,loss obj : 183761.140625，loss cls : 767.3101196289062,loss l2 : 58284.5703125,lr : 0.0007163929600000001 2020-12-22 04:57:37,232 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 04:57:37,702 train.py[line:453] INFO: Checkpoint 93 saved ! 2020-12-22 05:04:04,172 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 05:04:04,645 train.py[line:453] INFO: Checkpoint 94 saved ! 2020-12-22 05:06:59,211 train.py[line:412] DEBUG: Train step_15040: loss : 180506.6875,loss xy : 558.74462890625,loss wh : 117.15518188476562,loss obj : 178898.71875，loss cls : 932.0557250976562,loss l2 : 56169.9140625,lr : 0.0007807489599999998 2020-12-22 05:10:34,240 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 05:10:34,703 train.py[line:453] INFO: Checkpoint 95 saved ! 2020-12-22 05:16:59,227 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 05:16:59,721 train.py[line:453] INFO: Checkpoint 96 saved ! 2020-12-22 05:19:54,987 train.py[line:412] DEBUG: Train step_15360: loss : 175359.625,loss xy : 584.7514038085938,loss wh : 104.47136688232422,loss obj : 173695.75，loss cls : 974.6533203125,loss l2 : 53750.87109375,lr : 0.0008493465599999999 2020-12-22 05:23:19,330 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 05:23:19,826 train.py[line:453] INFO: Checkpoint 97 saved ! 2020-12-22 05:29:41,635 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 05:29:42,130 train.py[line:453] INFO: Checkpoint 98 saved ! 2020-12-22 05:32:41,878 train.py[line:412] DEBUG: Train step_15680: loss : 169072.90625,loss xy : 469.10260009765625,loss wh : 77.5811538696289,loss obj : 167746.78125，loss cls : 779.444091796875,loss l2 : 50857.31640625,lr : 0.0009223681599999999 2020-12-22 05:36:03,954 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 05:36:04,412 train.py[line:453] INFO: Checkpoint 99 saved ! 2020-12-22 05:42:34,379 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 05:42:34,912 train.py[line:453] INFO: Checkpoint 100 saved ! 2020-12-22 05:45:39,952 train.py[line:412] DEBUG: Train step_16000: loss : 163320.8125,loss xy : 447.79705810546875,loss wh : 51.33760070800781,loss obj : 162094.640625，loss cls : 727.0296020507812,loss l2 : 48208.8671875,lr : 0.001 2020-12-22 05:48:58,316 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 05:48:58,798 train.py[line:453] INFO: Checkpoint 101 saved ! 2020-12-22 05:55:26,906 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 05:55:27,427 train.py[line:453] INFO: Checkpoint 102 saved ! 2020-12-22 05:58:37,722 train.py[line:412] DEBUG: Train step_16320: loss : 158637.59375,loss xy : 542.1254272460938,loss wh : 57.35289764404297,loss obj : 157150.828125，loss cls : 887.2739868164062,loss l2 : 46022.11328125,lr : 0.001 2020-12-22 06:01:52,721 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 06:01:53,231 train.py[line:453] INFO: Checkpoint 103 saved ! 2020-12-22 06:08:16,007 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 06:08:16,507 train.py[line:453] INFO: Checkpoint 104 saved ! 2020-12-22 06:11:30,064 train.py[line:412] DEBUG: Train step_16640: loss : 152922.296875,loss xy : 569.4273681640625,loss wh : 87.36087799072266,loss obj : 151350.1875，loss cls : 915.335693359375,loss l2 : 43480.578125,lr : 0.001 2020-12-22 06:14:37,629 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 06:14:38,063 train.py[line:453] INFO: Checkpoint 105 saved ! 2020-12-22 06:20:59,556 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 06:21:00,035 train.py[line:453] INFO: Checkpoint 106 saved ! 2020-12-22 06:24:16,322 train.py[line:412] DEBUG: Train step_16960: loss : 150142.015625,loss xy : 536.94677734375,loss wh : 118.37590789794922,loss obj : 148633.78125，loss cls : 852.9093627929688,loss l2 : 42312.671875,lr : 0.001 2020-12-22 06:27:30,843 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 06:27:31,315 train.py[line:453] INFO: Checkpoint 107 saved ! 2020-12-22 06:33:46,729 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 06:33:47,197 train.py[line:453] INFO: Checkpoint 108 saved ! 2020-12-22 06:37:04,377 train.py[line:412] DEBUG: Train step_17280: loss : 146230.59375,loss xy : 566.9522705078125,loss wh : 98.50648498535156,loss obj : 144672.34375，loss cls : 892.7972412109375,loss l2 : 40510.6640625,lr : 0.001 2020-12-22 06:40:13,727 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 06:40:14,310 train.py[line:453] INFO: Checkpoint 109 saved ! 2020-12-22 06:46:35,792 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 06:46:36,302 train.py[line:453] INFO: Checkpoint 110 saved ! 2020-12-22 06:50:01,358 train.py[line:412] DEBUG: Train step_17600: loss : 141624.171875,loss xy : 518.603271484375,loss wh : 72.64222717285156,loss obj : 140230.296875，loss cls : 802.6242065429688,loss l2 : 38493.5078125,lr : 0.001 2020-12-22 06:52:59,545 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 06:53:00,059 train.py[line:453] INFO: Checkpoint 111 saved ! 2020-12-22 06:59:18,727 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 06:59:19,208 train.py[line:453] INFO: Checkpoint 112 saved ! 2020-12-22 07:02:44,090 train.py[line:412] DEBUG: Train step_17920: loss : 137487.65625,loss xy : 432.05987548828125,loss wh : 46.396705627441406,loss obj : 136344.609375，loss cls : 664.6012573242188,loss l2 : 36670.39453125,lr : 0.001 2020-12-22 07:05:39,574 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 07:05:40,130 train.py[line:453] INFO: Checkpoint 113 saved ! 2020-12-22 07:12:02,667 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 07:12:03,214 train.py[line:453] INFO: Checkpoint 114 saved ! 2020-12-22 07:15:34,459 train.py[line:412] DEBUG: Train step_18240: loss : 135058.0,loss xy : 372.59417724609375,loss wh : 47.59729766845703,loss obj : 134067.6875，loss cls : 570.1260375976562,loss l2 : 35652.2421875,lr : 0.001 2020-12-22 07:18:29,709 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 07:18:30,192 train.py[line:453] INFO: Checkpoint 115 saved ! 2020-12-22 07:24:56,225 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 07:24:56,730 train.py[line:453] INFO: Checkpoint 116 saved ! 2020-12-22 07:28:30,362 train.py[line:412] DEBUG: Train step_18560: loss : 132074.96875,loss xy : 513.9873046875,loss wh : 80.48649597167969,loss obj : 130684.8046875，loss cls : 795.688720703125,loss l2 : 34377.30078125,lr : 0.001 2020-12-22 07:31:12,867 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 07:31:13,344 train.py[line:453] INFO: Checkpoint 117 saved ! 2020-12-22 07:37:36,233 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 07:37:36,713 train.py[line:453] INFO: Checkpoint 118 saved ! 2020-12-22 07:41:13,940 train.py[line:412] DEBUG: Train step_18880: loss : 130985.046875,loss xy : 514.0846557617188,loss wh : 35.394405364990234,loss obj : 129656.6484375，loss cls : 778.9251708984375,loss l2 : 33839.265625,lr : 0.001 2020-12-22 07:43:54,879 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 07:43:55,398 train.py[line:453] INFO: Checkpoint 119 saved ! 2020-12-22 07:50:26,128 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 07:50:26,610 train.py[line:453] INFO: Checkpoint 120 saved ! 2020-12-22 07:54:07,999 train.py[line:412] DEBUG: Train step_19200: loss : 128256.453125,loss xy : 611.4891357421875,loss wh : 100.76364135742188,loss obj : 126605.6328125，loss cls : 938.5733642578125,loss l2 : 32780.1640625,lr : 0.001 2020-12-22 07:56:47,439 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 07:56:47,900 train.py[line:453] INFO: Checkpoint 121 saved ! 2020-12-22 08:03:06,794 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 08:03:07,302 train.py[line:453] INFO: Checkpoint 122 saved ! 2020-12-22 08:06:53,007 train.py[line:412] DEBUG: Train step_19520: loss : 130906.71875,loss xy : 556.4761962890625,loss wh : 160.22706604003906,loss obj : 129333.9609375，loss cls : 856.0540771484375,loss l2 : 34039.8828125,lr : 0.001 2020-12-22 08:09:26,693 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 08:09:27,188 train.py[line:453] INFO: Checkpoint 123 saved ! 2020-12-22 08:15:44,308 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 08:15:44,810 train.py[line:453] INFO: Checkpoint 124 saved ! 2020-12-22 08:19:32,864 train.py[line:412] DEBUG: Train step_19840: loss : 123652.8671875,loss xy : 547.7973022460938,loss wh : 70.6855697631836,loss obj : 122205.609375，loss cls : 828.775146484375,loss l2 : 30867.66796875,lr : 0.001 2020-12-22 08:22:03,152 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 08:22:03,651 train.py[line:453] INFO: Checkpoint 125 saved ! 2020-12-22 08:28:25,931 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 08:28:26,430 train.py[line:453] INFO: Checkpoint 126 saved ! 2020-12-22 08:32:20,669 train.py[line:412] DEBUG: Train step_20160: loss : 123649.171875,loss xy : 607.2474975585938,loss wh : 70.25634765625,loss obj : 122063.5703125，loss cls : 908.09912109375,loss l2 : 30837.783203125,lr : 0.001 2020-12-22 08:34:46,504 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 08:34:46,974 train.py[line:453] INFO: Checkpoint 127 saved ! 2020-12-22 08:41:04,747 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 08:41:05,338 train.py[line:453] INFO: Checkpoint 128 saved ! 2020-12-22 08:44:57,198 train.py[line:412] DEBUG: Train step_20480: loss : 120828.875,loss xy : 557.9169921875,loss wh : 80.22119140625,loss obj : 119377.109375，loss cls : 813.6236572265625,loss l2 : 29732.01953125,lr : 0.001 2020-12-22 08:47:23,248 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 08:47:23,742 train.py[line:453] INFO: Checkpoint 129 saved ! 2020-12-22 08:53:52,771 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 08:53:53,262 train.py[line:453] INFO: Checkpoint 130 saved ! 2020-12-22 08:57:51,795 train.py[line:412] DEBUG: Train step_20800: loss : 119358.765625,loss xy : 489.08331298828125,loss wh : 28.90752601623535,loss obj : 118119.1796875，loss cls : 721.5936889648438,loss l2 : 29088.455078125,lr : 0.001 2020-12-22 09:00:07,916 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 09:00:08,351 train.py[line:453] INFO: Checkpoint 131 saved ! 2020-12-22 09:06:26,573 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 09:06:27,092 train.py[line:453] INFO: Checkpoint 132 saved ! 2020-12-22 09:10:29,957 train.py[line:412] DEBUG: Train step_21120: loss : 118288.4609375,loss xy : 559.6806030273438,loss wh : 67.71033477783203,loss obj : 116850.9609375，loss cls : 810.1102294921875,loss l2 : 28703.30859375,lr : 0.001 2020-12-22 09:12:55,397 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 09:12:55,861 train.py[line:453] INFO: Checkpoint 133 saved ! 2020-12-22 09:19:12,029 train.py[line:448] INFO: Created checkpoint directory 2020-12-22 09:19:12,533 train.py[line:453] INFO: Checkpoint 134 saved ! 2020-12-22 09:23:22,100 train.py[line:412] DEBUG: Train step_21440: loss : 117128.4140625,loss xy : 573.6002197265625,loss wh : 92.7562255859375,loss obj : 115635.984375，loss cls : 826.07080078125,loss l2 : 28275.53515625,lr : 0.001

[snowyseaons]

Have you tried bigger learning rate?

[adrien-jacquot]

Try the following config (suggested in https://github.com/Tianxiaomo/pytorch-YOLOv4/blob/master/Use_yolov4_to_train_your_own_data.md)

Cfg.batch = 4
Cfg.subdivisions = 1

[APPLE-XMT]

您好，请问这个问题您解决了吗·