mindyolo icon indicating copy to clipboard operation
mindyolo copied to clipboard

Published 20 hours ago • verified publisher icon mindspore-lab

finetune visdrone

Open WangFengtu1996 opened this issue 10 months ago • 20 comments

  • using example finetue visdrone to train model in visdrone dataset.
  • can not load yolov8 pretrain model to train model ?

env

single nvidia geforce rtx 4090

WangFengtu1996 avatar Apr 01 '24 02:04 WangFengtu1996

在GPU 上面，使用这个example ，loss 处于一个不断震荡，无法减小loss的值。

WangFengtu1996 avatar Apr 02 '24 02:04 WangFengtu1996

可以尝试降低学习率试一下

zhanghuiyao avatar Apr 07 '24 07:04 zhanghuiyao

降低学习率，没有办法解决这个问题

WangFengtu1996 avatar Apr 07 '24 07:04 WangFengtu1996

pr 280跟这个是一个场景吗 是的话 每次学习的loss应该是不对的 https://github.com/mindspore-lab/mindyolo/issues/280

zhanghuiyao avatar Apr 07 '24 07:04 zhanghuiyao

是的。这个是我这边训练的loss。

2024-04-07 06:56:19,709 [INFO] Epoch 1/300, Step 100/101, imgsize (640, 640), loss: 0.7016, lbox: 0.1512, lobj: 0.1086, lcls: 0.4418, is_group_lr  , cur_lr: 0.07029703259468079
2024-04-07 06:56:19,712 [INFO] Epoch 1/300, Step 100/101, step time: 1042.46 ms
2024-04-07 06:56:20,739 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-1_101.ckpt
2024-04-07 06:56:20,739 [INFO] Epoch 1/300, epoch time: 1.75 min.
2024-04-07 06:57:50,282 [INFO] Epoch 2/300, Step 100/101, imgsize (640, 640), loss: 0.7020, lbox: 0.1528, lobj: 0.1076, lcls: 0.4416, is_group_lr  , cur_lr: 0.04027712717652321
2024-04-07 06:57:50,288 [INFO] Epoch 2/300, Step 100/101, step time: 895.49 ms
2024-04-07 06:57:51,361 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-2_101.ckpt
2024-04-07 06:57:51,364 [INFO] Epoch 2/300, epoch time: 1.51 min.
2024-04-07 06:59:20,665 [INFO] Epoch 3/300, Step 100/101, imgsize (640, 640), loss: 0.7034, lbox: 0.1516, lobj: 0.1119, lcls: 0.4399, is_group_lr  , cur_lr: 0.010237228125333786
2024-04-07 06:59:20,669 [INFO] Epoch 3/300, Step 100/101, step time: 893.02 ms
2024-04-07 06:59:21,536 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-3_101.ckpt
2024-04-07 06:59:21,536 [INFO] Epoch 3/300, epoch time: 1.50 min.
2024-04-07 07:00:50,036 [INFO] Epoch 4/300, Step 100/101, imgsize (640, 640), loss: 0.6930, lbox: 0.1509, lobj: 0.1024, lcls: 0.4398, is_group_lr  , cur_lr: 0.009909999556839466
2024-04-07 07:00:50,053 [INFO] Epoch 4/300, Step 100/101, step time: 885.17 ms
2024-04-07 07:00:51,080 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-4_101.ckpt
2024-04-07 07:00:51,084 [INFO] Epoch 4/300, epoch time: 1.49 min.
2024-04-07 07:02:21,884 [INFO] Epoch 5/300, Step 100/101, imgsize (640, 640), loss: 0.7084, lbox: 0.1547, lobj: 0.1141, lcls: 0.4397, is_group_lr  , cur_lr: 0.009879999794065952
2024-04-07 07:02:22,001 [INFO] Epoch 5/300, Step 100/101, step time: 909.13 ms
2024-04-07 07:02:22,789 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-5_101.ckpt
2024-04-07 07:02:22,790 [INFO] Epoch 5/300, epoch time: 1.53 min.
2024-04-07 07:03:52,419 [INFO] Epoch 6/300, Step 100/101, imgsize (640, 640), loss: 0.7078, lbox: 0.1517, lobj: 0.1185, lcls: 0.4376, is_group_lr  , cur_lr: 0.009850000031292439
2024-04-07 07:03:52,422 [INFO] Epoch 6/300, Step 100/101, step time: 896.33 ms
2024-04-07 07:03:53,325 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-6_101.ckpt
2024-04-07 07:03:53,347 [INFO] Epoch 6/300, epoch time: 1.51 min.
2024-04-07 07:05:22,812 [INFO] Epoch 7/300, Step 100/101, imgsize (640, 640), loss: 0.6980, lbox: 0.1520, lobj: 0.1063, lcls: 0.4397, is_group_lr  , cur_lr: 0.009820000268518925
2024-04-07 07:05:22,814 [INFO] Epoch 7/300, Step 100/101, step time: 894.62 ms
2024-04-07 07:05:23,583 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-7_101.ckpt
2024-04-07 07:05:23,583 [INFO] Epoch 7/300, epoch time: 1.50 min.
2024-04-07 07:06:51,892 [INFO] Epoch 8/300, Step 100/101, imgsize (640, 640), loss: 0.6983, lbox: 0.1493, lobj: 0.1112, lcls: 0.4378, is_group_lr  , cur_lr: 0.009789999574422836
2024-04-07 07:06:51,896 [INFO] Epoch 8/300, Step 100/101, step time: 883.12 ms
2024-04-07 07:06:52,962 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-8_101.ckpt
2024-04-07 07:06:52,962 [INFO] Epoch 8/300, epoch time: 1.49 min.
2024-04-07 07:08:23,364 [INFO] Epoch 9/300, Step 100/101, imgsize (640, 640), loss: 0.7062, lbox: 0.1504, lobj: 0.1164, lcls: 0.4394, is_group_lr  , cur_lr: 0.009759999811649323
2024-04-07 07:08:23,365 [INFO] Epoch 9/300, Step 100/101, step time: 904.02 ms
2024-04-07 07:08:24,201 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-9_101.ckpt
2024-04-07 07:08:24,201 [INFO] Epoch 9/300, epoch time: 1.52 min.
2024-04-07 07:09:53,871 [INFO] Epoch 10/300, Step 100/101, imgsize (640, 640), loss: 0.7013, lbox: 0.1518, lobj: 0.1102, lcls: 0.4393, is_group_lr  , cur_lr: 0.009730000048875809
2024-04-07 07:09:53,874 [INFO] Epoch 10/300, Step 100/101, step time: 896.72 ms
2024-04-07 07:09:54,637 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-10_101.ckpt
2024-04-07 07:09:54,638 [INFO] Epoch 10/300, epoch time: 1.51 min.
2024-04-07 07:11:23,666 [INFO] Epoch 11/300, Step 100/101, imgsize (640, 640), loss: 0.6950, lbox: 0.1518, lobj: 0.1055, lcls: 0.4377, is_group_lr  , cur_lr: 0.009700000286102295
2024-04-07 07:11:23,668 [INFO] Epoch 11/300, Step 100/101, step time: 890.30 ms
2024-04-07 07:11:24,582 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-11_101.ckpt
2024-04-07 07:11:24,582 [INFO] Epoch 11/300, epoch time: 1.50 min.
2024-04-07 07:12:53,847 [INFO] Epoch 12/300, Step 100/101, imgsize (640, 640), loss: 0.7008, lbox: 0.1494, lobj: 0.1138, lcls: 0.4376, is_group_lr  , cur_lr: 0.009669999592006207
2024-04-07 07:12:53,848 [INFO] Epoch 12/300, Step 100/101, step time: 892.66 ms
2024-04-07 07:12:54,870 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-12_101.ckpt
2024-04-07 07:12:54,870 [INFO] Epoch 12/300, epoch time: 1.50 min.
2024-04-07 07:14:23,942 [INFO] Epoch 13/300, Step 100/101, imgsize (640, 640), loss: 0.7061, lbox: 0.1507, lobj: 0.1180, lcls: 0.4374, is_group_lr  , cur_lr: 0.009639999829232693
2024-04-07 07:14:23,944 [INFO] Epoch 13/300, Step 100/101, step time: 890.73 ms
2024-04-07 07:14:24,742 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-13_101.ckpt
2024-04-07 07:14:24,743 [INFO] Epoch 13/300, epoch time: 1.50 min.
2024-04-07 07:15:55,605 [INFO] Epoch 14/300, Step 100/101, imgsize (640, 640), loss: 0.6863, lbox: 0.1518, lobj: 0.0980, lcls: 0.4365, is_group_lr  , cur_lr: 0.009610000066459179
2024-04-07 07:15:55,607 [INFO] Epoch 14/300, Step 100/101, step time: 908.64 ms
2024-04-07 07:15:56,599 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-14_101.ckpt
2024-04-07 07:15:56,599 [INFO] Epoch 14/300, epoch time: 1.53 min.
2024-04-07 07:17:26,110 [INFO] Epoch 15/300, Step 100/101, imgsize (640, 640), loss: 0.6954, lbox: 0.1504, lobj: 0.1081, lcls: 0.4369, is_group_lr  , cur_lr: 0.009580000303685665
2024-04-07 07:17:26,112 [INFO] Epoch 15/300, Step 100/101, step time: 894.97 ms
2024-04-07 07:17:26,940 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-15_101.ckpt
2024-04-07 07:17:26,940 [INFO] Epoch 15/300, epoch time: 1.51 min.
2024-04-07 07:18:54,888 [INFO] Epoch 16/300, Step 100/101, imgsize (640, 640), loss: 0.7003, lbox: 0.1516, lobj: 0.1127, lcls: 0.4360, is_group_lr  , cur_lr: 0.009549999609589577
2024-04-07 07:18:54,889 [INFO] Epoch 16/300, Step 100/101, step time: 879.49 ms
2024-04-07 07:18:55,985 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-16_101.ckpt
2024-04-07 07:18:55,985 [INFO] Epoch 16/300, epoch time: 1.48 min.
2024-04-07 07:20:24,867 [INFO] Epoch 17/300, Step 100/101, imgsize (640, 640), loss: 0.6888, lbox: 0.1515, lobj: 0.1020, lcls: 0.4354, is_group_lr  , cur_lr: 0.009519999846816063
2024-04-07 07:20:24,869 [INFO] Epoch 17/300, Step 100/101, step time: 888.83 ms
2024-04-07 07:20:25,817 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-17_101.ckpt
2024-04-07 07:20:25,818 [INFO] Epoch 17/300, epoch time: 1.50 min.
2024-04-07 07:21:54,060 [INFO] Epoch 18/300, Step 100/101, imgsize (640, 640), loss: 0.6966, lbox: 0.1519, lobj: 0.1099, lcls: 0.4349, is_group_lr  , cur_lr: 0.00949000008404255
2024-04-07 07:21:54,066 [INFO] Epoch 18/300, Step 100/101, step time: 882.48 ms
2024-04-07 07:21:54,941 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-18_101.ckpt
2024-04-07 07:21:54,942 [INFO] Epoch 18/300, epoch time: 1.49 min.
2024-04-07 07:23:25,204 [INFO] Epoch 19/300, Step 100/101, imgsize (640, 640), loss: 0.6930, lbox: 0.1511, lobj: 0.1065, lcls: 0.4354, is_group_lr  , cur_lr: 0.009460000321269035
2024-04-07 07:23:25,206 [INFO] Epoch 19/300, Step 100/101, step time: 902.64 ms
2024-04-07 07:23:26,142 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-19_101.ckpt
2024-04-07 07:23:26,142 [INFO] Epoch 19/300, epoch time: 1.52 min.
2024-04-07 07:24:56,520 [INFO] Epoch 20/300, Step 100/101, imgsize (640, 640), loss: 0.6985, lbox: 0.1507, lobj: 0.1130, lcls: 0.4349, is_group_lr  , cur_lr: 0.009429999627172947
2024-04-07 07:24:56,525 [INFO] Epoch 20/300, Step 100/101, step time: 903.82 ms
2024-04-07 07:24:57,456 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-20_101.ckpt
2024-04-07 07:24:57,457 [INFO] Epoch 20/300, epoch time: 1.52 min.
2024-04-07 07:26:27,775 [INFO] Epoch 21/300, Step 100/101, imgsize (640, 640), loss: 0.6962, lbox: 0.1507, lobj: 0.1116, lcls: 0.4338, is_group_lr  , cur_lr: 0.009399999864399433
2024-04-07 07:26:27,777 [INFO] Epoch 21/300, Step 100/101, step time: 903.19 ms
2024-04-07 07:26:28,683 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-21_101.ckpt
2024-04-07 07:26:28,684 [INFO] Epoch 21/300, epoch time: 1.52 min.
2024-04-07 07:27:58,161 [INFO] Epoch 22/300, Step 100/101, imgsize (640, 640), loss: 0.6921, lbox: 0.1505, lobj: 0.1075, lcls: 0.4340, is_group_lr  , cur_lr: 0.00937000010162592
2024-04-07 07:27:58,167 [INFO] Epoch 22/300, Step 100/101, step time: 894.83 ms
2024-04-07 07:27:59,300 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-22_101.ckpt
2024-04-07 07:27:59,300 [INFO] Epoch 22/300, epoch time: 1.51 min.
2024-04-07 07:29:28,303 [INFO] Epoch 23/300, Step 100/101, imgsize (640, 640), loss: 0.6943, lbox: 0.1509, lobj: 0.1101, lcls: 0.4334, is_group_lr  , cur_lr: 0.009340000338852406
2024-04-07 07:29:28,306 [INFO] Epoch 23/300, Step 100/101, step time: 890.05 ms
2024-04-07 07:29:29,152 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-23_101.ckpt
2024-04-07 07:29:29,153 [INFO] Epoch 23/300, epoch time: 1.50 min.
2024-04-07 07:30:59,926 [INFO] Epoch 24/300, Step 100/101, imgsize (640, 640), loss: 0.6889, lbox: 0.1511, lobj: 0.1039, lcls: 0.4339, is_group_lr  , cur_lr: 0.009309999644756317
2024-04-07 07:31:00,059 [INFO] Epoch 24/300, Step 100/101, step time: 909.04 ms
2024-04-07 07:31:00,849 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-24_101.ckpt
2024-04-07 07:31:00,849 [INFO] Epoch 24/300, epoch time: 1.53 min.
2024-04-07 07:32:30,450 [INFO] Epoch 25/300, Step 100/101, imgsize (640, 640), loss: 0.6983, lbox: 0.1504, lobj: 0.1148, lcls: 0.4331, is_group_lr  , cur_lr: 0.009279999881982803
2024-04-07 07:32:30,549 [INFO] Epoch 25/300, Step 100/101, step time: 896.98 ms
2024-04-07 07:32:31,322 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-25_101.ckpt
2024-04-07 07:32:31,322 [INFO] Epoch 25/300, epoch time: 1.51 min.
2024-04-07 07:34:00,766 [INFO] Epoch 26/300, Step 100/101, imgsize (640, 640), loss: 0.6983, lbox: 0.1507, lobj: 0.1140, lcls: 0.4336, is_group_lr  , cur_lr: 0.00925000011920929
2024-04-07 07:34:00,770 [INFO] Epoch 26/300, Step 100/101, step time: 894.47 ms
2024-04-07 07:34:01,724 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-26_101.ckpt
2024-04-07 07:34:01,724 [INFO] Epoch 26/300, epoch time: 1.51 min.
2024-04-07 07:35:29,713 [INFO] Epoch 27/300, Step 100/101, imgsize (640, 640), loss: 0.6931, lbox: 0.1499, lobj: 0.1103, lcls: 0.4329, is_group_lr  , cur_lr: 0.009220000356435776
2024-04-07 07:35:29,714 [INFO] Epoch 27/300, Step 100/101, step time: 879.90 ms
2024-04-07 07:35:30,824 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-27_101.ckpt
2024-04-07 07:35:30,828 [INFO] Epoch 27/300, epoch time: 1.49 min.
2024-04-07 07:36:58,688 [INFO] Epoch 28/300, Step 100/101, imgsize (640, 640), loss: 0.6985, lbox: 0.1500, lobj: 0.1153, lcls: 0.4332, is_group_lr  , cur_lr: 0.009189999662339687
2024-04-07 07:36:58,690 [INFO] Epoch 28/300, Step 100/101, step time: 878.62 ms
2024-04-07 07:36:59,719 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-28_101.ckpt
2024-04-07 07:36:59,719 [INFO] Epoch 28/300, epoch time: 1.48 min.
2024-04-07 07:38:29,504 [INFO] Epoch 29/300, Step 100/101, imgsize (640, 640), loss: 0.6942, lbox: 0.1517, lobj: 0.1084, lcls: 0.4342, is_group_lr  , cur_lr: 0.009159999899566174
2024-04-07 07:38:29,508 [INFO] Epoch 29/300, Step 100/101, step time: 897.89 ms
2024-04-07 07:38:30,385 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-29_101.ckpt
2024-04-07 07:38:30,386 [INFO] Epoch 29/300, epoch time: 1.51 min.
2024-04-07 07:39:57,640 [INFO] Epoch 30/300, Step 100/101, imgsize (640, 640), loss: 0.6960, lbox: 0.1510, lobj: 0.1126, lcls: 0.4324, is_group_lr  , cur_lr: 0.00913000013679266
2024-04-07 07:39:57,642 [INFO] Epoch 30/300, Step 100/101, step time: 872.56 ms
2024-04-07 07:39:58,753 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-30_101.ckpt
2024-04-07 07:39:58,754 [INFO] Epoch 30/300, epoch time: 1.47 min.
2024-04-07 07:41:27,361 [INFO] Epoch 31/300, Step 100/101, imgsize (640, 640), loss: 0.6840, lbox: 0.1514, lobj: 0.0997, lcls: 0.4328, is_group_lr  , cur_lr: 0.009100000374019146
2024-04-07 07:41:27,367 [INFO] Epoch 31/300, Step 100/101, step time: 886.13 ms
2024-04-07 07:41:28,295 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-31_101.ckpt
2024-04-07 07:41:28,296 [INFO] Epoch 31/300, epoch time: 1.49 min.
2024-04-07 07:42:58,320 [INFO] Epoch 32/300, Step 100/101, imgsize (640, 640), loss: 0.7004, lbox: 0.1523, lobj: 0.1153, lcls: 0.4327, is_group_lr  , cur_lr: 0.009069999679923058
2024-04-07 07:42:58,324 [INFO] Epoch 32/300, Step 100/101, step time: 900.28 ms
2024-04-07 07:42:59,093 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-32_101.ckpt
2024-04-07 07:42:59,093 [INFO] Epoch 32/300, epoch time: 1.51 min.
2024-04-07 07:44:28,881 [INFO] Epoch 33/300, Step 100/101, imgsize (640, 640), loss: 0.6895, lbox: 0.1507, lobj: 0.1059, lcls: 0.4330, is_group_lr  , cur_lr: 0.009039999917149544
2024-04-07 07:44:28,884 [INFO] Epoch 33/300, Step 100/101, step time: 897.90 ms
2024-04-07 07:44:29,951 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-33_101.ckpt
2024-04-07 07:44:29,952 [INFO] Epoch 33/300, epoch time: 1.51 min.
2024-04-07 07:45:59,500 [INFO] Epoch 34/300, Step 100/101, imgsize (640, 640), loss: 0.6950, lbox: 0.1497, lobj: 0.1135, lcls: 0.4318, is_group_lr  , cur_lr: 0.00901000015437603
2024-04-07 07:45:59,502 [INFO] Epoch 34/300, Step 100/101, step time: 895.50 ms
2024-04-07 07:46:00,410 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-34_101.ckpt
2024-04-07 07:46:00,410 [INFO] Epoch 34/300, epoch time: 1.51 min.
2024-04-07 07:47:30,100 [INFO] Epoch 35/300, Step 100/101, imgsize (640, 640), loss: 0.6920, lbox: 0.1515, lobj: 0.1099, lcls: 0.4306, is_group_lr  , cur_lr: 0.008980000391602516
2024-04-07 07:47:30,101 [INFO] Epoch 35/300, Step 100/101, step time: 896.90 ms
2024-04-07 07:47:31,258 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-35_101.ckpt
2024-04-07 07:47:31,259 [INFO] Epoch 35/300, epoch time: 1.51 min.
2024-04-07 07:49:00,188 [INFO] Epoch 36/300, Step 100/101, imgsize (640, 640), loss: 0.6905, lbox: 0.1495, lobj: 0.1094, lcls: 0.4317, is_group_lr  , cur_lr: 0.008949999697506428
2024-04-07 07:49:00,192 [INFO] Epoch 36/300, Step 100/101, step time: 889.33 ms
2024-04-07 07:49:01,247 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-36_101.ckpt
2024-04-07 07:49:01,247 [INFO] Epoch 36/300, epoch time: 1.50 min.
2024-04-07 07:50:30,681 [INFO] Epoch 37/300, Step 100/101, imgsize (640, 640), loss: 0.7041, lbox: 0.1511, lobj: 0.1220, lcls: 0.4309, is_group_lr  , cur_lr: 0.008919999934732914
2024-04-07 07:50:30,686 [INFO] Epoch 37/300, Step 100/101, step time: 894.38 ms
2024-04-07 07:50:31,637 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-37_101.ckpt
2024-04-07 07:50:31,637 [INFO] Epoch 37/300, epoch time: 1.51 min.
2024-04-07 07:52:00,231 [INFO] Epoch 38/300, Step 100/101, imgsize (640, 640), loss: 0.6761, lbox: 0.1517, lobj: 0.0937, lcls: 0.4307, is_group_lr  , cur_lr: 0.0088900001719594
2024-04-07 07:52:00,235 [INFO] Epoch 38/300, Step 100/101, step time: 885.98 ms
2024-04-07 07:52:01,122 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-38_101.ckpt
2024-04-07 07:52:01,122 [INFO] Epoch 38/300, epoch time: 1.49 min.
2024-04-07 07:53:29,943 [INFO] Epoch 39/300, Step 100/101, imgsize (640, 640), loss: 0.6893, lbox: 0.1514, lobj: 0.1063, lcls: 0.4316, is_group_lr  , cur_lr: 0.008860000409185886
2024-04-07 07:53:29,945 [INFO] Epoch 39/300, Step 100/101, step time: 888.23 ms
2024-04-07 07:53:30,858 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-39_101.ckpt
2024-04-07 07:53:30,859 [INFO] Epoch 39/300, epoch time: 1.50 min.
2024-04-07 07:55:01,174 [INFO] Epoch 40/300, Step 100/101, imgsize (640, 640), loss: 0.6975, lbox: 0.1513, lobj: 0.1152, lcls: 0.4310, is_group_lr  , cur_lr: 0.008829999715089798
2024-04-07 07:55:01,180 [INFO] Epoch 40/300, Step 100/101, step time: 903.18 ms
2024-04-07 07:55:02,061 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-40_101.ckpt
2024-04-07 07:55:02,061 [INFO] Epoch 40/300, epoch time: 1.52 min.
2024-04-07 07:56:30,041 [INFO] Epoch 41/300, Step 100/101, imgsize (640, 640), loss: 0.7023, lbox: 0.1504, lobj: 0.1207, lcls: 0.4312, is_group_lr  , cur_lr: 0.008799999952316284
2024-04-07 07:56:30,044 [INFO] Epoch 41/300, Step 100/101, step time: 879.83 ms
2024-04-07 07:56:30,916 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-41_101.ckpt
2024-04-07 07:56:30,917 [INFO] Epoch 41/300, epoch time: 1.48 min.
2024-04-07 07:58:00,916 [INFO] Epoch 42/300, Step 100/101, imgsize (640, 640), loss: 0.6902, lbox: 0.1495, lobj: 0.1096, lcls: 0.4311, is_group_lr  , cur_lr: 0.00877000018954277
2024-04-07 07:58:00,919 [INFO] Epoch 42/300, Step 100/101, step time: 900.01 ms
2024-04-07 07:58:01,765 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-42_101.ckpt
2024-04-07 07:58:01,765 [INFO] Epoch 42/300, epoch time: 1.51 min.

WangFengtu1996 avatar Apr 07 '24 08:04 WangFengtu1996

现在的情况是，基于这个yolov5s 使用这个预训练的模型，超参对齐到 官方的yolov5, 训练这个模型。

python train.py  --config examples/finetune_visdrone/yolov5-s-visdrone.yaml --device_target=GPU  --weight yolov5s_300e_mAP376-860bcf3b.ckpt  --strict_load=False
  • 关于这个loss的定义，我理解除了这个api，应该是对齐到这个官方的repo，对吧？这个参数分组，与官方的分组顺序是不一样的这个有啥考虑么？
def _group_param_common3(params):
    pg0, pg1, pg2 = [], [], []  # optimizer parameter groups
    for p in params:
        if "bias" in p.name or "beta" in p.name:
            pg0.append(p)
        elif "weight" in p.name:
            pg1.append(p)
        else:
            pg2.append(p)

    return pg0, pg1, pg2  # bias/beta, weight, others

WangFengtu1996 avatar Apr 07 '24 08:04 WangFengtu1996

分组的逻辑和官方是一致的 这个地方应该只是命名不同 只是为了保证多个模型的一致性

zhanghuiyao avatar Apr 07 '24 08:04 zhanghuiyao

  • 对齐下，我目前尝试的一些方法
  • 使用这个yolov5预训练模型去在visdrone 上面进行训练
python train.py  --config examples/finetune_visdrone/yolov5-s-visdrone.yaml --device_target=GPU  --weight yolov5s_300e_mAP376-860bcf3b.ckpt  --strict_load=False
  • 将mindyolo 中的超参与 官方yolov5的超参做对齐
loss:
  cls: 0.5
  obj: 1.0
  iou_t: 0.20
  • 使用官方的yolov5的 在visdrone 上面重新聚类，重新生成这个anchors。
 - [3.2203, 4.5185,  4.4992, 8.9727, 8.3438, 14.641]
    - [16.131, 9.4856, 16.131,9.4856,  16.004, 21.081 ]
    - [ 33.011, 16.956, 29.098, 36.673,  61.252, 62.739]
  • 我这边检查过训练集，以及数据增前后的数据集，发现没啥问题 https://github.com/mindspore-lab/mindyolo/issues/282#issue-2222399451
  • 但是我这边还是遇到上面的问题，可以指导下我如何去完

WangFengtu1996 avatar Apr 07 '24 08:04 WangFengtu1996

可以尝试在 def build_targets link 函数中debug看下是哪一步导致的 tmasks 出现全0

从输入的 predict 和 target 到return的 tmasks

zhanghuiyao avatar Apr 07 '24 08:04 zhanghuiyao

这个是我的训练log， 这个loss 已经非nan了，也就没有出现全零了。 是的。这个是我这边训练的loss。

2024-04-07 06:56:19,709 [INFO] Epoch 1/300, Step 100/101, imgsize (640, 640), loss: 0.7016, lbox: 0.1512, lobj: 0.1086, lcls: 0.4418, is_group_lr  , cur_lr: 0.07029703259468079
2024-04-07 06:56:19,712 [INFO] Epoch 1/300, Step 100/101, step time: 1042.46 ms
2024-04-07 06:56:20,739 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-1_101.ckpt
2024-04-07 06:56:20,739 [INFO] Epoch 1/300, epoch time: 1.75 min.
2024-04-07 06:57:50,282 [INFO] Epoch 2/300, Step 100/101, imgsize (640, 640), loss: 0.7020, lbox: 0.1528, lobj: 0.1076, lcls: 0.4416, is_group_lr  , cur_lr: 0.04027712717652321
2024-04-07 06:57:50,288 [INFO] Epoch 2/300, Step 100/101, step time: 895.49 ms
2024-04-07 06:57:51,361 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-2_101.ckpt
2024-04-07 06:57:51,364 [INFO] Epoch 2/300, epoch time: 1.51 min.
2024-04-07 06:59:20,665 [INFO] Epoch 3/300, Step 100/101, imgsize (640, 640), loss: 0.7034, lbox: 0.1516, lobj: 0.1119, lcls: 0.4399, is_group_lr  , cur_lr: 0.010237228125333786
2024-04-07 06:59:20,669 [INFO] Epoch 3/300, Step 100/101, step time: 893.02 ms
2024-04-07 06:59:21,536 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-3_101.ckpt
2024-04-07 06:59:21,536 [INFO] Epoch 3/300, epoch time: 1.50 min.
2024-04-07 07:00:50,036 [INFO] Epoch 4/300, Step 100/101, imgsize (640, 640), loss: 0.6930, lbox: 0.1509, lobj: 0.1024, lcls: 0.4398, is_group_lr  , cur_lr: 0.009909999556839466
2024-04-07 07:00:50,053 [INFO] Epoch 4/300, Step 100/101, step time: 885.17 ms
2024-04-07 07:00:51,080 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-4_101.ckpt
2024-04-07 07:00:51,084 [INFO] Epoch 4/300, epoch time: 1.49 min.
2024-04-07 07:02:21,884 [INFO] Epoch 5/300, Step 100/101, imgsize (640, 640), loss: 0.7084, lbox: 0.1547, lobj: 0.1141, lcls: 0.4397, is_group_lr  , cur_lr: 0.009879999794065952
2024-04-07 07:02:22,001 [INFO] Epoch 5/300, Step 100/101, step time: 909.13 ms
2024-04-07 07:02:22,789 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-5_101.ckpt
2024-04-07 07:02:22,790 [INFO] Epoch 5/300, epoch time: 1.53 min.
2024-04-07 07:03:52,419 [INFO] Epoch 6/300, Step 100/101, imgsize (640, 640), loss: 0.7078, lbox: 0.1517, lobj: 0.1185, lcls: 0.4376, is_group_lr  , cur_lr: 0.009850000031292439
2024-04-07 07:03:52,422 [INFO] Epoch 6/300, Step 100/101, step time: 896.33 ms
2024-04-07 07:03:53,325 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-6_101.ckpt
2024-04-07 07:03:53,347 [INFO] Epoch 6/300, epoch time: 1.51 min.
2024-04-07 07:05:22,812 [INFO] Epoch 7/300, Step 100/101, imgsize (640, 640), loss: 0.6980, lbox: 0.1520, lobj: 0.1063, lcls: 0.4397, is_group_lr  , cur_lr: 0.009820000268518925
2024-04-07 07:05:22,814 [INFO] Epoch 7/300, Step 100/101, step time: 894.62 ms
2024-04-07 07:05:23,583 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-7_101.ckpt
2024-04-07 07:05:23,583 [INFO] Epoch 7/300, epoch time: 1.50 min.
2024-04-07 07:06:51,892 [INFO] Epoch 8/300, Step 100/101, imgsize (640, 640), loss: 0.6983, lbox: 0.1493, lobj: 0.1112, lcls: 0.4378, is_group_lr  , cur_lr: 0.009789999574422836
2024-04-07 07:06:51,896 [INFO] Epoch 8/300, Step 100/101, step time: 883.12 ms
2024-04-07 07:06:52,962 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-8_101.ckpt
2024-04-07 07:06:52,962 [INFO] Epoch 8/300, epoch time: 1.49 min.
2024-04-07 07:08:23,364 [INFO] Epoch 9/300, Step 100/101, imgsize (640, 640), loss: 0.7062, lbox: 0.1504, lobj: 0.1164, lcls: 0.4394, is_group_lr  , cur_lr: 0.009759999811649323
2024-04-07 07:08:23,365 [INFO] Epoch 9/300, Step 100/101, step time: 904.02 ms
2024-04-07 07:08:24,201 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-9_101.ckpt
2024-04-07 07:08:24,201 [INFO] Epoch 9/300, epoch time: 1.52 min.
2024-04-07 07:09:53,871 [INFO] Epoch 10/300, Step 100/101, imgsize (640, 640), loss: 0.7013, lbox: 0.1518, lobj: 0.1102, lcls: 0.4393, is_group_lr  , cur_lr: 0.009730000048875809
2024-04-07 07:09:53,874 [INFO] Epoch 10/300, Step 100/101, step time: 896.72 ms
2024-04-07 07:09:54,637 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-10_101.ckpt
2024-04-07 07:09:54,638 [INFO] Epoch 10/300, epoch time: 1.51 min.
2024-04-07 07:11:23,666 [INFO] Epoch 11/300, Step 100/101, imgsize (640, 640), loss: 0.6950, lbox: 0.1518, lobj: 0.1055, lcls: 0.4377, is_group_lr  , cur_lr: 0.009700000286102295
2024-04-07 07:11:23,668 [INFO] Epoch 11/300, Step 100/101, step time: 890.30 ms
2024-04-07 07:11:24,582 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-11_101.ckpt
2024-04-07 07:11:24,582 [INFO] Epoch 11/300, epoch time: 1.50 min.
2024-04-07 07:12:53,847 [INFO] Epoch 12/300, Step 100/101, imgsize (640, 640), loss: 0.7008, lbox: 0.1494, lobj: 0.1138, lcls: 0.4376, is_group_lr  , cur_lr: 0.009669999592006207
2024-04-07 07:12:53,848 [INFO] Epoch 12/300, Step 100/101, step time: 892.66 ms
2024-04-07 07:12:54,870 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-12_101.ckpt
2024-04-07 07:12:54,870 [INFO] Epoch 12/300, epoch time: 1.50 min.
2024-04-07 07:14:23,942 [INFO] Epoch 13/300, Step 100/101, imgsize (640, 640), loss: 0.7061, lbox: 0.1507, lobj: 0.1180, lcls: 0.4374, is_group_lr  , cur_lr: 0.009639999829232693
2024-04-07 07:14:23,944 [INFO] Epoch 13/300, Step 100/101, step time: 890.73 ms
2024-04-07 07:14:24,742 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-13_101.ckpt
2024-04-07 07:14:24,743 [INFO] Epoch 13/300, epoch time: 1.50 min.
2024-04-07 07:15:55,605 [INFO] Epoch 14/300, Step 100/101, imgsize (640, 640), loss: 0.6863, lbox: 0.1518, lobj: 0.0980, lcls: 0.4365, is_group_lr  , cur_lr: 0.009610000066459179
2024-04-07 07:15:55,607 [INFO] Epoch 14/300, Step 100/101, step time: 908.64 ms
2024-04-07 07:15:56,599 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-14_101.ckpt
2024-04-07 07:15:56,599 [INFO] Epoch 14/300, epoch time: 1.53 min.
2024-04-07 07:17:26,110 [INFO] Epoch 15/300, Step 100/101, imgsize (640, 640), loss: 0.6954, lbox: 0.1504, lobj: 0.1081, lcls: 0.4369, is_group_lr  , cur_lr: 0.009580000303685665
2024-04-07 07:17:26,112 [INFO] Epoch 15/300, Step 100/101, step time: 894.97 ms
2024-04-07 07:17:26,940 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-15_101.ckpt
2024-04-07 07:17:26,940 [INFO] Epoch 15/300, epoch time: 1.51 min.
2024-04-07 07:18:54,888 [INFO] Epoch 16/300, Step 100/101, imgsize (640, 640), loss: 0.7003, lbox: 0.1516, lobj: 0.1127, lcls: 0.4360, is_group_lr  , cur_lr: 0.009549999609589577
2024-04-07 07:18:54,889 [INFO] Epoch 16/300, Step 100/101, step time: 879.49 ms
2024-04-07 07:18:55,985 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-16_101.ckpt
2024-04-07 07:18:55,985 [INFO] Epoch 16/300, epoch time: 1.48 min.
2024-04-07 07:20:24,867 [INFO] Epoch 17/300, Step 100/101, imgsize (640, 640), loss: 0.6888, lbox: 0.1515, lobj: 0.1020, lcls: 0.4354, is_group_lr  , cur_lr: 0.009519999846816063
2024-04-07 07:20:24,869 [INFO] Epoch 17/300, Step 100/101, step time: 888.83 ms
2024-04-07 07:20:25,817 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-17_101.ckpt
2024-04-07 07:20:25,818 [INFO] Epoch 17/300, epoch time: 1.50 min.
2024-04-07 07:21:54,060 [INFO] Epoch 18/300, Step 100/101, imgsize (640, 640), loss: 0.6966, lbox: 0.1519, lobj: 0.1099, lcls: 0.4349, is_group_lr  , cur_lr: 0.00949000008404255
2024-04-07 07:21:54,066 [INFO] Epoch 18/300, Step 100/101, step time: 882.48 ms
2024-04-07 07:21:54,941 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-18_101.ckpt
2024-04-07 07:21:54,942 [INFO] Epoch 18/300, epoch time: 1.49 min.
2024-04-07 07:23:25,204 [INFO] Epoch 19/300, Step 100/101, imgsize (640, 640), loss: 0.6930, lbox: 0.1511, lobj: 0.1065, lcls: 0.4354, is_group_lr  , cur_lr: 0.009460000321269035
2024-04-07 07:23:25,206 [INFO] Epoch 19/300, Step 100/101, step time: 902.64 ms
2024-04-07 07:23:26,142 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-19_101.ckpt
2024-04-07 07:23:26,142 [INFO] Epoch 19/300, epoch time: 1.52 min.
2024-04-07 07:24:56,520 [INFO] Epoch 20/300, Step 100/101, imgsize (640, 640), loss: 0.6985, lbox: 0.1507, lobj: 0.1130, lcls: 0.4349, is_group_lr  , cur_lr: 0.009429999627172947
2024-04-07 07:24:56,525 [INFO] Epoch 20/300, Step 100/101, step time: 903.82 ms
2024-04-07 07:24:57,456 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-20_101.ckpt
2024-04-07 07:24:57,457 [INFO] Epoch 20/300, epoch time: 1.52 min.
2024-04-07 07:26:27,775 [INFO] Epoch 21/300, Step 100/101, imgsize (640, 640), loss: 0.6962, lbox: 0.1507, lobj: 0.1116, lcls: 0.4338, is_group_lr  , cur_lr: 0.009399999864399433
2024-04-07 07:26:27,777 [INFO] Epoch 21/300, Step 100/101, step time: 903.19 ms
2024-04-07 07:26:28,683 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-21_101.ckpt
2024-04-07 07:26:28,684 [INFO] Epoch 21/300, epoch time: 1.52 min.
2024-04-07 07:27:58,161 [INFO] Epoch 22/300, Step 100/101, imgsize (640, 640), loss: 0.6921, lbox: 0.1505, lobj: 0.1075, lcls: 0.4340, is_group_lr  , cur_lr: 0.00937000010162592
2024-04-07 07:27:58,167 [INFO] Epoch 22/300, Step 100/101, step time: 894.83 ms
2024-04-07 07:27:59,300 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-22_101.ckpt
2024-04-07 07:27:59,300 [INFO] Epoch 22/300, epoch time: 1.51 min.
2024-04-07 07:29:28,303 [INFO] Epoch 23/300, Step 100/101, imgsize (640, 640), loss: 0.6943, lbox: 0.1509, lobj: 0.1101, lcls: 0.4334, is_group_lr  , cur_lr: 0.009340000338852406
2024-04-07 07:29:28,306 [INFO] Epoch 23/300, Step 100/101, step time: 890.05 ms
2024-04-07 07:29:29,152 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-23_101.ckpt
2024-04-07 07:29:29,153 [INFO] Epoch 23/300, epoch time: 1.50 min.
2024-04-07 07:30:59,926 [INFO] Epoch 24/300, Step 100/101, imgsize (640, 640), loss: 0.6889, lbox: 0.1511, lobj: 0.1039, lcls: 0.4339, is_group_lr  , cur_lr: 0.009309999644756317
2024-04-07 07:31:00,059 [INFO] Epoch 24/300, Step 100/101, step time: 909.04 ms
2024-04-07 07:31:00,849 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-24_101.ckpt
2024-04-07 07:31:00,849 [INFO] Epoch 24/300, epoch time: 1.53 min.
2024-04-07 07:32:30,450 [INFO] Epoch 25/300, Step 100/101, imgsize (640, 640), loss: 0.6983, lbox: 0.1504, lobj: 0.1148, lcls: 0.4331, is_group_lr  , cur_lr: 0.009279999881982803
2024-04-07 07:32:30,549 [INFO] Epoch 25/300, Step 100/101, step time: 896.98 ms
2024-04-07 07:32:31,322 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-25_101.ckpt
2024-04-07 07:32:31,322 [INFO] Epoch 25/300, epoch time: 1.51 min.
2024-04-07 07:34:00,766 [INFO] Epoch 26/300, Step 100/101, imgsize (640, 640), loss: 0.6983, lbox: 0.1507, lobj: 0.1140, lcls: 0.4336, is_group_lr  , cur_lr: 0.00925000011920929
2024-04-07 07:34:00,770 [INFO] Epoch 26/300, Step 100/101, step time: 894.47 ms
2024-04-07 07:34:01,724 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-26_101.ckpt
2024-04-07 07:34:01,724 [INFO] Epoch 26/300, epoch time: 1.51 min.
2024-04-07 07:35:29,713 [INFO] Epoch 27/300, Step 100/101, imgsize (640, 640), loss: 0.6931, lbox: 0.1499, lobj: 0.1103, lcls: 0.4329, is_group_lr  , cur_lr: 0.009220000356435776
2024-04-07 07:35:29,714 [INFO] Epoch 27/300, Step 100/101, step time: 879.90 ms
2024-04-07 07:35:30,824 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-27_101.ckpt
2024-04-07 07:35:30,828 [INFO] Epoch 27/300, epoch time: 1.49 min.
2024-04-07 07:36:58,688 [INFO] Epoch 28/300, Step 100/101, imgsize (640, 640), loss: 0.6985, lbox: 0.1500, lobj: 0.1153, lcls: 0.4332, is_group_lr  , cur_lr: 0.009189999662339687
2024-04-07 07:36:58,690 [INFO] Epoch 28/300, Step 100/101, step time: 878.62 ms
2024-04-07 07:36:59,719 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-28_101.ckpt
2024-04-07 07:36:59,719 [INFO] Epoch 28/300, epoch time: 1.48 min.
2024-04-07 07:38:29,504 [INFO] Epoch 29/300, Step 100/101, imgsize (640, 640), loss: 0.6942, lbox: 0.1517, lobj: 0.1084, lcls: 0.4342, is_group_lr  , cur_lr: 0.009159999899566174
2024-04-07 07:38:29,508 [INFO] Epoch 29/300, Step 100/101, step time: 897.89 ms
2024-04-07 07:38:30,385 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-29_101.ckpt
2024-04-07 07:38:30,386 [INFO] Epoch 29/300, epoch time: 1.51 min.
2024-04-07 07:39:57,640 [INFO] Epoch 30/300, Step 100/101, imgsize (640, 640), loss: 0.6960, lbox: 0.1510, lobj: 0.1126, lcls: 0.4324, is_group_lr  , cur_lr: 0.00913000013679266
2024-04-07 07:39:57,642 [INFO] Epoch 30/300, Step 100/101, step time: 872.56 ms
2024-04-07 07:39:58,753 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-30_101.ckpt
2024-04-07 07:39:58,754 [INFO] Epoch 30/300, epoch time: 1.47 min.
2024-04-07 07:41:27,361 [INFO] Epoch 31/300, Step 100/101, imgsize (640, 640), loss: 0.6840, lbox: 0.1514, lobj: 0.0997, lcls: 0.4328, is_group_lr  , cur_lr: 0.009100000374019146
2024-04-07 07:41:27,367 [INFO] Epoch 31/300, Step 100/101, step time: 886.13 ms
2024-04-07 07:41:28,295 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-31_101.ckpt
2024-04-07 07:41:28,296 [INFO] Epoch 31/300, epoch time: 1.49 min.
2024-04-07 07:42:58,320 [INFO] Epoch 32/300, Step 100/101, imgsize (640, 640), loss: 0.7004, lbox: 0.1523, lobj: 0.1153, lcls: 0.4327, is_group_lr  , cur_lr: 0.009069999679923058
2024-04-07 07:42:58,324 [INFO] Epoch 32/300, Step 100/101, step time: 900.28 ms
2024-04-07 07:42:59,093 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-32_101.ckpt
2024-04-07 07:42:59,093 [INFO] Epoch 32/300, epoch time: 1.51 min.
2024-04-07 07:44:28,881 [INFO] Epoch 33/300, Step 100/101, imgsize (640, 640), loss: 0.6895, lbox: 0.1507, lobj: 0.1059, lcls: 0.4330, is_group_lr  , cur_lr: 0.009039999917149544
2024-04-07 07:44:28,884 [INFO] Epoch 33/300, Step 100/101, step time: 897.90 ms
2024-04-07 07:44:29,951 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-33_101.ckpt
2024-04-07 07:44:29,952 [INFO] Epoch 33/300, epoch time: 1.51 min.
2024-04-07 07:45:59,500 [INFO] Epoch 34/300, Step 100/101, imgsize (640, 640), loss: 0.6950, lbox: 0.1497, lobj: 0.1135, lcls: 0.4318, is_group_lr  , cur_lr: 0.00901000015437603
2024-04-07 07:45:59,502 [INFO] Epoch 34/300, Step 100/101, step time: 895.50 ms
2024-04-07 07:46:00,410 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-34_101.ckpt
2024-04-07 07:46:00,410 [INFO] Epoch 34/300, epoch time: 1.51 min.
2024-04-07 07:47:30,100 [INFO] Epoch 35/300, Step 100/101, imgsize (640, 640), loss: 0.6920, lbox: 0.1515, lobj: 0.1099, lcls: 0.4306, is_group_lr  , cur_lr: 0.008980000391602516
2024-04-07 07:47:30,101 [INFO] Epoch 35/300, Step 100/101, step time: 896.90 ms
2024-04-07 07:47:31,258 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-35_101.ckpt
2024-04-07 07:47:31,259 [INFO] Epoch 35/300, epoch time: 1.51 min.
2024-04-07 07:49:00,188 [INFO] Epoch 36/300, Step 100/101, imgsize (640, 640), loss: 0.6905, lbox: 0.1495, lobj: 0.1094, lcls: 0.4317, is_group_lr  , cur_lr: 0.008949999697506428
2024-04-07 07:49:00,192 [INFO] Epoch 36/300, Step 100/101, step time: 889.33 ms
2024-04-07 07:49:01,247 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-36_101.ckpt
2024-04-07 07:49:01,247 [INFO] Epoch 36/300, epoch time: 1.50 min.
2024-04-07 07:50:30,681 [INFO] Epoch 37/300, Step 100/101, imgsize (640, 640), loss: 0.7041, lbox: 0.1511, lobj: 0.1220, lcls: 0.4309, is_group_lr  , cur_lr: 0.008919999934732914
2024-04-07 07:50:30,686 [INFO] Epoch 37/300, Step 100/101, step time: 894.38 ms
2024-04-07 07:50:31,637 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-37_101.ckpt
2024-04-07 07:50:31,637 [INFO] Epoch 37/300, epoch time: 1.51 min.
2024-04-07 07:52:00,231 [INFO] Epoch 38/300, Step 100/101, imgsize (640, 640), loss: 0.6761, lbox: 0.1517, lobj: 0.0937, lcls: 0.4307, is_group_lr  , cur_lr: 0.0088900001719594
2024-04-07 07:52:00,235 [INFO] Epoch 38/300, Step 100/101, step time: 885.98 ms
2024-04-07 07:52:01,122 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-38_101.ckpt
2024-04-07 07:52:01,122 [INFO] Epoch 38/300, epoch time: 1.49 min.
2024-04-07 07:53:29,943 [INFO] Epoch 39/300, Step 100/101, imgsize (640, 640), loss: 0.6893, lbox: 0.1514, lobj: 0.1063, lcls: 0.4316, is_group_lr  , cur_lr: 0.008860000409185886
2024-04-07 07:53:29,945 [INFO] Epoch 39/300, Step 100/101, step time: 888.23 ms
2024-04-07 07:53:30,858 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-39_101.ckpt
2024-04-07 07:53:30,859 [INFO] Epoch 39/300, epoch time: 1.50 min.
2024-04-07 07:55:01,174 [INFO] Epoch 40/300, Step 100/101, imgsize (640, 640), loss: 0.6975, lbox: 0.1513, lobj: 0.1152, lcls: 0.4310, is_group_lr  , cur_lr: 0.008829999715089798
2024-04-07 07:55:01,180 [INFO] Epoch 40/300, Step 100/101, step time: 903.18 ms
2024-04-07 07:55:02,061 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-40_101.ckpt
2024-04-07 07:55:02,061 [INFO] Epoch 40/300, epoch time: 1.52 min.
2024-04-07 07:56:30,041 [INFO] Epoch 41/300, Step 100/101, imgsize (640, 640), loss: 0.7023, lbox: 0.1504, lobj: 0.1207, lcls: 0.4312, is_group_lr  , cur_lr: 0.008799999952316284
2024-04-07 07:56:30,044 [INFO] Epoch 41/300, Step 100/101, step time: 879.83 ms
2024-04-07 07:56:30,916 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-41_101.ckpt
2024-04-07 07:56:30,917 [INFO] Epoch 41/300, epoch time: 1.48 min.
2024-04-07 07:58:00,916 [INFO] Epoch 42/300, Step 100/101, imgsize (640, 640), loss: 0.6902, lbox: 0.1495, lobj: 0.1096, lcls: 0.4311, is_group_lr  , cur_lr: 0.00877000018954277
2024-04-07 07:58:00,919 [INFO] Epoch 42/300, Step 100/101, step time: 900.01 ms
2024-04-07 07:58:01,765 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-42_101.ckpt
2024-04-07 07:58:01,765 [INFO] Epoch 42/300, epoch time: 1.51 min.

WangFengtu1996 avatar Apr 07 '24 08:04 WangFengtu1996

这个是我的训练log， 这个loss 已经非nan了，也就没有出现全零了。 是的。这个是我这边训练的loss。

2024-04-07 06:56:19,709 [INFO] Epoch 1/300, Step 100/101, imgsize (640, 640), loss: 0.7016, lbox: 0.1512, lobj: 0.1086, lcls: 0.4418, is_group_lr  , cur_lr: 0.07029703259468079
2024-04-07 06:56:19,712 [INFO] Epoch 1/300, Step 100/101, step time: 1042.46 ms
2024-04-07 06:56:20,739 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-1_101.ckpt
2024-04-07 06:56:20,739 [INFO] Epoch 1/300, epoch time: 1.75 min.
2024-04-07 06:57:50,282 [INFO] Epoch 2/300, Step 100/101, imgsize (640, 640), loss: 0.7020, lbox: 0.1528, lobj: 0.1076, lcls: 0.4416, is_group_lr  , cur_lr: 0.04027712717652321
2024-04-07 06:57:50,288 [INFO] Epoch 2/300, Step 100/101, step time: 895.49 ms
2024-04-07 06:57:51,361 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-2_101.ckpt
2024-04-07 06:57:51,364 [INFO] Epoch 2/300, epoch time: 1.51 min.
2024-04-07 06:59:20,665 [INFO] Epoch 3/300, Step 100/101, imgsize (640, 640), loss: 0.7034, lbox: 0.1516, lobj: 0.1119, lcls: 0.4399, is_group_lr  , cur_lr: 0.010237228125333786
2024-04-07 06:59:20,669 [INFO] Epoch 3/300, Step 100/101, step time: 893.02 ms
2024-04-07 06:59:21,536 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-3_101.ckpt
2024-04-07 06:59:21,536 [INFO] Epoch 3/300, epoch time: 1.50 min.
2024-04-07 07:00:50,036 [INFO] Epoch 4/300, Step 100/101, imgsize (640, 640), loss: 0.6930, lbox: 0.1509, lobj: 0.1024, lcls: 0.4398, is_group_lr  , cur_lr: 0.009909999556839466
2024-04-07 07:00:50,053 [INFO] Epoch 4/300, Step 100/101, step time: 885.17 ms
2024-04-07 07:00:51,080 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-4_101.ckpt
2024-04-07 07:00:51,084 [INFO] Epoch 4/300, epoch time: 1.49 min.
2024-04-07 07:02:21,884 [INFO] Epoch 5/300, Step 100/101, imgsize (640, 640), loss: 0.7084, lbox: 0.1547, lobj: 0.1141, lcls: 0.4397, is_group_lr  , cur_lr: 0.009879999794065952
2024-04-07 07:02:22,001 [INFO] Epoch 5/300, Step 100/101, step time: 909.13 ms
2024-04-07 07:02:22,789 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-5_101.ckpt
2024-04-07 07:02:22,790 [INFO] Epoch 5/300, epoch time: 1.53 min.
2024-04-07 07:03:52,419 [INFO] Epoch 6/300, Step 100/101, imgsize (640, 640), loss: 0.7078, lbox: 0.1517, lobj: 0.1185, lcls: 0.4376, is_group_lr  , cur_lr: 0.009850000031292439
2024-04-07 07:03:52,422 [INFO] Epoch 6/300, Step 100/101, step time: 896.33 ms
2024-04-07 07:03:53,325 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-6_101.ckpt
2024-04-07 07:03:53,347 [INFO] Epoch 6/300, epoch time: 1.51 min.
2024-04-07 07:05:22,812 [INFO] Epoch 7/300, Step 100/101, imgsize (640, 640), loss: 0.6980, lbox: 0.1520, lobj: 0.1063, lcls: 0.4397, is_group_lr  , cur_lr: 0.009820000268518925
2024-04-07 07:05:22,814 [INFO] Epoch 7/300, Step 100/101, step time: 894.62 ms
2024-04-07 07:05:23,583 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-7_101.ckpt
2024-04-07 07:05:23,583 [INFO] Epoch 7/300, epoch time: 1.50 min.
2024-04-07 07:06:51,892 [INFO] Epoch 8/300, Step 100/101, imgsize (640, 640), loss: 0.6983, lbox: 0.1493, lobj: 0.1112, lcls: 0.4378, is_group_lr  , cur_lr: 0.009789999574422836
2024-04-07 07:06:51,896 [INFO] Epoch 8/300, Step 100/101, step time: 883.12 ms
2024-04-07 07:06:52,962 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-8_101.ckpt
2024-04-07 07:06:52,962 [INFO] Epoch 8/300, epoch time: 1.49 min.
2024-04-07 07:08:23,364 [INFO] Epoch 9/300, Step 100/101, imgsize (640, 640), loss: 0.7062, lbox: 0.1504, lobj: 0.1164, lcls: 0.4394, is_group_lr  , cur_lr: 0.009759999811649323
2024-04-07 07:08:23,365 [INFO] Epoch 9/300, Step 100/101, step time: 904.02 ms
2024-04-07 07:08:24,201 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-9_101.ckpt
2024-04-07 07:08:24,201 [INFO] Epoch 9/300, epoch time: 1.52 min.
2024-04-07 07:09:53,871 [INFO] Epoch 10/300, Step 100/101, imgsize (640, 640), loss: 0.7013, lbox: 0.1518, lobj: 0.1102, lcls: 0.4393, is_group_lr  , cur_lr: 0.009730000048875809
2024-04-07 07:09:53,874 [INFO] Epoch 10/300, Step 100/101, step time: 896.72 ms
2024-04-07 07:09:54,637 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-10_101.ckpt
2024-04-07 07:09:54,638 [INFO] Epoch 10/300, epoch time: 1.51 min.
2024-04-07 07:11:23,666 [INFO] Epoch 11/300, Step 100/101, imgsize (640, 640), loss: 0.6950, lbox: 0.1518, lobj: 0.1055, lcls: 0.4377, is_group_lr  , cur_lr: 0.009700000286102295
2024-04-07 07:11:23,668 [INFO] Epoch 11/300, Step 100/101, step time: 890.30 ms
2024-04-07 07:11:24,582 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-11_101.ckpt
2024-04-07 07:11:24,582 [INFO] Epoch 11/300, epoch time: 1.50 min.
2024-04-07 07:12:53,847 [INFO] Epoch 12/300, Step 100/101, imgsize (640, 640), loss: 0.7008, lbox: 0.1494, lobj: 0.1138, lcls: 0.4376, is_group_lr  , cur_lr: 0.009669999592006207
2024-04-07 07:12:53,848 [INFO] Epoch 12/300, Step 100/101, step time: 892.66 ms
2024-04-07 07:12:54,870 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-12_101.ckpt
2024-04-07 07:12:54,870 [INFO] Epoch 12/300, epoch time: 1.50 min.
2024-04-07 07:14:23,942 [INFO] Epoch 13/300, Step 100/101, imgsize (640, 640), loss: 0.7061, lbox: 0.1507, lobj: 0.1180, lcls: 0.4374, is_group_lr  , cur_lr: 0.009639999829232693
2024-04-07 07:14:23,944 [INFO] Epoch 13/300, Step 100/101, step time: 890.73 ms
2024-04-07 07:14:24,742 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-13_101.ckpt
2024-04-07 07:14:24,743 [INFO] Epoch 13/300, epoch time: 1.50 min.
2024-04-07 07:15:55,605 [INFO] Epoch 14/300, Step 100/101, imgsize (640, 640), loss: 0.6863, lbox: 0.1518, lobj: 0.0980, lcls: 0.4365, is_group_lr  , cur_lr: 0.009610000066459179
2024-04-07 07:15:55,607 [INFO] Epoch 14/300, Step 100/101, step time: 908.64 ms
2024-04-07 07:15:56,599 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-14_101.ckpt
2024-04-07 07:15:56,599 [INFO] Epoch 14/300, epoch time: 1.53 min.
2024-04-07 07:17:26,110 [INFO] Epoch 15/300, Step 100/101, imgsize (640, 640), loss: 0.6954, lbox: 0.1504, lobj: 0.1081, lcls: 0.4369, is_group_lr  , cur_lr: 0.009580000303685665
2024-04-07 07:17:26,112 [INFO] Epoch 15/300, Step 100/101, step time: 894.97 ms
2024-04-07 07:17:26,940 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-15_101.ckpt
2024-04-07 07:17:26,940 [INFO] Epoch 15/300, epoch time: 1.51 min.
2024-04-07 07:18:54,888 [INFO] Epoch 16/300, Step 100/101, imgsize (640, 640), loss: 0.7003, lbox: 0.1516, lobj: 0.1127, lcls: 0.4360, is_group_lr  , cur_lr: 0.009549999609589577
2024-04-07 07:18:54,889 [INFO] Epoch 16/300, Step 100/101, step time: 879.49 ms
2024-04-07 07:18:55,985 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-16_101.ckpt
2024-04-07 07:18:55,985 [INFO] Epoch 16/300, epoch time: 1.48 min.
2024-04-07 07:20:24,867 [INFO] Epoch 17/300, Step 100/101, imgsize (640, 640), loss: 0.6888, lbox: 0.1515, lobj: 0.1020, lcls: 0.4354, is_group_lr  , cur_lr: 0.009519999846816063
2024-04-07 07:20:24,869 [INFO] Epoch 17/300, Step 100/101, step time: 888.83 ms
2024-04-07 07:20:25,817 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-17_101.ckpt
2024-04-07 07:20:25,818 [INFO] Epoch 17/300, epoch time: 1.50 min.
2024-04-07 07:21:54,060 [INFO] Epoch 18/300, Step 100/101, imgsize (640, 640), loss: 0.6966, lbox: 0.1519, lobj: 0.1099, lcls: 0.4349, is_group_lr  , cur_lr: 0.00949000008404255
2024-04-07 07:21:54,066 [INFO] Epoch 18/300, Step 100/101, step time: 882.48 ms
2024-04-07 07:21:54,941 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-18_101.ckpt
2024-04-07 07:21:54,942 [INFO] Epoch 18/300, epoch time: 1.49 min.
2024-04-07 07:23:25,204 [INFO] Epoch 19/300, Step 100/101, imgsize (640, 640), loss: 0.6930, lbox: 0.1511, lobj: 0.1065, lcls: 0.4354, is_group_lr  , cur_lr: 0.009460000321269035
2024-04-07 07:23:25,206 [INFO] Epoch 19/300, Step 100/101, step time: 902.64 ms
2024-04-07 07:23:26,142 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-19_101.ckpt
2024-04-07 07:23:26,142 [INFO] Epoch 19/300, epoch time: 1.52 min.
2024-04-07 07:24:56,520 [INFO] Epoch 20/300, Step 100/101, imgsize (640, 640), loss: 0.6985, lbox: 0.1507, lobj: 0.1130, lcls: 0.4349, is_group_lr  , cur_lr: 0.009429999627172947
2024-04-07 07:24:56,525 [INFO] Epoch 20/300, Step 100/101, step time: 903.82 ms
2024-04-07 07:24:57,456 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-20_101.ckpt
2024-04-07 07:24:57,457 [INFO] Epoch 20/300, epoch time: 1.52 min.
2024-04-07 07:26:27,775 [INFO] Epoch 21/300, Step 100/101, imgsize (640, 640), loss: 0.6962, lbox: 0.1507, lobj: 0.1116, lcls: 0.4338, is_group_lr  , cur_lr: 0.009399999864399433
2024-04-07 07:26:27,777 [INFO] Epoch 21/300, Step 100/101, step time: 903.19 ms
2024-04-07 07:26:28,683 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-21_101.ckpt
2024-04-07 07:26:28,684 [INFO] Epoch 21/300, epoch time: 1.52 min.
2024-04-07 07:27:58,161 [INFO] Epoch 22/300, Step 100/101, imgsize (640, 640), loss: 0.6921, lbox: 0.1505, lobj: 0.1075, lcls: 0.4340, is_group_lr  , cur_lr: 0.00937000010162592
2024-04-07 07:27:58,167 [INFO] Epoch 22/300, Step 100/101, step time: 894.83 ms
2024-04-07 07:27:59,300 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-22_101.ckpt
2024-04-07 07:27:59,300 [INFO] Epoch 22/300, epoch time: 1.51 min.
2024-04-07 07:29:28,303 [INFO] Epoch 23/300, Step 100/101, imgsize (640, 640), loss: 0.6943, lbox: 0.1509, lobj: 0.1101, lcls: 0.4334, is_group_lr  , cur_lr: 0.009340000338852406
2024-04-07 07:29:28,306 [INFO] Epoch 23/300, Step 100/101, step time: 890.05 ms
2024-04-07 07:29:29,152 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-23_101.ckpt
2024-04-07 07:29:29,153 [INFO] Epoch 23/300, epoch time: 1.50 min.
2024-04-07 07:30:59,926 [INFO] Epoch 24/300, Step 100/101, imgsize (640, 640), loss: 0.6889, lbox: 0.1511, lobj: 0.1039, lcls: 0.4339, is_group_lr  , cur_lr: 0.009309999644756317
2024-04-07 07:31:00,059 [INFO] Epoch 24/300, Step 100/101, step time: 909.04 ms
2024-04-07 07:31:00,849 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-24_101.ckpt
2024-04-07 07:31:00,849 [INFO] Epoch 24/300, epoch time: 1.53 min.
2024-04-07 07:32:30,450 [INFO] Epoch 25/300, Step 100/101, imgsize (640, 640), loss: 0.6983, lbox: 0.1504, lobj: 0.1148, lcls: 0.4331, is_group_lr  , cur_lr: 0.009279999881982803
2024-04-07 07:32:30,549 [INFO] Epoch 25/300, Step 100/101, step time: 896.98 ms
2024-04-07 07:32:31,322 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-25_101.ckpt
2024-04-07 07:32:31,322 [INFO] Epoch 25/300, epoch time: 1.51 min.
2024-04-07 07:34:00,766 [INFO] Epoch 26/300, Step 100/101, imgsize (640, 640), loss: 0.6983, lbox: 0.1507, lobj: 0.1140, lcls: 0.4336, is_group_lr  , cur_lr: 0.00925000011920929
2024-04-07 07:34:00,770 [INFO] Epoch 26/300, Step 100/101, step time: 894.47 ms
2024-04-07 07:34:01,724 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-26_101.ckpt
2024-04-07 07:34:01,724 [INFO] Epoch 26/300, epoch time: 1.51 min.
2024-04-07 07:35:29,713 [INFO] Epoch 27/300, Step 100/101, imgsize (640, 640), loss: 0.6931, lbox: 0.1499, lobj: 0.1103, lcls: 0.4329, is_group_lr  , cur_lr: 0.009220000356435776
2024-04-07 07:35:29,714 [INFO] Epoch 27/300, Step 100/101, step time: 879.90 ms
2024-04-07 07:35:30,824 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-27_101.ckpt
2024-04-07 07:35:30,828 [INFO] Epoch 27/300, epoch time: 1.49 min.
2024-04-07 07:36:58,688 [INFO] Epoch 28/300, Step 100/101, imgsize (640, 640), loss: 0.6985, lbox: 0.1500, lobj: 0.1153, lcls: 0.4332, is_group_lr  , cur_lr: 0.009189999662339687
2024-04-07 07:36:58,690 [INFO] Epoch 28/300, Step 100/101, step time: 878.62 ms
2024-04-07 07:36:59,719 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-28_101.ckpt
2024-04-07 07:36:59,719 [INFO] Epoch 28/300, epoch time: 1.48 min.
2024-04-07 07:38:29,504 [INFO] Epoch 29/300, Step 100/101, imgsize (640, 640), loss: 0.6942, lbox: 0.1517, lobj: 0.1084, lcls: 0.4342, is_group_lr  , cur_lr: 0.009159999899566174
2024-04-07 07:38:29,508 [INFO] Epoch 29/300, Step 100/101, step time: 897.89 ms
2024-04-07 07:38:30,385 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-29_101.ckpt
2024-04-07 07:38:30,386 [INFO] Epoch 29/300, epoch time: 1.51 min.
2024-04-07 07:39:57,640 [INFO] Epoch 30/300, Step 100/101, imgsize (640, 640), loss: 0.6960, lbox: 0.1510, lobj: 0.1126, lcls: 0.4324, is_group_lr  , cur_lr: 0.00913000013679266
2024-04-07 07:39:57,642 [INFO] Epoch 30/300, Step 100/101, step time: 872.56 ms
2024-04-07 07:39:58,753 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-30_101.ckpt
2024-04-07 07:39:58,754 [INFO] Epoch 30/300, epoch time: 1.47 min.
2024-04-07 07:41:27,361 [INFO] Epoch 31/300, Step 100/101, imgsize (640, 640), loss: 0.6840, lbox: 0.1514, lobj: 0.0997, lcls: 0.4328, is_group_lr  , cur_lr: 0.009100000374019146
2024-04-07 07:41:27,367 [INFO] Epoch 31/300, Step 100/101, step time: 886.13 ms
2024-04-07 07:41:28,295 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-31_101.ckpt
2024-04-07 07:41:28,296 [INFO] Epoch 31/300, epoch time: 1.49 min.
2024-04-07 07:42:58,320 [INFO] Epoch 32/300, Step 100/101, imgsize (640, 640), loss: 0.7004, lbox: 0.1523, lobj: 0.1153, lcls: 0.4327, is_group_lr  , cur_lr: 0.009069999679923058
2024-04-07 07:42:58,324 [INFO] Epoch 32/300, Step 100/101, step time: 900.28 ms
2024-04-07 07:42:59,093 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-32_101.ckpt
2024-04-07 07:42:59,093 [INFO] Epoch 32/300, epoch time: 1.51 min.
2024-04-07 07:44:28,881 [INFO] Epoch 33/300, Step 100/101, imgsize (640, 640), loss: 0.6895, lbox: 0.1507, lobj: 0.1059, lcls: 0.4330, is_group_lr  , cur_lr: 0.009039999917149544
2024-04-07 07:44:28,884 [INFO] Epoch 33/300, Step 100/101, step time: 897.90 ms
2024-04-07 07:44:29,951 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-33_101.ckpt
2024-04-07 07:44:29,952 [INFO] Epoch 33/300, epoch time: 1.51 min.
2024-04-07 07:45:59,500 [INFO] Epoch 34/300, Step 100/101, imgsize (640, 640), loss: 0.6950, lbox: 0.1497, lobj: 0.1135, lcls: 0.4318, is_group_lr  , cur_lr: 0.00901000015437603
2024-04-07 07:45:59,502 [INFO] Epoch 34/300, Step 100/101, step time: 895.50 ms
2024-04-07 07:46:00,410 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-34_101.ckpt
2024-04-07 07:46:00,410 [INFO] Epoch 34/300, epoch time: 1.51 min.
2024-04-07 07:47:30,100 [INFO] Epoch 35/300, Step 100/101, imgsize (640, 640), loss: 0.6920, lbox: 0.1515, lobj: 0.1099, lcls: 0.4306, is_group_lr  , cur_lr: 0.008980000391602516
2024-04-07 07:47:30,101 [INFO] Epoch 35/300, Step 100/101, step time: 896.90 ms
2024-04-07 07:47:31,258 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-35_101.ckpt
2024-04-07 07:47:31,259 [INFO] Epoch 35/300, epoch time: 1.51 min.
2024-04-07 07:49:00,188 [INFO] Epoch 36/300, Step 100/101, imgsize (640, 640), loss: 0.6905, lbox: 0.1495, lobj: 0.1094, lcls: 0.4317, is_group_lr  , cur_lr: 0.008949999697506428
2024-04-07 07:49:00,192 [INFO] Epoch 36/300, Step 100/101, step time: 889.33 ms
2024-04-07 07:49:01,247 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-36_101.ckpt
2024-04-07 07:49:01,247 [INFO] Epoch 36/300, epoch time: 1.50 min.
2024-04-07 07:50:30,681 [INFO] Epoch 37/300, Step 100/101, imgsize (640, 640), loss: 0.7041, lbox: 0.1511, lobj: 0.1220, lcls: 0.4309, is_group_lr  , cur_lr: 0.008919999934732914
2024-04-07 07:50:30,686 [INFO] Epoch 37/300, Step 100/101, step time: 894.38 ms
2024-04-07 07:50:31,637 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-37_101.ckpt
2024-04-07 07:50:31,637 [INFO] Epoch 37/300, epoch time: 1.51 min.
2024-04-07 07:52:00,231 [INFO] Epoch 38/300, Step 100/101, imgsize (640, 640), loss: 0.6761, lbox: 0.1517, lobj: 0.0937, lcls: 0.4307, is_group_lr  , cur_lr: 0.0088900001719594
2024-04-07 07:52:00,235 [INFO] Epoch 38/300, Step 100/101, step time: 885.98 ms
2024-04-07 07:52:01,122 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-38_101.ckpt
2024-04-07 07:52:01,122 [INFO] Epoch 38/300, epoch time: 1.49 min.
2024-04-07 07:53:29,943 [INFO] Epoch 39/300, Step 100/101, imgsize (640, 640), loss: 0.6893, lbox: 0.1514, lobj: 0.1063, lcls: 0.4316, is_group_lr  , cur_lr: 0.008860000409185886
2024-04-07 07:53:29,945 [INFO] Epoch 39/300, Step 100/101, step time: 888.23 ms
2024-04-07 07:53:30,858 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-39_101.ckpt
2024-04-07 07:53:30,859 [INFO] Epoch 39/300, epoch time: 1.50 min.
2024-04-07 07:55:01,174 [INFO] Epoch 40/300, Step 100/101, imgsize (640, 640), loss: 0.6975, lbox: 0.1513, lobj: 0.1152, lcls: 0.4310, is_group_lr  , cur_lr: 0.008829999715089798
2024-04-07 07:55:01,180 [INFO] Epoch 40/300, Step 100/101, step time: 903.18 ms
2024-04-07 07:55:02,061 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-40_101.ckpt
2024-04-07 07:55:02,061 [INFO] Epoch 40/300, epoch time: 1.52 min.
2024-04-07 07:56:30,041 [INFO] Epoch 41/300, Step 100/101, imgsize (640, 640), loss: 0.7023, lbox: 0.1504, lobj: 0.1207, lcls: 0.4312, is_group_lr  , cur_lr: 0.008799999952316284
2024-04-07 07:56:30,044 [INFO] Epoch 41/300, Step 100/101, step time: 879.83 ms
2024-04-07 07:56:30,916 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-41_101.ckpt
2024-04-07 07:56:30,917 [INFO] Epoch 41/300, epoch time: 1.48 min.
2024-04-07 07:58:00,916 [INFO] Epoch 42/300, Step 100/101, imgsize (640, 640), loss: 0.6902, lbox: 0.1495, lobj: 0.1096, lcls: 0.4311, is_group_lr  , cur_lr: 0.00877000018954277
2024-04-07 07:58:00,919 [INFO] Epoch 42/300, Step 100/101, step time: 900.01 ms
2024-04-07 07:58:01,765 [INFO] Saving model to ./runs/2024.04.07-06.54.32/weights/yolov5-s-visdrone-42_101.ckpt
2024-04-07 07:58:01,765 [INFO] Epoch 42/300, epoch time: 1.51 min.

如果loss是正常的，可以尝试缩小数据集，构造过拟合实验看下模型是不是能正常学到这个新数据集的样本数据

zhanghuiyao avatar Apr 07 '24 08:04 zhanghuiyao

  • 选取其中128张作为这个数据集，目前，现在还是loss不收敛(150epochs). image

  • 接下来，需要什么样的工作，指导一下？多谢

log (2).txt

WangFengtu1996 avatar Apr 07 '24 09:04 WangFengtu1996

@zhanghuiyao 关于 yolov5 中，没张图片的标注的框的数量，填充到160 的原因是什么？ 是问了加快运算速度么？

WangFengtu1996 avatar Apr 07 '24 10:04 WangFengtu1996

  • 选取其中128张作为这个数据集，目前，现在还是loss不收敛(150epochs). image
  • 接下来，需要什么样的工作，指导一下？多谢

log (2).txt

建议其中一个思路可以尝试换成1张图片不做数据增强过拟合查看loss和结果，确认模型是否可以正常学习，另外可以再调调超参

zhanghuiyao avatar Apr 08 '24 03:04 zhanghuiyao

@zhanghuiyao 关于 yolov5 中，没张图片的标注的框的数量，填充到160 的原因是什么？ 是问了加快运算速度么？

MindSpore 2.1 里面对动态shape支持可能不那么完善 这里是为了保持shape一致

zhanghuiyao avatar Apr 08 '24 03:04 zhanghuiyao

  • 使用动态图与静态图 对于超参是否存在影响？
  • 下面代码中的 ops.stop_gradient 与has_aux=True 是否存在冗余 https://github.com/mindspore-lab/mindyolo/blob/8228636c35b52d571d63713af7f2eb199e2698c1/mindyolo/utils/train_step_factory.py#L49

https://github.com/mindspore-lab/mindyolo/blob/8228636c35b52d571d63713af7f2eb199e2698c1/mindyolo/utils/train_step_factory.py#L51

WangFengtu1996 avatar Apr 09 '24 07:04 WangFengtu1996

@zhanghuiyao hello 我调了一顿学习率，和这个优化器哈，128张图片我这边loss 下降不下去？ 我这现在啥状态, 还有啥改进的地方么? 下面是我的一些记录 https://kj-smart.feishu.cn/sheets/Eeo1szYKJh0t9htgv1mcW9A2nFf?from=from_copylink

WangFengtu1996 avatar Apr 10 '24 10:04 WangFengtu1996

@zhanghuiyao

  1. 通过 nn.Cell.trainable_params() 与 value_and_grad 返回的tensor 以及 梯度， 顺序是对应的么？
  2. 我这边尝试将这个梯度打印出来, 我这边发现里面存在这个 梯度 L2 存在 None 值
'model.model.0.conv.weight: 28.409107'
'model.model.0.bn.gamma: 25.16164'
'model.model.0.bn.beta: 52.067745'
'model.model.1.conv.weight: 25.011356'
'model.model.1.bn.gamma: 55.50659'
'model.model.1.bn.beta: 35.82057'
'model.model.2.conv1.conv.weight: 19.484802'
'model.model.2.conv1.bn.gamma: 59.45546'
'model.model.2.conv1.bn.beta: 59.131954'
'model.model.2.conv2.conv.weight: 33.815678'
'model.model.2.conv2.bn.gamma: 93.1563'
'model.model.2.conv2.bn.beta: 32.32318'
'model.model.2.conv3.conv.weight: 61.322353'
'model.model.2.conv3.bn.gamma: 50.653046'
'model.model.2.conv3.bn.beta: 41.558666'
'model.model.2.m.0.conv1.conv.weight: 73.942024'
'model.model.2.m.0.conv1.bn.gamma: 84.80962'
'model.model.2.m.0.conv1.bn.beta: 34.03752'
'model.model.2.m.0.conv2.conv.weight: 74.77053'
'model.model.2.m.0.conv2.bn.gamma: 35.539707'
'model.model.2.m.0.conv2.bn.beta: 70.96884'
'model.model.3.conv.weight: 39.5306'
'model.model.3.bn.gamma: 37.03664'
'model.model.3.bn.beta: 38.14776'
'model.model.4.conv1.conv.weight: 27.400778'
'model.model.4.conv1.bn.gamma: 56.76263'
'model.model.4.conv1.bn.beta: 81.48478'
'model.model.4.conv2.conv.weight: 19.893522'
'model.model.4.conv2.bn.gamma: 42.75126'
'model.model.4.conv2.bn.beta: 31.2061'
'model.model.4.conv3.conv.weight: 31.563835'
'model.model.4.conv3.bn.gamma: 13.421359'
'model.model.4.conv3.bn.beta: 60.414085'
'model.model.4.m.0.conv1.conv.weight: 52.073772'
'model.model.4.m.0.conv1.bn.gamma: 36.193817'
'model.model.4.m.0.conv1.bn.beta: 52.057217'
'model.model.4.m.0.conv2.conv.weight: 77.50617'
'model.model.4.m.0.conv2.bn.gamma: 34.686867'
'model.model.4.m.0.conv2.bn.beta: 38.52361'
'model.model.4.m.1.conv1.conv.weight: 67.78816'
'model.model.4.m.1.conv1.bn.gamma: 63.454662'
'model.model.4.m.1.conv1.bn.beta: 64.39841'
'model.model.4.m.1.conv2.conv.weight: 114.59657'
'model.model.4.m.1.conv2.bn.gamma: 69.78127'
'model.model.4.m.1.conv2.bn.beta: 89.10402'
'model.model.5.conv.weight: 24.393505'
'model.model.5.bn.gamma: 21.33894'
'model.model.5.bn.beta: 36.423946'
'model.model.6.conv1.conv.weight: 89.5058'
'model.model.6.conv1.bn.gamma: 40.08446'
'model.model.6.conv1.bn.beta: 36.509342'
'model.model.6.conv2.conv.weight: 24.525108'
'model.model.6.conv2.bn.gamma: 15.451342'
'model.model.6.conv2.bn.beta: 23.851812'
'model.model.6.conv3.conv.weight: 123.799065'
'model.model.6.conv3.bn.gamma: 16.519405'
'model.model.6.conv3.bn.beta: 21.27131'
'model.model.6.m.0.conv1.conv.weight: 380.0305'
'model.model.6.m.0.conv1.bn.gamma: 338.2177'
'model.model.6.m.0.conv1.bn.beta: 414.5485'
'model.model.6.m.0.conv2.conv.weight: None'
'model.model.6.m.0.conv2.bn.gamma: None'
'model.model.6.m.0.conv2.bn.beta: None'
'model.model.6.m.1.conv1.conv.weight: None'
'model.model.6.m.1.conv1.bn.gamma: None'
'model.model.6.m.1.conv1.bn.beta: None'
'model.model.6.m.1.conv2.conv.weight: None'
'model.model.6.m.1.conv2.bn.gamma: None'
'model.model.6.m.1.conv2.bn.beta: None'
'model.model.6.m.2.conv1.conv.weight: None'
'model.model.6.m.2.conv1.bn.gamma: None'
'model.model.6.m.2.conv1.bn.beta: None'
'model.model.6.m.2.conv2.conv.weight: None'
'model.model.6.m.2.conv2.bn.gamma: None'
'model.model.6.m.2.conv2.bn.beta: None'
'model.model.7.conv.weight: None'
'model.model.7.bn.gamma: None'
'model.model.7.bn.beta: None'
'model.model.8.conv1.conv.weight: None'
'model.model.8.conv1.bn.gamma: None'
'model.model.8.conv1.bn.beta: None'
'model.model.8.conv2.conv.weight: None'
'model.model.8.conv2.bn.gamma: None'
'model.model.8.conv2.bn.beta: None'
'model.model.8.conv3.conv.weight: None'
'model.model.8.conv3.bn.gamma: None'
'model.model.8.conv3.bn.beta: None'
'model.model.8.m.0.conv1.conv.weight: None'
'model.model.8.m.0.conv1.bn.gamma: None'
'model.model.8.m.0.conv1.bn.beta: None'
'model.model.8.m.0.conv2.conv.weight: None'
'model.model.8.m.0.conv2.bn.gamma: None'
'model.model.8.m.0.conv2.bn.beta: None'
'model.model.9.conv1.conv.weight: None'
'model.model.9.conv1.bn.gamma: None'
'model.model.9.conv1.bn.beta: None'
'model.model.9.conv2.conv.weight: None'
'model.model.9.conv2.bn.gamma: None'
'model.model.9.conv2.bn.beta: None'
'model.model.10.conv.weight: None'
'model.model.10.bn.gamma: None'
'model.model.10.bn.beta: None'
'model.model.13.conv1.conv.weight: None'
'model.model.13.conv1.bn.gamma: None'
'model.model.13.conv1.bn.beta: None'
'model.model.13.conv2.conv.weight: None'
'model.model.13.conv2.bn.gamma: None'
'model.model.13.conv2.bn.beta: None'
'model.model.13.conv3.conv.weight: None'
'model.model.13.conv3.bn.gamma: None'
'model.model.13.conv3.bn.beta: None'
'model.model.13.m.0.conv1.conv.weight: None'
'model.model.13.m.0.conv1.bn.gamma: None'
'model.model.13.m.0.conv1.bn.beta: None'
'model.model.13.m.0.conv2.conv.weight: None'
'model.model.13.m.0.conv2.bn.gamma: None'
'model.model.13.m.0.conv2.bn.beta: None'
'model.model.14.conv.weight: None'
'model.model.14.bn.gamma: None'
'model.model.14.bn.beta: None'
'model.model.17.conv1.conv.weight: 79.45166'
'model.model.17.conv1.bn.gamma: 38.256844'
'model.model.17.conv1.bn.beta: 118.1539'
'model.model.17.conv2.conv.weight: 64.416306'
'model.model.17.conv2.bn.gamma: 85.62841'
'model.model.17.conv2.bn.beta: 71.489876'
'model.model.17.conv3.conv.weight: 59.796646'
'model.model.17.conv3.bn.gamma: 151.73509'
'model.model.17.conv3.bn.beta: 171.28897'
'model.model.17.m.0.conv1.conv.weight: 92.80423'
'model.model.17.m.0.conv1.bn.gamma: 211.45561'
'model.model.17.m.0.conv1.bn.beta: 67.332794'
'model.model.17.m.0.conv2.conv.weight: 185.01366'
'model.model.17.m.0.conv2.bn.gamma: 105.695786'
'model.model.17.m.0.conv2.bn.beta: 84.220436'
'model.model.18.conv.weight: 145.49171'
'model.model.18.bn.gamma: 238.93004'
'model.model.18.bn.beta: 64.76699'
'model.model.20.conv1.conv.weight: 155.5591'
'model.model.20.conv1.bn.gamma: 56.384068'
'model.model.20.conv1.bn.beta: 139.33545'
'model.model.20.conv2.conv.weight: 72.94961'
'model.model.20.conv2.bn.gamma: 78.70151'
'model.model.20.conv2.bn.beta: 66.37617'
'model.model.20.conv3.conv.weight: 42.435318'
'model.model.20.conv3.bn.gamma: 98.58793'
'model.model.20.conv3.bn.beta: 160.712'
'model.model.20.m.0.conv1.conv.weight: 40.51687'
'model.model.20.m.0.conv1.bn.gamma: 70.46126'
'model.model.20.m.0.conv1.bn.beta: 50.581604'
'model.model.20.m.0.conv2.conv.weight: 47.56846'
'model.model.20.m.0.conv2.bn.gamma: 86.207924'
'model.model.20.m.0.conv2.bn.beta: 91.0685'
'model.model.21.conv.weight: 87.84189'
'model.model.21.bn.gamma: 77.00267'
'model.model.21.bn.beta: 97.46887'
'model.model.23.conv1.conv.weight: 125.83838'
'model.model.23.conv1.bn.gamma: 72.54489'
'model.model.23.conv1.bn.beta: 86.57055'
'model.model.23.conv2.conv.weight: 138.32869'
'model.model.23.conv2.bn.gamma: 206.33005'
'model.model.23.conv2.bn.beta: 172.28896'
'model.model.23.conv3.conv.weight: 196.95154'
'model.model.23.conv3.bn.gamma: 186.8771'
'model.model.23.conv3.bn.beta: 136.93318'
'model.model.23.m.0.conv1.conv.weight: 49.525772'
'model.model.23.m.0.conv1.bn.gamma: 36.685146'
'model.model.23.m.0.conv1.bn.beta: 79.369934'
'model.model.23.m.0.conv2.conv.weight: 106.352974'
'model.model.23.m.0.conv2.bn.gamma: 61.913635'
'model.model.23.m.0.conv2.bn.beta: 54.91199'
'model.model.24.m.0.weight: 38.514282'
'model.model.24.m.0.bias: 24.594584'
'model.model.24.m.1.weight: 36.383896'
'model.model.24.m.1.bias: 102.33092'
'model.model.24.m.2.weight: 25.18849'
'model.model.24.m.2.bias: 25.324461'

WangFengtu1996 avatar Apr 11 '24 08:04 WangFengtu1996

@zhanghuiyao hello 我调了一顿学习率，和这个优化器哈，128张图片我这边loss 下降不下去？ 我这现在啥状态, 还有啥改进的地方么? 下面是我的一些记录 https://kj-smart.feishu.cn/sheets/Eeo1szYKJh0t9htgv1mcW9A2nFf?from=from_copylink

表里看起来不同的学习率的差异还是比较大，可以尝试用 随机初始权重 和 coco pretrain的权重 分别跑一组实验看看效果

zhanghuiyao avatar Apr 17 '24 03:04 zhanghuiyao

@zhanghuiyao

  1. 通过 nn.Cell.trainable_params() 与 value_and_grad 返回的tensor 以及 梯度， 顺序是对应的么？
  2. 我这边尝试将这个梯度打印出来, 我这边发现里面存在这个 梯度 L2 存在 None 值

value_and_grad 返回的梯度应该和 optimizer.parameters 里面是对应的，grad为None的可能是对应的parameter的require_grad为False https://github.com/mindspore-lab/mindyolo/blob/master/mindyolo/utils/train_step_factory.py#L51

zhanghuiyao avatar Apr 17 '24 03:04 zhanghuiyao