DexiNed
DexiNed copied to clipboard
Published
20 hours ago •
xavysp
xavysp
Training loss does not converge
Hi, i have problem in training process of Pytorch version. When using hed_loss2/bdcn_loss/bdcn_loss2 in losses，training loss is large and does not converge，model can predict edge map but the result is not very good. But when i use weighted_cross_entropy_loss in losses where pos_weight should pass to F.binary_cross_entropy_with_logits as pos_weight rather than weight, loss could converge. when i pass pos_weight to F.binary_cross_entropy_with_logits as pos_weight, loss could not converge. What is the problem?
Hi, i have problem in training process of Pytorch version. When using hed_loss2/bdcn_loss/bdcn_loss2 in losses，training loss is large and does not converge，model can predict edge map but the result is not very good. But when i use weighted_cross_entropy_loss in losses where pos_weight should pass to F.binary_cross_entropy_with_logits as pos_weight rather than weight, loss could converge. when i pass pos_weight to F.binary_cross_entropy_with_logits as pos_weight, loss could not converge. What is the problem?
Hi @xiaozr could I see the graphs of each training? Since the quantitative results does not change, and following the previous works I did not pay much attention on that, because it converge, not near to zero but still converge.
Xavier
Indeed, the value of the loss function is large, and after 25 epochs of training, the loss has barely decreased
Thanks for your reply! Here is the training loss of weight_cross_entropy where pos_weight = beta/(1-beta) passing to F.binary_cross_entropy_with_logits as weight param，
Fri Apr 16 17:54:21 2021 Epoch: 0 Sample 15/20 Loss: 3484692.0
Fri Apr 16 17:54:33 2021 Epoch: 1 Sample 0/20 Loss: 3507439.0
Fri Apr 16 17:54:42 2021 Epoch: 1 Sample 5/20 Loss: 3476176.75
Fri Apr 16 17:54:52 2021 Epoch: 1 Sample 10/20 Loss: 3433098.75
Fri Apr 16 17:55:01 2021 Epoch: 1 Sample 15/20 Loss: 3403802.5
Fri Apr 16 17:55:13 2021 Epoch: 2 Sample 0/20 Loss: 3416200.0
Fri Apr 16 17:55:23 2021 Epoch: 2 Sample 5/20 Loss: 3418759.0
Fri Apr 16 17:55:32 2021 Epoch: 2 Sample 10/20 Loss: 3396917.5
Fri Apr 16 17:55:41 2021 Epoch: 2 Sample 15/20 Loss: 3348021.25
Fri Apr 16 17:55:54 2021 Epoch: 3 Sample 0/20 Loss: 3312441.0
Fri Apr 16 17:56:03 2021 Epoch: 3 Sample 5/20 Loss: 3347686.25
Fri Apr 16 17:56:13 2021 Epoch: 3 Sample 10/20 Loss: 3344589.0
Fri Apr 16 17:56:22 2021 Epoch: 3 Sample 15/20 Loss: 3305243.25
Fri Apr 16 17:56:34 2021 Epoch: 4 Sample 0/20 Loss: 3284784.0
Fri Apr 16 17:56:44 2021 Epoch: 4 Sample 5/20 Loss: 3276130.75
Fri Apr 16 17:56:53 2021 Epoch: 4 Sample 10/20 Loss: 3229647.25
Fri Apr 16 17:57:03 2021 Epoch: 4 Sample 15/20 Loss: 3247599.5
Fri Apr 16 17:57:15 2021 Epoch: 5 Sample 0/20 Loss: 3237678.5
Fri Apr 16 17:57:24 2021 Epoch: 5 Sample 5/20 Loss: 3198486.75
Fri Apr 16 17:57:34 2021 Epoch: 5 Sample 10/20 Loss: 3187369.0
Fri Apr 16 17:57:43 2021 Epoch: 5 Sample 15/20 Loss: 3164659.0
Fri Apr 16 17:57:55 2021 Epoch: 6 Sample 0/20 Loss: 3148827.75
Fri Apr 16 17:58:05 2021 Epoch: 6 Sample 5/20 Loss: 3136897.0
Fri Apr 16 17:58:14 2021 Epoch: 6 Sample 10/20 Loss: 3119184.0
Fri Apr 16 17:58:24 2021 Epoch: 6 Sample 15/20 Loss: 3107234.0
Fri Apr 16 17:58:36 2021 Epoch: 7 Sample 0/20 Loss: 3061088.5
Fri Apr 16 17:58:45 2021 Epoch: 7 Sample 5/20 Loss: 3064931.5
Fri Apr 16 17:58:55 2021 Epoch: 7 Sample 10/20 Loss: 3021220.5
Fri Apr 16 17:59:04 2021 Epoch: 7 Sample 15/20 Loss: 3016984.25
Fri Apr 16 17:59:17 2021 Epoch: 8 Sample 0/20 Loss: 2998871.0
Fri Apr 16 17:59:26 2021 Epoch: 8 Sample 5/20 Loss: 2974223.75
Fri Apr 16 17:59:35 2021 Epoch: 8 Sample 10/20 Loss: 2962066.25
Fri Apr 16 17:59:45 2021 Epoch: 8 Sample 15/20 Loss: 2934750.75
Fri Apr 16 17:59:57 2021 Epoch: 9 Sample 0/20 Loss: 2920681.0
Fri Apr 16 18:00:07 2021 Epoch: 9 Sample 5/20 Loss: 2908385.5
Fri Apr 16 18:00:16 2021 Epoch: 9 Sample 10/20 Loss: 2893500.5
Fri Apr 16 18:00:25 2021 Epoch: 9 Sample 15/20 Loss: 2874969.5
Fri Apr 16 18:00:38 2021 Epoch: 10 Sample 0/20 Loss: 2845558.0
Fri Apr 16 18:00:47 2021 Epoch: 10 Sample 5/20 Loss: 2832489.5
Fri Apr 16 18:00:57 2021 Epoch: 10 Sample 10/20 Loss: 2810812.75
Fri Apr 16 18:01:06 2021 Epoch: 10 Sample 15/20 Loss: 2793566.0
Fri Apr 16 18:01:18 2021 Epoch: 11 Sample 0/20 Loss: 2776006.25
Fri Apr 16 18:01:28 2021 Epoch: 11 Sample 5/20 Loss: 2753394.0
Fri Apr 16 18:01:37 2021 Epoch: 11 Sample 10/20 Loss: 2740514.25
Fri Apr 16 18:01:47 2021 Epoch: 11 Sample 15/20 Loss: 2715585.0
Fri Apr 16 18:01:59 2021 Epoch: 12 Sample 0/20 Loss: 2696917.25
Fri Apr 16 18:02:08 2021 Epoch: 12 Sample 5/20 Loss: 2672573.25
Fri Apr 16 18:02:18 2021 Epoch: 12 Sample 10/20 Loss: 2664077.5
Fri Apr 16 18:02:27 2021 Epoch: 12 Sample 15/20 Loss: 2634007.25
Fri Apr 16 18:02:39 2021 Epoch: 13 Sample 0/20 Loss: 2610438.5
Fri Apr 16 18:02:49 2021 Epoch: 13 Sample 5/20 Loss: 2615285.75
…..
Fri Apr 16 18:30:04 2021 Epoch: 53 Sample 10/20 Loss: 1522760.875
Fri Apr 16 18:30:14 2021 Epoch: 53 Sample 15/20 Loss: 1541034.875
Fri Apr 16 18:30:27 2021 Epoch: 54 Sample 0/20 Loss: 1568336.625
Fri Apr 16 18:30:36 2021 Epoch: 54 Sample 5/20 Loss: 1491055.125
Fri Apr 16 18:30:46 2021 Epoch: 54 Sample 10/20 Loss: 1461484.625
Fri Apr 16 18:30:55 2021 Epoch: 54 Sample 15/20 Loss: 1502256.125
Fri Apr 16 18:31:07 2021 Epoch: 55 Sample 0/20 Loss: 1528008.125
Fri Apr 16 18:31:17 2021 Epoch: 55 Sample 5/20 Loss: 1443430.0
Fri Apr 16 18:31:26 2021 Epoch: 55 Sample 10/20 Loss: 1621077.5
Fri Apr 16 18:31:36 2021 Epoch: 55 Sample 15/20 Loss: 1515412.5
Fri Apr 16 18:31:48 2021 Epoch: 56 Sample 0/20 Loss: 1441033.5
Fri Apr 16 18:31:58 2021 Epoch: 56 Sample 5/20 Loss: 1485266.0
Fri Apr 16 18:32:07 2021 Epoch: 56 Sample 10/20 Loss: 1466353.0
Fri Apr 16 18:32:16 2021 Epoch: 56 Sample 15/20 Loss: 1533353.0
Fri Apr 16 18:32:29 2021 Epoch: 57 Sample 0/20 Loss: 1425835.75
Fri Apr 16 18:32:38 2021 Epoch: 57 Sample 5/20 Loss: 1463604.5
Fri Apr 16 18:32:48 2021 Epoch: 57 Sample 10/20 Loss: 1497420.625
Fri Apr 16 18:32:57 2021 Epoch: 57 Sample 15/20 Loss: 1539935.875
Fri Apr 16 18:33:09 2021 Epoch: 58 Sample 0/20 Loss: 1448864.25
Fri Apr 16 18:33:19 2021 Epoch: 58 Sample 5/20 Loss: 1449366.0
Fri Apr 16 18:33:28 2021 Epoch: 58 Sample 10/20 Loss: 1358913.75
Fri Apr 16 18:33:38 2021 Epoch: 58 Sample 15/20 Loss: 1441650.625
Fri Apr 16 18:33:50 2021 Epoch: 59 Sample 0/20 Loss: 1514837.375
Fri Apr 16 18:33:59 2021 Epoch: 59 Sample 5/20 Loss: 1406704.25
Fri Apr 16 18:34:09 2021 Epoch: 59 Sample 10/20 Loss: 1430945.75
Fri Apr 16 18:34:18 2021 Epoch: 59 Sample 15/20 Loss: 1427504.25
Fri Apr 16 18:34:31 2021 Epoch: 60 Sample 0/20 Loss: 1452333.875
Fri Apr 16 18:34:40 2021 Epoch: 60 Sample 5/20 Loss: 1394662.25
Fri Apr 16 18:34:50 2021 Epoch: 60 Sample 10/20 Loss: 1464023.0
Fri Apr 16 18:34:59 2021 Epoch: 60 Sample 15/20 Loss: 1425483.625
And here is the training loss of hed_loss2，
Sat Apr 17 17:00:07 2021 Epoch: 0 Sample 0/20 Loss: 3690127.5
Sat Apr 17 17:00:17 2021 Epoch: 0 Sample 5/20 Loss: 2853033.75
Sat Apr 17 17:00:27 2021 Epoch: 0 Sample 10/20 Loss: 5095746.0
Sat Apr 17 17:00:37 2021 Epoch: 0 Sample 15/20 Loss: 4105941.0
Sat Apr 17 17:00:49 2021 Epoch: 1 Sample 0/20 Loss: 3193714.0
Sat Apr 17 17:00:59 2021 Epoch: 1 Sample 5/20 Loss: 4195813.5
Sat Apr 17 17:01:08 2021 Epoch: 1 Sample 10/20 Loss: 3855421.25
Sat Apr 17 17:01:18 2021 Epoch: 1 Sample 15/20 Loss: 4358175.0
Sat Apr 17 17:01:31 2021 Epoch: 2 Sample 0/20 Loss: 3867448.75
Sat Apr 17 17:01:40 2021 Epoch: 2 Sample 5/20 Loss: 4186718.75
Sat Apr 17 17:01:50 2021 Epoch: 2 Sample 10/20 Loss: 3772531.75
Sat Apr 17 17:02:00 2021 Epoch: 2 Sample 15/20 Loss: 4265946.0
Sat Apr 17 17:02:12 2021 Epoch: 3 Sample 0/20 Loss: 4002610.25
Sat Apr 17 17:02:22 2021 Epoch: 3 Sample 5/20 Loss: 4056114.75
Sat Apr 17 17:02:32 2021 Epoch: 3 Sample 10/20 Loss: 3727825.5
Sat Apr 17 17:02:41 2021 Epoch: 3 Sample 15/20 Loss: 3586216.75
Sat Apr 17 17:02:54 2021 Epoch: 4 Sample 0/20 Loss: 4335299.0
Sat Apr 17 17:03:04 2021 Epoch: 4 Sample 5/20 Loss: 3556628.25
Sat Apr 17 17:03:13 2021 Epoch: 4 Sample 10/20 Loss: 2945395.5
Sat Apr 17 17:03:23 2021 Epoch: 4 Sample 15/20 Loss: 4332093.0
Sat Apr 17 17:03:36 2021 Epoch: 5 Sample 0/20 Loss: 4248689.5
Sat Apr 17 17:03:45 2021 Epoch: 5 Sample 5/20 Loss: 3688312.5
Sat Apr 17 17:03:55 2021 Epoch: 5 Sample 10/20 Loss: 4320783.5
Sat Apr 17 17:04:05 2021 Epoch: 5 Sample 15/20 Loss: 5098256.5
Sat Apr 17 17:04:18 2021 Epoch: 6 Sample 0/20 Loss: 3876798.0
Sat Apr 17 17:04:27 2021 Epoch: 6 Sample 5/20 Loss: 3466649.5
Sat Apr 17 17:04:37 2021 Epoch: 6 Sample 10/20 Loss: 3483822.5
Sat Apr 17 17:04:47 2021 Epoch: 6 Sample 15/20 Loss: 4658633.5
Sat Apr 17 17:04:59 2021 Epoch: 7 Sample 0/20 Loss: 4217309.0
Sat Apr 17 17:05:09 2021 Epoch: 7 Sample 5/20 Loss: 3610711.75
Sat Apr 17 17:05:19 2021 Epoch: 7 Sample 10/20 Loss: 4094682.0
Sat Apr 17 17:05:28 2021 Epoch: 7 Sample 15/20 Loss: 3380293.0
Sat Apr 17 17:05:41 2021 Epoch: 8 Sample 0/20 Loss: 3841871.5
Sat Apr 17 17:05:51 2021 Epoch: 8 Sample 5/20 Loss: 4018657.5
Sat Apr 17 17:06:01 2021 Epoch: 8 Sample 10/20 Loss: 3675544.5
Sat Apr 17 17:06:10 2021 Epoch: 8 Sample 15/20 Loss: 3921705.75
Sat Apr 17 17:06:23 2021 Epoch: 9 Sample 0/20 Loss: 3376372.25
Sat Apr 17 17:06:33 2021 Epoch: 9 Sample 5/20 Loss: 3627306.0
Sat Apr 17 17:06:42 2021 Epoch: 9 Sample 10/20 Loss: 3312043.75
Sat Apr 17 17:06:52 2021 Epoch: 9 Sample 15/20 Loss: 3863103.0
Sat Apr 17 17:07:05 2021 Epoch: 10 Sample 0/20 Loss: 3315958.0
Sat Apr 17 17:07:14 2021 Epoch: 10 Sample 5/20 Loss: 4149814.75
Sat Apr 17 17:07:24 2021 Epoch: 10 Sample 10/20 Loss: 4267039.0
Sat Apr 17 17:07:34 2021 Epoch: 10 Sample 15/20 Loss: 3774072.5
Sat Apr 17 17:07:46 2021 Epoch: 11 Sample 0/20 Loss: 3625838.0
Sat Apr 17 17:07:56 2021 Epoch: 11 Sample 5/20 Loss: 3039867.0
Sat Apr 17 17:08:06 2021 Epoch: 11 Sample 10/20 Loss: 3045588.0
Sat Apr 17 17:08:16 2021 Epoch: 11 Sample 15/20 Loss: 3517692.75
Sat Apr 17 17:08:28 2021 Epoch: 12 Sample 0/20 Loss: 3343667.75
Sat Apr 17 17:08:38 2021 Epoch: 12 Sample 5/20 Loss: 3714051.5
Sat Apr 17 17:08:48 2021 Epoch: 12 Sample 10/20 Loss: 2952007.0
Sat Apr 17 17:08:57 2021 Epoch: 12 Sample 15/20 Loss: 4002049.0
Sat Apr 17 17:09:10 2021 Epoch: 13 Sample 0/20 Loss: 2692313.5
Sat Apr 17 17:09:20 2021 Epoch: 13 Sample 5/20 Loss: 3333743.25
Sat Apr 17 17:09:29 2021 Epoch: 13 Sample 10/20 Loss: 3646270.0
Sat Apr 17 17:09:39 2021 Epoch: 13 Sample 15/20 Loss: 3297957.75
…
Sat Apr 17 17:40:51 2021 Epoch: 58 Sample 10/20 Loss: 3831380.5
Sat Apr 17 17:41:01 2021 Epoch: 58 Sample 15/20 Loss: 2155563.75
Sat Apr 17 17:41:14 2021 Epoch: 59 Sample 0/20 Loss: 3212187.5
Sat Apr 17 17:41:23 2021 Epoch: 59 Sample 5/20 Loss: 3154757.0
Sat Apr 17 17:41:33 2021 Epoch: 59 Sample 10/20 Loss: 2515013.25
Sat Apr 17 17:41:43 2021 Epoch: 59 Sample 15/20 Loss: 3030167.5
Sat Apr 17 17:41:55 2021 Epoch: 60 Sample 0/20 Loss: 2730586.0
Sat Apr 17 17:42:05 2021 Epoch: 60 Sample 5/20 Loss: 2763092.5
Sat Apr 17 17:42:15 2021 Epoch: 60 Sample 10/20 Loss: 2375105.5
Sat Apr 17 17:42:25 2021 Epoch: 60 Sample 15/20 Loss: 2622136.75
Sat Apr 17 17:42:37 2021 Epoch: 61 Sample 0/20 Loss: 2397398.5
Sat Apr 17 17:42:47 2021 Epoch: 61 Sample 5/20 Loss: 2538762.0
Sat Apr 17 17:42:57 2021 Epoch: 61 Sample 10/20 Loss: 3410674.5
Sat Apr 17 17:43:06 2021 Epoch: 61 Sample 15/20 Loss: 2614472.0
Sat Apr 17 17:43:19 2021 Epoch: 62 Sample 0/20 Loss: 2580796.25
Sat Apr 17 17:43:29 2021 Epoch: 62 Sample 5/20 Loss: 2344824.75
Sat Apr 17 17:43:38 2021 Epoch: 62 Sample 10/20 Loss: 2768735.5
Sat Apr 17 17:43:48 2021 Epoch: 62 Sample 15/20 Loss: 3354243.0
Sat Apr 17 17:44:01 2021 Epoch: 63 Sample 0/20 Loss: 3354437.75
Sat Apr 17 17:44:11 2021 Epoch: 63 Sample 5/20 Loss: 3092955.75
The second does not seem to converge.
Have you solved this problem? I'm running without changing any code, my loss has been around 2, can you communicate if it's convenient for you
