hed icon indicating copy to clipboard operation
hed copied to clipboard

All of the Loss is zero

Open 09876qwrte opened this issue 8 years ago • 2 comments

when I train the hed,all of the loss is zero,the next is my log of train: I1204 13:19:24.325491 2461 solver.cpp:242] Iteration 8, loss = 0 I1204 13:19:24.325518 2461 solver.cpp:258] Train net output #0: dsn1_loss = 0 (* 1 = 0 loss) I1204 13:19:24.325541 2461 solver.cpp:258] Train net output #1: dsn2_loss = 0 (* 1 = 0 loss) I1204 13:19:24.325552 2461 solver.cpp:258] Train net output #2: dsn3_loss = 0 (* 1 = 0 loss) I1204 13:19:24.325563 2461 solver.cpp:258] Train net output #3: dsn4_loss = 0 (* 1 = 0 loss) I1204 13:19:24.325574 2461 solver.cpp:258] Train net output #4: dsn5_loss = 0 (* 1 = 0 loss) I1204 13:19:24.325584 2461 solver.cpp:258] Train net output #5: fuse_loss = 0 (* 1 = 0 loss) I1204 13:19:24.325598 2461 solver.cpp:571] Iteration 8, lr = 1e-14 I1204 13:19:24.401729 2461 net.cpp:754] [Update] Layer Edgeconv1_1, param 0 data: 0.145995; diff: 1.41403e-11 I1204 13:19:24.401800 2461 net.cpp:754] [Update] Layer Edgeconv1_1, param 1 data: 0.50245; diff: 0 I1204 13:19:24.401882 2461 net.cpp:754] [Update] Layer Edgeconv1_2, param 0 data: 0.026326; diff: 2.54981e-12 I1204 13:19:24.401904 2461 net.cpp:754] [Update] Layer Edgeconv1_2, param 1 data: 0.270118; diff: 0 I1204 13:19:24.402009 2461 net.cpp:754] [Update] Layer Edgeconv2_1, param 0 data: 0.0211515; diff: 2.04863e-12 I1204 13:19:24.402036 2461 net.cpp:754] [Update] Layer Edgeconv2_1, param 1 data: 0.133851; diff: 0 I1204 13:19:24.402231 2461 net.cpp:754] [Update] Layer Edgeconv2_2, param 0 data: 0.0164008; diff: 1.5885e-12 I1204 13:19:24.402281 2461 net.cpp:754] [Update] Layer Edgeconv2_2, param 1 data: 0.148366; diff: 0 I1204 13:19:24.402647 2461 net.cpp:754] [Update] Layer Edgeconv3_1, param 0 data: 0.0121919; diff: 1.18084e-12 I1204 13:19:24.402775 2461 net.cpp:754] [Update] Layer Edgeconv3_1, param 1 data: 0.0556366; diff: 0 I1204 13:19:24.403424 2461 net.cpp:754] [Update] Layer Edgeconv3_2, param 0 data: 0.00905248; diff: 8.76779e-13 I1204 13:19:24.403977 2461 net.cpp:754] [Update] Layer Edgeconv3_2, param 1 data: 0.0638749; diff: 0 I1204 13:19:24.404670 2461 net.cpp:754] [Update] Layer Edgeconv3_3, param 0 data: 0.00922374; diff: 8.93368e-13 I1204 13:19:24.405221 2461 net.cpp:754] [Update] Layer Edgeconv3_3, param 1 data: 0.0597807; diff: 0 I1204 13:19:24.406379 2461 net.cpp:754] [Update] Layer Edgeconv4_1, param 0 data: 0.00740561; diff: 7.17272e-13 I1204 13:19:24.407716 2461 net.cpp:754] [Update] Layer Edgeconv4_1, param 1 data: 0.0415128; diff: 0 I1204 13:19:24.409811 2461 net.cpp:754] [Update] Layer Edgeconv4_2, param 0 data: 0.00582409; diff: 5.64094e-13 I1204 13:19:24.412645 2461 net.cpp:754] [Update] Layer Edgeconv4_2, param 1 data: 0.042761; diff: 0 I1204 13:19:24.414752 2461 net.cpp:754] [Update] Layer Edgeconv4_3, param 0 data: 0.00603477; diff: 5.84498e-13 I1204 13:19:24.417565 2461 net.cpp:754] [Update] Layer Edgeconv4_3, param 1 data: 0.0570726; diff: 0 I1204 13:19:24.419639 2461 net.cpp:754] [Update] Layer Edgeconv5_1, param 0 data: 0.0118411; diff: 1.14688e-10 I1204 13:19:24.422469 2461 net.cpp:754] [Update] Layer Edgeconv5_1, param 1 data: 0.113729; diff: 0 I1204 13:19:24.424614 2461 net.cpp:754] [Update] Layer Edgeconv5_2, param 0 data: 0.0091425; diff: 8.85499e-11 I1204 13:19:24.427412 2461 net.cpp:754] [Update] Layer Edgeconv5_2, param 1 data: 0.174423; diff: 0 I1204 13:19:24.429507 2461 net.cpp:754] [Update] Layer Edgeconv5_3, param 0 data: 0.00717594; diff: 6.95028e-11 I1204 13:19:24.432281 2461 net.cpp:754] [Update] Layer Edgeconv5_3, param 1 data: 0.22697; diff: 0 I1204 13:19:24.432334 2461 net.cpp:754] [Update] Layer Edgescore-dsn1, param 0 data: 0.00322575; diff: 3.12431e-15 I1204 13:19:24.432346 2461 net.cpp:754] [Update] Layer Edgescore-dsn1, param 1 data: 0.0334441; diff: 0 I1204 13:19:24.432356 2461 net.cpp:754] [Update] Layer Edgescore-dsn2, param 0 data: 0.00306832; diff: 2.97182e-15 I1204 13:19:24.432366 2461 net.cpp:754] [Update] Layer Edgescore-dsn2, param 1 data: 0.021047; diff: 0 I1204 13:19:24.432376 2461 net.cpp:754] [Update] Layer Edgeupsample_2, param 0 data: 0.25; diff: 0 I1204 13:19:24.432404 2461 net.cpp:754] [Update] Layer Edgeupsample_2, param 1 data: 0; diff: 0 I1204 13:19:24.432417 2461 net.cpp:754] [Update] Layer Edgescore-dsn3, param 0 data: 0.0035274; diff: 3.41647e-15 I1204 13:19:24.432441 2461 net.cpp:754] [Update] Layer Edgescore-dsn3, param 1 data: 0.00768858; diff: 0 I1204 13:19:24.432451 2461 net.cpp:754] [Update] Layer Edgeupsample_4, param 0 data: 0.25; diff: 0 I1204 13:19:24.432461 2461 net.cpp:754] [Update] Layer Edgeupsample_4, param 1 data: 0; diff: 0 I1204 13:19:24.432471 2461 net.cpp:754] [Update] Layer Edgescore-dsn4, param 0 data: 0.00384611; diff: 3.72515e-15 I1204 13:19:24.432482 2461 net.cpp:754] [Update] Layer Edgescore-dsn4, param 1 data: 0.00425628; diff: 0 I1204 13:19:24.432492 2461 net.cpp:754] [Update] Layer Edgeupsample_8, param 0 data: 0.25; diff: 0 I1204 13:19:24.432502 2461 net.cpp:754] [Update] Layer Edgeupsample_8, param 1 data: 0; diff: 0 I1204 13:19:24.432512 2461 net.cpp:754] [Update] Layer Edgescore-dsn5, param 0 data: 0.00102744; diff: 9.9513e-16 I1204 13:19:24.432521 2461 net.cpp:754] [Update] Layer Edgescore-dsn5, param 1 data: 0.000218881; diff: 0 I1204 13:19:24.432533 2461 net.cpp:754] [Update] Layer Edgeupsample_16, param 0 data: 0.25; diff: 0 I1204 13:19:24.432543 2461 net.cpp:754] [Update] Layer Edgeupsample_16, param 1 data: 0; diff: 0 I1204 13:19:24.432552 2461 net.cpp:754] [Update] Layer Edgenew-score-weighting, param 0 data: 0.205404; diff: 1.98945e-14 I1204 13:19:24.432562 2461 net.cpp:754] [Update] Layer Edgenew-score-weighting, param 1 data: 0.00135704; diff: 0 I1204 13:19:24.444571 2461 solver.cpp:346] Iteration 9, Testing net (#0) I1204 13:19:28.286509 2461 solver.cpp:414] Test net output #0: dsn1_loss = 0 (* 1 = 0 loss) I1204 13:19:28.286576 2461 solver.cpp:414] Test net output #1: dsn2_loss = 0 (* 1 = 0 loss) I1204 13:19:28.286936 2461 solver.cpp:414] Test net output #2: dsn3_loss = 0 (* 1 = 0 loss) I1204 13:19:28.286955 2461 solver.cpp:414] Test net output #3: dsn4_loss = 0 (* 1 = 0 loss) I1204 13:19:28.286972 2461 solver.cpp:414] Test net output #4: dsn5_loss = 0 (* 1 = 0 loss) I1204 13:19:28.286984 2461 solver.cpp:414] Test net output #5: fuse_loss = 0 (* 1 = 0 loss) E1204 13:19:38.964378 2461 net.cpp:828] [Backward] All net params (data, diff): L1 norm = (123540, 0); L2 norm = (51.1185, 0) E1204 13:19:49.602077 2461 net.cpp:828] [Backward] All net params (data, diff): L1 norm = (123540, 0); L2 norm = (51.1185, 0) E1204 13:20:00.292870 2461 net.cpp:828] [Backward] All net params (data, diff): L1 norm = (123540, 0); L2 norm = (51.1185, 0) E1204 13:20:10.969599 2461 net.cpp:828] [Backward] All net params (data, diff): L1 norm = (123540, 0); L2 norm = (51.1185, 0) E1204 13:20:21.624631 2461 net.cpp:828] [Backward] All net params (data, diff): L1 norm = (123540, 0); L2 norm = (51.1185, 0) E1204 13:20:32.288601 2461 net.cpp:828] [Backward] All net params (data, diff): L1 norm = (123540, 0); L2 norm = (51.1185, 0) E1204 13:20:42.971149 2461 net.cpp:828] [Backward] All net params (data, diff): L1 norm = (123540, 0); L2 norm = (51.1185, 0) E1204 13:20:53.662241 2461 net.cpp:828] [Backward] All net params (data, diff): L1 norm = (123540, 0); L2 norm = (51.1185, 0) E1204 13:21:04.363137 2461 net.cpp:828] [Backward] All net params (data, diff): L1 norm = (123540, 0); L2 norm = (51.1185, 0) E1204 13:21:15.084235 2461 net.cpp:828] [Backward] All net params (data, diff): L1 norm = (123540, 0); L2 norm = (51.1185, 0) I1204 13:21:15.084326 2461 solver.cpp:242] Iteration 9, loss = 0 I1204 13:21:15.084352 2461 solver.cpp:258] Train net output #0: dsn1_loss = 0 (* 1 = 0 loss) I1204 13:21:15.084369 2461 solver.cpp:258] Train net output #1: dsn2_loss = 0 (* 1 = 0 loss) I1204 13:21:15.084381 2461 solver.cpp:258] Train net output #2: dsn3_loss = 0 (* 1 = 0 loss) I1204 13:21:15.084393 2461 solver.cpp:258] Train net output #3: dsn4_loss = 0 (* 1 = 0 loss) I1204 13:21:15.084403 2461 solver.cpp:258] Train net output #4: dsn5_loss = 0 (* 1 = 0 loss) I1204 13:21:15.084434 2461 solver.cpp:258] Train net output #5: fuse_loss = 0 (* 1 = 0 loss) I1204 13:21:15.084448 2461 solver.cpp:571] Iteration 9, lr = 1e-15 I1204 13:21:15.160527 2461 net.cpp:754] [Update] Layer Edgeconv1_1, param 0 data: 0.145995; diff: 1.27263e-11 I1204 13:21:15.160596 2461 net.cpp:754] [Update] Layer Edgeconv1_1, param 1 data: 0.50245; diff: 0 I1204 13:21:15.160677 2461 net.cpp:754] [Update] Layer Edgeconv1_2, param 0 data: 0.026326; diff: 2.29483e-12 I1204 13:21:15.160699 2461 net.cpp:754] [Update] Layer Edgeconv1_2, param 1 data: 0.270118; diff: 0 I1204 13:21:15.160804 2461 net.cpp:754] [Update] Layer Edgeconv2_1, param 0 data: 0.0211515; diff: 1.84377e-12 I1204 13:21:15.160830 2461 net.cpp:754] [Update] Layer Edgeconv2_1, param 1 data: 0.133851; diff: 0 I1204 13:21:15.161026 2461 net.cpp:754] [Update] Layer Edgeconv2_2, param 0 data: 0.0164008; diff: 1.42965e-12 I1204 13:21:15.161074 2461 net.cpp:754] [Update] Layer Edgeconv2_2, param 1 data: 0.148366; diff: 0 I1204 13:21:15.161442 2461 net.cpp:754] [Update] Layer Edgeconv3_1, param 0 data: 0.0121919; diff: 1.06276e-12 I1204 13:21:15.161569 2461 net.cpp:754] [Update] Layer Edgeconv3_1, param 1 data: 0.0556366; diff: 0 I1204 13:21:15.162219 2461 net.cpp:754] [Update] Layer Edgeconv3_2, param 0 data: 0.00905248; diff: 7.89101e-13 I1204 13:21:15.162770 2461 net.cpp:754] [Update] Layer Edgeconv3_2, param 1 data: 0.0638749; diff: 0 I1204 13:21:15.163455 2461 net.cpp:754] [Update] Layer Edgeconv3_3, param 0 data: 0.00922374; diff: 8.04029e-13 I1204 13:21:15.164005 2461 net.cpp:754] [Update] Layer Edgeconv3_3, param 1 data: 0.0597807; diff: 0 I1204 13:21:15.165179 2461 net.cpp:754] [Update] Layer Edgeconv4_1, param 0 data: 0.00740561; diff: 6.45545e-13 I1204 13:21:15.166517 2461 net.cpp:754] [Update] Layer Edgeconv4_1, param 1 data: 0.0415128; diff: 0 I1204 13:21:15.168608 2461 net.cpp:754] [Update] Layer Edgeconv4_2, param 0 data: 0.00582409; diff: 5.07684e-13 I1204 13:21:15.171435 2461 net.cpp:754] [Update] Layer Edgeconv4_2, param 1 data: 0.042761; diff: 0 I1204 13:21:15.173553 2461 net.cpp:754] [Update] Layer Edgeconv4_3, param 0 data: 0.00603477; diff: 5.26049e-13 I1204 13:21:15.176348 2461 net.cpp:754] [Update] Layer Edgeconv4_3, param 1 data: 0.0570726; diff: 0 I1204 13:21:15.178428 2461 net.cpp:754] [Update] Layer Edgeconv5_1, param 0 data: 0.0118411; diff: 1.03219e-10 I1204 13:21:15.181217 2461 net.cpp:754] [Update] Layer Edgeconv5_1, param 1 data: 0.113729; diff: 0 I1204 13:21:15.183348 2461 net.cpp:754] [Update] Layer Edgeconv5_2, param 0 data: 0.0091425; diff: 7.96948e-11 I1204 13:21:15.186158 2461 net.cpp:754] [Update] Layer Edgeconv5_2, param 1 data: 0.174423; diff: 0 I1204 13:21:15.188238 2461 net.cpp:754] [Update] Layer Edgeconv5_3, param 0 data: 0.00717594; diff: 6.25525e-11 I1204 13:21:15.191023 2461 net.cpp:754] [Update] Layer Edgeconv5_3, param 1 data: 0.22697; diff: 0 I1204 13:21:15.191078 2461 net.cpp:754] [Update] Layer Edgescore-dsn1, param 0 data: 0.00322575; diff: 2.81188e-15 I1204 13:21:15.191090 2461 net.cpp:754] [Update] Layer Edgescore-dsn1, param 1 data: 0.0334441; diff: 0 I1204 13:21:15.191100 2461 net.cpp:754] [Update] Layer Edgescore-dsn2, param 0 data: 0.00306832; diff: 2.67464e-15 I1204 13:21:15.191110 2461 net.cpp:754] [Update] Layer Edgescore-dsn2, param 1 data: 0.021047; diff: 0 I1204 13:21:15.191138 2461 net.cpp:754] [Update] Layer Edgeupsample_2, param 0 data: 0.25; diff: 0 I1204 13:21:15.191149 2461 net.cpp:754] [Update] Layer Edgeupsample_2, param 1 data: 0; diff: 0 I1204 13:21:15.191159 2461 net.cpp:754] [Update] Layer Edgescore-dsn3, param 0 data: 0.0035274; diff: 3.07482e-15 I1204 13:21:15.191169 2461 net.cpp:754] [Update] Layer Edgescore-dsn3, param 1 data: 0.00768858; diff: 0 I1204 13:21:15.191179 2461 net.cpp:754] [Update] Layer Edgeupsample_4, param 0 data: 0.25; diff: 0 I1204 13:21:15.191189 2461 net.cpp:754] [Update] Layer Edgeupsample_4, param 1 data: 0; diff: 0 I1204 13:21:15.191202 2461 net.cpp:754] [Update] Layer Edgescore-dsn4, param 0 data: 0.00384611; diff: 3.35263e-15 I1204 13:21:15.191213 2461 net.cpp:754] [Update] Layer Edgescore-dsn4, param 1 data: 0.00425628; diff: 0 I1204 13:21:15.191223 2461 net.cpp:754] [Update] Layer Edgeupsample_8, param 0 data: 0.25; diff: 0 I1204 13:21:15.191233 2461 net.cpp:754] [Update] Layer Edgeupsample_8, param 1 data: 0; diff: 0 I1204 13:21:15.191243 2461 net.cpp:754] [Update] Layer Edgescore-dsn5, param 0 data: 0.00102744; diff: 8.95617e-16 I1204 13:21:15.191253 2461 net.cpp:754] [Update] Layer Edgescore-dsn5, param 1 data: 0.000218881; diff: 0 I1204 13:21:15.191264 2461 net.cpp:754] [Update] Layer Edgeupsample_16, param 0 data: 0.25; diff: 0 I1204 13:21:15.191274 2461 net.cpp:754] [Update] Layer Edgeupsample_16, param 1 data: 0; diff: 0 I1204 13:21:15.191284 2461 net.cpp:754] [Update] Layer Edgenew-score-weighting, param 0 data: 0.205404; diff: 1.7905e-14 I1204 13:21:15.191294 2461 net.cpp:754] [Update] Layer Edgenew-score-weighting, param 1 data: 0.00135704; diff: 0

09876qwrte avatar Dec 04 '16 12:12 09876qwrte

@09876qwrte ,I have the same problem,Did you solve it ?

optimal16 avatar Sep 05 '17 10:09 optimal16

@09876qwrte @optimal16 @s9xie I have the same problem,Did you solve it ?

zhuozhongshuo avatar Dec 25 '18 08:12 zhuozhongshuo