DifferentiableBinarization icon indicating copy to clipboard operation
DifferentiableBinarization copied to clipboard

loss keeps 4.96 all along

Open white2018 opened this issue 4 years ago • 7 comments

@xuannianz Excellent work.

I have encounter such problems in training process. 2189/2250 [============================>.] - ETA: 27s - loss: 4.96312020-03-09 17:47:41 [INFO] [Thread-6] [geos.py:219] Self-intersection at or near point 507.97936313701194 440.22383226394601 2020-03-09 17:47:41 [INFO] [Thread-6] [geos.py:219] Self-intersection at or near point 639.2710706150342 574.59295807232354 2190/2250 [============================>.] - ETA: 27s - loss: 4.96312020-03-09 17:47:41 [INFO] [Thread-6] [geos.py:219] Self-intersection at or near point 482.31532442427749 637.21180633522147 2196/2250 [============================>.] - ETA: 24s - loss: 4.96312020-03-09 17:47:44 [INFO] [Thread-6] [geos.py:219] Self-intersection at or near point 36.365901518633024 639.57219251336892 2200/2250 [============================>.] - ETA: 22s - loss: 4.9631^C

and the loss keeps 4.9631. any advice to me?

Thanks a lot

white2018 avatar Mar 09 '20 09:03 white2018

Hi! have u fixed the problem? i got the same issue. My loss was able to converge to 1.xx but when i trained it again i got 4.9xx. I believe i did not make any change between those two training procedures.

xxxxxxxiao avatar May 26 '20 02:05 xxxxxxxiao

Hi, were you able to solve the problem? I am also stuck at this step. Loss always stays at 4.95xxx Any thoughts what might go wrong in the training?

its-jd avatar Jun 11 '20 11:06 its-jd

setting a smaller learning rate may work

xxxxxxxiao avatar Jun 12 '20 00:06 xxxxxxxiao

have you done it yet?? same error

nguyenquanghieu2000d avatar Aug 14 '20 02:08 nguyenquanghieu2000d

setting a smaller learning rate may work

setting a smaller learning rate may work

i set it to 1e-7 but not work

nguyenquanghieu2000d avatar Aug 14 '20 04:08 nguyenquanghieu2000d

i got the same problem but my loss is nan

duydangg avatar Nov 05 '20 04:11 duydangg

i got the same problem but my loss is nan

my datasets caused this problem, i've fixed this but now my val loss is 1.00 at every epoch

duydangg avatar Jan 26 '21 07:01 duydangg