web-traffic-forecasting icon indicating copy to clipboard operation
web-traffic-forecasting copied to clipboard

During training, train loss and validation loss became nan, does this matter?

Open nce3xin opened this issue 6 years ago • 4 comments

When I ran cnn.py, during training, train loss and validation loss became nan after step 50, is this normal? I wonder why losses remains nan……

nce3xin avatar Jan 15 '19 09:01 nce3xin

Me too, Do you use gpu ? I had this problem when I use CPU to train.

leonJCY avatar Sep 16 '20 13:09 leonJCY

If you use gpu, it is 2080ti for me, then all the problems will be gone

leonJCY avatar Sep 17 '20 02:09 leonJCY

10 step becaome nan.. using cpu~~~

wentixiaogege avatar Feb 27 '21 07:02 wentixiaogege

change lr ,10 times smaller , work~;

wentixiaogege avatar Feb 27 '21 08:02 wentixiaogege