age-gender-estimation icon indicating copy to clipboard operation
age-gender-estimation copied to clipboard

Epoch 00001: val_loss did not improve from inf

Open yimjunhyuck2 opened this issue 6 years ago • 7 comments

I tried training through the following command but failed : sudo python3 ./train.py --input data/imdb_db.mat --nb_epochs 10 --depth 10. Every epoch went on with message "Epoch 000XX: val_loss did not improve from inf". Also after train.py was done, I tried plotting by "sudo python3 ./plot.history.py" but it also failed with a message "File models/history_16_8.h5 does not exist". I wonder if it's because I'm missing something.

yimjunhyuck2 avatar Jul 22 '19 05:07 yimjunhyuck2

During training, the value of loss become nan. I can't find how to solve this problem.

yimjunhyuck2 avatar Aug 08 '19 10:08 yimjunhyuck2

How did you create data/imdb_db.ma?

Try different learning rate lr.

yu4u avatar Aug 08 '19 11:08 yu4u

I created data/imdb_db.mat and tried lr 0.6, 0.8, 1.5. But the nan loss problem still occur. :(

yimjunhyuck2 avatar Aug 13 '19 08:08 yimjunhyuck2

Is this a problem with tensorflow or keras?

yimjunhyuck2 avatar Aug 13 '19 08:08 yimjunhyuck2

I want to know if there's solution from it? I have the same problem here.

youngmihuang avatar Aug 25 '20 08:08 youngmihuang

In my case, the default was too high. There was some improvement when I set the weight 0.005~0.01. Sorry for my poor English and I wish you get the result you wanted. -----Original Message----- From: "Shao Hsuan Huang"[email protected] To: "yu4u/age-gender-estimation"[email protected]; Cc: "yimjunhyuck2"[email protected]; "Author"[email protected]; Sent: 2020-08-25 (화) 17:35:21 (GMT+09:00) Subject: Re: [yu4u/age-gender-estimation] Epoch 00001: val_loss did not improve from inf (#97)

I want to know if there's solution from it? I have the same problem here. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

yimjunhyuck2 avatar Aug 25 '20 18:08 yimjunhyuck2

Thank for your suggestion. It seems that decreasing the learning rate lower than 0.01, and the loss would be improved (not NaN)?

youngmihuang avatar Aug 31 '20 15:08 youngmihuang