detecto icon indicating copy to clipboard operation
detecto copied to clipboard

Loss is nan on validation

Open maiiabocharova opened this issue 2 years ago • 2 comments

I have created a custom dataset using labelImg, everything looks fine, but while training I get Loss: nan after every epoch.

I reduced the size of the dataset for better manual checking, checked the names of the files, everything is fine, but the model is not training.

Thought that maybe the thing was that I was using png format and not jpg, but converting my files and labels didn't solve the issue.

Can you help me? Maybe I can send you the dataset? (just 30 images and 30 labels for easier debugging)

maiiabocharova avatar Sep 03 '21 04:09 maiiabocharova

Do any of the suggestions in #91 and #36 work? This issue occasionally comes up for some people and I'm not sure the best solution besides maybe double-checking/recreating environments to see if that helps.

alankbi avatar Jan 07 '22 19:01 alankbi

Just decrease the learning rate, it might getting large gradient descent. In my case I reduced it to 0.001

Tanmay-Trantor avatar May 13 '22 11:05 Tanmay-Trantor