detecto
detecto copied to clipboard
Loss is nan on validation
I have created a custom dataset using labelImg, everything looks fine, but while training I get Loss: nan
after every epoch.
I reduced the size of the dataset for better manual checking, checked the names of the files, everything is fine, but the model is not training.
Thought that maybe the thing was that I was using png format and not jpg, but converting my files and labels didn't solve the issue.
Can you help me? Maybe I can send you the dataset? (just 30 images and 30 labels for easier debugging)
Do any of the suggestions in #91 and #36 work? This issue occasionally comes up for some people and I'm not sure the best solution besides maybe double-checking/recreating environments to see if that helps.
Just decrease the learning rate, it might getting large gradient descent. In my case I reduced it to 0.001