convNet.pytorch icon indicating copy to clipboard operation
convNet.pytorch copied to clipboard

Published 20 hours ago • verified publisher icon eladhoffer

Nan loss for quantization

Open syorami opened this issue 5 years ago • 5 comments

I'm running the code for 8-bit quantization but found that the training loss always gets NAN while I didn't make a slight modification to the original code. Wondering why this could happen and hoping for your clarification.

syorami avatar Jul 18 '19 03:07 syorami

Same issue, using PyTorch 1.1.0

stgzr avatar Aug 06 '19 04:08 stgzr

Same issue, using PyTorch 1.1.0

Change PyTorch version to 1.0.1 may solve the NaN loss issue.

stgzr avatar Aug 06 '19 05:08 stgzr

@stgzr Thanks! I would have a try. And this seems strange to me. What's the difference between Pytorch 1.1.0 and 1.0.1 that could lead to this Nan loss?

syorami avatar Aug 06 '19 14:08 syorami

The same problem, have you solved it? Finding help.

chenhongluo avatar Jun 03 '21 11:06 chenhongluo

Try changing the learning rate, decrease it or increase it by factor of 10 and see

TonojiKiobya avatar Apr 03 '24 12:04 TonojiKiobya