Loss becomes very big suddenly when after epoch 10

[liamsun2019]

Dear author,

I encountered an issue when training on coco keypoints dataset. I just adjusted num_layers from 50 to 18, i.e, res18, and started to train based on coco keypoints dataset. The loss went down quickly and became negative soon. But after epoch 10, the loss became much bigger(over 30000) suddenly and then dropped again in next epoch. How do I understand such phenomenon? Any tips for that? Thanks!!

[Jeff-sjtu]

Hi @liamsun2019, I think the large loss value may be caused by the abnormal value of sigma. You can try adding a bigger eps to sigma (e.g. 1e-5) to see whether this problem happen again.

[liamsun2019]

Thanks for your suggestion. Will try it out.

[Tau-J]

Hi @liamsun2019, I think the large loss value may be caused by the abnormal value of sigma. You can try adding a bigger eps to sigma (e.g. 1e-5) to see whether this problem happen again.

Sorry to bother you, could I add a super big eps (e.g. 1e-3) to sigma? and is there any problem?