Switchable-Whitening icon indicating copy to clipboard operation
Switchable-Whitening copied to clipboard

Published 20 hours ago • verified publisher icon XingangPan

Train model with apex

Open ivylinden opened this issue 5 years ago • 3 comments

Hello, I use apex train the model with the level 'O1',ie fp16 ,but the loss is nan, if i train the model with fp32,the loss is normal,Do you have the same problem

ivylinden avatar Jan 08 '20 09:01 ivylinden

Sorry I haven't tried training with apex. It seems that there is a numerical issue with fp16, you may simply use fp32.

XingangPan avatar Jan 20 '20 05:01 XingangPan

I had the same problem. We need a way to exclude SW layer from O2 just like BN. But I have not found a proper way

hiyijian avatar Jan 20 '20 06:01 hiyijian

the problem may be cause by numerical underflow in https://github.com/XingangPan/Switchable-Whitening/blob/master/models/ops/switchwhiten.py#L175 ?

I think maybe add a eps here can solve it ?

DTennant avatar Feb 27 '20 15:02 DTennant