LightXML icon indicating copy to clipboard operation
LightXML copied to clipboard

Published 20 hours ago • verified publisher icon kongds

some problem in training

Open leileihl opened this issue 4 years ago • 2 comments

I encountered an anomaly during training. At a certain batch size, P@1 dropped from 0.9 to 0.1.

leileihl avatar Jun 24 '21 14:06 leileihl

Maybe gradient has exploration during training, what dataset did you use ?

kongds avatar Jun 29 '21 12:06 kongds

AmazonCat13k, i used apex with O1. When i used O0, model has not gradient explosion.

leileihl avatar Jul 05 '21 13:07 leileihl