Rajeev Goel

Tempe, Arizona PhD student in CS @ Arizona State University

Results 2 comments of


                                            Rajeev Goel

trafficstars

Got a nan loss and gradient norm when training swin-l on imagenet22k with O1

Use "torch.bfloat16" instead of "torch.float16" in AMP.

After several iterations gradient norm and loss becomes nan

@neurosynapse @byronyi This issue is due to AMP using the torch.float16 dtype by default. Use torch.bfloat16 instead.