Single-DGOD icon indicating copy to clipboard operation
Single-DGOD copied to clipboard

Published 20 hours ago • verified publisher icon AmingWu

Loss is

Open HuaYuexia opened this issue 1 year ago • 4 comments

HuaYuexia avatar Feb 03 '24 05:02 HuaYuexia

Loss is nan image

HuaYuexia avatar Feb 03 '24 05:02 HuaYuexia

The learning rate is set to 0.001. And the batchsize is set to 4. Welcome to communicate with me. Thanks.

AmingWu avatar Feb 03 '24 05:02 AmingWu

The learning rate is set to 0.001. And the batchsize is set to 4. Welcome to communicate with me. Thanks.

My learning rate is set to 0.001 too.The difference is that I used two gpus with a batch size of 8.The only way to get it to work properly is to reduce the learning rate to 1e-5, but then it won't converge, why is that?Sincerely look forward to your guidance, although this may be a naive question for you.

HuaYuexia avatar Feb 05 '24 02:02 HuaYuexia

Thank you for your reply. The problem has been successfully resolved.

xiao-song2022 avatar Feb 06 '24 17:02 xiao-song2022