grad_norm Nan question

Open 937739823 opened this issue 7 months ago • 1 comments

{'loss': 0.2051, 'grad_norm': nan, 'learning_rate': 9.774074342405685e-06, 'epoch': 0.53} {'loss': 0.0, 'grad_norm': nan, 'learning_rate': 9.773696745450087e-06, 'epoch': 0.53} {'loss': 0.0, 'grad_norm': nan, 'learning_rate': 9.773318840518906e-06, 'epoch': 0.53} How to adjust the gradient such as Nan loss and 0 when training to epoch=0.53?

May 21 '25 08:05 937739823

{'loss': 0.2051, 'grad_norm': nan, 'learning_rate': 9.774074342405685e-06, 'epoch': 0.53} {'loss': 0.0, 'grad_norm': nan, 'learning_rate': 9.773696745450087e-06, 'epoch': 0.53} {'loss': 0.0, 'grad_norm': nan, 'learning_rate': 9.773318840518906e-06, 'epoch': 0.53} How to adjust the gradient such as Nan loss and 0 when training to epoch=0.53?

Hello, I'm also experiencing the same issue. Could you please tell me how you resolved it?

Jun 07 '25 09:06 heweapon