LLFlow icon indicating copy to clipboard operation
LLFlow copied to clipboard

NaN or Inf found in input tensor.

Open ph0316 opened this issue 2 years ago • 2 comments

Hello, sorry to bother you again. I recently encountered a new error in training, do you know why? image The number of wrong epoch is different each time.

ph0316 avatar Jul 21 '22 02:07 ph0316

I have met the same problem, too~

Nitrogens avatar Mar 20 '23 07:03 Nitrogens

Hi, thanks for your interest in our work.

Maybe you could try the following work to alleviate this problem

https://arxiv.org/pdf/2212.04319.pdf

Besides, a trivial way may be that you could try to use a smaller learning rate.

wyf0912 avatar Sep 28 '23 08:09 wyf0912