academic-budget-bert icon indicating copy to clipboard operation
academic-budget-bert copied to clipboard

Grad overflow and null validation loss

Open NewDriverLee opened this issue 1 year ago • 0 comments

In the first epoch of pretraining, grad overflow happened in every iteration. Also, the evaluation loss of some epochs is null, after about the 17th epoch. It looks like the evaluation was not performed for these epochs. But the other epochs still exhibit valid evaluation loss. Anybody met the same issue? Is it normal?

All the commands I used are the same to the examples of README.md.

NewDriverLee avatar May 19 '23 07:05 NewDriverLee