academic-budget-bert
academic-budget-bert copied to clipboard
Grad overflow and null validation loss
In the first epoch of pretraining, grad overflow happened in every iteration. Also, the evaluation loss of some epochs is null, after about the 17th epoch. It looks like the evaluation was not performed for these epochs. But the other epochs still exhibit valid evaluation loss. Anybody met the same issue? Is it normal?
All the commands I used are the same to the examples of README.md.