academic-budget-bert icon indicating copy to clipboard operation
academic-budget-bert copied to clipboard

Published 20 hours ago • verified publisher icon IntelLabs
→

Metadata

Repository containing code for "How to Train BERT with an Academic Budget" paper

Results 11 academic-budget-bert issues
Sort by recently updated
recently updated
newest added

Grad overflow and null validation loss

In the first epoch of pretraining, grad overflow happened in every iteration. Also, the evaluation loss of some epochs is null, after about the 17th epoch. It looks like the...

NewDriverLee avatar NewDriverLee

← Metadata

298
Stars
47
Forks
Watchers

Owner

verified publisher icon IntelLabs

Metadata

Repository containing code for "How to Train BERT with an Academic Budget" paper

Back