icefall icon indicating copy to clipboard operation
icefall copied to clipboard

Published 20 hours ago • verified publisher icon k2-fsa

getting nans in lm training

Open Manjunath-mlp opened this issue 10 months ago • 0 comments

I am training a transformer LM ,and at some intermediate batch i am gettting loss nan and ppl nan. with hooks -ValueError: The sum of module.input_embedding.output is not finite:

Manjunath-mlp avatar Jan 29 '25 10:01 Manjunath-mlp