wav2letter icon indicating copy to clipboard operation
wav2letter copied to clipboard

Published 20 hours ago • verified publisher icon flashlight

Error while training RASR

Open innarid opened this issue 4 years ago • 3 comments
trafficstars

I got error(Floating point exception) while training 10 epoch. error.txt Log of 9 epoch looks good 009_log.txt I tried to change lr and lr decay, but it didn't help. Could you please help me to find the reason of this error? Thanks!

innarid avatar Apr 20 '21 15:04 innarid

Could you run without distributed training? Do you have all 1-9 epochs passed fine?

tlikhomanenko avatar Apr 23 '21 04:04 tlikhomanenko

I got the same error with -enable_distributed=false. Yes, 9 epochs passed fine.

innarid avatar Apr 23 '21 09:04 innarid

Can you confirm that if you rerun training from epoch 1 it is still working?

tlikhomanenko avatar Apr 23 '21 20:04 tlikhomanenko