Error during train stage2_e2e

[DeepBehavier]

I used 6 A100 to train the model of stage2_e2e. After completing the fourth epoch, an error occurred.Error as shown below。 This mistake is easy to repeat. Please help me to solve this problem.

[7bbjungle]

Have you solved the problem？

[duanmushuangquan]

same error Have you solved the problem

[generalchan825]

same issue occured. Did you solve it?

[YTEP-ZHI]

Hi @DeepBehavier @7bbjungle @duanmushuangquan @generalchan825. It happens occasionally during training but we did find the cause of this issue unfortunately. A recommended workaround is to resume the checkpoint of the 4th epoch and continue training for the next epochs.