Validation-Loss is infinite when running egs/aishell/conformer-mmi

[davidlin409]

Dear all:

Recently I tries to adopt LF-MMI in training AISHELL dataset inside model "tdnn-lstm", and I found that validation error is always infinity, while training-loss is perfectly normal.

In order to debug it, I tries to run "conformer-mmi" script in egs/aishell, to ensure that there is no programming error that I introduce (I only change validation interval to 1000, from original 3000). The training result still shows that in egs/aishell/conformer-mmi training script, validation-loss is still infinity. So it means that original script has already the issue of "validation error to be infiinity".

Commit hash for "icefall" is a7fbb18b.

Curently I am out of direction in debugging this issue, so don't know if there is anything that I am missing?