The difference between forward score and backward score exceeds 1.0

[somniumism]

trafficstars

Thank you always. It's been very helpful since kaldi : )

I am trying to train `librispeech/ASR/conformer_mmi with a private dataset to check the recipe works well.

I checked that loss is falling and learning is going well.

However, after crossing certain steps, the following warning appears:

[W] /opt/k2/k2/csrc/intersect_dense.cu:906:k2::Array1<float> k2::MultiGraphDenseIntersect::GetScoreCutoffs() The difference between forward score and backward score exceeds 1.0, the value is : 1.500000

Does the warning above mean that the total score of the numerator lattice or denominator lattice exceeded 1.0 when computing loss? Is this because of lattice determinism(ex. the lattice is non-deterministic)?

If not, I want to know what kind of problem causes that warning, and if it adversely affects the learning, and how to solve the problem.

[danpovey]

I think that warning would normally happen because of numerical roundoff, although the value seems a little too high for that. There could be something strange about the lattice involved. It's been a while since I touched that recipe or that code, so I can't give more precise instructions, but if you could find a way to save the lattice involved, and maybe find an easy way to reproduce the error, it would be helpful. [since the error looks too large to be normal.]

[somniumism]

I think that warning would normally happen because of numerical roundoff, although the value seems a little too high for that. There could be something strange about the lattice involved. It's been a while since I touched that recipe or that code, so I can't give more precise instructions, but if you could find a way to save the lattice involved, and maybe find an easy way to reproduce the error, it would be helpful. [since the error looks too large to be normal.]

Thank you a lot. I think I need to check the process of making that error first. And as you said, I'll find a way to easily reproduce errors. If there is an update, I will leave a comment :)