How to choose the best training model

[Sixy1204]

Hello,

In the training process, we save one model every 5000 iterations. Besides training loss, are there any criteria to help choose the best model? Because I think even if training loss is low, the performance is probably not good. For example, I trained two models. One is model_400k iteration, the other is model_500k iteration. But the actual decoded content 400k's model is more reasonable than 500k's.

Thanks a lot！