Fix the bug when the number of val/test samples is small

[classicsong]

Issue #, if available: #959

Description of changes:

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.