Enhancement | Storing best checkpoint

[deimsdeutsch]

Is there a way to store the best checkpoint among many so if somebody wants to store the best and not necessarily among every 20,000 checkpoints ? This way should be ideal if the dataset is not balanced.

[yangxue0827]

This is not practical, as a full evaluation will be very time consuming. And after the training is stable, the results are basically the same. @deimsdeutsch

[deimsdeutsch]

@yangxue0827 Very much practical and has been implemented by many frameworks. Just save the output of checkpoint in a best variable and replace it with the next if it is better. Keep the check for every best interval in the configuration file.

[yangxue0827]

I mean a full evaluation will be very time consuming.  @deimsdeutsch

[deimsdeutsch]

Can you look into this and add this as enhancement ?

Thanks.

[yangxue0827]

I don't have this plan for the time being, you can try it yourself. @deimsdeutsch