litgpt

litgpt copied to clipboard

For quick dev and testing, the new final validation step can be very expensive, especially with larger checkpoints. This adds an option to disable it. (This is analogous to the initial_validation argument; since some people don't need the initial validation loss some times, they may also not need the final validation loss some times).

This wouldn't change any default behaviors, and it would be analogous to the analogous to the initial_validation argument). Does it look ok to you @awaelchli and @carmocca or do you have any objections?

Thanks for the feedback. In my opinion, it's worth adding it because the initial evaluation was considered too expensive, so it's optional. For the same reason, the final validation can also be sometimes too expensive when wanting to do a quick run for testing purposes (e.g. figuring out what the "Memory used" value is or if the files are saved correctly etc.). So, it's nice to have some easy control over that and be able to skip the final validation in certain cases.

I was updating the Loom demos for the Studios in the Readme, and the validation can take up a relative long portion of time when doing quick demos (even longer for larger models): https://www.loom.com/share/4d5558cfaf874573b9c309878a36cf89

So I suggest bringing back this option to skip