Add support to continue pratraining

[hanyin88]

🚀 The feature, motivation and pitch

Currently the LLaMA-receipts do not support pretraining out of box. For example, one could not save the pretrained model based on the current checkpoint saving logic which requires run_validation = True, which is not needed during continued pretraining phase. It would be nice to add this feature down the road as it's a common use case in the process of domain adaption.

Thanks.

Alternatives

No response

Additional context

No response

[thisisisheanesu]

@hanyin88 Did you find anything for this elsewhere?

[HamidShojanazeri]

@hanyin88 the original focus of this repo was about fine-tuning, so it has not been designed to support pertaining, but this is something we are investigating as well.