where is the train file?

[liukaiyueyuo]

where is the train file? I want to learn how to train.

[StephennFernandes]

@liukaiyueyuo hey any luck finding the pretraining docs and scripts ? i am looking forward to actually pretrain the 7B model on my own data on some cloud TPUs would be helpful to find the pretraining scripts

[bokesyo]

also interested in training data composition, as it was not mentioned in technical report :)

[subramen]

there's a finetuning script at https://github.com/facebookresearch/llama-recipes/blob/main/llama_finetuning.py which you could adapt for pretraining. Section 2 of the paper (https://arxiv.org/pdf/2307.09288.pdf) has the hyperparams used for pretraining