how to pretrain llama2?

[wen020]

We read the pretrain code in this repo and find that the pretrain code is gpt model. how to pretrain llama2?

[fireyanci]

You can read the tutorials

[rasbt]

Actually, the GPT here is just a Python class. All other LLMs are very similar to GPT, which is why they are derived from that class in code. To choose the LLM you want to pretrain, check the pretrain tutorials here: https://github.com/Lightning-AI/litgpt/blob/main/tutorials/pretrain.md

E.g., litgpt pretrain ... --model_name Llama-2-7b-hf ...

[wen020]

ok

[rasbt]

I hope this helped. Please feel free to reopen if you have a follow up question.