support for older models

[qwenzo]

Hello,

I was wondering if it is straightforward to bring older models such as GPT-2 to lit-gpt. If so, what files/configs do I need to change?

Thank you!

[rasbt]

Good point, and it should be. I use GPT-2 myself privately a lot as well, and it'd be nice to have it in LitGPT as well.

I think the architecture is similar to GPTNeo, so you can probably copy and adapt the GPTNeo config. The general todo list I use for adding new configs is:

• [ ] Implement model download
• [ ] Implement HF checkpoint conversion
• [ ] Make sure generate.py produces reasonable outputs
• [ ] Update model_download docs
• [ ] Test pretraining
• [ ] Test finetuning
  • [ ] Full finetuning
  • [ ] LoRA
  • [ ] Adapter + Adapter v2
• [ ] Add tests