nebuly
nebuly copied to clipboard
[Chatllama] Add support to other open-source models
Description
LLaMA is a great model, however the weights license can be a huge barrier for users interested in the open-source LLM capabilities. The actual RLHF is designed around the LLaMA model, but it can be extended to other open-source models, like Flan-T5 or OPT.
TODO
- [x] Extend support of ChatLLaMA to HuggingFace models
- [ ] Add support for the encoder-decoder architectures (AutoModelsForSeq2Seq)
- [x] Add support for LLaMA from HF.
- [x] Add support for Cerebras GPT.
- [ ] Test the support for
- [ ] FlanT5
- [x] GPTJ
- [x] GPTNeoX
- [x] OPT
- [x] BLOOM
- [x] BLOOMZ
- [ ] Galactica
- [ ] Implement unittest for the code