Have you considered Cerebras-GPT-13B

[aamir-gmail]

Would this be suitable as the base model https://huggingface.co/cerebras/Cerebras-GPT-13B seems to be free of any licensing limitations

[zhisbug]

We have done some study and our temporal conclusion is that the backbone model Cerebras-GPT has much lower quality than the llama, due to pertaining dataset size, and data filtering strategies.

See the page 4, footnote 1 of the Cerebras-GPT paper: https://arxiv.org/pdf/2304.03208.pdf

Hence we prefer to look into models like Flan-T5 at this moment.

More discussion is welcome.

[zhisbug]

Closing as it is not planned.

[zhisbug]

Meanwhile, we have supported Dolly-v2, koala, and we're considering open assistant and probably the stabilityLM