FastChat
FastChat copied to clipboard
[feature] support codellama 70b
Why are these changes needed?
CodeLlama 70B Instruct uses a different format for the chat prompt than previous Llama 2 or CodeLlama models.
In this branch:
$ python3 -m fastchat.serve.cli --model codellama/CodeLlama-70b-Instruct-hf --num-gpus 2
user: hi
assistant: 😊 Hi! I'm a large language model, and I'm here to assist you with your writing. What would you like to write about today? Is there a specific topic or theme you're interested in? 📝💡
user:
In main branch:
$ python3 -m fastchat.serve.cli --model codellama/CodeLlama-70b-Instruct-hf --num-gpus 2
[INST]: hi
[/INST]: [INST] hi [/INST]
[INST] hi [/INST]
[INST] hi [/INST]
[INST] hi [/INST]
[INST] hi [/INST]
[INST] hi [/INST]
Related issue number (if applicable)
Closes #3135 Closes #3012
Checks
- [x] I've run
format.sh
to lint the changes in this PR. - [x] I've included any doc changes needed.
- [ ] I've made sure the relevant tests are passing (if applicable).