FastChat icon indicating copy to clipboard operation
FastChat copied to clipboard

[feature] support codellama 70b

Open kota-iizuka opened this issue 11 months ago • 0 comments

Why are these changes needed?

CodeLlama 70B Instruct uses a different format for the chat prompt than previous Llama 2 or CodeLlama models.

In this branch:

$ python3 -m fastchat.serve.cli --model codellama/CodeLlama-70b-Instruct-hf --num-gpus 2
user: hi
assistant: 😊 Hi! I'm a large language model, and I'm here to assist you with your writing. What would you like to write about today? Is there a specific topic or theme you're interested in? 📝💡
user: 

In main branch:

$ python3 -m fastchat.serve.cli --model codellama/CodeLlama-70b-Instruct-hf --num-gpus 2
[INST]: hi
[/INST]: [INST] hi [/INST]

[INST] hi [/INST]

[INST] hi [/INST]

[INST] hi [/INST]

[INST] hi [/INST]

[INST] hi [/INST]

Related issue number (if applicable)

Closes #3135 Closes #3012

Checks

  • [x] I've run format.sh to lint the changes in this PR.
  • [x] I've included any doc changes needed.
  • [ ] I've made sure the relevant tests are passing (if applicable).

kota-iizuka avatar Apr 04 '24 03:04 kota-iizuka