[Model Request] OpenChat 3.5 -

[alpayariyak]

trafficstars

⚙️ Request New Models

• Link to an existing implementation (e.g. Hugging Face/Github): https://huggingface.co/openchat/openchat_3.5
• Is this model architecture supported by MLC-LLM? (the list of supported models) Yes

Additional context

Current best 7b OSS model, beating March GPT-3.5 on most benchmarks

[MikeLP]

I've converted it, and it works well

python3 -m mlc_llm.build --hf-path  openchat/openchat_3.5 --target metal_x86_64 --quantization q4f16_1

I have only 1 issue the model doesn't stop streaming sometimes (like stop word doesn't work), looking for a solution.

[alpayariyak]

Have you uploaded it somewhere on HuggingFace? Seems like an eos token issue, might be fixable with a change to the config file

[MikeLP]

@alpayariyak Uploaded to HF

OpenChat 3.5

https://huggingface.co/iyanello/openchat_3.5-q4f16_1-mlc-llm

OpenChat 3.5 - 1210 update

https://huggingface.co/iyanello/openchat_3.5-1210-q4f16_1-mlc-llm/tree/main

[MikeLP]

https://github.com/mlc-ai/mlc-llm/issues/1776#issuecomment-1992550167