mlc-llm
mlc-llm copied to clipboard
[Bug] Mistral-Nemo-Instruct-2407 The results were confused
When I converted the mistralai/Mistral-Nemo-Instruct-2407 model and used it later, I found that it was basically unable to answer Chinese questions. I checked and found that in the converted file, there seemed to be a lack of a dictionary related to Chinese, and the dictionary structure was different from that of the gemma model. I found the model converted by others from hf, and this problem also occurred. But I used ollama to run this model and didn't find this problem.
I tried adjusting conv_template and found that this problem would occur no matter which one.
This is the pr that supports this model: https://github.com/mlc-ai/mlc-llm/pull/2676
Models converted by others on hf: https://huggingface.co/imi2/Mistral-Nemo-12B-Instruct-q4f16_1-MLC
My platform is mac m1 running on metal with q416_1 quantization
This is the result of mlc_llm:
This is the result of ollama: