mlc-llm
mlc-llm copied to clipboard
mlc-ai
Phi 3 128
🚀 Feature
Introduce phi 3 mini 128k instruct
https://huggingface.co/microsoft/Phi-3-mini-128k-instruct
Mini can run on phones (there is a 4k version and 128k version)
Motivation
Alternatives
Additional context
The 128k version could allow us to analyze a pdf from the phone. the 4k version would probably be less cpu intensive and would work out of the box.
sorry.i want to known that are yiu run phi3 mini model on the phone?i i run it with onnxruntime genai but it's too slow
Well I use phi2 all the time using ml. Android on my phone with Snapdragon 2nd gen and it works really good and smooth. That is why I would like to know how I can put phi 3 on mlc.
Tried following the instructions, and got the error:
ValueError: Unknown model type: phi3. Available ones: ['llama', 'mistral', 'gemma', 'gpt2', 'mixtral', 'gpt_neox', 'gpt_bigcode', 'phi-msft', 'phi', 'qwen', 'qwen2', 'stablelm', 'baichuan', 'internlm', 'rwkv5', 'orion', 'llava', 'rwkv6', 'chatglm', 'eagle']
@kripper thanks for your advice, the MODELS is not really just a list with a model name, it is a dict with the model name as key and the corresponding loader and implementation.
I've tried specify the --model-type phi-msft but without luck.
I would first research if the model loader is compatible with previous versions. If so, map the ID to the existing loader. Otherwise, create a new loader and have fun hacking, ie:
- research what changed by comparing (diff) the official loader for the previous and new versions.
- replicate those changes in mlc-llm's loader
Always prefere to share a same code base for the old and new model and use conditions (if(version == 1) {...} else if(version == 3) {...}).
Once this is figured out please update the android app apk with phi3. ❤️
