EricLBuehler
Model Wishlist
Please let us know what model architectures you would like to be added!
Up to date todo list below. Please feel free to contribute any model, a PR without device mapping, ISQ, etc. will still be merged!
Language models
- [ ] snowflake-arctic-instruct: Snowflake/snowflake-arctic-instruct
- [ ] WizardLM-2: alpindale/WizardLM-2-8x22B
- [ ] Command R: CohereForAI/c4ai-command-r-v01
- [ ] Command R+: CohereForAI/c4ai-command-r-plus
Multimodal models
- [x] Llava: liuhaotian/llava-v1.5-7b
- [x] Idefics2: HuggingFaceM4/idefics2-8b
- [ ] deepseek-vl: deepseek-ai/deepseek-vl-7b-chat
- [ ] MiniCPM: openbmb/MiniCPM-V-2
- [x] Phi 3 Vision: https://huggingface.co/microsoft/Phi-3-vision-128k-instruct
Embedding models
- [ ] T5: google-t5/t5-base
- [ ] nomic-text-embed: nomic-ai/nomic-embed-text-v1
@NiuBlibing, we have llama3 support ready: the README has a few examples. I will add Qwen support shortly.
@NiuBlibing, I just added Qwen2 support. Quantized Qwen2 support will be added in the next few days.
Hello! Any plans for adding multimodal (e.g. llava) and embedding models?
Can you add https://huggingface.co/Snowflake/snowflake-arctic-instruct?
@cargecla1, yes! It will be a great use case for ISQ.
Hello! Any plans for adding multimodal (e.g. llava) and embedding models?
@francis2tm, yes. I plan on supporting Llava and embedding models this week.
Would be nice to support at least one strong vision-language model: https://huggingface.co/openbmb/MiniCPM-V-2 https://huggingface.co/OpenGVLab/InternVL-Chat-V1-5 with an option to compute visual frontend model on CPU. You might find it easier to ship visual transformer part via onnx.
Would love to see some DeepSeek-VL, this model is better than Llava and spupports multiple images per prompt https://huggingface.co/collections/deepseek-ai/deepseek-vl-65f295948133d9cf92b706d3
Also, outside the LLM world, would love to see support for https://github.com/cvg/LightGlue :) but not sure if that's possible ...
Could you add support to for GGUF quantized Phi-3-Mini to the wishlist? Currently, this fails (built from master):
Running `./mistralrs-server gguf -m PrunaAI/Phi-3-mini-128k-instruct-GGUF-Imatrix-smashed -t microsoft/Phi-3-mini-128k-instruct -f /home/jett/Downloads/llms/Phi-3-mini-128k-instruct-q3_K_S.gguf`
2024-04-29T03:08:35.180939Z INFO mistralrs_server: avx: true, neon: false, simd128: false, f16c: false
2024-04-29T03:08:35.180975Z INFO mistralrs_server: Sampling method: penalties -> temperature -> topk -> topp -> multinomial
2024-04-29T03:08:35.180982Z INFO mistralrs_server: Loading model `microsoft/Phi-3-mini-128k-instruct` on Cpu...
2024-04-29T03:08:35.180989Z INFO mistralrs_server: Model kind is: quantized from gguf (no adapters)
2024-04-29T03:08:35.181017Z INFO hf_hub: Token file not found "/home/jett/.cache/huggingface/token"
2024-04-29T03:08:35.181048Z INFO mistralrs_core::utils::tokens: Could not load token at "/home/jett/.cache/huggingface/token", using no HF token.
2024-04-29T03:08:35.181122Z INFO hf_hub: Token file not found "/home/jett/.cache/huggingface/token"
2024-04-29T03:08:35.181133Z INFO mistralrs_core::utils::tokens: Could not load token at "/home/jett/.cache/huggingface/token", using no HF token.
Error: Unknown GGUF architecture `phi3`
@kir-gadjello
Would be nice to support at least one strong vision-language model: https://huggingface.co/openbmb/MiniCPM-V-2 https://huggingface.co/OpenGVLab/InternVL-Chat-V1-5 with an option to compute visual frontend model on CPU. You might find it easier to ship visual transformer part via onnx.
Supporting a vision+language or multimodal model is very high priority right now.
@chelbos
Would love to see some DeepSeek-VL, this model is better than Llava and spupports multiple images per prompt https://huggingface.co/collections/deepseek-ai/deepseek-vl-65f295948133d9cf92b706d3
I'll add this one too.
Also, outside the LLM world, would love to see support for https://github.com/cvg/LightGlue :) but not sure if that's possible ...
I will look into it!
@jett06
Could you add support to for GGUF quantized Phi-3-Mini to the wishlist?
Yes, absolutely, I think it should be easy. In the meantime, you can use ISQ to get the same speed.
@rodion-m
It'll be great to see WizardLM-2 and suzume. And thanks for a great tool!
Thanks! I think suzume is just finetuned Llama so that can be used already. I'll add WizardLM.
@W4G1
Command-R and Command-R+ from Cohere would be amazing 🙏
Yes, I'll add those.
@yongkangzhao
T5 and LLaVA
Yes, I'll add those. T5 will be a nice smaller model.
@EricLBuehler Thanks for your reply, for adding my suggestion to the model wishlist, and for developing such an awesome project! It's very appreciated :)
Congrats for your great work! +1 for vision models like Idefics2-8b or better would be awesome
Hello, first of all, I want to express my appreciation for the excellent work your team has accomplished on the mistral.rs engine. It's a great project.
I am currently developing a personal AI assistant using Rust, and I believe integrating additional features into your engine could significantly enhance its utility and appeal. Specifically, adding support for Whisper and incorporating Text-to-Speech (TTS) functionalities, such as StyleTTS or similar technologies, would be incredibly beneficial. This would enable the engine to handle LLM inference, speech-to-text, and text-to-speech processes in a unified system very fast (near runtime).
Implementing these features could transform the engine into a more versatile tool for developers like myself, who are keen on building more integrated and efficient AI applications.
@jett06, I just added quantized GGUF Phi-3 support in #276! That is without LongRope support currently, but you can use a plain model with ISQ.
@EricLBuehler Woah, thank you so much! This will be lovely for us folks with less powerful computers or size constraints, you're awesome :)
@jett06, my pleasure! I just fixed a small bug (in case you saw the strange behavior), so it should be all ready to go now!
@NeroHin
IBM's Granite series Code Models.
The 3b and 8b variants should already be supported as they are just based on the llama architecture.
The 20b and 34b variants are based on the GPTBigCode architecture which currently isn't implemented in mistral.rs.
Hello! Any plans for adding multimodal (e.g. llava) and embedding models?
I'm working on it now.chenwanqq/candle-llava It's not easy dude, tons of image preprocess and tensor concat.
I'm working on it now.chenwanqq/candle-llava It's not easy dude, tons of image preprocess and tensor concat.
Yes! Would you be willing to contribute your implementation here once it's done?
I'm working on it now.chenwanqq/candle-llava It's not easy dude, tons of image preprocess and tensor concat.
Yes! Would you be willing to contribute your implementation here once it's done?
Yes of course!
Yes of course!
Great, looking forward to that! I'm working on Idefics 2 here: #309.