mlc-llm
mlc-llm copied to clipboard
mlc-ai
Universal LLM Deployment Engine with ML Compilation
## ⚙️ Request New Models Huggingface implementation: https://huggingface.co/urchade/gliner_multi-v2.1 Is this model architecture supported by MLC-AI? Context: GLiNER (Generalist and Lightweight Model for Named Entity Recognition) is a NER model designed...
## 🐛 Bug It looks like all supported Gemma 2 models are failing right now. ## To Reproduce ```python from mlc_llm import MLCEngine # Create engine model = "HF://mlc-ai/gemma-2-2b-it-q4f16_1-MLC" engine...
## ❓ General Questions I followed the instructions to build for Orange Pi. https://blog.mlc.ai/2024/04/20/GPU-Accelerated-LLM-on-Orange-Pi I have the Radxa Rock 5B with the same SoC, but I think that should be...
I have errors compiling TVM with cuda 12.8 
## ❓ General Questions The Android app is crashing with an exception - ``` channel '1cce162 ai.mlc.mlcchat/ai.mlc.mlcchat.MainActivity (server)' ~ Channel is unrecoverably broken and will be disposed! 2025-01-15 20:28:24.710 1424-1541...
I checked multiple converted deepseek r1 distill qwen 1.5B models on MLCChat app on iPhone 15 Plus and Google Pixel 8 pro. But all of them have a very high...
## ⚙️ Request New Models - Link to an existing implementation (e.g. Hugging Face/Github): - Is this model architecture supported by MLC-LLM? (the list of [supported models](https://llm.mlc.ai/docs/prebuilt_models.html)) ## Additional context...
When I converted the mistralai/Mistral-Nemo-Instruct-2407 model and used it later, I found that it was basically unable to answer Chinese questions. I checked and found that in the converted file,...
Hello MLC-LLM team, I would appreciate it if you could implement KV cache offloading in the near future. Thanks in advance!
Comprehensive CPP Cli developed over existing JSONEngine interface. Intended to use in environments with only cli access like Android ADB shells.
