Zihao Ye
I have already mentioned, all models under mlc-ai (including the mlc-ai/demo-vicuna-v1-7b-int4 you used) are already compiled by MLC-LLM, and you should find some pre-compiled models in raw huggingface format (like...
> Tagging along here, i tried with my iPhone 13 with testflight, it also crashes after showing the system initializing message. iPhone 13 does not have enough RAM size (6GB...
Another future working item is to transform the conversion template into a YAML/JSON file we can load from disk, rather than hardcoded in C++ file.
Duplicate of #357
No DGL still uses CuSparse.
Sounds good to me.
Hi @MarcelDelhez, what do we mean by "tuning" here is tuning the kernel performance (to be faster) instead of fine-tuning weights. Support fine-tuning in MLC-LLM is indeed very important but...
> It seems there is no official support for interacting with MLC LLM using python api yet Actually, there is one: https://github.com/mlc-ai/mlc-llm/blob/main/tests/chat.py
Some of the functionalities for dynamic shape have not been upstreamed yet, please use the [relax repo](https://github.com/mlc-ai/relax) for now.
Hi @lucasjinreal , to build models on your own, you should clone the models' original huggingface repository, for example, for LLaMA, you should: ``` git clone https://huggingface.co/decapoda-research/llama-7b-hf.git dist/models/llama-7b-hf ``` The...