On A770，vllm and llama.cpp which brings better performance for MiniCPM-v-2.6 ?

[yangqing-yq]

Any data table for benchmark?

[liu-shaojun]

Currently, our VLLM does not support multimodal models. Support for multimodal models is ongoing in the 0.5.x version of VLLM. We will notify you once it's ready.

[yangqing-yq]

Is there a schedule when we could get a vllm supported version to run MiniCPM-v ？

[glorysdj]

Is there a schedule when we could get a vllm supported version to run MiniCPM-v ？

hi, the upgrade of IPEX-LLM vLLM to 0.5.x is in progress, we will let you know once it's ready, and we will also try to support MiniCPM-v in the new version.

[yangqing-yq]

Check progress, how is the status of vllm supporting MLLM like MiniCPM-V-2.6? @glorysdj

[glorysdj]

the upgrade of IPEX-LLM vLLM to 0.5.4 is finished, and MiniCPM-V-2.6 is supported, please refer to

https://github.com/intel-analytics/ipex-llm/tree/main/python/llm/example/GPU/vLLM-Serving#image-input