swift icon indicating copy to clipboard operation
swift copied to clipboard

Published 20 hours ago • verified publisher icon modelscope

API support for multi-modal model inference

Open babla9 opened this issue 1 month ago • 1 comments

Current code only supports single or batch inference for multi-modal models (Llava1.6, cogvlm etc) due to lack of vllm support. Any plans to add feature support to enable API support for these models? Maybe with something like https://github.com/sgl-project/sglang?

babla9 avatar May 12 '24 01:05 babla9