runpod-workers
Are visual LLMs supported?
I am trying to run https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct on serverless but it throws error
2024-10-09 18:06:06.525
[my1jfmdyltrax5]
[error]
worker exited with exit code 1
2024-10-09 18:05:50.528
[my1jfmdyltrax5]
[error]
worker exited with exit code 1
2024-10-09 18:05:34.515
[my1jfmdyltrax5]
[error]
worker exited with exit code 1
2024-10-09 18:05:19.851
[my1jfmdyltrax5]
[error]
worker exited with exit code 1
2024-10-09 18:04:18.168
[eau4o046d9uc4d]
[error]
worker exited with exit code 1
2024-10-09 18:04:01.663
[eau4o046d9uc4d]
[error]
worker exited with exit code 1
2024-10-09 18:03:45.158
[eau4o046d9uc4d]
[error]
worker exited with exit code 1
2024-10-09 18:03:28.734
[eau4o046d9uc4d]
[error]
worker exited with exit code 1
2024-10-09 18:03:12.261
[eau4o046d9uc4d]
[error]
worker exited with exit code 1
2024-10-09 18:02:55.743
[eau4o046d9uc4d]
[error]
worker exited with exit code 1
2024-10-09 18:02:39.224
[eau4o046d9uc4d]
[error]
worker exited with exit code 1
multi-modal models like Qwen2-VL is supported in pods, but it seems not in serverless endpoints. see my comment here: https://github.com/runpod-workers/worker-vllm/issues/114#issuecomment-2412967748
yes, this is fixed in the dev branch of vllm which when released in the next version should be fixed for endpoints as well.
Hi there!, At the moment we do not support visual-LLMs, but we are working on support for these in the future
It is supported now!
what model did you manage to run? I've tried multiple ones but none seems to work with the current vllm-worker
