Daniel J Walsh

I believe this is fixed now, closing.

@flouthoc PTAL

Yes go ahead. Flouthoc is not a full time student, so he has limited time to contribute. You can take it over.

@ygalblum could you rebase and repush.

Please run with debug to show us the podman line. I believe the problem is we are not grabbing the vllm image when attempting to run vllm. I tried this...

docker.io/vllm/vllm-openai:latest

This got me further: $ ramalama --image **docker.io/vllm/vllm-openai:latest** --debug --runtime vllm serve -c 16384 --temp 0.8 --ngl 999 --threads 9 --host 0.0.0.0 --port 9999 --device /dev/accel --device /dev/dri granite3-dense:8b exec_cmd:...

@robertgshaw2-redhat any idea what is going on here, or someone who could check?

No it looks like vllm requires that this device exists in its environment at least with the image I pulled.

I do not see this with the ramalama is main branch, I do see that --runtime=vllm is not working though. ramalama --runtime=vllm serve granite3-dense ERROR (catatonit:50): failed to exec pid1:...