Aaron Pham

Can you dumped the whole stack trace in a new issue?

Containerizing Bento with podman should already be supported. See https://docs.bentoml.com/en/latest/guides/containerization.html#containerization-with-different-container-engines ```bash bentoml containerize llm-bento --backend podman --opt ... ```

Tho there is an internal bug that I just discovered recently wrt running within the container. I will post updates about this soon

> Hey @aarnphm, I ran into the exact same behavior. I tried to deploy openllm within a podman container, with registry.redhat.io/ubi8/python-39:latest as base image. Are there plans for containerizing openllm...

Version 0.0.209 of langchain has include OpenLLM integration. Please try it out

Did you login with huggingface hub?

Can you try with the latest version to see if you can still run into these issues?

at least the tests were coverred for linux and macos

we need to support CPU (esp GGUF and GGML for it to run on MacOS) This is on our roadmap but currently not very high priority atm.

qq: Can you check if on Linux the following path exists `/opt/rocm/libexec/rocm_smi`?