ollama
ollama copied to clipboard
ollama
Make a second docker image for "NVidia GPUs"
I wanted to setup Ollama, (great project, the CPU variant ran out of the box and I had it doing its thing, deserve a 🥇 ), HOWEVER, I have spent 7 hours flailing with docker, (I am on a Windows 11 machine with a 4070 TI card), to get this to work.
I even had chatGPT, Bing Copilot, Rider AI Assistant and Linux Guru friend help me, and no joy, I just can't get docker with the nvidia toolkit and Ollama to run togheter in a container
PLEASE make a "ready to run" docker image that is already 100% ready to go for "Nvidia GPU mode", because I am probably missing something, but either its deprecated dependencies, or something else, and the simple solution here is to have multiple docker images with dedicated "optimizations".
Thank you for an awesome platform for hosting local AI models 😸
Edit: To be clear, we the WSL installation variant worked great
Not directly addressing the Docker image ask, but FYI, we're working on a native Windows install which might suit your needs. It's not quite ready yet, but if you're comfortable building from source, you can try it out on main. https://github.com/jmorganca/ollama/blob/main/docs/development.md#windows
Not directly addressing the Docker image ask, but FYI, we're working on a native Windows install which might suit your needs. It's not quite ready yet, but if you're comfortable building from source, you can try it out on main. https://github.com/jmorganca/ollama/blob/main/docs/development.md#windows
My bad for not stating it clearer, I got it running, and have been having lots of fun, I just was frustrated by the rabbithole of wasted time trying to get it to work with Docker, the WSL-variant works, and my GPU have never been so loud over such a long time before 😃
The current docker image should work out of the box with CUDA provided the prerequisites (nvidia-container-toolkit and --gpus=all) are met. If that's not the case, please describe how you're running the docker container and what errors you're seeing
The current docker image should work out of the box with CUDA provided the prerequisites (nvidia-container-toolkit and
--gpus=all) are met. If that's not the case, please describe how you're running the docker container and what errors you're seeing
Well, the toolkit refuses to install on the docker image as provided when following the official Nvidia instructions: Link.
My errors are that "sudo could not be found", then I install sudo, then something else is no longer supported in det package dependency tree, and then I get a apt-get error that sources are not found or the file is corrupted. I met so much friction trying to do this, that I was concidering just hooking up my Semantic Kernal project to OpenAI APIs and burn money, instead of running Ollama 😆
I would have saved hours of frustration if the docker image included the setup of cuda already, and I the most user-friendly variant of that, is to have two docker images, where one is CPU and one is GPU.
The nvidia-container-toolkit must be installed on the Docker host, Windows WSL2 in your case. It's required for Docker to expose the GPU to the container. The Ollama Docker image contains the runtime requires to use an NVIDIA GPU but if the GPU isn't passed through Docker, it's as if there's no GPU installed.
See this for more information
The nvidia-container-toolkit must be installed on the Docker host, Windows WSL2 in your case. It's required for Docker to expose the GPU to the container. The Ollama Docker image contains the runtime requires to use an NVIDIA GPU but if the GPU isn't passed through Docker, it's as if there's no GPU installed.
See this for more information
... In my defense it didn't say "on the host" explicitly anywhere, but that small detail has probably cost me a couple of years of my lifespan in frustrations 😆