alpine-docker
Include AVX Runner
without AVX runner the performance is going to be crippled as discussed in https://github.com/ollama/ollama/issues/2821
yes, I am working on it.
The problem here is, I need use same Dockerfile, build for both ARM64 and AMD64.
https://github.com/alpine-docker/ollama/blob/master/Dockerfile#L10-L12
I commented out below lines
# In arm64 ollama/ollama image, there is no avx libraries and seems they are not must-have (#2903, #3891)
COPY --from=ollama /usr/lib/ollama/runners/cpu_avx /usr/lib/ollama/runners/cpu_avx
COPY --from=ollama /usr/lib/ollama/runners/cpu_avx2 /usr/lib/ollama/runners/cpu_avx2
But in ARM build, above two folders are not exist.
In Dockerfile , COPY doesn't support wildcard yet, I don't get folders, but files under destination folder
COPY --from=ollama /usr/lib/ollama/runners/cpu_* /usr/lib/ollama/runners/
Need find way to add them back
@ozbillwang The prototype of one file for both platforms. Contains CPU optimized build targeted for host arch (NEON - arm64, AVX SSE etc - amd64):
FROM ollama/ollama AS base
FROM ubuntu:20.04
RUN apt-get update \
&& apt-get install -y ca-certificates \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
COPY --from=base /bin /usr/bin
COPY --from=base /lib/ollama/*.so /usr/lib/ollama/
ENV PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ENV OLLAMA_HOST=0.0.0.0:11434
EXPOSE 11434
ENTRYPOINT ["/bin/ollama"]
CMD ["serve"]
It has a minor ~(imho)~ con - final size is roughly 140mb
Basic idea behind this is to remove all gpu related stuff.
docker run --rm --entrypoint bash ollama/ollama -c 'du -sxmc /usr/lib/ollama/*'
I did not try alpine and I don't think it's a good idea, honestly
The thing is it's highly reduced runtime and musl lib, which can lead to performance degradation and bugs
And it is also quiet painful to maintain dependencies
I'm surprised the official ollama/ollama is still based on Ubuntu 20.04 which will reach end of life this month.