iv-org
dockerfile: compile openssl instead of using the one bundled on the crystal alpine image.
May fix https://github.com/iv-org/invidious/issues/1438
Supersede, close #5336
This has been running on my instance for a long time now and the memory usage has been reduced significantly as it can be seen here: https://github.com/iv-org/invidious/issues/1438#issuecomment-3190651286
Although it still increases slowly for some reason, is way better than letting it crash by out of memory.
The memory drops are from Docker daemon restarts while I was configuring some things, Invidious didn't crash by OOM and I didn't restart it manually either
Since this graphs are from my fork that has some added things that upstream Invidious doesn't have, this may completely fix the memory leak, but I'm not really sure.
I haven't tested arm64 at all so I didn't make any changes to docker/Dockerfile.arm64
I would be interested in the ARM64 version :)
There it is, feel free to test it
I wonder if compiling OpenSSL ourselves is really the better option compared to just switching the image to be based off of Debian or something
Yeah it's most likely a better idea. Though we should see if it doesn't increase the final docker image size by too much.
I found this docker image which offer amd64 and arm64 with either debian or ubuntu: https://hub.docker.com/r/84codes/crystal/tags. Found it here: https://github.com/crystal-lang/distribution-scripts/issues/125#issuecomment-1298548759. They also have .deb files: https://packagecloud.io/84codes/crystal
Could finally unify our Dockerfile for both cpu architectures.
84codes is also one of Crystal's main sponsors so it should be stable for the foreseeable future too.
Though the Debian images still rely on a static Crystal compiler compiled on Alpine... https://github.com/84codes/crystal-packages/blob/main/debian-static/Dockerfile so I wonder if that can potentially cause this leak to resurface. To my knowledge it shouldn't but this leak is also really strange.
I have not personally found anything else distributing debian crystal with arm64 support. Everything else is only for amd64. https://packages.debian.org/trixie/crystal https://build.opensuse.org/project/show/devel:languages:crystal https://launchpad.net/ubuntu/+source/crystal
But indeed it would be better to have full glibc. I wonder if their .deb is full glibc or also based on alpine: https://packagecloud.io/84codes/crystal
I'll take some time to make a Debian based Dockerfile and see how well it works, I'm also worried about the final image size, but if is not that much, that will be fine. The current image seems to weight about ~75MB
I'll take some time to make a Debian based Dockerfile and see how well it works, I'm also worried about the final image size, but if is not that much, that will be fine. The current image seems to weight about ~75MB
It now weights 225MB, too much :/ Here is the Dockerfile modified: https://github.com/Fijxu/invidious/commit/d954a5f25228d0eb3d43e49b5b3b3ebe1fcf3aa1
There is https://github.com/84codes/crystal-packages/pull/35, so I will try to use LibreSSL instead on my Invidious instance and see how's the memory usage
I have squeezed 20MB by replacing your apt by. So I got 204MB.
RUN apt-get update && \
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
librsvg2-bin \
fonts-open-sans \
tini \
tzdata \
libyaml-0-2 \
libsqlite3-0 \
libssl3 \
liblzma5 \
ca-certificates && \
apt-get clean && \
rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
I used dive to check what's using the most of MB. It's the base system and the apt-get install.
75 MB FROM blobs
96 MB RUN /bin/sh -c apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install
27 MB COPY /invidious/invidious .
200MB uncompressed it's not that bad to be fair. Invidious companion size uncompressed is 133MB.
There is 84codes/crystal-packages#35, so I will try to use LibreSSL instead on my Invidious instance and see how's the memory usage
I'm now testing LibreSSL on my instance without memory limits to see how's the memory usage ;)
Oh well, switching to LibreSSL doesn't seem to work, memory just raises and raises:
I'll try the Debian based image and see how many memory it uses after a few minutes of use.
For now, compiling OpenSSL ourselves seem to be the best option.
I'll try the Debian based image and see how many memory it uses after a few minutes of use.
It also leaks, it used up to 1.63GB on an Invidious process that is just used to receive feed updates from Youtube PubSub. With self compiling OpenSSL it never used more than 400MB
Which flags do you use when compiling openssl?
Just ./Configure --openssldir=/etc/ssl && make from https://forum.crystal-lang.org/t/memory-usage-of-http-one-shot-vs-maintaining-a-persistent-connection/8237/3?u=fijxu
It now weights 225MB, too much :/ Here is the Dockerfile modified: Fijxu@d954a5f
There is 84codes/crystal-packages#35, so I will try to use LibreSSL instead on my Invidious instance and see how's the memory usage
What a about using a static glibc binary with alpine as the final image? The size of my arm64 image is 109MB.
RUN --mount=type=cache,target=/root/.cache/crystal if [[ "${release}" == 1 ]] ; then \
crystal build ./src/invidious.cr \
--release \
--warnings all \
--static \
--cross-compile; \
else \
crystal build ./src/invidious.cr \
--warnings all \
--static \
--cross-compile; \
fi
RUN cc ./invidious.o -o ./invidious \
-rdynamic -static \
-Wl,--gc-sections \
-lyaml -lxml2 -licuuc -licudata -lstdc++ -lgcc -lz -llzma \
-lsqlite3 -lssl -lcrypto -lpcre2-8 -lm -lpthread -ldl \
-L/usr/lib/crystal -lgc
I added --cross-compile and do the linking myself so I can save time debugging the linking stage, which is error-prone when building a static binary.
One thing to note is that the binary built on Debian looks /usr/lib/ssl/cert.pem (instead of /etc/ssl/certs/ca-certificates.crt) for CA certificates.
RUN mkdir -p /usr/lib/ssl/
RUN ln -s /etc/ssl/certs/ca-certificates.crt /usr/lib/ssl/cert.pem
Complete Dockerfile
FROM 84codes/crystal:1.16.3-debian-12 AS builder
RUN apt update && \
apt install -y --no-install-recommends --no-install-suggests \
libsqlite3-dev libyaml-dev libssl-dev \
libxml2-dev liblzma-dev \
libicu-dev libstdc++-12-dev
ARG release
WORKDIR /invidious
COPY ./shard.yml ./shard.yml
COPY ./shard.lock ./shard.lock
RUN shards install --production
COPY ./src/ ./src/
# TODO: .git folder is required for building – this is destructive.
# See definition of CURRENT_BRANCH, CURRENT_COMMIT and CURRENT_VERSION.
COPY ./.git/ ./.git/
# Required for fetching player dependencies
COPY ./scripts/ ./scripts/
COPY ./assets/ ./assets/
COPY ./videojs-dependencies.yml ./videojs-dependencies.yml
RUN crystal spec --warnings all \
--link-flags "-lxml2 -llzma"
# Use bash to avoid syntax error on Debian (/bin/sh: 1: [[: not found)
SHELL ["bash","-c"]
RUN --mount=type=cache,target=/root/.cache/crystal if [[ "${release}" == 1 ]] ; then \
crystal build ./src/invidious.cr \
--release \
--warnings all \
--static \
--cross-compile; \
else \
crystal build ./src/invidious.cr \
--warnings all \
--static \
--cross-compile; \
fi
RUN cc ./invidious.o -o ./invidious \
-rdynamic -static \
-Wl,--gc-sections \
-lyaml -lxml2 -licuuc -licudata -lstdc++ -lgcc -lz -llzma \
-lsqlite3 -lssl -lcrypto -lpcre2-8 -lm -lpthread -ldl \
-L/usr/lib/crystal -lgc
RUN ls -l ./invidious
RUN ldd ./invidious || true
FROM alpine:3.22
RUN apk add --no-cache rsvg-convert ttf-opensans tini tzdata
WORKDIR /invidious
RUN addgroup -g 1000 -S invidious && \
adduser -u 1000 -S invidious -G invidious
COPY --chown=invidious ./config/config.* ./config/
RUN mv -n config/config.example.yml config/config.yml
RUN sed -i 's/host: \(127.0.0.1\|localhost\)/host: invidious-db/' config/config.yml
COPY ./config/sql/ ./config/sql/
COPY ./locales/ ./locales/
COPY --from=builder /invidious/assets ./assets/
COPY --from=builder /invidious/invidious .
RUN chmod o+rX -R ./assets ./config ./locales
RUN mkdir -p /usr/lib/ssl/
RUN ln -s /etc/ssl/certs/ca-certificates.crt /usr/lib/ssl/cert.pem
EXPOSE 3000
USER invidious
ENTRYPOINT ["/sbin/tini", "--"]
CMD [ "/invidious/invidious" ]
ps.
The Debian image from 84codes does use dynamic glibc Crystal compiler. You can check with ldd.
docker run --rm --entrypoint bash 84codes/crystal:1.16.3-debian-12 -c 'ldd $(which crystal shards)'
ldd result
/usr/bin/crystal:
linux-vdso.so.1 (0x0000e2c2585d5000)
libLLVM.so.19.1 => /lib/aarch64-linux-gnu/libLLVM.so.19.1 (0x0000e2c250340000)
libpcre2-8.so.0 => /lib/aarch64-linux-gnu/libpcre2-8.so.0 (0x0000e2c250290000)
libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000e2c2501f0000)
libffi.so.8 => /lib/aarch64-linux-gnu/libffi.so.8 (0x0000e2c2501c0000)
libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000e2c250180000)
libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000e2c24ffd0000)
libedit.so.2 => /lib/aarch64-linux-gnu/libedit.so.2 (0x0000e2c24ff70000)
libz3.so.4 => /lib/aarch64-linux-gnu/libz3.so.4 (0x0000e2c24ea40000)
libz.so.1 => /lib/aarch64-linux-gnu/libz.so.1 (0x0000e2c24ea00000)
libzstd.so.1 => /lib/aarch64-linux-gnu/libzstd.so.1 (0x0000e2c24e940000)
libxml2.so.2 => /lib/aarch64-linux-gnu/libxml2.so.2 (0x0000e2c24e780000)
libstdc++.so.6 => /lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000e2c24e560000)
/lib/ld-linux-aarch64.so.1 (0x0000e2c258590000)
libtinfo.so.6 => /lib/aarch64-linux-gnu/libtinfo.so.6 (0x0000e2c24e510000)
libbsd.so.0 => /lib/aarch64-linux-gnu/libbsd.so.0 (0x0000e2c24e4d0000)
libicuuc.so.72 => /lib/aarch64-linux-gnu/libicuuc.so.72 (0x0000e2c24e2b0000)
liblzma.so.5 => /lib/aarch64-linux-gnu/liblzma.so.5 (0x0000e2c24e260000)
libmd.so.0 => /lib/aarch64-linux-gnu/libmd.so.0 (0x0000e2c24e230000)
libicudata.so.72 => /lib/aarch64-linux-gnu/libicudata.so.72 (0x0000e2c24c440000)
/usr/bin/shards:
linux-vdso.so.1 (0x0000e25b3a7b4000)
libyaml-0.so.2 => /lib/aarch64-linux-gnu/libyaml-0.so.2 (0x0000e25b3a4c0000)
libpcre2-8.so.0 => /lib/aarch64-linux-gnu/libpcre2-8.so.0 (0x0000e25b3a410000)
libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000e25b3a3d0000)
libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000e25b3a220000)
/lib/ld-linux-aarch64.so.1 (0x0000e25b3a770000)
The debian-static is probably for building .deb packages for "Any Deb based distributions" as on https://packagecloud.io/84codes/crystal.
Edit: It might be possible to squeeze further by making it distroless with static rsvg-convert and tini-static. The rest (ttf-opensans, tzdata) are just architecture-independent files. In fact, my private instance is running distroless without rsvg-convert, ttf-opensans or even tzdata, since there is no need for new accounts. If I ever need a new account I can pull the official image to create one, and switch back to mine later anyway.
Maybe a stupid question: has anyone tried to incrementally add the configure options from Alpine's APKBUILD file, to determine which one is in cause and so be able to upstream a fix? https://gitlab.alpinelinux.org/alpine/aports/-/blob/3.22-stable/main/openssl/APKBUILD#L167-187
Maybe a stupid question: has anyone tried to incrementally add the configure options from Alpine's APKBUILD file, to determine which one is in cause and so be able to upstream a fix? https://gitlab.alpinelinux.org/alpine/aports/-/blob/3.22-stable/main/openssl/APKBUILD#L167-187
For that we would need to create a POC Crystal program to test it's memory usage against different flags, because it takes around ~30 minutes (or less, depends of the traffic of the instance) to make a single Invidious process go beyond 1GB of memory. Since I applied this on my instance, the memory of my 4 processes has never used more than 1GB of memory constantly
I'm testing this PR and it seems there's no large spike on ram usage over 2 days so far (though mine is not public instance so not sure for public ones).
Is there any way to do stress test or imitate the behavior of public instance?
(8GB Raspi4)
Got inspired by this idea and decided to see what we can compile further. As well as a SVG symlink.
Image size is 68~ MB now with these tweaks (which is still quite small!), and I don't actively monitor via graph but I've noticed a decrease of memory with this below Dockerfile.
I'm still not sure if it fixes the memory leak, but it's worth a try. Since this hasn't been extensively tested (and my fork is kinda a mess), I'm not making a pull request just yet.
# syntax=docker/dockerfile:1.4
ARG OPENSSL_VERSION=3.5.2
ARG ZLIB_VERSION=1.3.1
ARG XZ_VERSION=5.6.2
ARG LIBXML2_VERSION=2.14.5
FROM mirror.gcr.io/84codes/crystal:1.16.3-alpine AS builder
ARG OPENSSL_VERSION
ARG ZLIB_VERSION
ARG XZ_VERSION
ARG LIBXML2_VERSION
ENV PREFIX=/usr/local
ENV PATH=$PREFIX/bin:$PATH
ENV PKG_CONFIG_PATH=$PREFIX/lib/pkgconfig:$PKG_CONFIG_PATH
# Build deps only in builder
RUN apk add --no-cache \
build-base curl perl linux-headers autoconf automake libtool pkgconfig musl-dev \
pngquant jpeg-dev libpng-dev freetype-dev fontconfig-dev \
sqlite-static yaml-static rsvg-convert xz
WORKDIR /usr/src
### Build zlib (static)
RUN set -eux; \
curl -fsSL "https://zlib.net/zlib-${ZLIB_VERSION}.tar.gz" | tar xz; \
cd "zlib-${ZLIB_VERSION}"; \
./configure --prefix=$PREFIX; \
make -j$(nproc); \
make install
### Build xz (liblzma) (static)
RUN set -eux; \
curl -fsSL "https://tukaani.org/xz/xz-${XZ_VERSION}.tar.gz" | tar xz; \
cd "xz-${XZ_VERSION}"; \
./configure --prefix=$PREFIX --disable-shared --enable-static; \
make -j$(nproc); \
make install
### Build libxml2 (static) — use .tar.xz and tar xJ
RUN set -eux; \
curl -fsSL "https://download.gnome.org/sources/libxml2/2.14/libxml2-${LIBXML2_VERSION}.tar.xz" | tar xJ; \
cd "libxml2-${LIBXML2_VERSION}"; \
./configure --prefix=$PREFIX --without-python --enable-static --disable-shared --with-zlib=$PREFIX; \
make -j$(nproc); \
make install
### Build OpenSSL into /usr/local
RUN set -eux; \
curl -fsSL "https://github.com/openssl/openssl/releases/download/openssl-${OPENSSL_VERSION}/openssl-${OPENSSL_VERSION}.tar.gz" | tar xz; \
cd "openssl-${OPENSSL_VERSION}"; \
./Configure linux-x86_64 --prefix=$PREFIX --openssldir=/etc/ssl && \
make -j$(nproc) && make install_sw
WORKDIR /invidious
COPY ./shard.yml ./shard.yml
COPY ./shard.lock ./shard.lock
RUN shards install --production
COPY ./src/ ./src/
COPY ./.git/ ./.git/
COPY ./scripts/ ./scripts/
COPY ./assets/ ./assets/
COPY ./config/config.* ./config/
COPY ./config/sql/ ./config/sql/
COPY ./locales/ ./locales/
COPY ./videojs-dependencies.yml ./videojs-dependencies.yml
RUN mkdir -p /invidious/config/sql /invidious/locales
RUN mkdir -p /invidious/assets/raster; \
for svg in $(find assets -name '*.svg' || true); do \
out="/invidious/assets/raster/$(basename "${svg%.*}.png")"; \
rsvg-convert "$svg" -o "$out" || echo "warning: rsvg-convert failed $svg"; \
done
RUN --mount=type=cache,target=/root/.cache/crystal \
PKG_CONFIG_PATH=$PREFIX/lib/pkgconfig \
crystal build ./src/invidious.cr \
--release -s -p -t --mcpu=x86-64-v2 \
--static --warnings all \
--link-flags "-lxml2 -llzma -lz -lssl -lcrypto -ldl -lpthread -lm"
RUN file ./invidious && readelf -h ./invidious || true
FROM mirror.gcr.io/alpine:3.22 AS runtime
RUN apk del --no-cache openssl openssl-dev || true
RUN apk add --no-cache tini tzdata ttf-opensans ca-certificates
WORKDIR /invidious
RUN addgroup -g 1000 -S invidious && \
adduser -u 1000 -S invidious -G invidious
COPY --from=builder --chown=invidious /invidious/invidious .
COPY --from=builder --chown=invidious /invidious/assets/raster ./assets/raster
COPY --from=builder --chown=invidious /invidious/assets ./assets
COPY --from=builder --chown=invidious /invidious/config/config.* ./config/
COPY --from=builder --chown=invidious /invidious/config/sql ./config/sql
COPY --from=builder --chown=invidious /invidious/locales ./locales
RUN mv -n config/config.example.yml config/config.yml && \
sed -i 's/host: \(127.0.0.1\|localhost\)/host: invidious-db/' config/config.yml && \
chmod o+rX -R ./assets ./config ./locales
EXPOSE 3000
USER invidious
ENTRYPOINT ["/sbin/tini", "--"]
CMD [ "/invidious/invidious" ]
