Implement docker caching to not unnecessarily recompile dependencies

[aaronleopold]

Update: Feb 25, 2023

I think proper docker caching is just the way to go here. The only reason Stump takes so long to compile is because all of the dependencies, even if they have not changed, get recompiled each build. If I can get caching of the dependencies to work, then this will drastically decrease the build times.

I want to investigate potentially creating my own base images with some of the heavy, time-consuming dependencies precompiled. This will alleviate some of the headaches I have spending 2+ hours building arm images for every iterative development change. I would only have to rebuild these base images when I update the big dependencies (rocket, prisma, etc).

Similar to how stump pulls a base image dependent of the platform, I would need to create an image per supported platform.

This would have to live in a separate repository I think, so this issue would get transferred over there if any progress gets made.

[ShadowXCC]

Have you looked into using a crosscompiler? It will likely be faster than if you are using a Pi or something similar to do the cross compilation.

Sorry if this is what you are already doing.

[aaronleopold]

Hey @ShadowXCC, I don't actually use a Pi to compile for those platforms. I don't even want to think about how much longer that would take 😅

If you take a look at the Dockerfile, I use messense/rust-musl-cross base images and use buildx to build for multiple platforms in parallel. What I am looking for with this issue is seeing if I can just extend those images to have some of the rust dependencies precompiled in the image. This way, I only have to build the base image when I bump versions of those specific dependencies, and I don't need compile them each Stump release.

I have no clue if this is even possible, just a thought I had

[aaronleopold]

Could also try some sort of caching: https://blog.mgattozzi.dev/caching-rust-docker-builds/

Trying it out here https://github.com/aaronleopold/stump/tree/docker-opt-59

[aaronleopold]

My latest changes on that branch are close to working as I want. It seems to only work on the server app directly, meaning when I make a small change to apps/server/src/main.rs, it will use the cached deps. If I make a small change to core/src/lib.rs, it will rebuilt all the deps (ignoring the cache).

Next step is figuring out how to make it work in a way that only rebuilds deps when the dependencies between the two crates actually change, and otherwise just build the crates themselves.

First build: 597s Second build (with small change to server): 63s Third build (with small change to lib.rs): 600s

[tuxuser]

How/where are Docker image builds managed from, docker hub? The past links in this issue are not available anymore as the branches are non-existent.

[aaronleopold]

How/where are Docker image builds managed from, docker hub?

Yeah it goes to docker hub, you can find the docker things here: https://github.com/aaronleopold/stump/tree/develop/scripts/release

The past links in this issue are not available anymore as the branches are non-existent.

Yeah I didn't get anywhere with my attempts, I saved the Dockerfile though from that last commit of that now gone branch (note, it will be out of date):

# ------------------------------------------------------------------------------
# Frontend Build Stage
# ------------------------------------------------------------------------------

FROM node:16-alpine3.14 as frontend
ARG TARGETARCH

WORKDIR /app

# Note: I don't like copying ~everything~ but since I now use types exported from
# the core, and use pnpm specific means of accessing it via the workspace, I kind
# of need to maintain the structure of the workspace and use pnpm
COPY . .

RUN npm install -g pnpm

RUN pnpm i
RUN pnpm web build

RUN mv ./apps/web/dist build

# ------------------------------------------------------------------------------
# Cargo Build Stage
# ------------------------------------------------------------------------------

######################
### aarch64 / arm64 ##
######################

FROM messense/rust-musl-cross:aarch64-musl AS arm64-backend

WORKDIR /app

COPY .cargo .cargo
COPY . .

ENV CARGO_NET_GIT_FETCH_WITH_CLI=true

RUN rustup target add aarch64-unknown-linux-musl

RUN cargo build --package stump_server --bin stump_server --release --target aarch64-unknown-linux-musl && \
    cp target/aarch64-unknown-linux-musl/release/stump .

######################
### armv7 / arm/v7 ###
######################

# Note: the name here isn't entirely accurate to my understanding. But I can't figure
# out how to have the name be v7 inclusive so
FROM messense/rust-musl-cross:armv7-musleabihf@sha256:3e133558686fd5059ce25749cece40a81d87dad2c7a68727c36a1bcacba6752c AS arm-backend

WORKDIR /app

COPY .cargo .cargo
COPY . .

ENV CARGO_NET_GIT_FETCH_WITH_CLI=true

RUN rustup target add armv7-unknown-linux-musleabihf

RUN cargo build --package stump_server --bin stump_server --release --target armv7-unknown-linux-musleabihf && \
    cp target/armv7-unknown-linux-musleabihf/release/stump .

######################
### x86_64 / amd64 ###
######################

FROM messense/rust-musl-cross:x86_64-musl AS amd64-backend

WORKDIR /app

ENV CARGO_NET_GIT_FETCH_WITH_CLI=true

RUN rustup update && rustup target add x86_64-unknown-linux-musl

COPY .cargo .cargo

# Hack to make docker cache the build dependencies
COPY Cargo.toml .
# COPY Cargo.lock .
COPY core core
COPY apps/server/Cargo.toml apps/server/
RUN set -ex ;\
        mkdir apps/server/src ;\
        echo 'fn main() { println!("Wow, such empty!"); }' > apps/server/src/main.rs ;\
        # Run sed command to remove the lines "core/integration-tests", "apps/desktop/src-tauri and "apps/tui" from Cargo.toml
        # FIXME: this nice one liner doesn't work with the new lines in the Cargo.toml... I want to eventually fix this
        # sed -i 's/members = \[.*\]/members = \[ "core", "core\/prisma", "apps\/server" \]/' Cargo.toml ;\
        sed -i '/core\/integration-tests/d' Cargo.toml; \
        sed -i '/apps\/desktop\/src-tauri/d' Cargo.toml; \
        sed -i '/apps\/tui/d' Cargo.toml; \
        # sed -i 's/stump_core = { path = "..\/..\/core" }/#stump_core = { path = "..\/..\/core" }/' apps/server/Cargo.toml; \
        cargo build --package stump_server --bin stump_server --release --target x86_64-unknown-linux-musl; \
        # sed -i 's/#stump_core = { path = "..\/..\/core" }/stump_core = { path = "..\/..\/core" }/' apps/server/Cargo.toml
        rm -rf apps/server/src

COPY . .

RUN sed -i '/core\/integration-tests/d' Cargo.toml; \
        sed -i '/apps\/desktop\/src-tauri/d' Cargo.toml; \
        sed -i '/apps\/tui/d' Cargo.toml

RUN cargo build --package stump_server --bin stump_server --release --target x86_64-unknown-linux-musl && \
    cp target/x86_64-unknown-linux-musl/release/stump_server ./stump

######################
## Conditional step ##
######################

# Conditional to skip non-targetarch build stages
FROM ${TARGETARCH}-backend AS core-builder

# ------------------------------------------------------------------------------
# Final Stage
# ------------------------------------------------------------------------------
FROM alpine:latest

# libc6-compat
RUN apk add --no-cache libstdc++ binutils

# Create the user/group for stump
RUN addgroup -g 1000 stump
RUN adduser -D -s /bin/sh -u 1000 -G stump stump

WORKDIR /

# create the config, data and app directories
RUN mkdir -p config && \
    mkdir -p data && \
    mkdir -p app

# FIXME: this does not seem to be working...
# make the stump user own the directories
RUN chown stump /config && \
    chown stump /data && \
    chown stump /app

USER stump

# copy the binary
COPY --chown=stump:stump --from=core-builder /app/stump ./app/stump

# copy the react build
COPY --from=frontend /app/build ./app/client

# TODO: replace this with something more elegant lol maybe a bash case statement
RUN ln -s /lib/ld-musl-aarch64.so.1 /lib/ld-linux-aarch64.so.1; exit 0

# Default Stump environment variables
ENV STUMP_CONFIG_DIR=/config
ENV STUMP_CLIENT_DIR=/app/client
ENV STUMP_PROFILE=release
ENV STUMP_PORT=10801
ENV STUMP_IN_DOCKER=true

WORKDIR /app

CMD ["./stump"]

[tuxuser]

How/where are Docker image builds managed from, docker hub?

Yeah it goes to docker hub, you can find the docker things here: https://github.com/aaronleopold/stump/tree/develop/scripts/release

Am I understanding correctly - Docker hub is building from a single Dockerfile -> images are build sequentially instead of parallel?

[aaronleopold]

I use buildx to build for multiple platforms in parallel, not sequentially. Then, those images get pushed to https://hub.docker.com/r/aaronleopold/stump-preview. I rarely build the ARM image because without any caching of dependencies it takes over 2 hours. The command for just amd64 would be:

docker buildx build -f ./scripts/release/Dockerfile --push --platform=linux/amd64 -t aaronleopold/stump-preview:latest .

Eventually, the ideal would be to have the CI do this for releases (releases from merges on main, and potentially nightly from develop)

[aaronleopold]

cargo-chef might be worth looking into for this

[aaronleopold]

This issue is very out of date 😓 but def needs work still. Taking a look at a recent build, e.g. https://github.com/stumpapp/stump/actions/runs/7944155485/job/21689335390?pr=273, caching for ~900 MB of cargo deps each time takes way too long