OOM during sysimage creation

[ValentinKaisermayer]

I'm trying to package a Julia app as a docker image. Since, as of writing this, package relocatability is not solved; hence, this is the only way to share an app.

In my dockerfile I run:

###############################################################################################
# Stage 1: Build Stage
FROM julia:1.11.2-bullseye AS builder

# Install C compiler required for PackageCompiler and clean up afterward to reduce image size
RUN apt-get update && \
    apt-get install -y --no-install-recommends g++ && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# Set environment for Julia compilation
ENV JULIA_CPU_TARGET=generic

# Create a non-root user and set up directories
RUN useradd --create-home --shell /bin/bash jl && mkdir /home/jl/app && chown -R jl:jl /home/jl/
WORKDIR /home/jl/app

# Switch to non-root user
USER jl

ENV JULIA_DEPOT_PATH "/home/jl/.julia"
ENV JULIA_REVISE "off"
ENV EARLYBIND "true"
ENV JULIA_NUM_PRECOMPILE_TASKS "4"

# Copy dependency files first to leverage Docker caching for dependencies
COPY --chown=jl:jl Project.toml .
COPY --chown=jl:jl deps/ deps/
RUN julia --project -e "using Pkg; Pkg.instantiate(); Pkg.precompile();"

# Copy the full application code and precompile with sysimage
COPY --chown=jl:jl compiled/ compiled/
COPY --chown=jl:jl App/ App/
# copy test data so that precompiling works
COPY --chown=jl:jl data/ data/
RUN julia --project -t auto -O3 --startup-file=no compiled/make.jl

###############################################################################################
# Stage 2: Production Stage
FROM debian:bullseye-slim  
# Install curl (for BinaryProvider) compiler required for PackageCompiler and clean up afterward to reduce image size
RUN apt-get update && \
    apt-get install -y --no-install-recommends curl && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# Set up a non-root user for running the app
RUN useradd --create-home --shell /bin/bash jl && mkdir /home/jl/app && chown -R jl:jl /home/jl/
WORKDIR /home/jl/app
USER jl

# Set environment variables
ENV JULIA_REVISE "off"
ENV EARLYBIND "true"

# Copy only essential files from the build stage to the production stage
COPY --from=builder /home/jl/app/compiled/App /home/jl/app

# Since Julia's relocatability depends on each package, simply make a symlink to the artifacts directory in the compiled app
RUN mkdir -p /home/jl/.julia/artifacts && ln -s /home/jl/app/share/julia/artifacts /home/jl/.julia/artifacts

# Command to run the application
ENTRYPOINT ["/home/jl/app/bin/App"]
CMD ["--julia-args", "-t", "auto", "-O3", "--startup-file=no"]

and my make.jl file is:

using PackageCompiler

PackageCompiler.create_app(
    "App",
    "compiled/App";
    cpu_target="generic",
    sysimage_build_args=`-O3`,
    precompile_execution_file="compiled/precompile.jl",
    include_transitive_dependencies=false,
    include_lazy_artifacts=true,
    incremental=true,
    # filter_stdlibs=true,
)

I also have a precompile.jl script with a representative workload. When I run docker build I get as far as to compiling the incremental sysimage. Then I get OOMs. I just have 8 GB of RAM and 2 GB of swap availiable. The same runs fine on a machine with 16 GB or 24 GB RAM.

My question: How can I reduce the resource need of PackageCompiler.jl?

BTW, the secrete sauce is this line:

RUN mkdir -p /home/jl/.julia/artifacts && ln -s /home/jl/app/share/julia/artifacts /home/jl/.julia/artifacts

which solves the relocatable issue, since I copy over the app from the build stage.

[ValentinKaisermayer]

I'm using WSL under Windows 11. Which by default has 50% of the host's memory available.

[ValentinKaisermayer]

I increased the swap space, now it works. But is very slow.

[PatrickHaecker]

I haven't checked this, but try to remove the -t auto. Lowering JULIA_NUM_PRECOMPILE_TASKS "4" might help, too. If this does not help, please try the other thread-related variables in the documentation and report back here.

As a rule of thumb, you should have around 1 GB of free RAM per Julia process. What is your free RAM for Julia? Is it the 8 GB or 50% of 8 GB or less due to other WSL processes? How many cores are available to Julia?

What is actionable about this issue?

• [ ] improve the documentation

We already warn the user when the problem is detected, but that probably does not help too much in cases like docker. We probably do not want to redefine the semantics of --threads auto. Reducing memory usage is always a fine goal, but a generic point here won't be actionable.

So I think after we learned what helped in this case, we can improve the documentation and close this issue.

[acabal33uab]

Be aware that you have a typo at JULIA_NUM_PRECOMPILE_TAKS (missing S before K)

[PatrickHaecker]

Thanks, I corrected it in my comment, but @ValentinKaisermayer needs to fix it, too.

[dpinol]

My sysimage creation also uses a lot of memory, specially after julia 1.12 I tried with export JULIA_NUM_PRECOMPILE_TASKS=1 and --threads=1, but there's no difference.

During the first 1.5minutes, it stays at 100%cpu and tops at 5Gb

But during the final 30s, it raises to 1500%cpu and reaches 10Gb

Is there anything I can do to remove the memory consumption?

thanks

[PatrickHaecker]

Have you tried all the thread-related variables in the documentation? I think one of them should solve the problem. You can check if you just set all of them to "1" and see whether this mitigates the problem. You can narrow it down afterwards.

[dpinol]

Have you tried all the thread-related variables in the documentation? I think one of them should solve the problem. You can check if you just set all of them to "1" and see whether this mitigates the problem. You can narrow it down afterwards.

Yes, with JULIA_CPU_THREADS=1 I could limit the CPU to 100% and the memory to 8GB. thanks! At the cost of making it 3 times slower 😮‍💨

[PatrickHaecker]

Maybe you can set JULIA_CPU_THREADS=2? This should already accelerate it a lot and might still not overload your system.

You seem to have a Linux system. You can also experiment with having a larger swap device / file. Another idea might be to use zram.

But I agree that Julia's RAM usage is really excessive and there is probably some room for improvement there.