Kaniko builds take consume double the resources and time to build than docker/podman

[droslean]

This is not necessarily a bug issue. I just want to get some advice about the best practices.

My build takes around 15-20 minutes to build and consumes around 1-2CPU and 2GB of memory.

With Kaniko, it takes around 50 minutes to build and uses around 3CPU and 4-6GB of memory.

I am already using a cache. Here are the args

    - --dockerfile=Dockerfile.prod
    - --context=.
    - --destination=ghcr.io/my-org/my-repo:mytag
    - --cache=true
    - --cache-dir=/cache
    - --compressed-caching=false
    - --use-new-run
    - --cleanup

I wonder if any of the arguments I am using make the build so expensive.

[aaron-prindle]

@droslean thanks for the flagging this issue here. Kaniko currently has a few open issues related to resource usage esp. when used with large multi-stage builds. You can view these open issues to understand a bit more about this w/ kaniko:

https://github.com/GoogleContainerTools/kaniko/issues/875 https://github.com/GoogleContainerTools/kaniko/issues/970 https://github.com/GoogleContainerTools/kaniko/issues/909

Kaniko does have flags that can reduce the resource usage // build time for an image. The main flags here are:

• --use-new-run (which you already used above so likely not helpful in this case)
• --snapshot-mode=redo

[pimperator]

I can confirm the huge memory consumption as described in #2275 build time is indirectly proportional to available ressources

[sebglon]

I have the same issue. When i try to build Python wheel with gclient, Kaniko use more than 1h and more than 15GB Where buildah take 1h and only 4GB

here is part of our Dockerfile:

ARG PYTHON_VERSION
FROM python:3.9-slim-bullseye as webrtc_builder
RUN apt-get update
RUN apt-get -y install git curl build-essential lsb-release \
                             libasound2 \
                             libatk1.0-0 \
                             libatspi2.0-0 \
                             libc6 \
                             libcairo2 \
                             libcap2 \
                             libcups2 \
                             libdrm2 \
                             libevdev2 \
                             libexpat1 \
                             libfontconfig1 \
                             libfreetype6 \
                             libgbm1 \
                             libglib2.0-0 \
                             libgtk-3-0 \
                             libpam0g \
                             libpango-1.0-0 \
                             libpci3 \
                             libpcre3 \
                             libpixman-1-0 \
                             libspeechd2 \
                             libstdc++6 \
                             libsqlite3-0 \
                             libuuid1 \
                             libwayland-egl1-mesa \
                             libx11-6 \
                             libx11-xcb1 \
                             libxau6 \
                             libxcb1 \
                             libxcomposite1 \
                             libxcursor1 \
                             libxdamage1 \
                             libxdmcp6 \
                             libxext6 \
                             libxfixes3 \
                             libxi6 \
                             libxinerama1 \
                             libxrandr2 \
                             libxrender1 \
                             libxtst6 \
                             zlib1g \
                             wget  \
                             clang-11  \
                             ninja-build

RUN mkdir -p /utils/depot_tools
RUN git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git /utils/depot_tools

ENV PATH=/utils/depot_tools:$PATH
ARG WEBRTC_PATH=/webrtc_checkout
RUN mkdir $WEBRTC_PATH
WORKDIR "$WEBRTC_PATH"

RUN fetch --nohooks webrtc
RUN  gclient sync
RUN cd $WEBRTC_PATH/src &&  \
    git checkout -b m105 branch-heads/5195
RUN gclient sync
RUN find . -type d -name ".git"  | xargs rm -rf && \
    gn gen out/m105 --args='is_debug=false is_component_build=false rtc_include_tests=false rtc_use_h264=true use_rtti=true use_custom_libcxx=false use_custom_libcxx_for_host=false treat_warnings_as_errors=false rtc_use_x11=false use_glib=false rtc_exclude_audio_processing_module=true rtc_include_opus=false rtc_include_ilbc=false rtc_audio_device_plays_sinus_tone=true rtc_use_dummy_audio_file_devices=true target_os="linux" rtc_include_builtin_audio_codecs=true is_clang=false use_gio=false rtc_enable_protobuf=false target_os="linux" enable_iterator_debugging=false rtc_build_examples=false rtc_use_pipewire=false use_ozone=true'