GoogleContainerTools
Difference container image output from Kaniko build
Hello, I acknowledged that Kaniko already mentioned "Running kaniko in any Docker image other than the official kaniko image is not supported due to implementation details".
I'm trying to achieve this for GitLab Runner to build container images with Kaniko.
Here is the example Dockerfile, https://github.com/aws-samples/cdk-fargate-gitlab-runner/blob/main/docker_images/kaniko/Dockerfile.
I extracted a part from my Dockerfile
RUN apk upgrade --no-cache &&
apk add --no-cache openssh git git-lfs ca-certificates envsubst &&
ssh-keygen -A &&
sed -i -e 's/^#HostKey/HostKey/g' /etc/ssh/sshd_config &&
sed -i -e 's/^#Port 22/Port 22/g' /etc/ssh/sshd_config &&
sed -i -e 's/^#PubkeyAuthentication yes/PubkeyAuthentication yes/g' /etc/ssh/sshd_config &&
sed -i -e 's/^#PasswordAuthentication yes/PasswordAuthentication no/g' /etc/ssh/sshd_config &&
sed -i -e 's/^AuthorizedKeysFile\t.ssh/authorized_keys/AuthorizedKeysFile\t%h/.ssh/authorized_keys/g' /etc/ssh/sshd_config
Here is the build logs on my laptop.
INFO[2025-02-11T04:08:24Z] Taking snapshot of full filesystem...
INFO[2025-02-11T04:08:24Z] RUN apk upgrade --no-cache && apk add --no-cache tini openssh git git-lfs ca-certificates envsubst && ssh-keygen -A && sed -i -e 's/^#HostKey/HostKey/g' /etc/ssh/sshd_config && sed -i -e 's/^#Port 22/Port 22/g' /etc/ssh/sshd_config && sed -i -e 's/^#PubkeyAuthentication yes/PubkeyAuthentication yes/g' /etc/ssh/sshd_config && sed -i -e 's/^#PasswordAuthentication yes/PasswordAuthentication no/g' /etc/ssh/sshd_config && sed -i -e 's/^AuthorizedKeysFile\t.ssh/authorized_keys/AuthorizedKeysFile\t%h/.ssh/authorized_keys/g' /etc/ssh/sshd_config INFO[2025-02-11T04:08:24Z] Cmd: /bin/sh
INFO[2025-02-11T04:08:24Z] Args: [-c apk upgrade --no-cache && apk add --no-cache tini openssh git git-lfs ca-certificates envsubst && ssh-keygen -A && sed -i -e 's/^#HostKey/HostKey/g' /etc/ssh/sshd_config && sed -i -e 's/^#Port 22/Port 22/g' /etc/ssh/sshd_config && sed -i -e 's/^#PubkeyAuthentication yes/PubkeyAuthentication yes/g' /etc/ssh/sshd_config && sed -i -e 's/^#PasswordAuthentication yes/PasswordAuthentication no/g' /etc/ssh/sshd_config && sed -i -e 's/^AuthorizedKeysFile\t.ssh/authorized_keys/AuthorizedKeysFile\t%h/.ssh/authorized_keys/g' /etc/ssh/sshd_config] INFO[2025-02-11T04:08:24Z] Running: [/bin/sh -c apk upgrade --no-cache && apk add --no-cache tini openssh git git-lfs ca-certificates envsubst && ssh-keygen -A && sed -i -e 's/^#HostKey/HostKey/g' /etc/ssh/sshd_config && sed -i -e 's/^#Port 22/Port 22/g' /etc/ssh/sshd_config && sed -i -e 's/^#PubkeyAuthentication yes/PubkeyAuthentication yes/g' /etc/ssh/sshd_config && sed -i -e 's/^#PasswordAuthentication yes/PasswordAuthentication no/g' /etc/ssh/sshd_config && sed -i -e 's/^AuthorizedKeysFile\t.ssh/authorized_keys/AuthorizedKeysFile\t%h/.ssh/authorized_keys/g' /etc/ssh/sshd_config] fetch https://dl-cdn.alpinelinux.org/alpine/v3.21/main/aarch64/APKINDEX.tar.gz fetch https://dl-cdn.alpinelinux.org/alpine/v3.21/community/aarch64/APKINDEX.tar.gz (1/5) Upgrading busybox (1.37.0-r9 -> 1.37.0-r12) Executing busybox-1.37.0-r12.post-upgrade (2/5) Upgrading busybox-binsh (1.37.0-r9 -> 1.37.0-r12) (3/5) Upgrading libcrypto3 (3.3.2-r4 -> 3.3.2-r6) (4/5) Upgrading libssl3 (3.3.2-r4 -> 3.3.2-r6) (5/5) Upgrading ssl_client (1.37.0-r9 -> 1.37.0-r12) Executing busybox-1.37.0-r12.trigger OK: 7 MiB in 15 packages fetch https://dl-cdn.alpinelinux.org/alpine/v3.21/main/aarch64/APKINDEX.tar.gz fetch https://dl-cdn.alpinelinux.org/alpine/v3.21/community/aarch64/APKINDEX.tar.gz (1/27) Installing ca-certificates (20241121-r1) (2/27) Installing libintl (0.22.5-r0) (3/27) Installing gettext-envsubst (0.22.5-r0) (4/27) Installing brotli-libs (1.1.0-r2) (5/27) Installing c-ares (1.34.3-r0) (6/27) Installing libunistring (1.2-r0) (7/27) Installing libidn2 (2.3.7-r0) (8/27) Installing nghttp2-libs (1.64.0-r0) (9/27) Installing libpsl (0.21.5-r3) (10/27) Installing zstd-libs (1.5.6-r2) (11/27) Installing libcurl (8.11.1-r1) (12/27) Installing libexpat (2.6.4-r0) (13/27) Installing pcre2 (10.43-r0) (14/27) Installing git (2.47.2-r0) (15/27) Installing git-init-template (2.47.2-r0) (16/27) Installing git-lfs (3.6.0-r2) Executing git-lfs-3.6.0-r2.post-install Git LFS initialized. (17/27) Installing openssh-keygen (9.9_p1-r2) (18/27) Installing ncurses-terminfo-base (6.5_p20241006-r3) (19/27) Installing libncursesw (6.5_p20241006-r3) (20/27) Installing libedit (20240808.3.1-r0) (21/27) Installing openssh-client-common (9.9_p1-r2) (22/27) Installing openssh-client-default (9.9_p1-r2) (23/27) Installing openssh-sftp-server (9.9_p1-r2) (24/27) Installing openssh-server-common (9.9_p1-r2) (25/27) Installing openssh-server (9.9_p1-r2) (26/27) Installing openssh (9.9_p1-r2) (27/27) Installing tini (0.19.0-r3) Executing busybox-1.37.0-r12.trigger Executing ca-certificates-20241121-r1.trigger OK: 39 MiB in 42 packages
There is no issues for the container image which built on my laptop.
However, when my GitLab runner execute the same command on the same container image (GitLab Runner + Kaniko) on AWS ECS (Fargate Task). I noticed that there is no installation logs of the step.
INFO[2025-02-11T03:42:15Z] Taking snapshot of full filesystem...
INFO[2025-02-11T03:42:15Z] RUN apk upgrade --no-cache && apk add --no-cache openssh git git-lfs ca-certificates envsubst && ssh-keygen -A && sed -i -e 's/^#HostKey/HostKey/g' /etc/ssh/sshd_config && sed -i -e 's/^#Port 22/Port 22/g' /etc/ssh/sshd_config && sed -i -e 's/^#PubkeyAuthentication yes/PubkeyAuthentication yes/g' /etc/ssh/sshd_config && sed -i -e 's/^#PasswordAuthentication yes/PasswordAuthentication no/g' /etc/ssh/sshd_config && sed -i -e 's/^AuthorizedKeysFile\t.ssh/authorized_keys/AuthorizedKeysFile\t%h/.ssh/authorized_keys/g' /etc/ssh/sshd_config INFO[2025-02-11T03:42:15Z] Cmd: /bin/sh
INFO[2025-02-11T03:42:15Z] Args: [-c apk upgrade --no-cache && apk add --no-cache openssh git git-lfs ca-certificates envsubst && ssh-keygen -A && sed -i -e 's/^#HostKey/HostKey/g' /etc/ssh/sshd_config && sed -i -e 's/^#Port 22/Port 22/g' /etc/ssh/sshd_config && sed -i -e 's/^#PubkeyAuthentication yes/PubkeyAuthentication yes/g' /etc/ssh/sshd_config && sed -i -e 's/^#PasswordAuthentication yes/PasswordAuthentication no/g' /etc/ssh/sshd_config && sed -i -e 's/^AuthorizedKeysFile\t.ssh/authorized_keys/AuthorizedKeysFile\t%h/.ssh/authorized_keys/g' /etc/ssh/sshd_config] INFO[2025-02-11T03:42:15Z] Running: [/bin/sh -c apk upgrade --no-cache && apk add --no-cache tini openssh git git-lfs ca-certificates envsubst && ssh-keygen -A && sed -i -e 's/^#HostKey/HostKey/g' /etc/ssh/sshd_config && sed -i -e 's/^#Port 22/Port 22/g' /etc/ssh/sshd_config && sed -i -e 's/^#PubkeyAuthentication yes/PubkeyAuthentication yes/g' /etc/ssh/sshd_config && sed -i -e 's/^#PasswordAuthentication yes/PasswordAuthentication no/g' /etc/ssh/sshd_config && sed -i -e 's/^AuthorizedKeysFile\t.ssh/authorized_keys/AuthorizedKeysFile\t%h/.ssh/authorized_keys/g' /etc/ssh/sshd_config] INFO[2025-02-11T03:42:17Z] Taking snapshot of full filesystem...
Other libraries look good except tini. Normally, it will installed in /sbin
/ # ls /sbin acpid blkid fdisk getty ifdown inotifyd iplink iptunnel logread mkdosfs modprobe poweroff route swapon tunctl zcip adjtimex blockdev findfs halt ifenslave insmod ipneigh klogd losetup mkfs.vfat nameif raidautorun setconsole switch_root udhcpc apk depmod fsck hwclock ifup ip iproute ldconfig lsmod mkswap nologin reboot slattach sysctl vconfig arp fbsplash fstrim ifconfig init ipaddr iprule loadkmap mdev modinfo pivot_root rmmod swapoff syslogd watchdog
/ # ls /bin arch chattr dd egrep getopt iostat linux64 makemime mountpoint ping reformime setpriv su uname ash chgrp df false grep ipcalc ln mkdir mpstat ping6 rev setserial sync usleep base64 chmod dmesg fatattr gunzip kbd_mode login mknod mv pipe_progress rm sh tar watch bbconfig chown dnsdomainname fdflush gzip kill ls mktemp netstat printenv rmdir sleep touch zcat busybox cp dumpkmap fgrep hostname link lsattr more nice ps run-parts stat true cat date echo fsync ionice linux32 lzop mount pidof pwd sed stty umount
/ # apk version tini WARNING: opening from cache https://dl-cdn.alpinelinux.org/alpine/v3.21/main: No such file or directory WARNING: opening from cache https://dl-cdn.alpinelinux.org/alpine/v3.21/community: No such file or directory Installed: Available: tini-0.19.0-r3 = 0.19.0-r3
apk is showing tini is installed but no executable file in /sbin path.
Could you please guide me what could be the cause of this issue?
My base image in Dockerfile is alpine:3.21, and the architecture is arm64.
Update... the build output is not reliable. I found some executable files are missing for each builds.
Yeah, I have noticed a similar thing. Multiple invocations with different AWS credentials lead to broken images. The first build correctly installs the required Ubuntu packages but subsequent builds seem to exclude them
