kaniko Gitlab runner k8s: The build failed when setting the flag --ignore-var-run=false

Actual behavior

Use gitlab runner (run on k8s) to build Docker image using kaniko.

The build failed when setting the flag --ignore-var-run=true.

error building image: error building stage: failed to get filesystem from image: error removing var/run to make way for new symlink: unlinkat /var/run/secrets/kubernetes.io/serviceaccount/..data: read-only file system

The log with debug enabled

INFO[0001] Retrieving image manifest debian:buster      
INFO[0001] Retrieving image debian:buster from registry index.docker.io 
INFO[0004] Built cross stage deps: map[]                
INFO[0004] Retrieving image manifest debian:buster      
INFO[0004] Returning cached image manifest              
INFO[0004] Executing 0 build triggers                   
INFO[0004] Building stage 'debian:buster' [idx: '0', base-idx: '-1'] 
INFO[0004] Unpacking rootfs as cmd RUN groupadd -g $GID $UNAME requires it. 
DEBU[0004] Ignore list: [{/kaniko false} {/etc/mtab false} {/tmp/apt-key-gpghome true} {/proc false} {/dev false} {/dev/pts false} {/sys false} {/sys/fs/cgroup false} {/sys/fs/cgroup/systemd false} {/sys/fs/cgroup/net_cls,net_prio false} {/sys/fs/cgroup/devices false} {/sys/fs/cgroup/pids false} {/sys/fs/cgroup/cpuset false} {/sys/fs/cgroup/cpu,cpuacct false} {/sys/fs/cgroup/perf_event false} {/sys/fs/cgroup/hugetlb false} {/sys/fs/cgroup/blkio false} {/sys/fs/cgroup/memory false} {/sys/fs/cgroup/freezer false} {/dev/mqueue false} {/scripts-282-64633 false} {/logs-282-64633 false} {/builds false} {/busybox false} {/certs/client false} {/dev/termination-log false} {/etc/resolv.conf false} {/etc/hostname false} {/etc/hosts false} {/dev/shm false} {/var/run/secrets/kubernetes.io/serviceaccount false}] 
DEBU[0004] Not adding /dev because it is ignored        
DEBU[0004] Not adding /etc/hostname because it is ignored 
DEBU[0004] Not adding /etc/resolv.conf because it is ignored 
DEBU[0004] Not adding /proc because it is ignored       
DEBU[0004] Not adding /sys because it is ignored        
error building image: error building stage: failed to get filesystem from image: error removing var/run to make way for new symlink: unlinkat /var/run/secrets/kubernetes.io/serviceaccount/..data: read-only file system

Expected behavior

Suggestion: Should provide the option to explicitly include some specific paths under /var/run instead of including all /var/run as we don't need all stuff under /var/run. There are cases in which we only need to keep state of some directories, such as /var/run/postgresql.

To Reproduce

Setup k8s gitlab runner for the project
Run the pipeline with provided Dockerfile and .gitlab-ci.yml in this ticket.

Additional Information

Dockerfile

FROM debian:buster

ARG UNAME=postgres
ARG UID=70
ARG GID=70

ENV LANG en_US.UTF-8 
ENV LANGUAGE en_US:en 
ENV LC_ALL en_US.UTF-8

RUN groupadd -g $GID $UNAME
RUN useradd -m -u $UID -g $GID -s /bin/bash $UNAME
RUN usermod -aG root $UNAME

USER postgres

Build Context

stages:
  - build

build-push-docker-image:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:v1.12.1-debug
    entrypoint: [""]
  before_script:
    - mkdir -p /kaniko/.docker
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
  script:
    - /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
      --destination "${CI_REGISTRY_IMAGE}"
      --ignore-var-run=false
      --verbosity debug

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing	No
Please check if the build works in docker but not in kaniko	Yes
Please check if this error is seen when you use `--cache` flag	No
Please check if your dockerfile is a multistage dockerfile	No

Jul 31 '23 08:07 quangthe

I was bitten by this today as well. Seems like kaniko makes incorrect assumptions.

Here is a simple example that demonstrates the problem and does not require GitLab, Kubernetes and --ignore-var-run:

% docker run -it --entrypoint /bin/sh gcr.io/kaniko-project/executor:debug
/workspace # cat <<EOF > Dockerfile
> FROM alpine
> RUN mkdir /var/bla
> EOF
/workspace # mkdir -p /var/bla
/workspace # /kaniko/executor --destination=image --no-push --tar-path=/tmp/image.tar
INFO[0000] Retrieving image manifest alpine
INFO[0000] Retrieving image alpine from registry index.docker.io
INFO[0001] Built cross stage deps: map[]
INFO[0001] Retrieving image manifest alpine
INFO[0001] Returning cached image manifest
INFO[0001] Executing 0 build triggers
INFO[0001] Building stage 'alpine' [idx: '0', base-idx: '-1']
INFO[0001] Unpacking rootfs as cmd RUN mkdir /var/bla requires it.
INFO[0001] RUN mkdir /var/bla
INFO[0001] Initializing snapshotter ...
INFO[0001] Taking snapshot of full filesystem...
INFO[0001] Cmd: /bin/sh
INFO[0001] Args: [-c mkdir /var/bla]
INFO[0001] Running: [/bin/sh -c mkdir /var/bla]
mkdir: can't create directory '/var/bla': File exists
error building image: error building stage: failed to execute command: waiting for process to exit: exit status 1
/workspace # rmdir /var/bla
/workspace # /kaniko/executor --destination=image --no-push --tar-path=/tmp/image.tar
INFO[0000] Retrieving image manifest alpine
INFO[0000] Retrieving image alpine from registry index.docker.io
INFO[0001] Built cross stage deps: map[]
INFO[0001] Retrieving image manifest alpine
INFO[0001] Returning cached image manifest
INFO[0001] Executing 0 build triggers
INFO[0001] Building stage 'alpine' [idx: '0', base-idx: '-1']
INFO[0001] Unpacking rootfs as cmd RUN mkdir /var/bla requires it.
INFO[0001] RUN mkdir /var/bla
INFO[0001] Initializing snapshotter ...
INFO[0001] Taking snapshot of full filesystem...
INFO[0001] Cmd: /bin/sh
INFO[0001] Args: [-c mkdir /var/bla]
INFO[0001] Running: [/bin/sh -c mkdir /var/bla]
INFO[0001] Taking snapshot of full filesystem...
INFO[0002] Skipping push to container registry due to --no-push flag

Kaniko extracts the source image contents in / instead of an empty directory:

https://github.com/GoogleContainerTools/kaniko/blob/d7f7bc11bf6772ea51ade04ca3bce6d58278d912/pkg/executor/build.go#L329

https://github.com/GoogleContainerTools/kaniko/blob/d7f7bc11bf6772ea51ade04ca3bce6d58278d912/pkg/config/init.go#L50

https://github.com/GoogleContainerTools/kaniko/blob/d7f7bc11bf6772ea51ade04ca3bce6d58278d912/pkg/constants/constants.go#L21

And that is the problem.

Note that you might get bitten by this in case you execute kaniko multiple times to build multiple images as well. The kaniko argument --cleanup might save you here, though.

A simple GitLab Kubernetes Runner test case.

Dockerfile:

FROM alpine
RUN touch /var/bla

.gitlab-ci.yml:

test:
  image: "gcr.io/kaniko-project/executor:debug"
  script:
    - >-
      /kaniko/executor
      --destination image
      --no-push
      --tar-path=/image.tar
      --ignore-var-run=false

Running this on a GitLab Kubernetes Runner fails because /var/run/secrets/kubernetes.io/serviceaccount is populated in the k8s container where you run kaniko.

Setting --ignore-var-run=false makes kaniko want to try to delete /var/run in case it is present in the source image and the Dockerfile contains some commands the require the image to be unpacked:

https://github.com/GoogleContainerTools/kaniko/blob/d7f7bc11bf6772ea51ade04ca3bce6d58278d912/pkg/executor/build.go#L309-L340

This part here seems to fail:

https://github.com/GoogleContainerTools/kaniko/blob/d7f7bc11bf6772ea51ade04ca3bce6d58278d912/pkg/util/fs_util.go#L395-L411

as it assumes that it can just delete stuff.

Mar 04 '24 18:03 ghost

These issues seem to relate in one way or another, some are rather old: https://github.com/GoogleContainerTools/kaniko/issues/1750 https://github.com/GoogleContainerTools/kaniko/issues/2793 https://github.com/GoogleContainerTools/kaniko/issues/2776 https://github.com/GoogleContainerTools/kaniko/issues/2701 https://github.com/GoogleContainerTools/kaniko/issues/1297 https://github.com/GoogleContainerTools/kaniko/issues/1353 https://github.com/GoogleContainerTools/kaniko/issues/2764

Mar 04 '24 18:03 ghost