Gitlab runner k8s: The build failed when setting the flag --ignore-var-run=false

[quangthe]

Actual behavior

Use gitlab runner (run on k8s) to build Docker image using kaniko.

The build failed when setting the flag --ignore-var-run=true.

error building image: error building stage: failed to get filesystem from image: error removing var/run to make way for new symlink: unlinkat /var/run/secrets/kubernetes.io/serviceaccount/..data: read-only file system

The log with debug enabled

INFO[0001] Retrieving image manifest debian:buster      
INFO[0001] Retrieving image debian:buster from registry index.docker.io 
INFO[0004] Built cross stage deps: map[]                
INFO[0004] Retrieving image manifest debian:buster      
INFO[0004] Returning cached image manifest              
INFO[0004] Executing 0 build triggers                   
INFO[0004] Building stage 'debian:buster' [idx: '0', base-idx: '-1'] 
INFO[0004] Unpacking rootfs as cmd RUN groupadd -g $GID $UNAME requires it. 
DEBU[0004] Ignore list: [{/kaniko false} {/etc/mtab false} {/tmp/apt-key-gpghome true} {/proc false} {/dev false} {/dev/pts false} {/sys false} {/sys/fs/cgroup false} {/sys/fs/cgroup/systemd false} {/sys/fs/cgroup/net_cls,net_prio false} {/sys/fs/cgroup/devices false} {/sys/fs/cgroup/pids false} {/sys/fs/cgroup/cpuset false} {/sys/fs/cgroup/cpu,cpuacct false} {/sys/fs/cgroup/perf_event false} {/sys/fs/cgroup/hugetlb false} {/sys/fs/cgroup/blkio false} {/sys/fs/cgroup/memory false} {/sys/fs/cgroup/freezer false} {/dev/mqueue false} {/scripts-282-64633 false} {/logs-282-64633 false} {/builds false} {/busybox false} {/certs/client false} {/dev/termination-log false} {/etc/resolv.conf false} {/etc/hostname false} {/etc/hosts false} {/dev/shm false} {/var/run/secrets/kubernetes.io/serviceaccount false}] 
DEBU[0004] Not adding /dev because it is ignored        
DEBU[0004] Not adding /etc/hostname because it is ignored 
DEBU[0004] Not adding /etc/resolv.conf because it is ignored 
DEBU[0004] Not adding /proc because it is ignored       
DEBU[0004] Not adding /sys because it is ignored        
error building image: error building stage: failed to get filesystem from image: error removing var/run to make way for new symlink: unlinkat /var/run/secrets/kubernetes.io/serviceaccount/..data: read-only file system

Expected behavior

Suggestion: Should provide the option to explicitly include some specific paths under /var/run instead of including all /var/run as we don't need all stuff under /var/run. There are cases in which we only need to keep state of some directories, such as /var/run/postgresql.

To Reproduce

• Setup k8s gitlab runner for the project
• Run the pipeline with provided Dockerfile and .gitlab-ci.yml in this ticket.

Additional Information

• Dockerfile

FROM debian:buster

ARG UNAME=postgres
ARG UID=70
ARG GID=70

ENV LANG en_US.UTF-8 
ENV LANGUAGE en_US:en 
ENV LC_ALL en_US.UTF-8

RUN groupadd -g $GID $UNAME
RUN useradd -m -u $UID -g $GID -s /bin/bash $UNAME
RUN usermod -aG root $UNAME

USER postgres

• Build Context

stages:
  - build

build-push-docker-image:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:v1.12.1-debug
    entrypoint: [""]
  before_script:
    - mkdir -p /kaniko/.docker
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
  script:
    - /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
      --destination "${CI_REGISTRY_IMAGE}"
      --ignore-var-run=false
      --verbosity debug

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing	No
Please check if the build works in docker but not in kaniko	Yes
Please check if this error is seen when you use --cache flag	No
Please check if your dockerfile is a multistage dockerfile	No