Kaniko without root user and priviliged container

[guyscher]

Hello,

I'm having a hard time running kaniko on openshift with as non root and non privileged.

In addition, Openshift blocks kaniko from doing chown to '/'.

thanks, Guy

[imjasonh]

I'm having a hard time running kaniko on openshift with as non root and non privileged.

What specific errors are you seeing, and what are you trying to do?

[guyscher]

permission denied to chown "/"

[jonpalle]

I have the same issue as @PoPi2 on Openshift 3.11.

The following pod spec snippet is not allowed:

apiVersion: v1
kind: Pod
metadata:
  name: security-context-demo
spec:
  securityContext:
    runAsUser: 0

When running in a Jenkins pipeline I get this Jenkins console output:

ERROR: Unable to create pod kubernetes ejerskifte-jenkins/ejerskifte-omregistrering-api-kaniko-108-f4hmq-2xggm-2rhr5.
Failure executing: POST at: https://kubernetes.default.svc/api/v1/namespaces/ejerskifte-jenkins/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "ejerskifte-omregistrering-api-kaniko-108-f4hmq-2xggm-2rhr5" is forbidden: unable to validate against any security context constraint: [spec.containers[2].securityContext.securityContext.runAsUser: Invalid value: 0: must be in the ranges: [1002440000, 1002449999] spec.containers[2].securityContext.securityContext.runAsUser: Invalid value: 0: must be in the ranges: [1002440000, 1002449999]].

If I don't explicitly set securityContext I have trouble building a container image because chown is not allowed:

[36mINFO[0m[0003] Unpacking rootfs as cmd RUN mkdir -p /app/config requires it. 
[37mDEBU[0m[0003] Mounted directories: [{/kaniko false} {/etc/mtab false} {/tmp/apt-key-gpghome true} {/proc false} {/dev false} {/dev/pts false} {/sys false} {/sys/fs/cgroup false} {/sys/fs/cgroup/systemd false} {/sys/fs/cgroup/net_prio,net_cls false} {/sys/fs/cgroup/cpuset false} {/sys/fs/cgroup/devices false} {/sys/fs/cgroup/pids false} {/sys/fs/cgroup/cpuacct,cpu false} {/sys/fs/cgroup/perf_event false} {/sys/fs/cgroup/freezer false} {/sys/fs/cgroup/hugetlb false} {/sys/fs/cgroup/memory false} {/sys/fs/cgroup/blkio false} {/dev/mqueue false} {/home/jenkins false} {/dev/termination-log false} {/run/secrets false} {/etc/resolv.conf false} {/etc/hostname false} {/etc/hosts false} {/dev/shm false} {/home/jenkins/.docker false} {/run/secrets/kubernetes.io/serviceaccount false} {/dev/console false} {/proc/bus false} {/proc/fs false} {/proc/irq false} {/proc/sys false} {/proc/sysrq-trigger false} {/proc/acpi false} {/proc/kcore false} {/proc/keys false} {/proc/timer_list false} {/proc/timer_stats false} {/proc/sched_debug false} {/proc/scsi false} {/sys/firmware false}] 
error building image: error building stage: failed to get filesystem from image: chown /bin: operation not permitted

[asdfzxcvqwerasdfzcxv]

I have the same issue as @PoPi2 on Openshift 3.11.

The following pod spec snippet is not allowed:

apiVersion: v1
kind: Pod
metadata:
  name: security-context-demo
spec:
  securityContext:
    runAsUser: 0

When running in a Jenkins pipeline I get this Jenkins console output:

ERROR: Unable to create pod kubernetes ejerskifte-jenkins/ejerskifte-omregistrering-api-kaniko-108-f4hmq-2xggm-2rhr5.
Failure executing: POST at: https://kubernetes.default.svc/api/v1/namespaces/ejerskifte-jenkins/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "ejerskifte-omregistrering-api-kaniko-108-f4hmq-2xggm-2rhr5" is forbidden: unable to validate against any security context constraint: [spec.containers[2].securityContext.securityContext.runAsUser: Invalid value: 0: must be in the ranges: [1002440000, 1002449999] spec.containers[2].securityContext.securityContext.runAsUser: Invalid value: 0: must be in the ranges: [1002440000, 1002449999]].

If I don't explicitly set securityContext I have trouble building a container image because chown is not allowed:

[36mINFO[0m[0003] Unpacking rootfs as cmd RUN mkdir -p /app/config requires it. 
[37mDEBU[0m[0003] Mounted directories: [{/kaniko false} {/etc/mtab false} {/tmp/apt-key-gpghome true} {/proc false} {/dev false} {/dev/pts false} {/sys false} {/sys/fs/cgroup false} {/sys/fs/cgroup/systemd false} {/sys/fs/cgroup/net_prio,net_cls false} {/sys/fs/cgroup/cpuset false} {/sys/fs/cgroup/devices false} {/sys/fs/cgroup/pids false} {/sys/fs/cgroup/cpuacct,cpu false} {/sys/fs/cgroup/perf_event false} {/sys/fs/cgroup/freezer false} {/sys/fs/cgroup/hugetlb false} {/sys/fs/cgroup/memory false} {/sys/fs/cgroup/blkio false} {/dev/mqueue false} {/home/jenkins false} {/dev/termination-log false} {/run/secrets false} {/etc/resolv.conf false} {/etc/hostname false} {/etc/hosts false} {/dev/shm false} {/home/jenkins/.docker false} {/run/secrets/kubernetes.io/serviceaccount false} {/dev/console false} {/proc/bus false} {/proc/fs false} {/proc/irq false} {/proc/sys false} {/proc/sysrq-trigger false} {/proc/acpi false} {/proc/kcore false} {/proc/keys false} {/proc/timer_list false} {/proc/timer_stats false} {/proc/sched_debug false} {/proc/scsi false} {/sys/firmware false}] 
error building image: error building stage: failed to get filesystem from image: chown /bin: operation not permitted

I'll supply with that fact that our image with kaniko is the same as https://github.com/ContainerSolutions/dockerfiles/tree/master/kaniko-openshift from https://github.com/GoogleContainerTools/kaniko/issues/379#issuecomment-537982757

[BobbyNie]

sloved by modify the code

see https://github.com/GoogleContainerTools/kaniko/pull/2125

docker pull docker.io/bobbynie/kaniko-bash

apiVersion: v1
kind: Pod
metadata:
  name: kaniko-build
  labels:
    app: build
spec:
  imagePullSecrets:
    - name: kaniko-pull-secret
  containers:
    - name: kaniko
      image: 'docker.io/bobbynie/kaniko-bash'
      env:
        - name: DOCKER_CONFIG
          value: /workspace/.docker
      command: ["/bin/sh","-c","mkdir -m 777 /workspace/wp ; \
            /kaniko/executor --dockerfile=/workspace/Dockerfile  \
            --context=/workspace/wp \
            --destination=docker.io/bobbynie/buildimg:latest \
            --skip-tls-verify \
            -v=trace"]
      volumeMounts:
        - name: kaniko-secret
          mountPath: /workspace/.docker 
        - name: context
          mountPath: /workspace
  restartPolicy: Never
  volumes:
    - name: kaniko-secret
      secret:
        secretName:  kaniko-pull-secret
        items:
          - key: .dockerconfigjson
            path: config.json
    - name: context
      persistentVolumeClaim:
        claimName: build-pvc-sc

[lvarin]

That fix did not solve the problem for me.

I am running this in OpenShift OKD v4, and got the same chown /bin: operation not permitted error. The road to rootless is going to be long.

[BobbyNie]

sloved by modify the code

see #2125

docker pull docker.io/bobbynie/kaniko-bash

apiVersion: v1
kind: Pod
metadata:
  name: kaniko-build
  labels:
    app: build
spec:
  imagePullSecrets:
    - name: kaniko-pull-secret
  containers:
    - name: kaniko
      image: 'docker.io/bobbynie/kaniko-bash'
      env:
        - name: DOCKER_CONFIG
          value: /workspace/.docker
      command: ["/bin/sh","-c","mkdir -m 777 /workspace/wp ; \
            /kaniko/executor --dockerfile=/workspace/Dockerfile  \
            --context=/workspace/wp \
            --destination=docker.io/bobbynie/buildimg:latest \
            --skip-tls-verify \
            -v=trace"]
      volumeMounts:
        - name: kaniko-secret
          mountPath: /workspace/.docker 
        - name: context
          mountPath: /workspace
  restartPolicy: Never
  volumes:
    - name: kaniko-secret
      secret:
        secretName:  kaniko-pull-secret
        items:
          - key: .dockerconfigjson
            path: config.json
    - name: context
      persistentVolumeClaim:
        claimName: build-pvc-sc

actually it's not work.
kaniko need anyuid scc