procMount: Unmasked is not working

[pranjaljain]

Can you please share the .yml file needed to deploy img on kubernetes pod, if available?

[pranjaljain]

while running a pod with following manifest

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: img
  name: img
  annotations:
    container.apparmor.security.beta.kubernetes.io/img: unconfined
spec:
  securityContext:
    runAsUser: 1000
  initContainers:
    # This container clones the desired git repo to the EmptyDir volume.
    - name: git-clone
      image: r.j3ss.co/jq
      args:
        - git
        - clone
        - --single-branch
        - --
        - https://github.com/jessfraz/dockerfiles
        - /repo # Put it in the volume
      securityContext:
        procMount: "Unmasked"
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
      volumeMounts:
        - name: git-repo
          mountPath: /repo
  containers:
  - image: r.j3ss.co/img
    imagePullPolicy: Always
    name: img
    resources: {}
    workingDir: /repo
    command:
    - img
    - build
    - -t
    - irssi
    - irssi/
    securityContext:
      procMount: "Unmasked"
      capabilities:
        add:
        - SYS_ADMIN
    volumeMounts:
    - name: cache-volume
      mountPath: /tmp
    - name: git-repo
      mountPath: /repo
  volumes:
  - name: cache-volume
    emptyDir: {}
  - name: git-repo
    emptyDir: {}
  restartPolicy: Never

I am getting error:

container_linux.go:344: starting container process caused "process_linux.go:424: container init caused \"rootfs_linux.go:58: mounting \\\"proc\\\" to rootfs \\\"/home/user/.local/share/img/runc/native/executor/26pojmt2cn65900kjlpt6217i/rootfs\\\" at \\\"/proc\\\" caused \\\"operation not permitted\\\"\""

@jessfraz It seems procMount: Unmasked is not working for me. Any ideas why? I am using kubernetes version 1.13 and docker server version 18.06.1-ce and docker api version 1.38. Feature Gate ProcMountType is enabled.

[towolf]

I think I get the same thing, just FYI:

#5 [2/4] RUN apk --no-cache add         --repository http://dl-cdn.alpinelinux.org...
#5       digest: sha256:421cf23668e2abb7ca968581f731eb73cf424b2d8da6358752244806ea2bef29
#5         name: "[2/4] RUN apk --no-cache add \t--repository http://dl-cdn.alpinelinux.org/alpine/latest-stable/community \tca-certificates \tperl-datetime \tperl-timedate"                                       
#5      started: 2019-02-18 23:40:05.985147894 +0000 UTC m=+3.836340248
#5    completed: 2019-02-18 23:40:06.479805956 +0000 UTC m=+4.330998269
#5     duration: 494.658021ms
#5        error: "executor failed running [/bin/sh -c apk --no-cache add \t--repository http://dl-cdn.alpinelinux.org/alpine/latest-stable/community \tca-certificates \tperl-datetime \tperl-timedate]: exit code: 1"
#5 0.444 container_linux.go:344: starting container process caused "process_linux.go:424: container init caused \"rootfs_linux.go:58: mounting \\\"proc\\\" to rootfs \\\"/home/user/.local/share/img/runc/native/executor/0r6g578bq05ai3n1w357a9r74/rootfs\\\" at \\\"/proc\\\" caused \\\"operation not permitted\\\"\""

failed to solve: executor failed running [/bin/sh -c apk --no-cache add         --repository http://dl-cdn.alpinelinux.org/alpine/latest-stable/community       ca-certificates         perl-datetime   perl-timedate]: exit code: 1

[AkihiroSuda]

Set seccomp.security.alpha.kubernetes.io/pod to unconfined explicitly?

[AkihiroSuda]

Also, what's your Kubernetes distribution?

Anyway, procMount will no longer be required: #221

[towolf]

For me adding seccomp.security.alpha.kubernetes.io/pod: unconfined does not change anything.

I have Kubernetes 1.13.3, with some PodSecurityPolicies. But I tried to make sure that the pod has all permissions it needs. Maybe the PSP is breaking this?

From kubectl describe pod img

Annotations:        cni.projectcalico.org/podIP: 172.31.4.187/32
                    container.apparmor.security.beta.kubernetes.io/img: unconfined
                    kubernetes.io/psp: privileged
                    seccomp.security.alpha.kubernetes.io/pod: unconfined

This is the content of the PSP:

apiVersion: extensions/v1beta1
kind: PodSecurityPolicy
metadata:
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*'
  name: privileged
spec:
  allowPrivilegeEscalation: true
  allowedCapabilities:
  - '*'
  allowedProcMountTypes:
  - Unmasked
  - Default
  allowedUnsafeSysctls:
  - net.core.somaxconn
  fsGroup:
    rule: RunAsAny
  hostIPC: true
  hostNetwork: true
  hostPID: true
  hostPorts:
  - max: 65535
    min: 0
  privileged: true
  runAsUser:
    rule: RunAsAny
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  volumes:
  - '*'

[martencassel]

The runtime seems to ignore the procMount Unmasked option for the pod. Docker ce 18.09.3, API Version 1.39, Kubernetes v.1.5.0 with the feature gate, no Psp was enabled.

The kubelet logs reports that it sets the Unmasked ProcMount as expected, but the runtime does not.

[towolf]

I tried replacing dockerd with containerd (1.2.6) today, and now it works with the same input YAML.

Successfully built docker.io/library/irssi:latest

[towolf]

However I get this message: time="2019-04-08T21:13:22Z" level=warning msg="Process sandbox is not available, consider unmasking procfs: mount: permission denied (are you root?)\n"

What does this mean @AkihiroSuda?

[AkihiroSuda]

It means PID namespace is not unshared because procfs is masked

[towolf]

@AkihiroSuda how can I tell from the host system or from the container runtime that the unmasking worked successfully or not?

Because I have this now and with containerd img build worked, however it printed this error message. And your PR #221 was not in 0.5.6 yet, was it?

containers:
 -  image: gitlab.example.com:5005/img:v0.5.6 
    securityContext:                               
      capabilities:                                
        add:                                       
        - SETUID                                   
        - SETGID                                   
      privileged: false                            
      procMount: Unmasked                          
      runAsUser: 1000                              

EDIT after testing more:

Okay, when I modify to use root, then the error disappears. But I thought that this is supposed to work without root?

securityContext: 
  runAsUser: 0   

[AkihiroSuda]

how can I tell from the host system or from the container runtime that the unmasking worked successfully or not?

kubectl exec foo grep "^tmpfs /proc/.*" /proc/mounts shows masks if masked

Because I have this now and with containerd img build worked, however it printed this error message. And your PR #221 was not in 0.5.6 yet, was it?

@jessfraz could you consider releasing v0.6.0?

[towolf]

@AkihiroSuda

kubectl exec foo grep "^tmpfs /proc/.*" /proc/mounts shows masks if masked

So I reckon that this is bad, from inside the container?

/ $ grep "^tmpfs /proc/.*" /proc/mounts
tmpfs /proc/acpi tmpfs ro,relatime 0 0
tmpfs /proc/kcore tmpfs rw,nosuid,size=65536k,mode=755 0 0
tmpfs /proc/keys tmpfs rw,nosuid,size=65536k,mode=755 0 0
tmpfs /proc/timer_list tmpfs rw,nosuid,size=65536k,mode=755 0 0
tmpfs /proc/sched_debug tmpfs rw,nosuid,size=65536k,mode=755 0 0
tmpfs /proc/scsi tmpfs ro,relatime 0 0

My containerd says this

crictl inspectp 04814c79697a4
        "maskedPaths": [
          "/proc/acpi",
          "/proc/asound",
          "/proc/kcore",
          "/proc/keys",
          "/proc/latency_stats",
          "/proc/timer_list",
          "/proc/timer_stats",
          "/proc/sched_debug",
          "/sys/firmware",
          "/proc/scsi"
        ],
        "readonlyPaths": [
          "/proc/bus",
          "/proc/fs",
          "/proc/irq",
          "/proc/sys",
          "/proc/sysrq-trigger"
        ]
      "mounts": [
        {         
          "destination": "/proc",
          "type": "proc",
          "source": "proc",
          "options": [
            "nosuid",
            "noexec",
            "nodev"
          ]
        },

But my Pod contains this:

    securityContext:
      capabilities:  
        add:                         
        - SETUID    
        - SETGID          
      privileged: false
      procMount: Unmasked              
      runAsUser: 1000

And in kubelet log I can spot this: &SecurityContext{Capabilities:&Capabilities{Add:[SETUID SETGID],Drop:[],},Privileged:*false,SELinuxOptions:nil,RunAsUser:*1000,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:*Unmasked,} true false false}]

So, I don't get it. Why does Unmasked not work?

Containerd is at version 1.2.6.

[oz123]

Later k8s version don't even have procMount: Unmasked, trying to set this value, it will be changed to Default. What works is:

securityContext:
   privileged: true

Which seems like an overkill. Also, in that case, the annotations are obsolete.

[czunker]

@oz123 Came across the same problem. According to this https://github.com/hpcng/singularity/issues/5857#issuecomment-791814313, you can enable with an API server feature gate.