kube-psp-advisor icon indicating copy to clipboard operation
kube-psp-advisor copied to clipboard

Published 20 hours ago • verified publisher icon sysdiglabs

WIP: Setting default psp

Open paper2 opened this issue 4 years ago • 11 comments

I want to set default PSP when converting.

kube-psp-advisor is very cool tool!! We use this tool for creating psp.

kube-psp-advisor convert however makes some fields which are not defined in a manifest loose rules. EX) Default setting of runAsUser is "RunAsAny".

We want to change these default settings. Therefore I have tried to make prototype which read default psp setting form a file.

If this idea is accepted, I do rest work to do.(Apply all fields, make tests...) How do you think?

I write example in following.

This is sample deployment manifest.

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  creationTimestamp: "2020-05-23T12:12:08Z"
  generation: 1
  labels:
    k8s-app: kube-dns
  name: coredns
  namespace: kube-system
  resourceVersion: "14048"
  selfLink: /apis/apps/v1/namespaces/kube-system/deployments/coredns
  uid: c0bb78cc-fbb3-4ede-9f95-1a54b87b8775
spec:
  progressDeadlineSeconds: 600
  replicas: 2
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: kube-dns
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        k8s-app: kube-dns
    spec:
      containers:
      - args:
        - -conf
        - /etc/coredns/Corefile
        image: k8s.gcr.io/coredns:1.6.2
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 5
          httpGet:
            path: /health
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        name: coredns
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        - containerPort: 9153
          name: metrics
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /ready
            port: 8181
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            memory: 170Mi
          requests:
            cpu: 100m
            memory: 70Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - all
          readOnlyRootFilesystem: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/coredns
          name: config-volume
          readOnly: true
      dnsPolicy: Default
      nodeSelector:
        beta.kubernetes.io/os: linux
      priorityClassName: system-cluster-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: coredns
      serviceAccountName: coredns
      terminationGracePeriodSeconds: 30
      tolerations:
      - key: CriticalAddonsOnly
        operator: Exists
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
      volumes:
      - configMap:
          defaultMode: 420
          items:
          - key: Corefile
            path: Corefile
          name: coredns
        name: config-volume
status:
  availableReplicas: 2
  conditions:
  - lastTransitionTime: "2020-05-23T12:12:15Z"
    lastUpdateTime: "2020-05-23T12:12:24Z"
    message: ReplicaSet "coredns-5644d7b6d9" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2020-05-30T07:17:32Z"
    lastUpdateTime: "2020-05-30T07:17:32Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 1
  readyReplicas: 2
  replicas: 2
  updatedReplicas: 2

This is sample default psp file.

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: example
spec:
  seLinux:
    rule: RunAsAny
  runAsUser:
    rule: MustRunAs
    ranges:
      - min: 1
        max: 65535
  fsGroup:
    rule: RunAsAny
  volumes:
  - '*'

Without default psp file.

 ./kube-psp-advisor convert --podFile test-deploy.yaml --pspFile output-psp.yaml && cat output-psp.yaml                                                                        [~/go/src/kube-psp-advisor]
INFO[2020-05-31T10:12:23+09:00] Wrote generated psp to output-psp.yaml
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  creationTimestamp: null
  name: pod-security-policy-default-20200531101223
spec:
  allowPrivilegeEscalation: false
  defaultAddCapabilities:
  - NET_BIND_SERVICE
  fsGroup:
    rule: RunAsAny
  hostPorts:
  - max: 0
    min: 0
  readOnlyRootFilesystem: true
  requiredDropCapabilities:
  - all
  runAsGroup:
    rule: RunAsAny
  runAsUser:
    rule: RunAsAny
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  volumes:
  - configMap
  - secret

with default psp file

$ ./kube-psp-advisor convert --podFile test-deploy.yaml --pspFile output-psp.yaml --defaultPspFile ./test-psp.yaml && cat output-psp.yaml                                       [~/go/src/kube-psp-advisor]
INFO[2020-05-31T10:13:05+09:00] Wrote generated psp to output-psp.yaml
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  creationTimestamp: null
  name: pod-security-policy-default-20200531101305
spec:
  allowPrivilegeEscalation: false
  defaultAddCapabilities:
  - NET_BIND_SERVICE
  fsGroup:
    rule: RunAsAny
  hostPorts:
  - max: 0
    min: 0
  readOnlyRootFilesystem: true
  requiredDropCapabilities:
  - all
  runAsGroup:
    rule: RunAsAny
  runAsUser:                           <= replaced by default setting
    ranges:
    - max: 65535
      min: 1
    rule: MustRunAs
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  volumes:
  - configMap
  - secret

paper2 avatar May 31 '20 01:05 paper2

@paper2 thanks for PR, let me think about it and get back to you soon.

Kaizhe avatar May 31 '20 05:05 Kaizhe

I have difficulty in making a psp because kube-psp-advisor sets required fields as loose policy automatically.

Therefore I want to set default psp firstly for only fsGroup, runAsUser, seLinux, supplementalGroups. If another want to set other fields, I make it.

paper2 avatar Jun 02 '20 11:06 paper2

in making a psp because kube-psp-advisor sets required fields as loose policy automatically.

I don't think that's the case. The default settings follow the psp default value. To me, your use case is having a customized default settings.

Kaizhe avatar Jun 02 '20 17:06 Kaizhe

I don't think that's the case. The default settings follow the psp default value. To me, your use case is having a customized default settings.

That makes sense.

paper2 avatar Jun 04 '20 12:06 paper2

I originally thought this was about a customized default settings will take the priority over the suggested one. However, If you look at your example:

  1. the runAsUser in the customized default takes the precedence
  2. the volumes in the customized default DID NOT takes the precedence.

This is some confusion I have. Probably you want to rethink about this PR as well as the customized default settings.

Kaizhe avatar Jun 04 '20 17:06 Kaizhe

I originally thought this was about a customized default settings will take the priority over the suggested one.

My prototype uses the default psp when it can't suggest fields. Please let me show example following.

This is default psp file.

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: example
spec:
  seLinux:
    rule: RunAsAny
  runAsUser:
    rule: MustRunAs
    ranges:
      - min: 1
        max: 65535
  fsGroup:
    rule: RunAsAny
  volumes:
  - '*'

This is a sample Deployment which has runAsUser.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: coredns
spec:
  selector:
    matchLabels:
      k8s-app: kube-dns
  template:
    metadata:
      labels:
        k8s-app: kube-dns
    spec:
      containers:
      - args:
        - -conf
        - /etc/coredns/Corefile
        image: k8s.gcr.io/coredns:1.6.2
        imagePullPolicy: IfNotPresent
        name: coredns
        securityContext:
          runAsUser: 1000
          allowPrivilegeEscalation: false
          capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - all
          readOnlyRootFilesystem: true

In this case, kube-psp-advisor can suggest RunAsUser. Even if I give a default psp file, kube-psp-advisor ignores the one.

$ ./kube-psp-advisor convert --podFile test-deploy.yaml --pspFile output-psp.yaml --defaultPspFile ./test-psp.yaml && cat output-psp.yaml                                       [~/go/src/kube-psp-advisor]
INFO[2020-06-05T06:53:07+09:00] Wrote generated psp to output-psp.yaml
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  creationTimestamp: null
  name: pod-security-policy-default-20200605065307
spec:
  allowPrivilegeEscalation: false
  defaultAddCapabilities:
  - NET_BIND_SERVICE
  fsGroup:
    rule: RunAsAny
  readOnlyRootFilesystem: true
  requiredDropCapabilities:
  - all
  runAsGroup:
    rule: RunAsAny
  runAsUser:
    ranges:               <= kube-psp-advisor can suggest this and ignore default psp.
    - max: 1000
      min: 1000
    rule: MustRunAs
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  volumes:
  - secret

On the other hand, kube-psp-advisor uses the default psp settings when deployment has not RunAsUser .

apiVersion: apps/v1
kind: Deployment
metadata:
  name: coredns
spec:
  selector:
    matchLabels:
      k8s-app: kube-dns
  template:
    metadata:
      labels:
        k8s-app: kube-dns
    spec:
      containers:
      - args:
        - -conf
        - /etc/coredns/Corefile
        image: k8s.gcr.io/coredns:1.6.2
        imagePullPolicy: IfNotPresent
        name: coredns
        securityContext:                    <= Delete RunAsUser
          allowPrivilegeEscalation: false
          capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - all
          readOnlyRootFilesystem: true

$ ./kube-psp-advisor convert --podFile test-deploy.yaml --pspFile output-psp.yaml --defaultPspFile ./test-psp.yaml && cat output-psp.yaml                                       [~/go/src/kube-psp-advisor]
INFO[2020-06-05T06:53:28+09:00] Wrote generated psp to output-psp.yaml
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  creationTimestamp: null
  name: pod-security-policy-default-20200605065328
spec:
  allowPrivilegeEscalation: false
  defaultAddCapabilities:
  - NET_BIND_SERVICE
  fsGroup:
    rule: RunAsAny
  readOnlyRootFilesystem: true
  requiredDropCapabilities:
  - all
  runAsGroup:
    rule: RunAsAny
  runAsUser:                    <= kube-psp-advisor uses the default psp.
    ranges:
    - max: 65535
      min: 1
    rule: MustRunAs
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  volumes:
  - secret

Sorry for the confusion :(

paper2 avatar Jun 04 '20 22:06 paper2 

    securityContext:                    <= Delete RunAsUser

This is using the default value which is root user.

Let discuss the use case here, assuming there is a customized default psp (please allow me to add customized), for individual constraint (runAsRoot, runAsGroup, volume etc.):

  1. If a constraint is more loose in the customized default psp, do you plan to stick with the original one suggested by psp-advisor?
  2. What if there are mismatches between the customized default psp and the original one suggested by psp-advisor: a. volumes: secret, configmap in default pap b. volumes: hostpath, secret in the original one suggested by psp-advisor

Kaizhe avatar Jun 05 '20 21:06 Kaizhe

If a constraint is more loose in the customized default psp, do you plan to stick with the original one suggested by psp-advisor?

Yes. In my case, I basically want to use more restricted policy as much as possible.

What if there are mismatches between the customized default psp and the original one suggested by psp-advisor:

I want the result includes only the original suggested by psp-advisor. I assume this default settings is used when psp-advisor can't find explicit fields.

paper2 avatar Jun 05 '20 23:06 paper2

I want the result includes only the original suggested by psp-advisor. I assume this default settings is used when psp-advisor can't find explicit fields.

I don't understand. Please take a look:

//PodSecurityPolicy Recommendation System help in the following attributes:
//	1. allowPrivilegeEscalation - done
//	2. allowedCapabilities - done
//	3. allowedHostPaths - done
//	4. hostIPC - done
//	5. hostNetwork - done
//	6. hostPID - done
//	7. hostPorts - done
//	8. privileged - done
//	9. readOnlyRootFilesystem - done
//  10. runAsUser - done
//  11. runAsGroup - done
//  12. Volume - done
//	13. seLinux and others - need further investigation

type ContainerSecuritySpec struct {
	Metadata                 Metadata `json:"parentMetadata"`
	ContainerID              string   `json:"containerID"`
	ContainerName            string   `json:"containerName"`
	PodName                  string   `json:"podName"`
	Namespace                string   `json:"namespace"`
	ImageName                string   `json:"imageName"`
	ImageSHA                 string   `json:"imageSHA"`
	HostName                 string   `json:"hostName"`
	Capabilities             []string `json:"effectiveCapabilities,omitempty"`
	DroppedCap               []string `json:"droppedCapabilities,omitempty"`
	AddedCap                 []string `json:"addedCapabilities,omitempty"`
	Privileged               bool     `json:"privileged,omitempty"`
	ReadOnlyRootFS           bool     `json:"readOnlyRootFileSystem,omitempty"`
	RunAsNonRoot             *bool    `json:"runAsNonRoot,omitempty"`
	AllowPrivilegeEscalation *bool    `json:"allowPrivilegeEscalation,omitempty"`
	RunAsUser                *int64   `json:"runAsUser,omitempty"`
	RunAsGroup               *int64   `json:"runAsGroup,omitempty"`
	HostPorts                []int32  `json:"hostPorts,omitempty"`
	ServiceAccount           string   `json:"serviceAccount,omitempty"`
}

Kaizhe avatar Jun 05 '20 23:06 Kaizhe

If a manifest read by psp-advisor has no volume field, psp-advisor set psp customized default psp. If a manifest read by psp-advisor has one or more volume field, psp-advisor set psp these volume and ignroe customized default psp.

a. volumes: secret, configmap in (customized) default pap b. volumes: hostpath, secret in the original one suggested by psp-advisor

Above case, psp-advisor ignores customized default psp and suggest only hostpath and secret only.

Convert command creates a psp using a source manifest. If source manifest has not explicit fields(e.g. runAsUser), psp-advisor use default value of securityContext. In this case only, I want to use customized default psp. If source manifest has an explicit fields(e.g. runAsUser), psp-advisor suggests psp which fit to the fields. In this case, I want to use suggested one and ignore customized default psp.

paper2 avatar Jun 06 '20 05:06 paper2

Convert command creates a psp using a source manifest. If source manifest has not explicit fields(e.g. runAsUser), psp-advisor use default value of securityContext. In this case only, I want to use customized default psp. If source manifest has an explicit fields(e.g. runAsUser), psp-advisor suggests psp which fit to the fields. In this case, I want to use suggested

Now I understand what you suggested. This may complicates the use case. I need to think about it.

Kaizhe avatar Jun 06 '20 06:06 Kaizhe