Correct Pod Security Policies (securityContext) for singleuser jupyter pods and containers

[scottyhq]

Our JupyterHub images are meant to run as a non-root jovyan user with uid:gid 1000:1000 (https://github.com/pangeo-data/pangeo-stacks-dev/blob/fc42935aa20003890d8aab48ef6475e7e10e82fb/base-image/Dockerfile#L24), but currently containers run with the following securityContext:

(abbreviated)

kind: Pod
  name: jupyter-scottyhq
spec:
  containers:
  - args:
    - jupyterhub-singleuser
    - --ip=0.0.0.0
    - --port=8888
    - --NotebookApp.default_url=/lab
    image: uwhackweeks/icesat2:latest
    imagePullPolicy: IfNotPresent
    lifecycle: {}
    name: notebook
    ports:
    - containerPort: 8888
      name: notebook-port
      protocol: TCP
    securityContext:
      runAsGroup: 0
      runAsUser: 1000
  securityContext:
    fsGroup: 100
  serviceAccount: daskkubernetes

Based on discussion for binderhub config https://github.com/pangeo-data/pangeo-binder/pull/136 I think we should change this to

  containers:
  - args:
    - jupyterhub-singleuser
    ...
    securityContext:
      runAsGroup: 1000
      runAsUser: 1000
  securityContext:
    fsGroup: 1000

These can be set under values.yaml here, but I'm not sure about runAsGroup? https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/2d435d64d4978380f68a1bda185add4376c29272/jupyterhub/values.yaml#L188-L189

cc @consideRatio , @jhamman , @tjcrone

[consideRatio]

Rubber duck writing...

Hmm jupyter/docker-stacks images have a history of using GID of 100, I don't know why or why not. But 1000 is common. The JupyterHub helm charts configuration options isn't so intuitive or systematic about this...

There is hub.uid (default: 1000) / hub.fsGid (default: 1000), and singleuser.uid (default: 1000) / singleuser.fsGid (default: 100), and no configuration of other pods (proxy, autohttps, prepuller pods, etc, user-placeholder pods)

wait... what is fsGroup in kubernetes really?

1. It is not something part of the container's security context: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#securitycontext-v1-core
2. It is something that is part of the pod security context: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#podsecuritycontext-v1-core, which is a bit different while they have similar options that will be possible to be overridden by the container's securityContext.

   PodSecurityContext holds pod-level security attributes and common container settings. Some fields are also present in container.securityContext. Field values of container.securityContext take precedence over field values of PodSecurityContext.

fsGroup:

A special supplemental group that applies to all containers in a pod. Some volume types allow the Kubelet to change the ownership of that volume to be owned by the pod:

1. The owning GID will be the FSGroup
2. The setgid bit is set (new files created in the volume will be owned by FSGroup)
3. The permission bits are OR'd with rw-rw---- If unset, the Kubelet will not modify the ownership and permissions of any volume.

So... Then we don't configure the runAsGroup field at all? No i don't think we do. And then, from reading the docs in kubespawner aboug gid and fs_gid I think we may have 0 here because our the Dockerimage contained such info? Hmm?

There are soo much nested stuff going on here. I'll just drop this rubber duck exploration as it is.

I think the z2jh helm chart should allow the run_as_gid to be set, which maps to the kubespawner gid configuration, but it currently isn't something that can be configured in the z2jh helm chart

[scottyhq]

There are soo much nested stuff going on here.

Thanks for looking into this @consideRatio. Agreed ;)

wait... what is fsGroup in kubernetes really?

Without digging into the details, it seems the fsGroup number doesn't really matter, we just need the field to be present. The reason we need to set this (at least on AWS) is to link a pod service accounts to IAM roles (https://github.com/pangeo-data/pangeo-binder/issues/137). The EKS docs explain:

By default, only containers that run as root have the proper file system permissions to read the web identity token file. You can provide these permissions by having your containers run as root, or by providing the following security context for the containers in your manifest. The fsGroup ID is arbitrary, and you can choose any valid group ID. For more information about the implications of setting a security context for your pods, see Configure a Security Context for a Pod or Container in the Kubernetes documentation.