Installation fails when PodSecurityPolicy applied with readOnlyRootFilesystem set to true

[marekzebro]

ISSUE TYPE

• Bug Report

SUMMARY

Operator and AWX Installation fails, when pod security policy is applied prohibiting pods running as root user, and write access to filesystem.

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: restricted
spec:
  allowPrivilegeEscalation: false
  defaultAllowPrivilegeEscalation: false
  fsGroup:
    rule: MustRunAs
    ranges:
    - min: 1000
      max: 65536
  hostPorts:
  - min: 0
    max: 0
  hostPID: false
  hostIPC: false
  hostNetwork: false
  privileged: false
  readOnlyRootFilesystem: true
  requiredDropCapabilities:
    - ALL
  runAsUser:
    rule: MustRunAsNonRoot
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: MustRunAs
    ranges:
    - min: 1000
      max: 65536
  volumes:
  - 'emptyDir'
  - 'persistentVolumeClaim'
  - 'secret'
  - 'downwardAPI'
  - 'configMap'
  - 'projected'
  - 'cephFS

ENVIRONMENT

• AWX version: 19.3.0
• Operator version: 0.13.0
• Kubernetes version: v1.21.2
• AWX install method: minikube on linux

STEPS TO REPRODUCE

Apply PSP policy to the cluster and follow standard installation procedure

EXPECTED RESULTS

It is a good security practice run pods as non root user, without write access to filesystem. AWX & Operator should be able cope with such requirements.

ADDITIONAL INFORMATION

Changes to default deployment.yaml.j2 file are needed to overwrite containers securityContext and run as non root user. It would be great to have an option in CRD to set default securityContext for all containers.

Changes applied to deployment.yaml.j2 allowing containers run with restrictions:

    ..
     initContainers:
        - name: init
         securityContext:
           runAsUser: 1000
      ..  
         name: redis
         securityContext:
           runAsUser: 999
      ..
         name: '{{ meta.name }}-web'
         securityContext:
           runAsUser: 1000
      ..
         name: '{{ meta.name }}-task'
         securityContext:
           runAsUser: 1000
      ..
         name: '{{ meta.name }}-ee'
         securityContext:
           runAsUser: 1000

Containers need numerous directories being writeable, possible fix is to attach them by default as emptyDir volumes:

web_extra_volume_mounts: |
  - name: web-tmp
     mountPath: /tmp
  - name: web-tower-logs
     mountPath: /var/log/tower
  - name: web-awx-public
     mountPath: /var/lib/awx/public
task_extra_volume_mounts: |
  - name: task-tmp
     mountPath: /tmp
  - name: task-tower-logs
     mountPath: /var/log/tower
  - name: ee-tmp
     mountPath: /tmp
ee_extra_volume_mounts: |
  - name: ee-awx-ansible
     mountPath: /var/lib/awx/.ansible
  - name: ee-home-runner-ansible
     mountPath: /home/runner/.ansible

Containers are modifying content of /etc/passwd files when UID is higher than 500. Without correct user mapping application would fail. Wouldn't be a better option to template passwd files and attach them as a configMap to each container?

This is part of a script failing when write restrictions are applied

/usr/bin/launch_awx.sh
#!/usr/bin/env bash
if [ `id -u` -ge 500 ]; then
    echo "awx:x:`id -u`:`id -g`:,,,:/var/lib/awx:/bin/bash" >> /tmp/passwd
    cat /tmp/passwd > /etc/passwd
    rm /tmp/passwd
fi

config-watcher (web & task) is trying write to /var/lib/awx/.configsha and fails to start as directory is non writable. It would be better to write into temporary directory instead.

cat /usr/bin/config-watcher
27 def main():
28     settings_file = "/etc/tower/settings.py"
29     hash_file = "/var/lib/awx/.configsha"

Supervisord on web & task container have a setting for uwsgi for writing a pipe to /var/lib/awx/awxfifo, again due to read only filesystem this would fail breaking functionality. Can this be moved to temporary directory i.e /tmp

/etc/supervisord.conf & /etc/supervisord_task.conf 
[program:uwsgi]
command = /var/lib/awx/venv/awx/bin/uwsgi --socket 127.0.0.1:8050 --module=awx.wsgi:application --vacuum --processes=5 --harakiri=120 --no-orphans --master --max-requests=1000 --master-fifo=/var/lib/awx/awxfifo --lazy-apps -b 32768
directory = /var/lib/awx

A worker container ( automation-job-* ) fails due to inability changing /etc/passwd file when restrictions are in place. This is being done by a script /usr/bin/entrypoint. Again having passwd file created accordingly before container creation would fix the problem.

Related issues: #560 #383

[adavid44]

Hello I'm also interested with the resolution of this issue Any chance to see it resolved one day ? :) Thanks !

[shubhamsingh987]

I'm getting [emerg] 43#43: mkdir() "/var/lib/nginx/tmp/client_body" failed (13: Permission denied) nginx: [emerg] mkdir() "/var/lib/nginx/tmp/client_body" failed (13: Permission denied) from awx-demo-web container inside awx pod log any workaround ?