Cannot set readOnlyRootFilesystem for ui container

[zsolooo]

Describe the bug As a mandatory K8S cluster hardening policy in our organisation it's required to set readOnlyRootFilesystem for each container inside a pod. The actual issues are with the docker-entrypoint script: https://github.com/opencost/opencost/blob/develop/ui/docker-entrypoint.sh#L21 It tries to write a few files to the root volume (/var/www and /etc/nginx/conf.d). In case of /var/www it was possible to workaround the issue by attaching a writable emptyDir volume to /var/www, however the same approach was not possible with /etc/nginx/conf.d, because mounting a volume on that path hides files that are already existing on that path (entrypoint script tries to read config template to use for envsubst).

This issue makes the pod unable to start (CrashLoopBackoff).

To Reproduce with the tried workaround Steps to reproduce the behavior:

1. Deploy opencost via this helmchart: https://github.com/opencost/opencost-helm-chart , chart-version: 1.32.0 Using the following helm values:

   opencost:
     exporter:
       startupProbe:
         enabled: false
       livenessProbe:
         enabled: false
       readinessProbe:
         enabled: false
     ui:
       securityContext:
         capabilities:
           drop:
           - ALL
         allowPrivilegeEscalation: false
         readOnlyRootFilesystem: true
         runAsNonRoot: true
         runAsUser: 1000
   

2. Check the ui container logs with kubectl
3. See error:

cp: can't create '/var/www/MaterialIcons-Regular.12b3b105.woff': Read-only file system
cp: can't create '/var/www/MaterialIcons-Regular.333251c4.ttf': Read-only file system
cp: can't create '/var/www/MaterialIcons-Regular.b99eb5ce.woff2': Read-only file system
cp: can't create '/var/www/MaterialIcons-Regular.e9e55c63.eot': Read-only file system
cp: can't create '/var/www/favicon.7eff484d.ico': Read-only file system
cp: can't create '/var/www/index.3406705b.css': Read-only file system
cp: can't create '/var/www/index.3406705b.css.map': Read-only file system
cp: can't create '/var/www/index.4170b679.js': Read-only file system
cp: can't create '/var/www/index.4170b679.js.map': Read-only file system
cp: can't create '/var/www/index.6d70e179.js': Read-only file system
cp: can't create '/var/www/index.6d70e179.js.map': Read-only file system
cp: can't create '/var/www/index.html': Read-only file system
cp: can't create '/var/www/index.runtime.256cff63.js': Read-only file system
cp: can't create '/var/www/index.runtime.256cff63.js.map': Read-only file system
cp: can't create '/var/www/index.runtime.997a1df7.js': Read-only file system
cp: can't create '/var/www/index.runtime.997a1df7.js.map': Read-only file system
cp: can't create '/var/www/logo.b9464e00.png': Read-only file system

To Reproduce with the described workaround (still failing) Steps to reproduce the behavior:

1. Deploy opencost via this helmchart: https://github.com/opencost/opencost-helm-chart , version: 1.32.0 Using the following helm values:

   opencost:
     exporter:
       startupProbe:
         enabled: false
       livenessProbe:
         enabled: false
       readinessProbe:
         enabled: false
     ui:
       securityContext:
         capabilities:
           drop:
           - ALL
         allowPrivilegeEscalation: false
         readOnlyRootFilesystem: true
         runAsNonRoot: true
         runAsUser: 1000
       extraVolumeMounts:
       - mountPath: /var/www
         name: ui-www
       - mountPath: /etc/nginx/conf.d
         name: ui-nginx-config
     extraVolumes:
     - name: ui-www
       emptyDir: {}
     - name: ui-nginx-config
       emptyDir: {}
   

2. Check the ui container logs with kubectl
3. See error:

'/opt/ui/dist/MaterialIcons-Regular.12b3b105.woff' -> '/var/www/MaterialIcons-Regular.12b3b105.woff'
'/opt/ui/dist/MaterialIcons-Regular.333251c4.ttf' -> '/var/www/MaterialIcons-Regular.333251c4.ttf'
'/opt/ui/dist/MaterialIcons-Regular.b99eb5ce.woff2' -> '/var/www/MaterialIcons-Regular.b99eb5ce.woff2'
'/opt/ui/dist/MaterialIcons-Regular.e9e55c63.eot' -> '/var/www/MaterialIcons-Regular.e9e55c63.eot'
'/opt/ui/dist/favicon.7eff484d.ico' -> '/var/www/favicon.7eff484d.ico'
'/opt/ui/dist/index.3406705b.css' -> '/var/www/index.3406705b.css'
'/opt/ui/dist/index.3406705b.css.map' -> '/var/www/index.3406705b.css.map'
'/opt/ui/dist/index.4170b679.js' -> '/var/www/index.4170b679.js'
'/opt/ui/dist/index.4170b679.js.map' -> '/var/www/index.4170b679.js.map'
'/opt/ui/dist/index.6d70e179.js' -> '/var/www/index.6d70e179.js'
'/opt/ui/dist/index.6d70e179.js.map' -> '/var/www/index.6d70e179.js.map'
'/opt/ui/dist/index.html' -> '/var/www/index.html'
'/opt/ui/dist/index.runtime.256cff63.js' -> '/var/www/index.runtime.256cff63.js'
'/opt/ui/dist/index.runtime.256cff63.js.map' -> '/var/www/index.runtime.256cff63.js.map'
'/opt/ui/dist/index.runtime.997a1df7.js' -> '/var/www/index.runtime.997a1df7.js'
'/opt/ui/dist/index.runtime.997a1df7.js.map' -> '/var/www/index.runtime.997a1df7.js.map'
'/opt/ui/dist/logo.b9464e00.png' -> '/var/www/logo.b9464e00.png'
running with BASE_URL=/model
/usr/local/bin/docker-entrypoint.sh: line 21: can't open /etc/nginx/conf.d/default.nginx.conf.template: no such file

Expected behavior Pod starts up correctly with readonly root filesystem

Screenshots Not relevant

Which version of OpenCost are you using? OpenCost: v1.109.0 Helm chart: v1.32.0

Additional context None

[mkilchhofer]

I also wanted to raise an issue for this. We have the same approach forcing every workload to set the field pod.spec.containers.securityContext.readOnlyRootFilesystem to true.

For /var/www there was a workaround as mentioned until

• https://github.com/opencost/opencost/pull/2366/

was implemented. Now we are unable to workaround this.

---

Update: Found a recent PR which should fix this:

• https://github.com/opencost/opencost/pull/2521

But it still seems not possible. @kaitimmer does it still work for you?

[kaitimmer]

But it still seems not possible. @kaitimmer does it still work for you?

Actually no, because I missed this second part. Currently, we run opencost with an exception to the rule. My hope is that this fixes all of the places where it is needed. But if you can check again, that would be great. 4-eyes more than 2 :)