universalforwarder container fails to start when using k8s securityContext

[flyattimes]

trafficstars

I really hope I didn't miss anything obvious (I might have..), here is the problem.

Description

When using the securityContext as described here, the container for the universalforwarder will fail to start due to insufficient permissions. I am using universalforwarder version 8.1.

An example error message looks like this: [0;31mtar: system/README/messages.conf.example: Cannot open: No such file or directory[0m [0;31mtar: system/README: Cannot mkdir: Permission denied[0m

Expected behavior

The universalforwarder container should start without error.

Config files

These are the files I am currently running.

The universalforwarder Dockerfile is quite vanilla

FROM splunk/universalforwarder:8.1
COPY server.conf /opt/splunkforwarder/etc/system/local/server.conf
COPY deploymentclient.conf /opt/splunkforwarder/etc/system/local/deploymentclient.conf
COPY outputs.conf /opt/splunkforwarder/etc/apps/app/local/outputs.conf
COPY certs/ /mnt/certs/
EXPOSE 9997

Deployment.yaml (universalforwarder container):

- image: REDACTED
        imagePullPolicy: Always
        name: splunk-forwarder
        env:
          - name: SPLUNK_HOME_OWNERSHIP_ENFORCEMENT
            value: "false"
          - name: SPLUNK_PASSWORD
            valueFrom:
                secretKeyRef:
                  name: splunk-forwarder-local-user
                  key: password
          - name: SPLUNK_START_ARGS
            value: "--accept-license"
        securityContext:
          privileged: false
          allowPrivilegeEscalation: false
        ports:
          - name: tcp
            containerPort: 9997
            protocol: TCP

Deployment.yaml (spec.template.spec.securityContext):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: REDACTED
  namespace: REDACTED
spec:
  replicas: 1
  selector:
    #These labels are required for referencing other k8s objects
    matchLabels:
      app: "APP"
      servicegroup: "APPAPP"
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: "APP"
        servicegroup: "APPAPP"
      annotations:
    spec:
      securityContext:
          runAsUser: 41812
          fsGroup: 41812
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault