"cp: cannot create regular file '/opt/bitnami/airflow/airflow.cfg': Permission denied" in k8s-executor-init-config container for Airflow worker pod

[MarijnMB]

Name and Version

bitnami/airflow

What architecture are you using?

amd64

What steps will reproduce the bug?

1. Install Airflow using the bitnami helm chart on a kubernetes cluster with the 'kubernetesExecutor'
2. Trigger an airflow pipeline
3. Watch the k8s-executor-init-config container in the Airflow worker pod (pod name is according to pipeline that is run).
4. The Container tries to copy the airflow.cfg file (as seen in the source code for the container)

Are you using any custom parameters or values?

No response

What is the expected behavior?

Container copies the cfg file correctly

What do you see instead?

cp: cannot create regular file '/opt/bitnami/airflow/airflow.cfg': Permission denied

Additional information

I think this is related to securityContext changes in the bitnami/airflow helm chart release. I previously had another permissions issue in this thread: https://github.com/bitnami/charts/issues/24935

[MarijnMB]

Possibly (probably?) related to this: https://github.com/bitnami/charts/issues/25374

[dgomezleon]

Hi @MarijnMB,

Sorry for the delay. Could you please let us know if you are using any custom values (apart from kubernetesExecutor) so we can try to reproduce the issue?

[MarijnMB]

Oops - that didn't work, give me a sec.

[MarijnMB]

Attached to this comment should be the values file I'm using, with our git repo links anonymized. Renamed to .txt because I can't seem to attach .yaml files. bitnami-airflow-values.txt

The file contains some parametrized values (look for ${.*}) that are filled using terraform.

[dgomezleon]

Hi @MarijnMB

You probably need to add a new initContainer (create-default-config) in templates/config/configmap.yaml as was done in this PR for bitnami/airflow/templates/worker/statefulset.yaml Could you please give a try?

[MarijnMB]

There already is a k8s-executor-init-config container for k8s, which is the container that is giving me the error (and which is also doing something similar to what the create-default-config container is doing).

The k8s-executor-init-config container runs the command airflow_generate_config, which tries to create the config file and fails, since the output file is being written to /opt/bitnami/airflow/airflow.cfg, which is outside of the volumemount scope of the /opt/bitnami/airflow/k8s-executor-config volumemount for that container.

Since the /opt/bitnami/airflow/ folder is ~probably~(confirmed) the folder that the airflow_generate_config command is trying to copy the default config from, we can't just mount an emptydir to that folder ~I presume~ (confirmed).

This is my config that fixes (part of, keep reading) it: bitnami/airflow/templates/config/configmap.yaml

      initContainers: {{- include "airflow.git.containers.clone" (dict "securityContext" .Values.worker.containerSecurityContext "context" $) | trim | nindent 8 }}
        - name: k8s-executor-init-config
          image: {{ include "airflow.workerImage" . }}
          imagePullPolicy: {{ .Values.worker.image.pullPolicy }}
          {{- if .Values.worker.containerSecurityContext.enabled }}
          securityContext: {{- include "common.compatibility.renderSecurityContext" (dict "secContext" .Values.worker.containerSecurityContext "context" $) | nindent 12 }}
          {{- end }}
          command:
            - /bin/bash
          args:
            - -ec
            - |
                . /opt/bitnami/scripts/airflow-worker-env.sh
                . /opt/bitnami/scripts/libairflowworker.sh
                export AIRFLOW_CONF_FILE=/tmp/airflow.cfg
                airflow_generate_config # Generate the config file
                cp /tmp/airflow.cfg /k8s-executor-conf/airflow.cfg
          env:
            {{- include "airflow.configure.airflow.common" . | nindent 12 }}
            {{- include "airflow.configure.database" . | nindent 12 }}
            {{- include "airflow.configure.redis" . | nindent 12 }}
            {{- include "airflow.configure.airflow.kubernetesExecutor" . | nindent 12 }}
            - name: AIRFLOW_EXECUTOR
              value: {{ .Values.executor }}
            - name: AIRFLOW_WEBSERVER_HOST
              value: {{ include "common.names.fullname" . }}
            - name: AIRFLOW_WEBSERVER_PORT_NUMBER
              value: {{ .Values.service.ports.http | quote }}
          {{- if .Values.worker.resources }}
          resources: {{- toYaml .Values.worker.resources | nindent 12 }}
          {{- else if ne .Values.worker.resourcesPreset "none" }}
          resources: {{- include "common.resources.preset" (dict "type" .Values.worker.resourcesPreset) | nindent 12 }}
          {{- end }}
          volumeMounts:
            - name: empty-dir
              mountPath: /tmp
              subPath: tmp-dir
            - name: empty-dir
              mountPath: /k8s-executor-conf
              subPath: app-k8s-executor-conf-dir
        {{- if .Values.initContainers }}
        {{- include "common.tplvalues.render" (dict "value" .Values.initContainers "context" $) | trim | nindent 8 }}
        {{- end }}
        {{- if .Values.worker.initContainers }}
        {{- include "common.tplvalues.render" (dict "value" .Values.worker.initContainers "context" $) | trim | nindent 8 }}
        {{- end }}

Now the init container succeeds, but the worker container fails on the following:

airflow-worker 17:06:16.79 INFO  ==> Enabling non-root system user with nss_wrapper                                                                                                    │
│ /opt/bitnami/scripts/airflow-worker/entrypoint.sh: line 23: /opt/bitnami/airflow/nss-wrapper/nss_passwd: Read-only file system

So I've added

          volumeMounts:
            - name: empty-dir
              mountPath: /opt/bitnami/airflow/nss-wrapper
              subPath: app-nss-wrapper-dir
            - name: empty-dir
              mountPath: /tmp
              subPath: tmp-dir
            - name: empty-dir
              mountPath: /opt/bitnami/airflow/logs
              subPath: app-logs-dir
            - name: empty-dir
              mountPath: /opt/bitnami/airflow/tmp
              subPath: app-tmp-dir

to the volumeMounts for the worker in bitnami/airflow/templates/config/configmap.yaml. This matches the volumeMounts in the PR you mentioned @dgomezleon. I omitted the volumeMount for the db here, I'm not sure it's needed and we're running a separate instance anyway.

This makes the runner start the job.

[dgomezleon]

Hi @MarijnMB

Thanks a lot for sharing your feedback. Would you like to contribute with a PR? The Bitnami team will be excited to review your submission and offer feedback. You can find the contributing guidelines here.

[MarijnMB]

The PR is here: https://github.com/bitnami/charts/pull/26307 but I'd rather not have my real name out there (sign the commits), so do with it what you will.