Airflow error while running this image from your helm chart #2

[papagala]

Hey,

I'm trying to use a postgres back-end but keep getting this error. Have you come across this?

kubectl -n dev logs -p mlflow-airflow-webserver-b6778f8db-pmzfs 
Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 25, in <module>
    from airflow.configuration import conf
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/__init__.py", line 31, in <module>
    from airflow.utils.log.logging_mixin import LoggingMixin
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/__init__.py", line 24, in <module>
    from .decorators import apply_defaults as _apply_defaults
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/decorators.py", line 36, in <module>
    from airflow import settings
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/settings.py", line 37, in <module>
    from airflow.configuration import conf, AIRFLOW_HOME, WEBSERVER_CONFIG  # NOQA F401
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/configuration.py", line 622, in <module>
    conf.read(AIRFLOW_CONFIG)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/configuration.py", line 348, in read
    self._validate()
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/configuration.py", line 204, in _validate
    self.get('core', 'executor')))
airflow.exceptions.AirflowConfigException: error: cannot use sqlite with the KubernetesExecutor

[KIRY4]

I have same issue but with official airflow image + some additional dependencies installed...

 Warning  Unhealthy  2m3s (x196 over 4h54m)  kubelet, aks-toolset-22416452-vmss000001  Liveness probe failed: Traceback (most recent call last):
  File "<string>", line 4, in <module>
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/__init__.py", line 31, in <module>
    from airflow.utils.log.logging_mixin import LoggingMixin
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/__init__.py", line 24, in <module>
    from .decorators import apply_defaults as _apply_defaults
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/decorators.py", line 36, in <module>
    from airflow import settings
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/settings.py", line 37, in <module>
    from airflow.configuration import conf, AIRFLOW_HOME, WEBSERVER_CONFIG  # NOQA F401
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/configuration.py", line 731, in <module>
    conf.read(AIRFLOW_CONFIG)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/configuration.py", line 421, in read
    self._validate()
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/configuration.py", line 213, in _validate
    self._validate_config_dependencies()
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/configuration.py", line 247, in _validate_config_dependencies
    self.get('core', 'executor')))
airflow.exceptions.AirflowConfigException: error: cannot use sqlite with the KubernetesExecutor

[thyarles]

I have the same problem and I saw that the /opt/airflow/airflow.cfg, that is dynamically generated, has this config inside:

sql_alchemy_conn = sqlite:////opt/airflow/airflow.db

As I don't know exactly how the airflow.cfg is generated, I made a patch on the airflow-scheduler deployment because I know exactly how Kubernetes works.

Original airflow-scheduler deployment:

[...]
- args:
  - bash
  - -c
  - |-
    true \
     && mkdir -p /home/airflow/.local/bin \
     && echo 'export PATH="/home/airflow/.local/bin:$PATH"' >> /home/airflow/airflow_env.sh \
     && echo 'export AIRFLOW__CORE__SQL_ALCHEMY_CONN="postgresql+psycopg2://${DATABASE_USER}:$( echo ${DATABASE_PASSWORD} | python3 -c "import urllib.parse; encoded_pass = urllib.parse.quote(input()); print(encoded_pass)" )@${DATABASE_HOST}:${DATABASE_PORT}/${DATABASE_DB}${DATABASE_PROPERTIES}"' >> /home/airflow/airflow_env.sh \
     && echo 'source /home/airflow/airflow_env.sh' >> /home/airflow/.profile \
     && echo 'source /home/airflow/airflow_env.sh' >> /home/airflow/.bashrc \
     && source /home/airflow/airflow_env.sh \
     && echo "*** installing global extra pip packages..." \
     && pip install --user  "ldap3==2.8.1"  \
     && echo "*** executing Airflow initdb..." \
     && airflow initdb \
     && echo "*** adding Airflow variables..." \
     && airflow variables -i /home/airflow/variables-pools/variables.json \
     && echo "*** adding Airflow connections..." \
     && /home/airflow/connections/add-connections.sh \
     && echo "*** adding Airflow pools..." \
     && airflow pool -i /home/airflow/variables-pools/pools.json \
     && echo "*** running scheduler..." \
     && exec airflow scheduler -n -1
[...]

Changed airflow-scheduler deployment:

[...]
- args:
  - bash
  - -c
  - |-
    true \
     && mkdir -p /home/airflow/.local/bin \
     && echo 'export PATH="/home/airflow/.local/bin:$PATH"' >> /home/airflow/airflow_env.sh \
     && echo 'export AIRFLOW__CORE__SQL_ALCHEMY_CONN="postgresql+psycopg2://${DATABASE_USER}:$( echo ${DATABASE_PASSWORD} | python3 -c "import urllib.parse; encoded_pass = urllib.parse.quote(input()); print(encoded_pass)" )@${DATABASE_HOST}:${DATABASE_PORT}/${DATABASE_DB}${DATABASE_PROPERTIES}"' >> /home/airflow/airflow_env.sh \
     && echo 'source /home/airflow/airflow_env.sh' >> /home/airflow/.profile \
     && echo 'source /home/airflow/airflow_env.sh' >> /home/airflow/.bashrc \
     && source /home/airflow/airflow_env.sh \
     && echo "*** installing global extra pip packages..." \
     && pip install --user  "ldap3==2.8.1"  \
     && echo "*** executing Airflow initdb..." \
     && airflow initdb \
     && echo "*** patching airflow.cfg sql_alchemy_conn section..." \
     && sed -i "s#sql_alchemy_conn.*#sql_alchemy_conn = $AIRFLOW__CORE__SQL_ALCHEMY_CONN#" /opt/airflow/airflow.cfg \
     && echo "*** adding Airflow variables..." \
     && airflow variables -i /home/airflow/variables-pools/variables.json \
     && echo "*** adding Airflow connections..." \
     && /home/airflow/connections/add-connections.sh \
     && echo "*** adding Airflow pools..." \
     && airflow pool -i /home/airflow/variables-pools/pools.json \
     && echo "*** running scheduler..." \
     && exec airflow scheduler -n -1
[...]

It works for me.