changes in airflow.cfg file is not getting reflected through my docker compose airflow 1.10.7 setup

[ishanverma96]

Hi Team,

I am currently working on data pipleine with data quality checks solution. For that i have installed docker, under which i have installed airflow and pulled the repo using (git clone repo name). everything is working well and automated process is going good in airflow UI.

I also have one more requirement to send log files to S3 and for that i have tried every possible way from the past 15 days, but still getting bad luck.

I have passed the values in airflow.cfg file for s3 and made connection with s3 using admin UI(conn id as s3log and all the required parameters)

remote_logging = True remote_log_conn_id = s3log remote_base_log_folder = s3://bucket-name/log encrypt_s3_logs = False

The problem is whatever the changes i am doing in .cfg file is not getting reflected. changes can be anything timeout, worker changes, remote s3 etc and also deleting the airflow.cfg file from my directory made no impact in my airflow solution. everything is working fine without .cfg file.

but to do remote s3 logging, i have to pass parameters in config file but it is not taking any changes. only read local files in UI log output.

below is my docker-compose file:- version: '3.7' services: postgres: image: postgres:9.6 environment: - POSTGRES_USER=airflow - POSTGRES_PASSWORD=airflow - POSTGRES_DB=airflow logging: options: max-size: 10m max-file: "3"

webserver:
    image: puckel/docker-airflow:1.10.7
    restart: always
    depends_on:
        - postgres 
    environment:
        - LOAD_EX= False
        - EXECUTOR=Local

    logging:
        options:
            max-size: 10m
            max-file: "3"
    volumes:
        - ./dags:/usr/local/airflow/dags
        # - ./plugins:/usr/local/airflow/plugins
    ports:
        - "8080:8080"
    command: webserver
    healthcheck:
        test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
        interval: 30s
        timeout: 300s
        retries: 1

and my docker file:-

VERSION 1.10.7

AUTHOR: Matthieu "Puckel_" Roisil

DESCRIPTION: Basic Airflow container

BUILD: docker build --rm -t puckel/docker-airflow .

SOURCE: https://github.com/puckel/docker-airflow

FROM python:3.7-slim-stretch LABEL maintainer="Puckel_"

Never prompt the user for choices on installation/configuration of packages

ENV DEBIAN_FRONTEND noninteractive ENV TERM linux

Airflow

ARG AIRFLOW_VERSION=1.10.7 ARG AIRFLOW_USER_HOME=/usr/local/airflow ARG AIRFLOW_DEPS="" ARG PYTHON_DEPS="" ENV AIRFLOW_HOME=${AIRFLOW_USER_HOME}

Define en_US.

ENV LANGUAGE en_US.UTF-8 ENV LANG en_US.UTF-8 ENV LC_ALL en_US.UTF-8 ENV LC_CTYPE en_US.UTF-8 ENV LC_MESSAGES en_US.UTF-8

Disable noisy "Handling signal" log messages:

ENV GUNICORN_CMD_ARGS --log-level WARNING

RUN set -ex
&& buildDeps='
freetds-dev
libkrb5-dev
libsasl2-dev
libssl-dev
libffi-dev
libpq-dev
git
'
&& apt-get update -yqq
&& apt-get upgrade -yqq
&& apt-get install -yqq --no-install-recommends
$buildDeps
freetds-bin
build-essential
default-libmysqlclient-dev
apt-utils
curl
rsync
netcat
locales
&& sed -i 's/^# en_US.UTF-8 UTF-8$/en_US.UTF-8 UTF-8/g' /etc/locale.gen
&& locale-gen
&& update-locale LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8
&& useradd -ms /bin/bash -d ${AIRFLOW_USER_HOME} airflow
&& pip install -U pip setuptools wheel
&& pip install pytz
&& pip install pyOpenSSL
&& pip install ndg-httpsclient
&& pip install pyasn1
&& pip install apache-airflow[crypto,celery,postgres,hive,jdbc,mysql,ssh${AIRFLOW_DEPS:+,}${AIRFLOW_DEPS}]==${AIRFLOW_VERSION}
&& pip install 'redis==3.2'
&& if [ -n "${PYTHON_DEPS}" ]; then pip install ${PYTHON_DEPS}; fi
&& apt-get purge --auto-remove -yqq $buildDeps
&& apt-get autoremove -yqq --purge
&& apt-get clean
&& rm -rf
/var/lib/apt/lists/*
/tmp/*
/var/tmp/*
/usr/share/man
/usr/share/doc
/usr/share/doc-base

COPY script/entrypoint.sh /entrypoint.sh COPY config/airflow.cfg ${AIRFLOW_USER_HOME}/airflow.cfg

RUN chown -R airflow: ${AIRFLOW_USER_HOME}

EXPOSE 8080 5555 8793

USER airflow WORKDIR ${AIRFLOW_USER_HOME} ENTRYPOINT ["/entrypoint.sh"] CMD ["webserver"]

is there anything i am missing or any configuration i need to do so that my docker will read my airflow.cfg file? Any leads will be appreciated and its very urgent for me.

Thanks in advance :)

[ishanverma96]

also i am using docker toolbox v19.03.1

• airflow v 1.10,7 and also i have checked with other version ranging from 1.10.1 but no luck
• already made s3 bucket on s3 with read & write permissions.

[apanagar]

I'm having this issue as well. I'm somewhat new to Docker, it's not clear to me how entrypoint.sh is called and whether Dockerfile is every called

[apanagar]

I was able to get this to work by pointing directly to the airflow.cfg in my docker-compose.yml file

       volumes:
            - /home/apanagar/etl/dags:/usr/local/airflow/dags
            - /home/apanagar/docker-airflow/config/airflow.cfg:/usr/local/airflow/airflow.cfg

[ishanverma96]

hey @apanagar , Thanks for looking into the issue. Though i didnt get a chance to use your solution in my poc as i had started working on other poc, once i get back on airflow will definitely try this. 👍

[zhangruiskyline]

Hi, I wonder to know if we use docker based Airflow, how can we find the airflow.cfg?

[jodmoreira]

I was able to get this to work by pointing directly to the airflow.cfg in my docker-compose.yml file

       volumes:
            - /home/apanagar/etl/dags:/usr/local/airflow/dags
            - /home/apanagar/docker-airflow/config/airflow.cfg:/usr/local/airflow/airflow.cfg

When i try to do that it starts rebooting endlessly.

[chevdor]

So the first thing you want to do is to fetch the current config. If you followed the doc, you should have a local folder called dags that is mapped. We will use that to temporarily extract the config (we could also use a new mapping for more safety...). You can fetch it using:

docker exec -it docker_airflow-webserver_1 bash

Once in the container:

cp /opt/airflow/airflow.cfg /opt/airflow/dags

On your system, you will now find airflow.cfg in the dags folder. Move it out of there in the same folder than your docker-compose.yml.

Now in your docker-compose.yml, you can add the LAST line below:

volumes:
    - ./dags:/opt/airflow/dags
    - ./logs:/opt/airflow/logs
    - ./plugins:/opt/airflow/plugins
    - ./airflow.cfg:/opt/airflow/airflow.cfg                <----- add this

Now edit your local airflow.cfg as you wish.

restart your env (there may be better options... happy to hear about that):

docker-compose restart

wait a bit for everything to start and your new config should now be active.

[bing007]

So the first thing you want to do is to fetch the current config. If you followed the doc, you should have a local folder called dags that is mapped. We will use that to temporarily extract the config (we could also use a new mapping for more safety...). You can fetch it using:

docker exec -it docker_airflow-webserver_1 bash

Once in the container:

cp /opt/airflow/airflow.cfg /opt/airflow/dags

On your system, you will now find airflow.cfg in the dags folder. Move it out of there in the same folder than your docker-compose.yml.

Now in your docker-compose.yml, you can add the LAST line below:

volumes:
    - ./dags:/opt/airflow/dags
    - ./logs:/opt/airflow/logs
    - ./plugins:/opt/airflow/plugins
    - ./airflow.cfg:/opt/airflow/airflow.cfg                <----- add this

Now edit your local airflow.cfg as you wish.

restart your env (there may be better options... happy to hear about that):

docker-compose restart

wait a bit for everything to start and your new config should now be active.

• ./airflow.cfg:/opt/airflow/airflow.cfg

Thank you @chevdor , this should've been in their quick start guide.

[masbayuw]

So the first thing you want to do is to fetch the current config. If you followed the doc, you should have a local folder called dags that is mapped. We will use that to temporarily extract the config (we could also use a new mapping for more safety...). You can fetch it using:

docker exec -it docker_airflow-webserver_1 bash

Once in the container:

cp /opt/airflow/airflow.cfg /opt/airflow/dags

On your system, you will now find airflow.cfg in the dags folder. Move it out of there in the same folder than your docker-compose.yml.

Now in your docker-compose.yml, you can add the LAST line below:

volumes:
    - ./dags:/opt/airflow/dags
    - ./logs:/opt/airflow/logs
    - ./plugins:/opt/airflow/plugins
    - ./airflow.cfg:/opt/airflow/airflow.cfg                <----- add this

Now edit your local airflow.cfg as you wish.

restart your env (there may be better options... happy to hear about that):

docker-compose restart

wait a bit for everything to start and your new config should now be active.

Hi, i've follow your guide and didn't work for me.

my docker-compose yaml like this volumes: - ./dags:/opt/airflow/dags - ./logs:/opt/airflow/logs - ./plugins:/opt/airflow/plugins - ./scripts:/opt/airflow/scripts - ./config:/opt/airflow/config - ./airflow.cfg:/opt/airflow/airflow.cfg

and i've been move the cfg into same folder with it.

I try to use my email operator, in the airflow cfg I've already put smtp config

[smtp]
smtp_host = smtp.googlemail.com
smtp_starttls = True
smtp_ssl = False
smtp_user = xxxxxxxxxxxxxxxxxx
smtp_password = xxxxxxxxxxxxxxxxxx
smtp_port = 587
smtp_mail_from = xxxxxxxxxxxxxxxxxx
smtp_timeout = 30
smtp_retry_limit = 5 

but the logs fail and show this error

My airflow image is 2.0.1