Airflow 2.1.2 daemon flag & configuration file bug

[MeltyBot]

Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/3526

Originally created by @connor62 on 2022-05-25 15:38:59

---

What is the current bug behavior?

1. When trying to start an Airflow Scheduler Instance with meltano invoke airflow scheduler -D and/or meltano invoke airflow scheduler --daemon. However, despite producing a normal CLI output... you'll see a 'scheduler does not have a heartbeat' warning message. Currently the only was to remove this warning message and produce a 'heartbeat' is to run meltano invoke airflow scheduler and leave the terminal window open.

2. Currently we are trying to set the Airflow configurations to match the following:

   a. executor = LocalExecutor

   b. sql_alchemy_conn = postgresql

However these should be set in the airflow.cfg, under the path .meltano/run/airflow There are not the correct files... Currently it only host the files; airflow-scheduler.err, airflow-scheduler.log, airflow-scheduler.out, and webserver_config.py

What is the expected correct behavior?

1. airflow.cfg should be located under the path and .meltano/run/airflow. However this file is not always present and seems to me removed and sporadically reproduced overwriting any previous airflow configurations applied using environmental variables.

Steps to reproduce

1. Open a meltano project directory
2. Run meltano invoke airflow scheduler -D or meltano invoke airflow scheduler --daemon
3. Open the Airflow UI

• Meltano version (output with meltano --version)

• 1.102.0

• Airflow Configurations (output with airflow info)

• apache-airflow-providers-ftp | 2.0.0

• apache-airflow-providers-http | 2.0.0

• apache-airflow-providers-imap | 2.0.0

• apache-airflow-providers-slack | 4.0.0

• apache-airflow-providers-sqlite | 2.0.0

Relevant logs and/or screenshots

1. Run meltano invoke airflow scheduler -D or meltano invoke airflow scheduler --daemon
2. Open the Airflow UI

1. Configure Airflow using the meltano.yml file

   a. meltano config airflow set core.executor LocalExecutor

   b. meltano config airflow set core.sql_alchemy_conn postgresql://airflow:[email protected]:5432/airflow

Possible fixes

Determine how airflow.cfg is created by the Meltano project.

[MeltyBot]

View 7 previous comments from the original issue on GitLab