airflow worker keeps exiting when sets executor to "Celery"

[APAC-DevOps]

Apache Airflow 1.9.0 Celery 4.1.1 Python 3.6

I tried to run my airflow cluster with Celery executor in Docker environment. However, my airflow worker keeps "booting" and "exiting".

I did some search on internet, and found that Airflow's support to "Celery" is not very good, Airflow Celery Cluster only works with a few specific Celery.

Can you tell you me if my Celery version (4.1.1) compatible with my Apache Airflow (1.9.0)

[2018-08-10 07:29:53 +0000] [901] [INFO] Booting worker with pid: 901 [2018-08-10 07:29:56 +0000] [24] [INFO] Handling signal: ttou [2018-08-10 07:29:56 +0000] [839] [INFO] Worker exiting (pid: 839) [2018-08-10 07:30:27 +0000] [24] [INFO] Handling signal: ttin [2018-08-10 07:30:27 +0000] [912] [INFO] Booting worker with pid: 912 [2018-08-10 07:30:29 +0000] [24] [INFO] Handling signal: ttou [2018-08-10 07:30:29 +0000] [850] [INFO] Worker exiting (pid: 850) [2018-08-10 07:31:00 +0000] [24] [INFO] Handling signal: ttin [2018-08-10 07:31:00 +0000] [922] [INFO] Booting worker with pid: 922 [2018-08-10 07:31:02 +0000] [24] [INFO] Handling signal: ttou [2018-08-10 07:31:02 +0000] [868] [INFO] Worker exiting (pid: 868) [2018-08-10 07:31:33 +0000] [24] [INFO] Handling signal: ttin [2018-08-10 07:31:33 +0000] [934] [INFO] Booting worker with pid: 934 [2018-08-10 07:31:36 +0000] [24] [INFO] Handling signal: ttou [2018-08-10 07:31:36 +0000] [890] [INFO] Worker exiting (pid: 890) [2018-08-10 07:32:07 +0000] [24] [INFO] Handling signal: ttin [2018-08-10 07:32:07 +0000] [946] [INFO] Booting worker with pid: 946 [2018-08-10 07:32:09 +0000] [24] [INFO] Handling signal: ttou [2018-08-10 07:32:09 +0000] [901] [INFO] Worker exiting (pid: 901)

[neonerd0]

Not sure if its related, but I have a similar issue with my airflow workers resetting constantly.

[japrogramer]

Same issue, but with regular airflow using sequential scheduler with sqlitedb

[2019-09-11 16:15:10 -0500] [10894] [INFO] Handling signal: winch [2019-09-11 16:15:17 -0500] [10894] [INFO] Handling signal: ttin [2019-09-11 16:15:17 -0500] [10959] [INFO] Booting worker with pid: 10959 [2019-09-11 16:15:17,770] {init.py:51} INFO - Using executor SequentialExecutor [2019-09-11 16:15:18,045] {dagbag.py:90} INFO - Filling up the DagBag from /home/archangel/airflow/dags [2019-09-11 16:15:18 -0500] [10894] [INFO] Handling signal: ttou [2019-09-11 16:15:18 -0500] [10898] [INFO] Worker exiting (pid: 10898)

[chrisbeach]

My worker(s) keep exiting too. Disappointed at how little diagnostics/reporting are available from the Airflow UI.

[bhavaniravi]

2023 and one of my clients faced this issue right now