docker-airflow
docker-airflow copied to clipboard
Published
20 hours ago •
puckel
puckel
worker pod crashing
I deployed airflow webserver, scheduler, postgres, redis, worker, flower on different openshift pods. webserver, scheduler, postgres, redis, flower are successfully deployed & running.
Scheduler is performing LPUSH into redis queue with task instance. Can see airflow UI with dags, dag runs & task instance. Status of Dagruns remains running & task instance queued.
Worker deployment is crashing shows only following logs in the log terminal of worker pod:
[2019-05-28 14:04:36,401] {{settings.py:174}} INFO - settings.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800, pid=1
| [2019-05-28 14:04:37,401] {{init.py:51}} INFO - Using executor CeleryExecutor
Logs on webserver's log terminal are:
[2019-05-28 13:15:15,595] {{models.py:273}} INFO - Filling up the DagBag from /usr/local/airflow/dags
| [2019-05-28 13:15:16,765] {{settings.py:174}} INFO - settings.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800, pid=192 | [2019-05-28 13:15:17 +0000] [192] [INFO] Starting gunicorn 19.9.0 | [2019-05-28 13:15:17 +0000] [192] [INFO] Listening at: http://0.0.0.0:8081 (192) | [2019-05-28 13:15:17 +0000] [192] [INFO] Using worker: sync | [2019-05-28 13:15:17 +0000] [197] [INFO] Booting worker with pid: 197 | [2019-05-28 13:15:17 +0000] [198] [INFO] Booting worker with pid: 198 | [2019-05-28 13:15:17 +0000] [199] [INFO] Booting worker with pid: 199 | [2019-05-28 13:15:17 +0000] [200] [INFO] Booting worker with pid: 200 | Running the Gunicorn Server with: | Workers: 4 sync | Host: 0.0.0.0:8081 | Timeout: 120 | Logfiles: - - | ================================================================= | [2019-05-28 13:15:18,036] {{init.py:51}} INFO - Using executor CeleryExecutor | [2019-05-28 13:15:18,529] {{init.py:51}} INFO - Using executor CeleryExecutor | [2019-05-28 13:15:18,743] {{init.py:51}} INFO - Using executor CeleryExecutor | [2019-05-28 13:15:18,929] {{init.py:51}} INFO - Using executor CeleryExecutor | [2019-05-28 13:15:20,740] {{models.py:273}} INFO - Filling up the DagBag from /usr/local/airflow/dags | [2019-05-28 13:15:21,029] {{models.py:273}} INFO - Filling up the DagBag from /usr/local/airflow/dags | [2019-05-28 13:15:21,047] {{models.py:273}} INFO - Filling up the DagBag from /usr/local/airflow/dags | [2019-05-28 13:15:21,132] {{models.py:273}} INFO - Filling up the DagBag from /usr/local/airflow/dags......
Anybody can assist on fixing this up.
Thanks In Advance. Hitesh
I am facing the same issue
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: apache-airflow[google_auth] in /home/airflow/.local/lib/python3.6/site-packages (1.10.15)
Requirement already satisfied: python-daemon>=2.1.1 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (2.2.4)
Requirement already satisfied: json-merge-patch==0.2 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (0.2)
Requirement already satisfied: cached-property~=1.5 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (1.5.2)
Requirement already satisfied: flask-swagger<0.3,>=0.2.13 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (0.2.14)
Requirement already satisfied: pygments<3.0,>=2.0.1 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (2.7.2)
Requirement already satisfied: flask-login<0.5,>=0.3 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (0.4.1)
Requirement already satisfied: pendulum==1.4.4 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (1.4.4)
Requirement already satisfied: tabulate<0.9,>=0.7.5 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (0.8.7)
Requirement already satisfied: dill<0.4,>=0.2.2 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (0.3.3)
Requirement already satisfied: lazy-object-proxy<1.5.0 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (1.4.3)
Requirement already satisfied: pandas<2.0,>=0.17.1 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (1.1.4)
Requirement already satisfied: psutil<6.0.0,>=4.2.0 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (5.7.3)
Requirement already satisfied: croniter<0.4,>=0.3.17 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (0.3.36)
Requirement already satisfied: cryptography>=0.9.3; python_version >= "3.0" in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (3.2.1)
Requirement already satisfied: packaging in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (20.7)
Requirement already satisfied: iso8601>=0.1.12 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (0.1.13)
Requirement already satisfied: importlib-metadata~=2.0; python_version < "3.9" in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (2.1.0)
Requirement already satisfied: sqlalchemy~=1.3 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (1.3.20)
Requirement already satisfied: cattrs<1.1.0,>=1.0; python_version <= "3.6" in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (1.0.0)
Requirement already satisfied: colorlog==4.0.2 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (4.0.2)
Requirement already satisfied: email-validator in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (1.1.2)
Requirement already satisfied: funcsigs<2.0.0,>=1.0.0 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (1.0.2)
Requirement already satisfied: python-dateutil<3,>=2.3 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (2.8.1)
Requirement already satisfied: configparser<3.6.0,>=3.5.0 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (3.5.3)
Requirement already satisfied: jsonschema~=3.0 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (3.2.0)
Requirement already satisfied: werkzeug<1.0.0 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (0.16.1)
Requirement already satisfied: flask-appbuilder~=2.2; python_version >= "3.6" in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (2.3.4)
Requirement already satisfied: argcomplete~=1.10 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (1.12.2)
Requirement already satisfied: markdown<3.0,>=2.5.2 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (2.6.11)
Requirement already satisfied: thrift>=0.11.0 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (0.13.0)
Requirement already satisfied: future<0.19,>=0.16.0 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (0.18.2)
Requirement already satisfied: setproctitle<2,>=1.1.8 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (1.2)
Requirement already satisfied: attrs<21.0,>=20.0 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (20.3.0)
Requirement already satisfied: marshmallow-sqlalchemy<0.24.0,>=0.16.1; python_version >= "3.6" in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (0.23.1)
Requirement already satisfied: pep562~=1.0; python_version < "3.7" in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (1.0)
Requirement already satisfied: flask-admin==1.5.4 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (1.5.4)
Requirement already satisfied: jinja2<2.12.0,>=2.10.1 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (2.11.2)
Requirement already satisfied: python-slugify<5.0,>=3.0.0 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (4.0.1)
Requirement already satisfied: flask-caching<1.4.0,>=1.3.3 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (1.3.3)
Requirement already satisfied: alembic<2.0,>=1.0 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (1.4.3)
Requirement already satisfied: importlib-resources~=1.4 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (1.5.0)
Requirement already satisfied: flask-wtf<0.15,>=0.14.2 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (0.14.3)
Requirement already satisfied: zope.deprecation<5.0,>=4.0 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (4.4.0)
Requirement already satisfied: typing-extensions>=3.7.4; python_version < "3.8" in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (3.7.4.3)
Requirement already satisfied: tzlocal<2.0.0,>=1.4 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (1.5.1)
Requirement already satisfied: requests<2.24.0,>=2.20.0; python_version >= "3.0" in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (2.23.0)
Requirement already satisfied: gunicorn<21.0,>=19.5.0 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (20.0.4)
Requirement already satisfied: flask<2.0,>=1.1.0 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (1.1.2)
Requirement already satisfied: tenacity==4.12.0 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (4.12.0)
Requirement already satisfied: graphviz>=0.12 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (0.15)
Requirement already satisfied: python-nvd3~=0.15.0 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (0.15.0)
Requirement already satisfied: unicodecsv>=0.14.1 in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (0.14.1)
Requirement already satisfied: sqlalchemy-jsonfield~=0.9; python_version >= "3.5" in /home/airflow/.local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (0.9.0)
Requirement already satisfied: Flask-OAuthlib<0.9.6,>=0.9.1; extra == "google_auth" in /usr/local/lib/python3.6/site-packages (from apache-airflow[google_auth]) (0.9.5)
Collecting requests-oauthlib==1.1.0; extra == "google_auth"
Downloading requests_oauthlib-1.1.0-py2.py3-none-any.whl (21 kB)
Collecting oauthlib!=2.0.3,!=2.0.4,!=2.0.5,<3.0.0,>=1.1.2; extra == "google_auth"
Downloading oauthlib-2.1.0-py2.py3-none-any.whl (121 kB)
Requirement already satisfied: docutils in /home/airflow/.local/lib/python3.6/site-packages (from python-daemon>=2.1.1->apache-airflow[google_auth]) (0.15.2)
Requirement already satisfied: setuptools in /usr/local/lib/python3.6/site-packages (from python-daemon>=2.1.1->apache-airflow[google_auth]) (54.1.1)
Requirement already satisfied: lockfile>=0.10 in /home/airflow/.local/lib/python3.6/site-packages (from python-daemon>=2.1.1->apache-airflow[google_auth]) (0.12.2)
Requirement already satisfied: PyYAML>=5.1 in /home/airflow/.local/lib/python3.6/site-packages (from flask-swagger<0.3,>=0.2.13->apache-airflow[google_auth]) (5.3.1)
Requirement already satisfied: pytzdata>=2018.3.0.0 in /home/airflow/.local/lib/python3.6/site-packages (from pendulum==1.4.4->apache-airflow[google_auth]) (2020.1)
Requirement already satisfied: numpy>=1.15.4 in /home/airflow/.local/lib/python3.6/site-packages (from pandas<2.0,>=0.17.1->apache-airflow[google_auth]) (1.19.4)
Requirement already satisfied: pytz>=2017.2 in /home/airflow/.local/lib/python3.6/site-packages (from pandas<2.0,>=0.17.1->apache-airflow[google_auth]) (2020.4)
Requirement already satisfied: natsort in /home/airflow/.local/lib/python3.6/site-packages (from croniter<0.4,>=0.3.17->apache-airflow[google_auth]) (7.1.0)
Requirement already satisfied: six>=1.4.1 in /home/airflow/.local/lib/python3.6/site-packages (from cryptography>=0.9.3; python_version >= "3.0"->apache-airflow[google_auth]) (1.15.0)
Requirement already satisfied: cffi!=1.11.3,>=1.8 in /home/airflow/.local/lib/python3.6/site-packages (from cryptography>=0.9.3; python_version >= "3.0"->apache-airflow[google_auth]) (1.14.4)
Requirement already satisfied: pyparsing>=2.0.2 in /home/airflow/.local/lib/python3.6/site-packages (from packaging->apache-airflow[google_auth]) (2.4.7)
Requirement already satisfied: zipp>=0.5 in /home/airflow/.local/lib/python3.6/site-packages (from importlib-metadata~=2.0; python_version < "3.9"->apache-airflow[google_auth]) (3.4.0)
Requirement already satisfied: dnspython>=1.15.0 in /home/airflow/.local/lib/python3.6/site-packages (from email-validator->apache-airflow[google_auth]) (1.16.0)
Requirement already satisfied: idna>=2.0.0 in /home/airflow/.local/lib/python3.6/site-packages (from email-validator->apache-airflow[google_auth]) (2.8)
Requirement already satisfied: pyrsistent>=0.14.0 in /home/airflow/.local/lib/python3.6/site-packages (from jsonschema~=3.0->apache-airflow[google_auth]) (0.17.3)
Requirement already satisfied: click<8,>=6.7 in /home/airflow/.local/lib/python3.6/site-packages (from flask-appbuilder~=2.2; python_version >= "3.6"->apache-airflow[google_auth]) (6.7)
Requirement already satisfied: PyJWT>=1.7.1 in /home/airflow/.local/lib/python3.6/site-packages (from flask-appbuilder~=2.2; python_version >= "3.6"->apache-airflow[google_auth]) (1.7.1)
Requirement already satisfied: marshmallow<3.0.0,>=2.18.0 in /home/airflow/.local/lib/python3.6/site-packages (from flask-appbuilder~=2.2; python_version >= "3.6"->apache-airflow[google_auth]) (2.21.0)
Requirement already satisfied: Flask-Babel<2,>=1 in /home/airflow/.local/lib/python3.6/site-packages (from flask-appbuilder~=2.2; python_version >= "3.6"->apache-airflow[google_auth]) (1.0.0)
Requirement already satisfied: Flask-SQLAlchemy<3,>=2.4 in /home/airflow/.local/lib/python3.6/site-packages (from flask-appbuilder~=2.2; python_version >= "3.6"->apache-airflow[google_auth]) (2.4.4)
Requirement already satisfied: apispec[yaml]<2,>=1.1.1 in /home/airflow/.local/lib/python3.6/site-packages (from flask-appbuilder~=2.2; python_version >= "3.6"->apache-airflow[google_auth]) (1.3.3)
Requirement already satisfied: Flask-OpenID<2,>=1.2.5 in /home/airflow/.local/lib/python3.6/site-packages (from flask-appbuilder~=2.2; python_version >= "3.6"->apache-airflow[google_auth]) (1.2.5)
Requirement already satisfied: Flask-JWT-Extended<4,>=3.18 in /home/airflow/.local/lib/python3.6/site-packages (from flask-appbuilder~=2.2; python_version >= "3.6"->apache-airflow[google_auth]) (3.25.0)
Requirement already satisfied: marshmallow-enum<2,>=1.4.1 in /home/airflow/.local/lib/python3.6/site-packages (from flask-appbuilder~=2.2; python_version >= "3.6"->apache-airflow[google_auth]) (1.5.1)
Requirement already satisfied: colorama<1,>=0.3.9 in /home/airflow/.local/lib/python3.6/site-packages (from flask-appbuilder~=2.2; python_version >= "3.6"->apache-airflow[google_auth]) (0.4.4)
Requirement already satisfied: sqlalchemy-utils<1,>=0.32.21 in /home/airflow/.local/lib/python3.6/site-packages (from flask-appbuilder~=2.2; python_version >= "3.6"->apache-airflow[google_auth]) (0.36.8)
Requirement already satisfied: prison<1.0.0,>=0.1.3 in /home/airflow/.local/lib/python3.6/site-packages (from flask-appbuilder~=2.2; python_version >= "3.6"->apache-airflow[google_auth]) (0.1.3)
Requirement already satisfied: wtforms in /home/airflow/.local/lib/python3.6/site-packages (from flask-admin==1.5.4->apache-airflow[google_auth]) (2.3.3)
Requirement already satisfied: MarkupSafe>=0.23 in /home/airflow/.local/lib/python3.6/site-packages (from jinja2<2.12.0,>=2.10.1->apache-airflow[google_auth]) (1.1.1)
Requirement already satisfied: text-unidecode>=1.3 in /home/airflow/.local/lib/python3.6/site-packages (from python-slugify<5.0,>=3.0.0->apache-airflow[google_auth]) (1.3)
Requirement already satisfied: Mako in /home/airflow/.local/lib/python3.6/site-packages (from alembic<2.0,>=1.0->apache-airflow[google_auth]) (1.1.3)
Requirement already satisfied: python-editor>=0.3 in /home/airflow/.local/lib/python3.6/site-packages (from alembic<2.0,>=1.0->apache-airflow[google_auth]) (1.0.4)
Requirement already satisfied: itsdangerous in /home/airflow/.local/lib/python3.6/site-packages (from flask-wtf<0.15,>=0.14.2->apache-airflow[google_auth]) (1.1.0)
Requirement already satisfied: certifi>=2017.4.17 in /home/airflow/.local/lib/python3.6/site-packages (from requests<2.24.0,>=2.20.0; python_version >= "3.0"->apache-airflow[google_auth]) (2020.11.8)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/airflow/.local/lib/python3.6/site-packages (from requests<2.24.0,>=2.20.0; python_version >= "3.0"->apache-airflow[google_auth]) (1.25.11)
Requirement already satisfied: chardet<4,>=3.0.2 in /home/airflow/.local/lib/python3.6/site-packages (from requests<2.24.0,>=2.20.0; python_version >= "3.0"->apache-airflow[google_auth]) (3.0.4)
Requirement already satisfied: typing>=3.6; python_version < "3.7" in /home/airflow/.local/lib/python3.6/site-packages (from sqlalchemy-jsonfield~=0.9; python_version >= "3.5"->apache-airflow[google_auth]) (3.7.4.3)
Requirement already satisfied: pycparser in /home/airflow/.local/lib/python3.6/site-packages (from cffi!=1.11.3,>=1.8->cryptography>=0.9.3; python_version >= "3.0"->apache-airflow[google_auth]) (2.20)
Requirement already satisfied: Babel>=2.3 in /home/airflow/.local/lib/python3.6/site-packages (from Flask-Babel<2,>=1->flask-appbuilder~=2.2; python_version >= "3.6"->apache-airflow[google_auth]) (2.9.0)
Requirement already satisfied: python3-openid>=2.0 in /home/airflow/.local/lib/python3.6/site-packages (from Flask-OpenID<2,>=1.2.5->flask-appbuilder~=2.2; python_version >= "3.6"->apache-airflow[google_auth]) (3.2.0)
Requirement already satisfied: defusedxml in /home/airflow/.local/lib/python3.6/site-packages (from python3-openid>=2.0->Flask-OpenID<2,>=1.2.5->flask-appbuilder~=2.2; python_version >= "3.6"->apache-airflow[google_auth]) (0.6.0)
Installing collected packages: oauthlib, requests-oauthlib
Attempting uninstall: oauthlib
Found existing installation: oauthlib 3.1.0
Uninstalling oauthlib-3.1.0:
Successfully uninstalled oauthlib-3.1.0
Attempting uninstall: requests-oauthlib
Found existing installation: requests-oauthlib 1.3.0
Uninstalling requests-oauthlib-1.3.0:
Successfully uninstalled requests-oauthlib-1.3.0
Successfully installed oauthlib-2.1.0 requests-oauthlib-1.1.0
WARNING: You are using pip version 20.2.4; however, version 21.2.4 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2021-08-15 15:22:37,380] {logging_config.py:47} INFO - Successfully imported user-defined logging config from logging_config.LOGGING_CONFIG
[2021-08-15 15:22:39,337] {__init__.py:50} INFO - Using executor KubernetesExecutor
[2021-08-15 15:22:39,338] {dagbag.py:417} INFO - Filling up the DagBag from /dev/null
[2021-08-15 15:22:39,969] {security.py:478} INFO - Start syncing user roles.
[2021-08-15 15:22:40,349] {security.py:388} INFO - Fetching a set of all permission, view_menu from FAB meta-table
[2021-08-15 15:22:40,507] {security.py:331} INFO - Cleaning faulty perms
[2021-08-15 15:22:42 +0000] [36] [INFO] Starting gunicorn 20.0.4
[2021-08-15 15:22:42 +0000] [36] [INFO] Listening at: http://0.0.0.0:8080 (36)
[2021-08-15 15:22:42 +0000] [36] [INFO] Using worker: sync
[2021-08-15 15:22:42 +0000] [40] [INFO] Booting worker with pid: 40
[2021-08-15 15:22:42 +0000] [41] [INFO] Booting worker with pid: 41
[2021-08-15 15:22:42 +0000] [42] [INFO] Booting worker with pid: 42
[2021-08-15 15:22:42,201] {logging_config.py:47} INFO - Successfully imported user-defined logging config from logging_config.LOGGING_CONFIG
[2021-08-15 15:22:42 +0000] [43] [INFO] Booting worker with pid: 43
[2021-08-15 15:22:42,252] {logging_config.py:47} INFO - Successfully imported user-defined logging config from logging_config.LOGGING_CONFIG
[2021-08-15 15:22:42,344] {logging_config.py:47} INFO - Successfully imported user-defined logging config from logging_config.LOGGING_CONFIG
[2021-08-15 15:22:42,468] {logging_config.py:47} INFO - Successfully imported user-defined logging config from logging_config.LOGGING_CONFIG
[2021-08-15 15:22:43,483] {__init__.py:50} INFO - Using executor KubernetesExecutor
[2021-08-15 15:22:43,485] {dagbag.py:417} INFO - Filling up the DagBag from /opt/airflow/dags/dags/dags
[2021-08-15 15:22:43,558] {__init__.py:50} INFO - Using executor KubernetesExecutor
[2021-08-15 15:22:43,559] {dagbag.py:417} INFO - Filling up the DagBag from /opt/airflow/dags/dags/dags
[2021-08-15 15:22:43,585] {__init__.py:50} INFO - Using executor KubernetesExecutor
[2021-08-15 15:22:43,587] {dagbag.py:417} INFO - Filling up the DagBag from /opt/airflow/dags/dags/dags
[2021-08-15 15:22:43,676] {__init__.py:50} INFO - Using executor KubernetesExecutor
[2021-08-15 15:22:43,678] {dagbag.py:417} INFO - Filling up the DagBag from /opt/airflow/dags/dags/dags
[2021-08-15 15:24:42 +0000] [36] [CRITICAL] WORKER TIMEOUT (pid:40)
[2021-08-15 15:24:42 +0000] [36] [CRITICAL] WORKER TIMEOUT (pid:41)
[2021-08-15 15:24:42 +0000] [36] [CRITICAL] WORKER TIMEOUT (pid:42)
[2021-08-15 15:24:42 +0000] [36] [CRITICAL] WORKER TIMEOUT (pid:43)
[2021-08-15 15:24:43 +0000] [112] [INFO] Booting worker with pid: 112
[2021-08-15 15:24:43 +0000] [113] [INFO] Booting worker with pid: 113
[2021-08-15 15:24:43 +0000] [114] [INFO] Booting worker with pid: 114
[2021-08-15 15:24:43 +0000] [115] [INFO] Booting worker with pid: 115
[2021-08-15 15:24:43,597] {logging_config.py:47} INFO - Successfully imported user-defined logging config from logging_config.LOGGING_CONFIG
[2021-08-15 15:24:43,616] {logging_config.py:47} INFO - Successfully imported user-defined logging config from logging_config.LOGGING_CONFIG
[2021-08-15 15:24:43,646] {logging_config.py:47} INFO - Successfully imported user-defined logging config from logging_config.LOGGING_CONFIG
[2021-08-15 15:24:43,691] {logging_config.py:47} INFO - Successfully imported user-defined logging config from logging_config.LOGGING_CONFIG
[2021-08-15 15:24:44,848] {__init__.py:50} INFO - Using executor KubernetesExecutor
[2021-08-15 15:24:44,850] {dagbag.py:417} INFO - Filling up the DagBag from /opt/airflow/dags/dags/dags
[2021-08-15 15:24:44,933] {__init__.py:50} INFO - Using executor KubernetesExecutor
[2021-08-15 15:24:44,935] {dagbag.py:417} INFO - Filling up the DagBag from /opt/airflow/dags/dags/dags
[2021-08-15 15:24:44,935] {__init__.py:50} INFO - Using executor KubernetesExecutor
[2021-08-15 15:24:44,937] {dagbag.py:417} INFO - Filling up the DagBag from /opt/airflow/dags/dags/dags
[2021-08-15 15:24:44,973] {__init__.py:50} INFO - Using executor KubernetesExecutor
[2021-08-15 15:24:44,975] {dagbag.py:417} INFO - Filling up the DagBag from /opt/airflow/dags/dags/dags
[2021-08-15 15:26:43 +0000] [36] [CRITICAL] WORKER TIMEOUT (pid:112)
[2021-08-15 15:26:43 +0000] [36] [CRITICAL] WORKER TIMEOUT (pid:113)
[2021-08-15 15:26:43 +0000] [36] [CRITICAL] WORKER TIMEOUT (pid:114)
[2021-08-15 15:26:43 +0000] [36] [CRITICAL] WORKER TIMEOUT (pid:115)
[2021-08-15 15:26:44 +0000] [184] [INFO] Booting worker with pid: 184
[2021-08-15 15:26:44 +0000] [185] [INFO] Booting worker with pid: 185
[2021-08-15 15:26:44 +0000] [186] [INFO] Booting worker with pid: 186
[2021-08-15 15:26:44 +0000] [187] [INFO] Booting worker with pid: 187
[2021-08-15 15:26:44,914] {logging_config.py:47} INFO - Successfully imported user-defined logging config from logging_config.LOGGING_CONFIG
[2021-08-15 15:26:44,924] {logging_config.py:47} INFO - Successfully imported user-defined logging config from logging_config.LOGGING_CONFIG
[2021-08-15 15:26:44,973] {logging_config.py:47} INFO - Successfully imported user-defined logging config from logging_config.LOGGING_CONFIG
[2021-08-15 15:26:45,059] {logging_config.py:47} INFO - Successfully imported user-defined logging config from logging_config.LOGGING_CONFIG
[2021-08-15 15:26:46,167] {__init__.py:50} INFO - Using executor KubernetesExecutor
[2021-08-15 15:26:46,169] {dagbag.py:417} INFO - Filling up the DagBag from /opt/airflow/dags/dags/dags
[2021-08-15 15:26:46,192] {__init__.py:50} INFO - Using executor KubernetesExecutor
[2021-08-15 15:26:46,194] {dagbag.py:417} INFO - Filling up the DagBag from /opt/airflow/dags/dags/dags
[2021-08-15 15:26:46,194] {__init__.py:50} INFO - Using executor KubernetesExecutor
[2021-08-15 15:26:46,196] {dagbag.py:417} INFO - Filling up the DagBag from /opt/airflow/dags/dags/dags
[2021-08-15 15:26:46,250] {__init__.py:50} INFO - Using executor KubernetesExecutor
[2021-08-15 15:26:46,252] {dagbag.py:417} INFO - Filling up the DagBag from /opt/airflow/dags/dags/dags
[2021-08-15 15:28:44 +0000] [36] [CRITICAL] WORKER TIMEOUT (pid:184)
[2021-08-15 15:28:44 +0000] [36] [CRITICAL] WORKER TIMEOUT (pid:185)
[2021-08-15 15:28:44 +0000] [36] [CRITICAL] WORKER TIMEOUT (pid:186)
[2021-08-15 15:28:44 +0000] [36] [CRITICAL] WORKER TIMEOUT (pid:187)
[2021-08-15 15:28:45 +0000] [256] [INFO] Booting worker with pid: 256
[2021-08-15 15:28:46 +0000] [257] [INFO] Booting worker with pid: 257
[2021-08-15 15:28:46 +0000] [258] [INFO] Booting worker with pid: 258
[2021-08-15 15:28:46 +0000] [259] [INFO] Booting worker with pid: 259
[2021-08-15 15:28:46,200] {logging_config.py:47} INFO - Successfully imported user-defined logging config from logging_config.LOGGING_CONFIG
[2021-08-15 15:28:46,243] {logging_config.py:47} INFO - Successfully imported user-defined logging config from logging_config.LOGGING_CONFIG
[2021-08-15 15:28:46,271] {logging_config.py:47} INFO - Successfully imported user-defined logging config from logging_config.LOGGING_CONFIG
[2021-08-15 15:28:46,277] {logging_config.py:47} INFO - Successfully imported user-defined logging config from logging_config.LOGGING_CONFIG
[2021-08-15 15:28:47,398] {__init__.py:50} INFO - Using executor KubernetesExecutor
[2021-08-15 15:28:47,400] {dagbag.py:417} INFO - Filling up the DagBag from /opt/airflow/dags/dags/dags
[2021-08-15 15:28:47,559] {__init__.py:50} INFO - Using executor KubernetesExecutor
[2021-08-15 15:28:47,561] {dagbag.py:417} INFO - Filling up the DagBag from /opt/airflow/dags/dags/dags
[2021-08-15 15:28:47,636] {__init__.py:50} INFO - Using executor KubernetesExecutor
[2021-08-15 15:28:47,638] {dagbag.py:417} INFO - Filling up the DagBag from /opt/airflow/dags/dags/dags
[2021-08-15 15:28:47,648] {__init__.py:50} INFO - Using executor KubernetesExecutor
[2021-08-15 15:28:47,650] {dagbag.py:417} INFO - Filling up the DagBag from /opt/airflow/dags/dags/dags
The airflow webserver workers keeps timing out
