astronomer-cosmos
astronomer-cosmos copied to clipboard
Cosmos tasks randomly getting marked as zombie tasks
Not sure if this should be here, but it happens mostly with cosmos dags.
My airflow instance is in a kubernetes cluster, so this might be a problem with the scheduler, not with cosmos itself. I'm using CeleryExecutor right now, might switch these dags to KubernetesExecutor to see if it works.
But anyways, if anyone knows how to help it would be great
[2024-05-27, 22:00:37 UTC] {task_context_logger.py:104} ERROR - Detected zombie job: {'full_filepath': '/opt/airflow/dags/repo/dags/dags/dbt/rtz.py', 'processor_subdir': None, 'msg': "{'DAG Id': 'transform___dbt-instagram__dev', 'Task Id': 'instagram_models.all_media_insights_run', 'Run Id': 'scheduled__2024-05-27T21:20:00+00:00', 'Hostname': 'airflow-worker-1.airflow-worker.orchestration.svc.cluster.local', 'External Executor Id': '72425eed-2b53-4d22-81f0-fee59bbb3c99'}", 'simple_task_instance': <airflow.models.taskinstance.SimpleTaskInstance object at 0x7f856ebd3010>, 'is_failure_callback': True} (See https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/tasks.html#zombie-undead-tasks)
Sometimes it is marked as zombie task even if it did execute correctly