Paramjeet B
Paramjeet B
After debugging for a while , I have also found that one of our scheduler was failed with liveliness probe and restarted at the same time. So , I'm guessing...
Found this log in airflow scheduler: ``` [2024-05-03T12:09:06.476+0000] {process_utils.py:100} INFO - Sending the signal Signals.SIGTERM to process 75 as process group is missing. [2024-05-03T12:09:05.937+0000] {process_utils.py:131} INFO - Sending Signals.SIGTERM to...
I believe I have identified the cause of the issue: We are using AWS Spot EC2 instances for the workloads in Airflow. When a spot instance is terminated, the pod...
I hope this is solved in latest version with this PR : https://github.com/apache/airflow/pull/37671/files @dirrao , can you confirm the above PR is to solve this issue
I can confirm that the issue is solved with the below code , we have added this as custom extract_xcom : This is also mentioned here : https://github.com/kubernetes-client/python-base/issues/190#issuecomment-805073981 , We...
@crabio , Were you able to find a solution ? We are also facing the task leak issue in v2.6.3
@crabio I have updated my comments here https://github.com/apache/airflow/issues/38968#issuecomment-2059521327 , I was able to improve the performance and the task no longer have longer queue duration
@crabio , Yes we run in single namespace.
@eladkal yes , I'll create a PR with the above suggest code. I'm afraid that I can't find the root cause of the issue in current code.
@cleivson , you can customise your airflow to call the above mentioned method till we have a fix