yuqian90 comments

Results 4 comments of


                                            yuqian90

Copy of [AIRFLOW-5071] JIRA: Thousands of Executor reports task instance X finished (success) although the task says its queued. Was the task killed externally?

Hi @turbaszek, any finding on this? We have a CeleryExecutor + Redis setup with three workers (apache-airflow 1.10.12). The airflow-scheduler log has a lot of lines like this. I remember...

Copy of [AIRFLOW-5071] JIRA: Thousands of Executor reports task instance X finished (success) although the task says its queued. Was the task killed externally?

> Ok @yuqian90 @sgrzemski-ias what is you setting for core.dagbag_import_timeout ? > > As I'm hitting: > > Traceback (most recent call last): File "/usr/local/lib/airflow/airflow/models/dagbag.py", line 237, in process_file m...

Copy of [AIRFLOW-5071] JIRA: Thousands of Executor reports task instance X finished (success) although the task says its queued. Was the task killed externally?

> After digging further, I think the slowness that causes the error for our case is in this function: `SchedulerJob._process_dags()`. If this function takes around 60s, those `reschedule` sensors will...

Copy of [AIRFLOW-5071] JIRA: Thousands of Executor reports task instance X finished (success) although the task says its queued. Was the task killed externally?

> > The bug can also be fixed if the rescheduled task instance use a different try number, but this will cause a lot of log files. > > I...