TaskInstances- "get instance batch" is getting timeout after 1 minute, status code- 504 (~1000 records)

[litalkat]

is there a way to increase this timeout? Does pulling data for other airflow instances affect it? I tried to use _request_timeout=180 but it didn't help

[ferruzzi]

Hey @litalkat, we're likely going to need a little more info for this. Is there any stack trace with the error or any indication of a line number we can use to track it down?

[litalkat]

@ferruzzi sure. the request is for specific airflow deployment. in general- the airflow deployment is running ~100K airflow tasks per day. the process already run at least 5000 times In the last 3 months. I already countered this "504 Service Exception" response when I made an API call for long-range date but I saw that it can handle a 7000-9000 records in the response. in the last two days I get this 504 exception almost for any call (even for a range of 3 minutes -> ~100 tasks response)

its important to mention that I am monitoring a lot of other airflow instances in the same time but I didn't saw any thing about rate limit

the exception is returning EXACTLY 1 min after the API call is sent the traceback looks like: (504) Reason: Gateway Timeout HTTP response headers: HTTPHeaderDict({'x-powered-by': 'Express', ..... HTTP response body: Error accured while trying to proxy: some private env name/api/v1/dags/~/dagRuns/~/taskInstances/list

the request is using the airflow_client.get_task_instances_batch(ListTaskInstancesForm(...),_check_return_type=False)

[Yaadto]

@ferruzzi any updates? or ideas how to solve it?

[pierrejeambrun]

This is due to the webserver timeout. More info here: https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#web-server-master-timeout.

Maybe we need to improve performance of this particular endpoint, but if you have a really huge task instance table, this is king of expected I believe.