OpenMetadata icon indicating copy to clipboard operation
OpenMetadata copied to clipboard

Published 20 hours ago • verified publisher icon open-metadata

[Airflow Lineage Backend] Add timeout on request client

Open nuang-ee opened this issue 1 year ago • 0 comments

Feature

  • Currently, Airflow Lineage Backend does not have a timeout configuration available. This results to task succeeding but stalling indefinitely due to connection problem between Airflow worker and OM.

Describe the task

  • Please add a timeout & retry configuration.
    • When requesting from Airflow worker to OM, after configured timeout duration, request should be closed, returning proper Exception.
    • When request fails, lineage ingestion should be retried for configured retrial counts.

nuang-ee avatar Apr 25 '24 23:04 nuang-ee