airflow-provider-fivetran
airflow-provider-fivetran copied to clipboard
fivetran
Patient operator
Adds a FivetranPatientOperator that waits for the Fievtran sync to finish before marking the task as completed. This is necessary for certain instances in which parallelism is limited.
thank you @elliotdes, looking in this https://github.com/fivetran/airflow-provider-fivetran/pull/56 now
@denimalpaca we have to use this operator in our DAG's due to limited parallelism in our Airflow instance.
The FivetranSensor works by checking the fivetran connectors initial status against future pokes. If the FivetranSensor begins after the connector has already finished its sync, then the sensor will hang. This is what was happening for us.
This FivetranPatientOperator would not be needed if the Fivetran API had unique id's for each triggered run that could then be checked explicitly. This is how dbt Cloud does it, and that works quite well for us.
+1 on this - we are also experiencing this issue where the FivetranSensor starts after the connector has already finished its sync and so the FivetranSensor keeps reporting sync_state = scheduled.
@PubChimps do you have any input here on using the Sensor to avoid starting after the connector has finished? My only thought is to pattern the DAG differently: have the sensor and connector run in parallel, then set the next task to trigger_rule=all_success.
@denimalpaca is there a way to guarantee two tasks running in parallel? If we take an extreme example where we have parallelism set to 1 on Airflow, then i'm not sure if the both tasks would trigger at the same time
@elliotdes if you have parallelism set to 1, or using the Sequential Executor, then no you can't have the tasks running in parallel. However, if you have the infra for parallelism, then you should be able to use priority weighting to make things run simultaneously.
@denimalpaca would this operator be worth committing for people limited to 1 parallelism / Sequential Operator then?
@elliotdes ideally people aren't running production pipelines with a sequential executor, but for those who are, yes this will probably be useful!
Hi all, sorry for the delays here, but could you please check v 1.1.1 of the provider? @kieronellis this will solve the issues that you mentioned you have experienced