airflow icon indicating copy to clipboard operation
airflow copied to clipboard

Make conn id parameters templated in GenericTransfer and also allow passing hook parameters like in BaseSQLOperator

Open dabla opened this issue 1 year ago • 4 comments

Today I've discovered that the conn id parameters (source_conn_id and destination_conn_id) aren't templated in the GenericTransfer. This PR will also allow passing hook parameters like in BaseSQLOperator. Refactored the get_hook method in BaseOperator so that in the future it can be used in a generic way, added tests to remove backward compatibility code for Airflow 2.8.x once min version of provider is Aiflow 3.0.0 or higher.


^ Add meaningful description above Read the Pull Request Guidelines for more information. In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed. In case of a new dependency, check compliance with the ASF 3rd Party License Policy. In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

dabla avatar Oct 10 '24 10:10 dabla

Looks like there is some problem with minimum version of one of microsoft dependencies. Bringing it down, brings opentelemetry dependency down and this seems to break imports. Unfold the failng test output for microsoft.azure and you will see it:

https://github.com/apache/airflow/actions/runs/11299097052/job/31429704576?pr=42891#step:7:6196

potiuk avatar Oct 14 '24 01:10 potiuk

Looks like there is some problem with minimum version of one of microsoft dependencies. Bringing it down, brings opentelemetry dependency down and this seems to break imports. Unfold the failng test output for microsoft.azure and you will see it:

https://github.com/apache/airflow/actions/runs/11299097052/job/31429704576?pr=42891#step:7:6196

Seems to be fine now after I updated the branch

dabla avatar Oct 14 '24 07:10 dabla

There are some compatibility Airflow 3 / Airflow 2 issues that need to be addressed :)

potiuk avatar Oct 14 '24 22:10 potiuk

There are some compatibility Airflow 3 / Airflow 2 issues that need to be addressed :)

All issues fixed

dabla avatar Oct 17 '24 12:10 dabla