airflow icon indicating copy to clipboard operation
airflow copied to clipboard

templated fields logic checks for cloud_storage_transfer_service

Open okirialbert opened this issue 1 year ago • 8 comments


The PR fixes assignment of parameters in constructor initialization for the google cloud_storage_transfer_service ensuring templated fields' parameters correspond to constructor initialization.

^ Add meaningful description above Read the Pull Request Guidelines for more information. In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed. In case of a new dependency, check compliance with the ASF 3rd Party License Policy. In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

okirialbert avatar Feb 18 '24 15:02 okirialbert

@romsharon98 Kindly review.

okirialbert avatar Feb 18 '24 15:02 okirialbert

And also required fix failing tests

Taragolis avatar Feb 18 '24 17:02 Taragolis

can u explain the purpose of this PR? In this issue https://github.com/apache/airflow/issues/36484 cloud_storage_transfer_service.py is shown as not implemented but when running pre-commit on it it looks like it already taken care. also in the pre-commit-config the file is not excluded (meaning template field works) maybe someone has fixed it but not related it to the story.

romsharon98 avatar Feb 18 '24 21:02 romsharon98

The list of files that needs to be fixed is in:

https://github.com/apache/airflow/blob/e5688b9ae9556f3e6da3b55437bea803fd0bb444/.pre-commit-config.yaml#L328-L335

Thus like what @romsharon98 mentioned. i am not sure how this PR relates to https://github.com/apache/airflow/issues/36484 ? Can you explain?

eladkal avatar Feb 19 '24 01:02 eladkal

The service had minor deprecating changes I opted to take care of that were relating to #36484

okirialbert avatar Feb 19 '24 03:02 okirialbert

Just wondering, anyone noticed this lines above of the changes 🙄 😵‍💫

https://github.com/apache/airflow/blob/2bcfe543c7781cd06430ff055864b811768b46ba/airflow/providers/google/cloud/operators/cloud_storage_transfer_service.py#L549-L556

Taragolis avatar Feb 19 '24 08:02 Taragolis

Hey @Taragolis, I presumed the already implemented filter check ensures parameter backward compatibility and your suggestion accounts for property declaration

okirialbert avatar Feb 19 '24 08:02 okirialbert

Hi @Taragolis should I add your code, replace or leave the changes as they are?

okirialbert avatar Feb 20 '24 09:02 okirialbert

@okirialbert can you rebase and fix conflicts?

eladkal avatar Mar 10 '24 19:03 eladkal

@eladkal yeah sure

okirialbert avatar Mar 11 '24 10:03 okirialbert

Needs rebase and resolve conflicts

eladkal avatar Mar 16 '24 09:03 eladkal