remove deprecations google cloud

[Bowrna]

Related: #39485

---

^ Add meaningful description above Read the Pull Request Guidelines for more information. In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed. In case of a new dependency, check compliance with the ASF 3rd Party License Policy. In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[Bowrna]

I will add the PR for other google cloud examples as each of the fix in it have different set of doubts. Raising multiple PR to avoid confusion

[Bowrna]

i am facing issue in celery executor tests though the changes I did didn't affect that part. any idea on how I can resolve that?

[VladaZakharova]

@VladaZakharova In the docs https://cloud.google.com/python/docs/reference/container/latest/google.cloud.container_v1.types.Cluster, I couldn't see attribute name node_pool being set. It's only node_pools. Can you let me know if there is any typo error in node_pool in this part?

https://github.com/apache/airflow/blob/cb57a677be8e189ad3c4a077c0d476cb9f250df2/airflow/providers/google/cloud/operators/kubernetes_engine.py#L389-L403

Can you please tell why you decided to change this initial_node_count? Was it failing?

[Bowrna]

yes @VladaZakharova I got a deprecation message in this file saying.

airflow.exceptions.AirflowProviderDeprecationWarning:The body field 'initial_node_count' is deprecated. Use 'node_pool.initial_node_count' instead.

But when I added the cluster config with value like this { "name": GKE_CLUSTER_NAME, "workload_identity_config": { "workload_pool": WORKLOAD_POOL, }, "initial_node_count": 1, "node_pool": { "initial_node_count": 1, } , } I got an error message saying Field body['node_pools'] is required if none of fields

Then I fixed the node_pool in the above json to node_pools

{ "name": GKE_CLUSTER_NAME, "workload_identity_config": { "workload_pool": WORKLOAD_POOL, }, "initial_node_count": 1, "node_pools": { "initial_node_count": 1, } , }` The final error is the one you shared when running this example dag in your gcp project. Hope this helps

[Bowrna]

#39727 - similar issue Facing issues in running these test dags in GCP environment. I thought that it was a deprecation warning removal and went with the configuration parsing as suggested in the deprecation warning message assuming it would be right. I didn't verify the dags by running on a GCP environment to verify if the config is right. In case if the issue is persisting, please feel free to fix this PR.

[shahar1]

@Bowrna Looks good overall :) If there are still issues that occur locally but aren't indicated by CI, I think that it would be better to define them as within a GitHub issue and handle them in a separate PR. I'd be happy to assist if necessary.

[github-actions[bot]]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.