Script failure : Due to warning message related to "gke-gcloud-auth-plugin"

[nagasukesh]

Issue: The composer_db_transfer.py script present in this repository is not able to adopt with warning messages mentioned as below.

WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.25+; use gcloud instead.\nTo learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke

The script fails at multiple places because of the warning messages and we have identified the places where it fails and have mentioned below:

1. In the method get_pods_config we are not able to load into json objects due to the above warning also being parsed into json object which throws an error for not meeting the json requirements.
2. In _import_tables method the csv import command is returning the operations id along with the above mentioned warning message due to which we are not able to execute the wait command for the respective SQL operation.

Other issues :

1. After resolving the above two issues with some minor changes like taking the sub string of the output( which removes the warning statement for that particular output) to the script we are experiencing errors related to airflow CLI command with exit code as 1. Sample airflow CLI commands(airflow connections delete airflow_db, airflow rotate-fernet-key, airflow db upgrade, airflow connections add airflow_db)

Thanks

[leahecole]

for those following along, @jacek-izykowski , ashires@ and I are in an email discussion with the reporter and I will put our findings here

[nagasukesh]

The major reason of the issue as i observed is because of the WARNING message mentioned above in the thread. Because the WARNING is being printed on std out, when storing the output of the command into variable and passing it to other methods. The WARNING message is also being passed and leading to errors.

Same has happened with fernet key . When i opened the key file generated it was storing the WARNING messages along with the key and same is being used in importing the database and throwing the invalid token error.

If the script is modified to grab these warning messages in the output then better results can be expected.

[dandhlee]

Friendly ping, any updates?

[dandhlee]

Friendly ping @jacek-izykowski