airflow icon indicating copy to clipboard operation
airflow copied to clipboard

Add new Compute Engine Operators and fix system tests

Open VladaZakharova opened this issue 2 years ago • 4 comments

Added new Compute Engine Operators and updated existing system tests Compute Engine operators.


^ Add meaningful description above

Read the Pull Request Guidelines for more information. In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed. In case of a new dependency, check compliance with the ASF 3rd Party License Policy. In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

VladaZakharova avatar Aug 08 '22 14:08 VladaZakharova

Some conflicts :(

potiuk avatar Aug 08 '22 21:08 potiuk

conflicts + errors to fix :(

potiuk avatar Aug 27 '22 19:08 potiuk

confilcts to be resolved - likely after future.annotation change.

potiuk avatar Sep 18 '22 20:09 potiuk

Docs failing :(

potiuk avatar Sep 20 '22 19:09 potiuk

Some tests are failing.

potiuk avatar Sep 26 '22 14:09 potiuk

Some tests are failing.

Yes, but the problem is that all these tests, that are failing in the CI, I can run locally without any errors for all Python versions. Can you please check what could be the reason of the errors? Thanks!

VladaZakharova avatar Oct 13 '22 08:10 VladaZakharova

Some tests are failing.

Yes, but the problem is that all these tests, that are failing in the CI, I can run locally without any errors for all Python versions.

How are you running the postgres CI tests?

Can you please check what could be the reason of the errors? Thanks!

The errors can be seen here:

 ==== Providers postgres: 9 failures ====

tests/providers/google/cloud/hooks/test_compute.py::TestGcpComputeHookApiCall::test_delete_instance_group_manager_should_not_throw_ex_when_project_id_none: google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started
tests/providers/google/cloud/hooks/test_compute.py::TestGcpComputeHookApiCall::test_delete_instance_should_not_throw_ex_when_project_id_none: google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started
tests/providers/google/cloud/hooks/test_compute.py::TestGcpComputeHookApiCall::test_delete_template_should_not_throw_ex_when_project_id_none: google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started
tests/providers/google/cloud/hooks/test_compute.py::TestGcpComputeHookApiCall::test_get_instance_group_manager_should_not_throw_ex_when_project_id_none: google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started
tests/providers/google/cloud/hooks/test_compute.py::TestGcpComputeHookApiCall::test_get_instance_should_not_throw_ex_when_project_id_none: google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started
tests/providers/google/cloud/hooks/test_compute.py::TestGcpComputeHookApiCall::test_get_template_should_not_throw_ex_when_project_id_none: google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started
tests/providers/google/cloud/hooks/test_compute.py::TestGcpComputeHookApiCall::test_insert_instance_group_manager_should_not_throw_ex_when_project_id_none: google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started
tests/providers/google/cloud/hooks/test_compute.py::TestGcpComputeHookApiCall::test_insert_instance_should_not_throw_ex_when_project_id_none: google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started
tests/providers/google/cloud/hooks/test_compute.py::TestGcpComputeHookApiCall::test_insert_template_should_not_throw_ex_when_project_id_none: google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started
Error: Process completed with exit code 1.

o-nikolas avatar Oct 13 '22 20:10 o-nikolas

Hi Team, I have resolved all the conflicts and errors in the tests but continue getting errors in Helm(https://github.com/apache/airflow/actions/runs/3263824766/jobs/5368266070#logs) and MSSQL(https://github.com/apache/airflow/actions/runs/3263824766/jobs/5366146241). Rebasing from main brach didn't work for me. Also i was trying to restart the build several times but it also didn't work for me. May be someone already faced this kind of problem?

VladaZakharova avatar Oct 19 '22 06:10 VladaZakharova

https://github.com/apache/airflow/issues/9941

lwyszomi avatar Oct 26 '22 06:10 lwyszomi

I think configure aws credentials update was not intentional. Please use "git submodule update" to update your version to the latest we have :)

potiuk avatar Oct 31 '22 05:10 potiuk

@potiuk These changes are open for a long time because we had some problems with build, maybe now when build passed is possibility to review and merge it?

lwyszomi avatar Nov 03 '22 10:11 lwyszomi