airflow icon indicating copy to clipboard operation
airflow copied to clipboard
verified publisher icon apache

Migrate Google system tests to new design

Open potiuk opened this issue 3 years ago • 7 comments

There is a new design of system tests that was introduced by the AIP-47.

All current system tests need to be migrated, so they can be run in the CI process automatically before releases.

This is an aggregated issue for all system tests related to Google provider.

It is created to track progress of their migration.

List of paths to test files with corresponding number of tests inside:

  • [x] tests/providers/google/leveldb/operators/test_leveldb_system.py (1)
  • [x] tests/providers/google/firebase/operators/test_firestore_system.py (1)
  • [x] tests/providers/google/suite/operators/test_sheets_system.py (1)
  • [x] tests/providers/google/suite/transfers/test_gcs_to_sheets_system.py (1)
  • [x] tests/providers/google/marketing_platform/operators/test_analytics_system.py (1)
  • [x] tests/providers/google/marketing_platform/operators/test_search_ads_system.py (1)
  • [ ] tests/providers/google/marketing_platform/operators/test_display_video_system.py (3)
  • [x] tests/providers/google/marketing_platform/operators/test_campaign_manager_system.py (1)
  • [ ] tests/providers/google/cloud/operators/test_dataprep_system.py (1)
  • [x] tests/providers/google/cloud/operators/test_kubernetes_engine_system.py (1)
  • [ ] tests/providers/google/cloud/operators/test_compute_system.py (2)
  • [ ] tests/providers/google/cloud/operators/test_functions_system.py (1)
  • [x] tests/providers/google/cloud/operators/test_spanner_system.py (1)
  • [ ] tests/providers/google/cloud/operators/test_cloud_composer_system.py (1)
  • [x] tests/providers/google/cloud/operators/test_speech_to_text_system.py (1)
  • [x] tests/providers/google/cloud/operators/test_cloud_memorystore_system.py (2)
  • [x] tests/providers/google/cloud/operators/test_life_sciences_system.py (1)
  • [x] tests/providers/google/cloud/operators/test_bigquery_system.py (5)
  • [x] tests/providers/google/cloud/operators/test_stackdriver_system.py (1)
  • [x] tests/providers/google/cloud/operators/test_translate_speech_system.py (1)
  • [x] tests/providers/google/cloud/operators/test_pubsub_system.py (2)
  • [ ] tests/providers/google/cloud/operators/test_cloud_build_system.py (2)
  • [ ] tests/providers/google/cloud/operators/test_vertex_ai_system.py (3)
  • [ ] tests/providers/google/cloud/operators/test_bigquery_dts_system.py (1)
  • [ ] tests/providers/google/cloud/operators/test_dataflow_system.py (6)
  • [ ] tests/providers/google/cloud/operators/test_datafusion_system.py (1)
  • [ ] tests/providers/google/cloud/operators/test_vision_system.py (3)
  • [x] tests/providers/google/cloud/operators/test_datacatalog_system.py (1)
  • [ ] tests/providers/google/cloud/operators/test_bigtable_system.py (1)
  • [x] tests/providers/google/cloud/operators/test_natural_language_system.py (1)
  • [x] tests/providers/google/cloud/operators/test_dataproc_system.py (2)
  • [ ] tests/providers/google/cloud/operators/test_dataproc_metastore_system.py (1)
  • [x] tests/providers/google/cloud/operators/test_cloud_sql_system.py (2)
  • [x] tests/providers/google/cloud/operators/test_gcs_timespan_file_transform_system.py (1)
  • [x] tests/providers/google/cloud/operators/test_tasks_system.py (1)
  • [x] tests/providers/google/cloud/operators/test_workflows_system.py (1)
  • [ ] tests/providers/google/cloud/operators/test_datastore_system.py (2)
  • [x] tests/providers/google/cloud/operators/test_text_to_speech_system.py (1)
  • [ ] tests/providers/google/cloud/operators/test_video_intelligence_system.py (1)
  • [x] tests/providers/google/cloud/operators/test_gcs_system.py (2)
  • [ ] tests/providers/google/cloud/operators/test_mlengine_system.py (1)
  • [ ] tests/providers/google/cloud/operators/test_automl_system.py (2)
  • [x] tests/providers/google/cloud/operators/test_translate_system.py (1)
  • [ ] tests/providers/google/cloud/operators/test_dlp_system.py (4)
  • [ ] tests/providers/google/cloud/operators/test_cloud_storage_transfer_service_system.py (1)
  • [x] tests/providers/google/cloud/transfers/test_gcs_to_local_system.py (1)
  • [x] tests/providers/google/cloud/transfers/test_bigquery_to_bigquery_system.py (1)
  • [x] tests/providers/google/cloud/transfers/test_bigquery_to_gcs_system.py (1)
  • [x] tests/providers/google/cloud/transfers/test_gcs_to_gcs_system.py (1)
  • [ ] tests/providers/google/cloud/transfers/test_salesforce_to_gcs_system.py (1)
  • [x] tests/providers/google/cloud/transfers/test_gcs_to_bigquery_system.py (1)
  • [x] tests/providers/google/cloud/transfers/test_azure_fileshare_to_gcs_system.py (1)
  • [x] tests/providers/google/cloud/transfers/test_s3_to_gcs_system.py (1)
  • [x] tests/providers/google/cloud/transfers/test_mysql_to_gcs_system.py (1)
  • [x] tests/providers/google/cloud/transfers/test_sheets_to_gcs_system.py (1)
  • [ ] tests/providers/google/cloud/transfers/test_presto_to_gcs_system.py (1)
  • [ ] tests/providers/google/cloud/transfers/test_gdrive_to_gcs_system.py (1)
  • [ ] tests/providers/google/cloud/transfers/test_sftp_to_gcs_system.py (1)
  • [ ] tests/providers/google/cloud/transfers/test_postgres_to_gcs_system.py (1)
  • [x] tests/providers/google/cloud/transfers/test_local_to_gcs_system.py (1)
  • [x] tests/providers/google/cloud/transfers/test_trino_to_gcs_system.py (1)
  • [x] tests/providers/google/cloud/transfers/test_calendar_to_gcs_system.py (1)
  • [x] tests/providers/google/cloud/transfers/test_gcs_to_sftp_system.py (1)
  • [ ] tests/providers/google/cloud/transfers/test_facebook_ads_to_gcs_system.py (1)
  • [ ] tests/providers/google/cloud/hooks/test_kms_system.py (2)
  • [ ] tests/providers/google/cloud/hooks/test_bigquery_system.py (5)
  • [ ] tests/providers/google/cloud/hooks/test_secret_manager_system.py (3)
  • [ ] tests/providers/google/cloud/hooks/test_compute_ssh_system.py (1)
  • [ ] tests/providers/google/cloud/secrets/test_secret_manager_system.py (2)
  • [ ] tests/providers/google/cloud/log/test_gcs_task_handler_system.py (1)
  • [ ] tests/providers/google/cloud/log/test_stackdriver_task_handler_system.py (2)

For anyone involved in working with this issue - please, make sure to also check if all example DAGs are migrated. The issue for them is stored separately. Search for Migrate Google example DAGs to new design

potiuk avatar Mar 22 '22 13:03 potiuk

Hey, please update the checklist.

These system tests are being removed by PR #22311:

  • tests/providers/google/cloud/operators/test_bigquery_system.py

And these are WIP (Work In Progress) (not sure how we are going to mark them, but it would be good to notify others so that we will avoid work duplication): tests/providers/google/cloud/operators/*:

  • tests/providers/google/cloud/operators/test_dataflow_system.py
  • tests/providers/google/cloud/operators/test_datafusion_system.py
  • tests/providers/google/cloud/operators/test_dataproc_metastore_system.py
  • tests/providers/google/cloud/operators/test_dataproc_system.py
  • tests/providers/google/cloud/operators/test_datastore_system.py
  • tests/providers/google/cloud/operators/test_gcs_system.py
  • tests/providers/google/cloud/operators/test_gcs_timespan_file_transform_system.py

tests/providers/google/cloud/transfers/*:

  • tests/providers/google/cloud/transfers/test_bigquery_to_bigquery_system.py
  • tests/providers/google/cloud/transfers/test_bigquery_to_gcs_system.py
  • tests/providers/google/cloud/transfers/test_gcs_to_gcs_system.py
  • tests/providers/google/cloud/transfers/test_gcs_to_local_system.py
  • tests/providers/google/cloud/transfers/test_local_to_gcs_system.py

mnojek avatar Mar 24 '22 14:03 mnojek

Updated!

potiuk avatar Mar 28 '22 13:03 potiuk

Hey, please set this issue as "in progress" (so it has better visibility in linked project). The following system tests are in progress (in review):

  • tests/providers/google/cloud/operators/test_text_to_speech_system.py #23247
  • tests/providers/google/cloud/operators/test_tasks_system.py #23282
  • tests/providers/google/cloud/operators/test_stackdriver_system.py #23320

bhirsz avatar Apr 28 '22 07:04 bhirsz

Again. I am thinking about automating those status updates :D. Stay tuned ....

potiuk avatar Apr 28 '22 09:04 potiuk

Watch out - it looks like you catched "I need to automate it disease" it's highly contagious! :)

bhirsz avatar Apr 28 '22 09:04 bhirsz

Watch out - it looks like you catched "I need to automate it disease" it's highly contagious! :)

Actually this is chronic disease of mine. I think I caught it some 20 years ago :)

potiuk avatar Apr 28 '22 14:04 potiuk

Tool to automatically sync all the issues adeded : https://github.com/apache/airflow/pull/23745 (cc: @bhirsz @mnojek )

potiuk avatar May 17 '22 11:05 potiuk