airflow icon indicating copy to clipboard operation
airflow copied to clipboard
verified publisher icon apache

Migrate Google example DAGs to new design

Open potiuk opened this issue 3 years ago • 5 comments

There is a new design of system tests that was introduced by the AIP-47.

All current example dags need to be migrated and converted into system tests, so they can be run in the CI process automatically before releases.

This is an aggregated issue for all example DAGs related to Google provider. It is created to track progress of their migration.

List of paths to example DAGs:

  • [x] airflow/providers/google/leveldb/example_dags/example_leveldb.py
  • [x] airflow/providers/google/firebase/example_dags/example_firestore.py
  • [x] airflow/providers/google/ads/example_dags/example_ads.py
  • [x] airflow/providers/google/suite/example_dags/example_sql_to_sheets.py
  • [x] airflow/providers/google/suite/example_dags/example_gcs_to_sheets.py
  • [x] airflow/providers/google/suite/example_dags/example_gcs_to_gdrive.py
  • [x] airflow/providers/google/suite/example_dags/example_sheets.py
  • [x] airflow/providers/google/marketing_platform/example_dags/example_analytics.py
  • [ ] airflow/providers/google/marketing_platform/example_dags/example_display_video.py
  • [x] airflow/providers/google/marketing_platform/example_dags/example_campaign_manager.py
  • [x] airflow/providers/google/marketing_platform/example_dags/example_search_ads.py
  • [x] airflow/providers/google/cloud/example_dags/example_bigquery_queries.py
  • [x] airflow/providers/google/cloud/example_dags/example_gcs_to_sftp.py
  • [x] airflow/providers/google/cloud/example_dags/example_oracle_to_gcs.py
  • [x] airflow/providers/google/cloud/example_dags/example_compute_igm.py
  • [x] airflow/providers/google/cloud/example_dags/example_automl_vision_classification.py
  • [x] airflow/providers/google/cloud/example_dags/example_datacatalog.py
  • [x] airflow/providers/google/cloud/example_dags/example_bigquery_to_mssql.py
  • [x] airflow/providers/google/cloud/example_dags/example_mssql_to_gcs.py
  • [x] airflow/providers/google/cloud/example_dags/example_mysql_to_gcs.py
  • [x] airflow/providers/google/cloud/example_dags/example_bigquery_to_bigquery.py
  • [x] airflow/providers/google/cloud/example_dags/example_natural_language.py
  • [x] airflow/providers/google/cloud/example_dags/example_life_sciences.py
  • [x] airflow/providers/google/cloud/example_dags/example_automl_nl_text_extraction.py
  • [ ] airflow/providers/google/cloud/example_dags/example_automl_video_intelligence_classification.py
  • [x] airflow/providers/google/cloud/example_dags/example_trino_to_gcs.py
  • [x] airflow/providers/google/cloud/example_dags/example_bigquery_transfer.py
  • [ ] airflow/providers/google/cloud/example_dags/example_cloud_storage_transfer_service_gcp.py
  • [x] airflow/providers/google/cloud/example_dags/example_translate.py
  • [x] airflow/providers/google/cloud/example_dags/example_local_to_gcs.py
  • [ ] airflow/providers/google/cloud/example_dags/example_compute_ssh.py
  • [ ] airflow/providers/google/cloud/example_dags/example_dataflow_sql.py
  • [x] airflow/providers/google/cloud/example_dags/example_text_to_speech.py
  • [ ] airflow/providers/google/cloud/example_dags/example_presto_to_gcs.py
  • [ ] airflow/providers/google/cloud/example_dags/example_looker.py
  • [ ] airflow/providers/google/cloud/example_dags/example_sftp_to_gcs.py
  • [ ] airflow/providers/google/cloud/example_dags/example_facebook_ads_to_gcs.py
  • [x] airflow/providers/google/cloud/example_dags/example_gcs_to_bigquery.py
  • [ ] airflow/providers/google/cloud/example_dags/example_postgres_to_gcs.py
  • [x] airflow/providers/google/cloud/example_dags/example_spanner.py
  • [ ] airflow/providers/google/cloud/example_dags/example_gdrive_to_gcs.py
  • [ ] airflow/providers/google/cloud/example_dags/example_dlp.py
  • [ ] airflow/providers/google/cloud/example_dags/example_bigquery_dts.py
  • [x] airflow/providers/google/cloud/example_dags/example_datastore.py
  • [x] airflow/providers/google/cloud/example_dags/example_bigquery_operations.py
  • [ ] airflow/providers/google/cloud/example_dags/example_automl_nl_text_classification.py
  • [x] airflow/providers/google/cloud/example_dags/example_bigquery_to_gcs.py
  • [ ] airflow/providers/google/cloud/example_dags/example_automl_vision_object_detection.py
  • [ ] airflow/providers/google/cloud/example_dags/example_bigtable.py
  • [x] airflow/providers/google/cloud/example_dags/example_gcs_to_local.py
  • [ ] airflow/providers/google/cloud/example_dags/example_vertex_ai.py
  • [ ] airflow/providers/google/cloud/example_dags/example_automl_video_intelligence_tracking.py
  • [x] airflow/providers/google/cloud/example_dags/example_gcs_timespan_file_transform.py
  • [ ] airflow/providers/google/cloud/example_dags/example_dataproc_metastore.py
  • [ ] airflow/providers/google/cloud/example_dags/example_salesforce_to_gcs.py
  • [ ] airflow/providers/google/cloud/example_dags/example_functions.py
  • [ ] airflow/providers/google/cloud/example_dags/example_compute.py
  • [x] airflow/providers/google/cloud/example_dags/example_sheets_to_gcs.py
  • [ ] airflow/providers/google/cloud/example_dags/example_dataflow.py
  • [ ] airflow/providers/google/cloud/example_dags/example_datafusion.py
  • [ ] airflow/providers/google/cloud/example_dags/example_cloud_sql_query.py
  • [x] airflow/providers/google/cloud/example_dags/example_translate_speech.py
  • [ ] airflow/providers/google/cloud/example_dags/example_automl_tables.py
  • [x] airflow/providers/google/cloud/example_dags/example_cloud_memorystore.py
  • [x] airflow/providers/google/cloud/example_dags/example_bigquery_sensors.py
  • [x] airflow/providers/google/cloud/example_dags/example_gcs.py
  • [ ] airflow/providers/google/cloud/example_dags/example_mlengine.py
  • [ ] airflow/providers/google/cloud/example_dags/example_dataprep.py
  • [ ] airflow/providers/google/cloud/example_dags/example_cloud_storage_transfer_service_aws.py
  • [x] airflow/providers/google/cloud/example_dags/example_pubsub.py
  • [x] airflow/providers/google/cloud/example_dags/example_gcs_to_gcs.py
  • [x] airflow/providers/google/cloud/example_dags/example_s3_to_gcs.py
  • [ ] airflow/providers/google/cloud/example_dags/example_gdrive_to_local.py
  • [x] airflow/providers/google/cloud/example_dags/example_azure_fileshare_to_gcs.py
  • [x] airflow/providers/google/cloud/example_dags/example_workflows.py
  • [x] airflow/providers/google/cloud/example_dags/example_kubernetes_engine.py
  • [x] airflow/providers/google/cloud/example_dags/example_dataproc.py
  • [x] airflow/providers/google/cloud/example_dags/example_speech_to_text.py
  • [ ] airflow/providers/google/cloud/example_dags/example_cloud_composer.py
  • [ ] airflow/providers/google/cloud/example_dags/example_automl_translation.py
  • [x] airflow/providers/google/cloud/example_dags/example_stackdriver.py
  • [x] airflow/providers/google/cloud/example_dags/example_tasks.py
  • [x] airflow/providers/google/cloud/example_dags/example_calendar_to_gcs.py
  • [ ] airflow/providers/google/cloud/example_dags/example_video_intelligence.py
  • [ ] airflow/providers/google/cloud/example_dags/example_cloud_build.py
  • [ ] airflow/providers/google/cloud/example_dags/example_dataflow_flex_template.py
  • [ ] airflow/providers/google/cloud/example_dags/example_automl_nl_text_sentiment.py
  • [x] airflow/providers/google/cloud/example_dags/example_cloud_sql.py
  • [ ] airflow/providers/google/cloud/example_dags/example_vision.py

In progress:

  • airflow/providers/google/cloud/example_dags/example_bigquery_to_bigquery.py
  • airflow/providers/google/cloud/example_dags/example_dataflow.py
  • airflow/providers/google/cloud/example_dags/example_datafusion.py
  • airflow/providers/google/cloud/example_dags/example_dataproc.py
  • airflow/providers/google/cloud/example_dags/example_dataproc_metastore.py
  • airflow/providers/google/cloud/example_dags/example_datastore.py
  • airflow/providers/google/cloud/example_dags/example_gcs.py
  • airflow/providers/google/cloud/example_dags/example_gcs_timespan_file_transform.py
  • airflow/providers/google/cloud/example_dags/example_gcs_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_gcs_to_local.py
  • airflow/providers/google/cloud/example_dags/example_local_to_gcs.py

potiuk avatar Mar 22 '22 14:03 potiuk

Hey, please update the checklist.

These example DAGs are being removed by PR #22311:

  • airflow/providers/google/cloud/example_dags/example_bigquery_operations.py
  • airflow/providers/google/cloud/example_dags/example_bigquery_queries.py
  • airflow/providers/google/cloud/example_dags/example_bigquery_sensors.py

And these are WIP (Work In Progress) (not sure how we are going to mark them, but it would be good to notify others so that we will avoid work duplication):

  • airflow/providers/google/cloud/example_dags/example_bigquery_to_bigquery.py
  • airflow/providers/google/cloud/example_dags/example_dataflow.py
  • airflow/providers/google/cloud/example_dags/example_datafusion.py
  • airflow/providers/google/cloud/example_dags/example_dataproc.py
  • airflow/providers/google/cloud/example_dags/example_dataproc_metastore.py
  • airflow/providers/google/cloud/example_dags/example_datastore.py
  • airflow/providers/google/cloud/example_dags/example_gcs.py
  • airflow/providers/google/cloud/example_dags/example_gcs_timespan_file_transform.py
  • airflow/providers/google/cloud/example_dags/example_gcs_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_gcs_to_local.py
  • airflow/providers/google/cloud/example_dags/example_local_to_gcs.py

mnojek avatar Mar 24 '22 14:03 mnojek

Updated

potiuk avatar Mar 28 '22 13:03 potiuk

BTW. I will automate refreshing the list shortly :) @bhirsz :)

potiuk avatar Apr 28 '22 09:04 potiuk

Hi @potiuk Can we update the checklist if and when possible [So Can know which needs to be done, which are WIP]. Thanks :)

chethanuk avatar Jun 02 '22 23:06 chethanuk

Sure. I merge all the green ones and re-run the script to update the checklists

potiuk avatar Jun 03 '22 10:06 potiuk