astronomer-providers icon indicating copy to clipboard operation
astronomer-providers copied to clipboard

Implement Async Azure Operators

Open phanikumv opened this issue 2 years ago • 1 comments

Please look at docs to see if there are other operators we should make Async (Principle - Only create Async operators for the “sync-version” of operators that do some level of polling; take more than a few seconds to complete) for the Azure Provider.

  • ADLSListOperator
  • AzureDataLakeStorageDeleteOperator
  • AzureDataLakeStorageListOperator
  • AzureDataExplorerQueryOperator
  • AzureBatchOperator
  • AzureContainerInstancesOperator
  • AzureCosmosInsertDocumentOperator
  • WasbDeleteBlobOperator

Acceptance Criteria:

  • Unit Tests coverage in the PR (80% Code Coverage -- We will need to add CodeCov separately to measure code cov) with all of them passing
  • Example DAG using the async Operator that can be used to run Integration tests that are parametrized via Environment variables. Example - https://github.com/apache/airflow/blob/8a03a505e1df0f9de276038c5509135ac569a667/airflow/providers/google/cloud/example_dags/example_bigquery_to_gcs.py#L33-L35
  • Add proper docstrings for each of the methods and functions including Example DAG on how it should be used (populate
  • Exception Handling in case of errors
  • Improve the OSS Docs to make sure it covers the following:
  • Has an example DAG for the sync version
  • How to add a connection via Environment Variable & explain each of the fields. Example - https://airflow.apache.org/docs/apache-airflow-providers-postgres/stable/connections/postgres.html
  • How to use Guide for the Operator - example: https://airflow.apache.org/docs/apache-airflow-providers-postgres/stable/operators/postgres_operator_howto_guide.html

phanikumv avatar Mar 07 '22 05:03 phanikumv

I have separated out the following operators from this issue:

  • https://github.com/astronomer/astronomer-providers/issues/186

kaxil avatar Apr 04 '22 19:04 kaxil