astronomer-providers
astronomer-providers copied to clipboard
Implement Async Azure Operators
Please look at docs to see if there are other operators we should make Async (Principle - Only create Async operators for the “sync-version” of operators that do some level of polling; take more than a few seconds to complete) for the Azure Provider.
- ADLSListOperator
- AzureDataLakeStorageDeleteOperator
- AzureDataLakeStorageListOperator
- AzureDataExplorerQueryOperator
- AzureBatchOperator
- AzureContainerInstancesOperator
- AzureCosmosInsertDocumentOperator
- WasbDeleteBlobOperator
Acceptance Criteria:
- Unit Tests coverage in the PR (80% Code Coverage -- We will need to add CodeCov separately to measure code cov) with all of them passing
- Example DAG using the async Operator that can be used to run Integration tests that are parametrized via Environment variables. Example - https://github.com/apache/airflow/blob/8a03a505e1df0f9de276038c5509135ac569a667/airflow/providers/google/cloud/example_dags/example_bigquery_to_gcs.py#L33-L35
- Add proper docstrings for each of the methods and functions including Example DAG on how it should be used (populate
- Exception Handling in case of errors
- Improve the OSS Docs to make sure it covers the following:
- Has an example DAG for the sync version
- How to add a connection via Environment Variable & explain each of the fields. Example - https://airflow.apache.org/docs/apache-airflow-providers-postgres/stable/connections/postgres.html
- How to use Guide for the Operator - example: https://airflow.apache.org/docs/apache-airflow-providers-postgres/stable/operators/postgres_operator_howto_guide.html
I have separated out the following operators from this issue:
- https://github.com/astronomer/astronomer-providers/issues/186