Use base aws classes in amazon provider Operators/Sensors/Triggers

[Taragolis]

Body

Right now some inconsistency exists between different Operators/Sensors/Triggers which are use AwsBaseGeneric hook. After #34784 and #34884 we could make all of them (except Transfer Operators) use same subset of generic attributes.

It might make more easier to end users use this objects (don't need to think is it region or region_name or is it even support this arguments), as well as make easier extend by new new hook parameters to all this operators in the one go.

General Approach for solve each case:

• Use AwsBaseOperator for operators, AwsBaseSensor for sensors. In some very specific case e.g. multi inheritance required to use AwsBaseHookMixin (see: AppflowRecordsShortCircuitOperator)
• Provide additional arguments to AwsBaseWaiterTrigger (if required)
• Wrap template_fields parameters
• Attach generic documentation into the service specific page into the Amazon Provider Documentation
• Validate that all parameters propagated to operator/sensor specific hook

All this steps already done for some of the operators, so it could be use as a reference

• [x] Use base aws classes in Amazon AppFlow Operators (https://github.com/apache/airflow/pull/35082)
  • operator(s)
• [x] Use base aws classes in Amazon Athena Operators/Sensors/Triggers (https://github.com/apache/airflow/pull/35133)
  • operator(s)
  • sensor(s)
  • trigger(s)
• [ ] Use base aws classes in AWS Batch Operators/Sensors/Triggers (https://github.com/apache/airflow/pull/35226)
  • operator(s) https://github.com/apache/airflow/pull/49172
  • sensor(s) https://github.com/apache/airflow/pull/49172
  • trigger(s)
• [x] Use base aws classes in AWS CloudFormation Operators/Sensors https://github.com/apache/airflow/pull/36771
  • operator(s)
  • sensor(s)
• [x] Use base aws classes in AWS Datasync Operators https://github.com/apache/airflow/pull/36766
  • operator(s)
• [x] Use base aws classes in AWS DMS Operators/Sensors https://github.com/apache/airflow/pull/36772
  • operator(s)
  • sensor(s)
• [x] Use base aws classes in Amazon DynamoDB Sensors https://github.com/apache/airflow/pull/36770
  • sensor(s)
• [x] Use base aws classes in Amazon EC2 Operators/Sensors/Triggers https://github.com/apache/airflow/pull/47931
  • operator(s)
  • sensor(s)
  • trigger(s)
• [x] Use base aws classes in Amazon ECS Operators/Sensors/Triggers (https://github.com/apache/airflow/pull/36393)
  • operator(s)
  • sensor(s)
  • trigger(s)
• [x] Use base aws classes in Amazon EKS Operators/Sensors/Triggers https://github.com/apache/airflow/pull/48192
  • operator(s)
  • sensor(s)
  • trigger(s)
• [x] Use base aws classes in Amazon EMR Operators/Sensors/Triggers https://github.com/apache/airflow/pull/49486
  • operator(s)
  • sensor(s)
  • trigger(s)
• [x] Use base aws classes in Amazon EventBridge Operators https://github.com/apache/airflow/pull/36765
  • operator(s)
• [ ] Use base aws classes in AWS Glue Operators/Sensors/Triggers
  • operator(s) https://github.com/apache/airflow/pull/49750
  • sensor(s)
  • trigger(s)
• [x] Use base aws classes in AWS Glue Data Catalog Sensors https://github.com/apache/airflow/pull/40492
  • sensor(s)
• [x] Use base aws classes in AWS Glue Crawlers Operators/Sensors/Triggers https://github.com/apache/airflow/pull/40504
  • operator(s)
  • sensor(s)
  • trigger(s)
• [x] Use base aws classes in AWS Glue DataBrew Operators/Triggers https://github.com/apache/airflow/pull/41848
  • operator(s)
  • trigger(s)
• [x] Use base aws classes in AWS Lambda Operators/Sensors/Triggers (https://github.com/apache/airflow/pull/34890)
  • operator(s)
  • sensor(s)
  • trigger(s)
• [x] Use base aws classes in Amazon QuickSight Operators/Sensors https://github.com/apache/airflow/pull/36776
  • operator(s)
  • sensor(s)
• [x] Use base aws classes in Amazon RDS Operators/Sensors/Triggers https://github.com/apache/airflow/pull/48872
  • operator(s)
  • sensor(s)
  • trigger(s)
• [ ] Use base aws classes in Amazon Redshift clusters Operators/Sensors/Triggers
  • operator(s) https://github.com/apache/airflow/pull/51129
  • sensor(s) https://github.com/apache/airflow/pull/51129
  • trigger(s)
• [x] Use base aws classes in AWS Redshift Data API Operators https://github.com/apache/airflow/pull/36764
  • operator(s)
• [x] Use base aws classes in Amazon S3 Operators/Sensors/Triggers https://github.com/apache/airflow/pull/47321
  • operator(s)
  • sensor(s)
  • trigger(s)
• [x] Use base aws classes in Amazon S3 Glacier Operators/Sensors (https://github.com/apache/airflow/pull/35108)
  • operator(s)
  • sensor(s)
• [ ] Use base aws classes in Amazon SageMaker Operators/Sensors/Triggers
  • operator(s) https://github.com/apache/airflow/pull/50321
  • sensor(s) https://github.com/apache/airflow/pull/50321
  • trigger(s)
• [x] Use base aws classes in Amazon SNS Operators https://github.com/apache/airflow/pull/36615
  • operator(s)
• [x] Use base aws classes in Amazon SQS Operators/Sensors/Triggers https://github.com/apache/airflow/pull/36613
  • operator(s)
  • sensor(s)
  • trigger(s)
• [x] Use base aws classes in AWS Step Functions Operators/Sensors/Triggers https://github.com/apache/airflow/pull/36468
  • operator(s)
  • sensor(s)
  • trigger(s)

Committer

• [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[ferruzzi]

Thanks for getting this set up!

[gopidesupavan]

@Taragolis i am happy to look into this Use base aws classes in AWS Glue Crawlers Operators/Sensors/Triggers ? or anyone is working already?

[prakharcode]

@Taragolis I am happy to take EMR, EC2 and S3 (operators) here

[gopidesupavan]

@vincbeck @eladkal Working on this Use base aws classes in AWS Glue Data Catalog Sensors didnt find anything in the documentation and example code for this. Planning to add them here https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/operators/glue.html is this fine?

[vincbeck]

@vincbeck @eladkal Working on this Use base aws classes in AWS Glue Data Catalog Sensors didnt find anything in the documentation and example code for this. Planning to add them here https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/operators/glue.html is this fine?

Please yes :)

[eladkal]

@gopidesupavan do you plan to address the other Glue services that were left?

[gopidesupavan]

@eladkal Yes will update other services, coming week.

[ellisms]

I'll take a look at EC2.

[hinchliff]

I'd love to see the RDS sensors extend from the AwsBaseSensor. We have a use-case where we need to support multiple regions, and requiring a new Connection for each region is painful.

[dominikhei]

Glue Databrew was added in #41848

I'll have a look at the remaining Glue ones.

[dominikhei]

Glue Databrew was added in #41848

I'll have a look at the remaining Glue ones.

Speaking on this. The GlueJobCompleteTrigger currently does not come with a waiter in boto3. @ellisms as you implemented the Operator which pattern do you think is preferable? Is there something I am overlooking.

1. Not letting the Trigger inherit from AwsBaseWaiterTrigger and just port the Sensor. This is like it is done with the GlueCatalogPartitionSensor which inherits from the base class, whilst its trigger does not

2. Or create a new custom waiter and port the Trigger to the AwsBaseWaiterTrigger?

---

I have now opted for the second option, will test all changes with dags and then open a Draft PR

[ellisms]

@dominikhei I'd go with option 2. A quick scan of the current waiter code in the hook doesn't reveal anything special. A custom waiter would be able to handle it.