Add ability to interact with OpenSearch in AWS

[eladkal]

Body

We have OpenSearch provider but currently it can't be used with AWS auth mechanisms. We need to create a version of it in AWS provider (overriding only that particular portion of the Hook)

This effort started in https://github.com/apache/airflow/pull/34693

Note this is about querying open search - it's not about submitting Airflow logs to open search (for that we have another task https://github.com/apache/airflow/issues/33619 )

Committer

• [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[Taragolis]

We have OpenSearch provider but currently it can't be used with AWS auth mechanisms.

My 2 cents, it is happen due to fundamental problem with Connections and Hooks, it is very tight coupled with hooks and can't easily extend in current implementation due to current Airflow design.

Some abstract example:

• 1 Hook which interact with some stuff, like AwsomeDbWhichEveryoneWantToUse
• 10 Operators, 5 sensors, 15 Triggers all of them use Hook

In this case it is required not only change implementation in hoop but also change in operators, sensors, triggers. For each cloud provider with own auth methods 🙄

[rawwar]

In this case it is required not only change implementation in hoop but also change in operators, sensors, triggers. For each cloud provider with own auth methods 🙄

Would it be a good idea to implement a dialect like sqlalchemy does? This would allow the user to provide the necessary details and the dialect, and internally, we would choose the appropriate hook. This approach would make it easier to maintain or extend the system, especially when multiple hooks cater to different types of backend systems while the user will always use a single operator class with varying arguments

[Taragolis]

It is close to idea of dialects in SA, but more related to the authentication. Some of the hooks supports more then one method of auth thought extras, which is hard to validate in the UI due to mutuality exclusions