Deployment: Airflow 2.9.1 Installed on Kubernetes via Helm Chart - Opensearch Connection Issue (Error 406)

[emirhanaydin61]

Apache Airflow version

2.9.1

If "Other Airflow 2 version" selected, which one?

No response

What happened?

Hello, we are using airflow 2.4.0 and were reading logs from opensearch. When we upgraded airflow to version 2.9.1, we encountered this error: "elasticsearch.ApiError: ApiError(406, 'Content-Type header [application/vnd.elasticsearch+json; compatible-with=8] is not supported', 'Content-Type header [application/vnd.elasticsearch+json; compatible-with=8] is not supported')". Could you please provide support regarding the solution to this issue?

What you think should happen instead?

No response

How to reproduce

Upgrade airflow from version 2.4.0 to 2.9.1. Read logs from Opensearch. Execute a specific workflow or perform a particular operation. Observe the error message.

Operating System

CentOS Linux 7 (Core) / kubernetes v1.21.5

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

Airflow version 2.9.1 deployed on Kubernetes using Helm chart. Opensearch version 2.11.1.

Anything else?

No response

Are you willing to submit PR?

• [X] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[adrienyhuel]

Hello @emirhanaydin61

It is related to #33619

Airflow use "apache-airflow-providers-elasticsearch" to read logs, and the latests versions aren't compatible with Opensearch.

You can downgrade this python package to version 5.0.0, it is the lastest version which still support Opensearch.

[eladkal]

Yes this will be solved when https://github.com/apache/airflow/issues/33619 is resolved but this reports indicate that this is more than just a feature request as https://github.com/apache/airflow/issues/33619 states as users who are unable to migrate are blocked. ~cc @shubham22 maybe worth considering for AWS to priortize this task. I think it's very close to the finish line as mentioned in https://github.com/apache/airflow/issues/33619#issuecomment-1985675070~ (on second thought this isn't really about the Amazon OpenSearch but about OpenSearch in general)