Add support for Elasticsearch 8

[masci]

Is your feature request related to a problem? Please describe. Haystack should be compatible with more recent Elasticsearch releases, possibly also leveraging new features

Describe the solution you'd like The kNN search API was changed with version 8 and most likely will speed up dense retrieval. Other features could also be interesting for Haystack, e.g. pytorch model support

• [ ] Checking compatibility with old elasticsearch versions and probably have two ElasticsearchDocumentStores (on version 8, one version 7)
• [ ] Try running the latest version 7.x version -> 7.17 https://github.com/deepset-ai/haystack/pull/2993
• [ ] Fix error: cannot import name 'RequestsHttpConnection' from 'elasticsearch' https://github.com/deepset-ai/haystack/pull/2991
• [ ] Check migration guides and identify blockers/necessary changes https://www.elastic.co/guide/en/elasticsearch/reference/master/migrating-8.0.html
• [ ] Implement/support new features added in release 8.x (kNN search API etc.)

[UmarEm]

Hey @masci @ZanSara @julian-risch Can we get the timeline/plan for Elasticsearch 8.x support. You had earlier mentioned, that the support for Elasticsearch 8.x is scheduled for this quater It would be great, if we could know what is the progress, and when can we expect the update.

Thank you

[nickchomey]

Above it says 8.3.3 is pinned for testing, but ES 8.4 moved the ANN mechanism into the _search endpoint, which probably would require various changes on your end. I'm just sharing this in case you missed the recent announcement/update.

And any guidance (however tentative) that you might be able to provide about when this will be supported by Haystack would be very appreciated. Thanks for the great work!

https://www.elastic.co/guide/en/elasticsearch/reference/current/release-highlights.html#integrate_ann_into_search_endpoint

[julian-risch]

I noticed that Haystack's dependency BEIR pins the version of the elasticsearch client to elasticsearch==7.9.1. However, BEIR is just an optional dependency (not mandatory) and is only installed for the installation options all, all-gpu, and beir.

[anakin87]

I noticed that Haystack's dependency BEIR pins the version of the elasticsearch client to elasticsearch==7.9.1. However, BEIR is just an optional dependency (not mandatory) and is only installed for the installation options all, all-gpu, and beir.

Similar problem (beir pins faiss-cpu): #3218

[julian-risch]

Slightly related: A user needed just a few lines of code to use Haystack with Elastic on GCP https://github.com/deepset-ai/haystack/discussions/4547

[julian-risch]

ES8 is now supported and the last related issue has been merged: https://github.com/deepset-ai/haystack/pull/5320

[btofan]

Does the current solution support the approximate KNN feature that elasticsearch offers? According to the elasticsearch documentation (cf. https://www.elastic.co/guide/en/elasticsearch/reference/current/knn-search.html#approximate-knn), a specific kind of indexing and querying is required for approximate KNN search. I couldn't find the solution in the current commits. (This was also one of the main intentions of this issue and the related issues https://github.com/deepset-ai/haystack/issues/2833, https://github.com/deepset-ai/haystack/issues/2368.)