Migration from elasticsearch 7.17 to elasticsearch 8

[bamthomas]

Current version of Elasticsearch is 9.2.1. Datashare version of Elasticsearch is 7.17.9.

For project health, security reasons, and performance reasons it would be nice to keep up with changes.

Furthermore, ES 8 is bringing new features that could be useful to the ICIJ like word2vect, kNN search (k nearest neighbors).

It is a breaking change:

• current code could not run with Elasticsearch 8
• when upgrading Elasticsearch libraries and lucene to 9.2.1 and 10.3.1 respectively the java API is broken so we have to migrate the client code

The migration will also break embedded elasticsearch mode because the ES server plugin libraries are not published since version 7.

We did a local test with @mvanzalu: local docker container opensearch 2 (i.e. compatible with version 7 of search API) a made it work with our java elasticsearch client (ElasticsearchIndexerTest was green). We had to add the plugins.security.disabled=true to the docker environment params.

So, one solution that could be the least impactful could be

1. upgrade the elasticsearch client to 8 and update java API
2. use opensearch server jar(s) aligned with search API version 8 (version 3 of opensearch)

Check that tests against external ES are working and tests against internal ES are also green.