kafka-connect-elasticsearch icon indicating copy to clipboard operation
kafka-connect-elasticsearch copied to clipboard

CCMSG-1958: Elasticsearch 8 support through compatibility mode.

Open snehashisp opened this issue 3 years ago • 1 comments

Problem

The current version of the connector does not support ES 8 and above.

Solution

The High level rest client 7.17 has a compatibility mode which adds some headers to work with ES 8. When using ES 8 and above set this compatibility mode as true.

Although elastic does not explicitly say that the HLRC in compatibility mode won't work with older elastic versions (7.11 and lower) they do mention here that, under compatibility mode HLRC can also work with 7.11 and higher versions. Its best to only enable compatibility mode only with major version as 8 and above.

The connector already the ability to retrieve the version so we are using it to selectively turn on compatibility mode support (or otherwise) based on the version.

Does this solution apply anywhere else?
  • [ ] yes
  • [x] no
If yes, where?

Test Strategy

Testing done:
  • [x] Unit tests
  • [x] Integration tests
  • [ ] System tests
  • [x] Manual tests

Release Plan

snehashisp avatar Jun 08 '22 05:06 snehashisp

Looks good so far, lets add some ITs to test with ES8 docker VM and some unit tests

kpatelatwork avatar Jun 08 '22 17:06 kpatelatwork