kafka-connect-elasticsearch
kafka-connect-elasticsearch copied to clipboard
CCMSG-1958: Elasticsearch 8 support through compatibility mode.
Problem
The current version of the connector does not support ES 8 and above.
Solution
The High level rest client 7.17 has a compatibility mode which adds some headers to work with ES 8. When using ES 8 and above set this compatibility mode as true.
Although elastic does not explicitly say that the HLRC in compatibility mode won't work with older elastic versions (7.11 and lower) they do mention here that, under compatibility mode HLRC can also work with 7.11 and higher versions. Its best to only enable compatibility mode only with major version as 8 and above.
The connector already the ability to retrieve the version so we are using it to selectively turn on compatibility mode support (or otherwise) based on the version.
Does this solution apply anywhere else?
- [ ] yes
- [x] no
If yes, where?
Test Strategy
Testing done:
- [x] Unit tests
- [x] Integration tests
- [ ] System tests
- [x] Manual tests
Release Plan
Looks good so far, lets add some ITs to test with ES8 docker VM and some unit tests