Add option to use autogenerated id on index requests

[tvm18860]

trafficstars

Problem

While setting the document ID when indexing does provide exactly once delivery, it does put more load on Elasticsearch and is not necessary for all use cases.

A PR for this issue has been made before (https://github.com/confluentinc/kafka-connect-elasticsearch/pull/393). This PR is largely an update to that one, as there were many merge conflicts that needed resolving there as it fell out of date.

Addresses https://github.com/confluentinc/kafka-connect-elasticsearch/issues/139 and https://github.com/confluentinc/kafka-connect-elasticsearch/issues/97

Solution

Add a new option to use the autogenerated document id on index requests. The new option (use.autogenerated.ids) will default to false and only be applicable when write.method is set to INSERT.

Note that the large diff in the DataCoverter class on the convertRecord method is a result of having to pull a chunk of that code out into a separate method. The checkstyle plugin was throwing errors when I added an extra if statement in that the cyclomatic complexity got too high. The more relevant change is on lines 169-173.

Does this solution apply anywhere else?

• [ ] yes
• [x] no

If yes, where?

Test Strategy

Testing done:

• [x] Unit tests
• [ ] Integration tests
• [ ] System tests
• [x] Manual tests

We are running live connectors with this change now.

Release Plan

[ghost]

@confluentinc It looks like @tvm18860 just signed our Contributor License Agreement. :+1:

Always at your service,

clabot

[azlev]

Hello!

I'm interested in this PR. How can I help to merge it?

[tvm18860]

I think this PR is back in a good state now, I had missed a checkstyle validation error after an earlier rebase.

[morrone]

We would also find this very useful.

Are there any comments from the reviewers about what it would take to get this added? Are there any issues with the approach? Does it just need to be updated again to pass the checks?

[cla-assistant[bot]]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
2 out of 3 committers have signed the CLA.

:white_check_mark: sp-gupta
:white_check_mark: gjw13
:x: ConfluentJenkins
_{You have signed the CLA already but the status is still pending? Let us recheck it.}