kafka-connect-elasticsearch icon indicating copy to clipboard operation
kafka-connect-elasticsearch copied to clipboard

support STRUCT message keys

Open deuscapturus opened this issue 3 years ago • 5 comments

Problem

There is currently no way to sink messages with composite keys without ignoring those keys.

Error "STRUCT is not supported as the document id."

https://github.com/confluentinc/kafka-connect-elasticsearch/issues/193

Solution

Convert message key STRUCT to a string for the doucment id.

Does this solution apply anywhere else?
  • [ ] yes
  • [ ] no

Test Strategy

unit test provided for DataConverter

Testing done:
  • [x] Unit tests
  • [ ] Integration tests
  • [ ] System tests
  • [ ] Manual tests

Release Plan

deuscapturus avatar Jul 16 '21 18:07 deuscapturus

It looks like @deuscapturus hasn't signed our Contributor License Agreement, yet.

The purpose of a CLA is to ensure that the guardian of a project's outputs has the necessary ownership or grants of rights over all contributions to allow them to distribute under the chosen licence. Wikipedia

You can read and sign our full Contributor License Agreement here.

Once you've signed reply with [clabot:check] to prove it.

Appreciation of efforts,

clabot

ghost avatar Jul 16 '21 18:07 ghost

[clabot:check]

deuscapturus avatar Jul 16 '21 23:07 deuscapturus

@confluentinc It looks like @deuscapturus just signed our Contributor License Agreement. :+1:

Always at your service,

clabot

ghost avatar Jul 16 '21 23:07 ghost

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

CLAassistant avatar Apr 03 '22 12:04 CLAassistant

I feel like a better solution is to just use a Transform on the Key

For example - https://jcustenborder.github.io/kafka-connect-documentation/projects/kafka-connect-transform-common/transformations/ToJSON.html

OneCricketeer avatar May 26 '22 14:05 OneCricketeer