kafka-connect-elasticsearch icon indicating copy to clipboard operation
kafka-connect-elasticsearch copied to clipboard

How to Convert JSON String field to ES Object?

Open jasoncodes42 opened this issue 2 years ago • 0 comments

What's the recommended way to convert a JSON String field (from Debezium MySQL JSON field) into an ES structured object field?

Here's my ES record:

"hits": [
            {
                "_index": "instance._b5523bb03a3d4d9cb2649a82a4e58ec5._00d4c37c0b493d1c9b68fd4f8fd4bc0d.record",
                "_id": "4c8a411a-6aac-41b1-8376-4301efe1f7ee",
                "_score": 1.0,
                "_ignored": [
                    "data.keyword"
                ],
                "_source": {
                    "startDateTime": 1687579920158,
                    "endDateTime": 1687579921872,
                    "data": "{\"CPU\":{\"user\":\"11.2\",\"system\":\"0.2\",\"userNice\":\"0.0\",\"idle\":\"88.3\",\"IOWait\":\"0.0\",\"hardwareIRQ\":\"0.0\",\"softwareIRQ\":\"0.2\",\"stealTime\":\"0.0\"},\"load\":{\"oneMinute\":\"4.68\",\"fiveMinute\":\"4.87\",\"fifteenMinute\":\"5.03\"},\"memory\":{\"total\":128570,\"free\":15776,\"used\":87385},\"swap\":{\"total\":4096,\"free\":1279,\"used\":2817}}"
                }
            },

As you can see from above I have 3 fields coming from MySQL:

  • startDateTime (timestamp)
  • endDateTime (timestamp)
  • data (string)

I want to convert the data field into an object from a JSON string. I'm not able to change it on the source so it must be transformed on the sink.

Does something exist for this? Or do I need to write a transformation for it?

jasoncodes42 avatar Jun 25 '23 06:06 jasoncodes42