camel-kafka-connector icon indicating copy to clipboard operation
camel-kafka-connector copied to clipboard

Splitting files through the Kafka connector

Open arjun180 opened this issue 3 years ago • 0 comments

I am using an AWS s3 source connector to transfer files from an s3 bucket to a Kafka topic. I'm running into a problem where the size of the messages is too big. And the size is well beyond the buffer size I can setup. After investigation, I realized this was because the messages are an aggregation of multiple messages. Is there way I can split the message within the Kafka connector (through a delimiter) and then send the messages one-by-one to the Kafka topic? Order of messages is not a priority.

I am using this Kafka connector : https://camel.apache.org/camel-kafka-connector/0.11.x/reference/connectors/camel-aws2-s3-kafka-source-connector.html

arjun180 avatar Apr 25 '22 19:04 arjun180