camel-kafka-connector
camel-kafka-connector copied to clipboard
Splitting files through the Kafka connector
I am using an AWS s3 source connector to transfer files from an s3 bucket to a Kafka topic. I'm running into a problem where the size of the messages is too big. And the size is well beyond the buffer size I can setup. After investigation, I realized this was because the messages are an aggregation of multiple messages. Is there way I can split the message within the Kafka connector (through a delimiter) and then send the messages one-by-one to the Kafka topic? Order of messages is not a priority.
I am using this Kafka connector : https://camel.apache.org/camel-kafka-connector/0.11.x/reference/connectors/camel-aws2-s3-kafka-source-connector.html