kafka-connect-field-and-time-partitioner
kafka-connect-field-and-time-partitioner copied to clipboard
canelmas
Kafka Connect Store Partitioner by custom fields and time
Hi. I have some questions to use your library. The type of kafka messages I'm getting is json string, (not Avro) and the format of the json is nested json....
I build your project and get connect-fieldandtime-partitioner-1.1.0-SNAPSHOT.jar . Then I copy that in /usr/share/confluent-hub-components/confluentinc-kafka-connect-s3/lib/. When I run connector with these configs: curl -X POST -H "Content-Type: application/json" --data '{ "name":...
I want to partitioning my data with key and hold just latest 10 minutes or 10 data for each key. Can I use your partitioner class?
Works fine. But how can we define our own log level ? Seems not possible for now. I would like to set WARN and don't display INFO messages
If the value of the custom partitioner field is an Long the casting to String throws the exception. "java.lang.ClassCastException: class java.lang.Long cannot be cast to class java.lang.String" This code should...
"org.apache.kafka.common.config.ConfigException: Invalid value -1 for configuration partition.duration.ms: Partition duration needs to be a positive. at io.confluent.connect.storage.partitioner.TimeBasedPartitioner.configure(TimeBasedPartitioner.java:117) at io.confluent.connect.s3.S3SinkTask.newPartitioner(S3SinkTask.java:207) at io.confluent.connect.s3.S3SinkTask.start(S3SinkTask.java:117) at org.apache.kafka.connect.runtime.WorkerSinkTask.initializeAndStart(WorkerSinkTask.java:309) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:186) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:237) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)...
Trying to get this plugin working (thanks very much by the way :) ) my s3-sink connector works when I comment out the lines, when I comment them back in...
https://github.com/canelmas/kafka-connect-field-and-time-partitioner/blob/a27263ad403caf3148b70da6d6d6b97213489c3c/src/main/java/com/canelmas/kafka/connect/FieldAndTimeBasedPartitioner.java#L50 It seems that it parses a `String` but the default value is a boolean, and it cannot cast it. ``` Task threw an uncaught and unrecoverable exception. Task is...
when we include data valued in the partitioner, wouldn't this cause the partitions to roll over more frequently because of the different data we receive? Is there any way to...
I deploy the code and I got the following error. Any helps? `org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception. at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:559) at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:315) at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:218) at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:186) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:170) at...
