Yury Bushmelev

May be helpful: https://stackoverflow.com/questions/45560255/commitfailedexception-commit-cannot-be-completed-since-the-group-has-already-reb

There was 27Gb of data. I wiped it out and started kafka-backup again. Let's see..

Hmm.. this time it restated just fine with 84Gb of data 🤔 Maybe issue was on kafka cluster side.. or maybe data in backup directory was not good.. 🤷‍♂️

unfortunately, I dropped old data files already.. so cannot reproduce anymore. I can guess there was something really huge.

I was considering adding jmx_exporter to kafka-backup anyway.. So maybe will do this later. UPD: just found jmx_exporter config file for kafka-connect: https://github.com/zenreach/docker-kafka-connect/blob/master/jmx_exporter.yaml

Hit into this again this weekend. Will try to add JMX exporter this week to see.. Pasting log between retries here just in case you can spot anything: ``` [2020-06-29...

Hmm... I was unable to bring it to working state for 2 days.. so I gave up on this but forgot to disable nightly backup script. This script stops backup,...

After upgrade I noticed small change in the behaviour! There is OOM now :-D ``` [2020-07-03 07:32:34,475] ERROR WorkerSinkTask{id=chrono_prod-backup-sink-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:179) java.lang.OutOfMemoryError: Java heap...

JFYI, 2.5Gb is not enough to success... Not sure I can allocate more though at the moment..

Well.. no OOM this time but still no luck. It seems 5min is not enough for kafka-backup to reset offsets for our topics amount on this VM. Few stats: -...