[BUG] Use logstash for consumption, offset submission will not be submitted after a period of time normally, restarting logstash is useless

[zcola]

Describe the bug A clear and concise description of what the bug is. Use logstash for consumption, offset submission will not be submitted after a period of time normally, restarting logstash is useless

To Reproduce Steps to reproduce the behavior: logstash 7.15.2

config file

input {
    kafka {
        bootstrap_servers => "iowa.brokerXXXom:9092"
        group_id => "logstash_zcola_cld_dnode_kubelet_log_2020090401"
        topics => "iowa_cld_dnode_kubelet_log"
        codec => plain
        consumer_threads => 2  # number (optional)， default: 1
    }
}
output {
  stdout {
        #codec => dots
       # codec => rubydebug
    }
}

run command logstash -f /etc/logstash/zcola_cld_dnode_kubelet_log --log.level debug

When I ran it yesterday, I observed that the offset was submitted normally. Today, I saw that CURRENT-OFFSET has not changed. Restarting logstash is invalid. The latest data is consumed by the same group.

pulsar 2.9.1 and pulsar-protocol-handler-kafka-2.9.1.1.nar

We have been using logstash for a few years and have been updating with the official version. There are basically no problems with kafka. The implementation uses the official kafka client.

[zcola]

logstash_run.log

[zcola]

broker.log

[zcola]

dump.log

[Demogorgon314]

What version of kafka admin are you using? Can you provide the command you used? Thanks.

[zcola]

What version of kafka admin are you using? Can you provide the command you used? Thanks.

2.0.0 and 3.1.0 are the same

[BewareMyPower]

After killing 3 of 4 all brokers, the offset commit works. It looks like there is something wrong with the read/write on the system topic public/__kafka/__consumer_offsets.

We need to review the existing code again to look into this problem.

[SunDapeng1]

How is the issue going ？ We have the same problem： logstash 7.10.0 kop：2.9.3.2 pulsar：2.9.3 There is no problem with a new consumer group， but offset submission will not be submitted after a period of time normally. And the same with kafka client this time. what's more, we have this in kop.log, I'm not sure whether it is related

[zcola]

https://github.com/streamnative/kop/pull/1501

fixed