An error has occurred during metrics gathering

[kirashet]

I'm using the official docker image of danielqsj/kafka-exporter.

Set it up, works like a charm.

Then we tested some resilience stuff (setup is 1xZookeeper, 2xKafka Brokers). We switched off the brokers by one another and started back on.

The kafka world was fine, just the kafka-exporter has some sniff.

If I call the metrics endpoint, I get

`An error has occurred during metrics gathering:

7 error(s) occurred:

• collected metric kafka_consumergroup_members label:<name:"consumergroup" value:"testconsumer" > gauge:<value:0 > was collected before with the same name and label values`

docker logs shows:

time="2019-07-11T08:51:49Z" level=error msg="Cannot get offset of group axTest: kafka: broker not connected" source="kafka_exporter.go:396" time="2019-07-11T08:51:49Z" level=error msg="Cannot get consumer group: kafka: broker not connected" source="kafka_exporter.go:370" time="2019-07-11T08:51:51Z" level=info msg="Refreshing client metadata" source="kafka_exporter.go:233"

Restarting the exporter didn't do the trick either.

Any ideas?

[kirashet]

Nobody?

[kirashet]

Ok guys, I deleted the consumer group via

kafka-consumer-groups --bootstrap-server localhost:9092 --delete --group testconsumer

and the kafka_exporter doesn't bitch anymore.

Unfortunately I don't know the root cause.

Does someone?

[simonasr]

Having the same issue 😢

[Allen-yan]

Having the same issue , But I cannot delete the group

[kirashet]

Having the same issue , But I cannot delete the group

Why not?

[moogzy]

We've started to observe this on our redpanda cluster.

What we've found is that when broker leadership changes or when pods/nodes restart the exporter can fall into this state. A reboot while interruptive....often resolves it which is less than ideal.

Did anyone make progress on this?