kafka-lag-exporter icon indicating copy to clipboard operation
kafka-lag-exporter copied to clipboard

Lags not reported for topics with no active member

Open andypp opened this issue 3 years ago • 2 comments

Describe the bug As per #66, consumer groups with no active members also export the lags for all the topics. However in case where a group consumes from multiple topics and it has active members in only some topic, no metrics are reported for the topics without active member.

when group has no active members

$ kafka-consumer-groups --bootstrap-server $BROKERS --describe --group my-group

Consumer group 'my-group' has no active members.

GROUP    TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET LAG CONSUMER-ID  HOST  CLIENT-ID
my-group topic-1 0          100             100            0   -            -     -
my-group topic-2 0          100             100            0   -            -     -
$ curl -s http://localhost:8000 | grep my-group | grep kafka_consumergroup_group_lag
kafka_consumergroup_group_lag{cluster_name="test", group="my-group", topic="topic-1", partition="0", member_host="unknown", consumer_id="unknown", client_id="unknown"} 0.0
kafka_consumergroup_group_lag{cluster_name="test", group="my-group", topic="topic-2", partition="0", member_host="unknown", consumer_id="unknown", client_id="unknown"} 0.0

when group has some active members

$ kafka-consumer-groups --bootstrap-server $BROKERS --describe --group my-group


GROUP    TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET LAG CONSUMER-ID     HOST        CLIENT-ID
my-group topic-1 0          100             100            0   my-consumer-id  /10.11.0.1  rdkafka
my-group topic-2 0          100             100            0                   -           -
$ curl -s http://localhost:8000 | grep my-group | grep kafka_consumergroup_group_lag
kafka_consumergroup_group_lag{cluster_name="test", group="my-group", topic="topic-1", partition="0", member_host="/10.11.0.1 ", consumer_id="my-consumer-id", client_id="rdkafka"} 0.0

To Reproduce application.conf

kafka-lag-exporter {
  port = 8000
  reporters.prometheus.port = 8000
  clusters = [
    {
      name = "test"
      bootstrap-brokers = "localhost:29092"
    }
  ]
}

Environment

  • Version: 0.6.8
  • Version of Apache Kafka cluster: Confluent-kafka-2.12-5.3.2
  • Run with Standalone

andypp avatar Dec 01 '21 06:12 andypp

any idea? i see same problem, metrics are empty

theanotherwise avatar Sep 04 '22 22:09 theanotherwise

Hi, I just got the same issue. I can't help on code, but let me know if there are some tests to do to help analysis

piif avatar Oct 07 '22 09:10 piif