Topic Whitelist not working

[idristarwala]

I have been trying to deploy the lag exporter in our prod env.. but I see that regardless of what I put in the topic whitelist the lag exporter tries to pull the data of all topics.

kafka-lag-exporter { port = 8000 poll-interval = "60 seconds" lookup-table-size = "60" client-group-id = "test" kafka-client-timeout = "60 seconds" clusters = [ { name = "test" bootstrap-brokers = "kafka-broker:9092" topic-whitelist = [ "thor-test-topic1" ] consumer-properties = { allow.auto.create.topics = false } admin-client-properties = { } labels = { } } ] watchers = { strimzi = "false" } metric-whitelist = [ ".*" ] }

akka { loggers = ["akka.event.slf4j.Slf4jLogger"] loglevel = "DEBUG" logging-filter = "akka.event.slf4j.Slf4jLoggingFilter" }

any help would be great if I am not configuring this correctly

[idristarwala]

using the latest version of the lag exporter 0.6.5

[idristarwala]

I do see this in the log :- 2020-11-05 17:25:28,957 INFO c.l.k.KafkaClusterManager$ akka://kafka-lag-exporter/user - Cluster Added: Cluster name: thor-hvfc-prod-inbound Cluster Kafka bootstrap brokers: kafka-** Consumer group whitelist: [.*] Topic whitelist: [thor-test-topic1] Topic blacklist: []

[idristarwala]

@seglo ^

[idristarwala]

can anyone pls help out with this issue?

[seglo]

@idristarwala Thanks for reporting. This is interesting because we have a test that asserts topic whitelists work as expected.

https://github.com/lightbend/kafka-lag-exporter/blob/v0.6.5/src/test/scala/com/lightbend/kafkalagexporter/KafkaClientSpec.scala#L134

Can you verify the symptom of the problem you're experiencing? Are you noticing that the endpoint includes metrics for topics that do not belong in the whitelist? If you're aggregating data in prometheus server is it possible you're looking at old metrics before the whitelist config was added?

[idristarwala]

@seglo , no that is not the case.. lag-exporter is exporting metrics from a new job which is different from the one we were using earlier.. I have validated this accross multiple clusters and neither whitelist nor black list works.. We have instead applied prometheus rules to forward only specific topics..

[iSerganov]

+1 Suffering the same issue.

[DanMelman]

+1 Experienced the same issue on version 0.6.7

[josiasnonato]

+1 with same problem.

[cobolbaby]

@seglo

+1 with same problem 0.7.1

application.conf:

kafka-lag-exporter {
  sinks = ["PrometheusEndpointSink"]
  
  reporters {
    prometheus {
      port = 8000
    }
  }

  poll-interval = "30 seconds"

  lookup-table-size = 120

  client-group-id = "kafkaLagExporter"

  clusters = [
    {
      name = "kafka-ipt-f6"
      bootstrap-brokers = "xxx:19092,xxx:19092,xxx:19092"
      topic-whitelist = [
        "f6-pca-fis-result"
      ]
      consumer-properties = {
        client.id = "kafkaLagExporter"
      }
      admin-client-properties = {
        client.id = "kafkaLagExporter"
      }
      labels = {
      }
    }
  ]
}

KafkaClient.scala:

https://github.com/seglo/kafka-lag-exporter/blob/74c4952118cfcc25cfc6bc88ceb9fbaad020a923/src/main/scala/com/lightbend/kafkalagexporter/KafkaClient.scala#L262-L276

It can be found that noMemberGroupsPartitionsInfo is attached to the indicator information, which is not filtered by conditions.