All metrics not showing up

[mahakg10]

I have kafka running on a server and i have the kafka_exporter running as a container. I am only getting 7 metrics in prometheus as well as server:9308/metrics. Im getting: kafka_brokers kafka_topic_partitions kafka_topic_partition_in_sync_replica kafka_topic_partition_leader kafka_topic_partition_leader_is_preferred kafka_topic_partition_replicas kafka_topic_partition_under_replicated_partition

I need:

kafka_topic_partition_current_offset kafka_topic_partition_oldest_offset kafka_consumergroup_current_offset kafka_consumergroup_lag

While running the container im getting these errors:

> ERRO[0716] Cannot get current offset of topic __consumer_offsets partition 41: dial tcp [::1]:9092: getsockopt: connection refused  source="kafka_exporter.go:274"
> ERRO[0716] Cannot get oldest offset of topic __consumer_offsets partition 41: dial tcp [::1]:9092: getsockopt: connection refused  source="kafka_exporter.go:286"
> ERRO[0716] Cannot get current offset of topic __consumer_offsets partition 42: dial tcp [::1]:9092: getsockopt: connection refused  source="kafka_exporter.go:274"
> ERRO[0716] Cannot get oldest offset of topic __consumer_offsets partition 42: dial tcp [::1]:9092: getsockopt: connection refused  source="kafka_exporter.go:286"

docker command:

docker run --add-host=boltallinone:<servername> -ti --rm -p 9308:9308 danielqsj/kafka-exporter --kafka.server=boltallinone:9092

> INFO[0000] Starting kafka_exporter (version=1.2.0, branch=HEAD, revision=d8a8460163f4fe9316044f4581503f30a739936f)  source="kafka_exporter.go:500"
> INFO[0000] Build context (go=go1.9, user=travis@travis-job-660f82aa-84c1-49d4-9dae-ac7ce251ddac, date=20191224-01:55:25)  source="kafka_exporter.go:501"
> INFO[0000] Done Init Clients                             source="kafka_exporter.go:187"
> INFO[0000] Listening on :9308                            source="kafka_exporter.go:624"
> INFO[0010] Refreshing client metadata                    source="kafka_exporter.go:233"
> ERRO[0010] Cannot get current offset of topic syslog partition 0: dial tcp [::1]:9092: getsockopt: connection refused  source="kafka_exporter.go:274"
> ERRO[0010] Cannot get current offset of topic __consumer_offsets partition 0: dial tcp [::1]:9092: getsockopt: connection refused  source="kafka_exporter.go:274"
> ERRO[0010] Cannot get oldest offset of topic syslog partition 0: dial tcp [::1]:9092: getsockopt: connection refused  source="kafka_exporter.go:286"
> ERRO[0010] Cannot get oldest offset of topic __consumer_offsets partition 0: dial tcp [::1]:9092: getsockopt: connection refused  source="kafka_exporter.go:286"
> ERRO[0010] Cannot get current offset of topic __consumer_offsets partition 1: dial tcp [::1]:9092: getsockopt: connection refused  source="kafka_exporter.go:274"

How to fix this?

[jhyang-v]

I have similiar issues.

Im getting: kafka_brokers kafka_topic_*

I need: kafka_consumergroup_*

Did you solve the issue? If then, could know me how to fix?

[puthrichard]

Anyone with and update on this? We recently have this issue for a few lag* metrics.

Cheers

[fernandocosta2020]

Hi! This has to do with the KAFKA_ADVERTISED_LISTENERS property. If you set it to localhost, kafka_exporter will try to look on localhost.. but is not your localhost! Set it to a known outside hostname. Hope it helps!

[puthrichard]

@fernandocosta2020; I don't think that would be it since our Kafkas are not deployed in containers. They are their own servers. Any more ideas? Thank you

[fernandocosta2020]

Hi @puthrichard , it is not related with being in containers. You must define in kafka how hosts can reach you.

Here is my kafka environment variables: You have to define a listener from the container(PLAINTEXT) and outside (PLAINTEXT_HOST) KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT

On your advertised listeners you have to describe how host from the outside and inside can communicate with kafka. My exporter accesses kafka via kafka:29092 (kafka here is a hostname that must be defined in you environment). KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092 KAFKA_LISTENERS: PLAINTEXT://:29092, PLAINTEXT_HOST://:9092

Hope I made my self clear, it is a bit trickey and not easy to explain.

Best Regards!

[puthrichard]

I'll have a look at that! It's just weird because nothing changed in the configurations of the exporter and neither on the Kafkas. The metrics simply stopped showing all of a sudden!

Thank you

[puthrichard]

So after verifying this, the listeners are defined and everything seems to be in order! any other things I could be missing?

[fernandocosta2020]

You say nothing changed and it suddendly stopped working.. that's strange. For me it as a fresh setup. Nothing changed on your hostnames? Can you post your kafka listeners and exporter configuration?

[nandha4083]

I had a similar issue. The problem is "KAFKA_ADVERTISED_HOST_NAME" in Kafka and "--kafka.server" in Kafka-exporter are different. So I set both to the same value and it started working

KAFKA_ADVERTISED_HOST_NAME: "kafka1.kakfa-ns.svc.cluster.local" --kafka.server: "kafka1.kakfa-ns.svc.cluster.local"

[pflong]

I think you never have a consumer group that consums data successfully. Once consumed, these indicators are available~

[niutendo]

I think you never have a consumer group that consums data successfully. Once consumed, these indicators are available~

i used kafka-console-producer.sh and kafka-console-consumer.sh, but consumergroup metrics still nothing...