kafka_exporter
kafka_exporter copied to clipboard
danielqsj
Missing ConsumerGroup metrics
I am using: kafka_exporter: 1.1.0 Kafka Version: 0.10.1.0
./kafka_exporter --kafka.server=127.0.0.1:9092 --kafka.version=0.10.1.0 --log.level=debug INFO[0000] Starting kafka_exporter (version=1.1.0, branch=HEAD, revision=84cee7e0672f0161c05a93557cc2794a48c8a024) source="kafka_exporter.go:463" INFO[0000] Build context (go=go1.10.1, user=root@1df3c8748bcd, date=20180407-12:31:56) source="kafka_exporter.go:464" INFO[0000] Done Init Clients source="kafka_exporter.go:212" INFO[0000] Listening on :9308 source="kafka_exporter.go:488"
I also tried --log.enable-sarama but I didn't get anything helpful:
[root@ip-10-89-1-90 kafka_exporter-1.1.0.linux-amd64]# ./kafka_exporter --kafka.server=127.0.0.1:9092 --kafka.version=0.10.1.0 --log.enable-sarama INFO[0000] Starting kafka_exporter (version=1.1.0, branch=HEAD, revision=84cee7e0672f0161c05a93557cc2794a48c8a024) source="kafka_exporter.go:463" INFO[0000] Build context (go=go1.10.1, user=root@1df3c8748bcd, date=20180407-12:31:56) source="kafka_exporter.go:464" [sarama] 2018/06/06 01:56:45 Initializing new client [sarama] 2018/06/06 01:56:45 client/metadata fetching metadata for all topics from broker 127.0.0.1:9092 [sarama] 2018/06/06 01:56:45 Connected to broker at 127.0.0.1:9092 (unregistered) [sarama] 2018/06/06 01:56:45 client/brokers registered new broker #0 at 127.0.0.1:9092 [sarama] 2018/06/06 01:56:45 Successfully initialized new client INFO[0000] Done Init Clients source="kafka_exporter.go:212" INFO[0000] Listening on :9308 source="kafka_exporter.go:488" [sarama] 2018/06/06 01:56:47 client/metadata fetching metadata for all topics from broker 127.0.0.1:9092 [sarama] 2018/06/06 01:56:47 Connected to broker at 127.0.0.1:9092 (registered as #0) [sarama] 2018/06/06 01:56:47 Closed connection to broker 127.0.0.1:9092 [sarama] 2018/06/06 01:56:54 client/metadata fetching metadata for all topics from broker 127.0.0.1:9092 [sarama] 2018/06/06 01:56:54 Connected to broker at 127.0.0.1:9092 (registered as #0) [sarama] 2018/06/06 01:56:54 Closed connection to broker 127.0.0.1:9092
Metrics output:
HELP go_gc_duration_seconds A summary of the GC invocation durations.
TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 3.7732e-05 go_gc_duration_seconds{quantile="0.25"} 3.7732e-05 go_gc_duration_seconds{quantile="0.5"} 4.5654e-05 go_gc_duration_seconds{quantile="0.75"} 0.000156033 go_gc_duration_seconds{quantile="1"} 0.000156033 go_gc_duration_seconds_sum 0.000239419 go_gc_duration_seconds_count 3
HELP go_goroutines Number of goroutines that currently exist.
TYPE go_goroutines gauge
go_goroutines 10
HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 568448
HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 7.617624e+06
HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.443617e+06
HELP go_memstats_frees_total Total number of frees.
TYPE go_memstats_frees_total counter
go_memstats_frees_total 16058
HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 405504
HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 568448
HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 4.431872e+06
HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 1.400832e+06
HELP go_memstats_heap_objects Number of allocated objects.
TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 5217
HELP go_memstats_heap_released_bytes_total Total number of heap bytes released to OS.
TYPE go_memstats_heap_released_bytes_total counter
go_memstats_heap_released_bytes_total 0
HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 5.832704e+06
HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.5282498239396126e+09
HELP go_memstats_lookups_total Total number of pointer lookups.
TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 80
HELP go_memstats_mallocs_total Total number of mallocs.
TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 21275
HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 3472
HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 24472
HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 49152
HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 4.194304e+06
HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 778711
HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 458752
HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 458752
HELP go_memstats_sys_bytes Number of bytes obtained by system. Sum of all system allocations.
TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 8.984824e+06
HELP kafka_brokers Number of Brokers in the Kafka Cluster.
TYPE kafka_brokers gauge
kafka_brokers 1
HELP kafka_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which kafka_exporter was built.
TYPE kafka_exporter_build_info gauge
kafka_exporter_build_info{branch="HEAD",goversion="go1.10.1",revision="84cee7e0672f0161c05a93557cc2794a48c8a024",version="1.1.0"} 1
HELP kafka_topic_partition_current_offset Current Offset of a Broker at Topic/Partition
TYPE kafka_topic_partition_current_offset gauge
kafka_topic_partition_current_offset{partition="0",topic="venkata1"} 6 kafka_topic_partition_current_offset{partition="1",topic="venkata1"} 6 kafka_topic_partition_current_offset{partition="2",topic="venkata1"} 6
HELP kafka_topic_partition_in_sync_replica Number of In-Sync Replicas for this Topic/Partition
TYPE kafka_topic_partition_in_sync_replica gauge
kafka_topic_partition_in_sync_replica{partition="0",topic="venkata1"} 1 kafka_topic_partition_in_sync_replica{partition="1",topic="venkata1"} 1 kafka_topic_partition_in_sync_replica{partition="2",topic="venkata1"} 1
HELP kafka_topic_partition_leader Leader Broker ID of this Topic/Partition
TYPE kafka_topic_partition_leader gauge
kafka_topic_partition_leader{partition="0",topic="venkata1"} 0 kafka_topic_partition_leader{partition="1",topic="venkata1"} 0 kafka_topic_partition_leader{partition="2",topic="venkata1"} 0
HELP kafka_topic_partition_leader_is_preferred 1 if Topic/Partition is using the Preferred Broker
TYPE kafka_topic_partition_leader_is_preferred gauge
kafka_topic_partition_leader_is_preferred{partition="0",topic="venkata1"} 1 kafka_topic_partition_leader_is_preferred{partition="1",topic="venkata1"} 1 kafka_topic_partition_leader_is_preferred{partition="2",topic="venkata1"} 1
HELP kafka_topic_partition_oldest_offset Oldest Offset of a Broker at Topic/Partition
TYPE kafka_topic_partition_oldest_offset gauge
kafka_topic_partition_oldest_offset{partition="0",topic="venkata1"} 0 kafka_topic_partition_oldest_offset{partition="1",topic="venkata1"} 0 kafka_topic_partition_oldest_offset{partition="2",topic="venkata1"} 0
HELP kafka_topic_partition_replicas Number of Replicas for this Topic/Partition
TYPE kafka_topic_partition_replicas gauge
kafka_topic_partition_replicas{partition="0",topic="venkata1"} 1 kafka_topic_partition_replicas{partition="1",topic="venkata1"} 1 kafka_topic_partition_replicas{partition="2",topic="venkata1"} 1
HELP kafka_topic_partition_under_replicated_partition 1 if Topic/Partition is under Replicated
TYPE kafka_topic_partition_under_replicated_partition gauge
kafka_topic_partition_under_replicated_partition{partition="0",topic="venkata1"} 0 kafka_topic_partition_under_replicated_partition{partition="1",topic="venkata1"} 0 kafka_topic_partition_under_replicated_partition{partition="2",topic="venkata1"} 0
HELP kafka_topic_partitions Number of partitions for this Topic
TYPE kafka_topic_partitions gauge
kafka_topic_partitions{topic="venkata1"} 3
HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0.02
HELP process_max_fds Maximum number of open file descriptors.
TYPE process_max_fds gauge
process_max_fds 1024
HELP process_open_fds Number of open file descriptors.
TYPE process_open_fds gauge
process_open_fds 8
HELP process_resident_memory_bytes Resident memory size in bytes.
TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 8.835072e+06
HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
TYPE process_start_time_seconds gauge
process_start_time_seconds 1.52824969805e+09
HELP process_virtual_memory_bytes Virtual memory size in bytes.
TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.7281024e+07
I cannot find the following metrics: kafka_consumergroup_current_offset kafka_consumergroup_lag
@danielqsj Any ideas on what I might be doing incorrectly? Or is this a known bug?
I'm seeing this same issue, I have the latest version of kafka_exporter, 1.1, deployed on 4 clusters. On 3 clusters it is working perfectly fine, I'm getting the consumer group metrics, but on the 4th I am not, which is quite strange.
The cluster is using the simpleAclAuthorizer on port 9093, but the ACL permissions I've added are the same on all clusters. (Might be worth adding this to the README), but it seems like to get a SSL cluster working you need to add
--topic=* --operation describe and --cluster --operation describe
There are no DENIED's in the kafka-authorizer logs.
The cluster is healthy with no under replicated or unavailable partitions. And the logs aren't yielding anything useful.
I'll keep an eye on it and keep doing some digging, thought it was worth mentioning that I'm seeing it too though.
Thanks for the great tool, I appreciate your work on this.
I'm embarrassed to say that our exporter has since started reporting consumer lag metrics on this cluster, the problem was that no consumers had recently committed offsets, so there were no metrics to report.
Having the same problem, but cluster is new without any data, so I hope when data is in, we will see some group metrics even though I would prefer to see 0 value versus no metrics at all :)
Having the same problem. No ConsumerGroup metric at all although there is data in there...any ideas welcome!
@deelthor have you tried consuming from a topic? Based on previous comments, i believe you need to have at least one committed offset from a consumer. Using the client to read out a topic should achieve this.
I'm running into this issue as well. Many consumer groups are not showing up, even though they are actively committing offsets. Or rather, for me, some are showing up and others aren't. I have a suspicion that the exporter is not seeing consumer groups that assign themselves partitions manually, rather than using the subscription API.
@dead10ck I don't know enough about the internals to explain why but I suspect it's because you are not on the supported version: https://github.com/danielqsj/kafka_exporter/blob/master/README.md#compatibility
Had similar issues with missing kafka_consumergroup_lag with ACL enabled and exporter user not in super.users.
The root cause is that this is not enough:
resource_type = "Topic"
acl_operation = "Describe"
You need also allow describe on Group
resource_type = "Group"
acl_operation = "Describe"
我也是遇到同样的问题，找不到consumer*相关的指标，exporter的日志一直是这样的内容
[sarama] 2023/06/16 06:14:22 SASL authentication successful with broker kafka:9092:4 - [0 0 0 0] [sarama] 2023/06/16 06:14:22 Connected to broker at kafka:9092 (registered as #1) [sarama] 2023/06/16 06:14:22 Closed connection to broker kafka:9092 [sarama] 2023/06/16 06:14:22 Closed connection to broker kafka:9092 [sarama] 2023/06/16 06:14:22 Closed connection to broker kafka:9092 [sarama] 2023/06/16 06:14:28 client/metadata fetching metadata for all topics from broker kafka:9092 [sarama] 2023/06/16 06:14:51 Completed pre-auth SASL handshake. Available mechanisms: [PLAIN SCRAM-SHA-512 SCRAM-SHA-256] [sarama] 2023/06/16 06:14:51 SASL authentication successful with broker kafka:4 - [0 0 0 0]
