kminion icon indicating copy to clipboard operation
kminion copied to clipboard

Export numerical kafka_topic_info parameters/labels as individual metrics

Open amuraru opened this issue 3 years ago • 1 comments

Currently kminion_kafka_topic_info metric exposes all topic parameters as labels. [1]

I've got an use-case where I'd need the replication_factor and min_insync_replicas values to be used in promql queries to verify if a given topic has under-min replicas, e.g: kafka_topic_partition_replicas < "min_insync_replicas" or a given topic has under-replicated partitions: kafka_topic_partition_replicas < "replication_factor"

To do so, the only solution I found is to have these numerical topic parameters exposed as separate metrics, e.g: kminion_kafka_topic_info_min_insync_replicas kminion_kafka_topic_info_replication_factor

Would this be a good improvement for the kminion @weeco ?

[1] Sample:

kminion_kafka_topic_info{cleanup_policy="delete", container="kafka-minion", endpoint="metrics", 
max_message_bytes="1048588", message_format_version="2.5-IV0", message_timestamp_type="CreateTime", 
min_insync_replicas="2", partition_count="1",  replication_factor="1", retention_bytes="-1", retention_ms="10000", 
segment_bytes="536870912",
topic_name="00faec4f386fc8dda9688d402cc92d0c1082ee0d426909841e6b4d11750deb65"}

amuraru avatar Sep 20 '21 10:09 amuraru

I think Kafka itself has a metric called under min isr. I think introducing a similiar metric would make sense. Literally you could expose via a metric how many under.min.isr.partitions exist on a given topic:

Example:

kminion_kafka_topic_under_min_isr_partitions{topic_name="xy"} 4

WDYT?

weeco avatar Sep 20 '21 11:09 weeco