redpanda icon indicating copy to clipboard operation
redpanda copied to clipboard

redpanda/kafka: add consumer groups committed offset timestamp metrics

Open fatpat opened this issue 1 year ago • 2 comments

our application must consume all messages, we can't afford to loose one by cleanup retention. Alongside with disk usage metrics, we need to measure the age of the lag to measure, time wise, the lateness we are having on each topic/partition.

Backports Required

  • [x] none - not a bug fix
  • [ ] none - this is a backport
  • [ ] none - issue does not exist in previous branches
  • [ ] none - papercut/not impactful enough to backport
  • [ ] v23.3.x
  • [ ] v23.2.x
  • [ ] v23.1.x

Release Notes

Improvements

  • add redpanda_kafka_consumer_group_committed_offset_timestamp_seconds and redpanda_kafka_consumer_group_committed_offset_age_seconds public metrics

fatpat avatar Feb 08 '24 17:02 fatpat

CLA assistant check
All committers have signed the CLA.

CLAassistant avatar Feb 08 '24 17:02 CLAassistant

I also wanted to add a metric that returns the current lag of a consumer group: redpanda_kafka_consumer_group_lag but I didn't find an easy way to do it. I can only easily get the topic name and the partition number from the model::topic_partition variable.

but I would need the log-end-offset to calculate the LAG but I don't know how to get it from groupe_probe.h

fatpat avatar Feb 08 '24 18:02 fatpat

@fatpat i marked this as a draft since there is a merge conflict. let us know if this is something that still should be addressed and if you're able to work with ben on addressing his feedback. thanks!

dotnwat avatar Jul 19 '24 17:07 dotnwat