redpanda
redpanda copied to clipboard
redpanda/kafka: add consumer groups committed offset timestamp metrics
our application must consume all messages, we can't afford to loose one by cleanup retention. Alongside with disk usage metrics, we need to measure the age of the lag to measure, time wise, the lateness we are having on each topic/partition.
Backports Required
- [x] none - not a bug fix
- [ ] none - this is a backport
- [ ] none - issue does not exist in previous branches
- [ ] none - papercut/not impactful enough to backport
- [ ] v23.3.x
- [ ] v23.2.x
- [ ] v23.1.x
Release Notes
Improvements
- add
redpanda_kafka_consumer_group_committed_offset_timestamp_seconds
andredpanda_kafka_consumer_group_committed_offset_age_seconds
public metrics
I also wanted to add a metric that returns the current lag of a consumer group: redpanda_kafka_consumer_group_lag
but I didn't find an easy way to do it. I can only easily get the topic name and the partition number from the model::topic_partition
variable.
but I would need the log-end-offset
to calculate the LAG but I don't know how to get it from groupe_probe.h
@fatpat i marked this as a draft since there is a merge conflict. let us know if this is something that still should be addressed and if you're able to work with ben on addressing his feedback. thanks!