kafka-go
kafka-go copied to clipboard
Intelligent health checks
Describe the solution you would like
I am looking to see if it is currently possible to implement this health check idea https://blog.cloudflare.com/intelligent-automatic-restarts-for-unhealthy-kafka-consumers/ with kakfa-go.
Some initial struggles I am having with implementing this:
- Detecting a rebalancing event. Based on this open issue, https://github.com/segmentio/kafka-go/issues/1023, it seems it may be possible, but requires creating a new GroupRebalancer just to be notified of the event.
- Listing the partitions assigned to a Reader that is within a Consumer Group. This closed issue https://github.com/segmentio/kafka-go/issues/717 mentions it is not available, but I think the newer ConsumerGroup type possibly makes this possible?
- Retrieving the offsets for the Consumer Group. I believe I can make use of the methods mentioned here https://github.com/segmentio/kafka-go/issues/862#issuecomment-1106721990 once I figure out how to retrieve the current partitions assigned to the Consumer Group.
If these are all currently possible, I would greatly appreciate some advice and tips on how to implement this. As these lower level API's are a little more involved than just using the Reader.