kafka-go icon indicating copy to clipboard operation
kafka-go copied to clipboard

Intelligent health checks

Open KasonBraley opened this issue 1 year ago • 0 comments

Describe the solution you would like

I am looking to see if it is currently possible to implement this health check idea https://blog.cloudflare.com/intelligent-automatic-restarts-for-unhealthy-kafka-consumers/ with kakfa-go.

Some initial struggles I am having with implementing this:

  • Detecting a rebalancing event. Based on this open issue, https://github.com/segmentio/kafka-go/issues/1023, it seems it may be possible, but requires creating a new GroupRebalancer just to be notified of the event.
  • Listing the partitions assigned to a Reader that is within a Consumer Group. This closed issue https://github.com/segmentio/kafka-go/issues/717 mentions it is not available, but I think the newer ConsumerGroup type possibly makes this possible?
  • Retrieving the offsets for the Consumer Group. I believe I can make use of the methods mentioned here https://github.com/segmentio/kafka-go/issues/862#issuecomment-1106721990 once I figure out how to retrieve the current partitions assigned to the Consumer Group.

If these are all currently possible, I would greatly appreciate some advice and tips on how to implement this. As these lower level API's are a little more involved than just using the Reader.

KasonBraley avatar Aug 15 '23 19:08 KasonBraley