librdkafka Support for ListConsumerGroupOffsets?

I'd like to add a kafka consumer lag input plugin to telegraf, and for that I need a Golang client that supports the ListConsumerGroupOffsets call. But as best I can tell this isn't currently available in librdkafka's AdminClient implementation (and therefore unavailable in confluent-kafka-go).

So this ticket is to track adding it.

Note that I'm a total n00b to C, so I may be just misreading the librdkafka code and this call may already be there...

One design comment, since I just added this method to the kafka-python client:

A common scenario in monitoring situations is to fetch the offsets/calculate consumer lag for many/all consumer groups in the cluster.
ListConsumerGroupOffsets is a very chatty call because it will first lookup the group's coordinator. So it requires two network round trips.
However, when listing consumer groups, if a broker returns a consumer group, you know it's the coordinator. (I don't think this is part of the official Kafka spec, but nevertheless I doubt it will change anytime soon).
If you allow ListConsumerGroups to have an optional param of specifying a single broker and ListConsumerGroupOffsets to have an optional param of specifying a known group coordinator than you can leverage this to nearly halve the number of network calls required to fetch offsets for all consumer groups:
1. Loop through all brokers, calling ListConsumerGroups against each one.
2. For the consumer groups returned, call ListConsumerGroupOffsets, using the group coordinator identified above.

Java doesn't currently support these overrides, but they were useful enough that we chose to include them in kafka-python, even though in general we try to follow the Java clients very closely: https://kafka-python.readthedocs.io/en/master/apidoc/KafkaAdminClient.html#kafka.admin.KafkaAdminClient.list_consumer_group_offsets

Similarly, I added optional support for specifying which partitions you want the offsets for. If omitted, then all known offsets are returned (requires KIP-88 which landed in brokers >= 0.10.2).

You can see how I then leveraged this when patching the Datadog check here: https://github.com/DataDog/integrations-core/pull/2730/files#diff-deed983f73a04522ac621d7f6d0c8403R248

Jan 04 '19 08:01 jeffwidman

If no one else picks this up, I'll take a stab at this...

Jan 05 '19 09:01 jeffwidman

Hey Jeff!

The plan is to add the missing Admin API pieces early this year. As for all-consumer-groups-in-cluster optimizations: that's an interesting problem.. and solution :) Will look into it more and ping you when the time comes.

Jan 09 '19 15:01 edenhill

Hi @edenhill any plans to extend admin client with this API in upcoming releases? Possibility to query consumer offsets would enable new interesting scenarios like automation of scaling, monitoring of lag before maintenance actions like re-partition and etc.

Jan 21 '20 08:01 plachor

Any updates?

Apr 17 '20 07:04 solarhell

Querying consumer offsets is a very useful feature! Please consider implementing without this workaround

Apr 24 '20 13:04 accelerated

Hi @edenhill, can you share what are the plans regarding this feature in terms of work planning? It would be really great to have feature parity with Java client in this area. Currently it is real pain because we need to develop dedicated app/microservice in Java just to be able to read consumer group offsets, and we are not a Java software house.

May 12 '20 14:05 KrzysztofBranicki

We'll be looking at catching up the Admin API in Q3

May 20 '20 08:05 edenhill

Is it still considered for Q3?

Oct 02 '20 10:10 plachor

@edenhill any update on Admin API parity with Java client, especially reading consumer group offsets?

Jan 18 '21 07:01 KrzysztofBranicki

@edenhill can you provide new planned date for releasing this feature? It's quite important for my team.

Jul 19 '21 12:07 marekott

Hi @edenhill any update on this topic, many of us still waiting for this feature. Could you pledge any reliable date when we could expect this to happen?

Seems you already use internally OffsetFetchRequest for many years so most probably it should not be hard for you expose this.

Feb 11 '22 11:02 plachor

+1

We're also waiting for this future. So any update regarding the release date highly appreciated.