kafka-rest
kafka-rest copied to clipboard
confluentinc
How can i get a list of active consumers?
Firstly, thanks for this great product!
I have many instances of REST proxy and i want to monitor how many consumers has each instance. How can i do it?
IMHO zookeeper is probably the eadiest place to get this. There are many zookeeper libraries out there for the most popular languages.
If you perform a GetChildren query on /consumers you'll get the list of consumer groups. A GetChildren in /consumers/[group]/ids will give you the consumer instances.
On Tue, 1 Dec 2015 at 20:54, Ivan Shumkov [email protected] wrote:
— Reply to this email directly or view it on GitHub https://github.com/confluentinc/kafka-rest/issues/131.
Yeah, makes sense to me. Maybe we can add:
GET /consumers GET /consumers/{groupId}/instances
Might also be helpful to have a way to query current offsets:
GET /consumers/{groupId}/instances/{instanceId}/offsets
@hachikuji Do we need to be careful about how we group these too under the resources? The current set of APIs under /consumers only support operations for consumers on the current instances, but that's because they are all mutating operations (e.g. create, delete, and reading from a topic is as well despite being a GET).
In contrast, I might expect GET /consumers to list all consumers in the cluster, i.e. be able to list them similar to the ConsumerGroupCommand.
I'm not sure if mixing both under /consumers is going to be confusing, if we should add another top-level resource, or maybe we can find a completely different home for the API endpoint for a cluster-wide listing of consumer groups.
@ewencp I'm not sure I see any problem with having GET /consumers return the full list of groups in the cluster, but it's definitely trickier if we wanted to implement GET /consumers/{groupId}/instances. A user might expect it to return the full list of instances in the group or just the ones hosted on the proxy receiving the request. Either option would probably be confusing for some.
Now that I'm thinking about it, there seems to be two use cases here. In one case, the user wants to see the full list of instances in the group (e.g. to view the group's assignments and lag). In the other, the user wants to see only the instances hosted on the proxy in order to monitor the proxy itself (this seems to be @Shumkov's use case).
Given these cases, I would probably favor having GET /consumers only return groups which have active instances on the proxy. Similarly for the instances call (if we actually want to support it). Then maybe there could be a separate path (e.g. under /cluster) to query the full list of groups.
@hachikuji Yeah, that's exactly what I was getting at. In retrospect, I probably would have preferred the cluster-wide version to be /consumers and the instances on the proxy to be under something like /proxy/consumers since that would be more consistent with the other top-level resources that are cluster-wide. But I guess that's what I get for lack of foresight...
Was this ever implemented? I assume not as it's not in the docs. I think it would be useful, especially when used with generated consumer names.
@joewood If you're referring to the ability to list all the consumers in the cluster, it hasn't been implemented yet. It'd probably fall under the admin API. There's a new Java AdminClient API that is coming soon in Apache Kafka which would probably be the best way to get this info, although I'm not sure when info about consumer groups will be exposed in it (it is part of KIP-4 which is a large, multi-stage effort).
Hi, folks. I've made pull request PR. It is not return all consumer infos from cluster, but show consumers are restracted by consumer group and topic wich the group reads. Any comments are welcome.
you can check the number of consumers and some information about consumers. like this : kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my-group
Information : TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID my-topic 2 0 0 0 consumer-3-75a0020f-a76f-492a-bb6e-cbd70faa8593 /172.30.1.58 consumer-3 my-topic 3 0 0 0 consumer-4-b7992669-a2b6-4355-ab8d-f869183e6b8e /172.30.1.58 consumer-4 my-topic 1 0 0 0 consumer-2-2292686a-a001-4c09-9224-ee9697b4932d /172.30.1.58 consumer-2
Does anybody know how to use those kafka sh scripts with the remote bootstrap server?
