timeplus-io
Reuse same pooled Kafka(Simple)Consumer
Describe what enhancement you'd like to have
librdkafka shall allow reuse an existing Kafka consumer for different topic partition.
For example, at time t1, user queries topic1 by using external stream, we create a Kafka consumer and pool it. at time t2 while t1 query is still running, user queries topic2 by using external stream, for now we will create another Kafka consumer and pool it.
We shall be able to reuse Kafka consumer at t1 for t2 query instead of recreating another Kafka consumer which is actually very expensive and most importantly it is not scale.
Probably same for producer if we create separate producer on the fly for every external stream and if there are hundreds of them.
A consumer is bound to a specific topic parition ( using the legacy simpler consumer API ) or a specific list of paritions (using the new consumer API ). So, I am not quite sure what it exactly means to resue a consumer, or maybe you are talking about the KafkaWALSimpleConsumer ? But I want to understand more about why "recreating another kafka consumer is very expensive". At the end of the day, in order to consume from another topic/partition, another consumer has to be created anyways ( from the pont of view of librdkafka ). Could you elaborate, please?
From librdkafka side, what cachable are rd_kafkt_t and rd_kafka_topic_t ( if we use the new consumer API, we don't even need to cache rd_kafka_topic_t ).
BTW, I think we should refactor the pooling/caching for kafka external stream. Currently it uses KafkaWALPool, which I think (from the code) was originally deisnged for logstore storage. It looks to me that it's not the best idea to mix logstore and external stream together. It also has leaking issues like this. There are some useful concepts/tools existing in the klog namespace that can be reused in external stream, but it does not mean we have to resue everything there to make it more complex than it has to be.
When hundreds of kafka streams are created, tens of thousands of threads will be created, and the number of brokers in the kafka cluster is 160. Is there any good solution?
Hi @zhongyuankai , would you share your use case with a little bit more details? For example, do you have a big Kafka cluster that has lots of topics and plan to create external streams for those topics? If that's your use case, I wonder if it's possible for you to give the timeplus enterprise a try? We have implemented some enhancements on that side to improve the resource consumption efficiency for such case. It will take time to port the improvements over to this repo. So, if for now, it will be nice if you could try it out. And we would love to hear more about your use cases.
Thanks for asking the question, @zhongyuankai , also, I recalled librdkafka spawn one thread per broker. So for 160 cluster, there will be 160 thread at minimum in one instance of librdkafka. 160 threads itself is probably fine. When there are hundreds of Kafka streams gets created, it is still fine since there will be only hundreds of threads if one Kafka topic has only one partition instead of hundreds of thousands. We have customers who has thousands of Kafka topics to consume from in Timeplus Enterprise, which works just fine.
On the other hand, in Timeplus Enterprise we can scale out the Kafka external streams across a cluster which further solves the consuming threads issue.
Thank you very much for your reply. It is still under testing and research. According to my understanding, librdKafka will create 1+ n broker threads for every consumer created by librdKafka. When the kafka cluster is large and there are many kafka consumers created, it will lead to the system. The number of threads is exhausted and the system is stuck. Is there any better solution to this in the enterprise version?