pykafka icon indicating copy to clipboard operation
pykafka copied to clipboard

Support librdkafka in the ManagedBalancedConsumer

Open waytai opened this issue 7 years ago • 3 comments

Traceback (most recent call last):                                                                          
  File "pyconsume.py", line 30, in <module>                                                                 
    use_rdkafka = True                                                                                      
  File "/usr/local/lib/python2.7/dist-packages/pykafka/topic.py", line 214, in get_balanced_consumer        
    return cls(self, self._cluster, consumer_group, **kwargs)                                               
  File "/usr/local/lib/python2.7/dist-packages/pykafka/managedbalancedconsumer.py", line 188, in __init__   
    self.__class__.__name__))                                                                               
ImportError: use_rdkafka is not available for ManagedBalancedConsumer                                       
Exception AttributeError: "'ManagedBalancedConsumer' object has no attribute '_running'" in <bound method Ma
nagedBalancedConsumer.__del__ of <pykafka.managedbalancedconsumer.ManagedBalancedConsumer at 0x7f5a65a31b10 
(consumer_group=KAFKA_01)>> ignored

waytai avatar Mar 26 '18 03:03 waytai

At the time the ManagedBalancedConsumer was written, librdkafka didn't support Kafka-managed consumer groups, so this ImportError was added to make that clear. We should reexamine where librdkafka's support is these days, and fix this if possible. This will probably happen during Parse.ly's internal upgrade to latest Kafka later this year.

emmettbutler avatar Mar 26 '18 16:03 emmettbutler

than you for your kindly reply

waytai avatar Mar 27 '18 02:03 waytai

seems like https://github.com/edenhill/librdkafka/issues/601#issuecomment-208354621 indicates some of how this works, and the issue overall explains that librdkafka supports this now, if I'm reading it correctly

razor-1 avatar Aug 16 '18 23:08 razor-1