confluent-kafka-python
confluent-kafka-python copied to clipboard
[Question] Reacting to deletion of a topic being read through a consumer
I encountered a case where while I was reading through a topic, the topic was deleted and then created from the beginning. My consumer using confluent_kafka looks somewhat like this:
from confluent_kafka import Consumer
consumer = Consumer(config)
consumer.assign([TopicPartition(topic, partition, offset)])
while True:
msgs = self._consumer.consume(1000, 0.1)
for msg in msgs:
...
Case happening here:
- The topic is single-partitioned.
- When the topic is deleted, it is recreated within a few seconds and then ingestion starts.
- The consumer does not crash on
consume()and continues to return emptymsgslist.
Currently, I only receive the following line on the STDERR when the topic is deleted. Is it possible from the consumer side to know if such event occured, so that the program could react to this event?
%5|1737553425.511|PARTCNT|rdkafka#consumer-3| [thrd:main]: Topic <topic> partition count changed from 1 to 0