confluent-kafka-python
confluent-kafka-python copied to clipboard
confluentinc
Prioritizing poll between topics in single consumer
Description
Hi , General guidance / question we couldn't find good source online:
we have a consumer who is working on 2 topics, internal_events and external_events. We want to prioritize the external_events as they are important for our state maintenance, however we can't find any way to prioritize between 2 topics, to make sure that while there are messages in the external_events topic, we try to get X messages first from it and if not enough msgs get also from "internal_events" topic.
The bucket assign approach make sense if its within the same topic, but if we separate between topics, is there a way to achieve this within a single consumer?
How to reproduce
consumer.subscribe(topics) consumer.poll(1) - it gets a msg from either topics... so if "internal_events" is flooded with events, the important external_events are not being processed.
You should use two separate threads for this, with unique consumers that you can pause/resume rather than one subscribed to both topics.
@YechielFino wanted to follow up to see if this issue should remain open. Did the previous suggestion help or are you still looking for guidance on the single consumer case?
