kafka-retry-job
kafka-retry-job copied to clipboard
Trendyol
Kafka Retry Job is an open-source project for moving Kafka messages from error topics to retry topics, developed by Trendyol with love :orange_heart:
#29, #30 and #44 change the error catching and message consumption limit mechanism to topic-partition-based and apply topic-based parallel programming to optimize program running time
We can limit the message consume limit based on Topics instead of Topic Partitions
We can extract the message consumption section to a async function and then wait all calls outside of the loop with a Task.WhenAll(). This will increase the parallelism and we...
right now, when we get any exception, the project shuts down. We can catch exception based on topic partitions and when we get an exception, we can move to next...
we have all the config properties in root, we have consumer configs and producer configs and they are starting to confuse user. We need to use a nested object config.