[Question] How to handle messages in parrallel for each partitions

[MoJI4yH]

Description

I run 1 consumer per process, but I want to handle messages in parrallel for each partitions, so I can create as much workers as consumer partitions assigned.

Then I start consume loop: pseudocode

while(true) 
{
    var msg = consumer.Consume();
    var worker = findWorkerForMessagePartition();
    worker.handleAsync(message);
    /* 
     so hire do I need to call Pause for worker Partition to maintain the processing order?
     and Resume them after worker handled message
    */
}

Don`t you have any examples of such situation? Or the better way is to run as much proccess as topic partitions we have?

[mhowlett]

It's a little involved - you need to keep track of a lot of stuff. I have actually written an async stream processor abstraction that I think is exactly what you want, but I haven't worked out how to release it yet.

[allantargino]

Hi all, I've created this sample to support the partition per worker pattern: https://github.com/allantargino/dotnet-kafka-parallel-consumer It is just a sample and probably not production ready, but it is a start. If you think it is valuable, we could improve it and merge it in this repository's sample directory.