Ian Cooper

Hi @honkuan86 and @phuaxinyuin I wondered if you had any feedback. Looking to close this issue if it is fixed. Thanks, Ian

If you have 15 pods and 236 partitions, then you have about 16 partitions per pod. If you are struggling to keep up, that is because you can't process those...

The only other option would be for us to enable you to turn on auto-commit instead of using our batch approach. At your throughput it might be better. Your risk...

Another way of looking at this: 236 partitions and 16 threads would require processing each item in 40ms.

PS @honkuan86 if you turn up your logs to Information you should see it flushing the buffer. You may need to pipe your logs somewhere to see that, probably scroll...

I'll see if I can crank my test up fast enough, and with enough partitions to break it.

An option might be to allow you to set auto-commit and trade off the possibility of not processing a message against throughput.

@honkuan86 There may be an interaction with your load and either our synchronization context or our offset management that we need to look at. However, we don't recommend the async...

@phuaxinyuin Did you get any feedback?

So, as the owner of Brighter: https://github.com/BrighterCommand/Brighter which also operates in this space a few thoughts. - Welcome to the party!! - As we already have Brighter, Mass Transit, NServiceBus,...