cookie-cutter icon indicating copy to clipboard operation
cookie-cutter copied to clipboard

Published 20 hours ago • verified publisher icon walmartlabs

Kafka Offsets Are Broken in RPC Mode

Open sklose opened this issue 5 years ago • 2 comments

RPC mode does not guarantee ordering. Currently if RPC mode is used with a Kafka source the committed offsets will jump back and forth. We need to address this by keeping track of offsets better in the Kafka consumer. We may only commit an offset if all previous offsets were processed as all and there are no gaps left.

sklose avatar Jan 27 '20 15:01 sklose

@sklose Should a kafka source be usable when the system is run in a parallelism mode of RPC?

What are the use cases for making RPC mode support kafka offsets?

cross311 avatar Jan 30 '20 16:01 cross311

Generally you'd want to preserve ordering with Kafka, however there is a use case where somebody has long running RPC calls for each incoming Kafka msg and it would be advantageous to be able to parallelize those (order does not matter). Currently that screws up the committed offsets. One might argue though that Azure Queues are a better fit here than Kafka.

sklose avatar Jan 30 '20 19:01 sklose