kafunk icon indicating copy to clipboard operation
kafunk copied to clipboard

Parallelize consumer fetch requests

Open eulerfx opened this issue 8 years ago • 0 comments

Currently, the logic to route requests to brokers is contained in the connection. Fetch, Produce and Offset requests can be addressed to multiple brokers based on the partitions involved. When the consumer sends fetch requests, it does so for the entire set of partitions assigned to it. The connection then routes them to the appropriate brokers in a fork-join fashion. This join however is needless and reduces utilization/parallelism. It is better to have a fetch process per broker independent of the fetch processes of other brokers. Either way, the fetch processes publish messages into an exchange which distributes into per-partition streams.

eulerfx avatar Jan 03 '17 23:01 eulerfx