aiokafka icon indicating copy to clipboard operation
aiokafka copied to clipboard

Refactor Group Coordination

Open tvoinarovskyi opened this issue 5 years ago • 0 comments

The current implementation is a bunch of hacks made to make sure aiokafka works with the same input parameters as Java's client while remaining asynchronous as much as possible. This was made with the attempt to maintain similarity between Java's internals and aiokafka's. But at this point it changed so drastically I think it's fine to let go that similarity and make sure internals follow asyncio's principals, rather than Java's.

Points I want to be fixed here:

  • [] Group coordination should only rejoin the group if we have fetchers, ready to consume that data. This will be similar to what happens with poll() in Java's client but will allow clients to just let fetchers hang around during rebalances and still rejoin. (for example, create a task to fetch and only after subscribing to a group)
  • [] Remove offset_refresh routines. This was a result of several tries to fix problems while maintaining subscription state as the centre of communication between components. At this point let's just let Fetcher do an RPC call on Coordinator. The reference graph would be like: Consumer -> Fetcher Consumer -> Coordinator Consumer -> SubscriptionState Fetcher -> SubscriptionState Coordinator -> SubscriptionState Fetcher -> Coordinator

While still straightforward, but with another link between Fetcher and Coordinator. SubscriptionState should not contain a hard reference on any of the parent objects.

  • [] Authorization errors should also communicate in a better way. Probably we should cancel all fetch requests, thus making Coordination stop (no pending fetches). And if another fetch will be created will we rejoin.

tvoinarovskyi avatar Feb 23 '19 17:02 tvoinarovskyi