[WIP] Multi-topics consumer and partitioned consumer support

[RobertIndie]

This PR adds support for the multi-topics consumer and partitioned consumer.

Co-authored-by: Zike Yang [email protected] Co-authored-by: Zixuan Liu [email protected]

[RagingKore]

Hi, I was going through the PR so what's the status of this?

[RagingKore]

Quick questions:

• why change MessageId class to IMessageId?
• why do we need Nito.AsyncEx?

[RagingKore]

My suggestion would be to simply had a Topic property to the MessageId and then add a new method to the consumer public static Task<MessageIds[]> GetTopicLatestMessageIds(Topic topic)

This gives us much more visibility, and reduces complexity. and points of failure.

[RagingKore]

On the AutoUpdatePartitions feature.

IMHO this behaviour should be native and always automatically be handled internally while raising an event to allow the user to react to the rebalance by finishing some pending processing of messages and acknowledging their ids.

Just like Kafka clients already allow.

Plus 1 minute interval to check for partition changes seems a bit to long for any type of stream processing. A default of 5 seconds would be a more advisable value.