sarama icon indicating copy to clipboard operation
sarama copied to clipboard

feat(consumer): support multiple balance strategies

Open napallday opened this issue 3 years ago • 0 comments

As shown below, Kafka supports specifying multiple rebalance protocol from the very begining. (source)

v0 supported in 0.9.0.0 and greater
JoinGroupRequest => GroupId SessionTimeout MemberId ProtocolType GroupProtocols
  GroupId => string
  SessionTimeout => int32
  MemberId => string
  ProtocolType => string
  GroupProtocols => [ProtocolName ProtocolMetadata]
    ProtocolName => string
    ProtocolMetadata => bytes
 
v1 supported in 0.10.1.0 and greater
JoinGroupRequest => GroupId SessionTimeout MemberId ProtocolType GroupProtocols
  GroupId => string
  SessionTimeout => int32
  RebalanceTimeout => int32
  MemberId => string
  ProtocolType => string
  GroupProtocols => [ProtocolName ProtocolMetadata]
    ProtocolName => string
    ProtocolMetadata => bytes

However, Sarama only supports one group protocol right now, by specifying Consumer.Group.Rebalance.Strategy. This prevents users from enjoying some benefits in Kafka. For example, if users wanna change the default one range to a more advanced one like sticky, when the new consumers with sticky protocol try to join in the consumer group, they will be rejected by the coordinator with error ErrInconsistentGroupProtocol(kafka server: The provider group protocol type is incompatible with the other members) and thus cannot consume any messages. Only when all the instances are upgraded with sticky protocol can they resume consume message again.

In contrast, the client supporting multiple group protocols enables rolling upgrades without downtime -- The upgraded member includes both the new protocol and the protocol, and the coordinator will choose a single protocol which all members support. Once all members have upgraded, the coordinator will choose whichever protocol is listed first in the GroupProtocols array.

In this PR, I introduce a new config Consumer.Group.Rebalance.GroupStrategies and mark the old one Consumer.Group.Rebalance.Strategy deprecated. To make it backward compatible:

  • The dafault value for Consumer.Group.Rebalance.GroupStrategies is []{BalanceStrategyRange}
  • Consumer.Group.Rebalance.Strategy takes precedence of Consumer.Group.Rebalance.GroupStrategies

Thus,

  • if Consumer.Group.Rebalance.Strategy isn't explicitly configured previously, then the new default value range in Consumer.Group.Rebalance.GroupStrategies will be used.
  • if Consumer.Group.Rebalance.Strategy is configured by users previously, Consumer.Group.Rebalance.Strategy is also chosen as the group protocol.

I believe it's a desirable feature in sarama, since other Go Kafka Libraries, like segmentio/kafka-go, already support it.

napallday avatar Sep 15 '22 16:09 napallday