v1.6.1 - kafka.AssignedPartitions or kafka.RevokedPartitions events not returned with Poll()

[richiejenkins]

Description

The behaviour of "go.application.rebalance.enable": true appears to have changed with v1.6.1, with neither kafka.AssignedPartitions or kafka.RevokedPartitions being returned by Poll().

The only way to capture these events is separately with a rebalanceCb method when calling Subscribe().

Is this an intended change?

How to reproduce

• Create a consumer with "go.application.rebalance.enable": true
• Poll for events
• Observe no kafka.AssignedPartitions or kafka.RevokedPartitions

Checklist

Please provide the following information:

• [X] confluent-kafka-go and librdkafka version (1.6.1):
• [ ] Apache Kafka broker version:
• [X] Client configuration: ConfigMap{...}
• [X] Operating system: macOS 10.15.7
• [ ] Provide client logs (with "debug": ".." as necessary)
• [ ] Provide broker log excerpts
• [ ] Critical issue

[larry-cdn77]

Same problem here, those notification no longer come out of the polling function. They may have broken or even been removed for the incremental rebalancing feature introduced in version 1.6.1. As shown in the incremental balancing example, rebalance callback function is another way of getting assign/revoke notifications, and works for me.

[nsagnett]

Same issue here. some statements has been cleared between two versions for the rebalance process Working code : https://github.com/confluentinc/confluent-kafka-go/blob/8bd1e41d4448e6f6afb8539ad3ce7ada168c2724/kafka/event.go#L154 New code : https://github.com/confluentinc/confluent-kafka-go/blob/f0d7516a9eb10d64e32bd8ea25607ce3930c7bf7/kafka/consumer.go#L838

@edenhill, will this be fixed ?

Many thanks in advance

Regards

[jliunyu]

@nsagnett Thanks for your comment, yes, we are working on the fix for this issue

[jliunyu]

If you don't use channel based consumer, please use the rebalance callback to handle the rebalance, it will be called from the Poll() function.

[chris-montesione-sp]

@jliunyu just to clarify the rebalance will be called from the Poll() function? Would this make the rebalance callback thread safe to use with Poll()? Judging by the blocking call in the event loop I am inclined to say yes but want to confirm.

https://github.com/confluentinc/confluent-kafka-go/blob/master/kafka/event.go#L182

[edenhill]

The rebalance callback will be triggered from within the Poll() call on the same thread, so there application does not need to take any special care for threading/concurrency.

[wiyu]

This change doesn't appear to support the case where an application uses static/manual assignment via the Assign() method. There doesn't appear to be a way to provide a rebalance callback when using this approach. Any thoughts on adding support for this either via adding a call back param to Assign() or another config to allow for the events to be returned in the poll?

[nkostoulas]

This change doesn't appear to support the case where an application uses static/manual assignment via the Assign() method. There doesn't appear to be a way to provide a rebalance callback when using this approach. Any thoughts on adding support for this either via adding a call back param to Assign() or another config to allow for the events to be returned in the poll?

Can't you still use the same callback via Subscribe and always Assign the same partitions?