Add cooperative rebalance support

[thynson]

[thynson]

incremetalAssign and incrementalUnassign are already exposed to Node, and anyone can call them in a rebalance callback when partition.assignment.strategy is cooperative-sticky. Does the KafkaConsumer need to be also modified to provide a default rebalance_cb for cooperative-sticky?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[johncsnyder]

I would love to see this feature added. 🙏

As far as I can tell, the cooperative-sticky partition assignment strategy can't work without adding the bindings for incremental assign/unassign.

I've tried but always received the error application *assign() call failed: Changes to the current assignment must be made using incremental_assign() or incremental_unassign() when rebalance protocol type is COOPERATIVE.

@thynson are there any plans to re-open this PR?

[thynson]

As far as I can tell, the cooperative-sticky partition assignment strategy can't work without adding the bindings for incremental assign/unassign.

They were implemented in this PR, https://github.com/Blizzard/node-rdkafka/pull/907/commits/21a222635127a04904a6f9d97c09edefde30dd87#diff-e574f6617d5a1bd9fb35fab6b02e906b120024e0dcae87b010eed716e23820d4R45, but you need to set partition.assignment.strategy to cooperative-sticky. Or if you're using custom rebalance_cb, call incrementalAssign and incrementalUnassign.

[johncsnyder]

@thynson Yeah, I saw, looks great.

Any chance this will be merged into master? I'd prefer not to maintain my own fork.

[thynson]

Any chance this will be merged into master? I'd prefer not to maintain my own fork.

If @iradul has time to take a look.

[johncsnyder]

Sweet, thanks 🙏

[iradul]

@thynson, thank you for the PR

[thynson]

I'll try to implement the missing parts, but I'm still busy for some days.

[micheleangioni]

Any hope this will get implemented? @thynson :) It would be extremely useful for us

[neuralspin]

@micheleangioni Did you find a way to solve this? We also really need it @thynson )

[spalax]

and we need it a lot :)

[serj026]

We are eagerly awaiting this feature 🙏

[notgosu]

we need it also :)

[votar1408]

It can save the world, great the feature! @micheleangioni

[yaroslav-fedyshyn-playson]

Thanks for the PR. It will significantly reduce our struggling and help us!

[thynson]

I did not have any experience of writing an e2e test for rdkafka. So any help would be great, or I have to take some time to finish the e2e test.