librdkafka
librdkafka copied to clipboard
create new topic-partition list comparison method based on hashing
the new method is much faster than the old, O(Na + Nb) compared to the older one, which is O(Na * Nb)
Background: The old one was causing problems with partition counts ~3600 in the cooperative-sticky partition assignment. The group leader ended up being kicked out of the group because he spent too much time calculating the partition assignment, and didn't sent any heartbeat inbetween.
reducing the complexity of this pushes the runtime down. Concrete example with 3600 partitions and 1800 consumers before: ~240s after: ~3s
fixes problem in https://github.com/confluentinc/librdkafka/issues/4629
I have tried to stick to the styles, but ran into problems with getting the right version of clang-format (10 doesn't seem to be available via homebrew)
I have have also run the unit tests, they pass.
I am currently running 0113-cooperative-rebalance.cpp test case.
I'm not sure if I should be completely replacing the old comparison function, but happy to do so based on comments
:tada: All Contributor License Agreements have been signed. Ready to merge.
:white_check_mark: lucas-sonnabend
Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.
so, I'm trying to run the 0113 test, but it keeps timing out locally, both with the changes and on main. I'm also running a kafka cluster via docker-compose for it rather than the recommended way