Consumers taking longer to connect to broker after updating from 2.8.0 to 2.10.0

[maxmartin-broadsign]

Description

We have a problem, specifically in the context of our automated tests, where it takes longer for our tests to create new unique topics and validate they're up and ready for our tests to be run.

For more info, we're using librdkafka through the cpp wrapper cppkafka (version 0.4.1, this version has remained the same with the different version of librdkafka that we've tested). We've initially upgraded the version of librdkafka to address crashes we had when the connection with the broker was lost (similar/same to this issue)

The tests we run are integration tests where we create topics, unique for each test case, on a kafka broker running in docker. The topics are created using the admin API of librdkafka directly, we then create another consumer (through cppkafka) subscribing to the new topic and wait unit the assignment callback has been called for the consumer. This is the part that has become slower, going from 30-40ms for the whole create-validation to 500-600ms for the same operation (with no code change other than upgrading the lib).

From the broker's log, I can see a delay between the time the topic is created and the time the validation consumer seems to connect to the broker for the first time. This, given we didn't change the broker version, leads us to think it's the connection to the broker that take longer after the upgrade.

We've also been digging into what commit might have introduce this, and identified this commit as the first where the slow down is observed. Though we haven't been able to identify what exactly could be the cause nor whether there could be a workaround.

How to reproduce

• Create consumer belonging to group.id goup_A
• Using the admin API, create a new topic and wait for the event queue to confirm the creation
• Create consumer belonging to group.id group_B
• Set Assignment callback to the consumer so you can verify it's been called
• Subscribe new consumer to newly created topic
• poll until the assignment callback has been called
• Measure execution time

Checklist

Please provide the following information:

• [x] librdkafka version (release number or git tag): v2.10.0>
• [x] Apache Kafka version: 3.6.1
• [ ] librdkafka client configuration: <REPLACE with e.g., message.timeout.ms=123, auto.reset.offset=earliest, ..>
• [x] Operating system: Ubuntu 22.04
• [ ] Provide logs (with debug=.. as necessary) from librdkafka
• [ ] Provide broker log excerpts
• [ ] Critical issue