Metadata refresh of freed topic

[davideberra]

Description

Scenario: I have one producer that sets up a persistent connection with kafka brokers. It creates and destroy dinamically rd_kafka_topic_t object tied to a SINGLE rd_kafka_t object that lasts for the entire life of the main process (i have processes up from months on field)

The problem occours when a topic was created, then destroyed. It seems that its metadata are still requested during periodic topic and broker list refresh.

In my case the problem is that i cannot delete a topic (e.g. with Kafka Manager) because brokers have auto.create.topics.enable set to true and metadata requests force topic re-creation

How to reproduce

• setup a broker with auto.create.topics.enable to true
• process creates new rd_kafka_t object - rd_kafka_new(...)
• process creates new rd_kafka_topic_t object tied to the latter rd_kafka_t object- rd_kafka_topic_new(....)
• process push data
• process destroy rd_kafka_topic_t object

process is still up and connection between producer and kafka brokers is still up (rd_kafka_t still valid)

• delete the topic with kafka manager or kafka-topics.sh tool
• observe that topic is successfully deleted
• wait minutes and observe that this line is logged: "Request metadata for 1 topic(s): per iodic topic and broker list refresh"
• observe that topic was recreated

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

• [x] librdkafka-1.3.0
• [x] Apache Kafka version: 2.0.0
• [x] librdkafka client configuration: enable.idempotence=true; retry.backoff.ms=100; max.in.flight.requests.per.connection=1; api.version.request=false; broker.version.fallback=2.1.1; linger.ms=0; batch.num.messages=10000; queue.buffering.max.messages=; message.max.bytes=10000; statistics.interval.ms=60; debug=all;
• [x] Operating system: CentOS 7 x64
• [ ] Provide logs (with debug=.. as necessary) from librdkafka
• [ ] Provide broker log excerpts
• [ ] Critical issue

[edenhill]

Your observations are correct, librdkafka never lets go of a rd_kafka_topic_t object, it will keep a reference count until the rd_kafka_t instance is destroyed. So when your application destroys and re-creates the topic object it will actually get the same object back.

I understand this is a problem in your case, even though auto topic creation is not really recommended anymore (since there is a create_topics Admin API), but making librdkafka destroy the object internally is too big of a change, what we could do is not include the topic in metadata requests if its refcount is 1 (only librdkafka's internal reference count). Can look into it after the next release.