Different per-topic timeout configuration

[swehner]

Description

I'm looking to use a single Kafka producer for two slightly differnt use cases/message types. One that is just fire-and-forget and that will simply enqueue a message without wait for any acknowledgment. For the second use case I'd like to make sure that the message was written to broker before returning a response - similar to what's described in #1174

It seems that the way to achieve this would be to set a low message timeout and wait for poll() to acknowledge message delivery via the callback.

Having a low global message timeout wouldn't be very good for the fire-and-forget use cause, so it seems that the best option would be to use the per-topic configuration. Was there any progress on implementing this since https://github.com/confluentinc/confluent-kafka-python/issues/328#issuecomment-371199811 ? It looks like none of the APIs are exposed through the python client.

If there's still no way to provide per-topic config I'd be open to work on it and make a pull request, but would like to understand what the preferred approach is. maybe exposing the topic creation methods on the produce with a config dict would be enough?

Thanks in advance

Checklist

Please provide the following information:

• [X] confluent-kafka-python==2.20 librdkafka==2.2.0
• [X] Apache Kafka broker version: 3.4.0
• [ ] Client configuration: {...}
• [ ] Operating system:
• [ ] Provide client logs (with 'debug': '..' as necessary)
• [ ] Provide broker log excerpts
• [ ] Critical issue

[swehner]

I've created a pull-request for exposing the topic_new to be able to set a custom configuration here: https://github.com/confluentinc/confluent-kafka-python/pull/1695

Let me know if that approach works for you