Partitioner does not work with one message instance Producer

[Youlounn]

Description

I'm working with a PostgreSQL plpython3u function for messages production, these messages are sent without key. In my case I'm must have this function able to send only one message at a time and can be called multiple times in a row for one topic, so a Producer is instanced as many times as I have message to send. But in this scenario, all messages are sent to only one partition, even if this one is lagging.
I tried all built-in partitioners according to this doc. Also to set sticky.partitioning.linger.ms = 0, but nothing changed.
See this link

How to reproduce

• Create a TOPIC in single queue consumer with 10 partitions
• Subscribe 3 or 4 consumers to this topic
• Create a PostgreSQL plpython3u function which creates a Producer and send a message to the topic created before without key
• Call this function multiples times in a row

Result : all messages are sent to the same partition.

Checklist

Please provide the following information:

• [ ] confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()): ('1.7.0', 17235968) and ('1.7.0', 17236223)
• [ ] Apache Kafka broker version: 2.4.38 (Debian)
• [ ] Client configuration: {...}
• [ ] Operating system:
• [ ] Provide client logs (with 'debug': '..' as necessary)
• [ ] Provide broker log excerpts
• [ ] Critical issue

[edenhill]

Can you show your code that calls produce()?

Please also reproduce this with 'debug': 'msg,topic,broker' and provide the logs, thanks.

Also note that instantiating a new producer is quite costly and if you expect to be doing this more often than every 10s we recommend reusing a producer instance.

[Youlounn]

I just added 'debug': 'msg,topic,broker' but nothing appears in the logs.

Thank you for this information, but in my case it will be very difficult to manage multiples messages with one producer instance.

Here the code that calls produce: produce_message.txt

[edenhill]

You will need a way to extract the logs. They are emitted to stderr by default.