kafka-docker
kafka-docker copied to clipboard
wurstmeister
Topic creator is too brittle during new cluster construction
I'm building an automated test suite for some broker maintenance automation involving multiple brokers. As part of this test suite, I'm building a three-broker Kafka cluster, and desire to create a topic with a replication factor of three.
Since the brokers are started by a container scheduler, they're not necessarily all started at the same time. However, the topic creator script is run in the background immediately after a broker is started. If there aren't yet enough brokers in the cluster, the topic creator will fail hard with:
creating topics: test:1:3
Error while executing topic command : Replication factor: 3 larger than available brokers: 2.
What I'd like to happen is for the topic-creator to wait for the broker count to reach the desired replication factor before attempting to create the topic, or to retry until successful.
I'm building an automated test suite for some broker maintenance automation involving multiple brokers.
I would have thought the broker maintenance automation should deal with topic creation / deletion / alteration?
Even so - there is some merit in making this less brittle, but on the other hand, for production use-cases it is likely you would want to manage topics outside of the create topics helper script. There are issues such as not being able to co-ordinate each broker trying to create topics concurrently which creates un-necessary errors.
Related to (very) old PR https://github.com/wurstmeister/kafka-docker/pull/160
Just came here to say I ran into this recently in a single broker setup. Is there a suggested path forward to avoid this error where topic creation happens before broker has completely started up?
