NodePort allocate failed since conflict occur when multiple Kafka instances with external NodePort services expose

[walk1ng]

Hi teams, When I use Kafka operator to install multiple Kafka instances in the same k8s cluster and all Kafka with external NodePort type services exposed, the nodePorts maybe conflicted assignment here. e.g.

• kafka-1 with 3 brokers and external expose with 3 nodePorts: 30902, 30903, 30904
• kafka-2 with 3 brokers and external expose with 3 nodePorts the same: 30902, 30903, 30904 that's the conflict assignment.

Check the plan status of instance, you can get the error similar as "Invalid value: 30902: provided port is already allocated".

I know there is a parameter "EXTERNAL_NODE_PORT" to specify the port, but it's a starting value and operator will count the other ports according to brokers count, it also have the chance to get the used ports then install Kafka failed.

Cloud you please help. Thanks!

[walk1ng]

Version information:

KUDO Version version.Info{GitVersion:"0.13.0", GitCommit:"900ca042", BuildDate:"2020-05-14T18:23:50Z", GoVersion:"go1.14.2", Compiler:"gc", Platform:"darwin/amd64"}

Operator version kafka-1.3.0
zookeeper-0.3.0

App version kafka 2.5.0 zookeeper 3.4.14

[zmalik]

@walk1ng we don't recommend using nodeports and this shouldn't be the first choice to run KUDO Kafka in production https://github.com/kudobuilder/operators/blob/master/repository/kafka/docs/latest/external-access.md#nodeports

but as you said you can specify the EXTERNAL_NODE_PORT and reserve ranges for two different instances.

[walk1ng]

https://github.com/kudobuilder/operators/blob/master/repository/kafka/docs/latest/external-access.md#nodeports

Hi zmalik, thanks for your quick response. Does any way to assign the nodePorts by k8s rather than specify the starting nodePort by myself? In our system, we consider to install/deploy the Kafka instances automatically and on-demand, so it's difficulty to figure out the starting value and sequential ports are available.

[zmalik]

@walk1ng right now we haven't implemented that. Because that would mean that for clients to connect to a newly created Kafka cluster they would need to know the dynamic nodeport that just got assigned to the services. And that would be different for each Kafka broker. We just didn't have a use-case for users asking for dynamic nodeports and how they would use them.