after operator upgrade , a new envoy LB pod/ loadbalancer is created and the old one is not deleted

[priyavj08]

Describe the bug On upgrading kafka operator (tried from 0.12.3 to 0.15.1) , a new envoy external pod/ loadbalancer service gets created but the old one remains. Ends up with 2 loadbalancer and using up additional IP

Steps to reproduce the issue:

1. rolling upgrade of operator

Expected behavior expect the old envoy pod to get terminated so there is only one envoy LB pod/service

Screenshots If applicable, add screenshots to help explain your problem.

Additional context Add any other context about the problem like release numberm version, branch, etc.

[priyavj08]

see new pod/LB service and the old one also remains

`envoy-744f5f4f99-kkqch 1/1 Running 0 13h envoy-external-kafka-659b58bcfd-zm8jm 1/1 Running 0 11h

envoy-loadbalancer LoadBalancer 10.106.244.4 10.163.x.x 19090:32186/TCP,19091:30528/TCP,19092:32313/TCP 13h envoy-loadbalancer-external-kafka LoadBalancer 10.99.79.70 10.163.x.x 19090:30586/TCP,19091:31146/TCP,19092:32386/TCP,29092:31497/TCP 11h`

[baluchicken]

Hi @priyavj08 sorry for the delayed response. We are aware that the operator in some cases leaves some orphan instances in the cluster. For example if you are switching from headless service to per broker service or switching from envoy ingress controller to istio. Although in case of client facing instances I think this is the proper way to operate since changing the loadbalancer address would impact the client applications as well. By not deleting the old loadbalancer the operator gives a chance to migrate your clients seamlessly to the new one. Once you are done you can safely delete the old unused loadbalancer.

[priyavj08]

thanks @baluchicken
have a question, is it possible to assign the envoy-loadbalancer-external-kafka service with the old external IP (one used by my previous envoy LB) because requesting client to point to new IP may not be an option

thanks

[stoader]

You can set IP on the LoadBalancer through:

apiVersion: kafka.banzaicloud.io/v1beta1
kind: KafkaCluster
...
spec:
  envoyConfig:
    loadBalancerIP: 'your static ip'

Note: This feature depends on whether the underlying cloud-provider supports specifying the loadBalancerIP when a load balancer is created. This field will be ignored if the cloud-provider does not support the feature.

For details see: https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer