failed to renew lease kafka/controller-leader-election-helper: timed out

[pmatheson-greenphire]

Describe the bug Kafka Operator restarts every 5 minutes with this error

E0318 15:30:52.925394       1 leaderelection.go:320] error retrieving resource lock kafka/controller-leader-election-helper: Get "https://172.20.0.1:443/api/v1/namespaces/kafka/configmaps/controller-leader-election-helper": context deadline exceeded
I0318 15:30:54.821644       1 leaderelection.go:277] failed to renew lease kafka/controller-leader-election-helper: timed out waiting for the condition

Steps to reproduce the issue: run the operator

Expected behavior don't crash every 5 minutes

Additional context Using chart version '0.4.6'. This didn't always happen it started sometime in the last couple months.

[baluchicken]

Hi @pmatheson-greenphire sorry for the delayed response. Can you please share more information about your setup? What is the kubernetes version you are using? Are you updated the k8s version as well when upgrading the operator?

[pmatheson-greenphire]

k8s version: v1.17.12-eks-7684af

[pmatheson-greenphire]

I think my cluster just got in a wonky state after I create and deleted several hundred topics a few times with low resources allocated to the controller. I recreated everything an bumped the controller resources. Seems to be working without any errors now. Thank you for responding.

[laukik85]

Hi,

I am facing same issue with operator where Kafka Operator keeps restarting after running couple of minutes. I have allocated enough resources on Kafka Operator.

E0202 12:05:55.694251 1 leaderelection.go:367] Failed to update lock: resource name may not be empty I0202 12:05:55.694312 1 leaderelection.go:283] failed to renew lease kb/controller-leader-election-helper: timed out waiting for the condition {"level":"error","ts":"2023-02-02T12:05:55.694Z","logger":"setup","msg":"problem running manager","error":"leader election lost","stacktrace":"main.main\n\t/workspace/main.go:207\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255"}

Attached operator values.yaml for more information.

I am using Kubernetes version 1.24.6.

Thanks, Laukik values.yaml.zip

[panyuenlau]

@laukik85 Thanks for the update on this, reopening this and will look into it soon