raft: Failed to make RequestVote

[SF2311]

We are running a deployment of the MySQL-Operator on a bare metal kubernetes cluster. As mentioned in this issue, the orchestrator pod in the deployment is causing some troubles. The log file shows the error raft: Failed to make RequestVote. It was suggested to reopen the issue here, to get help with this. The situation is as follows:

Current Setup

We are running an operator deployment defined by this template in namespace mysql-operator:

helm template presslabs presslabs/mysql-operator -n mysql-operator \
     --version 0.4.0 \
     --include-crds \
     --set antiAffinity=hard \
     --set orchestrator.persistence.storageClass=local-path \
     > cluster01/mysql-operator/mysql-operator.yaml

This works without problems. Now we want to scale the deployment up by using the following template:

helm template presslabs presslabs/mysql-operator -n mysql-operator \
     --version 0.4.0 \
     --include-crds \
     --set antiAffinity=hard \
     --set orchestrator.persistence.storageClass=local-path \
     --set orchestrator.topologyPassword=<REDACTED>\
     --set replicas=3\
     > cluster01/mysql-operator/mysql-operator.yaml

Problem

After applying the new template the operator is replicated as expected. But the pods don't get ready anymore:

$ kubectl get pods
NAME                         READY   STATUS    RESTARTS   AGE
presslabs-mysql-operator-0   2/2     Running   0          7d
presslabs-mysql-operator-1   1/2     Running   0          21m
presslabs-mysql-operator-2   1/2     Running   0          21m

By taking a look at the logs I found that this is a problem in the orchestrator container: Output of $ kubectl logs presslabs-mysql-operator-1 -c orchestrator is attached. log.txt

[voarsh2]

Same here........ Or I see Duplicate term