redis-operator
redis-operator copied to clipboard
OT-CONTAINER-KIT
New master/slave nodes are not joining the cluster
I bought up the redis cluster with 3 masters and 3 slaves initially, everything went fine and cluster is formed.
root@rkota-k8s-1:~/redis-operator/helm-charts# kubectl get pods -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
...
redis-operator redis-master-0 2/2 Running 0 69m 10.244.1.9 rkota-k8s-2
root@rkota-k8s-1:~/redis-operator/helm-charts# kubectl exec -it redis-slave-0 -n redis-operator -- /bin/bash Defaulted container "redis-slave" out of: redis-slave, redis-exporter bash-4.4# redis-cli -a Opstree@1234 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. 127.0.0.1:6379> CLUSTER nodes 5d140c693a4c4bf3ba850348e6375c8815052688 10.244.1.8:6379@16379 slave 5bafacf844a073ddd83a9cff4291980374416964 0 1621930603798 3 connected 6f395a21e04a7e2a70effd0f78590547b2c20cc7 10.244.1.3:6379@16379 myself,master - 0 1621930602000 4 connected 0-5460 5bafacf844a073ddd83a9cff4291980374416964 10.244.1.7:6379@16379 master - 0 1621930603000 3 connected 10923-16383 71985a3577721d4b3c3487d65b778de99c61322d 10.244.1.5:6379@16379 master - 0 1621930603297 2 connected 5461-10922 217b0f6b58e45f0504e4ace91fd25aa73532da40 10.244.1.6:6379@16379 slave 71985a3577721d4b3c3487d65b778de99c61322d 0 1621930602000 2 connected 57b093a18faa5c4241deb7a8bcf1121e59aff3b9 10.244.1.9:6379@16379 slave 6f395a21e04a7e2a70effd0f78590547b2c20cc7 0 1621930602295 4 connected
Now i tried to upgrade the cluster size to 4 as below. root@rkota-k8s-1:~/redis-operator/helm-charts# helm upgrade redis-cluster ./charts/redis-setup/ --set setupMode="cluster" --set cluster.size=4 --set "persistence.size=1Gi" --set "persistence.storageClass=local-storage" --install --namespace redis-operator Release "redis-cluster" has been upgraded. Happy Helming! NAME: redis-cluster LAST DEPLOYED: Tue May 25 08:25:15 2021 NAMESPACE: redis-operator STATUS: deployed REVISION: 2 TEST SUITE: None
But i see that new master and slave has joined the cluster.
root@rkota-k8s-1:~/redis-operator/helm-charts# kubectl get pods -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
...
redis-operator redis-master-0 2/2 Running 0 80m 10.244.1.9 rkota-k8s-2
root@rkota-k8s-1:~# kubectl exec -it redis-master-3 -n redis-operator -- /bin/bash Defaulted container "redis-master" out of: redis-master, redis-exporter bash-4.4# redis-cli -a Opstree@1234 -c Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. 127.0.0.1:6379> CLUSTER nodes beb625801bdf1d27e2a004edbe36c1f90392dd75 10.244.1.12:6379@16379 myself,master - 0 0 0 connected
We can also confirm this. The operator tries to create a new cluster when scaled instead of adding the new node to the existing cluster. Error log:
1.6608151132622335e+09 ERROR controller_redis Could not execute command {"Request.RedisManager.Namespace": "redis-cluster", "Request.RedisManager.Name": "test2-redis", "Command": ["redis-cli", "--cluster", "create", "172.25.55.11:6379", "172.25.49.11:6379", "172.25.54.7:6379", "172.25.53.12:6379", "--cluster-yes"], "Output": "[ERR] Node 172.25.55.11:6379 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.\n", "Error": "", "error": "command terminated with exit code 1"}
One can add the nodes manually (including resharding) and the operator is satisfied when done.
Overall we are also interested in this feature:
- increase cluster size
- add new nodes to the cluster
- leader and its follower
- reshard the cluster
- distributing keys equally between the leader nodes
