Unable to connect to redis cluster from outside

[revathyr13]

Facing issue mentioned in https://github.com/OT-CONTAINER-KIT/redis-operator/issues/88

I have exposed the redis leader service as Loadbalancer. But still it is redirecting to pod IP.

172.3x.xx.xxx :6379> get key8 -> Redirected to slot [13004] located at 10.x.x.x:6379 Could not connect to Redis at 10.233.80.113:6379: Connection timed out

Is there any way to access this cluster from outside the cluster?

[davidcollom]

This looks like the redirection is handled (from what I can see and understand) to be from Redis itself, meaning its impossible for the operator to handle this traffic requirement/pattern and that there's an assumption that the Redis instances deployed/managed MUST all reside within the same Kubernetes cluster.

[zengzhengrong]

i have the same with python code

from rediscluster import RedisCluster


def main():
    startup_nodes = [{"host": "172.19.212.13", "port": "31999"}]
    conn = RedisCluster(startup_nodes=startup_nodes, decode_responses=True,password="password")
    # conn = RedisCluster(host="172.19.212.13", port=30946, decode_responses=True,password="password")
    result = conn.get('name')
    print(result)
    result2 = conn.get('name2')
    print(result2)



if __name__ == "__main__":
    main()

Traceback (most recent call last):
  File "C:\Users\Administrator\yaml\olm\redis-operator\test\test.py", line 16, in <module>
    main()
  File "C:\Users\Administrator\yaml\olm\redis-operator\test\test.py", line 6, in main
    conn = RedisCluster(startup_nodes=startup_nodes, decode_responses=True,password="password")
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\rediscluster\client.py", line 368, in __init__
    pool = connection_pool_cls(
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\rediscluster\connection.py", line 160, in __init__
    self.nodes.initialize()
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\rediscluster\nodemanager.py", line 288, in initialize
    need_full_slots_coverage = self.cluster_require_full_coverage(nodes_cache)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\rediscluster\nodemanager.py", line 378, in cluster_require_full_coverage
    return any(node_require_full_coverage(node) for node in nodes.values())
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\rediscluster\nodemanager.py", line 378, in <genexpr>
    return any(node_require_full_coverage(node) for node in nodes.values())
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\rediscluster\nodemanager.py", line 375, in node_require_full_coverage
    raise RedisClusterException("ERROR sending 'config get cluster-require-full-coverage' command to redis server: {0}".format(node))
rediscluster.exceptions.RedisClusterException: ERROR sending 'config get cluster-require-full-coverage' command to redis server: {'host': '10.42.0.68', 'port': 6379, 'name': '10.42.0.68:6379', 'server_type': 'master'}

/mnt/c/Users/Administrator/yaml/olm # kubectl get svc -n redis
NAME                     TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)          AGE
redis-cluster-leaders    LoadBalancer   10.43.150.215   172.19.212.13   6379:31999/TCP   5s
redis-cluster-follower   LoadBalancer   10.43.54.73     172.19.212.13   6378:32006/TCP   5s

pod

redis-operator-78c6c95b78-2kgft      1/1     Running   0          4h48m   10.42.0.64   desktop-ta1r1u1   <none>           <none>
redis-cluster-leader-0               1/1     Running   0          4h42m   10.42.0.68   desktop-ta1r1u1   <none>           <none>
redis-cluster-follower-0             1/1     Running   0          4h42m   10.42.0.69   desktop-ta1r1u1   <none>           <none>
redis-cluster-leader-1               1/1     Running   0          4h38m   10.42.0.72   desktop-ta1r1u1   <none>           <none>
redis-cluster-follower-1             1/1     Running   0          4h38m   10.42.0.73   desktop-ta1r1u1   <none>           <none>
redis-cluster-follower-2             1/1     Running   0          4h38m   10.42.0.76   desktop-ta1r1u1   <none>           <none>
redis-cluster-leader-2               1/1     Running   0          4h38m   10.42.0.77   desktop-ta1r1u1   <none>           <none>
svclb-redis-cluster-leaders-22n48    1/1     Running   0          3h14m   10.42.0.84   desktop-ta1r1u1   <none>           <none>
svclb-redis-cluster-follower-w5gqh   1/1     Running   0          3h14m   10.42.0.85   desktop-ta1r1u1   <none>           <none>

the code wolud connect pod ip but it can not access

[zmrdltl]

In example/external_service, cluster-svc cannot properly enforce redirection policy of redis cluster. When the redis server stores data, the data value is entered as an argument to the hash_function to determine the hash slot to be stored. At this time, if there is no corresponding hash slot in the leader redis instance that sent the save request, the redis api returns the ip and port address that should be stored correctly with a moved error. The k8s service object does not have the logic to save data after making a re-request to the correct redis instance with the returned address. Therefore, it seems that the cluster-related example yaml file should be deleted.

Separately, redirection test with redis-proxy (predixy, RedisLab) Has confirmed.

[lvyanru8200]

to use envoy or not

[rdxmb]

this is possible with redis announce ips (using the loadBalancerIPs or the hostnames), but this is kind of tricky and not implemented in this operator, so initContainers or stuff like that is needed.

We are switching to a simple statefulSet leaving the operator ;)

[drivebyer]

We could use hostnetwork as another lightly solution.

We could add hostnetwork && port to crd.