aredis icon indicating copy to clipboard operation
aredis copied to clipboard

StrictRedisCluster client hangs with AWS elasticache - connection pool remains uninitialized

Open adazzi93933 opened this issue 3 years ago • 1 comments

Checklist

  • Python version 3.7
  • Using hiredis or just Python parser no
  • Using uvloop or just asyncio event loop asyncio
  • Does issue exists against the master branch of aredis? yes git clone [email protected]:NoneGG/aredis.git cd aredis/ python3 setup.py install

Steps to reproduce

using the latest aredis v1.1.8, tried python 3.7,3.8,3.9 and downgraded aredis to lower versions as well ... same problem.

set up load cluster on mac

export REDIS_CLUSTER_IP=0.0.0.0 docker run -e "IP=0.0.0.0" -p 7000-7005:7000-7005 grokzen/redis-cluster:latest ran the script below and i can connect to it (no ssl, auth, etc) and i can set/get a key initializing the pool seems optional

output ...

local redis cluster
False
dict_keys([])
init connection pool
True
nodes:
127.0.0.1:7001
127.0.0.1:7004
127.0.0.1:7000
127.0.0.1:7003
127.0.0.1:7002
127.0.0.1:7005
True
True
b'bar' <== worked
True. <== pool got initialized whether i ask for it explicitly or not

###set-up a working cluster on aws elastic cache My production go service running on istio/k8s works fine and can connect to aws elastic cache! My production python admin service that used to work and connect to bitnami redis on kubernetes is now unable to work with aws elastic cache??? so i wrote this little script to debug ...

#!/usr/bin/python
# -*- coding: utf-8 -*-

import asyncio
from aredis import StrictRedisCluster
#from rediscluster import RedisCluster

REDIS_MASTER_URL = "clustercfg.mycluster.fmeera.use1.cache.amazonaws.com"
REDIS_PORT = "6379"

def find_node_ip_based_on_port(cluster_client):
    print("nodes:")
    for node_name, node_data in cluster_client.connection_pool.nodes.nodes.items():
        print(node_name)

async def example():
    cl = StrictRedisCluster(host="0.0.0.0", port=7000)
    print("local redis cluster")
    print(cl.connection_pool.initialized)
    print(cl.connection_pool.nodes.slots.keys())
    print("init connection pool")
    await cl.connection_pool.initialize()
    print(cl.connection_pool.initialized)
    #print(cl.connection_pool.nodes.slots.keys())
    find_node_ip_based_on_port(cl)
    #print(await cl.cluster_slots())
    print(cl.connection_pool.initialized)
    await cl.set('foo', 'bar')
    print(cl.connection_pool.initialized)
    result = await cl.get('foo')
    print(result)
    print(cl.connection_pool.initialized)
    #
    print("-----------------------")
    redis = StrictRedisCluster(host=REDIS_MASTER_URL, port=REDIS_PORT, ssl=True, password="some passwd", skip_full_coverage_check=True, decode_responses=True)
    print("aws elastic cache")
    print(redis.connection_pool.initialized)
    #print(redis.connection_pool.nodes.slots.keys())
    print("init connection pool")
    await redis.connection_pool.initialize()
    print(redis.connection_pool.initialized)
    find_node_ip_based_on_port(redis)
    print(await redis.cluster_slots())
    await redis.set('foo', 'bar')
    result = await redis.get('foo')
    print(result)

loop = asyncio.get_event_loop()
loop.run_until_complete(example())

Expected behavior

output is ...


aws elastic cache
False
init connection pool
False
nodes:

it hangs on 
 await redis.connection_pool.initialize()
or 
await redis.cluster_slots()

dumping the connection pool show no configured nodes, slots?

expected behavior is to get a map of nodes and slots.

in this case i am using TLS, and encryption.


connecting to aws elastic cache works using the cli connecting to aws elastic cache works using the blocking client -> rediscluster.RedisCluster (redis-py-cluster)

Actual behavior

local redis cluster
False
dict_keys([])
init connection pool
True
nodes:
127.0.0.1:7001
127.0.0.1:7004
127.0.0.1:7000
127.0.0.1:7003
127.0.0.1:7002
127.0.0.1:7005
True
True
b'bar'
True
-----------------------
aws elastic cache
False
init connection pool
False
nodes:
hangs ...

adazzi93933 avatar Apr 23 '22 01:04 adazzi93933

update: i created an aws elasticcache cluster without "Encryption in transit" enabled and auth and it is working now. i will narrow down the problem further. but it seems to be related to encryption. also i had to remove ssl=True to make it work (i was getting an error).

adazzi93933 avatar Apr 23 '22 16:04 adazzi93933