django-rq icon indicating copy to clipboard operation
django-rq copied to clipboard

add doc for redis cluster usage if support

Open alwqx opened this issue 6 years ago • 4 comments

Hi, I see rq support redis cluster from https://github.com/rq/rq/pull/741, but django-rq does not provide doc or details for using redis cluster.

I see code from https://github.com/rq/django-rq/blob/v1.0.1/django_rq/queues.py#L65-L107

def get_redis_connection(config, use_strict_redis=False):
    """
    Returns a redis connection from a connection config
    """
   redis_cls = redis.StrictRedis if use_strict_redis else redis.Redis

    if 'URL' in config:
        return redis_cls.from_url(config['URL'], db=config.get('DB'))
    if 'USE_REDIS_CACHE' in config.keys():

        try:
            from django.core.cache import caches
            cache = caches[config['USE_REDIS_CACHE']]
        except ImportError:
            from django.core.cache import get_cache
            cache = get_cache(config['USE_REDIS_CACHE'])

        if hasattr(cache, 'client'):
            # We're using django-redis. The cache's `client` attribute
            # is a pluggable backend that return its Redis connection as
            # its `client`
            try:
                # To get Redis connection on django-redis >= 3.4.0
                # we need to use cache.client.get_client() instead of
                # cache.client.client used in older versions
                try:
                    return cache.client.get_client()
                except AttributeError:
                    return cache.client.client
            except NotImplementedError:
                pass
        else:
            # We're using django-redis-cache
            try:
                return cache._client
            except AttributeError:
                # For django-redis-cache > 0.13.1
                return cache.get_master_client()

    if 'UNIX_SOCKET_PATH' in config:
        return redis_cls(unix_socket_path=config['UNIX_SOCKET_PATH'], db=config['DB'])

    return redis_cls(host=config['HOST'], port=config['PORT'], db=config['DB'], password=config.get('PASSWORD', None))

Django-rq checks URL,USE_REDIS_CACHE to determine which connection to use. So I use below django settings:

RQ_DB = int(os.getenv('REDIS_DB_NAME', 0))
# RQ_PASSWORD = os.getenv('REDIS_PASSWORD', '')
RQ_TIMEOUT = int(os.getenv('REDIS_TIMEOUT', 360))
REDIS_URL = os.getenv('REDIS_URL','localhost:6379')
RQ_QUEUES = {
    'default': {
        'URL': 'redis://{}'.format(REDIS_URL),
        'DB': RQ_DB,
        'DEFAULT_TIMEOUT': RQ_TIMEOUT
    },
    'high': {
        'URL': 'redis://{}'.format(REDIS_URL),
        'DB': RQ_DB,
        'DEFAULT_TIMEOUT': RQ_TIMEOUT
    },
    'low': {
        'URL': 'redis://{}'.format(REDIS_URL),        
        'DB': RQ_DB,
        'DEFAULT_TIMEOUT': RQ_TIMEOUT
    }
}
RQ = {
    'DEFAULT_RESULT_TTL': int(os.getenv('RQ_RESULT_TTL', 5000)),
}

REDIS_URL is 10.24.0.7:14020,10.24.0.7:14021,10.24.0.7:14022,10.24.0.7:14023,10.24.0.7:14024,10.24.0.7:14025 which is cluster url. But django-rq raise exception.

Django-rq seems not support redis cluster? So can django-rq support redis cluster or provide detailed doc if it already support.

alwqx avatar Mar 22 '18 15:03 alwqx

@adolphlwq I would like to use redis cluster. Have you solved this? Thanks.

michaelluk avatar Aug 20 '18 04:08 michaelluk

@michaelluk No, Have some trouble in pr https://github.com/Grokzen/redis-py-cluster/pull/251 I closed it.

alwqx avatar Aug 20 '18 07:08 alwqx

@adolphlwq Ok, thanks. I think I will try to use the redis cluster proxy (https://github.com/eleme/corvus) then.

michaelluk avatar Aug 20 '18 09:08 michaelluk

@adolphlwq Any update here? We want to use it with a cluster, but we think this is still not officially supported, isn't it? Would be great to get any update here.

severindellsperger avatar Feb 25 '21 09:02 severindellsperger