Redis UTF-8 cause hard to debug issue on group_send

[insspb]

All code is reproducible in multichat example app with last django and channels_redis.

It is possible to make settings.py configuration like this:

CHANNEL_LAYERS = {
    "default": {
        # This example app uses the Redis channel layer implementation channels_redis
        "BACKEND": "channels_redis.core.RedisChannelLayer",
        "CONFIG": {
            "hosts": [
                "redis://redis:6379/0?encoding=utf-8"
            ]
        },
    },
}

Let's take code that will create fake group and send message there. Code example for pycharm django console.

from asgiref.sync import async_to_sync
from channels.layers import get_channel_layer

layer = get_channel_layer()
add = async_to_sync(layer.group_add)
send = async_to_sync(layer.group_send)
res1 = add("some", "random")
res2 = send("some", {"message": "message",})

You will get this traceback:

Traceback (most recent call last):
  File "<input>", line 8, in <module>
  File "/home/insspb/.venvs/quick/lib/python3.7/site-packages/asgiref/sync.py", line 139, in __call__
    return call_result.result()
  File "/usr/lib/python3.7/concurrent/futures/_base.py", line 428, in result
    return self.__get_result()
  File "/usr/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/home/insspb/.venvs/quick/lib/python3.7/site-packages/asgiref/sync.py", line 204, in main_wrap
    result = await self.awaitable(*args, **kwargs)
  File "/home/insspb/.venvs/quick/lib/python3.7/site-packages/channels_redis/core.py", line 634, in group_send
    x.decode("utf8") for x in await connection.zrange(key, 0, -1)
  File "/home/insspb/.venvs/quick/lib/python3.7/site-packages/channels_redis/core.py", line 634, in <listcomp>
    x.decode("utf8") for x in await connection.zrange(key, 0, -1)
AttributeError: 'str' object has no attribute 'decode'

And it is very hard to guess that problem in connection string. Because item in redis will be created.

If replace connection code with same, but without encoding mention all will work without any issues.

CHANNEL_LAYERS = {
    "default": {
        # This example app uses the Redis channel layer implementation channels_redis
        "BACKEND": "channels_redis.core.RedisChannelLayer",
        "CONFIG": {
            "hosts": [
                "redis://redis:6379/0"
            ]
        },
    },
}

Here run with this string, without any issue.

import sys; print('Python %s on %s' % (sys.version, sys.platform))
import django; print('Django %s' % django.get_version())
sys.path.extend(['/home/insspb/git/quick', '/snap/pycharm-professional/213/plugins/python/helpers/pycharm', '/snap/pycharm-professional/213/plugins/python/helpers/pydev'])
if 'setup' in dir(django): django.setup()
import django_manage_shell; django_manage_shell.run("/home/insspb/git/quick")
PyDev console: starting.
Python 3.7.9 (default, Aug 18 2020, 06:22:45) 
[GCC 7.5.0] on linux
Django 3.1
from asgiref.sync import async_to_sync
... from channels.layers import get_channel_layer
... 
... layer = get_channel_layer()
... add = async_to_sync(layer.group_add)
... send = async_to_sync(layer.group_send)
... res1 = add("some", "random")
... res2 = send("some", {"message": "message",})
... print("Done without issues")
Done without issues

[sevdog]

The root cause of this is aioredis, since it parses redis URL to extract config options. When encoding is passed it is used to automatically decode any redis output before returning it to the outer scope, while when it is not present it returns bytes objects.

As of now RedisChannelLayer is not aware of connection encoding, thus expecting to always receive bytes from it when a string should be returned. Forcing any encoding may also give errors in other places (like deserialize).

[carltongibson]

Thanks @sevdog — do you think #195 is the way forwards here?

(I'm going to close this, thinking there's not really much we can do.)

[sevdog]

@carltongibson I think thank aredis can help, from its code it does not parse encoding from redis URL

https://github.com/NoneGG/aredis/blob/b46e67163692cd0796763e5c9e17394821d9280c/aredis/pool.py#L33-L37

Also Connection classes there uses an explicit decode_responses parameter (along with encoding).

IMO it should be documented that encoding url parameter is supported or not.

In aioredis there is this decode utility which is ussed by clients/connections.

[carltongibson]

OK, let's reopen at least for documentation. Thanks @sevdog