channels_redis
channels_redis copied to clipboard
django
multi process with websocket have more 8,000 time wait
and today I use Redis monitor command, found that so much to execute ZREMRANGEBYSCORE and ZRANGE
I use python3.6+django2.1+channel_redsi to make a chat project, channe_redis's configration are default.
when I use 16 processes, each process have 4 threads, each thread loop for 100 times to push message to frontend，that I found that there are more than 8000 time_wait connection at 127.0.0.1:6379. And i use redis monitor command found that they all are ZREMRANGEBYSCORE and ZRANGE, i think these two command are from group_send when i look deep into the channel_redis code.
anything else should i tell?
But when I change configuration host (localhost, 6379) to unix sock, there are only thirty time_wait connection
I managed to find out what was the root cause here. (Not sure how to fix though).
dango-channels (async_to_sycn() more specifically) does not work nice together with the connection pooling of channels_redis (https://github.com/django/channels_redis/pull/117)
Many callers of group_send() are synchronous. Django-channels/channels_redis document says you should wrap your call in async_to_sync().
Although that seems to work. (functionally that works...) it breaks the connection pooling implemented in channels_redis.
aioredis (1.3.1)/channels_redis implements connection pooling. ( on top of aioredis.create_redis_pool()) but that implementation uses a pool per event loop.
And ..... if you use async_to_sync(), a new event loop is spawed for every invocation of async_to_sync().... So when the group_send() is done... the even loop is destroyed, and so is the connection pool -> closing the socket to Redis.
This leaves you with one closed Redis socket, lingering in TIME_WAIT per invocation of group_send().....
This workaround worked for me:
# Don't use async_to_sync(), as you'll end up with thousands of TIME_WAIT sockets
loop = asyncio.get_event_loop()
task = loop.create_task(self.channel_layer.group_send(
"chat",
{
"type": "chat.message",
"text": text_data,
}
))
loop.run_until_complete(task)
Instead of
# THIS LEAVES TIME_WAIT sockets
async_to_sync(self.channel_layer.group_send)(
"chat",
{
"type": "chat.message",
"text": text_data,
},
)
Versions
python 3.7.13
channels==3.0.4
aioredis==1.3.1
channels-redis==3.4.0
django-redis==5.2.0
