kue icon indicating copy to clipboard operation
kue copied to clipboard

Err: Max Connections

Open russelltaylor05 opened this issue 8 years ago • 2 comments

I'm running into a max connections error when running on Heroku: ERR max number of clients reached

When I do a client list in Redis I see a ton of connections that look like this:

id=28 addr=23.20.17.18:59295 fd=33 name= age=14 idle=14 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=114 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=blpop
id=29 addr=23.20.17.18:59296 fd=34 name= age=14 idle=14 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=450 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=blpop
id=30 addr=23.20.17.18:59302 fd=35 name= age=14 idle=14 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=477 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=blpop
id=31 addr=23.20.17.18:59298 fd=36 name= age=14 idle=14 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=188 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=blpop
id=32 addr=23.20.17.18:59297 fd=37 name= age=14 idle=14 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=188 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=blpop`
  • How many connections does Kue.js normally need to create?
  • Are the number of connections directly related to how many job types I'm creating/processing?
  • If i have 10 different job types, does that mean I need 10 different connections to listen for each type?

russelltaylor05 avatar Aug 28 '17 16:08 russelltaylor05

I have the same problem. I see a new connection opened for each job type. I think this is not manageable if the application needs many job types, especially if using redis solutions from the cloud (e.g. redis labs has maximum 30 connections for free plan).

A work around may be to implement the redis createClientFactory option to return always one client. However, I am not sure if having a shared client may create problems.

Any help on this?

rewop avatar Oct 31 '18 12:10 rewop

It would be great to document how many redis connections are used. I have an app with multiple workers, and with 6 workers I somehow have 39 connections open to redis. I only have one job type, but ~3k jobs of that type in the queue.

rossvz avatar Oct 31 '18 19:10 rossvz