Redis maximum number of connections

[joshio1]

Hi,

I have more of a question than a bug or a feature request. We use Coverband in our Rails application which is deployed on Heroku. We use Puma as our web server and Heroku Data for Redis which is needed for Coverband.

We have 4 Heroku web dynos, 2 Puma workers and 5 threads in each of those Puma workers. Does this mean Coverband theoretically can have up to (4 * 2 * 5) 40 connections with Redis?

If yes, is there a way we can limit this? If no, how many connections can Coverband theoretically have with Redis?

For reference, I checked CLIENT LIST once on Redis when we had this issue of Redis Err: Max number of clients and Coverband had 5 connections open with Redis (which still seems high to me)

[danmayer]

yeah, I would need to dig in to be 100%, but I believe at the moment we could make a connection per process so if you have 2 puma per machine and 4 machines I would expect a max usage of 8.

Why we don't always hold them is we do periodic reporting to redis keeping data in process memory and updating from time to time... So connections come and go...

I should probably review if a redis connection pool should be used or if we should do anything else to mitigate exploring redis clients... You can edit the reporting time and the jitter to also help with this... if you aren't actively checking all the time it is fine to move reporting to 5-10m and connections will be used and closed.

[joshio1]

Thanks @danmayer! Please let me know what you find.

I'll try editing the reporting time and jitter to see how much it helps reduce the number of clients. Thanks!

[danmayer]

so the current code should end up with one connection per process... it would be pretty easy to add a connection pool but since we aren't having shared and multiple workers report to redis but a single reporting process for a collection of threaded workers it shouldn't really matter.

This connection pool would be easy enough to support https://github.com/mperham/connection_pool.

I think the other option that might help is if I closed the connection when not in use, as it would save some connections for any process that isn't seeing any new coverage. It would mean with a large reporting period and reporting wiggle the connections would be unlikely to be open at the same time. Let me know if you think that would help or is worth implementing @joshio1

[danmayer]

closing this as it is more about how you configure your redis web server or background job server we can just use whatever redis connection is provided.