Optimizing for used Connections

[pauline-koch]

Hi,

first of all, thanks for Solid Queue, we're currently using it in production and really like its simplicity and Rails-native design!

One challenge we've run into: when running IO-heavy jobs concurrently, it creates big connection spikes.

For example, with:

• 20 Solid Queue threads per worker

• 3 worker processes

On Aurora Serverless, this leads to CPU spikes, scaling churn, and connection storms, even though the jobs are barely touching the database.

We switched from sidekiq, to profit from the security solidqueue offers and from the simplicity.

But this connection storm issue is a problem for us, because we want to be able to perform many IO-heavy jobs like mailers or document generation at once, without causing CPU intensive connection spikes.

With sidekiq, we were able to handle these workloads without issue. Sidekiq Threads share one connection pool: https://github.com/sidekiq/sidekiq/wiki/Scaling-Sidekiq/00d09b1b8a8ae5b4c6c10ccde16bf0552bafa666 and do not need a connection per thread.

Is any optimization in that direction planned / possible?

Thanks and best regards

Pauline

[c0mrade]

Try RDS Proxy (or PgBouncer) to prevent connection storms with many worker threads

[pauline-koch]

Thank you, RDS-Proxy would alliviate our issue, but is unfortunately not compatible with our setup. We have to fullfill special authentication processes that incompatible with the proxy.