Dramatiq uses more RabbitMQ connections than necessary

[WillDaSilva]

We use RabbitMQ as our Dramatiq broker, and we're always dangerously close to the connection limit (which is 500 in our case). This is because Dramatiq creates a connection in each worker process for each queue we have, and we have a queue per Dramatiq task.

We think Dramatiq can be modified to use fewer connections. According to CloudAMQP's RabbitMQ best practices, you should generally limit connections to one connection per process, and use channels within that connection for per-thread use. Instead of doing this, Dramatiq creates a new connection with a single channel for each RabbitMQ consumer it creates, and it creates a consumer for each queue per process, when it could probably be made to just use one connection which each consumer within a process shares, and then each consumer can just create a channel for itself.

What OS are you using?

Debian 12 (via the official Python Docker image)

What version of Dramatiq are you using?

v1.17.0

[spumer]

You can use https://github.com/spumer/dramatiq-kombu-broker/ It's implement connection pooling and channel pooling

And soon we are publish video and article about this solution: https://piterpy.com/talks/d5516075e54d41c68d7d3be0c191b09b/

[spumer]

Also, you can try to use this: https://github.com/cloudamqp/amqproxy

[spumer]

Also remoulade (fork of dramatiq) implement thread-safe broker via amqpstorm, channelpool and separate thread for I/O

https://github.com/wiremind/remoulade/blob/adf142467867a99b10dee7d32de2591ceafe8504/remoulade/brokers/rabbitmq.py#L45