cloudamqp
amqproxy scalability questions
Hello and thanks for creating this awesome proxy!
I'm a CloudAMQP client and am considering using the proxy for a slightly different use case than the initial one for which it was created (which I assume being a local proxy for PHP-like applications), so I'm reaching out to understand if this can be the right tool for the job. I thought reaching out here instead of the CloudAMQP support for better visibility to anyone interested.
In my setup I have one RabbitMQ instance (deployed on CloudAMQP) which is consumed by thousands of separate VMs (sometimes even tens of thousands!). Each VM uses a single connection. In the past, especially when going beyond 100k VMs, we observed long instance freezes due to connection churn when creating or deleting thousands of VMs at a time during infrastructure rollouts. I'm considering putting the amqproxy in front of RabbitMQ to cache connections. The main points I'd like to discuss are:
- Setup: would the proxy deployed as a separate VM to which all other 100k VMs connect be ok?
- Scalability: would the proxy be able to handle as many connections?
- If not, do you have a better connection limit to suggest?
- If not, would it be ok to deploy N amqproxy VMs and let the consumers pick one randomly on start?
- Do you have plans for CloudAMQP provisioning an amqproxy in front of RabbitMQ in a managed way?
Hi,
You are correct that the current project is designed to act as a local proxy. We will (in the future) have another proxy that is targeting the server side. And yes that one will be provisioned in a managed way in front of RabbitMQ.
So right now you would suggest not to use the proxy as a remote component?
@ralbertazzi We encourage you to try it out and let us know how it goes, we will try to be helpful!
@ralbertazzi - In Kubernetes, I run AMQProxy as a Deployment (standalone service), and I put a Kubernetes Service in front of it, and I have all my other applications connect to that service as a remote/non-local component.
It works without issue...And I do realize, again, that this is designed to run as a proxy that's local to the PHP application which uses it (for performance gains), but even in that configuration, it works great.
The VM equivalent might look similar :)
@ralbertazzi We encourage you to try it out and let us know how it goes, we will try to be helpful!
Hey! Currently I am also interested in trying out amqproxy on the server side. I have the following setup:
- NGINX as TCP proxy -> rabbitmq We are managing approximately 30k connections simultaneously and if all connections are established everything runs fine. But we have problems if NGINX or rabbitmq must be restarted as TCP connections are being dropped and every client must reconnect. This happens pretty much at the same time, so 30k clients want to establish a new connection. NGINX as a Load-Balancer handles this fine, but rabbitmq is overwhelmed and more unresponsive than responsive. It takes more than 45 minutes to reconnect all clients and that we are fully up and running again.
My questions now:
- Would we benefit form amqproxy in terms of "rabbitmq would be better working while 30k clients trying to connect"?
- Or do you have other rabbitmq settings that might help us?
THX, Georg
Probably not today. But we could rewrite amqproxy to cram in as many channels as possible per connection, so that multiple different connections used a single upstream connection.
Channel pooling has now been implemented! https://github.com/cloudamqp/amqproxy/commit/98c0bc7bdc2509d62c2191bc3038ce8ddcb0563d
@carlhoerberg - SUPER excited for this. Going to pull it into our latest infra Docker container builds and then benchmark it.
Thanks for updating this thread, I would never have known otherwise!