arena icon indicating copy to clipboard operation
arena copied to clipboard

discovery of queue ?

Open maxired opened this issue 5 years ago • 8 comments

Hey,

from what I understand so far, it looks it need to configure all the queue that I want to monitor. It that right ?

The application I am working on is creating queues on the fly, so I was wondering whether I could dynamically monitor all queues, starting with specific prefix ?

maxired avatar Oct 24 '18 16:10 maxired

Could you expand a bit more on your use case here? Definitely interested! We use an auto-discovery method ourselves internally at the moment, but it requires auto-restart of arena. Would love to make it more reactive!

ttacon avatar Oct 28 '18 18:10 ttacon

We create queues on demand for specific complex jobs. This means we can't add them all to config index.json file.

dwightkelly avatar Nov 20 '18 00:11 dwightkelly

Hey @ttacon sorry, been off for a while ;)

I use bee queue for an ecommerce project I am working on. Basically, computing the price is the job done in the queue.

What we do is creating a new queue for each different orders. This allow us to control were the job is processed and so on

maxired avatar Nov 28 '18 15:11 maxired

Another use case here. Each "queue" is a different pub/sub message type in one of our services and there are many of them. Would be great if arena could just figure out all the queues without having to specify them in the json.

JonDum avatar Dec 09 '18 08:12 JonDum

Same here! We generate queues for each type of job and new job types get added all the time

shortcircuit3 avatar Dec 28 '18 21:12 shortcircuit3

Same here. I create queues on the fly dynamically so I can't account for them all when running Arena. @ttacon is it possible to show queues in arena as they are created?

adamsoffer avatar Jan 13 '19 00:01 adamsoffer

same here. Required to create queue dynamically. Any idea?

DanielChuDC avatar Aug 18 '19 13:08 DanielChuDC

even without having dynamic queue creation its just a pain in the ass to configure and maintain the index.json. If developers add or remove queues to our application we need to manually update the index, redeploy and restart arena.

there is this post from Mixmax: https://engineering.mixmax.com/blog/bull-queue-aws-autodiscovery

in which they too feel this pain and have created a helper script to pull a list of queues from redis on aws.

Why is this functionality is not built into Arena?

kevzettler avatar Feb 07 '20 20:02 kevzettler