Uniquely named queues

[farhadnowzari]

Hi,

I have a proposal for supporting multiple queue workers.

At the moment queueing jobs in Coravel will put everything in one queue. Now the issue is that it can happen which we need to queue different jobs in their own queues so they should not delay each other but still should run in a sequence. I'm not sure if this is already possible or not.

Could be cool if we could do it like this:

services.AddQueueWorkers(options => {
    options.AddQueue("unique_name");
    options.AddQueue("unique_name");
})

And on queuing we can say:

_queue.QueueInvocableWithPayload<,>(payload, /* optional name */ queueName);

[jamesmh]

Yes, this is a feature that's been planned for a while - just haven't had the time yet lol

#139

[farhadnowzari]

Hi @jamesmh Well I can add this feature if it is ok with you. I may have some time on weekends for these kind of stuff.

I promise to do it cleanly ^^

[jamesmh]

Sure, I'll have to take a look at the code and give you an outline of how it should be done. There are a few different ways I have in mind so I'll do a bit of thinking and then write-up some thoughts here at that point :)

[farhadnowzari]

@jamesmh Hi James, What's up? Any news on this ? ☺️

[jamesmh]

Haven't had a chance to look at this yet. Will update once I do 👍

[jamesmh]

So let's try this as a first stab at having multiple queues. There are a few ways that we could modify the existing Queue... but they all have issues.

So, what if we introduced a new IQueues with a usage like:

public Controller(IQueues queueWorkers)
{
     this.queueWorkers = queueWorkers;
}

public ActionResult Index(IndexModel model)
{
     IQueue queue = this.queueWorkers.Get("Worker1");
     // do stuff with queue like normal.
}

In Program.cs, like you suggested:

services.AddQueues(options => {
    options.AddQueue("unique_name");
    options.AddQueue("unique_name");
})

So, options.AddQueue can literally call an overload of this method to register an entire isolated background service, except we modify the new method to accept a string workerName and make that a public field on the Queue class and pass the worker name to it's constructor too. The existing AddQueue can just have a static readonly field used here (using a value of "$default" like the scheduler uses.

In Program.cs we'd need to add a new method like IQueues with a new class QueueFinder (can think of a better name later)

// Note: Registering multiple instances of `IQueue` in the service container allows injecting a collection
// of that type - and .NET will automatically pass in a collection with ALL registered concrete classes that are bound to
// this interface `IQueue`.
public QueueFinder(IEnumerable<IQueue> queues)
{
     this._queues = queue;
}

public IQueue Get(string queueName)
{
     // maybe some error handling?
     return this._queues.SingleOrDefault(q => q.QueueName == queueName);
}

This is a rough sketch and def. will consider this a prototype to be built first, and then played around with a bit to see how that feels in terms of developer experience. But I think it's in the right direction.

Are you still okay with trying this out @farhadnowzari ?

[jamesmh]

I also add - I like this approach because it opens up the door to have multiple queues configurable via the appsettings.json file too. Each queue is a "full" isolated background service so that all existing queue configurable properties can be changed per queue.

[jamesmh]

Also, you'll probably need to pass in the IQueues or possibly another interface bound to the same concrete class QueueFinder (or whatever its called) to the background service too and register an unregistered queue with thread safety too...

[jamesmh]

Let me know if you have any questions!

[farhadnowzari]

Hi @jamesmh

Thanks for getting back. I also like this approach. I'm still on it. I will make the change and you can check it out. I just saw your message, will start tomorrow morning 😊

[jamesmh]

Hi @jamesmh

Thanks for getting back. I also like this approach. I'm still on it. I will make the change and you can check it out. I just saw your message, will start tomorrow morning 😊

Cool! No rush at all. Just take your time and let me know whenever 👍

[farhadnowzari]

Hi @jamesmh

I implemented a prototype for this feature as we discussed here. Although there are some concerns which we may need to talk about.

1. Like making the Queue delay duration be configurable for each queue
2. Handling the services.AddQueue() and services.AddQueues(). I think we can prevent the user from registering the same queue twice

Here is the link to the fork: https://github.com/farhadnowzari/coravel/tree/master/Src/Coravel

Regards Farhad

[jamesmh]

@farhadnowzari Could you create a pull request using your local branch? That way I'll get immediate line-of-sight into what your changes were, etc. and we can continue the conversation there. Plus I can highlight lines of code etc. 👍

[bCamba]

This would be so useful