Documentation is unclear about interaction between queues and multiple servers

[Brondahl]

See this StackOverflow question: https://stackoverflow.com/questions/42201809/hangfire-recurring-job-on-every-server/42202844

We know that if you have multiple servers, and they all schedule the same RecurringJob, then they will use locks to ensure that only one of the servers runs the Job at a time.

Naturally, sometimes we want to change that behaviour - run the job on EVERY server, EVERY time.

The accepted answer on the SO question asserts that if you schedule the same a RecurringJob on multiple servers, but on a different queue on each server, then every server will run the job.

But the documentation around Queues doesn't say anything about how the queue system behaves if the same Job is scheduled on different queues on different servers.

Please can the behaviour be A) confirmed, and B) documented.

[Brondahl]

I'm happy to submit a PR for the doc update, if you can confirm the behaviour :D

[odinserj]

Behavior of recurring jobs is very similar to the behavior of .NET's System.Threading.Timer that's based on thread pool. When it's time to run a recurring job, a new background job is enqueued to the specified queue. Recurring job scheduler doesn't wait until that background job is completed by default (but it's possible to emulate this behavior by using a job filter), and if actual execution takes longer than the specified interval between the recurring schedule, new background job will be enqueued again when it's time to do so.

it's not possible to run something on every server in Hangfire, because it works with jobs, not servers. Such a logic should be implemented manually to determine what servers are active and create as many jobs as required. "Server" term in Hangfire is used only to implement fault detection and show what entities are running.

[Brondahl]

So, if I tell Hangfire enqueue the same job to 3 different queues simultaneously, will it be executed 3 times? or just once? i.e. does "different Queue" mean "different Job", or is it still the same job, listed 3 times?

[burningice2866]

Unsure what you mean when you say "the same job". There is never multiple jobs of the same job - each job is always unique and will run independently of each other.

If you enque a method or a piece of code 5 times, you'll have 5 jobs.

You'll be using queues to control what servers run what jobs. So by enqueing a method 3 times, each into a different queue, you can make sure that it will run on every server. But technically it's 3 unique jobs with no relation to each other.

[zlotnleo]

@burningice2866 "the same job" refers to the same recurring job. And the question is whether it's possible to schedule a recurring job to run on multiple queues at the same time, or if there can only be one queue associated with a recurring job.

From some trial and error, I suspect each recurring job can only be scheduled to run on one queue, and the solution to the problem of having to run the same job on multiple queues is to create a copy of the recurring job with a different ID, but we'd still like to confirm that this is the case

[burningice2866]

@zlotnleo for recurring jobs, what happens, technically, when the the scheduled time has been reached, an actual job is created as if you had called BackgroundJob.Enqueue with no reference to the originating recurring job that triggered it, and that job is fetched by a server and executed.

You would therefor need to have three recurring jobs, all pointing to the same c# code but added with each their own queue.

RecurringJob.AddOrUpdate(() => HandleCommand(), Cron.Daily, null, "server1");
RecurringJob.AddOrUpdate(() => HandleCommand(), Cron.Daily, null, "server2");
RecurringJob.AddOrUpdate(() => HandleCommand(), Cron.Daily, null, "server3");