delayed_job icon indicating copy to clipboard operation
delayed_job copied to clipboard

Pull from queues matching a pattern

Open glacials opened this issue 5 years ago • 0 comments

We're running delayed_job on a beta environment and a production environment simultaneously. Both access the same database.

When we push a change to beta that e.g. deletes a job type, beta will continue to pull those job types from the queues (put there by production) and try to process them, failing every time, until we promote beta to production. Same in reverse if we add a job type on beta, production starts failing to process jobs.

The natural solution seems to be

# ensure the environment doesn't pull jobs it doesn't know about
export QUEUES=job1,job2,job3

However, manually keeping QUEUES in sync with the various jobs we have seems subject to forgetfulness and poor knowledge transfer when jobs are added or deleted.

Additionally as the number of jobs increases, the environment variable gets more and more unwieldy to manage.

I would love a DRY way to achieve this same result, like matching against the existing queue prefix:

module MyApp
  class Application < Rails::Application
    config.active_job.queue_name_prefix = ENV['QUEUE_PREFIX'] # beta or production
  end
end

such that any job that starts with the prefix is processed, and others are not. This would enforce that an environment can only process jobs it knows how to without needing to keep QUEUES in sync manually.

glacials avatar Jul 01 '19 05:07 glacials