delayed_job
delayed_job copied to clipboard
Pull from queues matching a pattern
We're running delayed_job on a beta environment and a production environment simultaneously. Both access the same database.
When we push a change to beta that e.g. deletes a job type, beta will continue to pull those job types from the queues (put there by production) and try to process them, failing every time, until we promote beta to production. Same in reverse if we add a job type on beta, production starts failing to process jobs.
The natural solution seems to be
# ensure the environment doesn't pull jobs it doesn't know about
export QUEUES=job1,job2,job3
However, manually keeping QUEUES
in sync with the various jobs we have seems subject to forgetfulness and poor knowledge transfer when jobs are added or deleted.
Additionally as the number of jobs increases, the environment variable gets more and more unwieldy to manage.
I would love a DRY way to achieve this same result, like matching against the existing queue prefix:
module MyApp
class Application < Rails::Application
config.active_job.queue_name_prefix = ENV['QUEUE_PREFIX'] # beta or production
end
end
such that any job that starts with the prefix is processed, and others are not. This would enforce that an environment can only process jobs it knows how to without needing to keep QUEUES
in sync manually.