same crontab on multiple servers

[bernabas]

I have exported whenever tasks on multiple servers, but most of the crontab tasks have an implicit assumption that they should only be run once in their given time frame. Having multiple servers with the same crontab tasks obviously would break this assumption, leading to all kinds of weird bugs. Instead, what if we let the servers use a middleware (in this implementation, redis is used) to communicate and decide which server will run the task.

in *.schedule.rb

set :redis_options, url: 'redis://localhost:6379/1'

the default value for :redis_options is nil

and if it is set to something other than nil, instead of

0 4 * * * /bin/bash -l -c 'cd /path && bundle exec rake sometask --silent >> log/cron.log 2>> log/cron-error.log' 

it would export

0 4 * * * /bin/bash -l -c 'cd /path && if whenever_server >> /dev/null ; then cd /path && bundle exec rake sometask --silent >> log/cron.log 2>> log/cron-error.log ; fi'

before executing the job, we would execute the whenever_server script that would basically use atomic operations to decide if the job should run on this server on not (guaranteeing that the job is executed only once)

[bernabas]

@javan can you take a look when you have time? thanks :smiley:

[sandstrom]

I think it would be better if whenever exposed a simple locking mechanism.

# schedule.rb

acquire_lock do
  # The user can decide what to do here, for example make an http request, 
  # inquiry with redis, check with consul, or use some other mechanism.

  # If this method return true whenever will run the job/task/item.
  # If defined, this block is invoked before every runner/task in the schedule.
  true
end

[ausangshukla]

How are people handling this? I too have the exact same issue.

[sandstrom]

@ausangshukla I would look at https://github.com/rails/solid_queue or https://github.com/bensheldon/good_job, both have support for scheduled jobs AFAIK.

If you want to use whenever, handle the locking outside, i.e. in application code in your job. For example like this: https://www.postgresql.org/docs/current/explicit-locking.html#ADVISORY-LOCKS