Combine scheduler with resque-lock

[zedtux]

Hi,

I'm using resque-scheduler to at least enqueue each minutes a job. This job could be called with different parameters (for example, I'm passing a different path).

Now when I'm restarting workers, this queue is growing very fast until the workers are back. The idea is that we should be able to send a parameter in order to enqueue only once a job like do resque-lock plugin.

What is your feeling with that idea ?

[ksinkar]

@zedtux I like the idea. I have faced this problem, in which I have had to manually clear the queues after a restart of the workers. Would you care to give a PR or at least a Unit Test case?

[zedtux]

@ksinkar I could do it but I don't have time right now.

[ksinkar]

@zedtux, if resque-lock already exists, why do you not use it? Why duplicate the functionality here in resque-scheduler? I was thinking of solving the above problem as follows:-

When the resque workers are down for a specfic queue/s (queues which are earmarked as such through a configuration setting), we stop pushing jobs to that queue, until its workers are up and running again.

@bugant @meatballhat Let me know your thoughts on this.

[zedtux]

@ksinkar resque-lock exists but doesn't work when a job has no arguments. This is why I opened this issue.

[epinault]

+1 on this to get fixed. I would love to be able to no pollute the queue with messages . Any work around meanwhile?

[zedtux]

A workaround is to manage yourself a key in redis for the queue using the redis client from Resque.

[epinault]

So what I am missing in the scheduler is that I am using a static config as cron... but I the before_schedule never gets called... and I am not sure where to put the lock code either....

[epinault]

So something is wrong with the way the plugin for the lock works.

if I do in my worker class extend Resque::Plugins::Lock

The before_enqueue_lock and lock are never called (not even time out while overriding)

but if I copy the code of that module in my class (without extend and define self.xxxx methods) then all the methods are called and working properly...

I suspect something is wrong on how it get extended (and maybe something with RAILS in the mix)

[epinault]

Yup as I thought, it was related to rails on my end.... had to add the require for the lock in my file and somehow the plugins was included correctly (I saw a message of missing constant when running in the console which never showed while running a rake task with the loaded environment ...)

Anyway, seems that they works fine now together for me without even changing the lock name

[yaauie]

Another workaround is to make the job itself responsible for scheduling the next iteration:

module Resque::Scheduler::RepeatInterval
  attr repeat_interval
  def around_perform_ensure_job_repeats(*args)
    yield
  ensure
    Resque.enqueue_in(repeat_interval, self, *args)
  end
end

Usage:

class SomeJob
  extend Resque::Scheduler::RepeatInterval
  @queue = :some_job
  @repeat_interval = 5.minutes

  def perform(*args)
    # ...
  end
end

Caveat: if your environment doesn't handle and recover dirty exits, you will lose jobs from the schedule when workers don't shut down cleanly. Alternatively, you could make that a before_perform hook, which would close the window in which the job could be lost.