resque-scheduler
resque-scheduler copied to clipboard
resque
Combine scheduler with resque-lock
Hi,
I'm using resque-scheduler to at least enqueue each minutes a job. This job could be called with different parameters (for example, I'm passing a different path).
Now when I'm restarting workers, this queue is growing very fast until the workers are back. The idea is that we should be able to send a parameter in order to enqueue only once a job like do resque-lock plugin.
What is your feeling with that idea ?
@zedtux I like the idea. I have faced this problem, in which I have had to manually clear the queues after a restart of the workers. Would you care to give a PR or at least a Unit Test case?
@zedtux, if resque-lock already exists, why do you not use it? Why duplicate the functionality here in resque-scheduler? I was thinking of solving the above problem as follows:-
When the resque workers are down for a specfic queue/s (queues which are earmarked as such through a configuration setting), we stop pushing jobs to that queue, until its workers are up and running again.
@bugant @meatballhat Let me know your thoughts on this.
@ksinkar resque-lock exists but doesn't work when a job has no arguments. This is why I opened this issue.
+1 on this to get fixed. I would love to be able to no pollute the queue with messages . Any work around meanwhile?
A workaround is to manage yourself a key in redis for the queue using the redis client from Resque.
So what I am missing in the scheduler is that I am using a static config as cron... but I the before_schedule never gets called... and I am not sure where to put the lock code either....
So something is wrong with the way the plugin for the lock works.
if I do in my worker class extend Resque::Plugins::Lock
The before_enqueue_lock and lock are never called (not even time out while overriding)
but if I copy the code of that module in my class (without extend and define self.xxxx methods) then all the methods are called and working properly...
I suspect something is wrong on how it get extended (and maybe something with RAILS in the mix)
Yup as I thought, it was related to rails on my end.... had to add the require for the lock in my file and somehow the plugins was included correctly (I saw a message of missing constant when running in the console which never showed while running a rake task with the loaded environment ...)
Anyway, seems that they works fine now together for me without even changing the lock name
Another workaround is to make the job itself responsible for scheduling the next iteration:
module Resque::Scheduler::RepeatInterval
attr repeat_interval
def around_perform_ensure_job_repeats(*args)
yield
ensure
Resque.enqueue_in(repeat_interval, self, *args)
end
end
Usage:
class SomeJob
extend Resque::Scheduler::RepeatInterval
@queue = :some_job
@repeat_interval = 5.minutes
def perform(*args)
# ...
end
end
Caveat: if your environment doesn't handle and recover dirty exits, you will lose jobs from the schedule when workers don't shut down cleanly. Alternatively, you could make that a before_perform hook, which would close the window in which the job could be lost.
