Can i run background sceduled jobs on master redis server ?

[abhishekthehulk]

Hi , Currently we have 2 resque servers running to balance the load coming on a single resuque server . Can it be possible to run background scheduled job on 1 resuqe server it can be on master redis or slave redis ??

[yaauie]

I'm not entirely sure I understand your question; could you describe your architecture in a little more depth?

The vast majority of resque-scheduler (and resque) commands mutate what is in redis (e.g., by popping an item off a queue, thereby changing the value held in the underlying redis list) and are therefore not well-suited to be connected to a slave. But from a thoroughput standpoint, redis probably isn't your bottleneck, at least not if it's hosted on a well-powered machine; I maintain several clusters of resque workers (each of which has an resque-scheduler running) and am able to see ~1200 connected active redis clients (12 resque worker processes on each of ~100 hosts) on a single redis host before we start seeing calls to redis itself bog down.

If you could explain your architecture in a bit more detail, I'd be glad to help you get sorted out :smile:

[abhishekthehulk]

Currently we are running with 8 application servers and two redis servers to handle the traffic coming on our site . We have a set of scheduled job which runs midnight every day . But sometimes we faced that both the redis servers pickup the same background job and process them . Hence we faced multiple duplicate entries in our database as both redis servers processed them . So can it be there any possible solution we can have so that all background jobs will be processed only by master redis server and it should not be picked up by slave redis server.

[yaauie]

The redis server is incapable of doing any processing; it is only used by resque (and resque-scheduler) as a storage mechanism. Do you perhaps have an issue with multiple resque-scheduler processes running, which enqueues duplicate work into resque?

[abhishekthehulk]

yes , multiple resque-scheduler jobs were getting processed by both redis servers. Like i have a job which process a csv file . In csv file there is records of users with email addresses . Job runs at midnight and app server inserts users in user table with email address specified in csv . But this resque scheduled job is picked up by both servers and they process same csv file and it ends up in having two users record with same email . I can check validations but still if there is way we can restrict other redis server to pick same job .

[yaauie]

I believe you are conflating terms, which leads to confusion:

redis: the storage backend used by resque and resque-scheduler resque: the project that encompasses putting work on a queue, and workers that pick work up off a queue to process asynchronously resque-scheduler: puts work on a queue on a configured schedule (note: resque-scheduler does not actually process the work)

So to my understanding, you have multiple resque-scheduler instances, which are each reading from your schedule and enqueueing work. Since multiple work is enqueued, it is performed multiple times. The solution here would be to eliminate one of your resque-scheduler instances, which would prevent schedule duplication