kue-scheduler every couldn't run on redis

[WUJJU]

Hi, is anyone having the same problem? I am trying to migrate my parse host backend to heroku. I am changing Parse.Cloud.Job to kue-scheduler. It seems the every() function doesn't work in heroku redis, while it work on local redis-server.

var kue=require('kue-scheduler');
var queue=kue.createQueue({redis:'redis://h:[email protected]',
    skipConfig: true
});


var job=queue.create('new_job').attempts(3).priority('normal');

queue.every('2 seconds',job);

queue.process('new_job',function(job,done){
   console.log('Job',job.id,'is done');
   done();
})

[gmcnaught]

Kue depends on keyspace notifications, does heroku redis support enabling keyspace expiration notifications?

On Friday, September 2, 2016, Hao Wu [email protected] wrote:

Hi, is anyone having the same problem? I am trying to migrate my parse host backend to heroku. I am changing Parse.Cloud.Job to kue-scheduler. It seems the every() function doesn't work in heroku redis, while it work on local redis-server.

var kue=require('kue-scheduler'); var queue=kue.createQueue({redis:'redis://h:[email protected] javascript:_e(%7B%7D,'cvml','h:[email protected]');', skipConfig: true });

var job=queue.create('new_job').attempts(3).priority('normal');

queue.every('2 seconds',job);

queue.process('new_job',function(job,done){ console.log('Job',job.id,'is done'); done(); })

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/lykmapipo/kue-scheduler/issues/46, or mute the thread https://github.com/notifications/unsubscribe-auth/ABTsfzJQTCPQ24KU9n-yJRnlVVC1Gxnqks5qmHL0gaJpZM4Jz_7u .

[WUJJU]

Based on my search, Heroku redis close the keyspace notification by default. I have log in my Heroku redis instance and "set notify-keyspace-events Ex", console shows "OK". But every() is still not work. Any idea?

[lykmapipo]

Hello. Have you manage to fix it on Heroku.

On 2 Sep 2016 23:21, "Hao Wu" [email protected] wrote:

Based on my search, Heroku redis close the keyspace notification by default. Do you have any idea to change the config. I try "redis-cli config set notify-keyspace-events Ex". But fail.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/lykmapipo/kue-scheduler/issues/46#issuecomment-244478249, or mute the thread https://github.com/notifications/unsubscribe-auth/ABiUaUDHCB_ZHXzoyAp5_qN75GxVzhOCks5qmIU_gaJpZM4Jz_7u .

[lykmapipo]

@WUJJU

Please ensure you have Redis 2.8.0+.

[lykmapipo]

@WUJJU

See requirements for kue-scheduler to work. By default kue-scheduler will try to enable kespace notification. If it fails please use

redis-cli config set notify-keyspace-events Ex

[WUJJU]

@lykmapipo all requirements meet and notify-keyspace-events set to Ex. But still fail.

[lykmapipo]

Hello.

I think we should add logger to kue-scheduler to log its inner working.

Try to listen for "schedule error" event in your kue instance and see it it fired.

On 5 Sep 2016 22:29, "Hao Wu" [email protected] wrote:

@lykmapipo https://github.com/lykmapipo all requirements meet and notify-keyspace-events set to Ex. But still fail.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lykmapipo/kue-scheduler/issues/46#issuecomment-244802595, or mute the thread https://github.com/notifications/unsubscribe-auth/ABiUacSbHOQWAi5m3Y0_HiJWxQS_ESbQks5qnG2hgaJpZM4Jz_7u .

[josephktcheung]

Hi,

Just want to chime in with a reply from Heroku support. I too faced the same issue that Heroku Redis cannot run every. I tried connecting to Heroku Redis using redis-cli and do config set notify-keyspace-events Ex but it said unknown command CONFIG instead of OK. Then I contacted Heroku support by creating a ticket on this issue and here's Heroku support's reply:

I'm afraid that Heroku Redis doesn't currently support keyspace notifications. However, I can loop in Redis To Go to see if their instances do support this. They will reach out as soon as they can.

@WUJJU I'm afraid that you did config setting wrongly. set and config set are two different commands. set sets key to hold the string value. config set is used in order to reconfigure the server at run time without the need to restart Redis. You can change both trivial parameters or switch from one to another persistence option using this command (Quoted from Redis documentation). You can find the documenation of set and config set here and here.

@WUJJU you can try redis-cli config set notify-keyspace-events Ex in your local Redis -> enter redis-cli console -> type config get notify-keyspace-events, it should show xE. Then try to change the config to something else e.g. e by set notify-keyspace-events e, it will show OK but it's because it just set a new key notify-keyspace-event to hold e but it's not changing the config. If you do config get notify-keyspace-events now it will still show xE. If you log in your Heroku Redis instance and input config set notify-keyspace-events Ex it will show (error) ERR unknown command 'config'

[WUJJU]

@josephktcheung Thanks lot for the correction! Didn't expect the difference between set and config set.

[josephktcheung]

@WUJJU Welcome! Besides, I found out that Redis Cloud, another Heorku Redis add-on provider, allows us to set notify-keyspace-events. Happily using it.

[ghost]

@josephktcheung How are you setting notify-keyspace-events for Redis Cloud?

[josephktcheung]

pretty straight forward,

1. Add Redis Cloud heroku add-on
2. Go to heroku settings page
3. Reveal Config Vars in Config Variables
4. Copy REDISCLOUD_URL, it should be something like redis://rediscloud:[email protected]:PORT_NUMBER
5. In terminal enter redis-cli -h xxx.redislabs.com -p PORT_NUMBER -a PASSWORD with variables from REDISCLOUD_URL
6. Once connected, enter config set notify-keyspace-events Ex
7. You can verify it is set correctly by entering config get notify-keyspace-events
8. Done!

[ghost]

thanks @josephktcheung I got the config notify-keyspace-events set. But I only see my job running one time, not repeatedly every 3 seconds. Do I need to run the following (as outlined here https://gist.github.com/mojodna/1251812/21455e1dae2e6723a06988ee1978e69f44d89f75): heroku scale web=1 worker=1

Is something wrong with my code? See below.

// Create a kue instance and a queue.
var redisUrl =  process.env.REDIS_URL;
var kue = require('kue-scheduler');
var Queue = kue.createQueue({
  redis: redisUrl,
  skipConfig: true
});

function emailDailyRecapHandler(request, response) 
{
    console.log("KUE RUNNING: emailDailyRecapHandler");
    done();
}
var jobName = "emailDailyRecap";
// create job instance 
var job = Queue
  .createJob(jobName)
  // Priority can be 'low', 'normal', 'medium', 'high' and 'critical'
  .priority('normal')
  // We don't want to keep the job in memory after it's completed.
  .removeOnComplete(true);

// Schedule it to run every 60 minutes. Function every(interval, job) accepts interval in either a human-interval String format or a cron String format.
Queue.every('3 seconds', job);
// Queue.now(job);

// Processing a scheduled job.
Queue.process(jobName, emailDailyRecapHandler);

[josephktcheung]

Have you tried the code locally @developermhayden? From what I saw, emailDailyRecapHandler's response parameter should be renamed to done. Otherwise you will get done is not defined

[josephktcheung]

Besides, @developermhayden are you sure REDIS_URL is the correct environment variable for redis cloud? From what I have in heroku settings, redis url environment variable name for redis cloud by default is REDISCLOUD_URL not REDIS_URL, REDIS_URL is for heroku redis, unless you renamed it. I suggest you to remove heroku redis if it's not used anymore to avoid confusion.

[ghost]

@josephktcheung I mapped the REDICCLOUD_URL to REDIS_URL. Your catch on done() was the solution. Thank you.

[ghost]

Based on the examples in kue-scheduler, I should be able to set my reoccurring job using cron syntax. I've tried and my jobs run every 2 seconds regardless of what cron syntax I use. After running the job 100 times or so, heroku goes idle and the jobs stop running. When I do a curl request to a parse server endpoint, the server responses correctly but jobs will not start running again until I restart the server by redploying.

I have the following now:

// Run daily at 5pm Queue.every('*/0 */0 */17 * * *', job);

I have also tried the following, still getting the same results (job is running every 2 seconds) Queue.every('0 17 * * *', job);

I need my job to run every day at 5pm

My Current code is as follows:

// Create a kue instance and a queue.
var jobTimeSchedule = '0 17 * * *';
var redisUrl =  process.env.REDIS_URL;
var kue = require('kue-scheduler');
var Queue = kue.createQueue({
  redis: redisUrl,
  skipConfig: true
});
var _ = require('underscore');

// jobs
function emailDailyRecapHandler(request, done) 
{
    console.log("KUE RUNNING: emailDailyRecapHandler");
    Parse.Cloud.useMasterKey();

    Parse.Cloud.httpRequest(
    {
        method: 'POST',
        url: urlBase + 'emailDailyRecap',
        headers: 
        {
            'Content-Type': 'application/json;charset=utf-8',
            'X-Parse-Application-Id': Parse.applicationId,
            'X-Parse-REST-API-Key': parseRestApiKey,
            'X-Parse-Webhook-Key': parseWebHookKey
        },
        body: 
        {
            params: {}
        }
    }).then(function(httpResponse) 
    {
        console.log("emailDailyRecap success: " + httpResponse.text);

        return done();
    }, 
    function(error) 
    {
        console.error('emailDailyRecap: request failed with response code ' + error);

        // Pass Error object to done() to mark this job as failed.
        return done(new Error(errorMessage));
    });
}
console.log("Parse.Cloud.job is undefined using Kue-scheduler");
var jobName = "emailDailyRecap";
// create job instance 
var job = Queue
  .createJob(jobName)
  // Priority can be 'low', 'normal', 'medium', 'high' and 'critical'
  .priority('normal')
  // We don't want to keep the job in memory after it's completed.
  .removeOnComplete(true);
  //.unique(jobName);

// Schedule it to run every 60 minutes. Function every(interval, job) accepts interval in either a human-interval String format or a cron String format.
console.log("Schedule job to run: " + jobTimeSchedule);
Queue.every(jobTimeSchedule, job);
// Queue.every('0 0 17 * * *', job);
// Queue.now(job);

// Processing a scheduled job.
Queue.process(jobName, emailDailyRecapHandler);

[lykmapipo]

@developermhayden May you please open a new issue regarding cron syntax. Also i will appreciate if you can update kue-scheduler spec to fail on your scenario. That will help me and other contributors to fix its a bug.

[siddheshsatoskar]

You can also try config set notify-keyspace-events Xe If still does not work.