schedule job fails to run, sometimes will randomly work on reboot, no errors shown

[fiddydiddy]

We are having issues getting a scheduled, recurring job to run when its supposed to. There are a slew of other jobs being scheduled (non-recurring, 1-time jobs) that seem to work fine to date. Killing and starting the server back up can SOMETIMES result in the job running like it should, but id say 95% of the time the job never runs, no logs indicating failures, etc.

The job is showing as run successfully in agenda-dash UI. We have a log inside the job just saying "hey i made it here" and even that is not firing.

We have this same issue running locally (Ubuntu and windows env's) as well as our production server (Ubuntu server). I saw a quick line in the agenda documentation about enabling logs in the ENV file for agenda itself, where do those get output to?

At this point we are pretty confident the issue is with agenda and not something upstream.

const agenda = new Agenda({
  db: { address: connectionString, collection: 'scheduled-tasks' },
  processEvery: '1 minute'
});



//-------------------------------------------------------------------------------------------------------
// Run once on server boot to have cred available before next refresh
(async function () {
  await agenda.start();
  await agenda.schedule('1 second', 'refresh-zoho-token');
})();


// WARNING- ONLY NEED TO SCHEDULE THIS ONCE THEN MANAGE IT FROM UI AS IT LIVES IN MONGO AFTER THIS IS INVOKED ONCE
/*
(async function() {
    const tokenRefresh = agenda.create('refresh-zoho-token', { to: '[email protected]' });
    await agenda.start();
    await tokenRefresh.repeatEvery('5 minutes').save();
  })();
*/


let zOAuthToken = '';

//-------------------------------------------------------------------------------------------------------
//
agenda.define('refresh-zoho-token', { priority: 'high', concurrency: 2 }, async (job, done) => {
  console.log('Got inside the refresh job');
  
  var data = new FormData();
  data.append('grant_type', 'refresh_token');
  data.append('client_id', `${process.env.ZOHO_CLIENT_ID}`);
  data.append('client_secret', `${process.env.ZOHO_CLIENT_SECRET}`);
  data.append('refresh_token', `${process.env.ZOHO_REFRESH_TOKEN}`);

  var config = {
    method: 'post',
    url: 'https://accounts.zoho.com/oauth/v2/token',
    headers: {
      ...data.getHeaders()
    },
    data: data
  };

  await axios(config)
    .then((response) => {

      console.log(response.data.access_token);
      zOAuthToken = response.data.access_token;
      console.log("NEW ZOHO TOKEN FETCHED: ", zOAuthToken)


    })
    .catch((error) => {
      console.log("ERROR GETTING ZOHO TOKEN: ", error);
    });

  done();

});

[simllll]

Can you try https://www.npmjs.com/package/@hokify/agenda if this fixes your issues?