How to delete job with lock?

[michael-land]

I would like to clean all queue when shutdown during development, but looks like i'm not able to clean jobs with lock.

process.on('SIGINT', async () => {
  console.log('\n');

  if (process.env.NODE_ENV !== 'production') {
    for (const task of taskManager.tasks.values()) {
      console.log(`=== TASK: ${task.name} GRACEFUL SHUTDOWN ===`);

      await task.queue.drain();
      await task.queue.clean(0, 99999, 'active');
      const activeJobs = await task.queue.getActive();
      console.log(`ActiveJobs: ${activeJobs.length}`);
      console.log(`=== TASK: ${task.name} CLOSED ===`);
    }
  }

  process.exit(0);
});

Logs

PROGRAM START
worker 1 processing job repeat:task_name:b164ec4efe3520dc2b79f245732c7dd6:1611432392000
worker 1 processing job repeat:task_name:b164ec4efe3520dc2b79f245732c7dd6:1611432393000
^C

=== TASK: queue GRACEFUL SHUTDOWN ===
ActiveJobs: 1
=== TASK: queue CLOSED ===

[manast]

This is a bit tricky actually. You would need to make sure that none of the workers is still locking the job before you can delete it. Force a deletion could be dangerous since the worker is still working on the job so it may fail down the road...

[michael-land]

Thanks, Is there any existing method for unlock the job?

I also tried to change the job status to failure, but the method moveToFail() required a token. Is there a way to get the job token?

[manast]

A lock on a job uses a unique token that is generated for every Worker, so that only the worker that actually tool the lock is able to unlock it. Currently there is no "deleteLock" method, this function is performed atomically when moving a job to completed or failed statuses, but if this function is necessary it would be easy to add it as a new method with an option like "override token" or "force" that always deletes even if you do not know the token used for locking it, however this would be a "use at your own risk" function.

[maximedupre]

This feature would be quite useful 😶

Currently, there is no way to delete a job that is locked? Or is there a way to get the worker token to move the job to completed/failed?

[ivnnv]

Hi, there are indeed cases where the jobs are stuck in an active state when the worker is 100% NOT performing any action. I have many of these right now and Im not being able to just remove them (Ive manually checked that their actual state is "success" [so thats it they did their thing correctly], but thats another story). So... is there ANY way to remove or move to completed without needed to stop/restart the worker?

[manast]

If the lock is still there, then the worker is still running for that job. Another thing is that that worker may be stuck doing nothing due to some wrong handled promise or something like that. It is possible to manually removing the lock key from Redis though. We may implement a force flag for remove job, however this could create issues with the worker if it tries to complete the job later in the future.

[ivnnv]

Hi @manast yeah that would be really helpful, obviously someone that uses something like job.releaseLock() should know that could have side effects and once specificied at the documentation / type @description, to use at each own risk.

[isaachinman]

@manast Any updates on this? Force removal of jobs is necessary in some cases, when user error (hung jobs) has created a mess to clean up.

As far as I can tell, the only way to achieve a clean up is to force obliterate the queue and recreate it, right?

[manast]

The locks will expire themselves automatically after max 30 seconds (default value).