Jobs are not removed from the internal list of running/locked jobs when their locks expire

[princjef]

I am trying to reconcile locks in agenda with the way I typically see them work in other systems. Namely, if the lock for a job expires while a worker is processing it, the worker should no longer be allowed to complete/fail processing of the job (via done() and done(err)) because it no longer owns the resource. Similarly, once expired, the job should no longer count against the overall concurrency or lock limit, since the current worker doesn't really own the processing of that job any more.

However, the current behavior does neither of these things. From what I can tell, the validity of the lock is only checked immediately before the job is handed to the worker for processing, but not any time after that. Likewise, jobs whose locks have expired continue to count against the concurrency and lock limits even though the processing of them should no longer be valid. The only way to remove them is to call done() or done(err) to finish processing, which can be dangerous if there is another worker processing the job after the expiration of the lock.

This combination leads to situations like the one described in https://github.com/agenda/agenda/issues/389. An expired job is continually delivered upon expiration of the lock (which is reasonable), but the previous deliveries are never cleared out. This makes the same job count against you multiple times even though there is really only one valid delivery of that job.

Is this expected behavior? If so, would it be possible to also have the ability to do something more like what I described at the beginning?

[DevBrent]

Call job.touch() to refresh the lockedAt time while a job is still processing. This is a double-start and cancellation problem where your code has started processing and is calling done after another worker has picked up the job per the lockedAt/expiration parameters. Ideally you need your own proprietary cancellation code if you would like the first worker to cancel itself after a second worker picks up the job. As an alternative, call touch if you want to ensure your job isn't picked up twice while being processed - this resets the lockedAt time to now preventing that behavior.