Configurable Behavior for Enqueuing Dependent Jobs on Job Failure

[jtfidje]

We have built a large job orchestration engine on top of RQ and we are encountering some issues with the way RQ handles failed jobs that have dependent child-jobs registered to them.

In the current implementation, as seen in the handle_job_failure method in worker.py, a boolean enqueue_dependents is set to True when a job has run out of retries. This leads to the automatic enqueueing of dependent jobs when the parent job fails.

if retry:
    job.retry(queue, pipeline)
    enqueue_dependents = False
else:
    enqueue_dependents = True

try:
    pipeline.execute()
    if enqueue_dependents:
        queue.enqueue_dependents(job)
except Exception:
    # Ensure that custom exception handlers are called
    # even if Redis is down
    pass

In our use case, we would like to have more control over this behavior. We propose making this behavior configurable, allowing users to decide whether or not to enqueue dependent jobs upon the failure of a parent job. We suggest adding a new setting that defaults to the current behavior, so as not to break existing code.

This feature would provide more flexibility in handling job failures, particularly in complex job orchestration scenarios.

I would be happy to contribute this feature by submitting a pull request. Please let me know if this is a change you would consider merging, or if you have any suggestions or requirements for implementing this feature.

[selwin]

@jtfidje sorry for replying so late. Next time feel free to ping me using @selwin to get my attention so a notification arrives in my inbox.

Have you looked at Dependency(allow_failure=True)? This should allow you to control which dependents get enqueued when parent job fails. https://python-rq.org/docs/#job-dependencies

If this is not sufficient, I'm open to adding more control to dependency enqueueing behavior. What kind of mechanism/API are you thinking about proposing?