Add argument `wait` of `remove_job` and `remove_all_jobs`

[Pandede]

Things to check first

• [X] I have searched the existing issues and didn't find my feature already requested there

Feature description

remove_job or remove_all_jobs of BackgroundScheduler should block when there is still running jobs, such as shutdown(wait=True).

Use case

Without blocking:

import time

from apscheduler.schedulers.background import BackgroundScheduler


def count(arr):
    time.sleep(0.2)
    arr[0] += 1


if __name__ == '__main__':
    # Scheduler with 5 workers
    gconfig = {
        'apscheduler.job_defaults.max_instances': 5
    }
    scheduler = BackgroundScheduler(gconfig)

    # The job is increment the first element every 0.1 seconds
    # Each increment requires 0.2 seconds
    arr = [0]
    scheduler.add_job(
        count,
        'interval',
        seconds=0.1,
        args=(arr,)
    )
    scheduler.start()

    # Do the jobs in a second
    time.sleep(1.0)

    print('Value', arr[0])  # Value 7

    # Remove the jobs and reset the value
    scheduler.remove_all_jobs()
    arr[0] = 0

    # However, as it is not blocked, there's still running jobs, the value keeps incrementing
    time.sleep(1.0)
    print('Value', arr[0])  # Value 2

With blocking:

import time

from apscheduler.schedulers.background import BackgroundScheduler


def count(arr):
    time.sleep(0.2)
    arr[0] += 1


if __name__ == '__main__':
    # Scheduler with 5 workers
    gconfig = {
        'apscheduler.job_defaults.max_instances': 5
    }
    scheduler = BackgroundScheduler(gconfig)

    # The job is increment the first element every 0.1 seconds
    # Each increment requires 0.2 seconds
    arr = [0]
    scheduler.add_job(
        count,
        'interval',
        seconds=0.1,
        args=(arr,)
    )
    scheduler.start()

    # Do the jobs in a second
    time.sleep(1.0)

    print('Value', arr[0])  # Value 7

    # Remove the jobs and reset the value
    scheduler.remove_all_jobs(wait=True)
    arr[0] = 0

    # The value is reset after all jobs are removed, it keeps 0
    time.sleep(1.0)
    print('Value', arr[0])  # Value 0

[agronholm]

I think this is a useful feature to have. I'll try to include it in v4.0 if possible. If it's too much work, then I'll target 4.1.

[Pandede]

I think this is a useful feature to have. I'll try to include it in v4.0 if possible. If it's too much work, then I'll target 4.1.

Is that possible to implement this feature in version 3.x?

[agronholm]

I don't intent to add any new features to the 3.x branch. I have my hands full with so many projects already that working on two APScheduler branches is just not feasible.

[gh-andre]

I ran into the same issue of not being able to stop scheduled jobs gracefully. Adding a wait parameter to remove_job would definitely be helpful, but if it's going to be similar to AsyncIOExecutor.shutdown(), it will likely ignore this parameter because there's no async in any of those methods.

There is one thing that can be done on its own, which sounds like it would be simpler than converting a bunch of methods to async, which is to provide a method on the job to indicate whether it is scheduled or not.

This would allow app-level callbacks to implement their own tracking of whether they are running or not, which is currently impossible because a job may be in transition from being scheduled to when the job callback is called, like this:

job.pause()               # disables further scheduling

if job.scheduled:         # indicates that the job was scheduled before it
                          # was paused (even before callback is called)

    await app_stop_on_start_or_wait_to_finish()

, where app_stop_on_start_or_wait_to_finish() is some app-level logic that would cancel the job callback that is about to be called or waits for it to finish (or cancels it) if it is in progress.

[agronholm]

I'm not sure about the terminology: what's a "job" here?

[gh-andre]

apscheduler.job.Job

https://apscheduler.readthedocs.io/en/3.x/modules/job.html#module-apscheduler.job

Knowing the job status would allow application code to anticipate whether the job callback is going to be called, so scheduled jobs can be shut down gracefully.

[agronholm]

Sorry, I thought the comment belonged to a very different project, hence the confusion :) What is your actual use case though?

[gh-andre]

Thanks for responding.

The use case is to be able to wait for (this issue) or be able to cancel (a more generic one) a job that has been scheduled, but the callback for which hasn't been called yet, which is cumbersome to implement reliably without some support from the framework (APscheduler).

In practical terms, what I'm running into is that on application shutdown, I remove a scheduled job and I check/wait for an asyncio event, which is cleared in the first line of the job callback and set when the job callback returns, effectively mimicking waiting on the running job, if there is one.

Where this approach fails is that if a job has been scheduled right before it was removed, but the callback hasn't been called yet, so the event says there's no scheduled job running and the app proceeds to shut down other app services, so when the scheduled job callback is called, it ends up with referencing partially destroyed application and with a weird exception like database service is gone, etc.

Technically, it is possible to rig the code to set some kind of a die-when-or-if-called flag for the scheduled job before it is removed, but the call may be made when the service handling this scheduled job is destroyed, so there's no way to report anything (i.e. logger service may be gone as well) or handle anything gracefully.

Having some job status flag, such as whether it has been scheduled to run, would allow app code to sleep for a moment or two while the just-callback is called and then wait for it to finish or, if allowed, to cancel it in the app code.

Thinking aloud, if job.remove() returned the underlying asyncio task, if one was scheduled, it would make it really simple for the app code to wait on this task or cancel the job gracefully. I realize that it's not as easy for concurrent futures, though.

Sorry for the long post. I hope it makes sense.