High-level background tasks

[torfsen]

The new background job system proposed in #66 is more-or-less in place (ckan/ckan#3165). It provides a low level interface for executing asynchronous jobs in separate worker processes. What it doesn't provide (by design) are high-level features that are often useful when dealing with background activities:

• Scheduling (e.g. run the same job each day at midnight or once at a specific time in the future)
• Information about past jobs, including failed jobs (e.g. exception messages, job durations)

I'd like to propose a high-level task interface on top of the new job system that provides this kind of functionality. Tasks would be persistent entities that are stored in PostgreSQL and could be manipulated similar to other CKAN objects like for example datasets. One would probably need to distinguish between a task (which may have a certain repetition frequency) and its executions (each of which can, for example, fail with a different error message).

Scheduling for RQ can be done via rq-scheduler.

[torfsen]

Here are some thoughts on the topic in no particular order:

• A task is a high-level description of a piece of asynchronous work that is to be executed once or repeatedly in the future.
• A task has a schedule which describes when it is to be executed.
• At each scheduled point in time, a job is used to perform the actual execution of the task.
• It seems we need (at least) models for a Task and for a single TaskExecution.
• Tasks can only be scheduled from code (core or plugin) and not via the API. However, listing, enabling/disabling and deleting tasks via the API should be possible.
• RQ jobs can return results. They're not used for CKAN jobs (since these are not persistent) but might be of interest for CKAN tasks and could be stored in TaskExecution.
• What can go wrong with scheduled tasks?
  • Some part of the system (CKAN, Redis, Worker) might not be running at the intended point in time. That should be recognized (in retrospect) and dealt with. Such a task could either be re-run once the system is functioning again or the scheduled execution could be skipped.
  • The job might fail. That needs to be recognized and stored.
  • Parts of the system might have changed since the task was scheduled. For example, CKAN might have been upgraded. There's probably little on can do in such a situation from within the task system, so CKAN update instructions should document how to deal with this. The best option is perhaps to disable all tasks before the update (or to delete them).
• To actually enqueue scheduled jobs, rq-scheduler provides a command-line tool that needs to run in the background (similar to a worker).