Flexible scheduling

[jpbruinsslot]

Please refer to https://github.com/minvws/nl-kat-coordination/issues/204 for more background on this PR

Changes

Since this is a big PR with substantial changes, I've added the following for more guidance on review and qa. It is helpful to read the above-linked issue to get more context. And whenever you need me I can give you this as well in a call.

The main changes are:

• Addition of a new Schedule model to support a more deterministic approach to (re)scheduling of tasks
• Removal of the PrioritizedItem model and its database table. The queue is not a separate table anymore, a 'view' of the database filtered on status queued is the current state of the queued items of a scheduler.

Specific code changes of interest:

• Model changes

  • Schedule model added to models/schedule.py
    • migrations added ( take specific note of the p_item change to data in the task table)
    • store added to storage/schedule_store.py
  • Removed PrioritizedItem
    • migration added for removal of items table

• BoefjeScheduler changes

  • added a new process that adds new tasks to be picked up by checking the schedule of prior tasks: push_tasks_for_rescheduling()

• NormalizerScheduler changes

  • minor changes made to support the new setup

• Abstract base class Scheduler changes - some methods have been moved around so below are the main points of interest:

  • minor changes made to support the new setup
  • The post_push() method creates a Schedule for when a task has been added to the queue.

• API server changes

  • Added endpoints for Schedule
  • Updated Task endpoints to support the new setup

• Boefje Runner changes to support the new setup

• Rocky changes to support the new setup

QA notes

• Since this pr is now a replacement for the rescheduling of tasks within the scheduler the functionality of KAT should remain the same. So the main issues that could arise is from starting, (automatic) rescheduling of scans. All other actions of scanning that would touch the scheduler.
• Migrations, since there are substantial data migrations with this pr, care should be given on checking whether migrations from data prior to this pr will applied correctly with this pr.