spinach icon indicating copy to clipboard operation
spinach copied to clipboard

Allow periodic tasks to define their exact wall-clock time to run

Open juledwar opened this issue 1 year ago • 2 comments

The periodics are currently effectively a 'time since it last ran, which in turn is time since it first ever ran'.

It would be useful for periodics to define a wall clock start and offset, for a precise execution time (assuming free workers are available).

Tasks are currently defined like:

    def task(self, func: Optional[Callable]=None, name: Optional[str]=None,
             queue: Optional[str]=None, max_retries: Optional[Number]=None,
             periodicity: Optional[timedelta]=None,
             max_concurrency: Optional[int]=None):
        """Decorator to register a task function.

I propose that we extend the parameters to accept a periodicity_start, another datetime.timedelta object that is used to state the offset from the next largest unit. For example:

  • periodicity_start=timedelta(minutes=1), periodicity=timedelta(minutes=5) would tell the code to run the task every 5 minutes, starting from 1 minute past the hour i.e. 1m, 6m, 11m past the hour, etc.
  • periodicity_start=timedelta(hours=3), periodicity=timedelta(hours=6) would tell the code to run the task every 6 hours at 3am, 9am, 3pm and 9pm.

juledwar avatar Jun 03 '24 01:06 juledwar

This should be simple to implement: the register_periodic_tasks can change the "now" value to the required starting offset as per the task definition.

The hard part is making sure the parameters make logical sense and rejecting bad ones.

juledwar avatar Jun 03 '24 01:06 juledwar

Additionally, any existing tasks already in Redis that are subsequently changed to include a periodicity_start would need updating in Redis. In practice, this means going over all of them on every worker startup to make sure they are defined correctly.

juledwar avatar Jun 03 '24 01:06 juledwar