spinach
spinach copied to clipboard
Allow periodic tasks to define their exact wall-clock time to run
The periodics are currently effectively a 'time since it last ran, which in turn is time since it first ever ran'.
It would be useful for periodics to define a wall clock start and offset, for a precise execution time (assuming free workers are available).
Tasks are currently defined like:
def task(self, func: Optional[Callable]=None, name: Optional[str]=None,
queue: Optional[str]=None, max_retries: Optional[Number]=None,
periodicity: Optional[timedelta]=None,
max_concurrency: Optional[int]=None):
"""Decorator to register a task function.
I propose that we extend the parameters to accept a periodicity_start, another datetime.timedelta object that is used to state the offset from the next largest unit. For example:
periodicity_start=timedelta(minutes=1), periodicity=timedelta(minutes=5)would tell the code to run the task every 5 minutes, starting from 1 minute past the hour i.e. 1m, 6m, 11m past the hour, etc.periodicity_start=timedelta(hours=3), periodicity=timedelta(hours=6)would tell the code to run the task every 6 hours at 3am, 9am, 3pm and 9pm.
This should be simple to implement: the register_periodic_tasks can change the "now" value to the required starting offset as per the task definition.
The hard part is making sure the parameters make logical sense and rejecting bad ones.
Additionally, any existing tasks already in Redis that are subsequently changed to include a periodicity_start would need updating in Redis. In practice, this means going over all of them on every worker startup to make sure they are defined correctly.