db-scheduler icon indicating copy to clipboard operation
db-scheduler copied to clipboard

Support simple form of task priority

Open kagkarlsson opened this issue 4 years ago • 2 comments

It would be good to support a basic form for task priority that would ensure that certain tasks would get priority if there suddenly is a long queue of due executions.

A bit worried about performance implications. If we allow priority to be an unbounded field, it will be difficult to create indexes for optimal polling.

SELECT * from scheduled_executions
WHERE execution_time > :now
ORDER BY priority asc, execution_time asc

Normally this is a perfect index for polling, but it will not be if we add priority to the ORDER BY clause:

create index on scheduled_tasks(execution_time asc);

A possible compromise is to support a small fixed set of priorities and optimize polling queries...

kagkarlsson avatar Oct 10 '21 18:10 kagkarlsson

Are you interested in PRs for this one?

Desislav-Petrov avatar Dec 29 '21 16:12 Desislav-Petrov

There is one already, though I have not had the time to analyze the performance implications of it. ( #183 )

The first thing I would do here is to create some synthetic data, say 10M executions in the table with a new field priority (varied data) and run the polling query to see how it performs. If bad, try to optimize using indices. Once we know how it performs, we can decide how the feature should be designed..

Would you like to contribute with some testing? And dump some EXPLAINs?

kagkarlsson avatar Dec 29 '21 17:12 kagkarlsson