procrastinate
procrastinate copied to clipboard
procrastinate-org
Increasing worker concurrency increases number of `fetch_job` queries to procrstatinate database tables
(mostly looking for a sanity check on this before I roll up my sleeves)
run in worker.py starts a single_worker() once for as many instances as the concurrency setting specifies. This means that each NOTIFY from postgres that a job is available causes n coroutines to wake up and check fetch_job (near) simultaneously. This means that the number of calls to fetch_job scales at number_of_workers x concurrency rather than number_of_workers.
Assuming this is an unintended side effect of the implementation, would a PR to change this so that fetch_job is only called once per NOTIFY be accepted?
Hi @ashleyheath. We completely refactored the worker for the upcoming v3 release (done mainly by @onlyann). Would you mind looking at the new implementation of the worker. Do you think it is already solved there?
Hi @medihack, I've had a quick look at the refactor and it looks likely that it does solve the problem (though perhaps @onlyann can provide an opinion). Is there a timeline planned for the v3 release?
Hi @ashleyheath,
You are correct this is how it currently works.
#933 was created to solve this and other related issues. As @medihack mentioned, work has been done as part of the next major version. I am afraid this won't help you much for v2.
I am not aware of a timeline for when v3 will be released. Conversations are happening now on what the migration from v2 to v3 will look like, some of it you can read about in #1181 as well as on the discord server
If a pre-release of v3 was released, would you be interested in trying it on?
I agree with everything @onlyann and @medihack have said.
I'll add that we have a v3 milestone that is getting completed and I plan to work on #1189 on the week-end.
