pg-boss icon indicating copy to clipboard operation
pg-boss copied to clipboard

Worker stops working if same singleton key exists in retry & created state an batch size > 1

Open klesgidis opened this issue 11 months ago • 1 comments

Steps to repro

  • You should have a stately queue
  • You should have batch_size > 1
  • 2 jobs with the same singleton key, one in created and another one in retry states

In the above case the fetch query here will fetch both jobs and attempt to mark them as active. This causes the db to fail with UniqueKeyViolationError since both jobs have the same singleton key. In this situation the worker won't fetch messages forever

klesgidis avatar Jan 14 '25 14:01 klesgidis

I added a comment to the PR.

timgit avatar Jan 26 '25 15:01 timgit

I've encountered an issue similar to this, with a queue with policy singleton and batchSize > 1. Two jobs sent in quick succession end up stuck in the created state (either with an explicit singletonKey or without).

sylvaingi avatar Sep 23 '25 11:09 sylvaingi

I will try and address this before v11 goes out. I just fixed the issue regarding different keys, but I haven't yet addressed the issue of batches with the same key

timgit avatar Sep 23 '25 15:09 timgit

I just pushed a fix for this. For batch sizes > 1, I added a window function so it will only pull 1 job per singleton key.

timgit avatar Sep 23 '25 21:09 timgit