Invalid job status RECEIVED, expected DONE

[IAkumaI]

I get this error with no reason. Sometimes after 5 minutes, sometimes after 30 and then. Error occur only on prod with much load. I have much jobs, but do not use any external storage.

I think is happend when I call limiter.jobStatus(key) Bottleneck version 2.19.5 (latest)

// Bottleneck options
let options = {
            maxConcurrent: 5,
            minTime: 22,
            reservoir: 12,
            reservoirIncreaseAmount: 3,
            reservoirIncreaseInterval: 250,
            reservoirIncreaseMaximum: 12,
        }

/opt/source/image-process/node_modules/bottleneck/lib/Job.js:77
      throw new BottleneckError(`Invalid job status ${status}, expected ${expected}. Please open an issue at https://github.com/SGrondin/bottleneck/issues`);
            ^
Error: Invalid job status RECEIVED, expected DONE. Please open an issue at https://github.com/SGrondin/bottleneck/issues
    at Job._assertStatus (/opt/source/image-process/node_modules/bottleneck/lib/Job.js:77:13)
    at /opt/source/image-process/node_modules/bottleneck/lib/Job.js:198:18
    at Generator.next (<anonymous>)
    at asyncGeneratorStep (/opt/source/image-process/node_modules/bottleneck/lib/Job.js:3:103)
    at _next (/opt/source/image-process/node_modules/bottleneck/lib/Job.js:5:194)
    at runNextTicks (node:internal/process/task_queues:59:5)
    at listOnTimeout (node:internal/timers:526:9)
    at processTimers (node:internal/timers:500:7)

[IAkumaI]

I think i found the reason.

I use limiter.on('failed', async() => {...}) for retrying jobs. After last try jobs marked as "error" in external key-value storage to prevent push it to queue again. But sometimes, if there are many jobs in small time, job with same ID added to queue in time of executing of async on-failed callback. So in that situation we have job ID which is not failed, but added again.

My english not so well. In another words, if you add job with same id before on-failed finished, error occurs.