conductor icon indicating copy to clipboard operation
conductor copied to clipboard

Wait for the task to expire without continuing to execute the next task

Open zengqinglei opened this issue 1 year ago • 2 comments

Describe the bug Occasionally, the waiting task is stuck, waiting for a long time, and the time expires and still does not continue to the next task

Details Conductor version:3.13.5 Persistence implementation: postgres Lock: No Task definition:

{
  "name": "数据写入中,注:执行期间.....排查",
  "taskReferenceName": "wait_data_write",
  "inputParameters": {
    "duration": "60seconds"
  },
  "type": "WAIT",
  "startDelay": 0,
  "optional": false,
  "asyncComplete": false
}

problem: image

Occasionally, the waiting task is stuck, waiting for a long time, and the time expires and still does not continue to the next task

This problem has appeared frequently and randomly in our production environment, causing serious business flow problems. I hope your team can help us analyze the possible causes and provide some solutions as soon as possible. Thank you very much!

zengqinglei avatar Apr 20 '23 03:04 zengqinglei

Hi @zengqinglei , WAIT tasks generally get marked completed by Sweeper. Have you disabled the sweeper by any chance?

manan164 avatar May 25 '23 17:05 manan164

Hi @zengqinglei if you are using the conductor-postgres-persistence as persistence me too facing the same issue in 3.13.5. The sweeper will not picks the WAIT task which is already popped from queue_message table and left in infinite state until it is manually approved. if you set the duration of the WAIT task as less than 30 seconds, then the WAIT task will be completed and will be moved to the next stage. I have made an temporary fix for it from my end, so that we would not be blocked.

narayanapadmanabhuni avatar Jun 08 '23 17:06 narayanapadmanabhuni