kestra icon indicating copy to clipboard operation
kestra copied to clipboard

Add a `ResumeExecution` execution task to resume paused execution by ID

Open tchiotludo opened this issue 1 year ago • 3 comments

After pausing an execution with Pause task If you want to unpaused it, for now you can only handle it an api call or from the ui.

We should add Unpaused task to do it directly from Kestra flow. By this way, we will be able to use for an example a trigger from a queue and to restart the execution based on the current message. The design on the task can be tricky since :

  • if the trigger is from the queue, we need to support internal storage
  • if the trigger is from another task, we need to support raw data from outputs

Related discussion in slack

tchiotludo avatar Mar 03 '23 17:03 tchiotludo

The motivation behind this feature is to offload async processing.

  1. Start a Kestra workflow execution
  2. The workflow gathers/preprocesses data
  3. The workflow sends the data for processing to an external system
  4. The workflow pauses itself (with a defined timeout)
  5. The external system sends an event signaling the processing finished (via a webhook or a MQ system)
  6. The workflow gets resumed
  7. Data received from the external system get evaluated and further processed

yuri1969 avatar Mar 06 '23 14:03 yuri1969

linked to https://github.com/kestra-io/kestra/issues/1581

Ben8t avatar Jun 22 '23 07:06 Ben8t

Proposed syntax:

  - id: pause
    type: io.kestra.core.tasks.flows.Resume
    executionId: "{{ execution.Id }}"  # the default should be the current execution ID of this flow, but it can be any execution ID

anna-geller avatar Nov 19 '23 23:11 anna-geller