Async action pattern for HTTP webhook/callback

[zhouzhuojie]

Is your feature request related to a problem? Please describe. For example:

It's common that some remote services are designed in an async communication pattern:

• Webhook workflow:
  • Send HTTP
  • Wait for HTTP callback (usually listen on POST request from remote servers)
  • continue

Describe the solution you'd like For example, define a generic callback endpoint that's capable of finding the task by deriving public id from the request. Upon getting the webhook, look for the task/resolution, pause it and modify the state or the step or the input of the task and reevaluate the task.

Describe alternatives you've considered Maybe some workaround is to build something outside utask, e.g. a simple webserver that talks to utask APIs.

Additional context

[loopfz]

That's actually a really good suggestion.

Initial thoughts:

• This could be implemented via an action plugin
• The callback instance should probably be uniquely identified, not just tied to the public task id (eg one task could have several callbacks) ?
• The plugin could create an expected callback row somewhere in the db, and keep a reference to the task it came from
• we could expose a non-authenticated callback endpoint referencing the callback unique identifier and task public id (e.g. POST /callback/{taskID}/{callbackID} makes it easy to use + hard-to-guess, limits the need for authentication)
• question: how to make the contents of the callback available in the main task (via templating) ? The output of the callback step should probably contain the contents of the callback POST. Unsure what do with it if it's not json / yaml.

Regarding the timing/workflow aspects:

• We don't really need to modify the timing primitives of the engine
• The step, when executed should check its callback object in db, return SERVER_ERROR if not received, DONE otherwise (similar to the subtask plugin)
• To make it more fluid, the POST /callback/... endpoint could force run the task, to make sure it's rescheduled instantly
• as such, callback steps could have very slow retry patterns (e.g. hours) and still be unblocked instantly

What do you think of this initial brainstorm, @rbeuque74 @zhouzhuojie ?

[zhouzhuojie]

Thanks, that generic callback endpoint is nice to have!

I do think that if we have some control over the remote server, e.g. setting/passing Auth headers or configuring callback URL/headers, then we can leverage existing resolution APIs. For example, I added a new endpoint with POST /resolution/:resolution_id/reinstitute that combines pause, update and run. Let me know what do you think?

https://github.com/ovh/utask/pull/151

[rclsilver]

Closing this issue as #333 provides the callback feature