asynq
asynq copied to clipboard
hibiken
[FEATURE REQUEST] Support Workflow
Is your feature request related to a problem? Please describe. We'd like to add support for creating a workflow composed of multiple tasks. A workflow should be a directed-acyclic graph and can be grouped into multiple phases (or stages). A good example is Github actions, where a user can compose a workflow from multiple tasks (aka jobs).
Describe the solution you'd like A workflow should be treated as a unit. If a failure happens during a workflow execution, the entire workflow should be retried. Web UI should support inspecting and visualizing a workflow.
Additional context Similar feature request #218
Just contributing some thoughts to this workflow idea, as I'm rolling my own implementation of this at the moment.
The simplest implementation I had was just specifying the next task that had to be scheduled inside the payload struct.
So when the worker was finished doing task X, it knew to schedule task Y. The treating workflow as a unit comes with a lot of overhead to get right - as can be seen in the temporal.io infrastructure. I like asynq as it is a versatile, to be used in many ways without bloating and complexity.
@aon108 thanks for your input. I currently have a rough idea on how I want to go about implementing this. I agree with you that we want to keep the simplicity of the library and my hope is that this workflow feature is a layer on top of what we currently have so that it doesn't change the core of the library :)
Temporal does dehydration of a failed workflow . That means whatever part of a workflow fails , it can pickup where it left off.
Async does not have this feature , but let me know if I am incorrect. But I think all async needs to do is let you know it failed and for your own data layer to be idempotent so that repeats of parts of the workflow are fine anyway .
Temporal does dehydration of a failed workflow . That means whatever part of a workflow fails , it can pickup where it left off.
Async does not have this feature , but let me know if I am incorrect. But I think all async needs to do is let you know it failed and for your own data layer to be idempotent so that repeats of parts of the workflow are fine anyway .
Yes agreed with the above. My approach to this has been to persist a state machine in PG and then have each async task check the state and execute the next corresponding task in workflow. if the entire queue fails the workflow is consider failed. first thing the asynq server checks on startup is if the queue is empty and there are workflows that are not complete then schedules the correct tasks to continue processing. Asynq has the great feature of unique tasks, so that even if multiple servers check the state on startup only 1 unique task per workflow is started.
Ok that makes sense , and nice and simple - I like simple.
In a workflow the retry then does not happen and you do that on purpose to avoid the side effects that could occur ?
Ok that makes sense , and nice and simple - I like simple. In a workflow the retry then does not happen and you do that on purpose to avoid the side effects that could occur ?
If i understand you correctly you mean that the entire worfklow does not get rerun from start on a failure and recovery.
If so then yes to avoid side effects. In my particular use case some of the workflow steps are quite computational intense and take time. so rather than waste resources by doing the work again even if it is idempotent I chose to skip and just continue from the latest step.
I was thinking about how to describe a workflow and ironically https://github.com/go-task/task markup looks really nice and well dog fooded.
It is currently designed to replace a makefile with one thing running before another thing using the conconcept of dependencies but i think it can also be used to model tasks for the Async project, and possibly as a controller perhaps.
@hibiken I tried to build something similar with https://github.com/s8sg/goflow. This library keeps the execution state in redis . It also support complex branching and looping strategy. Lot of the components/ideas from this project can be reused
@hibiken I developed a dag workflow library based on asynq, It's simple and easy to use. https://github.com/yuyang0/dagflow
