digdag
digdag copied to clipboard
[Question] Dependencies between parallel subtasks?
Is there any way to run two tasks (with subtasks) in parallel, but setting some dependencies between the subtasks?
A concrete example:
- task A: A1, A2, A3
- task B: B1, B2, B3
Requirements:
- A and B run in parallel
- B3 must wait for B2 and A2 to finish
Using Airflow (the only other orchestrator I have used), this would be done in the following way:
A1 >> A2 >> A3
B1 >> B2 >> B3
A2 >> B3
Is there any way to replicate this functionality using DigDag's declarative syntax?
Thank you.
EDIT: I have thought about two options,
option 1: doesn't exactly work because it makes a3 also wait for b2:
+task1:
_parallel: true
+a:
+a1: ...
+a2: ...
+b:
+b1: ...
+b2: ...
+task2:
_parallel: true
+a:
+a3: ...
+b:
+b3: ...
option 2: doesn't exactly work because then b3 also has to wait for a3
+task1:
_parallel: true
+a:
+a1: ...
+a2: ...
+a3: ...
+b:
+b1: ...
+b2: ...
+task2:
+b:
+b3: ...