digdag icon indicating copy to clipboard operation
digdag copied to clipboard

[Question] Dependencies between parallel subtasks?

Open FredericoCoelhoNunes opened this issue 3 years ago • 0 comments

Is there any way to run two tasks (with subtasks) in parallel, but setting some dependencies between the subtasks?

A concrete example:

  • task A: A1, A2, A3
  • task B: B1, B2, B3

Requirements:

  • A and B run in parallel
  • B3 must wait for B2 and A2 to finish

Using Airflow (the only other orchestrator I have used), this would be done in the following way:

A1 >> A2 >> A3
B1 >> B2 >> B3
A2 >> B3

Is there any way to replicate this functionality using DigDag's declarative syntax?

Thank you.


EDIT: I have thought about two options,

option 1: doesn't exactly work because it makes a3 also wait for b2:

+task1:
  _parallel: true
  +a:
    +a1: ...
    +a2: ...
  +b:
    +b1: ...
    +b2: ...

+task2:
  _parallel: true
  +a:
    +a3: ...
  +b:
    +b3: ...

option 2: doesn't exactly work because then b3 also has to wait for a3

+task1:
  _parallel: true
  +a:
    +a1: ...
    +a2: ...
    +a3: ...
  +b:
    +b1: ...
    +b2: ...

+task2:
  +b:
    +b3: ...

FredericoCoelhoNunes avatar Oct 08 '21 09:10 FredericoCoelhoNunes