Feature Request: Looping and Parallel Processing in DAG Steps

[rrottmann]

Currently, in order to run code in a loop and do parallel processing, a shell script is required. However, this happens all in the same step and can make the output difficult to read and troubleshoot. It would be useful to have the ability to loop a step and do parallel processing within the DAG itself.

Suggested Solution: Add a new feature to the DAG engine that allows users to loop a step and do parallel processing within the DAG. You would specify the code to execute, how many parallel executions and how the input and output data should be handled. Like feeding line by line or using a FIFO buffer.

[yohamta]

Thank you for the interesting suggestion.

Maybe this could be related: #439