Workflow Processor - DAG Execution Ordering #2599

[jem-davies]

This Draft Pull Request is to request interim feedback on the proposed solution to the issue #2599.

A new Benthos Processor workflow_v2 has been created here.

Some example benthos config files have been included in ./benthos/cmd/benthos.

• config_old.yaml - runs the example DAG from the workflow processor docs using the workflow processor
• config.yaml - runs the example DAG from the workflow processor docs using the new workflow_v2 processor

The config_old.yaml will output:

INFO STAGE A FINISHED   
INFO STAGE C FINISHED   
INFO STAGE B FINISHED   
INFO STAGE E FINISHED   
INFO STAGE D FINISHED   
INFO STAGE F FINISHED   

The config.yaml will output:

INFO STAGE A FINISHED   
INFO STAGE C FINISHED   
INFO STAGE E FINISHED   
INFO STAGE F FINISHED   
INFO STAGE B FINISHED   
INFO STAGE D FINISHED   

Notice that E will start / finish before B is started.

I would greatly appreciate some feedback regarding the proposed solution before I spend more time fixing some of the things that I know that I need to do:

• unit tests
• writing docs
• implementing the ability to restart DAG execution at particular places described on the workflow documentation under structured metadata
• Currently there is no logic to process batches > 1 .

Also I am a quite new to Go - so could well have made some newbie mistakes.

Specific questions that I have are:

• Should this be a new processor i.e V2 or should the existing workflow be altered?
• The ability to infer the DAG from the request_map fields of the branch - Is this something that the V2 needs to have?