goflow
goflow copied to clipboard
Allow to schedule a node as kubernetes job.
In the DAG asynchronously we're creating a monolith structure when n the Distributed ETL use case it will be nice to have a flag to distribute the node of a graph as job, doing effectively a pipeline effect using redis queuing mechanism. Suppose we have a DAG:
- A -> B ->C
- The user trigger the dag from A,
- the triggering event schedule 3 types jobs A, B, C to be executed in the kubernetes cluster wherease each job type has a cardinality (number of pods associated)
This needs to be evalutated in design meeting and follow up discussion and check what currentely kubeflow does.