goflow icon indicating copy to clipboard operation
goflow copied to clipboard

Allow to schedule a node as kubernetes job.

Open giorgiozoppi opened this issue 1 year ago • 1 comments

In the DAG asynchronously we're creating a monolith structure when n the Distributed ETL use case it will be nice to have a flag to distribute the node of a graph as job, doing effectively a pipeline effect using redis queuing mechanism. Suppose we have a DAG:

  1. A -> B ->C
  2. The user trigger the dag from A,
  3. the triggering event schedule 3 types jobs A, B, C to be executed in the kubernetes cluster wherease each job type has a cardinality (number of pods associated)

This needs to be evalutated in design meeting and follow up discussion and check what currentely kubeflow does.

giorgiozoppi avatar Mar 18 '23 16:03 giorgiozoppi