kis-flow icon indicating copy to clipboard operation
kis-flow copied to clipboard

Support DAG glow

Open hanxuanliang opened this issue 1 year ago • 2 comments

Will the kis-flow support DAG scheduling later?

pipeline:
    - depend_on: flow1

like the above config

hanxuanliang avatar Feb 28 '24 03:02 hanxuanliang

The current Kis-flow does not yet support DAG (Directed Acyclic Graph) scheduling, but we can perform parallel streaming operations.

For example, in cases like combining flow1 and flow2 to generate flow3:

We can have FunctionS and FunctionL nodes, where FunctionS acts as a storage node and FunctionL serves as a read/write node. These nodes can link the two flows together. Alternatively, we can have flow1's result sent to a message queue, and then flow2 can use this as a data source, initiating another flow to execute.

For configurations like:

pipeline:
    - depend_on: flow1
like the above config

This configuration format is not yet supported. However, it's a very good suggestion. The approaches mentioned for Kis-flow can be adapted to a configuration-based approach.

aceld avatar Mar 01 '24 05:03 aceld

Maybe DAG is a little more common. It may also be necessary to introduce the docker runtime environment

hanxuanliang avatar Mar 05 '24 01:03 hanxuanliang