storage EFFICIENCY

[patricxu]

trafficstars

I have noticed FATE-flow divide a pipeline job to several component which will be executed as a task. However, each task is a subprocess of the job meaning that the intermediate outputs cannot be shared by memory between the upstream and downstream components. Instead, FATE-flow store the outputs on hard disc which incurs low efficiency.

Is there anyway to imporve this?