shore icon indicating copy to clipboard operation
shore copied to clipboard

Parallelize Nested Pipeline save

Open eyal-mor opened this issue 1 year ago • 0 comments

Current implementation of the nested pipeline save uses a naive DFS implementation.

Meaning, each new pipeline stage & pipeline is saved sequentially.

For highly nested pipelines or pipelines with many stages this sequential save could cause performance issues.

To mitigate this problem, we should use a BFS approach and save pipelines siblings concurrently. This will mitigate the performance problems to some extent.

In addition, it might be wise to add logging in case of very nested pipelines (Depth > 3) to notify the developers of a possible issue.

eyal-mor avatar Feb 27 '23 16:02 eyal-mor