shore
shore copied to clipboard
Parallelize Nested Pipeline save
Current implementation of the nested pipeline save uses a naive DFS implementation.
Meaning, each new pipeline stage & pipeline is saved sequentially.
For highly nested pipelines or pipelines with many stages this sequential save could cause performance issues.
To mitigate this problem, we should use a BFS approach and save pipelines siblings concurrently. This will mitigate the performance problems to some extent.
In addition, it might be wise to add logging in case of very nested pipelines (Depth > 3) to notify the developers of a possible issue.