spinnaker
feat(pipelineRef): Replace PipelineTrigger with PipelineRef for Spinnaker UI
Feature: Performance Optimization for Nested Pipelines in Spinnaker Orca In Spinnaker, pipelines with many nested child pipelines (e.g., Pipeline → Pipeline → Pipeline → Pipeline) can cause significant delays in loading times, particularly for large instances. This is because the Spinnaker UI often needs to load parent executions for any pipeline that was triggered by another, which unnecessarily increases the load on the Orca execution repository.
Solution:
To address this issue, I’ve introduced an optimization using the PipelineRef feature that was previously delivered. The idea is to propagate the PipelineRef trigger instead of the full PipelineTrigger for nested pipelines. This reduces the load on Orca and improves UI performance.
How It Works:
ExecutionRepository Update:
I added a new flag, includeNestedExecutions, to the retrieve method of the ExecutionRepository interface.
Default Behavior: Existing execution repository implementations don’t need to change.
- The default behavior remains to use the normal retrieve method.
SqlExecutionRepository Logic:
The only repository that requires an update is the SqlExecutionRepository, where the business logic for includeNestedExecutions is implemented.
-
If includeNestedExecutions is true, Orca will convert any
PipelineRefTriggerinto aPipelineTriggerto ensure nested executions are returned. -
If the flag is false, Orca will return executions with the
PipelineRefTrigger.
SPeL Compatibility:
To maintain backward compatibility with Spinnaker’s SPeL, the OrcaMessageHandler ensures that nested executions are included when evaluating any expressions. This guarantees that SPeL expressions still resolve correctly.
SPeLAutoComplete Compatibility:
The SPeLAutoComplete feature continues to work as expected. The endpoint that retrieves previous executions to feed the autocomplete is still retrieving executions with the full execution context, including nested executions. This ensures that the autocomplete functionality is unaffected by the performance improvement.
Summary: The main idea is to minimize unnecessary load on Orca by returning a PipelineRefTrigger for external requests, while keeping full execution context for internal modules that need it. This change significantly reduces the pressure on the Orca execution repository and improves UI performance, especially in large Spinnaker instances.
This performance improvement is backward compatible, and SPeLAutoComplete continues to function as expected, with no disruption to existing Spinnaker functionality.
How to enable it
executionRepository:
redis:
enabled: false
sql:
enabled: true
pipelineRef:
enabled: true
Here is an example on how the Spinnaker UI looks like when a pipeline is triggered by another pipeline:
Here is another example on how the execution looks like if we inspect it in the Spinnaker UI:
Finally here is another example that probes the SPeLAutoComplete still works by converting PipelineRefTrigger to PipelineTrigger and existing SPeLs still works as expected:
@dbyron-sf Since the 1.37.x has been released do you think we can start the review process for this?
@christosarvanitis Thanks for the review. I tried to put this improvement behind a feature flag to allow for easy disabling or rollback in case of issues. However, since my changes update functionality in orca-core, fully encapsulating them is challenging but not impossible. Doing so would require additional changes to my PR that are not directly related to this improvement. Additionally, my changes are already substantial, making the review process more complex. :(
However, I remember I hear there is a initiative to make more easy to write experimental code behind a feature flag but not sure if we can use that for this or what is the state for that work
