TEP-0056: Pipelines in Pipelines. Progress and ideas tracking as a living document.

[twoGiants]

Summary

Progress tracking on TEP-0056 including resources to study, advice, ideas and pull requests.

[!Note] Comments, suggestions and discussions are welcome.

Working Branch

This is the branch in my fork with all the changes and a PR to have an overview of everything:

• TEP-0056: PipelineSpec happy path working branch.
• 🥳 PR in fork with all changes for overview. 🥳

Pull Requests

• My PRs:
  • Fixing typos and improving names:
    • #8806
  • Extract common status testing logic:
    • #8850
  • Pipeline-in-Pipeline task resolution:
    • #8873
  • Pipeline-in-Pipeline state updates:
    • #8878
  • Pipeline-in-Pipeline reconciliation:
    • #8879
  • Fixing typos and removing skipped tests:
    • #8874
• Other PRs:
  • API changes, PipelineRef + PipelineSpec, adds validation and documentation:
    • #7055.
  • Docs update, more explicit that its not yet implemented:
    • #8767
  • "Additional Options" docs update:
    • #7904

Tasks

• [x] study first PR #7055
• [x] study first experimental implementation which uses a controller and the CustomTask feature >> all clear: summary
• [x] study second experimental implementation which executes the pipeline in a TaskRun >> all clear: summary
• [x] write down first implementation ideas
• [x] re-read controller-logic.md with its links
  • [x] look into Kubebuilder getting started tutorial >> very good: sample controller, tests, README.md
  • [x] watch video on building reconciler with Knative >> ok, to see the pieces
  • [x] read the links in Generated Code >> partly clear
• [x] study current implementation of TaskRun reconciler >> all clear: detailed summary
  • [x] study TaskRun reconciler video
  • [x] study actual TaskRun code
  • [x] summarize functionality with code snippets
• [x] study the code of the CustomTask feature >> partly clear: summary
  • [x] find related PRs
  • [x] study actual code in PipelineRun reconciler
• [x] finish first draft: focus on existing code base, experimental is outdated
• [x] implementation e2e happy path PipelineSpec
  • [x] implement PIP pipeline resolution
  • [x] implement PIP pipeline creation
  • [x] implement getting PIP pipeline status
  • [x] implement parent PipelineRun stop on child success
  • [x] unit tests
  • [x] e2e tests
• [x] open pull requests
• [ ] docs
• [ ] refactorings, code improvements, new issues <== you are here
  • [x] refactor new e2e tests
  • [ ] create issue for: using polymorphism to streamline reconciliation of child pipelines, custom tasks, and regular tasks, eliminating extensive switch statements
  • [ ] create issue for: test refactorings, see comment on #8556
• [ ] resolve/remove your TODOs left in the code

Next Tasks

Will be handled when the next TEP-0056 features will be implemented:

• [ ] implement results propagation

Implementation

General theme: prefer minimal e2e implementation and expand on that over separate steps which are connected together at the end.

✅ Option 1: Extend PipelineRun reconciler.

Reuse existing logic in PipelineRun reconciler by extending it and using its methods and functions. Basically add conditions and methods everywhere which handle a PipelineTask of the type Pipeline:

• identify in PipelineRef || PipelineSpec in PipelineRun reconciler
• propagate params, perform validations, ... same as for TaskRun and CustomRun
• collect status via informer, ... same as for TaskRun and CustomRun
• resolve embedded pipeline in PipelineTask in resources.ResolvePipelineTask, ... same as for TaskRun and CustomRun
• call createPipelineRuns in runNextSchedulableTask, ... same as for TaskRun and CustomRun

Pro	Con
consistent with Task / CustomTask handling	adds complexity to existing reconciler
less operational overhead
leverages existing logic

🚫 Option 2: Delegate to PipelineInPipeline reconciler.

Similar to First Experimental Implementation create a piprun.go reconciler:

• PipelineRun reconciler creates a new CRD, e.g. PipelinesInPipelinesRun which is picked up by the pip reconciler
• new reconciler performs params, validations, substitutions, etc.
• collect status via informer
• ... >> don't like the idea, feels like a lot of duplicated logic

Pro	Con
maybe less complexity than Opt. 1	still adds complexity to existing reconciler
PipelineRun reconciler cleanup	shared logic must be extracted
	more operational overhead

Links

• TEP-0056 Proposal
• Original tracking issue #7166.

/assign

/kind feature

[khrm]

Some of the components which are effected by this:

1. CLI
2. UI
3. Chains
4. Pruner
5. Results
6. PipelinesAsCode
7. Maybe, we might also need to improve observability around this.

Before going beta, we need to have access need for the support of Pipelines in Pipelines within these components.

[waveywaves]

@twoGiants how would timeouts be affected when running multiple pipelineruns in a chain ? I think we need more thought on this and maybe the etcd and memory footprint as well of a single pipelinerun when it chains multiple pipelines.

[twoGiants]

Some of the components which are effected by this: ... Before going beta, we need to have access need for the support of Pipelines in Pipelines within these components.

Yeah, it's a big feature. Once the MVP of it is implemented it makes sense to start looking into what needs to be done in the other components. Wdys?

I think the MVP should contain PipelineRef, parameters and workspaces and maybe service accounts.

@waveywaves the timeouts would work the same like for the TaskRuns, like in the proposal section Timeouts. And as for the memory footprint, not sure why this should be an issue, every Pipeline in Pipeline is just another PipelineRun in the cluster. Or what do you mean exactly?

[vdemeester]

@twoGiants I think now it's fully implemented isn't it ?