Avoid to start pipelines that we don’t need to run

[afrittoli]

Today every CI pipeline includes two shared tasks:

• one to handle the /test <job-name> command
• one to inspect the files changed in a PR against a filter defined in the CI job

The CI event listener has two group interceptors (pull and comment) with shared logic. Every repo repo has then two triggers (pull and comment) and one trigger template, which includes the list of CI jobs to execute.

This works fine, but it means that all CI jobs in the template are always started. If any of the checks fail, the end quickly. Ideally we would like to not start CI jobs at all if they do not need to run, to consume fewer resources, as all CI jobs will allocated a PVC and clone the repo just to perform the checks.

A few options:

• Change the files changed task to use the GitHub API instead of files on disk. Use Pipeline in a Pod when available. This way even if jobs are executed, the footprint is much smaller
• Move checks to a trigger interceptor and avoid starting pipelines that we don't need
  • this would require having a trigger + template for every pipeline
  • this would require moving the file filter from the pipeline definition into the pipeline specific trigger
• Use a Pipeline to aggregate the CI jobs, instead of a trigger template.
  • Use Pipeline in Pipeline if some CI job requires more than Task. We could start using CI jobs as they are today and then factor up common parts like cloning a repo, build artefacts
• Introduce a config file with a list of CI jobs
  • Use a pipeline with matrix to run all CI jobs configured in the config map
  • The config file would be handy for other things like setting required jobs in GitHub, sync GitHub notification, send GitHub comments etc
• Introduce PipelineRun level when expressions, so that if a certain input matches, the Pipeline is not started at all.

/cc @vdemeester @abayer @dibyom @lbernick @jerop

[tekton-robot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale with a justification. Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with /close with a justification. If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[afrittoli]

/remove-lifecycle stale

[afrittoli]

/lifecycle frozen