Polling a repository to detect changes and trigger a pipeline

[sm43]

To Poll a repository at a certain interval to look for any changes and trigger a pipelinerun/taskrun.

Use cases:

• Some users might not have permission to setup a webhook so this could be helpful to trigger a pipelinerun/taskrun after some changes.
• another use case https://github.com/tektoncd/triggers/issues/480#issuecomment-620605920

This can be done using a cronjob but could to a part of triggers. Similar Issue: https://github.com/tektoncd/triggers/issues/480

Proposal: To have this feature as a part of triggers. I did a poc around this where I have a new SyncRepo CRD

apiVersion: triggers.tekton.dev/v1alpha1
kind: SyncRepo
metadata:
  name: test
spec:
  repo: https://github.com/tektoncd/hub
  branch: main
  frequency: 3m
  binding: pipeline-binding
  template: pipeline-template

Approach:

• This looks for changes on repository using GitHub Rest API and triggers a pipelines.
• The API returns the response in json format for example or https://api.github.com/repos/sm43/hub/commits/tekton-ci
• It uses trigger binding to get values from the json response and use them in the trigger template and triggers the pipelinerun.
• Later it saves the last commit on which it was triggered and use it in next reconcilation to look for changes.

Things can be improved:

• Support for private repositories
• Gitlab also have similar API which we can use
• We can also directly allow users to pass pipelineRef which can be directly used if a user doesn't want to use TriggerBinding/TriggerTemplate
• A user can provide multiple pipeline templates which can be triggered together

Available solution: There is an operator by Kevin https://github.com/bigkevmcd/tekton-polling-operator which polls a repo and create a pipelinerun/taskrun if something is changes. This is independent of TektonBinding and TektonTemplate, it take PipelineRef as input.

cc @vdemeester

[sm43]

@dibyom @wlynch @savitaashture @khrm let me know what do you think? :upside_down_face:

[vdemeester]

We should also probably support "any" git repositories (and not only github/gitlab/…).

[dibyom]

We discussed this at the last WG and decided that the next steps would be to open up a TEP with the problem statement/use cases so that we can gather more requirements/feedback for this feature.

[bigkevmcd]

I was thinking about this, part of the complexity of this is integrating with Triggers and TriggerTemplates, but maybe it shouldn't, and instead, when it sees a new commit, it should send the commit as a "hook" to an EventListener, it would be "internal" ?

[dibyom]

but maybe it shouldn't, and instead, when it sees a new commit, it should send the commit as a "hook" to an EventListener, it would be "internal" ?

that is definitely an approach we should consider!

[tekton-robot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale with a justification. Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with /close with a justification. If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tisaiful31]

To Poll a repository at a certain interval to look for any changes and trigger a pipelinerun/taskrun.

Use cases:

• Some users might not have permission to setup a webhook so this could be helpful to trigger a pipelinerun/taskrun after some changes.
• another use case Poll based change detection? #480 (comment)

This can be done using a cronjob but could to a part of triggers. Similar Issue: #480

Proposal: To have this feature as a part of triggers. I did a poc around this where I have a new SyncRepo CRD

apiVersion: triggers.tekton.dev/v1alpha1
kind: SyncRepo
metadata:
  name: test
spec:
  repo: https://github.com/tektoncd/hub
  branch: main
  frequency: 3m
  binding: pipeline-binding
  template: pipeline-template

Approach:

• This looks for changes on repository using GitHub Rest API and triggers a pipelines.
• The API returns the response in json format for example or https://api.github.com/repos/sm43/hub/commits/tekton-ci
• It uses trigger binding to get values from the json response and use them in the trigger template and triggers the pipelinerun.
• Later it saves the last commit on which it was triggered and use it in next reconcilation to look for changes.

Things can be improved:

• Support for private repositories
• Gitlab also have similar API which we can use
• We can also directly allow users to pass pipelineRef which can be directly used if a user doesn't want to use TriggerBinding/TriggerTemplate
• A user can provide multiple pipeline templates which can be triggered together

Available solution: There is an operator by Kevin https://github.com/bigkevmcd/tekton-polling-operator which polls a repo and create a pipelinerun/taskrun if something is changes. This is independent of TektonBinding and TektonTemplate, it take PipelineRef as input.

cc @vdemeester

found this error : " no matches for kind "SyncRepo" in version "triggers.tekton.dev/v1alpha1""

[vdemeester]

[…] found this error : " no matches for kind "SyncRepo" in version "triggers.tekton.dev/v1alpha1""

@tisaiful31 this is to be expected, what I wrote is just a proposal, that's still being discussed and thought trough.

[bigkevmcd]

@vdemeester I have working code that I'm literally just writing examples and docs for, that simplifies this to just a CR that detects changes in a ref in a git repository and sends a CloudEvent on change...not Tekton specific at all

[dibyom]

@bigkevmcd That sounds like a really nice solution to this

[triantium]

There is a implementation that works in combination with FluxCD https://github.com/jquad-group/pipeline-trigger-operator

They use the Flux Sourcecontroller to track changes.

[bigkevmcd]

@triantium that is a nice idea, it could of course be connected to triggers via flux notifications like this https://bigkevmcd.github.io/tekton/flux/notifications/2020/12/06/flux-tekton.html

[tekton-robot]

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten with a justification. Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with /close with a justification. If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[vdemeester]

/area roadmap

[Syntax3rror404]

Are there any news regarding this feature? We also cannot use webhooks because of a corporate proxy :/ This issue is now 2 years old ...