Update of Included Git Repository doesn't trigger Reconciliation

[jceb]

Version:

• flux: v0.24.0
• helm-controller: v0.14.0
• kustomize-controller: v0.18.1
• notification-controller: v0.19.0
• source-controller: v0.19.0

I noticed that an update of a Git Repository that's included in another one (https://fluxcd.io/docs/components/source/gitrepositories/#including-gitrepository) doesn't trigger an update of the including repository and subsequently also not the reconciliation of the kustomization.

I'd like to use a webhook to trigger a reconciliation chain for every change in a repository as described here. The following configuration is applied in the cluster:

Receiver:

apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Receiver
metadata:
  name: gitlab-receiver
  namespace: flux-system
spec:
  events:
  - Push Hook
  resources:
  - apiVersion: source.toolkit.fluxcd.io/v1beta1
    kind: GitRepository
    name: helm-charts
  - apiVersion: source.toolkit.fluxcd.io/v1beta1
    kind: GitRepository
    name: chart-values
  - apiVersion: source.toolkit.fluxcd.io/v1beta1
    kind: GitRepository
    name: flux-system
  secretRef:
    name: gitlab-webhook-token
  type: gitlab

3 repositories: the chart-values repository is included in the helm-charts repository and provides additional values to the helm charts in this particular cluster:

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: helm-charts
  namespace: flux-system
spec:
  include:
  - fromPath: /dev
    repository:
      name: chart-values
    toPath: /values
  interval: 10m
  ref:
    branch: main
  secretRef:
    name: credentials
  url: ssh://[email protected]/.../helm-charts.git

When a webhook is triggered for the chart-values repository, the following happens - no kustomization is triggered and therefore the changes aren't applied:

2021-12-13T14:41:24.281Z info  - handling request: 932e27392a7a9251055428ff63c02e08636ecc961048f9c22e95020b7d3ef51e
2021-12-13T14:41:24.281Z info Receiver/gitlab-receiver.flux-system - handling GitLab event: Push Hook
2021-12-13T14:41:24.314Z info Receiver/gitlab-receiver.flux-system - resource 'GitRepository/chart-values.' annotated
2021-12-13T14:41:24.336Z info GitRepository/helm-charts.flux-system - All dependencies area ready, proceeding with reconciliation
2021-12-13T14:41:24.338Z info Receiver/gitlab-receiver.flux-system - resource 'GitRepository/helm-charts.' annotated
2021-12-13T14:41:24.360Z info Receiver/gitlab-receiver.flux-system - resource 'GitRepository/flux-system.' annotated
2021-12-13T14:41:24.727Z info GitRepository/helm-charts.flux-system - Reconciliation finished in 390.975721ms, next run in 10m0s
2021-12-13T14:41:25.119Z info GitRepository/flux-system.flux-system - Reconciliation finished in 391.698925ms, next run in 3m0s
2021-12-13T14:41:25.142Z info GitRepository/chart-values.flux-system - Reconciliation finished in 828.708696ms, next run in 10m0s
2021-12-13T14:41:25.142Z info GitRepository/chart-values.flux-system - Dispatching event: Fetched revision: master/88c6f1508da72cb25388614dd647a105b4c1e66b

The desired behavior is shown when a webhook triggers a reconciliation for the not-included flux-system repository - the kustomization is properly triggered after the repository has been updated:

2021-12-13T14:40:08.163Z info  - handling request: 932e27392a7a9251055428ff63c02e08636ecc961048f9c22e95020b7d3ef51e
2021-12-13T14:40:08.163Z info Receiver/gitlab-receiver.flux-system - handling GitLab event: Push Hook
2021-12-13T14:40:08.194Z info Receiver/gitlab-receiver.flux-system - resource 'GitRepository/chart-values.' annotated
2021-12-13T14:40:08.224Z info Receiver/gitlab-receiver.flux-system - resource 'GitRepository/flux-system.' annotated
2021-12-13T14:40:08.247Z info Receiver/gitlab-receiver.flux-system - resource 'GitRepository/helm-charts.' annotated
2021-12-13T14:40:08.744Z info GitRepository/flux-system.flux-system - Reconciliation finished in 521.284147ms, next run in 3m0s
2021-12-13T14:40:08.744Z info GitRepository/helm-charts.flux-system - All dependencies area ready, proceeding with reconciliation
2021-12-13T14:40:08.745Z info GitRepository/flux-system.flux-system - Dispatching event: Fetched revision: master/461dcd92f98fdd11c9f05172b53047f2815c6243
2021-12-13T14:40:09.030Z info GitRepository/chart-values.flux-system - Reconciliation finished in 837.729476ms, next run in 10m0s
2021-12-13T14:40:09.438Z info Kustomization/flux-system.flux-system - server-side apply completed
2021-12-13T14:40:09.438Z info Receiver/gitlab-receiver.flux-system - Receiver initialized
2021-12-13T14:40:09.451Z info Kustomization/flux-system.flux-system - Dispatching event: Receiver/flux-system/gitlab-receiver configured
2021-12-13T14:40:09.452Z info Receiver/gitlab-receiver.flux-system - Receiver initialized
2021-12-13T14:40:09.465Z info Kustomization/flux-system.flux-system - Reconciliation finished in 732.82896ms, next run in 10m0s
2021-12-13T14:40:09.466Z info Kustomization/flux-system.flux-system - Dispatching event: Reconciliation finished in 732.82896ms, next run in 10m0s
2021-12-13T14:40:09.469Z info GitRepository/helm-charts.flux-system - Dispatching event: Fetched revision: master/461dcd92f98fdd11c9f05172b53047f2815c6243
2021-12-13T14:40:09.468Z info GitRepository/helm-charts.flux-system - Reconciliation finished in 723.408451ms, next run in 10m0s

[danfromtitan]

+1

[danfromtitan]

Turns out my issue was with one of the inner git repo manifests. The sequence of troubleshoot this was:

• check flux picked up on the latest git changes in the inner repo: kubectl get gitrepositories.source.toolkit.fluxcd.io
• check the reconcile status of helmreleases or kustomizations coming from the inner repo: kubectl get kustomizations.kustomize.toolkit.fluxcd.io

That should reveal the reason for the failure to apply changes and provide the meaning for further investigation.

[pizzqc]

I am having the same issue. Recently I started to use a GitRepository that uses included repository and when the webhook receiver update the included repo, the including one isn't updated automatically. It needs to be reconciled manually.

But the chain reaction doesn't end there. The kustomization using that git repository will not trigger automatically because the "main" repository didn't update only the included repository did. So we also need to manually force a reconciliation on the kustomization after to get it executed and apply the changes

[earl-warren]

Same issue here on flux version 2.4.0 and k3s version v1.30.5+k3s1 (9b586704). Attached are the description of two GitRepositories, one included in the other.

• one.txt
• two.txt

---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: forgejo-next-digest
  namespace: flux-system
spec:
  interval: 15m
  ref:
    branch: main
  url: https://code.forgejo.org/infrastructure/next-digest.git
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 15m
  ref:
    branch: main
  url: https://code.forgejo.org/infrastructure/k8s-cluster.git
  # https://fluxcd.io/flux/components/source/gitrepositories/#include
  include:
    - repository:
        name: forgejo-next-digest
      toPath: flux/apps/digest