Mixed updated images in kustomization.yaml

[yogeek]

Describe the bug Image-updater mixed up images in its commits : it updated images from other applications and sometimes even removed kustomization patches.

To Reproduce Steps to reproduce the behavior:

• each microservice from https://github.com/GoogleCloudPlatform/microservices-demo has been extracted into its own gitlab repository
• all images have been mirrored into a private harbor registry
• kustomize is used to add some resources (hpa, istio gateway,...), patch some others and to replace original image by private registry one
• the master branch is used to store yaml manifests and kustomization file, and a Jenkins pipeline is launched at every modification to generate "final" YAML manifests into a "state" branch (this is the one synched by Argo Applications)
• an ApplicationSet has been created to generate 1 Application per repository

ApplicationSet (click to expand)

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: microdemo
  namespace: argocd
spec:
  generators:
  - scmProvider:
      cloneProtocol: ssh
      gitlab:
        group: "gtp/microdemo/cd"
        api: https://<GITLAB_URL>/
        allBranches: true
        includeSubgroups: true
        tokenRef:
          secretName: gitlab-appset-token
          key: token
      filters:
        - branchMatch: ^state
          pathsExist: [${K8S_ENVIRONMENT}/${K8S_REGION}/${K8S_CLASSIFICATION}]
  template:
    metadata:
      name: '{{ repository }}'
      annotations:
        #------------ ArgoCD Notifications config
        # https://argocd-notifications.readthedocs.io/en/stable/subscriptions/
        # send to slack if sync failed
        notifications.argoproj.io/subscribe.on-sync-failed.slack: gtp_cicd

        #------------ ArgoCD Image Updater config
        # https://argocd-image-updater.readthedocs.io/en/stable/configuration/images/#configuring-images-for-update
        argocd-image-updater.argoproj.io/image-list: server=${DOCKER_REGISTRY_HOSTNAME}/gtp/google-samples/microservices-demo/{{ repository }}
        argocd-image-updater.argoproj.io/server.update-strategy: semver
        argocd-image-updater.argoproj.io/server.allow-tags: regexp:^v0.3.[0-9]+$
        argocd-image-updater.argoproj.io/write-back-method: git:secret:argocd/git-creds
        # By default, image-updater is committing to the branch specified in the '.spec.source.targetRevision' of the Application spec.
        # But we want to commit to master to keep coherence between master and state 
        argocd-image-updater.argoproj.io/git-branch: master
        # WARNING: the image updater will only work if a "kustomization.yaml" is present also in the state branch (even if this it not the one that is used to generate manifests)
        # Path of the kustomization.yaml to commit to in master git-branch
        argocd-image-updater.argoproj.io/write-back-target: "kustomization:/base"
    spec:
      project: "microdemo"
      source:
        repoURL: '{{ url }}'
        targetRevision: '{{ branch }}'
        path: ${K8S_ENVIRONMENT}/${K8S_REGION}/${K8S_CLASSIFICATION}
      destination:
        server: https://kubernetes.default.svc
        namespace: 'gtp-microdemo'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
        - CreateNamespace=true

• deploy first with the 0.2.4 version
• then push the 0.3.0
• wait for image updater to detect the new version
• at first, is was not working because my "state" branch (synched by argo applications) did not contain a kustomization.yaml file (I guess because the documentation mentions that only helm and kustomize application are recognized by image updater...) so I added a minimal one just for image updater to work
• some services are updated successfully
• but for some others, there seems to be a mix up between images as shown below

for example, the frontend service has been updated correctly first :

but then, it has been updated again with another image, and image-updater has also removed some of my patches from the kustomization file :

Sme thing for the productcatalog service :

Expected behavior I expect that image updater only updates the images corresponding to the deployment of the current reporitory.

Additional context

Not sure if relevant but some errors appear regularly in image-updater logs as explained in Slack (cf. https://github.com/argoproj-labs/argocd-image-updater/issues/240#issuecomment-938748437)

Version argoprojlabs/argocd-image-updater:latest

Logs I do not have the logs from that period anymore...

[oleg-glushak]

Same here. Image updater remove the whole generators: section from the kustomize.yaml while setting a new image.

-generators:
-- secret-generator.yaml
-
 namespace: flink-staging-01
 images:
-- digest: sha256:52fefde3e903f8b0decc03b65bd68fd215fcaa18c1916c5d18c63b3b91362aeb
+- digest: sha256:635f23be44ab6a13d73097d66f82df7da1dcb78a114799e514b676792671cabd

Set of used annotations:

        argocd-image-updater.argoproj.io/image-list: app=<mycontainer>
        argocd-image-updater.argoproj.io/app.update-strategy: digest
        argocd-image-updater.argoproj.io/credsexpire: 5h
        argocd-image-updater.argoproj.io/write-back-method: git
        argocd-image-updater.argoproj.io/write-back-target: kustomization