Git commit fails when auto-sync is disabled in ArgoCD

[janpieper]

Describe the bug

We set up an alert in Prometheus based on the argocd_image_updater_images_errors_total metric to be informed in case there is a problem with any image update. For some applications, especially for prod environments, we disable auto-sync in ArgoCD.

So in case someone pushes a new image to the docker registry, the ArgoCD Image Updater updates the kustomization.yaml to use the newly pushed image. That's fine. But if your application is configured to not auto-sync these changes, the ArgoCD Image Updater will try to update the kustomization.yaml over and over again. This fails, because it cannot commit the change to the kustomization.yaml because git does not detect any changes.

This problem causes the argocd_image_updater_images_errors_total to increase which then causes an alert in our alerting system although it is a false-positive in this case.

To Reproduce

1. Disable auto-sync in ArgoCD for an application that uses the ArgoCD Image Updater
2. Push a new image for that application to the docker registry
3. Do NOT sync the application manually!
4. Wait until the ArgoCD Image Updater runs the next time

Expected behavior

In theory, the ArgoCD Image Update should be fine with not updating the kustomization.yaml, but only in case the change has already been committed before.

Additional context

None

Version

0.11.0

Logs

time="2022-01-31T15:54:57Z" level=info msg="Starting image update cycle, considering 12 annotated application(s) for update"
time="2022-01-31T15:54:57Z" level=info msg="Setting new image to example.azurecr.io/git-example-frontend:v3.2.0" alias=frontend application=example-frontend image_name=git-example-frontend image_tag=v3.1.0 registry=example.azurecr.io
time="2022-01-31T15:54:57Z" level=info msg="Successfully updated image 'example.azurecr.io/git-example-frontend:v3.1.0' to 'example.azurecr.io/git-example-frontend:v3.2.0', but pending spec update (dry run=false)" alias=frontend application=example-frontend image_name=git-example-frontend image_tag=v3.1.0 registry=example.azurecr.io
time="2022-01-31T15:54:57Z" level=info msg="Committing 1 parameter update(s) for application example-frontend" application=example-frontend
time="2022-01-31T15:54:57Z" level=info msg="Starting configmap/secret informers"
time="2022-01-31T15:54:57Z" level=info msg="Configmap/secret informer synced"
time="2022-01-31T15:54:57Z" level=info msg="Initializing ssh://[email protected]/example-org/example.git to /tmp/git-example-frontend568615438"
time="2022-01-31T15:54:57Z" level=info msg="secrets informer cancelled"
time="2022-01-31T15:54:57Z" level=info msg="rm -rf /tmp/git-example-frontend568615438" dir= execID=CvPm3
time="2022-01-31T15:54:57Z" level=info msg="configmap informer cancelled"
time="2022-01-31T15:54:57Z" level=info msg=Trace args="[rm -rf /tmp/git-example-frontend568615438]" dir= operation_name="exec rm" time_ms=5.9614519999999995
time="2022-01-31T15:54:57Z" level=info msg="git fetch origin --tags --force" dir=/tmp/git-example-frontend568615438 execID=gFVJi
time="2022-01-31T15:55:02Z" level=info msg=Trace args="[git fetch origin --tags --force]" dir=/tmp/git-example-frontend568615438 operation_name="exec git" time_ms=4870.133201
time="2022-01-31T15:55:02Z" level=info msg="git config user.name argocd-image-updater" dir=/tmp/git-example-frontend568615438 execID=TdORS
time="2022-01-31T15:55:02Z" level=info msg=Trace args="[git config user.name argocd-image-updater]" dir=/tmp/git-example-frontend568615438 operation_name="exec git" time_ms=1.366011
time="2022-01-31T15:55:02Z" level=info msg="git config user.email [email protected]" dir=/tmp/git-example-frontend568615438 execID=OzOxV
time="2022-01-31T15:55:02Z" level=info msg=Trace args="[git config user.email [email protected]]" dir=/tmp/git-example-frontend568615438 operation_name="exec git" time_ms=1.2873109999999999
time="2022-01-31T15:55:02Z" level=info msg="git checkout --force main" dir=/tmp/git-example-frontend568615438 execID=YwsXY
time="2022-01-31T15:55:02Z" level=info msg=Trace args="[git checkout --force main]" dir=/tmp/git-example-frontend568615438 operation_name="exec git" time_ms=312.668819
time="2022-01-31T15:55:02Z" level=info msg="git clean -fdx" dir=/tmp/git-example-frontend568615438 execID=RPIud
time="2022-01-31T15:55:02Z" level=info msg=Trace args="[git clean -fdx]" dir=/tmp/git-example-frontend568615438 operation_name="exec git" time_ms=16.529333
time="2022-01-31T15:55:02Z" level=info msg="updating base /tmp/git-example-frontend568615438/example/frontend"
time="2022-01-31T15:55:02Z" level=info msg="git commit -a -F /tmp/image-updater-commit-msg971480176" dir=/tmp/git-example-frontend568615438 execID=PQdAM
time="2022-01-31T15:55:02Z" level=error msg="`git commit -a -F /tmp/image-updater-commit-msg971480176` failed exit status 1" execID=PQdAM
time="2022-01-31T15:55:02Z" level=info msg=Trace args="[git commit -a -F /tmp/image-updater-commit-msg971480176]" dir=/tmp/git-example-frontend568615438 operation_name="exec git" time_ms=162.115006
time="2022-01-31T15:55:02Z" level=error msg="On branch main\nYour branch is up to date with 'origin/main'.\n\nnothing to commit, working tree clean"
time="2022-01-31T15:55:02Z" level=info msg="Processing results: applications=12 images_considered=11 images_skipped=4 images_updated=0 errors=1"
time="2022-01-31T15:55:02Z" level=error msg="Could not update application spec: `git commit -a -F /tmp/image-updater-commit-msg971480176` failed exit status 1" application=example-frontend

[jannfis]

Oh, I agree, this should be fixed. For the write-back of .argocd-source-<app>.yaml, we actually check whether there are differences between the existing and generated file and only perform the commit when there are changes.

Code can be found here: https://github.com/argoproj-labs/argocd-image-updater/blob/e46a206038282c41bdbab06fce3f84ce9fe33173/pkg/argocd/git.go#L271-L280

We should probably do something similar for kustomization write-back.

[tesharp]

Have the same problem, enabled auto sync just to get around it for now. However this issue also cause another problem. It seems the image list ArgoCD keeps for an application includes older images as well if it is a cronjob for instance. We have cronjobs that only run once a day and even though git kustomize file has correct version and the manifest for cronjob is correct the images list on ArgoCD application lists an older version. Probably it uses the image version on last run pod or something..

This causes image updater to always try to update the image and failing because there is actually nothing to commit. Fixing this problem would at least make it not fail, but it will continue to say that there is an update to the image as long as ArgoCD application is listing older versions and it compares to those.

For kustomize could it check for updates against the image defined in kustomize file?

[jannfis]

For kustomize could it check for updates against the image defined in kustomize file?

The long term fix will be to migrate to a dedicated API (e.g. make argocd-image-updater a real controller, using a dedicated CRD) and store such information in the CR's .status field I guess :)

[mfmsky]

Hello, sorry for the bump, but i also found this bug in 0.12.0 of ArgoCD Image Updater. is there any work around it?