Update attempts when image is already up to date when no container exist

[jimgus]

Describe the bug I have seen this issue for deployments that are scaled down to 0 replica by Keda and for CronJobs when no Job is running in combination with force-update=true. The image update part works when there is a new image available but it continuously tries to update it over and over again even though it is already up to date. I have not seen this issue with other applications with the same config that don't scale down to 0 so I believe this happens that there is no container up and running and force-update=true.

To Reproduce Use a CronJob and set force-update=true

Expected behavior The image update attempts shall only happen when the image is not up to date.

Additional context We are using Helm to deploy deployments and CronJobs

Version 0.14.0

Logs Update attempt when image already is up to date:

2024-08-13 11:26:30.325	
body=time="2024-08-13T09:26:30Z" level=info msg="Processing results: applications=10 images_considered=10 images_skipped=0 images_updated=1 errors=0"
2024-08-13 11:26:30.319	
body=time="2024-08-13T09:26:30Z" level=info msg="Successfully updated the live application spec" application=xxx-report
2024-08-13 11:26:30.268	
body=time="2024-08-13T09:26:30Z" level=info msg=Trace args="[git clean -ffdx]" dir=/tmp/git-xxx-report1109256959 operation_name="exec git" time_ms=13.148676
2024-08-13 11:26:30.255	
body=time="2024-08-13T09:26:30Z" level=info msg="git clean -ffdx" dir=/tmp/git-xxx-report1109256959 execID=da92d
2024-08-13 11:26:30.255	
body=time="2024-08-13T09:26:30Z" level=info msg=Trace args="[git checkout --force master]" dir=/tmp/git-xxx-report1109256959 operation_name="exec git" time_ms=213.936409
2024-08-13 11:26:30.040	
body=time="2024-08-13T09:26:30Z" level=info msg="git checkout --force master" dir=/tmp/git-xxx-report1109256959 execID=0292b
2024-08-13 11:26:30.040	
body=time="2024-08-13T09:26:30Z" level=info msg=Trace args="[git config user.email [email protected]]" dir=/tmp/git-xxx-report1109256959 operation_name="exec git" time_ms=1.514835
2024-08-13 11:26:30.039	
body=time="2024-08-13T09:26:30Z" level=info msg="git config user.email [email protected]" dir=/tmp/git-xxx-report1109256959 execID=d0dc8
2024-08-13 11:26:30.039	
body=time="2024-08-13T09:26:30Z" level=info msg=Trace args="[git config user.name argocd-image-updater]" dir=/tmp/git-xxx-report1109256959 operation_name="exec git" time_ms=1.5754059999999999
2024-08-13 11:26:30.037	
body=time="2024-08-13T09:26:30Z" level=info msg="git config user.name argocd-image-updater" dir=/tmp/git-xxx-report1109256959 execID=c8200
2024-08-13 11:26:30.037	
body=time="2024-08-13T09:26:30Z" level=info msg=Trace args="[git fetch origin --tags --force --prune]" dir=/tmp/git-xxx-report1109256959 operation_name="exec git" time_ms=8646.557649
2024-08-13 11:26:21.391	
body=time="2024-08-13T09:26:21Z" level=info msg="git fetch origin --tags --force --prune" dir=/tmp/git-xxx-report1109256959 execID=55f76
2024-08-13 11:26:21.390	
body=time="2024-08-13T09:26:21Z" level=warning msg="temporarily disabling strict host key checking (i.e. '-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null'), please don't use in production"
2024-08-13 11:26:21.390	
body=time="2024-08-13T09:26:21Z" level=info msg="Initializing [email protected]:annotell/annotell-cd.git to /tmp/git-xxx-report1109256959"
2024-08-13 11:26:21.385	
body=time="2024-08-13T09:26:21Z" level=info msg="Committing 1 parameter update(s) for application xxx-report" application=xxx-report
2024-08-13 11:26:21.385	
body=time="2024-08-13T09:26:21Z" level=info msg="Successfully updated image 'eu.gcr.io/repo/zzz-cronjobs' to 'eu.gcr.io/repo/zzz-cronjobs:master.20240807T0752.867e52aac7a88db533c4daaa7b58795b0dd14d6e', but pending spec update (dry run=false)" alias=yyy-zzz-cronjobs application=xxx-report image_name=repo/zzz-cronjobs image_tag= registry=eu.gcr.io
2024-08-13 11:26:21.385	
body=time="2024-08-13T09:26:21Z" level=info msg="Setting new image to eu.gcr.io/repo/zzz-cronjobs:master.20240807T0752.867e52aac7a88db533c4daaa7b58795b0dd14d6e" alias=yyy-zzz-cronjobs application=xxx-report image_name=repo/zzz-cronjobs image_tag= registry=eu.gcr.io
2024-08-13 11:26:21.269	
body=time="2024-08-13T09:26:21Z" level=info msg="Starting image update cycle, considering 10 annotated application(s) for update"

[chengfang]

Looking at https://github.com/argoproj-labs/argocd-image-updater/blob/0ad5b94ff483bd2045e7adb3f54b1ca4c700d3da/pkg/argocd/argocd.go#L452 , image-updater doesn't know the current image version from image-list value, since its tag part is just a version constraint, not a valid tag value. The update is done to honor the force-update setting, without knowing the current version.

One workaround is to set force-update to false for those images used in cronjobs and jobs.

[crabique]

This is a problem even without force-update as ArgoCD Application resource exports all images owned by the managed resources, so for a cronjob this means:

• argocd-image-updater sees the exported old image tag
• an update is pushed to git
• ArgoCD reconciles the image value, setting it in the CronJob object
• the exported image tag at this point is still the old one, if a new image update cycle starts during that time it will try to update it again
• the CronJob finally spawns a new Job, which creates a new Pod with the new image tag
• there are now 2 images in ArgoCD Application objects, and the old one will stay there until all jobs have been rotated out due to retention limits
• argocd-image-updater will attempt to update the image for as long as the old image is present in the list reported by the Application

---

I think the proper solution here would be to introduce some "non-greedy" annotation that would be satisfied seeing the updated image in the list exported by ArgoCD Application, while the current mode (greedy/eager) stays the default.

[chansuke]

I am facing same problem. I'd like to work on this.