Multiple images update

[IvanKuchin]

Hello team,

Sorry for the long post, I tried to be as concise and as precise as possible.

Describe the bug Problem with multiple images update.

My setup: Annotation tracks list of 4 images: https://github.com/IvanKuchin/k8s-argocd/blob/60c1dd2a72a47e4b4bcd560e2f805d7563a15461/argocd/microk8s/connme.yaml#L7

Base image version configured in kustomize config is v1.1.14 https://github.com/IvanKuchin/k8s-argocd/blob/master/overlay/microk8s/connme/app/kustomization.yaml

New images with v1.1.15 has been posted to registry.

argocd-image-updater notice change in all versions, but git commit contains just single image. https://github.com/IvanKuchin/k8s-argocd/commit/60c1dd2a72a47e4b4bcd560e2f805d7563a15461

To Reproduce

• Track multiple images in argocd-image-updater.argoproj.io/image-list annotation
• Bump version up in all tracked images at once or consequently
• Check commit on github

Expected behavior Created git commit should contain all 4 changes, otherwise ArgoCD can't properly update spec.

Additional context I've tried to post new version of images to registry consequently, but image-updater still create commits by replacing old image to a new one. Here are examples:

• 1-st commit added xxx-app:v1.1.14 https://github.com/IvanKuchin/k8s-argocd/commit/e738619e2bc0f93a17f9650100e85ac2ffc87c93
• bump up version v1.1.14 -> v1.1.15
• 2-nd commit replaced xxx-app to v1.1.15 - OK (https://github.com/IvanKuchin/k8s-argocd/commit/d30edeefb804e2ee0c5da388a008ad33f7e2e136)
• 3-rd commit bump-up version of xxx-images to v1.1.15 , this one replaced xxx-app - NOT OK https://github.com/IvanKuchin/k8s-argocd/commit/17b9977c38a37304199463c83a35302f4f71733a At this point kustomize build . will see images version:
• xxx-images == v1.1.15 (OK)
• xxx-app == v1.1.14 (NOT OK) ArgoCD will downgrade that container to v1.1.14

Similar behavior for two other images. Summary: 3 images sits on v1.1.14 and image updater pending on ArgoCD to update spec (but it never happens) 1 image updated properly

Version

ikuchin@microk8s-test:~/k8s/k8s-argocd/overlay/microk8s/connme$ microk8s.kubectl exec -n argocd -it argocd-image-updater-7f85f5d75b-bnlzl  -- argocd-image-updater version
argocd-image-updater: v0.11.3+f62b7d7
  BuildDate: 2022-01-23T04:22:23Z
  GitCommit: f62b7d7841a42030fe4c75de1951c46b0e7fca2e
  GoVersion: go1.16.13
  GoCompiler: gc
  Platform: linux/amd64

Logs Debug level logs I'm not sure that those log from exact timestamp of the first update, but image-updater keep posting same logs over and over waiting for ArgoCD update, but due to ArgoCD sees 3 images on v1.1.14 and one image on v1.1.15 - it never bumps image version of other containers up.

time="2022-01-23T04:39:28Z" level=info msg="Starting image update cycle, considering 1 annotated application(s) for update"
time="2022-01-23T04:39:28Z" level=debug msg="Processing application connme"
time="2022-01-23T04:39:28Z" level=debug msg="Considering this image for update" alias= application=connme image_name=connme.ru-app image_tag=v1.1.14 registry="registry.conn-me.ru:5000"
time="2022-01-23T04:39:28Z" level=debug msg="Using version constraint '1.x.x' when looking for a new tag" alias= application=connme image_name=connme.ru-app image_tag=v1.1.14 registry="registry.conn-me.ru:5000"
time="2022-01-23T04:39:28Z" level=debug msg="could not parse input tag latest as semver: Invalid Semantic Version"
time="2022-01-23T04:39:28Z" level=debug msg="found 9 from 9 tags eligible for consideration" image="registry.conn-me.ru:5000/connme.ru-app:v1.1.14"
time="2022-01-23T04:39:28Z" level=info msg="Setting new image to registry.conn-me.ru:5000/connme.ru-app:v1.1.15" alias= application=connme image_name=connme.ru-app image_tag=v1.1.14 registry="registry.conn-me.ru:5000"
time="2022-01-23T04:39:28Z" level=info msg="Successfully updated image 'registry.conn-me.ru:5000/connme.ru-app:v1.1.14' to 'registry.conn-me.ru:5000/connme.ru-app:v1.1.15', but pending spec update (dry run=false)" alias= application=connme image_name=connme.ru-app image_tag=v1.1.14 registry="registry.conn-me.ru:5000"
time="2022-01-23T04:39:28Z" level=debug msg="Considering this image for update" alias= application=connme image_name=connme.ru-images image_tag=v1.1.14 registry="registry.conn-me.ru:5000"
time="2022-01-23T04:39:28Z" level=debug msg="Using version constraint '1.x.x' when looking for a new tag" alias= application=connme image_name=connme.ru-images image_tag=v1.1.14 registry="registry.conn-me.ru:5000"
time="2022-01-23T04:39:28Z" level=debug msg="found 7 from 7 tags eligible for consideration" image="registry.conn-me.ru:5000/connme.ru-images:v1.1.14"
time="2022-01-23T04:39:28Z" level=info msg="Setting new image to registry.conn-me.ru:5000/connme.ru-images:v1.1.15" alias= application=connme image_name=connme.ru-images image_tag=v1.1.14 registry="registry.conn-me.ru:5000"
time="2022-01-23T04:39:28Z" level=info msg="Successfully updated image 'registry.conn-me.ru:5000/connme.ru-images:v1.1.14' to 'registry.conn-me.ru:5000/connme.ru-images:v1.1.15', but pending spec update (dry run=false)" alias= application=connme image_name=connme.ru-images image_tag=v1.1.14 registry="registry.conn-me.ru:5000"
time="2022-01-23T04:39:28Z" level=debug msg="Considering this image for update" alias= application=connme image_name=connme.ru-cron image_tag=v1.1.14 registry="registry.conn-me.ru:5000"
time="2022-01-23T04:39:28Z" level=debug msg="Using version constraint '1.x.x' when looking for a new tag" alias= application=connme image_name=connme.ru-cron image_tag=v1.1.14 registry="registry.conn-me.ru:5000"
time="2022-01-23T04:39:28Z" level=debug msg="found 7 from 7 tags eligible for consideration" image="registry.conn-me.ru:5000/connme.ru-cron:v1.1.14"
time="2022-01-23T04:39:28Z" level=info msg="Setting new image to registry.conn-me.ru:5000/connme.ru-cron:v1.1.15" alias= application=connme image_name=connme.ru-cron image_tag=v1.1.14 registry="registry.conn-me.ru:5000"
time="2022-01-23T04:39:28Z" level=info msg="Successfully updated image 'registry.conn-me.ru:5000/connme.ru-cron:v1.1.14' to 'registry.conn-me.ru:5000/connme.ru-cron:v1.1.15', but pending spec update (dry run=false)" alias= application=connme image_name=connme.ru-cron image_tag=v1.1.14 registry="registry.conn-me.ru:5000"
time="2022-01-23T04:39:28Z" level=debug msg="Considering this image for update" alias= application=connme image_name=connme.ru-chat image_tag=v1.1.15 registry="registry.conn-me.ru:5000"
time="2022-01-23T04:39:28Z" level=debug msg="Using version constraint '1.x.x' when looking for a new tag" alias= application=connme image_name=connme.ru-chat image_tag=v1.1.15 registry="registry.conn-me.ru:5000"
time="2022-01-23T04:39:28Z" level=debug msg="found 7 from 7 tags eligible for consideration" image="registry.conn-me.ru:5000/connme.ru-chat:v1.1.15"
time="2022-01-23T04:39:28Z" level=debug msg="Image 'registry.conn-me.ru:5000/connme.ru-chat:v1.1.15' already on latest allowed version" alias= application=connme image_name=connme.ru-chat image_tag=v1.1.15 registry="registry.conn-me.ru:5000"
time="2022-01-23T04:39:28Z" level=debug msg="Using commit message: build: automatic update of connme\n\nupdates image connme.ru-app tag 'v1.1.14' to 'v1.1.15'\nupdates image connme.ru-images tag 'v1.1.14' to 'v1.1.15'\nupdates image connme.ru-cron tag 'v1.1.14' to 'v1.1.15'\n"
time="2022-01-23T04:39:28Z" level=info msg="Committing 3 parameter update(s) for application connme" application=connme
time="2022-01-23T04:39:28Z" level=info msg="Initializing https://github.com/IvanKuchin/k8s-argocd to /tmp/git-connme197574263"
time="2022-01-23T04:39:28Z" level=info msg="rm -rf /tmp/git-connme197574263" dir= execID=MmVwm
time="2022-01-23T04:39:28Z" level=info msg=Trace args="[rm -rf /tmp/git-connme197574263]" dir= operation_name="exec rm" time_ms=1.000023
time="2022-01-23T04:39:28Z" level=info msg="git fetch origin --tags --force" dir=/tmp/git-connme197574263 execID=JIXnS
time="2022-01-23T04:39:29Z" level=info msg=Trace args="[git fetch origin --tags --force]" dir=/tmp/git-connme197574263 operation_name="exec git" time_ms=382.017535
time="2022-01-23T04:39:29Z" level=info msg="git config user.name argocd-image-updater" dir=/tmp/git-connme197574263 execID=IetJg
time="2022-01-23T04:39:29Z" level=info msg=Trace args="[git config user.name argocd-image-updater]" dir=/tmp/git-connme197574263 operation_name="exec git" time_ms=1.8038159999999999
time="2022-01-23T04:39:29Z" level=info msg="git config user.email [email protected]" dir=/tmp/git-connme197574263 execID=YpTWW
time="2022-01-23T04:39:29Z" level=info msg=Trace args="[git config user.email [email protected]]" dir=/tmp/git-connme197574263 operation_name="exec git" time_ms=2.748527
time="2022-01-23T04:39:29Z" level=info msg="git checkout --force master" dir=/tmp/git-connme197574263 execID=XPEw3
time="2022-01-23T04:39:29Z" level=info msg=Trace args="[git checkout --force master]" dir=/tmp/git-connme197574263 operation_name="exec git" time_ms=7.314295
time="2022-01-23T04:39:29Z" level=info msg="git clean -fdx" dir=/tmp/git-connme197574263 execID=ekU7i
time="2022-01-23T04:39:29Z" level=debug msg="target parameter file and marshaled data are the same, skipping commit."
time="2022-01-23T04:39:29Z" level=info msg="Successfully updated the live application spec" application=connme
time="2022-01-23T04:39:29Z" level=info msg=Trace args="[git clean -fdx]" dir=/tmp/git-connme197574263 operation_name="exec git" time_ms=3.2311319999999997
time="2022-01-23T04:39:29Z" level=info msg="Processing results: applications=1 images_considered=4 images_skipped=0 images_updated=3 errors=0"

[DucretJe]

I nearly filed the same post. I have the same issue on helm based apps.

[jannfis]

Thanks. I was thinking we had a similar issue reported (and fixed) recently. So either this is a regression, or my memory fails on me. Let me cross check.

[IvanKuchin]

Thank you for quick response.

Just out of curiosity, I've tested same setup with write-back-target: kustomization, result still the same (https://github.com/IvanKuchin/k8s-argocd/commit/e8ad025f7e1094b92577ce27d72fc62395d428b9)

[tobiaszuercher]

any update on this? we are facing the same issue. are there any workarounds?

[RolandRitt]

+1

[stephaneetje]

+1

[naviat]

+1

[thinkbigdatanalytics]

Hi,

Any updates on this issue, I am using the argo workflow template which has multiple images, the argo image updater is not able to update multiple images in the kustomize template. Please let me know if there is any workaround. Thanks

[Assaf-konnecto]

+1

[valves-ops]

+1

[peturgq]

I'm having this exact issue now in 0.12.2

[robbmue]

+1

[s-ledyakhov]

This is not a bug, you just specified the list without adding aliases. Here is an example of how it would look right (and work)

  annotations:
    argocd-image-updater.argoproj.io/image-list: app=registry.conn-me.ru:5000/connme.ru-app:1.x.x, images=registry.conn-me.ru:5000/connme.ru-images:1.x.x, cron=registry.conn-me.ru:5000/connme.ru-cron:1.x.x, chat=registry.conn-me.ru:5000/connme.ru-chat:1.x.x
    argocd-image-updater.argoproj.io/app.helm.image-name: deployment.app.image.repository
    argocd-image-updater.argoproj.io/app.helm.image-tag: deployment.app.image.tag
    argocd-image-updater.argoproj.io/app.update-strategy: latest
    argocd-image-updater.argoproj.io/images.helm.image-name: deployment.images.image.repository
    argocd-image-updater.argoproj.io/images.helm.image-tag: deployment.images.image.tag
    ...

It's not entirely obvious, but if you look at the examples and read the documentation more carefully, I did it. https://argocd-image-updater.readthedocs.io/en/latest/basics/update-strategies/